> Training Environment: | > Backend: Torch | > Mixed precision: True | > Precision: fp16 | > Current device: 0 | > Num. of GPUs: 1 | > Num. of CPUs: 12 | > Num. of Torch Threads: 6 | > Torch seed: 54321 | > Torch CUDNN: True | > Torch CUDNN deterministic: False | > Torch CUDNN benchmark: False | > Torch TF32 MatMul: False > Start Tensorboard: tensorboard --logdir=voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a > Restoring from model_file.pth ... > Restoring Model... > Partial model initialization... | > Layer missing in the model definition: posterior_encoder.enc.in_layers.0.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.0.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.1.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.1.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.2.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.2.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.3.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.3.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.4.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.4.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.5.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.5.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.6.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.6.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.7.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.7.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.8.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.8.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.9.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.9.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.10.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.10.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.11.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.11.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.12.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.12.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.13.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.13.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.14.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.14.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.15.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.15.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.4.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.4.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.5.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.5.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.6.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.6.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.7.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.7.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.8.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.8.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.9.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.9.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.10.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.10.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.11.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.11.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.12.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.12.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.13.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.13.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.14.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.14.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.15.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.15.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: waveform_decoder.ups.0.weight_g | > Layer missing in the model definition: waveform_decoder.ups.0.weight_v | > Layer missing in the model definition: waveform_decoder.ups.1.weight_g | > Layer missing in the model definition: waveform_decoder.ups.1.weight_v | > Layer missing in the model definition: waveform_decoder.ups.2.weight_g | > Layer missing in the model definition: waveform_decoder.ups.2.weight_v | > Layer missing in the model definition: waveform_decoder.ups.3.weight_g | > Layer missing in the model definition: waveform_decoder.ups.3.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.2.weight_v | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.6.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.6.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.7.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.7.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.8.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.8.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.9.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.9.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.10.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.10.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.11.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.11.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.12.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.12.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.13.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.13.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.14.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.14.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.15.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.15.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.6.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.6.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.7.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.7.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.8.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.8.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.9.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.9.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.10.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.10.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.11.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.11.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.12.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.12.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.13.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.13.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.14.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.14.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.15.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.15.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.0.bias | > Layer missing in the checkpoint: disc.nets.0.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.1.bias | > Layer missing in the checkpoint: disc.nets.0.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.2.bias | > Layer missing in the checkpoint: disc.nets.0.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.3.bias | > Layer missing in the checkpoint: disc.nets.0.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.4.bias | > Layer missing in the checkpoint: disc.nets.0.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.5.bias | > Layer missing in the checkpoint: disc.nets.0.convs.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.conv_post.bias | > Layer missing in the checkpoint: disc.nets.0.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.0.bias | > Layer missing in the checkpoint: disc.nets.1.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.1.bias | > Layer missing in the checkpoint: disc.nets.1.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.2.bias | > Layer missing in the checkpoint: disc.nets.1.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.3.bias | > Layer missing in the checkpoint: disc.nets.1.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.4.bias | > Layer missing in the checkpoint: disc.nets.1.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.conv_post.bias | > Layer missing in the checkpoint: disc.nets.1.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.0.bias | > Layer missing in the checkpoint: disc.nets.2.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.1.bias | > Layer missing in the checkpoint: disc.nets.2.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.2.bias | > Layer missing in the checkpoint: disc.nets.2.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.3.bias | > Layer missing in the checkpoint: disc.nets.2.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.4.bias | > Layer missing in the checkpoint: disc.nets.2.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.conv_post.bias | > Layer missing in the checkpoint: disc.nets.2.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.0.bias | > Layer missing in the checkpoint: disc.nets.3.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.1.bias | > Layer missing in the checkpoint: disc.nets.3.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.2.bias | > Layer missing in the checkpoint: disc.nets.3.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.3.bias | > Layer missing in the checkpoint: disc.nets.3.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.4.bias | > Layer missing in the checkpoint: disc.nets.3.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.conv_post.bias | > Layer missing in the checkpoint: disc.nets.3.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.0.bias | > Layer missing in the checkpoint: disc.nets.4.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.1.bias | > Layer missing in the checkpoint: disc.nets.4.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.2.bias | > Layer missing in the checkpoint: disc.nets.4.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.3.bias | > Layer missing in the checkpoint: disc.nets.4.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.4.bias | > Layer missing in the checkpoint: disc.nets.4.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.conv_post.bias | > Layer missing in the checkpoint: disc.nets.4.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.0.bias | > Layer missing in the checkpoint: disc.nets.5.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.1.bias | > Layer missing in the checkpoint: disc.nets.5.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.2.bias | > Layer missing in the checkpoint: disc.nets.5.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.3.bias | > Layer missing in the checkpoint: disc.nets.5.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.4.bias | > Layer missing in the checkpoint: disc.nets.5.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.conv_post.bias | > Layer missing in the checkpoint: disc.nets.5.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.conv_post.parametrizations.weight.original1 | > Layer dimention missmatch between model definition and checkpoint: text_encoder.emb.weight | > 557 / 949 layers are restored. > Model restored from step 1000000 > Model has 83047468 parameters  > EPOCH: 0/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:10:48)   --> TIME: 2025-02-25 08:11:16 -- STEP: 49/72 -- GLOBAL_STEP: 1000050 | > loss_disc: 2.9825031757354736 (3.2188744788267174) | > loss_disc_real_0: 0.25394102931022644 (0.29492562142561896) | > loss_disc_real_1: 0.25017398595809937 (0.2998270029012038) | > loss_disc_real_2: 0.22751484811306 (0.29802338064325107) | > loss_disc_real_3: 0.2826526463031769 (0.3007126067365919) | > loss_disc_real_4: 0.26421108841896057 (0.30721953146311703) | > loss_disc_real_5: 0.23740074038505554 (0.30391768442124734) | > loss_0: 2.9825031757354736 (3.2188744788267174) | > grad_norm_0: tensor(0.9411, device='cuda:0') (tensor(1.4486, device='cuda:0')) | > loss_gen: 1.522012710571289 (1.7242362146474877) | > loss_kl: 3.9697844982147217 (6.595462774743839) | > loss_feat: 0.35030442476272583 (0.3189420134437327) | > loss_mel: 21.26411247253418 (23.159969485535914) | > loss_duration: 1.8520582914352417 (1.9332660023047) | > amp_scaler: 256.0 (882.9387755102041) | > loss_1: 28.95827293395996 (33.73187649006747) | > grad_norm_1: tensor(141.9976, device='cuda:0') (tensor(158.1107, device='cuda:0')) | > current_lr_0: 0.0002 | > current_lr_1: 0.0002 | > step_time: 0.469 (0.4867572833080681) | > loader_time: 0.0096 (0.008112372184286312) --> EVAL PERFORMANCE | > avg_loader_time: 0.00901026121327575 (+0) | > avg_step_time: 0.4985209955296046 (+0) | > avg_loss_disc: 3.1403814436684194 (+0) | > avg_loss_disc_real_0: 0.2810367999572149 (+0) | > avg_loss_disc_real_1: 0.2837229065701995 (+0) | > avg_loss_disc_real_2: 0.28172466316273514 (+0) | > avg_loss_disc_real_3: 0.28414903129910085 (+0) | > avg_loss_disc_real_4: 0.2883562760453829 (+0) | > avg_loss_disc_real_5: 0.2869413927407332 (+0) | > avg_loss_0: 3.1403814436684194 (+0) | > avg_grad_norm_0: tensor(1.7452, device='cuda:0') (+0) | > avg_loss_gen: 1.670924420927612 (+0) | > avg_loss_kl: 5.7890785781430525 (+0) | > avg_loss_feat: 0.3709548791529427 (+0) | > avg_loss_mel: 22.693415440304175 (+0) | > avg_loss_duration: 1.8996801141282202 (+0) | > avg_amp_scaler: 688.6760563380282 (+0) | > avg_loss_1: 32.42405346077934 (+0) | > avg_grad_norm_1: tensor(175.1228, device='cuda:0') (+0) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000073.pth  > EPOCH: 1/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:11:32)   --> TIME: 2025-02-25 08:11:46 -- STEP: 27/72 -- GLOBAL_STEP: 1000100 | > loss_disc: 2.925985813140869 (2.9307760927412243) | > loss_disc_real_0: 0.2283693552017212 (0.24704700266873394) | > loss_disc_real_1: 0.2995283603668213 (0.21307367231282923) | > loss_disc_real_2: 0.26690712571144104 (0.24293037503957748) | > loss_disc_real_3: 0.16060686111450195 (0.24704200884810198) | > loss_disc_real_4: 0.26388227939605713 (0.2572009099854364) | > loss_disc_real_5: 0.2014242261648178 (0.24466190459551634) | > loss_0: 2.925985813140869 (2.9307760927412243) | > grad_norm_0: tensor(7.6913, device='cuda:0') (tensor(11.0084, device='cuda:0')) | > loss_gen: 1.581876277923584 (1.622780817526358) | > loss_kl: 3.5429091453552246 (3.8475298086802163) | > loss_feat: 0.789517343044281 (0.8173273007074991) | > loss_mel: 23.02496910095215 (21.421504621152526) | > loss_duration: 1.8326542377471924 (1.836715203744394) | > amp_scaler: 256.0 (256.0) | > loss_1: 30.771926879882812 (29.545857747395832) | > grad_norm_1: tensor(108.3841, device='cuda:0') (tensor(147.1433, device='cuda:0')) | > current_lr_0: 0.000199975 | > current_lr_1: 0.000199975 | > step_time: 0.4305 (0.42321381745515047) | > loader_time: 0.0081 (0.006800201204087999) --> EVAL PERFORMANCE | > avg_loader_time: 0.008423361979739766 (-0.0005868992335359841) | > avg_step_time: 0.44578703356460786 (-0.052733961964996745) | > avg_loss_disc: 2.935559648863027 (-0.20482179480539253) | > avg_loss_disc_real_0: 0.24788679539317815 (-0.03315000456403677) | > avg_loss_disc_real_1: 0.22574999469371748 (-0.057972911876482014) | > avg_loss_disc_real_2: 0.24085514981981734 (-0.0408695133429178) | > avg_loss_disc_real_3: 0.24977426472264277 (-0.03437476657645808) | > avg_loss_disc_real_4: 0.24848778566844026 (-0.03986849037694262) | > avg_loss_disc_real_5: 0.25090073154006204 (-0.036040661200671176) | > avg_loss_0: 2.935559648863027 (-0.20482179480539253) | > avg_grad_norm_0: tensor(11.2242, device='cuda:0') (+tensor(9.4790, device='cuda:0')) | > avg_loss_gen: 1.6193270632918453 (-0.05159735763576667) | > avg_loss_kl: 3.6470820635137424 (-2.14199651462931) | > avg_loss_feat: 0.8079535020908839 (+0.43699862293794123) | > avg_loss_mel: 21.401046457424968 (-1.292368982879207) | > avg_loss_duration: 1.8941579402332576 (-0.005522173894962679) | > avg_amp_scaler: 256.0 (-432.67605633802816) | > avg_loss_1: 29.369566930851466 (-3.0544865299278747) | > avg_grad_norm_1: tensor(173.4342, device='cuda:0') (tensor(-1.6886, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000145.pth  > EPOCH: 2/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:12:11)   --> TIME: 2025-02-25 08:12:15 -- STEP: 5/72 -- GLOBAL_STEP: 1000150 | > loss_disc: 2.957272529602051 (2.9675930023193358) | > loss_disc_real_0: 0.2359703928232193 (0.245746248960495) | > loss_disc_real_1: 0.2659362554550171 (0.23980973660945892) | > loss_disc_real_2: 0.2497364580631256 (0.2524750679731369) | > loss_disc_real_3: 0.24600060284137726 (0.242550528049469) | > loss_disc_real_4: 0.37197819352149963 (0.2659307271242142) | > loss_disc_real_5: 0.2499941885471344 (0.25404737889766693) | > loss_0: 2.957272529602051 (2.9675930023193358) | > grad_norm_0: tensor(4.4228, device='cuda:0') (tensor(6.1251, device='cuda:0')) | > loss_gen: 1.4387426376342773 (1.5379149913787842) | > loss_kl: 3.8326852321624756 (3.602376699447632) | > loss_feat: 0.7235944867134094 (0.7265236973762512) | > loss_mel: 22.401273727416992 (21.876456451416015) | > loss_duration: 1.9294607639312744 (1.9299361944198608) | > amp_scaler: 256.0 (256.0) | > loss_1: 30.32575798034668 (29.67320785522461) | > grad_norm_1: tensor(374.4723, device='cuda:0') (tensor(278.3387, device='cuda:0')) | > current_lr_0: 0.000199950003125 | > current_lr_1: 0.000199950003125 | > step_time: 0.4118 (0.4143852233886719) | > loader_time: 0.0059 (0.007408428192138672)  --> TIME: 2025-02-25 08:12:38 -- STEP: 55/72 -- GLOBAL_STEP: 1000200 | > loss_disc: 2.9496872425079346 (2.9714270895177664) | > loss_disc_real_0: 0.23269852995872498 (0.247294771400365) | > loss_disc_real_1: 0.2480759471654892 (0.24936744652011178) | > loss_disc_real_2: 0.25224682688713074 (0.25010719326409425) | > loss_disc_real_3: 0.2053726762533188 (0.24796133908358486) | > loss_disc_real_4: 0.2475266456604004 (0.24789060977372257) | > loss_disc_real_5: 0.21277084946632385 (0.2479921517047015) | > loss_0: 2.9496872425079346 (2.9714270895177664) | > grad_norm_0: tensor(2.1139, device='cuda:0') (tensor(6.5297, device='cuda:0')) | > loss_gen: 1.3628805875778198 (1.5691591999747536) | > loss_kl: 3.1818618774414062 (3.311189543117176) | > loss_feat: 0.6071171164512634 (0.693370153687217) | > loss_mel: 22.24699592590332 (21.16485925154253) | > loss_duration: 1.9517102241516113 (1.920708120952953) | > amp_scaler: 256.0 (256.0) | > loss_1: 29.35056495666504 (28.65928622159091) | > grad_norm_1: tensor(62.5079, device='cuda:0') (tensor(205.5087, device='cuda:0')) | > current_lr_0: 0.000199950003125 | > current_lr_1: 0.000199950003125 | > step_time: 0.4633 (0.4420865665782582) | > loader_time: 0.0092 (0.007749860936945135) --> EVAL PERFORMANCE | > avg_loader_time: 0.008473221684845397 (+4.985970510563119e-05) | > avg_step_time: 0.45470458353069465 (+0.00891754996608679) | > avg_loss_disc: 2.9702808689063698 (+0.03472122004334288) | > avg_loss_disc_real_0: 0.24721921854455706 (-0.0006675768486210931) | > avg_loss_disc_real_1: 0.2482473801978877 (+0.022497385504170214) | > avg_loss_disc_real_2: 0.250255586395801 (+0.009400436575983634) | > avg_loss_disc_real_3: 0.24880759250110304 (-0.0009666722215397328) | > avg_loss_disc_real_4: 0.24763951419104993 (-0.0008482714773903333) | > avg_loss_disc_real_5: 0.24635876022594075 (-0.00454197131412129) | > avg_loss_0: 2.9702808689063698 (+0.03472122004334288) | > avg_grad_norm_0: tensor(6.5843, device='cuda:0') (tensor(-4.6399, device='cuda:0')) | > avg_loss_gen: 1.5697072962640037 (-0.049619767027841544) | > avg_loss_kl: 3.2820434234511677 (-0.3650386400625747) | > avg_loss_feat: 0.6836673575387874 (-0.12428614455209652) | > avg_loss_mel: 21.23391457678567 (-0.16713188063929962) | > avg_loss_duration: 1.9250469560354528 (+0.030889015802195274) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 28.69437956474197 (-0.6751873661094976) | > avg_grad_norm_1: tensor(225.2051, device='cuda:0') (+tensor(51.7709, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000217.pth  > EPOCH: 3/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:12:50)   --> TIME: 2025-02-25 08:13:06 -- STEP: 33/72 -- GLOBAL_STEP: 1000250 | > loss_disc: 2.995230197906494 (2.9239247928966177) | > loss_disc_real_0: 0.2960442304611206 (0.25134173848412256) | > loss_disc_real_1: 0.39282822608947754 (0.24443596962726477) | > loss_disc_real_2: 0.20393028855323792 (0.24824705268397476) | > loss_disc_real_3: 0.2923261821269989 (0.24795765813552972) | > loss_disc_real_4: 0.3965698480606079 (0.2557710047924157) | > loss_disc_real_5: 0.28516554832458496 (0.23493806837183057) | > loss_0: 2.995230197906494 (2.9239247928966177) | > grad_norm_0: tensor(15.1858, device='cuda:0') (tensor(5.6415, device='cuda:0')) | > loss_gen: 1.662196159362793 (1.605379758459149) | > loss_kl: 2.84491229057312 (3.160514484752308) | > loss_feat: 0.8706418871879578 (0.7842760989160249) | > loss_mel: 19.98088264465332 (21.18256158539743) | > loss_duration: 1.899662733078003 (1.9039002837556782) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.258296966552734 (28.63663239912553) | > grad_norm_1: tensor(79.2487, device='cuda:0') (tensor(196.5991, device='cuda:0')) | > current_lr_0: 0.00019992500937460937 | > current_lr_1: 0.00019992500937460937 | > step_time: 0.4304 (0.42303753621650464) | > loader_time: 0.0078 (0.006899826454393791) --> EVAL PERFORMANCE | > avg_loader_time: 0.008448533608879842 (-2.4688075965555167e-05) | > avg_step_time: 0.4434272369868319 (-0.011277346543862754) | > avg_loss_disc: 2.9282614110221323 (-0.04201945788423744) | > avg_loss_disc_real_0: 0.24977532385940282 (+0.0025561053148457635) | > avg_loss_disc_real_1: 0.24855356065320297 (+0.0003061804553152747) | > avg_loss_disc_real_2: 0.2500136068589251 (-0.00024197953687588036) | > avg_loss_disc_real_3: 0.2461773285143812 (-0.002630263986721837) | > avg_loss_disc_real_4: 0.24869454389726606 (+0.0010550297062161262) | > avg_loss_disc_real_5: 0.23802789458086793 (-0.008330865645072821) | > avg_loss_0: 2.9282614110221323 (-0.04201945788423744) | > avg_grad_norm_0: tensor(5.4481, device='cuda:0') (tensor(-1.1362, device='cuda:0')) | > avg_loss_gen: 1.6043520024125004 (+0.034644706148496685) | > avg_loss_kl: 3.072064967222617 (-0.20997845622855094) | > avg_loss_feat: 0.7786891426838619 (+0.09502178514507453) | > avg_loss_mel: 20.817215127004705 (-0.41669944978096396) | > avg_loss_duration: 1.9170460768148934 (-0.008000879220559476) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 28.189367348039653 (-0.5050122167023154) | > avg_grad_norm_1: tensor(183.3010, device='cuda:0') (tensor(-41.9041, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000289.pth  > EPOCH: 4/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:13:29)   --> TIME: 2025-02-25 08:13:36 -- STEP: 11/72 -- GLOBAL_STEP: 1000300 | > loss_disc: 2.879227638244629 (2.902175881645896) | > loss_disc_real_0: 0.23267687857151031 (0.24812510203231464) | > loss_disc_real_1: 0.2110191434621811 (0.24586922065778213) | > loss_disc_real_2: 0.2280118316411972 (0.24365519799969412) | > loss_disc_real_3: 0.25330987572669983 (0.23963796279647134) | > loss_disc_real_4: 0.2743997275829315 (0.24286146461963654) | > loss_disc_real_5: 0.19379922747612 (0.2390293600884351) | > loss_0: 2.879227638244629 (2.902175881645896) | > grad_norm_0: tensor(3.5757, device='cuda:0') (tensor(2.4715, device='cuda:0')) | > loss_gen: 1.762873649597168 (1.6156580664894797) | > loss_kl: 3.112015724182129 (3.026738491925326) | > loss_feat: 0.9788455963134766 (0.9223772395740856) | > loss_mel: 21.10115623474121 (20.705883026123047) | > loss_duration: 1.8683559894561768 (1.9002541953867131) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.8232479095459 (28.170911268754438) | > grad_norm_1: tensor(231.1476, device='cuda:0') (tensor(144.6830, device='cuda:0')) | > current_lr_0: 0.00019990001874843754 | > current_lr_1: 0.00019990001874843754 | > step_time: 0.412 (0.41687536239624023) | > loader_time: 0.0055 (0.006056547164916992)  --> TIME: 2025-02-25 08:13:59 -- STEP: 61/72 -- GLOBAL_STEP: 1000350 | > loss_disc: 2.9949309825897217 (2.9275278732424876) | > loss_disc_real_0: 0.19715148210525513 (0.24802992334131335) | > loss_disc_real_1: 0.1990196853876114 (0.24823996302534324) | > loss_disc_real_2: 0.18832583725452423 (0.24527351729205396) | > loss_disc_real_3: 0.26448971033096313 (0.24341604748710258) | > loss_disc_real_4: 0.2670530080795288 (0.24608897600994736) | > loss_disc_real_5: 0.21023260056972504 (0.2456031514972937) | > loss_0: 2.9949309825897217 (2.9275278732424876) | > grad_norm_0: tensor(7.2462, device='cuda:0') (tensor(3.8344, device='cuda:0')) | > loss_gen: 1.538955569267273 (1.6051671661314417) | > loss_kl: 2.8710379600524902 (2.917222882880539) | > loss_feat: 0.8065652847290039 (0.8815001894216068) | > loss_mel: 20.197891235351562 (20.596212887373124) | > loss_duration: 1.8930835723876953 (1.8882950133964664) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.307533264160156 (27.88839834244525) | > grad_norm_1: tensor(309.7876, device='cuda:0') (tensor(183.4985, device='cuda:0')) | > current_lr_0: 0.00019990001874843754 | > current_lr_1: 0.00019990001874843754 | > step_time: 0.4916 (0.4392431055913206) | > loader_time: 0.0108 (0.00774496500609351) --> EVAL PERFORMANCE | > avg_loader_time: 0.008269555132154008 (-0.00017897847672583393) | > avg_step_time: 0.45216000248009053 (+0.008732765493258632) | > avg_loss_disc: 2.9316021925966504 (+0.0033407815745181146) | > avg_loss_disc_real_0: 0.2479266923078349 (-0.0018486315515679197) | > avg_loss_disc_real_1: 0.24831844971213543 (-0.00023511094106754005) | > avg_loss_disc_real_2: 0.24607928642924404 (-0.00393432042968106) | > avg_loss_disc_real_3: 0.24283116846017436 (-0.003346160054206848) | > avg_loss_disc_real_4: 0.24561430723734304 (-0.003080236659923019) | > avg_loss_disc_real_5: 0.24668525872935712 (+0.008657364148489194) | > avg_loss_0: 2.9316021925966504 (+0.0033407815745181146) | > avg_grad_norm_0: tensor(3.7589, device='cuda:0') (tensor(-1.6892, device='cuda:0')) | > avg_loss_gen: 1.6033664908207639 (-0.0009855115917365431) | > avg_loss_kl: 2.912233651523859 (-0.15983131569875786) | > avg_loss_feat: 0.8629949571381154 (+0.08430581445425345) | > avg_loss_mel: 20.59236462015501 (-0.22485050684969465) | > avg_loss_duration: 1.8902590644191688 (-0.026787012395724563) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.861218949438822 (-0.3281483986008311) | > avg_grad_norm_1: tensor(185.9581, device='cuda:0') (+tensor(2.6571, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000361.pth  > EPOCH: 5/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:14:09)   --> TIME: 2025-02-25 08:14:28 -- STEP: 39/72 -- GLOBAL_STEP: 1000400 | > loss_disc: 2.90788197517395 (2.928270584497696) | > loss_disc_real_0: 0.24960829317569733 (0.24170295550273016) | > loss_disc_real_1: 0.2572973668575287 (0.2436230717561184) | > loss_disc_real_2: 0.2461818903684616 (0.24636340523377442) | > loss_disc_real_3: 0.22269019484519958 (0.240693449591979) | > loss_disc_real_4: 0.2480708658695221 (0.2436519910891851) | > loss_disc_real_5: 0.2448655068874359 (0.24569719533125559) | > loss_0: 2.90788197517395 (2.928270584497696) | > grad_norm_0: tensor(1.9227, device='cuda:0') (tensor(5.0588, device='cuda:0')) | > loss_gen: 1.5842304229736328 (1.5921040100929065) | > loss_kl: 2.9740002155303955 (2.9450195263593617) | > loss_feat: 0.935753345489502 (0.8882377468622648) | > loss_mel: 20.40302276611328 (20.475824698423732) | > loss_duration: 1.852944254875183 (1.858114936412909) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.749950408935547 (27.759301063341972) | > grad_norm_1: tensor(208.6264, device='cuda:0') (tensor(185.2366, device='cuda:0')) | > current_lr_0: 0.00019987503124609398 | > current_lr_1: 0.00019987503124609398 | > step_time: 0.4458 (0.42758582799862593) | > loader_time: 0.0079 (0.007141602345002003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008404939946993975 (+0.00013538481483996688) | > avg_step_time: 0.4467850369466862 (-0.00537496553340433) | > avg_loss_disc: 2.9441230800789846 (+0.012520887482334153) | > avg_loss_disc_real_0: 0.2441791142796127 (-0.0037475780282222038) | > avg_loss_disc_real_1: 0.25010623940279786 (+0.001787789690662428) | > avg_loss_disc_real_2: 0.24738614450038318 (+0.0013068580711391442) | > avg_loss_disc_real_3: 0.24494413569779463 (+0.002112967237620278) | > avg_loss_disc_real_4: 0.2478619782437741 (+0.0022476710064310568) | > avg_loss_disc_real_5: 0.24742765535771008 (+0.0007423966283529593) | > avg_loss_0: 2.9441230800789846 (+0.012520887482334153) | > avg_grad_norm_0: tensor(5.3387, device='cuda:0') (+tensor(1.5798, device='cuda:0')) | > avg_loss_gen: 1.6001049867818053 (-0.0032615040389585293) | > avg_loss_kl: 2.9171111281488975 (+0.004877476625038568) | > avg_loss_feat: 0.8564649257861393 (-0.006530031351976051) | > avg_loss_mel: 20.468097767359772 (-0.12426685279523753) | > avg_loss_duration: 1.8697114192264181 (-0.020547645192750652) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.711490335598796 (-0.14972861384002556) | > avg_grad_norm_1: tensor(207.6699, device='cuda:0') (+tensor(21.7118, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000433.pth  > EPOCH: 6/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:14:48)   --> TIME: 2025-02-25 08:14:57 -- STEP: 17/72 -- GLOBAL_STEP: 1000450 | > loss_disc: 2.9398820400238037 (2.9305135642780975) | > loss_disc_real_0: 0.29226112365722656 (0.24547380121315227) | > loss_disc_real_1: 0.2757374346256256 (0.24457197592538946) | > loss_disc_real_2: 0.2659173607826233 (0.24676142895922942) | > loss_disc_real_3: 0.27626511454582214 (0.24606756427708795) | > loss_disc_real_4: 0.2918429672718048 (0.24552355005460627) | > loss_disc_real_5: 0.2863673269748688 (0.24438966021818273) | > loss_0: 2.9398820400238037 (2.9305135642780975) | > grad_norm_0: tensor(4.2697, device='cuda:0') (tensor(3.5832, device='cuda:0')) | > loss_gen: 1.5584704875946045 (1.579591288286097) | > loss_kl: 3.1111977100372314 (2.8222697061650894) | > loss_feat: 0.6928841471672058 (0.834976918557111) | > loss_mel: 20.777359008789062 (20.34934358035817) | > loss_duration: 1.8818970918655396 (1.8357375018736894) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.021806716918945 (27.421918981215534) | > grad_norm_1: tensor(239.1420, device='cuda:0') (tensor(195.1130, device='cuda:0')) | > current_lr_0: 0.0001998500468671882 | > current_lr_1: 0.0001998500468671882 | > step_time: 0.4185 (0.4211083720712101) | > loader_time: 0.0061 (0.006507046082440545)  --> TIME: 2025-02-25 08:15:21 -- STEP: 67/72 -- GLOBAL_STEP: 1000500 | > loss_disc: 2.966071844100952 (2.9377699182994337) | > loss_disc_real_0: 0.19781923294067383 (0.2453056179098229) | > loss_disc_real_1: 0.20561492443084717 (0.24635118913294665) | > loss_disc_real_2: 0.24311192333698273 (0.24661420130017978) | > loss_disc_real_3: 0.24655815958976746 (0.24587138041631498) | > loss_disc_real_4: 0.21554210782051086 (0.24345664613282503) | > loss_disc_real_5: 0.23191604018211365 (0.24542015501812323) | > loss_0: 2.966071844100952 (2.9377699182994337) | > grad_norm_0: tensor(4.2959, device='cuda:0') (tensor(5.0403, device='cuda:0')) | > loss_gen: 1.38261878490448 (1.5883239187411409) | > loss_kl: 2.8736062049865723 (2.8745713091608303) | > loss_feat: 0.6620239615440369 (0.8715708211286745) | > loss_mel: 20.178743362426758 (20.38734658084699) | > loss_duration: 1.8832614421844482 (1.8419423726067614) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.980253219604492 (27.563755063868282) | > grad_norm_1: tensor(303.9269, device='cuda:0') (tensor(205.5945, device='cuda:0')) | > current_lr_0: 0.0001998500468671882 | > current_lr_1: 0.0001998500468671882 | > step_time: 0.5439 (0.44295274321712663) | > loader_time: 0.0121 (0.00807895945079291) --> EVAL PERFORMANCE | > avg_loader_time: 0.008262546969131685 (-0.00014239297786228927) | > avg_step_time: 0.44863293876110666 (+0.0018479018144204606) | > avg_loss_disc: 2.9382684533025185 (-0.0058546267764660875) | > avg_loss_disc_real_0: 0.2451584871176263 (+0.000979372838013609) | > avg_loss_disc_real_1: 0.24566233850700755 (-0.004443900895790304) | > avg_loss_disc_real_2: 0.2461598236376131 (-0.0012263208627700806) | > avg_loss_disc_real_3: 0.24536493672451504 (+0.0004208010267204021) | > avg_loss_disc_real_4: 0.24385106605543216 (-0.004010912188341931) | > avg_loss_disc_real_5: 0.2450024556945747 (-0.002425199663135369) | > avg_loss_0: 2.9382684533025185 (-0.0058546267764660875) | > avg_grad_norm_0: tensor(5.0772, device='cuda:0') (tensor(-0.2615, device='cuda:0')) | > avg_loss_gen: 1.5883359606836884 (-0.01176902609811692) | > avg_loss_kl: 2.8649174193261375 (-0.05219370882275998) | > avg_loss_feat: 0.8726533852832418 (+0.01618845949710246) | > avg_loss_mel: 20.36292452207754 (-0.10517324528223071) | > avg_loss_duration: 1.8423974211786833 (-0.027313998047734822) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.531228804252518 (-0.18026153134627876) | > avg_grad_norm_1: tensor(208.9674, device='cuda:0') (+tensor(1.2975, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000505.pth  > EPOCH: 7/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:15:27)   --> TIME: 2025-02-25 08:15:49 -- STEP: 45/72 -- GLOBAL_STEP: 1000550 | > loss_disc: 2.9972050189971924 (2.9300031661987305) | > loss_disc_real_0: 0.23143596947193146 (0.2461431662241618) | > loss_disc_real_1: 0.22900594770908356 (0.24338801370726693) | > loss_disc_real_2: 0.2566421329975128 (0.24632740848594242) | > loss_disc_real_3: 0.27437376976013184 (0.24567623701360491) | > loss_disc_real_4: 0.33497583866119385 (0.24791279170248243) | > loss_disc_real_5: 0.33535492420196533 (0.24807819128036498) | > loss_0: 2.9972050189971924 (2.9300031661987305) | > grad_norm_0: tensor(3.2957, device='cuda:0') (tensor(5.0880, device='cuda:0')) | > loss_gen: 1.4765925407409668 (1.6007155100504558) | > loss_kl: 2.767991304397583 (2.882087177700467) | > loss_feat: 0.7573830485343933 (0.934130068620046) | > loss_mel: 19.508838653564453 (20.08430035909017) | > loss_duration: 1.851638674736023 (1.8174825562371149) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.362445831298828 (27.31871566772461) | > grad_norm_1: tensor(147.1764, device='cuda:0') (tensor(241.4984, device='cuda:0')) | > current_lr_0: 0.00019982506561132978 | > current_lr_1: 0.00019982506561132978 | > step_time: 0.4722 (0.4402787950303819) | > loader_time: 0.0085 (0.007095162073771159) --> EVAL PERFORMANCE | > avg_loader_time: 0.008433187511605277 (+0.00017064054247359177) | > avg_step_time: 0.45163247283075897 (+0.00299953406965231) | > avg_loss_disc: 2.9271783022813396 (-0.01109015102117894) | > avg_loss_disc_real_0: 0.24753031344480916 (+0.0023718263271828577) | > avg_loss_disc_real_1: 0.24352492476013346 (-0.002137413746874095) | > avg_loss_disc_real_2: 0.24574485652043787 (-0.00041496711717523005) | > avg_loss_disc_real_3: 0.24520082813753208 (-0.00016410858698295838) | > avg_loss_disc_real_4: 0.24561880780777462 (+0.0017677417523424555) | > avg_loss_disc_real_5: 0.24555425912561551 (+0.0005518034310408038) | > avg_loss_0: 2.9271783022813396 (-0.01109015102117894) | > avg_grad_norm_0: tensor(5.1919, device='cuda:0') (+tensor(0.1147, device='cuda:0')) | > avg_loss_gen: 1.6024325884563821 (+0.014096627772693715) | > avg_loss_kl: 2.830833784291442 (-0.03408363503469536) | > avg_loss_feat: 0.92838226573568 (+0.05572888045243818) | > avg_loss_mel: 20.030793525803258 (-0.3321309962742838) | > avg_loss_duration: 1.825426073141501 (-0.016971348037182254) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.217868375106598 (-0.31336042914592) | > avg_grad_norm_1: tensor(236.6461, device='cuda:0') (+tensor(27.6787, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000577.pth  > EPOCH: 8/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:16:06)   --> TIME: 2025-02-25 08:16:18 -- STEP: 23/72 -- GLOBAL_STEP: 1000600 | > loss_disc: 2.8464109897613525 (2.9138681370279063) | > loss_disc_real_0: 0.24167530238628387 (0.24325829873914304) | > loss_disc_real_1: 0.22613608837127686 (0.2467479265254477) | > loss_disc_real_2: 0.23937265574932098 (0.24493600233741428) | > loss_disc_real_3: 0.24034398794174194 (0.245130448885586) | > loss_disc_real_4: 0.27436548471450806 (0.24418555718401205) | > loss_disc_real_5: 0.2806076407432556 (0.24111954803052155) | > loss_0: 2.8464109897613525 (2.9138681370279063) | > grad_norm_0: tensor(2.6419, device='cuda:0') (tensor(5.3694, device='cuda:0')) | > loss_gen: 1.6769295930862427 (1.6183695793151855) | > loss_kl: 2.6130993366241455 (2.827208767766538) | > loss_feat: 0.9449037313461304 (0.9738812420679175) | > loss_mel: 18.87936782836914 (20.06368637084961) | > loss_duration: 1.828916072845459 (1.803760544113491) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.94321632385254 (27.286906283834707) | > grad_norm_1: tensor(212.1963, device='cuda:0') (tensor(205.3542, device='cuda:0')) | > current_lr_0: 0.00019980008747812837 | > current_lr_1: 0.00019980008747812837 | > step_time: 0.4296 (0.424426503803419) | > loader_time: 0.0068 (0.00653417214103367) --> EVAL PERFORMANCE | > avg_loader_time: 0.008400698782692492 (-3.248872891278495e-05) | > avg_step_time: 0.4488682511826636 (-0.0027642216480953663) | > avg_loss_disc: 2.9093105289298045 (-0.01786777335153511) | > avg_loss_disc_real_0: 0.24363864349647307 (-0.0038916699483360906) | > avg_loss_disc_real_1: 0.24503066317296365 (+0.0015057384128301932) | > avg_loss_disc_real_2: 0.24405031682739795 (-0.001694539693039926) | > avg_loss_disc_real_3: 0.24530617032252566 (+0.00010534218499358428) | > avg_loss_disc_real_4: 0.2442776814732753 (-0.0013411263344993152) | > avg_loss_disc_real_5: 0.24436620906205245 (-0.0011880500635630675) | > avg_loss_0: 2.9093105289298045 (-0.01786777335153511) | > avg_grad_norm_0: tensor(5.4159, device='cuda:0') (+tensor(0.2240, device='cuda:0')) | > avg_loss_gen: 1.628979313541466 (+0.026546725085083933) | > avg_loss_kl: 2.7761729435181954 (-0.05466084077324673) | > avg_loss_feat: 1.0186714588756292 (+0.09028919313994921) | > avg_loss_mel: 20.05015007878693 (+0.019356552983673225) | > avg_loss_duration: 1.8126053491108853 (-0.012820724030615738) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.286579212672272 (+0.06871083756567486) | > avg_grad_norm_1: tensor(229.4813, device='cuda:0') (tensor(-7.1648, device='cuda:0'))  > EPOCH: 9/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:16:43)   --> TIME: 2025-02-25 08:16:45 -- STEP: 1/72 -- GLOBAL_STEP: 1000650 | > loss_disc: 2.8769631385803223 (2.8769631385803223) | > loss_disc_real_0: 0.21832013130187988 (0.21832013130187988) | > loss_disc_real_1: 0.2879323959350586 (0.2879323959350586) | > loss_disc_real_2: 0.26962506771087646 (0.26962506771087646) | > loss_disc_real_3: 0.20471589267253876 (0.20471589267253876) | > loss_disc_real_4: 0.2201765775680542 (0.2201765775680542) | > loss_disc_real_5: 0.24503391981124878 (0.24503391981124878) | > loss_0: 2.8769631385803223 (2.8769631385803223) | > grad_norm_0: tensor(5.0745, device='cuda:0') (tensor(5.0745, device='cuda:0')) | > loss_gen: 1.6040257215499878 (1.6040257215499878) | > loss_kl: 2.6145920753479004 (2.6145920753479004) | > loss_feat: 1.1193921566009521 (1.1193921566009521) | > loss_mel: 20.217857360839844 (20.217857360839844) | > loss_duration: 1.8253167867660522 (1.8253167867660522) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.381183624267578 (27.381183624267578) | > grad_norm_1: tensor(452.1778, device='cuda:0') (tensor(452.1778, device='cuda:0')) | > current_lr_0: 0.0001997751124671936 | > current_lr_1: 0.0001997751124671936 | > step_time: 0.4185 (0.4185006618499756) | > loader_time: 0.0057 (0.0056955814361572266)  --> TIME: 2025-02-25 08:17:07 -- STEP: 51/72 -- GLOBAL_STEP: 1000700 | > loss_disc: 2.9296107292175293 (2.91784771750955) | > loss_disc_real_0: 0.2645227909088135 (0.25209840752330487) | > loss_disc_real_1: 0.2308952957391739 (0.24437146619254468) | > loss_disc_real_2: 0.28264302015304565 (0.2506018593615176) | > loss_disc_real_3: 0.2688547670841217 (0.24860463890374873) | > loss_disc_real_4: 0.281634122133255 (0.24531287626892911) | > loss_disc_real_5: 0.22337229549884796 (0.24581001406791164) | > loss_0: 2.9296107292175293 (2.91784771750955) | > grad_norm_0: tensor(3.5661, device='cuda:0') (tensor(7.2315, device='cuda:0')) | > loss_gen: 1.7124643325805664 (1.649853460928973) | > loss_kl: 2.73689603805542 (2.816495988883225) | > loss_feat: 0.9427319765090942 (1.0305962176883923) | > loss_mel: 19.385393142700195 (20.07251219655953) | > loss_duration: 1.8310108184814453 (1.8051764474195593) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.608495712280273 (27.374634499643363) | > grad_norm_1: tensor(198.5641, device='cuda:0') (tensor(243.3296, device='cuda:0')) | > current_lr_0: 0.0001997751124671936 | > current_lr_1: 0.0001997751124671936 | > step_time: 0.4582 (0.43444059409347235) | > loader_time: 0.0089 (0.007505533741969688) --> EVAL PERFORMANCE | > avg_loader_time: 0.008385356043426084 (-1.5342739266408292e-05) | > avg_step_time: 0.454880523010039 (+0.006012271827375404) | > avg_loss_disc: 2.916189825031119 (+0.006879296101314658) | > avg_loss_disc_real_0: 0.25026441816712786 (+0.006625774670654783) | > avg_loss_disc_real_1: 0.24393459713794816 (-0.0010960660350154972) | > avg_loss_disc_real_2: 0.2500373620382497 (+0.005987045210851777) | > avg_loss_disc_real_3: 0.24862659439234666 (+0.0033204240698209986) | > avg_loss_disc_real_4: 0.2461848410082535 (+0.0019071595349781922) | > avg_loss_disc_real_5: 0.247456410401304 (+0.003090201339251558) | > avg_loss_0: 2.916189825031119 (+0.006879296101314658) | > avg_grad_norm_0: tensor(6.8484, device='cuda:0') (+tensor(1.4324, device='cuda:0')) | > avg_loss_gen: 1.6439085325724643 (+0.014929219030998198) | > avg_loss_kl: 2.7870324296011058 (+0.010859486082910319) | > avg_loss_feat: 1.0238529863491868 (+0.005181527473557601) | > avg_loss_mel: 20.1327751589493 (+0.08262508016236936) | > avg_loss_duration: 1.8063769256564932 (-0.006228423454392118) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.393946150658834 (+0.10736693798656205) | > avg_grad_norm_1: tensor(249.6992, device='cuda:0') (+tensor(20.2178, device='cuda:0'))  > EPOCH: 10/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:17:19)   --> TIME: 2025-02-25 08:17:34 -- STEP: 29/72 -- GLOBAL_STEP: 1000750 | > loss_disc: 2.8289976119995117 (2.8804984174925705) | > loss_disc_real_0: 0.2652198076248169 (0.25028102202662106) | > loss_disc_real_1: 0.17679286003112793 (0.24071406752898775) | > loss_disc_real_2: 0.22767159342765808 (0.24491295167084398) | > loss_disc_real_3: 0.24257221817970276 (0.24743170162727093) | > loss_disc_real_4: 0.20909617841243744 (0.24212476662520704) | > loss_disc_real_5: 0.20427633821964264 (0.2409916951738555) | > loss_0: 2.8289976119995117 (2.8804984174925705) | > grad_norm_0: tensor(7.4484, device='cuda:0') (tensor(7.8095, device='cuda:0')) | > loss_gen: 1.6185669898986816 (1.6746325986138706) | > loss_kl: 3.2731237411499023 (2.906676095107506) | > loss_feat: 1.2397093772888184 (1.165622024700559) | > loss_mel: 21.1178035736084 (20.386256777007006) | > loss_duration: 1.7780410051345825 (1.7875927275624768) | > amp_scaler: 256.0 (256.0) | > loss_1: 29.027244567871094 (27.920780116114123) | > grad_norm_1: tensor(295.2816, device='cuda:0') (tensor(237.2145, device='cuda:0')) | > current_lr_0: 0.00019975014057813518 | > current_lr_1: 0.00019975014057813518 | > step_time: 0.4281 (0.4391281111487027) | > loader_time: 0.0076 (0.006659367988849509) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640114690216494 (+0.0002547586467904102) | > avg_step_time: 0.4522293890026254 (-0.0026511340074136247) | > avg_loss_disc: 2.885640520445058 (-0.030549304586060977) | > avg_loss_disc_real_0: 0.24804982291141026 (-0.0022145952557175996) | > avg_loss_disc_real_1: 0.24293688527295287 (-0.000997711864995282) | > avg_loss_disc_real_2: 0.24643660410189291 (-0.0036007579363568076) | > avg_loss_disc_real_3: 0.24562926359579595 (-0.0029973307965507068) | > avg_loss_disc_real_4: 0.24302487935818418 (-0.0031599616500693206) | > avg_loss_disc_real_5: 0.24211507699858975 (-0.0053413334027142545) | > avg_loss_0: 2.885640520445058 (-0.030549304586060977) | > avg_grad_norm_0: tensor(6.6878, device='cuda:0') (tensor(-0.1605, device='cuda:0')) | > avg_loss_gen: 1.669094099125392 (+0.025185566552927696) | > avg_loss_kl: 2.839984806490616 (+0.052952376889510155) | > avg_loss_feat: 1.1472828707224882 (+0.12342988437330149) | > avg_loss_mel: 20.204558547113987 (+0.0717833881646861) | > avg_loss_duration: 1.8000499329096835 (-0.006326992746809701) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.660970204313035 (+0.26702405365420034) | > avg_grad_norm_1: tensor(224.7787, device='cuda:0') (tensor(-24.9205, device='cuda:0'))  > EPOCH: 11/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:17:56)   --> TIME: 2025-02-25 08:18:01 -- STEP: 7/72 -- GLOBAL_STEP: 1000800 | > loss_disc: 3.022501230239868 (2.9057939052581787) | > loss_disc_real_0: 0.41650375723838806 (0.25090691447257996) | > loss_disc_real_1: 0.34224194288253784 (0.2608590636934553) | > loss_disc_real_2: 0.31523391604423523 (0.2509922427790506) | > loss_disc_real_3: 0.26713624596595764 (0.2468951599938529) | > loss_disc_real_4: 0.2502875328063965 (0.25770252517291475) | > loss_disc_real_5: 0.3119344413280487 (0.24473634362220764) | > loss_0: 3.022501230239868 (2.9057939052581787) | > grad_norm_0: tensor(15.6255, device='cuda:0') (tensor(10.2598, device='cuda:0')) | > loss_gen: 1.9256906509399414 (1.7630830662591117) | > loss_kl: 2.5568668842315674 (2.8391561848776683) | > loss_feat: 0.9208410978317261 (1.1693613018308366) | > loss_mel: 19.406442642211914 (19.87429509844099) | > loss_duration: 1.7682902812957764 (1.792463711329869) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.5781307220459 (27.43835994175502) | > grad_norm_1: tensor(179.4094, device='cuda:0') (tensor(292.6533, device='cuda:0')) | > current_lr_0: 0.00019972517181056292 | > current_lr_1: 0.00019972517181056292 | > step_time: 0.4171 (0.41687491961887907) | > loader_time: 0.0069 (0.006185293197631836)  --> TIME: 2025-02-25 08:18:23 -- STEP: 57/72 -- GLOBAL_STEP: 1000850 | > loss_disc: 2.89868426322937 (2.8832635210271467) | > loss_disc_real_0: 0.2129419445991516 (0.24560617433305373) | > loss_disc_real_1: 0.23200975358486176 (0.23927714910946393) | > loss_disc_real_2: 0.2414505034685135 (0.24273562418264255) | > loss_disc_real_3: 0.27278319001197815 (0.24064500446905168) | > loss_disc_real_4: 0.21306613087654114 (0.23999869039184168) | > loss_disc_real_5: 0.2478225976228714 (0.24174921659001133) | > loss_0: 2.89868426322937 (2.8832635210271467) | > grad_norm_0: tensor(3.6950, device='cuda:0') (tensor(9.2457, device='cuda:0')) | > loss_gen: 1.5882189273834229 (1.6605840524037678) | > loss_kl: 2.4520556926727295 (2.6633925730722) | > loss_feat: 1.018439531326294 (1.1950912977519788) | > loss_mel: 19.213224411010742 (19.642249659488083) | > loss_duration: 1.7990543842315674 (1.7815925008372258) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.070993423461914 (26.942910177665844) | > grad_norm_1: tensor(220.1996, device='cuda:0') (tensor(242.6119, device='cuda:0')) | > current_lr_0: 0.00019972517181056292 | > current_lr_1: 0.00019972517181056292 | > step_time: 0.451 (0.43276616146689967) | > loader_time: 0.0093 (0.0077864914609674825) --> EVAL PERFORMANCE | > avg_loader_time: 0.008427807982538787 (-0.000212306707677707) | > avg_step_time: 0.45022679382646585 (-0.0020025951761595273) | > avg_loss_disc: 2.8900426911636137 (+0.0044021707185555314) | > avg_loss_disc_real_0: 0.24702910827079289 (-0.0010207146406173706) | > avg_loss_disc_real_1: 0.24157449130860853 (-0.001362393964344344) | > avg_loss_disc_real_2: 0.24327565341348378 (-0.003160950688409131) | > avg_loss_disc_real_3: 0.2416333709384354 (-0.003995892657360556) | > avg_loss_disc_real_4: 0.2420301720709868 (-0.0009947072871973883) | > avg_loss_disc_real_5: 0.24228711741071352 (+0.00017204041212376397) | > avg_loss_0: 2.8900426911636137 (+0.0044021707185555314) | > avg_grad_norm_0: tensor(8.4805, device='cuda:0') (+tensor(1.7927, device='cuda:0')) | > avg_loss_gen: 1.6651574473985484 (-0.003936651726843543) | > avg_loss_kl: 2.6701318243859524 (-0.16985298210466349) | > avg_loss_feat: 1.1840431186514841 (+0.036760247928995904) | > avg_loss_mel: 19.74468499841825 (-0.45987354869573593) | > avg_loss_duration: 1.7837246925058499 (-0.016325240403833652) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.047742172026297 (-0.6132280322867381) | > avg_grad_norm_1: tensor(240.1803, device='cuda:0') (+tensor(15.4016, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000865.pth  > EPOCH: 12/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:18:35)   --> TIME: 2025-02-25 08:18:52 -- STEP: 35/72 -- GLOBAL_STEP: 1000900 | > loss_disc: 2.7877750396728516 (2.8686034611293247) | > loss_disc_real_0: 0.18705663084983826 (0.2402053062404905) | > loss_disc_real_1: 0.15801887214183807 (0.24070787600108556) | > loss_disc_real_2: 0.20947931706905365 (0.24327957119260515) | > loss_disc_real_3: 0.25703567266464233 (0.24586929295744214) | > loss_disc_real_4: 0.20974689722061157 (0.2456662280218942) | > loss_disc_real_5: 0.23456788063049316 (0.24148969990866526) | > loss_0: 2.7877750396728516 (2.8686034611293247) | > grad_norm_0: tensor(14.2712, device='cuda:0') (tensor(7.3042, device='cuda:0')) | > loss_gen: 1.5024340152740479 (1.6767275026866368) | > loss_kl: 2.979099750518799 (2.8799008641924178) | > loss_feat: 1.530431866645813 (1.2490832941872734) | > loss_mel: 20.60323715209961 (19.93271484375) | > loss_duration: 1.7908602952957153 (1.7640734672546388) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.406063079833984 (27.50249993460519) | > grad_norm_1: tensor(331.6520, device='cuda:0') (tensor(223.0181, device='cuda:0')) | > current_lr_0: 0.0001997002061640866 | > current_lr_1: 0.0001997002061640866 | > step_time: 0.4292 (0.4262429101126535) | > loader_time: 0.008 (0.006985133034842355) --> EVAL PERFORMANCE | > avg_loader_time: 0.008422898574614191 (-4.9094079245960015e-06) | > avg_step_time: 0.45759253770532743 (+0.007365743878861575) | > avg_loss_disc: 2.885740746914501 (-0.004301944249112566) | > avg_loss_disc_real_0: 0.24375186734636065 (-0.00327724092443224) | > avg_loss_disc_real_1: 0.24129905614634634 (-0.0002754351622621942) | > avg_loss_disc_real_2: 0.24566206797747545 (+0.0023864145639916623) | > avg_loss_disc_real_3: 0.24597520454668662 (+0.0043418336082512166) | > avg_loss_disc_real_4: 0.24588175621670735 (+0.0038515841457205657) | > avg_loss_disc_real_5: 0.24239242328724392 (+0.0001053058765304038) | > avg_loss_0: 2.885740746914501 (-0.004301944249112566) | > avg_grad_norm_0: tensor(7.0046, device='cuda:0') (tensor(-1.4759, device='cuda:0')) | > avg_loss_gen: 1.6706872842681237 (+0.005529836869575311) | > avg_loss_kl: 2.7855866056093026 (+0.11545478122335018) | > avg_loss_feat: 1.2023540013272997 (+0.0183108826758156) | > avg_loss_mel: 19.925112039270534 (+0.18042704085228323) | > avg_loss_duration: 1.776716521088506 (-0.007008171417343778) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.36045643981074 (+0.3127142677844432) | > avg_grad_norm_1: tensor(228.9953, device='cuda:0') (tensor(-11.1849, device='cuda:0'))  > EPOCH: 13/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:19:12)   --> TIME: 2025-02-25 08:19:19 -- STEP: 13/72 -- GLOBAL_STEP: 1000950 | > loss_disc: 2.9315714836120605 (2.859658663089459) | > loss_disc_real_0: 0.3524682819843292 (0.24717491062787864) | > loss_disc_real_1: 0.3101600110530853 (0.24358717294839713) | > loss_disc_real_2: 0.2592356503009796 (0.24688566762667435) | > loss_disc_real_3: 0.28543463349342346 (0.24767281458928034) | > loss_disc_real_4: 0.3111870586872101 (0.2436544127189196) | > loss_disc_real_5: 0.3164878785610199 (0.24179265934687394) | > loss_0: 2.9315714836120605 (2.859658663089459) | > grad_norm_0: tensor(10.1561, device='cuda:0') (tensor(7.3742, device='cuda:0')) | > loss_gen: 1.775195598602295 (1.7189474014135508) | > loss_kl: 2.8190438747406006 (2.7079458970289965) | > loss_feat: 1.055268406867981 (1.261477727156419) | > loss_mel: 18.10359001159668 (20.316599185650166) | > loss_duration: 1.748166561126709 (1.7785894137162428) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.501264572143555 (27.783559652475212) | > grad_norm_1: tensor(146.8176, device='cuda:0') (tensor(198.4345, device='cuda:0')) | > current_lr_0: 0.00019967524363831608 | > current_lr_1: 0.00019967524363831608 | > step_time: 0.4553 (0.4244197148543138) | > loader_time: 0.0114 (0.0070725954495943506)  --> TIME: 2025-02-25 08:19:43 -- STEP: 63/72 -- GLOBAL_STEP: 1001000 | > loss_disc: 2.9039359092712402 (2.876328426694113) | > loss_disc_real_0: 0.19481578469276428 (0.24034472330222054) | > loss_disc_real_1: 0.20156902074813843 (0.24279885774567014) | > loss_disc_real_2: 0.19965270161628723 (0.2450341416729821) | > loss_disc_real_3: 0.21853460371494293 (0.2426593710505773) | > loss_disc_real_4: 0.25568804144859314 (0.24422236354578108) | > loss_disc_real_5: 0.25536811351776123 (0.24431820615889535) | > loss_0: 2.9039359092712402 (2.876328426694113) | > grad_norm_0: tensor(7.1405, device='cuda:0') (tensor(7.2322, device='cuda:0')) | > loss_gen: 1.5807216167449951 (1.6851773999986195) | > loss_kl: 2.6276352405548096 (2.7272086408403187) | > loss_feat: 1.294409990310669 (1.2512692705033317) | > loss_mel: 19.652387619018555 (20.016976765223905) | > loss_duration: 1.750496506690979 (1.7735278606414795) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.905649185180664 (27.454159872872488) | > grad_norm_1: tensor(159.6012, device='cuda:0') (tensor(197.0115, device='cuda:0')) | > current_lr_0: 0.00019967524363831608 | > current_lr_1: 0.00019967524363831608 | > step_time: 0.4941 (0.4467459860302153) | > loader_time: 0.0116 (0.008144367308843704) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515069182489959 (+9.217060787576788e-05) | > avg_step_time: 0.45296536029224665 (-0.004627177413080774) | > avg_loss_disc: 2.877741343538526 (-0.007999403375975156) | > avg_loss_disc_real_0: 0.2416839341462498 (-0.0020679332001108386) | > avg_loss_disc_real_1: 0.241957415367516 (+0.0006583592211696632) | > avg_loss_disc_real_2: 0.24446731197162413 (-0.0011947560058513107) | > avg_loss_disc_real_3: 0.24346530773270297 (-0.002509896813983642) | > avg_loss_disc_real_4: 0.24341142702270563 (-0.0024703291940017247) | > avg_loss_disc_real_5: 0.24444520914218795 (+0.0020527858549440336) | > avg_loss_0: 2.877741343538526 (-0.007999403375975156) | > avg_grad_norm_0: tensor(7.1236, device='cuda:0') (+tensor(0.1190, device='cuda:0')) | > avg_loss_gen: 1.684932341038341 (+0.014245056770217257) | > avg_loss_kl: 2.713794449685325 (-0.07179215592397759) | > avg_loss_feat: 1.2489054740314751 (+0.04655147270417537) | > avg_loss_mel: 19.97574373701928 (+0.0506316977487451) | > avg_loss_duration: 1.776198706156771 (-0.0005178149317350833) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.399574682746135 (+0.039118242935394676) | > avg_grad_norm_1: tensor(202.5832, device='cuda:0') (tensor(-26.4121, device='cuda:0'))  > EPOCH: 14/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:19:48)   --> TIME: 2025-02-25 08:20:08 -- STEP: 41/72 -- GLOBAL_STEP: 1001050 | > loss_disc: 2.977388858795166 (2.878531293171208) | > loss_disc_real_0: 0.25983405113220215 (0.2464263141155243) | > loss_disc_real_1: 0.191773921251297 (0.24152139229018513) | > loss_disc_real_2: 0.2701978087425232 (0.24393404257006762) | > loss_disc_real_3: 0.27118754386901855 (0.24190911687001948) | > loss_disc_real_4: 0.2590712904930115 (0.24080294952159975) | > loss_disc_real_5: 0.2845865488052368 (0.2433111318727819) | > loss_0: 2.977388858795166 (2.878531293171208) | > grad_norm_0: tensor(8.9156, device='cuda:0') (tensor(7.1205, device='cuda:0')) | > loss_gen: 1.6471076011657715 (1.6864519439092496) | > loss_kl: 2.553969383239746 (2.7621085876371803) | > loss_feat: 1.1011830568313599 (1.312230002589342) | > loss_mel: 18.994234085083008 (19.960613064649625) | > loss_duration: 1.7527351379394531 (1.7656976653308403) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.049230575561523 (27.48710143856886) | > grad_norm_1: tensor(132.5952, device='cuda:0') (tensor(205.4990, device='cuda:0')) | > current_lr_0: 0.0001996502842328613 | > current_lr_1: 0.0001996502842328613 | > step_time: 0.4414 (0.4329438674740675) | > loader_time: 0.0089 (0.007291270465385623) --> EVAL PERFORMANCE | > avg_loader_time: 0.00846931967936771 (-4.574950312224817e-05) | > avg_step_time: 0.44765994246576873 (-0.005305417826477921) | > avg_loss_disc: 2.885775539237009 (+0.008034195698483071) | > avg_loss_disc_real_0: 0.24507415882298644 (+0.003390224676736636) | > avg_loss_disc_real_1: 0.2413995689909223 (-0.0005578463765937014) | > avg_loss_disc_real_2: 0.24442541599273682 (-4.189597888731855e-05) | > avg_loss_disc_real_3: 0.2437166735320024 (+0.0002513657992994356) | > avg_loss_disc_real_4: 0.24321623104558865 (-0.00019519597711697978) | > avg_loss_disc_real_5: 0.2426734900390598 (-0.0017717191031281498) | > avg_loss_0: 2.885775539237009 (+0.008034195698483071) | > avg_grad_norm_0: tensor(6.7890, device='cuda:0') (tensor(-0.3345, device='cuda:0')) | > avg_loss_gen: 1.678617450552927 (-0.006314890485414004) | > avg_loss_kl: 2.704093714834939 (-0.009700734850385917) | > avg_loss_feat: 1.2663006589446266 (+0.017395184913151507) | > avg_loss_mel: 19.912659443600077 (-0.06308429341920174) | > avg_loss_duration: 1.7729368562429724 (-0.003261849913798631) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.334608266051386 (-0.06496641669474812) | > avg_grad_norm_1: tensor(202.0191, device='cuda:0') (tensor(-0.5641, device='cuda:0'))  > EPOCH: 15/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:20:25)   --> TIME: 2025-02-25 08:20:35 -- STEP: 19/72 -- GLOBAL_STEP: 1001100 | > loss_disc: 2.9587786197662354 (2.891740736208464) | > loss_disc_real_0: 0.27034133672714233 (0.24588708579540253) | > loss_disc_real_1: 0.34924983978271484 (0.24602184562306656) | > loss_disc_real_2: 0.2560771703720093 (0.243091513451777) | > loss_disc_real_3: 0.27094972133636475 (0.24866846988075658) | > loss_disc_real_4: 0.32028138637542725 (0.24683895001285955) | > loss_disc_real_5: 0.31681591272354126 (0.24171258117023267) | > loss_0: 2.9587786197662354 (2.891740736208464) | > grad_norm_0: tensor(6.8855, device='cuda:0') (tensor(7.5957, device='cuda:0')) | > loss_gen: 1.734466791152954 (1.7025118187854165) | > loss_kl: 2.6958789825439453 (2.621646655233283) | > loss_feat: 1.0892359018325806 (1.2940272657494796) | > loss_mel: 19.261457443237305 (20.187122746517783) | > loss_duration: 1.712044596672058 (1.7523123590569747) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.49308204650879 (27.557620901810495) | > grad_norm_1: tensor(70.3258, device='cuda:0') (tensor(167.9343, device='cuda:0')) | > current_lr_0: 0.00019962532794733217 | > current_lr_1: 0.00019962532794733217 | > step_time: 0.4331 (0.4235148555354068) | > loader_time: 0.0066 (0.006843265734220806)  --> TIME: 2025-02-25 08:20:58 -- STEP: 69/72 -- GLOBAL_STEP: 1001150 | > loss_disc: 2.8806498050689697 (2.8952288282090337) | > loss_disc_real_0: 0.2649874687194824 (0.24395019420679065) | > loss_disc_real_1: 0.26336660981178284 (0.24373457850753397) | > loss_disc_real_2: 0.24689912796020508 (0.2459157549816629) | > loss_disc_real_3: 0.24482598900794983 (0.24489527012127033) | > loss_disc_real_4: 0.25887614488601685 (0.2448471415302028) | > loss_disc_real_5: 0.23329280316829681 (0.24248343878898068) | > loss_0: 2.8806498050689697 (2.8952288282090337) | > grad_norm_0: tensor(2.8627, device='cuda:0') (tensor(5.9190, device='cuda:0')) | > loss_gen: 1.6929004192352295 (1.6703599501347197) | > loss_kl: 2.634803056716919 (2.6027731930000195) | > loss_feat: 1.1456425189971924 (1.2149052343506739) | > loss_mel: 19.230792999267578 (19.65604370227758) | > loss_duration: 1.8460369110107422 (1.763031285742055) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.550174713134766 (26.90711333786232) | > grad_norm_1: tensor(278.4644, device='cuda:0') (tensor(227.6208, device='cuda:0')) | > current_lr_0: 0.00019962532794733217 | > current_lr_1: 0.00019962532794733217 | > step_time: 0.5385 (0.44761247911315033) | > loader_time: 0.0126 (0.008410688759624094) --> EVAL PERFORMANCE | > avg_loader_time: 0.008482197640647351 (+1.2877961279640637e-05) | > avg_step_time: 0.4496731355156697 (+0.002013193049900941) | > avg_loss_disc: 2.8944109695058473 (+0.008635430268838284) | > avg_loss_disc_real_0: 0.24340019805330626 (-0.0016739607696801828) | > avg_loss_disc_real_1: 0.24362085444826476 (+0.002221285457342459) | > avg_loss_disc_real_2: 0.24575299696183542 (+0.0013275809690986018) | > avg_loss_disc_real_3: 0.24498131367522227 (+0.001264640143219864) | > avg_loss_disc_real_4: 0.24501936389526852 (+0.001803132849679867) | > avg_loss_disc_real_5: 0.24291912581719144 (+0.0002456357781316365) | > avg_loss_0: 2.8944109695058473 (+0.008635430268838284) | > avg_grad_norm_0: tensor(5.8704, device='cuda:0') (tensor(-0.9186, device='cuda:0')) | > avg_loss_gen: 1.6680736239527312 (-0.010543826600195816) | > avg_loss_kl: 2.6072503311533324 (-0.09684338368160672) | > avg_loss_feat: 1.2130633743715953 (-0.05323728457303134) | > avg_loss_mel: 19.660011989969597 (-0.25264745363048036) | > avg_loss_duration: 1.762927406270739 (-0.010009449972233408) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.911326690458914 (-0.4232815755924726) | > avg_grad_norm_1: tensor(226.3823, device='cuda:0') (+tensor(24.3631, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001153.pth  > EPOCH: 16/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:21:04)   --> TIME: 2025-02-25 08:21:27 -- STEP: 47/72 -- GLOBAL_STEP: 1001200 | > loss_disc: 2.8801558017730713 (2.89119850321019) | > loss_disc_real_0: 0.22632938623428345 (0.24236279662619245) | > loss_disc_real_1: 0.24657349288463593 (0.2369601436434908) | > loss_disc_real_2: 0.2877485454082489 (0.24102419646496467) | > loss_disc_real_3: 0.2370375245809555 (0.24164117302032226) | > loss_disc_real_4: 0.2510968744754791 (0.24444667551111668) | > loss_disc_real_5: 0.21609003841876984 (0.2449251632107065) | > loss_0: 2.8801558017730713 (2.89119850321019) | > grad_norm_0: tensor(3.4926, device='cuda:0') (tensor(7.1661, device='cuda:0')) | > loss_gen: 1.5476938486099243 (1.6523948755670101) | > loss_kl: 2.5381743907928467 (2.70546947134302) | > loss_feat: 1.0153754949569702 (1.2175817299396432) | > loss_mel: 19.034391403198242 (19.714518689094707) | > loss_duration: 1.774019479751587 (1.752899134412725) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.909652709960938 (27.042863724079538) | > grad_norm_1: tensor(87.6172, device='cuda:0') (tensor(227.5675, device='cuda:0')) | > current_lr_0: 0.00019960037478133875 | > current_lr_1: 0.00019960037478133875 | > step_time: 0.4417 (0.43663811683654785) | > loader_time: 0.0088 (0.007510621496971617) --> EVAL PERFORMANCE | > avg_loader_time: 0.008463342424849389 (-1.8855215797962746e-05) | > avg_step_time: 0.45154077570203327 (+0.001867640186363595) | > avg_loss_disc: 2.887251558438154 (-0.007159411067693355) | > avg_loss_disc_real_0: 0.24612293633776652 (+0.0027227382844602632) | > avg_loss_disc_real_1: 0.23776750612846562 (-0.00585334831979914) | > avg_loss_disc_real_2: 0.24274722476240615 (-0.0030057721994292685) | > avg_loss_disc_real_3: 0.24198563417918245 (-0.0029956794960398214) | > avg_loss_disc_real_4: 0.24403912374671077 (-0.0009802401485577428) | > avg_loss_disc_real_5: 0.24497131394668364 (+0.002052188129492205) | > avg_loss_0: 2.887251558438154 (-0.007159411067693355) | > avg_grad_norm_0: tensor(8.2155, device='cuda:0') (+tensor(2.3451, device='cuda:0')) | > avg_loss_gen: 1.670580568447919 (+0.002506944495187735) | > avg_loss_kl: 2.681766782008426 (+0.07451645085509373) | > avg_loss_feat: 1.2062283981014306 (-0.0068349762701647165) | > avg_loss_mel: 19.66550966719506 (+0.005497677225463349) | > avg_loss_duration: 1.7629009585984996 (-2.6447672239360287e-05) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.98698621400645 (+0.07565952354753591) | > avg_grad_norm_1: tensor(234.5394, device='cuda:0') (+tensor(8.1571, device='cuda:0'))  > EPOCH: 17/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:21:40)   --> TIME: 2025-02-25 08:21:53 -- STEP: 25/72 -- GLOBAL_STEP: 1001250 | > loss_disc: 2.790039539337158 (2.8882436561584472) | > loss_disc_real_0: 0.23388923704624176 (0.24141636848449707) | > loss_disc_real_1: 0.27879729866981506 (0.2392246311903) | > loss_disc_real_2: 0.2542441487312317 (0.24435311675071716) | > loss_disc_real_3: 0.2034023553133011 (0.2452193319797516) | > loss_disc_real_4: 0.2253572791814804 (0.24818501949310304) | > loss_disc_real_5: 0.26610299944877625 (0.2477766352891922) | > loss_0: 2.790039539337158 (2.8882436561584472) | > grad_norm_0: tensor(4.4893, device='cuda:0') (tensor(6.9446, device='cuda:0')) | > loss_gen: 1.4962618350982666 (1.685491557121277) | > loss_kl: 2.8698010444641113 (2.6294270038604735) | > loss_feat: 1.403889536857605 (1.287045631408691) | > loss_mel: 20.34833526611328 (19.278455886840824) | > loss_duration: 1.7233526706695557 (1.7481592416763305) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.84164047241211 (26.628579177856444) | > grad_norm_1: tensor(396.0205, device='cuda:0') (tensor(254.9120, device='cuda:0')) | > current_lr_0: 0.00019957542473449108 | > current_lr_1: 0.00019957542473449108 | > step_time: 0.4222 (0.4245296096801758) | > loader_time: 0.0071 (0.00649531364440918) --> EVAL PERFORMANCE | > avg_loader_time: 0.008334824736689176 (-0.0001285176881602127) | > avg_step_time: 0.45264697074890137 (+0.001106195046868097) | > avg_loss_disc: 2.8851246531580537 (-0.0021269052801002886) | > avg_loss_disc_real_0: 0.24287784057603756 (-0.003245095761728961) | > avg_loss_disc_real_1: 0.24051104087225147 (+0.002743534743785858) | > avg_loss_disc_real_2: 0.24369787456284106 (+0.0009506498004349107) | > avg_loss_disc_real_3: 0.24398707378078516 (+0.002001439601602706) | > avg_loss_disc_real_4: 0.24507518385497618 (+0.001036060108265402) | > avg_loss_disc_real_5: 0.24514119935707307 (+0.0001698854103894254) | > avg_loss_0: 2.8851246531580537 (-0.0021269052801002886) | > avg_grad_norm_0: tensor(6.5905, device='cuda:0') (tensor(-1.6250, device='cuda:0')) | > avg_loss_gen: 1.6853060755931155 (+0.014725507145196604) | > avg_loss_kl: 2.6858854327403323 (+0.004118650731906204) | > avg_loss_feat: 1.2748170201207552 (+0.06858862201932459) | > avg_loss_mel: 19.55238898371308 (-0.11312068348198068) | > avg_loss_duration: 1.7624613617507505 (-0.0004395968477490797) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.960858842016943 (-0.026127371989506543) | > avg_grad_norm_1: tensor(246.7722, device='cuda:0') (+tensor(12.2328, device='cuda:0'))  > EPOCH: 18/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:22:17)   --> TIME: 2025-02-25 08:22:20 -- STEP: 3/72 -- GLOBAL_STEP: 1001300 | > loss_disc: 3.022946834564209 (2.973973035812378) | > loss_disc_real_0: 0.21725699305534363 (0.2384794851144155) | > loss_disc_real_1: 0.19852706789970398 (0.26173823078473407) | > loss_disc_real_2: 0.24702955782413483 (0.2474996397892634) | > loss_disc_real_3: 0.29018011689186096 (0.27646850049495697) | > loss_disc_real_4: 0.29478195309638977 (0.28743542234102887) | > loss_disc_real_5: 0.32211995124816895 (0.27543029685815174) | > loss_0: 3.022946834564209 (2.973973035812378) | > grad_norm_0: tensor(3.5352, device='cuda:0') (tensor(4.5872, device='cuda:0')) | > loss_gen: 1.5852434635162354 (1.7098170916239421) | > loss_kl: 2.736971139907837 (2.7424144744873047) | > loss_feat: 1.158319115638733 (1.158014436562856) | > loss_mel: 19.42810821533203 (19.05272674560547) | > loss_duration: 1.7827937602996826 (1.7700860897699993) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.691434860229492 (26.433059056599934) | > grad_norm_1: tensor(352.0777, device='cuda:0') (tensor(311.0700, device='cuda:0')) | > current_lr_0: 0.00019955047780639926 | > current_lr_1: 0.00019955047780639926 | > step_time: 0.4377 (0.43405946095784503) | > loader_time: 0.0056 (0.006072282791137695)  --> TIME: 2025-02-25 08:22:43 -- STEP: 53/72 -- GLOBAL_STEP: 1001350 | > loss_disc: 2.9231977462768555 (2.8913787041070327) | > loss_disc_real_0: 0.3182792067527771 (0.24189460755519146) | > loss_disc_real_1: 0.21800488233566284 (0.2383382224811698) | > loss_disc_real_2: 0.19074992835521698 (0.2451438288081367) | > loss_disc_real_3: 0.24464261531829834 (0.2444407231402847) | > loss_disc_real_4: 0.25461989641189575 (0.2451840048691012) | > loss_disc_real_5: 0.22932298481464386 (0.2429240306593337) | > loss_0: 2.9231977462768555 (2.8913787041070327) | > grad_norm_0: tensor(7.9937, device='cuda:0') (tensor(6.5563, device='cuda:0')) | > loss_gen: 1.7604336738586426 (1.663887914621605) | > loss_kl: 2.6514511108398438 (2.5678084886298995) | > loss_feat: 1.0244704484939575 (1.2207508818158561) | > loss_mel: 17.696496963500977 (19.30402539811044) | > loss_duration: 1.8110160827636719 (1.7488588926927098) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.943866729736328 (26.50533150726894) | > grad_norm_1: tensor(103.9918, device='cuda:0') (tensor(242.3885, device='cuda:0')) | > current_lr_0: 0.00019955047780639926 | > current_lr_1: 0.00019955047780639926 | > step_time: 0.476 (0.4415345012017016) | > loader_time: 0.0093 (0.007609421352170549) --> EVAL PERFORMANCE | > avg_loader_time: 0.008441867962689467 (+0.00010704322600029077) | > avg_step_time: 0.4570131906321351 (+0.004366219883233713) | > avg_loss_disc: 2.8850599544149054 (-6.469874314829482e-05) | > avg_loss_disc_real_0: 0.2408868305280175 (-0.0019910100480200754) | > avg_loss_disc_real_1: 0.23664685307254255 (-0.003864187799708929) | > avg_loss_disc_real_2: 0.2445809673255598 (+0.0008830927627187513) | > avg_loss_disc_real_3: 0.24319868524309615 (-0.0007883885376890054) | > avg_loss_disc_real_4: 0.24389103537713977 (-0.0011841484778364053) | > avg_loss_disc_real_5: 0.24323827951726779 (-0.0019029198398052838) | > avg_loss_0: 2.8850599544149054 (-6.469874314829482e-05) | > avg_grad_norm_0: tensor(6.4609, device='cuda:0') (tensor(-0.1297, device='cuda:0')) | > avg_loss_gen: 1.664563251213289 (-0.02074282437982644) | > avg_loss_kl: 2.582585391863971 (-0.10330004087636135) | > avg_loss_feat: 1.2314131755224416 (-0.04340384459831359) | > avg_loss_mel: 19.358875220930067 (-0.19351376278301302) | > avg_loss_duration: 1.7536705933826073 (-0.008790768368143187) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.591107543085663 (-0.36975129893128056) | > avg_grad_norm_1: tensor(246.4767, device='cuda:0') (tensor(-0.2955, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001369.pth  > EPOCH: 19/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:22:57)   --> TIME: 2025-02-25 08:23:12 -- STEP: 31/72 -- GLOBAL_STEP: 1001400 | > loss_disc: 2.9657487869262695 (2.8861963748931885) | > loss_disc_real_0: 0.3554186522960663 (0.2408942493700212) | > loss_disc_real_1: 0.420213907957077 (0.24729587570313485) | > loss_disc_real_2: 0.24035559594631195 (0.24597587123993905) | > loss_disc_real_3: 0.23452018201351166 (0.24270666366623295) | > loss_disc_real_4: 0.22193938493728638 (0.2465059281356873) | > loss_disc_real_5: 0.21424835920333862 (0.24475495036571257) | > loss_0: 2.9657487869262695 (2.8861963748931885) | > grad_norm_0: tensor(17.0801, device='cuda:0') (tensor(7.9736, device='cuda:0')) | > loss_gen: 1.712208867073059 (1.7029488240518877) | > loss_kl: 2.682657480239868 (2.6941521167755127) | > loss_feat: 1.3996262550354004 (1.4023534482525242) | > loss_mel: 20.190542221069336 (19.872536382367535) | > loss_duration: 1.7450640201568604 (1.7386954369083527) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.730100631713867 (27.41068636986517) | > grad_norm_1: tensor(337.0285, device='cuda:0') (tensor(212.1895, device='cuda:0')) | > current_lr_0: 0.00019952553399667344 | > current_lr_1: 0.00019952553399667344 | > step_time: 0.448 (0.4353668612818564) | > loader_time: 0.0084 (0.0069086551666259766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008506358509332361 (+6.449054664289453e-05) | > avg_step_time: 0.45076894760131836 (-0.006244243030816721) | > avg_loss_disc: 2.9073612723552005 (+0.022301317940295107) | > avg_loss_disc_real_0: 0.24147499310718457 (+0.0005881625791670786) | > avg_loss_disc_real_1: 0.2435164761165498 (+0.006869623044007261) | > avg_loss_disc_real_2: 0.24652379079603812 (+0.0019428234704783054) | > avg_loss_disc_real_3: 0.2444078842519035 (+0.0012091990088073379) | > avg_loss_disc_real_4: 0.2468671998087789 (+0.0029761644316391167) | > avg_loss_disc_real_5: 0.2469807806569086 (+0.003742501139640808) | > avg_loss_0: 2.9073612723552005 (+0.022301317940295107) | > avg_grad_norm_0: tensor(6.9748, device='cuda:0') (+tensor(0.5139, device='cuda:0')) | > avg_loss_gen: 1.664663877285702 (+0.00010062607241301258) | > avg_loss_kl: 2.6745133299223136 (+0.09192793805834265) | > avg_loss_feat: 1.2391708652738116 (+0.007757689751370034) | > avg_loss_mel: 19.71130373780156 (+0.3524285168714947) | > avg_loss_duration: 1.758972812706316 (+0.005302219323708668) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.048624683433854 (+0.457517140348191) | > avg_grad_norm_1: tensor(186.7496, device='cuda:0') (tensor(-59.7271, device='cuda:0'))  > EPOCH: 20/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:23:33)   --> TIME: 2025-02-25 08:23:39 -- STEP: 9/72 -- GLOBAL_STEP: 1001450 | > loss_disc: 2.9057536125183105 (2.8519533475240073) | > loss_disc_real_0: 0.19697996973991394 (0.23669269680976868) | > loss_disc_real_1: 0.2975824177265167 (0.24019617007838356) | > loss_disc_real_2: 0.2127843052148819 (0.23714648849434322) | > loss_disc_real_3: 0.18719737231731415 (0.23879917959372202) | > loss_disc_real_4: 0.24907398223876953 (0.2450561308198505) | > loss_disc_real_5: 0.27047187089920044 (0.24283565249707964) | > loss_0: 2.9057536125183105 (2.8519533475240073) | > grad_norm_0: tensor(8.0786, device='cuda:0') (tensor(6.7126, device='cuda:0')) | > loss_gen: 1.6266931295394897 (1.6845524046156142) | > loss_kl: 2.3659915924072266 (2.481327348285251) | > loss_feat: 1.4207643270492554 (1.306417531437344) | > loss_mel: 20.02596092224121 (19.61341094970703) | > loss_duration: 1.759803056716919 (1.7548447317547269) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.199214935302734 (26.840553283691406) | > grad_norm_1: tensor(56.8906, device='cuda:0') (tensor(197.0267, device='cuda:0')) | > current_lr_0: 0.00019950059330492385 | > current_lr_1: 0.00019950059330492385 | > step_time: 0.4221 (0.4261559910244412) | > loader_time: 0.0068 (0.006579902436998155)  --> TIME: 2025-02-25 08:24:02 -- STEP: 59/72 -- GLOBAL_STEP: 1001500 | > loss_disc: 2.850343942642212 (2.903893834453518) | > loss_disc_real_0: 0.2358885109424591 (0.2425661152702267) | > loss_disc_real_1: 0.16816596686840057 (0.2410505739814144) | > loss_disc_real_2: 0.2285175770521164 (0.24695299717329317) | > loss_disc_real_3: 0.20940260589122772 (0.2452142031516059) | > loss_disc_real_4: 0.2107834815979004 (0.24606611036648185) | > loss_disc_real_5: 0.22869591414928436 (0.24516853210279496) | > loss_0: 2.850343942642212 (2.903893834453518) | > grad_norm_0: tensor(9.7535, device='cuda:0') (tensor(6.2345, device='cuda:0')) | > loss_gen: 1.7142466306686401 (1.660745638912007) | > loss_kl: 2.4597110748291016 (2.6089320990998868) | > loss_feat: 1.3537002801895142 (1.196908824524637) | > loss_mel: 19.05359649658203 (19.6433622069278) | > loss_duration: 1.750388503074646 (1.752715280500509) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.33164405822754 (26.862664109569486) | > grad_norm_1: tensor(345.9215, device='cuda:0') (tensor(201.6814, device='cuda:0')) | > current_lr_0: 0.00019950059330492385 | > current_lr_1: 0.00019950059330492385 | > step_time: 0.4598 (0.4391410876128633) | > loader_time: 0.0097 (0.007933745949955308) --> EVAL PERFORMANCE | > avg_loader_time: 0.008486969370237537 (-1.9389139094824065e-05) | > avg_step_time: 0.44967846467461386 (-0.0010904829267044946) | > avg_loss_disc: 2.913110464391574 (+0.005749192036373696) | > avg_loss_disc_real_0: 0.24356493752607158 (+0.0020899444188870187) | > avg_loss_disc_real_1: 0.24344738058640925 (-6.909553014056158e-05) | > avg_loss_disc_real_2: 0.24606376088840862 (-0.0004600299076295) | > avg_loss_disc_real_3: 0.246014116515576 (+0.0016062322636725135) | > avg_loss_disc_real_4: 0.24706691566487432 (+0.00019971585609543374) | > avg_loss_disc_real_5: 0.24677321524687215 (-0.00020756541003644213) | > avg_loss_0: 2.913110464391574 (+0.005749192036373696) | > avg_grad_norm_0: tensor(5.9363, device='cuda:0') (tensor(-1.0385, device='cuda:0')) | > avg_loss_gen: 1.6528658161700611 (-0.011798061115640968) | > avg_loss_kl: 2.606387517821621 (-0.06812581210069268) | > avg_loss_feat: 1.1746448370772349 (-0.06452602819657671) | > avg_loss_mel: 19.608869848117024 (-0.10243388968453715) | > avg_loss_duration: 1.7544645759421336 (-0.004508236764182438) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.797232614436616 (-0.2513920689972373) | > avg_grad_norm_1: tensor(218.1075, device='cuda:0') (+tensor(31.3579, device='cuda:0'))  > EPOCH: 21/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:24:10)   --> TIME: 2025-02-25 08:24:28 -- STEP: 37/72 -- GLOBAL_STEP: 1001550 | > loss_disc: 2.9448161125183105 (2.8914007302877067) | > loss_disc_real_0: 0.28195828199386597 (0.24145464075578227) | > loss_disc_real_1: 0.3972407877445221 (0.24000603927148356) | > loss_disc_real_2: 0.23614472150802612 (0.24690183795787193) | > loss_disc_real_3: 0.2474469393491745 (0.24386362368996078) | > loss_disc_real_4: 0.24078160524368286 (0.24511757291652062) | > loss_disc_real_5: 0.268418550491333 (0.24630653254083684) | > loss_0: 2.9448161125183105 (2.8914007302877067) | > grad_norm_0: tensor(15.5872, device='cuda:0') (tensor(4.6089, device='cuda:0')) | > loss_gen: 1.8741023540496826 (1.660048181946213) | > loss_kl: 2.743349552154541 (2.611531502491719) | > loss_feat: 1.009792447090149 (1.2089776219548405) | > loss_mel: 19.08313751220703 (19.38737389848039) | > loss_duration: 1.7342770099639893 (1.732447582322198) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.444658279418945 (26.60037865509858) | > grad_norm_1: tensor(91.5304, device='cuda:0') (tensor(217.5773, device='cuda:0')) | > current_lr_0: 0.00019947565573076072 | > current_lr_1: 0.00019947565573076072 | > step_time: 0.4424 (0.42828393626857447) | > loader_time: 0.0086 (0.007540071332776868) --> EVAL PERFORMANCE | > avg_loader_time: 0.008637079050843143 (+0.00015010968060560556) | > avg_step_time: 0.44622098895865425 (-0.0034574757159596126) | > avg_loss_disc: 2.8921016303586287 (-0.02100883403294551) | > avg_loss_disc_real_0: 0.24245233225150847 (-0.001112605274563111) | > avg_loss_disc_real_1: 0.24015515476045474 (-0.003292225825954509) | > avg_loss_disc_real_2: 0.24692973494529724 (+0.0008659740568886243) | > avg_loss_disc_real_3: 0.24394275101137833 (-0.0020713655041976753) | > avg_loss_disc_real_4: 0.24475297877486324 (-0.0023139368900110813) | > avg_loss_disc_real_5: 0.2461578434621784 (-0.0006153717846937579) | > avg_loss_0: 2.8921016303586287 (-0.02100883403294551) | > avg_grad_norm_0: tensor(5.5973, device='cuda:0') (tensor(-0.3390, device='cuda:0')) | > avg_loss_gen: 1.6586350840581974 (+0.005769267888136254) | > avg_loss_kl: 2.5842671495088387 (-0.02212036831278219) | > avg_loss_feat: 1.1947253321258118 (+0.020080495048576896) | > avg_loss_mel: 19.36354575358646 (-0.24532409453056303) | > avg_loss_duration: 1.7461506346581688 (-0.008313941283964787) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.547323898530344 (-0.2499087159062725) | > avg_grad_norm_1: tensor(240.3257, device='cuda:0') (+tensor(22.2182, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001585.pth  > EPOCH: 22/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:24:48)   --> TIME: 2025-02-25 08:24:56 -- STEP: 15/72 -- GLOBAL_STEP: 1001600 | > loss_disc: 2.9230260848999023 (2.9044661362965902) | > loss_disc_real_0: 0.26215627789497375 (0.250490923722585) | > loss_disc_real_1: 0.19402411580085754 (0.22839810848236083) | > loss_disc_real_2: 0.26670971512794495 (0.2391862432161967) | > loss_disc_real_3: 0.25199875235557556 (0.24446295499801635) | > loss_disc_real_4: 0.2547849714756012 (0.24881505270799) | > loss_disc_real_5: 0.256616473197937 (0.24873322943846385) | > loss_0: 2.9230260848999023 (2.9044661362965902) | > grad_norm_0: tensor(6.9145, device='cuda:0') (tensor(6.3935, device='cuda:0')) | > loss_gen: 1.6616921424865723 (1.6617403745651245) | > loss_kl: 2.4423415660858154 (2.540050919850667) | > loss_feat: 1.0093644857406616 (1.230627751350403) | > loss_mel: 19.70926284790039 (19.273007202148438) | > loss_duration: 1.697892665863037 (1.7411004940668742) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.520553588867188 (26.446527099609376) | > grad_norm_1: tensor(317.4110, device='cuda:0') (tensor(314.8491, device='cuda:0')) | > current_lr_0: 0.00019945072127379438 | > current_lr_1: 0.00019945072127379438 | > step_time: 0.4238 (0.42617629369099935) | > loader_time: 0.0058 (0.006667598088582357)  --> TIME: 2025-02-25 08:25:20 -- STEP: 65/72 -- GLOBAL_STEP: 1001650 | > loss_disc: 2.921804189682007 (2.906099653244018) | > loss_disc_real_0: 0.30117878317832947 (0.24386619444076832) | > loss_disc_real_1: 0.28007960319519043 (0.23443187394967446) | > loss_disc_real_2: 0.2356089949607849 (0.24304747306383573) | > loss_disc_real_3: 0.2165457010269165 (0.24201442209573892) | > loss_disc_real_4: 0.23861585557460785 (0.24585791069727678) | > loss_disc_real_5: 0.22613444924354553 (0.24501121090008662) | > loss_0: 2.921804189682007 (2.906099653244018) | > grad_norm_0: tensor(5.7179, device='cuda:0') (tensor(7.2561, device='cuda:0')) | > loss_gen: 1.6835285425186157 (1.644925198188195) | > loss_kl: 2.5612876415252686 (2.5255652390993557) | > loss_feat: 1.1194744110107422 (1.1925956010818481) | > loss_mel: 19.547809600830078 (18.938758351252638) | > loss_duration: 1.7696552276611328 (1.7377421709207388) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.68175506591797 (26.039586844811073) | > grad_norm_1: tensor(497.3328, device='cuda:0') (tensor(277.7334, device='cuda:0')) | > current_lr_0: 0.00019945072127379438 | > current_lr_1: 0.00019945072127379438 | > step_time: 0.4902 (0.4549895506638747) | > loader_time: 0.0115 (0.008243193993201626) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850734912173849 (-0.0001297299291046529) | > avg_step_time: 0.460912586937488 (+0.014691597978833737) | > avg_loss_disc: 2.905481429167196 (+0.013379798808567323) | > avg_loss_disc_real_0: 0.24452441280156795 (+0.002072080550059474) | > avg_loss_disc_real_1: 0.2356960837480048 (-0.004459071012449939) | > avg_loss_disc_real_2: 0.2435456266705419 (-0.003384108274755354) | > avg_loss_disc_real_3: 0.2420432443769885 (-0.0018995066343898415) | > avg_loss_disc_real_4: 0.24546054068585516 (+0.0007075619109919162) | > avg_loss_disc_real_5: 0.2446866297805813 (-0.0014712136815970833) | > avg_loss_0: 2.905481429167196 (+0.013379798808567323) | > avg_grad_norm_0: tensor(7.1260, device='cuda:0') (+tensor(1.5287, device='cuda:0')) | > avg_loss_gen: 1.6456351532063014 (-0.012999930851895991) | > avg_loss_kl: 2.5346831200828013 (-0.04958402942603746) | > avg_loss_feat: 1.1953330862690026 (+0.0006077541431908084) | > avg_loss_mel: 19.04634712111783 (-0.31719863246863156) | > avg_loss_duration: 1.740200036008593 (-0.005950598649575811) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.162198778609156 (-0.38512511992118803) | > avg_grad_norm_1: tensor(291.7075, device='cuda:0') (+tensor(51.3817, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001657.pth  > EPOCH: 23/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:25:28)   --> TIME: 2025-02-25 08:25:49 -- STEP: 43/72 -- GLOBAL_STEP: 1001700 | > loss_disc: 2.9389889240264893 (2.9011113643646236) | > loss_disc_real_0: 0.18256796896457672 (0.2411255053309507) | > loss_disc_real_1: 0.21409055590629578 (0.23725411607775576) | > loss_disc_real_2: 0.2704111933708191 (0.24729260005230128) | > loss_disc_real_3: 0.27913230657577515 (0.24738186909708865) | > loss_disc_real_4: 0.2609252631664276 (0.24790116035661033) | > loss_disc_real_5: 0.28942105174064636 (0.24735097587108612) | > loss_0: 2.9389889240264893 (2.9011113643646236) | > grad_norm_0: tensor(6.4940, device='cuda:0') (tensor(5.9958, device='cuda:0')) | > loss_gen: 1.523191213607788 (1.6604462501614592) | > loss_kl: 2.463573932647705 (2.6068627778873887) | > loss_feat: 1.3138432502746582 (1.2005345447118891) | > loss_mel: 18.837017059326172 (19.157601733540382) | > loss_duration: 1.7183700799942017 (1.72453605019769) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.855995178222656 (26.349981352340343) | > grad_norm_1: tensor(369.1953, device='cuda:0') (tensor(254.5417, device='cuda:0')) | > current_lr_0: 0.00019942578993363514 | > current_lr_1: 0.00019942578993363514 | > step_time: 0.4554 (0.4330055270084115) | > loader_time: 0.0096 (0.0073826701142067136) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522933637592154 (+1.5584515853664382e-05) | > avg_step_time: 0.45705856403834383 (-0.0038540228991441605) | > avg_loss_disc: 2.9063831416653914 (+0.0009017124981953728) | > avg_loss_disc_real_0: 0.2417506029908086 (-0.0027738098107593367) | > avg_loss_disc_real_1: 0.2354227019027925 (-0.000273381845212306) | > avg_loss_disc_real_2: 0.24400462441041437 (+0.0004589977398724854) | > avg_loss_disc_real_3: 0.24621158586421482 (+0.004168341487226335) | > avg_loss_disc_real_4: 0.24701110955695033 (+0.0015505688710951704) | > avg_loss_disc_real_5: 0.2468491983665547 (+0.0021625685859733845) | > avg_loss_0: 2.9063831416653914 (+0.0009017124981953728) | > avg_grad_norm_0: tensor(6.3835, device='cuda:0') (tensor(-0.7425, device='cuda:0')) | > avg_loss_gen: 1.648840388781588 (+0.0032052355752865136) | > avg_loss_kl: 2.567896611253979 (+0.03321349117117789) | > avg_loss_feat: 1.1878893308236567 (-0.007443755445345923) | > avg_loss_mel: 19.235340763145768 (+0.18899364202793834) | > avg_loss_duration: 1.733704793621117 (-0.006495242387476008) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.373672001798386 (+0.21147322318923045) | > avg_grad_norm_1: tensor(279.7248, device='cuda:0') (tensor(-11.9827, device='cuda:0'))  > EPOCH: 24/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:26:05)   --> TIME: 2025-02-25 08:26:15 -- STEP: 21/72 -- GLOBAL_STEP: 1001750 | > loss_disc: 2.8914008140563965 (2.894137461980184) | > loss_disc_real_0: 0.23588694632053375 (0.24143440240905398) | > loss_disc_real_1: 0.26899799704551697 (0.2443155717282068) | > loss_disc_real_2: 0.2539578378200531 (0.24783170010362351) | > loss_disc_real_3: 0.24179720878601074 (0.2451774598587127) | > loss_disc_real_4: 0.23163126409053802 (0.2511644583372843) | > loss_disc_real_5: 0.25078728795051575 (0.24738918173880803) | > loss_0: 2.8914008140563965 (2.894137461980184) | > grad_norm_0: tensor(4.5616, device='cuda:0') (tensor(7.9450, device='cuda:0')) | > loss_gen: 1.5127112865447998 (1.654058882168361) | > loss_kl: 2.279688835144043 (2.4455625670296803) | > loss_feat: 1.0835776329040527 (1.2152665853500366) | > loss_mel: 18.296741485595703 (19.25797798520043) | > loss_duration: 1.752542495727539 (1.717810034751892) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.925260543823242 (26.290676026117232) | > grad_norm_1: tensor(280.0089, device='cuda:0') (tensor(312.6554, device='cuda:0')) | > current_lr_0: 0.00019940086170989343 | > current_lr_1: 0.00019940086170989343 | > step_time: 0.4277 (0.42572488103594097) | > loader_time: 0.0077 (0.006925696418398903)  --> TIME: 2025-02-25 08:26:39 -- STEP: 71/72 -- GLOBAL_STEP: 1001800 | > loss_disc: 2.7988154888153076 (2.909002508915646) | > loss_disc_real_0: 0.18274395167827606 (0.2420362695841722) | > loss_disc_real_1: 0.2537137567996979 (0.24012519293267962) | > loss_disc_real_2: 0.21837392449378967 (0.2450858241235706) | > loss_disc_real_3: 0.2159198671579361 (0.24332928804444595) | > loss_disc_real_4: 0.24884147942066193 (0.2466041926347034) | > loss_disc_real_5: 0.2403886467218399 (0.24577752931017272) | > loss_0: 2.7988154888153076 (2.909002508915646) | > grad_norm_0: tensor(6.7448, device='cuda:0') (tensor(6.8917, device='cuda:0')) | > loss_gen: 1.405028223991394 (1.6372650777789908) | > loss_kl: 2.322584390640259 (2.579905849107554) | > loss_feat: 1.049839735031128 (1.1836473589212122) | > loss_mel: 18.84657859802246 (19.19130070108763) | > loss_duration: 1.7304630279541016 (1.7336897531025846) | > amp_scaler: 128.0 (227.1549295774648) | > loss_1: 25.3544921875 (26.325808860886262) | > grad_norm_1: tensor(144.4796, device='cuda:0') (tensor(299.6810, device='cuda:0')) | > current_lr_0: 0.00019940086170989343 | > current_lr_1: 0.00019940086170989343 | > step_time: 0.4663 (0.4488499634702441) | > loader_time: 0.0078 (0.008547447097133582) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547447097133582 (+2.4513459541427793e-05) | > avg_step_time: 0.4488499634702441 (-0.008208600568099744) | > avg_loss_disc: 2.909002508915646 (+0.0026193672502548004) | > avg_loss_disc_real_0: 0.2420362695841722 (+0.00028566659336359823) | > avg_loss_disc_real_1: 0.24012519293267962 (+0.004702491029887124) | > avg_loss_disc_real_2: 0.2450858241235706 (+0.0010811997131562334) | > avg_loss_disc_real_3: 0.24332928804444595 (-0.00288229781976887) | > avg_loss_disc_real_4: 0.2466041926347034 (-0.0004069169222469249) | > avg_loss_disc_real_5: 0.24577752931017272 (-0.0010716690563819797) | > avg_loss_0: 2.909002508915646 (+0.0026193672502548004) | > avg_grad_norm_0: tensor(6.8917, device='cuda:0') (+tensor(0.5082, device='cuda:0')) | > avg_loss_gen: 1.6372650777789908 (-0.011575311002597122) | > avg_loss_kl: 2.579905849107554 (+0.012009237853574994) | > avg_loss_feat: 1.1836473589212122 (-0.0042419719024444635) | > avg_loss_mel: 19.19130070108763 (-0.04404006205813715) | > avg_loss_duration: 1.7336897531025846 (-1.50405185324054e-05) | > avg_amp_scaler: 227.1549295774648 (-28.8450704225352) | > avg_loss_1: 26.325808860886262 (-0.04786314091212418) | > avg_grad_norm_1: tensor(299.6810, device='cuda:0') (+tensor(19.9562, device='cuda:0'))  > EPOCH: 25/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:26:41)   --> TIME: 2025-02-25 08:27:05 -- STEP: 49/72 -- GLOBAL_STEP: 1001850 | > loss_disc: 2.8556389808654785 (2.895669635461301) | > loss_disc_real_0: 0.18077844381332397 (0.2396995613769609) | > loss_disc_real_1: 0.27560096979141235 (0.23867878196190814) | > loss_disc_real_2: 0.2606188654899597 (0.24328728750043985) | > loss_disc_real_3: 0.25425902009010315 (0.24187231489590236) | > loss_disc_real_4: 0.27478623390197754 (0.24526824482849666) | > loss_disc_real_5: 0.2694644033908844 (0.24516464587377043) | > loss_0: 2.8556389808654785 (2.895669635461301) | > grad_norm_0: tensor(8.2706, device='cuda:0') (tensor(7.1980, device='cuda:0')) | > loss_gen: 1.6491925716400146 (1.6565431551057466) | > loss_kl: 2.399681806564331 (2.750666000405136) | > loss_feat: 1.2994184494018555 (1.2535591356608329) | > loss_mel: 17.94442367553711 (19.61210262532137) | > loss_duration: 1.8006691932678223 (1.7377094857546749) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.093385696411133 (27.01058037427007) | > grad_norm_1: tensor(125.5050, device='cuda:0') (tensor(204.5266, device='cuda:0')) | > current_lr_0: 0.0001993759366021797 | > current_lr_1: 0.0001993759366021797 | > step_time: 0.4408 (0.4379293237413679) | > loader_time: 0.0095 (0.007641982059089505) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854810526673223 (+6.581695986482283e-07) | > avg_step_time: 0.449549332470961 (+0.0006993690007169318) | > avg_loss_disc: 2.8968704653457857 (-0.012132043569860418) | > avg_loss_disc_real_0: 0.2431541687585938 (+0.0011178991744215938) | > avg_loss_disc_real_1: 0.23700751091392946 (-0.003117682018750151) | > avg_loss_disc_real_2: 0.24426208835252575 (-0.000823735771044859) | > avg_loss_disc_real_3: 0.24248939332827715 (-0.0008398947161687986) | > avg_loss_disc_real_4: 0.24520270694309557 (-0.0014014856916078344) | > avg_loss_disc_real_5: 0.2454278827972815 (-0.00034964651289121074) | > avg_loss_0: 2.8968704653457857 (-0.012132043569860418) | > avg_grad_norm_0: tensor(7.0618, device='cuda:0') (+tensor(0.1701, device='cuda:0')) | > avg_loss_gen: 1.658010931082175 (+0.020745853303184125) | > avg_loss_kl: 2.6929909074810188 (+0.11308505837346461) | > avg_loss_feat: 1.2413004023928036 (+0.05765304347159139) | > avg_loss_mel: 19.55610339742311 (+0.3648026963354809) | > avg_loss_duration: 1.740564715694374 (+0.006874962591789435) | > avg_amp_scaler: 128.0 (-99.1549295774648) | > avg_loss_1: 26.888970334764938 (+0.5631614738786759) | > avg_grad_norm_1: tensor(216.2965, device='cuda:0') (tensor(-83.3845, device='cuda:0'))  > EPOCH: 26/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:27:18)   --> TIME: 2025-02-25 08:27:31 -- STEP: 27/72 -- GLOBAL_STEP: 1001900 | > loss_disc: 2.8695228099823 (2.904220607545641) | > loss_disc_real_0: 0.26708269119262695 (0.2392360700501336) | > loss_disc_real_1: 0.20630159974098206 (0.24170565053268714) | > loss_disc_real_2: 0.22509101033210754 (0.24667198956012723) | > loss_disc_real_3: 0.21079277992248535 (0.2459065682358212) | > loss_disc_real_4: 0.21353182196617126 (0.2432436529133055) | > loss_disc_real_5: 0.21402543783187866 (0.24523563462275047) | > loss_0: 2.8695228099823 (2.904220607545641) | > grad_norm_0: tensor(4.0151, device='cuda:0') (tensor(6.3086, device='cuda:0')) | > loss_gen: 1.6267127990722656 (1.6586766331284135) | > loss_kl: 2.5772647857666016 (2.592590888341268) | > loss_feat: 1.3629419803619385 (1.2265390599215473) | > loss_mel: 19.445457458496094 (19.550475791648584) | > loss_duration: 1.7088394165039062 (1.7162652368898745) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.721216201782227 (26.744547808611834) | > grad_norm_1: tensor(293.4725, device='cuda:0') (tensor(185.3245, device='cuda:0')) | > current_lr_0: 0.00019935101461010442 | > current_lr_1: 0.00019935101461010442 | > step_time: 0.4334 (0.42764713146068434) | > loader_time: 0.0084 (0.0068843188109221284) --> EVAL PERFORMANCE | > avg_loader_time: 0.00844210638126857 (-0.00010599888546366024) | > avg_step_time: 0.44634522182840697 (-0.0032041106425540478) | > avg_loss_disc: 2.912003010091648 (+0.01513254474586212) | > avg_loss_disc_real_0: 0.24467630537462906 (+0.0015221366160352579) | > avg_loss_disc_real_1: 0.2403399060729524 (+0.0033323951590229217) | > avg_loss_disc_real_2: 0.24538843354708711 (+0.0011263451945613678) | > avg_loss_disc_real_3: 0.2452800800682793 (+0.0027906867400021573) | > avg_loss_disc_real_4: 0.24644167553371107 (+0.001238968590615508) | > avg_loss_disc_real_5: 0.2465311200685904 (+0.001103237271308899) | > avg_loss_0: 2.912003010091648 (+0.01513254474586212) | > avg_grad_norm_0: tensor(6.5527, device='cuda:0') (tensor(-0.5092, device='cuda:0')) | > avg_loss_gen: 1.6564736567752463 (-0.0015372743069286265) | > avg_loss_kl: 2.5982511580829883 (-0.09473974939803043) | > avg_loss_feat: 1.2037514305450552 (-0.037548971847748414) | > avg_loss_mel: 19.39437296692754 (-0.16173043049557023) | > avg_loss_duration: 1.7291814743633 (-0.011383241331073934) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.582030658990565 (-0.3069396757743732) | > avg_grad_norm_1: tensor(203.6570, device='cuda:0') (tensor(-12.6396, device='cuda:0'))  > EPOCH: 27/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:27:54)   --> TIME: 2025-02-25 08:27:58 -- STEP: 5/72 -- GLOBAL_STEP: 1001950 | > loss_disc: 2.880256175994873 (2.9262388229370115) | > loss_disc_real_0: 0.20891675353050232 (0.25772765576839446) | > loss_disc_real_1: 0.26630711555480957 (0.2540987581014633) | > loss_disc_real_2: 0.18986926972866058 (0.2537607401609421) | > loss_disc_real_3: 0.23387284576892853 (0.26375583112239837) | > loss_disc_real_4: 0.24224036931991577 (0.2527202367782593) | > loss_disc_real_5: 0.2546103894710541 (0.2588008642196655) | > loss_0: 2.880256175994873 (2.9262388229370115) | > grad_norm_0: tensor(4.0717, device='cuda:0') (tensor(6.6429, device='cuda:0')) | > loss_gen: 1.6446689367294312 (1.6761852979660035) | > loss_kl: 2.6150712966918945 (2.671172571182251) | > loss_feat: 1.2884923219680786 (1.2545649766921998) | > loss_mel: 19.72877311706543 (19.12577590942383) | > loss_duration: 1.680324673652649 (1.735061240196228) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.95732879638672 (26.46275978088379) | > grad_norm_1: tensor(410.9273, device='cuda:0') (tensor(354.3331, device='cuda:0')) | > current_lr_0: 0.00019932609573327815 | > current_lr_1: 0.00019932609573327815 | > step_time: 0.4233 (0.42119503021240234) | > loader_time: 0.006 (0.006618404388427734)  --> TIME: 2025-02-25 08:28:20 -- STEP: 55/72 -- GLOBAL_STEP: 1002000 | > loss_disc: 3.0089752674102783 (2.907105168429288) | > loss_disc_real_0: 0.3770425021648407 (0.2445363919843327) | > loss_disc_real_1: 0.25121235847473145 (0.24085338684645566) | > loss_disc_real_2: 0.27587640285491943 (0.24784060201861638) | > loss_disc_real_3: 0.2669304311275482 (0.24532388069412925) | > loss_disc_real_4: 0.2648363709449768 (0.246510297330943) | > loss_disc_real_5: 0.2876167595386505 (0.2465935157103972) | > loss_0: 3.0089752674102783 (2.907105168429288) | > grad_norm_0: tensor(8.9664, device='cuda:0') (tensor(6.0818, device='cuda:0')) | > loss_gen: 1.7046538591384888 (1.6561946218663997) | > loss_kl: 2.5846939086914062 (2.664400477842851) | > loss_feat: 0.970173180103302 (1.2314907930114052) | > loss_mel: 19.04538917541504 (19.01426089893688) | > loss_duration: 1.7613420486450195 (1.725287628173828) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.066253662109375 (26.29163440357555) | > grad_norm_1: tensor(171.7437, device='cuda:0') (tensor(224.7897, device='cuda:0')) | > current_lr_0: 0.00019932609573327815 | > current_lr_1: 0.00019932609573327815 | > step_time: 0.459 (0.4345296079462225) | > loader_time: 0.0093 (0.007889921014959162) --> EVAL PERFORMANCE | > avg_loader_time: 0.008554230273609428 (+0.00011212389234085789) | > avg_step_time: 0.44428781052710303 (-0.0020574113013039352) | > avg_loss_disc: 2.8994698457314936 (-0.01253316436015428) | > avg_loss_disc_real_0: 0.24434050141086042 (-0.00033580396376864385) | > avg_loss_disc_real_1: 0.24105809104274695 (+0.0007181849697945608) | > avg_loss_disc_real_2: 0.2476593358835704 (+0.002270902336483277) | > avg_loss_disc_real_3: 0.24421324171650577 (-0.0010668383517735414) | > avg_loss_disc_real_4: 0.24537062707921148 (-0.0010710484544995957) | > avg_loss_disc_real_5: 0.24581523827264007 (-0.000715881795950335) | > avg_loss_0: 2.8994698457314936 (-0.01253316436015428) | > avg_grad_norm_0: tensor(5.9078, device='cuda:0') (tensor(-0.6449, device='cuda:0')) | > avg_loss_gen: 1.658694346186141 (+0.002220689410894705) | > avg_loss_kl: 2.6509465331762607 (+0.05269537509327238) | > avg_loss_feat: 1.2220029822537597 (+0.018251551708704516) | > avg_loss_mel: 19.1393998105761 (-0.2549731563514399) | > avg_loss_duration: 1.7305336871617276 (+0.0013522127984275656) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.40157734508246 (-0.1804533139081066) | > avg_grad_norm_1: tensor(230.7323, device='cuda:0') (+tensor(27.0753, device='cuda:0'))  > EPOCH: 28/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:28:30)   --> TIME: 2025-02-25 08:28:47 -- STEP: 33/72 -- GLOBAL_STEP: 1002050 | > loss_disc: 2.976036310195923 (2.9309029145674272) | > loss_disc_real_0: 0.20587162673473358 (0.23745529082688419) | > loss_disc_real_1: 0.1931743323802948 (0.23799870253512354) | > loss_disc_real_2: 0.27978792786598206 (0.24669118180419458) | > loss_disc_real_3: 0.28441089391708374 (0.2463208558884534) | > loss_disc_real_4: 0.2771759033203125 (0.24800060237898972) | > loss_disc_real_5: 0.2335912138223648 (0.24903935781030945) | > loss_0: 2.976036310195923 (2.9309029145674272) | > grad_norm_0: tensor(5.0772, device='cuda:0') (tensor(6.7029, device='cuda:0')) | > loss_gen: 1.6068952083587646 (1.6447518955577503) | > loss_kl: 2.9824321269989014 (2.6017336267413516) | > loss_feat: 1.1848703622817993 (1.2218602986046765) | > loss_mel: 18.921384811401367 (19.441229444561582) | > loss_duration: 1.7305535078048706 (1.7117823145606301) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.426136016845703 (26.621357599894207) | > grad_norm_1: tensor(148.0924, device='cuda:0') (tensor(181.7621, device='cuda:0')) | > current_lr_0: 0.0001993011799713115 | > current_lr_1: 0.0001993011799713115 | > step_time: 0.4367 (0.44811372323469684) | > loader_time: 0.0076 (0.00698749946825432) --> EVAL PERFORMANCE | > avg_loader_time: 0.008497459787718008 (-5.677048589142024e-05) | > avg_step_time: 0.4559218984254649 (+0.011634087898361845) | > avg_loss_disc: 2.916231155395508 (+0.016761309664014234) | > avg_loss_disc_real_0: 0.2406132393319842 (-0.00372726207887622) | > avg_loss_disc_real_1: 0.24099179881979044 (-6.629222295650994e-05) | > avg_loss_disc_real_2: 0.2462198795147345 (-0.0014394563688358841) | > avg_loss_disc_real_3: 0.24556239893738652 (+0.0013491572208807479) | > avg_loss_disc_real_4: 0.24798361852135456 (+0.002612991442143081) | > avg_loss_disc_real_5: 0.24840382413125375 (+0.0025885858586136845) | > avg_loss_0: 2.916231155395508 (+0.016761309664014234) | > avg_grad_norm_0: tensor(6.1530, device='cuda:0') (+tensor(0.2453, device='cuda:0')) | > avg_loss_gen: 1.655425572059524 (-0.00326877412661708) | > avg_loss_kl: 2.609278057662534 (-0.041668475513726655) | > avg_loss_feat: 1.2108437259432296 (-0.011159256310530052) | > avg_loss_mel: 19.29311628744636 (+0.15371647687025813) | > avg_loss_duration: 1.7276055275554387 (-0.0029281596062888937) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.49626922607422 (+0.09469188099176051) | > avg_grad_norm_1: tensor(211.5025, device='cuda:0') (tensor(-19.2298, device='cuda:0'))  > EPOCH: 29/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:29:07)   --> TIME: 2025-02-25 08:29:13 -- STEP: 11/72 -- GLOBAL_STEP: 1002100 | > loss_disc: 2.8044023513793945 (2.8665858182040127) | > loss_disc_real_0: 0.19525587558746338 (0.23262278464707462) | > loss_disc_real_1: 0.2704111635684967 (0.24143631078980185) | > loss_disc_real_2: 0.2375972718000412 (0.2398239726370031) | > loss_disc_real_3: 0.24933883547782898 (0.23597031154415823) | > loss_disc_real_4: 0.2570349872112274 (0.2354002743959427) | > loss_disc_real_5: 0.2277580052614212 (0.24056207456372) | > loss_0: 2.8044023513793945 (2.8665858182040127) | > grad_norm_0: tensor(5.0177, device='cuda:0') (tensor(6.4595, device='cuda:0')) | > loss_gen: 1.69826078414917 (1.6630829247561367) | > loss_kl: 2.561954975128174 (2.5437773140993984) | > loss_feat: 1.3805233240127563 (1.3183138804002241) | > loss_mel: 19.541391372680664 (19.84244138544256) | > loss_duration: 1.7258974313735962 (1.7264876040545376) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.90802764892578 (27.094103032892402) | > grad_norm_1: tensor(323.5125, device='cuda:0') (tensor(205.3059, device='cuda:0')) | > current_lr_0: 0.00019927626732381507 | > current_lr_1: 0.00019927626732381507 | > step_time: 0.4236 (0.4238014437935569) | > loader_time: 0.0062 (0.0062371384013782845)  --> TIME: 2025-02-25 08:29:36 -- STEP: 61/72 -- GLOBAL_STEP: 1002150 | > loss_disc: 2.961754322052002 (2.8894941142347994) | > loss_disc_real_0: 0.15305183827877045 (0.23720981034099078) | > loss_disc_real_1: 0.16426782310009003 (0.23510585163460404) | > loss_disc_real_2: 0.22591879963874817 (0.24480328989810632) | > loss_disc_real_3: 0.18942636251449585 (0.24150941801852868) | > loss_disc_real_4: 0.21312831342220306 (0.24316920242348655) | > loss_disc_real_5: 0.22157585620880127 (0.24319493672886833) | > loss_0: 2.961754322052002 (2.8894941142347994) | > grad_norm_0: tensor(15.1715, device='cuda:0') (tensor(6.0190, device='cuda:0')) | > loss_gen: 1.4736782312393188 (1.6540922375976062) | > loss_kl: 2.588895797729492 (2.545302051012633) | > loss_feat: 1.4081127643585205 (1.2811126992350723) | > loss_mel: 17.904130935668945 (18.973327167698596) | > loss_duration: 1.7446707487106323 (1.7199998664074256) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.119489669799805 (26.17383390958192) | > grad_norm_1: tensor(419.5436, device='cuda:0') (tensor(252.9745, device='cuda:0')) | > current_lr_0: 0.00019927626732381507 | > current_lr_1: 0.00019927626732381507 | > step_time: 0.4553 (0.44164360155824756) | > loader_time: 0.0103 (0.008093755753313909) --> EVAL PERFORMANCE | > avg_loader_time: 0.008529233260893486 (+3.177347317547853e-05) | > avg_step_time: 0.447637913932263 (-0.00828398449320189) | > avg_loss_disc: 2.8930417752601727 (-0.023189380135335114) | > avg_loss_disc_real_0: 0.23991943685941294 (-0.0006938024725712533) | > avg_loss_disc_real_1: 0.2363490254526407 (-0.004642773367149744) | > avg_loss_disc_real_2: 0.2443354113001219 (-0.0018844682146126057) | > avg_loss_disc_real_3: 0.24259746137639165 (-0.0029649375609948625) | > avg_loss_disc_real_4: 0.24398707126227903 (-0.003996547259075528) | > avg_loss_disc_real_5: 0.24447126119909152 (-0.003932562932162237) | > avg_loss_0: 2.8930417752601727 (-0.023189380135335114) | > avg_grad_norm_0: tensor(6.6523, device='cuda:0') (+tensor(0.4992, device='cuda:0')) | > avg_loss_gen: 1.664705002811593 (+0.009279430752069162) | > avg_loss_kl: 2.5548539329582534 (-0.05442412470428071) | > avg_loss_feat: 1.2804931998252873 (+0.06964947388205767) | > avg_loss_mel: 19.107457093789545 (-0.18565919365681438) | > avg_loss_duration: 1.7235060225070362 (-0.0040995050484025075) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.331015170460017 (-0.16525405561420214) | > avg_grad_norm_1: tensor(253.0182, device='cuda:0') (+tensor(41.5157, device='cuda:0'))  > EPOCH: 30/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:29:43)   --> TIME: 2025-02-25 08:30:02 -- STEP: 39/72 -- GLOBAL_STEP: 1002200 | > loss_disc: 2.9051389694213867 (2.9101743392455273) | > loss_disc_real_0: 0.20175959169864655 (0.23791853701457) | > loss_disc_real_1: 0.17041468620300293 (0.238667294383049) | > loss_disc_real_2: 0.26307928562164307 (0.2494499786542012) | > loss_disc_real_3: 0.27091455459594727 (0.24691254435441432) | > loss_disc_real_4: 0.26733770966529846 (0.2487466656244718) | > loss_disc_real_5: 0.26704129576683044 (0.24769389858612648) | > loss_0: 2.9051389694213867 (2.9101743392455273) | > grad_norm_0: tensor(5.9875, device='cuda:0') (tensor(5.8237, device='cuda:0')) | > loss_gen: 1.633084774017334 (1.6568134686885736) | > loss_kl: 2.701265335083008 (2.5768408958728495) | > loss_feat: 1.2087936401367188 (1.250619414525154) | > loss_mel: 18.595983505249023 (19.05056200271998) | > loss_duration: 1.7110475301742554 (1.7184196068690374) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.850175857543945 (26.253255795209835) | > grad_norm_1: tensor(147.7195, device='cuda:0') (tensor(230.1922, device='cuda:0')) | > current_lr_0: 0.00019925135779039958 | > current_lr_1: 0.00019925135779039958 | > step_time: 0.4368 (0.4307803740868202) | > loader_time: 0.0081 (0.007277464255308494) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515730710096762 (-1.3502550796724508e-05) | > avg_step_time: 0.44476431188449056 (-0.0028736020477724278) | > avg_loss_disc: 2.903841475365867 (+0.01079970010569431) | > avg_loss_disc_real_0: 0.2404928488630644 (+0.0005734120036514634) | > avg_loss_disc_real_1: 0.24171439694686675 (+0.005365371494226057) | > avg_loss_disc_real_2: 0.2464573924810114 (+0.002121981180889493) | > avg_loss_disc_real_3: 0.24473163464539488 (+0.0021341732690032256) | > avg_loss_disc_real_4: 0.2472165606811013 (+0.0032294894188222567) | > avg_loss_disc_real_5: 0.24697495514238385 (+0.0025036939432923344) | > avg_loss_0: 2.903841475365867 (+0.01079970010569431) | > avg_grad_norm_0: tensor(6.3135, device='cuda:0') (tensor(-0.3388, device='cuda:0')) | > avg_loss_gen: 1.6598677467292464 (-0.0048372560823466415) | > avg_loss_kl: 2.5550847456488803 (+0.00023081269062696919) | > avg_loss_feat: 1.2457197971746956 (-0.0347734026505917) | > avg_loss_mel: 18.97614307134923 (-0.1313140224403142) | > avg_loss_duration: 1.727341161647313 (+0.0038351391402768886) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.16415668541277 (-0.16685848504724632) | > avg_grad_norm_1: tensor(245.1848, device='cuda:0') (tensor(-7.8333, device='cuda:0'))  > EPOCH: 31/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:30:19)   --> TIME: 2025-02-25 08:30:29 -- STEP: 17/72 -- GLOBAL_STEP: 1002250 | > loss_disc: 2.864018440246582 (2.9019783524905933) | > loss_disc_real_0: 0.29712992906570435 (0.24934719327618093) | > loss_disc_real_1: 0.2080416977405548 (0.23401911206105175) | > loss_disc_real_2: 0.2316843420267105 (0.24270843407687018) | > loss_disc_real_3: 0.23602695763111115 (0.2472345522221397) | > loss_disc_real_4: 0.24820038676261902 (0.24654424891752355) | > loss_disc_real_5: 0.2626597583293915 (0.24605477382155025) | > loss_0: 2.864018440246582 (2.9019783524905933) | > grad_norm_0: tensor(7.9901, device='cuda:0') (tensor(6.2393, device='cuda:0')) | > loss_gen: 1.6216422319412231 (1.6755483781590181) | > loss_kl: 2.676908254623413 (2.5828358005074894) | > loss_feat: 1.4776495695114136 (1.30576790080351) | > loss_mel: 20.867889404296875 (19.58941526973949) | > loss_duration: 1.733891248703003 (1.7103996767717249) | > amp_scaler: 128.0 (128.0) | > loss_1: 28.377981185913086 (26.863967110128964) | > grad_norm_1: tensor(443.6864, device='cuda:0') (tensor(221.4569, device='cuda:0')) | > current_lr_0: 0.00019922645137067577 | > current_lr_1: 0.00019922645137067577 | > step_time: 0.425 (0.4569481681374943) | > loader_time: 0.006 (0.006879834567799288)  --> TIME: 2025-02-25 08:30:52 -- STEP: 67/72 -- GLOBAL_STEP: 1002300 | > loss_disc: 2.9482715129852295 (2.903370501390144) | > loss_disc_real_0: 0.2792314291000366 (0.24132370125891558) | > loss_disc_real_1: 0.24953202903270721 (0.23955076198969313) | > loss_disc_real_2: 0.26623016595840454 (0.24525578653634483) | > loss_disc_real_3: 0.24527955055236816 (0.24463464564351894) | > loss_disc_real_4: 0.254688560962677 (0.24538969103969746) | > loss_disc_real_5: 0.2742651402950287 (0.24757257637692925) | > loss_0: 2.9482715129852295 (2.903370501390144) | > grad_norm_0: tensor(5.1398, device='cuda:0') (tensor(6.8617, device='cuda:0')) | > loss_gen: 1.4519479274749756 (1.6653758127297928) | > loss_kl: 2.39963960647583 (2.5770961846878278) | > loss_feat: 1.0688042640686035 (1.2812355836825586) | > loss_mel: 18.657459259033203 (19.282149272178536) | > loss_duration: 1.7560886144638062 (1.7200310052330814) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.333940505981445 (26.525887816699584) | > grad_norm_1: tensor(189.3038, device='cuda:0') (tensor(203.0663, device='cuda:0')) | > current_lr_0: 0.00019922645137067577 | > current_lr_1: 0.00019922645137067577 | > step_time: 0.4928 (0.4522637395716425) | > loader_time: 0.0117 (0.00831432840717373) --> EVAL PERFORMANCE | > avg_loader_time: 0.008490837795633668 (-2.4892914463094107e-05) | > avg_step_time: 0.45534938154086263 (+0.01058506965637207) | > avg_loss_disc: 2.906167181444839 (+0.0023257060789720896) | > avg_loss_disc_real_0: 0.24016203322041202 (-0.00033081564265238783) | > avg_loss_disc_real_1: 0.2399887868636091 (-0.0017256100832576393) | > avg_loss_disc_real_2: 0.2443715558505394 (-0.0020858366304719878) | > avg_loss_disc_real_3: 0.24354209005832672 (-0.0011895445870681587) | > avg_loss_disc_real_4: 0.24402057222077544 (-0.0031959884603258437) | > avg_loss_disc_real_5: 0.24675788845814456 (-0.0002170666842392921) | > avg_loss_0: 2.906167181444839 (+0.0023257060789720896) | > avg_grad_norm_0: tensor(7.0259, device='cuda:0') (+tensor(0.7124, device='cuda:0')) | > avg_loss_gen: 1.6610669183059477 (+0.0011991715767012412) | > avg_loss_kl: 2.5747504234313965 (+0.019665677782516156) | > avg_loss_feat: 1.271702481827266 (+0.025982684652570365) | > avg_loss_mel: 19.24200439453125 (+0.26586132318201905) | > avg_loss_duration: 1.721077364935002 (-0.006263796712311143) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.470601498241155 (+0.306444812828385) | > avg_grad_norm_1: tensor(201.8349, device='cuda:0') (tensor(-43.3500, device='cuda:0'))  > EPOCH: 32/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:30:56)   --> TIME: 2025-02-25 08:31:18 -- STEP: 45/72 -- GLOBAL_STEP: 1002350 | > loss_disc: 2.884291172027588 (2.9003218015034986) | > loss_disc_real_0: 0.2938843071460724 (0.23647078739272223) | > loss_disc_real_1: 0.2794573903083801 (0.23785293069150712) | > loss_disc_real_2: 0.2398110330104828 (0.24190349049038357) | > loss_disc_real_3: 0.23810052871704102 (0.2425958385070165) | > loss_disc_real_4: 0.2505849003791809 (0.2458759864171346) | > loss_disc_real_5: 0.24859175086021423 (0.24908506770928712) | > loss_0: 2.884291172027588 (2.9003218015034986) | > grad_norm_0: tensor(6.0207, device='cuda:0') (tensor(7.4678, device='cuda:0')) | > loss_gen: 1.6572412252426147 (1.652497861120436) | > loss_kl: 2.621734380722046 (2.615407429801093) | > loss_feat: 1.1476025581359863 (1.2692739990022448) | > loss_mel: 18.350601196289062 (19.220060687594945) | > loss_duration: 1.737242341041565 (1.7137410296334161) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.514421463012695 (26.470981004503038) | > grad_norm_1: tensor(98.5380, device='cuda:0') (tensor(250.6062, device='cuda:0')) | > current_lr_0: 0.00019920154806425444 | > current_lr_1: 0.00019920154806425444 | > step_time: 0.4684 (0.4415375656551785) | > loader_time: 0.0088 (0.007398870256212022) --> EVAL PERFORMANCE | > avg_loader_time: 0.008495404686726311 (+4.566891092643369e-06) | > avg_step_time: 0.4539651870727539 (-0.0013841944681087237) | > avg_loss_disc: 2.906656550689482 (+0.0004893692446428233) | > avg_loss_disc_real_0: 0.23717319482648877 (-0.002988838393923249) | > avg_loss_disc_real_1: 0.23805482622603297 (-0.0019339606375761431) | > avg_loss_disc_real_2: 0.243036741731872 (-0.001334814118667399) | > avg_loss_disc_real_3: 0.24332003001595887 (-0.00022206004236785537) | > avg_loss_disc_real_4: 0.24677020122467633 (+0.002749629003900883) | > avg_loss_disc_real_5: 0.24882020874762198 (+0.0020623202894774206) | > avg_loss_0: 2.906656550689482 (+0.0004893692446428233) | > avg_grad_norm_0: tensor(6.9867, device='cuda:0') (tensor(-0.0392, device='cuda:0')) | > avg_loss_gen: 1.6450922422006096 (-0.015974676105338093) | > avg_loss_kl: 2.568316624198162 (-0.006433799233234616) | > avg_loss_feat: 1.2207241721556221 (-0.05097830967164385) | > avg_loss_mel: 19.14258024726115 (-0.09942414727009918) | > avg_loss_duration: 1.7226826389070968 (+0.0016052739720948672) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.299395923883143 (-0.1712055743580123) | > avg_grad_norm_1: tensor(250.0830, device='cuda:0') (+tensor(48.2481, device='cuda:0'))  > EPOCH: 33/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:31:33)   --> TIME: 2025-02-25 08:31:44 -- STEP: 23/72 -- GLOBAL_STEP: 1002400 | > loss_disc: 2.908190965652466 (2.892306203427522) | > loss_disc_real_0: 0.14946052432060242 (0.24219107563080994) | > loss_disc_real_1: 0.2198607176542282 (0.24348034482935202) | > loss_disc_real_2: 0.25268834829330444 (0.24236279142939526) | > loss_disc_real_3: 0.23690499365329742 (0.2442789518314859) | > loss_disc_real_4: 0.25148075819015503 (0.2464315133250278) | > loss_disc_real_5: 0.2514364421367645 (0.24728670392347418) | > loss_0: 2.908190965652466 (2.892306203427522) | > grad_norm_0: tensor(11.6372, device='cuda:0') (tensor(8.0796, device='cuda:0')) | > loss_gen: 1.5768721103668213 (1.660927264586739) | > loss_kl: 2.687213659286499 (2.5258127813753872) | > loss_feat: 1.1658961772918701 (1.2670804106670877) | > loss_mel: 17.522146224975586 (18.779415794040847) | > loss_duration: 1.7199056148529053 (1.6979675085648247) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.672033309936523 (25.931203759234883) | > grad_norm_1: tensor(71.3032, device='cuda:0') (tensor(246.1433, device='cuda:0')) | > current_lr_0: 0.0001991766478707464 | > current_lr_1: 0.0001991766478707464 | > step_time: 0.4465 (0.4302411286727242) | > loader_time: 0.0074 (0.006673864696336829) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850578764794578 (+1.0382961219468356e-05) | > avg_step_time: 0.44912782521315026 (-0.0048373618596036505) | > avg_loss_disc: 2.894388229074612 (-0.012268321614870015) | > avg_loss_disc_real_0: 0.240359602259918 (+0.003186407433429239) | > avg_loss_disc_real_1: 0.24551359196783792 (+0.007458765741804957) | > avg_loss_disc_real_2: 0.244194003687778 (+0.001157261955905986) | > avg_loss_disc_real_3: 0.2437408602153751 (+0.00042083019941624444) | > avg_loss_disc_real_4: 0.24425759575736355 (-0.002512605467312773) | > avg_loss_disc_real_5: 0.24472065185996847 (-0.0040995568876535105) | > avg_loss_0: 2.894388229074612 (-0.012268321614870015) | > avg_grad_norm_0: tensor(7.2764, device='cuda:0') (+tensor(0.2897, device='cuda:0')) | > avg_loss_gen: 1.665026327254067 (+0.019934085053457506) | > avg_loss_kl: 2.6065539608539 (+0.03823733665573803) | > avg_loss_feat: 1.2614871187948848 (+0.04076294663926272) | > avg_loss_mel: 18.950428438858246 (-0.19215180840290458) | > avg_loss_duration: 1.716140500256713 (-0.0065421386503838885) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.199636190709935 (-0.09975973317320808) | > avg_grad_norm_1: tensor(256.9482, device='cuda:0') (+tensor(6.8652, device='cuda:0'))  > EPOCH: 34/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:32:09)   --> TIME: 2025-02-25 08:32:11 -- STEP: 1/72 -- GLOBAL_STEP: 1002450 | > loss_disc: 2.8423664569854736 (2.8423664569854736) | > loss_disc_real_0: 0.2607113718986511 (0.2607113718986511) | > loss_disc_real_1: 0.24790246784687042 (0.24790246784687042) | > loss_disc_real_2: 0.22919221222400665 (0.22919221222400665) | > loss_disc_real_3: 0.2247556746006012 (0.2247556746006012) | > loss_disc_real_4: 0.23735632002353668 (0.23735632002353668) | > loss_disc_real_5: 0.21411725878715515 (0.21411725878715515) | > loss_0: 2.8423664569854736 (2.8423664569854736) | > grad_norm_0: tensor(4.0799, device='cuda:0') (tensor(4.0799, device='cuda:0')) | > loss_gen: 1.6912994384765625 (1.6912994384765625) | > loss_kl: 2.400812864303589 (2.400812864303589) | > loss_feat: 1.2697324752807617 (1.2697324752807617) | > loss_mel: 18.468006134033203 (18.468006134033203) | > loss_duration: 1.746065378189087 (1.746065378189087) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.575916290283203 (25.575916290283203) | > grad_norm_1: tensor(128.0807, device='cuda:0') (tensor(128.0807, device='cuda:0')) | > current_lr_0: 0.00019915175078976256 | > current_lr_1: 0.00019915175078976256 | > step_time: 0.4235 (0.423541784286499) | > loader_time: 0.0084 (0.008402109146118164)  --> TIME: 2025-02-25 08:32:34 -- STEP: 51/72 -- GLOBAL_STEP: 1002500 | > loss_disc: 2.8109755516052246 (2.896084808835796) | > loss_disc_real_0: 0.26899591088294983 (0.24024933722673678) | > loss_disc_real_1: 0.19879592955112457 (0.24062372481121735) | > loss_disc_real_2: 0.25860342383384705 (0.24542982350377476) | > loss_disc_real_3: 0.2585229277610779 (0.2449799650440029) | > loss_disc_real_4: 0.2529723346233368 (0.24723587579586925) | > loss_disc_real_5: 0.256896048784256 (0.24644788895167558) | > loss_0: 2.8109755516052246 (2.896084808835796) | > grad_norm_0: tensor(5.4665, device='cuda:0') (tensor(7.7320, device='cuda:0')) | > loss_gen: 1.5834040641784668 (1.661729003868851) | > loss_kl: 2.558084726333618 (2.5358614267087445) | > loss_feat: 1.378963828086853 (1.2566610492911994) | > loss_mel: 19.313610076904297 (18.69888025171617) | > loss_duration: 1.7452231645584106 (1.7111848405763215) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.579286575317383 (25.864316641115675) | > grad_norm_1: tensor(137.3377, device='cuda:0') (tensor(203.6909, device='cuda:0')) | > current_lr_0: 0.00019915175078976256 | > current_lr_1: 0.00019915175078976256 | > step_time: 0.4383 (0.4441534070407643) | > loader_time: 0.0088 (0.007602149364995021) --> EVAL PERFORMANCE | > avg_loader_time: 0.008481761099587022 (-2.4026548358757616e-05) | > avg_step_time: 0.4527546251323861 (+0.003626799919235868) | > avg_loss_disc: 2.8963712235571633 (+0.001982994482551348) | > avg_loss_disc_real_0: 0.23736315315038387 (-0.002996449109534144) | > avg_loss_disc_real_1: 0.23939945895067402 (-0.006114133017163903) | > avg_loss_disc_real_2: 0.2442288572939349 (+3.4853606156903805e-05) | > avg_loss_disc_real_3: 0.24536814760993902 (+0.0016272873945639144) | > avg_loss_disc_real_4: 0.24759930225325302 (+0.0033417064958894682) | > avg_loss_disc_real_5: 0.24690124854235582 (+0.002180596682387348) | > avg_loss_0: 2.8963712235571633 (+0.001982994482551348) | > avg_grad_norm_0: tensor(7.3070, device='cuda:0') (+tensor(0.0306, device='cuda:0')) | > avg_loss_gen: 1.6587880547617522 (-0.0062382724923149) | > avg_loss_kl: 2.5481925984503517 (-0.05836136240354817) | > avg_loss_feat: 1.25940705101255 (-0.002080067782334938) | > avg_loss_mel: 18.852543602527028 (-0.09788483633121814) | > avg_loss_duration: 1.7170042336826594 (+0.000863733425946478) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.0359355228048 (-0.16370066790513604) | > avg_grad_norm_1: tensor(222.5524, device='cuda:0') (tensor(-34.3958, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1002521.pth  > EPOCH: 35/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:32:48)   --> TIME: 2025-02-25 08:33:03 -- STEP: 29/72 -- GLOBAL_STEP: 1002550 | > loss_disc: 2.856727123260498 (2.8876393910112053) | > loss_disc_real_0: 0.1809854507446289 (0.23966546099761438) | > loss_disc_real_1: 0.2111508846282959 (0.23144900079431205) | > loss_disc_real_2: 0.23800809681415558 (0.2441820964731019) | > loss_disc_real_3: 0.2589728534221649 (0.24373502977963152) | > loss_disc_real_4: 0.24753575026988983 (0.24467133756341605) | > loss_disc_real_5: 0.22881537675857544 (0.2447178204511774) | > loss_0: 2.856727123260498 (2.8876393910112053) | > grad_norm_0: tensor(6.6879, device='cuda:0') (tensor(8.2423, device='cuda:0')) | > loss_gen: 1.659881591796875 (1.6745629433927864) | > loss_kl: 2.5777201652526855 (2.5365766574596535) | > loss_feat: 1.4067702293395996 (1.3852744513544544) | > loss_mel: 18.895090103149414 (19.264241909158645) | > loss_duration: 1.6840174198150635 (1.6979748545021847) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.223480224609375 (26.558631041954303) | > grad_norm_1: tensor(319.6001, device='cuda:0') (tensor(286.3028, device='cuda:0')) | > current_lr_0: 0.00019912685682091382 | > current_lr_1: 0.00019912685682091382 | > step_time: 0.4345 (0.4327324505510001) | > loader_time: 0.0066 (0.006853580474853516) --> EVAL PERFORMANCE | > avg_loader_time: 0.008461932061423717 (-1.9829038163304835e-05) | > avg_step_time: 0.45157818727090326 (-0.0011764378614828597) | > avg_loss_disc: 2.900446885068652 (+0.004075661511488526) | > avg_loss_disc_real_0: 0.23851767958889544 (+0.0011545264385115772) | > avg_loss_disc_real_1: 0.23914111185241754 (-0.0002583470982564784) | > avg_loss_disc_real_2: 0.24482285241845628 (+0.0005939951245213793) | > avg_loss_disc_real_3: 0.24687115701151566 (+0.0015030094015766327) | > avg_loss_disc_real_4: 0.2470531551770761 (-0.0005461470761769227) | > avg_loss_disc_real_5: 0.24610370565468156 (-0.0007975428876742519) | > avg_loss_0: 2.900446885068652 (+0.004075661511488526) | > avg_grad_norm_0: tensor(7.7387, device='cuda:0') (+tensor(0.4317, device='cuda:0')) | > avg_loss_gen: 1.6797300896174474 (+0.020942034855695235) | > avg_loss_kl: 2.5411424334620087 (-0.0070501649883429884) | > avg_loss_feat: 1.3374154785989036 (+0.07800842758635373) | > avg_loss_mel: 18.930296803863957 (+0.07775320133692887) | > avg_loss_duration: 1.714193342437207 (-0.0028108912454525292) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.202778265509803 (+0.1668427427050041) | > avg_grad_norm_1: tensor(269.7105, device='cuda:0') (+tensor(47.1581, device='cuda:0'))  > EPOCH: 36/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:33:25)   --> TIME: 2025-02-25 08:33:29 -- STEP: 7/72 -- GLOBAL_STEP: 1002600 | > loss_disc: 2.8667664527893066 (2.88627028465271) | > loss_disc_real_0: 0.2500670552253723 (0.2367938416344779) | > loss_disc_real_1: 0.2654327154159546 (0.24287656588213785) | > loss_disc_real_2: 0.29045844078063965 (0.24687757236616953) | > loss_disc_real_3: 0.26566082239151 (0.24493798400674546) | > loss_disc_real_4: 0.27817636728286743 (0.2484007946082524) | > loss_disc_real_5: 0.274057000875473 (0.2513157916920526) | > loss_0: 2.8667664527893066 (2.88627028465271) | > grad_norm_0: tensor(3.3105, device='cuda:0') (tensor(5.4473, device='cuda:0')) | > loss_gen: 1.6583184003829956 (1.6439352887017387) | > loss_kl: 2.5192031860351562 (2.6230450017111644) | > loss_feat: 1.222636342048645 (1.2221744060516357) | > loss_mel: 18.13262176513672 (18.596065521240234) | > loss_duration: 1.7022522687911987 (1.7110803808484758) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.235031127929688 (25.796300615583146) | > grad_norm_1: tensor(231.2368, device='cuda:0') (tensor(160.4467, device='cuda:0')) | > current_lr_0: 0.0001991019659638112 | > current_lr_1: 0.0001991019659638112 | > step_time: 0.4312 (0.426018306187221) | > loader_time: 0.0062 (0.006353071757725307)  --> TIME: 2025-02-25 08:33:52 -- STEP: 57/72 -- GLOBAL_STEP: 1002650 | > loss_disc: 2.8935060501098633 (2.9016979117142525) | > loss_disc_real_0: 0.20483753085136414 (0.23610955084625043) | > loss_disc_real_1: 0.25361406803131104 (0.23755801691297898) | > loss_disc_real_2: 0.22584111988544464 (0.2445740033137171) | > loss_disc_real_3: 0.222314715385437 (0.24277185452611824) | > loss_disc_real_4: 0.22632993757724762 (0.24565634994130386) | > loss_disc_real_5: 0.2158605009317398 (0.2459621220304255) | > loss_0: 2.8935060501098633 (2.9016979117142525) | > grad_norm_0: tensor(4.2383, device='cuda:0') (tensor(6.5752, device='cuda:0')) | > loss_gen: 1.759027123451233 (1.643209783654464) | > loss_kl: 2.350792646408081 (2.591226862187971) | > loss_feat: 1.2179936170578003 (1.2464403991113628) | > loss_mel: 18.89056968688965 (19.03568903605144) | > loss_duration: 1.7459139823913574 (1.71345009092699) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.964298248291016 (26.230015905279863) | > grad_norm_1: tensor(192.7784, device='cuda:0') (tensor(161.7884, device='cuda:0')) | > current_lr_0: 0.0001991019659638112 | > current_lr_1: 0.0001991019659638112 | > step_time: 0.4649 (0.438385055776228) | > loader_time: 0.0099 (0.00777938073141533) --> EVAL PERFORMANCE | > avg_loader_time: 0.008423200795348262 (-3.8731266075455253e-05) | > avg_step_time: 0.44654263576991127 (-0.005035551500991997) | > avg_loss_disc: 2.8969989326638235 (-0.003447952404828314) | > avg_loss_disc_real_0: 0.23634897046525713 (-0.0021687091236383127) | > avg_loss_disc_real_1: 0.2396126647650356 (+0.00047155291261805066) | > avg_loss_disc_real_2: 0.24450705609690976 (-0.0003157963215465187) | > avg_loss_disc_real_3: 0.24280718031903387 (-0.004063976692481791) | > avg_loss_disc_real_4: 0.24537390953218433 (-0.0016792456448917747) | > avg_loss_disc_real_5: 0.24628156360606074 (+0.00017785795137917337) | > avg_loss_0: 2.8969989326638235 (-0.003447952404828314) | > avg_grad_norm_0: tensor(6.4573, device='cuda:0') (tensor(-1.2814, device='cuda:0')) | > avg_loss_gen: 1.6458558734034148 (-0.03387421621403264) | > avg_loss_kl: 2.5713299193852386 (+0.03018748592322984) | > avg_loss_feat: 1.2480308245605147 (-0.08938465403838891) | > avg_loss_mel: 18.981151016665184 (+0.050854212801226595) | > avg_loss_duration: 1.7153057531571725 (+0.0011124107199655953) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.161673102580327 (-0.04110516292947608) | > avg_grad_norm_1: tensor(187.0147, device='cuda:0') (tensor(-82.6958, device='cuda:0'))  > EPOCH: 37/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:34:01)   --> TIME: 2025-02-25 08:34:18 -- STEP: 35/72 -- GLOBAL_STEP: 1002700 | > loss_disc: 2.951052188873291 (2.899826431274414) | > loss_disc_real_0: 0.2397899031639099 (0.23907052448817662) | > loss_disc_real_1: 0.22351621091365814 (0.23524734846183232) | > loss_disc_real_2: 0.255282461643219 (0.2478327989578247) | > loss_disc_real_3: 0.25030407309532166 (0.24767263233661652) | > loss_disc_real_4: 0.25108450651168823 (0.2486804587500436) | > loss_disc_real_5: 0.24816501140594482 (0.24912603497505187) | > loss_0: 2.951052188873291 (2.899826431274414) | > grad_norm_0: tensor(3.5902, device='cuda:0') (tensor(8.8642, device='cuda:0')) | > loss_gen: 1.6397666931152344 (1.672728327342442) | > loss_kl: 2.8145251274108887 (2.5310484068734307) | > loss_feat: 1.199871301651001 (1.311248448916844) | > loss_mel: 19.887409210205078 (19.08282721383231) | > loss_duration: 1.7528481483459473 (1.6962104865482874) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.29442024230957 (26.29406291416713) | > grad_norm_1: tensor(363.2078, device='cuda:0') (tensor(254.1047, device='cuda:0')) | > current_lr_0: 0.0001990770782180657 | > current_lr_1: 0.0001990770782180657 | > step_time: 0.4369 (0.4269232954297747) | > loader_time: 0.008 (0.006995596204485212) --> EVAL PERFORMANCE | > avg_loader_time: 0.008485747055268623 (+6.25462599203614e-05) | > avg_step_time: 0.45271511480841836 (+0.006172479038507095) | > avg_loss_disc: 2.89765229359479 (+0.0006533609309666844) | > avg_loss_disc_real_0: 0.24032508088669308 (+0.003976110421435947) | > avg_loss_disc_real_1: 0.23709680454831727 (-0.0025158602167183186) | > avg_loss_disc_real_2: 0.24631255403370925 (+0.0018054979367994883) | > avg_loss_disc_real_3: 0.2459780412240767 (+0.003170860905042844) | > avg_loss_disc_real_4: 0.2477081558654006 (+0.002334246333216272) | > avg_loss_disc_real_5: 0.24925528466701508 (+0.0029737210609543374) | > avg_loss_0: 2.89765229359479 (+0.0006533609309666844) | > avg_grad_norm_0: tensor(8.3795, device='cuda:0') (+tensor(1.9222, device='cuda:0')) | > avg_loss_gen: 1.6659990330816994 (+0.020143159678284617) | > avg_loss_kl: 2.5085790090157962 (-0.06275091036944236) | > avg_loss_feat: 1.2586709915752141 (+0.010640167014699387) | > avg_loss_mel: 18.70788552727498 (-0.2732654893902051) | > avg_loss_duration: 1.70960293185543 (-0.005702821301742489) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.850737558284276 (-0.31093554429605064) | > avg_grad_norm_1: tensor(251.6705, device='cuda:0') (+tensor(64.6557, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1002737.pth  > EPOCH: 38/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:34:40)   --> TIME: 2025-02-25 08:34:47 -- STEP: 13/72 -- GLOBAL_STEP: 1002750 | > loss_disc: 3.0055484771728516 (2.8755079966325026) | > loss_disc_real_0: 0.22288353741168976 (0.2369491045291607) | > loss_disc_real_1: 0.2985437512397766 (0.23908131970809057) | > loss_disc_real_2: 0.2452249974012375 (0.24233025770920974) | > loss_disc_real_3: 0.21887502074241638 (0.24406903982162476) | > loss_disc_real_4: 0.2258436679840088 (0.24650141940667078) | > loss_disc_real_5: 0.2527886629104614 (0.23852456762240484) | > loss_0: 3.0055484771728516 (2.8755079966325026) | > grad_norm_0: tensor(5.4951, device='cuda:0') (tensor(8.0467, device='cuda:0')) | > loss_gen: 1.61788010597229 (1.6887958783369799) | > loss_kl: 2.606147289276123 (2.524373641380897) | > loss_feat: 1.1971420049667358 (1.3356104630690355) | > loss_mel: 18.116416931152344 (18.67556674663837) | > loss_duration: 1.6831932067871094 (1.706328951395475) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.220779418945312 (25.93067580003005) | > grad_norm_1: tensor(195.8675, device='cuda:0') (tensor(252.2846, device='cuda:0')) | > current_lr_0: 0.00019905219358328844 | > current_lr_1: 0.00019905219358328844 | > step_time: 0.4646 (0.4298483408414401) | > loader_time: 0.0058 (0.006454082635732798)  --> TIME: 2025-02-25 08:35:11 -- STEP: 63/72 -- GLOBAL_STEP: 1002800 | > loss_disc: 2.9186460971832275 (2.887507052648635) | > loss_disc_real_0: 0.33474427461624146 (0.2384207873117356) | > loss_disc_real_1: 0.26759934425354004 (0.23606733008036537) | > loss_disc_real_2: 0.2595338225364685 (0.2444451820282709) | > loss_disc_real_3: 0.2544243037700653 (0.24451400647087704) | > loss_disc_real_4: 0.23641154170036316 (0.24526399539576638) | > loss_disc_real_5: 0.2637921869754791 (0.24631069198487296) | > loss_0: 2.9186460971832275 (2.887507052648635) | > grad_norm_0: tensor(8.5830, device='cuda:0') (tensor(7.2486, device='cuda:0')) | > loss_gen: 1.5917366743087769 (1.669511736385406) | > loss_kl: 2.6982381343841553 (2.5730634227631586) | > loss_feat: 1.1201635599136353 (1.299097646796514) | > loss_mel: 20.49252700805664 (18.709368569510325) | > loss_duration: 1.6882994174957275 (1.7059230236780076) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.59096336364746 (25.9569643111456) | > grad_norm_1: tensor(253.4525, device='cuda:0') (tensor(252.2742, device='cuda:0')) | > current_lr_0: 0.00019905219358328844 | > current_lr_1: 0.00019905219358328844 | > step_time: 0.4721 (0.4446299189612979) | > loader_time: 0.0114 (0.008347288010612368) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869715381676043 (+0.00021140676149180616) | > avg_step_time: 0.4499743253412381 (-0.00274078946718026) | > avg_loss_disc: 2.8915209904522965 (-0.00613130314249366) | > avg_loss_disc_real_0: 0.23723011390424112 (-0.0030949669824519577) | > avg_loss_disc_real_1: 0.23688898623829158 (-0.00020781831002569806) | > avg_loss_disc_real_2: 0.24504428787130705 (-0.001268266162402193) | > avg_loss_disc_real_3: 0.24510972138861536 (-0.0008683198354613453) | > avg_loss_disc_real_4: 0.24584866325620194 (-0.0018594926091986541) | > avg_loss_disc_real_5: 0.246394748628979 (-0.002860536038036071) | > avg_loss_0: 2.8915209904522965 (-0.00613130314249366) | > avg_grad_norm_0: tensor(7.1095, device='cuda:0') (tensor(-1.2700, device='cuda:0')) | > avg_loss_gen: 1.6675650485804383 (+0.0015660154987389152) | > avg_loss_kl: 2.584043287895095 (+0.07546427887929896) | > avg_loss_feat: 1.2865189599319244 (+0.02784796835671033) | > avg_loss_mel: 18.786716380589446 (+0.07883085331446793) | > avg_loss_duration: 1.7107877764903323 (+0.0011848446349023156) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.035631421586157 (+0.1848938633018804) | > avg_grad_norm_1: tensor(255.5769, device='cuda:0') (+tensor(3.9065, device='cuda:0'))  > EPOCH: 39/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:35:16)   --> TIME: 2025-02-25 08:35:37 -- STEP: 41/72 -- GLOBAL_STEP: 1002850 | > loss_disc: 3.0008182525634766 (2.885454154596096) | > loss_disc_real_0: 0.17912451922893524 (0.23796710335626836) | > loss_disc_real_1: 0.16504396498203278 (0.23464490582303302) | > loss_disc_real_2: 0.24343591928482056 (0.2430060600362173) | > loss_disc_real_3: 0.2568128705024719 (0.24328307662068344) | > loss_disc_real_4: 0.25048819184303284 (0.245180285195025) | > loss_disc_real_5: 0.21473179757595062 (0.24634405316376104) | > loss_0: 3.0008182525634766 (2.885454154596096) | > grad_norm_0: tensor(12.6568, device='cuda:0') (tensor(8.0668, device='cuda:0')) | > loss_gen: 1.5556846857070923 (1.6768495774850614) | > loss_kl: 2.6604721546173096 (2.6116410639227885) | > loss_feat: 1.278821587562561 (1.3875658046908494) | > loss_mel: 19.452062606811523 (19.160874669144793) | > loss_duration: 1.6958014965057373 (1.6975565392796585) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.64284324645996 (26.534487840605944) | > grad_norm_1: tensor(52.8584, device='cuda:0') (tensor(214.4780, device='cuda:0')) | > current_lr_0: 0.0001990273120590905 | > current_lr_1: 0.0001990273120590905 | > step_time: 0.4469 (0.4359679919917409) | > loader_time: 0.009 (0.007419440804458246) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591799668862784 (-0.00010535414789764573) | > avg_step_time: 0.45192300433843907 (+0.00194867899720097) | > avg_loss_disc: 2.8873597937570485 (-0.0041611966952479484) | > avg_loss_disc_real_0: 0.23185359676119308 (-0.005376517143048043) | > avg_loss_disc_real_1: 0.23730705468587474 (+0.00041806844758315864) | > avg_loss_disc_real_2: 0.24346858619804113 (-0.001575701673265928) | > avg_loss_disc_real_3: 0.24255247561024948 (-0.0025572457783658853) | > avg_loss_disc_real_4: 0.24492915247527647 (-0.0009195107809254754) | > avg_loss_disc_real_5: 0.2465752172218242 (+0.00018046859284520655) | > avg_loss_0: 2.8873597937570485 (-0.0041611966952479484) | > avg_grad_norm_0: tensor(8.1866, device='cuda:0') (+tensor(1.0771, device='cuda:0')) | > avg_loss_gen: 1.666193728715601 (-0.0013713198648372948) | > avg_loss_kl: 2.5705034564918194 (-0.013539831403275748) | > avg_loss_feat: 1.3344925339792815 (+0.047973574047357026) | > avg_loss_mel: 18.96608290873782 (+0.17936652814837473) | > avg_loss_duration: 1.7086140018113902 (-0.0021737746789420864) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.245886816105372 (+0.2102553945192156) | > avg_grad_norm_1: tensor(222.7237, device='cuda:0') (tensor(-32.8533, device='cuda:0'))  > EPOCH: 40/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:35:53)   --> TIME: 2025-02-25 08:36:03 -- STEP: 19/72 -- GLOBAL_STEP: 1002900 | > loss_disc: 2.9154891967773438 (2.89368704745644) | > loss_disc_real_0: 0.28646570444107056 (0.23665664854802584) | > loss_disc_real_1: 0.21670906245708466 (0.2365753399698358) | > loss_disc_real_2: 0.25536462664604187 (0.23817101669938942) | > loss_disc_real_3: 0.23781736195087433 (0.24707183555552834) | > loss_disc_real_4: 0.24804139137268066 (0.24843331779304303) | > loss_disc_real_5: 0.26004424691200256 (0.2493541021096079) | > loss_0: 2.9154891967773438 (2.89368704745644) | > grad_norm_0: tensor(6.5301, device='cuda:0') (tensor(9.3092, device='cuda:0')) | > loss_gen: 1.6064261198043823 (1.6481602819342362) | > loss_kl: 2.652101755142212 (2.5585928841641077) | > loss_feat: 1.0612691640853882 (1.281225091532657) | > loss_mel: 18.801801681518555 (18.777525751214277) | > loss_duration: 1.6822309494018555 (1.6921884197937815) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.803829193115234 (25.957692196494655) | > grad_norm_1: tensor(213.9314, device='cuda:0') (tensor(201.4572, device='cuda:0')) | > current_lr_0: 0.00019900243364508313 | > current_lr_1: 0.00019900243364508313 | > step_time: 0.4238 (0.42608691516675445) | > loader_time: 0.0078 (0.006770071230436626)  --> TIME: 2025-02-25 08:36:27 -- STEP: 69/72 -- GLOBAL_STEP: 1002950 | > loss_disc: 2.9397640228271484 (2.8920901409093887) | > loss_disc_real_0: 0.31501972675323486 (0.23591444004273068) | > loss_disc_real_1: 0.18162667751312256 (0.23790693391060483) | > loss_disc_real_2: 0.21184244751930237 (0.24180181367673736) | > loss_disc_real_3: 0.22006931900978088 (0.24440574473229007) | > loss_disc_real_4: 0.21064414083957672 (0.24764852960040604) | > loss_disc_real_5: 0.2073180079460144 (0.24803654309632123) | > loss_0: 2.9397640228271484 (2.8920901409093887) | > grad_norm_0: tensor(12.2510, device='cuda:0') (tensor(8.2266, device='cuda:0')) | > loss_gen: 1.6142956018447876 (1.6629310483517852) | > loss_kl: 2.522202730178833 (2.5246390708978623) | > loss_feat: 1.4210301637649536 (1.3104851781458093) | > loss_mel: 18.670785903930664 (18.81560430664947) | > loss_duration: 1.7754371166229248 (1.7090267288512078) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.003751754760742 (26.022686585136082) | > grad_norm_1: tensor(72.7598, device='cuda:0') (tensor(199.9034, device='cuda:0')) | > current_lr_0: 0.00019900243364508313 | > current_lr_1: 0.00019900243364508313 | > step_time: 0.5057 (0.4564408495806266) | > loader_time: 0.0127 (0.008519887924194336) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573874621324137 (-1.7925047538646863e-05) | > avg_step_time: 0.45765567833269144 (+0.0057326739942523686) | > avg_loss_disc: 2.8925361969101595 (+0.0051764031531109644) | > avg_loss_disc_real_0: 0.23482853551985514 (+0.00297493875866206) | > avg_loss_disc_real_1: 0.23980808404969498 (+0.0025010293638202397) | > avg_loss_disc_real_2: 0.24167298661990905 (-0.0017955995781320788) | > avg_loss_disc_real_3: 0.2451263775288219 (+0.00257390191857243) | > avg_loss_disc_real_4: 0.24764981005393283 (+0.0027206575786563603) | > avg_loss_disc_real_5: 0.24815177413779246 (+0.0015765569159682524) | > avg_loss_0: 2.8925361969101595 (+0.0051764031531109644) | > avg_grad_norm_0: tensor(8.2969, device='cuda:0') (+tensor(0.1103, device='cuda:0')) | > avg_loss_gen: 1.6632831096649168 (-0.0029106190506842378) | > avg_loss_kl: 2.5255965582081967 (-0.04490689828362271) | > avg_loss_feat: 1.3134511927483785 (-0.021041341230902955) | > avg_loss_mel: 18.828214161832566 (-0.13786874690525508) | > avg_loss_duration: 1.7089055541535498 (+0.0002915523421596067) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.039450846927267 (-0.2064359691781057) | > avg_grad_norm_1: tensor(208.6986, device='cuda:0') (tensor(-14.0250, device='cuda:0'))  > EPOCH: 41/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:36:30)   --> TIME: 2025-02-25 08:36:53 -- STEP: 47/72 -- GLOBAL_STEP: 1003000 | > loss_disc: 2.947558879852295 (2.8962346239292875) | > loss_disc_real_0: 0.12309382855892181 (0.23628902038995256) | > loss_disc_real_1: 0.24981635808944702 (0.2383508948569602) | > loss_disc_real_2: 0.18802505731582642 (0.24026400231300515) | > loss_disc_real_3: 0.19522930681705475 (0.24284774382063684) | > loss_disc_real_4: 0.18729637563228607 (0.2448044390754497) | > loss_disc_real_5: 0.1825147420167923 (0.24488963250150073) | > loss_0: 2.947558879852295 (2.8962346239292875) | > grad_norm_0: tensor(12.9737, device='cuda:0') (tensor(8.6104, device='cuda:0')) | > loss_gen: 1.5444605350494385 (1.6541968685515382) | > loss_kl: 2.5974507331848145 (2.477771449596324) | > loss_feat: 1.471407175064087 (1.3506589519216663) | > loss_mel: 19.19529151916504 (18.82419456319606) | > loss_duration: 1.7301366329193115 (1.6973669249960717) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.538745880126953 (26.00418882167086) | > grad_norm_1: tensor(227.3046, device='cuda:0') (tensor(223.7910, device='cuda:0')) | > current_lr_0: 0.0001989775583408775 | > current_lr_1: 0.0001989775583408775 | > step_time: 0.4502 (0.436750295314383) | > loader_time: 0.0088 (0.007418632507324219) --> EVAL PERFORMANCE | > avg_loader_time: 0.008436605963908453 (-0.0001372686574156838) | > avg_step_time: 0.448800825736892 (-0.008854852595799434) | > avg_loss_disc: 2.8936255246820584 (+0.0010893277718988514) | > avg_loss_disc_real_0: 0.2343843429231308 (-0.0004441925967243465) | > avg_loss_disc_real_1: 0.23899204945060568 (-0.0008160345990892992) | > avg_loss_disc_real_2: 0.24182242071124868 (+0.00014943409133963415) | > avg_loss_disc_real_3: 0.24301569793425815 (-0.0021106795945637635) | > avg_loss_disc_real_4: 0.24545157396457565 (-0.0021982360893571806) | > avg_loss_disc_real_5: 0.24605546555888486 (-0.0020963085789076075) | > avg_loss_0: 2.8936255246820584 (+0.0010893277718988514) | > avg_grad_norm_0: tensor(8.7471, device='cuda:0') (+tensor(0.4502, device='cuda:0')) | > avg_loss_gen: 1.657123522019722 (-0.006159587645194753) | > avg_loss_kl: 2.471911238952422 (-0.05368531925577491) | > avg_loss_feat: 1.324663667611673 (+0.011212474863294464) | > avg_loss_mel: 18.79912508037729 (-0.029089081455275334) | > avg_loss_duration: 1.7054581625360838 (-0.003447391617465989) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.958281772237427 (-0.08116907468983925) | > avg_grad_norm_1: tensor(229.2313, device='cuda:0') (+tensor(20.5327, device='cuda:0'))  > EPOCH: 42/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:37:06)   --> TIME: 2025-02-25 08:37:19 -- STEP: 25/72 -- GLOBAL_STEP: 1003050 | > loss_disc: 2.818114757537842 (2.904469861984253) | > loss_disc_real_0: 0.23891188204288483 (0.2417030704021454) | > loss_disc_real_1: 0.1539781242609024 (0.23532809734344481) | > loss_disc_real_2: 0.18976281583309174 (0.24730103552341462) | > loss_disc_real_3: 0.23741480708122253 (0.25005557358264924) | > loss_disc_real_4: 0.2372739464044571 (0.2505961400270462) | > loss_disc_real_5: 0.27071112394332886 (0.25353073596954345) | > loss_0: 2.818114757537842 (2.904469861984253) | > grad_norm_0: tensor(12.9953, device='cuda:0') (tensor(9.4887, device='cuda:0')) | > loss_gen: 1.5268961191177368 (1.667661828994751) | > loss_kl: 2.7083396911621094 (2.5137719440460207) | > loss_feat: 1.5528819561004639 (1.312841796875) | > loss_mel: 20.044565200805664 (18.794414749145506) | > loss_duration: 1.6597567796707153 (1.691067295074463) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.49243927001953 (25.97975730895996) | > grad_norm_1: tensor(575.3130, device='cuda:0') (tensor(238.0475, device='cuda:0')) | > current_lr_0: 0.00019895268614608487 | > current_lr_1: 0.00019895268614608487 | > step_time: 0.4436 (0.43138593673706055) | > loader_time: 0.007 (0.006910915374755859) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584868740028059 (+0.00014826277611960578) | > avg_step_time: 0.45127621502943444 (+0.0024753892925424292) | > avg_loss_disc: 2.9084765978262452 (+0.014851073144186877) | > avg_loss_disc_real_0: 0.23877461726816607 (+0.004390274345035278) | > avg_loss_disc_real_1: 0.2395874211485957 (+0.0005953716979900103) | > avg_loss_disc_real_2: 0.24465073372276736 (+0.002828313011518674) | > avg_loss_disc_real_3: 0.2466773443238836 (+0.0036616463896254614) | > avg_loss_disc_real_4: 0.24657023078958754 (+0.0011186568250118878) | > avg_loss_disc_real_5: 0.24840165066047454 (+0.0023461851015896817) | > avg_loss_0: 2.9084765978262452 (+0.014851073144186877) | > avg_grad_norm_0: tensor(9.4743, device='cuda:0') (+tensor(0.7272, device='cuda:0')) | > avg_loss_gen: 1.6820978362795334 (+0.024974314259811337) | > avg_loss_kl: 2.548582647887754 (+0.07667140893533197) | > avg_loss_feat: 1.3363836869387558 (+0.011720019327082776) | > avg_loss_mel: 18.81923457938181 (+0.020109499004519193) | > avg_loss_duration: 1.7128654785559212 (+0.007407316019837351) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.099164103118465 (+0.14088233088103763) | > avg_grad_norm_1: tensor(247.8350, device='cuda:0') (+tensor(18.6037, device='cuda:0'))  > EPOCH: 43/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:37:43)   --> TIME: 2025-02-25 08:37:46 -- STEP: 3/72 -- GLOBAL_STEP: 1003100 | > loss_disc: 2.9403505325317383 (2.96921706199646) | > loss_disc_real_0: 0.23142710328102112 (0.21864000459512076) | > loss_disc_real_1: 0.1648915708065033 (0.1924086461464564) | > loss_disc_real_2: 0.22881264984607697 (0.24120193223158518) | > loss_disc_real_3: 0.2887676954269409 (0.26154786845048267) | > loss_disc_real_4: 0.24470873177051544 (0.24889853099981943) | > loss_disc_real_5: 0.2556806206703186 (0.26012086868286133) | > loss_0: 2.9403505325317383 (2.96921706199646) | > grad_norm_0: tensor(8.4137, device='cuda:0') (tensor(9.3301, device='cuda:0')) | > loss_gen: 1.7896199226379395 (1.6127580006917317) | > loss_kl: 2.411421298980713 (2.50573468208313) | > loss_feat: 1.0142271518707275 (1.2310385306676228) | > loss_mel: 18.20500946044922 (18.530005772908527) | > loss_duration: 1.7441813945770264 (1.731704592704773) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.164457321166992 (25.611241658528645) | > grad_norm_1: tensor(281.9260, device='cuda:0') (tensor(366.4611, device='cuda:0')) | > current_lr_0: 0.0001989278170603166 | > current_lr_1: 0.0001989278170603166 | > step_time: 0.4426 (0.5832704703013102) | > loader_time: 0.0057 (0.006773630777994792)  --> TIME: 2025-02-25 08:38:09 -- STEP: 53/72 -- GLOBAL_STEP: 1003150 | > loss_disc: 2.940631151199341 (2.893475393079362) | > loss_disc_real_0: 0.20699578523635864 (0.2280019348803556) | > loss_disc_real_1: 0.32910919189453125 (0.2376335159225284) | > loss_disc_real_2: 0.2192654013633728 (0.24386557205668036) | > loss_disc_real_3: 0.26762646436691284 (0.2444103275829891) | > loss_disc_real_4: 0.2668669521808624 (0.24631401587207363) | > loss_disc_real_5: 0.26010364294052124 (0.2473062532690336) | > loss_0: 2.940631151199341 (2.893475393079362) | > grad_norm_0: tensor(7.9519, device='cuda:0') (tensor(8.3712, device='cuda:0')) | > loss_gen: 1.6991814374923706 (1.6589474273177813) | > loss_kl: 2.634423017501831 (2.4799524658131156) | > loss_feat: 1.184514045715332 (1.3163384064188544) | > loss_mel: 18.046506881713867 (18.531624344159972) | > loss_duration: 1.7598135471343994 (1.70070865694082) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.324438095092773 (25.687571039739645) | > grad_norm_1: tensor(137.1463, device='cuda:0') (tensor(223.7863, device='cuda:0')) | > current_lr_0: 0.0001989278170603166 | > current_lr_1: 0.0001989278170603166 | > step_time: 0.4484 (0.45179431393461406) | > loader_time: 0.0094 (0.007812814892462964) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599875678478835 (+1.5006938450776858e-05) | > avg_step_time: 0.4600253071583493 (+0.008749092128914837) | > avg_loss_disc: 2.8978457786667513 (-0.010630819159493932) | > avg_loss_disc_real_0: 0.2274057847303404 (-0.011368832537825668) | > avg_loss_disc_real_1: 0.23900813032203996 (-0.0005792908265557228) | > avg_loss_disc_real_2: 0.24399467924950827 (-0.0006560544732590878) | > avg_loss_disc_real_3: 0.2461491860974003 (-0.0005281582264833051) | > avg_loss_disc_real_4: 0.2471614660511554 (+0.000591235261567874) | > avg_loss_disc_real_5: 0.24841774895157612 (+1.6098291101579454e-05) | > avg_loss_0: 2.8978457786667513 (-0.010630819159493932) | > avg_grad_norm_0: tensor(8.5173, device='cuda:0') (tensor(-0.9571, device='cuda:0')) | > avg_loss_gen: 1.6583092867488591 (-0.02378854953067422) | > avg_loss_kl: 2.4748548689022876 (-0.07372777898546623) | > avg_loss_feat: 1.3061906032159296 (-0.03019308372282614) | > avg_loss_mel: 18.540560735783107 (-0.2786738435987033) | > avg_loss_duration: 1.7041007582570467 (-0.00876472029887454) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.68401607996981 (-0.4151480231486566) | > avg_grad_norm_1: tensor(255.6069, device='cuda:0') (+tensor(7.7719, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1003169.pth  > EPOCH: 44/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:38:22)   --> TIME: 2025-02-25 08:38:38 -- STEP: 31/72 -- GLOBAL_STEP: 1003200 | > loss_disc: 2.939303398132324 (2.875089914568009) | > loss_disc_real_0: 0.30073031783103943 (0.24000662613299587) | > loss_disc_real_1: 0.2653368413448334 (0.23572373678607325) | > loss_disc_real_2: 0.31895485520362854 (0.2417568737460721) | > loss_disc_real_3: 0.2817632257938385 (0.245354222193841) | > loss_disc_real_4: 0.2741537392139435 (0.24746703236333786) | > loss_disc_real_5: 0.2894754111766815 (0.24618300555213804) | > loss_0: 2.939303398132324 (2.875089914568009) | > grad_norm_0: tensor(10.1559, device='cuda:0') (tensor(7.5570, device='cuda:0')) | > loss_gen: 1.735785722732544 (1.6912942624861194) | > loss_kl: 2.784484624862671 (2.5395240091508438) | > loss_feat: 1.226628065109253 (1.3979881463512298) | > loss_mel: 18.46434783935547 (18.780331642396987) | > loss_duration: 1.7027747631072998 (1.6862470988304383) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.914020538330078 (26.0953851515247) | > grad_norm_1: tensor(86.6828, device='cuda:0') (tensor(228.9849, device='cuda:0')) | > current_lr_0: 0.00019890295108318404 | > current_lr_1: 0.00019890295108318404 | > step_time: 0.4405 (0.43141011268861834) | > loader_time: 0.0085 (0.00695816163093813) --> EVAL PERFORMANCE | > avg_loader_time: 0.008512527170315596 (-8.73485081632399e-05) | > avg_step_time: 0.44880166859693926 (-0.01122363856141001) | > avg_loss_disc: 2.8766211086595557 (-0.021224670007195634) | > avg_loss_disc_real_0: 0.2340098608967284 (+0.006604076166388001) | > avg_loss_disc_real_1: 0.23750455635534207 (-0.0015035739666978964) | > avg_loss_disc_real_2: 0.24252837854371945 (-0.00146630070578882) | > avg_loss_disc_real_3: 0.24340595745704544 (-0.0027432286403548667) | > avg_loss_disc_real_4: 0.24628941064149562 (-0.0008720554096597888) | > avg_loss_disc_real_5: 0.24574191847317656 (-0.00267583047839956) | > avg_loss_0: 2.8766211086595557 (-0.021224670007195634) | > avg_grad_norm_0: tensor(8.1585, device='cuda:0') (tensor(-0.3587, device='cuda:0')) | > avg_loss_gen: 1.6774607124462935 (+0.019151425697434332) | > avg_loss_kl: 2.5258905686123274 (+0.05103569971003985) | > avg_loss_feat: 1.3680544386447313 (+0.061863835428801695) | > avg_loss_mel: 18.58660174087739 (+0.04604100509428477) | > avg_loss_duration: 1.7027565811721372 (-0.0013441770849094148) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.860763979629734 (+0.17674789965992588) | > avg_grad_norm_1: tensor(268.7787, device='cuda:0') (+tensor(13.1718, device='cuda:0'))  > EPOCH: 45/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:38:59)   --> TIME: 2025-02-25 08:39:05 -- STEP: 9/72 -- GLOBAL_STEP: 1003250 | > loss_disc: 2.646784543991089 (2.8273055023617215) | > loss_disc_real_0: 0.1738201081752777 (0.2386423581176334) | > loss_disc_real_1: 0.16098329424858093 (0.2287102590004603) | > loss_disc_real_2: 0.1747227907180786 (0.24048001401954228) | > loss_disc_real_3: 0.20019173622131348 (0.23843086262543997) | > loss_disc_real_4: 0.20826944708824158 (0.23992736637592316) | > loss_disc_real_5: 0.21118971705436707 (0.23988738656044006) | > loss_0: 2.646784543991089 (2.8273055023617215) | > grad_norm_0: tensor(12.7252, device='cuda:0') (tensor(9.7021, device='cuda:0')) | > loss_gen: 1.6134250164031982 (1.6711890035205417) | > loss_kl: 2.6135306358337402 (2.445402834150526) | > loss_feat: 1.8814626932144165 (1.5001313553916082) | > loss_mel: 19.817413330078125 (18.74275694953071) | > loss_duration: 1.693155288696289 (1.7004101276397705) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.618986129760742 (26.059890535142685) | > grad_norm_1: tensor(56.3285, device='cuda:0') (tensor(244.2708, device='cuda:0')) | > current_lr_0: 0.00019887808821429862 | > current_lr_1: 0.00019887808821429862 | > step_time: 0.4187 (0.43324096997578937) | > loader_time: 0.0056 (0.0067484113905164934)  --> TIME: 2025-02-25 08:39:27 -- STEP: 59/72 -- GLOBAL_STEP: 1003300 | > loss_disc: 2.822441816329956 (2.890110419968427) | > loss_disc_real_0: 0.21276003122329712 (0.23551169770248867) | > loss_disc_real_1: 0.22396120429039001 (0.24144939320572353) | > loss_disc_real_2: 0.22922886908054352 (0.24237462751946207) | > loss_disc_real_3: 0.2114994078874588 (0.24607559280880428) | > loss_disc_real_4: 0.2254524976015091 (0.24736884135310933) | > loss_disc_real_5: 0.23838333785533905 (0.2485586046162298) | > loss_0: 2.822441816329956 (2.890110419968427) | > grad_norm_0: tensor(3.1128, device='cuda:0') (tensor(8.1617, device='cuda:0')) | > loss_gen: 1.663820743560791 (1.6878795462139582) | > loss_kl: 2.4285333156585693 (2.524227510064335) | > loss_feat: 1.3081238269805908 (1.3851286799220714) | > loss_mel: 17.74974822998047 (18.68600185846878) | > loss_duration: 1.7167314291000366 (1.700428063586607) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.86695671081543 (25.98366566027625) | > grad_norm_1: tensor(65.6288, device='cuda:0') (tensor(234.8575, device='cuda:0')) | > current_lr_0: 0.00019887808821429862 | > current_lr_1: 0.00019887808821429862 | > step_time: 0.4677 (0.44011697930804755) | > loader_time: 0.0096 (0.00790944745985128) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847877583033602 (-3.375133997957601e-05) | > avg_step_time: 0.44790293129397113 (-0.0008987373029681334) | > avg_loss_disc: 2.8830115459334684 (+0.006390437273912752) | > avg_loss_disc_real_0: 0.23274323344230652 (-0.0012666274544218814) | > avg_loss_disc_real_1: 0.24128652626359967 (+0.003781969908257604) | > avg_loss_disc_real_2: 0.24059803909818891 (-0.0019303394455305323) | > avg_loss_disc_real_3: 0.24422513221351194 (+0.0008191747564665064) | > avg_loss_disc_real_4: 0.24608030075758275 (-0.00020910988391287244) | > avg_loss_disc_real_5: 0.24622799456119537 (+0.0004860760880188164) | > avg_loss_0: 2.8830115459334684 (+0.006390437273912752) | > avg_grad_norm_0: tensor(8.0972, device='cuda:0') (tensor(-0.0613, device='cuda:0')) | > avg_loss_gen: 1.6842956610128914 (+0.006834948566597943) | > avg_loss_kl: 2.550172234924746 (+0.024281666312418793) | > avg_loss_feat: 1.3746284347184947 (+0.006573996073763411) | > avg_loss_mel: 18.744575204983562 (+0.15797346410617052) | > avg_loss_duration: 1.7050039969699486 (+0.002247415797811403) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.058675524214625 (+0.19791154458489046) | > avg_grad_norm_1: tensor(240.0144, device='cuda:0') (tensor(-28.7643, device='cuda:0'))  > EPOCH: 46/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:39:35)   --> TIME: 2025-02-25 08:39:54 -- STEP: 37/72 -- GLOBAL_STEP: 1003350 | > loss_disc: 2.8979668617248535 (2.8636864455970557) | > loss_disc_real_0: 0.2839125990867615 (0.2249577625377758) | > loss_disc_real_1: 0.26902487874031067 (0.24193409488007828) | > loss_disc_real_2: 0.24222590029239655 (0.24017002735589002) | > loss_disc_real_3: 0.265756756067276 (0.24762066755745862) | > loss_disc_real_4: 0.27062422037124634 (0.25031611766364126) | > loss_disc_real_5: 0.2820366621017456 (0.25236445786179723) | > loss_0: 2.8979668617248535 (2.8636864455970557) | > grad_norm_0: tensor(10.1637, device='cuda:0') (tensor(9.4545, device='cuda:0')) | > loss_gen: 1.6816964149475098 (1.714730591387362) | > loss_kl: 2.445549249649048 (2.649545328037159) | > loss_feat: 1.3527675867080688 (1.5032858848571777) | > loss_mel: 17.754079818725586 (18.91060478622849) | > loss_duration: 1.707586407661438 (1.6976449006312602) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.941679000854492 (26.47581131393845) | > grad_norm_1: tensor(78.7176, device='cuda:0') (tensor(201.2099, device='cuda:0')) | > current_lr_0: 0.00019885322845327182 | > current_lr_1: 0.00019885322845327182 | > step_time: 0.4433 (0.4478271136412749) | > loader_time: 0.0078 (0.007058826652733055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008485461624575332 (+6.685794239312537e-06) | > avg_step_time: 0.45794134072854487 (+0.010038409434573736) | > avg_loss_disc: 2.8597958625202446 (-0.023215683413223775) | > avg_loss_disc_real_0: 0.2283068943611333 (-0.004436339081173218) | > avg_loss_disc_real_1: 0.24022025709420863 (-0.0010662691693910376) | > avg_loss_disc_real_2: 0.24008131384010045 (-0.0005167252580884629) | > avg_loss_disc_real_3: 0.24536776374763167 (+0.0011426315341197257) | > avg_loss_disc_real_4: 0.24800000518140658 (+0.0019197044238238314) | > avg_loss_disc_real_5: 0.24843557997488638 (+0.00220758541369101) | > avg_loss_0: 2.8597958625202446 (-0.023215683413223775) | > avg_grad_norm_0: tensor(9.2330, device='cuda:0') (+tensor(1.1358, device='cuda:0')) | > avg_loss_gen: 1.7133887267448533 (+0.02909306573196191) | > avg_loss_kl: 2.5995849857867603 (+0.04941275086201413) | > avg_loss_feat: 1.49255612366636 (+0.11792768894786532) | > avg_loss_mel: 18.68272692720655 (-0.061848277777013294) | > avg_loss_duration: 1.709373244097535 (+0.004369247127586462) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.19763003604513 (+0.1389545118305051) | > avg_grad_norm_1: tensor(232.7051, device='cuda:0') (tensor(-7.3092, device='cuda:0'))  > EPOCH: 47/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:40:12)   --> TIME: 2025-02-25 08:40:21 -- STEP: 15/72 -- GLOBAL_STEP: 1003400 | > loss_disc: 2.8146872520446777 (2.8671706994374593) | > loss_disc_real_0: 0.29857194423675537 (0.2514100442330042) | > loss_disc_real_1: 0.21484600007534027 (0.23387620747089385) | > loss_disc_real_2: 0.25981175899505615 (0.24032157162825266) | > loss_disc_real_3: 0.2396831065416336 (0.2437069833278656) | > loss_disc_real_4: 0.253944456577301 (0.24540110528469086) | > loss_disc_real_5: 0.24574807286262512 (0.24585119783878326) | > loss_0: 2.8146872520446777 (2.8671706994374593) | > grad_norm_0: tensor(7.4521, device='cuda:0') (tensor(13.1856, device='cuda:0')) | > loss_gen: 1.6995728015899658 (1.7359413067499796) | > loss_kl: 2.767496109008789 (2.5406121095021565) | > loss_feat: 1.558987021446228 (1.5104121367136638) | > loss_mel: 20.031877517700195 (18.562950770060223) | > loss_duration: 1.6662287712097168 (1.690004579226176) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.724163055419922 (26.039921061197916) | > grad_norm_1: tensor(298.5754, device='cuda:0') (tensor(280.3304, device='cuda:0')) | > current_lr_0: 0.00019882837179971516 | > current_lr_1: 0.00019882837179971516 | > step_time: 0.4395 (0.43281795183817545) | > loader_time: 0.0073 (0.006944767634073893)  --> TIME: 2025-02-25 08:40:44 -- STEP: 65/72 -- GLOBAL_STEP: 1003450 | > loss_disc: 2.832529067993164 (2.862437835106483) | > loss_disc_real_0: 0.24491502344608307 (0.23718275061020486) | > loss_disc_real_1: 0.23842865228652954 (0.23644892470194742) | > loss_disc_real_2: 0.22397038340568542 (0.23727310552046849) | > loss_disc_real_3: 0.25183194875717163 (0.24233553340801825) | > loss_disc_real_4: 0.2477438747882843 (0.24501486397706546) | > loss_disc_real_5: 0.26136282086372375 (0.24488739187900838) | > loss_0: 2.832529067993164 (2.862437835106483) | > grad_norm_0: tensor(3.5607, device='cuda:0') (tensor(11.4007, device='cuda:0')) | > loss_gen: 1.7797825336456299 (1.700800035550044) | > loss_kl: 2.5843262672424316 (2.5420049850757307) | > loss_feat: 1.5713251829147339 (1.4675233969321617) | > loss_mel: 19.44862937927246 (18.411784685575046) | > loss_duration: 1.7542240619659424 (1.7000962587503285) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.138288497924805 (25.822209314199593) | > grad_norm_1: tensor(622.8371, device='cuda:0') (tensor(242.7848, device='cuda:0')) | > current_lr_0: 0.00019882837179971516 | > current_lr_1: 0.00019882837179971516 | > step_time: 0.488 (0.4448237822606013) | > loader_time: 0.0117 (0.008327520810640775) --> EVAL PERFORMANCE | > avg_loader_time: 0.008595124096937582 (+0.00010966247236224996) | > avg_step_time: 0.4489479602222711 (-0.00899338050627374) | > avg_loss_disc: 2.860755164858321 (+0.0009593023380762844) | > avg_loss_disc_real_0: 0.23551394842879872 (+0.007207054067665419) | > avg_loss_disc_real_1: 0.23666446236237673 (-0.0035557947318319016) | > avg_loss_disc_real_2: 0.23757033952524964 (-0.0025109743148508112) | > avg_loss_disc_real_3: 0.24223548992418906 (-0.0031322738234426106) | > avg_loss_disc_real_4: 0.24481920289321685 (-0.0031808022881897324) | > avg_loss_disc_real_5: 0.24499900177331038 (-0.0034365782015760016) | > avg_loss_0: 2.860755164858321 (+0.0009593023380762844) | > avg_grad_norm_0: tensor(10.9332, device='cuda:0') (+tensor(1.7002, device='cuda:0')) | > avg_loss_gen: 1.7033580508030637 (-0.010030675941789635) | > avg_loss_kl: 2.553265074609031 (-0.04631991117772927) | > avg_loss_feat: 1.4687315443871727 (-0.023824579279187352) | > avg_loss_mel: 18.482019988583847 (-0.20070693862270161) | > avg_loss_duration: 1.703241229057312 (-0.006132015040223093) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.91061586729238 (-0.28701416875275143) | > avg_grad_norm_1: tensor(240.4052, device='cuda:0') (+tensor(7.7001, device='cuda:0'))  > EPOCH: 48/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:40:49)   --> TIME: 2025-02-25 08:41:09 -- STEP: 43/72 -- GLOBAL_STEP: 1003500 | > loss_disc: 2.843982696533203 (2.864780758702478) | > loss_disc_real_0: 0.20399914681911469 (0.23195321546044462) | > loss_disc_real_1: 0.28321602940559387 (0.23843460616677306) | > loss_disc_real_2: 0.264424204826355 (0.2361950444620709) | > loss_disc_real_3: 0.25498080253601074 (0.24618076619713805) | > loss_disc_real_4: 0.23743127286434174 (0.24892677505349) | > loss_disc_real_5: 0.22532594203948975 (0.2489300938539727) | > loss_0: 2.843982696533203 (2.864780758702478) | > grad_norm_0: tensor(11.2623, device='cuda:0') (tensor(10.0008, device='cuda:0')) | > loss_gen: 1.7409377098083496 (1.6974506599958552) | > loss_kl: 2.48175311088562 (2.4807616056397905) | > loss_feat: 1.3873295783996582 (1.4586920849112575) | > loss_mel: 18.130325317382812 (18.616135619407476) | > loss_duration: 1.6956539154052734 (1.686986659848413) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.43600082397461 (25.940026615941246) | > grad_norm_1: tensor(262.3169, device='cuda:0') (tensor(185.7880, device='cuda:0')) | > current_lr_0: 0.00019880351825324018 | > current_lr_1: 0.00019880351825324018 | > step_time: 0.443 (0.4331376829812693) | > loader_time: 0.0093 (0.00730283870253452) --> EVAL PERFORMANCE | > avg_loader_time: 0.008456777518903705 (-0.00013834657803387697) | > avg_step_time: 0.4479703533817345 (-0.0009776068405366045) | > avg_loss_disc: 2.8676659322120774 (+0.006910767353756508) | > avg_loss_disc_real_0: 0.2313360979019756 (-0.004177850526823124) | > avg_loss_disc_real_1: 0.23902884440522798 (+0.0023643820428512485) | > avg_loss_disc_real_2: 0.23929029808078014 (+0.0017199585555305041) | > avg_loss_disc_real_3: 0.24726918318741758 (+0.005033693263228528) | > avg_loss_disc_real_4: 0.24841686516580447 (+0.0035976622725876206) | > avg_loss_disc_real_5: 0.24857499565876706 (+0.0035759938854566797) | > avg_loss_0: 2.8676659322120774 (+0.006910767353756508) | > avg_grad_norm_0: tensor(11.1977, device='cuda:0') (+tensor(0.2646, device='cuda:0')) | > avg_loss_gen: 1.705435872077942 (+0.002077821274878211) | > avg_loss_kl: 2.4976443270562405 (-0.05562074755279056) | > avg_loss_feat: 1.4733394837715257 (+0.004607939384352999) | > avg_loss_mel: 18.76886794936489 (+0.28684796078104213) | > avg_loss_duration: 1.712100087756842 (+0.008858858699529959) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.157387746891505 (+0.24677187959912672) | > avg_grad_norm_1: tensor(237.7227, device='cuda:0') (tensor(-2.6825, device='cuda:0'))  > EPOCH: 49/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:41:25)   --> TIME: 2025-02-25 08:41:36 -- STEP: 21/72 -- GLOBAL_STEP: 1003550 | > loss_disc: 2.8330655097961426 (2.85355696224031) | > loss_disc_real_0: 0.26272493600845337 (0.2119575118025144) | > loss_disc_real_1: 0.21884548664093018 (0.24097807421570733) | > loss_disc_real_2: 0.23027147352695465 (0.23494964838027954) | > loss_disc_real_3: 0.23666052520275116 (0.24286356426420666) | > loss_disc_real_4: 0.23979555070400238 (0.24558264371894656) | > loss_disc_real_5: 0.24807222187519073 (0.2452851796434039) | > loss_0: 2.8330655097961426 (2.85355696224031) | > grad_norm_0: tensor(6.9243, device='cuda:0') (tensor(10.8866, device='cuda:0')) | > loss_gen: 1.7839207649230957 (1.7015479746318998) | > loss_kl: 2.5677988529205322 (2.5302657853989374) | > loss_feat: 1.7102044820785522 (1.5602056980133057) | > loss_mel: 18.555932998657227 (18.46960213070824) | > loss_duration: 1.7208198308944702 (1.6875181936082386) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.33867645263672 (25.949139640444802) | > grad_norm_1: tensor(123.2232, device='cuda:0') (tensor(266.1277, device='cuda:0')) | > current_lr_0: 0.00019877866781345852 | > current_lr_1: 0.00019877866781345852 | > step_time: 0.4336 (0.4545227686564128) | > loader_time: 0.0065 (0.006838684990292504)  --> TIME: 2025-02-25 08:42:00 -- STEP: 71/72 -- GLOBAL_STEP: 1003600 | > loss_disc: 2.89853572845459 (2.8624046621188306) | > loss_disc_real_0: 0.16972503066062927 (0.2223449048232025) | > loss_disc_real_1: 0.2392481565475464 (0.23915575885436904) | > loss_disc_real_2: 0.21741829812526703 (0.24074578935831364) | > loss_disc_real_3: 0.2162218540906906 (0.2431833649605093) | > loss_disc_real_4: 0.21728208661079407 (0.24512076713669467) | > loss_disc_real_5: 0.21963876485824585 (0.24552265313309682) | > loss_0: 2.89853572845459 (2.8624046621188306) | > grad_norm_0: tensor(10.7640, device='cuda:0') (tensor(11.8857, device='cuda:0')) | > loss_gen: 1.6798393726348877 (1.6979806641457786) | > loss_kl: 2.7096710205078125 (2.467305028942269) | > loss_feat: 1.3938382863998413 (1.5230268323925178) | > loss_mel: 19.081186294555664 (18.26540487584934) | > loss_duration: 1.7335236072540283 (1.7007559655417859) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.598060607910156 (25.654473398772762) | > grad_norm_1: tensor(546.3431, device='cuda:0') (tensor(302.2272, device='cuda:0')) | > current_lr_0: 0.00019877866781345852 | > current_lr_1: 0.00019877866781345852 | > step_time: 0.4798 (0.4585583612952434) | > loader_time: 0.0084 (0.008793545440888742) --> EVAL PERFORMANCE | > avg_loader_time: 0.008793545440888742 (+0.0003367679219850369) | > avg_step_time: 0.4585583612952434 (+0.01058800791350889) | > avg_loss_disc: 2.8624046621188306 (-0.005261270093246839) | > avg_loss_disc_real_0: 0.2223449048232025 (-0.008991193078773108) | > avg_loss_disc_real_1: 0.23915575885436904 (+0.00012691444914106342) | > avg_loss_disc_real_2: 0.24074578935831364 (+0.0014554912775334994) | > avg_loss_disc_real_3: 0.2431833649605093 (-0.004085818226908289) | > avg_loss_disc_real_4: 0.24512076713669467 (-0.003296098029109795) | > avg_loss_disc_real_5: 0.24552265313309682 (-0.003052342525670243) | > avg_loss_0: 2.8624046621188306 (-0.005261270093246839) | > avg_grad_norm_0: tensor(11.8857, device='cuda:0') (+tensor(0.6880, device='cuda:0')) | > avg_loss_gen: 1.6979806641457786 (-0.007455207932163255) | > avg_loss_kl: 2.467305028942269 (-0.030339298113971402) | > avg_loss_feat: 1.5230268323925178 (+0.04968734862099211) | > avg_loss_mel: 18.26540487584934 (-0.5034630735155474) | > avg_loss_duration: 1.7007559655417859 (-0.011344122215056096) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.654473398772762 (-0.5029143481187432) | > avg_grad_norm_1: tensor(302.2272, device='cuda:0') (+tensor(64.5045, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1003601.pth  > EPOCH: 50/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:42:04)   --> TIME: 2025-02-25 08:42:28 -- STEP: 49/72 -- GLOBAL_STEP: 1003650 | > loss_disc: 2.8710122108459473 (2.862981450800993) | > loss_disc_real_0: 0.16540732979774475 (0.22800267989538153) | > loss_disc_real_1: 0.22782503068447113 (0.2380243822628138) | > loss_disc_real_2: 0.25439703464508057 (0.2379571734642496) | > loss_disc_real_3: 0.24070747196674347 (0.24563393696230285) | > loss_disc_real_4: 0.24992845952510834 (0.2474924514488298) | > loss_disc_real_5: 0.25572362542152405 (0.2479044685558397) | > loss_0: 2.8710122108459473 (2.862981450800993) | > grad_norm_0: tensor(11.7441, device='cuda:0') (tensor(9.4183, device='cuda:0')) | > loss_gen: 1.7307097911834717 (1.7087044204984392) | > loss_kl: 2.337610960006714 (2.5943098457492115) | > loss_feat: 1.4402676820755005 (1.5127015649055948) | > loss_mel: 18.44094467163086 (18.55779285820163) | > loss_duration: 1.7683517932891846 (1.6979069052910318) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.717884063720703 (26.071415492466517) | > grad_norm_1: tensor(273.9621, device='cuda:0') (tensor(271.0672, device='cuda:0')) | > current_lr_0: 0.00019875382047998183 | > current_lr_1: 0.00019875382047998183 | > step_time: 0.4355 (0.44118268635808205) | > loader_time: 0.0092 (0.0074847425733293805) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438026401358592 (-0.0003555190395301503) | > avg_step_time: 0.4514532727254948 (-0.007105088569748619) | > avg_loss_disc: 2.8556988440768825 (-0.006705818041948142) | > avg_loss_disc_real_0: 0.22597643355248678 (+0.0036315287292842946) | > avg_loss_disc_real_1: 0.23761422407459204 (-0.001541534779777004) | > avg_loss_disc_real_2: 0.23761921680309403 (-0.0031265725552196144) | > avg_loss_disc_real_3: 0.245493739213742 (+0.0023103742532327165) | > avg_loss_disc_real_4: 0.2472763376336702 (+0.0021555704969755396) | > avg_loss_disc_real_5: 0.24717335004202076 (+0.0016506969089239432) | > avg_loss_0: 2.8556988440768825 (-0.006705818041948142) | > avg_grad_norm_0: tensor(9.8929, device='cuda:0') (tensor(-1.9928, device='cuda:0')) | > avg_loss_gen: 1.7078065032690344 (+0.009825839123255742) | > avg_loss_kl: 2.591995823551231 (+0.12469079460896193) | > avg_loss_feat: 1.5046760951968985 (-0.018350737195619304) | > avg_loss_mel: 18.552727578391487 (+0.287322702542145) | > avg_loss_duration: 1.703164278621405 (+0.0024083130796190844) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.060370190042846 (+0.4058967912700844) | > avg_grad_norm_1: tensor(268.3074, device='cuda:0') (tensor(-33.9198, device='cuda:0'))  > EPOCH: 51/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:42:41)   --> TIME: 2025-02-25 08:42:55 -- STEP: 27/72 -- GLOBAL_STEP: 1003700 | > loss_disc: 2.876434803009033 (2.883258916713573) | > loss_disc_real_0: 0.3018462657928467 (0.23213826230278722) | > loss_disc_real_1: 0.3132675886154175 (0.23285501974600334) | > loss_disc_real_2: 0.23092719912528992 (0.24235519159723212) | > loss_disc_real_3: 0.23115064203739166 (0.24308387990351077) | > loss_disc_real_4: 0.2434183806180954 (0.24708536763985953) | > loss_disc_real_5: 0.25741031765937805 (0.24963231495133156) | > loss_0: 2.876434803009033 (2.883258916713573) | > grad_norm_0: tensor(12.3626, device='cuda:0') (tensor(11.1850, device='cuda:0')) | > loss_gen: 1.8585574626922607 (1.7092338932885065) | > loss_kl: 2.629019021987915 (2.4782949641898826) | > loss_feat: 1.5578551292419434 (1.6003786855273778) | > loss_mel: 18.226396560668945 (18.653272770069265) | > loss_duration: 1.6582531929016113 (1.6752867742821023) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.93008041381836 (26.116467157999676) | > grad_norm_1: tensor(105.6782, device='cuda:0') (tensor(270.3232, device='cuda:0')) | > current_lr_0: 0.00019872897625242182 | > current_lr_1: 0.00019872897625242182 | > step_time: 0.4375 (0.42607739236619735) | > loader_time: 0.0077 (0.006783229333383066) --> EVAL PERFORMANCE | > avg_loader_time: 0.008451522236138998 (+1.3495834780406779e-05) | > avg_step_time: 0.44719600005888605 (-0.004257272666608747) | > avg_loss_disc: 2.864859900004427 (+0.00916105592754457) | > avg_loss_disc_real_0: 0.22996641685005645 (+0.003989983297569666) | > avg_loss_disc_real_1: 0.2352440013012416 (-0.0023702227733504444) | > avg_loss_disc_real_2: 0.23638162184769 (-0.0012375949554040422) | > avg_loss_disc_real_3: 0.2443475022282399 (-0.0011462369855021193) | > avg_loss_disc_real_4: 0.24808857789341832 (+0.0008122402597481038) | > avg_loss_disc_real_5: 0.2494460100858984 (+0.0022726600438776334) | > avg_loss_0: 2.864859900004427 (+0.00916105592754457) | > avg_grad_norm_0: tensor(10.4022, device='cuda:0') (+tensor(0.5092, device='cuda:0')) | > avg_loss_gen: 1.7037857888450085 (-0.004020714424025851) | > avg_loss_kl: 2.5242335359815136 (-0.06776228756971747) | > avg_loss_feat: 1.548454284667969 (+0.04377818947107048) | > avg_loss_mel: 18.822659076099665 (+0.26993149770817837) | > avg_loss_duration: 1.7087561311856123 (+0.005591852564207356) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.307888863791884 (+0.24751867374903824) | > avg_grad_norm_1: tensor(237.9628, device='cuda:0') (tensor(-30.3447, device='cuda:0'))  > EPOCH: 52/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:43:17)   --> TIME: 2025-02-25 08:43:21 -- STEP: 5/72 -- GLOBAL_STEP: 1003750 | > loss_disc: 2.922886371612549 (2.8656113147735596) | > loss_disc_real_0: 0.25867772102355957 (0.23703727424144744) | > loss_disc_real_1: 0.18860110640525818 (0.22338390052318574) | > loss_disc_real_2: 0.2404131144285202 (0.21481280624866486) | > loss_disc_real_3: 0.27346184849739075 (0.22117140293121337) | > loss_disc_real_4: 0.26687008142471313 (0.21719287633895873) | > loss_disc_real_5: 0.2629414498806 (0.21666019260883332) | > loss_0: 2.922886371612549 (2.8656113147735596) | > grad_norm_0: tensor(14.4928, device='cuda:0') (tensor(15.7252, device='cuda:0')) | > loss_gen: 1.8617291450500488 (1.6512876749038696) | > loss_kl: 2.474952459335327 (2.4330774307250977) | > loss_feat: 1.3631813526153564 (1.6117158889770509) | > loss_mel: 17.737857818603516 (18.999121475219727) | > loss_duration: 1.659480333328247 (1.704973578453064) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.09720230102539 (26.40017623901367) | > grad_norm_1: tensor(58.7169, device='cuda:0') (tensor(138.4009, device='cuda:0')) | > current_lr_0: 0.00019870413513039026 | > current_lr_1: 0.00019870413513039026 | > step_time: 0.4391 (0.4278424739837646) | > loader_time: 0.0054 (0.0063037872314453125)  --> TIME: 2025-02-25 08:43:44 -- STEP: 55/72 -- GLOBAL_STEP: 1003800 | > loss_disc: 2.8232173919677734 (2.840140490098432) | > loss_disc_real_0: 0.21242257952690125 (0.22883179865100167) | > loss_disc_real_1: 0.23530729115009308 (0.23032048344612122) | > loss_disc_real_2: 0.2826930582523346 (0.23155391839417544) | > loss_disc_real_3: 0.27391359210014343 (0.24123704406348143) | > loss_disc_real_4: 0.29590296745300293 (0.2451066640290347) | > loss_disc_real_5: 0.31175658106803894 (0.24510091759941793) | > loss_0: 2.8232173919677734 (2.840140490098432) | > grad_norm_0: tensor(4.1290, device='cuda:0') (tensor(12.5051, device='cuda:0')) | > loss_gen: 1.8125609159469604 (1.7180314974351363) | > loss_kl: 2.4199378490448 (2.484680643948641) | > loss_feat: 1.6108816862106323 (1.5938519694588404) | > loss_mel: 17.531253814697266 (18.503572758761326) | > loss_duration: 1.7316795587539673 (1.6944588661193847) | > amp_scaler: 256.0 (165.23636363636365) | > loss_1: 25.106313705444336 (25.994596030495384) | > grad_norm_1: tensor(155.4395, device='cuda:0') (tensor(214.5426, device='cuda:0')) | > current_lr_0: 0.00019870413513039026 | > current_lr_1: 0.00019870413513039026 | > step_time: 0.4623 (0.4477624589746649) | > loader_time: 0.0094 (0.0077034343372691765) --> EVAL PERFORMANCE | > avg_loader_time: 0.008448328770382303 (-3.193465756695657e-06) | > avg_step_time: 0.45633438943137583 (+0.009138389372489786) | > avg_loss_disc: 2.8410626800966927 (-0.023797219907734313) | > avg_loss_disc_real_0: 0.22716005335391407 (-0.0028063634961423833) | > avg_loss_disc_real_1: 0.23368660727856863 (-0.0015573940226729643) | > avg_loss_disc_real_2: 0.22912801550307743 (-0.0072536063446125565) | > avg_loss_disc_real_3: 0.24107383086647785 (-0.0032736713617620428) | > avg_loss_disc_real_4: 0.2442383476546113 (-0.003850230238807012) | > avg_loss_disc_real_5: 0.24458342307889966 (-0.0048625870069987365) | > avg_loss_0: 2.8410626800966927 (-0.023797219907734313) | > avg_grad_norm_0: tensor(12.1434, device='cuda:0') (+tensor(1.7412, device='cuda:0')) | > avg_loss_gen: 1.7130757318416112 (+0.009289942996602685) | > avg_loss_kl: 2.467907771258287 (-0.0563257647232267) | > avg_loss_feat: 1.5856822477260109 (+0.03722796305804188) | > avg_loss_mel: 18.40871156773098 (-0.4139475083686861) | > avg_loss_duration: 1.697152965505358 (-0.011603165680254213) | > avg_amp_scaler: 185.69014084507043 (+57.69014084507043) | > avg_loss_1: 25.872530467073684 (-0.4353583967182004) | > avg_grad_norm_1: tensor(231.7841, device='cuda:0') (tensor(-6.1786, device='cuda:0'))  > EPOCH: 53/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:43:54)   --> TIME: 2025-02-25 08:44:11 -- STEP: 33/72 -- GLOBAL_STEP: 1003850 | > loss_disc: 2.8291704654693604 (2.8410057082320703) | > loss_disc_real_0: 0.22575980424880981 (0.23471452702175488) | > loss_disc_real_1: 0.20638981461524963 (0.24029435352845627) | > loss_disc_real_2: 0.17428001761436462 (0.23350605684699435) | > loss_disc_real_3: 0.2411458045244217 (0.2473932301456278) | > loss_disc_real_4: 0.25171637535095215 (0.2472227956309463) | > loss_disc_real_5: 0.25100642442703247 (0.24994690309871326) | > loss_0: 2.8291704654693604 (2.8410057082320703) | > grad_norm_0: tensor(3.2173, device='cuda:0') (tensor(11.2205, device='cuda:0')) | > loss_gen: 1.6728993654251099 (1.7414439952734746) | > loss_kl: 2.2892262935638428 (2.5096492911830093) | > loss_feat: 1.6173135042190552 (1.606650558384982) | > loss_mel: 17.3621768951416 (18.791937336777195) | > loss_duration: 1.6744953393936157 (1.6780779686841099) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.616111755371094 (26.32775913585316) | > grad_norm_1: tensor(226.8251, device='cuda:0') (tensor(240.6501, device='cuda:0')) | > current_lr_0: 0.00019867929711349895 | > current_lr_1: 0.00019867929711349895 | > step_time: 0.4459 (0.4323695789683949) | > loader_time: 0.0077 (0.006842779390739672) --> EVAL PERFORMANCE | > avg_loader_time: 0.008425145082070793 (-2.3183688311509756e-05) | > avg_step_time: 0.4499679551997655 (-0.006366434231610318) | > avg_loss_disc: 2.8389585152478287 (-0.0021041648488639986) | > avg_loss_disc_real_0: 0.2317033402726684 (+0.00454328691875433) | > avg_loss_disc_real_1: 0.23829698037933295 (+0.0046103731007643145) | > avg_loss_disc_real_2: 0.23518538957750293 (+0.006057374074425498) | > avg_loss_disc_real_3: 0.24537892072973116 (+0.004305089863253314) | > avg_loss_disc_real_4: 0.24740141105483954 (+0.003163063400228233) | > avg_loss_disc_real_5: 0.24704188087456663 (+0.002458457795666974) | > avg_loss_0: 2.8389585152478287 (-0.0021041648488639986) | > avg_grad_norm_0: tensor(10.7259, device='cuda:0') (tensor(-1.4175, device='cuda:0')) | > avg_loss_gen: 1.7423351146805455 (+0.029259382838934256) | > avg_loss_kl: 2.5094609831420462 (+0.04155321188375938) | > avg_loss_feat: 1.6394206654857582 (+0.0537384177597473) | > avg_loss_mel: 18.783744301594474 (+0.37503273386349534) | > avg_loss_duration: 1.6979223321860946 (+0.0007693666807364608) | > avg_amp_scaler: 256.0 (+70.30985915492957) | > avg_loss_1: 26.372883380298884 (+0.5003529132251998) | > avg_grad_norm_1: tensor(252.7056, device='cuda:0') (+tensor(20.9215, device='cuda:0'))  > EPOCH: 54/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:44:31)   --> TIME: 2025-02-25 08:44:37 -- STEP: 11/72 -- GLOBAL_STEP: 1003900 | > loss_disc: 2.8390491008758545 (2.809998555616899) | > loss_disc_real_0: 0.24820472300052643 (0.21746381575411017) | > loss_disc_real_1: 0.29022741317749023 (0.235430584712462) | > loss_disc_real_2: 0.18071788549423218 (0.22483301027254624) | > loss_disc_real_3: 0.25632840394973755 (0.24990602108565244) | > loss_disc_real_4: 0.27307066321372986 (0.25515993210402405) | > loss_disc_real_5: 0.25807955861091614 (0.2486349113962867) | > loss_0: 2.8390491008758545 (2.809998555616899) | > grad_norm_0: tensor(5.1474, device='cuda:0') (tensor(10.5850, device='cuda:0')) | > loss_gen: 1.8149034976959229 (1.7783371101726184) | > loss_kl: 2.713578224182129 (2.561538739637895) | > loss_feat: 1.685654878616333 (1.72059119831432) | > loss_mel: 17.898466110229492 (18.921592365611684) | > loss_duration: 1.6562552452087402 (1.7110092531551013) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.768857955932617 (26.693068417635832) | > grad_norm_1: tensor(211.6792, device='cuda:0') (tensor(207.2721, device='cuda:0')) | > current_lr_0: 0.00019865446220135974 | > current_lr_1: 0.00019865446220135974 | > step_time: 0.4357 (0.4292106411673806) | > loader_time: 0.0056 (0.00659357417713512)  --> TIME: 2025-02-25 08:45:00 -- STEP: 61/72 -- GLOBAL_STEP: 1003950 | > loss_disc: 2.9197168350219727 (2.8314742690227073) | > loss_disc_real_0: 0.2974034547805786 (0.22742814729448224) | > loss_disc_real_1: 0.2563638687133789 (0.2400175638374735) | > loss_disc_real_2: 0.33517763018608093 (0.23498108250195862) | > loss_disc_real_3: 0.22436362504959106 (0.24466327638899693) | > loss_disc_real_4: 0.2417599856853485 (0.24666830188915378) | > loss_disc_real_5: 0.24490393698215485 (0.24746411574668573) | > loss_0: 2.9197168350219727 (2.8314742690227073) | > grad_norm_0: tensor(14.7711, device='cuda:0') (tensor(10.2747, device='cuda:0')) | > loss_gen: 1.6931356191635132 (1.756776653352331) | > loss_kl: 2.4830381870269775 (2.5689633869733965) | > loss_feat: 1.511232852935791 (1.6898288081903927) | > loss_mel: 18.315509796142578 (18.733044421086547) | > loss_duration: 1.7186152935028076 (1.6996793160673047) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.72153091430664 (26.44829256026471) | > grad_norm_1: tensor(303.6135, device='cuda:0') (tensor(211.9670, device='cuda:0')) | > current_lr_0: 0.00019865446220135974 | > current_lr_1: 0.00019865446220135974 | > step_time: 0.4525 (0.44083140717178093) | > loader_time: 0.0107 (0.008123307931618614) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596950853374643 (+0.00017180577130384948) | > avg_step_time: 0.44750347607572316 (-0.0024644791240423514) | > avg_loss_disc: 2.8362633241734034 (-0.002695191074425285) | > avg_loss_disc_real_0: 0.22737950737207707 (-0.004323832900591323) | > avg_loss_disc_real_1: 0.2396739748582034 (+0.0013769944788704636) | > avg_loss_disc_real_2: 0.2349526472914387 (-0.00023274228606423586) | > avg_loss_disc_real_3: 0.2456502870354854 (+0.0002713663057542226) | > avg_loss_disc_real_4: 0.247494605855203 (+9.319480036346084e-05) | > avg_loss_disc_real_5: 0.24822976870436064 (+0.0011878878297940099) | > avg_loss_0: 2.8362633241734034 (-0.002695191074425285) | > avg_grad_norm_0: tensor(10.1320, device='cuda:0') (tensor(-0.5940, device='cuda:0')) | > avg_loss_gen: 1.753883338310349 (+0.011548223629803456) | > avg_loss_kl: 2.562764224871783 (+0.05330324172973677) | > avg_loss_feat: 1.6699942383967654 (+0.030573572911007263) | > avg_loss_mel: 18.761046879728074 (-0.022697421866400447) | > avg_loss_duration: 1.7026982441754408 (+0.004775911989346238) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.450386960741504 (+0.07750358044262029) | > avg_grad_norm_1: tensor(224.8676, device='cuda:0') (tensor(-27.8380, device='cuda:0'))  > EPOCH: 55/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:45:07)   --> TIME: 2025-02-25 08:45:27 -- STEP: 39/72 -- GLOBAL_STEP: 1004000 | > loss_disc: 2.8274149894714355 (2.809869063206208) | > loss_disc_real_0: 0.2420286238193512 (0.22611171962359014) | > loss_disc_real_1: 0.3058733642101288 (0.23887924009408706) | > loss_disc_real_2: 0.24179354310035706 (0.23582057081736052) | > loss_disc_real_3: 0.26911139488220215 (0.24182145870648897) | > loss_disc_real_4: 0.2875801622867584 (0.24515837201705346) | > loss_disc_real_5: 0.27110254764556885 (0.24277650584013033) | > loss_0: 2.8274149894714355 (2.809869063206208) | > grad_norm_0: tensor(11.6332, device='cuda:0') (tensor(10.0473, device='cuda:0')) | > loss_gen: 1.939686894416809 (1.7678647591517522) | > loss_kl: 2.583892345428467 (2.6027668439424954) | > loss_feat: 1.621698021888733 (1.7160679560441237) | > loss_mel: 19.389938354492188 (18.431508895678398) | > loss_duration: 1.6710264682769775 (1.6812487351588714) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.206241607666016 (26.199457266391853) | > grad_norm_1: tensor(316.0740, device='cuda:0') (tensor(288.2885, device='cuda:0')) | > current_lr_0: 0.00019862963039358455 | > current_lr_1: 0.00019862963039358455 | > step_time: 0.4425 (0.44506332201835436) | > loader_time: 0.0079 (0.007094169274354592) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438382350223168 (-0.0001585685031514749) | > avg_step_time: 0.45573922278175893 (+0.008235746706035774) | > avg_loss_disc: 2.8284649815357907 (-0.0077983426376126985) | > avg_loss_disc_real_0: 0.2260677715842153 (-0.0013117357878617841) | > avg_loss_disc_real_1: 0.2382962468644263 (-0.0013777279937771114) | > avg_loss_disc_real_2: 0.23471089169173173 (-0.00024175559970696092) | > avg_loss_disc_real_3: 0.24312621795795333 (-0.002524069077532054) | > avg_loss_disc_real_4: 0.24574244043356935 (-0.0017521654216336446) | > avg_loss_disc_real_5: 0.24631267702075796 (-0.0019170916836026841) | > avg_loss_0: 2.8284649815357907 (-0.0077983426376126985) | > avg_grad_norm_0: tensor(10.2783, device='cuda:0') (+tensor(0.1463, device='cuda:0')) | > avg_loss_gen: 1.747396811633043 (-0.0064865266773059815) | > avg_loss_kl: 2.5468571387546164 (-0.015907086117166624) | > avg_loss_feat: 1.674157758833657 (+0.004163520436891499) | > avg_loss_mel: 18.535234679638492 (-0.2258122000895817) | > avg_loss_duration: 1.6975415609252285 (-0.005156683250212346) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.201187966575084 (-0.2491989941664201) | > avg_grad_norm_1: tensor(266.9349, device='cuda:0') (+tensor(42.0674, device='cuda:0'))  > EPOCH: 56/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:45:44)   --> TIME: 2025-02-25 08:45:53 -- STEP: 17/72 -- GLOBAL_STEP: 1004050 | > loss_disc: 2.850959062576294 (2.8532026094548844) | > loss_disc_real_0: 0.26864439249038696 (0.22590986069510965) | > loss_disc_real_1: 0.27228501439094543 (0.24040743357994976) | > loss_disc_real_2: 0.2502411901950836 (0.2296527212156969) | > loss_disc_real_3: 0.2579598128795624 (0.24400335550308228) | > loss_disc_real_4: 0.23263022303581238 (0.24886796930257013) | > loss_disc_real_5: 0.282838374376297 (0.24547813131528742) | > loss_0: 2.850959062576294 (2.8532026094548844) | > grad_norm_0: tensor(9.7204, device='cuda:0') (tensor(10.7671, device='cuda:0')) | > loss_gen: 1.8409923315048218 (1.7234803508309757) | > loss_kl: 2.61967396736145 (2.5912282466888428) | > loss_feat: 1.5690757036209106 (1.5920091937570011) | > loss_mel: 19.00156593322754 (19.37078588149127) | > loss_duration: 1.6723436117172241 (1.677321567254908) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.703651428222656 (26.954825233010684) | > grad_norm_1: tensor(309.1191, device='cuda:0') (tensor(263.6670, device='cuda:0')) | > current_lr_0: 0.00019860480168978534 | > current_lr_1: 0.00019860480168978534 | > step_time: 0.429 (0.42956994561588063) | > loader_time: 0.0061 (0.006524394540225758)  --> TIME: 2025-02-25 08:46:16 -- STEP: 67/72 -- GLOBAL_STEP: 1004100 | > loss_disc: 2.903355360031128 (2.845446636427694) | > loss_disc_real_0: 0.17568282783031464 (0.22500418771558733) | > loss_disc_real_1: 0.1431373953819275 (0.23561240955075222) | > loss_disc_real_2: 0.2313353717327118 (0.23608715743271272) | > loss_disc_real_3: 0.20376087725162506 (0.242917586618395) | > loss_disc_real_4: 0.24868646264076233 (0.24822743192537508) | > loss_disc_real_5: 0.30446696281433105 (0.24890507246131327) | > loss_0: 2.903355360031128 (2.845446636427694) | > grad_norm_0: tensor(25.5205, device='cuda:0') (tensor(10.8287, device='cuda:0')) | > loss_gen: 1.5615925788879395 (1.7316452229200905) | > loss_kl: 2.4633607864379883 (2.538651679878804) | > loss_feat: 1.5203430652618408 (1.6496383218622919) | > loss_mel: 18.495412826538086 (18.740086484311238) | > loss_duration: 1.7513504028320312 (1.6945284053460876) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.79205894470215 (26.354550176592017) | > grad_norm_1: tensor(170.7461, device='cuda:0') (tensor(224.1139, device='cuda:0')) | > current_lr_0: 0.00019860480168978534 | > current_lr_1: 0.00019860480168978534 | > step_time: 0.5154 (0.44692260115893917) | > loader_time: 0.0126 (0.008289849580223883) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847814116679447 (+3.9758816571302744e-05) | > avg_step_time: 0.4500385069511306 (-0.005700715830628311) | > avg_loss_disc: 2.8434033830400915 (+0.014938401504300813) | > avg_loss_disc_real_0: 0.22634720046755294 (+0.00027942888333765126) | > avg_loss_disc_real_1: 0.23605645878214232 (-0.0022397880822839777) | > avg_loss_disc_real_2: 0.23575562343630993 (+0.0010447317445781978) | > avg_loss_disc_real_3: 0.24135736659379073 (-0.0017688513641626047) | > avg_loss_disc_real_4: 0.24665843475032861 (+0.000915994316759261) | > avg_loss_disc_real_5: 0.249178581254583 (+0.002865904233825045) | > avg_loss_0: 2.8434033830400915 (+0.014938401504300813) | > avg_grad_norm_0: tensor(10.8003, device='cuda:0') (+tensor(0.5220, device='cuda:0')) | > avg_loss_gen: 1.7326455166642094 (-0.01475129496883354) | > avg_loss_kl: 2.535481123857095 (-0.01137601489752127) | > avg_loss_feat: 1.6473720745301583 (-0.02678568430349859) | > avg_loss_mel: 18.746114623378705 (+0.21087994374021335) | > avg_loss_duration: 1.696244669632173 (-0.0012968912930555465) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.35785806682748 (+0.15667010025239492) | > avg_grad_norm_1: tensor(232.9565, device='cuda:0') (tensor(-33.9785, device='cuda:0'))  > EPOCH: 57/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:46:20)   --> TIME: 2025-02-25 08:46:42 -- STEP: 45/72 -- GLOBAL_STEP: 1004150 | > loss_disc: 2.861344575881958 (2.870514620674981) | > loss_disc_real_0: 0.2781733572483063 (0.235092830657959) | > loss_disc_real_1: 0.30178365111351013 (0.23994836409886677) | > loss_disc_real_2: 0.18483597040176392 (0.23300662967893812) | > loss_disc_real_3: 0.27137359976768494 (0.24638639953401353) | > loss_disc_real_4: 0.25539466738700867 (0.2473494162162145) | > loss_disc_real_5: 0.2716522812843323 (0.24620074265533023) | > loss_0: 2.861344575881958 (2.870514620674981) | > grad_norm_0: tensor(9.9816, device='cuda:0') (tensor(10.3528, device='cuda:0')) | > loss_gen: 1.789426326751709 (1.7287935256958007) | > loss_kl: 2.747429609298706 (2.4937745306226944) | > loss_feat: 1.4477758407592773 (1.5944007529152764) | > loss_mel: 18.864137649536133 (18.586970138549802) | > loss_duration: 1.7225972414016724 (1.6838519705666437) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.571367263793945 (26.087790722317166) | > grad_norm_1: tensor(76.5541, device='cuda:0') (tensor(213.7755, device='cuda:0')) | > current_lr_0: 0.0001985799760895741 | > current_lr_1: 0.0001985799760895741 | > step_time: 0.4694 (0.43571342362297905) | > loader_time: 0.011 (0.007364866468641493) --> EVAL PERFORMANCE | > avg_loader_time: 0.008446982209111603 (-3.115895768286692e-05) | > avg_step_time: 0.45034594267186984 (+0.0003074357207392131) | > avg_loss_disc: 2.8594674627545853 (+0.01606407971449375) | > avg_loss_disc_real_0: 0.230784352606451 (+0.0044371521388980595) | > avg_loss_disc_real_1: 0.23900249600410461 (+0.0029460372219622943) | > avg_loss_disc_real_2: 0.23415280018054263 (-0.0016028232557672994) | > avg_loss_disc_real_3: 0.2430287540798456 (+0.0016713874860548594) | > avg_loss_disc_real_4: 0.24613768743797088 (-0.0005207473123577389) | > avg_loss_disc_real_5: 0.2450832875681595 (-0.004095293686423501) | > avg_loss_0: 2.8594674627545853 (+0.01606407971449375) | > avg_grad_norm_0: tensor(10.7078, device='cuda:0') (tensor(-0.0925, device='cuda:0')) | > avg_loss_gen: 1.7237159517449392 (-0.008929564919270216) | > avg_loss_kl: 2.5045993999696123 (-0.03088172388748278) | > avg_loss_feat: 1.5991627112240858 (-0.048209363306072506) | > avg_loss_mel: 18.617052669256502 (-0.12906195412220356) | > avg_loss_duration: 1.6946068296969776 (-0.0016378399351952844) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.13913737552267 (-0.2187206913048101) | > avg_grad_norm_1: tensor(215.7676, device='cuda:0') (tensor(-17.1888, device='cuda:0'))  > EPOCH: 58/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:46:57)   --> TIME: 2025-02-25 08:47:09 -- STEP: 23/72 -- GLOBAL_STEP: 1004200 | > loss_disc: 2.824972629547119 (2.850419179252956) | > loss_disc_real_0: 0.16606342792510986 (0.22450004064518472) | > loss_disc_real_1: 0.2720545828342438 (0.23280823489894037) | > loss_disc_real_2: 0.20776326954364777 (0.24807631321575335) | > loss_disc_real_3: 0.2412312775850296 (0.2443483335816342) | > loss_disc_real_4: 0.24763767421245575 (0.24648144711618838) | > loss_disc_real_5: 0.2551850378513336 (0.24983096964981244) | > loss_0: 2.824972629547119 (2.850419179252956) | > grad_norm_0: tensor(9.6150, device='cuda:0') (tensor(9.8007, device='cuda:0')) | > loss_gen: 1.7068145275115967 (1.723663568496704) | > loss_kl: 2.3551597595214844 (2.5570314137831978) | > loss_feat: 1.731397032737732 (1.6242151882337488) | > loss_mel: 18.156301498413086 (18.76614056462827) | > loss_duration: 1.6812187433242798 (1.6789785623550415) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.630891799926758 (26.350029323412024) | > grad_norm_1: tensor(245.7071, device='cuda:0') (tensor(204.7007, device='cuda:0')) | > current_lr_0: 0.0001985551535925629 | > current_lr_1: 0.0001985551535925629 | > step_time: 0.425 (0.4295343212459398) | > loader_time: 0.0064 (0.0067210715749989386) --> EVAL PERFORMANCE | > avg_loader_time: 0.008725737182187363 (+0.0002787549730757597) | > avg_step_time: 0.45750942700345754 (+0.0071634843315876995) | > avg_loss_disc: 2.8403656046155477 (-0.019101858139037642) | > avg_loss_disc_real_0: 0.22795291958560407 (-0.0028314330208469296) | > avg_loss_disc_real_1: 0.23518175600280225 (-0.003820740001302364) | > avg_loss_disc_real_2: 0.24052547883819525 (+0.0063726786576526195) | > avg_loss_disc_real_3: 0.2434733170019069 (+0.00044456292206132564) | > avg_loss_disc_real_4: 0.24530965719424502 (-0.0008280302437258524) | > avg_loss_disc_real_5: 0.2466594868562591 (+0.0015761992880996) | > avg_loss_0: 2.8403656046155477 (-0.019101858139037642) | > avg_grad_norm_0: tensor(9.4937, device='cuda:0') (tensor(-1.2141, device='cuda:0')) | > avg_loss_gen: 1.731620279835983 (+0.0079043280910438) | > avg_loss_kl: 2.517164173260541 (+0.012564773290928866) | > avg_loss_feat: 1.654944057195959 (+0.05578134597187323) | > avg_loss_mel: 18.51704707615812 (-0.10000559309838053) | > avg_loss_duration: 1.6911450221504964 (-0.003461807546481177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.11192071941537 (-0.027216656107299286) | > avg_grad_norm_1: tensor(189.1198, device='cuda:0') (tensor(-26.6478, device='cuda:0'))  > EPOCH: 59/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:47:34)   --> TIME: 2025-02-25 08:47:36 -- STEP: 1/72 -- GLOBAL_STEP: 1004250 | > loss_disc: 2.872736930847168 (2.872736930847168) | > loss_disc_real_0: 0.17937824130058289 (0.17937824130058289) | > loss_disc_real_1: 0.21341586112976074 (0.21341586112976074) | > loss_disc_real_2: 0.2454749047756195 (0.2454749047756195) | > loss_disc_real_3: 0.2670848071575165 (0.2670848071575165) | > loss_disc_real_4: 0.26246923208236694 (0.26246923208236694) | > loss_disc_real_5: 0.2555696666240692 (0.2555696666240692) | > loss_0: 2.872736930847168 (2.872736930847168) | > grad_norm_0: tensor(7.9025, device='cuda:0') (tensor(7.9025, device='cuda:0')) | > loss_gen: 1.910260796546936 (1.910260796546936) | > loss_kl: 2.208247661590576 (2.208247661590576) | > loss_feat: 1.8697916269302368 (1.8697916269302368) | > loss_mel: 18.896337509155273 (18.896337509155273) | > loss_duration: 1.697526454925537 (1.697526454925537) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.582164764404297 (26.582164764404297) | > grad_norm_1: tensor(188.6122, device='cuda:0') (tensor(188.6122, device='cuda:0')) | > current_lr_0: 0.00019853033419836382 | > current_lr_1: 0.00019853033419836382 | > step_time: 0.4408 (0.4407680034637451) | > loader_time: 0.0061 (0.006110429763793945)  --> TIME: 2025-02-25 08:47:59 -- STEP: 51/72 -- GLOBAL_STEP: 1004300 | > loss_disc: 2.856458902359009 (2.8542189083847345) | > loss_disc_real_0: 0.26557785272598267 (0.23186246407966987) | > loss_disc_real_1: 0.2748275101184845 (0.23637130184500826) | > loss_disc_real_2: 0.2313685417175293 (0.2387404918086295) | > loss_disc_real_3: 0.246766597032547 (0.24443562650213055) | > loss_disc_real_4: 0.23315663635730743 (0.2454245070032045) | > loss_disc_real_5: 0.2389172613620758 (0.2463738561845293) | > loss_0: 2.856458902359009 (2.8542189083847345) | > grad_norm_0: tensor(8.1220, device='cuda:0') (tensor(11.4338, device='cuda:0')) | > loss_gen: 1.7168796062469482 (1.7248358656378353) | > loss_kl: 2.180349349975586 (2.474487519731708) | > loss_feat: 1.799719214439392 (1.644172275767607) | > loss_mel: 17.428001403808594 (18.233774035584688) | > loss_duration: 1.7134040594100952 (1.6805101796692492) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.83835220336914 (25.757780075073242) | > grad_norm_1: tensor(76.7111, device='cuda:0') (tensor(166.6694, device='cuda:0')) | > current_lr_0: 0.00019853033419836382 | > current_lr_1: 0.00019853033419836382 | > step_time: 0.4395 (0.4406360551422718) | > loader_time: 0.0089 (0.0075727257074094285) --> EVAL PERFORMANCE | > avg_loader_time: 0.00844518567474795 (-0.0002805515074394138) | > avg_step_time: 0.45161121663912923 (-0.005898210364328305) | > avg_loss_disc: 2.853910597277359 (+0.013544992661811417) | > avg_loss_disc_real_0: 0.2334076227016852 (+0.005454703116081122) | > avg_loss_disc_real_1: 0.2358943375063614 (+0.0007125815035591565) | > avg_loss_disc_real_2: 0.23870757998714984 (-0.001817898851045413) | > avg_loss_disc_real_3: 0.2434026389054849 (-7.067809642199996e-05) | > avg_loss_disc_real_4: 0.24596219184532972 (+0.0006525346510846963) | > avg_loss_disc_real_5: 0.2466427895804526 (-1.6697275806498713e-05) | > avg_loss_0: 2.853910597277359 (+0.013544992661811417) | > avg_grad_norm_0: tensor(11.4007, device='cuda:0') (+tensor(1.9070, device='cuda:0')) | > avg_loss_gen: 1.7233196648073865 (-0.00830061502859647) | > avg_loss_kl: 2.456272461044956 (-0.060891712215585425) | > avg_loss_feat: 1.622558252912172 (-0.03238580428378701) | > avg_loss_mel: 18.288350924639634 (-0.22869615151848777) | > avg_loss_duration: 1.6869194222168185 (-0.004225599933677993) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.77742084986727 (-0.3344998695481003) | > avg_grad_norm_1: tensor(171.7975, device='cuda:0') (tensor(-17.3223, device='cuda:0'))  > EPOCH: 60/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:48:10)   --> TIME: 2025-02-25 08:48:25 -- STEP: 29/72 -- GLOBAL_STEP: 1004350 | > loss_disc: 2.777634382247925 (2.866813840537236) | > loss_disc_real_0: 0.2598886489868164 (0.23080620991772618) | > loss_disc_real_1: 0.24233366549015045 (0.2322248076570445) | > loss_disc_real_2: 0.22247719764709473 (0.2439245016410433) | > loss_disc_real_3: 0.2227313220500946 (0.24706807578432152) | > loss_disc_real_4: 0.2146698385477066 (0.2522254966456315) | > loss_disc_real_5: 0.19193801283836365 (0.25381252940358784) | > loss_0: 2.777634382247925 (2.866813840537236) | > grad_norm_0: tensor(6.4398, device='cuda:0') (tensor(10.8130, device='cuda:0')) | > loss_gen: 1.7063363790512085 (1.7300617119361614) | > loss_kl: 2.5314719676971436 (2.487925603471954) | > loss_feat: 1.8102024793624878 (1.6217390586589944) | > loss_mel: 17.7999210357666 (18.280840643521014) | > loss_duration: 1.65604567527771 (1.6649644210420806) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.503978729248047 (25.785531734598095) | > grad_norm_1: tensor(323.2068, device='cuda:0') (tensor(228.8155, device='cuda:0')) | > current_lr_0: 0.000198505517906589 | > current_lr_1: 0.000198505517906589 | > step_time: 0.4541 (0.43583541080869476) | > loader_time: 0.0078 (0.006984299626843682) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614419211804027 (+0.00016923353705607797) | > avg_step_time: 0.4520303699332224 (+0.00041915329409314417) | > avg_loss_disc: 2.8619985278223603 (+0.008087930545001232) | > avg_loss_disc_real_0: 0.23158702422195757 (-0.0018205984797276253) | > avg_loss_disc_real_1: 0.23617469655795836 (+0.00028035905159695673) | > avg_loss_disc_real_2: 0.24107376874332698 (+0.00236618875617714) | > avg_loss_disc_real_3: 0.24459158114984003 (+0.0011889422443551179) | > avg_loss_disc_real_4: 0.24853664489699082 (+0.0025744530516610964) | > avg_loss_disc_real_5: 0.25022017640966765 (+0.0035773868292150457) | > avg_loss_0: 2.8619985278223603 (+0.008087930545001232) | > avg_grad_norm_0: tensor(10.9427, device='cuda:0') (tensor(-0.4580, device='cuda:0')) | > avg_loss_gen: 1.7228196657879251 (-0.0004999990194614057) | > avg_loss_kl: 2.4681603606318085 (+0.011887899586852768) | > avg_loss_feat: 1.600060115397816 (-0.02249813751435603) | > avg_loss_mel: 18.33300437389966 (+0.04465344926002501) | > avg_loss_duration: 1.6878136396408079 (+0.0008942174239894296) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.81185813688896 (+0.03443728702169224) | > avg_grad_norm_1: tensor(230.8321, device='cuda:0') (+tensor(59.0346, device='cuda:0'))  > EPOCH: 61/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:48:47)   --> TIME: 2025-02-25 08:48:52 -- STEP: 7/72 -- GLOBAL_STEP: 1004400 | > loss_disc: 2.8701884746551514 (2.8194477217538014) | > loss_disc_real_0: 0.2740136981010437 (0.23684240451880864) | > loss_disc_real_1: 0.22888076305389404 (0.2161211563008172) | > loss_disc_real_2: 0.29014045000076294 (0.23206558184964315) | > loss_disc_real_3: 0.24549447000026703 (0.2325520600591387) | > loss_disc_real_4: 0.24958039820194244 (0.23158423815454757) | > loss_disc_real_5: 0.24731671810150146 (0.23505450359412602) | > loss_0: 2.8701884746551514 (2.8194477217538014) | > grad_norm_0: tensor(7.8413, device='cuda:0') (tensor(11.0581, device='cuda:0')) | > loss_gen: 1.8828563690185547 (1.787545987537929) | > loss_kl: 2.5251543521881104 (2.489682810647147) | > loss_feat: 1.7630857229232788 (1.7972713197980608) | > loss_mel: 19.125898361206055 (18.733982631138392) | > loss_duration: 1.6957038640975952 (1.6939109563827515) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.992698669433594 (26.502393450055802) | > grad_norm_1: tensor(418.0108, device='cuda:0') (tensor(288.0775, device='cuda:0')) | > current_lr_0: 0.00019848070471685067 | > current_lr_1: 0.00019848070471685067 | > step_time: 0.4367 (0.42416347776140484) | > loader_time: 0.0054 (0.006590230124337333)  --> TIME: 2025-02-25 08:49:15 -- STEP: 57/72 -- GLOBAL_STEP: 1004450 | > loss_disc: 2.8416781425476074 (2.8624584591179563) | > loss_disc_real_0: 0.2916838526725769 (0.23484328153886294) | > loss_disc_real_1: 0.2713256776332855 (0.23829672054240578) | > loss_disc_real_2: 0.23875586688518524 (0.23929786865125624) | > loss_disc_real_3: 0.25790461897850037 (0.2404914885236506) | > loss_disc_real_4: 0.25565868616104126 (0.24566401959511272) | > loss_disc_real_5: 0.2640838325023651 (0.2460113870992995) | > loss_0: 2.8416781425476074 (2.8624584591179563) | > grad_norm_0: tensor(9.1305, device='cuda:0') (tensor(9.8427, device='cuda:0')) | > loss_gen: 1.7293109893798828 (1.7306309603808219) | > loss_kl: 2.39324951171875 (2.4945873377615944) | > loss_feat: 1.503561019897461 (1.6345755321937694) | > loss_mel: 17.338451385498047 (18.676382800988975) | > loss_duration: 1.7135088443756104 (1.687729051238612) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.678081512451172 (26.223905630279006) | > grad_norm_1: tensor(93.7430, device='cuda:0') (tensor(227.3337, device='cuda:0')) | > current_lr_0: 0.00019848070471685067 | > current_lr_1: 0.00019848070471685067 | > step_time: 0.4647 (0.4512659206725003) | > loader_time: 0.0097 (0.008141082629822848) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738420378993932 (+0.0001240011671899046) | > avg_step_time: 0.4585682842093454 (+0.006537914276123047) | > avg_loss_disc: 2.8667772521435375 (+0.004778724321177208) | > avg_loss_disc_real_0: 0.2338855912987615 (+0.0022985670768039257) | > avg_loss_disc_real_1: 0.23804532998884229 (+0.0018706334308839223) | > avg_loss_disc_real_2: 0.23925969718207776 (-0.0018140715612492142) | > avg_loss_disc_real_3: 0.24024709450526976 (-0.004344486644570267) | > avg_loss_disc_real_4: 0.24570766825911025 (-0.0028289766378805647) | > avg_loss_disc_real_5: 0.24530047199255983 (-0.0049197044171078175) | > avg_loss_0: 2.8667772521435375 (+0.004778724321177208) | > avg_grad_norm_0: tensor(10.0036, device='cuda:0') (tensor(-0.9391, device='cuda:0')) | > avg_loss_gen: 1.7176188885326116 (-0.005200777255313538) | > avg_loss_kl: 2.4830545942548294 (+0.014894233623020892) | > avg_loss_feat: 1.601419430383494 (+0.001359314985678095) | > avg_loss_mel: 18.560929956570472 (+0.22792558267081375) | > avg_loss_duration: 1.691064068968867 (+0.003250429328059079) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.054086900093186 (+0.24222876320422415) | > avg_grad_norm_1: tensor(214.3399, device='cuda:0') (tensor(-16.4922, device='cuda:0'))  > EPOCH: 62/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:49:24)   --> TIME: 2025-02-25 08:49:41 -- STEP: 35/72 -- GLOBAL_STEP: 1004500 | > loss_disc: 2.9002840518951416 (2.846085330418178) | > loss_disc_real_0: 0.30090266466140747 (0.23294003201382502) | > loss_disc_real_1: 0.2763217091560364 (0.2346105511699404) | > loss_disc_real_2: 0.25452473759651184 (0.2461565222058978) | > loss_disc_real_3: 0.2362712025642395 (0.2458256244659424) | > loss_disc_real_4: 0.24372273683547974 (0.24924805802958352) | > loss_disc_real_5: 0.2305029034614563 (0.2518812733037131) | > loss_0: 2.9002840518951416 (2.846085330418178) | > grad_norm_0: tensor(8.4497, device='cuda:0') (tensor(9.5595, device='cuda:0')) | > loss_gen: 1.556093692779541 (1.7470856870923723) | > loss_kl: 2.717165946960449 (2.5234905924115862) | > loss_feat: 1.51024329662323 (1.6754490579877581) | > loss_mel: 19.380311965942383 (18.637416676112576) | > loss_duration: 1.7056243419647217 (1.6644700390951974) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.86943817138672 (26.247911998203822) | > grad_norm_1: tensor(292.1721, device='cuda:0') (tensor(209.5055, device='cuda:0')) | > current_lr_0: 0.00019845589462876104 | > current_lr_1: 0.00019845589462876104 | > step_time: 0.4381 (0.4339841978890555) | > loader_time: 0.0081 (0.007036215918404715) --> EVAL PERFORMANCE | > avg_loader_time: 0.0084755454264896 (-0.0002628749525043321) | > avg_step_time: 0.4498582054191912 (-0.00871007879015423) | > avg_loss_disc: 2.847654362799416 (-0.01912288934412132) | > avg_loss_disc_real_0: 0.23215246169080198 (-0.001733129607959516) | > avg_loss_disc_real_1: 0.23705532706119645 (-0.0009900029276458389) | > avg_loss_disc_real_2: 0.24340605295040238 (+0.004146355768324617) | > avg_loss_disc_real_3: 0.24483468519969726 (+0.00458759069442749) | > avg_loss_disc_real_4: 0.2485941060831849 (+0.0028864378240746613) | > avg_loss_disc_real_5: 0.2488128009816291 (+0.00351232898906928) | > avg_loss_0: 2.847654362799416 (-0.01912288934412132) | > avg_grad_norm_0: tensor(8.7683, device='cuda:0') (tensor(-1.2352, device='cuda:0')) | > avg_loss_gen: 1.740967122601791 (+0.02334823406917952) | > avg_loss_kl: 2.4985023720163695 (+0.015447777761540049) | > avg_loss_feat: 1.6377336592741416 (+0.036314228890647504) | > avg_loss_mel: 18.577768675038506 (+0.01683871846803342) | > avg_loss_duration: 1.6881357313881458 (-0.0029283375807211964) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.143107454541703 (+0.08902055444851698) | > avg_grad_norm_1: tensor(187.8404, device='cuda:0') (tensor(-26.4995, device='cuda:0'))  > EPOCH: 63/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:50:00)   --> TIME: 2025-02-25 08:50:08 -- STEP: 13/72 -- GLOBAL_STEP: 1004550 | > loss_disc: 2.8554892539978027 (2.8627369770636926) | > loss_disc_real_0: 0.23794470727443695 (0.22462574105996352) | > loss_disc_real_1: 0.26056423783302307 (0.2396454650622148) | > loss_disc_real_2: 0.21130014955997467 (0.23067319622406593) | > loss_disc_real_3: 0.22366821765899658 (0.23704678049454322) | > loss_disc_real_4: 0.2264612317085266 (0.23933845873062426) | > loss_disc_real_5: 0.24977856874465942 (0.24663019868043753) | > loss_0: 2.8554892539978027 (2.8627369770636926) | > grad_norm_0: tensor(3.4827, device='cuda:0') (tensor(10.2442, device='cuda:0')) | > loss_gen: 1.7472915649414062 (1.6893713015776415) | > loss_kl: 2.5043370723724365 (2.454436173805824) | > loss_feat: 1.6029244661331177 (1.6023420095443726) | > loss_mel: 18.427526473999023 (18.728913967426006) | > loss_duration: 1.639389991760254 (1.6723472430155828) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.921470642089844 (26.14741090627817) | > grad_norm_1: tensor(153.3463, device='cuda:0') (tensor(176.8872, device='cuda:0')) | > current_lr_0: 0.00019843108764193245 | > current_lr_1: 0.00019843108764193245 | > step_time: 0.4468 (0.4272147325369028) | > loader_time: 0.0057 (0.006582278471726637)  --> TIME: 2025-02-25 08:50:31 -- STEP: 63/72 -- GLOBAL_STEP: 1004600 | > loss_disc: 2.870388984680176 (2.853951930999756) | > loss_disc_real_0: 0.288534939289093 (0.23153495906837404) | > loss_disc_real_1: 0.23725174367427826 (0.23891032901075152) | > loss_disc_real_2: 0.2877478301525116 (0.23532549161759633) | > loss_disc_real_3: 0.2730226218700409 (0.24101202166269695) | > loss_disc_real_4: 0.25947096943855286 (0.24394385824127804) | > loss_disc_real_5: 0.25441083312034607 (0.24531913394965824) | > loss_0: 2.870388984680176 (2.853951930999756) | > grad_norm_0: tensor(7.0676, device='cuda:0') (tensor(8.8865, device='cuda:0')) | > loss_gen: 1.689012050628662 (1.7053220707272727) | > loss_kl: 2.3674001693725586 (2.4460487782009066) | > loss_feat: 1.5088763236999512 (1.586884337758261) | > loss_mel: 18.23388671875 (18.433041254679363) | > loss_duration: 1.6560373306274414 (1.6812881969270252) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.455211639404297 (25.852584657214937) | > grad_norm_1: tensor(287.7455, device='cuda:0') (tensor(184.3099, device='cuda:0')) | > current_lr_0: 0.00019843108764193245 | > current_lr_1: 0.00019843108764193245 | > step_time: 0.4791 (0.4429842366112603) | > loader_time: 0.0112 (0.008217663992018929) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617474999226318 (+0.00014192957273671813) | > avg_step_time: 0.44994398237953726 (+8.57769603460623e-05) | > avg_loss_disc: 2.854769686578025 (+0.007115323778609017) | > avg_loss_disc_real_0: 0.22978890582289493 (-0.002363555867907041) | > avg_loss_disc_real_1: 0.23935972573891492 (+0.0023043986777184777) | > avg_loss_disc_real_2: 0.2353704334564612 (-0.008035619493941187) | > avg_loss_disc_real_3: 0.24096726049000108 (-0.003867424709696171) | > avg_loss_disc_real_4: 0.24436134540698898 (-0.004232760676195935) | > avg_loss_disc_real_5: 0.24686209626600775 (-0.0019507047156213653) | > avg_loss_0: 2.854769686578025 (+0.007115323778609017) | > avg_grad_norm_0: tensor(9.4041, device='cuda:0') (+tensor(0.6358, device='cuda:0')) | > avg_loss_gen: 1.7092274074823084 (-0.0317397151194827) | > avg_loss_kl: 2.4498759659243303 (-0.048626406092039165) | > avg_loss_feat: 1.5855896002809766 (-0.05214405899316499) | > avg_loss_mel: 18.44052086413746 (-0.13724781090104443) | > avg_loss_duration: 1.6848667171639455 (-0.0032690142242002285) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.870080625507192 (-0.27302682903451014) | > avg_grad_norm_1: tensor(192.5205, device='cuda:0') (+tensor(4.6801, device='cuda:0'))  > EPOCH: 64/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:50:37)   --> TIME: 2025-02-25 08:50:57 -- STEP: 41/72 -- GLOBAL_STEP: 1004650 | > loss_disc: 2.8572237491607666 (2.8409224545083394) | > loss_disc_real_0: 0.21074309945106506 (0.23030883491766163) | > loss_disc_real_1: 0.261011004447937 (0.23335536587529065) | > loss_disc_real_2: 0.2357894331216812 (0.23741284521614633) | > loss_disc_real_3: 0.2752818763256073 (0.24478118993887088) | > loss_disc_real_4: 0.28517016768455505 (0.24545439605305835) | > loss_disc_real_5: 0.2936679422855377 (0.24447020997361438) | > loss_0: 2.8572237491607666 (2.8409224545083394) | > grad_norm_0: tensor(8.1055, device='cuda:0') (tensor(9.5895, device='cuda:0')) | > loss_gen: 1.7107226848602295 (1.73188011529969) | > loss_kl: 2.4179322719573975 (2.4015861429819254) | > loss_feat: 1.5209451913833618 (1.691061118753945) | > loss_mel: 17.831892013549805 (18.35915156108577) | > loss_duration: 1.6808242797851562 (1.6665447572382486) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.162315368652344 (25.85022400646675) | > grad_norm_1: tensor(109.8781, device='cuda:0') (tensor(197.9867, device='cuda:0')) | > current_lr_0: 0.0001984062837559772 | > current_lr_1: 0.0001984062837559772 | > step_time: 0.4423 (0.43671309657213164) | > loader_time: 0.0094 (0.00751876249545958) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865440301492181 (+3.692801569549209e-05) | > avg_step_time: 0.45526038760870274 (+0.005316405229165488) | > avg_loss_disc: 2.856462609600013 (+0.0016929230219879798) | > avg_loss_disc_real_0: 0.22909657499739822 (-0.0006923308254967175) | > avg_loss_disc_real_1: 0.23551096190029466 (-0.0038487638386202616) | > avg_loss_disc_real_2: 0.2426007457182441 (+0.007230312261782917) | > avg_loss_disc_real_3: 0.24255770822645914 (+0.001590447736458056) | > avg_loss_disc_real_4: 0.24403172794362188 (-0.00032961746336709896) | > avg_loss_disc_real_5: 0.24350593795239087 (-0.0033561583136168804) | > avg_loss_0: 2.856462609600013 (+0.0016929230219879798) | > avg_grad_norm_0: tensor(9.7988, device='cuda:0') (+tensor(0.3947, device='cuda:0')) | > avg_loss_gen: 1.7129415693417402 (+0.0037141618594318526) | > avg_loss_kl: 2.4015224315750765 (-0.048353534349253824) | > avg_loss_feat: 1.637951026500111 (+0.05236142621913431) | > avg_loss_mel: 18.231682387875836 (-0.20883847626162577) | > avg_loss_duration: 1.6810328826098375 (-0.0038338345541080177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.665130400321853 (-0.204950225185339) | > avg_grad_norm_1: tensor(229.4055, device='cuda:0') (+tensor(36.8850, device='cuda:0'))  > EPOCH: 65/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:51:14)   --> TIME: 2025-02-25 08:51:24 -- STEP: 19/72 -- GLOBAL_STEP: 1004700 | > loss_disc: 2.751725673675537 (2.852010639090287) | > loss_disc_real_0: 0.2116757333278656 (0.23506777537496468) | > loss_disc_real_1: 0.19135650992393494 (0.22957163421731247) | > loss_disc_real_2: 0.27084484696388245 (0.2385956292089663) | > loss_disc_real_3: 0.22498273849487305 (0.24168984591960907) | > loss_disc_real_4: 0.23547933995723724 (0.24471713680970042) | > loss_disc_real_5: 0.22996635735034943 (0.2479256571907746) | > loss_0: 2.751725673675537 (2.852010639090287) | > grad_norm_0: tensor(11.6683, device='cuda:0') (tensor(9.9817, device='cuda:0')) | > loss_gen: 1.7862660884857178 (1.7088341650209928) | > loss_kl: 2.6650943756103516 (2.504326293342992) | > loss_feat: 1.7522495985031128 (1.5971430414601375) | > loss_mel: 17.57408332824707 (18.433240539149235) | > loss_duration: 1.6519901752471924 (1.667219318841633) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.429685592651367 (25.910763288799085) | > grad_norm_1: tensor(152.1625, device='cuda:0') (tensor(208.7059, device='cuda:0')) | > current_lr_0: 0.00019838148297050769 | > current_lr_1: 0.00019838148297050769 | > step_time: 0.4311 (0.43252350154675934) | > loader_time: 0.0062 (0.006803675701743678)  --> TIME: 2025-02-25 08:51:48 -- STEP: 69/72 -- GLOBAL_STEP: 1004750 | > loss_disc: 2.883967161178589 (2.8524872565615005) | > loss_disc_real_0: 0.22916509211063385 (0.2332861114671265) | > loss_disc_real_1: 0.19253751635551453 (0.23628274103005728) | > loss_disc_real_2: 0.2302854210138321 (0.241100642344226) | > loss_disc_real_3: 0.2712898254394531 (0.24406951449919437) | > loss_disc_real_4: 0.27113524079322815 (0.2469087839126587) | > loss_disc_real_5: 0.2826646864414215 (0.24946702152922534) | > loss_0: 2.883967161178589 (2.8524872565615005) | > grad_norm_0: tensor(5.3920, device='cuda:0') (tensor(9.2762, device='cuda:0')) | > loss_gen: 1.7774717807769775 (1.7270621475966081) | > loss_kl: 2.485835075378418 (2.4471186002095546) | > loss_feat: 1.589167833328247 (1.61906297483306) | > loss_mel: 18.964723587036133 (18.100604416667547) | > loss_duration: 1.7587512731552124 (1.6809539138406948) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.57594871520996 (25.5748021222543) | > grad_norm_1: tensor(467.2714, device='cuda:0') (tensor(213.3185, device='cuda:0')) | > current_lr_0: 0.00019838148297050769 | > current_lr_1: 0.00019838148297050769 | > step_time: 0.5011 (0.449846561404242) | > loader_time: 0.0136 (0.008757017660832063) --> EVAL PERFORMANCE | > avg_loader_time: 0.008819271141374619 (+0.0001648681264528086) | > avg_step_time: 0.45141903447433257 (-0.0038413531343701734) | > avg_loss_disc: 2.8518801037694366 (-0.004582505830576622) | > avg_loss_disc_real_0: 0.23331576103055982 (+0.004219186033161598) | > avg_loss_disc_real_1: 0.23602354946270795 (+0.0005125875624132914) | > avg_loss_disc_real_2: 0.24119206150652658 (-0.0014086842117175258) | > avg_loss_disc_real_3: 0.2435859269659284 (+0.001028218739469261) | > avg_loss_disc_real_4: 0.24697781301720043 (+0.002946085073578547) | > avg_loss_disc_real_5: 0.24951797870682998 (+0.0060120407544391186) | > avg_loss_0: 2.8518801037694366 (-0.004582505830576622) | > avg_grad_norm_0: tensor(9.1970, device='cuda:0') (tensor(-0.6018, device='cuda:0')) | > avg_loss_gen: 1.7252002249301321 (+0.012258655588391898) | > avg_loss_kl: 2.4500964560978855 (+0.04857402452280901) | > avg_loss_feat: 1.619010352752578 (-0.018940673747532832) | > avg_loss_mel: 18.125743879398826 (-0.10593850847700992) | > avg_loss_duration: 1.6806666532032928 (-0.000366229406544738) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.600717598283794 (-0.06441280203805988) | > avg_grad_norm_1: tensor(214.5087, device='cuda:0') (tensor(-14.8968, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1004753.pth  > EPOCH: 66/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:51:53)   --> TIME: 2025-02-25 08:52:16 -- STEP: 47/72 -- GLOBAL_STEP: 1004800 | > loss_disc: 2.961214065551758 (2.870808408615437) | > loss_disc_real_0: 0.1899324208498001 (0.2305303653503986) | > loss_disc_real_1: 0.19863244891166687 (0.24118986472170403) | > loss_disc_real_2: 0.26501980423927307 (0.24783141530574632) | > loss_disc_real_3: 0.20319916307926178 (0.24458488536641954) | > loss_disc_real_4: 0.22528700530529022 (0.24730165968550014) | > loss_disc_real_5: 0.23124881088733673 (0.24659027190918617) | > loss_0: 2.961214065551758 (2.870808408615437) | > grad_norm_0: tensor(9.7361, device='cuda:0') (tensor(11.1613, device='cuda:0')) | > loss_gen: 1.5274100303649902 (1.730169435764881) | > loss_kl: 2.3185319900512695 (2.47647947453438) | > loss_feat: 1.6841219663619995 (1.6108984490658373) | > loss_mel: 19.01351547241211 (18.797482023847863) | > loss_duration: 1.6970081329345703 (1.6725865095219714) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.240589141845703 (26.287615918098613) | > grad_norm_1: tensor(261.0907, device='cuda:0') (tensor(197.0942, device='cuda:0')) | > current_lr_0: 0.00019835668528513637 | > current_lr_1: 0.00019835668528513637 | > step_time: 0.4504 (0.43595314025878906) | > loader_time: 0.0087 (0.00735347828966506) --> EVAL PERFORMANCE | > avg_loader_time: 0.008363381238050863 (-0.00045588990332375533) | > avg_step_time: 0.45075371231831296 (-0.0006653221560196099) | > avg_loss_disc: 2.863606587262221 (+0.011726483492784379) | > avg_loss_disc_real_0: 0.2321007938032419 (-0.0012149672273179257) | > avg_loss_disc_real_1: 0.2384666922226758 (+0.002443142759967848) | > avg_loss_disc_real_2: 0.2448567420663968 (+0.0036646805598702092) | > avg_loss_disc_real_3: 0.24202673347063466 (-0.001559193495293737) | > avg_loss_disc_real_4: 0.24579566654185175 (-0.0011821464753486721) | > avg_loss_disc_real_5: 0.24489610232937503 (-0.004621876377454953) | > avg_loss_0: 2.863606587262221 (+0.011726483492784379) | > avg_grad_norm_0: tensor(11.0308, device='cuda:0') (+tensor(1.8338, device='cuda:0')) | > avg_loss_gen: 1.72164725753623 (-0.0035529673939020956) | > avg_loss_kl: 2.4595747195499045 (+0.009478263452018965) | > avg_loss_feat: 1.6023239803985811 (-0.016686372353996948) | > avg_loss_mel: 18.59838969270948 (+0.4726458133106526) | > avg_loss_duration: 1.6825711676772213 (+0.0019045144739284758) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.064507068043024 (+0.46378946975923085) | > avg_grad_norm_1: tensor(193.0348, device='cuda:0') (tensor(-21.4739, device='cuda:0'))  > EPOCH: 67/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:52:30)   --> TIME: 2025-02-25 08:52:42 -- STEP: 25/72 -- GLOBAL_STEP: 1004850 | > loss_disc: 2.838111162185669 (2.8587242126464845) | > loss_disc_real_0: 0.2646409273147583 (0.23010312795639037) | > loss_disc_real_1: 0.25168710947036743 (0.24857015430927276) | > loss_disc_real_2: 0.21542489528656006 (0.24474615693092347) | > loss_disc_real_3: 0.2494901865720749 (0.24472183346748352) | > loss_disc_real_4: 0.2582859992980957 (0.24486838042736053) | > loss_disc_real_5: 0.27310651540756226 (0.2496925449371337) | > loss_0: 2.838111162185669 (2.8587242126464845) | > grad_norm_0: tensor(6.6539, device='cuda:0') (tensor(8.6613, device='cuda:0')) | > loss_gen: 1.6985331773757935 (1.748354001045227) | > loss_kl: 2.6169345378875732 (2.504551067352296) | > loss_feat: 1.6943621635437012 (1.6653272819519043) | > loss_mel: 19.2904109954834 (18.767690429687505) | > loss_duration: 1.6339256763458252 (1.6678301525115966) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.934167861938477 (26.353753128051757) | > grad_norm_1: tensor(77.7440, device='cuda:0') (tensor(209.6896, device='cuda:0')) | > current_lr_0: 0.00019833189069947573 | > current_lr_1: 0.00019833189069947573 | > step_time: 0.4228 (0.4308187961578369) | > loader_time: 0.0076 (0.006964130401611328) --> EVAL PERFORMANCE | > avg_loader_time: 0.008604456001604106 (+0.0002410747635532423) | > avg_step_time: 0.45123613720208827 (+0.000482424883775312) | > avg_loss_disc: 2.8653656899089555 (+0.0017591026467345117) | > avg_loss_disc_real_0: 0.2361263311664823 (+0.004025537363240406) | > avg_loss_disc_real_1: 0.23941778234193023 (+0.0009510901192544274) | > avg_loss_disc_real_2: 0.2430641575598381 (-0.0017925845065586976) | > avg_loss_disc_real_3: 0.24318679516584102 (+0.0011600616952063547) | > avg_loss_disc_real_4: 0.24618024872222416 (+0.0003845821803724059) | > avg_loss_disc_real_5: 0.24942749780668338 (+0.004531395477308353) | > avg_loss_0: 2.8653656899089555 (+0.0017591026467345117) | > avg_grad_norm_0: tensor(10.1630, device='cuda:0') (tensor(-0.8678, device='cuda:0')) | > avg_loss_gen: 1.727036793467025 (+0.005389535930794942) | > avg_loss_kl: 2.4751670293405024 (+0.015592309790597891) | > avg_loss_feat: 1.59926919198372 (-0.003054788414861065) | > avg_loss_mel: 18.404993379619764 (-0.19339631308971406) | > avg_loss_duration: 1.6836964647534867 (+0.001125297076265408) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.89016291121362 (-0.17434415682940596) | > avg_grad_norm_1: tensor(196.3245, device='cuda:0') (+tensor(3.2897, device='cuda:0'))  > EPOCH: 68/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:53:06)   --> TIME: 2025-02-25 08:53:10 -- STEP: 3/72 -- GLOBAL_STEP: 1004900 | > loss_disc: 2.792980909347534 (2.8005477587381997) | > loss_disc_real_0: 0.2047598510980606 (0.19731472929318747) | > loss_disc_real_1: 0.16917984187602997 (0.18932140370210013) | > loss_disc_real_2: 0.16995784640312195 (0.24051589767138162) | > loss_disc_real_3: 0.21965377032756805 (0.23553876082102457) | > loss_disc_real_4: 0.23361361026763916 (0.24067358672618866) | > loss_disc_real_5: 0.23979723453521729 (0.24127432703971863) | > loss_0: 2.792980909347534 (2.8005477587381997) | > grad_norm_0: tensor(12.8941, device='cuda:0') (tensor(9.7003, device='cuda:0')) | > loss_gen: 1.751641869544983 (1.667905290921529) | > loss_kl: 2.5913009643554688 (2.617438316345215) | > loss_feat: 2.0342490673065186 (1.835221568743388) | > loss_mel: 19.666044235229492 (18.620386759440105) | > loss_duration: 1.7034912109375 (1.6847174564997356) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.746728897094727 (26.425670623779297) | > grad_norm_1: tensor(378.3166, device='cuda:0') (tensor(308.3279, device='cuda:0')) | > current_lr_0: 0.0001983070992131383 | > current_lr_1: 0.0001983070992131383 | > step_time: 0.4231 (0.4295247395833333) | > loader_time: 0.0054 (0.005953311920166016)  --> TIME: 2025-02-25 08:53:33 -- STEP: 53/72 -- GLOBAL_STEP: 1004950 | > loss_disc: 2.8841655254364014 (2.8577777889539613) | > loss_disc_real_0: 0.2060757577419281 (0.22993211701231184) | > loss_disc_real_1: 0.20003622770309448 (0.23140764714411968) | > loss_disc_real_2: 0.22752971947193146 (0.24110156269568317) | > loss_disc_real_3: 0.21575085818767548 (0.2433890242059276) | > loss_disc_real_4: 0.23478594422340393 (0.24618661291194413) | > loss_disc_real_5: 0.22109901905059814 (0.2467680427263368) | > loss_0: 2.8841655254364014 (2.8577777889539613) | > grad_norm_0: tensor(13.7719, device='cuda:0') (tensor(8.8410, device='cuda:0')) | > loss_gen: 1.6201080083847046 (1.7134176222783215) | > loss_kl: 2.3424248695373535 (2.4300130268312863) | > loss_feat: 1.5010193586349487 (1.688233049410694) | > loss_mel: 17.707239151000977 (18.310369491577152) | > loss_duration: 1.7286468744277954 (1.6716177058669757) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.899438858032227 (25.813650814992076) | > grad_norm_1: tensor(61.8485, device='cuda:0') (tensor(219.3347, device='cuda:0')) | > current_lr_0: 0.0001983070992131383 | > current_lr_1: 0.0001983070992131383 | > step_time: 0.4455 (0.4409812891258384) | > loader_time: 0.0091 (0.007657001603324458) --> EVAL PERFORMANCE | > avg_loader_time: 0.008450007774460484 (-0.00015444822714362136) | > avg_step_time: 0.45316291527009345 (+0.001926778068005175) | > avg_loss_disc: 2.8521756823633755 (-0.013190007545579974) | > avg_loss_disc_real_0: 0.22817978011050694 (-0.00794655105597536) | > avg_loss_disc_real_1: 0.2351843645455132 (-0.004233417796417033) | > avg_loss_disc_real_2: 0.24244674305680772 (-0.0006174145030303779) | > avg_loss_disc_real_3: 0.2421194892114317 (-0.00106730595440932) | > avg_loss_disc_real_4: 0.24524168208451338 (-0.0009385666377107749) | > avg_loss_disc_real_5: 0.2457114728823514 (-0.0037160249243319843) | > avg_loss_0: 2.8521756823633755 (-0.013190007545579974) | > avg_grad_norm_0: tensor(8.6315, device='cuda:0') (tensor(-1.5314, device='cuda:0')) | > avg_loss_gen: 1.7205414889563977 (-0.006495304510627298) | > avg_loss_kl: 2.4539920947921114 (-0.021174934548390922) | > avg_loss_feat: 1.6737407395537471 (+0.07447154757002705) | > avg_loss_mel: 18.383905491358803 (-0.021087888260961307) | > avg_loss_duration: 1.6795125863921474 (-0.004183878361339266) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.911692498435436 (+0.02152958722181708) | > avg_grad_norm_1: tensor(225.9081, device='cuda:0') (+tensor(29.5836, device='cuda:0'))  > EPOCH: 69/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:53:43)   --> TIME: 2025-02-25 08:53:59 -- STEP: 31/72 -- GLOBAL_STEP: 1005000 | > loss_disc: 2.88020658493042 (2.8558029743932907) | > loss_disc_real_0: 0.25921809673309326 (0.23349157889043132) | > loss_disc_real_1: 0.1984739750623703 (0.23444981776898907) | > loss_disc_real_2: 0.21922732889652252 (0.23742005853883683) | > loss_disc_real_3: 0.23514579236507416 (0.2429834839797789) | > loss_disc_real_4: 0.21972054243087769 (0.24366771261538228) | > loss_disc_real_5: 0.22057785093784332 (0.24282492264624564) | > loss_0: 2.88020658493042 (2.8558029743932907) | > grad_norm_0: tensor(3.3912, device='cuda:0') (tensor(8.0085, device='cuda:0')) | > loss_gen: 1.7458689212799072 (1.7149726806148406) | > loss_kl: 2.5564374923706055 (2.434836556834559) | > loss_feat: 1.5610660314559937 (1.6764402850981681) | > loss_mel: 19.012895584106445 (18.24597518674789) | > loss_duration: 1.6750812530517578 (1.6553473203412947) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.551349639892578 (25.727572164227887) | > grad_norm_1: tensor(82.0240, device='cuda:0') (tensor(233.6124, device='cuda:0')) | > current_lr_0: 0.00019828231082573666 | > current_lr_1: 0.00019828231082573666 | > step_time: 0.4462 (0.4321405656876103) | > loader_time: 0.0085 (0.00693509655614053) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1005000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008527282258154639 (+7.727448369415461e-05) | > avg_step_time: 0.45036829693216673 (-0.002794618337926713) | > avg_loss_disc: 2.861509269391987 (+0.009333587028611312) | > avg_loss_disc_real_0: 0.23181007676561113 (+0.003630296655104198) | > avg_loss_disc_real_1: 0.23565732845118348 (+0.0004729639056702817) | > avg_loss_disc_real_2: 0.24018584632537734 (-0.002260896731430373) | > avg_loss_disc_real_3: 0.24245458064784467 (+0.0003350914364129709) | > avg_loss_disc_real_4: 0.24452068919027356 (-0.0007209928942398247) | > avg_loss_disc_real_5: 0.24501653519314778 (-0.0006949376892036174) | > avg_loss_0: 2.861509269391987 (+0.009333587028611312) | > avg_grad_norm_0: tensor(8.2683, device='cuda:0') (tensor(-0.3632, device='cuda:0')) | > avg_loss_gen: 1.7024013089462064 (-0.018140180010191242) | > avg_loss_kl: 2.4353465765294895 (-0.018645518262621952) | > avg_loss_feat: 1.6176793541706784 (-0.05606138538306871) | > avg_loss_mel: 18.107040230657013 (-0.27686526070178985) | > avg_loss_duration: 1.6755447102264618 (-0.003967876165685613) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.538012276233083 (-0.37368022220235275) | > avg_grad_norm_1: tensor(216.6890, device='cuda:0') (tensor(-9.2191, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1005041.pth  > EPOCH: 70/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:54:25)   --> TIME: 2025-02-25 08:54:31 -- STEP: 9/72 -- GLOBAL_STEP: 1005050 | > loss_disc: 2.8284387588500977 (2.8409921328226724) | > loss_disc_real_0: 0.2338678389787674 (0.24406239059236315) | > loss_disc_real_1: 0.2625011205673218 (0.23919519947634804) | > loss_disc_real_2: 0.23587952554225922 (0.23931928310129377) | > loss_disc_real_3: 0.23338717222213745 (0.23986412915918562) | > loss_disc_real_4: 0.26560285687446594 (0.24692904286914402) | > loss_disc_real_5: 0.2600603997707367 (0.24635240932305655) | > loss_0: 2.8284387588500977 (2.8409921328226724) | > grad_norm_0: tensor(4.9568, device='cuda:0') (tensor(9.4487, device='cuda:0')) | > loss_gen: 1.6667253971099854 (1.740802076127794) | > loss_kl: 2.2921295166015625 (2.5866204367743597) | > loss_feat: 1.6664109230041504 (1.62271867858039) | > loss_mel: 18.493732452392578 (18.123147540622288) | > loss_duration: 1.6680991649627686 (1.6830500496758356) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.78709602355957 (25.75633896721734) | > grad_norm_1: tensor(47.8397, device='cuda:0') (tensor(290.1562, device='cuda:0')) | > current_lr_0: 0.00019825752553688343 | > current_lr_1: 0.00019825752553688343 | > step_time: 0.4225 (0.4297192361619737) | > loader_time: 0.0056 (0.0064800050523546)  --> TIME: 2025-02-25 08:54:54 -- STEP: 59/72 -- GLOBAL_STEP: 1005100 | > loss_disc: 2.8100428581237793 (2.8573180376473126) | > loss_disc_real_0: 0.2021401971578598 (0.2312783242787345) | > loss_disc_real_1: 0.1943504959344864 (0.23456210024276022) | > loss_disc_real_2: 0.243153378367424 (0.2409320261518834) | > loss_disc_real_3: 0.23977723717689514 (0.24512599029783474) | > loss_disc_real_4: 0.26211512088775635 (0.24627139507713963) | > loss_disc_real_5: 0.2670755684375763 (0.24910253690460982) | > loss_0: 2.8100428581237793 (2.8573180376473126) | > grad_norm_0: tensor(6.2045, device='cuda:0') (tensor(7.7693, device='cuda:0')) | > loss_gen: 1.8488173484802246 (1.7188405283426835) | > loss_kl: 2.342414379119873 (2.495618266574407) | > loss_feat: 1.5830343961715698 (1.6348160121400477) | > loss_mel: 17.121068954467773 (18.313494827787746) | > loss_duration: 1.6916083097457886 (1.6747149794788685) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.586944580078125 (25.837484941644185) | > grad_norm_1: tensor(213.7298, device='cuda:0') (tensor(175.2543, device='cuda:0')) | > current_lr_0: 0.00019825752553688343 | > current_lr_1: 0.00019825752553688343 | > step_time: 0.485 (0.4408773123207739) | > loader_time: 0.0097 (0.007970797813544837) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855339748758665 (+2.611522943201175e-05) | > avg_step_time: 0.45162401065020497 (+0.0012557137180382316) | > avg_loss_disc: 2.856547436244051 (-0.004961833147935746) | > avg_loss_disc_real_0: 0.2311108794430612 (-0.0006991973225499437) | > avg_loss_disc_real_1: 0.23525757772821776 (-0.00039975072296571934) | > avg_loss_disc_real_2: 0.2407837302332193 (+0.0005978839078419618) | > avg_loss_disc_real_3: 0.24387338295788832 (+0.0014188023100436542) | > avg_loss_disc_real_4: 0.24495098645418462 (+0.00043029726391105583) | > avg_loss_disc_real_5: 0.246791680933724 (+0.0017751457405762172) | > avg_loss_0: 2.856547436244051 (-0.004961833147935746) | > avg_grad_norm_0: tensor(8.2463, device='cuda:0') (tensor(-0.0220, device='cuda:0')) | > avg_loss_gen: 1.7111253402602504 (+0.008724031314043978) | > avg_loss_kl: 2.5006303686491202 (+0.06528379211963076) | > avg_loss_feat: 1.6254788368520603 (+0.007799482681381864) | > avg_loss_mel: 18.29026501615282 (+0.18322478549580623) | > avg_loss_duration: 1.67900430148756 (+0.0034595912610981827) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.80650407495633 (+0.2684917987232467) | > avg_grad_norm_1: tensor(179.5515, device='cuda:0') (tensor(-37.1374, device='cuda:0'))  > EPOCH: 71/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:55:02)   --> TIME: 2025-02-25 08:55:20 -- STEP: 37/72 -- GLOBAL_STEP: 1005150 | > loss_disc: 2.810821533203125 (2.8530902798111373) | > loss_disc_real_0: 0.27082765102386475 (0.230627505763157) | > loss_disc_real_1: 0.21721482276916504 (0.2297666473565875) | > loss_disc_real_2: 0.17970117926597595 (0.23854572627995466) | > loss_disc_real_3: 0.19739145040512085 (0.24240086731073018) | > loss_disc_real_4: 0.20575544238090515 (0.2431505435221904) | > loss_disc_real_5: 0.20660565793514252 (0.24734629892014168) | > loss_0: 2.810821533203125 (2.8530902798111373) | > grad_norm_0: tensor(6.5514, device='cuda:0') (tensor(8.5888, device='cuda:0')) | > loss_gen: 1.6635339260101318 (1.7017445822019832) | > loss_kl: 2.4243862628936768 (2.4802605654742265) | > loss_feat: 1.7951290607452393 (1.6670856250298989) | > loss_mel: 18.675933837890625 (18.6642349346264) | > loss_duration: 1.6867473125457764 (1.6654378175735474) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.245729446411133 (26.178763363812422) | > grad_norm_1: tensor(61.0072, device='cuda:0') (tensor(189.8332, device='cuda:0')) | > current_lr_0: 0.0001982327433461913 | > current_lr_1: 0.0001982327433461913 | > step_time: 0.4419 (0.4342161835850896) | > loader_time: 0.0084 (0.007117761147988809) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864141423937301 (+8.801675178635865e-05) | > avg_step_time: 0.4517893824778812 (+0.0001653718276762306) | > avg_loss_disc: 2.8533990013767303 (-0.0031484348673207485) | > avg_loss_disc_real_0: 0.23298955677260816 (+0.0018786773295469683) | > avg_loss_disc_real_1: 0.2350516878593136 (-0.00020588986890415772) | > avg_loss_disc_real_2: 0.24218710627354367 (+0.0014033760403243667) | > avg_loss_disc_real_3: 0.24298608177144762 (-0.0008873011864407032) | > avg_loss_disc_real_4: 0.24549511977484528 (+0.0005441333206606669) | > avg_loss_disc_real_5: 0.2480824004176637 (+0.0012907194839396896) | > avg_loss_0: 2.8533990013767303 (-0.0031484348673207485) | > avg_grad_norm_0: tensor(8.5679, device='cuda:0') (+tensor(0.3216, device='cuda:0')) | > avg_loss_gen: 1.7186290697312692 (+0.0075037294710187386) | > avg_loss_kl: 2.5099289988128226 (+0.00929863016370236) | > avg_loss_feat: 1.6524961028300535 (+0.027017265977993254) | > avg_loss_mel: 18.59331896607305 (+0.3030539499202298) | > avg_loss_duration: 1.6832626218527136 (+0.004258320365153612) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.15763586339816 (+0.3511317884418297) | > avg_grad_norm_1: tensor(201.9945, device='cuda:0') (+tensor(22.4429, device='cuda:0'))  > EPOCH: 72/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:55:39)   --> TIME: 2025-02-25 08:55:47 -- STEP: 15/72 -- GLOBAL_STEP: 1005200 | > loss_disc: 2.850466728210449 (2.8452314058939616) | > loss_disc_real_0: 0.2432892620563507 (0.22665836612383525) | > loss_disc_real_1: 0.18925677239894867 (0.2336742450793584) | > loss_disc_real_2: 0.2692889869213104 (0.24280764957269033) | > loss_disc_real_3: 0.2821803689002991 (0.24566096564133963) | > loss_disc_real_4: 0.29261624813079834 (0.246475014090538) | > loss_disc_real_5: 0.2873423993587494 (0.24285610616207123) | > loss_0: 2.850466728210449 (2.8452314058939616) | > grad_norm_0: tensor(5.6809, device='cuda:0') (tensor(7.9221, device='cuda:0')) | > loss_gen: 1.7407225370407104 (1.7304233630498251) | > loss_kl: 2.5074830055236816 (2.4565542856852214) | > loss_feat: 1.5671511888504028 (1.693858806292216) | > loss_mel: 20.90087127685547 (18.555100123087566) | > loss_duration: 1.6358002424240112 (1.6670158863067628) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.352027893066406 (26.10295232137044) | > grad_norm_1: tensor(145.7686, device='cuda:0') (tensor(188.5617, device='cuda:0')) | > current_lr_0: 0.00019820796425327303 | > current_lr_1: 0.00019820796425327303 | > step_time: 0.4289 (0.4353130976359049) | > loader_time: 0.0074 (0.006827052434285482)  --> TIME: 2025-02-25 08:56:10 -- STEP: 65/72 -- GLOBAL_STEP: 1005250 | > loss_disc: 2.792635440826416 (2.8572427309476414) | > loss_disc_real_0: 0.2534359395503998 (0.2294968598164045) | > loss_disc_real_1: 0.27439844608306885 (0.23509204823237198) | > loss_disc_real_2: 0.20111443102359772 (0.238500632918798) | > loss_disc_real_3: 0.2269127517938614 (0.24213187946723058) | > loss_disc_real_4: 0.24194316565990448 (0.24541323781013488) | > loss_disc_real_5: 0.23769402503967285 (0.2459368448991042) | > loss_0: 2.792635440826416 (2.8572427309476414) | > grad_norm_0: tensor(10.3410, device='cuda:0') (tensor(10.2111, device='cuda:0')) | > loss_gen: 1.8235875368118286 (1.7118660413301907) | > loss_kl: 2.568059206008911 (2.4649172452779915) | > loss_feat: 1.8164069652557373 (1.642987821652339) | > loss_mel: 19.155546188354492 (18.4417466383714) | > loss_duration: 1.6965388059616089 (1.675748872756958) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.060138702392578 (25.93726639380822) | > grad_norm_1: tensor(272.6232, device='cuda:0') (tensor(191.6394, device='cuda:0')) | > current_lr_0: 0.00019820796425327303 | > current_lr_1: 0.00019820796425327303 | > step_time: 0.4868 (0.44580775040846604) | > loader_time: 0.0116 (0.008385984714214619) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647717220682492 (+6.302981309483022e-06) | > avg_step_time: 0.45084655788582817 (-0.0009428245920530265) | > avg_loss_disc: 2.8548914210897096 (+0.0014924197129793093) | > avg_loss_disc_real_0: 0.23020297141981796 (-0.002786585352790194) | > avg_loss_disc_real_1: 0.23421539618095882 (-0.000836291678354778) | > avg_loss_disc_real_2: 0.23830550762129502 (-0.003881598652248658) | > avg_loss_disc_real_3: 0.24198359104109482 (-0.0010024907303528008) | > avg_loss_disc_real_4: 0.24454964991186706 (-0.000945469862978221) | > avg_loss_disc_real_5: 0.24505568956825094 (-0.0030267108494127504) | > avg_loss_0: 2.8548914210897096 (+0.0014924197129793093) | > avg_grad_norm_0: tensor(10.1709, device='cuda:0') (+tensor(1.6030, device='cuda:0')) | > avg_loss_gen: 1.711098869081954 (-0.0075302006493152) | > avg_loss_kl: 2.463611992312149 (-0.0463170065006735) | > avg_loss_feat: 1.6408635814424972 (-0.011632521387556327) | > avg_loss_mel: 18.429836380649622 (-0.1634825854234272) | > avg_loss_duration: 1.6782698765606947 (-0.004992745292018874) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.92368053382551 (-0.2339553295726482) | > avg_grad_norm_1: tensor(197.5942, device='cuda:0') (tensor(-4.4003, device='cuda:0'))  > EPOCH: 73/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:56:15)   --> TIME: 2025-02-25 08:56:36 -- STEP: 43/72 -- GLOBAL_STEP: 1005300 | > loss_disc: 2.890089273452759 (2.8594547537870185) | > loss_disc_real_0: 0.23613108694553375 (0.23001156989918198) | > loss_disc_real_1: 0.23362191021442413 (0.23614915304405745) | > loss_disc_real_2: 0.23431651294231415 (0.24176117050093274) | > loss_disc_real_3: 0.24075812101364136 (0.24422294282635978) | > loss_disc_real_4: 0.24321606755256653 (0.24827921598456623) | > loss_disc_real_5: 0.2528892755508423 (0.252504637075025) | > loss_0: 2.890089273452759 (2.8594547537870185) | > grad_norm_0: tensor(8.1580, device='cuda:0') (tensor(8.6529, device='cuda:0')) | > loss_gen: 1.6736762523651123 (1.7279714429101278) | > loss_kl: 2.3221781253814697 (2.3956856699877) | > loss_feat: 1.580487608909607 (1.692995739537616) | > loss_mel: 16.75711441040039 (18.388033622919128) | > loss_duration: 1.6722352504730225 (1.6621904428615126) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.005691528320312 (25.866876823957575) | > grad_norm_1: tensor(150.3445, device='cuda:0') (tensor(247.3045, device='cuda:0')) | > current_lr_0: 0.00019818318825774137 | > current_lr_1: 0.00019818318825774137 | > step_time: 0.4518 (0.43354374308918797) | > loader_time: 0.0094 (0.007300077482711437) --> EVAL PERFORMANCE | > avg_loader_time: 0.008452166973705022 (-0.00019555024697747066) | > avg_step_time: 0.4482398335362824 (-0.002606724349545786) | > avg_loss_disc: 2.857885679728548 (+0.0029942586388385095) | > avg_loss_disc_real_0: 0.23090795018303562 (+0.0007049787632176507) | > avg_loss_disc_real_1: 0.23705801787510725 (+0.002842621694148423) | > avg_loss_disc_real_2: 0.243241247996478 (+0.00493574037518299) | > avg_loss_disc_real_3: 0.2445204199199945 (+0.002536828878899694) | > avg_loss_disc_real_4: 0.24669256575510534 (+0.00214291584323828) | > avg_loss_disc_real_5: 0.24935563307412914 (+0.004299943505878201) | > avg_loss_0: 2.857885679728548 (+0.0029942586388385095) | > avg_grad_norm_0: tensor(8.5734, device='cuda:0') (tensor(-1.5976, device='cuda:0')) | > avg_loss_gen: 1.7236461320393521 (+0.012547262957398164) | > avg_loss_kl: 2.4131851548879917 (-0.050426837424157434) | > avg_loss_feat: 1.6663751669333016 (+0.025511585490804345) | > avg_loss_mel: 18.215060543006576 (-0.21477583764304597) | > avg_loss_duration: 1.675156076189498 (-0.0031138003711967954) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.693423015970584 (-0.23025751785492687) | > avg_grad_norm_1: tensor(228.4740, device='cuda:0') (+tensor(30.8799, device='cuda:0'))  > EPOCH: 74/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:56:52)   --> TIME: 2025-02-25 08:57:03 -- STEP: 21/72 -- GLOBAL_STEP: 1005350 | > loss_disc: 2.909548282623291 (2.835058632351103) | > loss_disc_real_0: 0.2182680070400238 (0.2183205158937545) | > loss_disc_real_1: 0.2520752251148224 (0.24671648300829388) | > loss_disc_real_2: 0.24684783816337585 (0.23934777293886458) | > loss_disc_real_3: 0.262095183134079 (0.24675360180082775) | > loss_disc_real_4: 0.2733018696308136 (0.24509248918011076) | > loss_disc_real_5: 0.2780856788158417 (0.24297322758606502) | > loss_0: 2.909548282623291 (2.835058632351103) | > grad_norm_0: tensor(4.4043, device='cuda:0') (tensor(9.8512, device='cuda:0')) | > loss_gen: 1.7130038738250732 (1.7490785746347337) | > loss_kl: 2.2985928058624268 (2.518406867980957) | > loss_feat: 1.3334720134735107 (1.739604910214742) | > loss_mel: 17.30977439880371 (18.675301143101287) | > loss_duration: 1.6956875324249268 (1.6658230395544142) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.35053062438965 (26.348214558192662) | > grad_norm_1: tensor(192.2877, device='cuda:0') (tensor(158.7008, device='cuda:0')) | > current_lr_0: 0.00019815841535920914 | > current_lr_1: 0.00019815841535920914 | > step_time: 0.4536 (0.4349487168448312) | > loader_time: 0.0066 (0.006708167848132905)  --> TIME: 2025-02-25 08:57:27 -- STEP: 71/72 -- GLOBAL_STEP: 1005400 | > loss_disc: 2.9909119606018066 (2.849827692542278) | > loss_disc_real_0: 0.3414369523525238 (0.22777043450886095) | > loss_disc_real_1: 0.19097881019115448 (0.24028252170119488) | > loss_disc_real_2: 0.2844803035259247 (0.2419626643540154) | > loss_disc_real_3: 0.26807576417922974 (0.24357833115147873) | > loss_disc_real_4: 0.27622291445732117 (0.24531069230025923) | > loss_disc_real_5: 0.2762675881385803 (0.24379730476459988) | > loss_0: 2.9909119606018066 (2.849827692542278) | > grad_norm_0: tensor(12.5598, device='cuda:0') (tensor(8.9664, device='cuda:0')) | > loss_gen: 1.785667896270752 (1.7307571139134152) | > loss_kl: 2.533198356628418 (2.473680073106792) | > loss_feat: 1.738784670829773 (1.6685777499642171) | > loss_mel: 18.718751907348633 (18.407632196453257) | > loss_duration: 1.6956455707550049 (1.6827924503407008) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.472047805786133 (25.96343972649373) | > grad_norm_1: tensor(199.3083, device='cuda:0') (tensor(191.6819, device='cuda:0')) | > current_lr_0: 0.00019815841535920914 | > current_lr_1: 0.00019815841535920914 | > step_time: 0.4976 (0.453129167288122) | > loader_time: 0.0083 (0.008542665293518926) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542665293518926 (+9.049831981390415e-05) | > avg_step_time: 0.453129167288122 (+0.004889333751839642) | > avg_loss_disc: 2.849827692542278 (-0.008057987186270044) | > avg_loss_disc_real_0: 0.22777043450886095 (-0.003137515674174668) | > avg_loss_disc_real_1: 0.24028252170119488 (+0.0032245038260876324) | > avg_loss_disc_real_2: 0.2419626643540154 (-0.0012785836424626107) | > avg_loss_disc_real_3: 0.24357833115147873 (-0.0009420887685157864) | > avg_loss_disc_real_4: 0.24531069230025923 (-0.001381873454846111) | > avg_loss_disc_real_5: 0.24379730476459988 (-0.005558328309529265) | > avg_loss_0: 2.849827692542278 (-0.008057987186270044) | > avg_grad_norm_0: tensor(8.9664, device='cuda:0') (+tensor(0.3930, device='cuda:0')) | > avg_loss_gen: 1.7307571139134152 (+0.007110981874063116) | > avg_loss_kl: 2.473680073106792 (+0.060494918218800375) | > avg_loss_feat: 1.6685777499642171 (+0.0022025830309155836) | > avg_loss_mel: 18.407632196453257 (+0.1925716534466808) | > avg_loss_duration: 1.6827924503407008 (+0.007636374151202885) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.96343972649373 (+0.27001671052314435) | > avg_grad_norm_1: tensor(191.6819, device='cuda:0') (tensor(-36.7921, device='cuda:0'))  > EPOCH: 75/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:57:29)   --> TIME: 2025-02-25 08:57:52 -- STEP: 49/72 -- GLOBAL_STEP: 1005450 | > loss_disc: 2.8605105876922607 (2.8710233046084035) | > loss_disc_real_0: 0.281941682100296 (0.23419511287796255) | > loss_disc_real_1: 0.2697787582874298 (0.236409420565683) | > loss_disc_real_2: 0.2415625900030136 (0.24183042438662783) | > loss_disc_real_3: 0.25373634696006775 (0.24508066323338723) | > loss_disc_real_4: 0.2764936089515686 (0.2493748515844345) | > loss_disc_real_5: 0.2728559970855713 (0.24935766507168203) | > loss_0: 2.8605105876922607 (2.8710233046084035) | > grad_norm_0: tensor(6.0903, device='cuda:0') (tensor(9.3501, device='cuda:0')) | > loss_gen: 1.6420700550079346 (1.7235968745484644) | > loss_kl: 2.2953133583068848 (2.427608227243229) | > loss_feat: 1.4242959022521973 (1.670667543703196) | > loss_mel: 17.92823600769043 (18.324967248099192) | > loss_duration: 1.7434343099594116 (1.6677255362880474) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.03335189819336 (25.814565658569336) | > grad_norm_1: tensor(81.7618, device='cuda:0') (tensor(211.8009, device='cuda:0')) | > current_lr_0: 0.00019813364555728923 | > current_lr_1: 0.00019813364555728923 | > step_time: 0.4398 (0.43651318063541333) | > loader_time: 0.0095 (0.007630207100693061) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850583466005997 (-3.683063345895528e-05) | > avg_step_time: 0.44820473899304025 (-0.004924428295081773) | > avg_loss_disc: 2.865070359807619 (+0.015242667265340959) | > avg_loss_disc_real_0: 0.23367451865908126 (+0.005904084150220312) | > avg_loss_disc_real_1: 0.23515723876550165 (-0.005125282935693226) | > avg_loss_disc_real_2: 0.2414244379795773 (-0.0005382263744380944) | > avg_loss_disc_real_3: 0.24412937924055986 (+0.0005510480890811298) | > avg_loss_disc_real_4: 0.24861213795735804 (+0.003301445657098806) | > avg_loss_disc_real_5: 0.2495169182058791 (+0.00571961344127922) | > avg_loss_0: 2.865070359807619 (+0.015242667265340959) | > avg_grad_norm_0: tensor(8.9754, device='cuda:0') (+tensor(0.0089, device='cuda:0')) | > avg_loss_gen: 1.719577109309989 (-0.011180004603426275) | > avg_loss_kl: 2.4273098220287914 (-0.046370251078000635) | > avg_loss_feat: 1.650300687467548 (-0.01827706249666905) | > avg_loss_mel: 18.380199244324597 (-0.027432952128659593) | > avg_loss_duration: 1.6762235601183395 (-0.00656889022236129) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.853610562606598 (-0.10982916388713093) | > avg_grad_norm_1: tensor(216.6683, device='cuda:0') (+tensor(24.9863, device='cuda:0'))  > EPOCH: 76/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:58:05)   --> TIME: 2025-02-25 08:58:19 -- STEP: 27/72 -- GLOBAL_STEP: 1005500 | > loss_disc: 2.7963969707489014 (2.8711316497237593) | > loss_disc_real_0: 0.272580623626709 (0.23786320123407576) | > loss_disc_real_1: 0.19737379252910614 (0.23526284429762098) | > loss_disc_real_2: 0.22215446829795837 (0.2432553497729478) | > loss_disc_real_3: 0.2220640331506729 (0.24332286693431712) | > loss_disc_real_4: 0.23229099810123444 (0.2487132449944814) | > loss_disc_real_5: 0.24183237552642822 (0.2518831232079754) | > loss_0: 2.7963969707489014 (2.8711316497237593) | > grad_norm_0: tensor(8.0373, device='cuda:0') (tensor(7.2512, device='cuda:0')) | > loss_gen: 1.6871098279953003 (1.7223981000758983) | > loss_kl: 2.326436758041382 (2.4240102149822094) | > loss_feat: 1.6185864210128784 (1.6425382340395893) | > loss_mel: 18.007118225097656 (18.115255214549876) | > loss_duration: 1.6438300609588623 (1.6688205798467) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.283079147338867 (25.573022489194518) | > grad_norm_1: tensor(201.0176, device='cuda:0') (tensor(224.3143, device='cuda:0')) | > current_lr_0: 0.00019810887885159456 | > current_lr_1: 0.00019810887885159456 | > step_time: 0.4291 (0.43099780435915347) | > loader_time: 0.008 (0.006773736741807725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008483799410537935 (-2.203524952203509e-05) | > avg_step_time: 0.44895440423992317 (+0.0007496652468829135) | > avg_loss_disc: 2.8616832041404616 (-0.0033871556671574155) | > avg_loss_disc_real_0: 0.23186053923318084 (-0.0018139794259004194) | > avg_loss_disc_real_1: 0.2353427231731549 (+0.00018548440765325802) | > avg_loss_disc_real_2: 0.2418747514906064 (+0.0004503135110290879) | > avg_loss_disc_real_3: 0.2429171105925466 (-0.0012122686480132705) | > avg_loss_disc_real_4: 0.2459047705354825 (-0.0027073674218755506) | > avg_loss_disc_real_5: 0.2474975671986459 (-0.002019351007233189) | > avg_loss_0: 2.8616832041404616 (-0.0033871556671574155) | > avg_grad_norm_0: tensor(8.3858, device='cuda:0') (tensor(-0.5895, device='cuda:0')) | > avg_loss_gen: 1.7168683132655185 (-0.002708796044470496) | > avg_loss_kl: 2.419711089470017 (-0.0075987325587743015) | > avg_loss_feat: 1.6511345463739315 (+0.0008338589063834334) | > avg_loss_mel: 18.077335250209753 (-0.3028639941148441) | > avg_loss_duration: 1.6807423292751043 (+0.004518769156764746) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.545791679704692 (-0.3078188829019055) | > avg_grad_norm_1: tensor(217.2332, device='cuda:0') (+tensor(0.5649, device='cuda:0'))  > EPOCH: 77/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:58:41)   --> TIME: 2025-02-25 08:58:45 -- STEP: 5/72 -- GLOBAL_STEP: 1005550 | > loss_disc: 2.72395920753479 (2.814996528625488) | > loss_disc_real_0: 0.2720147371292114 (0.23146049976348876) | > loss_disc_real_1: 0.23396673798561096 (0.2358856290578842) | > loss_disc_real_2: 0.21150429546833038 (0.2283957153558731) | > loss_disc_real_3: 0.20758260786533356 (0.23739745914936067) | > loss_disc_real_4: 0.23475193977355957 (0.24222363233566285) | > loss_disc_real_5: 0.24117229878902435 (0.24578551054000855) | > loss_0: 2.72395920753479 (2.814996528625488) | > grad_norm_0: tensor(7.3141, device='cuda:0') (tensor(8.3490, device='cuda:0')) | > loss_gen: 1.815287470817566 (1.726225233078003) | > loss_kl: 2.2854294776916504 (2.436672019958496) | > loss_feat: 1.9383132457733154 (1.6435854196548463) | > loss_mel: 18.006906509399414 (17.977390670776366) | > loss_duration: 1.632140874862671 (1.6760910511016847) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.678075790405273 (25.459964752197266) | > grad_norm_1: tensor(139.9421, device='cuda:0') (tensor(184.9381, device='cuda:0')) | > current_lr_0: 0.0001980841152417381 | > current_lr_1: 0.0001980841152417381 | > step_time: 0.4203 (0.41947221755981445) | > loader_time: 0.0061 (0.006465387344360351)  --> TIME: 2025-02-25 08:59:08 -- STEP: 55/72 -- GLOBAL_STEP: 1005600 | > loss_disc: 2.8589742183685303 (2.8442533536390826) | > loss_disc_real_0: 0.2649068236351013 (0.22729939317161388) | > loss_disc_real_1: 0.17416824400424957 (0.23175730217586865) | > loss_disc_real_2: 0.2464473694562912 (0.23964316519823942) | > loss_disc_real_3: 0.2502548098564148 (0.2418990284204483) | > loss_disc_real_4: 0.2625226378440857 (0.24719381440769542) | > loss_disc_real_5: 0.2460917979478836 (0.24682031815702266) | > loss_0: 2.8589742183685303 (2.8442533536390826) | > grad_norm_0: tensor(19.4921, device='cuda:0') (tensor(9.1696, device='cuda:0')) | > loss_gen: 1.7728049755096436 (1.7226262721148404) | > loss_kl: 2.378443717956543 (2.448891271244396) | > loss_feat: 1.4947761297225952 (1.7039690537886185) | > loss_mel: 17.756275177001953 (18.06703903891824) | > loss_duration: 1.7048263549804688 (1.6719985723495483) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.107126235961914 (25.61452435580167) | > grad_norm_1: tensor(47.7852, device='cuda:0') (tensor(191.5772, device='cuda:0')) | > current_lr_0: 0.0001980841152417381 | > current_lr_1: 0.0001980841152417381 | > step_time: 0.4559 (0.4391085841438987) | > loader_time: 0.0092 (0.007928076657381926) --> EVAL PERFORMANCE | > avg_loader_time: 0.008604885826648121 (+0.00012108641611018565) | > avg_step_time: 0.4564982602294062 (+0.007543855989483039) | > avg_loss_disc: 2.8592332450436873 (-0.002449959096774368) | > avg_loss_disc_real_0: 0.23302411728761566 (+0.0011635780544348162) | > avg_loss_disc_real_1: 0.23427298287270773 (-0.0010697403004471806) | > avg_loss_disc_real_2: 0.2407339057871993 (-0.001140845703407084) | > avg_loss_disc_real_3: 0.24313014493861668 (+0.00021303434607009009) | > avg_loss_disc_real_4: 0.24797631758199612 (+0.002071547046513633) | > avg_loss_disc_real_5: 0.24778120891309122 (+0.0002836417144453096) | > avg_loss_0: 2.8592332450436873 (-0.002449959096774368) | > avg_grad_norm_0: tensor(9.1560, device='cuda:0') (+tensor(0.7702, device='cuda:0')) | > avg_loss_gen: 1.721922199491044 (+0.005053886225525606) | > avg_loss_kl: 2.468578597189675 (+0.048867507719657866) | > avg_loss_feat: 1.6862331108308175 (+0.03509856445688597) | > avg_loss_mel: 18.13225888534331 (+0.05492363513355869) | > avg_loss_duration: 1.6778263004732803 (-0.0029160288018239555) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68681910340215 (+0.1410274236974587) | > avg_grad_norm_1: tensor(208.1187, device='cuda:0') (tensor(-9.1146, device='cuda:0'))  > EPOCH: 78/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:59:18)   --> TIME: 2025-02-25 08:59:35 -- STEP: 33/72 -- GLOBAL_STEP: 1005650 | > loss_disc: 2.790376663208008 (2.8566062811649213) | > loss_disc_real_0: 0.17264452576637268 (0.23312380322904297) | > loss_disc_real_1: 0.2591272294521332 (0.23009059781377966) | > loss_disc_real_2: 0.23839600384235382 (0.24054804308847946) | > loss_disc_real_3: 0.23863646388053894 (0.24346944057580197) | > loss_disc_real_4: 0.23291859030723572 (0.24457054743261047) | > loss_disc_real_5: 0.23612362146377563 (0.24668677512443427) | > loss_0: 2.790376663208008 (2.8566062811649213) | > grad_norm_0: tensor(6.6469, device='cuda:0') (tensor(8.9917, device='cuda:0')) | > loss_gen: 1.781074047088623 (1.7125915469545308) | > loss_kl: 2.461029529571533 (2.4425124977574204) | > loss_feat: 1.897339105606079 (1.6338420817346284) | > loss_mel: 19.391494750976562 (18.339348590735234) | > loss_duration: 1.6695436239242554 (1.6657922448533955) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.200481414794922 (25.794087207678594) | > grad_norm_1: tensor(143.2126, device='cuda:0') (tensor(140.2878, device='cuda:0')) | > current_lr_0: 0.00019805935472733287 | > current_lr_1: 0.00019805935472733287 | > step_time: 0.449 (0.43367233420863294) | > loader_time: 0.0078 (0.00736074736624053) --> EVAL PERFORMANCE | > avg_loader_time: 0.008648163835767289 (+4.327800911916799e-05) | > avg_step_time: 0.44983687870939015 (-0.006661381520016052) | > avg_loss_disc: 2.8581796061824747 (-0.0010536388612125691) | > avg_loss_disc_real_0: 0.232213051596158 (-0.0008110656914576686) | > avg_loss_disc_real_1: 0.23258839505659024 (-0.0016845878161174943) | > avg_loss_disc_real_2: 0.24204493450446868 (+0.0013110287172693746) | > avg_loss_disc_real_3: 0.24198980461543715 (-0.0011403403231795284) | > avg_loss_disc_real_4: 0.24413669822921216 (-0.0038396193527839573) | > avg_loss_disc_real_5: 0.24475536044214813 (-0.0030258484709430877) | > avg_loss_0: 2.8581796061824747 (-0.0010536388612125691) | > avg_grad_norm_0: tensor(8.6579, device='cuda:0') (tensor(-0.4980, device='cuda:0')) | > avg_loss_gen: 1.7079949865878468 (-0.013927212903197317) | > avg_loss_kl: 2.421348330000757 (-0.047230267188917985) | > avg_loss_feat: 1.6364348337683878 (-0.049798277062429674) | > avg_loss_mel: 18.217871115241252 (+0.08561222989794004) | > avg_loss_duration: 1.6808108527895431 (+0.002984552316262823) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.66446038367043 (-0.0223587197317201) | > avg_grad_norm_1: tensor(173.9222, device='cuda:0') (tensor(-34.1965, device='cuda:0'))  > EPOCH: 79/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:59:55)   --> TIME: 2025-02-25 09:00:01 -- STEP: 11/72 -- GLOBAL_STEP: 1005700 | > loss_disc: 2.7831950187683105 (2.838224107568914) | > loss_disc_real_0: 0.21808256208896637 (0.22882152958349747) | > loss_disc_real_1: 0.2626716196537018 (0.2366937886584889) | > loss_disc_real_2: 0.24606268107891083 (0.2318264530463652) | > loss_disc_real_3: 0.21672222018241882 (0.23693794282999905) | > loss_disc_real_4: 0.24890807271003723 (0.24557337300343948) | > loss_disc_real_5: 0.24879519641399384 (0.2455155524340543) | > loss_0: 2.7831950187683105 (2.838224107568914) | > grad_norm_0: tensor(5.5364, device='cuda:0') (tensor(9.3621, device='cuda:0')) | > loss_gen: 1.7399327754974365 (1.7067810730500654) | > loss_kl: 2.069121837615967 (2.340772520412098) | > loss_feat: 1.7643928527832031 (1.6640893762761897) | > loss_mel: 18.985366821289062 (18.164338805458762) | > loss_duration: 1.6322674751281738 (1.6578261202031916) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.191082000732422 (25.533808274702594) | > grad_norm_1: tensor(382.9642, device='cuda:0') (tensor(135.5414, device='cuda:0')) | > current_lr_0: 0.00019803459730799195 | > current_lr_1: 0.00019803459730799195 | > step_time: 0.4368 (0.4272426041689786) | > loader_time: 0.0057 (0.006252874027598988)  --> TIME: 2025-02-25 09:00:24 -- STEP: 61/72 -- GLOBAL_STEP: 1005750 | > loss_disc: 2.8677265644073486 (2.8578592480206098) | > loss_disc_real_0: 0.29782193899154663 (0.22972287921631923) | > loss_disc_real_1: 0.2489088922739029 (0.23633389443647665) | > loss_disc_real_2: 0.26545628905296326 (0.24194612390682346) | > loss_disc_real_3: 0.24783086776733398 (0.24494173096828772) | > loss_disc_real_4: 0.25405168533325195 (0.2464844877602624) | > loss_disc_real_5: 0.2515145242214203 (0.24723643113355168) | > loss_0: 2.8677265644073486 (2.8578592480206098) | > grad_norm_0: tensor(8.9503, device='cuda:0') (tensor(8.3542, device='cuda:0')) | > loss_gen: 1.6459076404571533 (1.7242548309388708) | > loss_kl: 2.2714312076568604 (2.4939640975389317) | > loss_feat: 1.4693413972854614 (1.6783210996721611) | > loss_mel: 16.473316192626953 (18.340850251619926) | > loss_duration: 1.6963646411895752 (1.6744895079096809) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.55636215209961 (25.91187989907186) | > grad_norm_1: tensor(370.9273, device='cuda:0') (tensor(212.5628, device='cuda:0')) | > current_lr_0: 0.00019803459730799195 | > current_lr_1: 0.00019803459730799195 | > step_time: 0.4632 (0.4423288869076088) | > loader_time: 0.0106 (0.00793285057192943) --> EVAL PERFORMANCE | > avg_loader_time: 0.008439369604621135 (-0.0002087942311461543) | > avg_step_time: 0.44941488118238854 (-0.0004219975270016163) | > avg_loss_disc: 2.857244175924382 (-0.0009354302580928398) | > avg_loss_disc_real_0: 0.22968186584996506 (-0.002531185746192932) | > avg_loss_disc_real_1: 0.2343789050486726 (+0.0017905099920823564) | > avg_loss_disc_real_2: 0.24239188873432052 (+0.00034695422985184243) | > avg_loss_disc_real_3: 0.2448527153948663 (+0.0028629107794291564) | > avg_loss_disc_real_4: 0.2470359017311687 (+0.0028992035019565487) | > avg_loss_disc_real_5: 0.247747315277516 (+0.002991954835367877) | > avg_loss_0: 2.857244175924382 (-0.0009354302580928398) | > avg_grad_norm_0: tensor(8.4869, device='cuda:0') (tensor(-0.1710, device='cuda:0')) | > avg_loss_gen: 1.7191599305246916 (+0.011164943936844862) | > avg_loss_kl: 2.4848850814389505 (+0.06353675143819348) | > avg_loss_feat: 1.6686405564697695 (+0.03220572270138167) | > avg_loss_mel: 18.31249047668886 (+0.0946193614476094) | > avg_loss_duration: 1.6793756132394495 (-0.0014352395500936144) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.864551705373838 (+0.2000913217034075) | > avg_grad_norm_1: tensor(218.1111, device='cuda:0') (+tensor(44.1889, device='cuda:0'))  > EPOCH: 80/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:00:31)   --> TIME: 2025-02-25 09:00:50 -- STEP: 39/72 -- GLOBAL_STEP: 1005800 | > loss_disc: 2.838954210281372 (2.8442571896773114) | > loss_disc_real_0: 0.21383817493915558 (0.22768254425281134) | > loss_disc_real_1: 0.1863125115633011 (0.23590000432271224) | > loss_disc_real_2: 0.2255401313304901 (0.2411131304808152) | > loss_disc_real_3: 0.22537493705749512 (0.24411095946263045) | > loss_disc_real_4: 0.22919875383377075 (0.24481368026672265) | > loss_disc_real_5: 0.22883909940719604 (0.24323275647102258) | > loss_0: 2.838954210281372 (2.8442571896773114) | > grad_norm_0: tensor(5.0043, device='cuda:0') (tensor(8.5572, device='cuda:0')) | > loss_gen: 1.839397668838501 (1.7344961502613165) | > loss_kl: 2.440429449081421 (2.4400625779078555) | > loss_feat: 1.9399868249893188 (1.7316534519195557) | > loss_mel: 18.383113861083984 (18.532332102457683) | > loss_duration: 1.66410493850708 (1.6636276978712816) | > amp_scaler: 256.0 (262.56410256410254) | > loss_1: 26.267032623291016 (26.102171579996746) | > grad_norm_1: tensor(289.8247, device='cuda:0') (tensor(232.0081, device='cuda:0')) | > current_lr_0: 0.00019800984298332845 | > current_lr_1: 0.00019800984298332845 | > step_time: 0.4498 (0.43264116385044193) | > loader_time: 0.0081 (0.007227671452057667) --> EVAL PERFORMANCE | > avg_loader_time: 0.008508981113702479 (+6.961150908134373e-05) | > avg_step_time: 0.44888504458145356 (-0.0005298366009349742) | > avg_loss_disc: 2.853648474518682 (-0.0035957014056999803) | > avg_loss_disc_real_0: 0.22868907535579844 (-0.0009927904941666177) | > avg_loss_disc_real_1: 0.23826176557742373 (+0.003882860528751142) | > avg_loss_disc_real_2: 0.2415032879987233 (-0.0008886007355972114) | > avg_loss_disc_real_3: 0.24453478191100375 (-0.0003179334838625536) | > avg_loss_disc_real_4: 0.24551392797852906 (-0.0015219737526396548) | > avg_loss_disc_real_5: 0.24583382316878144 (-0.0019134921087345658) | > avg_loss_0: 2.853648474518682 (-0.0035957014056999803) | > avg_grad_norm_0: tensor(8.2074, device='cuda:0') (tensor(-0.2795, device='cuda:0')) | > avg_loss_gen: 1.7206603087170023 (+0.0015003781923106363) | > avg_loss_kl: 2.4243003952671107 (-0.06058468617183976) | > avg_loss_feat: 1.6822016877187809 (+0.013561131249011371) | > avg_loss_mel: 18.32088244800836 (+0.008391971319497316) | > avg_loss_duration: 1.6765349448566707 (-0.002840668382778855) | > avg_amp_scaler: 259.6056338028169 (+3.6056338028168966) | > avg_loss_1: 25.824579507532253 (-0.039972197841585455) | > avg_grad_norm_1: tensor(222.6794, device='cuda:0') (+tensor(4.5683, device='cuda:0'))  > EPOCH: 81/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:01:07)   --> TIME: 2025-02-25 09:01:17 -- STEP: 17/72 -- GLOBAL_STEP: 1005850 | > loss_disc: 2.817281723022461 (2.840534238254323) | > loss_disc_real_0: 0.2390151023864746 (0.22502156829132752) | > loss_disc_real_1: 0.22698631882667542 (0.22972943064044504) | > loss_disc_real_2: 0.1927202343940735 (0.23569690479951746) | > loss_disc_real_3: 0.21733137965202332 (0.24076282452134526) | > loss_disc_real_4: 0.22457954287528992 (0.24517312119988835) | > loss_disc_real_5: 0.2415163516998291 (0.24356597837279825) | > loss_0: 2.817281723022461 (2.840534238254323) | > grad_norm_0: tensor(3.6905, device='cuda:0') (tensor(7.3973, device='cuda:0')) | > loss_gen: 1.9419257640838623 (1.732550445724936) | > loss_kl: 2.5454444885253906 (2.410373533473295) | > loss_feat: 1.9768235683441162 (1.746274099630468) | > loss_mel: 18.746431350708008 (18.336212831384994) | > loss_duration: 1.6682466268539429 (1.6495556130128748) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.87887191772461 (25.874966677497415) | > grad_norm_1: tensor(431.0878, device='cuda:0') (tensor(299.2061, device='cuda:0')) | > current_lr_0: 0.00019798509175295552 | > current_lr_1: 0.00019798509175295552 | > step_time: 0.4281 (0.43836090143989115) | > loader_time: 0.0061 (0.006636465297025793)  --> TIME: 2025-02-25 09:01:41 -- STEP: 67/72 -- GLOBAL_STEP: 1005900 | > loss_disc: 2.8841617107391357 (2.847749742109384) | > loss_disc_real_0: 0.21129730343818665 (0.22772507747607446) | > loss_disc_real_1: 0.2889651656150818 (0.23306900863327198) | > loss_disc_real_2: 0.2256956696510315 (0.24679844859820693) | > loss_disc_real_3: 0.21158535778522491 (0.24371891368680926) | > loss_disc_real_4: 0.25270524621009827 (0.24654603849596052) | > loss_disc_real_5: 0.2544425129890442 (0.24607362542579422) | > loss_0: 2.8841617107391357 (2.847749742109384) | > grad_norm_0: tensor(5.2303, device='cuda:0') (tensor(8.4994, device='cuda:0')) | > loss_gen: 1.7967164516448975 (1.7436806248195136) | > loss_kl: 2.382844924926758 (2.4308629391798338) | > loss_feat: 1.7660163640975952 (1.729317279004339) | > loss_mel: 18.498598098754883 (18.22575295861087) | > loss_duration: 1.7327824831008911 (1.6699604027306856) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.176958084106445 (25.79957421146222) | > grad_norm_1: tensor(312.4117, device='cuda:0') (tensor(232.7052, device='cuda:0')) | > current_lr_0: 0.00019798509175295552 | > current_lr_1: 0.00019798509175295552 | > step_time: 0.4969 (0.45248287115524066) | > loader_time: 0.0121 (0.008291201805000877) --> EVAL PERFORMANCE | > avg_loader_time: 0.008468359289034992 (-4.062182466748644e-05) | > avg_step_time: 0.45613767395556815 (+0.007252629374114583) | > avg_loss_disc: 2.8494625057972653 (-0.004185968721416522) | > avg_loss_disc_real_0: 0.22876387016034463 (+7.479480454619658e-05) | > avg_loss_disc_real_1: 0.23244980061557932 (-0.0058119649618444125) | > avg_loss_disc_real_2: 0.2461767576529946 (+0.0046734696542712895) | > avg_loss_disc_real_3: 0.24486444302847687 (+0.00032966111747312343) | > avg_loss_disc_real_4: 0.24723654250863572 (+0.0017226145301066642) | > avg_loss_disc_real_5: 0.24643492656694332 (+0.0006011033981618841) | > avg_loss_0: 2.8494625057972653 (-0.004185968721416522) | > avg_grad_norm_0: tensor(8.6424, device='cuda:0') (+tensor(0.4349, device='cuda:0')) | > avg_loss_gen: 1.7455559498827222 (+0.02489564116571996) | > avg_loss_kl: 2.430800538667491 (+0.006500143400380409) | > avg_loss_feat: 1.7313730313744344 (+0.049171343655653565) | > avg_loss_mel: 18.20157634036641 (-0.11930610764194682) | > avg_loss_duration: 1.6716373386517378 (-0.004897606204932847) | > avg_amp_scaler: 256.0 (-3.6056338028168966) | > avg_loss_1: 25.780943158646703 (-0.043636348885549125) | > avg_grad_norm_1: tensor(229.2429, device='cuda:0') (+tensor(6.5635, device='cuda:0'))  > EPOCH: 82/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:01:44)   --> TIME: 2025-02-25 09:02:07 -- STEP: 45/72 -- GLOBAL_STEP: 1005950 | > loss_disc: 2.8944900035858154 (2.833508666356405) | > loss_disc_real_0: 0.2580908536911011 (0.23095714218086666) | > loss_disc_real_1: 0.24486194550991058 (0.23254381749365066) | > loss_disc_real_2: 0.1967284083366394 (0.23888639509677886) | > loss_disc_real_3: 0.2243981808423996 (0.23963924811946022) | > loss_disc_real_4: 0.2420007586479187 (0.24301706453164418) | > loss_disc_real_5: 0.22485961019992828 (0.24520366423659853) | > loss_0: 2.8944900035858154 (2.833508666356405) | > grad_norm_0: tensor(5.8805, device='cuda:0') (tensor(9.8931, device='cuda:0')) | > loss_gen: 1.4626922607421875 (1.7282514810562133) | > loss_kl: 2.5747323036193848 (2.509362379709878) | > loss_feat: 1.62249755859375 (1.7904404083887735) | > loss_mel: 18.34469985961914 (18.39048733181423) | > loss_duration: 1.6824324131011963 (1.6605517599317763) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.687053680419922 (26.079093509250217) | > grad_norm_1: tensor(316.5132, device='cuda:0') (tensor(208.1762, device='cuda:0')) | > current_lr_0: 0.0001979603436164864 | > current_lr_1: 0.0001979603436164864 | > step_time: 0.4629 (0.43998600112067326) | > loader_time: 0.009 (0.007560364405314128) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591709002642564 (+0.00012334971360757237) | > avg_step_time: 0.45431200551315093 (-0.0018256684424172165) | > avg_loss_disc: 2.8462462492392104 (-0.0032162565580549085) | > avg_loss_disc_real_0: 0.23146004177315135 (+0.0026961716128067192) | > avg_loss_disc_real_1: 0.23306398219625715 (+0.0006141815806778306) | > avg_loss_disc_real_2: 0.23973775392686816 (-0.0064390037261264355) | > avg_loss_disc_real_3: 0.2411585811158301 (-0.0037058619126467685) | > avg_loss_disc_real_4: 0.2431486878596561 (-0.004087854648979622) | > avg_loss_disc_real_5: 0.2452859234222224 (-0.0011490031447209115) | > avg_loss_0: 2.8462462492392104 (-0.0032162565580549085) | > avg_grad_norm_0: tensor(9.9579, device='cuda:0') (+tensor(1.3155, device='cuda:0')) | > avg_loss_gen: 1.7223130917884935 (-0.02324285809422877) | > avg_loss_kl: 2.524731934910088 (+0.09393139624259694) | > avg_loss_feat: 1.740874985573997 (+0.009501954199562457) | > avg_loss_mel: 18.275856609075838 (+0.07428026870942617) | > avg_loss_duration: 1.6743024419730819 (+0.0026651033213440645) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.93807921611088 (+0.15713605746417514) | > avg_grad_norm_1: tensor(238.9791, device='cuda:0') (+tensor(9.7362, device='cuda:0'))  > EPOCH: 83/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:02:21)   --> TIME: 2025-02-25 09:02:33 -- STEP: 23/72 -- GLOBAL_STEP: 1006000 | > loss_disc: 2.738790988922119 (2.844216263812521) | > loss_disc_real_0: 0.24594828486442566 (0.22717083666635596) | > loss_disc_real_1: 0.209656223654747 (0.2344916767400244) | > loss_disc_real_2: 0.26063770055770874 (0.24892906650252963) | > loss_disc_real_3: 0.24813802540302277 (0.2442489013723705) | > loss_disc_real_4: 0.28192153573036194 (0.24740267512590988) | > loss_disc_real_5: 0.2506193220615387 (0.24504438172216) | > loss_0: 2.738790988922119 (2.844216263812521) | > grad_norm_0: tensor(6.8288, device='cuda:0') (tensor(7.6796, device='cuda:0')) | > loss_gen: 1.8838251829147339 (1.7314931102420972) | > loss_kl: 2.4465363025665283 (2.4344017194664995) | > loss_feat: 1.8921529054641724 (1.655924854071244) | > loss_mel: 17.764602661132812 (18.346592778744906) | > loss_duration: 1.6598494052886963 (1.6561098720716394) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.6469669342041 (25.824522516001824) | > grad_norm_1: tensor(64.3773, device='cuda:0') (tensor(261.3699, device='cuda:0')) | > current_lr_0: 0.00019793559857353432 | > current_lr_1: 0.00019793559857353432 | > step_time: 0.4333 (0.43310849562935205) | > loader_time: 0.0064 (0.006783848223478898) --> EVAL PERFORMANCE | > avg_loader_time: 0.008589428915104395 (-2.280087538169759e-06) | > avg_step_time: 0.4509244804650965 (-0.0033875250480544516) | > avg_loss_disc: 2.8558765465105083 (+0.009630297271297827) | > avg_loss_disc_real_0: 0.23311128813615986 (+0.0016512463630085072) | > avg_loss_disc_real_1: 0.2374413850861536 (+0.004377402889896437) | > avg_loss_disc_real_2: 0.24322544688909825 (+0.003487692962230088) | > avg_loss_disc_real_3: 0.2426596850576535 (+0.0015011039418234007) | > avg_loss_disc_real_4: 0.2472593364581256 (+0.004110648598469491) | > avg_loss_disc_real_5: 0.24930379529234395 (+0.004017871870121537) | > avg_loss_0: 2.8558765465105083 (+0.009630297271297827) | > avg_grad_norm_0: tensor(9.0351, device='cuda:0') (tensor(-0.9228, device='cuda:0')) | > avg_loss_gen: 1.7293063821926924 (+0.0069932904041989374) | > avg_loss_kl: 2.396164803437783 (-0.12856713147230492) | > avg_loss_feat: 1.6757589914429356 (-0.06511599413106128) | > avg_loss_mel: 18.290321699330505 (+0.014465090254667246) | > avg_loss_duration: 1.6762725769633977 (+0.0019701349903158416) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76782449534242 (-0.17025472076845816) | > avg_grad_norm_1: tensor(222.6369, device='cuda:0') (tensor(-16.3421, device='cuda:0'))  > EPOCH: 84/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:02:58)   --> TIME: 2025-02-25 09:03:00 -- STEP: 1/72 -- GLOBAL_STEP: 1006050 | > loss_disc: 2.9282140731811523 (2.9282140731811523) | > loss_disc_real_0: 0.2183455228805542 (0.2183455228805542) | > loss_disc_real_1: 0.1405745893716812 (0.1405745893716812) | > loss_disc_real_2: 0.23146413266658783 (0.23146413266658783) | > loss_disc_real_3: 0.23153740167617798 (0.23153740167617798) | > loss_disc_real_4: 0.22167439758777618 (0.22167439758777618) | > loss_disc_real_5: 0.24115917086601257 (0.24115917086601257) | > loss_0: 2.9282140731811523 (2.9282140731811523) | > grad_norm_0: tensor(27.0323, device='cuda:0') (tensor(27.0323, device='cuda:0')) | > loss_gen: 1.614777684211731 (1.614777684211731) | > loss_kl: 2.3541746139526367 (2.3541746139526367) | > loss_feat: 1.8990113735198975 (1.8990113735198975) | > loss_mel: 18.279239654541016 (18.279239654541016) | > loss_duration: 1.673421025276184 (1.673421025276184) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.82062530517578 (25.82062530517578) | > grad_norm_1: tensor(241.6967, device='cuda:0') (tensor(241.6967, device='cuda:0')) | > current_lr_0: 0.00019791085662371262 | > current_lr_1: 0.00019791085662371262 | > step_time: 0.4453 (0.44533205032348633) | > loader_time: 0.006 (0.006005764007568359)  --> TIME: 2025-02-25 09:03:23 -- STEP: 51/72 -- GLOBAL_STEP: 1006100 | > loss_disc: 2.892099142074585 (2.8552628591948865) | > loss_disc_real_0: 0.2790597081184387 (0.2345593191829382) | > loss_disc_real_1: 0.21762381494045258 (0.2368882155301524) | > loss_disc_real_2: 0.2504127621650696 (0.24261855407088412) | > loss_disc_real_3: 0.2264476716518402 (0.24589942804738588) | > loss_disc_real_4: 0.2458173632621765 (0.24601367787987577) | > loss_disc_real_5: 0.24954885244369507 (0.2477161393446081) | > loss_0: 2.892099142074585 (2.8552628591948865) | > grad_norm_0: tensor(7.6902, device='cuda:0') (tensor(8.9535, device='cuda:0')) | > loss_gen: 1.5649921894073486 (1.7404673754000197) | > loss_kl: 2.3304624557495117 (2.451911725249945) | > loss_feat: 1.6212108135223389 (1.6928525625490676) | > loss_mel: 19.034324645996094 (18.51754813100778) | > loss_duration: 1.686582326889038 (1.6717022937886856) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.237573623657227 (26.074481964111328) | > grad_norm_1: tensor(113.5037, device='cuda:0') (tensor(236.1090, device='cuda:0')) | > current_lr_0: 0.00019791085662371262 | > current_lr_1: 0.00019791085662371262 | > step_time: 0.4462 (0.44507090718138453) | > loader_time: 0.009 (0.007654844545850567) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852726546811386 (-6.216344699053536e-05) | > avg_step_time: 0.4578740294550506 (+0.006949548989954146) | > avg_loss_disc: 2.8606332993843187 (+0.004756752873810388) | > avg_loss_disc_real_0: 0.23005860760598115 (-0.0030526805301787086) | > avg_loss_disc_real_1: 0.23606939529868917 (-0.0013719897874644171) | > avg_loss_disc_real_2: 0.24272921702391664 (-0.0004962298651816077) | > avg_loss_disc_real_3: 0.24456377411392374 (+0.0019040890562702362) | > avg_loss_disc_real_4: 0.2457938097732168 (-0.0014655266849087911) | > avg_loss_disc_real_5: 0.24752802999926285 (-0.0017757652930810996) | > avg_loss_0: 2.8606332993843187 (+0.004756752873810388) | > avg_grad_norm_0: tensor(8.7510, device='cuda:0') (tensor(-0.2841, device='cuda:0')) | > avg_loss_gen: 1.723023339056633 (-0.00628304313605943) | > avg_loss_kl: 2.4542252177923496 (+0.058060414354566436) | > avg_loss_feat: 1.663132029519954 (-0.012626961922981517) | > avg_loss_mel: 18.417953047953866 (+0.12763134862336045) | > avg_loss_duration: 1.6779593447564354 (+0.001686767793037669) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.93629291695608 (+0.16846842161366027) | > avg_grad_norm_1: tensor(235.8760, device='cuda:0') (+tensor(13.2391, device='cuda:0'))  > EPOCH: 85/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:03:35)   --> TIME: 2025-02-25 09:03:50 -- STEP: 29/72 -- GLOBAL_STEP: 1006150 | > loss_disc: 2.8701343536376953 (2.834670560113315) | > loss_disc_real_0: 0.1733030378818512 (0.23010013946171465) | > loss_disc_real_1: 0.2905268967151642 (0.23875944162237234) | > loss_disc_real_2: 0.2382815182209015 (0.24070084403301106) | > loss_disc_real_3: 0.2461562156677246 (0.2396396552694255) | > loss_disc_real_4: 0.21975797414779663 (0.2468794707594247) | > loss_disc_real_5: 0.2423006296157837 (0.24146245831045612) | > loss_0: 2.8701343536376953 (2.834670560113315) | > grad_norm_0: tensor(11.2189, device='cuda:0') (tensor(8.4126, device='cuda:0')) | > loss_gen: 1.6390775442123413 (1.7407010752579262) | > loss_kl: 2.2617533206939697 (2.4628754237602495) | > loss_feat: 1.7658865451812744 (1.7774986316417825) | > loss_mel: 18.46258544921875 (18.463224345240096) | > loss_duration: 1.6375397443771362 (1.6496071404424206) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.766841888427734 (26.09390666567046) | > grad_norm_1: tensor(277.0778, device='cuda:0') (tensor(196.2700, device='cuda:0')) | > current_lr_0: 0.00019788611776663464 | > current_lr_1: 0.00019788611776663464 | > step_time: 0.4374 (0.4325218282896897) | > loader_time: 0.0079 (0.006900409172321188) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570288268613144 (+4.302280049928511e-05) | > avg_step_time: 0.45108516787139463 (-0.006788861583655992) | > avg_loss_disc: 2.8457175375710073 (-0.014915761813311335) | > avg_loss_disc_real_0: 0.2306113024832497 (+0.0005526948772685558) | > avg_loss_disc_real_1: 0.23573304020183186 (-0.00033635509685731035) | > avg_loss_disc_real_2: 0.23919713686049823 (-0.0035320801634184107) | > avg_loss_disc_real_3: 0.24238315560448337 (-0.002180618509440374) | > avg_loss_disc_real_4: 0.24614831994117145 (+0.0003545101679546525) | > avg_loss_disc_real_5: 0.24596966257397557 (-0.0015583674252872826) | > avg_loss_0: 2.8457175375710073 (-0.014915761813311335) | > avg_grad_norm_0: tensor(9.1981, device='cuda:0') (+tensor(0.4471, device='cuda:0')) | > avg_loss_gen: 1.7265300566041972 (+0.0035067175475642642) | > avg_loss_kl: 2.4131988639562905 (-0.041026353836059126) | > avg_loss_feat: 1.7306787430400579 (+0.06754671352010377) | > avg_loss_mel: 18.296700034343026 (-0.12125301361083984) | > avg_loss_duration: 1.6727035582905085 (-0.005255786465926837) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.83981121761698 (-0.09648169933910111) | > avg_grad_norm_1: tensor(193.0138, device='cuda:0') (tensor(-42.8622, device='cuda:0'))  > EPOCH: 86/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:04:12)   --> TIME: 2025-02-25 09:04:17 -- STEP: 7/72 -- GLOBAL_STEP: 1006200 | > loss_disc: 2.7610316276550293 (2.7999659946986606) | > loss_disc_real_0: 0.19535070657730103 (0.23302748799324036) | > loss_disc_real_1: 0.2591384947299957 (0.25550136821610586) | > loss_disc_real_2: 0.26773422956466675 (0.24306118488311768) | > loss_disc_real_3: 0.2249167561531067 (0.2381300734622138) | > loss_disc_real_4: 0.2301509827375412 (0.24564845860004425) | > loss_disc_real_5: 0.25502562522888184 (0.23269700578280858) | > loss_0: 2.7610316276550293 (2.7999659946986606) | > grad_norm_0: tensor(7.9260, device='cuda:0') (tensor(7.8157, device='cuda:0')) | > loss_gen: 1.6956878900527954 (1.7945424999509538) | > loss_kl: 2.387800455093384 (2.3400176082338606) | > loss_feat: 2.0046169757843018 (2.0136820588793074) | > loss_mel: 18.0709285736084 (18.946001052856445) | > loss_duration: 1.6587438583374023 (1.670241458075387) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.817779541015625 (26.76448495047433) | > grad_norm_1: tensor(336.9724, device='cuda:0') (tensor(171.0379, device='cuda:0')) | > current_lr_0: 0.0001978613820019138 | > current_lr_1: 0.0001978613820019138 | > step_time: 0.4282 (0.43251729011535645) | > loader_time: 0.0054 (0.006013563701084682)  --> TIME: 2025-02-25 09:04:39 -- STEP: 57/72 -- GLOBAL_STEP: 1006250 | > loss_disc: 2.838599920272827 (2.841506891083299) | > loss_disc_real_0: 0.18903875350952148 (0.23246988654136658) | > loss_disc_real_1: 0.23110732436180115 (0.23908410652687676) | > loss_disc_real_2: 0.2400628924369812 (0.2394902695689285) | > loss_disc_real_3: 0.21258004009723663 (0.24105793316113322) | > loss_disc_real_4: 0.24586927890777588 (0.24517643216409182) | > loss_disc_real_5: 0.24971386790275574 (0.2456591754106053) | > loss_0: 2.838599920272827 (2.841506891083299) | > grad_norm_0: tensor(8.8800, device='cuda:0') (tensor(9.5912, device='cuda:0')) | > loss_gen: 1.6966314315795898 (1.7438703085246838) | > loss_kl: 2.3983519077301025 (2.4276626465613376) | > loss_feat: 1.8148961067199707 (1.7819558235636928) | > loss_mel: 17.692373275756836 (18.209991254304583) | > loss_duration: 1.697073221206665 (1.6659641893286454) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.299325942993164 (25.829444282933284) | > grad_norm_1: tensor(253.9631, device='cuda:0') (tensor(221.2857, device='cuda:0')) | > current_lr_0: 0.0001978613820019138 | > current_lr_1: 0.0001978613820019138 | > step_time: 0.4492 (0.4403759094706753) | > loader_time: 0.0097 (0.00774014623541581) --> EVAL PERFORMANCE | > avg_loader_time: 0.008413106622830244 (-0.0001571816457829004) | > avg_step_time: 0.45151684989391916 (+0.0004316820225245266) | > avg_loss_disc: 2.853258683647908 (+0.007541146076900596) | > avg_loss_disc_real_0: 0.2338726048318433 (+0.003261302348593592) | > avg_loss_disc_real_1: 0.24168113230819432 (+0.0059480921063624625) | > avg_loss_disc_real_2: 0.23946057364974224 (+0.00026343678924400926) | > avg_loss_disc_real_3: 0.24289938261811161 (+0.0005162270136282454) | > avg_loss_disc_real_4: 0.24637979793716486 (+0.00023147799599340657) | > avg_loss_disc_real_5: 0.24729578238977512 (+0.0013261198157995535) | > avg_loss_0: 2.853258683647908 (+0.007541146076900596) | > avg_grad_norm_0: tensor(9.4912, device='cuda:0') (+tensor(0.2931, device='cuda:0')) | > avg_loss_gen: 1.7409763655192416 (+0.014446308915044392) | > avg_loss_kl: 2.4328268128381643 (+0.019627948881873802) | > avg_loss_feat: 1.7557537303843969 (+0.025074987344338995) | > avg_loss_mel: 18.244040636949137 (-0.052659397393888696) | > avg_loss_duration: 1.6717125261333625 (-0.0009910321571460212) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.84531015745351 (+0.005498939836531491) | > avg_grad_norm_1: tensor(235.1172, device='cuda:0') (+tensor(42.1034, device='cuda:0'))  > EPOCH: 87/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:04:49)   --> TIME: 2025-02-25 09:05:06 -- STEP: 35/72 -- GLOBAL_STEP: 1006300 | > loss_disc: 2.8999404907226562 (2.8627241611480714) | > loss_disc_real_0: 0.25515055656433105 (0.24550126876149858) | > loss_disc_real_1: 0.24208100140094757 (0.23429017748151507) | > loss_disc_real_2: 0.2391432821750641 (0.2414703735283443) | > loss_disc_real_3: 0.26938581466674805 (0.2424354293516704) | > loss_disc_real_4: 0.25618529319763184 (0.24593753133501325) | > loss_disc_real_5: 0.2559240460395813 (0.24676197682108197) | > loss_0: 2.8999404907226562 (2.8627241611480714) | > grad_norm_0: tensor(6.3681, device='cuda:0') (tensor(9.4983, device='cuda:0')) | > loss_gen: 1.6077457666397095 (1.7279506547110421) | > loss_kl: 2.7808611392974854 (2.464648757662092) | > loss_feat: 1.6653039455413818 (1.7068392719541277) | > loss_mel: 19.819435119628906 (18.337733404976984) | > loss_duration: 1.714146614074707 (1.652941073690142) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.587493896484375 (25.890113285609655) | > grad_norm_1: tensor(331.2830, device='cuda:0') (tensor(202.1999, device='cuda:0')) | > current_lr_0: 0.00019783664932916355 | > current_lr_1: 0.00019783664932916355 | > step_time: 0.4453 (0.4395534924098424) | > loader_time: 0.008 (0.007381459644862583) --> EVAL PERFORMANCE | > avg_loader_time: 0.00881180964725118 (+0.000398703024420936) | > avg_step_time: 0.4554023474035129 (+0.003885497509593716) | > avg_loss_disc: 2.863372067330589 (+0.01011338368268122) | > avg_loss_disc_real_0: 0.23584418114222272 (+0.0019715763103794193) | > avg_loss_disc_real_1: 0.23384545785440525 (-0.007835674453789077) | > avg_loss_disc_real_2: 0.2421877086162567 (+0.0027271349665144717) | > avg_loss_disc_real_3: 0.24291995860321422 (+2.05759851026055e-05) | > avg_loss_disc_real_4: 0.24707031816663877 (+0.0006905202294739121) | > avg_loss_disc_real_5: 0.24735679424984355 (+6.1011860068432844e-05) | > avg_loss_0: 2.863372067330589 (+0.01011338368268122) | > avg_grad_norm_0: tensor(9.6442, device='cuda:0') (+tensor(0.1530, device='cuda:0')) | > avg_loss_gen: 1.7159030017718462 (-0.02507336374739544) | > avg_loss_kl: 2.4556731774773395 (+0.02284636463917522) | > avg_loss_feat: 1.6968503552423397 (-0.058903375142057124) | > avg_loss_mel: 18.12675617110561 (-0.11728446584352525) | > avg_loss_duration: 1.6736498852850685 (+0.0019373591517060085) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.668832644610337 (-0.17647751284317437) | > avg_grad_norm_1: tensor(205.3715, device='cuda:0') (tensor(-29.7457, device='cuda:0'))  > EPOCH: 88/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:05:25)   --> TIME: 2025-02-25 09:05:33 -- STEP: 13/72 -- GLOBAL_STEP: 1006350 | > loss_disc: 2.8792171478271484 (2.8135959001687856) | > loss_disc_real_0: 0.22929495573043823 (0.22259219678548667) | > loss_disc_real_1: 0.16809841990470886 (0.2348888092316114) | > loss_disc_real_2: 0.2420896291732788 (0.23673973404444182) | > loss_disc_real_3: 0.269193172454834 (0.24563161341043618) | > loss_disc_real_4: 0.25656622648239136 (0.2472249945768943) | > loss_disc_real_5: 0.25637462735176086 (0.24227781364550957) | > loss_0: 2.8792171478271484 (2.8135959001687856) | > grad_norm_0: tensor(10.2062, device='cuda:0') (tensor(9.3042, device='cuda:0')) | > loss_gen: 1.7955272197723389 (1.7903658059927134) | > loss_kl: 2.348168134689331 (2.397504403040959) | > loss_feat: 1.7215862274169922 (1.8663749236326952) | > loss_mel: 17.716148376464844 (18.01930985083947) | > loss_duration: 1.572614073753357 (1.6453483196405263) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.15404510498047 (25.718903908362755) | > grad_norm_1: tensor(248.8394, device='cuda:0') (tensor(233.8456, device='cuda:0')) | > current_lr_0: 0.0001978119197479974 | > current_lr_1: 0.0001978119197479974 | > step_time: 0.4409 (0.42978770916278547) | > loader_time: 0.0058 (0.00686038457430326)  --> TIME: 2025-02-25 09:05:56 -- STEP: 63/72 -- GLOBAL_STEP: 1006400 | > loss_disc: 2.979435443878174 (2.8406670093536377) | > loss_disc_real_0: 0.2522873878479004 (0.23186473997812423) | > loss_disc_real_1: 0.24919335544109344 (0.23501692570391156) | > loss_disc_real_2: 0.2851322889328003 (0.2419796803640941) | > loss_disc_real_3: 0.27651160955429077 (0.24527411967042892) | > loss_disc_real_4: 0.2751944363117218 (0.24504560303120387) | > loss_disc_real_5: 0.2781367003917694 (0.24683719802470433) | > loss_0: 2.979435443878174 (2.8406670093536377) | > grad_norm_0: tensor(5.5146, device='cuda:0') (tensor(9.0713, device='cuda:0')) | > loss_gen: 1.7725441455841064 (1.7551023979035636) | > loss_kl: 2.46433687210083 (2.4066026097252258) | > loss_feat: 1.4583135843276978 (1.7885641181279743) | > loss_mel: 17.801424026489258 (18.151214902363126) | > loss_duration: 1.643946647644043 (1.6602561795522297) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.140567779541016 (25.761740578545464) | > grad_norm_1: tensor(388.3426, device='cuda:0') (tensor(234.3666, device='cuda:0')) | > current_lr_0: 0.0001978119197479974 | > current_lr_1: 0.0001978119197479974 | > step_time: 0.4939 (0.4464765465448773) | > loader_time: 0.0111 (0.008162271408807664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008546302016352264 (-0.00026550763089891634) | > avg_step_time: 0.45245935883320554 (-0.0029429885703073366) | > avg_loss_disc: 2.842169227734418 (-0.021202839596170975) | > avg_loss_disc_real_0: 0.23207849033281835 (-0.003765690809404365) | > avg_loss_disc_real_1: 0.2358920345423927 (+0.002046576687987439) | > avg_loss_disc_real_2: 0.24057673841295107 (-0.0016109702033056394) | > avg_loss_disc_real_3: 0.24463343956101108 (+0.001713480957796859) | > avg_loss_disc_real_4: 0.24587441980838776 (-0.001195898358251013) | > avg_loss_disc_real_5: 0.24740410606626054 (+4.73118164169839e-05) | > avg_loss_0: 2.842169227734418 (-0.021202839596170975) | > avg_grad_norm_0: tensor(9.2561, device='cuda:0') (tensor(-0.3880, device='cuda:0')) | > avg_loss_gen: 1.7452172343160066 (+0.029314232544160435) | > avg_loss_kl: 2.423621298561634 (-0.0320518789157056) | > avg_loss_feat: 1.7629170619266135 (+0.06606670668427372) | > avg_loss_mel: 18.14453032318975 (+0.017774152084136574) | > avg_loss_duration: 1.6651700567191756 (-0.00847982856589291) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.74145626014387 (+0.07262361553353358) | > avg_grad_norm_1: tensor(234.8506, device='cuda:0') (+tensor(29.4790, device='cuda:0'))  > EPOCH: 89/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:06:02)   --> TIME: 2025-02-25 09:06:23 -- STEP: 41/72 -- GLOBAL_STEP: 1006450 | > loss_disc: 2.9040732383728027 (2.84514991830035) | > loss_disc_real_0: 0.25273779034614563 (0.23208104364755677) | > loss_disc_real_1: 0.22609162330627441 (0.23352660347775714) | > loss_disc_real_2: 0.24658502638339996 (0.24462011492833857) | > loss_disc_real_3: 0.2505195438861847 (0.24416981564789283) | > loss_disc_real_4: 0.24195295572280884 (0.24828591070524075) | > loss_disc_real_5: 0.24500368535518646 (0.2476124665359171) | > loss_0: 2.9040732383728027 (2.84514991830035) | > grad_norm_0: tensor(8.3701, device='cuda:0') (tensor(9.7839, device='cuda:0')) | > loss_gen: 1.784749984741211 (1.7591700844648408) | > loss_kl: 2.3241617679595947 (2.435089716097204) | > loss_feat: 1.7413135766983032 (1.7711134247663545) | > loss_mel: 18.207857131958008 (18.154651037076626) | > loss_duration: 1.6634197235107422 (1.6564990078530661) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.72150230407715 (25.776523264443004) | > grad_norm_1: tensor(224.5845, device='cuda:0') (tensor(196.4666, device='cuda:0')) | > current_lr_0: 0.0001977871932580289 | > current_lr_1: 0.0001977871932580289 | > step_time: 0.4396 (0.4368053238566329) | > loader_time: 0.0089 (0.007446056459008194) --> EVAL PERFORMANCE | > avg_loader_time: 0.008635870167906855 (+8.956815155459173e-05) | > avg_step_time: 0.4532308511331048 (+0.0007714922998992568) | > avg_loss_disc: 2.8517555955430165 (+0.00958636780859834) | > avg_loss_disc_real_0: 0.23604993627104962 (+0.003971445938231261) | > avg_loss_disc_real_1: 0.23376037935975572 (-0.002131655182636971) | > avg_loss_disc_real_2: 0.24440345944653094 (+0.0038267210335798663) | > avg_loss_disc_real_3: 0.24316036701202393 (-0.0014730725489871532) | > avg_loss_disc_real_4: 0.24767197752502604 (+0.0017975577166382817) | > avg_loss_disc_real_5: 0.24663576084963032 (-0.0007683452166302174) | > avg_loss_0: 2.8517555955430165 (+0.00958636780859834) | > avg_grad_norm_0: tensor(9.8615, device='cuda:0') (+tensor(0.6054, device='cuda:0')) | > avg_loss_gen: 1.7435075044631958 (-0.0017097298528108151) | > avg_loss_kl: 2.4347989827814236 (+0.011177684219789707) | > avg_loss_feat: 1.730300439915187 (-0.032616622011426566) | > avg_loss_mel: 18.103502031782984 (-0.04102829140676434) | > avg_loss_duration: 1.6719742341780326 (+0.006804177458856975) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68408321326887 (-0.057373046875) | > avg_grad_norm_1: tensor(223.4882, device='cuda:0') (tensor(-11.3624, device='cuda:0'))  > EPOCH: 90/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:06:39)   --> TIME: 2025-02-25 09:06:50 -- STEP: 19/72 -- GLOBAL_STEP: 1006500 | > loss_disc: 2.8341684341430664 (2.8467202814001786) | > loss_disc_real_0: 0.22207742929458618 (0.22897656967765406) | > loss_disc_real_1: 0.21857404708862305 (0.2429870743500559) | > loss_disc_real_2: 0.20325839519500732 (0.24033376025526146) | > loss_disc_real_3: 0.24378570914268494 (0.24023226452501198) | > loss_disc_real_4: 0.22039145231246948 (0.2419865186277189) | > loss_disc_real_5: 0.21705526113510132 (0.2429573151626085) | > loss_0: 2.8341684341430664 (2.8467202814001786) | > grad_norm_0: tensor(5.2269, device='cuda:0') (tensor(8.9425, device='cuda:0')) | > loss_gen: 1.6974616050720215 (1.735395544453671) | > loss_kl: 2.4156100749969482 (2.4058561827007092) | > loss_feat: 1.909623384475708 (1.8726198861473484) | > loss_mel: 17.914682388305664 (18.68711903220729) | > loss_duration: 1.629659652709961 (1.6430472700219405) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.56703758239746 (26.344038009643555) | > grad_norm_1: tensor(116.6928, device='cuda:0') (tensor(221.6084, device='cuda:0')) | > current_lr_0: 0.00019776246985887165 | > current_lr_1: 0.00019776246985887165 | > step_time: 0.4258 (0.4350703515504536) | > loader_time: 0.0065 (0.006777035562615646)  --> TIME: 2025-02-25 09:07:13 -- STEP: 69/72 -- GLOBAL_STEP: 1006550 | > loss_disc: 2.918752908706665 (2.8612664402395063) | > loss_disc_real_0: 0.2478901743888855 (0.23279014704883963) | > loss_disc_real_1: 0.3469572067260742 (0.2405726378378661) | > loss_disc_real_2: 0.26681607961654663 (0.24206166530864826) | > loss_disc_real_3: 0.26818329095840454 (0.24294510246187018) | > loss_disc_real_4: 0.25641316175460815 (0.24615242299826248) | > loss_disc_real_5: 0.28028759360313416 (0.24704133190106656) | > loss_0: 2.918752908706665 (2.8612664402395063) | > grad_norm_0: tensor(8.8906, device='cuda:0') (tensor(8.9738, device='cuda:0')) | > loss_gen: 1.773284912109375 (1.7300319049669348) | > loss_kl: 2.574341058731079 (2.4412692733432935) | > loss_feat: 1.6759419441223145 (1.755070434100386) | > loss_mel: 18.21282196044922 (18.48491508373316) | > loss_duration: 1.7445648908615112 (1.6709448427393816) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.980953216552734 (26.08223149396371) | > grad_norm_1: tensor(330.1239, device='cuda:0') (tensor(248.3322, device='cuda:0')) | > current_lr_0: 0.00019776246985887165 | > current_lr_1: 0.00019776246985887165 | > step_time: 0.5153 (0.4524563913759978) | > loader_time: 0.0126 (0.008469550505928373) --> EVAL PERFORMANCE | > avg_loader_time: 0.008527503886692963 (-0.00010836628121389281) | > avg_step_time: 0.4539017375086395 (+0.0006708863755346961) | > avg_loss_disc: 2.8626308206101547 (+0.010875225067138228) | > avg_loss_disc_real_0: 0.2319525958786548 (-0.004097340392394816) | > avg_loss_disc_real_1: 0.2404449441483323 (+0.006684564788576597) | > avg_loss_disc_real_2: 0.24218082574891373 (-0.0022226336976172156) | > avg_loss_disc_real_3: 0.24360353741007792 (+0.00044317039805399916) | > avg_loss_disc_real_4: 0.24630035816783635 (-0.001371619357189685) | > avg_loss_disc_real_5: 0.24771016059626996 (+0.0010743997466396427) | > avg_loss_0: 2.8626308206101547 (+0.010875225067138228) | > avg_grad_norm_0: tensor(8.9494, device='cuda:0') (tensor(-0.9122, device='cuda:0')) | > avg_loss_gen: 1.7287977698823096 (-0.014709734580886202) | > avg_loss_kl: 2.4455443939692536 (+0.010745411187830012) | > avg_loss_feat: 1.7476395200675643 (+0.017339080152377395) | > avg_loss_mel: 18.445632773385924 (+0.34213074160294) | > avg_loss_duration: 1.670929683765895 (-0.0010445504121376903) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.038544050404724 (+0.3544608371358535) | > avg_grad_norm_1: tensor(245.4543, device='cuda:0') (+tensor(21.9661, device='cuda:0'))  > EPOCH: 91/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:07:16)   --> TIME: 2025-02-25 09:07:39 -- STEP: 47/72 -- GLOBAL_STEP: 1006600 | > loss_disc: 2.830890417098999 (2.8514384158114168) | > loss_disc_real_0: 0.2766723930835724 (0.2301349658915337) | > loss_disc_real_1: 0.23910745978355408 (0.23885884976133387) | > loss_disc_real_2: 0.2891020178794861 (0.24076678492921463) | > loss_disc_real_3: 0.21676598489284515 (0.241262471739282) | > loss_disc_real_4: 0.238656684756279 (0.2443364826922721) | > loss_disc_real_5: 0.23516952991485596 (0.245204359292984) | > loss_0: 2.830890417098999 (2.8514384158114168) | > grad_norm_0: tensor(6.3734, device='cuda:0') (tensor(8.7641, device='cuda:0')) | > loss_gen: 1.6013948917388916 (1.7248294556394537) | > loss_kl: 2.4343888759613037 (2.4210688560567006) | > loss_feat: 1.8378955125808716 (1.7241496963703886) | > loss_mel: 18.268346786499023 (18.192724511978476) | > loss_duration: 1.6874648332595825 (1.6608499694377818) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.829490661621094 (25.723622585864778) | > grad_norm_1: tensor(128.6683, device='cuda:0') (tensor(211.8081, device='cuda:0')) | > current_lr_0: 0.0001977377495501393 | > current_lr_1: 0.0001977377495501393 | > step_time: 0.4503 (0.4409451687589605) | > loader_time: 0.0112 (0.007870638624150703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754387707777426 (+0.00022688382108446321) | > avg_step_time: 0.452153054761215 (-0.0017486827474245126) | > avg_loss_disc: 2.860117432097314 (-0.002513388512840553) | > avg_loss_disc_real_0: 0.23128781446688612 (-0.0006647814117686779) | > avg_loss_disc_real_1: 0.23891668861180965 (-0.0015282555365226658) | > avg_loss_disc_real_2: 0.2411172129738499 (-0.001063612775063838) | > avg_loss_disc_real_3: 0.24379451761783008 (+0.00019098020775215607) | > avg_loss_disc_real_4: 0.2466701097051862 (+0.00036975153734986055) | > avg_loss_disc_real_5: 0.24689148786202283 (-0.0008186727342471278) | > avg_loss_0: 2.860117432097314 (-0.002513388512840553) | > avg_grad_norm_0: tensor(9.3476, device='cuda:0') (+tensor(0.3982, device='cuda:0')) | > avg_loss_gen: 1.7308207968590963 (+0.0020230269767866904) | > avg_loss_kl: 2.4101092848979255 (-0.035435109071328075) | > avg_loss_feat: 1.71273805557842 (-0.03490146448914433) | > avg_loss_mel: 18.14619045526209 (-0.29944231812383393) | > avg_loss_duration: 1.6787345929884574 (+0.007804909222562495) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.67859321916607 (-0.35995083123865257) | > avg_grad_norm_1: tensor(232.5493, device='cuda:0') (tensor(-12.9049, device='cuda:0'))  > EPOCH: 92/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:07:53)   --> TIME: 2025-02-25 09:08:06 -- STEP: 25/72 -- GLOBAL_STEP: 1006650 | > loss_disc: 2.8274426460266113 (2.8599162006378176) | > loss_disc_real_0: 0.1420454978942871 (0.22999772548675537) | > loss_disc_real_1: 0.24092477560043335 (0.23516590595245362) | > loss_disc_real_2: 0.22451792657375336 (0.24321955740451812) | > loss_disc_real_3: 0.23469123244285583 (0.2417353367805481) | > loss_disc_real_4: 0.242137148976326 (0.24519217133522034) | > loss_disc_real_5: 0.2579938769340515 (0.2469542133808136) | > loss_0: 2.8274426460266113 (2.8599162006378176) | > grad_norm_0: tensor(11.0699, device='cuda:0') (tensor(9.3645, device='cuda:0')) | > loss_gen: 1.6936757564544678 (1.7228529119491578) | > loss_kl: 2.4821853637695312 (2.410423984527588) | > loss_feat: 1.8881858587265015 (1.7513498306274413) | > loss_mel: 17.9317569732666 (17.93092491149902) | > loss_duration: 1.62685227394104 (1.6458809661865235) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.622655868530273 (25.461432571411134) | > grad_norm_1: tensor(159.8380, device='cuda:0') (tensor(181.0744, device='cuda:0')) | > current_lr_0: 0.0001977130323314455 | > current_lr_1: 0.0001977130323314455 | > step_time: 0.4355 (0.43032413482666015) | > loader_time: 0.0071 (0.006814556121826172) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577219197447871 (-0.00017716851032955477) | > avg_step_time: 0.44885055112167144 (-0.0033025036395435414) | > avg_loss_disc: 2.8454757878478145 (-0.014641644249499652) | > avg_loss_disc_real_0: 0.2330808847303122 (+0.0017930702634260665) | > avg_loss_disc_real_1: 0.23895351987489513 (+3.683126308548501e-05) | > avg_loss_disc_real_2: 0.24152690591946455 (+0.0004096929456146592) | > avg_loss_disc_real_3: 0.24091832465688948 (-0.0028761929609406045) | > avg_loss_disc_real_4: 0.2458768944505235 (-0.0007932152546627214) | > avg_loss_disc_real_5: 0.2473212369311024 (+0.0004297490690795547) | > avg_loss_0: 2.8454757878478145 (-0.014641644249499652) | > avg_grad_norm_0: tensor(9.2595, device='cuda:0') (tensor(-0.0881, device='cuda:0')) | > avg_loss_gen: 1.7472589469291795 (+0.0164381500700832) | > avg_loss_kl: 2.395110190754206 (-0.014999094143719471) | > avg_loss_feat: 1.7766044223812265 (+0.06386636680280655) | > avg_loss_mel: 17.9309247110931 (-0.2152657441689918) | > avg_loss_duration: 1.6646953686861925 (-0.014039224302264897) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.514593715399084 (-0.1639995037669877) | > avg_grad_norm_1: tensor(225.2302, device='cuda:0') (tensor(-7.3191, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1006697.pth  > EPOCH: 93/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:08:32)   --> TIME: 2025-02-25 09:08:35 -- STEP: 3/72 -- GLOBAL_STEP: 1006700 | > loss_disc: 2.8367953300476074 (2.8063789208730063) | > loss_disc_real_0: 0.19466467201709747 (0.21596665680408478) | > loss_disc_real_1: 0.2220536470413208 (0.22947006424268088) | > loss_disc_real_2: 0.2399943619966507 (0.23755035797754923) | > loss_disc_real_3: 0.237724170088768 (0.2311839759349823) | > loss_disc_real_4: 0.22815214097499847 (0.22337565819422403) | > loss_disc_real_5: 0.22781401872634888 (0.24018486340840658) | > loss_0: 2.8367953300476074 (2.8063789208730063) | > grad_norm_0: tensor(4.0722, device='cuda:0') (tensor(5.0585, device='cuda:0')) | > loss_gen: 1.7885655164718628 (1.7440050840377808) | > loss_kl: 2.4285025596618652 (2.441890239715576) | > loss_feat: 1.7339575290679932 (1.8061317602793376) | > loss_mel: 17.54124641418457 (18.37698809305827) | > loss_duration: 1.6628772020339966 (1.663833498954773) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.155149459838867 (26.03284962972005) | > grad_norm_1: tensor(269.1956, device='cuda:0') (tensor(450.3511, device='cuda:0')) | > current_lr_0: 0.00019768831820240408 | > current_lr_1: 0.00019768831820240408 | > step_time: 0.4306 (0.4376231829325358) | > loader_time: 0.0051 (0.006353934605916341)  --> TIME: 2025-02-25 09:08:58 -- STEP: 53/72 -- GLOBAL_STEP: 1006750 | > loss_disc: 2.846574306488037 (2.8428224257703096) | > loss_disc_real_0: 0.2150961458683014 (0.23093452178082377) | > loss_disc_real_1: 0.2926884591579437 (0.2367954338496586) | > loss_disc_real_2: 0.2568556070327759 (0.2388865902738751) | > loss_disc_real_3: 0.24152976274490356 (0.24310872268001987) | > loss_disc_real_4: 0.2339305430650711 (0.2461618314374168) | > loss_disc_real_5: 0.23761236667633057 (0.24497585785838794) | > loss_0: 2.846574306488037 (2.8428224257703096) | > grad_norm_0: tensor(5.0923, device='cuda:0') (tensor(8.9066, device='cuda:0')) | > loss_gen: 1.7958133220672607 (1.7454660838505007) | > loss_kl: 2.407600164413452 (2.4061545920821863) | > loss_feat: 1.7982659339904785 (1.808483049554645) | > loss_mel: 18.06688690185547 (18.243879426200426) | > loss_duration: 1.712228536605835 (1.6561295783744667) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.780794143676758 (25.860112640092957) | > grad_norm_1: tensor(261.2036, device='cuda:0') (tensor(278.1767, device='cuda:0')) | > current_lr_0: 0.00019768831820240408 | > current_lr_1: 0.00019768831820240408 | > step_time: 0.4494 (0.44338099011835064) | > loader_time: 0.009 (0.008024984935544571) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722140755451897 (+0.0001449215580040259) | > avg_step_time: 0.4624447621090311 (+0.01359421098735969) | > avg_loss_disc: 2.8467465756644663 (+0.0012707878166517972) | > avg_loss_disc_real_0: 0.23057640522298678 (-0.002504479507325408) | > avg_loss_disc_real_1: 0.23659045511568097 (-0.002363064759214162) | > avg_loss_disc_real_2: 0.24164469603081823 (+0.00011779011135368278) | > avg_loss_disc_real_3: 0.24240049216109263 (+0.001482167504203158) | > avg_loss_disc_real_4: 0.2459575309300087 (+8.063647948519659e-05) | > avg_loss_disc_real_5: 0.24572619732836604 (-0.0015950396027363534) | > avg_loss_0: 2.8467465756644663 (+0.0012707878166517972) | > avg_grad_norm_0: tensor(9.0253, device='cuda:0') (tensor(-0.2342, device='cuda:0')) | > avg_loss_gen: 1.7361758964162477 (-0.011083050512931791) | > avg_loss_kl: 2.3879541376946687 (-0.007156053059537371) | > avg_loss_feat: 1.7827810821398882 (+0.006176659758661662) | > avg_loss_mel: 18.132516592321256 (+0.20159188122815763) | > avg_loss_duration: 1.6633330600362428 (-0.001362308649949684) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.702760669547068 (+0.18816695414798446) | > avg_grad_norm_1: tensor(286.7740, device='cuda:0') (+tensor(61.5438, device='cuda:0'))  > EPOCH: 94/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:09:09)   --> TIME: 2025-02-25 09:09:25 -- STEP: 31/72 -- GLOBAL_STEP: 1006800 | > loss_disc: 2.8985135555267334 (2.8390608449136057) | > loss_disc_real_0: 0.3355867564678192 (0.22906597343183333) | > loss_disc_real_1: 0.2344847321510315 (0.23761290456018141) | > loss_disc_real_2: 0.2637145519256592 (0.2412447621745448) | > loss_disc_real_3: 0.26698336005210876 (0.2425222747748898) | > loss_disc_real_4: 0.2599111795425415 (0.2435181136092832) | > loss_disc_real_5: 0.27140989899635315 (0.24423762098435434) | > loss_0: 2.8985135555267334 (2.8390608449136057) | > grad_norm_0: tensor(11.7400, device='cuda:0') (tensor(9.0351, device='cuda:0')) | > loss_gen: 1.836849331855774 (1.7441184328448387) | > loss_kl: 2.4690277576446533 (2.3537788006567184) | > loss_feat: 1.5864038467407227 (1.8038962848724858) | > loss_mel: 18.240737915039062 (18.149824942311934) | > loss_duration: 1.6816942691802979 (1.641942447231662) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.814712524414062 (25.693560815626576) | > grad_norm_1: tensor(279.7000, device='cuda:0') (tensor(274.9311, device='cuda:0')) | > current_lr_0: 0.00019766360716262876 | > current_lr_1: 0.00019766360716262876 | > step_time: 0.4383 (0.43319436042539533) | > loader_time: 0.0089 (0.00670063111089891) --> EVAL PERFORMANCE | > avg_loader_time: 0.008530143281103858 (-0.00019199747434803932) | > avg_step_time: 0.45175812277995364 (-0.010686639329077485) | > avg_loss_disc: 2.8449327844968986 (-0.0018137911675677465) | > avg_loss_disc_real_0: 0.23155355159665497 (+0.0009771463736681918) | > avg_loss_disc_real_1: 0.23621763414900068 (-0.0003728209666802873) | > avg_loss_disc_real_2: 0.23996328279166154 (-0.0016814132391566872) | > avg_loss_disc_real_3: 0.2410225316252507 (-0.0013779605358419378) | > avg_loss_disc_real_4: 0.24444571976930324 (-0.0015118111607054507) | > avg_loss_disc_real_5: 0.2449525004961121 (-0.0007736968322539228) | > avg_loss_0: 2.8449327844968986 (-0.0018137911675677465) | > avg_grad_norm_0: tensor(9.3399, device='cuda:0') (+tensor(0.3145, device='cuda:0')) | > avg_loss_gen: 1.7298745756417933 (-0.006301320774454444) | > avg_loss_kl: 2.3840578106087698 (-0.003896327085898932) | > avg_loss_feat: 1.7952363004147167 (+0.012455218274828539) | > avg_loss_mel: 18.157969998641754 (+0.025453406320497862) | > avg_loss_duration: 1.6658587136738736 (+0.002525653637630798) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.732997168957347 (+0.03023649941027884) | > avg_grad_norm_1: tensor(248.7052, device='cuda:0') (tensor(-38.0687, device='cuda:0'))  > EPOCH: 95/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:09:46)   --> TIME: 2025-02-25 09:09:52 -- STEP: 9/72 -- GLOBAL_STEP: 1006850 | > loss_disc: 2.892669916152954 (2.8752741283840604) | > loss_disc_real_0: 0.21708309650421143 (0.2615746044450336) | > loss_disc_real_1: 0.24253129959106445 (0.2466398643122779) | > loss_disc_real_2: 0.19138957560062408 (0.23543846110502878) | > loss_disc_real_3: 0.20450440049171448 (0.244712480240398) | > loss_disc_real_4: 0.2548876106739044 (0.24019917514589098) | > loss_disc_real_5: 0.2555510401725769 (0.23660814762115479) | > loss_0: 2.892669916152954 (2.8752741283840604) | > grad_norm_0: tensor(6.3658, device='cuda:0') (tensor(11.6261, device='cuda:0')) | > loss_gen: 1.6023749113082886 (1.724878986676534) | > loss_kl: 2.460114002227783 (2.4189844131469727) | > loss_feat: 1.8752925395965576 (1.6887321339713202) | > loss_mel: 19.30060386657715 (18.058408313327366) | > loss_duration: 1.645197868347168 (1.648135119014316) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.883583068847656 (25.539138582017685) | > grad_norm_1: tensor(67.7587, device='cuda:0') (tensor(207.9868, device='cuda:0')) | > current_lr_0: 0.00019763889921173343 | > current_lr_1: 0.00019763889921173343 | > step_time: 0.4192 (0.43000859684414333) | > loader_time: 0.0055 (0.006274806128607856)  --> TIME: 2025-02-25 09:10:15 -- STEP: 59/72 -- GLOBAL_STEP: 1006900 | > loss_disc: 2.7553534507751465 (2.835366685511702) | > loss_disc_real_0: 0.18585681915283203 (0.2324216999744965) | > loss_disc_real_1: 0.2476375848054886 (0.23611663888066503) | > loss_disc_real_2: 0.24573974311351776 (0.24062686097823968) | > loss_disc_real_3: 0.2772822678089142 (0.2414062010534739) | > loss_disc_real_4: 0.26988962292671204 (0.24400596775240818) | > loss_disc_real_5: 0.2764981687068939 (0.2443834957429918) | > loss_0: 2.7553534507751465 (2.835366685511702) | > grad_norm_0: tensor(5.3082, device='cuda:0') (tensor(9.6949, device='cuda:0')) | > loss_gen: 1.865618348121643 (1.7534478761382022) | > loss_kl: 2.2684638500213623 (2.401330972121934) | > loss_feat: 1.7652645111083984 (1.8352544934062633) | > loss_mel: 18.037229537963867 (18.055635710894055) | > loss_duration: 1.6821601390838623 (1.6587785381381794) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.61873435974121 (25.704447406833456) | > grad_norm_1: tensor(81.6665, device='cuda:0') (tensor(211.3747, device='cuda:0')) | > current_lr_0: 0.00019763889921173343 | > current_lr_1: 0.00019763889921173343 | > step_time: 0.4694 (0.4461076259613037) | > loader_time: 0.0099 (0.008189738807031664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008720921798491143 (+0.00019077851738728488) | > avg_step_time: 0.45346606953043334 (+0.0017079467504796941) | > avg_loss_disc: 2.837461851012539 (-0.007470933484359676) | > avg_loss_disc_real_0: 0.22911332684083724 (-0.0024402247558177326) | > avg_loss_disc_real_1: 0.23517805715681803 (-0.0010395769921826559) | > avg_loss_disc_real_2: 0.23932227387394703 (-0.000641008917714514) | > avg_loss_disc_real_3: 0.24059642011850652 (-0.0004261115067441812) | > avg_loss_disc_real_4: 0.24401527580241084 (-0.00043044396689240205) | > avg_loss_disc_real_5: 0.24470106438851694 (-0.00025143610759517654) | > avg_loss_0: 2.837461851012539 (-0.007470933484359676) | > avg_grad_norm_0: tensor(9.8117, device='cuda:0') (+tensor(0.4718, device='cuda:0')) | > avg_loss_gen: 1.7474645879906667 (+0.017590012348873474) | > avg_loss_kl: 2.4035563301032696 (+0.019498519494499877) | > avg_loss_feat: 1.8321851122547204 (+0.03694881184000365) | > avg_loss_mel: 18.03970745247854 (-0.11826254616321208) | > avg_loss_duration: 1.6639474395295264 (-0.0019112741443472192) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68686079643142 (-0.046136372525925395) | > avg_grad_norm_1: tensor(207.6618, device='cuda:0') (tensor(-41.0434, device='cuda:0'))  > EPOCH: 96/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:10:23)   --> TIME: 2025-02-25 09:10:41 -- STEP: 37/72 -- GLOBAL_STEP: 1006950 | > loss_disc: 2.8688712120056152 (2.843111921001125) | > loss_disc_real_0: 0.22090743482112885 (0.23408113480419726) | > loss_disc_real_1: 0.23979586362838745 (0.23713008374781222) | > loss_disc_real_2: 0.23762498795986176 (0.23916640555536425) | > loss_disc_real_3: 0.29993945360183716 (0.2445331111147597) | > loss_disc_real_4: 0.26724886894226074 (0.24486500628896662) | > loss_disc_real_5: 0.29528436064720154 (0.24817387719412107) | > loss_0: 2.8688712120056152 (2.843111921001125) | > grad_norm_0: tensor(4.3237, device='cuda:0') (tensor(8.9973, device='cuda:0')) | > loss_gen: 1.8229628801345825 (1.7528375387191772) | > loss_kl: 2.4020400047302246 (2.4256414013939933) | > loss_feat: 1.6297051906585693 (1.7923862257519283) | > loss_mel: 17.57550621032715 (18.290893760887354) | > loss_duration: 1.6861450672149658 (1.6469355853828225) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.11635971069336 (25.908694653897673) | > grad_norm_1: tensor(293.0597, device='cuda:0') (tensor(221.7886, device='cuda:0')) | > current_lr_0: 0.00019761419434933197 | > current_lr_1: 0.00019761419434933197 | > step_time: 0.4437 (0.4356011828860721) | > loader_time: 0.0083 (0.0074168282586175045) --> EVAL PERFORMANCE | > avg_loader_time: 0.008675961427285635 (-4.496037120550744e-05) | > avg_step_time: 0.4517632471004003 (-0.0017028224300330574) | > avg_loss_disc: 2.8428470047426897 (+0.0053851537301508046) | > avg_loss_disc_real_0: 0.23257285839235278 (+0.0034595315515155434) | > avg_loss_disc_real_1: 0.23584668422249003 (+0.0006686270656720006) | > avg_loss_disc_real_2: 0.24101419641938007 (+0.0016919225454330444) | > avg_loss_disc_real_3: 0.24190733571287612 (+0.001310915594369605) | > avg_loss_disc_real_4: 0.2443800818332484 (+0.0003648060308375778) | > avg_loss_disc_real_5: 0.2454557949808282 (+0.0007547305923112646) | > avg_loss_0: 2.8428470047426897 (+0.0053851537301508046) | > avg_grad_norm_0: tensor(9.1927, device='cuda:0') (tensor(-0.6190, device='cuda:0')) | > avg_loss_gen: 1.7479556312023754 (+0.0004910432117086749) | > avg_loss_kl: 2.40049899799723 (-0.0030573321060396452) | > avg_loss_feat: 1.7888586571518803 (-0.043326455102840056) | > avg_loss_mel: 18.247278401549433 (+0.20757094907089169) | > avg_loss_duration: 1.6650676878405288 (+0.0011202483110024364) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.849659530209824 (+0.16279873377840204) | > avg_grad_norm_1: tensor(233.2506, device='cuda:0') (+tensor(25.5888, device='cuda:0'))  > EPOCH: 97/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:10:59)   --> TIME: 2025-02-25 09:11:08 -- STEP: 15/72 -- GLOBAL_STEP: 1007000 | > loss_disc: 2.8938674926757812 (2.826928981145223) | > loss_disc_real_0: 0.2119663655757904 (0.22602806190649669) | > loss_disc_real_1: 0.210460364818573 (0.2364251305659612) | > loss_disc_real_2: 0.2578149735927582 (0.2348844329516093) | > loss_disc_real_3: 0.22701798379421234 (0.2398663212855657) | > loss_disc_real_4: 0.25966474413871765 (0.24332582155863444) | > loss_disc_real_5: 0.2730017602443695 (0.24354623953501384) | > loss_0: 2.8938674926757812 (2.826928981145223) | > grad_norm_0: tensor(7.6621, device='cuda:0') (tensor(9.1606, device='cuda:0')) | > loss_gen: 1.6426771879196167 (1.7303386847178142) | > loss_kl: 2.4135079383850098 (2.3973111788431805) | > loss_feat: 1.6746193170547485 (1.822993262608846) | > loss_mel: 19.252784729003906 (18.65646947224935) | > loss_duration: 1.6388096809387207 (1.6615913073221842) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.622398376464844 (26.268704223632813) | > grad_norm_1: tensor(177.3422, device='cuda:0') (tensor(237.9268, device='cuda:0')) | > current_lr_0: 0.0001975894925750383 | > current_lr_1: 0.0001975894925750383 | > step_time: 0.4368 (0.4360663890838623) | > loader_time: 0.0058 (0.006370671590169271)  --> TIME: 2025-02-25 09:11:32 -- STEP: 65/72 -- GLOBAL_STEP: 1007050 | > loss_disc: 2.790252685546875 (2.840381915752704) | > loss_disc_real_0: 0.20105916261672974 (0.2307037245768767) | > loss_disc_real_1: 0.23580436408519745 (0.2377942871588927) | > loss_disc_real_2: 0.2451656311750412 (0.23865460661741403) | > loss_disc_real_3: 0.2621800899505615 (0.24223673802155715) | > loss_disc_real_4: 0.2545139193534851 (0.24498347915135898) | > loss_disc_real_5: 0.23012188076972961 (0.24619783392319314) | > loss_0: 2.790252685546875 (2.840381915752704) | > grad_norm_0: tensor(6.1809, device='cuda:0') (tensor(8.6530, device='cuda:0')) | > loss_gen: 1.8153076171875 (1.7438565529309786) | > loss_kl: 2.380336284637451 (2.3728033982790433) | > loss_feat: 1.9162089824676514 (1.8206022079174335) | > loss_mel: 18.83551597595215 (18.500929612379807) | > loss_duration: 1.697409987449646 (1.6715445811931904) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.644779205322266 (26.10973645723783) | > grad_norm_1: tensor(67.4119, device='cuda:0') (tensor(228.1180, device='cuda:0')) | > current_lr_0: 0.0001975894925750383 | > current_lr_1: 0.0001975894925750383 | > step_time: 0.4986 (0.45002280748807466) | > loader_time: 0.0118 (0.00830127275907076) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570110294180854 (-0.00010585113310478123) | > avg_step_time: 0.45565544719427403 (+0.0038922000938737544) | > avg_loss_disc: 2.843399531404737 (+0.0005525266620471925) | > avg_loss_disc_real_0: 0.22960002405542723 (-0.0029728343369255505) | > avg_loss_disc_real_1: 0.23787932538650405 (+0.0020326411640140263) | > avg_loss_disc_real_2: 0.2386679508736436 (-0.0023462455457364806) | > avg_loss_disc_real_3: 0.24097493983490367 (-0.0009323958779724473) | > avg_loss_disc_real_4: 0.24548503798498233 (+0.0011049561517339213) | > avg_loss_disc_real_5: 0.2482771982609386 (+0.002821403280110407) | > avg_loss_0: 2.843399531404737 (+0.0005525266620471925) | > avg_grad_norm_0: tensor(9.4130, device='cuda:0') (+tensor(0.2203, device='cuda:0')) | > avg_loss_gen: 1.7410573086268466 (-0.006898322575528804) | > avg_loss_kl: 2.3744717550949312 (-0.02602724290229874) | > avg_loss_feat: 1.8143460498729223 (+0.025487392721041946) | > avg_loss_mel: 18.459961098684392 (+0.2126826971349587) | > avg_loss_duration: 1.673531090709525 (+0.008463402868996273) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.063367360074754 (+0.21370782986492998) | > avg_grad_norm_1: tensor(232.8707, device='cuda:0') (tensor(-0.3799, device='cuda:0'))  > EPOCH: 98/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:11:36)   --> TIME: 2025-02-25 09:11:58 -- STEP: 43/72 -- GLOBAL_STEP: 1007100 | > loss_disc: 2.90559458732605 (2.8422831712767134) | > loss_disc_real_0: 0.26194947957992554 (0.2245746837105862) | > loss_disc_real_1: 0.21439257264137268 (0.23159696750862654) | > loss_disc_real_2: 0.2498631626367569 (0.23851790261823078) | > loss_disc_real_3: 0.250087171792984 (0.2420199589673863) | > loss_disc_real_4: 0.240674689412117 (0.24491920270199) | > loss_disc_real_5: 0.2523328959941864 (0.24853708439095076) | > loss_0: 2.90559458732605 (2.8422831712767134) | > grad_norm_0: tensor(11.5058, device='cuda:0') (tensor(11.6345, device='cuda:0')) | > loss_gen: 1.6270594596862793 (1.7236547026523323) | > loss_kl: 2.2883377075195312 (2.491924363513325) | > loss_feat: 1.4845948219299316 (1.841895347417787) | > loss_mel: 18.055747985839844 (18.009279339812522) | > loss_duration: 1.669986605644226 (1.6539397101069606) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.1257266998291 (25.720693765684615) | > grad_norm_1: tensor(243.5566, device='cuda:0') (tensor(219.4954, device='cuda:0')) | > current_lr_0: 0.0001975647938884664 | > current_lr_1: 0.0001975647938884664 | > step_time: 0.4443 (0.438190171884936) | > loader_time: 0.0093 (0.007677139237869618) --> EVAL PERFORMANCE | > avg_loader_time: 0.008736059699260014 (+0.00016594940507915976) | > avg_step_time: 0.4528855001422721 (-0.002769947052001953) | > avg_loss_disc: 2.8565252834642436 (+0.013125752059506723) | > avg_loss_disc_real_0: 0.2263391336924593 (-0.003260890362967922) | > avg_loss_disc_real_1: 0.23354977012520106 (-0.004329555261302992) | > avg_loss_disc_real_2: 0.24125480420992407 (+0.0025868533362804758) | > avg_loss_disc_real_3: 0.24580359291022932 (+0.004828653075325651) | > avg_loss_disc_real_4: 0.2466210580207932 (+0.0011360200358108696) | > avg_loss_disc_real_5: 0.248907498280767 (+0.0006303000198283892) | > avg_loss_0: 2.8565252834642436 (+0.013125752059506723) | > avg_grad_norm_0: tensor(11.2558, device='cuda:0') (+tensor(1.8428, device='cuda:0')) | > avg_loss_gen: 1.7235490839246292 (-0.017508224702217356) | > avg_loss_kl: 2.4719751922177595 (+0.09750343712282827) | > avg_loss_feat: 1.7854540045832243 (-0.02889204528969791) | > avg_loss_mel: 18.11053149800905 (-0.349429600675343) | > avg_loss_duration: 1.6664663972988936 (-0.0070646934106315395) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.757976290205836 (-0.30539106986891795) | > avg_grad_norm_1: tensor(238.2716, device='cuda:0') (+tensor(5.4009, device='cuda:0'))  > EPOCH: 99/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:12:13)   --> TIME: 2025-02-25 09:12:24 -- STEP: 21/72 -- GLOBAL_STEP: 1007150 | > loss_disc: 2.8524246215820312 (2.834554978779384) | > loss_disc_real_0: 0.2061687558889389 (0.23294606450058164) | > loss_disc_real_1: 0.2931002080440521 (0.22785461374691554) | > loss_disc_real_2: 0.21958033740520477 (0.23475337738082522) | > loss_disc_real_3: 0.23738320171833038 (0.23951083776496707) | > loss_disc_real_4: 0.27166032791137695 (0.25006349100953057) | > loss_disc_real_5: 0.2498731017112732 (0.24640616987432754) | > loss_0: 2.8524246215820312 (2.834554978779384) | > grad_norm_0: tensor(5.7035, device='cuda:0') (tensor(9.6345, device='cuda:0')) | > loss_gen: 1.7755186557769775 (1.7582676297142392) | > loss_kl: 2.444927453994751 (2.369567042305356) | > loss_feat: 1.7122039794921875 (1.8887797650836764) | > loss_mel: 17.273435592651367 (18.281419299897692) | > loss_duration: 1.691765546798706 (1.6440935645784651) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.897850036621094 (25.942127136957076) | > grad_norm_1: tensor(85.1113, device='cuda:0') (tensor(228.1577, device='cuda:0')) | > current_lr_0: 0.00019754009828923033 | > current_lr_1: 0.00019754009828923033 | > step_time: 0.4356 (0.43337660744076684) | > loader_time: 0.0072 (0.00678933234441848)  --> TIME: 2025-02-25 09:12:48 -- STEP: 71/72 -- GLOBAL_STEP: 1007200 | > loss_disc: 2.8531057834625244 (2.8485229854852383) | > loss_disc_real_0: 0.28748002648353577 (0.23352243929681643) | > loss_disc_real_1: 0.28929802775382996 (0.23454856935521246) | > loss_disc_real_2: 0.2309664785861969 (0.23719295691436446) | > loss_disc_real_3: 0.24078072607517242 (0.24182128213660817) | > loss_disc_real_4: 0.2560441792011261 (0.24663994702654826) | > loss_disc_real_5: 0.2868243157863617 (0.24710354754622554) | > loss_0: 2.8531057834625244 (2.8485229854852383) | > grad_norm_0: tensor(13.0257, device='cuda:0') (tensor(9.9222, device='cuda:0')) | > loss_gen: 1.7380576133728027 (1.7452263378761184) | > loss_kl: 2.361119508743286 (2.4339760793766496) | > loss_feat: 1.4999747276306152 (1.8194151310853555) | > loss_mel: 16.93572235107422 (18.33741746821874) | > loss_duration: 1.6862624883651733 (1.666449335259451) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.22113609313965 (26.00248422757001) | > grad_norm_1: tensor(80.9031, device='cuda:0') (tensor(189.6463, device='cuda:0')) | > current_lr_0: 0.00019754009828923033 | > current_lr_1: 0.00019754009828923033 | > step_time: 0.4937 (0.4530146726420228) | > loader_time: 0.0079 (0.008605235059496385) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605235059496385 (-0.0001308246397636291) | > avg_step_time: 0.4530146726420228 (+0.000129172499750696) | > avg_loss_disc: 2.8485229854852383 (-0.008002297979005313) | > avg_loss_disc_real_0: 0.23352243929681643 (+0.007183305604357121) | > avg_loss_disc_real_1: 0.23454856935521246 (+0.0009987992300113935) | > avg_loss_disc_real_2: 0.23719295691436446 (-0.004061847295559612) | > avg_loss_disc_real_3: 0.24182128213660817 (-0.003982310773621156) | > avg_loss_disc_real_4: 0.24663994702654826 (+1.8889005755057253e-05) | > avg_loss_disc_real_5: 0.24710354754622554 (-0.001803950734541454) | > avg_loss_0: 2.8485229854852383 (-0.008002297979005313) | > avg_grad_norm_0: tensor(9.9222, device='cuda:0') (tensor(-1.3336, device='cuda:0')) | > avg_loss_gen: 1.7452263378761184 (+0.021677253951489117) | > avg_loss_kl: 2.4339760793766496 (-0.037999112841109906) | > avg_loss_feat: 1.8194151310853555 (+0.03396112650213112) | > avg_loss_mel: 18.33741746821874 (+0.2268859702096897) | > avg_loss_duration: 1.666449335259451 (-1.7062039442627963e-05) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.00248422757001 (+0.2445079373641761) | > avg_grad_norm_1: tensor(189.6463, device='cuda:0') (tensor(-48.6253, device='cuda:0'))  > EPOCH: 100/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:12:50)   --> TIME: 2025-02-25 09:13:14 -- STEP: 49/72 -- GLOBAL_STEP: 1007250 | > loss_disc: 2.823737382888794 (2.839214334682542) | > loss_disc_real_0: 0.16721117496490479 (0.22325584322822337) | > loss_disc_real_1: 0.2602999806404114 (0.2347907150278286) | > loss_disc_real_2: 0.217144176363945 (0.2413884091133974) | > loss_disc_real_3: 0.21508049964904785 (0.238876968013997) | > loss_disc_real_4: 0.23488403856754303 (0.24576900990641848) | > loss_disc_real_5: 0.2557973861694336 (0.2458621102328203) | > loss_0: 2.823737382888794 (2.839214334682542) | > grad_norm_0: tensor(9.6467, device='cuda:0') (tensor(11.1958, device='cuda:0')) | > loss_gen: 1.876655101776123 (1.7452094019675741) | > loss_kl: 2.466073513031006 (2.443119238833991) | > loss_feat: 1.8262351751327515 (1.8607412868616533) | > loss_mel: 18.982280731201172 (18.46838230989417) | > loss_duration: 1.7476279735565186 (1.6567903100227823) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.89887237548828 (26.17424248675911) | > grad_norm_1: tensor(196.3090, device='cuda:0') (tensor(238.9874, device='cuda:0')) | > current_lr_0: 0.00019751540577694416 | > current_lr_1: 0.00019751540577694416 | > step_time: 0.45 (0.44232458970984634) | > loader_time: 0.0094 (0.007754379389237384) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864849627857477 (+4.326121907838494e-05) | > avg_step_time: 0.4542885733322358 (+0.0012739006902130479) | > avg_loss_disc: 2.8436963591777094 (-0.004826626307528947) | > avg_loss_disc_real_0: 0.22759023077890905 (-0.005932208517907378) | > avg_loss_disc_real_1: 0.23530582810791445 (+0.0007572587527019947) | > avg_loss_disc_real_2: 0.24196379999039877 (+0.004770843076034315) | > avg_loss_disc_real_3: 0.24110915963078888 (-0.0007121225058192848) | > avg_loss_disc_real_4: 0.24567506535792014 (-0.0009648816686281259) | > avg_loss_disc_real_5: 0.2462439348160381 (-0.0008596127301874479) | > avg_loss_0: 2.8436963591777094 (-0.004826626307528947) | > avg_grad_norm_0: tensor(10.5195, device='cuda:0') (+tensor(0.5972, device='cuda:0')) | > avg_loss_gen: 1.735730550658535 (-0.009495787217583374) | > avg_loss_kl: 2.43525339180315 (+0.001277312426500199) | > avg_loss_feat: 1.80723443837233 (-0.012180692713025465) | > avg_loss_mel: 18.355855511947414 (+0.018438043728675524) | > avg_loss_duration: 1.6642404794692993 (-0.0022088557901516204) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.998314306769572 (-0.004169920800439542) | > avg_grad_norm_1: tensor(228.1575, device='cuda:0') (+tensor(38.5111, device='cuda:0'))  > EPOCH: 101/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:13:27)   --> TIME: 2025-02-25 09:13:41 -- STEP: 27/72 -- GLOBAL_STEP: 1007300 | > loss_disc: 2.9130473136901855 (2.839447913346467) | > loss_disc_real_0: 0.16437718272209167 (0.22074398784725754) | > loss_disc_real_1: 0.2843588888645172 (0.2388601518339581) | > loss_disc_real_2: 0.24052879214286804 (0.24335202464350947) | > loss_disc_real_3: 0.2558841109275818 (0.24039436490447433) | > loss_disc_real_4: 0.2594814896583557 (0.24758682703530346) | > loss_disc_real_5: 0.25027427077293396 (0.24674482643604279) | > loss_0: 2.9130473136901855 (2.839447913346467) | > grad_norm_0: tensor(13.8868, device='cuda:0') (tensor(8.6741, device='cuda:0')) | > loss_gen: 1.6216976642608643 (1.7444869456467804) | > loss_kl: 2.6685941219329834 (2.4343695375654426) | > loss_feat: 1.60382878780365 (1.8443154405664515) | > loss_mel: 16.319990158081055 (18.282650770964445) | > loss_duration: 1.5954315662384033 (1.6382906701829698) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.80954360961914 (25.944113484135382) | > grad_norm_1: tensor(275.9817, device='cuda:0') (tensor(229.8331, device='cuda:0')) | > current_lr_0: 0.00019749071635122203 | > current_lr_1: 0.00019749071635122203 | > step_time: 0.4402 (0.4376282603652389) | > loader_time: 0.0083 (0.006901449627346463) --> EVAL PERFORMANCE | > avg_loader_time: 0.008516335151564905 (-0.00013216112700986486) | > avg_step_time: 0.45405506080305075 (-0.00023351252918507592) | > avg_loss_disc: 2.849330035733505 (+0.00563367655579583) | > avg_loss_disc_real_0: 0.22454684038817044 (-0.003043390390738615) | > avg_loss_disc_real_1: 0.23716555997519426 (+0.0018597318672798069) | > avg_loss_disc_real_2: 0.24285782705730116 (+0.000894027066902392) | > avg_loss_disc_real_3: 0.24203988049231784 (+0.0009307208615289553) | > avg_loss_disc_real_4: 0.24684574507491688 (+0.0011706797169967476) | > avg_loss_disc_real_5: 0.2479454797338432 (+0.001701544917805109) | > avg_loss_0: 2.849330035733505 (+0.00563367655579583) | > avg_grad_norm_0: tensor(9.3251, device='cuda:0') (tensor(-1.1944, device='cuda:0')) | > avg_loss_gen: 1.735814549553562 (+8.3998895026971e-05) | > avg_loss_kl: 2.37954499687947 (-0.05570839492367963) | > avg_loss_feat: 1.8058968530574315 (-0.0013375853148984707) | > avg_loss_mel: 17.970017795831374 (-0.3858377161160398) | > avg_loss_duration: 1.659783553069746 (-0.004456926399553307) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.551057788687693 (-0.44725651808187905) | > avg_grad_norm_1: tensor(222.7006, device='cuda:0') (tensor(-5.4568, device='cuda:0'))  > EPOCH: 102/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:14:04)   --> TIME: 2025-02-25 09:14:08 -- STEP: 5/72 -- GLOBAL_STEP: 1007350 | > loss_disc: 2.7876462936401367 (2.807144021987915) | > loss_disc_real_0: 0.2054121047258377 (0.2160310924053192) | > loss_disc_real_1: 0.16360267996788025 (0.2235386848449707) | > loss_disc_real_2: 0.270195335149765 (0.2293948858976364) | > loss_disc_real_3: 0.26691734790802 (0.23060463666915892) | > loss_disc_real_4: 0.24212367832660675 (0.24251160323619841) | > loss_disc_real_5: 0.27408209443092346 (0.2507777541875839) | > loss_0: 2.7876462936401367 (2.807144021987915) | > grad_norm_0: tensor(6.3718, device='cuda:0') (tensor(7.0482, device='cuda:0')) | > loss_gen: 1.8364083766937256 (1.7388893365859985) | > loss_kl: 2.1916956901550293 (2.352883529663086) | > loss_feat: 1.907353162765503 (1.8559232234954834) | > loss_mel: 18.04445457458496 (17.834932327270508) | > loss_duration: 1.6220827102661133 (1.6530301094055175) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.601993560791016 (25.43565788269043) | > grad_norm_1: tensor(134.7484, device='cuda:0') (tensor(310.5513, device='cuda:0')) | > current_lr_0: 0.00019746603001167813 | > current_lr_1: 0.00019746603001167813 | > step_time: 0.4404 (0.4325725555419922) | > loader_time: 0.0083 (0.006185340881347656)  --> TIME: 2025-02-25 09:14:31 -- STEP: 55/72 -- GLOBAL_STEP: 1007400 | > loss_disc: 2.8358118534088135 (2.843480027805675) | > loss_disc_real_0: 0.27639225125312805 (0.23068587211045352) | > loss_disc_real_1: 0.21288229525089264 (0.23533127036961643) | > loss_disc_real_2: 0.2381116896867752 (0.23916460302743045) | > loss_disc_real_3: 0.2429223358631134 (0.2400801282037388) | > loss_disc_real_4: 0.2546062469482422 (0.24587540382688697) | > loss_disc_real_5: 0.265064537525177 (0.25039361796595816) | > loss_0: 2.8358118534088135 (2.843480027805675) | > grad_norm_0: tensor(10.1980, device='cuda:0') (tensor(9.6001, device='cuda:0')) | > loss_gen: 1.81718111038208 (1.7457669128071178) | > loss_kl: 2.535944938659668 (2.4278060609644108) | > loss_feat: 1.7169570922851562 (1.8228506109931253) | > loss_mel: 18.783634185791016 (18.075966887040572) | > loss_duration: 1.7034908533096313 (1.6650612766092474) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.557209014892578 (25.73745158802379) | > grad_norm_1: tensor(320.6658, device='cuda:0') (tensor(192.2118, device='cuda:0')) | > current_lr_0: 0.00019746603001167813 | > current_lr_1: 0.00019746603001167813 | > step_time: 0.46 (0.4414457928050648) | > loader_time: 0.0094 (0.00789555636319247) --> EVAL PERFORMANCE | > avg_loader_time: 0.008622176210645219 (+0.00010584105908031417) | > avg_step_time: 0.4517888586286088 (-0.002266202174441956) | > avg_loss_disc: 2.8444476362685083 (-0.004882399464996912) | > avg_loss_disc_real_0: 0.23178553623212894 (+0.007238695843958504) | > avg_loss_disc_real_1: 0.23660522258617508 (-0.0005603373890191721) | > avg_loss_disc_real_2: 0.2382602912019676 (-0.004597535855333568) | > avg_loss_disc_real_3: 0.23974048566650336 (-0.0022993948258144825) | > avg_loss_disc_real_4: 0.24657154314115015 (-0.0002742019337667323) | > avg_loss_disc_real_5: 0.25053579412715526 (+0.00259031439331206) | > avg_loss_0: 2.8444476362685083 (-0.004882399464996912) | > avg_grad_norm_0: tensor(9.5807, device='cuda:0') (+tensor(0.2556, device='cuda:0')) | > avg_loss_gen: 1.741906681530912 (+0.006092131977350146) | > avg_loss_kl: 2.4244095809023145 (+0.04486458402284432) | > avg_loss_feat: 1.800058467287413 (-0.005838385770018606) | > avg_loss_mel: 18.18203036885866 (+0.21201257302728393) | > avg_loss_duration: 1.6695900030539068 (+0.009806449984160803) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.817994937090806 (+0.26693714840311245) | > avg_grad_norm_1: tensor(203.4392, device='cuda:0') (tensor(-19.2614, device='cuda:0'))  > EPOCH: 103/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:14:41)   --> TIME: 2025-02-25 09:14:58 -- STEP: 33/72 -- GLOBAL_STEP: 1007450 | > loss_disc: 2.8544790744781494 (2.858577793294733) | > loss_disc_real_0: 0.15998345613479614 (0.22927260647217432) | > loss_disc_real_1: 0.2641802430152893 (0.23187588832595132) | > loss_disc_real_2: 0.20875151455402374 (0.24021179296753623) | > loss_disc_real_3: 0.22348318994045258 (0.24097912781166309) | > loss_disc_real_4: 0.20963580906391144 (0.24801855421427524) | > loss_disc_real_5: 0.21699146926403046 (0.2478064849521174) | > loss_0: 2.8544790744781494 (2.858577793294733) | > grad_norm_0: tensor(8.9274, device='cuda:0') (tensor(9.2810, device='cuda:0')) | > loss_gen: 1.750420331954956 (1.7334271416519627) | > loss_kl: 2.417335271835327 (2.4565224358529756) | > loss_feat: 1.905555009841919 (1.8400613394650547) | > loss_mel: 18.059268951416016 (18.069743185332328) | > loss_duration: 1.6205530166625977 (1.6349279374787302) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.753131866455078 (25.734681967532996) | > grad_norm_1: tensor(138.8483, device='cuda:0') (tensor(173.9261, device='cuda:0')) | > current_lr_0: 0.00019744134675792665 | > current_lr_1: 0.00019744134675792665 | > step_time: 0.4395 (0.4401511929251931) | > loader_time: 0.0077 (0.0070432605165423766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534216544997524 (-8.795966564769525e-05) | > avg_step_time: 0.45467884775618433 (+0.002889989127575543) | > avg_loss_disc: 2.860469955793569 (+0.016022319525060613) | > avg_loss_disc_real_0: 0.23413856988641576 (+0.0023530336542868235) | > avg_loss_disc_real_1: 0.2343029109105258 (-0.002302311675649288) | > avg_loss_disc_real_2: 0.24191395749508496 (+0.0036536662931173636) | > avg_loss_disc_real_3: 0.24317574941776168 (+0.0034352637512583273) | > avg_loss_disc_real_4: 0.24730040300899828 (+0.0007288598678481251) | > avg_loss_disc_real_5: 0.248378917365007 (-0.002156876762148263) | > avg_loss_0: 2.860469955793569 (+0.016022319525060613) | > avg_grad_norm_0: tensor(9.7641, device='cuda:0') (+tensor(0.1834, device='cuda:0')) | > avg_loss_gen: 1.7344495914351772 (-0.0074570900957349195) | > avg_loss_kl: 2.44023829446712 (+0.01582871356480542) | > avg_loss_feat: 1.7767246189251753 (-0.02333384836223762) | > avg_loss_mel: 17.937352516281766 (-0.244677852576892) | > avg_loss_duration: 1.659590502859841 (-0.009999500194065813) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.54835550550004 (-0.26963943159076464) | > avg_grad_norm_1: tensor(189.3575, device='cuda:0') (tensor(-14.0817, device='cuda:0'))  > EPOCH: 104/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:15:18)   --> TIME: 2025-02-25 09:15:24 -- STEP: 11/72 -- GLOBAL_STEP: 1007500 | > loss_disc: 2.8669273853302 (2.8435234373266045) | > loss_disc_real_0: 0.2657109200954437 (0.23328714885494925) | > loss_disc_real_1: 0.17984087765216827 (0.24402513558214362) | > loss_disc_real_2: 0.2876445949077606 (0.24051441929557107) | > loss_disc_real_3: 0.21357019245624542 (0.24002885005690835) | > loss_disc_real_4: 0.24179551005363464 (0.23778190531513907) | > loss_disc_real_5: 0.24409635365009308 (0.23762290179729462) | > loss_0: 2.8669273853302 (2.8435234373266045) | > grad_norm_0: tensor(8.0176, device='cuda:0') (tensor(10.0650, device='cuda:0')) | > loss_gen: 1.7640044689178467 (1.7394300265745684) | > loss_kl: 2.5514864921569824 (2.391209689053622) | > loss_feat: 1.92684006690979 (1.8513339974663474) | > loss_mel: 18.010841369628906 (18.01348079334606) | > loss_duration: 1.6136125326156616 (1.6406025994907727) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.866785049438477 (25.636057593605734) | > grad_norm_1: tensor(270.2138, device='cuda:0') (tensor(270.3638, device='cuda:0')) | > current_lr_0: 0.0001974166665895819 | > current_lr_1: 0.0001974166665895819 | > step_time: 0.4205 (0.4293569868261164) | > loader_time: 0.0056 (0.005980751731178977)  --> TIME: 2025-02-25 09:15:48 -- STEP: 61/72 -- GLOBAL_STEP: 1007550 | > loss_disc: 2.837066888809204 (2.8421464435389785) | > loss_disc_real_0: 0.20827746391296387 (0.22294951706636149) | > loss_disc_real_1: 0.24826978147029877 (0.23922080925253572) | > loss_disc_real_2: 0.20882844924926758 (0.24060768011163491) | > loss_disc_real_3: 0.21605843305587769 (0.24360844908190554) | > loss_disc_real_4: 0.25853294134140015 (0.24749178598161603) | > loss_disc_real_5: 0.271754652261734 (0.2489130333310268) | > loss_0: 2.837066888809204 (2.8421464435389785) | > grad_norm_0: tensor(4.8941, device='cuda:0') (tensor(9.8465, device='cuda:0')) | > loss_gen: 1.7650128602981567 (1.7547430171341192) | > loss_kl: 2.431561231613159 (2.4072728704233644) | > loss_feat: 1.7889947891235352 (1.8978632121789651) | > loss_mel: 17.63347816467285 (18.144542881699866) | > loss_duration: 1.6936476230621338 (1.6551161887215788) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.312694549560547 (25.859538312818184) | > grad_norm_1: tensor(185.7710, device='cuda:0') (tensor(220.4965, device='cuda:0')) | > current_lr_0: 0.0001974166665895819 | > current_lr_1: 0.0001974166665895819 | > step_time: 0.4588 (0.4447429883675497) | > loader_time: 0.0109 (0.007955844285058194) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438395782255791 (-9.582076274173255e-05) | > avg_step_time: 0.45467489873859246 (-3.949017591875492e-06) | > avg_loss_disc: 2.8443479034262644 (-0.016122052367304462) | > avg_loss_disc_real_0: 0.22755096391053267 (-0.006587605975883093) | > avg_loss_disc_real_1: 0.23889093499788097 (+0.004588024087355175) | > avg_loss_disc_real_2: 0.2402386982256258 (-0.0016752592694591617) | > avg_loss_disc_real_3: 0.2442008376541272 (+0.0010250882363655178) | > avg_loss_disc_real_4: 0.24613200484866826 (-0.0011683981603300142) | > avg_loss_disc_real_5: 0.24804471978839016 (-0.0003341975766168459) | > avg_loss_0: 2.8443479034262644 (-0.016122052367304462) | > avg_grad_norm_0: tensor(9.8203, device='cuda:0') (+tensor(0.0561, device='cuda:0')) | > avg_loss_gen: 1.7500778990732115 (+0.015628307638034356) | > avg_loss_kl: 2.410270237586868 (-0.029968056880251837) | > avg_loss_feat: 1.8572701115003774 (+0.08054549257520205) | > avg_loss_mel: 18.02633399694739 (+0.08898148066562328) | > avg_loss_duration: 1.6600601740286387 (+0.0004696711687977029) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.70401250812369 (+0.15565700262364857) | > avg_grad_norm_1: tensor(213.6638, device='cuda:0') (+tensor(24.3062, device='cuda:0'))  > EPOCH: 105/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:15:55)   --> TIME: 2025-02-25 09:16:14 -- STEP: 39/72 -- GLOBAL_STEP: 1007600 | > loss_disc: 2.8024344444274902 (2.8461049581185365) | > loss_disc_real_0: 0.1536007523536682 (0.23002698100530183) | > loss_disc_real_1: 0.2104606330394745 (0.23465312520662943) | > loss_disc_real_2: 0.2579183876514435 (0.23968128936413008) | > loss_disc_real_3: 0.27902206778526306 (0.24266690206833375) | > loss_disc_real_4: 0.27571901679039 (0.24720699359209108) | > loss_disc_real_5: 0.27056610584259033 (0.2480660046522434) | > loss_0: 2.8024344444274902 (2.8461049581185365) | > grad_norm_0: tensor(9.3940, device='cuda:0') (tensor(8.3558, device='cuda:0')) | > loss_gen: 1.7897504568099976 (1.7430633459335718) | > loss_kl: 2.4756314754486084 (2.4317407669165187) | > loss_feat: 1.841991662979126 (1.841716048045036) | > loss_mel: 17.357627868652344 (18.143593030098156) | > loss_duration: 1.6514816284179688 (1.6438265519264417) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.116483688354492 (25.80393981933594) | > grad_norm_1: tensor(402.2325, device='cuda:0') (tensor(206.6382, device='cuda:0')) | > current_lr_0: 0.0001973919895062582 | > current_lr_1: 0.0001973919895062582 | > step_time: 0.4744 (0.43547707948929226) | > loader_time: 0.008 (0.007330643825041942) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640702341643857 (+0.00020230655938806644) | > avg_step_time: 0.45082865970235475 (-0.0038462390362377086) | > avg_loss_disc: 2.846992449021675 (+0.002644545595410719) | > avg_loss_disc_real_0: 0.22824404357184827 (+0.0006930796613155987) | > avg_loss_disc_real_1: 0.23581665691355586 (-0.003074278084325116) | > avg_loss_disc_real_2: 0.24078652682438703 (+0.0005478285987612352) | > avg_loss_disc_real_3: 0.24077987230159867 (-0.003420965352528532) | > avg_loss_disc_real_4: 0.24741625408051718 (+0.0012842492318489163) | > avg_loss_disc_real_5: 0.24847173334007533 (+0.00042701355168517363) | > avg_loss_0: 2.846992449021675 (+0.002644545595410719) | > avg_grad_norm_0: tensor(8.4207, device='cuda:0') (tensor(-1.3995, device='cuda:0')) | > avg_loss_gen: 1.738144624401146 (-0.011933274672065508) | > avg_loss_kl: 2.3871273960865724 (-0.023142841500295663) | > avg_loss_feat: 1.825586648054526 (-0.03168346344585138) | > avg_loss_mel: 17.806628818243322 (-0.2197051787040678) | > avg_loss_duration: 1.6586344661846968 (-0.0014257078439419146) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.41612197983433 (-0.2878905282893598) | > avg_grad_norm_1: tensor(228.0280, device='cuda:0') (+tensor(14.3642, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1007633.pth  > EPOCH: 106/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:16:34)   --> TIME: 2025-02-25 09:16:43 -- STEP: 17/72 -- GLOBAL_STEP: 1007650 | > loss_disc: 2.930431365966797 (2.8200972781461826) | > loss_disc_real_0: 0.27856338024139404 (0.2180304211728713) | > loss_disc_real_1: 0.18485286831855774 (0.21779660617603974) | > loss_disc_real_2: 0.2537737488746643 (0.23350243708666632) | > loss_disc_real_3: 0.24093389511108398 (0.23845696098664226) | > loss_disc_real_4: 0.23608139157295227 (0.24469436968074126) | > loss_disc_real_5: 0.2565968334674835 (0.24333354217164657) | > loss_0: 2.930431365966797 (2.8200972781461826) | > grad_norm_0: tensor(10.0925, device='cuda:0') (tensor(11.2876, device='cuda:0')) | > loss_gen: 1.7477271556854248 (1.7287739024442785) | > loss_kl: 2.6169745922088623 (2.355369077009313) | > loss_feat: 2.031388998031616 (1.9615579142290003) | > loss_mel: 19.398773193359375 (18.327765296487247) | > loss_duration: 1.6833162307739258 (1.6315066183314604) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.478179931640625 (26.004972457885742) | > grad_norm_1: tensor(223.1129, device='cuda:0') (tensor(226.0231, device='cuda:0')) | > current_lr_0: 0.0001973673155075699 | > current_lr_1: 0.0001973673155075699 | > step_time: 0.4347 (0.43439900173860435) | > loader_time: 0.006 (0.006933717166676241)  --> TIME: 2025-02-25 09:17:07 -- STEP: 67/72 -- GLOBAL_STEP: 1007700 | > loss_disc: 2.882356882095337 (2.829213153070478) | > loss_disc_real_0: 0.16426804661750793 (0.2238285763939815) | > loss_disc_real_1: 0.23433716595172882 (0.23195294880155307) | > loss_disc_real_2: 0.24507689476013184 (0.23654807548024762) | > loss_disc_real_3: 0.23871085047721863 (0.23857185911776416) | > loss_disc_real_4: 0.2327325940132141 (0.24388691871913512) | > loss_disc_real_5: 0.2213994264602661 (0.24603439022356005) | > loss_0: 2.882356882095337 (2.829213153070478) | > grad_norm_0: tensor(11.6232, device='cuda:0') (tensor(11.4239, device='cuda:0')) | > loss_gen: 1.7739561796188354 (1.7382516594075446) | > loss_kl: 2.3664000034332275 (2.3883745385639705) | > loss_feat: 1.8301893472671509 (1.877578797625072) | > loss_mel: 18.318208694458008 (18.05661275493565) | > loss_duration: 1.7170159816741943 (1.6551833241733152) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.005769729614258 (25.716000912794428) | > grad_norm_1: tensor(354.2353, device='cuda:0') (tensor(248.4685, device='cuda:0')) | > current_lr_0: 0.0001973673155075699 | > current_lr_1: 0.0001973673155075699 | > step_time: 0.4949 (0.4513668053185762) | > loader_time: 0.0122 (0.008428231993717936) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860809944045376 (-3.260290119009787e-05) | > avg_step_time: 0.45521030291705067 (+0.004381643214695918) | > avg_loss_disc: 2.8320605788432376 (-0.014931870178437556) | > avg_loss_disc_real_0: 0.22566790878772736 (-0.002576134784120915) | > avg_loss_disc_real_1: 0.23277710294219808 (-0.0030395539713577724) | > avg_loss_disc_real_2: 0.23742779690614887 (-0.003358729918238157) | > avg_loss_disc_real_3: 0.23893147308222004 (-0.001848399219378627) | > avg_loss_disc_real_4: 0.24398590792232835 (-0.003430346158188824) | > avg_loss_disc_real_5: 0.2460482697671568 (-0.0024234635729185328) | > avg_loss_0: 2.8320605788432376 (-0.014931870178437556) | > avg_grad_norm_0: tensor(11.6722, device='cuda:0') (+tensor(3.2514, device='cuda:0')) | > avg_loss_gen: 1.7369145658654226 (-0.001230058535723444) | > avg_loss_kl: 2.386414615201279 (-0.0007127808852933626) | > avg_loss_feat: 1.8630145485972014 (+0.03742790054267542) | > avg_loss_mel: 17.97258514081928 (+0.16595632257595838) | > avg_loss_duration: 1.6567185160139914 (-0.0019159501707053384) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.615647248818842 (+0.1995252689845124) | > avg_grad_norm_1: tensor(247.1932, device='cuda:0') (+tensor(19.1652, device='cuda:0'))  > EPOCH: 107/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:17:11)   --> TIME: 2025-02-25 09:17:33 -- STEP: 45/72 -- GLOBAL_STEP: 1007750 | > loss_disc: 2.8413710594177246 (2.8494567447238492) | > loss_disc_real_0: 0.12762093544006348 (0.23481813172499338) | > loss_disc_real_1: 0.23570066690444946 (0.23316698670387268) | > loss_disc_real_2: 0.24592141807079315 (0.24081867867045934) | > loss_disc_real_3: 0.26255080103874207 (0.24302663405736288) | > loss_disc_real_4: 0.2657429277896881 (0.24574284685982598) | > loss_disc_real_5: 0.28803807497024536 (0.24730823006894853) | > loss_0: 2.8413710594177246 (2.8494567447238492) | > grad_norm_0: tensor(14.5392, device='cuda:0') (tensor(9.7459, device='cuda:0')) | > loss_gen: 1.7175562381744385 (1.7432745218276977) | > loss_kl: 2.5241432189941406 (2.4205391830868193) | > loss_feat: 1.721670150756836 (1.8707411024305556) | > loss_mel: 17.517959594726562 (18.09235280354818) | > loss_duration: 1.685192346572876 (1.6419787247975668) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.166521072387695 (25.768886184692384) | > grad_norm_1: tensor(101.1369, device='cuda:0') (tensor(205.1009, device='cuda:0')) | > current_lr_0: 0.00019734264459313146 | > current_lr_1: 0.00019734264459313146 | > step_time: 0.4958 (0.4423593150244819) | > loader_time: 0.0091 (0.007435168160332573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534508691707127 (-7.359074874663252e-05) | > avg_step_time: 0.4549156276273056 (-0.0002946752897450833) | > avg_loss_disc: 2.844775854701727 (+0.012715275858489417) | > avg_loss_disc_real_0: 0.2346844922908595 (+0.009016583503132142) | > avg_loss_disc_real_1: 0.23574947345424707 (+0.0029723705120489885) | > avg_loss_disc_real_2: 0.24084806715098905 (+0.003420270244840179) | > avg_loss_disc_real_3: 0.24280834575773966 (+0.003876872675519616) | > avg_loss_disc_real_4: 0.2452333577921693 (+0.001247449869840933) | > avg_loss_disc_real_5: 0.24671437748720948 (+0.0006661077200526833) | > avg_loss_0: 2.844775854701727 (+0.012715275858489417) | > avg_grad_norm_0: tensor(10.1120, device='cuda:0') (tensor(-1.5602, device='cuda:0')) | > avg_loss_gen: 1.7496709991508805 (+0.012756433285457947) | > avg_loss_kl: 2.426007264096972 (+0.03959264889569303) | > avg_loss_feat: 1.8474668902410587 (-0.01554765835614269) | > avg_loss_mel: 18.083644759487104 (+0.11105961866782366) | > avg_loss_duration: 1.6576116874184408 (+0.0008931714044493333) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76440136869189 (+0.14875411987304688) | > avg_grad_norm_1: tensor(218.6526, device='cuda:0') (tensor(-28.5406, device='cuda:0'))  > EPOCH: 108/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:17:48)   --> TIME: 2025-02-25 09:18:00 -- STEP: 23/72 -- GLOBAL_STEP: 1007800 | > loss_disc: 2.8159573078155518 (2.849819971167523) | > loss_disc_real_0: 0.11489056050777435 (0.21903846121352652) | > loss_disc_real_1: 0.2503643035888672 (0.22743596071782318) | > loss_disc_real_2: 0.21655771136283875 (0.23622417190800543) | > loss_disc_real_3: 0.20800042152404785 (0.24026077856188235) | > loss_disc_real_4: 0.23388463258743286 (0.2483350641053656) | > loss_disc_real_5: 0.22466182708740234 (0.2515014235092246) | > loss_0: 2.8159573078155518 (2.849819971167523) | > grad_norm_0: tensor(17.8659, device='cuda:0') (tensor(9.7046, device='cuda:0')) | > loss_gen: 1.544012427330017 (1.735317209492559) | > loss_kl: 2.370511770248413 (2.4380961293759555) | > loss_feat: 1.957208514213562 (1.8752455504044243) | > loss_mel: 17.714975357055664 (18.29243742901346) | > loss_duration: 1.6613378524780273 (1.6410664682802947) | > amp_scaler: 512.0 (422.95652173913044) | > loss_1: 25.248046875 (25.98216264144234) | > grad_norm_1: tensor(66.0955, device='cuda:0') (tensor(244.5417, device='cuda:0')) | > current_lr_0: 0.0001973179767625573 | > current_lr_1: 0.0001973179767625573 | > step_time: 0.4411 (0.4338551085928212) | > loader_time: 0.0073 (0.006532793459684953) --> EVAL PERFORMANCE | > avg_loader_time: 0.00848100890576 (-5.349978594712708e-05) | > avg_step_time: 0.4532460226139552 (-0.0016696050133503948) | > avg_loss_disc: 2.85054561118005 (+0.005769756478323007) | > avg_loss_disc_real_0: 0.22634762168770106 (-0.008336870603158436) | > avg_loss_disc_real_1: 0.23503579765977994 (-0.0007136757944671279) | > avg_loss_disc_real_2: 0.24185521229052206 (+0.001007145139533011) | > avg_loss_disc_real_3: 0.2433410492581381 (+0.0005327035003984404) | > avg_loss_disc_real_4: 0.24593995343631422 (+0.0007065956441449361) | > avg_loss_disc_real_5: 0.2473746957913251 (+0.0006603183041156147) | > avg_loss_0: 2.85054561118005 (+0.005769756478323007) | > avg_grad_norm_0: tensor(10.4674, device='cuda:0') (+tensor(0.3554, device='cuda:0')) | > avg_loss_gen: 1.7554032718631583 (+0.0057322727122777195) | > avg_loss_kl: 2.432891748320889 (+0.0068844842239168536) | > avg_loss_feat: 1.886845523202923 (+0.03937863296186439) | > avg_loss_mel: 18.15611346338837 (+0.07246870390126503) | > avg_loss_duration: 1.6602891482098001 (+0.002677460791359332) | > avg_amp_scaler: 335.32394366197184 (+79.32394366197184) | > avg_loss_1: 25.891543106294016 (+0.12714173760212688) | > avg_grad_norm_1: tensor(225.0956, device='cuda:0') (+tensor(6.4431, device='cuda:0'))  > EPOCH: 109/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:18:24)   --> TIME: 2025-02-25 09:18:27 -- STEP: 1/72 -- GLOBAL_STEP: 1007850 | > loss_disc: 2.9343760013580322 (2.9343760013580322) | > loss_disc_real_0: 0.2629437744617462 (0.2629437744617462) | > loss_disc_real_1: 0.3002200722694397 (0.3002200722694397) | > loss_disc_real_2: 0.21628569066524506 (0.21628569066524506) | > loss_disc_real_3: 0.21876762807369232 (0.21876762807369232) | > loss_disc_real_4: 0.22328251600265503 (0.22328251600265503) | > loss_disc_real_5: 0.22973254323005676 (0.22973254323005676) | > loss_0: 2.9343760013580322 (2.9343760013580322) | > grad_norm_0: tensor(7.6624, device='cuda:0') (tensor(7.6624, device='cuda:0')) | > loss_gen: 1.5716511011123657 (1.5716511011123657) | > loss_kl: 2.4250073432922363 (2.4250073432922363) | > loss_feat: 1.8564859628677368 (1.8564859628677368) | > loss_mel: 18.498414993286133 (18.498414993286133) | > loss_duration: 1.6567213535308838 (1.6567213535308838) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.00827980041504 (26.00827980041504) | > grad_norm_1: tensor(329.1075, device='cuda:0') (tensor(329.1075, device='cuda:0')) | > current_lr_0: 0.00019729331201546197 | > current_lr_1: 0.00019729331201546197 | > step_time: 0.4156 (0.41564393043518066) | > loader_time: 0.0047 (0.004715442657470703)  --> TIME: 2025-02-25 09:18:50 -- STEP: 51/72 -- GLOBAL_STEP: 1007900 | > loss_disc: 2.860442638397217 (2.8445000414754826) | > loss_disc_real_0: 0.13472649455070496 (0.23257673166546167) | > loss_disc_real_1: 0.290700763463974 (0.23463445112985723) | > loss_disc_real_2: 0.19597584009170532 (0.23984075586001077) | > loss_disc_real_3: 0.22618569433689117 (0.24189897465939617) | > loss_disc_real_4: 0.22608302533626556 (0.24273434018387513) | > loss_disc_real_5: 0.2311989665031433 (0.2454964796117708) | > loss_0: 2.860442638397217 (2.8445000414754826) | > grad_norm_0: tensor(16.4867, device='cuda:0') (tensor(11.0125, device='cuda:0')) | > loss_gen: 1.6950701475143433 (1.7433481099558812) | > loss_kl: 2.269840717315674 (2.4600358850815724) | > loss_feat: 1.9558706283569336 (1.8714287327785117) | > loss_mel: 18.291481018066406 (17.95526799968645) | > loss_duration: 1.7086482048034668 (1.6540989852419086) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.92091178894043 (25.68417956782322) | > grad_norm_1: tensor(233.2939, device='cuda:0') (tensor(242.6102, device='cuda:0')) | > current_lr_0: 0.00019729331201546197 | > current_lr_1: 0.00019729331201546197 | > step_time: 0.4501 (0.45122630923402074) | > loader_time: 0.0088 (0.007701579262228573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611843619548093 (+0.00013083471378809268) | > avg_step_time: 0.461457759561673 (+0.008211736947717818) | > avg_loss_disc: 2.853520319495402 (+0.0029747083153521814) | > avg_loss_disc_real_0: 0.23144076869521343 (+0.005093147007512372) | > avg_loss_disc_real_1: 0.2353316218080655 (+0.0002958241482855506) | > avg_loss_disc_real_2: 0.24016726289836454 (-0.0016879493921575228) | > avg_loss_disc_real_3: 0.2413092765589835 (-0.0020317726991546103) | > avg_loss_disc_real_4: 0.2433583872838759 (-0.0025815661524383193) | > avg_loss_disc_real_5: 0.2463965012993611 (-0.0009781944919639851) | > avg_loss_0: 2.853520319495402 (+0.0029747083153521814) | > avg_grad_norm_0: tensor(10.4985, device='cuda:0') (+tensor(0.0311, device='cuda:0')) | > avg_loss_gen: 1.7329934079882126 (-0.0224098638749457) | > avg_loss_kl: 2.4384935674533046 (+0.005601819132415642) | > avg_loss_feat: 1.8436361652025035 (-0.043209358000419584) | > avg_loss_mel: 17.883805718220454 (-0.27230774516791456) | > avg_loss_duration: 1.6647333779805142 (+0.004444229770714125) | > avg_amp_scaler: 256.0 (-79.32394366197184) | > avg_loss_1: 25.563662166326818 (-0.3278809399671978) | > avg_grad_norm_1: tensor(237.6201, device='cuda:0') (+tensor(12.5244, device='cuda:0'))  > EPOCH: 110/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:19:02)   --> TIME: 2025-02-25 09:19:17 -- STEP: 29/72 -- GLOBAL_STEP: 1007950 | > loss_disc: 2.768181800842285 (2.826312056903182) | > loss_disc_real_0: 0.2208687961101532 (0.22872266327512675) | > loss_disc_real_1: 0.24305333197116852 (0.23372577541861042) | > loss_disc_real_2: 0.2856875956058502 (0.24451476592442084) | > loss_disc_real_3: 0.256611168384552 (0.24410057119254408) | > loss_disc_real_4: 0.24351386725902557 (0.2451074349469152) | > loss_disc_real_5: 0.2262411266565323 (0.24910007417201996) | > loss_0: 2.768181800842285 (2.826312056903182) | > grad_norm_0: tensor(3.8209, device='cuda:0') (tensor(9.3898, device='cuda:0')) | > loss_gen: 1.6706459522247314 (1.7690532577448879) | > loss_kl: 2.566333293914795 (2.568418182175735) | > loss_feat: 1.8034948110580444 (1.9290122451453373) | > loss_mel: 18.421382904052734 (18.474558468522705) | > loss_duration: 1.648412823677063 (1.6371448779928273) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.11026954650879 (26.37818685071222) | > grad_norm_1: tensor(72.4000, device='cuda:0') (tensor(209.7753, device='cuda:0')) | > current_lr_0: 0.00019726865035146003 | > current_lr_1: 0.00019726865035146003 | > step_time: 0.44 (0.43775509965830833) | > loader_time: 0.0078 (0.006688381063527074) --> EVAL PERFORMANCE | > avg_loader_time: 0.008478144524802625 (-0.0001336990947454677) | > avg_step_time: 0.4561233419767567 (-0.005334417584916318) | > avg_loss_disc: 2.83224136728636 (-0.021278952209042412) | > avg_loss_disc_real_0: 0.22587328771470297 (-0.005567480980510464) | > avg_loss_disc_real_1: 0.2360154392433838 (+0.0006838174353183124) | > avg_loss_disc_real_2: 0.23915932799728823 (-0.0010079349010763128) | > avg_loss_disc_real_3: 0.24128983306213164 (-1.944349685184532e-05) | > avg_loss_disc_real_4: 0.24624605812657047 (+0.002887670842694562) | > avg_loss_disc_real_5: 0.24775645451646455 (+0.0013599532171034434) | > avg_loss_0: 2.83224136728636 (-0.021278952209042412) | > avg_grad_norm_0: tensor(9.8001, device='cuda:0') (tensor(-0.6984, device='cuda:0')) | > avg_loss_gen: 1.7514155078941667 (+0.018422099905954115) | > avg_loss_kl: 2.48492732182355 (+0.04643375437024533) | > avg_loss_feat: 1.853724887673284 (+0.010088722470780409) | > avg_loss_mel: 17.936225810521087 (+0.052420092300632604) | > avg_loss_duration: 1.6574624457829434 (-0.007270932197570801) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.683755767177527 (+0.12009360085070853) | > avg_grad_norm_1: tensor(253.0788, device='cuda:0') (+tensor(15.4587, device='cuda:0'))  > EPOCH: 111/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:19:39)   --> TIME: 2025-02-25 09:19:44 -- STEP: 7/72 -- GLOBAL_STEP: 1008000 | > loss_disc: 2.7599377632141113 (2.8145037037985667) | > loss_disc_real_0: 0.2737751305103302 (0.2267725850854601) | > loss_disc_real_1: 0.23086251318454742 (0.2370507653270449) | > loss_disc_real_2: 0.2784525752067566 (0.2373275033065251) | > loss_disc_real_3: 0.21464698016643524 (0.23563880154064723) | > loss_disc_real_4: 0.23824641108512878 (0.235230558684894) | > loss_disc_real_5: 0.22189265489578247 (0.23892885872295924) | > loss_0: 2.7599377632141113 (2.8145037037985667) | > grad_norm_0: tensor(8.5473, device='cuda:0') (tensor(8.3631, device='cuda:0')) | > loss_gen: 1.8438777923583984 (1.7799374886921473) | > loss_kl: 2.2471580505371094 (2.329643590109689) | > loss_feat: 2.3345141410827637 (2.005366563796997) | > loss_mel: 18.588909149169922 (18.056053706577845) | > loss_duration: 1.6261787414550781 (1.6374057701655798) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.64063835144043 (25.808407374790736) | > grad_norm_1: tensor(185.7324, device='cuda:0') (tensor(244.1194, device='cuda:0')) | > current_lr_0: 0.0001972439917701661 | > current_lr_1: 0.0001972439917701661 | > step_time: 0.4309 (0.4332024710518973) | > loader_time: 0.0055 (0.00607405390058245)  --> TIME: 2025-02-25 09:20:07 -- STEP: 57/72 -- GLOBAL_STEP: 1008050 | > loss_disc: 2.7762787342071533 (2.845695027133875) | > loss_disc_real_0: 0.1834718883037567 (0.22969752079562136) | > loss_disc_real_1: 0.26440584659576416 (0.23637585420357554) | > loss_disc_real_2: 0.23313000798225403 (0.24171577524720578) | > loss_disc_real_3: 0.23119235038757324 (0.2427886361092852) | > loss_disc_real_4: 0.24208411574363708 (0.24561883666013418) | > loss_disc_real_5: 0.23627975583076477 (0.24621669425253281) | > loss_0: 2.7762787342071533 (2.845695027133875) | > grad_norm_0: tensor(7.3399, device='cuda:0') (tensor(9.1556, device='cuda:0')) | > loss_gen: 1.6752504110336304 (1.752515345288996) | > loss_kl: 2.5861504077911377 (2.4282026332721376) | > loss_feat: 2.0930027961730957 (1.886854939293443) | > loss_mel: 18.337343215942383 (17.951652426468694) | > loss_duration: 1.6951284408569336 (1.647788919900593) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.38687515258789 (25.667014239127177) | > grad_norm_1: tensor(56.5068, device='cuda:0') (tensor(211.1691, device='cuda:0')) | > current_lr_0: 0.0001972439917701661 | > current_lr_1: 0.0001972439917701661 | > step_time: 0.4712 (0.44389316073635166) | > loader_time: 0.0107 (0.007927593431974711) --> EVAL PERFORMANCE | > avg_loader_time: 0.008569079385676854 (+9.093486087422865e-05) | > avg_step_time: 0.45336197127758615 (-0.00276137069917054) | > avg_loss_disc: 2.8410918611875724 (+0.00885049390121262) | > avg_loss_disc_real_0: 0.22637461923377614 (+0.0005013315190731671) | > avg_loss_disc_real_1: 0.2360687919065986 (+5.335266321479515e-05) | > avg_loss_disc_real_2: 0.24099671462891806 (+0.0018373866316298282) | > avg_loss_disc_real_3: 0.24196230462739166 (+0.0006724715652600133) | > avg_loss_disc_real_4: 0.244798535402392 (-0.0014475227241784738) | > avg_loss_disc_real_5: 0.2459044955985647 (-0.0018519589178998541) | > avg_loss_0: 2.8410918611875724 (+0.00885049390121262) | > avg_grad_norm_0: tensor(9.2828, device='cuda:0') (tensor(-0.5173, device='cuda:0')) | > avg_loss_gen: 1.7511217594146729 (-0.0002937484794938161) | > avg_loss_kl: 2.4469717724222533 (-0.03795554940129664) | > avg_loss_feat: 1.889943540935785 (+0.036218653262501066) | > avg_loss_mel: 18.026469861957384 (+0.09024405143629721) | > avg_loss_duration: 1.6552323573072192 (-0.0022300884757242567) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76973923159317 (+0.0859834644156443) | > avg_grad_norm_1: tensor(211.6181, device='cuda:0') (tensor(-41.4607, device='cuda:0'))  > EPOCH: 112/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:20:15)   --> TIME: 2025-02-25 09:20:34 -- STEP: 35/72 -- GLOBAL_STEP: 1008100 | > loss_disc: 2.8356149196624756 (2.8475781645093647) | > loss_disc_real_0: 0.2440621703863144 (0.2438704954726356) | > loss_disc_real_1: 0.2944531738758087 (0.23064357936382293) | > loss_disc_real_2: 0.24804671108722687 (0.23921925510678974) | > loss_disc_real_3: 0.21548964083194733 (0.2403488895722798) | > loss_disc_real_4: 0.229297935962677 (0.24815717126641954) | > loss_disc_real_5: 0.2323397547006607 (0.24794057139328549) | > loss_0: 2.8356149196624756 (2.8475781645093647) | > grad_norm_0: tensor(6.4713, device='cuda:0') (tensor(11.9515, device='cuda:0')) | > loss_gen: 1.663158655166626 (1.744713384764535) | > loss_kl: 2.5191903114318848 (2.3838492461613248) | > loss_feat: 1.848059892654419 (1.87889575277056) | > loss_mel: 19.169713973999023 (18.165097372872488) | > loss_duration: 1.6528213024139404 (1.6294525691441126) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.852943420410156 (25.8020081111363) | > grad_norm_1: tensor(168.7471, device='cuda:0') (tensor(237.4523, device='cuda:0')) | > current_lr_0: 0.0001972193362711948 | > current_lr_1: 0.0001972193362711948 | > step_time: 0.4391 (0.4505204541342599) | > loader_time: 0.0079 (0.007203033992222377) --> EVAL PERFORMANCE | > avg_loader_time: 0.008553196007097268 (-1.5883378579585605e-05) | > avg_step_time: 0.46180758341937 (+0.008445612141783854) | > avg_loss_disc: 2.836773476130525 (-0.0043183850570471805) | > avg_loss_disc_real_0: 0.2368370042720311 (+0.010462385038254973) | > avg_loss_disc_real_1: 0.2366616085801326 (+0.0005928166735340024) | > avg_loss_disc_real_2: 0.2377729657250391 (-0.00322374890387897) | > avg_loss_disc_real_3: 0.2414066189191711 (-0.0005556857082205657) | > avg_loss_disc_real_4: 0.2443257899771274 (-0.00047274542526459795) | > avg_loss_disc_real_5: 0.2446818039031096 (-0.0012226916954551081) | > avg_loss_0: 2.836773476130525 (-0.0043183850570471805) | > avg_grad_norm_0: tensor(11.5204, device='cuda:0') (+tensor(2.2376, device='cuda:0')) | > avg_loss_gen: 1.751807562062438 (+0.000685802647765188) | > avg_loss_kl: 2.3756365037300204 (-0.07133526869223283) | > avg_loss_feat: 1.871145145993837 (-0.018798394941947905) | > avg_loss_mel: 18.008981865896306 (-0.017487996061078093) | > avg_loss_duration: 1.6524010846312618 (-0.002831272675957397) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.659972177424894 (-0.1097670541682767) | > avg_grad_norm_1: tensor(223.8470, device='cuda:0') (+tensor(12.2289, device='cuda:0'))  > EPOCH: 113/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:20:53)   --> TIME: 2025-02-25 09:21:00 -- STEP: 13/72 -- GLOBAL_STEP: 1008150 | > loss_disc: 2.8656907081604004 (2.8588195397303653) | > loss_disc_real_0: 0.3362360894680023 (0.23505998632082573) | > loss_disc_real_1: 0.25221529603004456 (0.2288002371788025) | > loss_disc_real_2: 0.20831698179244995 (0.2330398937830558) | > loss_disc_real_3: 0.22395534813404083 (0.2374524210508053) | > loss_disc_real_4: 0.2422725260257721 (0.24785451934887812) | > loss_disc_real_5: 0.2245326191186905 (0.25007906212256503) | > loss_0: 2.8656907081604004 (2.8588195397303653) | > grad_norm_0: tensor(17.0258, device='cuda:0') (tensor(16.4770, device='cuda:0')) | > loss_gen: 1.8750324249267578 (1.7404053578009973) | > loss_kl: 2.640188694000244 (2.363160280080942) | > loss_feat: 1.9006191492080688 (1.8612838983535767) | > loss_mel: 16.355329513549805 (17.8304936335637) | > loss_duration: 1.556899070739746 (1.6300490636091967) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.328067779541016 (25.425392150878906) | > grad_norm_1: tensor(260.1764, device='cuda:0') (tensor(244.0704, device='cuda:0')) | > current_lr_0: 0.0001971946838541609 | > current_lr_1: 0.0001971946838541609 | > step_time: 0.4571 (0.43451578800494856) | > loader_time: 0.0058 (0.0068054932814378)  --> TIME: 2025-02-25 09:21:24 -- STEP: 63/72 -- GLOBAL_STEP: 1008200 | > loss_disc: 2.8482818603515625 (2.8442244718945218) | > loss_disc_real_0: 0.23444345593452454 (0.23491960905847095) | > loss_disc_real_1: 0.25713810324668884 (0.23603737094099558) | > loss_disc_real_2: 0.26617899537086487 (0.23743958747576152) | > loss_disc_real_3: 0.2514830231666565 (0.2418707606811372) | > loss_disc_real_4: 0.2621331810951233 (0.24466810221709903) | > loss_disc_real_5: 0.26985669136047363 (0.24679783388735757) | > loss_0: 2.8482818603515625 (2.8442244718945218) | > grad_norm_0: tensor(4.2566, device='cuda:0') (tensor(11.4963, device='cuda:0')) | > loss_gen: 1.7138876914978027 (1.758885902071756) | > loss_kl: 2.3827364444732666 (2.412426993960426) | > loss_feat: 1.4119220972061157 (1.9055185696435353) | > loss_mel: 17.538307189941406 (17.830531589568608) | > loss_duration: 1.6412920951843262 (1.6463252097841292) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.688146591186523 (25.553688291519407) | > grad_norm_1: tensor(59.3488, device='cuda:0') (tensor(257.4180, device='cuda:0')) | > current_lr_0: 0.0001971946838541609 | > current_lr_1: 0.0001971946838541609 | > step_time: 0.5306 (0.4482163663894411) | > loader_time: 0.0113 (0.008259273710704987) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863645781933422 (+8.326181223695245e-05) | > avg_step_time: 0.455153334308678 (-0.006654249110692012) | > avg_loss_disc: 2.8457514601694025 (+0.008977984038877285) | > avg_loss_disc_real_0: 0.23207997016503779 (-0.004757034106993324) | > avg_loss_disc_real_1: 0.23616317250359226 (-0.0004984360765403484) | > avg_loss_disc_real_2: 0.23796649235235134 (+0.00019352662731225712) | > avg_loss_disc_real_3: 0.2412748332594482 (-0.0001317856597228828) | > avg_loss_disc_real_4: 0.24458462629519717 (+0.00025883631806977725) | > avg_loss_disc_real_5: 0.24661170198044308 (+0.0019298980773334862) | > avg_loss_0: 2.8457514601694025 (+0.008977984038877285) | > avg_grad_norm_0: tensor(11.2540, device='cuda:0') (tensor(-0.2664, device='cuda:0')) | > avg_loss_gen: 1.755697895103777 (+0.0038903330413389003) | > avg_loss_kl: 2.4216121720596093 (+0.04597566832958888) | > avg_loss_feat: 1.8945605385471398 (+0.023415392553302716) | > avg_loss_mel: 17.853978009291097 (-0.15500385660520877) | > avg_loss_duration: 1.6511857308132547 (-0.001215353818007081) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.577034318950815 (-0.08293785847407875) | > avg_grad_norm_1: tensor(264.2580, device='cuda:0') (+tensor(40.4110, device='cuda:0'))  > EPOCH: 114/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:21:30)   --> TIME: 2025-02-25 09:21:50 -- STEP: 41/72 -- GLOBAL_STEP: 1008250 | > loss_disc: 2.841259479522705 (2.849920644992736) | > loss_disc_real_0: 0.1319190412759781 (0.22731279745334532) | > loss_disc_real_1: 0.21139760315418243 (0.24077644835158094) | > loss_disc_real_2: 0.26856130361557007 (0.24086391126237264) | > loss_disc_real_3: 0.23457330465316772 (0.24424696768202434) | > loss_disc_real_4: 0.22225895524024963 (0.2468769063309925) | > loss_disc_real_5: 0.21643173694610596 (0.24976333212561724) | > loss_0: 2.841259479522705 (2.849920644992736) | > grad_norm_0: tensor(12.9751, device='cuda:0') (tensor(8.4179, device='cuda:0')) | > loss_gen: 1.7291232347488403 (1.7611194500109044) | > loss_kl: 2.3008480072021484 (2.418987593999723) | > loss_feat: 2.0026485919952393 (1.8901751332166719) | > loss_mel: 18.743955612182617 (18.02838044050263) | > loss_duration: 1.6530710458755493 (1.641310741261738) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.429645538330078 (25.739973207799398) | > grad_norm_1: tensor(281.1378, device='cuda:0') (tensor(228.2634, device='cuda:0')) | > current_lr_0: 0.0001971700345186791 | > current_lr_1: 0.0001971700345186791 | > step_time: 0.4432 (0.43552856910519483) | > loader_time: 0.0094 (0.007273860094023914) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852830645064233 (-0.00010815136869189029) | > avg_step_time: 0.45111009772394745 (-0.004043236584730547) | > avg_loss_disc: 2.8528249532404084 (+0.00707349307100591) | > avg_loss_disc_real_0: 0.23097070820734533 (-0.0011092619576924534) | > avg_loss_disc_real_1: 0.23731685618699436 (+0.0011536836834021014) | > avg_loss_disc_real_2: 0.24005949203397187 (+0.002092999681620522) | > avg_loss_disc_real_3: 0.2417385886672517 (+0.00046375540780349556) | > avg_loss_disc_real_4: 0.2451618834280632 (+0.0005772571328660214) | > avg_loss_disc_real_5: 0.24751487122455113 (+0.0009031692441080486) | > avg_loss_0: 2.8528249532404084 (+0.00707349307100591) | > avg_grad_norm_0: tensor(9.8374, device='cuda:0') (tensor(-1.4167, device='cuda:0')) | > avg_loss_gen: 1.7474162595372804 (-0.008281635566496526) | > avg_loss_kl: 2.3741662938829875 (-0.047445878176621825) | > avg_loss_feat: 1.8598317159733302 (-0.0347288225738096) | > avg_loss_mel: 18.00067430146983 (+0.14669629217873137) | > avg_loss_duration: 1.6561956170579077 (+0.005009886244653039) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.638284038489974 (+0.06124971953915903) | > avg_grad_norm_1: tensor(275.3025, device='cuda:0') (+tensor(11.0445, device='cuda:0'))  > EPOCH: 115/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:22:06)   --> TIME: 2025-02-25 09:22:16 -- STEP: 19/72 -- GLOBAL_STEP: 1008300 | > loss_disc: 2.754560947418213 (2.8582475059910823) | > loss_disc_real_0: 0.22257420420646667 (0.21513276507979945) | > loss_disc_real_1: 0.21435615420341492 (0.24457337197504544) | > loss_disc_real_2: 0.1729259490966797 (0.25001145036597) | > loss_disc_real_3: 0.18955859541893005 (0.24219105745616712) | > loss_disc_real_4: 0.19606377184391022 (0.2531610539084987) | > loss_disc_real_5: 0.21630282700061798 (0.25417085933057887) | > loss_0: 2.754560947418213 (2.8582475059910823) | > grad_norm_0: tensor(10.6270, device='cuda:0') (tensor(10.4612, device='cuda:0')) | > loss_gen: 1.7089124917984009 (1.7556414478703548) | > loss_kl: 2.4712159633636475 (2.4230987649214897) | > loss_feat: 2.1064951419830322 (1.8676715022639225) | > loss_mel: 18.084848403930664 (18.082428279675934) | > loss_duration: 1.6007236242294312 (1.6315484046936035) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.97219467163086 (25.760388424522) | > grad_norm_1: tensor(109.4730, device='cuda:0') (tensor(288.1319, device='cuda:0')) | > current_lr_0: 0.00019714538826436426 | > current_lr_1: 0.00019714538826436426 | > step_time: 0.4277 (0.4323293786299856) | > loader_time: 0.0087 (0.006720555456061112)  --> TIME: 2025-02-25 09:22:41 -- STEP: 69/72 -- GLOBAL_STEP: 1008350 | > loss_disc: 2.981799840927124 (2.855333169301351) | > loss_disc_real_0: 0.21996895968914032 (0.22305147965317187) | > loss_disc_real_1: 0.26769956946372986 (0.2384771166936211) | > loss_disc_real_2: 0.2264125794172287 (0.24060446088728699) | > loss_disc_real_3: 0.24062439799308777 (0.23872732381889786) | > loss_disc_real_4: 0.23894385993480682 (0.24589966453503873) | > loss_disc_real_5: 0.2237127125263214 (0.24671128759349603) | > loss_0: 2.981799840927124 (2.855333169301351) | > grad_norm_0: tensor(6.3152, device='cuda:0') (tensor(10.1373, device='cuda:0')) | > loss_gen: 1.7735099792480469 (1.7368394153705542) | > loss_kl: 2.423961639404297 (2.4198045592377158) | > loss_feat: 1.8128573894500732 (1.8498570884483447) | > loss_mel: 17.638233184814453 (17.96442331784013) | > loss_duration: 1.7362972497940063 (1.6552902028180552) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.38486099243164 (25.626214428224426) | > grad_norm_1: tensor(370.8735, device='cuda:0') (tensor(236.2611, device='cuda:0')) | > current_lr_0: 0.00019714538826436426 | > current_lr_1: 0.00019714538826436426 | > step_time: 0.5449 (0.46130937769793084) | > loader_time: 0.0127 (0.008624398190042248) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680007827114049 (+0.000151701376471719) | > avg_step_time: 0.4626445837423835 (+0.011534486018436041) | > avg_loss_disc: 2.855088724216945 (+0.0022637709765365166) | > avg_loss_disc_real_0: 0.2249699410208514 (-0.006000767186493922) | > avg_loss_disc_real_1: 0.2376517607292659 (+0.0003349045422715302) | > avg_loss_disc_real_2: 0.24141322034345547 (+0.001353728309483604) | > avg_loss_disc_real_3: 0.2391456919656673 (-0.002592896701584413) | > avg_loss_disc_real_4: 0.24629530163717941 (+0.0011334182091162215) | > avg_loss_disc_real_5: 0.2467525926694064 (-0.0007622785551447131) | > avg_loss_0: 2.855088724216945 (+0.0022637709765365166) | > avg_grad_norm_0: tensor(10.2623, device='cuda:0') (+tensor(0.4250, device='cuda:0')) | > avg_loss_gen: 1.738913655281067 (-0.00850260425621352) | > avg_loss_kl: 2.422271748663675 (+0.048105454780687396) | > avg_loss_feat: 1.8529737482608204 (-0.006857967712509794) | > avg_loss_mel: 17.966098154094855 (-0.03457614737497394) | > avg_loss_duration: 1.6553005735639115 (-0.0008950434939962726) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.635557685099858 (-0.0027263533901162873) | > avg_grad_norm_1: tensor(239.5593, device='cuda:0') (tensor(-35.7432, device='cuda:0'))  > EPOCH: 116/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:22:44)   --> TIME: 2025-02-25 09:23:07 -- STEP: 47/72 -- GLOBAL_STEP: 1008400 | > loss_disc: 2.8215041160583496 (2.8532977763642657) | > loss_disc_real_0: 0.25183677673339844 (0.23375887296935347) | > loss_disc_real_1: 0.24361024796962738 (0.23573335276005117) | > loss_disc_real_2: 0.2294919341802597 (0.24205463997861174) | > loss_disc_real_3: 0.23350399732589722 (0.24201425053971878) | > loss_disc_real_4: 0.23600399494171143 (0.24631227806527564) | > loss_disc_real_5: 0.24259136617183685 (0.24966364369747487) | > loss_0: 2.8215041160583496 (2.8532977763642657) | > grad_norm_0: tensor(8.0469, device='cuda:0') (tensor(11.7176, device='cuda:0')) | > loss_gen: 1.7453616857528687 (1.7424163590086268) | > loss_kl: 2.489428758621216 (2.394781112670899) | > loss_feat: 1.6858950853347778 (1.82342469945867) | > loss_mel: 17.570348739624023 (18.066239458449342) | > loss_duration: 1.674207091331482 (1.6388860661932763) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.165241241455078 (25.66574770338992) | > grad_norm_1: tensor(134.3820, device='cuda:0') (tensor(224.4611, device='cuda:0')) | > current_lr_0: 0.0001971207450908312 | > current_lr_1: 0.0001971207450908312 | > step_time: 0.4496 (0.4386223833611671) | > loader_time: 0.0088 (0.007549514161779526) --> EVAL PERFORMANCE | > avg_loader_time: 0.008541681397129113 (-0.00013832642998493593) | > avg_step_time: 0.4524253556426142 (-0.010219228099769273) | > avg_loss_disc: 2.848419273403329 (-0.006669450813615807) | > avg_loss_disc_real_0: 0.23384467659282013 (+0.008874735571968717) | > avg_loss_disc_real_1: 0.2374693018869615 (-0.00018245884230438936) | > avg_loss_disc_real_2: 0.242315766047424 (+0.000902545703968527) | > avg_loss_disc_real_3: 0.24288230777626307 (+0.0037366158105957836) | > avg_loss_disc_real_4: 0.2463759337092789 (+8.063207209949019e-05) | > avg_loss_disc_real_5: 0.24930991022519663 (+0.002557317555790223) | > avg_loss_0: 2.848419273403329 (-0.006669450813615807) | > avg_grad_norm_0: tensor(11.3420, device='cuda:0') (+tensor(1.0796, device='cuda:0')) | > avg_loss_gen: 1.7558787742131192 (+0.016965118932052325) | > avg_loss_kl: 2.423644149807138 (+0.0013724011434632644) | > avg_loss_feat: 1.8580498426732883 (+0.005076094412467924) | > avg_loss_mel: 18.250557026392976 (+0.2844588722981207) | > avg_loss_duration: 1.6583979264111586 (+0.0030973528472471035) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.94652761539943 (+0.31096993029957076) | > avg_grad_norm_1: tensor(250.1299, device='cuda:0') (+tensor(10.5706, device='cuda:0'))  > EPOCH: 117/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:23:20)   --> TIME: 2025-02-25 09:23:33 -- STEP: 25/72 -- GLOBAL_STEP: 1008450 | > loss_disc: 2.821437120437622 (2.8368984031677247) | > loss_disc_real_0: 0.1989038586616516 (0.22317952066659927) | > loss_disc_real_1: 0.2357052117586136 (0.23011987924575805) | > loss_disc_real_2: 0.23919562995433807 (0.23893481850624085) | > loss_disc_real_3: 0.2511695623397827 (0.23868126332759856) | > loss_disc_real_4: 0.25425878167152405 (0.2445218765735626) | > loss_disc_real_5: 0.24680757522583008 (0.24983967483043668) | > loss_0: 2.821437120437622 (2.8368984031677247) | > grad_norm_0: tensor(3.4851, device='cuda:0') (tensor(9.2777, device='cuda:0')) | > loss_gen: 1.7281603813171387 (1.7387859916687012) | > loss_kl: 2.4487991333007812 (2.3268232297897335) | > loss_feat: 1.9109336137771606 (1.8551423788070678) | > loss_mel: 16.471696853637695 (18.02052696228027) | > loss_duration: 1.61146080493927 (1.6257069540023803) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.171051025390625 (25.56698539733887) | > grad_norm_1: tensor(194.9069, device='cuda:0') (tensor(283.4067, device='cuda:0')) | > current_lr_0: 0.00019709610499769482 | > current_lr_1: 0.00019709610499769482 | > step_time: 0.4279 (0.42900221824645995) | > loader_time: 0.0071 (0.006830778121948242) --> EVAL PERFORMANCE | > avg_loader_time: 0.00858163162016533 (+3.9950223036216634e-05) | > avg_step_time: 0.4499325248557077 (-0.002492830786906486) | > avg_loss_disc: 2.8437539456595835 (-0.0046653277437456175) | > avg_loss_disc_real_0: 0.2256449359193654 (-0.00819974067345472) | > avg_loss_disc_real_1: 0.23472179703309504 (-0.0027475048538664615) | > avg_loss_disc_real_2: 0.2401134021265406 (-0.00220236392088341) | > avg_loss_disc_real_3: 0.24052443722604025 (-0.0023578705502228237) | > avg_loss_disc_real_4: 0.24523925172611022 (-0.0011366819831686858) | > avg_loss_disc_real_5: 0.24610626172851507 (-0.0032036484966815604) | > avg_loss_0: 2.8437539456595835 (-0.0046653277437456175) | > avg_grad_norm_0: tensor(9.8035, device='cuda:0') (tensor(-1.5385, device='cuda:0')) | > avg_loss_gen: 1.7383264605428133 (-0.017552313670305963) | > avg_loss_kl: 2.3317461399964885 (-0.09189800981064966) | > avg_loss_feat: 1.853635957543279 (-0.004413885130009287) | > avg_loss_mel: 17.803705000541587 (-0.4468520258513884) | > avg_loss_duration: 1.649935655190911 (-0.008462271220247564) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.377349208778064 (-0.569178406621365) | > avg_grad_norm_1: tensor(286.5043, device='cuda:0') (+tensor(36.3744, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1008497.pth  > EPOCH: 118/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:24:00)   --> TIME: 2025-02-25 09:24:03 -- STEP: 3/72 -- GLOBAL_STEP: 1008500 | > loss_disc: 2.894160747528076 (2.867253303527832) | > loss_disc_real_0: 0.24751628935337067 (0.17545977234840393) | > loss_disc_real_1: 0.2275703400373459 (0.23204304774602255) | > loss_disc_real_2: 0.23340630531311035 (0.2319840838511785) | > loss_disc_real_3: 0.24728922545909882 (0.24403955042362213) | > loss_disc_real_4: 0.24977436661720276 (0.25737373034159344) | > loss_disc_real_5: 0.2749484181404114 (0.2797236243883769) | > loss_0: 2.894160747528076 (2.867253303527832) | > grad_norm_0: tensor(11.2020, device='cuda:0') (tensor(14.6360, device='cuda:0')) | > loss_gen: 1.7761930227279663 (1.7304871479670207) | > loss_kl: 2.474766492843628 (2.480562130610148) | > loss_feat: 1.6419074535369873 (1.8909279902776082) | > loss_mel: 18.33090591430664 (18.736294428507488) | > loss_duration: 1.6798553466796875 (1.658381422360738) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.903629302978516 (26.496653238932293) | > grad_norm_1: tensor(469.8001, device='cuda:0') (tensor(468.3873, device='cuda:0')) | > current_lr_0: 0.0001970714679845701 | > current_lr_1: 0.0001970714679845701 | > step_time: 0.4384 (0.43627746899922687) | > loader_time: 0.0081 (0.0070798397064208984)  --> TIME: 2025-02-25 09:24:26 -- STEP: 53/72 -- GLOBAL_STEP: 1008550 | > loss_disc: 2.85949969291687 (2.832202677456838) | > loss_disc_real_0: 0.2356933057308197 (0.22151349203766518) | > loss_disc_real_1: 0.3041488528251648 (0.2372463786939405) | > loss_disc_real_2: 0.23038825392723083 (0.23736074088879353) | > loss_disc_real_3: 0.2394043356180191 (0.24158462769580338) | > loss_disc_real_4: 0.24375875294208527 (0.2451431062423958) | > loss_disc_real_5: 0.22603552043437958 (0.2492716559823954) | > loss_0: 2.85949969291687 (2.832202677456838) | > grad_norm_0: tensor(8.0741, device='cuda:0') (tensor(10.8261, device='cuda:0')) | > loss_gen: 1.8146692514419556 (1.7561552861951433) | > loss_kl: 2.2114832401275635 (2.3503655577605627) | > loss_feat: 1.7957677841186523 (1.922858305697171) | > loss_mel: 16.021474838256836 (17.86690215344699) | > loss_duration: 1.6887145042419434 (1.6418468974671274) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.532108306884766 (25.538128402997863) | > grad_norm_1: tensor(362.3083, device='cuda:0') (tensor(246.6369, device='cuda:0')) | > current_lr_0: 0.0001970714679845701 | > current_lr_1: 0.0001970714679845701 | > step_time: 0.4446 (0.44638066021901257) | > loader_time: 0.0092 (0.007879189725192089) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624244743669542 (+4.261312350421244e-05) | > avg_step_time: 0.45626184973918216 (+0.00632932488347443) | > avg_loss_disc: 2.8409877662927334 (-0.0027661793668500856) | > avg_loss_disc_real_0: 0.22574731266834366 (+0.00010237674897825566) | > avg_loss_disc_real_1: 0.2349691395188721 (+0.00024734248577706897) | > avg_loss_disc_real_2: 0.2405374654581849 (+0.0004240633316443254) | > avg_loss_disc_real_3: 0.24228677699263668 (+0.0017623397665964269) | > avg_loss_disc_real_4: 0.24624002462541553 (+0.0010007728993053078) | > avg_loss_disc_real_5: 0.25076069882218266 (+0.004654437093667585) | > avg_loss_0: 2.8409877662927334 (-0.0027661793668500856) | > avg_grad_norm_0: tensor(12.2859, device='cuda:0') (+tensor(2.4824, device='cuda:0')) | > avg_loss_gen: 1.7553307338499686 (+0.01700427330715537) | > avg_loss_kl: 2.3723920399034526 (+0.0406458999069641) | > avg_loss_feat: 1.8996559784445963 (+0.04602002090131729) | > avg_loss_mel: 17.88940061649806 (+0.08569561595647457) | > avg_loss_duration: 1.6565859351359622 (+0.006650279945051185) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.573365520423568 (+0.19601631164550426) | > avg_grad_norm_1: tensor(244.4012, device='cuda:0') (tensor(-42.1032, device='cuda:0'))  > EPOCH: 119/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:24:37)   --> TIME: 2025-02-25 09:24:52 -- STEP: 31/72 -- GLOBAL_STEP: 1008600 | > loss_disc: 2.7467639446258545 (2.8261131086657123) | > loss_disc_real_0: 0.19425860047340393 (0.2296447403007938) | > loss_disc_real_1: 0.24392031133174896 (0.23280471082656615) | > loss_disc_real_2: 0.1798020601272583 (0.23257405190698563) | > loss_disc_real_3: 0.17511296272277832 (0.23697300111093828) | > loss_disc_real_4: 0.23907606303691864 (0.24324173936920782) | > loss_disc_real_5: 0.2534257173538208 (0.24372657941233727) | > loss_0: 2.7467639446258545 (2.8261131086657123) | > grad_norm_0: tensor(4.2238, device='cuda:0') (tensor(9.5215, device='cuda:0')) | > loss_gen: 1.7544376850128174 (1.7530966535691292) | > loss_kl: 2.4657609462738037 (2.4795248969908683) | > loss_feat: 2.175786018371582 (1.96461647172128) | > loss_mel: 18.138710021972656 (18.34734744410361) | > loss_duration: 1.6441351175308228 (1.6359678045395882) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.178831100463867 (26.18055312864242) | > grad_norm_1: tensor(106.4552, device='cuda:0') (tensor(198.9850, device='cuda:0')) | > current_lr_0: 0.000197046834051072 | > current_lr_1: 0.000197046834051072 | > step_time: 0.4365 (0.4357381636096585) | > loader_time: 0.0083 (0.006980265340497417) --> EVAL PERFORMANCE | > avg_loader_time: 0.008546624385135279 (-7.762035853426351e-05) | > avg_step_time: 0.4528685522751069 (-0.00339329746407524) | > avg_loss_disc: 2.8348600058488445 (-0.006127760443888963) | > avg_loss_disc_real_0: 0.22853265608280476 (+0.0027853434144611) | > avg_loss_disc_real_1: 0.2318355529241159 (-0.0031335865947562103) | > avg_loss_disc_real_2: 0.23653092287795643 (-0.004006542580228478) | > avg_loss_disc_real_3: 0.24117613930097767 (-0.0011106376916590055) | > avg_loss_disc_real_4: 0.24412164029101252 (-0.0021183843344030062) | > avg_loss_disc_real_5: 0.24438129974083161 (-0.006379399081351045) | > avg_loss_0: 2.8348600058488445 (-0.006127760443888963) | > avg_grad_norm_0: tensor(9.6819, device='cuda:0') (tensor(-2.6040, device='cuda:0')) | > avg_loss_gen: 1.7513314401599722 (-0.003999293689996408) | > avg_loss_kl: 2.403053465023846 (+0.030661425120393382) | > avg_loss_feat: 1.9306956882208166 (+0.03103970977622028) | > avg_loss_mel: 18.034275860853604 (+0.14487524435554278) | > avg_loss_duration: 1.6569933773766101 (+0.0004074422406479261) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.77634969899352 (+0.20298417856995243) | > avg_grad_norm_1: tensor(217.6151, device='cuda:0') (tensor(-26.7861, device='cuda:0'))  > EPOCH: 120/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:25:13)   --> TIME: 2025-02-25 09:25:19 -- STEP: 9/72 -- GLOBAL_STEP: 1008650 | > loss_disc: 2.7602410316467285 (2.8634965419769287) | > loss_disc_real_0: 0.18801233172416687 (0.2279668649037679) | > loss_disc_real_1: 0.19738785922527313 (0.2358551886346605) | > loss_disc_real_2: 0.20161154866218567 (0.24587341811921862) | > loss_disc_real_3: 0.22512730956077576 (0.23812495006455314) | > loss_disc_real_4: 0.25076788663864136 (0.25050120221243966) | > loss_disc_real_5: 0.23611928522586823 (0.2526626338561376) | > loss_0: 2.7602410316467285 (2.8634965419769287) | > grad_norm_0: tensor(8.0837, device='cuda:0') (tensor(8.1664, device='cuda:0')) | > loss_gen: 1.781408429145813 (1.7347343630260892) | > loss_kl: 2.4365992546081543 (2.37141489982605) | > loss_feat: 2.019899845123291 (2.037071466445923) | > loss_mel: 19.11353874206543 (19.352385414971245) | > loss_duration: 1.6509077548980713 (1.644220921728346) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.00235366821289 (27.139827304416233) | > grad_norm_1: tensor(122.5363, device='cuda:0') (tensor(315.0980, device='cuda:0')) | > current_lr_0: 0.00019702220319681561 | > current_lr_1: 0.00019702220319681561 | > step_time: 0.4348 (0.4274527496761746) | > loader_time: 0.0054 (0.006619400448269314)  --> TIME: 2025-02-25 09:25:42 -- STEP: 59/72 -- GLOBAL_STEP: 1008700 | > loss_disc: 2.7980360984802246 (2.850463519662114) | > loss_disc_real_0: 0.15279725193977356 (0.22696075676861455) | > loss_disc_real_1: 0.214533731341362 (0.2373217485718808) | > loss_disc_real_2: 0.2337779700756073 (0.24066312863665112) | > loss_disc_real_3: 0.2007354497909546 (0.2406584433579849) | > loss_disc_real_4: 0.22827160358428955 (0.24645020446534885) | > loss_disc_real_5: 0.23611602187156677 (0.24734556775982097) | > loss_0: 2.7980360984802246 (2.850463519662114) | > grad_norm_0: tensor(9.5139, device='cuda:0') (tensor(9.7824, device='cuda:0')) | > loss_gen: 1.8444968461990356 (1.7421852956383914) | > loss_kl: 2.2195627689361572 (2.3366117073317705) | > loss_feat: 2.0944182872772217 (1.933535630420103) | > loss_mel: 18.1267032623291 (18.24131471019681) | > loss_duration: 1.675676703453064 (1.645889647936417) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.960859298706055 (25.899536973339018) | > grad_norm_1: tensor(335.2340, device='cuda:0') (tensor(256.6339, device='cuda:0')) | > current_lr_0: 0.00019702220319681561 | > current_lr_1: 0.00019702220319681561 | > step_time: 0.4604 (0.44099465063062765) | > loader_time: 0.0097 (0.008014998193514549) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575012986089143 (+2.838860095386378e-05) | > avg_step_time: 0.45131635665893555 (-0.0015521956161713701) | > avg_loss_disc: 2.8518433604441906 (+0.016983354595346167) | > avg_loss_disc_real_0: 0.2311632473703841 (+0.002630591287579337) | > avg_loss_disc_real_1: 0.23977089704762042 (+0.007935344123504523) | > avg_loss_disc_real_2: 0.23958340455108965 (+0.0030524816731332116) | > avg_loss_disc_real_3: 0.2407108960856854 (-0.0004652432152922603) | > avg_loss_disc_real_4: 0.2448109442919073 (+0.0006893040008947859) | > avg_loss_disc_real_5: 0.24511553305135647 (+0.0007342333105248566) | > avg_loss_0: 2.8518433604441906 (+0.016983354595346167) | > avg_grad_norm_0: tensor(10.6144, device='cuda:0') (+tensor(0.9325, device='cuda:0')) | > avg_loss_gen: 1.7457694772263648 (-0.005561962933607445) | > avg_loss_kl: 2.329848477538203 (-0.07320498748564308) | > avg_loss_feat: 1.912504343919351 (-0.01819134430146563) | > avg_loss_mel: 18.176603129212292 (+0.14232726835868803) | > avg_loss_duration: 1.6511461717981688 (-0.005847205578441317) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.815871601373377 (+0.039521902379856755) | > avg_grad_norm_1: tensor(276.5771, device='cuda:0') (+tensor(58.9620, device='cuda:0'))  > EPOCH: 121/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:25:50)   --> TIME: 2025-02-25 09:26:09 -- STEP: 37/72 -- GLOBAL_STEP: 1008750 | > loss_disc: 2.8199124336242676 (2.8423519134521484) | > loss_disc_real_0: 0.1586817502975464 (0.23312612079285286) | > loss_disc_real_1: 0.18874363601207733 (0.24012436496244893) | > loss_disc_real_2: 0.23485703766345978 (0.24036914069910306) | > loss_disc_real_3: 0.24498650431632996 (0.24548189865576253) | > loss_disc_real_4: 0.26509934663772583 (0.24714384650861895) | > loss_disc_real_5: 0.21738648414611816 (0.24922517707218994) | > loss_0: 2.8199124336242676 (2.8423519134521484) | > grad_norm_0: tensor(11.7273, device='cuda:0') (tensor(8.9337, device='cuda:0')) | > loss_gen: 1.6884304285049438 (1.771258215646486) | > loss_kl: 2.2555599212646484 (2.500305820155788) | > loss_feat: 1.8382556438446045 (2.0001140381838822) | > loss_mel: 18.965133666992188 (18.41322898864746) | > loss_duration: 1.6471484899520874 (1.6494729712202743) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.394527435302734 (26.334380072516364) | > grad_norm_1: tensor(224.2146, device='cuda:0') (tensor(247.9037, device='cuda:0')) | > current_lr_0: 0.000196997575421416 | > current_lr_1: 0.000196997575421416 | > step_time: 0.4744 (0.43719295553258947) | > loader_time: 0.008 (0.007230700673283757) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568609264534963 (-6.4037215541795645e-06) | > avg_step_time: 0.4534051922005667 (+0.00208883554163114) | > avg_loss_disc: 2.8616149694147244 (+0.009771608970533752) | > avg_loss_disc_real_0: 0.23760006923071095 (+0.006436821860326847) | > avg_loss_disc_real_1: 0.24123322040262357 (+0.0014623233550031534) | > avg_loss_disc_real_2: 0.24518702189687272 (+0.005603617345783074) | > avg_loss_disc_real_3: 0.2455651814668951 (+0.004854285381209689) | > avg_loss_disc_real_4: 0.24763844865308682 (+0.0028275043611795114) | > avg_loss_disc_real_5: 0.24896077454929622 (+0.0038452414979397442) | > avg_loss_0: 2.8616149694147244 (+0.009771608970533752) | > avg_grad_norm_0: tensor(9.7702, device='cuda:0') (tensor(-0.8442, device='cuda:0')) | > avg_loss_gen: 1.766239901663552 (+0.020470424437187162) | > avg_loss_kl: 2.4737033038072185 (+0.14385482626901558) | > avg_loss_feat: 1.9320362319409008 (+0.019531888021549815) | > avg_loss_mel: 18.34322226887018 (+0.166619139657886) | > avg_loss_duration: 1.6655654554635706 (+0.014419283665401794) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.1807671936465 (+0.36489559227312185) | > avg_grad_norm_1: tensor(252.9825, device='cuda:0') (tensor(-23.5946, device='cuda:0'))  > EPOCH: 122/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:26:27)   --> TIME: 2025-02-25 09:26:35 -- STEP: 15/72 -- GLOBAL_STEP: 1008800 | > loss_disc: 2.923950672149658 (2.831955655415853) | > loss_disc_real_0: 0.37146639823913574 (0.22648312548796337) | > loss_disc_real_1: 0.25366106629371643 (0.2268927663564682) | > loss_disc_real_2: 0.2783689498901367 (0.2472155143817266) | > loss_disc_real_3: 0.24856501817703247 (0.23652251561482748) | > loss_disc_real_4: 0.25629523396492004 (0.24809630513191222) | > loss_disc_real_5: 0.2547435462474823 (0.24711072444915771) | > loss_0: 2.923950672149658 (2.831955655415853) | > grad_norm_0: tensor(20.2905, device='cuda:0') (tensor(9.7726, device='cuda:0')) | > loss_gen: 1.8229321241378784 (1.7345330635706584) | > loss_kl: 2.4480032920837402 (2.2713330427805585) | > loss_feat: 1.8094085454940796 (1.8212752183278402) | > loss_mel: 17.61872100830078 (17.903762435913087) | > loss_duration: 1.5943093299865723 (1.622736668586731) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.293373107910156 (25.353639984130858) | > grad_norm_1: tensor(95.4269, device='cuda:0') (tensor(242.9855, device='cuda:0')) | > current_lr_0: 0.00019697295072448832 | > current_lr_1: 0.00019697295072448832 | > step_time: 0.4258 (0.43217503229777016) | > loader_time: 0.0058 (0.006463766098022461)  --> TIME: 2025-02-25 09:26:59 -- STEP: 65/72 -- GLOBAL_STEP: 1008850 | > loss_disc: 2.838252544403076 (2.846258489902203) | > loss_disc_real_0: 0.1798495501279831 (0.22610808908939362) | > loss_disc_real_1: 0.20493249595165253 (0.2340854117503533) | > loss_disc_real_2: 0.2038046270608902 (0.24309938481220833) | > loss_disc_real_3: 0.21209362149238586 (0.24223960844370035) | > loss_disc_real_4: 0.22142428159713745 (0.24912587450100826) | > loss_disc_real_5: 0.24162685871124268 (0.2497988139207546) | > loss_0: 2.838252544403076 (2.846258489902203) | > grad_norm_0: tensor(12.6263, device='cuda:0') (tensor(8.4953, device='cuda:0')) | > loss_gen: 1.571721076965332 (1.745486017373892) | > loss_kl: 2.4094760417938232 (2.4321760911207924) | > loss_feat: 1.861517071723938 (1.8762896537780762) | > loss_mel: 18.608257293701172 (18.183348171527577) | > loss_duration: 1.6886506080627441 (1.6532926082611084) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.139619827270508 (25.890592604417066) | > grad_norm_1: tensor(91.4543, device='cuda:0') (tensor(225.9605, device='cuda:0')) | > current_lr_0: 0.00019697295072448832 | > current_lr_1: 0.00019697295072448832 | > step_time: 0.4916 (0.4458884422595684) | > loader_time: 0.0116 (0.008319979447584882) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577682602573446 (+9.07333803848269e-06) | > avg_step_time: 0.450555337986476 (-0.0028498542140906746) | > avg_loss_disc: 2.8470702775767154 (-0.014544691838008994) | > avg_loss_disc_real_0: 0.22667445410305345 (-0.010925615127657495) | > avg_loss_disc_real_1: 0.2355778391512347 (-0.005655381251388869) | > avg_loss_disc_real_2: 0.24280611058356058 (-0.0023809113133121396) | > avg_loss_disc_real_3: 0.24281379895311006 (-0.002751382513785039) | > avg_loss_disc_real_4: 0.24810087806741957 (+0.000462429414332749) | > avg_loss_disc_real_5: 0.24879437706000362 (-0.00016639748929259124) | > avg_loss_0: 2.8470702775767154 (-0.014544691838008994) | > avg_grad_norm_0: tensor(8.5918, device='cuda:0') (tensor(-1.1784, device='cuda:0')) | > avg_loss_gen: 1.745465767215675 (-0.020774134447876902) | > avg_loss_kl: 2.4297727094569668 (-0.043930594350251706) | > avg_loss_feat: 1.8608976098853098 (-0.07113862205559096) | > avg_loss_mel: 18.215603626949694 (-0.12761864192048478) | > avg_loss_duration: 1.6563808817259023 (-0.009184573737668345) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.908120625455613 (-0.2726465681908863) | > avg_grad_norm_1: tensor(218.0619, device='cuda:0') (tensor(-34.9206, device='cuda:0'))  > EPOCH: 123/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:27:03)   --> TIME: 2025-02-25 09:27:25 -- STEP: 43/72 -- GLOBAL_STEP: 1008900 | > loss_disc: 2.8561251163482666 (2.838756910590238) | > loss_disc_real_0: 0.13409560918807983 (0.22877460926078086) | > loss_disc_real_1: 0.2331743836402893 (0.23214110212270603) | > loss_disc_real_2: 0.23000793159008026 (0.24140830407308977) | > loss_disc_real_3: 0.2176816761493683 (0.24288022379542507) | > loss_disc_real_4: 0.22239650785923004 (0.24872612745262856) | > loss_disc_real_5: 0.21443088352680206 (0.24781790063824766) | > loss_0: 2.8561251163482666 (2.838756910590238) | > grad_norm_0: tensor(12.8151, device='cuda:0') (tensor(10.5397, device='cuda:0')) | > loss_gen: 1.5940794944763184 (1.7504641399827114) | > loss_kl: 2.1333577632904053 (2.3482635575671527) | > loss_feat: 2.076111078262329 (1.8960026668947796) | > loss_mel: 17.52189826965332 (17.83137064201887) | > loss_duration: 1.636040210723877 (1.637688811435256) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.96148681640625 (25.46378956284634) | > grad_norm_1: tensor(365.6432, device='cuda:0') (tensor(230.1749, device='cuda:0')) | > current_lr_0: 0.00019694832910564775 | > current_lr_1: 0.00019694832910564775 | > step_time: 0.4553 (0.43779050472170805) | > loader_time: 0.0094 (0.007476734560589457) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585778760238435 (+8.096157664989542e-06) | > avg_step_time: 0.45257450157487894 (+0.002019163588402928) | > avg_loss_disc: 2.838785319261148 (-0.008284958315567525) | > avg_loss_disc_real_0: 0.22723000528107226 (+0.0005555511780188094) | > avg_loss_disc_real_1: 0.23440956292857587 (-0.0011682762226588317) | > avg_loss_disc_real_2: 0.24236580645534353 (-0.00044030412821705056) | > avg_loss_disc_real_3: 0.2428501223174619 (+3.6323364351831255e-05) | > avg_loss_disc_real_4: 0.24680267862031158 (-0.0012981994471079894) | > avg_loss_disc_real_5: 0.24652250404928772 (-0.002271873010715908) | > avg_loss_0: 2.838785319261148 (-0.008284958315567525) | > avg_grad_norm_0: tensor(9.4997, device='cuda:0') (+tensor(0.9079, device='cuda:0')) | > avg_loss_gen: 1.7502692937850952 (+0.00480352656942018) | > avg_loss_kl: 2.3477806272641035 (-0.08199208219286325) | > avg_loss_feat: 1.8945434563596484 (+0.03364584647433855) | > avg_loss_mel: 17.88371499827211 (-0.3318886286775822) | > avg_loss_duration: 1.657033898461033 (+0.0006530167351308513) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.53334211967361 (-0.3747785057820039) | > avg_grad_norm_1: tensor(220.7684, device='cuda:0') (+tensor(2.7065, device='cuda:0'))  > EPOCH: 124/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:27:40)   --> TIME: 2025-02-25 09:27:51 -- STEP: 21/72 -- GLOBAL_STEP: 1008950 | > loss_disc: 2.890005111694336 (2.8216183412642706) | > loss_disc_real_0: 0.18214856088161469 (0.22428369664010547) | > loss_disc_real_1: 0.3023427426815033 (0.23201486397357213) | > loss_disc_real_2: 0.2648037374019623 (0.23834200983955747) | > loss_disc_real_3: 0.2893903851509094 (0.2396103214649927) | > loss_disc_real_4: 0.2806307077407837 (0.24531052864733197) | > loss_disc_real_5: 0.27087149024009705 (0.24588077408926828) | > loss_0: 2.890005111694336 (2.8216183412642706) | > grad_norm_0: tensor(8.9487, device='cuda:0') (tensor(8.2139, device='cuda:0')) | > loss_gen: 1.784650206565857 (1.7629162413733346) | > loss_kl: 2.2863447666168213 (2.278300466991606) | > loss_feat: 1.8982492685317993 (2.016915957132975) | > loss_mel: 18.275358200073242 (18.364219574701217) | > loss_duration: 1.6508169174194336 (1.6310684453873407) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.89542007446289 (26.053420566377184) | > grad_norm_1: tensor(145.9300, device='cuda:0') (tensor(194.0081, device='cuda:0')) | > current_lr_0: 0.00019692371056450955 | > current_lr_1: 0.00019692371056450955 | > step_time: 0.428 (0.4254894256591797) | > loader_time: 0.0071 (0.006669021788097564)  --> TIME: 2025-02-25 09:28:15 -- STEP: 71/72 -- GLOBAL_STEP: 1009000 | > loss_disc: 2.9104552268981934 (2.8279456857224585) | > loss_disc_real_0: 0.15555015206336975 (0.22338720253655608) | > loss_disc_real_1: 0.35701116919517517 (0.23550900134822012) | > loss_disc_real_2: 0.27141276001930237 (0.23971073413398905) | > loss_disc_real_3: 0.2282170057296753 (0.24114617222631482) | > loss_disc_real_4: 0.24343590438365936 (0.24586972489323414) | > loss_disc_real_5: 0.23319217562675476 (0.24703326338613532) | > loss_0: 2.9104552268981934 (2.8279456857224585) | > grad_norm_0: tensor(15.8892, device='cuda:0') (tensor(9.0383, device='cuda:0')) | > loss_gen: 1.9135470390319824 (1.7621658788600438) | > loss_kl: 2.427581310272217 (2.328605651855468) | > loss_feat: 1.9769450426101685 (1.9850555322539638) | > loss_mel: 18.264122009277344 (18.105255046360934) | > loss_duration: 1.6481826305389404 (1.6494275381867314) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.230377197265625 (25.8305095081598) | > grad_norm_1: tensor(400.1477, device='cuda:0') (tensor(239.0849, device='cuda:0')) | > current_lr_0: 0.00019692371056450955 | > current_lr_1: 0.00019692371056450955 | > step_time: 0.4987 (0.44952500705987636) | > loader_time: 0.0079 (0.008514505037119691) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514505037119691 (-7.127372311874387e-05) | > avg_step_time: 0.44952500705987636 (-0.003049494515002582) | > avg_loss_disc: 2.8279456857224585 (-0.01083963353868933) | > avg_loss_disc_real_0: 0.22338720253655608 (-0.003842802744516177) | > avg_loss_disc_real_1: 0.23550900134822012 (+0.0010994384196442442) | > avg_loss_disc_real_2: 0.23971073413398905 (-0.002655072321354479) | > avg_loss_disc_real_3: 0.24114617222631482 (-0.0017039500911470717) | > avg_loss_disc_real_4: 0.24586972489323414 (-0.0009329537270774402) | > avg_loss_disc_real_5: 0.24703326338613532 (+0.0005107593368476049) | > avg_loss_0: 2.8279456857224585 (-0.01083963353868933) | > avg_grad_norm_0: tensor(9.0383, device='cuda:0') (tensor(-0.4614, device='cuda:0')) | > avg_loss_gen: 1.7621658788600438 (+0.011896585074948618) | > avg_loss_kl: 2.328605651855468 (-0.019174975408635664) | > avg_loss_feat: 1.9850555322539638 (+0.09051207589431542) | > avg_loss_mel: 18.105255046360934 (+0.22154004808882277) | > avg_loss_duration: 1.6494275381867314 (-0.007606360274301727) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.8305095081598 (+0.29716738848619073) | > avg_grad_norm_1: tensor(239.0849, device='cuda:0') (+tensor(18.3165, device='cuda:0'))  > EPOCH: 125/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:28:17)   --> TIME: 2025-02-25 09:28:41 -- STEP: 49/72 -- GLOBAL_STEP: 1009050 | > loss_disc: 2.818774700164795 (2.8248624266410363) | > loss_disc_real_0: 0.24909698963165283 (0.22409907363507212) | > loss_disc_real_1: 0.13779830932617188 (0.23242102076812665) | > loss_disc_real_2: 0.22807903587818146 (0.2422110420100543) | > loss_disc_real_3: 0.26300811767578125 (0.24065057568404138) | > loss_disc_real_4: 0.23879583179950714 (0.24436758489024882) | > loss_disc_real_5: 0.23150192201137543 (0.24425655208071884) | > loss_0: 2.818774700164795 (2.8248624266410363) | > grad_norm_0: tensor(18.1052, device='cuda:0') (tensor(9.2884, device='cuda:0')) | > loss_gen: 1.7628936767578125 (1.7565648677397747) | > loss_kl: 2.4074044227600098 (2.399165966072861) | > loss_feat: 1.9943798780441284 (1.980989626475743) | > loss_mel: 18.12413787841797 (17.988769219846148) | > loss_duration: 1.730853796005249 (1.6389616338574156) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.019670486450195 (25.764451357783102) | > grad_norm_1: tensor(141.2494, device='cuda:0') (tensor(212.5877, device='cuda:0')) | > current_lr_0: 0.000196899095100689 | > current_lr_1: 0.000196899095100689 | > step_time: 0.4417 (0.4531414022251051) | > loader_time: 0.0095 (0.007816601772697605) --> EVAL PERFORMANCE | > avg_loader_time: 0.008695649429106375 (+0.00018114439198668383) | > avg_step_time: 0.46439408248579 (+0.014869075425913647) | > avg_loss_disc: 2.833773549173919 (+0.00582786345146058) | > avg_loss_disc_real_0: 0.225186879156341 (+0.0017996766197849179) | > avg_loss_disc_real_1: 0.23343821583499372 (-0.0020707855132263975) | > avg_loss_disc_real_2: 0.24146947705409896 (+0.0017587429201099125) | > avg_loss_disc_real_3: 0.24174195905806314 (+0.0005957868317483239) | > avg_loss_disc_real_4: 0.24561249391293863 (-0.0002572309802955086) | > avg_loss_disc_real_5: 0.24814506924488175 (+0.0011118058587464286) | > avg_loss_0: 2.833773549173919 (+0.00582786345146058) | > avg_grad_norm_0: tensor(9.6313, device='cuda:0') (+tensor(0.5930, device='cuda:0')) | > avg_loss_gen: 1.758869881361303 (-0.003295997498740766) | > avg_loss_kl: 2.38931036667085 (+0.06070471481538231) | > avg_loss_feat: 1.9563452844888392 (-0.028710247765124652) | > avg_loss_mel: 17.9992759328493 (-0.10597911351163347) | > avg_loss_duration: 1.6514460318525073 (+0.002018493665775889) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.755247411593587 (-0.07526209656621319) | > avg_grad_norm_1: tensor(235.2316, device='cuda:0') (tensor(-3.8533, device='cuda:0'))  > EPOCH: 126/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:28:54)   --> TIME: 2025-02-25 09:29:08 -- STEP: 27/72 -- GLOBAL_STEP: 1009100 | > loss_disc: 2.841951370239258 (2.8134781078056053) | > loss_disc_real_0: 0.12627965211868286 (0.216663952226992) | > loss_disc_real_1: 0.24331654608249664 (0.2367878661111549) | > loss_disc_real_2: 0.2124910056591034 (0.23998147745927176) | > loss_disc_real_3: 0.24501897394657135 (0.23968066330309268) | > loss_disc_real_4: 0.2212524563074112 (0.24555735565997935) | > loss_disc_real_5: 0.23941448330879211 (0.24693304962582058) | > loss_0: 2.841951370239258 (2.8134781078056053) | > grad_norm_0: tensor(13.5117, device='cuda:0') (tensor(9.1342, device='cuda:0')) | > loss_gen: 1.666780948638916 (1.766521524499964) | > loss_kl: 2.339459180831909 (2.349463939666748) | > loss_feat: 1.9811453819274902 (2.0267884422231606) | > loss_mel: 17.373579025268555 (18.04697333441841) | > loss_duration: 1.5870237350463867 (1.6428396701812744) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.947986602783203 (25.832586853592485) | > grad_norm_1: tensor(103.2812, device='cuda:0') (tensor(249.7748, device='cuda:0')) | > current_lr_0: 0.0001968744827138014 | > current_lr_1: 0.0001968744827138014 | > step_time: 0.4488 (0.43421280825579606) | > loader_time: 0.0082 (0.006987359788682725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625460342622137 (-7.018908648423819e-05) | > avg_step_time: 0.45181326463188926 (-0.01258081785390075) | > avg_loss_disc: 2.8220823146927523 (-0.011691234481166823) | > avg_loss_disc_real_0: 0.22281127737861284 (-0.0023756017777281646) | > avg_loss_disc_real_1: 0.23461623552819372 (+0.0011780196931999998) | > avg_loss_disc_real_2: 0.24214960865571467 (+0.0006801316016157022) | > avg_loss_disc_real_3: 0.24050946890468328 (-0.0012324901533798671) | > avg_loss_disc_real_4: 0.2457625752603504 (+0.00015008134741176238) | > avg_loss_disc_real_5: 0.2459797246355406 (-0.0021653446093411466) | > avg_loss_0: 2.8220823146927523 (-0.011691234481166823) | > avg_grad_norm_0: tensor(10.9283, device='cuda:0') (+tensor(1.2970, device='cuda:0')) | > avg_loss_gen: 1.7666843155740013 (+0.007814434212698229) | > avg_loss_kl: 2.3826140585079996 (-0.006696308162850606) | > avg_loss_feat: 1.988475646771175 (+0.0321303622823359) | > avg_loss_mel: 17.883978722800684 (-0.1152972100486167) | > avg_loss_duration: 1.661323280401633 (+0.009877248549125639) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68307594514229 (-0.07217146645129802) | > avg_grad_norm_1: tensor(242.0226, device='cuda:0') (+tensor(6.7910, device='cuda:0'))  > EPOCH: 127/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:29:31)   --> TIME: 2025-02-25 09:29:35 -- STEP: 5/72 -- GLOBAL_STEP: 1009150 | > loss_disc: 2.8277738094329834 (2.8060186386108397) | > loss_disc_real_0: 0.2313452959060669 (0.2341061294078827) | > loss_disc_real_1: 0.18672674894332886 (0.22637152373790742) | > loss_disc_real_2: 0.26778197288513184 (0.22767284214496614) | > loss_disc_real_3: 0.23328058421611786 (0.24511174261569976) | > loss_disc_real_4: 0.262572318315506 (0.24833742082118987) | > loss_disc_real_5: 0.27166369557380676 (0.2655049741268158) | > loss_0: 2.8277738094329834 (2.8060186386108397) | > grad_norm_0: tensor(6.4152, device='cuda:0') (tensor(16.9097, device='cuda:0')) | > loss_gen: 1.8896304368972778 (1.7843027114868164) | > loss_kl: 2.386305570602417 (2.3622018814086916) | > loss_feat: 1.8696812391281128 (2.0358415842056274) | > loss_mel: 17.370006561279297 (17.472645568847657) | > loss_duration: 1.616824984550476 (1.6263823509216309) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.132450103759766 (25.281374359130858) | > grad_norm_1: tensor(305.6591, device='cuda:0') (tensor(344.6317, device='cuda:0')) | > current_lr_0: 0.00019684987340346216 | > current_lr_1: 0.00019684987340346216 | > step_time: 0.4247 (0.4272500514984131) | > loader_time: 0.0071 (0.006715822219848633)  --> TIME: 2025-02-25 09:29:58 -- STEP: 55/72 -- GLOBAL_STEP: 1009200 | > loss_disc: 2.901193618774414 (2.8282512751492583) | > loss_disc_real_0: 0.1904871165752411 (0.2267525705424222) | > loss_disc_real_1: 0.21548619866371155 (0.23644959222186696) | > loss_disc_real_2: 0.26972219347953796 (0.24150672473690726) | > loss_disc_real_3: 0.22535575926303864 (0.24210273135792126) | > loss_disc_real_4: 0.2556599974632263 (0.24528904448856007) | > loss_disc_real_5: 0.2389320582151413 (0.24863870739936828) | > loss_0: 2.901193618774414 (2.8282512751492583) | > grad_norm_0: tensor(5.4132, device='cuda:0') (tensor(10.0025, device='cuda:0')) | > loss_gen: 1.8313612937927246 (1.7792131900787354) | > loss_kl: 2.4013736248016357 (2.367713295329701) | > loss_feat: 2.310512065887451 (2.0266890309073715) | > loss_mel: 19.17637062072754 (17.99987248507413) | > loss_duration: 1.693312406539917 (1.6382179086858577) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.41292953491211 (25.811705675992098) | > grad_norm_1: tensor(566.7577, device='cuda:0') (tensor(323.7551, device='cuda:0')) | > current_lr_0: 0.00019684987340346216 | > current_lr_1: 0.00019684987340346216 | > step_time: 0.4639 (0.4420209191062234) | > loader_time: 0.0093 (0.007883587750521573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596165079466053 (-2.9295263156084092e-05) | > avg_step_time: 0.4537681525861713 (+0.0019548879542820585) | > avg_loss_disc: 2.8278767155929345 (+0.005794400900182239) | > avg_loss_disc_real_0: 0.2288575096869133 (+0.006046232308300453) | > avg_loss_disc_real_1: 0.23778735856774827 (+0.003171123039554552) | > avg_loss_disc_real_2: 0.23999357076597885 (-0.0021560378897358123) | > avg_loss_disc_real_3: 0.2402767646060863 (-0.00023270429859698072) | > avg_loss_disc_real_4: 0.24575288572781523 (-9.689532535162027e-06) | > avg_loss_disc_real_5: 0.24772927500832248 (+0.0017495503727818773) | > avg_loss_0: 2.8278767155929345 (+0.005794400900182239) | > avg_grad_norm_0: tensor(10.1226, device='cuda:0') (tensor(-0.8057, device='cuda:0')) | > avg_loss_gen: 1.7733929325157487 (+0.006708616941747403) | > avg_loss_kl: 2.377646160797334 (-0.004967897710665525) | > avg_loss_feat: 1.9946658023646189 (+0.006190155593443825) | > avg_loss_mel: 17.94717431404221 (+0.06319559124152718) | > avg_loss_duration: 1.6497476252031997 (-0.011575655198433177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.742626673738723 (+0.05955072859643451) | > avg_grad_norm_1: tensor(337.1365, device='cuda:0') (+tensor(95.1138, device='cuda:0'))  > EPOCH: 128/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:30:08)   --> TIME: 2025-02-25 09:30:25 -- STEP: 33/72 -- GLOBAL_STEP: 1009250 | > loss_disc: 2.795694589614868 (2.83488181143096) | > loss_disc_real_0: 0.18799112737178802 (0.24212682721289722) | > loss_disc_real_1: 0.21013467013835907 (0.23326307941566815) | > loss_disc_real_2: 0.2391277700662613 (0.23963144918282828) | > loss_disc_real_3: 0.23620478808879852 (0.241625320279237) | > loss_disc_real_4: 0.2484552413225174 (0.24426100064407696) | > loss_disc_real_5: 0.24903687834739685 (0.2468256385940494) | > loss_0: 2.795694589614868 (2.83488181143096) | > grad_norm_0: tensor(4.9793, device='cuda:0') (tensor(10.7701, device='cuda:0')) | > loss_gen: 1.830545425415039 (1.7814846652926821) | > loss_kl: 2.368116617202759 (2.3098214973102915) | > loss_feat: 2.0367233753204346 (2.0415604620268857) | > loss_mel: 17.88150978088379 (17.902870004827324) | > loss_duration: 1.6137900352478027 (1.6200078653566765) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.73068618774414 (25.655744726007637) | > grad_norm_1: tensor(360.7883, device='cuda:0') (tensor(280.6485, device='cuda:0')) | > current_lr_0: 0.00019682526716928672 | > current_lr_1: 0.00019682526716928672 | > step_time: 0.4362 (0.43516073804913147) | > loader_time: 0.0078 (0.007060513351902817) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855813227908712 (-3.803280037893361e-05) | > avg_step_time: 0.45493515108672666 (+0.0011669985005553496) | > avg_loss_disc: 2.844413374511289 (+0.01653665891835443) | > avg_loss_disc_real_0: 0.23211425070611524 (+0.003256741019201953) | > avg_loss_disc_real_1: 0.24124150691737592 (+0.0034541483496276504) | > avg_loss_disc_real_2: 0.2408251982759422 (+0.0008316275099633508) | > avg_loss_disc_real_3: 0.24090599048305567 (+0.0006292258769693693) | > avg_loss_disc_real_4: 0.24498013711311448 (-0.0007727486147007523) | > avg_loss_disc_real_5: 0.24747921106681017 (-0.0002500639415123074) | > avg_loss_0: 2.844413374511289 (+0.01653665891835443) | > avg_grad_norm_0: tensor(9.6806, device='cuda:0') (tensor(-0.4420, device='cuda:0')) | > avg_loss_gen: 1.7587670591515554 (-0.014625873364193298) | > avg_loss_kl: 2.359549495535837 (-0.01809666526149689) | > avg_loss_feat: 1.9531441204984419 (-0.041521681866176996) | > avg_loss_mel: 17.8444962165725 (-0.10267809746970968) | > avg_loss_duration: 1.6467616843505644 (-0.0029859408526353715) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.562718780947407 (-0.17990789279131647) | > avg_grad_norm_1: tensor(270.7306, device='cuda:0') (tensor(-66.4059, device='cuda:0'))  > EPOCH: 129/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:30:45)   --> TIME: 2025-02-25 09:30:52 -- STEP: 11/72 -- GLOBAL_STEP: 1009300 | > loss_disc: 2.78977108001709 (2.8036254752766) | > loss_disc_real_0: 0.1699061244726181 (0.21109688011082736) | > loss_disc_real_1: 0.24348947405815125 (0.24026238918304443) | > loss_disc_real_2: 0.2030487060546875 (0.22087119790640744) | > loss_disc_real_3: 0.22969383001327515 (0.24797844344919379) | > loss_disc_real_4: 0.23330380022525787 (0.2543270831758326) | > loss_disc_real_5: 0.26339346170425415 (0.2521623522043228) | > loss_0: 2.78977108001709 (2.8036254752766) | > grad_norm_0: tensor(11.5306, device='cuda:0') (tensor(9.4941, device='cuda:0')) | > loss_gen: 1.8000524044036865 (1.802463022145358) | > loss_kl: 2.1783273220062256 (2.4183027744293213) | > loss_feat: 2.001811981201172 (2.0385421731255273) | > loss_mel: 18.585689544677734 (19.18806093389338) | > loss_duration: 1.6582152843475342 (1.670241724361073) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.224096298217773 (27.117610931396484) | > grad_norm_1: tensor(245.5950, device='cuda:0') (tensor(275.0335, device='cuda:0')) | > current_lr_0: 0.00019680066401089056 | > current_lr_1: 0.00019680066401089056 | > step_time: 0.4353 (0.43292823704806244) | > loader_time: 0.0066 (0.006322622299194336)  --> TIME: 2025-02-25 09:31:15 -- STEP: 61/72 -- GLOBAL_STEP: 1009350 | > loss_disc: 2.7552599906921387 (2.824674696218772) | > loss_disc_real_0: 0.26640552282333374 (0.22035052920462656) | > loss_disc_real_1: 0.2476954609155655 (0.23595801191251786) | > loss_disc_real_2: 0.22691334784030914 (0.23862036500797898) | > loss_disc_real_3: 0.21638743579387665 (0.24299652508047762) | > loss_disc_real_4: 0.22897575795650482 (0.24643884525924434) | > loss_disc_real_5: 0.24214543402194977 (0.2468535264984506) | > loss_0: 2.7552599906921387 (2.824674696218772) | > grad_norm_0: tensor(11.9358, device='cuda:0') (tensor(10.4034, device='cuda:0')) | > loss_gen: 1.7756869792938232 (1.7684538892058075) | > loss_kl: 2.282668113708496 (2.3992686467092543) | > loss_feat: 2.0358409881591797 (2.0037090465670726) | > loss_mel: 18.027223587036133 (18.519225855342686) | > loss_duration: 1.6698039770126343 (1.652024245652996) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.791223526000977 (26.342681665889554) | > grad_norm_1: tensor(557.5346, device='cuda:0') (tensor(236.3007, device='cuda:0')) | > current_lr_0: 0.00019680066401089056 | > current_lr_1: 0.00019680066401089056 | > step_time: 0.4622 (0.44979213495723536) | > loader_time: 0.0107 (0.008213778011134414) --> EVAL PERFORMANCE | > avg_loader_time: 0.008664027066297934 (+0.00010589478721081437) | > avg_step_time: 0.45815833521560884 (+0.0032231841288821728) | > avg_loss_disc: 2.825978222027631 (-0.0184351524836579) | > avg_loss_disc_real_0: 0.22131369304908832 (-0.010800557657026921) | > avg_loss_disc_real_1: 0.23578767772291748 (-0.005453829194458443) | > avg_loss_disc_real_2: 0.23801871000880925 (-0.0028064882671329505) | > avg_loss_disc_real_3: 0.24272167074008727 (+0.0018156802570316044) | > avg_loss_disc_real_4: 0.24572474247133227 (+0.000744605358217798) | > avg_loss_disc_real_5: 0.24673093960318768 (-0.0007482714636224963) | > avg_loss_0: 2.825978222027631 (-0.0184351524836579) | > avg_grad_norm_0: tensor(10.6788, device='cuda:0') (+tensor(0.9982, device='cuda:0')) | > avg_loss_gen: 1.765042889286095 (+0.006275830134539584) | > avg_loss_kl: 2.398116884097247 (+0.03856738856140973) | > avg_loss_feat: 1.991447260682012 (+0.03830314018357006) | > avg_loss_mel: 18.48884128516829 (+0.6443450685957899) | > avg_loss_duration: 1.6573263171692967 (+0.010564632818732322) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.300774560847753 (+0.7380557799003462) | > avg_grad_norm_1: tensor(239.3034, device='cuda:0') (tensor(-31.4272, device='cuda:0'))  > EPOCH: 130/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:31:22)   --> TIME: 2025-02-25 09:31:41 -- STEP: 39/72 -- GLOBAL_STEP: 1009400 | > loss_disc: 2.854703903198242 (2.836874821247199) | > loss_disc_real_0: 0.2777777314186096 (0.23970111058308527) | > loss_disc_real_1: 0.23330865800380707 (0.22663091466977045) | > loss_disc_real_2: 0.2220493108034134 (0.23347890109587938) | > loss_disc_real_3: 0.2404615581035614 (0.23817608180718544) | > loss_disc_real_4: 0.25792229175567627 (0.24195512517904624) | > loss_disc_real_5: 0.24362115561962128 (0.24404680117582664) | > loss_0: 2.854703903198242 (2.836874821247199) | > grad_norm_0: tensor(10.8773, device='cuda:0') (tensor(15.2846, device='cuda:0')) | > loss_gen: 1.7953062057495117 (1.760822669053689) | > loss_kl: 2.5362441539764404 (2.3219537123655667) | > loss_feat: 1.9046980142593384 (2.0258622352893534) | > loss_mel: 18.275896072387695 (18.06908724858211) | > loss_duration: 1.6343228816986084 (1.6284836591818395) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.146469116210938 (25.806209564208984) | > grad_norm_1: tensor(487.4705, device='cuda:0') (tensor(244.5166, device='cuda:0')) | > current_lr_0: 0.00019677606392788917 | > current_lr_1: 0.00019677606392788917 | > step_time: 0.4698 (0.43443083151792866) | > loader_time: 0.0081 (0.007491276814387395) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872055913361025 (+5.653206731231694e-05) | > avg_step_time: 0.4516584638138892 (-0.006499871401719648) | > avg_loss_disc: 2.840115570686233 (+0.014137348658601834) | > avg_loss_disc_real_0: 0.23548902319350712 (+0.014175330144418796) | > avg_loss_disc_real_1: 0.2332895020364036 (-0.002498175686513865) | > avg_loss_disc_real_2: 0.2391448868832118 (+0.0011261768744025569) | > avg_loss_disc_real_3: 0.23897037917459515 (-0.0037512915654921153) | > avg_loss_disc_real_4: 0.24342097195101456 (-0.002303770520317716) | > avg_loss_disc_real_5: 0.24588205004242106 (-0.0008488895607666191) | > avg_loss_0: 2.840115570686233 (+0.014137348658601834) | > avg_grad_norm_0: tensor(13.5839, device='cuda:0') (+tensor(2.9051, device='cuda:0')) | > avg_loss_gen: 1.7569750413088732 (-0.00806784797722182) | > avg_loss_kl: 2.330323746506596 (-0.06779313759065086) | > avg_loss_feat: 1.9764804218856387 (-0.014966838796373283) | > avg_loss_mel: 17.9433631628332 (-0.5454781223350906) | > avg_loss_duration: 1.6474531771431509 (-0.00987314002614581) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.65459554967746 (-0.6461790111702932) | > avg_grad_norm_1: tensor(236.7361, device='cuda:0') (tensor(-2.5673, device='cuda:0'))  > EPOCH: 131/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:31:59)   --> TIME: 2025-02-25 09:32:08 -- STEP: 17/72 -- GLOBAL_STEP: 1009450 | > loss_disc: 2.9192001819610596 (2.843988867367015) | > loss_disc_real_0: 0.1237250491976738 (0.2533407535623102) | > loss_disc_real_1: 0.20552374422550201 (0.2327004101346521) | > loss_disc_real_2: 0.24258895218372345 (0.24838190306635463) | > loss_disc_real_3: 0.2832757234573364 (0.2472307322656407) | > loss_disc_real_4: 0.2916824221611023 (0.25123152925687675) | > loss_disc_real_5: 0.29022544622421265 (0.24811378471991596) | > loss_0: 2.9192001819610596 (2.843988867367015) | > grad_norm_0: tensor(19.0308, device='cuda:0') (tensor(17.2816, device='cuda:0')) | > loss_gen: 1.681026577949524 (1.8132842358420878) | > loss_kl: 2.4187827110290527 (2.2966018003575943) | > loss_feat: 1.8437730073928833 (2.0902851048637836) | > loss_mel: 17.84027671813965 (18.181388854980465) | > loss_duration: 1.650977373123169 (1.6165850372875439) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.434837341308594 (25.99814515955308) | > grad_norm_1: tensor(311.5041, device='cuda:0') (tensor(346.3202, device='cuda:0')) | > current_lr_0: 0.00019675146691989817 | > current_lr_1: 0.00019675146691989817 | > step_time: 0.4499 (0.4298720219556023) | > loader_time: 0.0097 (0.0066027921788832725)  --> TIME: 2025-02-25 09:32:32 -- STEP: 67/72 -- GLOBAL_STEP: 1009500 | > loss_disc: 2.76936674118042 (2.842462667778357) | > loss_disc_real_0: 0.19923323392868042 (0.23684030926939267) | > loss_disc_real_1: 0.2628706395626068 (0.23650044291766723) | > loss_disc_real_2: 0.1872919201850891 (0.24058602861504055) | > loss_disc_real_3: 0.22039221227169037 (0.24262158706117032) | > loss_disc_real_4: 0.24038349092006683 (0.24710359991486394) | > loss_disc_real_5: 0.23486429452896118 (0.24658530038684162) | > loss_0: 2.76936674118042 (2.842462667778357) | > grad_norm_0: tensor(4.3069, device='cuda:0') (tensor(12.9178, device='cuda:0')) | > loss_gen: 1.7349181175231934 (1.7699736748168717) | > loss_kl: 2.5546247959136963 (2.3080992022557045) | > loss_feat: 2.0890445709228516 (1.9691182054690457) | > loss_mel: 18.944236755371094 (17.891168537424566) | > loss_duration: 1.730094075202942 (1.6494652531040248) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.052919387817383 (25.58782500651345) | > grad_norm_1: tensor(487.4405, device='cuda:0') (tensor(292.3690, device='cuda:0')) | > current_lr_0: 0.00019675146691989817 | > current_lr_1: 0.00019675146691989817 | > step_time: 0.5183 (0.45499912304664725) | > loader_time: 0.0127 (0.008450234114234128) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627115840643226 (-9.344329296702425e-05) | > avg_step_time: 0.45879929838046224 (+0.007140834566573051) | > avg_loss_disc: 2.8406774359689626 (+0.0005618652827297232) | > avg_loss_disc_real_0: 0.23754593373184474 (+0.0020569105383376196) | > avg_loss_disc_real_1: 0.23303027094249995 (-0.0002592310939036613) | > avg_loss_disc_real_2: 0.24073463846260393 (+0.001589751579392118) | > avg_loss_disc_real_3: 0.2426416966277109 (+0.0036713174531157466) | > avg_loss_disc_real_4: 0.2477110015674376 (+0.004290029616423052) | > avg_loss_disc_real_5: 0.2466600816434538 (+0.0007780316010327359) | > avg_loss_0: 2.8406774359689626 (+0.0005618652827297232) | > avg_grad_norm_0: tensor(13.2027, device='cuda:0') (tensor(-0.3812, device='cuda:0')) | > avg_loss_gen: 1.769227190756462 (+0.0122521494475889) | > avg_loss_kl: 2.3192345760238005 (-0.011089170482795563) | > avg_loss_feat: 1.9718399031061518 (-0.004640518779486902) | > avg_loss_mel: 17.89809302209128 (-0.04527014074191982) | > avg_loss_duration: 1.6518560429693947 (+0.004402865826243829) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.610250875983443 (-0.044344673694016734) | > avg_grad_norm_1: tensor(296.3337, device='cuda:0') (+tensor(59.5976, device='cuda:0'))  > EPOCH: 132/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:32:36)   --> TIME: 2025-02-25 09:32:58 -- STEP: 45/72 -- GLOBAL_STEP: 1009550 | > loss_disc: 2.897432327270508 (2.8523542298210987) | > loss_disc_real_0: 0.24203816056251526 (0.22614674154255127) | > loss_disc_real_1: 0.29788321256637573 (0.23584299269649717) | > loss_disc_real_2: 0.251619428396225 (0.2428182386689716) | > loss_disc_real_3: 0.2658636271953583 (0.2425035552846061) | > loss_disc_real_4: 0.26436182856559753 (0.2481593516137865) | > loss_disc_real_5: 0.2627350091934204 (0.2505447510215971) | > loss_0: 2.897432327270508 (2.8523542298210987) | > grad_norm_0: tensor(4.8258, device='cuda:0') (tensor(13.0285, device='cuda:0')) | > loss_gen: 1.7957158088684082 (1.764339804649353) | > loss_kl: 2.5138118267059326 (2.4203649044036863) | > loss_feat: 1.7484569549560547 (1.9986816432740953) | > loss_mel: 18.09029769897461 (18.30493498908149) | > loss_duration: 1.6723265647888184 (1.634066065152486) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.820608139038086 (26.12238731384277) | > grad_norm_1: tensor(136.3620, device='cuda:0') (tensor(245.2998, device='cuda:0')) | > current_lr_0: 0.00019672687298653317 | > current_lr_1: 0.00019672687298653317 | > step_time: 0.4534 (0.4404179573059082) | > loader_time: 0.0085 (0.007535823186238607) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593831263797383 (-3.3284576845843405e-05) | > avg_step_time: 0.45378775327978 (-0.0050115451006822465) | > avg_loss_disc: 2.8452725444041502 (+0.004595108435187623) | > avg_loss_disc_real_0: 0.2270181414107202 (-0.010527792321124552) | > avg_loss_disc_real_1: 0.23449189425774022 (+0.0014616233152402736) | > avg_loss_disc_real_2: 0.24261137171530386 (+0.0018767332526999358) | > avg_loss_disc_real_3: 0.24181548033801603 (-0.0008262162896948699) | > avg_loss_disc_real_4: 0.2449181659120909 (-0.002792835655346715) | > avg_loss_disc_real_5: 0.2467648987199219 (+0.00010481707646811667) | > avg_loss_0: 2.8452725444041502 (+0.004595108435187623) | > avg_grad_norm_0: tensor(11.5317, device='cuda:0') (tensor(-1.6710, device='cuda:0')) | > avg_loss_gen: 1.7622879592465683 (-0.0069392315098937285) | > avg_loss_kl: 2.4233035503978466 (+0.10406897437404616) | > avg_loss_feat: 1.9745692232964744 (+0.0027293201903226816) | > avg_loss_mel: 18.183202098792705 (+0.28510907670142416) | > avg_loss_duration: 1.6644076293622945 (+0.012551586392899772) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.007770404009744 (+0.39751952802630086) | > avg_grad_norm_1: tensor(263.3813, device='cuda:0') (tensor(-32.9523, device='cuda:0'))  > EPOCH: 133/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:33:13)   --> TIME: 2025-02-25 09:33:25 -- STEP: 23/72 -- GLOBAL_STEP: 1009600 | > loss_disc: 2.8756520748138428 (2.8265850336655327) | > loss_disc_real_0: 0.16098102927207947 (0.23725140548270682) | > loss_disc_real_1: 0.28288930654525757 (0.23127807482429172) | > loss_disc_real_2: 0.2569682002067566 (0.23061426178268765) | > loss_disc_real_3: 0.23499056696891785 (0.23478753476039224) | > loss_disc_real_4: 0.23918795585632324 (0.2405690362919932) | > loss_disc_real_5: 0.2502601444721222 (0.2441473758738974) | > loss_0: 2.8756520748138428 (2.8265850336655327) | > grad_norm_0: tensor(17.3131, device='cuda:0') (tensor(12.0028, device='cuda:0')) | > loss_gen: 1.7840118408203125 (1.7555421953615935) | > loss_kl: 2.419403553009033 (2.3599337598551875) | > loss_feat: 2.0561838150024414 (1.9952605392621912) | > loss_mel: 16.454378128051758 (17.9064358420994) | > loss_duration: 1.649675726890564 (1.634746199068816) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.3636531829834 (25.65191882589589) | > grad_norm_1: tensor(504.0211, device='cuda:0') (tensor(171.3791, device='cuda:0')) | > current_lr_0: 0.00019670228212740986 | > current_lr_1: 0.00019670228212740986 | > step_time: 0.43 (0.43518830382305645) | > loader_time: 0.0065 (0.006442246229752251) --> EVAL PERFORMANCE | > avg_loader_time: 0.008500740561686771 (-9.30907021106115e-05) | > avg_step_time: 0.4564606270319979 (+0.002672873752217919) | > avg_loss_disc: 2.8361975374356123 (-0.009075006968537913) | > avg_loss_disc_real_0: 0.23502511100869783 (+0.008006969597977642) | > avg_loss_disc_real_1: 0.2322585406437726 (-0.0022333536139676202) | > avg_loss_disc_real_2: 0.23832651993758242 (-0.004284851777721449) | > avg_loss_disc_real_3: 0.23806848723283955 (-0.0037469931051764827) | > avg_loss_disc_real_4: 0.244078451478985 (-0.0008397144331059037) | > avg_loss_disc_real_5: 0.2478858280769536 (+0.0011209293570317025) | > avg_loss_0: 2.8361975374356123 (-0.009075006968537913) | > avg_grad_norm_0: tensor(13.1938, device='cuda:0') (+tensor(1.6621, device='cuda:0')) | > avg_loss_gen: 1.7699894132748457 (+0.007701454028277377) | > avg_loss_kl: 2.3768205844180685 (-0.04648296597977808) | > avg_loss_feat: 2.0136856159693752 (+0.03911639267290079) | > avg_loss_mel: 17.972262637715943 (-0.2109394610767623) | > avg_loss_duration: 1.6529560089111328 (-0.01145162045116166) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.785714243499324 (-0.22205616051041943) | > avg_grad_norm_1: tensor(243.3534, device='cuda:0') (tensor(-20.0280, device='cuda:0'))  > EPOCH: 134/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:33:50)   --> TIME: 2025-02-25 09:33:52 -- STEP: 1/72 -- GLOBAL_STEP: 1009650 | > loss_disc: 2.710556983947754 (2.710556983947754) | > loss_disc_real_0: 0.19979532063007355 (0.19979532063007355) | > loss_disc_real_1: 0.21159707009792328 (0.21159707009792328) | > loss_disc_real_2: 0.2423340529203415 (0.2423340529203415) | > loss_disc_real_3: 0.2418859601020813 (0.2418859601020813) | > loss_disc_real_4: 0.22933326661586761 (0.22933326661586761) | > loss_disc_real_5: 0.2596051096916199 (0.2596051096916199) | > loss_0: 2.710556983947754 (2.710556983947754) | > grad_norm_0: tensor(3.9770, device='cuda:0') (tensor(3.9770, device='cuda:0')) | > loss_gen: 1.9906151294708252 (1.9906151294708252) | > loss_kl: 2.4061012268066406 (2.4061012268066406) | > loss_feat: 2.512977123260498 (2.512977123260498) | > loss_mel: 18.813064575195312 (18.813064575195312) | > loss_duration: 1.6563551425933838 (1.6563551425933838) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.379114151000977 (27.379114151000977) | > grad_norm_1: tensor(680.3429, device='cuda:0') (tensor(680.3429, device='cuda:0')) | > current_lr_0: 0.00019667769434214392 | > current_lr_1: 0.00019667769434214392 | > step_time: 0.4264 (0.426361083984375) | > loader_time: 0.0057 (0.0056629180908203125)  --> TIME: 2025-02-25 09:34:15 -- STEP: 51/72 -- GLOBAL_STEP: 1009700 | > loss_disc: 2.7825868129730225 (2.81874995138131) | > loss_disc_real_0: 0.2361644208431244 (0.2153460437468454) | > loss_disc_real_1: 0.22416524589061737 (0.23336034253531812) | > loss_disc_real_2: 0.21329496800899506 (0.243333755754957) | > loss_disc_real_3: 0.22334600985050201 (0.24141934251084046) | > loss_disc_real_4: 0.2407170832157135 (0.2464595036763771) | > loss_disc_real_5: 0.2414645403623581 (0.24697022899693133) | > loss_0: 2.7825868129730225 (2.81874995138131) | > grad_norm_0: tensor(6.7867, device='cuda:0') (tensor(9.4233, device='cuda:0')) | > loss_gen: 1.6602494716644287 (1.7776702315199608) | > loss_kl: 2.140697956085205 (2.274106254764632) | > loss_feat: 2.0754966735839844 (2.02573009332021) | > loss_mel: 17.355518341064453 (17.796886986377196) | > loss_duration: 1.6631851196289062 (1.6332849731632308) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.8951473236084 (25.50767827501484) | > grad_norm_1: tensor(140.7742, device='cuda:0') (tensor(360.6353, device='cuda:0')) | > current_lr_0: 0.00019667769434214392 | > current_lr_1: 0.00019667769434214392 | > step_time: 0.4541 (0.4541955695432775) | > loader_time: 0.0095 (0.007805492363724054) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653771709388412 (+0.0001530311477016405) | > avg_step_time: 0.46273529697471943 (+0.0062746699427215225) | > avg_loss_disc: 2.8282901602731623 (-0.007907377162450047) | > avg_loss_disc_real_0: 0.21711310943667317 (-0.01791200157202466) | > avg_loss_disc_real_1: 0.23331358000426225 (+0.0010550393604896458) | > avg_loss_disc_real_2: 0.24370407680390585 (+0.005377556866323435) | > avg_loss_disc_real_3: 0.24019395759407902 (+0.002125470361239473) | > avg_loss_disc_real_4: 0.24673287654426737 (+0.0026544250652823786) | > avg_loss_disc_real_5: 0.24806484663990183 (+0.00017901856294821883) | > avg_loss_0: 2.8282901602731623 (-0.007907377162450047) | > avg_grad_norm_0: tensor(9.7220, device='cuda:0') (tensor(-3.4717, device='cuda:0')) | > avg_loss_gen: 1.7655102249602197 (-0.004479188314626015) | > avg_loss_kl: 2.275737571044707 (-0.10108301337336156) | > avg_loss_feat: 1.965605316027789 (-0.048080299941586135) | > avg_loss_mel: 17.613941891092654 (-0.3583207466232885) | > avg_loss_duration: 1.6422339207689527 (-0.01072208814218012) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.26302872241383 (-0.5226855210854957) | > avg_grad_norm_1: tensor(353.2978, device='cuda:0') (+tensor(109.9444, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1009721.pth  > EPOCH: 135/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:34:30)   --> TIME: 2025-02-25 09:34:45 -- STEP: 29/72 -- GLOBAL_STEP: 1009750 | > loss_disc: 2.861896514892578 (2.8311282519636483) | > loss_disc_real_0: 0.27467572689056396 (0.2249989838435732) | > loss_disc_real_1: 0.2274363785982132 (0.24008114533177738) | > loss_disc_real_2: 0.20017068088054657 (0.2379121528617267) | > loss_disc_real_3: 0.22157292068004608 (0.2451820476301785) | > loss_disc_real_4: 0.25006046891212463 (0.25170138375512485) | > loss_disc_real_5: 0.2478359490633011 (0.2524345178028633) | > loss_0: 2.861896514892578 (2.8311282519636483) | > grad_norm_0: tensor(8.7374, device='cuda:0') (tensor(13.0623, device='cuda:0')) | > loss_gen: 1.596498727798462 (1.7792007224313144) | > loss_kl: 2.128965377807617 (2.211370879206164) | > loss_feat: 1.8362300395965576 (2.0402707601415706) | > loss_mel: 18.720600128173828 (17.47503741034146) | > loss_duration: 1.6002087593078613 (1.6029294441486228) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.88250160217285 (25.10880917516248) | > grad_norm_1: tensor(365.1267, device='cuda:0') (tensor(371.1061, device='cuda:0')) | > current_lr_0: 0.00019665310963035113 | > current_lr_1: 0.00019665310963035113 | > step_time: 0.4404 (0.4440453463587268) | > loader_time: 0.0078 (0.007364388169913456) --> EVAL PERFORMANCE | > avg_loader_time: 0.009030919679453676 (+0.0003771479700652638) | > avg_step_time: 0.45941186958635355 (-0.003323427388365885) | > avg_loss_disc: 2.8272063832887455 (-0.001083776984416751) | > avg_loss_disc_real_0: 0.2224367491795983 (+0.005323639742925129) | > avg_loss_disc_real_1: 0.23957019246799846 (+0.006256612463736211) | > avg_loss_disc_real_2: 0.2386218636808261 (-0.005082213123079743) | > avg_loss_disc_real_3: 0.24218901278267443 (+0.0019950551885954126) | > avg_loss_disc_real_4: 0.2466386275392183 (-9.424900504906297e-05) | > avg_loss_disc_real_5: 0.24825531055389996 (+0.000190463913998129) | > avg_loss_0: 2.8272063832887455 (-0.001083776984416751) | > avg_grad_norm_0: tensor(10.5948, device='cuda:0') (+tensor(0.8728, device='cuda:0')) | > avg_loss_gen: 1.7712540979116735 (+0.005743872951453843) | > avg_loss_kl: 2.2923342879389375 (+0.01659671689423048) | > avg_loss_feat: 2.012896784594361 (+0.04729146856657196) | > avg_loss_mel: 17.5746735586247 (-0.03926833246795525) | > avg_loss_duration: 1.6369886935596736 (-0.005245227209279113) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.288147456209426 (+0.025118733795597592) | > avg_grad_norm_1: tensor(322.9191, device='cuda:0') (tensor(-30.3786, device='cuda:0'))  > EPOCH: 136/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:35:07)   --> TIME: 2025-02-25 09:35:12 -- STEP: 7/72 -- GLOBAL_STEP: 1009800 | > loss_disc: 2.768542528152466 (2.8221448148999895) | > loss_disc_real_0: 0.1710284948348999 (0.24262057883398874) | > loss_disc_real_1: 0.18435129523277283 (0.2264264758144106) | > loss_disc_real_2: 0.2556527554988861 (0.26036695710250307) | > loss_disc_real_3: 0.21915271878242493 (0.24181594380310603) | > loss_disc_real_4: 0.22331365942955017 (0.2371715498822076) | > loss_disc_real_5: 0.21552108228206635 (0.23643659480980464) | > loss_0: 2.768542528152466 (2.8221448148999895) | > grad_norm_0: tensor(14.3402, device='cuda:0') (tensor(12.6251, device='cuda:0')) | > loss_gen: 1.5741692781448364 (1.7384786094938005) | > loss_kl: 2.4400086402893066 (2.3477886063711986) | > loss_feat: 2.1507222652435303 (2.0034797191619873) | > loss_mel: 18.533781051635742 (18.37784549168178) | > loss_duration: 1.6390528678894043 (1.6300223895481654) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.33773422241211 (26.09761483328683) | > grad_norm_1: tensor(441.5283, device='cuda:0') (tensor(241.3542, device='cuda:0')) | > current_lr_0: 0.00019662852799164733 | > current_lr_1: 0.00019662852799164733 | > step_time: 0.4392 (0.4282752105167934) | > loader_time: 0.0064 (0.006194864000592913)  --> TIME: 2025-02-25 09:35:35 -- STEP: 57/72 -- GLOBAL_STEP: 1009850 | > loss_disc: 2.8552558422088623 (2.830325486367209) | > loss_disc_real_0: 0.13489174842834473 (0.22988829278109366) | > loss_disc_real_1: 0.1780242919921875 (0.2273263889446593) | > loss_disc_real_2: 0.21836474537849426 (0.24369875851430392) | > loss_disc_real_3: 0.25556623935699463 (0.2399679260295734) | > loss_disc_real_4: 0.26937028765678406 (0.24340216632474931) | > loss_disc_real_5: 0.2808400094509125 (0.24557055611359446) | > loss_0: 2.8552558422088623 (2.830325486367209) | > grad_norm_0: tensor(21.1643, device='cuda:0') (tensor(13.5011, device='cuda:0')) | > loss_gen: 1.5930097103118896 (1.7605566685659844) | > loss_kl: 2.2626395225524902 (2.3992368505712145) | > loss_feat: 1.8951694965362549 (2.0249105252717685) | > loss_mel: 17.141233444213867 (18.023676219739407) | > loss_duration: 1.6693835258483887 (1.6435298208604778) | > amp_scaler: 256.0 (345.82456140350877) | > loss_1: 24.56143569946289 (25.851910105922766) | > grad_norm_1: tensor(534.7872, device='cuda:0') (tensor(223.8504, device='cuda:0')) | > current_lr_0: 0.00019662852799164733 | > current_lr_1: 0.00019662852799164733 | > step_time: 0.4611 (0.44267697083322627) | > loader_time: 0.0099 (0.007920353036177784) --> EVAL PERFORMANCE | > avg_loader_time: 0.008576634904028664 (-0.00045428477542501164) | > avg_step_time: 0.45373255769971393 (-0.005679311886639615) | > avg_loss_disc: 2.8366148639732685 (+0.009408480684522935) | > avg_loss_disc_real_0: 0.2320747043884976 (+0.00963795520889929) | > avg_loss_disc_real_1: 0.23268872184652678 (-0.00688147062147168) | > avg_loss_disc_real_2: 0.2427392799249837 (+0.004117416244157596) | > avg_loss_disc_real_3: 0.24007962958913454 (-0.0021093831935398988) | > avg_loss_disc_real_4: 0.2437387161271673 (-0.0028999114120510094) | > avg_loss_disc_real_5: 0.24629920385253262 (-0.0019561067013673383) | > avg_loss_0: 2.8366148639732685 (+0.009408480684522935) | > avg_grad_norm_0: tensor(13.9155, device='cuda:0') (+tensor(3.3207, device='cuda:0')) | > avg_loss_gen: 1.7678787372481655 (-0.0033753606635080757) | > avg_loss_kl: 2.3860050792425453 (+0.09367079130360789) | > avg_loss_feat: 1.9960607945079551 (-0.016835990086405905) | > avg_loss_mel: 17.958703135100887 (+0.38402957647618763) | > avg_loss_duration: 1.6490770309743747 (+0.012088337414701167) | > avg_amp_scaler: 328.11267605633805 (+72.11267605633805) | > avg_loss_1: 25.757724896283218 (+0.46957744007379176) | > avg_grad_norm_1: tensor(246.1574, device='cuda:0') (tensor(-76.7617, device='cuda:0'))  > EPOCH: 137/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:35:44)   --> TIME: 2025-02-25 09:36:02 -- STEP: 35/72 -- GLOBAL_STEP: 1009900 | > loss_disc: 2.8713607788085938 (2.828493118286133) | > loss_disc_real_0: 0.2567070424556732 (0.21991627982684545) | > loss_disc_real_1: 0.26514363288879395 (0.2353156247309276) | > loss_disc_real_2: 0.24018029868602753 (0.2386885553598404) | > loss_disc_real_3: 0.2789873778820038 (0.2391030192375183) | > loss_disc_real_4: 0.25800445675849915 (0.24427165516785213) | > loss_disc_real_5: 0.2668767273426056 (0.24783368536404202) | > loss_0: 2.8713607788085938 (2.828493118286133) | > grad_norm_0: tensor(10.2864, device='cuda:0') (tensor(9.9256, device='cuda:0')) | > loss_gen: 1.718691110610962 (1.7583267790930612) | > loss_kl: 2.5690746307373047 (2.365027175630842) | > loss_feat: 1.691635012626648 (1.9972746201923914) | > loss_mel: 17.761280059814453 (17.90809440612793) | > loss_duration: 1.671602725982666 (1.6233419792992727) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.412282943725586 (25.65206484113421) | > grad_norm_1: tensor(297.4482, device='cuda:0') (tensor(302.9407, device='cuda:0')) | > current_lr_0: 0.00019660394942564837 | > current_lr_1: 0.00019660394942564837 | > step_time: 0.4436 (0.4496155738830566) | > loader_time: 0.0082 (0.0071084295000348775) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859726650614134 (+2.0631602112675812e-05) | > avg_step_time: 0.46144092250877705 (+0.007708364809063117) | > avg_loss_disc: 2.827982281295347 (-0.008632582677921441) | > avg_loss_disc_real_0: 0.22065716310286185 (-0.01141754128563574) | > avg_loss_disc_real_1: 0.23629313455501072 (+0.0036044127084839395) | > avg_loss_disc_real_2: 0.23827758536372387 (-0.0044616945612598324) | > avg_loss_disc_real_3: 0.23884522746986067 (-0.0012344021192738641) | > avg_loss_disc_real_4: 0.2445663482370511 (+0.0008276321098837913) | > avg_loss_disc_real_5: 0.24816364406699865 (+0.0018644402144660233) | > avg_loss_0: 2.827982281295347 (-0.008632582677921441) | > avg_grad_norm_0: tensor(10.9048, device='cuda:0') (tensor(-3.0107, device='cuda:0')) | > avg_loss_gen: 1.7618102053521385 (-0.006068531896026963) | > avg_loss_kl: 2.346409905124718 (-0.039595174117827536) | > avg_loss_feat: 1.9803674338569104 (-0.015693360651044763) | > avg_loss_mel: 17.732399188296906 (-0.2263039468039807) | > avg_loss_duration: 1.647541128413778 (-0.001535902560596769) | > avg_amp_scaler: 256.0 (-72.11267605633805) | > avg_loss_1: 25.46852778045224 (-0.2891971158309765) | > avg_grad_norm_1: tensor(292.3065, device='cuda:0') (+tensor(46.1491, device='cuda:0'))  > EPOCH: 138/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:36:21)   --> TIME: 2025-02-25 09:36:29 -- STEP: 13/72 -- GLOBAL_STEP: 1009950 | > loss_disc: 2.8544983863830566 (2.8313169479370117) | > loss_disc_real_0: 0.2267635464668274 (0.22340784737697014) | > loss_disc_real_1: 0.2995845079421997 (0.2476878475684386) | > loss_disc_real_2: 0.24697259068489075 (0.24672755484397596) | > loss_disc_real_3: 0.24712292850017548 (0.25307172536849976) | > loss_disc_real_4: 0.2390882521867752 (0.24652618857530448) | > loss_disc_real_5: 0.23547735810279846 (0.24525105609343603) | > loss_0: 2.8544983863830566 (2.8313169479370117) | > grad_norm_0: tensor(17.5018, device='cuda:0') (tensor(12.2401, device='cuda:0')) | > loss_gen: 1.7897610664367676 (1.805829644203186) | > loss_kl: 2.397636651992798 (2.238850006690392) | > loss_feat: 1.7610934972763062 (2.075173543049739) | > loss_mel: 16.936077117919922 (17.82818779578576) | > loss_duration: 1.5414361953735352 (1.611594796180725) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.426002502441406 (25.559635455791767) | > grad_norm_1: tensor(80.7520, device='cuda:0') (tensor(340.4609, device='cuda:0')) | > current_lr_0: 0.00019657937393197016 | > current_lr_1: 0.00019657937393197016 | > step_time: 0.4635 (0.4358795422774095) | > loader_time: 0.006 (0.006599206190842848)  --> TIME: 2025-02-25 09:36:52 -- STEP: 63/72 -- GLOBAL_STEP: 1010000 | > loss_disc: 2.857452869415283 (2.8426608320266484) | > loss_disc_real_0: 0.32703977823257446 (0.23016961462913998) | > loss_disc_real_1: 0.22586359083652496 (0.2393414926907373) | > loss_disc_real_2: 0.2359848916530609 (0.24182585094656264) | > loss_disc_real_3: 0.1921551674604416 (0.24231387485587408) | > loss_disc_real_4: 0.2160922735929489 (0.2449574354622099) | > loss_disc_real_5: 0.2207537442445755 (0.24730099643033648) | > loss_0: 2.857452869415283 (2.8426608320266484) | > grad_norm_0: tensor(12.8472, device='cuda:0') (tensor(12.7171, device='cuda:0')) | > loss_gen: 1.5519753694534302 (1.7659411524969435) | > loss_kl: 2.130521059036255 (2.315867923554919) | > loss_feat: 1.8131606578826904 (2.0110155552152613) | > loss_mel: 17.15725326538086 (17.678776029556523) | > loss_duration: 1.6321113109588623 (1.6370983142701407) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.28502082824707 (25.408698944818404) | > grad_norm_1: tensor(69.6190, device='cuda:0') (tensor(308.9344, device='cuda:0')) | > current_lr_0: 0.00019657937393197016 | > current_lr_1: 0.00019657937393197016 | > step_time: 0.4896 (0.4484676330808609) | > loader_time: 0.0111 (0.008230792151557075) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1010000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008593851411846322 (-3.4150942950176427e-06) | > avg_step_time: 0.45519634031913647 (-0.0062445821896405795) | > avg_loss_disc: 2.8437778546776573 (+0.0157955733823103) | > avg_loss_disc_real_0: 0.2276460696274126 (+0.006988906524550753) | > avg_loss_disc_real_1: 0.2379639681795953 (+0.001670833624584589) | > avg_loss_disc_real_2: 0.24162125545488278 (+0.003343670091158907) | > avg_loss_disc_real_3: 0.24284898731070506 (+0.00400375984084439) | > avg_loss_disc_real_4: 0.24479749945687576 (+0.00023115121982467124) | > avg_loss_disc_real_5: 0.24621437196160706 (-0.0019492721053915862) | > avg_loss_0: 2.8437778546776573 (+0.0157955733823103) | > avg_grad_norm_0: tensor(12.4204, device='cuda:0') (+tensor(1.5155, device='cuda:0')) | > avg_loss_gen: 1.7631172092867566 (+0.001307003934618134) | > avg_loss_kl: 2.323721418918018 (-0.02268848620669983) | > avg_loss_feat: 1.9955864070167013 (+0.015218973159790927) | > avg_loss_mel: 17.687008199557457 (-0.0453909887394488) | > avg_loss_duration: 1.6424606551586742 (-0.005080473255103746) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.41189389833262 (-0.05663388211961973) | > avg_grad_norm_1: tensor(294.9803, device='cuda:0') (+tensor(2.6737, device='cuda:0'))  > EPOCH: 139/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:37:01)   --> TIME: 2025-02-25 09:37:21 -- STEP: 41/72 -- GLOBAL_STEP: 1010050 | > loss_disc: 2.992640495300293 (2.838063920416483) | > loss_disc_real_0: 0.24104087054729462 (0.2240713918354453) | > loss_disc_real_1: 0.28458863496780396 (0.23821085146287593) | > loss_disc_real_2: 0.2494010031223297 (0.24147710858321772) | > loss_disc_real_3: 0.28993695974349976 (0.24353467545858243) | > loss_disc_real_4: 0.23676882684230804 (0.24390281046309123) | > loss_disc_real_5: 0.24064764380455017 (0.24583279541352904) | > loss_0: 2.992640495300293 (2.838063920416483) | > grad_norm_0: tensor(8.6000, device='cuda:0') (tensor(10.4356, device='cuda:0')) | > loss_gen: 1.8033027648925781 (1.7753884443422643) | > loss_kl: 2.2942492961883545 (2.4502132694895673) | > loss_feat: 2.129857301712036 (2.0602536463155974) | > loss_mel: 18.66592025756836 (18.281537730519364) | > loss_duration: 1.6504976749420166 (1.6295413622042028) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.543827056884766 (26.196934583710462) | > grad_norm_1: tensor(406.0226, device='cuda:0') (tensor(277.9958, device='cuda:0')) | > current_lr_0: 0.00019655480151022865 | > current_lr_1: 0.00019655480151022865 | > step_time: 0.4548 (0.43803603474686786) | > loader_time: 0.0095 (0.00735998735195253) --> EVAL PERFORMANCE | > avg_loader_time: 0.008578535536645164 (-1.5315875201158194e-05) | > avg_step_time: 0.4532583196398238 (-0.0019380206793126464) | > avg_loss_disc: 2.83944872399451 (-0.00432913068314722) | > avg_loss_disc_real_0: 0.2263940239036587 (-0.0012520457237538973) | > avg_loss_disc_real_1: 0.23612549523232687 (-0.001838472947268438) | > avg_loss_disc_real_2: 0.2416061670847342 (-1.5088370148574892e-05) | > avg_loss_disc_real_3: 0.24212108531468352 (-0.000727901996021546) | > avg_loss_disc_real_4: 0.24519769469617117 (+0.0004001952392954089) | > avg_loss_disc_real_5: 0.24739777454188172 (+0.0011834025802746617) | > avg_loss_0: 2.83944872399451 (-0.00432913068314722) | > avg_grad_norm_0: tensor(10.7113, device='cuda:0') (tensor(-1.7091, device='cuda:0')) | > avg_loss_gen: 1.7609443563810538 (-0.0021728529057027934) | > avg_loss_kl: 2.4117427443114803 (+0.0880213253934623) | > avg_loss_feat: 1.998116112091171 (+0.0025297050744697014) | > avg_loss_mel: 18.236038288600007 (+0.5490300890425495) | > avg_loss_duration: 1.6476065978197987 (+0.005145942661124447) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.054448141178614 (+0.6425542428459927) | > avg_grad_norm_1: tensor(284.3350, device='cuda:0') (tensor(-10.6453, device='cuda:0'))  > EPOCH: 140/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:37:37)   --> TIME: 2025-02-25 09:37:48 -- STEP: 19/72 -- GLOBAL_STEP: 1010100 | > loss_disc: 2.779688596725464 (2.824057842555799) | > loss_disc_real_0: 0.2224903255701065 (0.21973215514107755) | > loss_disc_real_1: 0.20119348168373108 (0.23479583153599187) | > loss_disc_real_2: 0.20269738137722015 (0.24266540768899417) | > loss_disc_real_3: 0.22997638583183289 (0.2407963785685991) | > loss_disc_real_4: 0.2265619933605194 (0.248397622453539) | > loss_disc_real_5: 0.2432398647069931 (0.2508070908094707) | > loss_0: 2.779688596725464 (2.824057842555799) | > grad_norm_0: tensor(4.5208, device='cuda:0') (tensor(8.2532, device='cuda:0')) | > loss_gen: 1.7234854698181152 (1.7652383289839093) | > loss_kl: 2.330142021179199 (2.3460394583250346) | > loss_feat: 2.206324577331543 (2.034228713888871) | > loss_mel: 17.95482063293457 (18.29882049560547) | > loss_duration: 1.585099458694458 (1.6139885814566362) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.79987335205078 (26.05831608019377) | > grad_norm_1: tensor(159.8241, device='cuda:0') (tensor(255.8688, device='cuda:0')) | > current_lr_0: 0.00019653023216003985 | > current_lr_1: 0.00019653023216003985 | > step_time: 0.4477 (0.4373303965518349) | > loader_time: 0.006 (0.006625012347572728)  --> TIME: 2025-02-25 09:38:12 -- STEP: 69/72 -- GLOBAL_STEP: 1010150 | > loss_disc: 2.8586344718933105 (2.838630303092624) | > loss_disc_real_0: 0.33603331446647644 (0.2257316226976505) | > loss_disc_real_1: 0.21966980397701263 (0.23483603458473648) | > loss_disc_real_2: 0.23904135823249817 (0.24042455549689307) | > loss_disc_real_3: 0.22987304627895355 (0.24061219847720602) | > loss_disc_real_4: 0.23786242306232452 (0.2486011429109435) | > loss_disc_real_5: 0.2277042716741562 (0.25059030345384625) | > loss_0: 2.8586344718933105 (2.838630303092624) | > grad_norm_0: tensor(13.1012, device='cuda:0') (tensor(11.2295, device='cuda:0')) | > loss_gen: 1.6951628923416138 (1.7693251405937085) | > loss_kl: 2.3010900020599365 (2.3545374455659287) | > loss_feat: 1.8806017637252808 (2.01293310738992) | > loss_mel: 19.05708122253418 (18.317525117293652) | > loss_duration: 1.7266881465911865 (1.6419776401657988) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.660625457763672 (26.096298577128977) | > grad_norm_1: tensor(149.5344, device='cuda:0') (tensor(309.3749, device='cuda:0')) | > current_lr_0: 0.00019653023216003985 | > current_lr_1: 0.00019653023216003985 | > step_time: 0.5351 (0.4548359953838846) | > loader_time: 0.0131 (0.008504283600959225) --> EVAL PERFORMANCE | > avg_loader_time: 0.008562568207861672 (-1.5967328783492163e-05) | > avg_step_time: 0.45690805475476765 (+0.0036497351149438195) | > avg_loss_disc: 2.838996400295848 (-0.0004523236986622159) | > avg_loss_disc_real_0: 0.22457660356877554 (-0.001817420334883163) | > avg_loss_disc_real_1: 0.23528115430348356 (-0.0008443409288433068) | > avg_loss_disc_real_2: 0.24085692053949329 (-0.0007492465452409169) | > avg_loss_disc_real_3: 0.2409626690434738 (-0.0011584162712097168) | > avg_loss_disc_real_4: 0.24829318993528124 (+0.0030954952391100687) | > avg_loss_disc_real_5: 0.2502590988723325 (+0.0028613243304507763) | > avg_loss_0: 2.838996400295848 (-0.0004523236986622159) | > avg_grad_norm_0: tensor(11.1621, device='cuda:0') (+tensor(0.4508, device='cuda:0')) | > avg_loss_gen: 1.768623969924282 (+0.007679613543228214) | > avg_loss_kl: 2.3502703183133837 (-0.06147242599809655) | > avg_loss_feat: 2.0027320771150188 (+0.004615965023847757) | > avg_loss_mel: 18.295438954527953 (+0.059400665927945795) | > avg_loss_duration: 1.641865053647001 (-0.005741544172797708) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.058930491057918 (+0.0044823498793036265) | > avg_grad_norm_1: tensor(309.1815, device='cuda:0') (+tensor(24.8465, device='cuda:0'))  > EPOCH: 141/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:38:15)   --> TIME: 2025-02-25 09:38:38 -- STEP: 47/72 -- GLOBAL_STEP: 1010200 | > loss_disc: 2.9035751819610596 (2.852880721396588) | > loss_disc_real_0: 0.2309970110654831 (0.22038371353707414) | > loss_disc_real_1: 0.2571257948875427 (0.2401981252305051) | > loss_disc_real_2: 0.2732023000717163 (0.24314691348278777) | > loss_disc_real_3: 0.23913517594337463 (0.2442061444546314) | > loss_disc_real_4: 0.24916984140872955 (0.24681487584367712) | > loss_disc_real_5: 0.25013959407806396 (0.24950894840220186) | > loss_0: 2.9035751819610596 (2.852880721396588) | > grad_norm_0: tensor(4.4666, device='cuda:0') (tensor(9.0127, device='cuda:0')) | > loss_gen: 1.6550525426864624 (1.7443702626735607) | > loss_kl: 2.4206154346466064 (2.4135262103790938) | > loss_feat: 1.8771312236785889 (1.9267804597286469) | > loss_mel: 18.21466636657715 (18.128917004199742) | > loss_duration: 1.661529779434204 (1.6359694993242304) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.828994750976562 (25.849563476887155) | > grad_norm_1: tensor(184.5683, device='cuda:0') (tensor(252.5371, device='cuda:0')) | > current_lr_0: 0.00019650566588101984 | > current_lr_1: 0.00019650566588101984 | > step_time: 0.4543 (0.44423914970235623) | > loader_time: 0.0086 (0.007453806856845288) --> EVAL PERFORMANCE | > avg_loader_time: 0.008490246786198146 (-7.232142166352547e-05) | > avg_step_time: 0.4568031002098405 (-0.0001049545449271494) | > avg_loss_disc: 2.8395100210753963 (+0.0005136207795484182) | > avg_loss_disc_real_0: 0.22258547662009656 (-0.0019911269486789784) | > avg_loss_disc_real_1: 0.23626261110037144 (+0.0009814567968878807) | > avg_loss_disc_real_2: 0.24057313778870543 (-0.00028378275078785875) | > avg_loss_disc_real_3: 0.24128282972624604 (+0.0003201606827722414) | > avg_loss_disc_real_4: 0.2459980610390784 (-0.0022951288962028416) | > avg_loss_disc_real_5: 0.24827686204037197 (-0.0019822368319605266) | > avg_loss_0: 2.8395100210753963 (+0.0005136207795484182) | > avg_grad_norm_0: tensor(11.5844, device='cuda:0') (+tensor(0.4223, device='cuda:0')) | > avg_loss_gen: 1.7503437794430154 (-0.01828019048126661) | > avg_loss_kl: 2.4013978897685746 (+0.05112757145519087) | > avg_loss_feat: 1.9387860264576657 (-0.06394605065735304) | > avg_loss_mel: 18.04876811067823 (-0.24667084384972426) | > avg_loss_duration: 1.650342026226957 (+0.008476972579956055) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.78963790141361 (-0.2692925896443086) | > avg_grad_norm_1: tensor(278.8854, device='cuda:0') (tensor(-30.2960, device='cuda:0'))  > EPOCH: 142/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:38:52)   --> TIME: 2025-02-25 09:39:05 -- STEP: 25/72 -- GLOBAL_STEP: 1010250 | > loss_disc: 2.738959550857544 (2.81833664894104) | > loss_disc_real_0: 0.2839054763317108 (0.22323533654212951) | > loss_disc_real_1: 0.22491750121116638 (0.23794742166996002) | > loss_disc_real_2: 0.23825855553150177 (0.23574246287345887) | > loss_disc_real_3: 0.26045292615890503 (0.2437783294916153) | > loss_disc_real_4: 0.24756449460983276 (0.2509370422363282) | > loss_disc_real_5: 0.24368177354335785 (0.2537623572349548) | > loss_0: 2.738959550857544 (2.81833664894104) | > grad_norm_0: tensor(16.2612, device='cuda:0') (tensor(10.4837, device='cuda:0')) | > loss_gen: 1.9073086977005005 (1.8001335954666138) | > loss_kl: 2.400804042816162 (2.3763937187194824) | > loss_feat: 2.1897573471069336 (2.0778820705413814) | > loss_mel: 17.606090545654297 (18.1616593170166) | > loss_duration: 1.6168882846832275 (1.6227320861816406) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.720848083496094 (26.038800506591798) | > grad_norm_1: tensor(402.4161, device='cuda:0') (tensor(290.8610, device='cuda:0')) | > current_lr_0: 0.0001964811026727847 | > current_lr_1: 0.0001964811026727847 | > step_time: 0.441 (0.43381868362426756) | > loader_time: 0.0063 (0.0068021869659423825) --> EVAL PERFORMANCE | > avg_loader_time: 0.008530264169397487 (+4.0017383199340145e-05) | > avg_step_time: 0.4542369439568318 (-0.002566156253008711) | > avg_loss_disc: 2.8164010820254477 (-0.023108939049948596) | > avg_loss_disc_real_0: 0.2186406372298657 (-0.003944839390230853) | > avg_loss_disc_real_1: 0.23617912619046763 (-8.348490990381374e-05) | > avg_loss_disc_real_2: 0.23892189457382954 (-0.0016512432148758915) | > avg_loss_disc_real_3: 0.24127738156788786 (-5.448158358178912e-06) | > avg_loss_disc_real_4: 0.24666596109598454 (+0.0006679000569061455) | > avg_loss_disc_real_5: 0.24830108104457319 (+2.42190042012147e-05) | > avg_loss_0: 2.8164010820254477 (-0.023108939049948596) | > avg_grad_norm_0: tensor(10.2127, device='cuda:0') (tensor(-1.3717, device='cuda:0')) | > avg_loss_gen: 1.7799520811564487 (+0.029608301713433205) | > avg_loss_kl: 2.392233254204333 (-0.009164635564241408) | > avg_loss_feat: 2.0357154943573637 (+0.09692946789969792) | > avg_loss_mel: 18.004490167322295 (-0.04427794335593305) | > avg_loss_duration: 1.6462868149851408 (-0.004055211241816181) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.85867782377861 (+0.06903992236500045) | > avg_grad_norm_1: tensor(295.7721, device='cuda:0') (+tensor(16.8866, device='cuda:0'))  > EPOCH: 143/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:39:29)   --> TIME: 2025-02-25 09:39:32 -- STEP: 3/72 -- GLOBAL_STEP: 1010300 | > loss_disc: 2.8365564346313477 (2.8274354139963784) | > loss_disc_real_0: 0.17730575799942017 (0.2666712502638499) | > loss_disc_real_1: 0.2462027221918106 (0.2488120198249817) | > loss_disc_real_2: 0.30043813586235046 (0.29602187871932983) | > loss_disc_real_3: 0.298664391040802 (0.2743614912033081) | > loss_disc_real_4: 0.26299476623535156 (0.2558912436167399) | > loss_disc_real_5: 0.25163641571998596 (0.25425058603286743) | > loss_0: 2.8365564346313477 (2.8274354139963784) | > grad_norm_0: tensor(17.1408, device='cuda:0') (tensor(21.3373, device='cuda:0')) | > loss_gen: 1.8201134204864502 (1.9129010836283367) | > loss_kl: 2.6372978687286377 (2.4659144083658853) | > loss_feat: 2.0274875164031982 (2.0465876261393228) | > loss_mel: 18.250944137573242 (18.20174725850423) | > loss_duration: 1.6802500486373901 (1.6442373593648274) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.416093826293945 (26.271387736002605) | > grad_norm_1: tensor(228.2780, device='cuda:0') (tensor(276.7006, device='cuda:0')) | > current_lr_0: 0.00019645654253495058 | > current_lr_1: 0.00019645654253495058 | > step_time: 0.4322 (0.43720579147338867) | > loader_time: 0.0095 (0.007415374120076497)  --> TIME: 2025-02-25 09:39:55 -- STEP: 53/72 -- GLOBAL_STEP: 1010350 | > loss_disc: 2.8263187408447266 (2.8348351244656547) | > loss_disc_real_0: 0.22939810156822205 (0.2322918657424315) | > loss_disc_real_1: 0.2765735685825348 (0.23864962863472272) | > loss_disc_real_2: 0.21155676245689392 (0.24143946170806885) | > loss_disc_real_3: 0.22339192032814026 (0.243029644466796) | > loss_disc_real_4: 0.26696982979774475 (0.24728674371287507) | > loss_disc_real_5: 0.269650399684906 (0.2488015929483018) | > loss_0: 2.8263187408447266 (2.8348351244656547) | > grad_norm_0: tensor(5.2853, device='cuda:0') (tensor(12.3913, device='cuda:0')) | > loss_gen: 1.565402865409851 (1.78164375278185) | > loss_kl: 2.233142852783203 (2.463051696993269) | > loss_feat: 2.1637911796569824 (2.0507798127408283) | > loss_mel: 17.338088989257812 (18.126453543609045) | > loss_duration: 1.6944921016693115 (1.6512514780152519) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.994918823242188 (26.073180324626417) | > grad_norm_1: tensor(86.6432, device='cuda:0') (tensor(241.2378, device='cuda:0')) | > current_lr_0: 0.00019645654253495058 | > current_lr_1: 0.00019645654253495058 | > step_time: 0.4508 (0.4527236245713144) | > loader_time: 0.0091 (0.007802486419677734) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630987624047508 (+0.00010072345465002124) | > avg_step_time: 0.4625888542390206 (+0.00835191028218879) | > avg_loss_disc: 2.8365266255929438 (+0.020125543567496074) | > avg_loss_disc_real_0: 0.23489326511470365 (+0.016252627884837945) | > avg_loss_disc_real_1: 0.2450662265780946 (+0.008887100387626967) | > avg_loss_disc_real_2: 0.2417766426650571 (+0.0028547480912275713) | > avg_loss_disc_real_3: 0.24136740661842723 (+9.002505053937182e-05) | > avg_loss_disc_real_4: 0.24466212702469087 (-0.0020038340712936753) | > avg_loss_disc_real_5: 0.24710624381689958 (-0.0011948372276736063) | > avg_loss_0: 2.8365266255929438 (+0.020125543567496074) | > avg_grad_norm_0: tensor(13.3221, device='cuda:0') (+tensor(3.1094, device='cuda:0')) | > avg_loss_gen: 1.786040245647162 (+0.006088164490713366) | > avg_loss_kl: 2.4391926644553594 (+0.046959410251026235) | > avg_loss_feat: 2.0340475868171364 (-0.0016679075402272758) | > avg_loss_mel: 18.003936458641373 (-0.0005537086809219716) | > avg_loss_duration: 1.6567887185325085 (+0.010501903547367686) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.920005664019516 (+0.06132784024090654) | > avg_grad_norm_1: tensor(240.2685, device='cuda:0') (tensor(-55.5036, device='cuda:0'))  > EPOCH: 144/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:40:06)   --> TIME: 2025-02-25 09:40:22 -- STEP: 31/72 -- GLOBAL_STEP: 1010400 | > loss_disc: 2.6696574687957764 (2.8280842458048174) | > loss_disc_real_0: 0.27391138672828674 (0.2371906445872399) | > loss_disc_real_1: 0.2612936496734619 (0.23744315149322634) | > loss_disc_real_2: 0.22034496068954468 (0.23615454281530074) | > loss_disc_real_3: 0.19885338842868805 (0.23982675710032064) | > loss_disc_real_4: 0.23966744542121887 (0.24376317185740318) | > loss_disc_real_5: 0.2537757158279419 (0.2425025446761039) | > loss_0: 2.6696574687957764 (2.8280842458048174) | > grad_norm_0: tensor(9.4004, device='cuda:0') (tensor(10.8981, device='cuda:0')) | > loss_gen: 1.677656888961792 (1.7874176079227078) | > loss_kl: 2.63149356842041 (2.392948527489939) | > loss_feat: 2.3765530586242676 (2.1385577763280557) | > loss_mel: 18.62601089477539 (18.26168386397823) | > loss_duration: 1.6559698581695557 (1.6209559286794355) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.967683792114258 (26.201563927435107) | > grad_norm_1: tensor(174.0485, device='cuda:0') (tensor(262.6663, device='cuda:0')) | > current_lr_0: 0.0001964319854671337 | > current_lr_1: 0.0001964319854671337 | > step_time: 0.4476 (0.4389396098352248) | > loader_time: 0.0084 (0.007111257122408959) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584778073807838 (-4.620955023966998e-05) | > avg_step_time: 0.4552733663102271 (-0.007315487928793496) | > avg_loss_disc: 2.8297721466548005 (-0.0067544789381432935) | > avg_loss_disc_real_0: 0.23074608988745113 (-0.004147175227252525) | > avg_loss_disc_real_1: 0.234386189302928 (-0.010680037275166587) | > avg_loss_disc_real_2: 0.23708503745811085 (-0.004691605206946253) | > avg_loss_disc_real_3: 0.24076723737615935 (-0.0006001692422678839) | > avg_loss_disc_real_4: 0.24491937626415575 (+0.00025724923946487954) | > avg_loss_disc_real_5: 0.2461816865793416 (-0.0009245572375579658) | > avg_loss_0: 2.8297721466548005 (-0.0067544789381432935) | > avg_grad_norm_0: tensor(11.9659, device='cuda:0') (tensor(-1.3562, device='cuda:0')) | > avg_loss_gen: 1.7678322808843263 (-0.018207964762835704) | > avg_loss_kl: 2.3729010434217863 (-0.06629162103357311) | > avg_loss_feat: 2.0562295090984284 (+0.02218192228129201) | > avg_loss_mel: 17.93449861230985 (-0.06943784633152461) | > avg_loss_duration: 1.643658868024047 (-0.013129850508461605) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.775120533687968 (-0.1448851303315486) | > avg_grad_norm_1: tensor(241.5215, device='cuda:0') (+tensor(1.2529, device='cuda:0'))  > EPOCH: 145/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:40:43)   --> TIME: 2025-02-25 09:40:49 -- STEP: 9/72 -- GLOBAL_STEP: 1010450 | > loss_disc: 2.8205394744873047 (2.8331111801995172) | > loss_disc_real_0: 0.21669265627861023 (0.2384082790878084) | > loss_disc_real_1: 0.17334912717342377 (0.22798186209466723) | > loss_disc_real_2: 0.2742358148097992 (0.24661015470822653) | > loss_disc_real_3: 0.2804219126701355 (0.24070093035697937) | > loss_disc_real_4: 0.2943693995475769 (0.241586082511478) | > loss_disc_real_5: 0.2790068984031677 (0.23527255323198107) | > loss_0: 2.8205394744873047 (2.8331111801995172) | > grad_norm_0: tensor(5.9243, device='cuda:0') (tensor(13.8994, device='cuda:0')) | > loss_gen: 1.8534494638442993 (1.789664665857951) | > loss_kl: 2.2976324558258057 (2.3491935994890003) | > loss_feat: 2.376591682434082 (2.1945354408688016) | > loss_mel: 20.086111068725586 (18.89566887749566) | > loss_duration: 1.6385650634765625 (1.6309241718716092) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.252347946166992 (26.85998598734538) | > grad_norm_1: tensor(324.6746, device='cuda:0') (tensor(306.0754, device='cuda:0')) | > current_lr_0: 0.0001964074314689503 | > current_lr_1: 0.0001964074314689503 | > step_time: 0.4277 (0.43236464924282497) | > loader_time: 0.0056 (0.005402088165283203)  --> TIME: 2025-02-25 09:41:13 -- STEP: 59/72 -- GLOBAL_STEP: 1010500 | > loss_disc: 2.8473381996154785 (2.8159300836466126) | > loss_disc_real_0: 0.2967348098754883 (0.2236987260186066) | > loss_disc_real_1: 0.2551575303077698 (0.23944337287191617) | > loss_disc_real_2: 0.2043851613998413 (0.23896715853173853) | > loss_disc_real_3: 0.23219923675060272 (0.23957321552907007) | > loss_disc_real_4: 0.2448875457048416 (0.24455170737484755) | > loss_disc_real_5: 0.2504802942276001 (0.24571346782021602) | > loss_0: 2.8473381996154785 (2.8159300836466126) | > grad_norm_0: tensor(9.5860, device='cuda:0') (tensor(11.4528, device='cuda:0')) | > loss_gen: 1.6429920196533203 (1.7883444236496748) | > loss_kl: 2.4865283966064453 (2.316159668615309) | > loss_feat: 1.9738479852676392 (2.0796207007715255) | > loss_mel: 17.582658767700195 (17.991236217951354) | > loss_duration: 1.6937984228134155 (1.637631375910872) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.379825592041016 (25.812992095947266) | > grad_norm_1: tensor(174.8284, device='cuda:0') (tensor(280.5393, device='cuda:0')) | > current_lr_0: 0.0001964074314689503 | > current_lr_1: 0.0001964074314689503 | > step_time: 0.4661 (0.44840421919095313) | > loader_time: 0.0098 (0.007888212042339778) --> EVAL PERFORMANCE | > avg_loader_time: 0.008483963952937596 (-0.00010081412087024205) | > avg_step_time: 0.457748789182851 (+0.0024754228726239225) | > avg_loss_disc: 2.817640999673118 (-0.012131146981682495) | > avg_loss_disc_real_0: 0.22382073534626357 (-0.006925354541187562) | > avg_loss_disc_real_1: 0.23856024129289977 (+0.004174051989971755) | > avg_loss_disc_real_2: 0.23884888916787966 (+0.00176385170976881) | > avg_loss_disc_real_3: 0.23806017385402195 (-0.0027070635221373984) | > avg_loss_disc_real_4: 0.24377333320362468 (-0.0011460430605310656) | > avg_loss_disc_real_5: 0.24562461821126266 (-0.0005570683680789501) | > avg_loss_0: 2.817640999673118 (-0.012131146981682495) | > avg_grad_norm_0: tensor(10.9398, device='cuda:0') (tensor(-1.0261, device='cuda:0')) | > avg_loss_gen: 1.7820266871385169 (+0.01419440625419055) | > avg_loss_kl: 2.3321556608441854 (-0.040745382577600964) | > avg_loss_feat: 2.0645852978800376 (+0.008355788781609252) | > avg_loss_mel: 17.95706534049879 (+0.022566728188941454) | > avg_loss_duration: 1.6443008939984818 (+0.0006420259744348566) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.780133636904434 (+0.005013103216466419) | > avg_grad_norm_1: tensor(277.5061, device='cuda:0') (+tensor(35.9847, device='cuda:0'))  > EPOCH: 146/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:41:20)   --> TIME: 2025-02-25 09:41:39 -- STEP: 37/72 -- GLOBAL_STEP: 1010550 | > loss_disc: 2.8619749546051025 (2.8276899376431026) | > loss_disc_real_0: 0.2243383228778839 (0.2247913663049002) | > loss_disc_real_1: 0.23599094152450562 (0.23134394553867546) | > loss_disc_real_2: 0.23408636450767517 (0.2362339142206553) | > loss_disc_real_3: 0.23573608696460724 (0.24160574497403325) | > loss_disc_real_4: 0.251048743724823 (0.24666712815697128) | > loss_disc_real_5: 0.2338012307882309 (0.24709218700189847) | > loss_0: 2.8619749546051025 (2.8276899376431026) | > grad_norm_0: tensor(16.1581, device='cuda:0') (tensor(13.6628, device='cuda:0')) | > loss_gen: 1.6000946760177612 (1.7659419356165706) | > loss_kl: 2.3684988021850586 (2.4020174129589185) | > loss_feat: 1.8160909414291382 (2.1195788737889885) | > loss_mel: 16.56398582458496 (17.908365301183753) | > loss_duration: 1.6712167263031006 (1.641590901323267) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.019886016845703 (25.837494360434043) | > grad_norm_1: tensor(226.8522, device='cuda:0') (tensor(255.7561, device='cuda:0')) | > current_lr_0: 0.00019638288054001668 | > current_lr_1: 0.00019638288054001668 | > step_time: 0.4463 (0.43728933463225494) | > loader_time: 0.0082 (0.007179240922670107) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857494246791786 (+9.097851498026353e-05) | > avg_step_time: 0.46147871689057685 (+0.003729927707725844) | > avg_loss_disc: 2.8238555444798 (+0.006214544806681932) | > avg_loss_disc_real_0: 0.22641790553297794 (+0.002597170186714376) | > avg_loss_disc_real_1: 0.2352966999923679 (-0.003263541300531858) | > avg_loss_disc_real_2: 0.24056486365660815 (+0.0017159744887284833) | > avg_loss_disc_real_3: 0.24421011729979178 (+0.006149943445769829) | > avg_loss_disc_real_4: 0.2460116878361769 (+0.0022383546325522308) | > avg_loss_disc_real_5: 0.24546920224814348 (-0.0001554159631191876) | > avg_loss_0: 2.8238555444798 (+0.006214544806681932) | > avg_grad_norm_0: tensor(13.2826, device='cuda:0') (+tensor(2.3428, device='cuda:0')) | > avg_loss_gen: 1.7802042037668362 (-0.001822483371680672) | > avg_loss_kl: 2.390780553011827 (+0.05862489216764155) | > avg_loss_feat: 2.0768280600158255 (+0.012242762135787899) | > avg_loss_mel: 17.95851218532508 (+0.0014468448262903166) | > avg_loss_duration: 1.6531295843527352 (+0.008828690354253377) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.859454437040945 (+0.07932080013651088) | > avg_grad_norm_1: tensor(233.5016, device='cuda:0') (tensor(-44.0045, device='cuda:0'))  > EPOCH: 147/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:41:58)   --> TIME: 2025-02-25 09:42:06 -- STEP: 15/72 -- GLOBAL_STEP: 1010600 | > loss_disc: 2.85044527053833 (2.8197267373402912) | > loss_disc_real_0: 0.12157441675662994 (0.22147135933240256) | > loss_disc_real_1: 0.17343002557754517 (0.2287265509366989) | > loss_disc_real_2: 0.22838032245635986 (0.23663777510325115) | > loss_disc_real_3: 0.2420026659965515 (0.23659908175468444) | > loss_disc_real_4: 0.22665002942085266 (0.24371779958407083) | > loss_disc_real_5: 0.24608327448368073 (0.24560099045435588) | > loss_0: 2.85044527053833 (2.8197267373402912) | > grad_norm_0: tensor(26.6791, device='cuda:0') (tensor(11.6313, device='cuda:0')) | > loss_gen: 1.6140962839126587 (1.7547003587086996) | > loss_kl: 2.2382113933563232 (2.304805135726929) | > loss_feat: 2.321953296661377 (2.1208300987879434) | > loss_mel: 18.377573013305664 (17.93372980753581) | > loss_duration: 1.6027817726135254 (1.6013170719146728) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.154613494873047 (25.715382639567057) | > grad_norm_1: tensor(206.5014, device='cuda:0') (tensor(306.0315, device='cuda:0')) | > current_lr_0: 0.00019635833267994917 | > current_lr_1: 0.00019635833267994917 | > step_time: 0.4279 (0.43873289426167805) | > loader_time: 0.0062 (0.00653840700785319)  --> TIME: 2025-02-25 09:42:30 -- STEP: 65/72 -- GLOBAL_STEP: 1010650 | > loss_disc: 2.8731184005737305 (2.85123034623953) | > loss_disc_real_0: 0.2071157544851303 (0.23440717126314456) | > loss_disc_real_1: 0.18387185037136078 (0.2391801600272839) | > loss_disc_real_2: 0.2208959460258484 (0.24428966893599582) | > loss_disc_real_3: 0.22420239448547363 (0.243908236347712) | > loss_disc_real_4: 0.22395442426204681 (0.2446652114391327) | > loss_disc_real_5: 0.2390446960926056 (0.24618737399578094) | > loss_0: 2.8731184005737305 (2.85123034623953) | > grad_norm_0: tensor(6.9244, device='cuda:0') (tensor(14.2758, device='cuda:0')) | > loss_gen: 1.7762360572814941 (1.776048390681927) | > loss_kl: 2.5144572257995605 (2.3723370405343864) | > loss_feat: 1.9168012142181396 (2.0343951371999887) | > loss_mel: 18.55462074279785 (18.043778580885668) | > loss_duration: 1.6830344200134277 (1.6378741117624136) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.445148468017578 (25.86443322988657) | > grad_norm_1: tensor(274.6646, device='cuda:0') (tensor(277.3019, device='cuda:0')) | > current_lr_0: 0.00019635833267994917 | > current_lr_1: 0.00019635833267994917 | > step_time: 0.5067 (0.4514339043543889) | > loader_time: 0.0115 (0.008282881516676685) --> EVAL PERFORMANCE | > avg_loader_time: 0.008550586834759781 (-2.435563315807826e-05) | > avg_step_time: 0.4561952402893926 (-0.005283476601184223) | > avg_loss_disc: 2.8488397228885702 (+0.02498417840877032) | > avg_loss_disc_real_0: 0.23454337483140783 (+0.008125469298429888) | > avg_loss_disc_real_1: 0.2393976672434471 (+0.004100967251079196) | > avg_loss_disc_real_2: 0.2444301190930353 (+0.0038652554364271563) | > avg_loss_disc_real_3: 0.2436845191347767 (-0.0005255981650150732) | > avg_loss_disc_real_4: 0.24481228623591678 (-0.001199401600260136) | > avg_loss_disc_real_5: 0.24668210493007175 (+0.0012129026819282795) | > avg_loss_0: 2.8488397228885702 (+0.02498417840877032) | > avg_grad_norm_0: tensor(13.9931, device='cuda:0') (+tensor(0.7105, device='cuda:0')) | > avg_loss_gen: 1.7761132247011429 (-0.004090979065693334) | > avg_loss_kl: 2.3748945786919387 (-0.015885974319888252) | > avg_loss_feat: 2.029446239202795 (-0.04738182081303055) | > avg_loss_mel: 18.109104640047317 (+0.15059245472223637) | > avg_loss_duration: 1.6417376054844386 (-0.01139197886829657) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.931296147091288 (+0.07184171005034301) | > avg_grad_norm_1: tensor(284.4741, device='cuda:0') (+tensor(50.9724, device='cuda:0'))  > EPOCH: 148/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:42:35)   --> TIME: 2025-02-25 09:42:56 -- STEP: 43/72 -- GLOBAL_STEP: 1010700 | > loss_disc: 2.9737906455993652 (2.825420706771141) | > loss_disc_real_0: 0.2564440667629242 (0.22023447966852852) | > loss_disc_real_1: 0.3463699519634247 (0.2392010962547258) | > loss_disc_real_2: 0.24803727865219116 (0.23917972660341927) | > loss_disc_real_3: 0.24229833483695984 (0.24003836720488791) | > loss_disc_real_4: 0.28079986572265625 (0.24692154277202694) | > loss_disc_real_5: 0.2557365298271179 (0.24700624790302542) | > loss_0: 2.9737906455993652 (2.825420706771141) | > grad_norm_0: tensor(12.0845, device='cuda:0') (tensor(11.1768, device='cuda:0')) | > loss_gen: 1.6518982648849487 (1.771976071734761) | > loss_kl: 2.330456256866455 (2.311952845994816) | > loss_feat: 1.668947696685791 (2.0537294066229523) | > loss_mel: 15.429706573486328 (17.712497910787896) | > loss_duration: 1.646453857421875 (1.622582565906436) | > amp_scaler: 256.0 (256.0) | > loss_1: 22.727462768554688 (25.47273884263149) | > grad_norm_1: tensor(165.1361, device='cuda:0') (tensor(299.1577, device='cuda:0')) | > current_lr_0: 0.00019633378788836418 | > current_lr_1: 0.00019633378788836418 | > step_time: 0.459 (0.4404757854550384) | > loader_time: 0.0097 (0.007298441820366438) --> EVAL PERFORMANCE | > avg_loader_time: 0.008482795366099184 (-6.779146866059751e-05) | > avg_step_time: 0.45494354610711757 (-0.0012516941822750538) | > avg_loss_disc: 2.844657162545433 (-0.004182560343137354) | > avg_loss_disc_real_0: 0.22496318649238267 (-0.009580188339025159) | > avg_loss_disc_real_1: 0.24050020290092683 (+0.0011025356574797252) | > avg_loss_disc_real_2: 0.23942218741900484 (-0.005007931674030464) | > avg_loss_disc_real_3: 0.2425323078330134 (-0.0011522113017632951) | > avg_loss_disc_real_4: 0.2467009509113473 (+0.0018886646754305292) | > avg_loss_disc_real_5: 0.24803868691686173 (+0.0013565819867899787) | > avg_loss_0: 2.844657162545433 (-0.004182560343137354) | > avg_grad_norm_0: tensor(12.0876, device='cuda:0') (tensor(-1.9055, device='cuda:0')) | > avg_loss_gen: 1.7715604808968557 (-0.004552743804287163) | > avg_loss_kl: 2.3735800863991323 (-0.0013144922928063352) | > avg_loss_feat: 2.022887590905309 (-0.0065586482974859805) | > avg_loss_mel: 18.01450526546424 (-0.09459937458307621) | > avg_loss_duration: 1.6427639907514546 (+0.001026385267016039) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.825297503404208 (-0.10599864368708012) | > avg_grad_norm_1: tensor(269.9528, device='cuda:0') (tensor(-14.5212, device='cuda:0'))  > EPOCH: 149/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:43:12)   --> TIME: 2025-02-25 09:43:23 -- STEP: 21/72 -- GLOBAL_STEP: 1010750 | > loss_disc: 2.8246986865997314 (2.841654595874605) | > loss_disc_real_0: 0.43459105491638184 (0.2383847442411241) | > loss_disc_real_1: 0.2516043186187744 (0.23664960123243786) | > loss_disc_real_2: 0.2548680007457733 (0.24855164473965055) | > loss_disc_real_3: 0.25854888558387756 (0.24287437115396773) | > loss_disc_real_4: 0.25113561749458313 (0.2436929259981428) | > loss_disc_real_5: 0.2442716360092163 (0.24856554184641158) | > loss_0: 2.8246986865997314 (2.841654595874605) | > grad_norm_0: tensor(30.2531, device='cuda:0') (tensor(14.5352, device='cuda:0')) | > loss_gen: 2.0640151500701904 (1.791391894930885) | > loss_kl: 2.1487395763397217 (2.2560235886346724) | > loss_feat: 2.2272300720214844 (2.0061972368331182) | > loss_mel: 17.34714698791504 (18.091169720604306) | > loss_duration: 1.6440953016281128 (1.6172302280153548) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.43122673034668 (25.762012663341704) | > grad_norm_1: tensor(308.6335, device='cuda:0') (tensor(242.0432, device='cuda:0')) | > current_lr_0: 0.00019630924616487812 | > current_lr_1: 0.00019630924616487812 | > step_time: 0.4286 (0.43015059970674063) | > loader_time: 0.0061 (0.006621042887369792)  --> TIME: 2025-02-25 09:43:47 -- STEP: 71/72 -- GLOBAL_STEP: 1010800 | > loss_disc: 2.781947135925293 (2.8341046084820385) | > loss_disc_real_0: 0.12489339709281921 (0.2256529562070336) | > loss_disc_real_1: 0.19942432641983032 (0.2363949283747606) | > loss_disc_real_2: 0.21525302529335022 (0.24243469074578353) | > loss_disc_real_3: 0.22207577526569366 (0.2414757749983962) | > loss_disc_real_4: 0.20920497179031372 (0.24645635212810946) | > loss_disc_real_5: 0.21335144340991974 (0.24815125847366495) | > loss_0: 2.781947135925293 (2.8341046084820385) | > grad_norm_0: tensor(19.2235, device='cuda:0') (tensor(12.5266, device='cuda:0')) | > loss_gen: 1.632520318031311 (1.7607527366826232) | > loss_kl: 2.072141647338867 (2.3113920755789312) | > loss_feat: 2.002027750015259 (1.9989040032239027) | > loss_mel: 17.563627243041992 (17.93336512337268) | > loss_duration: 1.645827293395996 (1.6432886912789144) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.91614532470703 (25.647702526038803) | > grad_norm_1: tensor(402.8837, device='cuda:0') (tensor(278.6466, device='cuda:0')) | > current_lr_0: 0.00019630924616487812 | > current_lr_1: 0.00019630924616487812 | > step_time: 0.5087 (0.4615211016695264) | > loader_time: 0.008 (0.008522272109985352) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522272109985352 (+3.9476743886168036e-05) | > avg_step_time: 0.4615211016695264 (+0.006577555562408854) | > avg_loss_disc: 2.8341046084820385 (-0.010552554063394393) | > avg_loss_disc_real_0: 0.2256529562070336 (+0.0006897697146509285) | > avg_loss_disc_real_1: 0.2363949283747606 (-0.0041052745261662416) | > avg_loss_disc_real_2: 0.24243469074578353 (+0.003012503326778687) | > avg_loss_disc_real_3: 0.2414757749983962 (-0.0010565328346172076) | > avg_loss_disc_real_4: 0.24645635212810946 (-0.00024459878323784823) | > avg_loss_disc_real_5: 0.24815125847366495 (+0.0001125715568032204) | > avg_loss_0: 2.8341046084820385 (-0.010552554063394393) | > avg_grad_norm_0: tensor(12.5266, device='cuda:0') (+tensor(0.4390, device='cuda:0')) | > avg_loss_gen: 1.7607527366826232 (-0.010807744214232473) | > avg_loss_kl: 2.3113920755789312 (-0.06218801082020109) | > avg_loss_feat: 1.9989040032239027 (-0.023983587681406293) | > avg_loss_mel: 17.93336512337268 (-0.0811401420915594) | > avg_loss_duration: 1.6432886912789144 (+0.0005247005274597871) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.647702526038803 (-0.17759497736540553) | > avg_grad_norm_1: tensor(278.6466, device='cuda:0') (+tensor(8.6938, device='cuda:0'))  > EPOCH: 150/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:43:49)   --> TIME: 2025-02-25 09:44:13 -- STEP: 49/72 -- GLOBAL_STEP: 1010850 | > loss_disc: 2.811985969543457 (2.8251526793655084) | > loss_disc_real_0: 0.17055697739124298 (0.22867421623395415) | > loss_disc_real_1: 0.24201565980911255 (0.2415617917265211) | > loss_disc_real_2: 0.2508834898471832 (0.2431392614938775) | > loss_disc_real_3: 0.22968356311321259 (0.24427721421329343) | > loss_disc_real_4: 0.25320664048194885 (0.24708905785667654) | > loss_disc_real_5: 0.25729888677597046 (0.2500040798771138) | > loss_0: 2.811985969543457 (2.8251526793655084) | > grad_norm_0: tensor(19.7644, device='cuda:0') (tensor(12.5011, device='cuda:0')) | > loss_gen: 1.606200098991394 (1.7781446734253241) | > loss_kl: 2.131453275680542 (2.300817971326867) | > loss_feat: 1.6531095504760742 (1.9845195814054837) | > loss_mel: 17.472288131713867 (17.683305273250653) | > loss_duration: 1.7205212116241455 (1.6285137935560576) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.583572387695312 (25.37530128323302) | > grad_norm_1: tensor(335.1149, device='cuda:0') (tensor(250.8007, device='cuda:0')) | > current_lr_0: 0.0001962847075091075 | > current_lr_1: 0.0001962847075091075 | > step_time: 0.4486 (0.4443554245695776) | > loader_time: 0.0095 (0.007548959887757593) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568840967097749 (+4.6568857112396986e-05) | > avg_step_time: 0.45653400958423884 (-0.004987092085287581) | > avg_loss_disc: 2.8225057158671634 (-0.011598892614875123) | > avg_loss_disc_real_0: 0.22310867758703903 (-0.0025442786199945666) | > avg_loss_disc_real_1: 0.23954045163913512 (+0.0031455232643745334) | > avg_loss_disc_real_2: 0.2418505159062399 (-0.0005841748395436219) | > avg_loss_disc_real_3: 0.2443598727105369 (+0.0028840977121407096) | > avg_loss_disc_real_4: 0.24622060190623915 (-0.0002357502218703067) | > avg_loss_disc_real_5: 0.24870591138450193 (+0.000554652910836978) | > avg_loss_0: 2.8225057158671634 (-0.011598892614875123) | > avg_grad_norm_0: tensor(12.2049, device='cuda:0') (tensor(-0.3218, device='cuda:0')) | > avg_loss_gen: 1.773548124541699 (+0.012795387859075857) | > avg_loss_kl: 2.3283083539613534 (+0.016916278382422156) | > avg_loss_feat: 1.996005760112279 (-0.0028982431116237883) | > avg_loss_mel: 17.800424387757204 (-0.13294073561547748) | > avg_loss_duration: 1.6457552305409608 (+0.00246653926204643) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.54404180822238 (-0.10366071781642106) | > avg_grad_norm_1: tensor(266.9423, device='cuda:0') (tensor(-11.7043, device='cuda:0'))  > EPOCH: 151/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:44:26)   --> TIME: 2025-02-25 09:44:40 -- STEP: 27/72 -- GLOBAL_STEP: 1010900 | > loss_disc: 2.8049156665802 (2.808858385792485) | > loss_disc_real_0: 0.23101410269737244 (0.2245210282228611) | > loss_disc_real_1: 0.2844814658164978 (0.22943645108629157) | > loss_disc_real_2: 0.25440335273742676 (0.23515319051565947) | > loss_disc_real_3: 0.2271779328584671 (0.23977695515862218) | > loss_disc_real_4: 0.25059807300567627 (0.24388319861005853) | > loss_disc_real_5: 0.2446843683719635 (0.24933203171800683) | > loss_0: 2.8049156665802 (2.808858385792485) | > grad_norm_0: tensor(9.2422, device='cuda:0') (tensor(11.0557, device='cuda:0')) | > loss_gen: 1.8281140327453613 (1.7861174212561712) | > loss_kl: 2.193303108215332 (2.270468561737626) | > loss_feat: 2.2272326946258545 (2.106531929086756) | > loss_mel: 17.95346450805664 (17.892893120094588) | > loss_duration: 1.568488597869873 (1.6111498541302152) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.77060317993164 (25.667161023175275) | > grad_norm_1: tensor(175.9390, device='cuda:0') (tensor(288.5121, device='cuda:0')) | > current_lr_0: 0.00019626017192066886 | > current_lr_1: 0.00019626017192066886 | > step_time: 0.459 (0.43582722875807023) | > loader_time: 0.0084 (0.007017223923294632) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627861318453936 (+5.9020351356187106e-05) | > avg_step_time: 0.45561278034263936 (-0.000921229241599486) | > avg_loss_disc: 2.827693059410848 (+0.005187343543684442) | > avg_loss_disc_real_0: 0.22638248620738446 (+0.003273808620345431) | > avg_loss_disc_real_1: 0.2376672772454544 (-0.0018731743936807321) | > avg_loss_disc_real_2: 0.2383476711914573 (-0.003502844714782599) | > avg_loss_disc_real_3: 0.24133782004806356 (-0.0030220526624733512) | > avg_loss_disc_real_4: 0.24479207690332977 (-0.001428525002909381) | > avg_loss_disc_real_5: 0.2482478616523071 (-0.0004580497321948207) | > avg_loss_0: 2.827693059410848 (+0.005187343543684442) | > avg_grad_norm_0: tensor(11.2167, device='cuda:0') (tensor(-0.9881, device='cuda:0')) | > avg_loss_gen: 1.7783235009287444 (+0.0047753763870452826) | > avg_loss_kl: 2.319859306577225 (-0.008449047384128328) | > avg_loss_feat: 2.0425567929173867 (+0.04655103280510775) | > avg_loss_mel: 17.821915263860998 (+0.021490876103793966) | > avg_loss_duration: 1.6402100929072205 (-0.005545137633740316) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.602864896747427 (+0.058823088525045364) | > avg_grad_norm_1: tensor(292.0284, device='cuda:0') (+tensor(25.0861, device='cuda:0'))  > EPOCH: 152/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:45:03)   --> TIME: 2025-02-25 09:45:07 -- STEP: 5/72 -- GLOBAL_STEP: 1010950 | > loss_disc: 2.8182828426361084 (2.8176127910614013) | > loss_disc_real_0: 0.17652608454227448 (0.22567334175109863) | > loss_disc_real_1: 0.26407572627067566 (0.23830504715442657) | > loss_disc_real_2: 0.25375887751579285 (0.2220944344997406) | > loss_disc_real_3: 0.24917738139629364 (0.24332163333892823) | > loss_disc_real_4: 0.2730486989021301 (0.24932295680046082) | > loss_disc_real_5: 0.25843843817710876 (0.2439560979604721) | > loss_0: 2.8182828426361084 (2.8176127910614013) | > grad_norm_0: tensor(9.7040, device='cuda:0') (tensor(12.7217, device='cuda:0')) | > loss_gen: 1.9242634773254395 (1.7968483924865724) | > loss_kl: 2.0737881660461426 (2.2328908443450928) | > loss_feat: 1.9622071981430054 (2.0616636991500856) | > loss_mel: 17.224285125732422 (17.71948699951172) | > loss_duration: 1.5824241638183594 (1.611806869506836) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.7669677734375 (25.42269630432129) | > grad_norm_1: tensor(75.3408, device='cuda:0') (tensor(307.9156, device='cuda:0')) | > current_lr_0: 0.00019623563939917877 | > current_lr_1: 0.00019623563939917877 | > step_time: 0.4344 (0.4290469169616699) | > loader_time: 0.0058 (0.006422233581542969)  --> TIME: 2025-02-25 09:45:31 -- STEP: 55/72 -- GLOBAL_STEP: 1011000 | > loss_disc: 2.923597812652588 (2.8304510333321313) | > loss_disc_real_0: 0.2474297434091568 (0.22699338427998803) | > loss_disc_real_1: 0.250255286693573 (0.23421285287900404) | > loss_disc_real_2: 0.2877504527568817 (0.24026577391407705) | > loss_disc_real_3: 0.2742791175842285 (0.24022211107340727) | > loss_disc_real_4: 0.2715352773666382 (0.24558260549198496) | > loss_disc_real_5: 0.2548193037509918 (0.24552931189537047) | > loss_0: 2.923597812652588 (2.8304510333321313) | > grad_norm_0: tensor(11.4796, device='cuda:0') (tensor(12.6693, device='cuda:0')) | > loss_gen: 1.8817808628082275 (1.778451926057989) | > loss_kl: 2.249772787094116 (2.320084129680286) | > loss_feat: 2.3798460960388184 (2.0683219627900558) | > loss_mel: 19.820083618164062 (17.80040040449662) | > loss_duration: 1.6798865795135498 (1.6301161636005748) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.011369705200195 (25.597374413230202) | > grad_norm_1: tensor(443.5691, device='cuda:0') (tensor(275.8018, device='cuda:0')) | > current_lr_0: 0.00019623563939917877 | > current_lr_1: 0.00019623563939917877 | > step_time: 0.4622 (0.4529725551605225) | > loader_time: 0.0093 (0.007893475619229403) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627616183858525 (-2.4513459541061766e-07) | > avg_step_time: 0.46112632079863214 (+0.0055135404559927825) | > avg_loss_disc: 2.836513223782392 (+0.008820164371543981) | > avg_loss_disc_real_0: 0.22574687308408845 (-0.0006356131232960194) | > avg_loss_disc_real_1: 0.23389705260035018 (-0.0037702246451042087) | > avg_loss_disc_real_2: 0.24008085253373 (+0.0017331813422726827) | > avg_loss_disc_real_3: 0.24054992219931642 (-0.0007878978487471378) | > avg_loss_disc_real_4: 0.24771086095084607 (+0.002918784047516304) | > avg_loss_disc_real_5: 0.24758835472691226 (-0.0006595069253948482) | > avg_loss_0: 2.836513223782392 (+0.008820164371543981) | > avg_grad_norm_0: tensor(12.7716, device='cuda:0') (+tensor(1.5549, device='cuda:0')) | > avg_loss_gen: 1.771548956212863 (-0.006774544715881348) | > avg_loss_kl: 2.333024693207001 (+0.013165386629776066) | > avg_loss_feat: 2.035815326260848 (-0.0067414666565386305) | > avg_loss_mel: 17.924696210404516 (+0.10278094654351833) | > avg_loss_duration: 1.6405083075375624 (+0.00029821463034185314) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.705593297179316 (+0.10272840043188936) | > avg_grad_norm_1: tensor(257.9885, device='cuda:0') (tensor(-34.0399, device='cuda:0'))  > EPOCH: 153/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:45:40)   --> TIME: 2025-02-25 09:45:57 -- STEP: 33/72 -- GLOBAL_STEP: 1011050 | > loss_disc: 2.732982635498047 (2.8147206739945845) | > loss_disc_real_0: 0.18846695125102997 (0.21787580531654935) | > loss_disc_real_1: 0.2228766828775406 (0.23269698746276624) | > loss_disc_real_2: 0.21177561581134796 (0.2361143831953858) | > loss_disc_real_3: 0.21660171449184418 (0.2358477133693117) | > loss_disc_real_4: 0.21937833726406097 (0.2417734062129801) | > loss_disc_real_5: 0.24351367354393005 (0.24436349941022467) | > loss_0: 2.732982635498047 (2.8147206739945845) | > grad_norm_0: tensor(6.6788, device='cuda:0') (tensor(12.5588, device='cuda:0')) | > loss_gen: 1.9910115003585815 (1.7653758561972417) | > loss_kl: 2.462674856185913 (2.348110552990075) | > loss_feat: 2.4910037517547607 (2.138222275358258) | > loss_mel: 18.52243423461914 (18.373838887070164) | > loss_duration: 1.6210858821868896 (1.6091776580521555) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.08820915222168 (26.234725027373344) | > grad_norm_1: tensor(258.7161, device='cuda:0') (tensor(239.2241, device='cuda:0')) | > current_lr_0: 0.00019621110994425385 | > current_lr_1: 0.00019621110994425385 | > step_time: 0.458 (0.4403552936785149) | > loader_time: 0.0082 (0.007019657077211322) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617592529511785 (-1.0023654346739616e-05) | > avg_step_time: 0.45636674719797055 (-0.004759573600661593) | > avg_loss_disc: 2.8236068940498456 (-0.01290632973254624) | > avg_loss_disc_real_0: 0.22497411743855814 (-0.0007727556455303097) | > avg_loss_disc_real_1: 0.23121201446358586 (-0.002685038136764323) | > avg_loss_disc_real_2: 0.23864746891276936 (-0.0014333836209606265) | > avg_loss_disc_real_3: 0.23947452302549926 (-0.0010753991738171598) | > avg_loss_disc_real_4: 0.2443273852408772 (-0.003383475709968886) | > avg_loss_disc_real_5: 0.24630579814104966 (-0.0012825565858625987) | > avg_loss_0: 2.8236068940498456 (-0.01290632973254624) | > avg_grad_norm_0: tensor(11.8924, device='cuda:0') (tensor(-0.8792, device='cuda:0')) | > avg_loss_gen: 1.7647400590735423 (-0.006808897139320758) | > avg_loss_kl: 2.3213463299710986 (-0.011678363235902545) | > avg_loss_feat: 2.0657287449903885 (+0.029913418729540453) | > avg_loss_mel: 17.978351243784722 (+0.05365503338020616) | > avg_loss_duration: 1.6358146852170918 (-0.004693622320470547) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.765980975728638 (+0.06038767854932203) | > avg_grad_norm_1: tensor(274.1141, device='cuda:0') (+tensor(16.1255, device='cuda:0'))  > EPOCH: 154/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:46:17)   --> TIME: 2025-02-25 09:46:24 -- STEP: 11/72 -- GLOBAL_STEP: 1011100 | > loss_disc: 2.814251661300659 (2.8352674787694756) | > loss_disc_real_0: 0.2291485071182251 (0.22644857249476694) | > loss_disc_real_1: 0.22355785965919495 (0.2360072989355434) | > loss_disc_real_2: 0.20159432291984558 (0.24268136783079666) | > loss_disc_real_3: 0.23182865977287292 (0.2344269738955931) | > loss_disc_real_4: 0.2281895875930786 (0.24324450303207745) | > loss_disc_real_5: 0.23514917492866516 (0.24777617779645053) | > loss_0: 2.814251661300659 (2.8352674787694756) | > grad_norm_0: tensor(4.0379, device='cuda:0') (tensor(11.5928, device='cuda:0')) | > loss_gen: 1.7969510555267334 (1.751383131200617) | > loss_kl: 2.760708808898926 (2.3817260482094507) | > loss_feat: 2.295675277709961 (2.0608663992448286) | > loss_mel: 18.712757110595703 (18.02136091752486) | > loss_duration: 1.6187198162078857 (1.644756718115373) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.184812545776367 (25.86009355024858) | > grad_norm_1: tensor(176.9978, device='cuda:0') (tensor(237.1260, device='cuda:0')) | > current_lr_0: 0.00019618658355551082 | > current_lr_1: 0.00019618658355551082 | > step_time: 0.4289 (0.4281332926316695) | > loader_time: 0.0065 (0.006301381371238015)  --> TIME: 2025-02-25 09:46:47 -- STEP: 61/72 -- GLOBAL_STEP: 1011150 | > loss_disc: 2.784963607788086 (2.8333972555692077) | > loss_disc_real_0: 0.24454951286315918 (0.2256453745677823) | > loss_disc_real_1: 0.2928204834461212 (0.2388856572205903) | > loss_disc_real_2: 0.2849762737751007 (0.24654810765727622) | > loss_disc_real_3: 0.28298473358154297 (0.24348315034733445) | > loss_disc_real_4: 0.2633724808692932 (0.24764757884330438) | > loss_disc_real_5: 0.2707344889640808 (0.24775248326239038) | > loss_0: 2.784963607788086 (2.8333972555692077) | > grad_norm_0: tensor(12.5500, device='cuda:0') (tensor(11.6367, device='cuda:0')) | > loss_gen: 1.9317855834960938 (1.7850288914852455) | > loss_kl: 2.363863706588745 (2.4655342258390824) | > loss_feat: 2.0315608978271484 (2.0505753110666736) | > loss_mel: 18.54360008239746 (18.066496614549976) | > loss_duration: 1.6724292039871216 (1.6423549847524674) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.543241500854492 (26.009990254386526) | > grad_norm_1: tensor(393.2936, device='cuda:0') (tensor(249.8816, device='cuda:0')) | > current_lr_0: 0.00019618658355551082 | > current_lr_1: 0.00019618658355551082 | > step_time: 0.4836 (0.4459136353164423) | > loader_time: 0.0113 (0.008049597505663262) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534162816867022 (-8.342971264476383e-05) | > avg_step_time: 0.45357215908211723 (-0.0027945881158533137) | > avg_loss_disc: 2.829953922352321 (+0.006347028302475355) | > avg_loss_disc_real_0: 0.2273248364807854 (+0.0023507190422272783) | > avg_loss_disc_real_1: 0.23689850850004546 (+0.0056864940364595995) | > avg_loss_disc_real_2: 0.24428207164918872 (+0.0056346027364193585) | > avg_loss_disc_real_3: 0.2425765489608469 (+0.003102025935347641) | > avg_loss_disc_real_4: 0.2463283853631624 (+0.0020010001222852125) | > avg_loss_disc_real_5: 0.2475191656972321 (+0.0012133675561824264) | > avg_loss_0: 2.829953922352321 (+0.006347028302475355) | > avg_grad_norm_0: tensor(12.6138, device='cuda:0') (+tensor(0.7214, device='cuda:0')) | > avg_loss_gen: 1.7840208016650778 (+0.01928074259153556) | > avg_loss_kl: 2.45371651649475 (+0.1323701865236515) | > avg_loss_feat: 2.0455462093084624 (-0.020182535681926073) | > avg_loss_mel: 18.0713471963372 (+0.09299595255247795) | > avg_loss_duration: 1.6468031708623323 (+0.010988485645240464) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.001433976939026 (+0.23545300121038792) | > avg_grad_norm_1: tensor(249.5795, device='cuda:0') (tensor(-24.5346, device='cuda:0'))  > EPOCH: 155/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:46:54)   --> TIME: 2025-02-25 09:47:14 -- STEP: 39/72 -- GLOBAL_STEP: 1011200 | > loss_disc: 2.8714447021484375 (2.837482800850502) | > loss_disc_real_0: 0.1366618126630783 (0.21542342179096663) | > loss_disc_real_1: 0.25755414366722107 (0.236550306280454) | > loss_disc_real_2: 0.2349572777748108 (0.23930992912023497) | > loss_disc_real_3: 0.25824084877967834 (0.24295972746152145) | > loss_disc_real_4: 0.24888980388641357 (0.24858596080388778) | > loss_disc_real_5: 0.23286104202270508 (0.24795119235148796) | > loss_0: 2.8714447021484375 (2.837482800850502) | > grad_norm_0: tensor(20.5280, device='cuda:0') (tensor(12.8478, device='cuda:0')) | > loss_gen: 1.6893426179885864 (1.7602595121432574) | > loss_kl: 2.384101152420044 (2.447701246310503) | > loss_feat: 2.1947922706604004 (2.092526998275365) | > loss_mel: 18.045902252197266 (18.247290195562893) | > loss_duration: 1.6362674236297607 (1.6163230798183343) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.95040512084961 (26.16410103822366) | > grad_norm_1: tensor(628.8716, device='cuda:0') (tensor(255.1768, device='cuda:0')) | > current_lr_0: 0.00019616206023256638 | > current_lr_1: 0.00019616206023256638 | > step_time: 0.4571 (0.4365669764005221) | > loader_time: 0.008 (0.007311026255289714) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651864360755598 (+0.00011770154388857611) | > avg_step_time: 0.45334670577250735 (-0.00022545330960987986) | > avg_loss_disc: 2.8320944712195604 (+0.002140548867239467) | > avg_loss_disc_real_0: 0.21885023832740919 (-0.008474598153376228) | > avg_loss_disc_real_1: 0.23546984602867718 (-0.001428662471368275) | > avg_loss_disc_real_2: 0.24176581098999775 (-0.002516260659190972) | > avg_loss_disc_real_3: 0.2414299768041557 (-0.0011465721566912002) | > avg_loss_disc_real_4: 0.2467794695370634 (+0.0004510841739009952) | > avg_loss_disc_real_5: 0.24778774779447368 (+0.0002685820972415931) | > avg_loss_0: 2.8320944712195604 (+0.002140548867239467) | > avg_grad_norm_0: tensor(12.1875, device='cuda:0') (tensor(-0.4263, device='cuda:0')) | > avg_loss_gen: 1.7659906820512155 (-0.01803011961386236) | > avg_loss_kl: 2.4146321793677097 (-0.0390843371270404) | > avg_loss_feat: 2.0570029171419817 (+0.011456707833519264) | > avg_loss_mel: 18.081482309690664 (+0.01013511335346351) | > avg_loss_duration: 1.6401219804522018 (-0.006681190410130489) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.95923015433298 (-0.04220382260604438) | > avg_grad_norm_1: tensor(259.3237, device='cuda:0') (+tensor(9.7442, device='cuda:0'))  > EPOCH: 156/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:47:31)   --> TIME: 2025-02-25 09:47:41 -- STEP: 17/72 -- GLOBAL_STEP: 1011250 | > loss_disc: 2.81426739692688 (2.8775577825658463) | > loss_disc_real_0: 0.2679263651371002 (0.24057664722204208) | > loss_disc_real_1: 0.1956290900707245 (0.22720492149100585) | > loss_disc_real_2: 0.286893755197525 (0.24313244135940776) | > loss_disc_real_3: 0.21668486297130585 (0.24056423411649816) | > loss_disc_real_4: 0.23025494813919067 (0.25110436888302073) | > loss_disc_real_5: 0.22363632917404175 (0.24779996100594015) | > loss_0: 2.81426739692688 (2.8775577825658463) | > grad_norm_0: tensor(8.7380, device='cuda:0') (tensor(16.3383, device='cuda:0')) | > loss_gen: 1.6404720544815063 (1.7364795698839075) | > loss_kl: 2.6426279544830322 (2.3380337322459503) | > loss_feat: 2.001232624053955 (2.023148768088397) | > loss_mel: 18.805143356323242 (18.02260645698098) | > loss_duration: 1.6962471008300781 (1.623495908344493) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.785724639892578 (25.743764877319336) | > grad_norm_1: tensor(482.8550, device='cuda:0') (tensor(167.1304, device='cuda:0')) | > current_lr_0: 0.0001961375399750373 | > current_lr_1: 0.0001961375399750373 | > step_time: 0.4549 (0.46610145007862763) | > loader_time: 0.006 (0.006390613668105181)  --> TIME: 2025-02-25 09:48:05 -- STEP: 67/72 -- GLOBAL_STEP: 1011300 | > loss_disc: 2.919921875 (2.844714680714394) | > loss_disc_real_0: 0.1846030056476593 (0.22784512467793563) | > loss_disc_real_1: 0.1477995663881302 (0.23235210464961492) | > loss_disc_real_2: 0.20149828493595123 (0.23803419519716235) | > loss_disc_real_3: 0.2000247836112976 (0.23922738922176076) | > loss_disc_real_4: 0.24162372946739197 (0.2437432888728469) | > loss_disc_real_5: 0.23777157068252563 (0.24489343455478327) | > loss_0: 2.919921875 (2.844714680714394) | > grad_norm_0: tensor(9.7584, device='cuda:0') (tensor(14.2873, device='cuda:0')) | > loss_gen: 1.8592735528945923 (1.7591325952045953) | > loss_kl: 2.6427111625671387 (2.348120486558373) | > loss_feat: 2.1144280433654785 (2.0658194698504544) | > loss_mel: 18.85245132446289 (17.916288375854492) | > loss_duration: 1.7586463689804077 (1.6452071310868903) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.227510452270508 (25.734568211569716) | > grad_norm_1: tensor(89.6018, device='cuda:0') (tensor(210.2214, device='cuda:0')) | > current_lr_0: 0.0001961375399750373 | > current_lr_1: 0.0001961375399750373 | > step_time: 0.5135 (0.4589636468175632) | > loader_time: 0.0121 (0.008277907300351271) --> EVAL PERFORMANCE | > avg_loader_time: 0.008469954342909262 (-0.00018191001784633593) | > avg_step_time: 0.46202174038954186 (+0.008675034617034505) | > avg_loss_disc: 2.8485256517437136 (+0.016431180524153266) | > avg_loss_disc_real_0: 0.23522336014979323 (+0.01637312182238404) | > avg_loss_disc_real_1: 0.23766506977484259 (+0.0021952237461654034) | > avg_loss_disc_real_2: 0.2381017548097691 (-0.0036640561802286364) | > avg_loss_disc_real_3: 0.24017835586843356 (-0.0012516209357221475) | > avg_loss_disc_real_4: 0.2445224322063822 (-0.002257037330681194) | > avg_loss_disc_real_5: 0.24510103380176382 (-0.002686713992709866) | > avg_loss_0: 2.8485256517437136 (+0.016431180524153266) | > avg_grad_norm_0: tensor(14.6298, device='cuda:0') (+tensor(2.4423, device='cuda:0')) | > avg_loss_gen: 1.7703296063651501 (+0.004338924313934678) | > avg_loss_kl: 2.357873923341992 (-0.056758256025717824) | > avg_loss_feat: 2.0590244380521097 (+0.002021520910127972) | > avg_loss_mel: 17.916879600202535 (-0.16460270948812905) | > avg_loss_duration: 1.6469178468408718 (+0.006795866388670024) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.75102561628315 (-0.20820453804983075) | > avg_grad_norm_1: tensor(208.4533, device='cuda:0') (tensor(-50.8704, device='cuda:0'))  > EPOCH: 157/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:48:08)   --> TIME: 2025-02-25 09:48:31 -- STEP: 45/72 -- GLOBAL_STEP: 1011350 | > loss_disc: 2.8168370723724365 (2.830147499508328) | > loss_disc_real_0: 0.2563115358352661 (0.21888099842601352) | > loss_disc_real_1: 0.22911787033081055 (0.23388041887018415) | > loss_disc_real_2: 0.2232561707496643 (0.24365934299098121) | > loss_disc_real_3: 0.21847796440124512 (0.24254778424898785) | > loss_disc_real_4: 0.24423177540302277 (0.24559695654445224) | > loss_disc_real_5: 0.2533480226993561 (0.24975892172919376) | > loss_0: 2.8168370723724365 (2.830147499508328) | > grad_norm_0: tensor(11.2035, device='cuda:0') (tensor(8.6132, device='cuda:0')) | > loss_gen: 1.6401244401931763 (1.7709928115208944) | > loss_kl: 2.43924880027771 (2.3433462063471477) | > loss_feat: 2.2058372497558594 (2.07199219332801) | > loss_mel: 17.475309371948242 (18.012601513332793) | > loss_duration: 1.6527986526489258 (1.621317278014289) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.413318634033203 (25.82024989657932) | > grad_norm_1: tensor(61.6821, device='cuda:0') (tensor(230.5162, device='cuda:0')) | > current_lr_0: 0.0001961130227825404 | > current_lr_1: 0.0001961130227825404 | > step_time: 0.4552 (0.4406112141079373) | > loader_time: 0.0087 (0.007655594083997939) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687828628110215 (+0.00021787428520095298) | > avg_step_time: 0.4544414130734726 (-0.00758032731606928) | > avg_loss_disc: 2.839058261522105 (-0.009467390221608696) | > avg_loss_disc_real_0: 0.23122027327477093 (-0.004003086875022294) | > avg_loss_disc_real_1: 0.2344082663176765 (-0.0032568034571660853) | > avg_loss_disc_real_2: 0.24346250421564344 (+0.005360749405874332) | > avg_loss_disc_real_3: 0.2408253273493807 (+0.0006469714809471394) | > avg_loss_disc_real_4: 0.24373979740579363 (-0.0007826348005885719) | > avg_loss_disc_real_5: 0.24906640762174634 (+0.003965373819982521) | > avg_loss_0: 2.839058261522105 (-0.009467390221608696) | > avg_grad_norm_0: tensor(12.0031, device='cuda:0') (tensor(-2.6267, device='cuda:0')) | > avg_loss_gen: 1.7739444181952677 (+0.0036148118301175813) | > avg_loss_kl: 2.3510105290883025 (-0.0068633942536893144) | > avg_loss_feat: 2.0323003883093174 (-0.0267240497427923) | > avg_loss_mel: 17.91254962330133 (-0.004329976901203736) | > avg_loss_duration: 1.6384802133264675 (-0.008437633514404297) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.708285103381524 (-0.04274051290162717) | > avg_grad_norm_1: tensor(240.0607, device='cuda:0') (+tensor(31.6075, device='cuda:0'))  > EPOCH: 158/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:48:45)   --> TIME: 2025-02-25 09:48:57 -- STEP: 23/72 -- GLOBAL_STEP: 1011400 | > loss_disc: 2.819446325302124 (2.7913407761117686) | > loss_disc_real_0: 0.16703712940216064 (0.20505085101594095) | > loss_disc_real_1: 0.1837926059961319 (0.22397050876980243) | > loss_disc_real_2: 0.2084101140499115 (0.2334192278592483) | > loss_disc_real_3: 0.1890021562576294 (0.23892895229484723) | > loss_disc_real_4: 0.22535070776939392 (0.24463920619176782) | > loss_disc_real_5: 0.21853409707546234 (0.24883270717185477) | > loss_0: 2.819446325302124 (2.7913407761117686) | > grad_norm_0: tensor(18.1028, device='cuda:0') (tensor(10.1210, device='cuda:0')) | > loss_gen: 1.820615530014038 (1.7784624203391697) | > loss_kl: 2.0579278469085693 (2.292023959367172) | > loss_feat: 2.3618478775024414 (2.2242572411246924) | > loss_mel: 17.161069869995117 (18.11131012958029) | > loss_duration: 1.6210085153579712 (1.6142262168552564) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.02246856689453 (26.020279594089676) | > grad_norm_1: tensor(39.4126, device='cuda:0') (tensor(250.5430, device='cuda:0')) | > current_lr_0: 0.00019608850865469258 | > current_lr_1: 0.00019608850865469258 | > step_time: 0.4335 (0.43371303185172705) | > loader_time: 0.0064 (0.006764515586521315) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869060234284737 (+2.77371473715593e-06) | > avg_step_time: 0.4565930332936032 (+0.002151620220130601) | > avg_loss_disc: 2.8174242872587394 (-0.021633974263365552) | > avg_loss_disc_real_0: 0.218113120068127 (-0.013107153206643934) | > avg_loss_disc_real_1: 0.23419138862633368 (-0.00021687769134282053) | > avg_loss_disc_real_2: 0.2387633031942475 (-0.0046992010213959545) | > avg_loss_disc_real_3: 0.24071403267517896 (-0.000111294674201734) | > avg_loss_disc_real_4: 0.24244780313800757 (-0.0012919942677860619) | > avg_loss_disc_real_5: 0.2432494665115652 (-0.005816941110181134) | > avg_loss_0: 2.8174242872587394 (-0.021633974263365552) | > avg_grad_norm_0: tensor(12.2664, device='cuda:0') (+tensor(0.2633, device='cuda:0')) | > avg_loss_gen: 1.7731693882337758 (-0.0007750299614919243) | > avg_loss_kl: 2.3576475365061156 (+0.0066370074178130345) | > avg_loss_feat: 2.1173166238086325 (+0.08501623549931514) | > avg_loss_mel: 17.94833438497194 (+0.035784761670608844) | > avg_loss_duration: 1.6425663602184242 (+0.004086146891956632) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.83903422825773 (+0.13074912487620693) | > avg_grad_norm_1: tensor(286.2799, device='cuda:0') (+tensor(46.2192, device='cuda:0'))  > EPOCH: 159/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:49:23)   --> TIME: 2025-02-25 09:49:25 -- STEP: 1/72 -- GLOBAL_STEP: 1011450 | > loss_disc: 2.914388656616211 (2.914388656616211) | > loss_disc_real_0: 0.16958549618721008 (0.16958549618721008) | > loss_disc_real_1: 0.23681974411010742 (0.23681974411010742) | > loss_disc_real_2: 0.32491999864578247 (0.32491999864578247) | > loss_disc_real_3: 0.30765730142593384 (0.30765730142593384) | > loss_disc_real_4: 0.28662025928497314 (0.28662025928497314) | > loss_disc_real_5: 0.2668186128139496 (0.2668186128139496) | > loss_0: 2.914388656616211 (2.914388656616211) | > grad_norm_0: tensor(9.0495, device='cuda:0') (tensor(9.0495, device='cuda:0')) | > loss_gen: 1.9356633424758911 (1.9356633424758911) | > loss_kl: 2.336153984069824 (2.336153984069824) | > loss_feat: 2.2007932662963867 (2.2007932662963867) | > loss_mel: 18.58694076538086 (18.58694076538086) | > loss_duration: 1.6569215059280396 (1.6569215059280396) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.716472625732422 (26.716472625732422) | > grad_norm_1: tensor(444.8731, device='cuda:0') (tensor(444.8731, device='cuda:0')) | > current_lr_0: 0.00019606399759111075 | > current_lr_1: 0.00019606399759111075 | > step_time: 0.4395 (0.4394798278808594) | > loader_time: 0.0061 (0.006062030792236328)  --> TIME: 2025-02-25 09:49:48 -- STEP: 51/72 -- GLOBAL_STEP: 1011500 | > loss_disc: 2.784581184387207 (2.826586639179903) | > loss_disc_real_0: 0.1705552041530609 (0.2107470909170076) | > loss_disc_real_1: 0.3073682188987732 (0.24061102025649128) | > loss_disc_real_2: 0.21607622504234314 (0.24435979624589285) | > loss_disc_real_3: 0.2153158336877823 (0.2406826206282073) | > loss_disc_real_4: 0.23570452630519867 (0.24640602048705607) | > loss_disc_real_5: 0.2368280440568924 (0.2473749816417694) | > loss_0: 2.784581184387207 (2.826586639179903) | > grad_norm_0: tensor(13.0590, device='cuda:0') (tensor(13.2773, device='cuda:0')) | > loss_gen: 1.9064359664916992 (1.7780816718643786) | > loss_kl: 2.149048089981079 (2.313818043353511) | > loss_feat: 2.1521074771881104 (2.1008020756291406) | > loss_mel: 17.919286727905273 (17.79314029917997) | > loss_duration: 1.681168794631958 (1.6278435669693292) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.808046340942383 (25.61368575750613) | > grad_norm_1: tensor(130.6628, device='cuda:0') (tensor(265.5417, device='cuda:0')) | > current_lr_0: 0.00019606399759111075 | > current_lr_1: 0.00019606399759111075 | > step_time: 0.452 (0.448655553892547) | > loader_time: 0.0088 (0.008059066884657918) --> EVAL PERFORMANCE | > avg_loader_time: 0.008930488371513261 (+0.00023988602866589076) | > avg_step_time: 0.45976500779810087 (+0.0031719745044976877) | > avg_loss_disc: 2.8261406052280478 (+0.008716317969308385) | > avg_loss_disc_real_0: 0.21665118739638534 (-0.0014619326717416614) | > avg_loss_disc_real_1: 0.239184063085368 (+0.004992674459034313) | > avg_loss_disc_real_2: 0.24343251531392757 (+0.004669212119680077) | > avg_loss_disc_real_3: 0.23966354929225545 (-0.0010504833829235172) | > avg_loss_disc_real_4: 0.2466292184003642 (+0.00418141526235663) | > avg_loss_disc_real_5: 0.2472645682348332 (+0.004015101723267994) | > avg_loss_0: 2.8261406052280478 (+0.008716317969308385) | > avg_grad_norm_0: tensor(13.2120, device='cuda:0') (+tensor(0.9456, device='cuda:0')) | > avg_loss_gen: 1.7762654767909518 (+0.0030960885571760244) | > avg_loss_kl: 2.324023770614409 (-0.03362376589170646) | > avg_loss_feat: 2.077470920455288 (-0.039845703353344586) | > avg_loss_mel: 17.819829524402888 (-0.12850486056905197) | > avg_loss_duration: 1.6385653220431906 (-0.004001038175233562) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.636155034454777 (-0.2028791938029535) | > avg_grad_norm_1: tensor(277.9568, device='cuda:0') (tensor(-8.3231, device='cuda:0'))  > EPOCH: 160/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:50:00)   --> TIME: 2025-02-25 09:50:15 -- STEP: 29/72 -- GLOBAL_STEP: 1011550 | > loss_disc: 2.7660303115844727 (2.8065073572356125) | > loss_disc_real_0: 0.23469404876232147 (0.2214517768087058) | > loss_disc_real_1: 0.2048013061285019 (0.23459944180373488) | > loss_disc_real_2: 0.2923962473869324 (0.24034335335780835) | > loss_disc_real_3: 0.26638129353523254 (0.24125814232332954) | > loss_disc_real_4: 0.26372501254081726 (0.24242732853725038) | > loss_disc_real_5: 0.26678070425987244 (0.2450352490975939) | > loss_0: 2.7660303115844727 (2.8065073572356125) | > grad_norm_0: tensor(9.8181, device='cuda:0') (tensor(12.8542, device='cuda:0')) | > loss_gen: 1.8965193033218384 (1.8019223788688923) | > loss_kl: 2.428286552429199 (2.3550239595873603) | > loss_feat: 2.0620951652526855 (2.1416715753489526) | > loss_mel: 17.3969783782959 (17.828083959119073) | > loss_duration: 1.611238718032837 (1.6083728979373801) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.395116806030273 (25.735074733865673) | > grad_norm_1: tensor(474.7248, device='cuda:0') (tensor(264.6970, device='cuda:0')) | > current_lr_0: 0.00019603948959141186 | > current_lr_1: 0.00019603948959141186 | > step_time: 0.4449 (0.43992263695289346) | > loader_time: 0.0076 (0.007355681781111093) --> EVAL PERFORMANCE | > avg_loader_time: 0.008774720447164186 (-0.0001557679243490758) | > avg_step_time: 0.45747990675375494 (-0.0022851010443459274) | > avg_loss_disc: 2.8238312157107073 (-0.002309389517340499) | > avg_loss_disc_real_0: 0.22014156634538945 (+0.0034903789490041137) | > avg_loss_disc_real_1: 0.23772901023777437 (-0.0014550528475936186) | > avg_loss_disc_real_2: 0.2379569671523403 (-0.005475548161587268) | > avg_loss_disc_real_3: 0.24008123870466797 (+0.00041768941241251967) | > avg_loss_disc_real_4: 0.24274881855702737 (-0.0038803998433368236) | > avg_loss_disc_real_5: 0.2480443453704807 (+0.0007797771356475025) | > avg_loss_0: 2.8238312157107073 (-0.002309389517340499) | > avg_grad_norm_0: tensor(12.5094, device='cuda:0') (tensor(-0.7027, device='cuda:0')) | > avg_loss_gen: 1.775230992008263 (-0.0010344847826888603) | > avg_loss_kl: 2.3294261845064836 (+0.00540241389207452) | > avg_loss_feat: 2.0884655344654135 (+0.01099461401012558) | > avg_loss_mel: 17.66059367757448 (-0.15923584682840897) | > avg_loss_duration: 1.6356599884973446 (-0.0029053335458459895) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.489376121843364 (-0.1467789126114134) | > avg_grad_norm_1: tensor(256.6306, device='cuda:0') (tensor(-21.3263, device='cuda:0'))  > EPOCH: 161/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:50:37)   --> TIME: 2025-02-25 09:50:42 -- STEP: 7/72 -- GLOBAL_STEP: 1011600 | > loss_disc: 2.7757468223571777 (2.8191472462245395) | > loss_disc_real_0: 0.2553856372833252 (0.21525028773716517) | > loss_disc_real_1: 0.16833338141441345 (0.2323371491261891) | > loss_disc_real_2: 0.22151073813438416 (0.24267084896564484) | > loss_disc_real_3: 0.23978696763515472 (0.2410492535148348) | > loss_disc_real_4: 0.2225203812122345 (0.24406900789056504) | > loss_disc_real_5: 0.24161705374717712 (0.23609083039419992) | > loss_0: 2.7757468223571777 (2.8191472462245395) | > grad_norm_0: tensor(6.0371, device='cuda:0') (tensor(8.2730, device='cuda:0')) | > loss_gen: 1.696871042251587 (1.7518741403307234) | > loss_kl: 2.451162576675415 (2.2208940982818604) | > loss_feat: 2.3867990970611572 (2.2093518461499895) | > loss_mel: 17.918363571166992 (17.92855099269322) | > loss_duration: 1.624518632888794 (1.6182527031217302) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.077714920043945 (25.7289240700858) | > grad_norm_1: tensor(150.3860, device='cuda:0') (tensor(299.7636, device='cuda:0')) | > current_lr_0: 0.0001960149846552129 | > current_lr_1: 0.0001960149846552129 | > step_time: 0.4374 (0.43501646178109304) | > loader_time: 0.0055 (0.006082262311662946)  --> TIME: 2025-02-25 09:51:05 -- STEP: 57/72 -- GLOBAL_STEP: 1011650 | > loss_disc: 2.783923864364624 (2.838168788374516) | > loss_disc_real_0: 0.1902572512626648 (0.21761357705844075) | > loss_disc_real_1: 0.22732050716876984 (0.23830946155807428) | > loss_disc_real_2: 0.26036688685417175 (0.24178827985336906) | > loss_disc_real_3: 0.2389041930437088 (0.243062064313052) | > loss_disc_real_4: 0.24722976982593536 (0.2474714975085175) | > loss_disc_real_5: 0.25500452518463135 (0.24873668530531096) | > loss_0: 2.783923864364624 (2.838168788374516) | > grad_norm_0: tensor(8.9699, device='cuda:0') (tensor(11.2912, device='cuda:0')) | > loss_gen: 1.8238552808761597 (1.7615085986622594) | > loss_kl: 2.381019353866577 (2.3388429942883944) | > loss_feat: 1.8097823858261108 (2.0705430005726058) | > loss_mel: 16.78580093383789 (17.831026947289185) | > loss_duration: 1.6826788187026978 (1.6338992077007628) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.483135223388672 (25.63582075687877) | > grad_norm_1: tensor(331.4911, device='cuda:0') (tensor(225.5446, device='cuda:0')) | > current_lr_0: 0.0001960149846552129 | > current_lr_1: 0.0001960149846552129 | > step_time: 0.4723 (0.4441086074762177) | > loader_time: 0.0096 (0.007918876514100193) --> EVAL PERFORMANCE | > avg_loader_time: 0.008548820522469535 (-0.0002258999246946506) | > avg_step_time: 0.4547841851140412 (-0.002695721639713766) | > avg_loss_disc: 2.827959460271916 (+0.004128244561208749) | > avg_loss_disc_real_0: 0.2182331574215016 (-0.0019084089238878554) | > avg_loss_disc_real_1: 0.23614488982818496 (-0.0015841204095894124) | > avg_loss_disc_real_2: 0.24254116374002377 (+0.00458419658768347) | > avg_loss_disc_real_3: 0.24221028723347354 (+0.002129048528805577) | > avg_loss_disc_real_4: 0.24622145945757207 (+0.0034726409005446934) | > avg_loss_disc_real_5: 0.24690345916110026 (-0.0011408862093804373) | > avg_loss_0: 2.827959460271916 (+0.004128244561208749) | > avg_grad_norm_0: tensor(11.7553, device='cuda:0') (tensor(-0.7541, device='cuda:0')) | > avg_loss_gen: 1.7618588696063404 (-0.013372122401922537) | > avg_loss_kl: 2.349189593758381 (+0.019763409251897457) | > avg_loss_feat: 2.060537996426434 (-0.027927538038979627) | > avg_loss_mel: 17.777748349686746 (+0.11715467211226738) | > avg_loss_duration: 1.641151980614998 (+0.005491992117653277) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.590486821993977 (+0.1011107001506133) | > avg_grad_norm_1: tensor(250.6088, device='cuda:0') (tensor(-6.0218, device='cuda:0'))  > EPOCH: 162/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:51:14)   --> TIME: 2025-02-25 09:51:32 -- STEP: 35/72 -- GLOBAL_STEP: 1011700 | > loss_disc: 2.834519386291504 (2.803168548856463) | > loss_disc_real_0: 0.21375715732574463 (0.21712261119059154) | > loss_disc_real_1: 0.19913779199123383 (0.2469868642943246) | > loss_disc_real_2: 0.23597529530525208 (0.23361072157110488) | > loss_disc_real_3: 0.24840714037418365 (0.23983759369168964) | > loss_disc_real_4: 0.2558634281158447 (0.2466627482857023) | > loss_disc_real_5: 0.23497086763381958 (0.24368213968617575) | > loss_0: 2.834519386291504 (2.803168548856463) | > grad_norm_0: tensor(6.4778, device='cuda:0') (tensor(9.7841, device='cuda:0')) | > loss_gen: 1.8452675342559814 (1.8128134727478027) | > loss_kl: 2.447301149368286 (2.2912383828844343) | > loss_feat: 2.0218262672424316 (2.2017575298036838) | > loss_mel: 18.447856903076172 (17.970534460885183) | > loss_duration: 1.6579358577728271 (1.6108933755329677) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.420188903808594 (25.887237112862724) | > grad_norm_1: tensor(502.9356, device='cuda:0') (tensor(300.7096, device='cuda:0')) | > current_lr_0: 0.000195990482782131 | > current_lr_1: 0.000195990482782131 | > step_time: 0.4399 (0.4436796188354492) | > loader_time: 0.0088 (0.007336051123482841) --> EVAL PERFORMANCE | > avg_loader_time: 0.008747913468051963 (+0.000199092945582428) | > avg_step_time: 0.4593001083588936 (+0.004515923244852449) | > avg_loss_disc: 2.8188645201669607 (-0.009094940104955374) | > avg_loss_disc_real_0: 0.21715915948152542 (-0.0010739979399761734) | > avg_loss_disc_real_1: 0.23903930711914118 (+0.002894417290956214) | > avg_loss_disc_real_2: 0.23588440011084919 (-0.0066567636291745835) | > avg_loss_disc_real_3: 0.23885110902114653 (-0.0033591782123270075) | > avg_loss_disc_real_4: 0.24579704521407544 (-0.0004244142434966236) | > avg_loss_disc_real_5: 0.24573303423297238 (-0.0011704249281278833) | > avg_loss_0: 2.8188645201669607 (-0.009094940104955374) | > avg_grad_norm_0: tensor(11.8741, device='cuda:0') (+tensor(0.1188, device='cuda:0')) | > avg_loss_gen: 1.7883617391049023 (+0.026502869498561843) | > avg_loss_kl: 2.3032371913883036 (-0.04595240237007747) | > avg_loss_feat: 2.1549359795073393 (+0.09439798308090541) | > avg_loss_mel: 17.786934167566425 (+0.009185817879679092) | > avg_loss_duration: 1.634325448895844 (-0.006826531719153817) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.667794348488393 (+0.07730752649441541) | > avg_grad_norm_1: tensor(250.8313, device='cuda:0') (+tensor(0.2225, device='cuda:0'))  > EPOCH: 163/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:51:52)   --> TIME: 2025-02-25 09:51:59 -- STEP: 13/72 -- GLOBAL_STEP: 1011750 | > loss_disc: 2.858150005340576 (2.807761632479154) | > loss_disc_real_0: 0.1984110176563263 (0.20868556659955245) | > loss_disc_real_1: 0.12355311214923859 (0.2223095756310683) | > loss_disc_real_2: 0.19022001326084137 (0.24819063681822556) | > loss_disc_real_3: 0.2298627346754074 (0.2413854908484679) | > loss_disc_real_4: 0.2223176211118698 (0.24084291664453653) | > loss_disc_real_5: 0.25303640961647034 (0.24420540034770966) | > loss_0: 2.858150005340576 (2.807761632479154) | > grad_norm_0: tensor(8.6319, device='cuda:0') (tensor(10.4653, device='cuda:0')) | > loss_gen: 1.8163264989852905 (1.784622155703031) | > loss_kl: 2.384033441543579 (2.266992348891038) | > loss_feat: 2.556781768798828 (2.2393561601638794) | > loss_mel: 16.9827880859375 (18.227444722102238) | > loss_duration: 1.5336153507232666 (1.6042893483088567) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.273544311523438 (26.12270501943735) | > grad_norm_1: tensor(102.8173, device='cuda:0') (tensor(358.3878, device='cuda:0')) | > current_lr_0: 0.00019596598397178324 | > current_lr_1: 0.00019596598397178324 | > step_time: 0.4634 (0.442453696177556) | > loader_time: 0.0064 (0.007640856962937575)  --> TIME: 2025-02-25 09:52:23 -- STEP: 63/72 -- GLOBAL_STEP: 1011800 | > loss_disc: 2.7747342586517334 (2.828175215494065) | > loss_disc_real_0: 0.24662473797798157 (0.21498932653949374) | > loss_disc_real_1: 0.21144366264343262 (0.2359586568578841) | > loss_disc_real_2: 0.2343469262123108 (0.24566409677740128) | > loss_disc_real_3: 0.24052509665489197 (0.2440469539354718) | > loss_disc_real_4: 0.2580290138721466 (0.24579895134010013) | > loss_disc_real_5: 0.24965247511863708 (0.24893966766576917) | > loss_0: 2.7747342586517334 (2.828175215494065) | > grad_norm_0: tensor(10.6074, device='cuda:0') (tensor(10.7205, device='cuda:0')) | > loss_gen: 1.871288537979126 (1.7846551565896898) | > loss_kl: 2.30556321144104 (2.3899884337470634) | > loss_feat: 2.2153100967407227 (2.126548481365992) | > loss_mel: 18.06870460510254 (17.973556912134566) | > loss_duration: 1.626189947128296 (1.6299936468639071) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.087055206298828 (25.904742710174077) | > grad_norm_1: tensor(403.3040, device='cuda:0') (tensor(260.6922, device='cuda:0')) | > current_lr_0: 0.00019596598397178324 | > current_lr_1: 0.00019596598397178324 | > step_time: 0.5037 (0.4521384049975683) | > loader_time: 0.0115 (0.008499262824891106) --> EVAL PERFORMANCE | > avg_loader_time: 0.008839607238769531 (+9.169377071756822e-05) | > avg_step_time: 0.4578926227462124 (-0.0014074856126812252) | > avg_loss_disc: 2.8267125445352472 (+0.007848024368286577) | > avg_loss_disc_real_0: 0.21511160949586142 (-0.0020475499856640045) | > avg_loss_disc_real_1: 0.23509519620680472 (-0.003944110912336457) | > avg_loss_disc_real_2: 0.24421677497071279 (+0.0083323748598636) | > avg_loss_disc_real_3: 0.24337420958868214 (+0.004523100567535604) | > avg_loss_disc_real_4: 0.24490781946921011 (-0.0008892257448653296) | > avg_loss_disc_real_5: 0.24768349570287784 (+0.0019504614699054623) | > avg_loss_0: 2.8267125445352472 (+0.007848024368286577) | > avg_grad_norm_0: tensor(10.4125, device='cuda:0') (tensor(-1.4616, device='cuda:0')) | > avg_loss_gen: 1.781103555585297 (-0.007258183519605277) | > avg_loss_kl: 2.3826908326484784 (+0.07945364126017473) | > avg_loss_feat: 2.1134977911559636 (-0.04143818835137569) | > avg_loss_mel: 17.916653055540273 (+0.12971888797384779) | > avg_loss_duration: 1.6356082331966346 (+0.0012827843007905404) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.82955363099004 (+0.16175928250164873) | > avg_grad_norm_1: tensor(259.7813, device='cuda:0') (+tensor(8.9500, device='cuda:0'))  > EPOCH: 164/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:52:29)   --> TIME: 2025-02-25 09:52:49 -- STEP: 41/72 -- GLOBAL_STEP: 1011850 | > loss_disc: 2.8791844844818115 (2.8313484889705007) | > loss_disc_real_0: 0.13079895079135895 (0.21628753822751162) | > loss_disc_real_1: 0.26881784200668335 (0.2316401291911195) | > loss_disc_real_2: 0.24067355692386627 (0.2388292932655753) | > loss_disc_real_3: 0.248407244682312 (0.24213862637194192) | > loss_disc_real_4: 0.25759157538414 (0.24659947614844252) | > loss_disc_real_5: 0.2522853910923004 (0.24962027625339783) | > loss_0: 2.8791844844818115 (2.8313484889705007) | > grad_norm_0: tensor(13.8352, device='cuda:0') (tensor(14.8751, device='cuda:0')) | > loss_gen: 1.648024559020996 (1.7547792603329915) | > loss_kl: 2.618849277496338 (2.3412825247136553) | > loss_feat: 2.044710874557495 (2.1373081178199946) | > loss_mel: 17.959487915039062 (17.83454108819729) | > loss_duration: 1.6428892612457275 (1.6260239118483009) | > amp_scaler: 256.0 (262.2439024390244) | > loss_1: 25.91396141052246 (25.693935068642222) | > grad_norm_1: tensor(173.2330, device='cuda:0') (tensor(240.6617, device='cuda:0')) | > current_lr_0: 0.00019594148822378677 | > current_lr_1: 0.00019594148822378677 | > step_time: 0.4636 (0.4435407940934344) | > loader_time: 0.0102 (0.007501887112129025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008805368987607285 (-3.4238251162246186e-05) | > avg_step_time: 0.45732899450919995 (-0.0005636282370124479) | > avg_loss_disc: 2.8258340157253645 (-0.000878528809882706) | > avg_loss_disc_real_0: 0.21700589274856405 (+0.0018942832527026332) | > avg_loss_disc_real_1: 0.2351770560506364 (+8.185984383168865e-05) | > avg_loss_disc_real_2: 0.23876059621992246 (-0.005456178750790325) | > avg_loss_disc_real_3: 0.24146965146064758 (-0.0019045581280345558) | > avg_loss_disc_real_4: 0.24704906491326614 (+0.002141245444056028) | > avg_loss_disc_real_5: 0.2491998594831413 (+0.0015163637802634578) | > avg_loss_0: 2.8258340157253645 (-0.000878528809882706) | > avg_grad_norm_0: tensor(13.0004, device='cuda:0') (+tensor(2.5879, device='cuda:0')) | > avg_loss_gen: 1.767270220837123 (-0.013833334748174053) | > avg_loss_kl: 2.336774513755046 (-0.04591631889343217) | > avg_loss_feat: 2.107334234345127 (-0.006163556810836468) | > avg_loss_mel: 17.785056933550774 (-0.13159612198949944) | > avg_loss_duration: 1.6440255457246806 (+0.008417312528046006) | > avg_amp_scaler: 259.6056338028169 (+3.6056338028168966) | > avg_loss_1: 25.64046153216295 (-0.18909209882709277) | > avg_grad_norm_1: tensor(248.5432, device='cuda:0') (tensor(-11.2381, device='cuda:0'))  > EPOCH: 165/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:53:06)   --> TIME: 2025-02-25 09:53:16 -- STEP: 19/72 -- GLOBAL_STEP: 1011900 | > loss_disc: 2.768052101135254 (2.7984929210261296) | > loss_disc_real_0: 0.22160035371780396 (0.21778396320970436) | > loss_disc_real_1: 0.2823691666126251 (0.2257017195224762) | > loss_disc_real_2: 0.2660314738750458 (0.24107009799856888) | > loss_disc_real_3: 0.2365369200706482 (0.2418120628909061) | > loss_disc_real_4: 0.246366947889328 (0.24568512173075424) | > loss_disc_real_5: 0.2479620724916458 (0.25041179672667857) | > loss_0: 2.768052101135254 (2.7984929210261296) | > grad_norm_0: tensor(7.0380, device='cuda:0') (tensor(9.7575, device='cuda:0')) | > loss_gen: 1.8657677173614502 (1.7877263709118492) | > loss_kl: 2.2902286052703857 (2.245674127026608) | > loss_feat: 2.3255417346954346 (2.156932040264732) | > loss_mel: 17.61492919921875 (17.528435857672445) | > loss_duration: 1.5755423307418823 (1.5930680914929038) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.67201042175293 (25.31183664422286) | > grad_norm_1: tensor(542.6116, device='cuda:0') (tensor(370.7210, device='cuda:0')) | > current_lr_0: 0.00019591699553775878 | > current_lr_1: 0.00019591699553775878 | > step_time: 0.4303 (0.4340549519187526) | > loader_time: 0.0072 (0.006708195334986637)  --> TIME: 2025-02-25 09:53:40 -- STEP: 69/72 -- GLOBAL_STEP: 1011950 | > loss_disc: 2.8994247913360596 (2.8212387181710508) | > loss_disc_real_0: 0.2880305051803589 (0.22025279441605444) | > loss_disc_real_1: 0.25933322310447693 (0.23269779936990875) | > loss_disc_real_2: 0.22328530251979828 (0.24053299556607785) | > loss_disc_real_3: 0.2540212571620941 (0.23984074311843817) | > loss_disc_real_4: 0.2526693046092987 (0.24351390138052512) | > loss_disc_real_5: 0.24906161427497864 (0.24777406907599905) | > loss_0: 2.8994247913360596 (2.8212387181710508) | > grad_norm_0: tensor(13.8027, device='cuda:0') (tensor(11.9984, device='cuda:0')) | > loss_gen: 1.8114261627197266 (1.7658710047818613) | > loss_kl: 2.434795618057251 (2.29240011650583) | > loss_feat: 2.1694657802581787 (2.1047259983809106) | > loss_mel: 19.304737091064453 (17.589949248493586) | > loss_duration: 1.7156351804733276 (1.6288365540297136) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.436059951782227 (25.381783056950223) | > grad_norm_1: tensor(453.2572, device='cuda:0') (tensor(313.0965, device='cuda:0')) | > current_lr_0: 0.00019591699553775878 | > current_lr_1: 0.00019591699553775878 | > step_time: 0.5366 (0.45689500933108124) | > loader_time: 0.0132 (0.008650886839714602) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870537757873535 (-9.999140887193524e-05) | > avg_step_time: 0.4583528243320089 (+0.0010238298228089637) | > avg_loss_disc: 2.8215065371822305 (-0.004327478543133978) | > avg_loss_disc_real_0: 0.21909242910398563 (+0.002086536355421581) | > avg_loss_disc_real_1: 0.23294400644134466 (-0.002233049609291743) | > avg_loss_disc_real_2: 0.2411153599829741 (+0.0023547637630516316) | > avg_loss_disc_real_3: 0.24021771948941997 (-0.00125193197122761) | > avg_loss_disc_real_4: 0.2439048661732338 (-0.0031441987400323557) | > avg_loss_disc_real_5: 0.24817124659746465 (-0.0010286128856766552) | > avg_loss_0: 2.8215065371822305 (-0.004327478543133978) | > avg_grad_norm_0: tensor(11.8449, device='cuda:0') (tensor(-1.1555, device='cuda:0')) | > avg_loss_gen: 1.766232665155975 (-0.0010375556811479392) | > avg_loss_kl: 2.297028677564272 (-0.03974583619077432) | > avg_loss_feat: 2.099895227123315 (-0.007439007221812233) | > avg_loss_mel: 17.603499063303776 (-0.18155787024699777) | > avg_loss_duration: 1.6289823508598438 (-0.01504319486483685) | > avg_amp_scaler: 256.0 (-3.6056338028168966) | > avg_loss_1: 25.395638103216466 (-0.24482342894648212) | > avg_grad_norm_1: tensor(309.5582, device='cuda:0') (+tensor(61.0150, device='cuda:0'))  > EPOCH: 166/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:53:43)   --> TIME: 2025-02-25 09:54:06 -- STEP: 47/72 -- GLOBAL_STEP: 1012000 | > loss_disc: 2.8452653884887695 (2.8097167877440756) | > loss_disc_real_0: 0.07764611393213272 (0.21471253069157295) | > loss_disc_real_1: 0.23012977838516235 (0.2343110001467644) | > loss_disc_real_2: 0.24937394261360168 (0.23739100929270399) | > loss_disc_real_3: 0.23408587276935577 (0.23839127478447367) | > loss_disc_real_4: 0.24364493787288666 (0.2441963090541515) | > loss_disc_real_5: 0.24998965859413147 (0.24544039590561645) | > loss_0: 2.8452653884887695 (2.8097167877440756) | > grad_norm_0: tensor(39.2952, device='cuda:0') (tensor(13.3537, device='cuda:0')) | > loss_gen: 1.6839168071746826 (1.773800588668661) | > loss_kl: 2.4051268100738525 (2.34708232068001) | > loss_feat: 1.9424057006835938 (2.1738231664008283) | > loss_mel: 17.392316818237305 (18.050730035660113) | > loss_duration: 1.6451163291931152 (1.6361984318875251) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.06888198852539 (25.981634383505963) | > grad_norm_1: tensor(391.7676, device='cuda:0') (tensor(245.6819, device='cuda:0')) | > current_lr_0: 0.00019589250591331656 | > current_lr_1: 0.00019589250591331656 | > step_time: 0.4516 (0.44362878799438477) | > loader_time: 0.0091 (0.007786476865727851) --> EVAL PERFORMANCE | > avg_loader_time: 0.008711237302968198 (+5.859724232847724e-06) | > avg_step_time: 0.45541791177131763 (-0.002934912560691283) | > avg_loss_disc: 2.8210388002261304 (-0.0004677369561001399) | > avg_loss_disc_real_0: 0.21920091795249724 (+0.000108488848511612) | > avg_loss_disc_real_1: 0.23416517905786005 (+0.0012211726165153869) | > avg_loss_disc_real_2: 0.24142947981894855 (+0.0003141198359744579) | > avg_loss_disc_real_3: 0.2395444587922432 (-0.0006732606971767696) | > avg_loss_disc_real_4: 0.244993195357457 (+0.001088329184223219) | > avg_loss_disc_real_5: 0.24592433387125043 (-0.0022469127262142174) | > avg_loss_0: 2.8210388002261304 (-0.0004677369561001399) | > avg_grad_norm_0: tensor(13.6076, device='cuda:0') (+tensor(1.7627, device='cuda:0')) | > avg_loss_gen: 1.7721699308341659 (+0.005937265678190862) | > avg_loss_kl: 2.336831458857362 (+0.03980278129309012) | > avg_loss_feat: 2.1345938830308517 (+0.034698655907536846) | > avg_loss_mel: 17.939508142605632 (+0.3360090793018564) | > avg_loss_duration: 1.6476161144149135 (+0.018633763555069738) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.83071942396567 (+0.43508132074920525) | > avg_grad_norm_1: tensor(278.4984, device='cuda:0') (tensor(-31.0599, device='cuda:0'))  > EPOCH: 167/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:54:20)   --> TIME: 2025-02-25 09:54:34 -- STEP: 25/72 -- GLOBAL_STEP: 1012050 | > loss_disc: 2.7713775634765625 (2.8132323455810546) | > loss_disc_real_0: 0.24346555769443512 (0.22189089715480803) | > loss_disc_real_1: 0.20818640291690826 (0.2396166878938675) | > loss_disc_real_2: 0.23249702155590057 (0.23881504952907562) | > loss_disc_real_3: 0.17852355539798737 (0.23889188647270201) | > loss_disc_real_4: 0.2458183914422989 (0.24350684881210327) | > loss_disc_real_5: 0.23340502381324768 (0.2416045081615448) | > loss_0: 2.7713775634765625 (2.8132323455810546) | > grad_norm_0: tensor(4.7728, device='cuda:0') (tensor(11.2379, device='cuda:0')) | > loss_gen: 1.6354060173034668 (1.7752600812911987) | > loss_kl: 2.516580820083618 (2.366994037628174) | > loss_feat: 2.3642630577087402 (2.170894346237182) | > loss_mel: 18.256248474121094 (18.0937548828125) | > loss_duration: 1.6193349361419678 (1.6092015790939331) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.391834259033203 (26.016105422973634) | > grad_norm_1: tensor(117.9772, device='cuda:0') (tensor(217.2050, device='cuda:0')) | > current_lr_0: 0.0001958680193500774 | > current_lr_1: 0.0001958680193500774 | > step_time: 0.4377 (0.4414076805114746) | > loader_time: 0.0074 (0.007143907546997071) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685387356180541 (-2.584994678765662e-05) | > avg_step_time: 0.45800797032638335 (+0.002590058555065722) | > avg_loss_disc: 2.8244050287864573 (+0.003366228560326867) | > avg_loss_disc_real_0: 0.21626056835684979 (-0.002940349595647457) | > avg_loss_disc_real_1: 0.2383311392975525 (+0.004165960239692457) | > avg_loss_disc_real_2: 0.24067535345822993 (-0.0007541263607186155) | > avg_loss_disc_real_3: 0.23975328912197705 (+0.00020883032973384452) | > avg_loss_disc_real_4: 0.24325022940904323 (-0.0017429659484137772) | > avg_loss_disc_real_5: 0.2452381429537921 (-0.0006861909174583347) | > avg_loss_0: 2.8244050287864573 (+0.003366228560326867) | > avg_grad_norm_0: tensor(11.1951, device='cuda:0') (tensor(-2.4125, device='cuda:0')) | > avg_loss_gen: 1.7643840363327887 (-0.007785894501377122) | > avg_loss_kl: 2.3404898811394066 (+0.0036584222820446044) | > avg_loss_feat: 2.1088138063189006 (-0.02578007671195115) | > avg_loss_mel: 17.87258338928223 (-0.06692475332340209) | > avg_loss_duration: 1.637784805096371 (-0.00983130931854248) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.72405602898396 (-0.1066633949817124) | > avg_grad_norm_1: tensor(259.3675, device='cuda:0') (tensor(-19.1309, device='cuda:0'))  > EPOCH: 168/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:54:58)   --> TIME: 2025-02-25 09:55:01 -- STEP: 3/72 -- GLOBAL_STEP: 1012100 | > loss_disc: 2.806565523147583 (2.810675303141276) | > loss_disc_real_0: 0.17261174321174622 (0.20676350593566895) | > loss_disc_real_1: 0.20017284154891968 (0.22741482655207315) | > loss_disc_real_2: 0.24233563244342804 (0.22381280859311423) | > loss_disc_real_3: 0.2257423996925354 (0.24363051851590475) | > loss_disc_real_4: 0.23180553317070007 (0.2478845218817393) | > loss_disc_real_5: 0.24858297407627106 (0.25081636011600494) | > loss_0: 2.806565523147583 (2.810675303141276) | > grad_norm_0: tensor(14.0230, device='cuda:0') (tensor(7.9257, device='cuda:0')) | > loss_gen: 1.7601407766342163 (1.750739296277364) | > loss_kl: 2.3959569931030273 (2.4025468031565347) | > loss_feat: 2.3690266609191895 (2.2540688514709473) | > loss_mel: 18.4644775390625 (18.579086939493816) | > loss_duration: 1.632885456085205 (1.6329087416330974) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.622488021850586 (26.61935043334961) | > grad_norm_1: tensor(325.7592, device='cuda:0') (tensor(334.5038, device='cuda:0')) | > current_lr_0: 0.00019584353584765863 | > current_lr_1: 0.00019584353584765863 | > step_time: 0.4267 (0.4372001489003499) | > loader_time: 0.0075 (0.007075786590576172)  --> TIME: 2025-02-25 09:55:24 -- STEP: 53/72 -- GLOBAL_STEP: 1012150 | > loss_disc: 2.890383243560791 (2.813645691241858) | > loss_disc_real_0: 0.26866471767425537 (0.22313731979086715) | > loss_disc_real_1: 0.22084325551986694 (0.23335643871775213) | > loss_disc_real_2: 0.2537275552749634 (0.2388021265560726) | > loss_disc_real_3: 0.22623108327388763 (0.2368180661831262) | > loss_disc_real_4: 0.2617863118648529 (0.2425081637108101) | > loss_disc_real_5: 0.25622648000717163 (0.24514655853217504) | > loss_0: 2.890383243560791 (2.813645691241858) | > grad_norm_0: tensor(6.1483, device='cuda:0') (tensor(11.8886, device='cuda:0')) | > loss_gen: 1.72802734375 (1.7697322391114145) | > loss_kl: 2.351485252380371 (2.3306877837990814) | > loss_feat: 1.9886480569839478 (2.1538476494123353) | > loss_mel: 16.63813591003418 (17.910568939065026) | > loss_duration: 1.6649446487426758 (1.6320934363131254) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.37124252319336 (25.796930169159513) | > grad_norm_1: tensor(298.4833, device='cuda:0') (tensor(247.4039, device='cuda:0')) | > current_lr_0: 0.00019584353584765863 | > current_lr_1: 0.00019584353584765863 | > step_time: 0.4519 (0.4451946492465037) | > loader_time: 0.0091 (0.00787533904021641) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865942659512372 (-2.5960761056820214e-05) | > avg_step_time: 0.45627238045276053 (-0.0017355898736228226) | > avg_loss_disc: 2.820389811421784 (-0.004015217364673251) | > avg_loss_disc_real_0: 0.22048890915974764 (+0.004228340802897856) | > avg_loss_disc_real_1: 0.23336588937631794 (-0.004965249921234566) | > avg_loss_disc_real_2: 0.23833267747516362 (-0.0023426759830663113) | > avg_loss_disc_real_3: 0.2377626056402502 (-0.001990683481726846) | > avg_loss_disc_real_4: 0.24413805108674816 (+0.0008878216777049308) | > avg_loss_disc_real_5: 0.24659464707676793 (+0.0013565041229758323) | > avg_loss_0: 2.820389811421784 (-0.004015217364673251) | > avg_grad_norm_0: tensor(12.0347, device='cuda:0') (+tensor(0.8396, device='cuda:0')) | > avg_loss_gen: 1.761109792010885 (-0.0032742443219038186) | > avg_loss_kl: 2.3190879385236283 (-0.021401942615778324) | > avg_loss_feat: 2.1178870956662674 (+0.009073289347366842) | > avg_loss_mel: 17.75339717596349 (-0.11918621331874135) | > avg_loss_duration: 1.6401564806280002 (+0.0023716755316292026) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.591638538199412 (-0.13241749078454745) | > avg_grad_norm_1: tensor(282.4809, device='cuda:0') (+tensor(23.1135, device='cuda:0'))  > EPOCH: 169/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:55:35)   --> TIME: 2025-02-25 09:55:50 -- STEP: 31/72 -- GLOBAL_STEP: 1012200 | > loss_disc: 2.8145689964294434 (2.8301757535626813) | > loss_disc_real_0: 0.20130571722984314 (0.2355683849703881) | > loss_disc_real_1: 0.14878658950328827 (0.2376346371827587) | > loss_disc_real_2: 0.25169140100479126 (0.24361256918599528) | > loss_disc_real_3: 0.24582059681415558 (0.241962896239373) | > loss_disc_real_4: 0.24811823666095734 (0.24394673301327613) | > loss_disc_real_5: 0.24141626060009003 (0.2443414503528226) | > loss_0: 2.8145689964294434 (2.8301757535626813) | > grad_norm_0: tensor(10.3504, device='cuda:0') (tensor(13.4496, device='cuda:0')) | > loss_gen: 1.7724262475967407 (1.7950935979043283) | > loss_kl: 2.275085687637329 (2.3042819846060967) | > loss_feat: 2.2049953937530518 (2.150259948545887) | > loss_mel: 18.080272674560547 (17.843368161109183) | > loss_duration: 1.6328837871551514 (1.6069296713798278) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.96566390991211 (25.699933144354052) | > grad_norm_1: tensor(418.8703, device='cuda:0') (tensor(348.2957, device='cuda:0')) | > current_lr_0: 0.00019581905540567768 | > current_lr_1: 0.00019581905540567768 | > step_time: 0.4405 (0.43589583519966374) | > loader_time: 0.0095 (0.007461332505749118) --> EVAL PERFORMANCE | > avg_loader_time: 0.008741331772065502 (+8.190517694178084e-05) | > avg_step_time: 0.4547091067676813 (-0.0015632736850792472) | > avg_loss_disc: 2.833986332718755 (+0.013596521296971087) | > avg_loss_disc_real_0: 0.22694835916791165 (+0.006459450008164003) | > avg_loss_disc_real_1: 0.23762447796237302 (+0.004258588586055073) | > avg_loss_disc_real_2: 0.24353094105149659 (+0.005198263576332962) | > avg_loss_disc_real_3: 0.24232184488168904 (+0.004559239241438834) | > avg_loss_disc_real_4: 0.24479707676759907 (+0.000659025680850911) | > avg_loss_disc_real_5: 0.24636399032364428 (-0.00023065675312364253) | > avg_loss_0: 2.833986332718755 (+0.013596521296971087) | > avg_grad_norm_0: tensor(12.7741, device='cuda:0') (+tensor(0.7394, device='cuda:0')) | > avg_loss_gen: 1.7753369808197021 (+0.014227188808817237) | > avg_loss_kl: 2.3034429499800773 (-0.01564498854355101) | > avg_loss_feat: 2.0955875306062297 (-0.022299565060037718) | > avg_loss_mel: 17.891854595130603 (+0.13845741916711418) | > avg_loss_duration: 1.6338996232395442 (-0.00625685738845605) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.700121704961212 (+0.10848316676180048) | > avg_grad_norm_1: tensor(345.9680, device='cuda:0') (+tensor(63.4871, device='cuda:0'))  > EPOCH: 170/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:56:12)   --> TIME: 2025-02-25 09:56:18 -- STEP: 9/72 -- GLOBAL_STEP: 1012250 | > loss_disc: 2.7413980960845947 (2.7853462431165905) | > loss_disc_real_0: 0.2075081467628479 (0.24453044020467335) | > loss_disc_real_1: 0.19037501513957977 (0.21852266126208836) | > loss_disc_real_2: 0.24546419084072113 (0.21527720822228325) | > loss_disc_real_3: 0.24067392945289612 (0.23504454559750027) | > loss_disc_real_4: 0.22863566875457764 (0.2289245989587572) | > loss_disc_real_5: 0.2230997234582901 (0.23789502680301666) | > loss_0: 2.7413980960845947 (2.7853462431165905) | > grad_norm_0: tensor(5.8194, device='cuda:0') (tensor(16.4311, device='cuda:0')) | > loss_gen: 1.7748408317565918 (1.742776870727539) | > loss_kl: 2.5547852516174316 (2.328525278303358) | > loss_feat: 2.168794631958008 (2.1686806546317206) | > loss_mel: 18.623552322387695 (18.28740225897895) | > loss_duration: 1.627553939819336 (1.6217942900127835) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.749526977539062 (26.14917967054579) | > grad_norm_1: tensor(226.7768, device='cuda:0') (tensor(272.8720, device='cuda:0')) | > current_lr_0: 0.00019579457802375197 | > current_lr_1: 0.00019579457802375197 | > step_time: 0.4254 (0.43320616086324054) | > loader_time: 0.0056 (0.006346013810899522)  --> TIME: 2025-02-25 09:56:41 -- STEP: 59/72 -- GLOBAL_STEP: 1012300 | > loss_disc: 2.9295849800109863 (2.8185226674807273) | > loss_disc_real_0: 0.2933250963687897 (0.21593426395270784) | > loss_disc_real_1: 0.21727615594863892 (0.23686810898578772) | > loss_disc_real_2: 0.26970845460891724 (0.2366494493464292) | > loss_disc_real_3: 0.27451983094215393 (0.2394763236328707) | > loss_disc_real_4: 0.2667056620121002 (0.24200157947459464) | > loss_disc_real_5: 0.27244728803634644 (0.24422166256581324) | > loss_0: 2.9295849800109863 (2.8185226674807273) | > grad_norm_0: tensor(14.9476, device='cuda:0') (tensor(11.6652, device='cuda:0')) | > loss_gen: 1.7623450756072998 (1.7639142295061532) | > loss_kl: 2.419826030731201 (2.3593842983245845) | > loss_feat: 2.0673561096191406 (2.1630416805461294) | > loss_mel: 17.925769805908203 (18.013993570360086) | > loss_duration: 1.6696863174438477 (1.6351236953573711) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.844982147216797 (25.935457294270144) | > grad_norm_1: tensor(531.8218, device='cuda:0') (tensor(260.1732, device='cuda:0')) | > current_lr_0: 0.00019579457802375197 | > current_lr_1: 0.00019579457802375197 | > step_time: 0.4794 (0.4496320708323333) | > loader_time: 0.0098 (0.007945780026710638) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855069764902894 (-0.00019063412303656213) | > avg_step_time: 0.4579336240258015 (+0.0032245172581202297) | > avg_loss_disc: 2.8206549496717854 (-0.01333138304696968) | > avg_loss_disc_real_0: 0.2162468437577637 (-0.010701515410147938) | > avg_loss_disc_real_1: 0.23581876532292703 (-0.0018057126394459833) | > avg_loss_disc_real_2: 0.2356107661002119 (-0.007920174951284692) | > avg_loss_disc_real_3: 0.23984444645089162 (-0.0024773984307974173) | > avg_loss_disc_real_4: 0.24251886950412266 (-0.0022782072634764117) | > avg_loss_disc_real_5: 0.24487522475316492 (-0.0014887655704793612) | > avg_loss_0: 2.8206549496717854 (-0.01333138304696968) | > avg_grad_norm_0: tensor(12.4281, device='cuda:0') (tensor(-0.3460, device='cuda:0')) | > avg_loss_gen: 1.7574591670237796 (-0.01787781379592257) | > avg_loss_kl: 2.361451501577672 (+0.05800855159759477) | > avg_loss_feat: 2.125166253304817 (+0.02957872269858708) | > avg_loss_mel: 17.99938483976982 (+0.1075302446392179) | > avg_loss_duration: 1.642593049667251 (+0.008693426427706763) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.88605464344293 (+0.18593293848171655) | > avg_grad_norm_1: tensor(256.3997, device='cuda:0') (tensor(-89.5683, device='cuda:0'))  > EPOCH: 171/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:56:49)   --> TIME: 2025-02-25 09:57:08 -- STEP: 37/72 -- GLOBAL_STEP: 1012350 | > loss_disc: 3.0043540000915527 (2.85477479084118) | > loss_disc_real_0: 0.12906619906425476 (0.21993805347262202) | > loss_disc_real_1: 0.27673283219337463 (0.24791324501102036) | > loss_disc_real_2: 0.29801419377326965 (0.25027930414354477) | > loss_disc_real_3: 0.26226291060447693 (0.25074097554425934) | > loss_disc_real_4: 0.22378364205360413 (0.24968210829270854) | > loss_disc_real_5: 0.22105464339256287 (0.250182578289831) | > loss_0: 3.0043540000915527 (2.85477479084118) | > grad_norm_0: tensor(17.6202, device='cuda:0') (tensor(12.4615, device='cuda:0')) | > loss_gen: 1.7752549648284912 (1.8045642150414956) | > loss_kl: 2.3052520751953125 (2.3135063326036605) | > loss_feat: 1.9289326667785645 (2.129045473562704) | > loss_mel: 17.333154678344727 (18.017278671264652) | > loss_duration: 1.6234567165374756 (1.6120559395970524) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.96605110168457 (25.876450873710013) | > grad_norm_1: tensor(78.5634, device='cuda:0') (tensor(303.9006, device='cuda:0')) | > current_lr_0: 0.000195770103701499 | > current_lr_1: 0.000195770103701499 | > step_time: 0.4429 (0.43818683882017395) | > loader_time: 0.0079 (0.007205834259858003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624345483914227 (+7.364783488528724e-05) | > avg_step_time: 0.4547416760887898 (-0.0031919479370117188) | > avg_loss_disc: 2.8547817149632415 (+0.034126765291456085) | > avg_loss_disc_real_0: 0.22616500124125413 (+0.009918157483490425) | > avg_loss_disc_real_1: 0.24012032864798963 (+0.004301563325062596) | > avg_loss_disc_real_2: 0.24930725215186544 (+0.01369648605165355) | > avg_loss_disc_real_3: 0.24830114421710156 (+0.008456697766209942) | > avg_loss_disc_real_4: 0.24837014834645768 (+0.005851278842335023) | > avg_loss_disc_real_5: 0.25183820766462417 (+0.006962982911459248) | > avg_loss_0: 2.8547817149632415 (+0.034126765291456085) | > avg_grad_norm_0: tensor(12.0555, device='cuda:0') (tensor(-0.3725, device='cuda:0')) | > avg_loss_gen: 1.786655174174779 (+0.029196007150999348) | > avg_loss_kl: 2.302938812215563 (-0.05851268936210907) | > avg_loss_feat: 2.0805239744589348 (-0.04464227884588201) | > avg_loss_mel: 17.788012934402687 (-0.2113719053671339) | > avg_loss_duration: 1.6362869756322513 (-0.006306074034999609) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.594417921254333 (-0.2916367221885956) | > avg_grad_norm_1: tensor(327.7130, device='cuda:0') (+tensor(71.3133, device='cuda:0'))  > EPOCH: 172/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:57:26)   --> TIME: 2025-02-25 09:57:35 -- STEP: 15/72 -- GLOBAL_STEP: 1012400 | > loss_disc: 2.768059730529785 (2.821458641688029) | > loss_disc_real_0: 0.15064756572246552 (0.21460827589035034) | > loss_disc_real_1: 0.29353150725364685 (0.25002606213092804) | > loss_disc_real_2: 0.2187410145998001 (0.23525602519512176) | > loss_disc_real_3: 0.263467937707901 (0.24464941918849945) | > loss_disc_real_4: 0.230699822306633 (0.2387971967458725) | > loss_disc_real_5: 0.2465294897556305 (0.24182697832584382) | > loss_0: 2.768059730529785 (2.821458641688029) | > grad_norm_0: tensor(14.9813, device='cuda:0') (tensor(11.8379, device='cuda:0')) | > loss_gen: 1.7063305377960205 (1.7698248545328776) | > loss_kl: 2.2882401943206787 (2.1641642570495603) | > loss_feat: 2.1310741901397705 (2.1306095759073895) | > loss_mel: 17.910634994506836 (17.86635551452637) | > loss_duration: 1.5826984643936157 (1.6056212107340495) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.61897850036621 (25.536575317382812) | > grad_norm_1: tensor(203.3456, device='cuda:0') (tensor(359.2890, device='cuda:0')) | > current_lr_0: 0.0001957456324385363 | > current_lr_1: 0.0001957456324385363 | > step_time: 0.4335 (0.43628260294596355) | > loader_time: 0.0062 (0.006720638275146485)  --> TIME: 2025-02-25 09:57:58 -- STEP: 65/72 -- GLOBAL_STEP: 1012450 | > loss_disc: 2.8432154655456543 (2.829165304624117) | > loss_disc_real_0: 0.2915049195289612 (0.22126435454075152) | > loss_disc_real_1: 0.22733506560325623 (0.23645724333249607) | > loss_disc_real_2: 0.2721327245235443 (0.23756678585822766) | > loss_disc_real_3: 0.26312530040740967 (0.2420023127244069) | > loss_disc_real_4: 0.258312851190567 (0.24389161811425136) | > loss_disc_real_5: 0.26491883397102356 (0.24516493036196782) | > loss_0: 2.8432154655456543 (2.829165304624117) | > grad_norm_0: tensor(18.8145, device='cuda:0') (tensor(12.1980, device='cuda:0')) | > loss_gen: 1.7914713621139526 (1.7606890935164232) | > loss_kl: 2.5086092948913574 (2.289100980758667) | > loss_feat: 1.8984630107879639 (2.0978215511028577) | > loss_mel: 17.551509857177734 (17.89929791964017) | > loss_duration: 1.692379355430603 (1.632840475669274) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.442432403564453 (25.679749767596906) | > grad_norm_1: tensor(199.7432, device='cuda:0') (tensor(303.1269, device='cuda:0')) | > current_lr_0: 0.0001957456324385363 | > current_lr_1: 0.0001957456324385363 | > step_time: 0.4871 (0.4497453322777381) | > loader_time: 0.0118 (0.008293757071861856) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579546297100232 (-4.479918681399471e-05) | > avg_step_time: 0.4560808161614646 (+0.001339140072674827) | > avg_loss_disc: 2.825090532571497 (-0.02969118239174451) | > avg_loss_disc_real_0: 0.22286051113001057 (-0.003304490111243563) | > avg_loss_disc_real_1: 0.23611863272290834 (-0.004001695925081289) | > avg_loss_disc_real_2: 0.23630504687906992 (-0.013002205272795525) | > avg_loss_disc_real_3: 0.24106278364926997 (-0.0072383605678315865) | > avg_loss_disc_real_4: 0.24279962585005962 (-0.005570522496398056) | > avg_loss_disc_real_5: 0.24522903561592102 (-0.006609172048703149) | > avg_loss_0: 2.825090532571497 (-0.02969118239174451) | > avg_grad_norm_0: tensor(12.4605, device='cuda:0') (+tensor(0.4049, device='cuda:0')) | > avg_loss_gen: 1.7599705155466643 (-0.026684658628114644) | > avg_loss_kl: 2.2975897621101056 (-0.005349050105457387) | > avg_loss_feat: 2.093104616017409 (+0.01258064155847416) | > avg_loss_mel: 17.865692165535936 (+0.07767923113324926) | > avg_loss_duration: 1.6365197964117562 (+0.0002328207795048165) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.652876571870188 (+0.058458650615854424) | > avg_grad_norm_1: tensor(294.3449, device='cuda:0') (tensor(-33.3680, device='cuda:0'))  > EPOCH: 173/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:58:03)   --> TIME: 2025-02-25 09:58:24 -- STEP: 43/72 -- GLOBAL_STEP: 1012500 | > loss_disc: 2.8524510860443115 (2.817459106445313) | > loss_disc_real_0: 0.2003340721130371 (0.21825682181258535) | > loss_disc_real_1: 0.22865180671215057 (0.23479199028292366) | > loss_disc_real_2: 0.26340848207473755 (0.2417025864124298) | > loss_disc_real_3: 0.276853084564209 (0.24122734610424484) | > loss_disc_real_4: 0.2553943395614624 (0.24595980900664663) | > loss_disc_real_5: 0.26485171914100647 (0.2449295184639997) | > loss_0: 2.8524510860443115 (2.817459106445313) | > grad_norm_0: tensor(9.5238, device='cuda:0') (tensor(13.6325, device='cuda:0')) | > loss_gen: 1.8107305765151978 (1.7867393133252165) | > loss_kl: 2.2595021724700928 (2.275202426799508) | > loss_feat: 1.8727790117263794 (2.1820507659468538) | > loss_mel: 16.965187072753906 (17.965318591095674) | > loss_duration: 1.6232104301452637 (1.614288989887681) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.531410217285156 (25.823600148045735) | > grad_norm_1: tensor(532.3546, device='cuda:0') (tensor(278.7784, device='cuda:0')) | > current_lr_0: 0.00019572116423448148 | > current_lr_1: 0.00019572116423448148 | > step_time: 0.4559 (0.4405643607294837) | > loader_time: 0.0096 (0.007539239040640897) --> EVAL PERFORMANCE | > avg_loader_time: 0.008673866030196068 (+9.431973309583647e-05) | > avg_step_time: 0.4627415764499718 (+0.006660760288507206) | > avg_loss_disc: 2.8228023085795653 (-0.002288223991931737) | > avg_loss_disc_real_0: 0.21840088163882915 (-0.004459629491181422) | > avg_loss_disc_real_1: 0.23639721916594975 (+0.00027858644304140645) | > avg_loss_disc_real_2: 0.24285081070913395 (+0.0065457638300640275) | > avg_loss_disc_real_3: 0.23970388568622966 (-0.001358897963040312) | > avg_loss_disc_real_4: 0.24479033514647416 (+0.001990709296414539) | > avg_loss_disc_real_5: 0.2447282264769917 (-0.0005008091389293312) | > avg_loss_0: 2.8228023085795653 (-0.002288223991931737) | > avg_grad_norm_0: tensor(15.1766, device='cuda:0') (+tensor(2.7162, device='cuda:0')) | > avg_loss_gen: 1.7737155044582529 (+0.013744988911588596) | > avg_loss_kl: 2.264140117336327 (-0.03344964477377843) | > avg_loss_feat: 2.134417711848944 (+0.04131309583153486) | > avg_loss_mel: 17.7619489347431 (-0.10374323079283698) | > avg_loss_duration: 1.630088383043316 (-0.00643141336844022) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.56431063799791 (-0.08856593387227818) | > avg_grad_norm_1: tensor(321.6850, device='cuda:0') (+tensor(27.3401, device='cuda:0'))  > EPOCH: 174/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:58:40)   --> TIME: 2025-02-25 09:58:52 -- STEP: 21/72 -- GLOBAL_STEP: 1012550 | > loss_disc: 2.7966747283935547 (2.799616802306402) | > loss_disc_real_0: 0.2247387170791626 (0.20223069616726466) | > loss_disc_real_1: 0.27465173602104187 (0.241015667007083) | > loss_disc_real_2: 0.2495521605014801 (0.24213514583451407) | > loss_disc_real_3: 0.2502221465110779 (0.23950592818714322) | > loss_disc_real_4: 0.28283584117889404 (0.24464228039696104) | > loss_disc_real_5: 0.3016551434993744 (0.24767879928861344) | > loss_0: 2.7966747283935547 (2.799616802306402) | > grad_norm_0: tensor(4.6414, device='cuda:0') (tensor(9.2441, device='cuda:0')) | > loss_gen: 1.7223161458969116 (1.7961967899685813) | > loss_kl: 2.5120749473571777 (2.2360505035945346) | > loss_feat: 2.088904857635498 (2.260378843262082) | > loss_mel: 17.20878791809082 (18.043525423322404) | > loss_duration: 1.6203514337539673 (1.6010729301543463) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.152437210083008 (25.937224615187873) | > grad_norm_1: tensor(64.7632, device='cuda:0') (tensor(291.2126, device='cuda:0')) | > current_lr_0: 0.00019569669908895215 | > current_lr_1: 0.00019569669908895215 | > step_time: 0.4479 (0.43669605255126953) | > loader_time: 0.0095 (0.006952387945992606)  --> TIME: 2025-02-25 09:59:16 -- STEP: 71/72 -- GLOBAL_STEP: 1012600 | > loss_disc: 2.815497875213623 (2.822155499122512) | > loss_disc_real_0: 0.23768389225006104 (0.22205158952676074) | > loss_disc_real_1: 0.2229541689157486 (0.2343519972243779) | > loss_disc_real_2: 0.2175367772579193 (0.23783189352129547) | > loss_disc_real_3: 0.2262951284646988 (0.23937611113971388) | > loss_disc_real_4: 0.24578243494033813 (0.24559952780394487) | > loss_disc_real_5: 0.26595351099967957 (0.24986243604774205) | > loss_0: 2.815497875213623 (2.822155499122512) | > grad_norm_0: tensor(5.6741, device='cuda:0') (tensor(13.4466, device='cuda:0')) | > loss_gen: 1.7790968418121338 (1.7914452485635246) | > loss_kl: 1.9610605239868164 (2.3600994667536783) | > loss_feat: 1.8569425344467163 (2.186795902923798) | > loss_mel: 15.541149139404297 (18.033573486435575) | > loss_duration: 1.6481425762176514 (1.6324359648664233) | > amp_scaler: 256.0 (256.0) | > loss_1: 22.78639030456543 (26.00435012495014) | > grad_norm_1: tensor(284.1098, device='cuda:0') (tensor(277.9865, device='cuda:0')) | > current_lr_0: 0.00019569669908895215 | > current_lr_1: 0.00019569669908895215 | > step_time: 0.5038 (0.4579942595790809) | > loader_time: 0.0081 (0.008714188992137637) --> EVAL PERFORMANCE | > avg_loader_time: 0.008714188992137637 (+4.032296194156869e-05) | > avg_step_time: 0.4579942595790809 (-0.004747316870890916) | > avg_loss_disc: 2.822155499122512 (-0.000646809457053088) | > avg_loss_disc_real_0: 0.22205158952676074 (+0.0036507078879315924) | > avg_loss_disc_real_1: 0.2343519972243779 (-0.0020452219415718342) | > avg_loss_disc_real_2: 0.23783189352129547 (-0.005018917187838479) | > avg_loss_disc_real_3: 0.23937611113971388 (-0.00032777454651578) | > avg_loss_disc_real_4: 0.24559952780394487 (+0.0008091926574707031) | > avg_loss_disc_real_5: 0.24986243604774205 (+0.005134209570750364) | > avg_loss_0: 2.822155499122512 (-0.000646809457053088) | > avg_grad_norm_0: tensor(13.4466, device='cuda:0') (tensor(-1.7301, device='cuda:0')) | > avg_loss_gen: 1.7914452485635246 (+0.017729744105271728) | > avg_loss_kl: 2.3600994667536783 (+0.09595934941735118) | > avg_loss_feat: 2.186795902923798 (+0.0523781910748542) | > avg_loss_mel: 18.033573486435575 (+0.2716245516924758) | > avg_loss_duration: 1.6324359648664233 (+0.0023475818231073475) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.00435012495014 (+0.44003948695223016) | > avg_grad_norm_1: tensor(277.9865, device='cuda:0') (tensor(-43.6985, device='cuda:0'))  > EPOCH: 175/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:59:18)   --> TIME: 2025-02-25 09:59:42 -- STEP: 49/72 -- GLOBAL_STEP: 1012650 | > loss_disc: 2.9417927265167236 (2.8231905917732076) | > loss_disc_real_0: 0.18014228343963623 (0.21612371139380396) | > loss_disc_real_1: 0.2492857426404953 (0.2387002898114068) | > loss_disc_real_2: 0.26155897974967957 (0.24376944528550518) | > loss_disc_real_3: 0.23413188755512238 (0.24535205473705213) | > loss_disc_real_4: 0.23768013715744019 (0.24562051801048979) | > loss_disc_real_5: 0.24223855137825012 (0.24437291889774557) | > loss_0: 2.9417927265167236 (2.8231905917732076) | > grad_norm_0: tensor(14.0042, device='cuda:0') (tensor(11.7058, device='cuda:0')) | > loss_gen: 1.7359219789505005 (1.7969343637933537) | > loss_kl: 2.2093069553375244 (2.303843357125107) | > loss_feat: 1.9057726860046387 (2.1508697952542986) | > loss_mel: 16.76009750366211 (17.7918171006806) | > loss_duration: 1.7204105854034424 (1.6211173656035442) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.331510543823242 (25.664582116263254) | > grad_norm_1: tensor(68.1669, device='cuda:0') (tensor(234.5765, device='cuda:0')) | > current_lr_0: 0.00019567223700156603 | > current_lr_1: 0.00019567223700156603 | > step_time: 0.4726 (0.4469800044079216) | > loader_time: 0.0096 (0.007834809167044503) --> EVAL PERFORMANCE | > avg_loader_time: 0.008872280658130918 (+0.00015809166599328045) | > avg_step_time: 0.45784411967640193 (-0.0001501399026789807) | > avg_loss_disc: 2.8261727514401285 (+0.004017252317616293) | > avg_loss_disc_real_0: 0.2214281711150223 (-0.0006234184117384356) | > avg_loss_disc_real_1: 0.2396472369281339 (+0.005295239703755977) | > avg_loss_disc_real_2: 0.24324323341880047 (+0.005411339897505002) | > avg_loss_disc_real_3: 0.24352187317022136 (+0.004145762030507483) | > avg_loss_disc_real_4: 0.24535938866541418 (-0.00024013913853068725) | > avg_loss_disc_real_5: 0.24473101887064921 (-0.00513141717709284) | > avg_loss_0: 2.8261727514401285 (+0.004017252317616293) | > avg_grad_norm_0: tensor(13.3251, device='cuda:0') (tensor(-0.1215, device='cuda:0')) | > avg_loss_gen: 1.7945610838876644 (+0.0031158353241398373) | > avg_loss_kl: 2.308298265430289 (-0.05180120132338928) | > avg_loss_feat: 2.147646234069072 (-0.03914966885472593) | > avg_loss_mel: 17.886846596086524 (-0.14672689034905062) | > avg_loss_duration: 1.6361625815781067 (+0.003726616711683395) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.773514949100118 (-0.23083517585002156) | > avg_grad_norm_1: tensor(253.8371, device='cuda:0') (tensor(-24.1493, device='cuda:0'))  > EPOCH: 176/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:59:55)   --> TIME: 2025-02-25 10:00:09 -- STEP: 27/72 -- GLOBAL_STEP: 1012700 | > loss_disc: 2.905198335647583 (2.8271034117098206) | > loss_disc_real_0: 0.26468682289123535 (0.21892105208502877) | > loss_disc_real_1: 0.2136104553937912 (0.2293742053486683) | > loss_disc_real_2: 0.20648327469825745 (0.23859302478807945) | > loss_disc_real_3: 0.22817492485046387 (0.23854332665602365) | > loss_disc_real_4: 0.22212114930152893 (0.24683972310136865) | > loss_disc_real_5: 0.20803844928741455 (0.2478123406569163) | > loss_0: 2.905198335647583 (2.8271034117098206) | > grad_norm_0: tensor(10.5798, device='cuda:0') (tensor(10.8352, device='cuda:0')) | > loss_gen: 1.7765543460845947 (1.7752992488719799) | > loss_kl: 2.341334342956543 (2.2575889340153448) | > loss_feat: 2.205997943878174 (2.231486960693642) | > loss_mel: 17.99836540222168 (17.94154188368055) | > loss_duration: 1.5903443098068237 (1.602970216009352) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.912595748901367 (25.808887199119287) | > grad_norm_1: tensor(544.0786, device='cuda:0') (tensor(254.7519, device='cuda:0')) | > current_lr_0: 0.00019564777797194082 | > current_lr_1: 0.00019564777797194082 | > step_time: 0.4443 (0.4571225378248427) | > loader_time: 0.0081 (0.0071895210831253616) --> EVAL PERFORMANCE | > avg_loader_time: 0.008822659371604383 (-4.962128652653483e-05) | > avg_step_time: 0.46477997470909443 (+0.006935855032692506) | > avg_loss_disc: 2.818496814915832 (-0.007675936524296656) | > avg_loss_disc_real_0: 0.21911867393154497 (-0.002309497183477338) | > avg_loss_disc_real_1: 0.23389220500076321 (-0.005755031927370674) | > avg_loss_disc_real_2: 0.24130436022516707 (-0.0019388731936333947) | > avg_loss_disc_real_3: 0.2410604911790767 (-0.002461381991144651) | > avg_loss_disc_real_4: 0.24496487055865812 (-0.0003945181067560588) | > avg_loss_disc_real_5: 0.24445458533058703 (-0.000276433540062182) | > avg_loss_0: 2.818496814915832 (-0.007675936524296656) | > avg_grad_norm_0: tensor(13.4924, device='cuda:0') (+tensor(0.1673, device='cuda:0')) | > avg_loss_gen: 1.7846666524108028 (-0.009894431476861643) | > avg_loss_kl: 2.267004820662485 (-0.041293444767803944) | > avg_loss_feat: 2.199047766940694 (+0.05140153287162175) | > avg_loss_mel: 17.822796848458303 (-0.06404974762822135) | > avg_loss_duration: 1.6367994429359973 (+0.0006368613578906501) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.71031554316131 (-0.06319940593880702) | > avg_grad_norm_1: tensor(306.9163, device='cuda:0') (+tensor(53.0792, device='cuda:0'))  > EPOCH: 177/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:00:32)   --> TIME: 2025-02-25 10:00:36 -- STEP: 5/72 -- GLOBAL_STEP: 1012750 | > loss_disc: 2.816584587097168 (2.8257405757904053) | > loss_disc_real_0: 0.14223118126392365 (0.2060187578201294) | > loss_disc_real_1: 0.24676136672496796 (0.2309532046318054) | > loss_disc_real_2: 0.23385420441627502 (0.24014154374599456) | > loss_disc_real_3: 0.22819672524929047 (0.2378053694963455) | > loss_disc_real_4: 0.2566602826118469 (0.26245457530021665) | > loss_disc_real_5: 0.2573731243610382 (0.2618259429931641) | > loss_0: 2.816584587097168 (2.8257405757904053) | > grad_norm_0: tensor(18.8121, device='cuda:0') (tensor(9.6548, device='cuda:0')) | > loss_gen: 1.7102606296539307 (1.766923999786377) | > loss_kl: 2.3369104862213135 (2.258838415145874) | > loss_feat: 2.0617828369140625 (2.120105028152466) | > loss_mel: 17.531150817871094 (18.142479705810548) | > loss_duration: 1.549399495124817 (1.6120486736297608) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.189504623413086 (25.900395965576173) | > grad_norm_1: tensor(181.1564, device='cuda:0') (tensor(241.9164, device='cuda:0')) | > current_lr_0: 0.00019562332199969432 | > current_lr_1: 0.00019562332199969432 | > step_time: 0.4329 (0.43597412109375) | > loader_time: 0.0062 (0.006321859359741211)  --> TIME: 2025-02-25 10:01:00 -- STEP: 55/72 -- GLOBAL_STEP: 1012800 | > loss_disc: 2.8850808143615723 (2.841423684900457) | > loss_disc_real_0: 0.18684959411621094 (0.23727373033761978) | > loss_disc_real_1: 0.2828323543071747 (0.23773465156555176) | > loss_disc_real_2: 0.24107564985752106 (0.24354747398333115) | > loss_disc_real_3: 0.24143604934215546 (0.24006196477196434) | > loss_disc_real_4: 0.24672165513038635 (0.24519952102140946) | > loss_disc_real_5: 0.2404061257839203 (0.24650859805670652) | > loss_0: 2.8850808143615723 (2.841423684900457) | > grad_norm_0: tensor(7.1959, device='cuda:0') (tensor(14.2313, device='cuda:0')) | > loss_gen: 1.7621108293533325 (1.8027719215913252) | > loss_kl: 2.4651401042938232 (2.454294798590919) | > loss_feat: 1.8585259914398193 (2.177790548584678) | > loss_mel: 18.505420684814453 (18.311367034912116) | > loss_duration: 1.670180082321167 (1.6262161645022306) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.261377334594727 (26.37244037281383) | > grad_norm_1: tensor(40.3609, device='cuda:0') (tensor(268.5282, device='cuda:0')) | > current_lr_0: 0.00019562332199969432 | > current_lr_1: 0.00019562332199969432 | > step_time: 0.4621 (0.4470415635542436) | > loader_time: 0.0097 (0.007878515937111594) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652556110435807 (-0.00017010326116857592) | > avg_step_time: 0.45655564858879844 (-0.008224326120296) | > avg_loss_disc: 2.830501287755832 (+0.012004472840000169) | > avg_loss_disc_real_0: 0.2315621735974097 (+0.012443499665864721) | > avg_loss_disc_real_1: 0.2352609821188618 (+0.0013687771180985742) | > avg_loss_disc_real_2: 0.2441279067539833 (+0.002823546528816223) | > avg_loss_disc_real_3: 0.23962289073937376 (-0.0014376004397029518) | > avg_loss_disc_real_4: 0.24459793890865755 (-0.00036693165000056815) | > avg_loss_disc_real_5: 0.245763307516004 (+0.0013087221854169717) | > avg_loss_0: 2.830501287755832 (+0.012004472840000169) | > avg_grad_norm_0: tensor(13.0072, device='cuda:0') (tensor(-0.4852, device='cuda:0')) | > avg_loss_gen: 1.790499290949862 (+0.005832638539059243) | > avg_loss_kl: 2.4654994884007406 (+0.19849466773825553) | > avg_loss_feat: 2.161549467436025 (-0.037498299504668875) | > avg_loss_mel: 18.22021374232333 (+0.397416893865028) | > avg_loss_duration: 1.6403615407540764 (+0.0035620978180790352) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.278123452629842 (+0.5678079094685309) | > avg_grad_norm_1: tensor(273.1767, device='cuda:0') (tensor(-33.7396, device='cuda:0'))  > EPOCH: 178/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:01:09)   --> TIME: 2025-02-25 10:01:26 -- STEP: 33/72 -- GLOBAL_STEP: 1012850 | > loss_disc: 2.72442626953125 (2.8132951693101362) | > loss_disc_real_0: 0.2692662179470062 (0.21920731934634122) | > loss_disc_real_1: 0.28774553537368774 (0.23342953035325714) | > loss_disc_real_2: 0.22339200973510742 (0.238809220718615) | > loss_disc_real_3: 0.23670218884944916 (0.2408261980974313) | > loss_disc_real_4: 0.19853022694587708 (0.24576810228102136) | > loss_disc_real_5: 0.22791941463947296 (0.24706653528141254) | > loss_0: 2.72442626953125 (2.8132951693101362) | > grad_norm_0: tensor(12.7911, device='cuda:0') (tensor(9.2883, device='cuda:0')) | > loss_gen: 1.905833125114441 (1.787759390744296) | > loss_kl: 2.31534481048584 (2.3207858042283487) | > loss_feat: 2.4878978729248047 (2.2035003033551304) | > loss_mel: 18.009952545166016 (17.837008505156547) | > loss_duration: 1.616304874420166 (1.6048170906124692) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.335330963134766 (25.753871108546402) | > grad_norm_1: tensor(417.8248, device='cuda:0') (tensor(309.7512, device='cuda:0')) | > current_lr_0: 0.00019559886908444434 | > current_lr_1: 0.00019559886908444434 | > step_time: 0.4595 (0.4370774066809452) | > loader_time: 0.01 (0.007267150011929599) --> EVAL PERFORMANCE | > avg_loader_time: 0.008816037379520041 (+0.0001634812690842341) | > avg_step_time: 0.45705263715394784 (+0.0004969885651494033) | > avg_loss_disc: 2.8233466786398016 (-0.007154609116030386) | > avg_loss_disc_real_0: 0.22375138166924596 (-0.007810791928163724) | > avg_loss_disc_real_1: 0.23065325974578588 (-0.004607722373075912) | > avg_loss_disc_real_2: 0.24147709868323636 (-0.0026508080707469406) | > avg_loss_disc_real_3: 0.24231079514597503 (+0.0026879044066012714) | > avg_loss_disc_real_4: 0.24521175803433 (+0.0006138191256724601) | > avg_loss_disc_real_5: 0.2485315524776217 (+0.002768244961617705) | > avg_loss_0: 2.8233466786398016 (-0.007154609116030386) | > avg_grad_norm_0: tensor(10.7380, device='cuda:0') (tensor(-2.2692, device='cuda:0')) | > avg_loss_gen: 1.7772882018290774 (-0.013211089120784614) | > avg_loss_kl: 2.3164113038022753 (-0.1490881845984653) | > avg_loss_feat: 2.13798870503063 (-0.023560762405395064) | > avg_loss_mel: 17.713598560279525 (-0.5066151820438058) | > avg_loss_duration: 1.6293133329337752 (-0.011048207820301137) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.574599924221843 (-0.7035235284079988) | > avg_grad_norm_1: tensor(292.6168, device='cuda:0') (+tensor(19.4401, device='cuda:0'))  > EPOCH: 179/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:01:46)   --> TIME: 2025-02-25 10:01:53 -- STEP: 11/72 -- GLOBAL_STEP: 1012900 | > loss_disc: 2.8983237743377686 (2.8167772726579146) | > loss_disc_real_0: 0.26087141036987305 (0.22605393149636008) | > loss_disc_real_1: 0.2250777631998062 (0.22656157477335495) | > loss_disc_real_2: 0.24508562684059143 (0.23510665378787302) | > loss_disc_real_3: 0.23935900628566742 (0.23988598178733478) | > loss_disc_real_4: 0.26173800230026245 (0.24601493775844574) | > loss_disc_real_5: 0.26334425806999207 (0.24276791919361462) | > loss_0: 2.8983237743377686 (2.8167772726579146) | > grad_norm_0: tensor(12.2856, device='cuda:0') (tensor(7.9108, device='cuda:0')) | > loss_gen: 1.6965373754501343 (1.7642888697710903) | > loss_kl: 2.419796943664551 (2.2991021763194692) | > loss_feat: 1.8691577911376953 (2.227215138348666) | > loss_mel: 17.698646545410156 (18.25387157093395) | > loss_duration: 1.5949530601501465 (1.6062157370827415) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.27909278869629 (26.150693720037285) | > grad_norm_1: tensor(154.6630, device='cuda:0') (tensor(223.5492, device='cuda:0')) | > current_lr_0: 0.00019557441922580878 | > current_lr_1: 0.00019557441922580878 | > step_time: 0.4295 (0.4351217746734619) | > loader_time: 0.0056 (0.0061709664084694605)  --> TIME: 2025-02-25 10:02:17 -- STEP: 61/72 -- GLOBAL_STEP: 1012950 | > loss_disc: 2.8266448974609375 (2.819542028864876) | > loss_disc_real_0: 0.23315615952014923 (0.21559470806454048) | > loss_disc_real_1: 0.23361244797706604 (0.2343698808404266) | > loss_disc_real_2: 0.19542567431926727 (0.2400292512334761) | > loss_disc_real_3: 0.21716083586215973 (0.24004560953280965) | > loss_disc_real_4: 0.22729049623012543 (0.24427748141718691) | > loss_disc_real_5: 0.23857608437538147 (0.24746742673584673) | > loss_0: 2.8266448974609375 (2.819542028864876) | > grad_norm_0: tensor(5.6499, device='cuda:0') (tensor(12.2511, device='cuda:0')) | > loss_gen: 1.6624023914337158 (1.7649387375253145) | > loss_kl: 2.2546989917755127 (2.2953718920223047) | > loss_feat: 1.8711349964141846 (2.1556134927468222) | > loss_mel: 17.697467803955078 (17.915856971115364) | > loss_duration: 1.6995666027069092 (1.6240162224066061) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.185270309448242 (25.755797026587313) | > grad_norm_1: tensor(57.2588, device='cuda:0') (tensor(291.2855, device='cuda:0')) | > current_lr_0: 0.00019557441922580878 | > current_lr_1: 0.00019557441922580878 | > step_time: 0.49 (0.4555828688574619) | > loader_time: 0.0135 (0.008126516811183242) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666139253428285 (-0.00014989812609175583) | > avg_step_time: 0.4632657212270817 (+0.006213084073133868) | > avg_loss_disc: 2.8200078883641204 (-0.0033387902756811805) | > avg_loss_disc_real_0: 0.21594576527115325 (-0.007805616398092713) | > avg_loss_disc_real_1: 0.23601798335431326 (+0.005364723608527383) | > avg_loss_disc_real_2: 0.2410313369522632 (-0.00044576173097315985) | > avg_loss_disc_real_3: 0.24082910531843213 (-0.0014816898275429036) | > avg_loss_disc_real_4: 0.24473456765564394 (-0.00047719037868607095) | > avg_loss_disc_real_5: 0.24729710082772752 (-0.0012344516498941915) | > avg_loss_0: 2.8200078883641204 (-0.0033387902756811805) | > avg_grad_norm_0: tensor(11.7028, device='cuda:0') (+tensor(0.9649, device='cuda:0')) | > avg_loss_gen: 1.7720877435845388 (-0.0052004582445386305) | > avg_loss_kl: 2.3109379217658246 (-0.005473382036450758) | > avg_loss_feat: 2.157721744456761 (+0.01973303942613125) | > avg_loss_mel: 18.00988057633521 (+0.2962820160556845) | > avg_loss_duration: 1.6310202806768284 (+0.0017069477430531332) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.88164796963544 (+0.3070480454135982) | > avg_grad_norm_1: tensor(288.6078, device='cuda:0') (tensor(-4.0090, device='cuda:0'))  > EPOCH: 180/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:02:24)   --> TIME: 2025-02-25 10:02:44 -- STEP: 39/72 -- GLOBAL_STEP: 1013000 | > loss_disc: 2.8532209396362305 (2.8330432023757544) | > loss_disc_real_0: 0.21604067087173462 (0.2110512409454737) | > loss_disc_real_1: 0.20651747286319733 (0.23660672513338235) | > loss_disc_real_2: 0.29672738909721375 (0.2441893911514527) | > loss_disc_real_3: 0.29117098450660706 (0.24741910359798333) | > loss_disc_real_4: 0.24596938490867615 (0.24503697111056402) | > loss_disc_real_5: 0.25077685713768005 (0.24469256018980956) | > loss_0: 2.8532209396362305 (2.8330432023757544) | > grad_norm_0: tensor(6.4804, device='cuda:0') (tensor(12.3364, device='cuda:0')) | > loss_gen: 1.907478928565979 (1.789122373629839) | > loss_kl: 2.297039270401001 (2.2806804822041444) | > loss_feat: 2.1430108547210693 (2.2002528967001505) | > loss_mel: 17.917192459106445 (17.957247465084762) | > loss_duration: 1.6402316093444824 (1.6173527882649348) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.904953002929688 (25.84465594169421) | > grad_norm_1: tensor(273.8488, device='cuda:0') (tensor(255.8382, device='cuda:0')) | > current_lr_0: 0.00019554997242340555 | > current_lr_1: 0.00019554997242340555 | > step_time: 0.4541 (0.4417396447597406) | > loader_time: 0.0084 (0.007330448199541141) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738322996757404 (+7.218374332911871e-05) | > avg_step_time: 0.4560317858843736 (-0.007233935342708109) | > avg_loss_disc: 2.8323901975658576 (+0.012382309201737218) | > avg_loss_disc_real_0: 0.22201254544123797 (+0.006066780170084718) | > avg_loss_disc_real_1: 0.2349097185151678 (-0.001108264839145473) | > avg_loss_disc_real_2: 0.24448044493164814 (+0.003449107979384941) | > avg_loss_disc_real_3: 0.24410851782476398 (+0.003279412506331847) | > avg_loss_disc_real_4: 0.24429735898132054 (-0.00043720867432339716) | > avg_loss_disc_real_5: 0.24551486696155977 (-0.0017822338661677428) | > avg_loss_0: 2.8323901975658576 (+0.012382309201737218) | > avg_grad_norm_0: tensor(14.1126, device='cuda:0') (+tensor(2.4098, device='cuda:0')) | > avg_loss_gen: 1.7889529013297927 (+0.016865157745253878) | > avg_loss_kl: 2.2702698153509213 (-0.04066810641490326) | > avg_loss_feat: 2.1798010963789176 (+0.022079351922156487) | > avg_loss_mel: 17.810363500890602 (-0.19951707544460717) | > avg_loss_duration: 1.6382855328036026 (+0.007265252126774202) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68767268221143 (-0.19397528742400993) | > avg_grad_norm_1: tensor(289.3773, device='cuda:0') (+tensor(0.7695, device='cuda:0'))  > EPOCH: 181/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:03:01)   --> TIME: 2025-02-25 10:03:10 -- STEP: 17/72 -- GLOBAL_STEP: 1013050 | > loss_disc: 2.7885804176330566 (2.8089454454534195) | > loss_disc_real_0: 0.1829015612602234 (0.2243847023038303) | > loss_disc_real_1: 0.19651077687740326 (0.2288406938314438) | > loss_disc_real_2: 0.2353145033121109 (0.23870311852763682) | > loss_disc_real_3: 0.2309829741716385 (0.23859884633737452) | > loss_disc_real_4: 0.2597010135650635 (0.24727054729181178) | > loss_disc_real_5: 0.2672449052333832 (0.25150367705261006) | > loss_0: 2.7885804176330566 (2.8089454454534195) | > grad_norm_0: tensor(9.3591, device='cuda:0') (tensor(8.6411, device='cuda:0')) | > loss_gen: 1.7926466464996338 (1.7902857345693253) | > loss_kl: 2.2014479637145996 (2.223806198905496) | > loss_feat: 2.1387760639190674 (2.1715180733624626) | > loss_mel: 18.881996154785156 (17.75858194687787) | > loss_duration: 1.639217495918274 (1.6020245832555435) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.654085159301758 (25.546216515933764) | > grad_norm_1: tensor(52.5904, device='cuda:0') (tensor(274.7605, device='cuda:0')) | > current_lr_0: 0.00019552552867685262 | > current_lr_1: 0.00019552552867685262 | > step_time: 0.4296 (0.43668847925522747) | > loader_time: 0.0069 (0.006642944672528435)  --> TIME: 2025-02-25 10:03:34 -- STEP: 67/72 -- GLOBAL_STEP: 1013100 | > loss_disc: 2.762543201446533 (2.817538314790868) | > loss_disc_real_0: 0.2542431950569153 (0.22465606723258744) | > loss_disc_real_1: 0.19023101031780243 (0.23458094049745531) | > loss_disc_real_2: 0.27797913551330566 (0.2390475920332012) | > loss_disc_real_3: 0.26291999220848083 (0.24234964665192277) | > loss_disc_real_4: 0.2563231289386749 (0.2448677814273692) | > loss_disc_real_5: 0.2801406979560852 (0.2471627063715636) | > loss_0: 2.762543201446533 (2.817538314790868) | > grad_norm_0: tensor(8.0650, device='cuda:0') (tensor(10.9391, device='cuda:0')) | > loss_gen: 1.7506364583969116 (1.7907279338409652) | > loss_kl: 2.34928822517395 (2.323325505897181) | > loss_feat: 2.3286123275756836 (2.1759766927405972) | > loss_mel: 18.539228439331055 (18.19364265897381) | > loss_duration: 1.7144666910171509 (1.634457995642477) | > amp_scaler: 128.0 (252.17910447761193) | > loss_1: 26.682231903076172 (26.118130840472322) | > grad_norm_1: tensor(359.9403, device='cuda:0') (tensor(267.8688, device='cuda:0')) | > current_lr_0: 0.00019552552867685262 | > current_lr_1: 0.00019552552867685262 | > step_time: 0.4908 (0.4524366997960788) | > loader_time: 0.012 (0.008557333875058302) --> EVAL PERFORMANCE | > avg_loader_time: 0.008821541154888316 (+8.321815813091236e-05) | > avg_step_time: 0.4564909364136172 (+0.00045915052924361266) | > avg_loss_disc: 2.8148553472169686 (-0.01753485034888902) | > avg_loss_disc_real_0: 0.22362328541110937 (+0.0016107399698714053) | > avg_loss_disc_real_1: 0.23400159075226581 (-0.0009081277629019724) | > avg_loss_disc_real_2: 0.23787471882894007 (-0.0066057261027080705) | > avg_loss_disc_real_3: 0.24178327515091694 (-0.002325242673847039) | > avg_loss_disc_real_4: 0.24488253849492947 (+0.0005851795136089244) | > avg_loss_disc_real_5: 0.24804933075333985 (+0.0025344637917800728) | > avg_loss_0: 2.8148553472169686 (-0.01753485034888902) | > avg_grad_norm_0: tensor(10.9715, device='cuda:0') (tensor(-3.1412, device='cuda:0')) | > avg_loss_gen: 1.7916255635274967 (+0.002672662197704012) | > avg_loss_kl: 2.3233948694148543 (+0.053125054063932975) | > avg_loss_feat: 2.179748555304299 (-5.254107461860613e-05) | > avg_loss_mel: 18.15639098261444 (+0.3460274817238371) | > avg_loss_duration: 1.6361203495885286 (-0.0021651832150739736) | > avg_amp_scaler: 245.18309859154928 (-10.816901408450718) | > avg_loss_1: 26.087280380893763 (+0.39960769868233115) | > avg_grad_norm_1: tensor(272.1920, device='cuda:0') (tensor(-17.1853, device='cuda:0'))  > EPOCH: 182/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:03:38)   --> TIME: 2025-02-25 10:04:00 -- STEP: 45/72 -- GLOBAL_STEP: 1013150 | > loss_disc: 2.945465564727783 (2.8177720334794785) | > loss_disc_real_0: 0.35705098509788513 (0.2127626069717937) | > loss_disc_real_1: 0.2648950219154358 (0.2340230898724662) | > loss_disc_real_2: 0.25286948680877686 (0.24381757577260338) | > loss_disc_real_3: 0.27178141474723816 (0.24367266562249926) | > loss_disc_real_4: 0.24270224571228027 (0.24430052240689595) | > loss_disc_real_5: 0.24337537586688995 (0.24831369386778934) | > loss_0: 2.945465564727783 (2.8177720334794785) | > grad_norm_0: tensor(30.5376, device='cuda:0') (tensor(13.4887, device='cuda:0')) | > loss_gen: 1.8420778512954712 (1.7943240695529514) | > loss_kl: 2.486372947692871 (2.293504810333252) | > loss_feat: 2.096285343170166 (2.170408389303419) | > loss_mel: 17.680212020874023 (17.651731999715174) | > loss_duration: 1.6263116598129272 (1.6144532362620037) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.731260299682617 (25.524422285291884) | > grad_norm_1: tensor(58.8785, device='cuda:0') (tensor(276.8541, device='cuda:0')) | > current_lr_0: 0.000195501087985768 | > current_lr_1: 0.000195501087985768 | > step_time: 0.4555 (0.4410916169484456) | > loader_time: 0.0089 (0.007640833324856228) --> EVAL PERFORMANCE | > avg_loader_time: 0.008965237039915273 (+0.0001436958850269572) | > avg_step_time: 0.45485530436878474 (-0.001635632044832469) | > avg_loss_disc: 2.8258722903023306 (+0.011016943085361941) | > avg_loss_disc_real_0: 0.21816483192460637 (-0.0054584534865030065) | > avg_loss_disc_real_1: 0.23586724278792529 (+0.0018656520356594708) | > avg_loss_disc_real_2: 0.2450352823230582 (+0.007160563494118144) | > avg_loss_disc_real_3: 0.2442934315809062 (+0.002510156429989252) | > avg_loss_disc_real_4: 0.2467495171536862 (+0.001866978658756735) | > avg_loss_disc_real_5: 0.24953416094813544 (+0.0014848301947955889) | > avg_loss_0: 2.8258722903023306 (+0.011016943085361941) | > avg_grad_norm_0: tensor(14.7585, device='cuda:0') (+tensor(3.7871, device='cuda:0')) | > avg_loss_gen: 1.791430567351865 (-0.00019499617563178973) | > avg_loss_kl: 2.3277900588344522 (+0.0043951894195979335) | > avg_loss_feat: 2.116796995552493 (-0.06295155975180622) | > avg_loss_mel: 17.68031997412024 (-0.47607100849419837) | > avg_loss_duration: 1.6456888276086727 (+0.009568478020144155) | > avg_amp_scaler: 128.0 (-117.18309859154928) | > avg_loss_1: 25.562026279073365 (-0.5252541018203978) | > avg_grad_norm_1: tensor(280.5410, device='cuda:0') (+tensor(8.3490, device='cuda:0'))  > EPOCH: 183/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:04:16)   --> TIME: 2025-02-25 10:04:28 -- STEP: 23/72 -- GLOBAL_STEP: 1013200 | > loss_disc: 2.8795294761657715 (2.7940106702887495) | > loss_disc_real_0: 0.24960246682167053 (0.2115975935821948) | > loss_disc_real_1: 0.2545185089111328 (0.22910849879617276) | > loss_disc_real_2: 0.2401944249868393 (0.24213405277418054) | > loss_disc_real_3: 0.23536573350429535 (0.2381360815918964) | > loss_disc_real_4: 0.22096122801303864 (0.24126022859759952) | > loss_disc_real_5: 0.24732142686843872 (0.23923279733761496) | > loss_0: 2.8795294761657715 (2.7940106702887495) | > grad_norm_0: tensor(6.6708, device='cuda:0') (tensor(11.9899, device='cuda:0')) | > loss_gen: 1.8150808811187744 (1.8215028099391772) | > loss_kl: 2.191866636276245 (2.2346163988113408) | > loss_feat: 2.4381415843963623 (2.3624744104302446) | > loss_mel: 18.079084396362305 (17.989530895067297) | > loss_duration: 1.6251391172409058 (1.6135107486144356) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.14931297302246 (26.021635304326598) | > grad_norm_1: tensor(412.3828, device='cuda:0') (tensor(357.9199, device='cuda:0')) | > current_lr_0: 0.00019547665034976976 | > current_lr_1: 0.00019547665034976976 | > step_time: 0.4333 (0.43709999582041864) | > loader_time: 0.0065 (0.006922991379447605) --> EVAL PERFORMANCE | > avg_loader_time: 0.008715881428248445 (-0.000249355611666828) | > avg_step_time: 0.45694927094687876 (+0.0020939665780940198) | > avg_loss_disc: 2.8240871530183607 (-0.0017851372839698953) | > avg_loss_disc_real_0: 0.22355722457590238 (+0.005392392651296013) | > avg_loss_disc_real_1: 0.23491823841148698 (-0.0009490043764383005) | > avg_loss_disc_real_2: 0.24141641989560195 (-0.0036188624274562653) | > avg_loss_disc_real_3: 0.24228799406071783 (-0.0020054375201883634) | > avg_loss_disc_real_4: 0.2448824014462216 (-0.0018671157074646172) | > avg_loss_disc_real_5: 0.24649742056786175 (-0.0030367403802736836) | > avg_loss_0: 2.8240871530183607 (-0.0017851372839698953) | > avg_grad_norm_0: tensor(13.2412, device='cuda:0') (tensor(-1.5174, device='cuda:0')) | > avg_loss_gen: 1.785104941314375 (-0.006325626037489984) | > avg_loss_kl: 2.276758872287375 (-0.05103118654707739) | > avg_loss_feat: 2.1670063566154165 (+0.05020936106292373) | > avg_loss_mel: 17.706972766929955 (+0.026652792809713333) | > avg_loss_duration: 1.6431438553501183 (-0.0025449722585544343) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.578986933533574 (+0.016960654460209668) | > avg_grad_norm_1: tensor(297.7238, device='cuda:0') (+tensor(17.1828, device='cuda:0'))  > EPOCH: 184/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:04:53)   --> TIME: 2025-02-25 10:04:55 -- STEP: 1/72 -- GLOBAL_STEP: 1013250 | > loss_disc: 2.8165504932403564 (2.8165504932403564) | > loss_disc_real_0: 0.12536215782165527 (0.12536215782165527) | > loss_disc_real_1: 0.22442635893821716 (0.22442635893821716) | > loss_disc_real_2: 0.2367728054523468 (0.2367728054523468) | > loss_disc_real_3: 0.21948957443237305 (0.21948957443237305) | > loss_disc_real_4: 0.22628192603588104 (0.22628192603588104) | > loss_disc_real_5: 0.2379523515701294 (0.2379523515701294) | > loss_0: 2.8165504932403564 (2.8165504932403564) | > grad_norm_0: tensor(14.2250, device='cuda:0') (tensor(14.2250, device='cuda:0')) | > loss_gen: 1.7455987930297852 (1.7455987930297852) | > loss_kl: 2.183584690093994 (2.183584690093994) | > loss_feat: 2.1285688877105713 (2.1285688877105713) | > loss_mel: 18.471378326416016 (18.471378326416016) | > loss_duration: 1.652996301651001 (1.652996301651001) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.182125091552734 (26.182125091552734) | > grad_norm_1: tensor(198.6499, device='cuda:0') (tensor(198.6499, device='cuda:0')) | > current_lr_0: 0.00019545221576847604 | > current_lr_1: 0.00019545221576847604 | > step_time: 0.4398 (0.43975138664245605) | > loader_time: 0.0069 (0.006907463073730469)  --> TIME: 2025-02-25 10:05:18 -- STEP: 51/72 -- GLOBAL_STEP: 1013300 | > loss_disc: 2.866271495819092 (2.8223111115249937) | > loss_disc_real_0: 0.16830982267856598 (0.22358737798298106) | > loss_disc_real_1: 0.2762807011604309 (0.23865161690057493) | > loss_disc_real_2: 0.21815945208072662 (0.23810476064682007) | > loss_disc_real_3: 0.2607807219028473 (0.24210742115974426) | > loss_disc_real_4: 0.26358258724212646 (0.24604650513798582) | > loss_disc_real_5: 0.26353976130485535 (0.2487004954440921) | > loss_0: 2.866271495819092 (2.8223111115249937) | > grad_norm_0: tensor(10.7471, device='cuda:0') (tensor(13.2584, device='cuda:0')) | > loss_gen: 1.7807146310806274 (1.7858534513735305) | > loss_kl: 2.089733600616455 (2.3242716625625013) | > loss_feat: 1.893848180770874 (2.1330146649304567) | > loss_mel: 16.74104881286621 (18.15645771400601) | > loss_duration: 1.646615982055664 (1.624447694011763) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.151960372924805 (26.024044971840055) | > grad_norm_1: tensor(167.2557, device='cuda:0') (tensor(312.6685, device='cuda:0')) | > current_lr_0: 0.00019545221576847604 | > current_lr_1: 0.00019545221576847604 | > step_time: 0.4731 (0.4462297710717893) | > loader_time: 0.0091 (0.00779885871737611) --> EVAL PERFORMANCE | > avg_loader_time: 0.008748014208296654 (+3.2132780048209006e-05) | > avg_step_time: 0.4595730170397691 (+0.0026237460928903444) | > avg_loss_disc: 2.8274372597815294 (+0.0033501067631687143) | > avg_loss_disc_real_0: 0.22707112469303775 (+0.003513900117135371) | > avg_loss_disc_real_1: 0.23691898983129314 (+0.002000751419806157) | > avg_loss_disc_real_2: 0.23888867883615092 (-0.0025277410594510274) | > avg_loss_disc_real_3: 0.24096811195494425 (-0.0013198821057735788) | > avg_loss_disc_real_4: 0.2451666813920921 (+0.0002842799458705192) | > avg_loss_disc_real_5: 0.2473992606703664 (+0.0009018401025046585) | > avg_loss_0: 2.8274372597815294 (+0.0033501067631687143) | > avg_grad_norm_0: tensor(13.6868, device='cuda:0') (+tensor(0.4456, device='cuda:0')) | > avg_loss_gen: 1.7832234762084316 (-0.0018814651059433363) | > avg_loss_kl: 2.3371325596957138 (+0.06037368740833893) | > avg_loss_feat: 2.101510494527683 (-0.0654958620877335) | > avg_loss_mel: 18.108755944480357 (+0.4017831775504028) | > avg_loss_duration: 1.6350231137074216 (-0.008120741642696716) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.965645346843022 (+0.3866584133094477) | > avg_grad_norm_1: tensor(288.8527, device='cuda:0') (tensor(-8.8711, device='cuda:0'))  > EPOCH: 185/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:05:30)   --> TIME: 2025-02-25 10:05:45 -- STEP: 29/72 -- GLOBAL_STEP: 1013350 | > loss_disc: 2.7689552307128906 (2.8088768597306877) | > loss_disc_real_0: 0.18826863169670105 (0.21757337707897711) | > loss_disc_real_1: 0.26069074869155884 (0.2358689714094688) | > loss_disc_real_2: 0.25932127237319946 (0.24428627922617155) | > loss_disc_real_3: 0.2508421242237091 (0.23840965844433884) | > loss_disc_real_4: 0.25407904386520386 (0.24422387830142317) | > loss_disc_real_5: 0.22970540821552277 (0.24489807363214164) | > loss_0: 2.7689552307128906 (2.8088768597306877) | > grad_norm_0: tensor(8.9344, device='cuda:0') (tensor(12.2053, device='cuda:0')) | > loss_gen: 1.8563566207885742 (1.7978954027438987) | > loss_kl: 2.386040210723877 (2.4154974427716485) | > loss_feat: 2.3559062480926514 (2.2533707577606727) | > loss_mel: 18.004009246826172 (18.32709891220619) | > loss_duration: 1.6132380962371826 (1.6091204429494923) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.215551376342773 (26.402983172186488) | > grad_norm_1: tensor(380.5507, device='cuda:0') (tensor(289.5966, device='cuda:0')) | > current_lr_0: 0.00019542778424150499 | > current_lr_1: 0.00019542778424150499 | > step_time: 0.4495 (0.44053360511516704) | > loader_time: 0.0075 (0.007003775958357186) --> EVAL PERFORMANCE | > avg_loader_time: 0.008788609168898889 (+4.059496060223461e-05) | > avg_step_time: 0.45845309781356597 (-0.0011199192262031366) | > avg_loss_disc: 2.824909314303331 (-0.0025279454781985677) | > avg_loss_disc_real_0: 0.22304567128000125 (-0.004025453413036506) | > avg_loss_disc_real_1: 0.23577671298678493 (-0.001142276844508211) | > avg_loss_disc_real_2: 0.24107349842367037 (+0.0021848195875194543) | > avg_loss_disc_real_3: 0.23984248747288342 (-0.0011256244820608274) | > avg_loss_disc_real_4: 0.24459129257101409 (-0.0005753888210780211) | > avg_loss_disc_real_5: 0.24610250348776158 (-0.0012967571826048296) | > avg_loss_0: 2.824909314303331 (-0.0025279454781985677) | > avg_grad_norm_0: tensor(14.8708, device='cuda:0') (+tensor(1.1840, device='cuda:0')) | > avg_loss_gen: 1.7844778500812155 (+0.0012543738727839315) | > avg_loss_kl: 2.3693935132362474 (+0.0322609535405336) | > avg_loss_feat: 2.1630078654893685 (+0.061497370961685505) | > avg_loss_mel: 17.97812632439841 (-0.13062962008194745) | > avg_loss_duration: 1.6377320088131326 (+0.00270889510571104) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.93273761910452 (-0.03290772773850392) | > avg_grad_norm_1: tensor(268.1009, device='cuda:0') (tensor(-20.7518, device='cuda:0'))  > EPOCH: 186/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:06:07)   --> TIME: 2025-02-25 10:06:12 -- STEP: 7/72 -- GLOBAL_STEP: 1013400 | > loss_disc: 2.6720495223999023 (2.7675596305302212) | > loss_disc_real_0: 0.20022352039813995 (0.20180882726396834) | > loss_disc_real_1: 0.24532738327980042 (0.25233182311058044) | > loss_disc_real_2: 0.2490909844636917 (0.23054452879088266) | > loss_disc_real_3: 0.23146574199199677 (0.23024757845061167) | > loss_disc_real_4: 0.2173873782157898 (0.22775166588170187) | > loss_disc_real_5: 0.2318917214870453 (0.22257221383707865) | > loss_0: 2.6720495223999023 (2.7675596305302212) | > grad_norm_0: tensor(3.6378, device='cuda:0') (tensor(8.6283, device='cuda:0')) | > loss_gen: 1.83955979347229 (1.7580887760434831) | > loss_kl: 2.6956939697265625 (2.3678014619009837) | > loss_feat: 2.427227020263672 (2.2763582978929793) | > loss_mel: 19.31241226196289 (18.5767947605678) | > loss_duration: 1.6370900869369507 (1.6238689933504378) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.911983489990234 (26.602912630353654) | > grad_norm_1: tensor(639.6496, device='cuda:0') (tensor(385.7869, device='cuda:0')) | > current_lr_0: 0.0001954033557684748 | > current_lr_1: 0.0001954033557684748 | > step_time: 0.437 (0.4366325650896345) | > loader_time: 0.0055 (0.006669725690569196)  --> TIME: 2025-02-25 10:06:36 -- STEP: 57/72 -- GLOBAL_STEP: 1013450 | > loss_disc: 2.8040988445281982 (2.7935259969610926) | > loss_disc_real_0: 0.2436688244342804 (0.21305716795879498) | > loss_disc_real_1: 0.22275260090827942 (0.24076791003085019) | > loss_disc_real_2: 0.21016426384449005 (0.23778379146467177) | > loss_disc_real_3: 0.20447850227355957 (0.24122378988224163) | > loss_disc_real_4: 0.25451380014419556 (0.24288553600771384) | > loss_disc_real_5: 0.25356096029281616 (0.2442333591088914) | > loss_0: 2.8040988445281982 (2.7935259969610926) | > grad_norm_0: tensor(5.9699, device='cuda:0') (tensor(10.9978, device='cuda:0')) | > loss_gen: 1.7122783660888672 (1.815747947023626) | > loss_kl: 2.2766828536987305 (2.3068903538218715) | > loss_feat: 2.073075771331787 (2.3078629677755784) | > loss_mel: 15.844361305236816 (18.099122398778025) | > loss_duration: 1.680849552154541 (1.6526763836542766) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.587247848510742 (26.182299931844074) | > grad_norm_1: tensor(318.3607, device='cuda:0') (tensor(314.7639, device='cuda:0')) | > current_lr_0: 0.0001954033557684748 | > current_lr_1: 0.0001954033557684748 | > step_time: 0.4599 (0.4467787658959104) | > loader_time: 0.0095 (0.00792770218430904) --> EVAL PERFORMANCE | > avg_loader_time: 0.008560811969595896 (-0.00022779719930299258) | > avg_step_time: 0.4575593034986039 (-0.0008937943149620597) | > avg_loss_disc: 2.7966547415290086 (-0.028254572774322195) | > avg_loss_disc_real_0: 0.21608302994093428 (-0.006962641339066961) | > avg_loss_disc_real_1: 0.23945471486994918 (+0.0036780018831642503) | > avg_loss_disc_real_2: 0.24039321067467542 (-0.000680287748994951) | > avg_loss_disc_real_3: 0.2407167612666815 (+0.0008742737937980916) | > avg_loss_disc_real_4: 0.24119926884140766 (-0.003392023729606425) | > avg_loss_disc_real_5: 0.24307712786634203 (-0.0030253756214195515) | > avg_loss_0: 2.7966547415290086 (-0.028254572774322195) | > avg_grad_norm_0: tensor(12.2484, device='cuda:0') (tensor(-2.6223, device='cuda:0')) | > avg_loss_gen: 1.8175415522615674 (+0.03306370218035193) | > avg_loss_kl: 2.3114780237976933 (-0.05791548943855407) | > avg_loss_feat: 2.2882515813263367 (+0.12524371583696814) | > avg_loss_mel: 18.058549343700154 (+0.08042301930174389) | > avg_loss_duration: 1.6591171093390022 (+0.021385100525869616) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.13493747442541 (+0.20219985532089169) | > avg_grad_norm_1: tensor(319.6751, device='cuda:0') (+tensor(51.5742, device='cuda:0'))  > EPOCH: 187/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:06:45)   --> TIME: 2025-02-25 10:07:02 -- STEP: 35/72 -- GLOBAL_STEP: 1013500 | > loss_disc: 2.7708685398101807 (2.8049801622118267) | > loss_disc_real_0: 0.14640414714813232 (0.20893014562981468) | > loss_disc_real_1: 0.20728209614753723 (0.23520839044025965) | > loss_disc_real_2: 0.24816659092903137 (0.2417561663048608) | > loss_disc_real_3: 0.27433541417121887 (0.2390553308384759) | > loss_disc_real_4: 0.26178327202796936 (0.24386982108865465) | > loss_disc_real_5: 0.25234147906303406 (0.24510351291724614) | > loss_0: 2.7708685398101807 (2.8049801622118267) | > grad_norm_0: tensor(12.4710, device='cuda:0') (tensor(13.0480, device='cuda:0')) | > loss_gen: 1.7773592472076416 (1.7845848526273456) | > loss_kl: 2.4250504970550537 (2.261784451348441) | > loss_feat: 2.330145835876465 (2.2633697986602788) | > loss_mel: 18.14997673034668 (17.97891998291015) | > loss_duration: 1.6799578666687012 (1.6087930985859462) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.362491607666016 (25.897452218191965) | > grad_norm_1: tensor(619.7748, device='cuda:0') (tensor(268.7164, device='cuda:0')) | > current_lr_0: 0.0001953789303490037 | > current_lr_1: 0.0001953789303490037 | > step_time: 0.4399 (0.4373656817844936) | > loader_time: 0.0082 (0.0072021007537841795) --> EVAL PERFORMANCE | > avg_loader_time: 0.008586967495125782 (+2.6155525529885162e-05) | > avg_step_time: 0.4540723746931049 (-0.003486928805499001) | > avg_loss_disc: 2.8118566761554127 (+0.015201934626404068) | > avg_loss_disc_real_0: 0.2136504138019723 (-0.0024326161389619794) | > avg_loss_disc_real_1: 0.2348977222409047 (-0.004556992629044493) | > avg_loss_disc_real_2: 0.2397849629042854 (-0.0006082477703900357) | > avg_loss_disc_real_3: 0.2389841509956709 (-0.0017326102710105984) | > avg_loss_disc_real_4: 0.24244793326082364 (+0.0012486644194159846) | > avg_loss_disc_real_5: 0.24479983579105055 (+0.001722707924708522) | > avg_loss_0: 2.8118566761554127 (+0.015201934626404068) | > avg_grad_norm_0: tensor(14.7243, device='cuda:0') (+tensor(2.4758, device='cuda:0')) | > avg_loss_gen: 1.7897777993914108 (-0.027763752870156644) | > avg_loss_kl: 2.309515647485223 (-0.0019623763124703153) | > avg_loss_feat: 2.2461905983132375 (-0.04206098301309913) | > avg_loss_mel: 18.0714071837949 (+0.012857840094746109) | > avg_loss_duration: 1.6348492880942116 (-0.024267821244790655) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.051740350857585 (-0.0831971235678246) | > avg_grad_norm_1: tensor(297.6307, device='cuda:0') (tensor(-22.0444, device='cuda:0'))  > EPOCH: 188/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:07:22)   --> TIME: 2025-02-25 10:07:29 -- STEP: 13/72 -- GLOBAL_STEP: 1013550 | > loss_disc: 2.77437162399292 (2.7794608152829685) | > loss_disc_real_0: 0.18079358339309692 (0.20035370439291) | > loss_disc_real_1: 0.30657821893692017 (0.23267370806290552) | > loss_disc_real_2: 0.19550183415412903 (0.23289938614918634) | > loss_disc_real_3: 0.2126413881778717 (0.2353205933020665) | > loss_disc_real_4: 0.22639696300029755 (0.24010958809119004) | > loss_disc_real_5: 0.2101621776819229 (0.23885688644189101) | > loss_0: 2.77437162399292 (2.7794608152829685) | > grad_norm_0: tensor(6.6892, device='cuda:0') (tensor(14.4619, device='cuda:0')) | > loss_gen: 1.7953778505325317 (1.766239202939547) | > loss_kl: 2.202536106109619 (2.202478977350088) | > loss_feat: 2.2640721797943115 (2.3218558935018683) | > loss_mel: 17.66518211364746 (17.91080797635592) | > loss_duration: 1.5251870155334473 (1.592934718498817) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.452354431152344 (25.794317098764274) | > grad_norm_1: tensor(75.3018, device='cuda:0') (tensor(371.5866, device='cuda:0')) | > current_lr_0: 0.00019535450798271008 | > current_lr_1: 0.00019535450798271008 | > step_time: 0.4621 (0.4352358121138353) | > loader_time: 0.008 (0.006792802077073317)  --> TIME: 2025-02-25 10:07:53 -- STEP: 63/72 -- GLOBAL_STEP: 1013600 | > loss_disc: 2.749089002609253 (2.8121286876617915) | > loss_disc_real_0: 0.19090843200683594 (0.2194938732991143) | > loss_disc_real_1: 0.2418755292892456 (0.2348336721696551) | > loss_disc_real_2: 0.22562749683856964 (0.23976556767546942) | > loss_disc_real_3: 0.23633885383605957 (0.23997293459990668) | > loss_disc_real_4: 0.23673565685749054 (0.24470390261165678) | > loss_disc_real_5: 0.24167756736278534 (0.24814826155465747) | > loss_0: 2.749089002609253 (2.8121286876617915) | > grad_norm_0: tensor(7.0184, device='cuda:0') (tensor(14.4494, device='cuda:0')) | > loss_gen: 1.8418800830841064 (1.7901076116259136) | > loss_kl: 2.3332061767578125 (2.2881065701681473) | > loss_feat: 2.158616542816162 (2.251716180453225) | > loss_mel: 17.748748779296875 (17.884314098055402) | > loss_duration: 1.617682695388794 (1.6205003015578738) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.700136184692383 (25.834744892423114) | > grad_norm_1: tensor(445.2870, device='cuda:0') (tensor(355.0100, device='cuda:0')) | > current_lr_0: 0.00019535450798271008 | > current_lr_1: 0.00019535450798271008 | > step_time: 0.4843 (0.4479553283207) | > loader_time: 0.0112 (0.008349096964276029) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087209251564993 (+0.00013395766137351897) | > avg_step_time: 0.4559304781363044 (+0.001858103443199477) | > avg_loss_disc: 2.8076811978514766 (-0.004175478303936053) | > avg_loss_disc_real_0: 0.21852330779525594 (+0.004872893993283639) | > avg_loss_disc_real_1: 0.23381571366753376 (-0.0010820085733709295) | > avg_loss_disc_real_2: 0.23981772824912004 (+3.276534483465099e-05) | > avg_loss_disc_real_3: 0.23939916428545832 (+0.00041501328978740815) | > avg_loss_disc_real_4: 0.24373092386924045 (+0.0012829906084168008) | > avg_loss_disc_real_5: 0.2473036099067876 (+0.0025037741157370452) | > avg_loss_0: 2.8076811978514766 (-0.004175478303936053) | > avg_grad_norm_0: tensor(13.5823, device='cuda:0') (tensor(-1.1420, device='cuda:0')) | > avg_loss_gen: 1.792044439785917 (+0.002266640394506192) | > avg_loss_kl: 2.2918051766677645 (-0.017710470817458468) | > avg_loss_feat: 2.25659880000101 (+0.010408201687772678) | > avg_loss_mel: 17.918801885255625 (-0.15260529853927451) | > avg_loss_duration: 1.6268629238639079 (-0.007986364230303744) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.88611336828957 (-0.16562698256801411) | > avg_grad_norm_1: tensor(357.5628, device='cuda:0') (+tensor(59.9321, device='cuda:0'))  > EPOCH: 189/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:07:59)   --> TIME: 2025-02-25 10:08:20 -- STEP: 41/72 -- GLOBAL_STEP: 1013650 | > loss_disc: 2.803727149963379 (2.822969756475309) | > loss_disc_real_0: 0.2288210093975067 (0.22051803676820383) | > loss_disc_real_1: 0.21557065844535828 (0.23649324367686017) | > loss_disc_real_2: 0.22643785178661346 (0.2354715339294294) | > loss_disc_real_3: 0.2200053334236145 (0.23957350377629444) | > loss_disc_real_4: 0.24155688285827637 (0.24536145133216203) | > loss_disc_real_5: 0.23364971578121185 (0.2458780771348534) | > loss_0: 2.803727149963379 (2.822969756475309) | > grad_norm_0: tensor(6.4612, device='cuda:0') (tensor(13.6468, device='cuda:0')) | > loss_gen: 1.6756389141082764 (1.7772637576591679) | > loss_kl: 2.108253002166748 (2.297210117665733) | > loss_feat: 2.431385040283203 (2.2352433757084174) | > loss_mel: 18.643787384033203 (18.142842781252984) | > loss_duration: 1.632336139678955 (1.6118452461754404) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.49139976501465 (26.06440520868069) | > grad_norm_1: tensor(369.4896, device='cuda:0') (tensor(324.7834, device='cuda:0')) | > current_lr_0: 0.00019533008866921223 | > current_lr_1: 0.00019533008866921223 | > step_time: 0.4476 (0.4443092171738787) | > loader_time: 0.0096 (0.007395017437818573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625228640059351 (-9.569651643994911e-05) | > avg_step_time: 0.45941966688129265 (+0.0034891887449882697) | > avg_loss_disc: 2.8080891320403194 (+0.00040793418884277344) | > avg_loss_disc_real_0: 0.21472978969694864 (-0.0037935180983073025) | > avg_loss_disc_real_1: 0.2361809940825046 (+0.002365280414970833) | > avg_loss_disc_real_2: 0.23642420139111264 (-0.003393526858007395) | > avg_loss_disc_real_3: 0.2395898527662519 (+0.00019068848079356604) | > avg_loss_disc_real_4: 0.24333677955076727 (-0.00039414431847317744) | > avg_loss_disc_real_5: 0.24438655943098203 (-0.002917050475805566) | > avg_loss_0: 2.8080891320403194 (+0.00040793418884277344) | > avg_grad_norm_0: tensor(12.5228, device='cuda:0') (tensor(-1.0595, device='cuda:0')) | > avg_loss_gen: 1.7788131942211742 (-0.013231245564742755) | > avg_loss_kl: 2.306213956483653 (+0.014408779815888284) | > avg_loss_feat: 2.227664851806533 (-0.028933948194477033) | > avg_loss_mel: 18.059912318914716 (+0.14111043365909026) | > avg_loss_duration: 1.6316842142964754 (+0.0048212904325675154) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.00428841819226 (+0.11817504990268901) | > avg_grad_norm_1: tensor(318.2854, device='cuda:0') (tensor(-39.2774, device='cuda:0'))  > EPOCH: 190/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:08:36)   --> TIME: 2025-02-25 10:08:47 -- STEP: 19/72 -- GLOBAL_STEP: 1013700 | > loss_disc: 2.7429990768432617 (2.8127084280315198) | > loss_disc_real_0: 0.23525065183639526 (0.21695519748486972) | > loss_disc_real_1: 0.21776536107063293 (0.23618727765585246) | > loss_disc_real_2: 0.2251586616039276 (0.24170075435387461) | > loss_disc_real_3: 0.22611932456493378 (0.23983478389288249) | > loss_disc_real_4: 0.257782906293869 (0.25593274282781703) | > loss_disc_real_5: 0.26682114601135254 (0.2590040721391377) | > loss_0: 2.7429990768432617 (2.8127084280315198) | > grad_norm_0: tensor(5.1934, device='cuda:0') (tensor(12.4188, device='cuda:0')) | > loss_gen: 1.7946854829788208 (1.829625167344746) | > loss_kl: 2.272033452987671 (2.2311894768162777) | > loss_feat: 2.3832485675811768 (2.2700181634802568) | > loss_mel: 17.926023483276367 (17.73932005229749) | > loss_duration: 1.572474718093872 (1.5973268998296637) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.948467254638672 (25.667479766042607) | > grad_norm_1: tensor(397.7280, device='cuda:0') (tensor(278.5099, device='cuda:0')) | > current_lr_0: 0.00019530567240812858 | > current_lr_1: 0.00019530567240812858 | > step_time: 0.4274 (0.4351046085357666) | > loader_time: 0.0065 (0.006715887471249229)  --> TIME: 2025-02-25 10:09:11 -- STEP: 69/72 -- GLOBAL_STEP: 1013750 | > loss_disc: 2.8019614219665527 (2.822668141213016) | > loss_disc_real_0: 0.2769244313240051 (0.22040068952069766) | > loss_disc_real_1: 0.2649182677268982 (0.2385330977647201) | > loss_disc_real_2: 0.2717907428741455 (0.2431867463865142) | > loss_disc_real_3: 0.260296493768692 (0.241875060658524) | > loss_disc_real_4: 0.2680203914642334 (0.2500651936600174) | > loss_disc_real_5: 0.27046510577201843 (0.2508434970741686) | > loss_0: 2.8019614219665527 (2.822668141213016) | > grad_norm_0: tensor(15.2878, device='cuda:0') (tensor(13.7170, device='cuda:0')) | > loss_gen: 1.838515281677246 (1.8119647468345752) | > loss_kl: 2.2837369441986084 (2.317139798316402) | > loss_feat: 2.064333200454712 (2.250143154807713) | > loss_mel: 16.985361099243164 (17.769329458043202) | > loss_duration: 1.7143259048461914 (1.632009798201962) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.886272430419922 (25.780586906101394) | > grad_norm_1: tensor(437.0605, device='cuda:0') (tensor(270.2231, device='cuda:0')) | > current_lr_0: 0.00019530567240812858 | > current_lr_1: 0.00019530567240812858 | > step_time: 0.5392 (0.4551370558531388) | > loader_time: 0.013 (0.00853207836980405) --> EVAL PERFORMANCE | > avg_loader_time: 0.008588384574567768 (-3.68440654915838e-05) | > avg_step_time: 0.45688329951863893 (-0.002536367362653724) | > avg_loss_disc: 2.8232728091763777 (+0.015183677136058282) | > avg_loss_disc_real_0: 0.21964473728562744 (+0.004914947588678803) | > avg_loss_disc_real_1: 0.23758141255714524 (+0.0014004184746406467) | > avg_loss_disc_real_2: 0.2444313067785451 (+0.008007105387432462) | > avg_loss_disc_real_3: 0.24204877207816486 (+0.0024589193119129715) | > avg_loss_disc_real_4: 0.2502547578072884 (+0.006917978256521129) | > avg_loss_disc_real_5: 0.25120969704339197 (+0.006823137612409935) | > avg_loss_0: 2.8232728091763777 (+0.015183677136058282) | > avg_grad_norm_0: tensor(13.5420, device='cuda:0') (+tensor(1.0192, device='cuda:0')) | > avg_loss_gen: 1.812676085552699 (+0.03386289133152487) | > avg_loss_kl: 2.3143818244128154 (+0.008167867929162576) | > avg_loss_feat: 2.2470132800894724 (+0.019348428282939256) | > avg_loss_mel: 17.78278939153108 (-0.27712292738363686) | > avg_loss_duration: 1.6320284386755717 (+0.000344224379096314) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.788888985002547 (-0.2153994331897131) | > avg_grad_norm_1: tensor(273.5464, device='cuda:0') (tensor(-44.7390, device='cuda:0'))  > EPOCH: 191/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:09:13)   --> TIME: 2025-02-25 10:09:37 -- STEP: 47/72 -- GLOBAL_STEP: 1013800 | > loss_disc: 2.795328140258789 (2.826555094820388) | > loss_disc_real_0: 0.18635980784893036 (0.22004594986743115) | > loss_disc_real_1: 0.20084835588932037 (0.23415611494094768) | > loss_disc_real_2: 0.22367122769355774 (0.2441870833330966) | > loss_disc_real_3: 0.22339271008968353 (0.24063466584428828) | > loss_disc_real_4: 0.23195213079452515 (0.24562491286308208) | > loss_disc_real_5: 0.26329711079597473 (0.2502452445791123) | > loss_0: 2.795328140258789 (2.826555094820388) | > grad_norm_0: tensor(4.7941, device='cuda:0') (tensor(11.6033, device='cuda:0')) | > loss_gen: 1.7422999143600464 (1.804704194373273) | > loss_kl: 2.3516805171966553 (2.3041400757241757) | > loss_feat: 2.1148874759674072 (2.2201355213814598) | > loss_mel: 18.930522918701172 (17.897056133189096) | > loss_duration: 1.657953143119812 (1.6243867747327114) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.797344207763672 (25.85042267657341) | > grad_norm_1: tensor(199.8322, device='cuda:0') (tensor(255.4970, device='cuda:0')) | > current_lr_0: 0.00019528125919907757 | > current_lr_1: 0.00019528125919907757 | > step_time: 0.4577 (0.4423944696466974) | > loader_time: 0.0091 (0.007769112891339241) --> EVAL PERFORMANCE | > avg_loader_time: 0.008809079586620062 (+0.00022069501205229484) | > avg_step_time: 0.4553331590034592 (-0.0015501405151797099) | > avg_loss_disc: 2.827442555360391 (+0.004169746184013334) | > avg_loss_disc_real_0: 0.2221995428414412 (+0.0025548055558137495) | > avg_loss_disc_real_1: 0.2350866370637652 (-0.002494775493380036) | > avg_loss_disc_real_2: 0.24293701560564443 (-0.0014942911729006747) | > avg_loss_disc_real_3: 0.2393484955102625 (-0.0027002765679023655) | > avg_loss_disc_real_4: 0.24460193116060444 (-0.005652826646683956) | > avg_loss_disc_real_5: 0.24758597263987636 (-0.0036237244035156047) | > avg_loss_0: 2.827442555360391 (+0.004169746184013334) | > avg_grad_norm_0: tensor(11.2106, device='cuda:0') (tensor(-2.3314, device='cuda:0')) | > avg_loss_gen: 1.790740036628616 (-0.021936048924083185) | > avg_loss_kl: 2.321974341298493 (+0.007592516885677636) | > avg_loss_feat: 2.180638706180411 (-0.06637457390906132) | > avg_loss_mel: 18.057342797937523 (+0.27455340640644366) | > avg_loss_duration: 1.6395541060138756 (+0.007525667338303954) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.99025003675004 (+0.20136105174749375) | > avg_grad_norm_1: tensor(271.1547, device='cuda:0') (tensor(-2.3917, device='cuda:0'))  > EPOCH: 192/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:09:51)   --> TIME: 2025-02-25 10:10:04 -- STEP: 25/72 -- GLOBAL_STEP: 1013850 | > loss_disc: 2.7373831272125244 (2.788196086883545) | > loss_disc_real_0: 0.2655118405818939 (0.21162112951278686) | > loss_disc_real_1: 0.23764759302139282 (0.23060535550117492) | > loss_disc_real_2: 0.23308852314949036 (0.23521233797073365) | > loss_disc_real_3: 0.22816769778728485 (0.24126408457756043) | > loss_disc_real_4: 0.2337135523557663 (0.23982621610164642) | > loss_disc_real_5: 0.22957777976989746 (0.24568940997123717) | > loss_0: 2.7373831272125244 (2.788196086883545) | > grad_norm_0: tensor(14.1983, device='cuda:0') (tensor(10.5779, device='cuda:0')) | > loss_gen: 1.780302882194519 (1.7894181871414185) | > loss_kl: 2.3949971199035645 (2.2482037448883054) | > loss_feat: 2.318606376647949 (2.29018410205841) | > loss_mel: 18.968891143798828 (17.92381656646729) | > loss_duration: 1.605839490890503 (1.5964625310897826) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.068635940551758 (25.8480850982666) | > grad_norm_1: tensor(288.1831, device='cuda:0') (tensor(364.3853, device='cuda:0')) | > current_lr_0: 0.00019525684904167766 | > current_lr_1: 0.00019525684904167766 | > step_time: 0.4419 (0.4393287467956543) | > loader_time: 0.0091 (0.006936349868774414) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655259307001678 (-0.0001538202796183847) | > avg_step_time: 0.4586526373742332 (+0.003319478370773954) | > avg_loss_disc: 2.8126282893436056 (-0.014814266016785371) | > avg_loss_disc_real_0: 0.2209253512637716 (-0.0012741915776695867) | > avg_loss_disc_real_1: 0.2325173173571976 (-0.0025693197065675966) | > avg_loss_disc_real_2: 0.23879064388678106 (-0.0041463717188633675) | > avg_loss_disc_real_3: 0.2422315390177176 (+0.0028830435074551075) | > avg_loss_disc_real_4: 0.24323839704755326 (-0.0013635341130511791) | > avg_loss_disc_real_5: 0.24615287969649677 (-0.0014330929433795936) | > avg_loss_0: 2.8126282893436056 (-0.014814266016785371) | > avg_grad_norm_0: tensor(12.3798, device='cuda:0') (+tensor(1.1692, device='cuda:0')) | > avg_loss_gen: 1.787249207496643 (-0.00349082913197285) | > avg_loss_kl: 2.2626322457488155 (-0.05934209554967751) | > avg_loss_feat: 2.209388534787675 (+0.02874982860726405) | > avg_loss_mel: 17.816721593829943 (-0.2406212041075797) | > avg_loss_duration: 1.6356166651551152 (-0.003937440858760466) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.711608295709315 (-0.27864174104072603) | > avg_grad_norm_1: tensor(328.5029, device='cuda:0') (+tensor(57.3482, device='cuda:0'))  > EPOCH: 193/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:10:28)   --> TIME: 2025-02-25 10:10:31 -- STEP: 3/72 -- GLOBAL_STEP: 1013900 | > loss_disc: 2.758619546890259 (2.7456893920898438) | > loss_disc_real_0: 0.21154242753982544 (0.19206871092319489) | > loss_disc_real_1: 0.18853315711021423 (0.20515245695908865) | > loss_disc_real_2: 0.2630275785923004 (0.24020355939865112) | > loss_disc_real_3: 0.23045165836811066 (0.23073655366897583) | > loss_disc_real_4: 0.23621228337287903 (0.2424019326766332) | > loss_disc_real_5: 0.24526633322238922 (0.24607884883880615) | > loss_0: 2.758619546890259 (2.7456893920898438) | > grad_norm_0: tensor(4.9892, device='cuda:0') (tensor(6.5068, device='cuda:0')) | > loss_gen: 1.9503720998764038 (1.8172277212142944) | > loss_kl: 2.4343957901000977 (2.557826360066732) | > loss_feat: 2.4148919582366943 (2.4950289726257324) | > loss_mel: 18.870702743530273 (19.696502685546875) | > loss_duration: 1.6676998138427734 (1.6986043055852253) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.338062286376953 (28.26519012451172) | > grad_norm_1: tensor(256.8175, device='cuda:0') (tensor(325.4071, device='cuda:0')) | > current_lr_0: 0.00019523244193554745 | > current_lr_1: 0.00019523244193554745 | > step_time: 0.4479 (0.4468994140625) | > loader_time: 0.0064 (0.006179650624593099)  --> TIME: 2025-02-25 10:10:54 -- STEP: 53/72 -- GLOBAL_STEP: 1013950 | > loss_disc: 2.932814121246338 (2.822112690727666) | > loss_disc_real_0: 0.29344600439071655 (0.21766588190254174) | > loss_disc_real_1: 0.2369133085012436 (0.2355933313099843) | > loss_disc_real_2: 0.26389867067337036 (0.24040634311595052) | > loss_disc_real_3: 0.23507985472679138 (0.2392061451133692) | > loss_disc_real_4: 0.24934861063957214 (0.24416740857205302) | > loss_disc_real_5: 0.26523420214653015 (0.2458847274757781) | > loss_0: 2.932814121246338 (2.822112690727666) | > grad_norm_0: tensor(16.0279, device='cuda:0') (tensor(13.3940, device='cuda:0')) | > loss_gen: 1.678283929824829 (1.780012013777247) | > loss_kl: 2.2851312160491943 (2.3332166716737572) | > loss_feat: 1.6401035785675049 (2.235187157145085) | > loss_mel: 15.12125015258789 (17.946987350032007) | > loss_duration: 1.6734578609466553 (1.6287260932742424) | > amp_scaler: 128.0 (128.0) | > loss_1: 22.39822769165039 (25.924129270157724) | > grad_norm_1: tensor(97.6696, device='cuda:0') (tensor(196.2645, device='cuda:0')) | > current_lr_0: 0.00019523244193554745 | > current_lr_1: 0.00019523244193554745 | > step_time: 0.4627 (0.44642186164855957) | > loader_time: 0.0095 (0.00786144328567217) --> EVAL PERFORMANCE | > avg_loader_time: 0.008636031352298358 (-1.9227954703320005e-05) | > avg_step_time: 0.45762533201298244 (-0.00102730536125073) | > avg_loss_disc: 2.819632651100696 (+0.007004361757090294) | > avg_loss_disc_real_0: 0.21654116949984725 (-0.004384181763924355) | > avg_loss_disc_real_1: 0.23417243571348592 (+0.0016551183562883154) | > avg_loss_disc_real_2: 0.24032802229196254 (+0.0015373784051814776) | > avg_loss_disc_real_3: 0.2387362689199582 (-0.0034952700977594064) | > avg_loss_disc_real_4: 0.2434576390914514 (+0.0002192420438981435) | > avg_loss_disc_real_5: 0.2446452666756133 (-0.0015076130208834804) | > avg_loss_0: 2.819632651100696 (+0.007004361757090294) | > avg_grad_norm_0: tensor(12.7055, device='cuda:0') (+tensor(0.3257, device='cuda:0')) | > avg_loss_gen: 1.781744701761595 (-0.005504505735048015) | > avg_loss_kl: 2.3242802821414585 (+0.061648036392643046) | > avg_loss_feat: 2.230237450398189 (+0.02084891561051405) | > avg_loss_mel: 17.960145990613473 (+0.14342439678353003) | > avg_loss_duration: 1.6367419471203442 (+0.0011252819652289858) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.933150412331162 (+0.22154211662184764) | > avg_grad_norm_1: tensor(228.0598, device='cuda:0') (tensor(-100.4431, device='cuda:0'))  > EPOCH: 194/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:11:05)   --> TIME: 2025-02-25 10:11:21 -- STEP: 31/72 -- GLOBAL_STEP: 1014000 | > loss_disc: 2.773080825805664 (2.803959823423816) | > loss_disc_real_0: 0.24583925306797028 (0.2132009474500533) | > loss_disc_real_1: 0.2590397894382477 (0.23175347572372806) | > loss_disc_real_2: 0.22449591755867004 (0.2395773789575023) | > loss_disc_real_3: 0.21745191514492035 (0.2416321236279703) | > loss_disc_real_4: 0.23754474520683289 (0.24276670765492223) | > loss_disc_real_5: 0.2300766557455063 (0.2441211482209544) | > loss_0: 2.773080825805664 (2.803959823423816) | > grad_norm_0: tensor(7.5525, device='cuda:0') (tensor(11.5374, device='cuda:0')) | > loss_gen: 1.6240630149841309 (1.7829002334225563) | > loss_kl: 2.596538782119751 (2.310485178424466) | > loss_feat: 2.377039909362793 (2.28346698899423) | > loss_mel: 17.733976364135742 (18.04927715178459) | > loss_duration: 1.6358450651168823 (1.5942443647692282) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.96746253967285 (26.020373867404075) | > grad_norm_1: tensor(135.9190, device='cuda:0') (tensor(265.9453, device='cuda:0')) | > current_lr_0: 0.0001952080378803055 | > current_lr_1: 0.0001952080378803055 | > step_time: 0.4363 (0.43547727215674614) | > loader_time: 0.009 (0.007193703805246661) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754562324201557 (+0.00011853097190319892) | > avg_step_time: 0.4557675005684436 (-0.0018578314445388266) | > avg_loss_disc: 2.8204939096746315 (+0.0008612585739355971) | > avg_loss_disc_real_0: 0.21703860058750904 (+0.0004974310876617871) | > avg_loss_disc_real_1: 0.23719864789868744 (+0.003026212185201521) | > avg_loss_disc_real_2: 0.2401792392344542 (-0.00014878305750834486) | > avg_loss_disc_real_3: 0.24388327711904553 (+0.005147008199087338) | > avg_loss_disc_real_4: 0.24532360405149595 (+0.0018659649600445416) | > avg_loss_disc_real_5: 0.24656419665880605 (+0.001918929983192763) | > avg_loss_0: 2.8204939096746315 (+0.0008612585739355971) | > avg_grad_norm_0: tensor(13.8626, device='cuda:0') (+tensor(1.1571, device='cuda:0')) | > avg_loss_gen: 1.787245582526838 (+0.005500880765243021) | > avg_loss_kl: 2.3113787845826486 (-0.012901497558809982) | > avg_loss_feat: 2.1864690327308547 (-0.043768417667334525) | > avg_loss_mel: 17.655697070376984 (-0.3044489202364886) | > avg_loss_duration: 1.6256540291745898 (-0.011087917945754366) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.566444289516394 (-0.3667061228147688) | > avg_grad_norm_1: tensor(276.8453, device='cuda:0') (+tensor(48.7855, device='cuda:0'))  > EPOCH: 195/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:11:42)   --> TIME: 2025-02-25 10:11:48 -- STEP: 9/72 -- GLOBAL_STEP: 1014050 | > loss_disc: 2.6816394329071045 (2.7527434296078153) | > loss_disc_real_0: 0.2601681351661682 (0.19573603901598188) | > loss_disc_real_1: 0.19709838926792145 (0.21516785356733534) | > loss_disc_real_2: 0.20402811467647552 (0.24162522123919594) | > loss_disc_real_3: 0.2228824496269226 (0.23642121089829338) | > loss_disc_real_4: 0.21003344655036926 (0.24456866913371617) | > loss_disc_real_5: 0.22692766785621643 (0.24746031396918827) | > loss_0: 2.6816394329071045 (2.7527434296078153) | > grad_norm_0: tensor(13.0028, device='cuda:0') (tensor(9.3308, device='cuda:0')) | > loss_gen: 1.9318581819534302 (1.833898451593187) | > loss_kl: 2.2491605281829834 (2.1955778996149697) | > loss_feat: 2.509096145629883 (2.393734653790792) | > loss_mel: 17.680509567260742 (18.013260735405815) | > loss_duration: 1.5976283550262451 (1.6052681340111628) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.968252182006836 (26.04173978169759) | > grad_norm_1: tensor(300.5465, device='cuda:0') (tensor(358.1451, device='cuda:0')) | > current_lr_0: 0.00019518363687557043 | > current_lr_1: 0.00019518363687557043 | > step_time: 0.4263 (0.4384254349602593) | > loader_time: 0.0055 (0.007097032335069444)  --> TIME: 2025-02-25 10:12:12 -- STEP: 59/72 -- GLOBAL_STEP: 1014100 | > loss_disc: 2.769805431365967 (2.8048565468545688) | > loss_disc_real_0: 0.31689590215682983 (0.218582939545987) | > loss_disc_real_1: 0.18794818222522736 (0.2308368622246435) | > loss_disc_real_2: 0.22997823357582092 (0.24054718371165001) | > loss_disc_real_3: 0.2198059856891632 (0.2375111120232081) | > loss_disc_real_4: 0.23207177221775055 (0.24372205845380235) | > loss_disc_real_5: 0.24076299369335175 (0.24613843681448597) | > loss_0: 2.769805431365967 (2.8048565468545688) | > grad_norm_0: tensor(20.2843, device='cuda:0') (tensor(15.9461, device='cuda:0')) | > loss_gen: 1.73177170753479 (1.7942295700816784) | > loss_kl: 2.247584581375122 (2.286218449220819) | > loss_feat: 2.301907539367676 (2.249887704849244) | > loss_mel: 17.246286392211914 (17.809594493801303) | > loss_duration: 1.6547746658325195 (1.6229173793631084) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.182323455810547 (25.76284764176708) | > grad_norm_1: tensor(399.9505, device='cuda:0') (tensor(284.0825, device='cuda:0')) | > current_lr_0: 0.00019518363687557043 | > current_lr_1: 0.00019518363687557043 | > step_time: 0.4693 (0.449960619716321) | > loader_time: 0.0097 (0.008120997477385958) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866517550508741 (-8.938681911414663e-05) | > avg_step_time: 0.45904886554664287 (+0.0032813649781992527) | > avg_loss_disc: 2.8057281165055827 (-0.014765793169048802) | > avg_loss_disc_real_0: 0.21719801824697307 (+0.00015941765946403397) | > avg_loss_disc_real_1: 0.23181860736558135 (-0.00538004053310609) | > avg_loss_disc_real_2: 0.23892938104313863 (-0.001249858191315567) | > avg_loss_disc_real_3: 0.2363979373599442 (-0.007485339759101345) | > avg_loss_disc_real_4: 0.24136879074741419 (-0.003954813304081761) | > avg_loss_disc_real_5: 0.24274776750047442 (-0.0038164291583316357) | > avg_loss_0: 2.8057281165055827 (-0.014765793169048802) | > avg_grad_norm_0: tensor(16.2433, device='cuda:0') (+tensor(2.3807, device='cuda:0')) | > avg_loss_gen: 1.7856630187639049 (-0.001582563762933198) | > avg_loss_kl: 2.2869071758968724 (-0.024471608685776136) | > avg_loss_feat: 2.235850948682974 (+0.04938191595211938) | > avg_loss_mel: 17.761723719852064 (+0.10602664947507989) | > avg_loss_duration: 1.6293019845452108 (+0.0036479553706210144) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.699446852778042 (+0.13300256326164828) | > avg_grad_norm_1: tensor(310.6667, device='cuda:0') (+tensor(33.8214, device='cuda:0'))  > EPOCH: 196/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:12:20)   --> TIME: 2025-02-25 10:12:39 -- STEP: 37/72 -- GLOBAL_STEP: 1014150 | > loss_disc: 2.845604419708252 (2.8255737214475065) | > loss_disc_real_0: 0.19398359954357147 (0.22165577717729518) | > loss_disc_real_1: 0.267084538936615 (0.23437670034331246) | > loss_disc_real_2: 0.28126680850982666 (0.24263918278990565) | > loss_disc_real_3: 0.249945729970932 (0.24005483010330717) | > loss_disc_real_4: 0.2739420235157013 (0.24543632285015002) | > loss_disc_real_5: 0.26941925287246704 (0.246866295869286) | > loss_0: 2.845604419708252 (2.8255737214475065) | > grad_norm_0: tensor(5.4931, device='cuda:0') (tensor(13.2169, device='cuda:0')) | > loss_gen: 1.796085238456726 (1.789786177712518) | > loss_kl: 2.225419759750366 (2.3501447793599723) | > loss_feat: 2.3685896396636963 (2.275626933252489) | > loss_mel: 18.917491912841797 (18.05873272870038) | > loss_duration: 1.6322391033172607 (1.6008812897914164) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.9398250579834 (26.075171857266813) | > grad_norm_1: tensor(160.0000, device='cuda:0') (tensor(236.8478, device='cuda:0')) | > current_lr_0: 0.00019515923892096098 | > current_lr_1: 0.00019515923892096098 | > step_time: 0.4715 (0.4446772240303658) | > loader_time: 0.0083 (0.00756480242754962) --> EVAL PERFORMANCE | > avg_loader_time: 0.008811601450745483 (+0.00014642594565807306) | > avg_step_time: 0.4592868576587086 (+0.00023799211206571025) | > avg_loss_disc: 2.8189988908633383 (+0.013270774357755588) | > avg_loss_disc_real_0: 0.2202013269276686 (+0.003003308680695538) | > avg_loss_disc_real_1: 0.23322023577253584 (+0.001401628406954486) | > avg_loss_disc_real_2: 0.23844934853029923 (-0.0004800325128394012) | > avg_loss_disc_real_3: 0.2386293843598433 (+0.002231446999899106) | > avg_loss_disc_real_4: 0.2447784037237436 (+0.0034096129763294125) | > avg_loss_disc_real_5: 0.24612416306012114 (+0.003376395559646722) | > avg_loss_0: 2.8189988908633383 (+0.013270774357755588) | > avg_grad_norm_0: tensor(11.9668, device='cuda:0') (tensor(-4.2765, device='cuda:0')) | > avg_loss_gen: 1.7796262200449555 (-0.006036798718949354) | > avg_loss_kl: 2.331134668538268 (+0.04422749264139547) | > avg_loss_feat: 2.2293349622001117 (-0.006515986482862335) | > avg_loss_mel: 17.96641461278352 (+0.20469089293145615) | > avg_loss_duration: 1.6284302053317217 (-0.0008717792134891322) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.934940472455093 (+0.23549361967705096) | > avg_grad_norm_1: tensor(280.6530, device='cuda:0') (tensor(-30.0138, device='cuda:0'))  > EPOCH: 197/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:12:57)   --> TIME: 2025-02-25 10:13:06 -- STEP: 15/72 -- GLOBAL_STEP: 1014200 | > loss_disc: 2.9011716842651367 (2.80472149848938) | > loss_disc_real_0: 0.2091156542301178 (0.20752179026603698) | > loss_disc_real_1: 0.2829422354698181 (0.22408847212791444) | > loss_disc_real_2: 0.22931651771068573 (0.25283300975958506) | > loss_disc_real_3: 0.24814893305301666 (0.24113376637299855) | > loss_disc_real_4: 0.2491905242204666 (0.24708198209603627) | > loss_disc_real_5: 0.25007835030555725 (0.24783620337645212) | > loss_0: 2.9011716842651367 (2.80472149848938) | > grad_norm_0: tensor(5.1216, device='cuda:0') (tensor(9.8122, device='cuda:0')) | > loss_gen: 1.7732040882110596 (1.7893497387568156) | > loss_kl: 2.1469826698303223 (2.147695215543111) | > loss_feat: 2.0703420639038086 (2.204743552207946) | > loss_mel: 18.335100173950195 (17.444169362386067) | > loss_duration: 1.5749603509902954 (1.583682107925415) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.900590896606445 (25.169639841715494) | > grad_norm_1: tensor(408.8224, device='cuda:0') (tensor(365.8278, device='cuda:0')) | > current_lr_0: 0.00019513484401609586 | > current_lr_1: 0.00019513484401609586 | > step_time: 0.4384 (0.4336717128753662) | > loader_time: 0.009 (0.00695341428120931)  --> TIME: 2025-02-25 10:13:29 -- STEP: 65/72 -- GLOBAL_STEP: 1014250 | > loss_disc: 2.805434226989746 (2.8181155131413385) | > loss_disc_real_0: 0.3266552686691284 (0.22138318751867) | > loss_disc_real_1: 0.24037960171699524 (0.2300591704937128) | > loss_disc_real_2: 0.21191711723804474 (0.24349047381144304) | > loss_disc_real_3: 0.23242777585983276 (0.2408076449082448) | > loss_disc_real_4: 0.23268471658229828 (0.24445815155139336) | > loss_disc_real_5: 0.22899717092514038 (0.24644197523593903) | > loss_0: 2.805434226989746 (2.8181155131413385) | > grad_norm_0: tensor(18.6604, device='cuda:0') (tensor(13.4107, device='cuda:0')) | > loss_gen: 1.846323013305664 (1.784759937799894) | > loss_kl: 2.5125317573547363 (2.2364084793971126) | > loss_feat: 2.3728458881378174 (2.2184754903499897) | > loss_mel: 17.76605796813965 (17.500552617586568) | > loss_duration: 1.6646173000335693 (1.6183152840687678) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.162376403808594 (25.358511763352613) | > grad_norm_1: tensor(507.4943, device='cuda:0') (tensor(374.7244, device='cuda:0')) | > current_lr_0: 0.00019513484401609586 | > current_lr_1: 0.00019513484401609586 | > step_time: 0.5258 (0.4493518535907452) | > loader_time: 0.0118 (0.008349451651939978) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613002132362043 (-0.0001985993183834401) | > avg_step_time: 0.45586212923828984 (-0.0034247284204187434) | > avg_loss_disc: 2.8166243761358123 (-0.0023745147275260337) | > avg_loss_disc_real_0: 0.22062597255891478 (+0.0004246456312461677) | > avg_loss_disc_real_1: 0.23029629277511382 (-0.002923942997422019) | > avg_loss_disc_real_2: 0.24280762336623501 (+0.004358274835935788) | > avg_loss_disc_real_3: 0.24079907444161427 (+0.0021696900817709808) | > avg_loss_disc_real_4: 0.24419832418502216 (-0.0005800795387214397) | > avg_loss_disc_real_5: 0.24572668927656094 (-0.00039747378356019825) | > avg_loss_0: 2.8166243761358123 (-0.0023745147275260337) | > avg_grad_norm_0: tensor(13.2882, device='cuda:0') (+tensor(1.3214, device='cuda:0')) | > avg_loss_gen: 1.782364890608989 (+0.0027386705640335407) | > avg_loss_kl: 2.2400956254609867 (-0.09103904307728117) | > avg_loss_feat: 2.2159528077488213 (-0.013382154451290429) | > avg_loss_mel: 17.48003018070274 (-0.4863844320807793) | > avg_loss_duration: 1.6226043667591794 (-0.0058258385725422634) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.341047851132675 (-0.5938926213224178) | > avg_grad_norm_1: tensor(380.7682, device='cuda:0') (+tensor(100.1152, device='cuda:0'))  > EPOCH: 198/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:13:34)   --> TIME: 2025-02-25 10:13:56 -- STEP: 43/72 -- GLOBAL_STEP: 1014300 | > loss_disc: 2.8313052654266357 (2.808699275172034) | > loss_disc_real_0: 0.27154332399368286 (0.2212454711281976) | > loss_disc_real_1: 0.1893731951713562 (0.2376004769358524) | > loss_disc_real_2: 0.2604963481426239 (0.24486705106358195) | > loss_disc_real_3: 0.257249653339386 (0.242319232502649) | > loss_disc_real_4: 0.27879759669303894 (0.24460652990396634) | > loss_disc_real_5: 0.2475912868976593 (0.25090152545030725) | > loss_0: 2.8313052654266357 (2.808699275172034) | > grad_norm_0: tensor(16.1901, device='cuda:0') (tensor(10.5084, device='cuda:0')) | > loss_gen: 1.843108892440796 (1.8334887360417567) | > loss_kl: 2.0976455211639404 (2.340338873308758) | > loss_feat: 2.075350761413574 (2.320079531780509) | > loss_mel: 16.56262969970703 (17.811458942502043) | > loss_duration: 1.6215007305145264 (1.608335442321245) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.20023536682129 (25.913701257040334) | > grad_norm_1: tensor(182.1573, device='cuda:0') (tensor(356.1390, device='cuda:0')) | > current_lr_0: 0.00019511045216059385 | > current_lr_1: 0.00019511045216059385 | > step_time: 0.4581 (0.45173165410063987) | > loader_time: 0.0119 (0.00738688402397688) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534935158742988 (-7.806697361905507e-05) | > avg_step_time: 0.46282352192301146 (+0.006961392684721623) | > avg_loss_disc: 2.8136276359289463 (-0.002996740206866022) | > avg_loss_disc_real_0: 0.22155047617327997 (+0.0009245036143651908) | > avg_loss_disc_real_1: 0.23580339781835047 (+0.005507105043236649) | > avg_loss_disc_real_2: 0.24193582190594204 (-0.0008718014602929758) | > avg_loss_disc_real_3: 0.24104444535685257 (+0.00024537091523829657) | > avg_loss_disc_real_4: 0.24447974919433324 (+0.0002814250093110815) | > avg_loss_disc_real_5: 0.2507712937576669 (+0.005044604481105941) | > avg_loss_0: 2.8136276359289463 (-0.002996740206866022) | > avg_grad_norm_0: tensor(10.8756, device='cuda:0') (tensor(-2.4127, device='cuda:0')) | > avg_loss_gen: 1.8136280019518356 (+0.03126311134284654) | > avg_loss_kl: 2.3078796359854685 (+0.0677840105244818) | > avg_loss_feat: 2.266455369935909 (+0.05050256218708782) | > avg_loss_mel: 17.74665515523561 (+0.2666249745328706) | > avg_loss_duration: 1.628185767523 (+0.005581400763820632) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.76280368213922 (+0.42175583100654634) | > avg_grad_norm_1: tensor(353.0258, device='cuda:0') (tensor(-27.7424, device='cuda:0'))  > EPOCH: 199/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:14:11)   --> TIME: 2025-02-25 10:14:23 -- STEP: 21/72 -- GLOBAL_STEP: 1014350 | > loss_disc: 2.851496696472168 (2.815706582296462) | > loss_disc_real_0: 0.1644604206085205 (0.21855807659171878) | > loss_disc_real_1: 0.18063576519489288 (0.2414836117199489) | > loss_disc_real_2: 0.25287702679634094 (0.23851526422160013) | > loss_disc_real_3: 0.2749384641647339 (0.24644519885381064) | > loss_disc_real_4: 0.2640209197998047 (0.24315603077411652) | > loss_disc_real_5: 0.25722944736480713 (0.2388976649159477) | > loss_0: 2.851496696472168 (2.815706582296462) | > grad_norm_0: tensor(11.4971, device='cuda:0') (tensor(12.8142, device='cuda:0')) | > loss_gen: 1.8370473384857178 (1.8170627298809232) | > loss_kl: 2.149500846862793 (2.2919799713861373) | > loss_feat: 2.344517946243286 (2.3979969989685785) | > loss_mel: 17.91280746459961 (18.29387619381859) | > loss_duration: 1.6218804121017456 (1.6100465286345709) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.865753173828125 (26.41096224103655) | > grad_norm_1: tensor(314.6283, device='cuda:0') (tensor(369.4414, device='cuda:0')) | > current_lr_0: 0.00019508606335407377 | > current_lr_1: 0.00019508606335407377 | > step_time: 0.4466 (0.4364780698503767) | > loader_time: 0.0071 (0.0065618469601585756)  --> TIME: 2025-02-25 10:14:47 -- STEP: 71/72 -- GLOBAL_STEP: 1014400 | > loss_disc: 2.8712751865386963 (2.8088919444822924) | > loss_disc_real_0: 0.20768144726753235 (0.21755969597839972) | > loss_disc_real_1: 0.23780910670757294 (0.23077700885248856) | > loss_disc_real_2: 0.27613797783851624 (0.23710882474838849) | > loss_disc_real_3: 0.2459930181503296 (0.24224535197439329) | > loss_disc_real_4: 0.2504010498523712 (0.243819191422261) | > loss_disc_real_5: 0.2330649197101593 (0.24525011383311848) | > loss_0: 2.8712751865386963 (2.8088919444822924) | > grad_norm_0: tensor(12.0056, device='cuda:0') (tensor(14.0256, device='cuda:0')) | > loss_gen: 1.9182175397872925 (1.7883820181161585) | > loss_kl: 2.3017430305480957 (2.3152974625708356) | > loss_feat: 2.1849937438964844 (2.2850835894195125) | > loss_mel: 17.27480125427246 (17.88852919994944) | > loss_duration: 1.635259747505188 (1.6319147818525068) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.315013885498047 (25.90920700825436) | > grad_norm_1: tensor(192.2084, device='cuda:0') (tensor(321.1282, device='cuda:0')) | > current_lr_0: 0.00019508606335407377 | > current_lr_1: 0.00019508606335407377 | > step_time: 0.4858 (0.4552547427969919) | > loader_time: 0.008 (0.008514256544516118) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514256544516118 (-2.0678614226870423e-05) | > avg_step_time: 0.4552547427969919 (-0.007568779126019554) | > avg_loss_disc: 2.8088919444822924 (-0.004735691446653867) | > avg_loss_disc_real_0: 0.21755969597839972 (-0.00399078019488025) | > avg_loss_disc_real_1: 0.23077700885248856 (-0.005026388965861911) | > avg_loss_disc_real_2: 0.23710882474838849 (-0.004826997157553553) | > avg_loss_disc_real_3: 0.24224535197439329 (+0.0012009066175407146) | > avg_loss_disc_real_4: 0.243819191422261 (-0.0006605577720722455) | > avg_loss_disc_real_5: 0.24525011383311848 (-0.005521179924548397) | > avg_loss_0: 2.8088919444822924 (-0.004735691446653867) | > avg_grad_norm_0: tensor(14.0256, device='cuda:0') (+tensor(3.1501, device='cuda:0')) | > avg_loss_gen: 1.7883820181161585 (-0.02524598383567711) | > avg_loss_kl: 2.3152974625708356 (+0.007417826585367049) | > avg_loss_feat: 2.2850835894195125 (+0.01862821948360338) | > avg_loss_mel: 17.88852919994944 (+0.14187404471382692) | > avg_loss_duration: 1.6319147818525068 (+0.0037290143295067857) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.90920700825436 (+0.14640332611513784) | > avg_grad_norm_1: tensor(321.1282, device='cuda:0') (tensor(-31.8976, device='cuda:0'))  > EPOCH: 200/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:14:48)   --> TIME: 2025-02-25 10:15:13 -- STEP: 49/72 -- GLOBAL_STEP: 1014450 | > loss_disc: 2.9242732524871826 (2.809063196182251) | > loss_disc_real_0: 0.22208106517791748 (0.2115383171001259) | > loss_disc_real_1: 0.24436981976032257 (0.23434689701819905) | > loss_disc_real_2: 0.23846356570720673 (0.24218960258425498) | > loss_disc_real_3: 0.23330600559711456 (0.24144999105103163) | > loss_disc_real_4: 0.24323955178260803 (0.24488265052133681) | > loss_disc_real_5: 0.22850055992603302 (0.24520865995056776) | > loss_0: 2.9242732524871826 (2.809063196182251) | > grad_norm_0: tensor(8.3614, device='cuda:0') (tensor(13.8548, device='cuda:0')) | > loss_gen: 1.7071411609649658 (1.7828099849272747) | > loss_kl: 2.4855926036834717 (2.372908193237927) | > loss_feat: 2.372742176055908 (2.2850222198330634) | > loss_mel: 18.90533447265625 (18.00410204517598) | > loss_duration: 1.69631826877594 (1.6216148624614792) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.16712760925293 (26.066457359158264) | > grad_norm_1: tensor(160.0697, device='cuda:0') (tensor(235.7988, device='cuda:0')) | > current_lr_0: 0.00019506167759615451 | > current_lr_1: 0.00019506167759615451 | > step_time: 0.4393 (0.443151128535368) | > loader_time: 0.0091 (0.0076584864635856785) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598586203346787 (+8.432965883066988e-05) | > avg_step_time: 0.45601745390556225 (+0.0007627111085703464) | > avg_loss_disc: 2.812282273467158 (+0.003390328984865576) | > avg_loss_disc_real_0: 0.2178584266926201 (+0.00029873071422037434) | > avg_loss_disc_real_1: 0.23444721459502904 (+0.0036702057425404833) | > avg_loss_disc_real_2: 0.24287483452911107 (+0.005766009780722586) | > avg_loss_disc_real_3: 0.24190544452465756 (-0.0003399074497357213) | > avg_loss_disc_real_4: 0.24471417006472468 (+0.0008949786424636841) | > avg_loss_disc_real_5: 0.2478458780637929 (+0.0025957642306744244) | > avg_loss_0: 2.812282273467158 (+0.003390328984865576) | > avg_grad_norm_0: tensor(15.0015, device='cuda:0') (+tensor(0.9759, device='cuda:0')) | > avg_loss_gen: 1.789977364137139 (+0.0015953460209805481) | > avg_loss_kl: 2.3701633466801177 (+0.05486588410928217) | > avg_loss_feat: 2.2415976272502434 (-0.04348596216926914) | > avg_loss_mel: 17.959537613559778 (+0.07100841361033972) | > avg_loss_duration: 1.6344251347259737 (+0.0025103528734669034) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.99570110482229 (+0.08649409656793239) | > avg_grad_norm_1: tensor(248.6075, device='cuda:0') (tensor(-72.5207, device='cuda:0'))  > EPOCH: 201/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:15:25)   --> TIME: 2025-02-25 10:15:40 -- STEP: 27/72 -- GLOBAL_STEP: 1014500 | > loss_disc: 2.830667018890381 (2.8099524709913464) | > loss_disc_real_0: 0.22474849224090576 (0.21011274059613547) | > loss_disc_real_1: 0.25023382902145386 (0.2407487283150355) | > loss_disc_real_2: 0.26650765538215637 (0.2402085993025038) | > loss_disc_real_3: 0.273488849401474 (0.2418949410871223) | > loss_disc_real_4: 0.24354605376720428 (0.2473416455365993) | > loss_disc_real_5: 0.22475172579288483 (0.24646739220177685) | > loss_0: 2.830667018890381 (2.8099524709913464) | > grad_norm_0: tensor(10.9265, device='cuda:0') (tensor(10.0102, device='cuda:0')) | > loss_gen: 1.6906813383102417 (1.7976041634877522) | > loss_kl: 2.4602468013763428 (2.2898119114063404) | > loss_feat: 2.1395785808563232 (2.292075713475545) | > loss_mel: 16.806396484375 (17.589997114958585) | > loss_duration: 1.580041766166687 (1.5941499648270783) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.676944732666016 (25.563638969703955) | > grad_norm_1: tensor(102.8880, device='cuda:0') (tensor(286.0075, device='cuda:0')) | > current_lr_0: 0.000195037294886455 | > current_lr_1: 0.000195037294886455 | > step_time: 0.4416 (0.454960929022895) | > loader_time: 0.0085 (0.006723792464644821) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855135917663574 (-4.722702671104695e-05) | > avg_step_time: 0.4600096655563569 (+0.003992211650794664) | > avg_loss_disc: 2.811420776474643 (-0.0008614969925151428) | > avg_loss_disc_real_0: 0.21191837837998295 (-0.005940048312637142) | > avg_loss_disc_real_1: 0.23883468458350277 (+0.0043874699884737245) | > avg_loss_disc_real_2: 0.24219902321486406 (-0.0006758113142470157) | > avg_loss_disc_real_3: 0.24206782310781344 (+0.0001623785831558755) | > avg_loss_disc_real_4: 0.24546674607505262 (+0.0007525760103279377) | > avg_loss_disc_real_5: 0.24442748452576113 (-0.0034183935380317776) | > avg_loss_0: 2.811420776474643 (-0.0008614969925151428) | > avg_grad_norm_0: tensor(10.6932, device='cuda:0') (tensor(-4.3083, device='cuda:0')) | > avg_loss_gen: 1.7969870130780716 (+0.007009648940932589) | > avg_loss_kl: 2.343008484638913 (-0.02715486204120454) | > avg_loss_feat: 2.2558191191982218 (+0.014221491947978393) | > avg_loss_mel: 17.697942894949044 (-0.26159471861073413) | > avg_loss_duration: 1.626104494215737 (-0.008320640510236732) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.719862091709192 (-0.27583901311309944) | > avg_grad_norm_1: tensor(303.4874, device='cuda:0') (+tensor(54.8799, device='cuda:0'))  > EPOCH: 202/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:16:03)   --> TIME: 2025-02-25 10:16:07 -- STEP: 5/72 -- GLOBAL_STEP: 1014550 | > loss_disc: 2.8214101791381836 (2.8683105945587157) | > loss_disc_real_0: 0.3052104115486145 (0.23530149459838867) | > loss_disc_real_1: 0.23279830813407898 (0.2524644911289215) | > loss_disc_real_2: 0.2274113893508911 (0.2401471734046936) | > loss_disc_real_3: 0.23751166462898254 (0.23726092875003815) | > loss_disc_real_4: 0.23509612679481506 (0.2595484733581543) | > loss_disc_real_5: 0.2434854358434677 (0.25572389662265776) | > loss_0: 2.8214101791381836 (2.8683105945587157) | > grad_norm_0: tensor(20.7196, device='cuda:0') (tensor(14.9607, device='cuda:0')) | > loss_gen: 1.7299482822418213 (1.7748101949691772) | > loss_kl: 2.432347297668457 (2.336027240753174) | > loss_feat: 2.063993215560913 (2.1518999576568603) | > loss_mel: 18.30889129638672 (18.119503784179688) | > loss_duration: 1.5968286991119385 (1.6194225549697876) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.132009506225586 (26.001663208007812) | > grad_norm_1: tensor(63.8774, device='cuda:0') (tensor(168.8613, device='cuda:0')) | > current_lr_0: 0.00019501291522459419 | > current_lr_1: 0.00019501291522459419 | > step_time: 0.42 (0.43685026168823243) | > loader_time: 0.0059 (0.006606292724609375)  --> TIME: 2025-02-25 10:16:30 -- STEP: 55/72 -- GLOBAL_STEP: 1014600 | > loss_disc: 2.8778607845306396 (2.820374809611928) | > loss_disc_real_0: 0.27946045994758606 (0.22071926973082803) | > loss_disc_real_1: 0.22049586474895477 (0.23976448449221524) | > loss_disc_real_2: 0.2539489269256592 (0.2392700111324137) | > loss_disc_real_3: 0.256276935338974 (0.23593189147385685) | > loss_disc_real_4: 0.2584141492843628 (0.24578697193752636) | > loss_disc_real_5: 0.270069420337677 (0.24681709706783295) | > loss_0: 2.8778607845306396 (2.820374809611928) | > grad_norm_0: tensor(15.0604, device='cuda:0') (tensor(13.8197, device='cuda:0')) | > loss_gen: 1.7152867317199707 (1.7877467090433294) | > loss_kl: 2.535593032836914 (2.359241208163174) | > loss_feat: 1.971165657043457 (2.2433256040919916) | > loss_mel: 15.997149467468262 (17.869066429138186) | > loss_duration: 1.6596591472625732 (1.6191007115624168) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.87885284423828 (25.878480460427024) | > grad_norm_1: tensor(224.0521, device='cuda:0') (tensor(248.7489, device='cuda:0')) | > current_lr_0: 0.00019501291522459419 | > current_lr_1: 0.00019501291522459419 | > step_time: 0.4692 (0.44673976464705034) | > loader_time: 0.0092 (0.00786095532503995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591047475035762 (+3.9688298400021235e-05) | > avg_step_time: 0.45852589943039584 (-0.0014837661259610724) | > avg_loss_disc: 2.82338538639982 (+0.011964609925177339) | > avg_loss_disc_real_0: 0.22033674993985136 (+0.00841837155986841) | > avg_loss_disc_real_1: 0.2407709968761659 (+0.0019363122926631393) | > avg_loss_disc_real_2: 0.2411735519556932 (-0.0010254712591708515) | > avg_loss_disc_real_3: 0.23813604007304554 (-0.003931783034767905) | > avg_loss_disc_real_4: 0.24570077909550198 (+0.00023403302044935903) | > avg_loss_disc_real_5: 0.2474189965238034 (+0.002991511998042262) | > avg_loss_0: 2.82338538639982 (+0.011964609925177339) | > avg_grad_norm_0: tensor(13.8650, device='cuda:0') (+tensor(3.1718, device='cuda:0')) | > avg_loss_gen: 1.7894489496526582 (-0.0075380634254134105) | > avg_loss_kl: 2.3575453388858847 (+0.014536854246971487) | > avg_loss_feat: 2.203098221563958 (-0.052720897634263686) | > avg_loss_mel: 17.832849193626725 (+0.13490629867768078) | > avg_loss_duration: 1.628908207718755 (+0.0028037135030181126) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.811849755300603 (+0.09198766359141075) | > avg_grad_norm_1: tensor(256.8165, device='cuda:0') (tensor(-46.6709, device='cuda:0'))  > EPOCH: 203/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:16:40)   --> TIME: 2025-02-25 10:16:57 -- STEP: 33/72 -- GLOBAL_STEP: 1014650 | > loss_disc: 2.818465232849121 (2.81660144256823) | > loss_disc_real_0: 0.14904505014419556 (0.2201237208915479) | > loss_disc_real_1: 0.22280380129814148 (0.2388091766924569) | > loss_disc_real_2: 0.22041627764701843 (0.2417234800981753) | > loss_disc_real_3: 0.22000913321971893 (0.23975576640981616) | > loss_disc_real_4: 0.27085286378860474 (0.24665706853071848) | > loss_disc_real_5: 0.2526242136955261 (0.24679324663046634) | > loss_0: 2.818465232849121 (2.81660144256823) | > grad_norm_0: tensor(11.1606, device='cuda:0') (tensor(13.9432, device='cuda:0')) | > loss_gen: 1.709263563156128 (1.7897581331657642) | > loss_kl: 2.2675681114196777 (2.253112265557954) | > loss_feat: 2.077515125274658 (2.2287177389318287) | > loss_mel: 17.00885009765625 (17.641497294108067) | > loss_duration: 1.6091222763061523 (1.599115707657554) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.672317504882812 (25.512201193607215) | > grad_norm_1: tensor(294.4186, device='cuda:0') (tensor(298.8468, device='cuda:0')) | > current_lr_0: 0.0001949885386101911 | > current_lr_1: 0.0001949885386101911 | > step_time: 0.44 (0.44167398683952563) | > loader_time: 0.0078 (0.0074920654296875) --> EVAL PERFORMANCE | > avg_loader_time: 0.008769774101149866 (+0.00017872662611410385) | > avg_step_time: 0.4587360738028943 (+0.0002101743724984284) | > avg_loss_disc: 2.8200658307948583 (-0.003319555604961888) | > avg_loss_disc_real_0: 0.22404802999865842 (+0.003711280058807054) | > avg_loss_disc_real_1: 0.23623670060449922 (-0.004534296271666682) | > avg_loss_disc_real_2: 0.24158109745509188 (+0.00040754549939867046) | > avg_loss_disc_real_3: 0.24011482508249685 (+0.0019787850094513115) | > avg_loss_disc_real_4: 0.24518568897751017 (-0.0005150901179918066) | > avg_loss_disc_real_5: 0.24664192216497072 (-0.0007770743588326745) | > avg_loss_0: 2.8200658307948583 (-0.003319555604961888) | > avg_grad_norm_0: tensor(13.7198, device='cuda:0') (tensor(-0.1452, device='cuda:0')) | > avg_loss_gen: 1.7868115129605147 (-0.0026374366921435577) | > avg_loss_kl: 2.250040047605273 (-0.10750529128061181) | > avg_loss_feat: 2.2174698466985996 (+0.01437162513464152) | > avg_loss_mel: 17.63578494165984 (-0.1970642519668857) | > avg_loss_duration: 1.6243122799295775 (-0.004595927789177612) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.514418534829584 (-0.2974312204710188) | > avg_grad_norm_1: tensor(315.0276, device='cuda:0') (+tensor(58.2111, device='cuda:0'))  > EPOCH: 204/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:17:17)   --> TIME: 2025-02-25 10:17:24 -- STEP: 11/72 -- GLOBAL_STEP: 1014700 | > loss_disc: 2.839815616607666 (2.772027470848777) | > loss_disc_real_0: 0.1568208932876587 (0.208775902336294) | > loss_disc_real_1: 0.21801327168941498 (0.21757278117266568) | > loss_disc_real_2: 0.22624525427818298 (0.22858275744048032) | > loss_disc_real_3: 0.20116737484931946 (0.2384117435325276) | > loss_disc_real_4: 0.22809815406799316 (0.247322139414874) | > loss_disc_real_5: 0.2622503638267517 (0.24725120311433618) | > loss_0: 2.839815616607666 (2.772027470848777) | > grad_norm_0: tensor(16.5378, device='cuda:0') (tensor(9.4820, device='cuda:0')) | > loss_gen: 1.662821888923645 (1.7805410515178333) | > loss_kl: 2.3204610347747803 (2.1982764222405176) | > loss_feat: 2.136085271835327 (2.364453662525524) | > loss_mel: 17.680883407592773 (17.69240309975364) | > loss_duration: 1.5831044912338257 (1.6025190570137717) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.383358001708984 (25.638193477283824) | > grad_norm_1: tensor(184.8569, device='cuda:0') (tensor(294.1278, device='cuda:0')) | > current_lr_0: 0.00019496416504286482 | > current_lr_1: 0.00019496416504286482 | > step_time: 0.4371 (0.477262171831998) | > loader_time: 0.0062 (0.0062322833321311255)  --> TIME: 2025-02-25 10:17:48 -- STEP: 61/72 -- GLOBAL_STEP: 1014750 | > loss_disc: 2.8028526306152344 (2.800652339810231) | > loss_disc_real_0: 0.35484862327575684 (0.2220441841932594) | > loss_disc_real_1: 0.18424957990646362 (0.23078712747722377) | > loss_disc_real_2: 0.1869923323392868 (0.2410243849773876) | > loss_disc_real_3: 0.26318076252937317 (0.24065177362473283) | > loss_disc_real_4: 0.24006474018096924 (0.24418226250859557) | > loss_disc_real_5: 0.2574779689311981 (0.2455255088747525) | > loss_0: 2.8028526306152344 (2.800652339810231) | > grad_norm_0: tensor(39.9302, device='cuda:0') (tensor(13.7952, device='cuda:0')) | > loss_gen: 1.8342156410217285 (1.8221687195730991) | > loss_kl: 2.2935855388641357 (2.2971987079401495) | > loss_feat: 2.2191526889801025 (2.3375040507707436) | > loss_mel: 16.739534378051758 (18.021636306262415) | > loss_duration: 1.7013068199157715 (1.6283741446792102) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.787796020507812 (26.10688184519283) | > grad_norm_1: tensor(319.8158, device='cuda:0') (tensor(300.5624, device='cuda:0')) | > current_lr_0: 0.00019496416504286482 | > current_lr_1: 0.00019496416504286482 | > step_time: 0.4683 (0.45423896586308715) | > loader_time: 0.0115 (0.008114064326051804) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655830168388258 (-0.00011394393276160757) | > avg_step_time: 0.4617318636934522 (+0.0029957898905579317) | > avg_loss_disc: 2.809145192025414 (-0.01092063876944449) | > avg_loss_disc_real_0: 0.22049721456329588 (-0.0035508154353625365) | > avg_loss_disc_real_1: 0.23270721880482956 (-0.003529481799669665) | > avg_loss_disc_real_2: 0.24119002319557567 (-0.00039107425951620534) | > avg_loss_disc_real_3: 0.24110966270238582 (+0.000994837619888972) | > avg_loss_disc_real_4: 0.24522398202352122 (+3.829304601105088e-05) | > avg_loss_disc_real_5: 0.24666699431311917 (+2.5072148148452822e-05) | > avg_loss_0: 2.809145192025414 (-0.01092063876944449) | > avg_grad_norm_0: tensor(14.1174, device='cuda:0') (+tensor(0.3976, device='cuda:0')) | > avg_loss_gen: 1.8185611241300341 (+0.031749611169519465) | > avg_loss_kl: 2.294536343762573 (+0.04449629615730011) | > avg_loss_feat: 2.316928799723235 (+0.09945895302463548) | > avg_loss_mel: 17.95357702819395 (+0.3177920865341122) | > avg_loss_duration: 1.6351612235458803 (+0.010848943616302842) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.018764388393347 (+0.5043458535637626) | > avg_grad_norm_1: tensor(309.5126, device='cuda:0') (tensor(-5.5150, device='cuda:0'))  > EPOCH: 205/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:17:55)   --> TIME: 2025-02-25 10:18:14 -- STEP: 39/72 -- GLOBAL_STEP: 1014800 | > loss_disc: 2.845625877380371 (2.824803627454318) | > loss_disc_real_0: 0.3727865517139435 (0.23203076307590192) | > loss_disc_real_1: 0.22573143243789673 (0.23294981435323373) | > loss_disc_real_2: 0.249736025929451 (0.23973804712295532) | > loss_disc_real_3: 0.23501260578632355 (0.24136741650410187) | > loss_disc_real_4: 0.25583821535110474 (0.24536651296493334) | > loss_disc_real_5: 0.24636821448802948 (0.2492338334902739) | > loss_0: 2.845625877380371 (2.824803627454318) | > grad_norm_0: tensor(17.8704, device='cuda:0') (tensor(11.9081, device='cuda:0')) | > loss_gen: 1.8161171674728394 (1.7982923984527588) | > loss_kl: 2.24284029006958 (2.2641551677997294) | > loss_feat: 2.2180628776550293 (2.2212062737880607) | > loss_mel: 18.33365821838379 (17.669696612235832) | > loss_duration: 1.6152534484863281 (1.6060243722720025) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.22593116760254 (25.559374833718323) | > grad_norm_1: tensor(402.3589, device='cuda:0') (tensor(221.5957, device='cuda:0')) | > current_lr_0: 0.00019493979452223445 | > current_lr_1: 0.00019493979452223445 | > step_time: 0.4573 (0.4423211476741693) | > loader_time: 0.0081 (0.00739754163301908) --> EVAL PERFORMANCE | > avg_loader_time: 0.008658812079631104 (+2.9819112428459277e-06) | > avg_step_time: 0.45749348989674743 (-0.004238373796704775) | > avg_loss_disc: 2.8207339400976474 (+0.011588748072233646) | > avg_loss_disc_real_0: 0.2274868315164472 (+0.006989616953151312) | > avg_loss_disc_real_1: 0.23281950031367826 (+0.00011228150884870503) | > avg_loss_disc_real_2: 0.2405765174140393 (-0.0006135057815363776) | > avg_loss_disc_real_3: 0.2409606137326066 (-0.0001490489697792141) | > avg_loss_disc_real_4: 0.24504259291669012 (-0.00018138910683110354) | > avg_loss_disc_real_5: 0.24822393752319713 (+0.00155694321007796) | > avg_loss_0: 2.8207339400976474 (+0.011588748072233646) | > avg_grad_norm_0: tensor(10.9693, device='cuda:0') (tensor(-3.1481, device='cuda:0')) | > avg_loss_gen: 1.7903456805457532 (-0.028215443584280964) | > avg_loss_kl: 2.305962162958065 (+0.011425819195491904) | > avg_loss_feat: 2.1952924829133798 (-0.12163631680985532) | > avg_loss_mel: 17.584275903836094 (-0.36930112435785745) | > avg_loss_duration: 1.6287108323943447 (-0.006450391151535673) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.504586958549393 (-0.5141774298439543) | > avg_grad_norm_1: tensor(225.9573, device='cuda:0') (tensor(-83.5553, device='cuda:0'))  > EPOCH: 206/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:18:32)   --> TIME: 2025-02-25 10:18:41 -- STEP: 17/72 -- GLOBAL_STEP: 1014850 | > loss_disc: 2.83064603805542 (2.8266918238471535) | > loss_disc_real_0: 0.22539269924163818 (0.21842357165673199) | > loss_disc_real_1: 0.25613638758659363 (0.2340890572351568) | > loss_disc_real_2: 0.24315385520458221 (0.23839904017308178) | > loss_disc_real_3: 0.26516127586364746 (0.24468336035223567) | > loss_disc_real_4: 0.24927078187465668 (0.24290049864965327) | > loss_disc_real_5: 0.2528354525566101 (0.2474127727396348) | > loss_0: 2.83064603805542 (2.8266918238471535) | > grad_norm_0: tensor(4.4309, device='cuda:0') (tensor(7.9400, device='cuda:0')) | > loss_gen: 1.6970030069351196 (1.7736108443316292) | > loss_kl: 2.545639991760254 (2.230699525159948) | > loss_feat: 1.953476071357727 (2.210040870834799) | > loss_mel: 18.491621017456055 (18.01643113528981) | > loss_duration: 1.6218204498291016 (1.5915045177235323) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.309560775756836 (25.822286942425897) | > grad_norm_1: tensor(456.6858, device='cuda:0') (tensor(245.7416, device='cuda:0')) | > current_lr_0: 0.00019491542704791915 | > current_lr_1: 0.00019491542704791915 | > step_time: 0.4446 (0.43977949198554545) | > loader_time: 0.006 (0.00698998395134421)  --> TIME: 2025-02-25 10:19:05 -- STEP: 67/72 -- GLOBAL_STEP: 1014900 | > loss_disc: 2.777465343475342 (2.8133981227874756) | > loss_disc_real_0: 0.15309743583202362 (0.21356408153452092) | > loss_disc_real_1: 0.15796029567718506 (0.23375561499773567) | > loss_disc_real_2: 0.24116958677768707 (0.24023216073192769) | > loss_disc_real_3: 0.24521289765834808 (0.24194353729931276) | > loss_disc_real_4: 0.2714967727661133 (0.24536126318262585) | > loss_disc_real_5: 0.2732904553413391 (0.24679280706306003) | > loss_0: 2.777465343475342 (2.8133981227874756) | > grad_norm_0: tensor(16.3971, device='cuda:0') (tensor(10.8455, device='cuda:0')) | > loss_gen: 1.8220351934432983 (1.778923680533224) | > loss_kl: 2.376009464263916 (2.2382459035560256) | > loss_feat: 2.143418073654175 (2.2105611491559167) | > loss_mel: 18.131431579589844 (17.708847871467242) | > loss_duration: 1.7059863805770874 (1.6227650606810158) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.17888069152832 (25.559343651159487) | > grad_norm_1: tensor(514.6911, device='cuda:0') (tensor(250.6587, device='cuda:0')) | > current_lr_0: 0.00019491542704791915 | > current_lr_1: 0.00019491542704791915 | > step_time: 0.5319 (0.4530143488698931) | > loader_time: 0.0123 (0.008622176611601416) --> EVAL PERFORMANCE | > avg_loader_time: 0.008787544680313326 (+0.00012873260068222217) | > avg_step_time: 0.4569628507318631 (-0.0005306391648843278) | > avg_loss_disc: 2.8114620430368773 (-0.00927189706077014) | > avg_loss_disc_real_0: 0.21369704299829376 (-0.013789788518153434) | > avg_loss_disc_real_1: 0.23425216175300975 (+0.0014326614393314896) | > avg_loss_disc_real_2: 0.23950154848501715 (-0.0010749689290221465) | > avg_loss_disc_real_3: 0.24077409736707178 (-0.00018651636553482231) | > avg_loss_disc_real_4: 0.2452105466328876 (+0.00016795371619748867) | > avg_loss_disc_real_5: 0.2470853366482426 (-0.0011386008749545429) | > avg_loss_0: 2.8114620430368773 (-0.00927189706077014) | > avg_grad_norm_0: tensor(10.6140, device='cuda:0') (tensor(-0.3554, device='cuda:0')) | > avg_loss_gen: 1.7780261258004417 (-0.012319554745311434) | > avg_loss_kl: 2.246777480756732 (-0.05918468220133288) | > avg_loss_feat: 2.2149222081815703 (+0.019629725268190512) | > avg_loss_mel: 17.784377595068698 (+0.20010169123260368) | > avg_loss_duration: 1.6256910763995749 (-0.0030197559947697794) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.64979448452802 (+0.14520752597862696) | > avg_grad_norm_1: tensor(253.9590, device='cuda:0') (+tensor(28.0016, device='cuda:0'))  > EPOCH: 207/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:19:09)   --> TIME: 2025-02-25 10:19:31 -- STEP: 45/72 -- GLOBAL_STEP: 1014950 | > loss_disc: 2.7261226177215576 (2.7998702685038257) | > loss_disc_real_0: 0.21526682376861572 (0.21142224785354402) | > loss_disc_real_1: 0.20845246315002441 (0.232940474152565) | > loss_disc_real_2: 0.2411414086818695 (0.24076917701297337) | > loss_disc_real_3: 0.2332380712032318 (0.2403761131895913) | > loss_disc_real_4: 0.2357737272977829 (0.24584353466828665) | > loss_disc_real_5: 0.23670554161071777 (0.2474629216723972) | > loss_0: 2.7261226177215576 (2.7998702685038257) | > grad_norm_0: tensor(10.4981, device='cuda:0') (tensor(12.4063, device='cuda:0')) | > loss_gen: 1.8833727836608887 (1.80143072075314) | > loss_kl: 2.538573980331421 (2.3263423018985323) | > loss_feat: 2.63360595703125 (2.282390756077236) | > loss_mel: 18.90207862854004 (17.841829723782006) | > loss_duration: 1.646328091621399 (1.614048261112637) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.603960037231445 (25.866041522555882) | > grad_norm_1: tensor(568.5533, device='cuda:0') (tensor(260.6366, device='cuda:0')) | > current_lr_0: 0.00019489106261953815 | > current_lr_1: 0.00019489106261953815 | > step_time: 0.4538 (0.43961819012959796) | > loader_time: 0.009 (0.0075753106011284725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626209178441004 (-0.00016133550187232178) | > avg_step_time: 0.45371048887011023 (-0.0032523618617528727) | > avg_loss_disc: 2.8087449846133388 (-0.002717058423538532) | > avg_loss_disc_real_0: 0.21363729574310947 (-5.974725518428925e-05) | > avg_loss_disc_real_1: 0.24029786611946535 (+0.006045704366455601) | > avg_loss_disc_real_2: 0.24123508136876873 (+0.0017335328837515818) | > avg_loss_disc_real_3: 0.2403567800639381 (-0.0004173173031336852) | > avg_loss_disc_real_4: 0.24425822538389286 (-0.0009523212489947475) | > avg_loss_disc_real_5: 0.24617913764127544 (-0.0009061990069671455) | > avg_loss_0: 2.8087449846133388 (-0.002717058423538532) | > avg_grad_norm_0: tensor(13.4466, device='cuda:0') (+tensor(2.8326, device='cuda:0')) | > avg_loss_gen: 1.8026353355864404 (+0.024609209785998676) | > avg_loss_kl: 2.3393230270332017 (+0.09254554627646971) | > avg_loss_feat: 2.260299806863489 (+0.04537759868191893) | > avg_loss_mel: 17.861477677251248 (+0.07710008218255027) | > avg_loss_duration: 1.6318869490019032 (+0.006195872602328345) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.895622656378947 (+0.24582817185092765) | > avg_grad_norm_1: tensor(289.1734, device='cuda:0') (+tensor(35.2144, device='cuda:0'))  > EPOCH: 208/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:19:46)   --> TIME: 2025-02-25 10:19:58 -- STEP: 23/72 -- GLOBAL_STEP: 1015000 | > loss_disc: 2.7852649688720703 (2.801151234170665) | > loss_disc_real_0: 0.17393076419830322 (0.2114224203902742) | > loss_disc_real_1: 0.2276306450366974 (0.22639380395412445) | > loss_disc_real_2: 0.2483019083738327 (0.24075294188831164) | > loss_disc_real_3: 0.2518734037876129 (0.23846710376117541) | > loss_disc_real_4: 0.26037970185279846 (0.24613793777382892) | > loss_disc_real_5: 0.27331167459487915 (0.2459697354099025) | > loss_0: 2.7852649688720703 (2.801151234170665) | > grad_norm_0: tensor(7.6457, device='cuda:0') (tensor(12.2179, device='cuda:0')) | > loss_gen: 1.7524288892745972 (1.793389045673868) | > loss_kl: 2.166224479675293 (2.284480364426323) | > loss_feat: 1.9054309129714966 (2.2671190085618393) | > loss_mel: 16.760618209838867 (17.67765642249066) | > loss_duration: 1.6175655126571655 (1.5938206392785776) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.202266693115234 (25.616465278293777) | > grad_norm_1: tensor(59.5310, device='cuda:0') (tensor(271.2231, device='cuda:0')) | > current_lr_0: 0.0001948667012367107 | > current_lr_1: 0.0001948667012367107 | > step_time: 0.4299 (0.4555306123650592) | > loader_time: 0.0061 (0.006658129070116126) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1015000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.00857430444636815 (-5.190473207285391e-05) | > avg_step_time: 0.4641614598287663 (+0.010450970958656047) | > avg_loss_disc: 2.8159802430112597 (+0.007235258397920941) | > avg_loss_disc_real_0: 0.21637164780371626 (+0.002734352060606793) | > avg_loss_disc_real_1: 0.23232269350072027 (-0.007975172618745086) | > avg_loss_disc_real_2: 0.24051224073054087 (-0.0007228406382278618) | > avg_loss_disc_real_3: 0.2409461652728873 (+0.0005893852089492169) | > avg_loss_disc_real_4: 0.24552440370472384 (+0.0012661783208309796) | > avg_loss_disc_real_5: 0.246146205235535 (-3.293240574045453e-05) | > avg_loss_0: 2.8159802430112597 (+0.007235258397920941) | > avg_grad_norm_0: tensor(12.7798, device='cuda:0') (tensor(-0.6667, device='cuda:0')) | > avg_loss_gen: 1.7929761409759521 (-0.009659194610488253) | > avg_loss_kl: 2.3540681751681047 (+0.014745148134903019) | > avg_loss_feat: 2.2762956199511675 (+0.01599581308767828) | > avg_loss_mel: 17.911868270014377 (+0.050390592763129405) | > avg_loss_duration: 1.6263976516857954 (-0.005489297316107811) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.96160560930279 (+0.06598295292384293) | > avg_grad_norm_1: tensor(279.2558, device='cuda:0') (tensor(-9.9176, device='cuda:0'))  > EPOCH: 209/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:20:26)   --> TIME: 2025-02-25 10:20:28 -- STEP: 1/72 -- GLOBAL_STEP: 1015050 | > loss_disc: 2.764949083328247 (2.764949083328247) | > loss_disc_real_0: 0.29500943422317505 (0.29500943422317505) | > loss_disc_real_1: 0.2293839305639267 (0.2293839305639267) | > loss_disc_real_2: 0.26290416717529297 (0.26290416717529297) | > loss_disc_real_3: 0.26859599351882935 (0.26859599351882935) | > loss_disc_real_4: 0.2594437897205353 (0.2594437897205353) | > loss_disc_real_5: 0.2757881283760071 (0.2757881283760071) | > loss_0: 2.764949083328247 (2.764949083328247) | > grad_norm_0: tensor(15.4750, device='cuda:0') (tensor(15.4750, device='cuda:0')) | > loss_gen: 2.0311436653137207 (2.0311436653137207) | > loss_kl: 2.4031827449798584 (2.4031827449798584) | > loss_feat: 2.2726659774780273 (2.2726659774780273) | > loss_mel: 18.29155158996582 (18.29155158996582) | > loss_duration: 1.6450785398483276 (1.6450785398483276) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.64362144470215 (26.64362144470215) | > grad_norm_1: tensor(168.6069, device='cuda:0') (tensor(168.6069, device='cuda:0')) | > current_lr_0: 0.0001948423428990561 | > current_lr_1: 0.0001948423428990561 | > step_time: 0.4256 (0.4256253242492676) | > loader_time: 0.0067 (0.006669044494628906)  --> TIME: 2025-02-25 10:20:51 -- STEP: 51/72 -- GLOBAL_STEP: 1015100 | > loss_disc: 2.8655834197998047 (2.829378258948233) | > loss_disc_real_0: 0.34696561098098755 (0.223488117257754) | > loss_disc_real_1: 0.27183523774147034 (0.24113425963065205) | > loss_disc_real_2: 0.2935735285282135 (0.24348359072909637) | > loss_disc_real_3: 0.29083847999572754 (0.24278749467111102) | > loss_disc_real_4: 0.2742980420589447 (0.24596510330835977) | > loss_disc_real_5: 0.239512100815773 (0.24914806263119565) | > loss_0: 2.8655834197998047 (2.829378258948233) | > grad_norm_0: tensor(35.3140, device='cuda:0') (tensor(10.3888, device='cuda:0')) | > loss_gen: 2.0651168823242188 (1.8033417650297576) | > loss_kl: 2.325896739959717 (2.274995953428979) | > loss_feat: 2.150219440460205 (2.255938766049403) | > loss_mel: 18.54639434814453 (18.021099576763078) | > loss_duration: 1.6370854377746582 (1.6143023757373585) | > amp_scaler: 256.0 (133.01960784313727) | > loss_1: 26.724712371826172 (25.96967831779929) | > grad_norm_1: tensor(577.4040, device='cuda:0') (tensor(285.0215, device='cuda:0')) | > current_lr_0: 0.0001948423428990561 | > current_lr_1: 0.0001948423428990561 | > step_time: 0.4578 (0.44504236707500383) | > loader_time: 0.0091 (0.00786120283837412) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688923338769191 (+0.00011461889240104058) | > avg_step_time: 0.45752547492443674 (-0.006635984904329539) | > avg_loss_disc: 2.8233909002492124 (+0.007410657237952645) | > avg_loss_disc_real_0: 0.22181355218652268 (+0.005441904382806423) | > avg_loss_disc_real_1: 0.2394776075658664 (+0.007154914065146123) | > avg_loss_disc_real_2: 0.2413085077849912 (+0.0007962670544503225) | > avg_loss_disc_real_3: 0.2427476189086135 (+0.0018014536357261746) | > avg_loss_disc_real_4: 0.24539442339413603 (-0.00012998031058780724) | > avg_loss_disc_real_5: 0.2461384463478142 (-7.75888772078237e-06) | > avg_loss_0: 2.8233909002492124 (+0.007410657237952645) | > avg_grad_norm_0: tensor(10.5679, device='cuda:0') (tensor(-2.2119, device='cuda:0')) | > avg_loss_gen: 1.792175724472798 (-0.0008004165031541) | > avg_loss_kl: 2.321543428259836 (-0.032524746908268565) | > avg_loss_feat: 2.250249808942767 (-0.02604581100840031) | > avg_loss_mel: 18.102455783897724 (+0.19058751388334727) | > avg_loss_duration: 1.6274358739315624 (+0.0010382222457669954) | > avg_amp_scaler: 167.66197183098595 (+39.66197183098595) | > avg_loss_1: 26.093860626220703 (+0.13225501691791308) | > avg_grad_norm_1: tensor(293.5024, device='cuda:0') (+tensor(14.2466, device='cuda:0'))  > EPOCH: 210/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:21:03)   --> TIME: 2025-02-25 10:21:18 -- STEP: 29/72 -- GLOBAL_STEP: 1015150 | > loss_disc: 2.7187576293945312 (2.8179669051334777) | > loss_disc_real_0: 0.17622286081314087 (0.2216796001483654) | > loss_disc_real_1: 0.21308118104934692 (0.22908566429697233) | > loss_disc_real_2: 0.2635459303855896 (0.24202691069964705) | > loss_disc_real_3: 0.24483245611190796 (0.23806801798014804) | > loss_disc_real_4: 0.24855707585811615 (0.24769203159315833) | > loss_disc_real_5: 0.25229883193969727 (0.25301342894291046) | > loss_0: 2.7187576293945312 (2.8179669051334777) | > grad_norm_0: tensor(8.2326, device='cuda:0') (tensor(10.4507, device='cuda:0')) | > loss_gen: 1.9554005861282349 (1.788107913115929) | > loss_kl: 2.166937828063965 (2.3044753321285905) | > loss_feat: 2.3317668437957764 (2.2374005687647855) | > loss_mel: 18.100339889526367 (17.757234441822973) | > loss_duration: 1.6058268547058105 (1.595830646054498) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.1602725982666 (25.683048905997442) | > grad_norm_1: tensor(173.2433, device='cuda:0') (tensor(211.5013, device='cuda:0')) | > current_lr_0: 0.0001948179876061937 | > current_lr_1: 0.0001948179876061937 | > step_time: 0.4644 (0.4386472537599761) | > loader_time: 0.0077 (0.006994502297763167) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687415593106982 (-1.5077456622086122e-06) | > avg_step_time: 0.45567854021636534 (-0.0018469347080714016) | > avg_loss_disc: 2.8160632999849993 (-0.0073276002642130855) | > avg_loss_disc_real_0: 0.22624870652044324 (+0.004435154333920555) | > avg_loss_disc_real_1: 0.23007071920683686 (-0.00940688835902953) | > avg_loss_disc_real_2: 0.23736008896794117 (-0.00394841881705002) | > avg_loss_disc_real_3: 0.23791433283140961 (-0.004833286077203874) | > avg_loss_disc_real_4: 0.24470938259447125 (-0.0006850407996647767) | > avg_loss_disc_real_5: 0.2479456692514284 (+0.0018072229036142085) | > avg_loss_0: 2.8160632999849993 (-0.0073276002642130855) | > avg_grad_norm_0: tensor(12.7522, device='cuda:0') (+tensor(2.1843, device='cuda:0')) | > avg_loss_gen: 1.7721343628117736 (-0.020041361661024437) | > avg_loss_kl: 2.31521500835956 (-0.0063284199002762165) | > avg_loss_feat: 2.1811046382071275 (-0.06914517073563964) | > avg_loss_mel: 17.674621541735153 (-0.4278342421625716) | > avg_loss_duration: 1.6312613268973122 (+0.003825452965749765) | > avg_amp_scaler: 256.0 (+88.33802816901405) | > avg_loss_1: 25.574336898158975 (-0.5195237280617278) | > avg_grad_norm_1: tensor(225.4165, device='cuda:0') (tensor(-68.0860, device='cuda:0'))  > EPOCH: 211/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:21:40)   --> TIME: 2025-02-25 10:21:46 -- STEP: 7/72 -- GLOBAL_STEP: 1015200 | > loss_disc: 2.7384276390075684 (2.794182402747018) | > loss_disc_real_0: 0.16817651689052582 (0.1955359641994749) | > loss_disc_real_1: 0.20360930263996124 (0.226401863353593) | > loss_disc_real_2: 0.22656989097595215 (0.24447604162352427) | > loss_disc_real_3: 0.21048344671726227 (0.2355088187115533) | > loss_disc_real_4: 0.23151104152202606 (0.2480573058128357) | > loss_disc_real_5: 0.23846757411956787 (0.24145509515489852) | > loss_0: 2.7384276390075684 (2.794182402747018) | > grad_norm_0: tensor(8.2011, device='cuda:0') (tensor(9.0090, device='cuda:0')) | > loss_gen: 1.9148244857788086 (1.7798999377659388) | > loss_kl: 2.3471269607543945 (2.1849264076777866) | > loss_feat: 2.6858792304992676 (2.3176156793321883) | > loss_mel: 18.738033294677734 (17.788706370762416) | > loss_duration: 1.6098535060882568 (1.6298709086009435) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.295717239379883 (25.701019287109375) | > grad_norm_1: tensor(165.8948, device='cuda:0') (tensor(379.6559, device='cuda:0')) | > current_lr_0: 0.00019479363535774292 | > current_lr_1: 0.00019479363535774292 | > step_time: 0.4329 (0.4371122292109898) | > loader_time: 0.0064 (0.0065724849700927734)  --> TIME: 2025-02-25 10:22:09 -- STEP: 57/72 -- GLOBAL_STEP: 1015250 | > loss_disc: 2.6801671981811523 (2.786061684290568) | > loss_disc_real_0: 0.1888183057308197 (0.20609426812121742) | > loss_disc_real_1: 0.19719573855400085 (0.23094126477576138) | > loss_disc_real_2: 0.24305078387260437 (0.24016525609451428) | > loss_disc_real_3: 0.2201903909444809 (0.24071781833966574) | > loss_disc_real_4: 0.22854170203208923 (0.24430022166486373) | > loss_disc_real_5: 0.2179415225982666 (0.24625958659146963) | > loss_0: 2.6801671981811523 (2.786061684290568) | > grad_norm_0: tensor(6.2718, device='cuda:0') (tensor(9.7531, device='cuda:0')) | > loss_gen: 1.990615963935852 (1.8037490405534442) | > loss_kl: 2.2735841274261475 (2.3143509856441558) | > loss_feat: 2.351884603500366 (2.347943713790492) | > loss_mel: 17.4219970703125 (17.873307529248677) | > loss_duration: 1.6767170429229736 (1.6181380539609675) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.714799880981445 (25.957489281369927) | > grad_norm_1: tensor(527.2079, device='cuda:0') (tensor(315.2852, device='cuda:0')) | > current_lr_0: 0.00019479363535774292 | > current_lr_1: 0.00019479363535774292 | > step_time: 0.4773 (0.4509760204114412) | > loader_time: 0.0096 (0.007991723846970942) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624133929400377 (-6.32816637066054e-05) | > avg_step_time: 0.45916272888720877 (+0.0034841886708434355) | > avg_loss_disc: 2.7919570493026513 (-0.02410625068234795) | > avg_loss_disc_real_0: 0.21000198600157885 (-0.016246720518864388) | > avg_loss_disc_real_1: 0.23153235102203532 (+0.0014616318151984553) | > avg_loss_disc_real_2: 0.23795080415799585 (+0.0005907151900546859) | > avg_loss_disc_real_3: 0.23975487452157787 (+0.001840541690168257) | > avg_loss_disc_real_4: 0.24415458780778965 (-0.0005547947866816061) | > avg_loss_disc_real_5: 0.2474397085082363 (-0.0005059607431921187) | > avg_loss_0: 2.7919570493026513 (-0.02410625068234795) | > avg_grad_norm_0: tensor(10.5254, device='cuda:0') (tensor(-2.2268, device='cuda:0')) | > avg_loss_gen: 1.8015628650154867 (+0.02942850220371307) | > avg_loss_kl: 2.3111943929967738 (-0.004020615362786195) | > avg_loss_feat: 2.338250135032224 (+0.1571454968250965) | > avg_loss_mel: 17.836196590477304 (+0.16157504874215078) | > avg_loss_duration: 1.6256693937409092 (-0.0055919331564029395) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.912873389015736 (+0.33853649085676096) | > avg_grad_norm_1: tensor(345.7459, device='cuda:0') (+tensor(120.3294, device='cuda:0'))  > EPOCH: 212/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:22:18)   --> TIME: 2025-02-25 10:22:35 -- STEP: 35/72 -- GLOBAL_STEP: 1015300 | > loss_disc: 2.844420909881592 (2.7991580826895577) | > loss_disc_real_0: 0.27808958292007446 (0.2273000140275274) | > loss_disc_real_1: 0.3010484278202057 (0.23343127710478648) | > loss_disc_real_2: 0.30277377367019653 (0.24315683586256845) | > loss_disc_real_3: 0.24233002960681915 (0.24034673145839147) | > loss_disc_real_4: 0.25269263982772827 (0.24535511604377203) | > loss_disc_real_5: 0.24216362833976746 (0.2460114734513419) | > loss_0: 2.844420909881592 (2.7991580826895577) | > grad_norm_0: tensor(13.9650, device='cuda:0') (tensor(12.0822, device='cuda:0')) | > loss_gen: 1.8299459218978882 (1.7997121504374913) | > loss_kl: 2.4994232654571533 (2.2285505192620407) | > loss_feat: 2.3855206966400146 (2.268848388535636) | > loss_mel: 18.764808654785156 (17.484579467773436) | > loss_duration: 1.6605918407440186 (1.5933299303054809) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.140291213989258 (25.375020272391183) | > grad_norm_1: tensor(550.5094, device='cuda:0') (tensor(325.7752, device='cuda:0')) | > current_lr_0: 0.0001947692861533232 | > current_lr_1: 0.0001947692861533232 | > step_time: 0.4423 (0.4394767897469657) | > loader_time: 0.0084 (0.007299225670950754) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862583979754381 (+1.7058681434332923e-06) | > avg_step_time: 0.4538407325744629 (-0.00532199631274588) | > avg_loss_disc: 2.8148582653260568 (+0.022901216023405446) | > avg_loss_disc_real_0: 0.229267960807807 (+0.01926597480622816) | > avg_loss_disc_real_1: 0.2330958133851978 (+0.0015634623631624878) | > avg_loss_disc_real_2: 0.24327288693945173 (+0.005322082781455878) | > avg_loss_disc_real_3: 0.2399402340533028 (+0.0001853595317249379) | > avg_loss_disc_real_4: 0.24356878790217387 (-0.0005857999056157748) | > avg_loss_disc_real_5: 0.24423381098559205 (-0.0032058975226442465) | > avg_loss_0: 2.8148582653260568 (+0.022901216023405446) | > avg_grad_norm_0: tensor(14.1319, device='cuda:0') (+tensor(3.6065, device='cuda:0')) | > avg_loss_gen: 1.7931685867443892 (-0.00839427827109751) | > avg_loss_kl: 2.291296188260468 (-0.019898204736305658) | > avg_loss_feat: 2.2487381478430515 (-0.08951198718917253) | > avg_loss_mel: 17.714067002417337 (-0.12212958805996621) | > avg_loss_duration: 1.6253408109638052 (-0.00032858277710401396) | > avg_amp_scaler: 237.9718309859155 (-18.02816901408451) | > avg_loss_1: 25.672610457514374 (-0.2402629315013627) | > avg_grad_norm_1: tensor(291.5497, device='cuda:0') (tensor(-54.1961, device='cuda:0'))  > EPOCH: 213/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:22:55)   --> TIME: 2025-02-25 10:23:02 -- STEP: 13/72 -- GLOBAL_STEP: 1015350 | > loss_disc: 2.746612071990967 (2.8197282277620754) | > loss_disc_real_0: 0.24158960580825806 (0.2145669529071221) | > loss_disc_real_1: 0.2529413402080536 (0.23877329436632302) | > loss_disc_real_2: 0.2379326969385147 (0.23819916408795577) | > loss_disc_real_3: 0.21466279029846191 (0.24605280848649833) | > loss_disc_real_4: 0.2443390041589737 (0.24293203078783476) | > loss_disc_real_5: 0.2302689254283905 (0.2451546581891867) | > loss_0: 2.746612071990967 (2.8197282277620754) | > grad_norm_0: tensor(8.2151, device='cuda:0') (tensor(10.2390, device='cuda:0')) | > loss_gen: 1.7637262344360352 (1.7781785543148334) | > loss_kl: 2.377830982208252 (2.3765564148242655) | > loss_feat: 2.502488613128662 (2.298810812143179) | > loss_mel: 17.704513549804688 (18.4480714064378) | > loss_duration: 1.5630755424499512 (1.6106833127828746) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.911636352539062 (26.512300931490383) | > grad_norm_1: tensor(245.2879, device='cuda:0') (tensor(345.5894, device='cuda:0')) | > current_lr_0: 0.000194744939992554 | > current_lr_1: 0.000194744939992554 | > step_time: 0.4495 (0.43462098561800444) | > loader_time: 0.0058 (0.007233234552236704)  --> TIME: 2025-02-25 10:23:26 -- STEP: 63/72 -- GLOBAL_STEP: 1015400 | > loss_disc: 2.765812635421753 (2.8030878778487915) | > loss_disc_real_0: 0.22098366916179657 (0.2179671438440444) | > loss_disc_real_1: 0.25347280502319336 (0.23074512089055682) | > loss_disc_real_2: 0.20109979808330536 (0.23636034249313295) | > loss_disc_real_3: 0.21611715853214264 (0.23747293934935615) | > loss_disc_real_4: 0.21030892431735992 (0.24079336415207575) | > loss_disc_real_5: 0.226384699344635 (0.24375828272766537) | > loss_0: 2.765812635421753 (2.8030878778487915) | > grad_norm_0: tensor(7.3152, device='cuda:0') (tensor(14.9922, device='cuda:0')) | > loss_gen: 1.7392576932907104 (1.783135832302154) | > loss_kl: 2.2518885135650635 (2.3514663775761924) | > loss_feat: 2.460819959640503 (2.2765490422173147) | > loss_mel: 18.291828155517578 (17.98217437380836) | > loss_duration: 1.6173655986785889 (1.6248687543566265) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.36115837097168 (26.018194228883775) | > grad_norm_1: tensor(250.1355, device='cuda:0') (tensor(338.6854, device='cuda:0')) | > current_lr_0: 0.000194744939992554 | > current_lr_1: 0.000194744939992554 | > step_time: 0.5022 (0.4494506593734499) | > loader_time: 0.0116 (0.008410692214965817) --> EVAL PERFORMANCE | > avg_loader_time: 0.00878124505701199 (+0.0001554052594681804) | > avg_step_time: 0.4580058346331959 (+0.0041651020587329945) | > avg_loss_disc: 2.8045656378840063 (-0.010292627442050506) | > avg_loss_disc_real_0: 0.21749135180258417 (-0.011776609005222843) | > avg_loss_disc_real_1: 0.2323618214315092 (-0.0007339919536885897) | > avg_loss_disc_real_2: 0.23766955880212112 (-0.00560332813733061) | > avg_loss_disc_real_3: 0.2384302620316895 (-0.0015099720216133206) | > avg_loss_disc_real_4: 0.2409647245642165 (-0.002604063337957374) | > avg_loss_disc_real_5: 0.24417921187172473 (-5.459911386732075e-05) | > avg_loss_0: 2.8045656378840063 (-0.010292627442050506) | > avg_grad_norm_0: tensor(14.2646, device='cuda:0') (+tensor(0.1326, device='cuda:0')) | > avg_loss_gen: 1.7893747746104924 (-0.0037938121338967346) | > avg_loss_kl: 2.344512769873713 (+0.05321658161324505) | > avg_loss_feat: 2.2814237215149564 (+0.032685573671904944) | > avg_loss_mel: 17.925115719647472 (+0.21104871723013474) | > avg_loss_duration: 1.630113485833289 (+0.0047726748694838506) | > avg_amp_scaler: 128.0 (-109.97183098591549) | > avg_loss_1: 25.970540328764578 (+0.2979298712502043) | > avg_grad_norm_1: tensor(352.3754, device='cuda:0') (+tensor(60.8256, device='cuda:0'))  > EPOCH: 214/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:23:32)   --> TIME: 2025-02-25 10:23:52 -- STEP: 41/72 -- GLOBAL_STEP: 1015450 | > loss_disc: 2.8799397945404053 (2.8285770590712382) | > loss_disc_real_0: 0.18264909088611603 (0.21174073728119455) | > loss_disc_real_1: 0.25038614869117737 (0.24118142847607776) | > loss_disc_real_2: 0.2553010582923889 (0.2416734310185037) | > loss_disc_real_3: 0.25134459137916565 (0.24127003405152297) | > loss_disc_real_4: 0.25274673104286194 (0.24469262543247966) | > loss_disc_real_5: 0.25866472721099854 (0.24590820873655925) | > loss_0: 2.8799397945404053 (2.8285770590712382) | > grad_norm_0: tensor(7.9401, device='cuda:0') (tensor(9.7576, device='cuda:0')) | > loss_gen: 1.852905511856079 (1.7894508402522018) | > loss_kl: 2.3103950023651123 (2.222395902726708) | > loss_feat: 1.8695087432861328 (2.3090293494666496) | > loss_mel: 16.14881706237793 (17.679095198468463) | > loss_duration: 1.6238861083984375 (1.6135855273502628) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.805511474609375 (25.61355693165849) | > grad_norm_1: tensor(373.7638, device='cuda:0') (tensor(285.6301, device='cuda:0')) | > current_lr_0: 0.00019472059687505495 | > current_lr_1: 0.00019472059687505495 | > step_time: 0.4441 (0.4402308231446801) | > loader_time: 0.0094 (0.007631476332501668) --> EVAL PERFORMANCE | > avg_loader_time: 0.008723796253472983 (-5.7448803539007776e-05) | > avg_step_time: 0.46314488666158327 (+0.005139052028387381) | > avg_loss_disc: 2.8269366311355375 (+0.022370993251531246) | > avg_loss_disc_real_0: 0.2137866012856994 (-0.003704750516884764) | > avg_loss_disc_real_1: 0.23622361728003327 (+0.0038617958485240578) | > avg_loss_disc_real_2: 0.23935587515293713 (+0.0016863163508160084) | > avg_loss_disc_real_3: 0.24110511323096048 (+0.0026748511992709945) | > avg_loss_disc_real_4: 0.24480059365151632 (+0.003835869087299826) | > avg_loss_disc_real_5: 0.24734900094253917 (+0.003169789070814444) | > avg_loss_0: 2.8269366311355375 (+0.022370993251531246) | > avg_grad_norm_0: tensor(11.8028, device='cuda:0') (tensor(-2.4617, device='cuda:0')) | > avg_loss_gen: 1.7723741967913131 (-0.017000577819179297) | > avg_loss_kl: 2.2810370451967477 (-0.0634757246769655) | > avg_loss_feat: 2.2444418836647353 (-0.03698183785022113) | > avg_loss_mel: 17.800043092647066 (-0.12507262700040656) | > avg_loss_duration: 1.634469514161768 (+0.0043560283284789225) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.73236570223956 (-0.2381746265250193) | > avg_grad_norm_1: tensor(274.5161, device='cuda:0') (tensor(-77.8593, device='cuda:0'))  > EPOCH: 215/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:24:09)   --> TIME: 2025-02-25 10:24:20 -- STEP: 19/72 -- GLOBAL_STEP: 1015500 | > loss_disc: 2.8264214992523193 (2.8128818963703357) | > loss_disc_real_0: 0.15345999598503113 (0.21768394622363543) | > loss_disc_real_1: 0.1634427309036255 (0.2447856570544996) | > loss_disc_real_2: 0.24064795672893524 (0.24910381357920797) | > loss_disc_real_3: 0.21735088527202606 (0.24393839114590696) | > loss_disc_real_4: 0.24783191084861755 (0.24308924690673225) | > loss_disc_real_5: 0.25903570652008057 (0.24248305904237846) | > loss_0: 2.8264214992523193 (2.8128818963703357) | > grad_norm_0: tensor(10.4224, device='cuda:0') (tensor(12.9672, device='cuda:0')) | > loss_gen: 1.80061936378479 (1.8101799550809359) | > loss_kl: 2.53153395652771 (2.336515878376208) | > loss_feat: 2.285240650177002 (2.3278419218565287) | > loss_mel: 16.784698486328125 (18.12357440747713) | > loss_duration: 1.5711544752120972 (1.5894317438727932) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.97324562072754 (26.187543869018555) | > grad_norm_1: tensor(518.3120, device='cuda:0') (tensor(314.1645, device='cuda:0')) | > current_lr_0: 0.00019469625680044555 | > current_lr_1: 0.00019469625680044555 | > step_time: 0.4308 (0.43678720373856394) | > loader_time: 0.0062 (0.006620658071417557)  --> TIME: 2025-02-25 10:24:44 -- STEP: 69/72 -- GLOBAL_STEP: 1015550 | > loss_disc: 2.84954833984375 (2.8051617732946426) | > loss_disc_real_0: 0.1498112976551056 (0.21604125730801318) | > loss_disc_real_1: 0.21774494647979736 (0.23905652545500491) | > loss_disc_real_2: 0.2448015809059143 (0.2430854012136874) | > loss_disc_real_3: 0.24001505970954895 (0.2414442555627961) | > loss_disc_real_4: 0.23384429514408112 (0.24572969130847766) | > loss_disc_real_5: 0.25645849108695984 (0.24726108990717624) | > loss_0: 2.84954833984375 (2.8051617732946426) | > grad_norm_0: tensor(13.3714, device='cuda:0') (tensor(11.9314, device='cuda:0')) | > loss_gen: 1.634419322013855 (1.8075300420539966) | > loss_kl: 2.4876558780670166 (2.431385171586189) | > loss_feat: 2.0501692295074463 (2.3335841151251318) | > loss_mel: 18.22677230834961 (18.197603142779805) | > loss_duration: 1.6920280456542969 (1.6296449588692707) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.09104347229004 (26.3997473232988) | > grad_norm_1: tensor(304.4391, device='cuda:0') (tensor(365.4581, device='cuda:0')) | > current_lr_0: 0.00019469625680044555 | > current_lr_1: 0.00019469625680044555 | > step_time: 0.5437 (0.45504780783169513) | > loader_time: 0.013 (0.00849839915399966) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555408934472312 (-0.00016838731900067037) | > avg_step_time: 0.4562980188450343 (-0.006846867816548974) | > avg_loss_disc: 2.8054685156110306 (-0.021468115524506892) | > avg_loss_disc_real_0: 0.2145606102028363 (+0.0007740089171368869) | > avg_loss_disc_real_1: 0.2380774887934537 (+0.0018538715134204242) | > avg_loss_disc_real_2: 0.2424515380825795 (+0.003095662929642362) | > avg_loss_disc_real_3: 0.24027721311005068 (-0.0008279001209098025) | > avg_loss_disc_real_4: 0.24464334778382746 (-0.0001572458676888655) | > avg_loss_disc_real_5: 0.2467076215945499 (-0.0006413793479892738) | > avg_loss_0: 2.8054685156110306 (-0.021468115524506892) | > avg_grad_norm_0: tensor(12.0113, device='cuda:0') (+tensor(0.2085, device='cuda:0')) | > avg_loss_gen: 1.8066863493180612 (+0.03431215252674802) | > avg_loss_kl: 2.430546421400259 (+0.14950937620351112) | > avg_loss_feat: 2.33035196552814 (+0.08591008186340465) | > avg_loss_mel: 18.19143378566688 (+0.3913906930198152) | > avg_loss_duration: 1.6298628709685634 (-0.004606643193204585) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.38888128038863 (+0.6565155781490724) | > avg_grad_norm_1: tensor(366.1317, device='cuda:0') (+tensor(91.6157, device='cuda:0'))  > EPOCH: 216/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:24:46)   --> TIME: 2025-02-25 10:25:10 -- STEP: 47/72 -- GLOBAL_STEP: 1015600 | > loss_disc: 2.764753818511963 (2.807094229028579) | > loss_disc_real_0: 0.24235811829566956 (0.21984236402080415) | > loss_disc_real_1: 0.16131547093391418 (0.23589086944752552) | > loss_disc_real_2: 0.23786866664886475 (0.2361573564245346) | > loss_disc_real_3: 0.23858769237995148 (0.24385479092597961) | > loss_disc_real_4: 0.24249418079853058 (0.24690484683564368) | > loss_disc_real_5: 0.24891532957553864 (0.24896154917301017) | > loss_0: 2.764753818511963 (2.807094229028579) | > grad_norm_0: tensor(16.0434, device='cuda:0') (tensor(14.8803, device='cuda:0')) | > loss_gen: 1.7460405826568604 (1.8112165547431784) | > loss_kl: 2.3030552864074707 (2.316977952389007) | > loss_feat: 2.2834160327911377 (2.3607327633715687) | > loss_mel: 18.04623031616211 (17.919740859498372) | > loss_duration: 1.6429741382598877 (1.6073410130561667) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.021717071533203 (26.01600910754914) | > grad_norm_1: tensor(124.1019, device='cuda:0') (tensor(262.7688, device='cuda:0')) | > current_lr_0: 0.0001946719197683455 | > current_lr_1: 0.0001946719197683455 | > step_time: 0.4489 (0.44274992638445915) | > loader_time: 0.0088 (0.0075884169720588845) --> EVAL PERFORMANCE | > avg_loader_time: 0.008594052892335703 (+3.86439578633907e-05) | > avg_step_time: 0.4555654122795857 (-0.0007326065654486014) | > avg_loss_disc: 2.811029702844753 (+0.005561187233722453) | > avg_loss_disc_real_0: 0.21931116713184706 (+0.004750556929010774) | > avg_loss_disc_real_1: 0.2339701148825632 (-0.004107373910890499) | > avg_loss_disc_real_2: 0.23895180687098436 (-0.0034997312115951362) | > avg_loss_disc_real_3: 0.24249868493684582 (+0.002221471826795135) | > avg_loss_disc_real_4: 0.24643425307643246 (+0.001790905292605005) | > avg_loss_disc_real_5: 0.24860478979600997 (+0.0018971682014600755) | > avg_loss_0: 2.811029702844753 (+0.005561187233722453) | > avg_grad_norm_0: tensor(13.4845, device='cuda:0') (+tensor(1.4732, device='cuda:0')) | > avg_loss_gen: 1.7936267802413082 (-0.013059569076752986) | > avg_loss_kl: 2.3220684864151644 (-0.10847793498509439) | > avg_loss_feat: 2.291176166332943 (-0.03917579919519687) | > avg_loss_mel: 17.865734543598872 (-0.3256992420680085) | > avg_loss_duration: 1.625653669867717 (-0.004209201100846327) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.898259606159908 (-0.4906216742287235) | > avg_grad_norm_1: tensor(284.0119, device='cuda:0') (tensor(-82.1198, device='cuda:0'))  > EPOCH: 217/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:25:23)   --> TIME: 2025-02-25 10:25:36 -- STEP: 25/72 -- GLOBAL_STEP: 1015650 | > loss_disc: 2.7534444332122803 (2.8219992446899416) | > loss_disc_real_0: 0.30333247780799866 (0.22850760638713838) | > loss_disc_real_1: 0.22034600377082825 (0.23857815563678741) | > loss_disc_real_2: 0.24738021194934845 (0.2428393530845642) | > loss_disc_real_3: 0.22750578820705414 (0.24402630150318147) | > loss_disc_real_4: 0.24739354848861694 (0.24257951200008393) | > loss_disc_real_5: 0.26569539308547974 (0.24366280257701875) | > loss_0: 2.7534444332122803 (2.8219992446899416) | > grad_norm_0: tensor(14.5430, device='cuda:0') (tensor(15.3659, device='cuda:0')) | > loss_gen: 1.9129379987716675 (1.821126708984375) | > loss_kl: 2.178395986557007 (2.2469081401824953) | > loss_feat: 2.294269561767578 (2.329029483795166) | > loss_mel: 17.310916900634766 (17.91217895507812) | > loss_duration: 1.5772535800933838 (1.5845337200164795) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.273773193359375 (25.893776931762694) | > grad_norm_1: tensor(34.4685, device='cuda:0') (tensor(324.5024, device='cuda:0')) | > current_lr_0: 0.00019464758577837445 | > current_lr_1: 0.00019464758577837445 | > step_time: 0.4395 (0.43280562400817874) | > loader_time: 0.0072 (0.006859111785888672) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555063059632205 (-3.898983270349786e-05) | > avg_step_time: 0.4520764115830542 (-0.0034890006965314635) | > avg_loss_disc: 2.815040521218743 (+0.004010818373989977) | > avg_loss_disc_real_0: 0.217650830325946 (-0.0016603368059010526) | > avg_loss_disc_real_1: 0.23596412698987504 (+0.0019940121073118433) | > avg_loss_disc_real_2: 0.24214177253380628 (+0.0031899656628219253) | > avg_loss_disc_real_3: 0.24120329089567694 (-0.0012953940411688791) | > avg_loss_disc_real_4: 0.24142292435740081 (-0.005011328719031649) | > avg_loss_disc_real_5: 0.24443983129212554 (-0.0041649585038844306) | > avg_loss_0: 2.815040521218743 (+0.004010818373989977) | > avg_grad_norm_0: tensor(12.9383, device='cuda:0') (tensor(-0.5462, device='cuda:0')) | > avg_loss_gen: 1.7998520555630537 (+0.006225275321745549) | > avg_loss_kl: 2.3092271173504044 (-0.012841369064759967) | > avg_loss_feat: 2.317596180338255 (+0.02642001400531191) | > avg_loss_mel: 17.85620213898135 (-0.009532404617523582) | > avg_loss_duration: 1.6242553885553923 (-0.0013982813123247784) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.907132806912273 (+0.008873200752365307) | > avg_grad_norm_1: tensor(299.1027, device='cuda:0') (+tensor(15.0908, device='cuda:0'))  > EPOCH: 218/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:26:00)   --> TIME: 2025-02-25 10:26:03 -- STEP: 3/72 -- GLOBAL_STEP: 1015700 | > loss_disc: 2.8262219429016113 (2.7984867095947266) | > loss_disc_real_0: 0.2960168719291687 (0.2461552768945694) | > loss_disc_real_1: 0.23153522610664368 (0.20855141679445902) | > loss_disc_real_2: 0.21420146524906158 (0.20332834621270499) | > loss_disc_real_3: 0.21043722331523895 (0.21283021569252014) | > loss_disc_real_4: 0.2700279951095581 (0.2565640062093735) | > loss_disc_real_5: 0.24801087379455566 (0.26244603594144184) | > loss_0: 2.8262219429016113 (2.7984867095947266) | > grad_norm_0: tensor(15.6861, device='cuda:0') (tensor(13.8318, device='cuda:0')) | > loss_gen: 1.7594208717346191 (1.811841328938802) | > loss_kl: 2.3702194690704346 (2.356300433476766) | > loss_feat: 2.281275987625122 (2.3564254442850747) | > loss_mel: 18.396739959716797 (18.31411616007487) | > loss_duration: 1.6491649150848389 (1.6191612482070923) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.456819534301758 (26.457844416300457) | > grad_norm_1: tensor(261.8070, device='cuda:0') (tensor(357.9698, device='cuda:0')) | > current_lr_0: 0.00019462325483015215 | > current_lr_1: 0.00019462325483015215 | > step_time: 0.4264 (0.4316554069519043) | > loader_time: 0.0085 (0.00718839963277181)  --> TIME: 2025-02-25 10:26:27 -- STEP: 53/72 -- GLOBAL_STEP: 1015750 | > loss_disc: 2.8207762241363525 (2.8139709751561) | > loss_disc_real_0: 0.27978986501693726 (0.21323724878284167) | > loss_disc_real_1: 0.2492971271276474 (0.23622091761175193) | > loss_disc_real_2: 0.22221611440181732 (0.23668484136743365) | > loss_disc_real_3: 0.2590428292751312 (0.2404721443945507) | > loss_disc_real_4: 0.2652781009674072 (0.24763160578484805) | > loss_disc_real_5: 0.2500953674316406 (0.2486617570778109) | > loss_0: 2.8207762241363525 (2.8139709751561) | > grad_norm_0: tensor(27.9813, device='cuda:0') (tensor(12.3752, device='cuda:0')) | > loss_gen: 1.9513673782348633 (1.79843655172384) | > loss_kl: 2.1247496604919434 (2.2803311887777076) | > loss_feat: 2.361629009246826 (2.292124053217329) | > loss_mel: 16.903818130493164 (17.814376795066973) | > loss_duration: 1.6511162519454956 (1.614483876048394) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.992679595947266 (25.799752469332713) | > grad_norm_1: tensor(310.9912, device='cuda:0') (tensor(308.5954, device='cuda:0')) | > current_lr_0: 0.00019462325483015215 | > current_lr_1: 0.00019462325483015215 | > step_time: 0.4529 (0.4543370435822685) | > loader_time: 0.0096 (0.008199291409186595) --> EVAL PERFORMANCE | > avg_loader_time: 0.008990223978606745 (+0.00043516091897453923) | > avg_step_time: 0.4638933766056115 (+0.011816965022557246) | > avg_loss_disc: 2.818734608905416 (+0.0036940876866728978) | > avg_loss_disc_real_0: 0.2142507249200848 (-0.00340010540586122) | > avg_loss_disc_real_1: 0.23667802609188457 (+0.0007138991020095298) | > avg_loss_disc_real_2: 0.23924479623075942 (-0.002896976303046861) | > avg_loss_disc_real_3: 0.2402010931515358 (-0.0010021977441411478) | > avg_loss_disc_real_4: 0.24646843634021115 (+0.005045511982810336) | > avg_loss_disc_real_5: 0.24828421334985276 (+0.003844382057727219) | > avg_loss_0: 2.818734608905416 (+0.0036940876866728978) | > avg_grad_norm_0: tensor(12.4673, device='cuda:0') (tensor(-0.4711, device='cuda:0')) | > avg_loss_gen: 1.7885821705133143 (-0.01126988504973947) | > avg_loss_kl: 2.291968150877617 (-0.017258966472787574) | > avg_loss_feat: 2.2548955376719078 (-0.06270064266634723) | > avg_loss_mel: 17.72119726261622 (-0.13500487636513014) | > avg_loss_duration: 1.624760365821946 (+0.0005049772665537411) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.681403469031967 (-0.22572933788030625) | > avg_grad_norm_1: tensor(343.1216, device='cuda:0') (+tensor(44.0189, device='cuda:0'))  > EPOCH: 219/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:26:37)   --> TIME: 2025-02-25 10:26:53 -- STEP: 31/72 -- GLOBAL_STEP: 1015800 | > loss_disc: 2.7660038471221924 (2.779794669920398) | > loss_disc_real_0: 0.2627866268157959 (0.2042162480854219) | > loss_disc_real_1: 0.24018225073814392 (0.23766791532116552) | > loss_disc_real_2: 0.20467638969421387 (0.23747599605591066) | > loss_disc_real_3: 0.21575899422168732 (0.2429326916894605) | > loss_disc_real_4: 0.22082895040512085 (0.24118013487708184) | > loss_disc_real_5: 0.2182636559009552 (0.2481206472842924) | > loss_0: 2.7660038471221924 (2.779794669920398) | > grad_norm_0: tensor(12.2533, device='cuda:0') (tensor(12.8762, device='cuda:0')) | > loss_gen: 1.6341094970703125 (1.8163871495954451) | > loss_kl: 2.35418438911438 (2.2904487579099593) | > loss_feat: 2.422492504119873 (2.3946987621245848) | > loss_mel: 18.391036987304688 (17.852851375456776) | > loss_duration: 1.6527249813079834 (1.6000841894457418) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.454547882080078 (25.954470049950384) | > grad_norm_1: tensor(166.0448, device='cuda:0') (tensor(337.7898, device='cuda:0')) | > current_lr_0: 0.00019459892692329838 | > current_lr_1: 0.00019459892692329838 | > step_time: 0.441 (0.43754466118351104) | > loader_time: 0.0083 (0.006939080453688099) --> EVAL PERFORMANCE | > avg_loader_time: 0.008552306134935836 (-0.00043791784367090865) | > avg_step_time: 0.45763721600384777 (-0.006256160601763705) | > avg_loss_disc: 2.8007683082365653 (-0.017966300668850632) | > avg_loss_disc_real_0: 0.20921831714435363 (-0.0050324077757311625) | > avg_loss_disc_real_1: 0.23522138091879832 (-0.0014566451730862462) | > avg_loss_disc_real_2: 0.23626729258349244 (-0.0029775036472669825) | > avg_loss_disc_real_3: 0.24068532695232983 (+0.00048423380079404277) | > avg_loss_disc_real_4: 0.24348079948358134 (-0.0029876368566298106) | > avg_loss_disc_real_5: 0.24652130146261672 (-0.0017629118872360405) | > avg_loss_0: 2.8007683082365653 (-0.017966300668850632) | > avg_grad_norm_0: tensor(15.5507, device='cuda:0') (+tensor(3.0834, device='cuda:0')) | > avg_loss_gen: 1.8026265896541969 (+0.014044419140882614) | > avg_loss_kl: 2.312755514198626 (+0.020787363321009167) | > avg_loss_feat: 2.348184698064563 (+0.09328916039265511) | > avg_loss_mel: 17.87675075799646 (+0.15555349538024288) | > avg_loss_duration: 1.6293842036959152 (+0.004623837873969139) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.969701525191187 (+0.28829805615922055) | > avg_grad_norm_1: tensor(363.5453, device='cuda:0') (+tensor(20.4237, device='cuda:0'))  > EPOCH: 220/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:27:15)   --> TIME: 2025-02-25 10:27:20 -- STEP: 9/72 -- GLOBAL_STEP: 1015850 | > loss_disc: 2.853621006011963 (2.778430117501153) | > loss_disc_real_0: 0.18828436732292175 (0.2066153883934021) | > loss_disc_real_1: 0.16570670902729034 (0.22874524195988974) | > loss_disc_real_2: 0.2926616668701172 (0.24272601968712276) | > loss_disc_real_3: 0.2396240383386612 (0.23965611888302696) | > loss_disc_real_4: 0.2763679027557373 (0.2523108936018414) | > loss_disc_real_5: 0.25101202726364136 (0.2437647200293011) | > loss_0: 2.853621006011963 (2.778430117501153) | > grad_norm_0: tensor(6.3092, device='cuda:0') (tensor(9.9751, device='cuda:0')) | > loss_gen: 2.043388843536377 (1.8499051729838054) | > loss_kl: 2.6365292072296143 (2.361308309766981) | > loss_feat: 2.675938606262207 (2.430252128177219) | > loss_mel: 19.40731430053711 (17.575074937608505) | > loss_duration: 1.6125115156173706 (1.6066135432985094) | > amp_scaler: 128.0 (128.0) | > loss_1: 28.375680923461914 (25.823153813680012) | > grad_norm_1: tensor(500.0658, device='cuda:0') (tensor(330.1396, device='cuda:0')) | > current_lr_0: 0.00019457460205743297 | > current_lr_1: 0.00019457460205743297 | > step_time: 0.4204 (0.42815044191148544) | > loader_time: 0.0054 (0.006466574139065213)  --> TIME: 2025-02-25 10:27:44 -- STEP: 59/72 -- GLOBAL_STEP: 1015900 | > loss_disc: 2.8584907054901123 (2.8115218291848394) | > loss_disc_real_0: 0.1930030882358551 (0.22054371856531854) | > loss_disc_real_1: 0.24383299052715302 (0.23661613893710962) | > loss_disc_real_2: 0.27138951420783997 (0.23982109381990918) | > loss_disc_real_3: 0.21241764724254608 (0.2423642234782041) | > loss_disc_real_4: 0.2490001916885376 (0.24663636644007794) | > loss_disc_real_5: 0.25951871275901794 (0.24855417871879318) | > loss_0: 2.8584907054901123 (2.8115218291848394) | > grad_norm_0: tensor(6.3487, device='cuda:0') (tensor(15.3381, device='cuda:0')) | > loss_gen: 1.850446343421936 (1.8099120693691706) | > loss_kl: 2.5725040435791016 (2.3337308067386444) | > loss_feat: 2.096783399581909 (2.31833051422895) | > loss_mel: 17.635635375976562 (17.773931761919442) | > loss_duration: 1.6527462005615234 (1.616602099547952) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.808116912841797 (25.852507348787988) | > grad_norm_1: tensor(127.9569, device='cuda:0') (tensor(322.2278, device='cuda:0')) | > current_lr_0: 0.00019457460205743297 | > current_lr_1: 0.00019457460205743297 | > step_time: 0.4664 (0.4465097249564478) | > loader_time: 0.0099 (0.008037930828029826) --> EVAL PERFORMANCE | > avg_loader_time: 0.008590668020114092 (+3.836188517825599e-05) | > avg_step_time: 0.4573071607401673 (-0.0003300552636804621) | > avg_loss_disc: 2.807935637487492 (+0.0071673292509268904) | > avg_loss_disc_real_0: 0.22191768811202386 (+0.012699370967670237) | > avg_loss_disc_real_1: 0.23411777384684118 (-0.0011036070719571411) | > avg_loss_disc_real_2: 0.24065843350450758 (+0.004391140921015141) | > avg_loss_disc_real_3: 0.24174240210526426 (+0.0010570751529344335) | > avg_loss_disc_real_4: 0.2458334827507046 (+0.0023526832671232623) | > avg_loss_disc_real_5: 0.2481696910001862 (+0.001648389537569489) | > avg_loss_0: 2.807935637487492 (+0.0071673292509268904) | > avg_grad_norm_0: tensor(14.5399, device='cuda:0') (tensor(-1.0108, device='cuda:0')) | > avg_loss_gen: 1.808089320088776 (+0.005462730434579122) | > avg_loss_kl: 2.3318999854611686 (+0.019144471262542595) | > avg_loss_feat: 2.3087804569324977 (-0.0394042411320652) | > avg_loss_mel: 17.835321909944778 (-0.041428848051683786) | > avg_loss_duration: 1.6248443563219528 (-0.004539847373962402) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.908936111020374 (-0.06076541417081316) | > avg_grad_norm_1: tensor(332.3581, device='cuda:0') (tensor(-31.1873, device='cuda:0'))  > EPOCH: 221/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:27:52)   --> TIME: 2025-02-25 10:28:10 -- STEP: 37/72 -- GLOBAL_STEP: 1015950 | > loss_disc: 2.894197463989258 (2.809937779967849) | > loss_disc_real_0: 0.12039177119731903 (0.21375853648862322) | > loss_disc_real_1: 0.25462934374809265 (0.24020173340230375) | > loss_disc_real_2: 0.2819291651248932 (0.24320374670866374) | > loss_disc_real_3: 0.26238924264907837 (0.2408615185602291) | > loss_disc_real_4: 0.2598930597305298 (0.24272599292768016) | > loss_disc_real_5: 0.2315032184123993 (0.2430845680269035) | > loss_0: 2.894197463989258 (2.809937779967849) | > grad_norm_0: tensor(24.4932, device='cuda:0') (tensor(12.5680, device='cuda:0')) | > loss_gen: 1.5952762365341187 (1.8090775947313051) | > loss_kl: 2.1113927364349365 (2.3496308294502466) | > loss_feat: 2.047985792160034 (2.3392720286910604) | > loss_mel: 17.14130401611328 (18.08854695912954) | > loss_duration: 1.6305649280548096 (1.6080801358094086) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.52652359008789 (26.194607605805267) | > grad_norm_1: tensor(249.0986, device='cuda:0') (tensor(248.9274, device='cuda:0')) | > current_lr_0: 0.00019455028023217577 | > current_lr_1: 0.00019455028023217577 | > step_time: 0.4536 (0.4390839628271154) | > loader_time: 0.0101 (0.007340630969485721) --> EVAL PERFORMANCE | > avg_loader_time: 0.008744911408760178 (+0.0001542433886460859) | > avg_step_time: 0.45652043315726265 (-0.000786727582904656) | > avg_loss_disc: 2.812221829320343 (+0.00428619183285095) | > avg_loss_disc_real_0: 0.21874462760670085 (-0.003173060505323011) | > avg_loss_disc_real_1: 0.23559308408851354 (+0.0014753102416723562) | > avg_loss_disc_real_2: 0.24162567522324307 (+0.0009672417187354954) | > avg_loss_disc_real_3: 0.24146063982600896 (-0.0002817622792553043) | > avg_loss_disc_real_4: 0.24291023549059748 (-0.00292324726010712) | > avg_loss_disc_real_5: 0.24396715071839345 (-0.00420254028179276) | > avg_loss_0: 2.812221829320343 (+0.00428619183285095) | > avg_grad_norm_0: tensor(13.3167, device='cuda:0') (tensor(-1.2232, device='cuda:0')) | > avg_loss_gen: 1.796317674744297 (-0.011771645344478943) | > avg_loss_kl: 2.3412330973316244 (+0.009333111870455824) | > avg_loss_feat: 2.2671912824603875 (-0.04158917447211019) | > avg_loss_mel: 17.97058258593922 (+0.13526067599444147) | > avg_loss_duration: 1.6299381642274453 (+0.00509380790549252) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.005262885295167 (+0.09632677427479308) | > avg_grad_norm_1: tensor(292.1891, device='cuda:0') (tensor(-40.1689, device='cuda:0'))  > EPOCH: 222/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:28:29)   --> TIME: 2025-02-25 10:28:38 -- STEP: 15/72 -- GLOBAL_STEP: 1016000 | > loss_disc: 2.7904574871063232 (2.823025703430176) | > loss_disc_real_0: 0.1566922664642334 (0.2247641419370969) | > loss_disc_real_1: 0.312007337808609 (0.23417288362979888) | > loss_disc_real_2: 0.24093016982078552 (0.24664024313290914) | > loss_disc_real_3: 0.2218688279390335 (0.2451537271340688) | > loss_disc_real_4: 0.23421192169189453 (0.24609961012999218) | > loss_disc_real_5: 0.23990967869758606 (0.24557175636291503) | > loss_0: 2.7904574871063232 (2.823025703430176) | > grad_norm_0: tensor(15.2108, device='cuda:0') (tensor(18.1725, device='cuda:0')) | > loss_gen: 1.8441487550735474 (1.812241760889689) | > loss_kl: 2.2305612564086914 (2.1749094009399412) | > loss_feat: 2.391368865966797 (2.2696792682011924) | > loss_mel: 18.39969253540039 (17.483641052246092) | > loss_duration: 1.5557904243469238 (1.582501769065857) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.421560287475586 (25.322973251342773) | > grad_norm_1: tensor(54.4615, device='cuda:0') (tensor(266.2762, device='cuda:0')) | > current_lr_0: 0.00019452596144714675 | > current_lr_1: 0.00019452596144714675 | > step_time: 0.4418 (0.4387934366861979) | > loader_time: 0.0073 (0.006792227427164714)  --> TIME: 2025-02-25 10:29:01 -- STEP: 65/72 -- GLOBAL_STEP: 1016050 | > loss_disc: 2.9447450637817383 (2.8422754764556886) | > loss_disc_real_0: 0.11896471679210663 (0.22207230914097567) | > loss_disc_real_1: 0.32673555612564087 (0.23478271250541394) | > loss_disc_real_2: 0.22786732017993927 (0.24448367196779985) | > loss_disc_real_3: 0.23139162361621857 (0.2439953088760376) | > loss_disc_real_4: 0.2509745657444 (0.24341789713272682) | > loss_disc_real_5: 0.24541586637496948 (0.24539659779805403) | > loss_0: 2.9447450637817383 (2.8422754764556886) | > grad_norm_0: tensor(26.2356, device='cuda:0') (tensor(16.1657, device='cuda:0')) | > loss_gen: 1.6933249235153198 (1.7748836315595187) | > loss_kl: 2.436290740966797 (2.2694241597102236) | > loss_feat: 1.8577632904052734 (2.2287161111831666) | > loss_mel: 17.99937629699707 (17.68636493682861) | > loss_duration: 1.683031678199768 (1.615539816709665) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.66978645324707 (25.57492881188026) | > grad_norm_1: tensor(76.6140, device='cuda:0') (tensor(289.9358, device='cuda:0')) | > current_lr_0: 0.00019452596144714675 | > current_lr_1: 0.00019452596144714675 | > step_time: 0.49 (0.4500404761387752) | > loader_time: 0.0116 (0.008384785285362833) --> EVAL PERFORMANCE | > avg_loader_time: 0.008644480100819764 (-0.00010043130794041427) | > avg_step_time: 0.45728913831039214 (+0.0007687051531294897) | > avg_loss_disc: 2.8450496028846417 (+0.03282777356429856) | > avg_loss_disc_real_0: 0.22219186716936004 (+0.003447239562659188) | > avg_loss_disc_real_1: 0.23560001270871767 (+6.928620204127434e-06) | > avg_loss_disc_real_2: 0.24480242355608603 (+0.0031767483328429547) | > avg_loss_disc_real_3: 0.2445326227117592 (+0.003071982885750252) | > avg_loss_disc_real_4: 0.24373121433694597 (+0.0008209788463484913) | > avg_loss_disc_real_5: 0.2454073019850422 (+0.0014401512666487593) | > avg_loss_0: 2.8450496028846417 (+0.03282777356429856) | > avg_grad_norm_0: tensor(15.9334, device='cuda:0') (+tensor(2.6167, device='cuda:0')) | > avg_loss_gen: 1.7716201291957372 (-0.02469754554855985) | > avg_loss_kl: 2.2839596405835216 (-0.0572734567481028) | > avg_loss_feat: 2.204043218787287 (-0.06314806367310055) | > avg_loss_mel: 17.756323142790457 (-0.21425944314876233) | > avg_loss_duration: 1.6206995366324841 (-0.00923862759496119) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.636645787198777 (-0.3686170980963901) | > avg_grad_norm_1: tensor(286.9075, device='cuda:0') (tensor(-5.2816, device='cuda:0'))  > EPOCH: 223/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:29:06)   --> TIME: 2025-02-25 10:29:28 -- STEP: 43/72 -- GLOBAL_STEP: 1016100 | > loss_disc: 2.739140510559082 (2.8047473874202993) | > loss_disc_real_0: 0.21716031432151794 (0.22807871862206347) | > loss_disc_real_1: 0.20729675889015198 (0.23237704121789268) | > loss_disc_real_2: 0.20042143762111664 (0.2365532032279081) | > loss_disc_real_3: 0.22732970118522644 (0.24038438880166343) | > loss_disc_real_4: 0.24809950590133667 (0.24801933072334112) | > loss_disc_real_5: 0.24441568553447723 (0.24860390397005303) | > loss_0: 2.739140510559082 (2.8047473874202993) | > grad_norm_0: tensor(12.6101, device='cuda:0') (tensor(13.9246, device='cuda:0')) | > loss_gen: 1.7209994792938232 (1.8045625908430232) | > loss_kl: 2.1797492504119873 (2.28677184082741) | > loss_feat: 2.4258995056152344 (2.272873601248098) | > loss_mel: 17.204713821411133 (18.01938580357751) | > loss_duration: 1.6048463582992554 (1.601297650226327) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.13620948791504 (25.984891536623934) | > grad_norm_1: tensor(85.6352, device='cuda:0') (tensor(293.1794, device='cuda:0')) | > current_lr_0: 0.00019450164570196585 | > current_lr_1: 0.00019450164570196585 | > step_time: 0.4591 (0.4440678773924362) | > loader_time: 0.0122 (0.007602159367051236) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710626145483739 (+6.614604466397521e-05) | > avg_step_time: 0.4587440087761677 (+0.0014548704657755818) | > avg_loss_disc: 2.8084277972369125 (-0.03662180564772921) | > avg_loss_disc_real_0: 0.2233524590101041 (+0.0011605918407440463) | > avg_loss_disc_real_1: 0.23743186639228336 (+0.0018318536835656984) | > avg_loss_disc_real_2: 0.2368539231763759 (-0.007948500379710122) | > avg_loss_disc_real_3: 0.23904714617930667 (-0.005485476532452543) | > avg_loss_disc_real_4: 0.24558300157667887 (+0.0018517872397328938) | > avg_loss_disc_real_5: 0.2477557314953334 (+0.0023484295102911834) | > avg_loss_0: 2.8084277972369125 (-0.03662180564772921) | > avg_grad_norm_0: tensor(14.1940, device='cuda:0') (tensor(-1.7394, device='cuda:0')) | > avg_loss_gen: 1.7945023976581198 (+0.022882268462382616) | > avg_loss_kl: 2.29883568387636 (+0.014876043292838315) | > avg_loss_feat: 2.247078024165732 (+0.043034805378445284) | > avg_loss_mel: 17.8206773140061 (+0.06435417121564413) | > avg_loss_duration: 1.6227987887154163 (+0.002099252082932157) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.78389210768149 (+0.14724632048271147) | > avg_grad_norm_1: tensor(271.6315, device='cuda:0') (tensor(-15.2760, device='cuda:0'))  > EPOCH: 224/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:29:43)   --> TIME: 2025-02-25 10:29:55 -- STEP: 21/72 -- GLOBAL_STEP: 1016150 | > loss_disc: 2.75608491897583 (2.7875252110617503) | > loss_disc_real_0: 0.1674942523241043 (0.20884157717227936) | > loss_disc_real_1: 0.15915264189243317 (0.2278356310867128) | > loss_disc_real_2: 0.2195538878440857 (0.23611673357940854) | > loss_disc_real_3: 0.2355376034975052 (0.2401874221506573) | > loss_disc_real_4: 0.21366946399211884 (0.2453256675175258) | > loss_disc_real_5: 0.22166091203689575 (0.24863421207382566) | > loss_0: 2.75608491897583 (2.7875252110617503) | > grad_norm_0: tensor(12.0081, device='cuda:0') (tensor(12.2620, device='cuda:0')) | > loss_gen: 1.6917530298233032 (1.7937973567417689) | > loss_kl: 2.110372543334961 (2.2497253645034063) | > loss_feat: 2.5019819736480713 (2.39355662890843) | > loss_mel: 17.388582229614258 (17.82573854355585) | > loss_duration: 1.6105743646621704 (1.5950708446048556) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.303264617919922 (25.85788890293666) | > grad_norm_1: tensor(130.3710, device='cuda:0') (tensor(262.6132, device='cuda:0')) | > current_lr_0: 0.0001944773329962531 | > current_lr_1: 0.0001944773329962531 | > step_time: 0.441 (0.43400321687970844) | > loader_time: 0.0069 (0.006679398672921317)  --> TIME: 2025-02-25 10:30:18 -- STEP: 71/72 -- GLOBAL_STEP: 1016200 | > loss_disc: 2.9966988563537598 (2.7996541782164237) | > loss_disc_real_0: 0.21642562747001648 (0.21494843842278064) | > loss_disc_real_1: 0.24727773666381836 (0.23509256142965504) | > loss_disc_real_2: 0.2584538459777832 (0.23647448105711333) | > loss_disc_real_3: 0.22918152809143066 (0.24020197211017072) | > loss_disc_real_4: 0.23325234651565552 (0.24284892665668273) | > loss_disc_real_5: 0.25729504227638245 (0.24759753405208318) | > loss_0: 2.9966988563537598 (2.7996541782164237) | > grad_norm_0: tensor(8.8751, device='cuda:0') (tensor(10.9684, device='cuda:0')) | > loss_gen: 1.7346264123916626 (1.8148315103960708) | > loss_kl: 2.437582492828369 (2.311682983183526) | > loss_feat: 2.149864912033081 (2.3684182418903834) | > loss_mel: 17.54765510559082 (17.622796475047803) | > loss_duration: 1.6212868690490723 (1.623630100572613) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.49101448059082 (25.741359307732377) | > grad_norm_1: tensor(561.9378, device='cuda:0') (tensor(317.3721, device='cuda:0')) | > current_lr_0: 0.0001944773329962531 | > current_lr_1: 0.0001944773329962531 | > step_time: 0.4712 (0.4534289769723382) | > loader_time: 0.0081 (0.008579509359010508) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579509359010508 (-0.00013111678647323086) | > avg_step_time: 0.4534289769723382 (-0.0053150318038295286) | > avg_loss_disc: 2.7996541782164237 (-0.008773619020488788) | > avg_loss_disc_real_0: 0.21494843842278064 (-0.008404020587323452) | > avg_loss_disc_real_1: 0.23509256142965504 (-0.0023393049626283247) | > avg_loss_disc_real_2: 0.23647448105711333 (-0.00037944211926257965) | > avg_loss_disc_real_3: 0.24020197211017072 (+0.0011548259308640507) | > avg_loss_disc_real_4: 0.24284892665668273 (-0.002734074919996138) | > avg_loss_disc_real_5: 0.24759753405208318 (-0.00015819744325021312) | > avg_loss_0: 2.7996541782164237 (-0.008773619020488788) | > avg_grad_norm_0: tensor(10.9684, device='cuda:0') (tensor(-3.2256, device='cuda:0')) | > avg_loss_gen: 1.8148315103960708 (+0.020329112737951016) | > avg_loss_kl: 2.311682983183526 (+0.012847299307165994) | > avg_loss_feat: 2.3684182418903834 (+0.1213402177246512) | > avg_loss_mel: 17.622796475047803 (-0.197880838958298) | > avg_loss_duration: 1.623630100572613 (+0.0008313118571967593) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.741359307732377 (-0.042532799949110967) | > avg_grad_norm_1: tensor(317.3721, device='cuda:0') (+tensor(45.7407, device='cuda:0'))  > EPOCH: 225/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:30:20)   --> TIME: 2025-02-25 10:30:45 -- STEP: 49/72 -- GLOBAL_STEP: 1016250 | > loss_disc: 2.8626976013183594 (2.805508871467746) | > loss_disc_real_0: 0.16473156213760376 (0.2123896464401362) | > loss_disc_real_1: 0.24258850514888763 (0.23438612478119986) | > loss_disc_real_2: 0.28527870774269104 (0.24111976824244674) | > loss_disc_real_3: 0.2608398497104645 (0.24561458461138666) | > loss_disc_real_4: 0.24536724388599396 (0.24824534265362483) | > loss_disc_real_5: 0.25735583901405334 (0.2470313730288525) | > loss_0: 2.8626976013183594 (2.805508871467746) | > grad_norm_0: tensor(14.7170, device='cuda:0') (tensor(12.9183, device='cuda:0')) | > loss_gen: 1.7471940517425537 (1.8015288309175141) | > loss_kl: 2.1490626335144043 (2.34426136892669) | > loss_feat: 2.1396050453186035 (2.304131247559372) | > loss_mel: 16.698322296142578 (17.70257490508409) | > loss_duration: 1.6936798095703125 (1.6150573102795347) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.42786407470703 (25.76755360194615) | > grad_norm_1: tensor(370.6385, device='cuda:0') (tensor(391.9002, device='cuda:0')) | > current_lr_0: 0.00019445302332962857 | > current_lr_1: 0.00019445302332962857 | > step_time: 0.4395 (0.4511062174427266) | > loader_time: 0.0094 (0.007699543116044025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008725800984342336 (+0.00014629162533182777) | > avg_step_time: 0.46169224927123165 (+0.008263272298893454) | > avg_loss_disc: 2.817902128461381 (+0.01824795024495751) | > avg_loss_disc_real_0: 0.21670686810369222 (+0.001758429680911583) | > avg_loss_disc_real_1: 0.2334201537387472 (-0.0016724076909078456) | > avg_loss_disc_real_2: 0.24274183494943968 (+0.006267353892326355) | > avg_loss_disc_real_3: 0.24376989880078276 (+0.003567926690612039) | > avg_loss_disc_real_4: 0.2465964440308826 (+0.0037475173741998713) | > avg_loss_disc_real_5: 0.24611936509609222 (-0.001478168955990955) | > avg_loss_0: 2.817902128461381 (+0.01824795024495751) | > avg_grad_norm_0: tensor(14.4568, device='cuda:0') (+tensor(3.4885, device='cuda:0')) | > avg_loss_gen: 1.7906987616713619 (-0.02413274872470894) | > avg_loss_kl: 2.325107880041633 (+0.013424896858106994) | > avg_loss_feat: 2.2608691319613388 (-0.10754910992904465) | > avg_loss_mel: 17.636434219252884 (+0.013637744205080793) | > avg_loss_duration: 1.6266014693488537 (+0.0029713687762407126) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.639711487461145 (-0.10164782027123209) | > avg_grad_norm_1: tensor(394.4012, device='cuda:0') (+tensor(77.0291, device='cuda:0'))  > EPOCH: 226/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:30:57)   --> TIME: 2025-02-25 10:31:11 -- STEP: 27/72 -- GLOBAL_STEP: 1016300 | > loss_disc: 2.925704002380371 (2.7997107329191984) | > loss_disc_real_0: 0.23501725494861603 (0.21283732078693532) | > loss_disc_real_1: 0.24652616679668427 (0.2322616590945809) | > loss_disc_real_2: 0.2515372931957245 (0.23913111399721215) | > loss_disc_real_3: 0.24805612862110138 (0.2316447639906848) | > loss_disc_real_4: 0.2442593276500702 (0.24266340059262734) | > loss_disc_real_5: 0.23054538667201996 (0.24466293277563872) | > loss_0: 2.925704002380371 (2.7997107329191984) | > grad_norm_0: tensor(7.5171, device='cuda:0') (tensor(11.1543, device='cuda:0')) | > loss_gen: 1.7038233280181885 (1.7941025937045063) | > loss_kl: 2.3756942749023438 (2.2578348936858004) | > loss_feat: 2.3088157176971436 (2.3597704834408226) | > loss_mel: 18.168235778808594 (17.672995673285588) | > loss_duration: 1.562948226928711 (1.5863809099903814) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.119516372680664 (25.671084227385343) | > grad_norm_1: tensor(89.9242, device='cuda:0') (tensor(264.7288, device='cuda:0')) | > current_lr_0: 0.00019442871670171237 | > current_lr_1: 0.00019442871670171237 | > step_time: 0.4391 (0.4380495106732404) | > loader_time: 0.0098 (0.00715823526735659) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693090626891232 (-3.271035745110347e-05) | > avg_step_time: 0.45612906066464703 (-0.0055631886065846126) | > avg_loss_disc: 2.80736062224482 (-0.010541506216561025) | > avg_loss_disc_real_0: 0.21028629533002074 (-0.0064205727736714735) | > avg_loss_disc_real_1: 0.2331718050048385 (-0.00024834873390869316) | > avg_loss_disc_real_2: 0.240055167129342 (-0.002686667820097688) | > avg_loss_disc_real_3: 0.2371771106837501 (-0.00659278811703265) | > avg_loss_disc_real_4: 0.2410906368577984 (-0.005505807173084187) | > avg_loss_disc_real_5: 0.2441714624284019 (-0.001947902667690321) | > avg_loss_0: 2.80736062224482 (-0.010541506216561025) | > avg_grad_norm_0: tensor(14.7403, device='cuda:0') (+tensor(0.2835, device='cuda:0')) | > avg_loss_gen: 1.78564431297947 (-0.005054448691891977) | > avg_loss_kl: 2.350111766600272 (+0.025003886558639277) | > avg_loss_feat: 2.287436784153253 (+0.02656765219191426) | > avg_loss_mel: 17.64550491118095 (+0.00907069192806631) | > avg_loss_duration: 1.6490515084333823 (+0.022450039084528584) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.71774909865688 (+0.07803761119573593) | > avg_grad_norm_1: tensor(317.2317, device='cuda:0') (tensor(-77.1696, device='cuda:0'))  > EPOCH: 227/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:31:34)   --> TIME: 2025-02-25 10:31:38 -- STEP: 5/72 -- GLOBAL_STEP: 1016350 | > loss_disc: 2.8352742195129395 (2.9116102695465087) | > loss_disc_real_0: 0.12021112442016602 (0.3046184957027435) | > loss_disc_real_1: 0.25866377353668213 (0.2587037980556488) | > loss_disc_real_2: 0.24664808809757233 (0.26699202954769136) | > loss_disc_real_3: 0.2537645101547241 (0.25006556510925293) | > loss_disc_real_4: 0.2503712475299835 (0.248379847407341) | > loss_disc_real_5: 0.2581418454647064 (0.24769812524318696) | > loss_0: 2.8352742195129395 (2.9116102695465087) | > grad_norm_0: tensor(18.0397, device='cuda:0') (tensor(23.3412, device='cuda:0')) | > loss_gen: 1.6947343349456787 (1.830525803565979) | > loss_kl: 1.9096155166625977 (2.1489660263061525) | > loss_feat: 2.274510145187378 (2.2164053440093996) | > loss_mel: 17.924549102783203 (17.90822830200195) | > loss_duration: 1.58184814453125 (1.6212954759597777) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.385257720947266 (25.725421142578124) | > grad_norm_1: tensor(98.0686, device='cuda:0') (tensor(327.6395, device='cuda:0')) | > current_lr_0: 0.00019440441311212466 | > current_lr_1: 0.00019440441311212466 | > step_time: 0.4214 (0.4279210090637207) | > loader_time: 0.0065 (0.0053692817687988285)  --> TIME: 2025-02-25 10:32:01 -- STEP: 55/72 -- GLOBAL_STEP: 1016400 | > loss_disc: 2.80698299407959 (2.835092410174283) | > loss_disc_real_0: 0.2270948588848114 (0.22107996182008224) | > loss_disc_real_1: 0.2168915718793869 (0.23879501223564148) | > loss_disc_real_2: 0.2580398917198181 (0.2423141747713089) | > loss_disc_real_3: 0.27250567078590393 (0.24373658055608924) | > loss_disc_real_4: 0.24941591918468475 (0.24520113359798085) | > loss_disc_real_5: 0.2553275227546692 (0.24699950299479745) | > loss_0: 2.80698299407959 (2.835092410174283) | > grad_norm_0: tensor(3.1821, device='cuda:0') (tensor(13.0001, device='cuda:0')) | > loss_gen: 1.7497458457946777 (1.7863962563601408) | > loss_kl: 2.276242971420288 (2.2388263637369326) | > loss_feat: 2.223452568054199 (2.2791215853257616) | > loss_mel: 17.658998489379883 (17.67743043032559) | > loss_duration: 1.6581213474273682 (1.6137042305686258) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.56656265258789 (25.59547868208452) | > grad_norm_1: tensor(272.0541, device='cuda:0') (tensor(338.2607, device='cuda:0')) | > current_lr_0: 0.00019440441311212466 | > current_lr_1: 0.00019440441311212466 | > step_time: 0.462 (0.4445472153750333) | > loader_time: 0.0093 (0.007838895104148171) --> EVAL PERFORMANCE | > avg_loader_time: 0.008583639709042832 (-0.00010945091784840023) | > avg_step_time: 0.45678990659579427 (+0.0006608459311472359) | > avg_loss_disc: 2.8319187332207045 (+0.02455811097588434) | > avg_loss_disc_real_0: 0.2217224080797652 (+0.01143611274974446) | > avg_loss_disc_real_1: 0.23650724388344188 (+0.0033354388786033806) | > avg_loss_disc_real_2: 0.24195210194923508 (+0.0018969348198930869) | > avg_loss_disc_real_3: 0.24375296982241348 (+0.006575859138663376) | > avg_loss_disc_real_4: 0.24536156717320562 (+0.0042709303154072065) | > avg_loss_disc_real_5: 0.24711039242610125 (+0.0029389299976993466) | > avg_loss_0: 2.8319187332207045 (+0.02455811097588434) | > avg_grad_norm_0: tensor(12.9742, device='cuda:0') (tensor(-1.7661, device='cuda:0')) | > avg_loss_gen: 1.7880435728690993 (+0.0023992598896294126) | > avg_loss_kl: 2.2721137882958 (-0.07799797830447242) | > avg_loss_feat: 2.260548883760479 (-0.026887900392774) | > avg_loss_mel: 17.78931354468977 (+0.14380863350881867) | > avg_loss_duration: 1.6255432787075847 (-0.02350822972579758) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.73556290210133 (+0.017813803444447274) | > avg_grad_norm_1: tensor(339.2987, device='cuda:0') (+tensor(22.0671, device='cuda:0'))  > EPOCH: 228/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:32:11)   --> TIME: 2025-02-25 10:32:28 -- STEP: 33/72 -- GLOBAL_STEP: 1016450 | > loss_disc: 2.8360791206359863 (2.8034579753875732) | > loss_disc_real_0: 0.1444581151008606 (0.21769871878804584) | > loss_disc_real_1: 0.18622079491615295 (0.23573438823223114) | > loss_disc_real_2: 0.2013087272644043 (0.23964839722170975) | > loss_disc_real_3: 0.23096594214439392 (0.24466344127149292) | > loss_disc_real_4: 0.2355935126543045 (0.24406644250407364) | > loss_disc_real_5: 0.23399782180786133 (0.24738369204781271) | > loss_0: 2.8360791206359863 (2.8034579753875732) | > grad_norm_0: tensor(15.1489, device='cuda:0') (tensor(13.6172, device='cuda:0')) | > loss_gen: 1.7154536247253418 (1.8214536652420505) | > loss_kl: 2.1910459995269775 (2.3143669619704736) | > loss_feat: 2.2438931465148926 (2.37390128771464) | > loss_mel: 17.659353256225586 (18.060424226703073) | > loss_duration: 1.5996919870376587 (1.598788062731425) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.40943717956543 (26.16893433079575) | > grad_norm_1: tensor(277.1800, device='cuda:0') (tensor(282.7936, device='cuda:0')) | > current_lr_0: 0.00019438011256048564 | > current_lr_1: 0.00019438011256048564 | > step_time: 0.4369 (0.43753139900438714) | > loader_time: 0.0078 (0.007038737788344874) --> EVAL PERFORMANCE | > avg_loader_time: 0.008603341142896195 (+1.9701433853363398e-05) | > avg_step_time: 0.452396013367344 (-0.004393893228450296) | > avg_loss_disc: 2.806253493671686 (-0.025665239549018448) | > avg_loss_disc_real_0: 0.2190970394393088 (-0.002625368640456399) | > avg_loss_disc_real_1: 0.2359279944023616 (-0.0005792494810802906) | > avg_loss_disc_real_2: 0.23724351781354824 (-0.0047085841356868385) | > avg_loss_disc_real_3: 0.2413035622784789 (-0.0024494075439345786) | > avg_loss_disc_real_4: 0.2448792470173097 (-0.0004823201558959156) | > avg_loss_disc_real_5: 0.24800073051116836 (+0.0008903380850671105) | > avg_loss_0: 2.806253493671686 (-0.025665239549018448) | > avg_grad_norm_0: tensor(13.5811, device='cuda:0') (+tensor(0.6068, device='cuda:0')) | > avg_loss_gen: 1.800507644532432 (+0.012464071663332632) | > avg_loss_kl: 2.327411728845515 (+0.055297940549715374) | > avg_loss_feat: 2.2981208579640993 (+0.03757197420362024) | > avg_loss_mel: 17.891323788065307 (+0.10201024337553832) | > avg_loss_duration: 1.6297993458492654 (+0.004256067141680697) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.947163434095785 (+0.21160053199445628) | > avg_grad_norm_1: tensor(284.4056, device='cuda:0') (tensor(-54.8932, device='cuda:0'))  > EPOCH: 229/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:32:48)   --> TIME: 2025-02-25 10:32:55 -- STEP: 11/72 -- GLOBAL_STEP: 1016500 | > loss_disc: 2.822441816329956 (2.81700814854015) | > loss_disc_real_0: 0.20584534108638763 (0.21271562305363742) | > loss_disc_real_1: 0.23506903648376465 (0.23659379373897205) | > loss_disc_real_2: 0.2632719874382019 (0.25226970694281836) | > loss_disc_real_3: 0.21894404292106628 (0.2440700043331493) | > loss_disc_real_4: 0.2668689787387848 (0.24453741176561875) | > loss_disc_real_5: 0.27669641375541687 (0.2497039328921925) | > loss_0: 2.822441816329956 (2.81700814854015) | > grad_norm_0: tensor(3.6024, device='cuda:0') (tensor(10.6770, device='cuda:0')) | > loss_gen: 1.7377333641052246 (1.8060502139004795) | > loss_kl: 2.3059797286987305 (2.181720018386841) | > loss_feat: 2.114701271057129 (2.3506149487061934) | > loss_mel: 17.379619598388672 (17.647808768532492) | > loss_duration: 1.5880550146102905 (1.5957877744327893) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.126089096069336 (25.581981658935547) | > grad_norm_1: tensor(67.2090, device='cuda:0') (tensor(330.2519, device='cuda:0')) | > current_lr_0: 0.00019435581504641556 | > current_lr_1: 0.00019435581504641556 | > step_time: 0.4699 (0.43135920437899505) | > loader_time: 0.0063 (0.006174737756902521)  --> TIME: 2025-02-25 10:33:19 -- STEP: 61/72 -- GLOBAL_STEP: 1016550 | > loss_disc: 2.8210043907165527 (2.813315063226418) | > loss_disc_real_0: 0.2566661536693573 (0.21286091110745414) | > loss_disc_real_1: 0.19301407039165497 (0.23800107836723328) | > loss_disc_real_2: 0.21153801679611206 (0.2425556497984245) | > loss_disc_real_3: 0.17836418747901917 (0.24030291032595713) | > loss_disc_real_4: 0.23918074369430542 (0.2427039195279606) | > loss_disc_real_5: 0.24701343476772308 (0.24664222632275254) | > loss_0: 2.8210043907165527 (2.813315063226418) | > grad_norm_0: tensor(9.6545, device='cuda:0') (tensor(10.0940, device='cuda:0')) | > loss_gen: 1.6285490989685059 (1.7922208543683662) | > loss_kl: 2.5005860328674316 (2.2990536220738136) | > loss_feat: 2.108444929122925 (2.3163200538666526) | > loss_mel: 18.108837127685547 (17.932888500026007) | > loss_duration: 1.669951319694519 (1.6371058733736883) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.016368865966797 (25.977589060048587) | > grad_norm_1: tensor(94.0524, device='cuda:0') (tensor(305.9000, device='cuda:0')) | > current_lr_0: 0.00019435581504641556 | > current_lr_1: 0.00019435581504641556 | > step_time: 0.4606 (0.45481166292409425) | > loader_time: 0.0106 (0.008279343120387344) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738544625295719 (+0.0001352034823995235) | > avg_step_time: 0.46148180961608887 (+0.009085796248744893) | > avg_loss_disc: 2.808672676623707 (+0.0024191829520208152) | > avg_loss_disc_real_0: 0.21252151872490493 (-0.006575520714403871) | > avg_loss_disc_real_1: 0.23849493663915447 (+0.0025669422367928796) | > avg_loss_disc_real_2: 0.2408078239417412 (+0.0035643061281929456) | > avg_loss_disc_real_3: 0.24004789179479571 (-0.0012556704836831911) | > avg_loss_disc_real_4: 0.2418148939458417 (-0.00306435307146799) | > avg_loss_disc_real_5: 0.24553465885175785 (-0.0024660716594105125) | > avg_loss_0: 2.808672676623707 (+0.0024191829520208152) | > avg_grad_norm_0: tensor(10.4228, device='cuda:0') (tensor(-3.1583, device='cuda:0')) | > avg_loss_gen: 1.790816839312164 (-0.009690805220267995) | > avg_loss_kl: 2.308876460706683 (-0.018535268138831995) | > avg_loss_feat: 2.3129571558723994 (+0.0148362979083001) | > avg_loss_mel: 17.930950836396548 (+0.039627048331240644) | > avg_loss_duration: 1.6415981776277782 (+0.011798831778512708) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.985199646211008 (+0.03803621211522312) | > avg_grad_norm_1: tensor(311.8501, device='cuda:0') (+tensor(27.4445, device='cuda:0'))  > EPOCH: 230/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:33:26)   --> TIME: 2025-02-25 10:33:45 -- STEP: 39/72 -- GLOBAL_STEP: 1016600 | > loss_disc: 2.736037015914917 (2.8197557498247194) | > loss_disc_real_0: 0.1485428512096405 (0.2272224770142482) | > loss_disc_real_1: 0.27131447196006775 (0.2323707002095687) | > loss_disc_real_2: 0.22399446368217468 (0.24355880419413248) | > loss_disc_real_3: 0.23422420024871826 (0.24865588278342515) | > loss_disc_real_4: 0.2472354769706726 (0.25203302579048353) | > loss_disc_real_5: 0.25056561827659607 (0.25038776107323474) | > loss_0: 2.736037015914917 (2.8197557498247194) | > grad_norm_0: tensor(12.9914, device='cuda:0') (tensor(14.4010, device='cuda:0')) | > loss_gen: 1.866302490234375 (1.8357821366725824) | > loss_kl: 2.3198044300079346 (2.3428159860464244) | > loss_feat: 2.447628974914551 (2.322267660727867) | > loss_mel: 19.012128829956055 (18.086551910791634) | > loss_duration: 1.618577003479004 (1.6029353661414905) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.264442443847656 (26.190353002303684) | > grad_norm_1: tensor(574.9836, device='cuda:0') (tensor(320.8414, device='cuda:0')) | > current_lr_0: 0.00019433152056953475 | > current_lr_1: 0.00019433152056953475 | > step_time: 0.4929 (0.4468701680501302) | > loader_time: 0.0083 (0.0075416992872189255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008692163816640074 (-4.638080865564456e-05) | > avg_step_time: 0.4605962558531425 (-0.0008855537629463561) | > avg_loss_disc: 2.8129548791428687 (+0.004282202519161782) | > avg_loss_disc_real_0: 0.22343669611383493 (+0.010915177388929997) | > avg_loss_disc_real_1: 0.2350696627522858 (-0.0034252738868686683) | > avg_loss_disc_real_2: 0.2404428548376325 (-0.00036496910410868666) | > avg_loss_disc_real_3: 0.2441019143017245 (+0.004054022506928795) | > avg_loss_disc_real_4: 0.24754914914218473 (+0.005734255196343019) | > avg_loss_disc_real_5: 0.24828667854759054 (+0.0027520196958326915) | > avg_loss_0: 2.8129548791428687 (+0.004282202519161782) | > avg_grad_norm_0: tensor(15.5164, device='cuda:0') (+tensor(5.0936, device='cuda:0')) | > avg_loss_gen: 1.8155264300359806 (+0.02470959072381662) | > avg_loss_kl: 2.3239273151881257 (+0.0150508544814425) | > avg_loss_feat: 2.312326075325549 (-0.0006310805468503489) | > avg_loss_mel: 17.791267395019524 (-0.1396834413770236) | > avg_loss_duration: 1.6247174236136424 (-0.0168807540141358) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.867764459529393 (-0.11743518668161457) | > avg_grad_norm_1: tensor(327.5977, device='cuda:0') (+tensor(15.7476, device='cuda:0'))  > EPOCH: 231/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:34:03)   --> TIME: 2025-02-25 10:34:12 -- STEP: 17/72 -- GLOBAL_STEP: 1016650 | > loss_disc: 2.7733500003814697 (2.8170699932995964) | > loss_disc_real_0: 0.21829524636268616 (0.20714155365439021) | > loss_disc_real_1: 0.22315043210983276 (0.2306373566389084) | > loss_disc_real_2: 0.25624263286590576 (0.24319322933168971) | > loss_disc_real_3: 0.24072568118572235 (0.2384682595729828) | > loss_disc_real_4: 0.24155905842781067 (0.2460729173001121) | > loss_disc_real_5: 0.26908642053604126 (0.24758989057120154) | > loss_0: 2.7733500003814697 (2.8170699932995964) | > grad_norm_0: tensor(5.0392, device='cuda:0') (tensor(14.5367, device='cuda:0')) | > loss_gen: 1.9450626373291016 (1.7965102476232193) | > loss_kl: 2.3497560024261475 (2.287960403105792) | > loss_feat: 2.419253349304199 (2.3672830076778637) | > loss_mel: 18.685022354125977 (18.235119539148666) | > loss_duration: 1.6167261600494385 (1.5900881150189567) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.01582145690918 (26.27696149489459) | > grad_norm_1: tensor(116.6161, device='cuda:0') (tensor(208.5776, device='cuda:0')) | > current_lr_0: 0.00019430722912946354 | > current_lr_1: 0.00019430722912946354 | > step_time: 0.4311 (0.4311639982111314) | > loader_time: 0.0112 (0.006967530531041762)  --> TIME: 2025-02-25 10:34:36 -- STEP: 67/72 -- GLOBAL_STEP: 1016700 | > loss_disc: 2.808626413345337 (2.7997641421076076) | > loss_disc_real_0: 0.1625138521194458 (0.2152906745227415) | > loss_disc_real_1: 0.23192733526229858 (0.23234878836283043) | > loss_disc_real_2: 0.2508814334869385 (0.24042660531713) | > loss_disc_real_3: 0.26220470666885376 (0.2390540377862418) | > loss_disc_real_4: 0.2641260027885437 (0.24436515480724733) | > loss_disc_real_5: 0.2772015333175659 (0.2460392566759195) | > loss_0: 2.808626413345337 (2.7997641421076076) | > grad_norm_0: tensor(14.2177, device='cuda:0') (tensor(13.6062, device='cuda:0')) | > loss_gen: 1.7455304861068726 (1.7988298490865906) | > loss_kl: 2.200472831726074 (2.2467595605707884) | > loss_feat: 1.6524325609207153 (2.322758845428922) | > loss_mel: 15.156706809997559 (17.60909215727849) | > loss_duration: 1.7024109363555908 (1.6182972513028044) | > amp_scaler: 128.0 (128.0) | > loss_1: 22.457555770874023 (25.59573779889007) | > grad_norm_1: tensor(189.0113, device='cuda:0') (tensor(339.5154, device='cuda:0')) | > current_lr_0: 0.00019430722912946354 | > current_lr_1: 0.00019430722912946354 | > step_time: 0.5292 (0.44865867984828667) | > loader_time: 0.0118 (0.008473086712965325) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638674104717416 (-5.348971192265829e-05) | > avg_step_time: 0.4527249369822757 (-0.007871318870866784) | > avg_loss_disc: 2.7994866975596255 (-0.013468181583243144) | > avg_loss_disc_real_0: 0.2150437168672051 (-0.008392979246629823) | > avg_loss_disc_real_1: 0.23212006079479003 (-0.0029496019574957733) | > avg_loss_disc_real_2: 0.24064275748293165 (+0.00019990264529914925) | > avg_loss_disc_real_3: 0.2393380258704575 (-0.004763888431267016) | > avg_loss_disc_real_4: 0.2446371144811872 (-0.0029120346609975423) | > avg_loss_disc_real_5: 0.24672511010102824 (-0.0015615684465623003) | > avg_loss_0: 2.7994866975596255 (-0.013468181583243144) | > avg_grad_norm_0: tensor(13.4030, device='cuda:0') (tensor(-2.1134, device='cuda:0')) | > avg_loss_gen: 1.7986416128319753 (-0.016884817204005254) | > avg_loss_kl: 2.2430406959963523 (-0.08088661919177342) | > avg_loss_feat: 2.3108832315659855 (-0.0014428437595634946) | > avg_loss_mel: 17.590943631991536 (-0.20032376302798838) | > avg_loss_duration: 1.620795641146915 (-0.003921782466727253) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.564304942816076 (-0.3034595167133176) | > avg_grad_norm_1: tensor(340.0863, device='cuda:0') (+tensor(12.4886, device='cuda:0'))  > EPOCH: 232/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:34:40)   --> TIME: 2025-02-25 10:35:02 -- STEP: 45/72 -- GLOBAL_STEP: 1016750 | > loss_disc: 2.945725679397583 (2.8166272746192083) | > loss_disc_real_0: 0.10087284445762634 (0.212954537736045) | > loss_disc_real_1: 0.22163358330726624 (0.235997012257576) | > loss_disc_real_2: 0.2011193037033081 (0.24118374354309505) | > loss_disc_real_3: 0.23606711626052856 (0.2468269232246611) | > loss_disc_real_4: 0.2722189724445343 (0.24612859785556793) | > loss_disc_real_5: 0.2755967974662781 (0.24925460716088613) | > loss_0: 2.945725679397583 (2.8166272746192083) | > grad_norm_0: tensor(29.5430, device='cuda:0') (tensor(13.0530, device='cuda:0')) | > loss_gen: 1.4693176746368408 (1.7987215412987603) | > loss_kl: 2.4046175479888916 (2.3189595222473143) | > loss_feat: 1.9391950368881226 (2.3025171014997694) | > loss_mel: 18.1086368560791 (17.75724071926541) | > loss_duration: 1.6455035209655762 (1.6120897690455118) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.567272186279297 (25.78952891031901) | > grad_norm_1: tensor(298.9997, device='cuda:0') (tensor(309.2866, device='cuda:0')) | > current_lr_0: 0.00019428294072582235 | > current_lr_1: 0.00019428294072582235 | > step_time: 0.4538 (0.43979706764221194) | > loader_time: 0.0089 (0.007517512639363607) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624698074770645 (-1.397602994677137e-05) | > avg_step_time: 0.45351537516419316 (+0.00079043818191743) | > avg_loss_disc: 2.8217661951629207 (+0.022279497603295173) | > avg_loss_disc_real_0: 0.2225845534826668 (+0.007540836615461705) | > avg_loss_disc_real_1: 0.23915412854140913 (+0.007034067746619105) | > avg_loss_disc_real_2: 0.23924207645402829 (-0.0014006810289033667) | > avg_loss_disc_real_3: 0.24086021474549468 (+0.0015221888750371892) | > avg_loss_disc_real_4: 0.24227852049008222 (-0.002358593991104968) | > avg_loss_disc_real_5: 0.24613126042023512 (-0.0005938496807931237) | > avg_loss_0: 2.8217661951629207 (+0.022279497603295173) | > avg_grad_norm_0: tensor(16.0347, device='cuda:0') (+tensor(2.6316, device='cuda:0')) | > avg_loss_gen: 1.7964841412826322 (-0.002157471549343093) | > avg_loss_kl: 2.3391813627431093 (+0.09614066674675703) | > avg_loss_feat: 2.28528547958589 (-0.025597751980095396) | > avg_loss_mel: 17.76062605414593 (+0.16968242215439489) | > avg_loss_duration: 1.6322844632914368 (+0.011488822144521738) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.813861632011307 (+0.24955668919523077) | > avg_grad_norm_1: tensor(296.9348, device='cuda:0') (tensor(-43.1515, device='cuda:0'))  > EPOCH: 233/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:35:17)   --> TIME: 2025-02-25 10:35:29 -- STEP: 23/72 -- GLOBAL_STEP: 1016800 | > loss_disc: 2.7800474166870117 (2.8215103356734565) | > loss_disc_real_0: 0.16306555271148682 (0.23102017265299093) | > loss_disc_real_1: 0.22778216004371643 (0.2288725998090661) | > loss_disc_real_2: 0.21497520804405212 (0.2402673681145129) | > loss_disc_real_3: 0.20747661590576172 (0.2386401204959206) | > loss_disc_real_4: 0.23173758387565613 (0.24019937929899796) | > loss_disc_real_5: 0.25247466564178467 (0.24869334503360416) | > loss_0: 2.7800474166870117 (2.8215103356734565) | > grad_norm_0: tensor(13.5307, device='cuda:0') (tensor(16.0265, device='cuda:0')) | > loss_gen: 1.6605771780014038 (1.7946299107178398) | > loss_kl: 2.1142773628234863 (2.2901499064072324) | > loss_feat: 2.625019073486328 (2.3479075483653853) | > loss_mel: 18.350923538208008 (18.425807704096254) | > loss_duration: 1.6167595386505127 (1.6021080328070598) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.367557525634766 (26.460603133491848) | > grad_norm_1: tensor(515.3152, device='cuda:0') (tensor(298.7917, device='cuda:0')) | > current_lr_0: 0.0001942586553582316 | > current_lr_1: 0.0001942586553582316 | > step_time: 0.433 (0.43380924929743225) | > loader_time: 0.0067 (0.0067053255827530574) --> EVAL PERFORMANCE | > avg_loader_time: 0.008587860725295375 (-3.683734947526954e-05) | > avg_step_time: 0.45784875036964956 (+0.004333375205456402) | > avg_loss_disc: 2.8086083606934884 (-0.013157834469432306) | > avg_loss_disc_real_0: 0.2159559577703476 (-0.0066285957123192185) | > avg_loss_disc_real_1: 0.23016348292290326 (-0.008990645618505877) | > avg_loss_disc_real_2: 0.23820876424581233 (-0.001033312208215953) | > avg_loss_disc_real_3: 0.2380801113978238 (-0.0027801033476708703) | > avg_loss_disc_real_4: 0.24430299707701508 (+0.0020244765869328607) | > avg_loss_disc_real_5: 0.2480908409810402 (+0.0019595805608050854) | > avg_loss_0: 2.8086083606934884 (-0.013157834469432306) | > avg_grad_norm_0: tensor(16.1804, device='cuda:0') (+tensor(0.1458, device='cuda:0')) | > avg_loss_gen: 1.7931339270632032 (-0.0033502142194290485) | > avg_loss_kl: 2.293500101062614 (-0.04568126168049513) | > avg_loss_feat: 2.350430686708907 (+0.06514520712301675) | > avg_loss_mel: 17.89984529790744 (+0.1392192437615094) | > avg_loss_duration: 1.624783114648201 (-0.007501348643235817) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.961693186155507 (+0.1478315541442008) | > avg_grad_norm_1: tensor(303.6395, device='cuda:0') (+tensor(6.7047, device='cuda:0'))  > EPOCH: 234/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:35:54)   --> TIME: 2025-02-25 10:35:56 -- STEP: 1/72 -- GLOBAL_STEP: 1016850 | > loss_disc: 2.821312665939331 (2.821312665939331) | > loss_disc_real_0: 0.23543711006641388 (0.23543711006641388) | > loss_disc_real_1: 0.2747855484485626 (0.2747855484485626) | > loss_disc_real_2: 0.24636872112751007 (0.24636872112751007) | > loss_disc_real_3: 0.2105817198753357 (0.2105817198753357) | > loss_disc_real_4: 0.2331257164478302 (0.2331257164478302) | > loss_disc_real_5: 0.24985575675964355 (0.24985575675964355) | > loss_0: 2.821312665939331 (2.821312665939331) | > grad_norm_0: tensor(7.2573, device='cuda:0') (tensor(7.2573, device='cuda:0')) | > loss_gen: 1.804575800895691 (1.804575800895691) | > loss_kl: 2.2496700286865234 (2.2496700286865234) | > loss_feat: 2.3985025882720947 (2.3985025882720947) | > loss_mel: 18.501110076904297 (18.501110076904297) | > loss_duration: 1.6326711177825928 (1.6326711177825928) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.586528778076172 (26.586528778076172) | > grad_norm_1: tensor(784.9601, device='cuda:0') (tensor(784.9601, device='cuda:0')) | > current_lr_0: 0.00019423437302631182 | > current_lr_1: 0.00019423437302631182 | > step_time: 0.4221 (0.4220855236053467) | > loader_time: 0.0057 (0.0056688785552978516)  --> TIME: 2025-02-25 10:36:19 -- STEP: 51/72 -- GLOBAL_STEP: 1016900 | > loss_disc: 2.8037331104278564 (2.809923854528689) | > loss_disc_real_0: 0.25766685605049133 (0.21079739270841374) | > loss_disc_real_1: 0.23628437519073486 (0.23505082551170797) | > loss_disc_real_2: 0.20276710391044617 (0.2423342857875076) | > loss_disc_real_3: 0.23082980513572693 (0.24022539076851865) | > loss_disc_real_4: 0.22388894855976105 (0.2478502021700728) | > loss_disc_real_5: 0.23267994821071625 (0.24905055586029504) | > loss_0: 2.8037331104278564 (2.809923854528689) | > grad_norm_0: tensor(16.9909, device='cuda:0') (tensor(11.4421, device='cuda:0')) | > loss_gen: 1.7656779289245605 (1.8083707024069393) | > loss_kl: 2.192211627960205 (2.2851528233172846) | > loss_feat: 2.133234977722168 (2.339063971650366) | > loss_mel: 16.354450225830078 (17.658898989359532) | > loss_duration: 1.6523959636688232 (1.6084648066875982) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.097970962524414 (25.699951022279027) | > grad_norm_1: tensor(296.7777, device='cuda:0') (tensor(366.3531, device='cuda:0')) | > current_lr_0: 0.00019423437302631182 | > current_lr_1: 0.00019423437302631182 | > step_time: 0.4549 (0.44124530810935825) | > loader_time: 0.0095 (0.007663170496622722) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555207453982928 (-3.265327131244701e-05) | > avg_step_time: 0.4550042555365764 (-0.002844494833073141) | > avg_loss_disc: 2.8140698923191554 (+0.005461531625666982) | > avg_loss_disc_real_0: 0.22039317444596493 (+0.004437216675617334) | > avg_loss_disc_real_1: 0.23328908459401468 (+0.003125601671111422) | > avg_loss_disc_real_2: 0.24152371937959966 (+0.003314955133787323) | > avg_loss_disc_real_3: 0.24216720671720907 (+0.004087095319385253) | > avg_loss_disc_real_4: 0.24808976306042202 (+0.0037867659834069367) | > avg_loss_disc_real_5: 0.24948688523030613 (+0.0013960442492659264) | > avg_loss_0: 2.8140698923191554 (+0.005461531625666982) | > avg_grad_norm_0: tensor(12.4979, device='cuda:0') (tensor(-3.6825, device='cuda:0')) | > avg_loss_gen: 1.813130131909545 (+0.019996204846341792) | > avg_loss_kl: 2.2963824372896005 (+0.002882336226986304) | > avg_loss_feat: 2.304196450072275 (-0.04623423663663173) | > avg_loss_mel: 17.65379106494742 (-0.24605423296002016) | > avg_loss_duration: 1.6226464409223744 (-0.002136673725826599) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.690146298475668 (-0.2715468876798397) | > avg_grad_norm_1: tensor(368.4248, device='cuda:0') (+tensor(64.7854, device='cuda:0'))  > EPOCH: 235/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:36:31)   --> TIME: 2025-02-25 10:36:46 -- STEP: 29/72 -- GLOBAL_STEP: 1016950 | > loss_disc: 2.8193321228027344 (2.79216690721183) | > loss_disc_real_0: 0.29338914155960083 (0.20618255693337012) | > loss_disc_real_1: 0.2331124246120453 (0.23681312593920478) | > loss_disc_real_2: 0.266546368598938 (0.24027058019720274) | > loss_disc_real_3: 0.2652973234653473 (0.24276051541854596) | > loss_disc_real_4: 0.2573617994785309 (0.24872939185849552) | > loss_disc_real_5: 0.24907436966896057 (0.2494854048408311) | > loss_0: 2.8193321228027344 (2.79216690721183) | > grad_norm_0: tensor(17.7862, device='cuda:0') (tensor(10.3092, device='cuda:0')) | > loss_gen: 1.8675243854522705 (1.8123885187609443) | > loss_kl: 2.2982563972473145 (2.3281406164169307) | > loss_feat: 2.3668110370635986 (2.3574718853522993) | > loss_mel: 18.466651916503906 (18.126803365247) | > loss_duration: 1.5953114032745361 (1.5871793113905805) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.59455680847168 (26.21198345052785) | > grad_norm_1: tensor(79.5974, device='cuda:0') (tensor(261.3857, device='cuda:0')) | > current_lr_0: 0.00019421009372968352 | > current_lr_1: 0.00019421009372968352 | > step_time: 0.4549 (0.4342505520787732) | > loader_time: 0.0079 (0.006889787213555698) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593878275911576 (+3.8670821928647736e-05) | > avg_step_time: 0.45282496197122923 (-0.0021792935653471845) | > avg_loss_disc: 2.7928968584033798 (-0.02117303391577563) | > avg_loss_disc_real_0: 0.21071341675771793 (-0.009679757688246998) | > avg_loss_disc_real_1: 0.23861232966604368 (+0.005323245072028998) | > avg_loss_disc_real_2: 0.23917337643428588 (-0.002350342945313777) | > avg_loss_disc_real_3: 0.2399720014400885 (-0.0021952052771205544) | > avg_loss_disc_real_4: 0.24448826510301777 (-0.0036014979574042483) | > avg_loss_disc_real_5: 0.24656296888707388 (-0.002923916343232247) | > avg_loss_0: 2.7928968584033798 (-0.02117303391577563) | > avg_grad_norm_0: tensor(11.4356, device='cuda:0') (tensor(-1.0623, device='cuda:0')) | > avg_loss_gen: 1.8102103841136878 (-0.0029197477958571305) | > avg_loss_kl: 2.3274465900071903 (+0.031064152717589888) | > avg_loss_feat: 2.355801607521487 (+0.05160515744921179) | > avg_loss_mel: 17.891194625639578 (+0.23740356069215807) | > avg_loss_duration: 1.6218782999146153 (-0.0007681410077591266) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.00653132586412 (+0.31638502738845276) | > avg_grad_norm_1: tensor(297.3816, device='cuda:0') (tensor(-71.0432, device='cuda:0'))  > EPOCH: 236/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:37:08)   --> TIME: 2025-02-25 10:37:13 -- STEP: 7/72 -- GLOBAL_STEP: 1017000 | > loss_disc: 2.8436734676361084 (2.858994109289987) | > loss_disc_real_0: 0.22614885866641998 (0.22957156598567963) | > loss_disc_real_1: 0.1861373782157898 (0.22443492923464095) | > loss_disc_real_2: 0.2530411183834076 (0.25935578771999906) | > loss_disc_real_3: 0.21964122354984283 (0.2546676588909967) | > loss_disc_real_4: 0.24083688855171204 (0.25881256375994) | > loss_disc_real_5: 0.2616901099681854 (0.26418899638312204) | > loss_0: 2.8436734676361084 (2.858994109289987) | > grad_norm_0: tensor(4.9118, device='cuda:0') (tensor(7.3894, device='cuda:0')) | > loss_gen: 1.65687894821167 (1.7974819455827986) | > loss_kl: 2.314490556716919 (2.2808541229793002) | > loss_feat: 2.246966600418091 (2.3637421812329973) | > loss_mel: 17.39654541015625 (17.797735486711776) | > loss_duration: 1.5874006748199463 (1.6038150787353516) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.202281951904297 (25.84362874712263) | > grad_norm_1: tensor(70.9393, device='cuda:0') (tensor(318.0617, device='cuda:0')) | > current_lr_0: 0.0001941858174679673 | > current_lr_1: 0.0001941858174679673 | > step_time: 0.4398 (0.429074832371303) | > loader_time: 0.0069 (0.006845951080322266)  --> TIME: 2025-02-25 10:37:36 -- STEP: 57/72 -- GLOBAL_STEP: 1017050 | > loss_disc: 2.777872085571289 (2.8140899173000404) | > loss_disc_real_0: 0.2921512722969055 (0.21780112111254743) | > loss_disc_real_1: 0.23850437998771667 (0.23305082634875648) | > loss_disc_real_2: 0.2535473108291626 (0.24304524534627012) | > loss_disc_real_3: 0.22411289811134338 (0.23978218619237868) | > loss_disc_real_4: 0.2319614440202713 (0.2456251435112535) | > loss_disc_real_5: 0.2291557937860489 (0.24770673638895938) | > loss_0: 2.777872085571289 (2.8140899173000404) | > grad_norm_0: tensor(16.1558, device='cuda:0') (tensor(11.6122, device='cuda:0')) | > loss_gen: 1.6851688623428345 (1.7966661557816623) | > loss_kl: 2.322903633117676 (2.277332860126831) | > loss_feat: 2.347771406173706 (2.3648749359867036) | > loss_mel: 17.99771499633789 (17.792389250638188) | > loss_duration: 1.6580750942230225 (1.6120342685465228) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.011634826660156 (25.84329725566663) | > grad_norm_1: tensor(276.8534, device='cuda:0') (tensor(260.6509, device='cuda:0')) | > current_lr_0: 0.0001941858174679673 | > current_lr_1: 0.0001941858174679673 | > step_time: 0.4681 (0.4430168344263445) | > loader_time: 0.01 (0.00824947942767227) --> EVAL PERFORMANCE | > avg_loader_time: 0.008847632878263232 (+0.00025375460235165653) | > avg_step_time: 0.4533738351204026 (+0.0005488731491733456) | > avg_loss_disc: 2.811578609574009 (+0.0186817511706292) | > avg_loss_disc_real_0: 0.21715730502152106 (+0.006443888263803127) | > avg_loss_disc_real_1: 0.23293216148732412 (-0.005680168178719552) | > avg_loss_disc_real_2: 0.24289512214526324 (+0.003721745710977359) | > avg_loss_disc_real_3: 0.2396760773910603 (-0.00029592404902820113) | > avg_loss_disc_real_4: 0.24534847429940398 (+0.0008602091963862135) | > avg_loss_disc_real_5: 0.24751695130072848 (+0.0009539824136546027) | > avg_loss_0: 2.811578609574009 (+0.0186817511706292) | > avg_grad_norm_0: tensor(11.5398, device='cuda:0') (+tensor(0.1041, device='cuda:0')) | > avg_loss_gen: 1.7955044088229326 (-0.014705975290755235) | > avg_loss_kl: 2.3092294830671505 (-0.018217106940039862) | > avg_loss_feat: 2.3389317334537774 (-0.016869874067709567) | > avg_loss_mel: 17.8121154274739 (-0.07907919816567954) | > avg_loss_duration: 1.6230740815820828 (+0.001195781667467477) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.8788549933635 (-0.12767633250062005) | > avg_grad_norm_1: tensor(291.7932, device='cuda:0') (tensor(-5.5884, device='cuda:0'))  > EPOCH: 237/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:37:45)   --> TIME: 2025-02-25 10:38:03 -- STEP: 35/72 -- GLOBAL_STEP: 1017100 | > loss_disc: 2.8846869468688965 (2.8079428604670933) | > loss_disc_real_0: 0.32912877202033997 (0.20497521502631053) | > loss_disc_real_1: 0.23778414726257324 (0.23153045432908195) | > loss_disc_real_2: 0.2497483193874359 (0.24366568624973298) | > loss_disc_real_3: 0.26583433151245117 (0.24195645536695207) | > loss_disc_real_4: 0.2581113576889038 (0.24332992264202663) | > loss_disc_real_5: 0.24836409091949463 (0.2483731687068939) | > loss_0: 2.8846869468688965 (2.8079428604670933) | > grad_norm_0: tensor(21.6558, device='cuda:0') (tensor(14.4840, device='cuda:0')) | > loss_gen: 1.9694311618804932 (1.8041016374315535) | > loss_kl: 2.509711503982544 (2.26711688041687) | > loss_feat: 2.0586888790130615 (2.3886305809021002) | > loss_mel: 17.851245880126953 (17.73023839678083) | > loss_duration: 1.6414400339126587 (1.5891958917890276) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.030517578125 (25.779283469063895) | > grad_norm_1: tensor(139.8960, device='cuda:0') (tensor(293.5958, device='cuda:0')) | > current_lr_0: 0.0001941615442407838 | > current_lr_1: 0.0001941615442407838 | > step_time: 0.4486 (0.4362505231584821) | > loader_time: 0.0083 (0.007232597896030971) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614731506562568 (-0.00023290137170066394) | > avg_step_time: 0.45325454523865605 (-0.00011928988174653155) | > avg_loss_disc: 2.8014665388725173 (-0.010112070701491671) | > avg_loss_disc_real_0: 0.2068903926812427 (-0.01026691234027835) | > avg_loss_disc_real_1: 0.23035745511592273 (-0.002574706371401392) | > avg_loss_disc_real_2: 0.24107991767601228 (-0.0018152044692509584) | > avg_loss_disc_real_3: 0.24027504803429187 (+0.0005989706432315556) | > avg_loss_disc_real_4: 0.24281225531873568 (-0.0025362189806683033) | > avg_loss_disc_real_5: 0.2466509069355441 (-0.000866044365184393) | > avg_loss_0: 2.8014665388725173 (-0.010112070701491671) | > avg_grad_norm_0: tensor(15.4809, device='cuda:0') (+tensor(3.9411, device='cuda:0')) | > avg_loss_gen: 1.80033476084051 (+0.004830352017577422) | > avg_loss_kl: 2.2929313149250725 (-0.016298168142077962) | > avg_loss_feat: 2.3575912797954723 (+0.018659546341694977) | > avg_loss_mel: 17.671657857760582 (-0.14045756971331613) | > avg_loss_duration: 1.6180849041737302 (-0.004989177408352585) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.740600102384324 (-0.13825489097917654) | > avg_grad_norm_1: tensor(312.7036, device='cuda:0') (+tensor(20.9104, device='cuda:0'))  > EPOCH: 238/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:38:22)   --> TIME: 2025-02-25 10:38:30 -- STEP: 13/72 -- GLOBAL_STEP: 1017150 | > loss_disc: 2.87497878074646 (2.810348620781532) | > loss_disc_real_0: 0.23649655282497406 (0.20517072081565857) | > loss_disc_real_1: 0.2423163801431656 (0.24005525272626144) | > loss_disc_real_2: 0.21826638281345367 (0.23593537967938644) | > loss_disc_real_3: 0.22754953801631927 (0.2414736174620115) | > loss_disc_real_4: 0.2337656021118164 (0.2453726209126986) | > loss_disc_real_5: 0.23867939412593842 (0.24973639731223768) | > loss_0: 2.87497878074646 (2.810348620781532) | > grad_norm_0: tensor(7.0617, device='cuda:0') (tensor(8.8109, device='cuda:0')) | > loss_gen: 1.7692773342132568 (1.7870931258568397) | > loss_kl: 2.3673620223999023 (2.262398591408363) | > loss_feat: 2.427095651626587 (2.3939578166374793) | > loss_mel: 18.384395599365234 (17.745881740863503) | > loss_duration: 1.5165491104125977 (1.5868044266333947) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.464679718017578 (25.776135224562424) | > grad_norm_1: tensor(229.6010, device='cuda:0') (tensor(303.6490, device='cuda:0')) | > current_lr_0: 0.0001941372740477537 | > current_lr_1: 0.0001941372740477537 | > step_time: 0.4582 (0.43896172596858096) | > loader_time: 0.0073 (0.006571659675011268)  --> TIME: 2025-02-25 10:38:53 -- STEP: 63/72 -- GLOBAL_STEP: 1017200 | > loss_disc: 2.8080692291259766 (2.8041018342214916) | > loss_disc_real_0: 0.16904938220977783 (0.2109912426935302) | > loss_disc_real_1: 0.22637760639190674 (0.23769126690569378) | > loss_disc_real_2: 0.20386041700839996 (0.23802384142837826) | > loss_disc_real_3: 0.2008092999458313 (0.23926850585710435) | > loss_disc_real_4: 0.25479620695114136 (0.24532656206025016) | > loss_disc_real_5: 0.25833651423454285 (0.24826498447902623) | > loss_0: 2.8080692291259766 (2.8041018342214916) | > grad_norm_0: tensor(18.3342, device='cuda:0') (tensor(11.5495, device='cuda:0')) | > loss_gen: 1.6783337593078613 (1.8069403777046809) | > loss_kl: 2.3755125999450684 (2.306973780904497) | > loss_feat: 2.1719977855682373 (2.3765205428713845) | > loss_mel: 18.204206466674805 (17.944208205692348) | > loss_duration: 1.6144485473632812 (1.6157746598834086) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.044498443603516 (26.05041733999101) | > grad_norm_1: tensor(284.6041, device='cuda:0') (tensor(283.4324, device='cuda:0')) | > current_lr_0: 0.0001941372740477537 | > current_lr_1: 0.0001941372740477537 | > step_time: 0.5037 (0.45199318916078596) | > loader_time: 0.0115 (0.008224975495111374) --> EVAL PERFORMANCE | > avg_loader_time: 0.008602474776791855 (-1.2256729770713029e-05) | > avg_step_time: 0.4590099126520291 (+0.005755367413373069) | > avg_loss_disc: 2.8089598266171736 (+0.007493287744656296) | > avg_loss_disc_real_0: 0.21431232684514892 (+0.007421934163906213) | > avg_loss_disc_real_1: 0.23706728345911268 (+0.00670982834318995) | > avg_loss_disc_real_2: 0.23849387487895052 (-0.0025860427970617605) | > avg_loss_disc_real_3: 0.24033296653922176 (+5.791850492989359e-05) | > avg_loss_disc_real_4: 0.24433468177284992 (+0.0015224264541142396) | > avg_loss_disc_real_5: 0.24805885091633864 (+0.0014079439807945515) | > avg_loss_0: 2.8089598266171736 (+0.007493287744656296) | > avg_grad_norm_0: tensor(12.1357, device='cuda:0') (tensor(-3.3452, device='cuda:0')) | > avg_loss_gen: 1.8063131285385348 (+0.0059783676980247336) | > avg_loss_kl: 2.316907172471704 (+0.023975857546631563) | > avg_loss_feat: 2.364149016393742 (+0.006557736598269859) | > avg_loss_mel: 17.98964720712581 (+0.31798934936522727) | > avg_loss_duration: 1.6239417908896865 (+0.0058568867159563) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.10095803166779 (+0.36035792928346666) | > avg_grad_norm_1: tensor(310.4404, device='cuda:0') (tensor(-2.2632, device='cuda:0'))  > EPOCH: 239/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:39:00)   --> TIME: 2025-02-25 10:39:20 -- STEP: 41/72 -- GLOBAL_STEP: 1017250 | > loss_disc: 2.821096658706665 (2.7892144656762845) | > loss_disc_real_0: 0.18263159692287445 (0.20985000635065684) | > loss_disc_real_1: 0.28086644411087036 (0.2289947089625568) | > loss_disc_real_2: 0.21862438321113586 (0.23826166623976172) | > loss_disc_real_3: 0.27753591537475586 (0.23881243278340594) | > loss_disc_real_4: 0.2572515606880188 (0.24544359670906532) | > loss_disc_real_5: 0.2864730954170227 (0.24819512701616053) | > loss_0: 2.821096658706665 (2.7892144656762845) | > grad_norm_0: tensor(6.8346, device='cuda:0') (tensor(9.6325, device='cuda:0')) | > loss_gen: 1.932519555091858 (1.8245111674797245) | > loss_kl: 2.1578965187072754 (2.3201680008958028) | > loss_feat: 2.381376028060913 (2.438423819658233) | > loss_mel: 18.180522918701172 (18.14501753085997) | > loss_duration: 1.6138712167739868 (1.6065090603944732) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.266185760498047 (26.334629384482778) | > grad_norm_1: tensor(370.8639, device='cuda:0') (tensor(262.7432, device='cuda:0')) | > current_lr_0: 0.0001941130068884977 | > current_lr_1: 0.0001941130068884977 | > step_time: 0.4616 (0.438948392868042) | > loader_time: 0.0098 (0.007429849810716583) --> EVAL PERFORMANCE | > avg_loader_time: 0.008658446056742064 (+5.5971279950208994e-05) | > avg_step_time: 0.4544340724676428 (-0.004575840184386337) | > avg_loss_disc: 2.796159055871023 (-0.01280077074615038) | > avg_loss_disc_real_0: 0.20998307558852183 (-0.004329251256627087) | > avg_loss_disc_real_1: 0.23534515101305195 (-0.0017221324460607312) | > avg_loss_disc_real_2: 0.23830270158572936 (-0.0001911732932211585) | > avg_loss_disc_real_3: 0.23881001623583512 (-0.0015229503033866443) | > avg_loss_disc_real_4: 0.2431502747283855 (-0.0011844070444644306) | > avg_loss_disc_real_5: 0.2457291798692354 (-0.0023296710471032434) | > avg_loss_0: 2.796159055871023 (-0.01280077074615038) | > avg_grad_norm_0: tensor(10.8004, device='cuda:0') (tensor(-1.3353, device='cuda:0')) | > avg_loss_gen: 1.8182283438427347 (+0.011915215304199922) | > avg_loss_kl: 2.3496882478955774 (+0.03278107542387332) | > avg_loss_feat: 2.415605215959145 (+0.051456199565402816) | > avg_loss_mel: 18.336363510346757 (+0.3467163032209477) | > avg_loss_duration: 1.6349616017140134 (+0.011019810824326903) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.554846938227264 (+0.4538889065594738) | > avg_grad_norm_1: tensor(307.0878, device='cuda:0') (tensor(-3.3526, device='cuda:0'))  > EPOCH: 240/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:39:37)   --> TIME: 2025-02-25 10:39:47 -- STEP: 19/72 -- GLOBAL_STEP: 1017300 | > loss_disc: 2.7547647953033447 (2.7751207602651498) | > loss_disc_real_0: 0.1431598663330078 (0.19871565778004496) | > loss_disc_real_1: 0.21075429022312164 (0.23284349080763364) | > loss_disc_real_2: 0.18781909346580505 (0.23992667543260673) | > loss_disc_real_3: 0.22775347530841827 (0.23762159206365285) | > loss_disc_real_4: 0.22685743868350983 (0.24281609372088783) | > loss_disc_real_5: 0.24161270260810852 (0.24624883893289065) | > loss_0: 2.7547647953033447 (2.7751207602651498) | > grad_norm_0: tensor(13.8068, device='cuda:0') (tensor(9.7815, device='cuda:0')) | > loss_gen: 1.675001859664917 (1.8170223549792641) | > loss_kl: 2.391969919204712 (2.275182021291633) | > loss_feat: 2.5280423164367676 (2.5295960903167725) | > loss_mel: 18.741193771362305 (18.591903084202816) | > loss_duration: 1.5757694244384766 (1.5865627464495207) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.911975860595703 (26.80026636625591) | > grad_norm_1: tensor(491.1740, device='cuda:0') (tensor(323.6433, device='cuda:0')) | > current_lr_0: 0.00019408874276263664 | > current_lr_1: 0.00019408874276263664 | > step_time: 0.4298 (0.43496451879802506) | > loader_time: 0.006 (0.006806411241230212)  --> TIME: 2025-02-25 10:40:11 -- STEP: 69/72 -- GLOBAL_STEP: 1017350 | > loss_disc: 2.839418888092041 (2.7985113634579424) | > loss_disc_real_0: 0.20469355583190918 (0.2043299504380295) | > loss_disc_real_1: 0.18980801105499268 (0.2337193309828855) | > loss_disc_real_2: 0.20547425746917725 (0.2404483928196672) | > loss_disc_real_3: 0.2337944507598877 (0.23793823606726053) | > loss_disc_real_4: 0.24818910658359528 (0.2446481423533481) | > loss_disc_real_5: 0.25754886865615845 (0.248233867512233) | > loss_0: 2.839418888092041 (2.7985113634579424) | > grad_norm_0: tensor(8.6576, device='cuda:0') (tensor(12.0079, device='cuda:0')) | > loss_gen: 1.7538511753082275 (1.8055446994477424) | > loss_kl: 2.3927066326141357 (2.264426841252092) | > loss_feat: 2.0009891986846924 (2.392533723858819) | > loss_mel: 17.982929229736328 (17.981873650481734) | > loss_duration: 1.7111358642578125 (1.6178388923838518) | > amp_scaler: 256.0 (172.52173913043478) | > loss_1: 25.841611862182617 (26.062217767687812) | > grad_norm_1: tensor(353.4288, device='cuda:0') (tensor(310.6217, device='cuda:0')) | > current_lr_0: 0.00019408874276263664 | > current_lr_1: 0.00019408874276263664 | > step_time: 0.5472 (0.4548897259477256) | > loader_time: 0.0126 (0.008733493694360706) --> EVAL PERFORMANCE | > avg_loader_time: 0.00877680241222113 (+0.00011835635547906634) | > avg_step_time: 0.45636593791800484 (+0.001931865450362058) | > avg_loss_disc: 2.8017595015781027 (+0.0056004457070795155) | > avg_loss_disc_real_0: 0.20475989873980133 (-0.005223176848720507) | > avg_loss_disc_real_1: 0.23408177411052541 (-0.0012633769025265362) | > avg_loss_disc_real_2: 0.2401791745927972 (+0.001876473007067836) | > avg_loss_disc_real_3: 0.23837538315376766 (-0.00043463308206745377) | > avg_loss_disc_real_4: 0.24489183618988788 (+0.0017415614615023944) | > avg_loss_disc_real_5: 0.2487105582381638 (+0.0029813783689283946) | > avg_loss_0: 2.8017595015781027 (+0.0056004457070795155) | > avg_grad_norm_0: tensor(11.8757, device='cuda:0') (+tensor(1.0754, device='cuda:0')) | > avg_loss_gen: 1.8044852270206935 (-0.013743116822041213) | > avg_loss_kl: 2.266311962839583 (-0.08337628505599426) | > avg_loss_feat: 2.385264205261015 (-0.03034101069812989) | > avg_loss_mel: 17.96768114600383 (-0.3686823643429271) | > avg_loss_duration: 1.6179796423710568 (-0.016981959342956543) | > avg_amp_scaler: 174.8732394366197 (+46.87323943661971) | > avg_loss_1: 26.041722176780162 (-0.5131247614471022) | > avg_grad_norm_1: tensor(307.1552, device='cuda:0') (+tensor(0.0674, device='cuda:0'))  > EPOCH: 241/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:40:14)   --> TIME: 2025-02-25 10:40:37 -- STEP: 47/72 -- GLOBAL_STEP: 1017400 | > loss_disc: 2.7707858085632324 (2.7937826400107526) | > loss_disc_real_0: 0.3226192593574524 (0.2143060641085848) | > loss_disc_real_1: 0.22926466166973114 (0.23730198119549042) | > loss_disc_real_2: 0.2501997649669647 (0.23826031862421237) | > loss_disc_real_3: 0.2524697780609131 (0.23786548984811662) | > loss_disc_real_4: 0.24548277258872986 (0.24318069853681198) | > loss_disc_real_5: 0.25621432065963745 (0.2457801715490666) | > loss_0: 2.7707858085632324 (2.7937826400107526) | > grad_norm_0: tensor(18.1434, device='cuda:0') (tensor(11.8435, device='cuda:0')) | > loss_gen: 1.9990655183792114 (1.8144448665862387) | > loss_kl: 2.1502978801727295 (2.2215741644514373) | > loss_feat: 2.3537344932556152 (2.3760656955394337) | > loss_mel: 16.644428253173828 (17.64674292219446) | > loss_duration: 1.6355464458465576 (1.6004213703439591) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.783071517944336 (25.659249204270385) | > grad_norm_1: tensor(145.8781, device='cuda:0') (tensor(306.1982, device='cuda:0')) | > current_lr_0: 0.0001940644816697913 | > current_lr_1: 0.0001940644816697913 | > step_time: 0.4573 (0.4429166519895513) | > loader_time: 0.0088 (0.007859321350746967) --> EVAL PERFORMANCE | > avg_loader_time: 0.00878768907466405 (+1.0886662442919845e-05) | > avg_step_time: 0.4562262649267492 (-0.00013967299125561272) | > avg_loss_disc: 2.791459268247577 (-0.010300233330525543) | > avg_loss_disc_real_0: 0.21130127585689787 (+0.006541377117096542) | > avg_loss_disc_real_1: 0.23469353246856745 (+0.0006117583580420305) | > avg_loss_disc_real_2: 0.23953691061953425 (-0.0006422639732629465) | > avg_loss_disc_real_3: 0.23761917650699615 (-0.0007562066467715067) | > avg_loss_disc_real_4: 0.24297470693856899 (-0.0019171292513188998) | > avg_loss_disc_real_5: 0.2442925429679978 (-0.004418015270165998) | > avg_loss_0: 2.791459268247577 (-0.010300233330525543) | > avg_grad_norm_0: tensor(11.8992, device='cuda:0') (+tensor(0.0234, device='cuda:0')) | > avg_loss_gen: 1.8085105754959752 (+0.004025348475281687) | > avg_loss_kl: 2.2278894236390028 (-0.03842253920058036) | > avg_loss_feat: 2.3709194962407496 (-0.014344709020265523) | > avg_loss_mel: 17.525476912377584 (-0.4422042336262457) | > avg_loss_duration: 1.6180910929827617 (+0.00011145061170481085) | > avg_amp_scaler: 256.0 (+81.12676056338029) | > avg_loss_1: 25.55088763169839 (-0.49083454508177127) | > avg_grad_norm_1: tensor(337.7295, device='cuda:0') (+tensor(30.5743, device='cuda:0'))  > EPOCH: 242/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:40:51)   --> TIME: 2025-02-25 10:41:04 -- STEP: 25/72 -- GLOBAL_STEP: 1017450 | > loss_disc: 2.719245672225952 (2.788860912322998) | > loss_disc_real_0: 0.17007102072238922 (0.205242520570755) | > loss_disc_real_1: 0.17566026747226715 (0.22725799322128296) | > loss_disc_real_2: 0.22061140835285187 (0.23870603501796722) | > loss_disc_real_3: 0.22794070839881897 (0.2409702545404434) | > loss_disc_real_4: 0.2393365502357483 (0.2479822701215744) | > loss_disc_real_5: 0.2516794800758362 (0.2503321099281311) | > loss_0: 2.719245672225952 (2.788860912322998) | > grad_norm_0: tensor(13.8948, device='cuda:0') (tensor(9.8250, device='cuda:0')) | > loss_gen: 1.8908309936523438 (1.7922059059143067) | > loss_kl: 2.3938355445861816 (2.178264832496643) | > loss_feat: 2.4994606971740723 (2.3865261173248293) | > loss_mel: 18.01952362060547 (17.664631652832036) | > loss_duration: 1.6060783863067627 (1.5748471784591676) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.40972900390625 (25.59647590637207) | > grad_norm_1: tensor(405.5194, device='cuda:0') (tensor(273.6544, device='cuda:0')) | > current_lr_0: 0.00019404022360958257 | > current_lr_1: 0.00019404022360958257 | > step_time: 0.4352 (0.4370351505279541) | > loader_time: 0.0077 (0.006744604110717773) --> EVAL PERFORMANCE | > avg_loader_time: 0.008574804789583447 (-0.0002128842850806032) | > avg_step_time: 0.4551769948341477 (-0.0010492700926015286) | > avg_loss_disc: 2.7908705120355313 (-0.0005887562120459044) | > avg_loss_disc_real_0: 0.2090605605446117 (-0.0022407153122861734) | > avg_loss_disc_real_1: 0.2316562166096459 (-0.0030373158589215576) | > avg_loss_disc_real_2: 0.2396661336992828 (+0.00012922307974855274) | > avg_loss_disc_real_3: 0.24005303626329128 (+0.0024338597562951203) | > avg_loss_disc_real_4: 0.2451869591440953 (+0.002212252205526316) | > avg_loss_disc_real_5: 0.2492973848547734 (+0.005004841886775607) | > avg_loss_0: 2.7908705120355313 (-0.0005887562120459044) | > avg_grad_norm_0: tensor(12.5298, device='cuda:0') (+tensor(0.6307, device='cuda:0')) | > avg_loss_gen: 1.8041918479220969 (-0.0043187275738783) | > avg_loss_kl: 2.2781089980837326 (+0.0502195744447298) | > avg_loss_feat: 2.354149747902239 (-0.016769748338510748) | > avg_loss_mel: 17.714550716776245 (+0.18907380439866017) | > avg_loss_duration: 1.6152806164513172 (-0.0028104765314445057) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.766282121900097 (+0.21539449020170665) | > avg_grad_norm_1: tensor(307.6341, device='cuda:0') (tensor(-30.0954, device='cuda:0'))  > EPOCH: 243/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:41:28)   --> TIME: 2025-02-25 10:41:31 -- STEP: 3/72 -- GLOBAL_STEP: 1017500 | > loss_disc: 2.785358428955078 (2.747980753580729) | > loss_disc_real_0: 0.12448962032794952 (0.19202885528405508) | > loss_disc_real_1: 0.2151722013950348 (0.24836819370587668) | > loss_disc_real_2: 0.2288220375776291 (0.22478821376959482) | > loss_disc_real_3: 0.2194758504629135 (0.22430277864138284) | > loss_disc_real_4: 0.22725136578083038 (0.23862573007742563) | > loss_disc_real_5: 0.22562897205352783 (0.23901050289471945) | > loss_0: 2.785358428955078 (2.747980753580729) | > grad_norm_0: tensor(18.5255, device='cuda:0') (tensor(12.1517, device='cuda:0')) | > loss_gen: 1.6359989643096924 (1.7295783360799153) | > loss_kl: 2.274099111557007 (2.173187176386515) | > loss_feat: 2.325171947479248 (2.340325911839803) | > loss_mel: 17.48981285095215 (17.888176600138348) | > loss_duration: 1.6164754629135132 (1.6066851218541462) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.34156036376953 (25.737953821818035) | > grad_norm_1: tensor(194.0122, device='cuda:0') (tensor(502.0913, device='cuda:0')) | > current_lr_0: 0.00019401596858163137 | > current_lr_1: 0.00019401596858163137 | > step_time: 0.432 (0.44476477305094403) | > loader_time: 0.0057 (0.007232189178466797)  --> TIME: 2025-02-25 10:41:54 -- STEP: 53/72 -- GLOBAL_STEP: 1017550 | > loss_disc: 2.960150957107544 (2.8014988179476754) | > loss_disc_real_0: 0.22974909842014313 (0.21339382906005067) | > loss_disc_real_1: 0.18074217438697815 (0.2308545211013758) | > loss_disc_real_2: 0.22028550505638123 (0.24064262014514995) | > loss_disc_real_3: 0.2550038695335388 (0.24006651088876543) | > loss_disc_real_4: 0.24894562363624573 (0.24351976924347427) | > loss_disc_real_5: 0.23305772244930267 (0.24603319139975421) | > loss_0: 2.960150957107544 (2.8014988179476754) | > grad_norm_0: tensor(5.8087, device='cuda:0') (tensor(12.1491, device='cuda:0')) | > loss_gen: 1.8444887399673462 (1.8109195209899038) | > loss_kl: 2.284895896911621 (2.2340498960243074) | > loss_feat: 2.4847614765167236 (2.4240143254118145) | > loss_mel: 18.084243774414062 (17.64681859286326) | > loss_duration: 1.6512784957885742 (1.6025252657116584) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.349666595458984 (25.718327504283977) | > grad_norm_1: tensor(325.2915, device='cuda:0') (tensor(408.3762, device='cuda:0')) | > current_lr_0: 0.00019401596858163137 | > current_lr_1: 0.00019401596858163137 | > step_time: 0.4589 (0.4485854247830949) | > loader_time: 0.0098 (0.008034683623403874) --> EVAL PERFORMANCE | > avg_loader_time: 0.008776228192826394 (+0.0002014234032429469) | > avg_step_time: 0.4582971049026704 (+0.003120110068522697) | > avg_loss_disc: 2.8143932718626212 (+0.023522759827089956) | > avg_loss_disc_real_0: 0.21202647392178922 (+0.002965913377177526) | > avg_loss_disc_real_1: 0.23571470484767162 (+0.004058488238025737) | > avg_loss_disc_real_2: 0.24648046997231496 (+0.006814336273032157) | > avg_loss_disc_real_3: 0.244457988881729 (+0.004404952618437735) | > avg_loss_disc_real_4: 0.2448224580623734 (-0.0003645010817218963) | > avg_loss_disc_real_5: 0.24537129343395503 (-0.003926091420818373) | > avg_loss_0: 2.8143932718626212 (+0.023522759827089956) | > avg_grad_norm_0: tensor(13.3473, device='cuda:0') (+tensor(0.8174, device='cuda:0')) | > avg_loss_gen: 1.8062586650042467 (+0.002066817082149841) | > avg_loss_kl: 2.2576901308247734 (-0.02041886725895914) | > avg_loss_feat: 2.344844811399218 (-0.009304936503020667) | > avg_loss_mel: 17.61402107292497 (-0.10052964385127439) | > avg_loss_duration: 1.616713676654117 (+0.001433060202799874) | > avg_amp_scaler: 237.9718309859155 (-18.02816901408451) | > avg_loss_1: 25.639528220808003 (-0.12675390109209417) | > avg_grad_norm_1: tensor(388.1844, device='cuda:0') (+tensor(80.5503, device='cuda:0'))  > EPOCH: 244/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:42:05)   --> TIME: 2025-02-25 10:42:21 -- STEP: 31/72 -- GLOBAL_STEP: 1017600 | > loss_disc: 2.799323081970215 (2.792378279470628) | > loss_disc_real_0: 0.3267469108104706 (0.21273575098283828) | > loss_disc_real_1: 0.23671352863311768 (0.22845389285395223) | > loss_disc_real_2: 0.26173046231269836 (0.2387208063756266) | > loss_disc_real_3: 0.24936985969543457 (0.2375069154847053) | > loss_disc_real_4: 0.25239092111587524 (0.2418993221175286) | > loss_disc_real_5: 0.22406379878520966 (0.24389267881070417) | > loss_0: 2.799323081970215 (2.792378279470628) | > grad_norm_0: tensor(22.4445, device='cuda:0') (tensor(12.7281, device='cuda:0')) | > loss_gen: 1.979994773864746 (1.8099398766794512) | > loss_kl: 2.271249294281006 (2.279960901506485) | > loss_feat: 2.4475162029266357 (2.442647868587125) | > loss_mel: 18.551376342773438 (18.027562295236898) | > loss_duration: 1.6321159601211548 (1.5964883258265834) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.882253646850586 (26.156599475491433) | > grad_norm_1: tensor(98.9605, device='cuda:0') (tensor(267.9101, device='cuda:0')) | > current_lr_0: 0.00019399171658555865 | > current_lr_1: 0.00019399171658555865 | > step_time: 0.4681 (0.44155158535126715) | > loader_time: 0.0091 (0.0070645501536707725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687116730381065 (-8.911146244532965e-05) | > avg_step_time: 0.45805853857121 (-0.00023856633146041029) | > avg_loss_disc: 2.8066948501157087 (-0.007698421746912487) | > avg_loss_disc_real_0: 0.22004832568722713 (+0.008021851765437904) | > avg_loss_disc_real_1: 0.23151931271586626 (-0.004195392131805364) | > avg_loss_disc_real_2: 0.2391424214755985 (-0.007338048496716459) | > avg_loss_disc_real_3: 0.24335699077223388 (-0.0011009981094951271) | > avg_loss_disc_real_4: 0.24412476554722853 (-0.000697692515144871) | > avg_loss_disc_real_5: 0.24620396150669582 (+0.0008326680727407942) | > avg_loss_0: 2.8066948501157087 (-0.007698421746912487) | > avg_grad_norm_0: tensor(15.4682, device='cuda:0') (+tensor(2.1209, device='cuda:0')) | > avg_loss_gen: 1.8164115657269115 (+0.010152900722664837) | > avg_loss_kl: 2.284572994205313 (+0.02688286338053958) | > avg_loss_feat: 2.3882687259727806 (+0.04342391457356243) | > avg_loss_mel: 17.83539424144046 (+0.22137316851549116) | > avg_loss_duration: 1.6324452044258655 (+0.015731527771748466) | > avg_amp_scaler: 128.0 (-109.97183098591549) | > avg_loss_1: 25.95709287616568 (+0.3175646553576783) | > avg_grad_norm_1: tensor(329.3368, device='cuda:0') (tensor(-58.8477, device='cuda:0'))  > EPOCH: 245/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:42:42)   --> TIME: 2025-02-25 10:42:48 -- STEP: 9/72 -- GLOBAL_STEP: 1017650 | > loss_disc: 2.920545816421509 (2.795192824469672) | > loss_disc_real_0: 0.32656797766685486 (0.21837446755833095) | > loss_disc_real_1: 0.23653648793697357 (0.21858816345532736) | > loss_disc_real_2: 0.24266359210014343 (0.243880624572436) | > loss_disc_real_3: 0.24758432805538177 (0.23696885175175136) | > loss_disc_real_4: 0.25517576932907104 (0.24276844494872624) | > loss_disc_real_5: 0.23083075881004333 (0.24270360834068722) | > loss_0: 2.920545816421509 (2.795192824469672) | > grad_norm_0: tensor(27.2718, device='cuda:0') (tensor(13.6098, device='cuda:0')) | > loss_gen: 1.9113531112670898 (1.8504139582316081) | > loss_kl: 2.030674457550049 (2.198749992582533) | > loss_feat: 2.653219223022461 (2.4656403329637318) | > loss_mel: 18.031084060668945 (17.619187037150066) | > loss_duration: 1.6312882900238037 (1.6505671077304416) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.257619857788086 (25.78455861409505) | > grad_norm_1: tensor(295.0895, device='cuda:0') (tensor(370.3676, device='cuda:0')) | > current_lr_0: 0.00019396746762098544 | > current_lr_1: 0.00019396746762098544 | > step_time: 0.4407 (0.4385897053612603) | > loader_time: 0.0067 (0.006767140494452583)  --> TIME: 2025-02-25 10:43:11 -- STEP: 59/72 -- GLOBAL_STEP: 1017700 | > loss_disc: 2.7132949829101562 (2.8021315841351524) | > loss_disc_real_0: 0.32448428869247437 (0.21128229885283162) | > loss_disc_real_1: 0.21971015632152557 (0.23062108255038827) | > loss_disc_real_2: 0.2221430093050003 (0.24106848997584843) | > loss_disc_real_3: 0.2017766386270523 (0.24295903237189276) | > loss_disc_real_4: 0.24520911276340485 (0.24540123166674274) | > loss_disc_real_5: 0.25949880480766296 (0.24589187411938684) | > loss_0: 2.7132949829101562 (2.8021315841351524) | > grad_norm_0: tensor(24.5442, device='cuda:0') (tensor(12.7584, device='cuda:0')) | > loss_gen: 2.0693106651306152 (1.8127867710792411) | > loss_kl: 2.14117693901062 (2.247092814768775) | > loss_feat: 2.334063768386841 (2.367091550665388) | > loss_mel: 17.587039947509766 (17.729137679277844) | > loss_duration: 1.6532015800476074 (1.6312570470874592) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.784793853759766 (25.7873658648992) | > grad_norm_1: tensor(346.3286, device='cuda:0') (tensor(299.6420, device='cuda:0')) | > current_lr_0: 0.00019396746762098544 | > current_lr_1: 0.00019396746762098544 | > step_time: 0.4921 (0.4487665831032446) | > loader_time: 0.01 (0.008028466822737355) --> EVAL PERFORMANCE | > avg_loader_time: 0.008594079756400955 (-9.303697398010957e-05) | > avg_step_time: 0.4566378794925314 (-0.00142065907867861) | > avg_loss_disc: 2.80068435131664 (-0.006010498799068564) | > avg_loss_disc_real_0: 0.21534495359994993 (-0.004703372087277197) | > avg_loss_disc_real_1: 0.23322252320571685 (+0.0017032104898505873) | > avg_loss_disc_real_2: 0.2399595083904938 (+0.0008170869148952931) | > avg_loss_disc_real_3: 0.24086582828575456 (-0.0024911624864793203) | > avg_loss_disc_real_4: 0.2442244056244971 (+9.964007726856461e-05) | > avg_loss_disc_real_5: 0.24554599779592434 (-0.0006579637107714809) | > avg_loss_0: 2.80068435131664 (-0.006010498799068564) | > avg_grad_norm_0: tensor(12.8642, device='cuda:0') (tensor(-2.6040, device='cuda:0')) | > avg_loss_gen: 1.8156727166243003 (-0.0007388491026112387) | > avg_loss_kl: 2.2574198195632076 (-0.027153174642105427) | > avg_loss_feat: 2.3654762926236015 (-0.02279243334917913) | > avg_loss_mel: 17.774827554192346 (-0.06056668724811587) | > avg_loss_duration: 1.6356779756680342 (+0.003232771242168697) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.849074430868658 (-0.10801844529702365) | > avg_grad_norm_1: tensor(307.1399, device='cuda:0') (tensor(-22.1968, device='cuda:0'))  > EPOCH: 246/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:43:19)   --> TIME: 2025-02-25 10:43:38 -- STEP: 37/72 -- GLOBAL_STEP: 1017750 | > loss_disc: 2.81669282913208 (2.7882399365708634) | > loss_disc_real_0: 0.20377975702285767 (0.19685164657798973) | > loss_disc_real_1: 0.24128130078315735 (0.22920701753448797) | > loss_disc_real_2: 0.2609356939792633 (0.24097446533473763) | > loss_disc_real_3: 0.2585163116455078 (0.24265551365710594) | > loss_disc_real_4: 0.2550201117992401 (0.24614138498499588) | > loss_disc_real_5: 0.24362611770629883 (0.24572629904424823) | > loss_0: 2.81669282913208 (2.7882399365708634) | > grad_norm_0: tensor(15.7874, device='cuda:0') (tensor(10.1829, device='cuda:0')) | > loss_gen: 1.8917529582977295 (1.8140243936229397) | > loss_kl: 2.2630152702331543 (2.2910794148574003) | > loss_feat: 2.213038921356201 (2.439869026880007) | > loss_mel: 17.661556243896484 (17.985925983738255) | > loss_duration: 1.6152217388153076 (1.5955606441240053) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.64458465576172 (26.12645943100388) | > grad_norm_1: tensor(617.5703, device='cuda:0') (tensor(280.0719, device='cuda:0')) | > current_lr_0: 0.0001939432216875328 | > current_lr_1: 0.0001939432216875328 | > step_time: 0.4472 (0.4396490664095492) | > loader_time: 0.0081 (0.007334825154897329) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669050646499848 (+7.497089009889275e-05) | > avg_step_time: 0.4629084157272124 (+0.006270536234681046) | > avg_loss_disc: 2.800200814932165 (-0.0004835363844750695) | > avg_loss_disc_real_0: 0.20726360128798954 (-0.008081352311960388) | > avg_loss_disc_real_1: 0.22988952255584824 (-0.003333000649868606) | > avg_loss_disc_real_2: 0.23945463354318913 (-0.0005048748473046594) | > avg_loss_disc_real_3: 0.23976654339004572 (-0.0010992848957088464) | > avg_loss_disc_real_4: 0.24494775290220555 (+0.0007233472777084526) | > avg_loss_disc_real_5: 0.24654236729715912 (+0.0009963695012347729) | > avg_loss_0: 2.800200814932165 (-0.0004835363844750695) | > avg_grad_norm_0: tensor(12.4643, device='cuda:0') (tensor(-0.3999, device='cuda:0')) | > avg_loss_gen: 1.8012846221386547 (-0.014388094485645597) | > avg_loss_kl: 2.3187007249241143 (+0.06128090536090669) | > avg_loss_feat: 2.370664282583855 (+0.005187989960253603) | > avg_loss_mel: 17.791678791314784 (+0.01685123712243808) | > avg_loss_duration: 1.6196141326931162 (-0.01606384297491803) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.901942562049545 (+0.052868131180886735) | > avg_grad_norm_1: tensor(289.5649, device='cuda:0') (tensor(-17.5750, device='cuda:0'))  > EPOCH: 247/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:43:57)   --> TIME: 2025-02-25 10:44:05 -- STEP: 15/72 -- GLOBAL_STEP: 1017800 | > loss_disc: 2.813751220703125 (2.794600073496501) | > loss_disc_real_0: 0.21152330935001373 (0.20870057841142017) | > loss_disc_real_1: 0.25426173210144043 (0.2276049037774404) | > loss_disc_real_2: 0.2968215346336365 (0.23839217821757) | > loss_disc_real_3: 0.2725769579410553 (0.24328451852003732) | > loss_disc_real_4: 0.27584025263786316 (0.24506732523441316) | > loss_disc_real_5: 0.2698332369327545 (0.2512912501891454) | > loss_0: 2.813751220703125 (2.794600073496501) | > grad_norm_0: tensor(4.9832, device='cuda:0') (tensor(11.0183, device='cuda:0')) | > loss_gen: 1.913027286529541 (1.8155757983525593) | > loss_kl: 2.2444558143615723 (2.2962881088256837) | > loss_feat: 2.3103649616241455 (2.4932707945505777) | > loss_mel: 17.36958122253418 (18.153810246785483) | > loss_duration: 1.5561484098434448 (1.572577174504598) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.393577575683594 (26.331522496541343) | > grad_norm_1: tensor(229.7464, device='cuda:0') (tensor(256.8354, device='cuda:0')) | > current_lr_0: 0.00019391897878482186 | > current_lr_1: 0.00019391897878482186 | > step_time: 0.4318 (0.43744171460469566) | > loader_time: 0.0063 (0.006499719619750976)  --> TIME: 2025-02-25 10:44:29 -- STEP: 65/72 -- GLOBAL_STEP: 1017850 | > loss_disc: 2.810316801071167 (2.7977827769059402) | > loss_disc_real_0: 0.19463883340358734 (0.2132144040786303) | > loss_disc_real_1: 0.2086988091468811 (0.2304158105300023) | > loss_disc_real_2: 0.20437252521514893 (0.23948322121913618) | > loss_disc_real_3: 0.2324824035167694 (0.2394480872612733) | > loss_disc_real_4: 0.24584956467151642 (0.2442036564533527) | > loss_disc_real_5: 0.2571203112602234 (0.2472876094854795) | > loss_0: 2.810316801071167 (2.7977827769059402) | > grad_norm_0: tensor(6.4052, device='cuda:0') (tensor(12.2313, device='cuda:0')) | > loss_gen: 1.9480071067810059 (1.8054408091765184) | > loss_kl: 2.3336195945739746 (2.3051213741302483) | > loss_feat: 2.382939338684082 (2.3936623316544754) | > loss_mel: 17.715688705444336 (17.82636404771071) | > loss_duration: 1.6657179594039917 (1.6166571983924278) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.04597282409668 (25.947245729886568) | > grad_norm_1: tensor(348.2278, device='cuda:0') (tensor(280.2708, device='cuda:0')) | > current_lr_0: 0.00019391897878482186 | > current_lr_1: 0.00019391897878482186 | > step_time: 0.5355 (0.4523579193995549) | > loader_time: 0.0118 (0.008362161196195162) --> EVAL PERFORMANCE | > avg_loader_time: 0.008639872913629236 (-2.9177732870611442e-05) | > avg_step_time: 0.4586907071127018 (-0.004217708614510596) | > avg_loss_disc: 2.798706890831531 (-0.001493924100634203) | > avg_loss_disc_real_0: 0.2134897849509414 (+0.006226183662951856) | > avg_loss_disc_real_1: 0.2325550997760934 (+0.0026655772202451578) | > avg_loss_disc_real_2: 0.24076879717094798 (+0.001314163627758852) | > avg_loss_disc_real_3: 0.2402543242968304 (+0.0004877809067846961) | > avg_loss_disc_real_4: 0.24379868070844193 (-0.0011490721937636172) | > avg_loss_disc_real_5: 0.24726812751360341 (+0.0007257602164442989) | > avg_loss_0: 2.798706890831531 (-0.001493924100634203) | > avg_grad_norm_0: tensor(12.2501, device='cuda:0') (tensor(-0.2142, device='cuda:0')) | > avg_loss_gen: 1.806660982924448 (+0.005376360785793288) | > avg_loss_kl: 2.3171786489621007 (-0.0015220759620135915) | > avg_loss_feat: 2.3880017005221945 (+0.017337417938339428) | > avg_loss_mel: 17.883963222235018 (+0.09228443092023397) | > avg_loss_duration: 1.6211480540289005 (+0.0015339213357843384) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.016952648968765 (+0.1150100869192201) | > avg_grad_norm_1: tensor(302.3003, device='cuda:0') (+tensor(12.7353, device='cuda:0'))  > EPOCH: 248/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:44:34)   --> TIME: 2025-02-25 10:44:56 -- STEP: 43/72 -- GLOBAL_STEP: 1017900 | > loss_disc: 2.7437031269073486 (2.8044272411701296) | > loss_disc_real_0: 0.18102188408374786 (0.22417346855928733) | > loss_disc_real_1: 0.19874681532382965 (0.2303453264541404) | > loss_disc_real_2: 0.23874272406101227 (0.2408200225164724) | > loss_disc_real_3: 0.2623109519481659 (0.2406037085971167) | > loss_disc_real_4: 0.23603396117687225 (0.24832838219265604) | > loss_disc_real_5: 0.2369081825017929 (0.2471996833418691) | > loss_0: 2.7437031269073486 (2.8044272411701296) | > grad_norm_0: tensor(9.5908, device='cuda:0') (tensor(14.2347, device='cuda:0')) | > loss_gen: 1.7623029947280884 (1.812208508336267) | > loss_kl: 2.3580496311187744 (2.2649787747582724) | > loss_feat: 2.4641735553741455 (2.356297878331916) | > loss_mel: 18.3320255279541 (17.857828894326847) | > loss_duration: 1.6340279579162598 (1.6094742015350696) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.550579071044922 (25.900788329368414) | > grad_norm_1: tensor(399.0244, device='cuda:0') (tensor(292.0501, device='cuda:0')) | > current_lr_0: 0.00019389473891247375 | > current_lr_1: 0.00019389473891247375 | > step_time: 0.4535 (0.4416192409604095) | > loader_time: 0.0122 (0.007638227107912995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008727150903621191 (+8.727798999195492e-05) | > avg_step_time: 0.4581108496222698 (-0.0005798574904320453) | > avg_loss_disc: 2.800466722166035 (+0.0017598313345041916) | > avg_loss_disc_real_0: 0.22221909096123468 (+0.008729306010293286) | > avg_loss_disc_real_1: 0.23010033536964739 (-0.002454764406446014) | > avg_loss_disc_real_2: 0.2375938057059973 (-0.003174991464950677) | > avg_loss_disc_real_3: 0.2373869691936063 (-0.002867355103224112) | > avg_loss_disc_real_4: 0.24687806590342184 (+0.003079385194979911) | > avg_loss_disc_real_5: 0.24653842247707744 (-0.0007297050365259738) | > avg_loss_0: 2.800466722166035 (+0.0017598313345041916) | > avg_grad_norm_0: tensor(14.0205, device='cuda:0') (+tensor(1.7704, device='cuda:0')) | > avg_loss_gen: 1.8081137529561218 (+0.0014527700316737668) | > avg_loss_kl: 2.3155465126037598 (-0.0016321363583409187) | > avg_loss_feat: 2.3387480957407356 (-0.049253604781458904) | > avg_loss_mel: 17.93890265343894 (+0.054939431203923306) | > avg_loss_duration: 1.6261240462182274 (+0.004975992189326872) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.02743522214218 (+0.010482573173415943) | > avg_grad_norm_1: tensor(309.2201, device='cuda:0') (+tensor(6.9198, device='cuda:0'))  > EPOCH: 249/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:45:11)   --> TIME: 2025-02-25 10:45:23 -- STEP: 21/72 -- GLOBAL_STEP: 1017950 | > loss_disc: 2.6931538581848145 (2.8141520704541887) | > loss_disc_real_0: 0.20570647716522217 (0.211898155155636) | > loss_disc_real_1: 0.21585573256015778 (0.24248754729827246) | > loss_disc_real_2: 0.22414526343345642 (0.2459807970694133) | > loss_disc_real_3: 0.22541779279708862 (0.24411802561510176) | > loss_disc_real_4: 0.25692251324653625 (0.24612989666916074) | > loss_disc_real_5: 0.2510101795196533 (0.24727113403025128) | > loss_0: 2.6931538581848145 (2.8141520704541887) | > grad_norm_0: tensor(7.7119, device='cuda:0') (tensor(13.3846, device='cuda:0')) | > loss_gen: 1.846922755241394 (1.810376871199835) | > loss_kl: 2.2177867889404297 (2.2829146725790843) | > loss_feat: 2.5076370239257812 (2.3557144573756625) | > loss_mel: 17.402111053466797 (17.847478957403276) | > loss_duration: 1.5933256149291992 (1.5863192194984077) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.56778335571289 (25.882803962344216) | > grad_norm_1: tensor(267.8399, device='cuda:0') (tensor(226.3450, device='cuda:0')) | > current_lr_0: 0.00019387050207010967 | > current_lr_1: 0.00019387050207010967 | > step_time: 0.4451 (0.4380765642438616) | > loader_time: 0.0064 (0.007019656045096261)  --> TIME: 2025-02-25 10:45:47 -- STEP: 71/72 -- GLOBAL_STEP: 1018000 | > loss_disc: 2.915482521057129 (2.796853122576861) | > loss_disc_real_0: 0.16202008724212646 (0.2105777268888245) | > loss_disc_real_1: 0.27171558141708374 (0.2360367228237676) | > loss_disc_real_2: 0.24649813771247864 (0.2431698567430738) | > loss_disc_real_3: 0.2586384415626526 (0.23966141401881902) | > loss_disc_real_4: 0.25843700766563416 (0.24433603798839407) | > loss_disc_real_5: 0.2831164002418518 (0.24640974234527266) | > loss_0: 2.915482521057129 (2.796853122576861) | > grad_norm_0: tensor(18.5020, device='cuda:0') (tensor(13.8765, device='cuda:0')) | > loss_gen: 1.6838874816894531 (1.8150049498383427) | > loss_kl: 2.0726358890533447 (2.268576954452085) | > loss_feat: 1.9302184581756592 (2.394935574330074) | > loss_mel: 16.437898635864258 (17.630458093025315) | > loss_duration: 1.6400940418243408 (1.6267663173272575) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.76473617553711 (25.735742004824356) | > grad_norm_1: tensor(442.2560, device='cuda:0') (tensor(319.6419, device='cuda:0')) | > current_lr_0: 0.00019387050207010967 | > current_lr_1: 0.00019387050207010967 | > step_time: 0.4833 (0.4579068808488443) | > loader_time: 0.008 (0.00874588858913368) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874588858913368 (+1.873768551248836e-05) | > avg_step_time: 0.4579068808488443 (-0.00020396877342548914) | > avg_loss_disc: 2.796853122576861 (-0.0036135995891739547) | > avg_loss_disc_real_0: 0.2105777268888245 (-0.011641364072410176) | > avg_loss_disc_real_1: 0.2360367228237676 (+0.005936387454120201) | > avg_loss_disc_real_2: 0.2431698567430738 (+0.005576051037076479) | > avg_loss_disc_real_3: 0.23966141401881902 (+0.002274444825212718) | > avg_loss_disc_real_4: 0.24433603798839407 (-0.002542027915027778) | > avg_loss_disc_real_5: 0.24640974234527266 (-0.00012868013180478144) | > avg_loss_0: 2.796853122576861 (-0.0036135995891739547) | > avg_grad_norm_0: tensor(13.8765, device='cuda:0') (tensor(-0.1440, device='cuda:0')) | > avg_loss_gen: 1.8150049498383427 (+0.006891196882220951) | > avg_loss_kl: 2.268576954452085 (-0.04696955815167492) | > avg_loss_feat: 2.394935574330074 (+0.05618747858933837) | > avg_loss_mel: 17.630458093025315 (-0.30844456041362633) | > avg_loss_duration: 1.6267663173272575 (+0.0006422711090301458) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.735742004824356 (-0.2916932173178246) | > avg_grad_norm_1: tensor(319.6419, device='cuda:0') (+tensor(10.4218, device='cuda:0'))  > EPOCH: 250/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:45:49)   --> TIME: 2025-02-25 10:46:13 -- STEP: 49/72 -- GLOBAL_STEP: 1018050 | > loss_disc: 2.809964418411255 (2.7954398855871085) | > loss_disc_real_0: 0.1439947783946991 (0.20894222980251118) | > loss_disc_real_1: 0.2318648248910904 (0.23163684107819382) | > loss_disc_real_2: 0.2407362014055252 (0.24439485462344424) | > loss_disc_real_3: 0.24791398644447327 (0.24146195485883828) | > loss_disc_real_4: 0.22707007825374603 (0.2464692756837728) | > loss_disc_real_5: 0.2520887553691864 (0.2512423067676777) | > loss_0: 2.809964418411255 (2.7954398855871085) | > grad_norm_0: tensor(17.2974, device='cuda:0') (tensor(13.6225, device='cuda:0')) | > loss_gen: 1.789933204650879 (1.828131347286458) | > loss_kl: 2.125013589859009 (2.2340750548304342) | > loss_feat: 2.3877620697021484 (2.4288996336411444) | > loss_mel: 16.743432998657227 (17.6434262334084) | > loss_duration: 1.6568365097045898 (1.6034064633505685) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.702980041503906 (25.737938900383153) | > grad_norm_1: tensor(469.0483, device='cuda:0') (tensor(321.6087, device='cuda:0')) | > current_lr_0: 0.0001938462682573509 | > current_lr_1: 0.0001938462682573509 | > step_time: 0.4424 (0.4432119252730389) | > loader_time: 0.0094 (0.007775326164401308) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653852301584164 (-9.203628754951565e-05) | > avg_step_time: 0.4553486730011416 (-0.002558207847702665) | > avg_loss_disc: 2.8044343632711493 (+0.007581240694288116) | > avg_loss_disc_real_0: 0.20870158884307027 (-0.0018761380457542332) | > avg_loss_disc_real_1: 0.23267310752835071 (-0.003363615295416872) | > avg_loss_disc_real_2: 0.24323747779282046 (+6.762104974666894e-05) | > avg_loss_disc_real_3: 0.2424765747617668 (+0.00281516074294777) | > avg_loss_disc_real_4: 0.2463633959562006 (+0.0020273579678065368) | > avg_loss_disc_real_5: 0.25073290794668057 (+0.004323165601407913) | > avg_loss_0: 2.8044343632711493 (+0.007581240694288116) | > avg_grad_norm_0: tensor(12.9242, device='cuda:0') (tensor(-0.9523, device='cuda:0')) | > avg_loss_gen: 1.812724254500698 (-0.0022806953376446337) | > avg_loss_kl: 2.252670479492402 (-0.015906474959682892) | > avg_loss_feat: 2.408434184504226 (+0.013498610174151882) | > avg_loss_mel: 17.667806598502146 (+0.03734850547683166) | > avg_loss_duration: 1.6187403000576395 (-0.008026017269618047) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.7603759765625 (+0.024633971738143856) | > avg_grad_norm_1: tensor(329.8126, device='cuda:0') (+tensor(10.1706, device='cuda:0'))  > EPOCH: 251/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:46:26)   --> TIME: 2025-02-25 10:46:40 -- STEP: 27/72 -- GLOBAL_STEP: 1018100 | > loss_disc: 2.8956918716430664 (2.7897025037694863) | > loss_disc_real_0: 0.1813262403011322 (0.21659743040800095) | > loss_disc_real_1: 0.25018227100372314 (0.2276830126841863) | > loss_disc_real_2: 0.26239582896232605 (0.2374196002880732) | > loss_disc_real_3: 0.2709905803203583 (0.24190436524373513) | > loss_disc_real_4: 0.2737487852573395 (0.2424585007958942) | > loss_disc_real_5: 0.25277769565582275 (0.24322887206519092) | > loss_0: 2.8956918716430664 (2.7897025037694863) | > grad_norm_0: tensor(8.7960, device='cuda:0') (tensor(14.0657, device='cuda:0')) | > loss_gen: 1.8988282680511475 (1.8079694421203047) | > loss_kl: 2.3302667140960693 (2.2197823921839404) | > loss_feat: 2.2202773094177246 (2.430606479997988) | > loss_mel: 16.724977493286133 (17.309910738909686) | > loss_duration: 1.5835766792297363 (1.5786874426735773) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.757925033569336 (25.34695667690701) | > grad_norm_1: tensor(193.3596, device='cuda:0') (tensor(320.1096, device='cuda:0')) | > current_lr_0: 0.00019382203747381871 | > current_lr_1: 0.00019382203747381871 | > step_time: 0.4384 (0.4404457763389305) | > loader_time: 0.0086 (0.0069654252794053816) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618720820252324 (-3.5131481331839715e-05) | > avg_step_time: 0.4569045154141708 (+0.001555842413029196) | > avg_loss_disc: 2.801607796843622 (-0.0028265664275273217) | > avg_loss_disc_real_0: 0.21557099351160963 (+0.006869404668539358) | > avg_loss_disc_real_1: 0.2303720194269234 (-0.0023010881014273177) | > avg_loss_disc_real_2: 0.24059591851603818 (-0.0026415592767822793) | > avg_loss_disc_real_3: 0.24282731829394757 (+0.00035074353218078613) | > avg_loss_disc_real_4: 0.24480734009977798 (-0.001556055856422628) | > avg_loss_disc_real_5: 0.24526152686333993 (-0.005471381083340637) | > avg_loss_0: 2.801607796843622 (-0.0028265664275273217) | > avg_grad_norm_0: tensor(13.3887, device='cuda:0') (+tensor(0.4645, device='cuda:0')) | > avg_loss_gen: 1.8059748797349526 (-0.006749374765745442) | > avg_loss_kl: 2.258501242583907 (+0.005830763091505098) | > avg_loss_feat: 2.394253762675003 (-0.014180421829222745) | > avg_loss_mel: 17.666325636313005 (-0.0014809621891416214) | > avg_loss_duration: 1.6187217907166818 (-1.85093409577064e-05) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.74377734224561 (-0.016598634316888905) | > avg_grad_norm_1: tensor(319.5006, device='cuda:0') (tensor(-10.3120, device='cuda:0'))  > EPOCH: 252/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:47:03)   --> TIME: 2025-02-25 10:47:07 -- STEP: 5/72 -- GLOBAL_STEP: 1018150 | > loss_disc: 2.768620491027832 (2.758290004730225) | > loss_disc_real_0: 0.14976966381072998 (0.19178707003593445) | > loss_disc_real_1: 0.1850736290216446 (0.21762464344501495) | > loss_disc_real_2: 0.22362223267555237 (0.24514293074607849) | > loss_disc_real_3: 0.2644272744655609 (0.244052854180336) | > loss_disc_real_4: 0.25399407744407654 (0.24000285267829896) | > loss_disc_real_5: 0.25535035133361816 (0.2502837508916855) | > loss_0: 2.768620491027832 (2.758290004730225) | > grad_norm_0: tensor(17.1729, device='cuda:0') (tensor(8.0507, device='cuda:0')) | > loss_gen: 1.8091665506362915 (1.8430314779281616) | > loss_kl: 2.4034643173217773 (2.359813690185547) | > loss_feat: 2.1978752613067627 (2.4039042949676515) | > loss_mel: 17.25000762939453 (17.77186737060547) | > loss_duration: 1.5745620727539062 (1.639134120941162) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.235074996948242 (26.017750549316407) | > grad_norm_1: tensor(325.5438, device='cuda:0') (tensor(325.9267, device='cuda:0')) | > current_lr_0: 0.0001937978097191345 | > current_lr_1: 0.0001937978097191345 | > step_time: 0.4263 (0.42678303718566896) | > loader_time: 0.0061 (0.00657505989074707)  --> TIME: 2025-02-25 10:47:30 -- STEP: 55/72 -- GLOBAL_STEP: 1018200 | > loss_disc: 2.9118709564208984 (2.8034777858040556) | > loss_disc_real_0: 0.23164165019989014 (0.21296903816136448) | > loss_disc_real_1: 0.2519885003566742 (0.23910920105197214) | > loss_disc_real_2: 0.252625972032547 (0.2423461206934669) | > loss_disc_real_3: 0.22815512120723724 (0.24344914636828682) | > loss_disc_real_4: 0.2564120292663574 (0.2427553653717041) | > loss_disc_real_5: 0.2572326362133026 (0.24550253369591452) | > loss_0: 2.9118709564208984 (2.8034777858040556) | > grad_norm_0: tensor(11.6302, device='cuda:0') (tensor(10.8717, device='cuda:0')) | > loss_gen: 1.9283051490783691 (1.8343975218859585) | > loss_kl: 2.383342981338501 (2.3441270741549407) | > loss_feat: 1.9963961839675903 (2.437569843639027) | > loss_mel: 16.565710067749023 (17.76077537536621) | > loss_duration: 1.660524606704712 (1.6132514303380794) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.534278869628906 (25.99012100913308) | > grad_norm_1: tensor(150.5703, device='cuda:0') (tensor(266.6216, device='cuda:0')) | > current_lr_0: 0.0001937978097191345 | > current_lr_1: 0.0001937978097191345 | > step_time: 0.4654 (0.44474756500937723) | > loader_time: 0.0094 (0.008124498887495562) --> EVAL PERFORMANCE | > avg_loader_time: 0.008802071423597742 (+0.00018335060334541756) | > avg_step_time: 0.4562836062740272 (-0.0006209091401436218) | > avg_loss_disc: 2.807984301741695 (+0.006376504898073065) | > avg_loss_disc_real_0: 0.21345610098100046 (-0.002114892530609175) | > avg_loss_disc_real_1: 0.23754847448476604 (+0.007176455057842646) | > avg_loss_disc_real_2: 0.24317863100851086 (+0.0025827124924726885) | > avg_loss_disc_real_3: 0.24265118782788936 (-0.00017613046605821636) | > avg_loss_disc_real_4: 0.24418767216339918 (-0.0006196679363787982) | > avg_loss_disc_real_5: 0.24633939165464588 (+0.0010778647913059491) | > avg_loss_0: 2.807984301741695 (+0.006376504898073065) | > avg_grad_norm_0: tensor(11.1472, device='cuda:0') (tensor(-2.2415, device='cuda:0')) | > avg_loss_gen: 1.8284210974061992 (+0.02244621767124655) | > avg_loss_kl: 2.359187535836663 (+0.10068629325275591) | > avg_loss_feat: 2.4153177469549045 (+0.021063984279901415) | > avg_loss_mel: 17.864255233549734 (+0.19792959723672965) | > avg_loss_duration: 1.6235652305710484 (+0.004843439854366638) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.090746651233083 (+0.3469693089874717) | > avg_grad_norm_1: tensor(275.7132, device='cuda:0') (tensor(-43.7874, device='cuda:0'))  > EPOCH: 253/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:47:40)   --> TIME: 2025-02-25 10:47:57 -- STEP: 33/72 -- GLOBAL_STEP: 1018250 | > loss_disc: 2.7972617149353027 (2.832630244168368) | > loss_disc_real_0: 0.15186890959739685 (0.22649925247286307) | > loss_disc_real_1: 0.19494958221912384 (0.2289438432816303) | > loss_disc_real_2: 0.23396487534046173 (0.2445471544157375) | > loss_disc_real_3: 0.2490616887807846 (0.2487760384877523) | > loss_disc_real_4: 0.23208387196063995 (0.24794157346089682) | > loss_disc_real_5: 0.24774785339832306 (0.24817370510462558) | > loss_0: 2.7972617149353027 (2.832630244168368) | > grad_norm_0: tensor(18.7999, device='cuda:0') (tensor(17.6443, device='cuda:0')) | > loss_gen: 1.7457014322280884 (1.7991018981644602) | > loss_kl: 2.0343780517578125 (2.1820730223800204) | > loss_feat: 2.2859396934509277 (2.3338337378068403) | > loss_mel: 17.21404266357422 (17.727890650431316) | > loss_duration: 1.610885739326477 (1.5836189660159024) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.890947341918945 (25.626518076116387) | > grad_norm_1: tensor(348.6467, device='cuda:0') (tensor(298.4356, device='cuda:0')) | > current_lr_0: 0.0001937735849929196 | > current_lr_1: 0.0001937735849929196 | > step_time: 0.4518 (0.43862847125891485) | > loader_time: 0.0078 (0.007252548680160985) --> EVAL PERFORMANCE | > avg_loader_time: 0.008723007121556242 (-7.906430204149967e-05) | > avg_step_time: 0.4567338211435667 (+0.0004502148695395003) | > avg_loss_disc: 2.8107265317943733 (+0.002742230052678263) | > avg_loss_disc_real_0: 0.22187052383809022 (+0.008414422857089765) | > avg_loss_disc_real_1: 0.23276604628059225 (-0.004782428204173794) | > avg_loss_disc_real_2: 0.2412999885182985 (-0.0018786424902123566) | > avg_loss_disc_real_3: 0.2412861552036984 (-0.001365032624190965) | > avg_loss_disc_real_4: 0.24428776284338724 (+0.00010009067998806298) | > avg_loss_disc_real_5: 0.24626933499960832 (-7.005665503756475e-05) | > avg_loss_0: 2.8107265317943733 (+0.002742230052678263) | > avg_grad_norm_0: tensor(14.4881, device='cuda:0') (+tensor(3.3409, device='cuda:0')) | > avg_loss_gen: 1.8102654890275338 (-0.018155608378665367) | > avg_loss_kl: 2.225375408857642 (-0.1338121269790209) | > avg_loss_feat: 2.351912223117452 (-0.06340552383745246) | > avg_loss_mel: 17.598548163830397 (-0.2657070697193369) | > avg_loss_duration: 1.6181250256551822 (-0.005440204915866165) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.604226179525885 (-0.48652047170719825) | > avg_grad_norm_1: tensor(357.4275, device='cuda:0') (+tensor(81.7143, device='cuda:0'))  > EPOCH: 254/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:48:17)   --> TIME: 2025-02-25 10:48:24 -- STEP: 11/72 -- GLOBAL_STEP: 1018300 | > loss_disc: 2.6935408115386963 (2.8321296951987525) | > loss_disc_real_0: 0.1696574091911316 (0.2055354965004054) | > loss_disc_real_1: 0.18374381959438324 (0.22877333516424353) | > loss_disc_real_2: 0.2206861525774002 (0.247195150364529) | > loss_disc_real_3: 0.22548826038837433 (0.2451134053143588) | > loss_disc_real_4: 0.23041768372058868 (0.250757711854848) | > loss_disc_real_5: 0.23271577060222626 (0.2485575960441069) | > loss_0: 2.6935408115386963 (2.8321296951987525) | > grad_norm_0: tensor(7.7536, device='cuda:0') (tensor(11.0335, device='cuda:0')) | > loss_gen: 1.973441481590271 (1.7790646119551226) | > loss_kl: 2.1369736194610596 (2.2384088689630683) | > loss_feat: 2.5622217655181885 (2.2370081814852627) | > loss_mel: 18.28745460510254 (17.583084973421965) | > loss_duration: 1.5765798091888428 (1.5918320200660012) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.536672592163086 (25.429398623379793) | > grad_norm_1: tensor(303.5620, device='cuda:0') (tensor(243.8851, device='cuda:0')) | > current_lr_0: 0.00019374936329479546 | > current_lr_1: 0.00019374936329479546 | > step_time: 0.4463 (0.4396959218111905) | > loader_time: 0.0057 (0.006278276443481445)  --> TIME: 2025-02-25 10:48:47 -- STEP: 61/72 -- GLOBAL_STEP: 1018350 | > loss_disc: 2.8296046257019043 (2.8068605329169602) | > loss_disc_real_0: 0.1691650152206421 (0.20884561404341556) | > loss_disc_real_1: 0.22516800463199615 (0.24177763269084399) | > loss_disc_real_2: 0.25331005454063416 (0.24131482155596623) | > loss_disc_real_3: 0.21226710081100464 (0.24070821604767784) | > loss_disc_real_4: 0.24539802968502045 (0.24547075468008636) | > loss_disc_real_5: 0.2424674928188324 (0.24626465141773224) | > loss_0: 2.8296046257019043 (2.8068605329169602) | > grad_norm_0: tensor(10.1390, device='cuda:0') (tensor(13.2599, device='cuda:0')) | > loss_gen: 1.8216838836669922 (1.801037561698038) | > loss_kl: 2.1719024181365967 (2.315900673631763) | > loss_feat: 2.3883519172668457 (2.3489492936212497) | > loss_mel: 17.100433349609375 (17.633521033115077) | > loss_duration: 1.6751093864440918 (1.6175479361268341) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.157480239868164 (25.716956404388927) | > grad_norm_1: tensor(600.5632, device='cuda:0') (tensor(305.1758, device='cuda:0')) | > current_lr_0: 0.00019374936329479546 | > current_lr_1: 0.00019374936329479546 | > step_time: 0.469 (0.4487961901993048) | > loader_time: 0.0106 (0.008065825603047356) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542655219494457 (-0.0001803519020617851) | > avg_step_time: 0.45693635604750943 (+0.00020253490394273044) | > avg_loss_disc: 2.807854991563609 (-0.0028715402307644133) | > avg_loss_disc_real_0: 0.2119807297704925 (-0.009889794067597713) | > avg_loss_disc_real_1: 0.23943620783762193 (+0.00667016155702968) | > avg_loss_disc_real_2: 0.24212411591704464 (+0.0008241273987461273) | > avg_loss_disc_real_3: 0.24178063428737748 (+0.0004944790836790836) | > avg_loss_disc_real_4: 0.24637766476248352 (+0.0020899019190962764) | > avg_loss_disc_real_5: 0.2476492471678156 (+0.0013799121682072923) | > avg_loss_0: 2.807854991563609 (-0.0028715402307644133) | > avg_grad_norm_0: tensor(13.2694, device='cuda:0') (tensor(-1.2187, device='cuda:0')) | > avg_loss_gen: 1.8066546446840528 (-0.0036108443434810233) | > avg_loss_kl: 2.319522182706377 (+0.09414677384873471) | > avg_loss_feat: 2.3593307968596333 (+0.007418573742181245) | > avg_loss_mel: 17.71986964051153 (+0.12132147668113191) | > avg_loss_duration: 1.6242696113989388 (+0.0061445857437565365) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.829646795568333 (+0.22542061604244878) | > avg_grad_norm_1: tensor(317.9304, device='cuda:0') (tensor(-39.4971, device='cuda:0'))  > EPOCH: 255/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:48:54)   --> TIME: 2025-02-25 10:49:14 -- STEP: 39/72 -- GLOBAL_STEP: 1018400 | > loss_disc: 2.785043478012085 (2.8034382416651797) | > loss_disc_real_0: 0.22811050713062286 (0.22278036196262407) | > loss_disc_real_1: 0.20226475596427917 (0.2277756195801955) | > loss_disc_real_2: 0.20899932086467743 (0.24668670770449516) | > loss_disc_real_3: 0.2237488031387329 (0.24216209886929926) | > loss_disc_real_4: 0.24087688326835632 (0.2458020100990931) | > loss_disc_real_5: 0.24964095652103424 (0.24856230081656042) | > loss_0: 2.785043478012085 (2.8034382416651797) | > grad_norm_0: tensor(10.7467, device='cuda:0') (tensor(14.8562, device='cuda:0')) | > loss_gen: 1.7360072135925293 (1.831309355222262) | > loss_kl: 2.2753970623016357 (2.1641379380837464) | > loss_feat: 2.5202009677886963 (2.4552966020046134) | > loss_mel: 18.458772659301758 (17.84914598709498) | > loss_duration: 1.608518123626709 (1.5904730344429991) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.59889793395996 (25.89036281292255) | > grad_norm_1: tensor(595.5637, device='cuda:0') (tensor(423.2494, device='cuda:0')) | > current_lr_0: 0.0001937251446243836 | > current_lr_1: 0.0001937251446243836 | > step_time: 0.4563 (0.4405419337443816) | > loader_time: 0.008 (0.007249220823630309) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086153292320144 (+7.267401251994342e-05) | > avg_step_time: 0.4566464524873545 (-0.0002899035601549027) | > avg_loss_disc: 2.8150387314003957 (+0.007183739836786884) | > avg_loss_disc_real_0: 0.22381505527546708 (+0.011834325504974569) | > avg_loss_disc_real_1: 0.23076018691062927 (-0.008676020926992656) | > avg_loss_disc_real_2: 0.2438579174414487 (+0.001733801524404055) | > avg_loss_disc_real_3: 0.24316722364492818 (+0.001386589357550705) | > avg_loss_disc_real_4: 0.24609299046053013 (-0.0002846743019533915) | > avg_loss_disc_real_5: 0.2481759211546938 (+0.0005266739868781956) | > avg_loss_0: 2.8150387314003957 (+0.007183739836786884) | > avg_grad_norm_0: tensor(14.1960, device='cuda:0') (+tensor(0.9266, device='cuda:0')) | > avg_loss_gen: 1.8089270709266125 (+0.002272426242559744) | > avg_loss_kl: 2.2205275078894378 (-0.09899467481693902) | > avg_loss_feat: 2.3674395655242484 (+0.008108768664615074) | > avg_loss_mel: 17.75810128870144 (+0.03823164818991032) | > avg_loss_duration: 1.6142046988849905 (-0.010064912513948254) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.76920006980359 (-0.06044672576474497) | > avg_grad_norm_1: tensor(376.8976, device='cuda:0') (+tensor(58.9672, device='cuda:0'))  > EPOCH: 256/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:49:31)   --> TIME: 2025-02-25 10:49:41 -- STEP: 17/72 -- GLOBAL_STEP: 1018450 | > loss_disc: 2.8364789485931396 (2.7522099719328037) | > loss_disc_real_0: 0.2631467580795288 (0.20658958878587275) | > loss_disc_real_1: 0.23682449758052826 (0.23739509810419643) | > loss_disc_real_2: 0.2313016951084137 (0.23207682809408972) | > loss_disc_real_3: 0.2468160092830658 (0.23227506350068486) | > loss_disc_real_4: 0.25052937865257263 (0.2382001017822939) | > loss_disc_real_5: 0.2476455718278885 (0.23928073223899393) | > loss_0: 2.8364789485931396 (2.7522099719328037) | > grad_norm_0: tensor(12.4570, device='cuda:0') (tensor(11.6452, device='cuda:0')) | > loss_gen: 1.799391746520996 (1.8263107608346378) | > loss_kl: 2.3923866748809814 (2.2819731095257927) | > loss_feat: 2.5568857192993164 (2.5413918214685776) | > loss_mel: 18.91865348815918 (17.779210707720587) | > loss_duration: 1.6341042518615723 (1.5789683496250826) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.301420211791992 (26.00785479826086) | > grad_norm_1: tensor(141.0576, device='cuda:0') (tensor(303.4793, device='cuda:0')) | > current_lr_0: 0.00019370092898130556 | > current_lr_1: 0.00019370092898130556 | > step_time: 0.437 (0.4353525638580322) | > loader_time: 0.0087 (0.007056600907269646)  --> TIME: 2025-02-25 10:50:05 -- STEP: 67/72 -- GLOBAL_STEP: 1018500 | > loss_disc: 2.781174898147583 (2.803590632196683) | > loss_disc_real_0: 0.26037830114364624 (0.21663171497743522) | > loss_disc_real_1: 0.24475771188735962 (0.23207205022448923) | > loss_disc_real_2: 0.22872135043144226 (0.24003720728319083) | > loss_disc_real_3: 0.22060483694076538 (0.2405555125937533) | > loss_disc_real_4: 0.2530916929244995 (0.24422377214502933) | > loss_disc_real_5: 0.23534809052944183 (0.2473852794561813) | > loss_0: 2.781174898147583 (2.803590632196683) | > grad_norm_0: tensor(15.9252, device='cuda:0') (tensor(13.3906, device='cuda:0')) | > loss_gen: 1.697908639907837 (1.8117399945187924) | > loss_kl: 2.3684241771698 (2.3397020510773157) | > loss_feat: 2.2309489250183105 (2.445387096547369) | > loss_mel: 16.704065322875977 (17.894110124502607) | > loss_duration: 1.6913176774978638 (1.6208121865542966) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.692665100097656 (26.111751499460706) | > grad_norm_1: tensor(169.4540, device='cuda:0') (tensor(315.4249, device='cuda:0')) | > current_lr_0: 0.00019370092898130556 | > current_lr_1: 0.00019370092898130556 | > step_time: 0.5013 (0.4513606882807034) | > loader_time: 0.0124 (0.008477840850602332) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669705458090336 (+5.437622607593583e-05) | > avg_step_time: 0.45555403534795197 (-0.0010924171394025572) | > avg_loss_disc: 2.8042957211883976 (-0.010743010211998083) | > avg_loss_disc_real_0: 0.2145092071781696 (-0.009305848097297481) | > avg_loss_disc_real_1: 0.2309365406842299 (+0.0001763537736006182) | > avg_loss_disc_real_2: 0.2392780493262788 (-0.004579868115169888) | > avg_loss_disc_real_3: 0.23948152006511958 (-0.0036857035798086024) | > avg_loss_disc_real_4: 0.2439052936896472 (-0.00218769677088293) | > avg_loss_disc_real_5: 0.24656584376180676 (-0.0016100773928870438) | > avg_loss_0: 2.8042957211883976 (-0.010743010211998083) | > avg_grad_norm_0: tensor(13.4258, device='cuda:0') (tensor(-0.7702, device='cuda:0')) | > avg_loss_gen: 1.8057790272672412 (-0.00314804365937138) | > avg_loss_kl: 2.33430716353403 (+0.11377965564459203) | > avg_loss_feat: 2.433545297300313 (+0.06610573177606449) | > avg_loss_mel: 17.8723080326134 (+0.11420674391196073) | > avg_loss_duration: 1.6227004595205818 (+0.00849576063559132) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.068640050753743 (+0.29943998095015445) | > avg_grad_norm_1: tensor(324.8621, device='cuda:0') (tensor(-52.0355, device='cuda:0'))  > EPOCH: 257/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:50:08)   --> TIME: 2025-02-25 10:50:31 -- STEP: 45/72 -- GLOBAL_STEP: 1018550 | > loss_disc: 2.787980556488037 (2.8106919129689536) | > loss_disc_real_0: 0.20333972573280334 (0.208600658012761) | > loss_disc_real_1: 0.24409878253936768 (0.23517333070437113) | > loss_disc_real_2: 0.24774813652038574 (0.24087735215822856) | > loss_disc_real_3: 0.22525569796562195 (0.23926854994561936) | > loss_disc_real_4: 0.25210705399513245 (0.2455852703915702) | > loss_disc_real_5: 0.24536240100860596 (0.2472015678882599) | > loss_0: 2.787980556488037 (2.8106919129689536) | > grad_norm_0: tensor(6.9522, device='cuda:0') (tensor(16.2977, device='cuda:0')) | > loss_gen: 1.909713864326477 (1.8073525057898627) | > loss_kl: 2.730968475341797 (2.2369549433390294) | > loss_feat: 2.3865864276885986 (2.4261028952068746) | > loss_mel: 18.7860050201416 (17.78387688530816) | > loss_duration: 1.6488571166992188 (1.5949554125467935) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.46213150024414 (25.849242613050674) | > grad_norm_1: tensor(99.4190, device='cuda:0') (tensor(303.6955, device='cuda:0')) | > current_lr_0: 0.00019367671636518288 | > current_lr_1: 0.00019367671636518288 | > step_time: 0.4631 (0.45267914136250814) | > loader_time: 0.0088 (0.007512272728814019) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580211182715184 (-8.949427537515223e-05) | > avg_step_time: 0.4625914600533499 (+0.007037424705397921) | > avg_loss_disc: 2.8046463207459786 (+0.0003505995575809884) | > avg_loss_disc_real_0: 0.21190357491583892 (-0.0026056322623306816) | > avg_loss_disc_real_1: 0.2326875196796068 (+0.0017509789953769062) | > avg_loss_disc_real_2: 0.23846074707911047 (-0.0008173022471683333) | > avg_loss_disc_real_3: 0.2384742221362154 (-0.0010072979289041661) | > avg_loss_disc_real_4: 0.24397377543885942 (+6.848174921222916e-05) | > avg_loss_disc_real_5: 0.24611025104220485 (-0.0004555927196019105) | > avg_loss_0: 2.8046463207459786 (+0.0003505995575809884) | > avg_grad_norm_0: tensor(15.6195, device='cuda:0') (+tensor(2.1936, device='cuda:0')) | > avg_loss_gen: 1.8020625315921406 (-0.003716495675100573) | > avg_loss_kl: 2.2908213642281545 (-0.04348579930587526) | > avg_loss_feat: 2.372979736663925 (-0.06056556063638796) | > avg_loss_mel: 17.97900245559047 (+0.10669442297707121) | > avg_loss_duration: 1.617329693176377 (-0.005370766344204858) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.06219568386884 (-0.006444366884903019) | > avg_grad_norm_1: tensor(307.1960, device='cuda:0') (tensor(-17.6662, device='cuda:0'))  > EPOCH: 258/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:50:46)   --> TIME: 2025-02-25 10:50:58 -- STEP: 23/72 -- GLOBAL_STEP: 1018600 | > loss_disc: 2.816824436187744 (2.794055223464966) | > loss_disc_real_0: 0.2629738748073578 (0.2161008683235749) | > loss_disc_real_1: 0.2532300353050232 (0.23417395547680234) | > loss_disc_real_2: 0.2628685534000397 (0.2374708652496338) | > loss_disc_real_3: 0.24503087997436523 (0.241817906498909) | > loss_disc_real_4: 0.268564373254776 (0.24433429733566617) | > loss_disc_real_5: 0.27009549736976624 (0.24769159697968027) | > loss_0: 2.816824436187744 (2.794055223464966) | > grad_norm_0: tensor(14.3040, device='cuda:0') (tensor(14.5730, device='cuda:0')) | > loss_gen: 1.946628212928772 (1.8300823906193608) | > loss_kl: 2.1361687183380127 (2.1921082113100137) | > loss_feat: 2.317037343978882 (2.4576909023782476) | > loss_mel: 16.98177146911621 (17.842853794927183) | > loss_duration: 1.6114561557769775 (1.5840953743976096) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.99306297302246 (25.906830912050992) | > grad_norm_1: tensor(195.2505, device='cuda:0') (tensor(250.2798, device='cuda:0')) | > current_lr_0: 0.0001936525067756372 | > current_lr_1: 0.0001936525067756372 | > step_time: 0.4399 (0.4375206076580545) | > loader_time: 0.0066 (0.0069696903228759766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625362960385605 (+4.5151777670421334e-05) | > avg_step_time: 0.45594176775972606 (-0.006649692293623832) | > avg_loss_disc: 2.791193552420173 (-0.013452768325805664) | > avg_loss_disc_real_0: 0.2111412588979157 (-0.0007623160179232036) | > avg_loss_disc_real_1: 0.23301388847995813 (+0.0003263688003513343) | > avg_loss_disc_real_2: 0.23864802697174986 (+0.0001872798926393915) | > avg_loss_disc_real_3: 0.2394633987839793 (+0.0009891766477638786) | > avg_loss_disc_real_4: 0.24341617440673666 (-0.0005576010321227676) | > avg_loss_disc_real_5: 0.24628097595463336 (+0.00017072491242850485) | > avg_loss_0: 2.791193552420173 (-0.013452768325805664) | > avg_grad_norm_0: tensor(13.0272, device='cuda:0') (tensor(-2.5923, device='cuda:0')) | > avg_loss_gen: 1.8102542229101692 (+0.00819169131802866) | > avg_loss_kl: 2.2099861376722085 (-0.080835226555946) | > avg_loss_feat: 2.414690355180015 (+0.04171061851609004) | > avg_loss_mel: 17.5978876704901 (-0.38111478510037244) | > avg_loss_duration: 1.6180823201864538 (+0.000752627010076834) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.65090090791944 (-0.41129477594940056) | > avg_grad_norm_1: tensor(359.1086, device='cuda:0') (+tensor(51.9127, device='cuda:0'))  > EPOCH: 259/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:51:23)   --> TIME: 2025-02-25 10:51:25 -- STEP: 1/72 -- GLOBAL_STEP: 1018650 | > loss_disc: 2.7454912662506104 (2.7454912662506104) | > loss_disc_real_0: 0.24895232915878296 (0.24895232915878296) | > loss_disc_real_1: 0.23258322477340698 (0.23258322477340698) | > loss_disc_real_2: 0.26047971844673157 (0.26047971844673157) | > loss_disc_real_3: 0.24259760975837708 (0.24259760975837708) | > loss_disc_real_4: 0.25740301609039307 (0.25740301609039307) | > loss_disc_real_5: 0.25664350390434265 (0.25664350390434265) | > loss_0: 2.7454912662506104 (2.7454912662506104) | > grad_norm_0: tensor(8.9382, device='cuda:0') (tensor(8.9382, device='cuda:0')) | > loss_gen: 1.9474743604660034 (1.9474743604660034) | > loss_kl: 2.2393624782562256 (2.2393624782562256) | > loss_feat: 2.622123956680298 (2.622123956680298) | > loss_mel: 17.1702880859375 (17.1702880859375) | > loss_duration: 1.6288390159606934 (1.6288390159606934) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.60808753967285 (25.60808753967285) | > grad_norm_1: tensor(261.9059, device='cuda:0') (tensor(261.9059, device='cuda:0')) | > current_lr_0: 0.00019362830021229025 | > current_lr_1: 0.00019362830021229025 | > step_time: 0.4277 (0.42766880989074707) | > loader_time: 0.0062 (0.006242513656616211)  --> TIME: 2025-02-25 10:51:48 -- STEP: 51/72 -- GLOBAL_STEP: 1018700 | > loss_disc: 2.7070584297180176 (2.784180902967266) | > loss_disc_real_0: 0.21710450947284698 (0.21596813216513278) | > loss_disc_real_1: 0.21056236326694489 (0.23213252951117122) | > loss_disc_real_2: 0.21351365745067596 (0.23702630254567839) | > loss_disc_real_3: 0.24142064154148102 (0.2339153459259108) | > loss_disc_real_4: 0.21784831583499908 (0.24407667973462274) | > loss_disc_real_5: 0.22465860843658447 (0.24709117208041398) | > loss_0: 2.7070584297180176 (2.784180902967266) | > grad_norm_0: tensor(9.2964, device='cuda:0') (tensor(15.2732, device='cuda:0')) | > loss_gen: 1.7481542825698853 (1.8162369587842155) | > loss_kl: 2.2162179946899414 (2.3183062286937943) | > loss_feat: 2.5882747173309326 (2.478709973540961) | > loss_mel: 17.307456970214844 (17.566756416769586) | > loss_duration: 1.6591300964355469 (1.607406310006684) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.51923370361328 (25.787415784948013) | > grad_norm_1: tensor(236.5868, device='cuda:0') (tensor(282.3938, device='cuda:0')) | > current_lr_0: 0.00019362830021229025 | > current_lr_1: 0.00019362830021229025 | > step_time: 0.4432 (0.4453304141175513) | > loader_time: 0.0089 (0.007925935820037245) --> EVAL PERFORMANCE | > avg_loader_time: 0.008726741226626117 (+0.00010137826624051147) | > avg_step_time: 0.4556953940593021 (-0.000246373700423963) | > avg_loss_disc: 2.795739942873028 (+0.004546390452854876) | > avg_loss_disc_real_0: 0.2221543597923198 (+0.011013100894404076) | > avg_loss_disc_real_1: 0.2332249384530833 (+0.00021104997312518248) | > avg_loss_disc_real_2: 0.24003478527908595 (+0.001386758307336089) | > avg_loss_disc_real_3: 0.23587410319858873 (-0.0035892955853905617) | > avg_loss_disc_real_4: 0.24552408532357553 (+0.002107910916838873) | > avg_loss_disc_real_5: 0.2475393064005274 (+0.0012583304458940459) | > avg_loss_0: 2.795739942873028 (+0.004546390452854876) | > avg_grad_norm_0: tensor(16.2734, device='cuda:0') (+tensor(3.2462, device='cuda:0')) | > avg_loss_gen: 1.825063833048646 (+0.014809610138476703) | > avg_loss_kl: 2.328764661936692 (+0.11877852426448365) | > avg_loss_feat: 2.459961041598253 (+0.04527068641823817) | > avg_loss_mel: 17.696831797210262 (+0.09894412672016273) | > avg_loss_duration: 1.6202605811642927 (+0.0021782609778389084) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.930881876341054 (+0.279980968421615) | > avg_grad_norm_1: tensor(309.5233, device='cuda:0') (tensor(-49.5853, device='cuda:0'))  > EPOCH: 260/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:52:00)   --> TIME: 2025-02-25 10:52:15 -- STEP: 29/72 -- GLOBAL_STEP: 1018750 | > loss_disc: 2.7774641513824463 (2.784177615724761) | > loss_disc_real_0: 0.12483349442481995 (0.19848735733278866) | > loss_disc_real_1: 0.18431922793388367 (0.22823060997601213) | > loss_disc_real_2: 0.23100008070468903 (0.23888039897228108) | > loss_disc_real_3: 0.25061020255088806 (0.23594754471861082) | > loss_disc_real_4: 0.23853179812431335 (0.24155102618809404) | > loss_disc_real_5: 0.22942358255386353 (0.24428328203743901) | > loss_0: 2.7774641513824463 (2.784177615724761) | > grad_norm_0: tensor(28.6944, device='cuda:0') (tensor(15.9463, device='cuda:0')) | > loss_gen: 1.734041452407837 (1.7989032474057427) | > loss_kl: 2.39958119392395 (2.275572727466452) | > loss_feat: 2.4918160438537598 (2.5209662503209604) | > loss_mel: 17.80398941040039 (17.919759947678134) | > loss_duration: 1.5898154973983765 (1.5842061536065464) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.019243240356445 (26.099408051063275) | > grad_norm_1: tensor(460.6196, device='cuda:0') (tensor(394.7140, device='cuda:0')) | > current_lr_0: 0.0001936040966747637 | > current_lr_1: 0.0001936040966747637 | > step_time: 0.4431 (0.437222382117962) | > loader_time: 0.0076 (0.00672624028962234) --> EVAL PERFORMANCE | > avg_loader_time: 0.008540979573424435 (-0.00018576165320168155) | > avg_step_time: 0.45619504216691137 (+0.000499648107609274) | > avg_loss_disc: 2.7883663949832105 (-0.007373547889817367) | > avg_loss_disc_real_0: 0.20736992285704947 (-0.014784436935270318) | > avg_loss_disc_real_1: 0.22997272770169755 (-0.0032522107513857645) | > avg_loss_disc_real_2: 0.23888068236935306 (-0.0011541029097328903) | > avg_loss_disc_real_3: 0.2391812497041595 (+0.0033071465055707627) | > avg_loss_disc_real_4: 0.2433604209775656 (-0.0021636643460099247) | > avg_loss_disc_real_5: 0.24540447873968474 (-0.002134827660842664) | > avg_loss_0: 2.7883663949832105 (-0.007373547889817367) | > avg_grad_norm_0: tensor(14.2920, device='cuda:0') (tensor(-1.9814, device='cuda:0')) | > avg_loss_gen: 1.8104053735733032 (-0.014658459475342722) | > avg_loss_kl: 2.2796651595075375 (-0.04909950242915473) | > avg_loss_feat: 2.456341256558055 (-0.00361978504019822) | > avg_loss_mel: 17.622642355905445 (-0.0741894413048172) | > avg_loss_duration: 1.6165535147760954 (-0.0037070663881972887) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.78560748570402 (-0.1452743906370344) | > avg_grad_norm_1: tensor(425.7735, device='cuda:0') (+tensor(116.2502, device='cuda:0'))  > EPOCH: 261/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:52:37)   --> TIME: 2025-02-25 10:52:42 -- STEP: 7/72 -- GLOBAL_STEP: 1018800 | > loss_disc: 2.8782687187194824 (2.836538689477103) | > loss_disc_real_0: 0.24602308869361877 (0.205823329942567) | > loss_disc_real_1: 0.2695816457271576 (0.235032890524183) | > loss_disc_real_2: 0.21981468796730042 (0.23289177673203604) | > loss_disc_real_3: 0.20957431197166443 (0.2437540399176734) | > loss_disc_real_4: 0.23701372742652893 (0.24720809289387294) | > loss_disc_real_5: 0.25687137246131897 (0.24816459204469407) | > loss_0: 2.8782687187194824 (2.836538689477103) | > grad_norm_0: tensor(10.8410, device='cuda:0') (tensor(8.1900, device='cuda:0')) | > loss_gen: 1.774317979812622 (1.8160631486347742) | > loss_kl: 2.360227108001709 (2.1932647057941983) | > loss_feat: 2.3831374645233154 (2.3294905935015) | > loss_mel: 17.916519165039062 (17.811064311436244) | > loss_duration: 1.5977346897125244 (1.5896231957844325) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.031936645507812 (25.739506040300643) | > grad_norm_1: tensor(402.7469, device='cuda:0') (tensor(213.4132, device='cuda:0')) | > current_lr_0: 0.00019357989616267934 | > current_lr_1: 0.00019357989616267934 | > step_time: 0.4318 (0.4274673802512033) | > loader_time: 0.0054 (0.006944860730852399)  --> TIME: 2025-02-25 10:53:05 -- STEP: 57/72 -- GLOBAL_STEP: 1018850 | > loss_disc: 2.86781644821167 (2.8062370325389656) | > loss_disc_real_0: 0.22873495519161224 (0.20578977949263758) | > loss_disc_real_1: 0.22437335550785065 (0.234451789866414) | > loss_disc_real_2: 0.25771084427833557 (0.24287236939396778) | > loss_disc_real_3: 0.24574899673461914 (0.23775197905406617) | > loss_disc_real_4: 0.24942384660243988 (0.2455164957464787) | > loss_disc_real_5: 0.2815893292427063 (0.2483808024410616) | > loss_0: 2.86781644821167 (2.8062370325389656) | > grad_norm_0: tensor(7.1789, device='cuda:0') (tensor(12.0782, device='cuda:0')) | > loss_gen: 1.8032416105270386 (1.812799428638659) | > loss_kl: 2.3744328022003174 (2.333043773969015) | > loss_feat: 2.182793617248535 (2.4303077354765783) | > loss_mel: 17.52008628845215 (18.055354804323432) | > loss_duration: 1.6675360202789307 (1.6186250155432182) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.5480899810791 (26.250130703574733) | > grad_norm_1: tensor(425.0364, device='cuda:0') (tensor(295.8089, device='cuda:0')) | > current_lr_0: 0.00019357989616267934 | > current_lr_1: 0.00019357989616267934 | > step_time: 0.476 (0.44531235778540895) | > loader_time: 0.0097 (0.007998453943352949) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625158121888066 (+8.417854846363114e-05) | > avg_step_time: 0.45659655920216735 (+0.0004015170352559827) | > avg_loss_disc: 2.8038073123340865 (+0.015440917350876049) | > avg_loss_disc_real_0: 0.20923982161871144 (+0.0018698987616619644) | > avg_loss_disc_real_1: 0.2364712869617301 (+0.006498559260032538) | > avg_loss_disc_real_2: 0.24198334884475653 (+0.0031026664754034705) | > avg_loss_disc_real_3: 0.23746491800731337 (-0.001716331696846124) | > avg_loss_disc_real_4: 0.24340469946323987 (+4.4278485674259516e-05) | > avg_loss_disc_real_5: 0.24649859335221036 (+0.0010941146125256207) | > avg_loss_0: 2.8038073123340865 (+0.015440917350876049) | > avg_grad_norm_0: tensor(12.3471, device='cuda:0') (tensor(-1.9449, device='cuda:0')) | > avg_loss_gen: 1.819572500779595 (+0.00916712720629187) | > avg_loss_kl: 2.3139908464861594 (+0.03432568697862193) | > avg_loss_feat: 2.4358825414953107 (-0.020458715062744215) | > avg_loss_mel: 18.055702343792984 (+0.43305998788753897) | > avg_loss_duration: 1.627015175953717 (+0.010461661177621595) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.252163416902786 (+0.4665559311987657) | > avg_grad_norm_1: tensor(319.3675, device='cuda:0') (tensor(-106.4060, device='cuda:0'))  > EPOCH: 262/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:53:14)   --> TIME: 2025-02-25 10:53:32 -- STEP: 35/72 -- GLOBAL_STEP: 1018900 | > loss_disc: 2.9400527477264404 (2.7912655762263707) | > loss_disc_real_0: 0.31159108877182007 (0.20288803492273602) | > loss_disc_real_1: 0.2915497124195099 (0.23448797166347504) | > loss_disc_real_2: 0.2727223038673401 (0.2378446774823325) | > loss_disc_real_3: 0.2768794596195221 (0.23503399533884867) | > loss_disc_real_4: 0.2513215243816376 (0.24094368857996804) | > loss_disc_real_5: 0.26259177923202515 (0.2474832249539239) | > loss_0: 2.9400527477264404 (2.7912655762263707) | > grad_norm_0: tensor(15.4214, device='cuda:0') (tensor(11.6057, device='cuda:0')) | > loss_gen: 1.7673243284225464 (1.8004301956721716) | > loss_kl: 2.5069918632507324 (2.2687132596969604) | > loss_feat: 2.12410306930542 (2.493037905011858) | > loss_mel: 18.003875732421875 (18.26372806004116) | > loss_duration: 1.6415085792541504 (1.6136513880320957) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.043804168701172 (26.439560644967216) | > grad_norm_1: tensor(188.6026, device='cuda:0') (tensor(305.6307, device='cuda:0')) | > current_lr_0: 0.000193555698675659 | > current_lr_1: 0.000193555698675659 | > step_time: 0.4445 (0.4379111494336809) | > loader_time: 0.0083 (0.007261984688895089) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618992818912994 (-6.165302975072798e-06) | > avg_step_time: 0.45640345022711953 (-0.00019310897504781765) | > avg_loss_disc: 2.7837999874437362 (-0.020007324890350286) | > avg_loss_disc_real_0: 0.20675936162891523 (-0.0024804599897962076) | > avg_loss_disc_real_1: 0.23328898763152914 (-0.0031822993302009495) | > avg_loss_disc_real_2: 0.2391450134381442 (-0.0028383354066123245) | > avg_loss_disc_real_3: 0.2378138456965836 (+0.0003489276892702231) | > avg_loss_disc_real_4: 0.24171659791133773 (-0.0016881015519021325) | > avg_loss_disc_real_5: 0.24535861674328924 (-0.0011399766089211227) | > avg_loss_0: 2.7837999874437362 (-0.020007324890350286) | > avg_grad_norm_0: tensor(14.4271, device='cuda:0') (+tensor(2.0800, device='cuda:0')) | > avg_loss_gen: 1.817872146485557 (-0.0017003542940381422) | > avg_loss_kl: 2.274987988069024 (-0.03900285841713558) | > avg_loss_feat: 2.4847190279356193 (+0.04883648644030858) | > avg_loss_mel: 17.944097451760737 (-0.111604892032247) | > avg_loss_duration: 1.6281039093581724 (+0.0010887334044553665) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.149780461485964 (-0.10238295541682163) | > avg_grad_norm_1: tensor(326.2087, device='cuda:0') (+tensor(6.8412, device='cuda:0'))  > EPOCH: 263/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:53:51)   --> TIME: 2025-02-25 10:53:59 -- STEP: 13/72 -- GLOBAL_STEP: 1018950 | > loss_disc: 2.8535258769989014 (2.7984838485717773) | > loss_disc_real_0: 0.20753103494644165 (0.21102230136211103) | > loss_disc_real_1: 0.22364717721939087 (0.2360487454212629) | > loss_disc_real_2: 0.23552206158638 (0.23917730496479914) | > loss_disc_real_3: 0.21895261108875275 (0.23898802124536955) | > loss_disc_real_4: 0.21966077387332916 (0.24039202011548555) | > loss_disc_real_5: 0.22777017951011658 (0.24190515623642847) | > loss_0: 2.8535258769989014 (2.7984838485717773) | > grad_norm_0: tensor(6.1550, device='cuda:0') (tensor(13.0218, device='cuda:0')) | > loss_gen: 1.9201250076293945 (1.8346052995094886) | > loss_kl: 2.325843334197998 (2.121527048257681) | > loss_feat: 2.628891944885254 (2.5208912996145396) | > loss_mel: 17.576522827148438 (17.45954550229586) | > loss_duration: 1.5017613172531128 (1.5716863228724554) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.953142166137695 (25.508255445040188) | > grad_norm_1: tensor(219.4947, device='cuda:0') (tensor(307.9044, device='cuda:0')) | > current_lr_0: 0.00019353150421332454 | > current_lr_1: 0.00019353150421332454 | > step_time: 0.4591 (0.4326949486365685) | > loader_time: 0.0059 (0.006892607762263372)  --> TIME: 2025-02-25 10:54:22 -- STEP: 63/72 -- GLOBAL_STEP: 1019000 | > loss_disc: 2.824575424194336 (2.7908045299469477) | > loss_disc_real_0: 0.3039020299911499 (0.22240135830546182) | > loss_disc_real_1: 0.23627908527851105 (0.23247694590735057) | > loss_disc_real_2: 0.23040835559368134 (0.24044491303345514) | > loss_disc_real_3: 0.2541738748550415 (0.24100884653273083) | > loss_disc_real_4: 0.2656916677951813 (0.2444405657431436) | > loss_disc_real_5: 0.2586174011230469 (0.24423537840918888) | > loss_0: 2.824575424194336 (2.7908045299469477) | > grad_norm_0: tensor(21.0945, device='cuda:0') (tensor(12.9259, device='cuda:0')) | > loss_gen: 1.7907390594482422 (1.8430547827766055) | > loss_kl: 2.2493929862976074 (2.193100482698471) | > loss_feat: 2.526000738143921 (2.5013478578083097) | > loss_mel: 17.612079620361328 (17.671073535132034) | > loss_duration: 1.599284052848816 (1.6061289367221652) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.777496337890625 (25.814705439976283) | > grad_norm_1: tensor(238.6438, device='cuda:0') (tensor(318.7593, device='cuda:0')) | > current_lr_0: 0.00019353150421332454 | > current_lr_1: 0.00019353150421332454 | > step_time: 0.5037 (0.4472045860593281) | > loader_time: 0.0119 (0.008280598928057955) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666260141721909 (+4.726732280891516e-05) | > avg_step_time: 0.45453819086853886 (-0.0018652593585806732) | > avg_loss_disc: 2.7870685349048023 (+0.0032685474610660847) | > avg_loss_disc_real_0: 0.2193129904253382 (+0.012553628796422966) | > avg_loss_disc_real_1: 0.23093344208220362 (-0.002355545549325516) | > avg_loss_disc_real_2: 0.24009775863566868 (+0.0009527451975244738) | > avg_loss_disc_real_3: 0.24057642884657415 (+0.0027625831499905606) | > avg_loss_disc_real_4: 0.2443495210207684 (+0.002632923109430668) | > avg_loss_disc_real_5: 0.24449066754797816 (-0.0008679491953110796) | > avg_loss_0: 2.7870685349048023 (+0.0032685474610660847) | > avg_grad_norm_0: tensor(12.4185, device='cuda:0') (tensor(-2.0086, device='cuda:0')) | > avg_loss_gen: 1.843858174874749 (+0.02598602838919195) | > avg_loss_kl: 2.194182899636282 (-0.08080508843274181) | > avg_loss_feat: 2.502011522440843 (+0.017292494505223743) | > avg_loss_mel: 17.663567301253206 (-0.2805301505075306) | > avg_loss_duration: 1.613092259621956 (-0.015011649736216492) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.816711989926624 (-0.33306847155933994) | > avg_grad_norm_1: tensor(320.3393, device='cuda:0') (tensor(-5.8694, device='cuda:0'))  > EPOCH: 264/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:54:28)   --> TIME: 2025-02-25 10:54:49 -- STEP: 41/72 -- GLOBAL_STEP: 1019050 | > loss_disc: 2.798943281173706 (2.811408426703476) | > loss_disc_real_0: 0.17167311906814575 (0.21406419902313045) | > loss_disc_real_1: 0.30454930663108826 (0.23370237830208568) | > loss_disc_real_2: 0.20283372700214386 (0.24078054653435219) | > loss_disc_real_3: 0.22650368511676788 (0.24139891782911813) | > loss_disc_real_4: 0.26831579208374023 (0.24694299770564568) | > loss_disc_real_5: 0.2651343047618866 (0.2526783081816464) | > loss_0: 2.798943281173706 (2.811408426703476) | > grad_norm_0: tensor(6.3817, device='cuda:0') (tensor(14.8791, device='cuda:0')) | > loss_gen: 1.8252646923065186 (1.81287234876214) | > loss_kl: 2.1455514430999756 (2.19703486779841) | > loss_feat: 2.6255061626434326 (2.443337469566159) | > loss_mel: 17.582056045532227 (17.629401555875447) | > loss_duration: 1.6162803173065186 (1.5933011857474721) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.79465675354004 (25.675947421934545) | > grad_norm_1: tensor(87.6777, device='cuda:0') (tensor(306.3398, device='cuda:0')) | > current_lr_0: 0.00019350731277529787 | > current_lr_1: 0.00019350731277529787 | > step_time: 0.4526 (0.45376699726756026) | > loader_time: 0.0094 (0.007383584976196289) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599247730953592 (-6.70124107683169e-05) | > avg_step_time: 0.4640935911259181 (+0.00955540025737922) | > avg_loss_disc: 2.8115773100248527 (+0.02450877512005034) | > avg_loss_disc_real_0: 0.21461353209656725 (-0.004699458328770945) | > avg_loss_disc_real_1: 0.23366747208884064 (+0.0027340300066370227) | > avg_loss_disc_real_2: 0.24164021183067644 (+0.0015424531950077591) | > avg_loss_disc_real_3: 0.24171818750844876 (+0.0011417586618746034) | > avg_loss_disc_real_4: 0.24668893427915975 (+0.0023394132583913485) | > avg_loss_disc_real_5: 0.24978698055509116 (+0.005296313007113002) | > avg_loss_0: 2.8115773100248527 (+0.02450877512005034) | > avg_grad_norm_0: tensor(14.5835, device='cuda:0') (+tensor(2.1650, device='cuda:0')) | > avg_loss_gen: 1.813632113832823 (-0.030226061041925822) | > avg_loss_kl: 2.238549653912934 (+0.04436675427665193) | > avg_loss_feat: 2.4317841563426277 (-0.07022736609821534) | > avg_loss_mel: 17.73833879282777 (+0.07477149157456253) | > avg_loss_duration: 1.6168588433467166 (+0.003766583724760686) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.839163632460043 (+0.022451642533418692) | > avg_grad_norm_1: tensor(276.8417, device='cuda:0') (tensor(-43.4976, device='cuda:0'))  > EPOCH: 265/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:55:06)   --> TIME: 2025-02-25 10:55:16 -- STEP: 19/72 -- GLOBAL_STEP: 1019100 | > loss_disc: 2.7747931480407715 (2.7884974981609143) | > loss_disc_real_0: 0.19028520584106445 (0.19701477257828964) | > loss_disc_real_1: 0.22533713281154633 (0.23583584239608363) | > loss_disc_real_2: 0.23045337200164795 (0.24055986969094528) | > loss_disc_real_3: 0.22701144218444824 (0.24095208158618525) | > loss_disc_real_4: 0.2545524835586548 (0.2483063646053013) | > loss_disc_real_5: 0.2502312660217285 (0.2465424043567557) | > loss_0: 2.7747931480407715 (2.7884974981609143) | > grad_norm_0: tensor(10.4073, device='cuda:0') (tensor(11.5159, device='cuda:0')) | > loss_gen: 1.7919427156448364 (1.8107125696383024) | > loss_kl: 2.188772439956665 (2.2267333896536576) | > loss_feat: 2.2589211463928223 (2.4791368936237537) | > loss_mel: 16.534971237182617 (18.092321295487253) | > loss_duration: 1.5732671022415161 (1.5887389747720015) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.347875595092773 (26.197643581189606) | > grad_norm_1: tensor(98.3441, device='cuda:0') (tensor(303.4876, device='cuda:0')) | > current_lr_0: 0.00019348312436120093 | > current_lr_1: 0.00019348312436120093 | > step_time: 0.4398 (0.436563767884907) | > loader_time: 0.0083 (0.006643935253745631)  --> TIME: 2025-02-25 10:55:40 -- STEP: 69/72 -- GLOBAL_STEP: 1019150 | > loss_disc: 2.8543410301208496 (2.790700746619183) | > loss_disc_real_0: 0.2662767469882965 (0.21129156281982642) | > loss_disc_real_1: 0.22274205088615417 (0.23415966094404028) | > loss_disc_real_2: 0.22603286802768707 (0.24193818996781888) | > loss_disc_real_3: 0.25740712881088257 (0.23995128900244617) | > loss_disc_real_4: 0.24917946755886078 (0.24565540815609088) | > loss_disc_real_5: 0.2562352120876312 (0.24690096274666165) | > loss_0: 2.8543410301208496 (2.790700746619183) | > grad_norm_0: tensor(23.3877, device='cuda:0') (tensor(12.7817, device='cuda:0')) | > loss_gen: 1.8600139617919922 (1.827202384022699) | > loss_kl: 2.413802146911621 (2.2660159079924873) | > loss_feat: 2.2248873710632324 (2.4663482956264327) | > loss_mel: 17.76970863342285 (17.81932590318762) | > loss_duration: 1.719549536705017 (1.62319839173469) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.987960815429688 (26.00209108988444) | > grad_norm_1: tensor(440.6869, device='cuda:0') (tensor(279.8121, device='cuda:0')) | > current_lr_0: 0.00019348312436120093 | > current_lr_1: 0.00019348312436120093 | > step_time: 0.5393 (0.4574371973673503) | > loader_time: 0.0126 (0.008569641389708586) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626538263240328 (+2.7290532286736516e-05) | > avg_step_time: 0.4592857293679681 (-0.004807861757950005) | > avg_loss_disc: 2.793324611556362 (-0.018252698468490536) | > avg_loss_disc_real_0: 0.21286184749972653 (-0.001751684596840719) | > avg_loss_disc_real_1: 0.2353914732664404 (+0.0017240011775997433) | > avg_loss_disc_real_2: 0.24189991661360566 (+0.00025970478292922095) | > avg_loss_disc_real_3: 0.24004425696084197 (-0.0016739305476067834) | > avg_loss_disc_real_4: 0.24625160471654275 (-0.00043732956261699485) | > avg_loss_disc_real_5: 0.24740854787154937 (-0.0023784326835417924) | > avg_loss_0: 2.793324611556362 (-0.018252698468490536) | > avg_grad_norm_0: tensor(12.8353, device='cuda:0') (tensor(-1.7483, device='cuda:0')) | > avg_loss_gen: 1.8259809319402132 (+0.01234881810739008) | > avg_loss_kl: 2.2702195056727237 (+0.03166985175978976) | > avg_loss_feat: 2.457098499150343 (+0.025314342807715384) | > avg_loss_mel: 17.808063238439424 (+0.06972444561165503) | > avg_loss_duration: 1.6232083078841089 (+0.006349464537392269) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.984570677851284 (+0.1454070453912415) | > avg_grad_norm_1: tensor(285.3270, device='cuda:0') (+tensor(8.4853, device='cuda:0'))  > EPOCH: 266/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:55:43)   --> TIME: 2025-02-25 10:56:07 -- STEP: 47/72 -- GLOBAL_STEP: 1019200 | > loss_disc: 2.73371958732605 (2.7782913065971213) | > loss_disc_real_0: 0.1915145367383957 (0.20867148659964826) | > loss_disc_real_1: 0.24310797452926636 (0.23197172962604684) | > loss_disc_real_2: 0.25080323219299316 (0.23609646964580455) | > loss_disc_real_3: 0.21380671858787537 (0.23816174490654723) | > loss_disc_real_4: 0.237012580037117 (0.24282430429407892) | > loss_disc_real_5: 0.22089073061943054 (0.24264393620034483) | > loss_0: 2.73371958732605 (2.7782913065971213) | > grad_norm_0: tensor(4.7682, device='cuda:0') (tensor(11.7604, device='cuda:0')) | > loss_gen: 1.8422846794128418 (1.8241602689661878) | > loss_kl: 2.123656988143921 (2.2694170779370255) | > loss_feat: 2.6896064281463623 (2.4876963534253704) | > loss_mel: 18.151573181152344 (17.874684272928437) | > loss_duration: 1.603271722793579 (1.5958136117204706) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.41039276123047 (26.051771488595517) | > grad_norm_1: tensor(207.6505, device='cuda:0') (tensor(312.6748, device='cuda:0')) | > current_lr_0: 0.0001934589389706558 | > current_lr_1: 0.0001934589389706558 | > step_time: 0.4464 (0.4452880544865385) | > loader_time: 0.0089 (0.007720521155824053) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688003244534342 (+6.146498129401372e-05) | > avg_step_time: 0.4588409276075766 (-0.0004448017603914667) | > avg_loss_disc: 2.792685418061807 (-0.0006391934945551014) | > avg_loss_disc_real_0: 0.21206459864764146 (-0.0007972488520850696) | > avg_loss_disc_real_1: 0.2364329567677538 (+0.0010414835013134172) | > avg_loss_disc_real_2: 0.23877290604819715 (-0.003127010565408511) | > avg_loss_disc_real_3: 0.24129382111656833 (+0.001249564155726357) | > avg_loss_disc_real_4: 0.24384205672942416 (-0.002409547987118593) | > avg_loss_disc_real_5: 0.24554603179575693 (-0.0018625160757924364) | > avg_loss_0: 2.792685418061807 (-0.0006391934945551014) | > avg_grad_norm_0: tensor(12.5859, device='cuda:0') (tensor(-0.2493, device='cuda:0')) | > avg_loss_gen: 1.8204107989727611 (-0.005570132967452013) | > avg_loss_kl: 2.285205659732014 (+0.014986154059290158) | > avg_loss_feat: 2.442193632394494 (-0.0149048667558489) | > avg_loss_mel: 17.852370329306165 (+0.04430709086674156) | > avg_loss_duration: 1.6155048675940071 (-0.007703440290101726) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.015685229234293 (+0.031114551383009115) | > avg_grad_norm_1: tensor(311.7816, device='cuda:0') (+tensor(26.4546, device='cuda:0'))  > EPOCH: 267/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:56:21)   --> TIME: 2025-02-25 10:56:34 -- STEP: 25/72 -- GLOBAL_STEP: 1019250 | > loss_disc: 2.8657124042510986 (2.7904735088348387) | > loss_disc_real_0: 0.15621516108512878 (0.20558788299560546) | > loss_disc_real_1: 0.2262723594903946 (0.2365476506948471) | > loss_disc_real_2: 0.23840385675430298 (0.2433670949935913) | > loss_disc_real_3: 0.24672670662403107 (0.23873712182044982) | > loss_disc_real_4: 0.255948543548584 (0.24648343920707702) | > loss_disc_real_5: 0.2583460211753845 (0.24736233711242675) | > loss_0: 2.8657124042510986 (2.7904735088348387) | > grad_norm_0: tensor(14.1943, device='cuda:0') (tensor(12.5627, device='cuda:0')) | > loss_gen: 1.721482276916504 (1.8274787378311157) | > loss_kl: 2.2157366275787354 (2.1782717561721805) | > loss_feat: 2.2162911891937256 (2.5210741424560554) | > loss_mel: 17.890701293945312 (17.586420822143555) | > loss_duration: 1.5670781135559082 (1.5742318153381347) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.611289978027344 (25.687477111816406) | > grad_norm_1: tensor(356.0092, device='cuda:0') (tensor(392.9459, device='cuda:0')) | > current_lr_0: 0.00019343475660328445 | > current_lr_1: 0.00019343475660328445 | > step_time: 0.4302 (0.4328083801269531) | > loader_time: 0.0064 (0.006822433471679688) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860661184284049 (-8.13914016938519e-05) | > avg_step_time: 0.45493555740571356 (-0.0039053702018630454) | > avg_loss_disc: 2.7832320340922183 (-0.009453383969588725) | > avg_loss_disc_real_0: 0.20392318909436885 (-0.00814140955327261) | > avg_loss_disc_real_1: 0.23196894937837628 (-0.004464007389377522) | > avg_loss_disc_real_2: 0.23879239802629176 (+1.949197809461567e-05) | > avg_loss_disc_real_3: 0.23640336033324122 (-0.004890460783327111) | > avg_loss_disc_real_4: 0.24345726635254605 (-0.000384790376878108) | > avg_loss_disc_real_5: 0.24714494276214655 (+0.0015989109663896162) | > avg_loss_0: 2.7832320340922183 (-0.009453383969588725) | > avg_grad_norm_0: tensor(12.2662, device='cuda:0') (tensor(-0.3198, device='cuda:0')) | > avg_loss_gen: 1.8162612931829103 (-0.004149505789850849) | > avg_loss_kl: 2.2885643075889255 (+0.003358647856911645) | > avg_loss_feat: 2.4893127058593327 (+0.04711907346483857) | > avg_loss_mel: 17.506102320174104 (-0.3462680091320607) | > avg_loss_duration: 1.6144815532254502 (-0.0010233143685569601) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.71472224383287 (-0.30096298540142286) | > avg_grad_norm_1: tensor(346.8440, device='cuda:0') (+tensor(35.0623, device='cuda:0'))  > EPOCH: 268/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:56:58)   --> TIME: 2025-02-25 10:57:01 -- STEP: 3/72 -- GLOBAL_STEP: 1019300 | > loss_disc: 2.7860841751098633 (2.7967259089152017) | > loss_disc_real_0: 0.28500252962112427 (0.22246742248535156) | > loss_disc_real_1: 0.19570569694042206 (0.22484159469604492) | > loss_disc_real_2: 0.2657780349254608 (0.2642980714639028) | > loss_disc_real_3: 0.2102639079093933 (0.2405269593000412) | > loss_disc_real_4: 0.24001479148864746 (0.2464956889549891) | > loss_disc_real_5: 0.25283709168434143 (0.2573460141817729) | > loss_0: 2.7860841751098633 (2.7967259089152017) | > grad_norm_0: tensor(17.6122, device='cuda:0') (tensor(13.5175, device='cuda:0')) | > loss_gen: 1.7475621700286865 (1.7910313606262207) | > loss_kl: 2.2595582008361816 (2.255664507548014) | > loss_feat: 2.660527229309082 (2.495690663655599) | > loss_mel: 17.899185180664062 (17.6724116007487) | > loss_duration: 1.62228524684906 (1.6094582080841064) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.189119338989258 (25.824256896972656) | > grad_norm_1: tensor(557.3786, device='cuda:0') (tensor(303.6593, device='cuda:0')) | > current_lr_0: 0.00019341057725870902 | > current_lr_1: 0.00019341057725870902 | > step_time: 0.4797 (0.45293108622233075) | > loader_time: 0.006 (0.006508588790893555)  --> TIME: 2025-02-25 10:57:24 -- STEP: 53/72 -- GLOBAL_STEP: 1019350 | > loss_disc: 2.8038153648376465 (2.7981351546521456) | > loss_disc_real_0: 0.22615289688110352 (0.2141912746823059) | > loss_disc_real_1: 0.21377941966056824 (0.22910808139252212) | > loss_disc_real_2: 0.223164364695549 (0.24127995011941442) | > loss_disc_real_3: 0.22995881736278534 (0.23832301795482635) | > loss_disc_real_4: 0.242198646068573 (0.2449449770293146) | > loss_disc_real_5: 0.22895005345344543 (0.24918019855922116) | > loss_0: 2.8038153648376465 (2.7981351546521456) | > grad_norm_0: tensor(7.4150, device='cuda:0') (tensor(13.6181, device='cuda:0')) | > loss_gen: 1.7177841663360596 (1.812737543627901) | > loss_kl: 2.349252700805664 (2.326595153448717) | > loss_feat: 2.6992878913879395 (2.4849364510122336) | > loss_mel: 18.290740966796875 (17.93493794495205) | > loss_duration: 1.655540943145752 (1.6011353123862788) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.71260643005371 (26.160342270473265) | > grad_norm_1: tensor(480.3504, device='cuda:0') (tensor(270.2049, device='cuda:0')) | > current_lr_0: 0.00019341057725870902 | > current_lr_1: 0.00019341057725870902 | > step_time: 0.4506 (0.4478196423008757) | > loader_time: 0.009 (0.008023788344185306) --> EVAL PERFORMANCE | > avg_loader_time: 0.008768279787520285 (+0.00016166794467979521) | > avg_step_time: 0.46108913757431674 (+0.006153580168603179) | > avg_loss_disc: 2.793526938263799 (+0.010294904171580743) | > avg_loss_disc_real_0: 0.21179995517915404 (+0.007876766084785186) | > avg_loss_disc_real_1: 0.22936253669396253 (-0.0026064126844137503) | > avg_loss_disc_real_2: 0.24119300342781444 (+0.0024006054015226763) | > avg_loss_disc_real_3: 0.23820847881511903 (+0.0018051184818778099) | > avg_loss_disc_real_4: 0.24502111320764247 (+0.001563846855096418) | > avg_loss_disc_real_5: 0.24851690526579467 (+0.0013719625036481276) | > avg_loss_0: 2.793526938263799 (+0.010294904171580743) | > avg_grad_norm_0: tensor(13.4532, device='cuda:0') (+tensor(1.1870, device='cuda:0')) | > avg_loss_gen: 1.812130330314099 (-0.0041309628688113165) | > avg_loss_kl: 2.33775815493624 (+0.04919384734731436) | > avg_loss_feat: 2.4551388445034834 (-0.03417386135584932) | > avg_loss_mel: 17.91407282923309 (+0.407970509058984) | > avg_loss_duration: 1.6143565715198787 (-0.0001249817055715141) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.13345656596439 (+0.4187343221315203) | > avg_grad_norm_1: tensor(282.1109, device='cuda:0') (tensor(-64.7331, device='cuda:0'))  > EPOCH: 269/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:57:35)   --> TIME: 2025-02-25 10:57:51 -- STEP: 31/72 -- GLOBAL_STEP: 1019400 | > loss_disc: 2.690685987472534 (2.795716639487974) | > loss_disc_real_0: 0.24207371473312378 (0.20960228337395576) | > loss_disc_real_1: 0.19550806283950806 (0.23048171088580163) | > loss_disc_real_2: 0.23386526107788086 (0.24348208981175576) | > loss_disc_real_3: 0.23308631777763367 (0.242260800734643) | > loss_disc_real_4: 0.23307682573795319 (0.24419940719681402) | > loss_disc_real_5: 0.23722977936267853 (0.24744098465288838) | > loss_0: 2.690685987472534 (2.795716639487974) | > grad_norm_0: tensor(6.2085, device='cuda:0') (tensor(11.0844, device='cuda:0')) | > loss_gen: 1.7288051843643188 (1.8188511287012408) | > loss_kl: 2.5160696506500244 (2.201588922931302) | > loss_feat: 2.509124755859375 (2.5104715824127197) | > loss_mel: 18.550683975219727 (17.627799372519217) | > loss_duration: 1.612969994544983 (1.573270555465452) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.917654037475586 (25.731981338993197) | > grad_norm_1: tensor(381.2701, device='cuda:0') (tensor(296.0647, device='cuda:0')) | > current_lr_0: 0.00019338640093655168 | > current_lr_1: 0.00019338640093655168 | > step_time: 0.4468 (0.44053562994926204) | > loader_time: 0.0093 (0.007335316750311082) --> EVAL PERFORMANCE | > avg_loader_time: 0.008798219788242394 (+2.9940000722109006e-05) | > avg_step_time: 0.4598179602287185 (-0.001271177345598229) | > avg_loss_disc: 2.7937530128049177 (+0.00022607454111867042) | > avg_loss_disc_real_0: 0.20727488797315408 (-0.004525067205999961) | > avg_loss_disc_real_1: 0.23101766879709673 (+0.0016551321031341992) | > avg_loss_disc_real_2: 0.23944756388664246 (-0.0017454395411719836) | > avg_loss_disc_real_3: 0.24025566078407665 (+0.0020471819689576176) | > avg_loss_disc_real_4: 0.2423346802382402 (-0.0026864329694022693) | > avg_loss_disc_real_5: 0.24438001782121793 (-0.004136887444576742) | > avg_loss_0: 2.7937530128049177 (+0.00022607454111867042) | > avg_grad_norm_0: tensor(13.6577, device='cuda:0') (+tensor(0.2045, device='cuda:0')) | > avg_loss_gen: 1.8122121616148612 (+8.183130076222511e-05) | > avg_loss_kl: 2.2423043771528857 (-0.09545377778335418) | > avg_loss_feat: 2.4679713232416502 (+0.012832478738166841) | > avg_loss_mel: 17.61867987941688 (-0.29539294981620756) | > avg_loss_duration: 1.61445813951358 (+0.00010156799370131253) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.755625899408905 (-0.3778306665554858) | > avg_grad_norm_1: tensor(301.2683, device='cuda:0') (+tensor(19.1574, device='cuda:0'))  > EPOCH: 270/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:58:13)   --> TIME: 2025-02-25 10:58:19 -- STEP: 9/72 -- GLOBAL_STEP: 1019450 | > loss_disc: 2.7027664184570312 (2.7550247775183783) | > loss_disc_real_0: 0.24548545479774475 (0.20474006566736433) | > loss_disc_real_1: 0.21110789477825165 (0.2286868608660168) | > loss_disc_real_2: 0.23162741959095 (0.23583313822746277) | > loss_disc_real_3: 0.2115199714899063 (0.22876685692204368) | > loss_disc_real_4: 0.23915491998195648 (0.2412685669130749) | > loss_disc_real_5: 0.24573299288749695 (0.2404312872224384) | > loss_0: 2.7027664184570312 (2.7550247775183783) | > grad_norm_0: tensor(15.0618, device='cuda:0') (tensor(7.4771, device='cuda:0')) | > loss_gen: 1.9038069248199463 (1.8062189949883356) | > loss_kl: 2.0830981731414795 (2.0996445417404175) | > loss_feat: 2.72202730178833 (2.5738343132866754) | > loss_mel: 18.30853271484375 (17.804648717244465) | > loss_duration: 1.5956943035125732 (1.586316426595052) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.6131591796875 (25.870663113064236) | > grad_norm_1: tensor(278.5457, device='cuda:0') (tensor(214.5548, device='cuda:0')) | > current_lr_0: 0.00019336222763643461 | > current_lr_1: 0.00019336222763643461 | > step_time: 0.4269 (0.4335385693444146) | > loader_time: 0.0055 (0.006255600187513564)  --> TIME: 2025-02-25 10:58:42 -- STEP: 59/72 -- GLOBAL_STEP: 1019500 | > loss_disc: 2.8124406337738037 (2.7866196955664684) | > loss_disc_real_0: 0.2114969789981842 (0.20755123309159684) | > loss_disc_real_1: 0.22250017523765564 (0.23394011794510533) | > loss_disc_real_2: 0.23355984687805176 (0.23976569736407974) | > loss_disc_real_3: 0.2411428987979889 (0.23734694845595602) | > loss_disc_real_4: 0.23745733499526978 (0.24411128158286466) | > loss_disc_real_5: 0.2489711046218872 (0.24647546218613448) | > loss_0: 2.8124406337738037 (2.7866196955664684) | > grad_norm_0: tensor(6.4013, device='cuda:0') (tensor(11.6631, device='cuda:0')) | > loss_gen: 1.75387704372406 (1.8136437504978504) | > loss_kl: 2.246408700942993 (2.27193482851578) | > loss_feat: 2.25864315032959 (2.5022074772139726) | > loss_mel: 16.900705337524414 (17.73047479532532) | > loss_duration: 1.65522038936615 (1.6071915202221627) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.814855575561523 (25.92545247482041) | > grad_norm_1: tensor(432.1879, device='cuda:0') (tensor(348.4449, device='cuda:0')) | > current_lr_0: 0.00019336222763643461 | > current_lr_1: 0.00019336222763643461 | > step_time: 0.4823 (0.4480685056266138) | > loader_time: 0.01 (0.00806837162729037) --> EVAL PERFORMANCE | > avg_loader_time: 0.008641414239373008 (-0.00015680554886938688) | > avg_step_time: 0.45700010783235784 (-0.002817852396360676) | > avg_loss_disc: 2.789391487417087 (-0.004361525387830856) | > avg_loss_disc_real_0: 0.21111105571330432 (+0.003836167740150248) | > avg_loss_disc_real_1: 0.23401561064619414 (+0.0029979418490974075) | > avg_loss_disc_real_2: 0.24062462235000773 (+0.0011770584633652714) | > avg_loss_disc_real_3: 0.23761260068752396 (-0.002643060096552685) | > avg_loss_disc_real_4: 0.24446501866192885 (+0.002130338423688649) | > avg_loss_disc_real_5: 0.24653112888336182 (+0.0021511110621438845) | > avg_loss_0: 2.789391487417087 (-0.004361525387830856) | > avg_grad_norm_0: tensor(11.7649, device='cuda:0') (tensor(-1.8928, device='cuda:0')) | > avg_loss_gen: 1.8139300161684062 (+0.0017178545535450063) | > avg_loss_kl: 2.267382319544402 (+0.025077942391516395) | > avg_loss_feat: 2.47749409205477 (+0.009522768813119864) | > avg_loss_mel: 17.717487039700355 (+0.09880716028347436) | > avg_loss_duration: 1.615197912068434 (+0.0007397725548541345) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.891491500424667 (+0.1358656010157624) | > avg_grad_norm_1: tensor(356.0659, device='cuda:0') (+tensor(54.7976, device='cuda:0'))  > EPOCH: 271/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:58:50)   --> TIME: 2025-02-25 10:59:09 -- STEP: 37/72 -- GLOBAL_STEP: 1019550 | > loss_disc: 2.8414535522460938 (2.7768675701038257) | > loss_disc_real_0: 0.1615384817123413 (0.21465196923629656) | > loss_disc_real_1: 0.12683047354221344 (0.22620866669190898) | > loss_disc_real_2: 0.25572827458381653 (0.23819843740076632) | > loss_disc_real_3: 0.26416894793510437 (0.23697744390449008) | > loss_disc_real_4: 0.2795366942882538 (0.24305241534838806) | > loss_disc_real_5: 0.2507254183292389 (0.24409140606184263) | > loss_0: 2.8414535522460938 (2.7768675701038257) | > grad_norm_0: tensor(12.4698, device='cuda:0') (tensor(12.0439, device='cuda:0')) | > loss_gen: 1.7607835531234741 (1.8307447755658948) | > loss_kl: 2.107106924057007 (2.1230430474152437) | > loss_feat: 2.481074333190918 (2.588199634809752) | > loss_mel: 17.266998291015625 (17.479541108414935) | > loss_duration: 1.6090502738952637 (1.57584755162935) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.225013732910156 (25.5973759728509) | > grad_norm_1: tensor(470.2355, device='cuda:0') (tensor(377.3282, device='cuda:0')) | > current_lr_0: 0.00019333805735798006 | > current_lr_1: 0.00019333805735798006 | > step_time: 0.4535 (0.4553076512104756) | > loader_time: 0.0081 (0.007182288814235378) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584180348356008 (-5.723389101700005e-05) | > avg_step_time: 0.46799561339364926 (+0.010995505561291419) | > avg_loss_disc: 2.79038960832945 (+0.0009981209123632873) | > avg_loss_disc_real_0: 0.21217001512856554 (+0.0010589594152612125) | > avg_loss_disc_real_1: 0.23324553017884914 (-0.0007700804673450024) | > avg_loss_disc_real_2: 0.2387447138907204 (-0.001879908459287316) | > avg_loss_disc_real_3: 0.23700871182159639 (-0.0006038888659275765) | > avg_loss_disc_real_4: 0.24217962127336315 (-0.002285397388565702) | > avg_loss_disc_real_5: 0.24364938240655712 (-0.0028817464768046974) | > avg_loss_0: 2.79038960832945 (+0.0009981209123632873) | > avg_grad_norm_0: tensor(11.8407, device='cuda:0') (+tensor(0.0757, device='cuda:0')) | > avg_loss_gen: 1.8102053118423678 (-0.0037247043260384416) | > avg_loss_kl: 2.226999669007852 (-0.0403826505365501) | > avg_loss_feat: 2.5113206581330627 (+0.0338265660782926) | > avg_loss_mel: 17.629396035637654 (-0.08809100406270076) | > avg_loss_duration: 1.608688576120726 (-0.006509335947708195) | > avg_amp_scaler: 174.8732394366197 (+46.87323943661971) | > avg_loss_1: 25.786610160075444 (-0.10488134034922325) | > avg_grad_norm_1: tensor(328.0999, device='cuda:0') (tensor(-27.9660, device='cuda:0'))  > EPOCH: 272/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 10:59:28)   --> TIME: 2025-02-25 10:59:36 -- STEP: 15/72 -- GLOBAL_STEP: 1019600 | > loss_disc: 2.7346444129943848 (2.8100091934204103) | > loss_disc_real_0: 0.22842058539390564 (0.2015215535958608) | > loss_disc_real_1: 0.2392561137676239 (0.24091099301973978) | > loss_disc_real_2: 0.2553258538246155 (0.2459296812613805) | > loss_disc_real_3: 0.2538708448410034 (0.24580354491869608) | > loss_disc_real_4: 0.23967857658863068 (0.24646811286608378) | > loss_disc_real_5: 0.2600070536136627 (0.24759635229905447) | > loss_0: 2.7346444129943848 (2.8100091934204103) | > grad_norm_0: tensor(6.3744, device='cuda:0') (tensor(8.5469, device='cuda:0')) | > loss_gen: 1.7864171266555786 (1.8144797325134276) | > loss_kl: 2.3467819690704346 (2.2766074736913047) | > loss_feat: 2.8414218425750732 (2.529336373011271) | > loss_mel: 19.389541625976562 (17.40292867024739) | > loss_duration: 1.604763388633728 (1.58646449247996) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.96892547607422 (25.609816869099934) | > grad_norm_1: tensor(424.0074, device='cuda:0') (tensor(381.3211, device='cuda:0')) | > current_lr_0: 0.0001933138901008103 | > current_lr_1: 0.0001933138901008103 | > step_time: 0.4537 (0.44196173350016277) | > loader_time: 0.0074 (0.006856981913248698)  --> TIME: 2025-02-25 11:00:00 -- STEP: 65/72 -- GLOBAL_STEP: 1019650 | > loss_disc: 2.8045108318328857 (2.8013700118431677) | > loss_disc_real_0: 0.26199376583099365 (0.20930752410338474) | > loss_disc_real_1: 0.24461601674556732 (0.23103887347074656) | > loss_disc_real_2: 0.24378596246242523 (0.23911268344292275) | > loss_disc_real_3: 0.2726442813873291 (0.24079556098351113) | > loss_disc_real_4: 0.25937554240226746 (0.24527037350031045) | > loss_disc_real_5: 0.24097245931625366 (0.24794609959308916) | > loss_0: 2.8045108318328857 (2.8013700118431677) | > grad_norm_0: tensor(21.6171, device='cuda:0') (tensor(14.4813, device='cuda:0')) | > loss_gen: 1.9874014854431152 (1.8113062968620888) | > loss_kl: 2.562875509262085 (2.3163596648436324) | > loss_feat: 2.3880090713500977 (2.493614515891442) | > loss_mel: 18.217529296875 (17.72217075641338) | > loss_duration: 1.6531890630722046 (1.6241333484649658) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.809005737304688 (25.967584698016825) | > grad_norm_1: tensor(136.1061, device='cuda:0') (tensor(354.5403, device='cuda:0')) | > current_lr_0: 0.0001933138901008103 | > current_lr_1: 0.0001933138901008103 | > step_time: 0.4945 (0.45135938571049616) | > loader_time: 0.0117 (0.008493647208580602) --> EVAL PERFORMANCE | > avg_loader_time: 0.008753319861183701 (+0.00016913951282769395) | > avg_step_time: 0.4578270744270002 (-0.010168538966649043) | > avg_loss_disc: 2.7999515701347675 (+0.00956196180531732) | > avg_loss_disc_real_0: 0.212465107650824 (+0.0002950925222584555) | > avg_loss_disc_real_1: 0.23213263968346823 (-0.0011128904953809082) | > avg_loss_disc_real_2: 0.238139958868564 (-0.0006047550221564002) | > avg_loss_disc_real_3: 0.24139464762009366 (+0.004385935798497276) | > avg_loss_disc_real_4: 0.2446829571690358 (+0.0025033358956726426) | > avg_loss_disc_real_5: 0.24739231714900112 (+0.0037429347424439985) | > avg_loss_0: 2.7999515701347675 (+0.00956196180531732) | > avg_grad_norm_0: tensor(14.7697, device='cuda:0') (+tensor(2.9291, device='cuda:0')) | > avg_loss_gen: 1.8105357230549128 (+0.00033041121254506756) | > avg_loss_kl: 2.3186180709113535 (+0.09161840190350157) | > avg_loss_feat: 2.479771164101614 (-0.03154949403144869) | > avg_loss_mel: 17.774749057393674 (+0.14535302175601927) | > avg_loss_duration: 1.6274725752817074 (+0.0187839991609815) | > avg_amp_scaler: 256.0 (+81.12676056338029) | > avg_loss_1: 26.011146679730484 (+0.2245365196550395) | > avg_grad_norm_1: tensor(348.7711, device='cuda:0') (+tensor(20.6711, device='cuda:0'))  > EPOCH: 273/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:00:05)   --> TIME: 2025-02-25 11:00:26 -- STEP: 43/72 -- GLOBAL_STEP: 1019700 | > loss_disc: 2.89148211479187 (2.7815969711126285) | > loss_disc_real_0: 0.25463002920150757 (0.20617577173682147) | > loss_disc_real_1: 0.267387717962265 (0.23029783472072246) | > loss_disc_real_2: 0.256526380777359 (0.24066322626069533) | > loss_disc_real_3: 0.2650165557861328 (0.23874906086644462) | > loss_disc_real_4: 0.2562217116355896 (0.24429693194322807) | > loss_disc_real_5: 0.2613602876663208 (0.2482523283986158) | > loss_0: 2.89148211479187 (2.7815969711126285) | > grad_norm_0: tensor(9.2173, device='cuda:0') (tensor(13.6981, device='cuda:0')) | > loss_gen: 1.7810050249099731 (1.8521747866342233) | > loss_kl: 2.08697509765625 (2.2267759622529493) | > loss_feat: 2.6311867237091064 (2.6388285659080326) | > loss_mel: 17.358240127563477 (17.78307249379712) | > loss_duration: 1.6198806762695312 (1.601203064585841) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.47728729248047 (26.10205481773199) | > grad_norm_1: tensor(120.8108, device='cuda:0') (tensor(402.3437, device='cuda:0')) | > current_lr_0: 0.0001932897258645477 | > current_lr_1: 0.0001932897258645477 | > step_time: 0.4506 (0.439546080522759) | > loader_time: 0.0096 (0.007504341214202171) --> EVAL PERFORMANCE | > avg_loader_time: 0.008623831708666305 (-0.0001294881525173968) | > avg_step_time: 0.45466409602635344 (-0.0031629784006467765) | > avg_loss_disc: 2.7925663128705094 (-0.007385257264258094) | > avg_loss_disc_real_0: 0.20680251192878668 (-0.005662595722037311) | > avg_loss_disc_real_1: 0.23537606084850474 (+0.0032434211650365086) | > avg_loss_disc_real_2: 0.24053818784968953 (+0.0023982289811255164) | > avg_loss_disc_real_3: 0.2415068185245487 (+0.00011217090445503342) | > avg_loss_disc_real_4: 0.24585694515369308 (+0.0011739879846572876) | > avg_loss_disc_real_5: 0.24928902362433952 (+0.0018967064753384055) | > avg_loss_0: 2.7925663128705094 (-0.007385257264258094) | > avg_grad_norm_0: tensor(14.6170, device='cuda:0') (tensor(-0.1528, device='cuda:0')) | > avg_loss_gen: 1.8273602841605603 (+0.01682456110564745) | > avg_loss_kl: 2.255778255596966 (-0.06283981531438743) | > avg_loss_feat: 2.538193057960188 (+0.05842189385857388) | > avg_loss_mel: 17.84959513704542 (+0.07484607965174561) | > avg_loss_duration: 1.620756468302767 (-0.006716106978940317) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.091683105683664 (+0.08053642595318067) | > avg_grad_norm_1: tensor(341.1898, device='cuda:0') (tensor(-7.5812, device='cuda:0'))  > EPOCH: 274/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:00:42)   --> TIME: 2025-02-25 11:00:53 -- STEP: 21/72 -- GLOBAL_STEP: 1019750 | > loss_disc: 2.729905128479004 (2.8246118000575473) | > loss_disc_real_0: 0.16897441446781158 (0.218954864357199) | > loss_disc_real_1: 0.21437332034111023 (0.2610744174037661) | > loss_disc_real_2: 0.243479922413826 (0.23935002798125857) | > loss_disc_real_3: 0.22575674951076508 (0.23222430688994272) | > loss_disc_real_4: 0.25530847907066345 (0.24368733167648315) | > loss_disc_real_5: 0.2539457082748413 (0.2449035325220653) | > loss_0: 2.729905128479004 (2.8246118000575473) | > grad_norm_0: tensor(4.5970, device='cuda:0') (tensor(18.8114, device='cuda:0')) | > loss_gen: 1.9767484664916992 (1.8638616630009242) | > loss_kl: 2.1041486263275146 (2.200740411168053) | > loss_feat: 2.587838649749756 (2.450400534130278) | > loss_mel: 18.162254333496094 (17.902723221551803) | > loss_duration: 1.6025220155715942 (1.5826391889935447) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.43351173400879 (26.000364939371746) | > grad_norm_1: tensor(108.9674, device='cuda:0') (tensor(297.6574, device='cuda:0')) | > current_lr_0: 0.00019326556464881463 | > current_lr_1: 0.00019326556464881463 | > step_time: 0.4422 (0.43447096007210867) | > loader_time: 0.0075 (0.0069895244780040925)  --> TIME: 2025-02-25 11:01:17 -- STEP: 71/72 -- GLOBAL_STEP: 1019800 | > loss_disc: 2.8288233280181885 (2.7970612485643844) | > loss_disc_real_0: 0.24932734668254852 (0.20993671303903555) | > loss_disc_real_1: 0.25181078910827637 (0.2403463280536759) | > loss_disc_real_2: 0.2735348343849182 (0.23910382746810643) | > loss_disc_real_3: 0.23424570262432098 (0.23862022434321928) | > loss_disc_real_4: 0.23047107458114624 (0.24400763759310817) | > loss_disc_real_5: 0.24765074253082275 (0.24736399130082468) | > loss_0: 2.8288233280181885 (2.7970612485643844) | > grad_norm_0: tensor(7.6433, device='cuda:0') (tensor(14.2313, device='cuda:0')) | > loss_gen: 1.776017665863037 (1.831615021531011) | > loss_kl: 2.176791191101074 (2.228224715716402) | > loss_feat: 2.2179207801818848 (2.4856633904954073) | > loss_mel: 18.206375122070312 (17.786258361708942) | > loss_duration: 1.6263742446899414 (1.6206641432265165) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.00347900390625 (25.952425647789322) | > grad_norm_1: tensor(185.1908, device='cuda:0') (tensor(331.6415, device='cuda:0')) | > current_lr_0: 0.00019326556464881463 | > current_lr_1: 0.00019326556464881463 | > step_time: 0.4966 (0.4565152517506774) | > loader_time: 0.008 (0.008839217709823393) --> EVAL PERFORMANCE | > avg_loader_time: 0.008839217709823393 (+0.00021538600115708802) | > avg_step_time: 0.4565152517506774 (+0.001851155724323983) | > avg_loss_disc: 2.7970612485643844 (+0.004494935693875046) | > avg_loss_disc_real_0: 0.20993671303903555 (+0.0031342011102488687) | > avg_loss_disc_real_1: 0.2403463280536759 (+0.004970267205171158) | > avg_loss_disc_real_2: 0.23910382746810643 (-0.001434360381583094) | > avg_loss_disc_real_3: 0.23862022434321928 (-0.002886594181329416) | > avg_loss_disc_real_4: 0.24400763759310817 (-0.0018493075605849063) | > avg_loss_disc_real_5: 0.24736399130082468 (-0.001925032323514847) | > avg_loss_0: 2.7970612485643844 (+0.004494935693875046) | > avg_grad_norm_0: tensor(14.2313, device='cuda:0') (tensor(-0.3857, device='cuda:0')) | > avg_loss_gen: 1.831615021531011 (+0.004254737370450679) | > avg_loss_kl: 2.228224715716402 (-0.02755353988056397) | > avg_loss_feat: 2.4856633904954073 (-0.052529667464780605) | > avg_loss_mel: 17.786258361708942 (-0.0633367753364773) | > avg_loss_duration: 1.6206641432265165 (-9.232507625056563e-05) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.952425647789322 (-0.13925745789434174) | > avg_grad_norm_1: tensor(331.6415, device='cuda:0') (tensor(-9.5483, device='cuda:0'))  > EPOCH: 275/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:01:19)   --> TIME: 2025-02-25 11:01:44 -- STEP: 49/72 -- GLOBAL_STEP: 1019850 | > loss_disc: 2.701420783996582 (2.7891638570902306) | > loss_disc_real_0: 0.14932817220687866 (0.2119789050549877) | > loss_disc_real_1: 0.236627995967865 (0.23934943943607564) | > loss_disc_real_2: 0.23906578123569489 (0.23985315920138844) | > loss_disc_real_3: 0.24051180481910706 (0.23887284282518892) | > loss_disc_real_4: 0.2385750561952591 (0.24420245052600392) | > loss_disc_real_5: 0.22827352583408356 (0.2489713156709866) | > loss_0: 2.701420783996582 (2.7891638570902306) | > grad_norm_0: tensor(9.9770, device='cuda:0') (tensor(14.1220, device='cuda:0')) | > loss_gen: 1.924122929573059 (1.8349381296002134) | > loss_kl: 2.3409371376037598 (2.2053775349441835) | > loss_feat: 2.2683048248291016 (2.500938201437191) | > loss_mel: 18.414867401123047 (17.86532865251814) | > loss_duration: 1.6490974426269531 (1.5934681795081314) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.59733009338379 (26.000050603126994) | > grad_norm_1: tensor(185.0667, device='cuda:0') (tensor(317.8329, device='cuda:0')) | > current_lr_0: 0.0001932414064532335 | > current_lr_1: 0.0001932414064532335 | > step_time: 0.4426 (0.445819261122723) | > loader_time: 0.0097 (0.0075317694216358415) --> EVAL PERFORMANCE | > avg_loader_time: 0.008504320198381448 (-0.000334897511441945) | > avg_step_time: 0.4562975587979169 (-0.0002176929527605287) | > avg_loss_disc: 2.792662167213333 (-0.0043990813510514926) | > avg_loss_disc_real_0: 0.21503783069865803 (+0.005101117659622484) | > avg_loss_disc_real_1: 0.236558413421604 (-0.003787914632071898) | > avg_loss_disc_real_2: 0.2401307655472151 (+0.001026938079108669) | > avg_loss_disc_real_3: 0.23917940405892654 (+0.0005591797157072642) | > avg_loss_disc_real_4: 0.2456964287959354 (+0.0016887912028272223) | > avg_loss_disc_real_5: 0.24873429557807009 (+0.00137030427724541) | > avg_loss_0: 2.792662167213333 (-0.0043990813510514926) | > avg_grad_norm_0: tensor(14.0654, device='cuda:0') (tensor(-0.1659, device='cuda:0')) | > avg_loss_gen: 1.8430260839596602 (+0.011411062428649199) | > avg_loss_kl: 2.2442855163359297 (+0.016060800619527527) | > avg_loss_feat: 2.4900477906348 (+0.004384400139392675) | > avg_loss_mel: 17.79549798831135 (+0.00923962660240818) | > avg_loss_duration: 1.6098206227933858 (-0.010843520433130749) | > avg_amp_scaler: 228.95774647887325 (-27.042253521126753) | > avg_loss_1: 25.982678023862164 (+0.03025237607284126) | > avg_grad_norm_1: tensor(351.7946, device='cuda:0') (+tensor(20.1531, device='cuda:0'))  > EPOCH: 276/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:01:56)   --> TIME: 2025-02-25 11:02:10 -- STEP: 27/72 -- GLOBAL_STEP: 1019900 | > loss_disc: 2.782161235809326 (2.801482015185886) | > loss_disc_real_0: 0.2137894630432129 (0.21705284273182904) | > loss_disc_real_1: 0.24343135952949524 (0.2303117557808205) | > loss_disc_real_2: 0.23497338593006134 (0.23887662644739505) | > loss_disc_real_3: 0.22138267755508423 (0.2368299624434224) | > loss_disc_real_4: 0.2373630255460739 (0.24419695359689217) | > loss_disc_real_5: 0.24122904241085052 (0.2491511967447069) | > loss_0: 2.782161235809326 (2.801482015185886) | > grad_norm_0: tensor(7.6912, device='cuda:0') (tensor(13.0440, device='cuda:0')) | > loss_gen: 1.6757221221923828 (1.7891887073163633) | > loss_kl: 2.170173406600952 (2.1076843297040018) | > loss_feat: 2.5629143714904785 (2.45538427653136) | > loss_mel: 16.976619720458984 (17.474507720382128) | > loss_duration: 1.5523200035095215 (1.5695650709999933) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.9377498626709 (25.39633009168837) | > grad_norm_1: tensor(488.4175, device='cuda:0') (tensor(400.1156, device='cuda:0')) | > current_lr_0: 0.00019321725127742685 | > current_lr_1: 0.00019321725127742685 | > step_time: 0.4399 (0.4393117604432283) | > loader_time: 0.0086 (0.006999404342086227) --> EVAL PERFORMANCE | > avg_loader_time: 0.008631256264700016 (+0.00012693606631856806) | > avg_step_time: 0.4575521341511901 (+0.001254575353273224) | > avg_loss_disc: 2.7904079799920742 (-0.0022541872212586966) | > avg_loss_disc_real_0: 0.2104151123006579 (-0.004622718398000131) | > avg_loss_disc_real_1: 0.23216860877795958 (-0.004389804643644413) | > avg_loss_disc_real_2: 0.24001767832628437 (-0.00011308722093072987) | > avg_loss_disc_real_3: 0.23687897370734684 (-0.002300430351579702) | > avg_loss_disc_real_4: 0.2444767996039189 (-0.001219629192016486) | > avg_loss_disc_real_5: 0.2479733686212083 (-0.0007609269568617794) | > avg_loss_0: 2.7904079799920742 (-0.0022541872212586966) | > avg_grad_norm_0: tensor(13.7053, device='cuda:0') (tensor(-0.3602, device='cuda:0')) | > avg_loss_gen: 1.8113544507765433 (-0.03167163318311683) | > avg_loss_kl: 2.1917516144228655 (-0.052533901913064174) | > avg_loss_feat: 2.4743820378478136 (-0.015665752786986342) | > avg_loss_mel: 17.515570842044447 (-0.2799271462669033) | > avg_loss_duration: 1.6102394973728018 (+0.0004188745794160287) | > avg_amp_scaler: 128.0 (-100.95774647887325) | > avg_loss_1: 25.603298375304316 (-0.37937964855784756) | > avg_grad_norm_1: tensor(363.4019, device='cuda:0') (+tensor(11.6073, device='cuda:0'))  > EPOCH: 277/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:02:33)   --> TIME: 2025-02-25 11:02:37 -- STEP: 5/72 -- GLOBAL_STEP: 1019950 | > loss_disc: 2.8983447551727295 (2.788467264175415) | > loss_disc_real_0: 0.20723369717597961 (0.22822748720645905) | > loss_disc_real_1: 0.28536489605903625 (0.23769571185111998) | > loss_disc_real_2: 0.3047921061515808 (0.24581678211688995) | > loss_disc_real_3: 0.2931423485279083 (0.24154384136199952) | > loss_disc_real_4: 0.2679903507232666 (0.2440067708492279) | > loss_disc_real_5: 0.261992871761322 (0.2485962301492691) | > loss_0: 2.8983447551727295 (2.788467264175415) | > grad_norm_0: tensor(5.3768, device='cuda:0') (tensor(9.4447, device='cuda:0')) | > loss_gen: 1.8659263849258423 (1.8910948038101196) | > loss_kl: 2.291922092437744 (2.179504227638245) | > loss_feat: 2.5929911136627197 (2.7834585666656495) | > loss_mel: 18.041183471679688 (18.88722038269043) | > loss_duration: 1.5629318952560425 (1.5909151554107666) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.354955673217773 (27.332192993164064) | > grad_norm_1: tensor(221.1824, device='cuda:0') (tensor(616.1683, device='cuda:0')) | > current_lr_0: 0.00019319309912101717 | > current_lr_1: 0.00019319309912101717 | > step_time: 0.4299 (0.4348261833190918) | > loader_time: 0.0065 (0.007084560394287109)  --> TIME: 2025-02-25 11:03:00 -- STEP: 55/72 -- GLOBAL_STEP: 1020000 | > loss_disc: 2.880307674407959 (2.79002235585993) | > loss_disc_real_0: 0.31878066062927246 (0.2088257066228173) | > loss_disc_real_1: 0.21035991609096527 (0.23443972034887833) | > loss_disc_real_2: 0.23301582038402557 (0.2405153139071031) | > loss_disc_real_3: 0.23033370077610016 (0.24057011333378878) | > loss_disc_real_4: 0.2606184780597687 (0.2444324168291959) | > loss_disc_real_5: 0.26202860474586487 (0.2466374546289444) | > loss_0: 2.880307674407959 (2.79002235585993) | > grad_norm_0: tensor(27.7654, device='cuda:0') (tensor(11.5020, device='cuda:0')) | > loss_gen: 1.8980216979980469 (1.8262245871803977) | > loss_kl: 2.3412973880767822 (2.2308657299388543) | > loss_feat: 2.1835999488830566 (2.5872785264795475) | > loss_mel: 17.476953506469727 (17.92885024330832) | > loss_duration: 1.641789436340332 (1.601135947487571) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.54166030883789 (26.174355004050515) | > grad_norm_1: tensor(168.5566, device='cuda:0') (tensor(328.9650, device='cuda:0')) | > current_lr_0: 0.00019319309912101717 | > current_lr_1: 0.00019319309912101717 | > step_time: 0.4617 (0.4440908908843994) | > loader_time: 0.0098 (0.007983641190962358) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1020000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008675403997931682 (+4.4147733231665945e-05) | > avg_step_time: 0.46344861849932606 (+0.005896484348135944) | > avg_loss_disc: 2.797000703677325 (+0.006592723685250768) | > avg_loss_disc_real_0: 0.21282446006654016 (+0.0024093477658822526) | > avg_loss_disc_real_1: 0.23361798050537916 (+0.0014493717274195739) | > avg_loss_disc_real_2: 0.2418727436116044 (+0.0018550652853200345) | > avg_loss_disc_real_3: 0.24127476840791567 (+0.004395794700568834) | > avg_loss_disc_real_4: 0.24487443373236858 (+0.0003976341284496754) | > avg_loss_disc_real_5: 0.2459056083585175 (-0.0020677602626908154) | > avg_loss_0: 2.797000703677325 (+0.006592723685250768) | > avg_grad_norm_0: tensor(12.6429, device='cuda:0') (tensor(-1.0624, device='cuda:0')) | > avg_loss_gen: 1.8136175786945181 (+0.002263127917974783) | > avg_loss_kl: 2.2536766999204403 (+0.061925085497574806) | > avg_loss_feat: 2.5235636200703366 (+0.049181582222523) | > avg_loss_mel: 17.808260460974463 (+0.2926896189300159) | > avg_loss_duration: 1.61334151113537 (+0.0031020137625681343) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.012459848968074 (+0.4091614736637581) | > avg_grad_norm_1: tensor(330.3232, device='cuda:0') (tensor(-33.0788, device='cuda:0'))  > EPOCH: 278/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:03:14)   --> TIME: 2025-02-25 11:03:30 -- STEP: 33/72 -- GLOBAL_STEP: 1020050 | > loss_disc: 2.791811227798462 (2.814264926043424) | > loss_disc_real_0: 0.23801974952220917 (0.21974036752274542) | > loss_disc_real_1: 0.29091131687164307 (0.24598254218245996) | > loss_disc_real_2: 0.25800272822380066 (0.24245510137442386) | > loss_disc_real_3: 0.2491730898618698 (0.2438678682753534) | > loss_disc_real_4: 0.24709665775299072 (0.24824371211456525) | > loss_disc_real_5: 0.251845121383667 (0.2508495066202048) | > loss_0: 2.791811227798462 (2.814264926043424) | > grad_norm_0: tensor(7.8343, device='cuda:0') (tensor(12.7693, device='cuda:0')) | > loss_gen: 1.7531471252441406 (1.8583008773399121) | > loss_kl: 2.38885235786438 (2.257126414414608) | > loss_feat: 2.191300630569458 (2.4741678093418926) | > loss_mel: 17.490880966186523 (17.515310518669363) | > loss_duration: 1.6129387617111206 (1.583694262938066) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.437118530273438 (25.688599499789152) | > grad_norm_1: tensor(228.6786, device='cuda:0') (tensor(321.6461, device='cuda:0')) | > current_lr_0: 0.00019316894998362705 | > current_lr_1: 0.00019316894998362705 | > step_time: 0.4397 (0.43711827018044214) | > loader_time: 0.0076 (0.007179354176376805) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865201882913079 (-2.3385168800892434e-05) | > avg_step_time: 0.4556957936622727 (-0.0077528248370533626) | > avg_loss_disc: 2.8142175372217744 (+0.01721683354444936) | > avg_loss_disc_real_0: 0.21944040238437518 (+0.006615942317835022) | > avg_loss_disc_real_1: 0.24321772095183253 (+0.00959974044645337) | > avg_loss_disc_real_2: 0.23900167203285325 (-0.002871071578751161) | > avg_loss_disc_real_3: 0.23878625392074315 (-0.0024885144871725218) | > avg_loss_disc_real_4: 0.24575133369842045 (+0.0008768999660518639) | > avg_loss_disc_real_5: 0.24875685824474814 (+0.0028512498862306446) | > avg_loss_0: 2.8142175372217744 (+0.01721683354444936) | > avg_grad_norm_0: tensor(13.3974, device='cuda:0') (+tensor(0.7545, device='cuda:0')) | > avg_loss_gen: 1.8273698511257979 (+0.013752272431279744) | > avg_loss_kl: 2.281117571911342 (+0.027440871990901616) | > avg_loss_feat: 2.4272137762795034 (-0.09634984379083322) | > avg_loss_mel: 17.669053467226707 (-0.1392069937477558) | > avg_loss_duration: 1.6210279783732453 (+0.007686467237875361) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.825782453510122 (-0.18667739545795214) | > avg_grad_norm_1: tensor(364.3919, device='cuda:0') (+tensor(34.0688, device='cuda:0'))  > EPOCH: 279/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:03:50)   --> TIME: 2025-02-25 11:03:57 -- STEP: 11/72 -- GLOBAL_STEP: 1020100 | > loss_disc: 2.7976627349853516 (2.72229359366677) | > loss_disc_real_0: 0.22454530000686646 (0.20330741595138202) | > loss_disc_real_1: 0.2068944126367569 (0.22063678232106296) | > loss_disc_real_2: 0.21238058805465698 (0.23887723006985404) | > loss_disc_real_3: 0.23933906853199005 (0.23672757230021738) | > loss_disc_real_4: 0.23546215891838074 (0.24534581059759314) | > loss_disc_real_5: 0.23967090249061584 (0.24141293357719074) | > loss_0: 2.7976627349853516 (2.72229359366677) | > grad_norm_0: tensor(7.8941, device='cuda:0') (tensor(10.5956, device='cuda:0')) | > loss_gen: 1.7436952590942383 (1.8558118343353271) | > loss_kl: 2.2217538356781006 (2.191658475182273) | > loss_feat: 2.4412038326263428 (2.593843525106257) | > loss_mel: 16.712825775146484 (17.708487250588156) | > loss_duration: 1.5826505422592163 (1.5926651196046309) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.702131271362305 (25.94246621565385) | > grad_norm_1: tensor(90.8495, device='cuda:0') (tensor(362.1129, device='cuda:0')) | > current_lr_0: 0.0001931448038648791 | > current_lr_1: 0.0001931448038648791 | > step_time: 0.422 (0.43094886432994495) | > loader_time: 0.0057 (0.005950104106556286)  --> TIME: 2025-02-25 11:04:20 -- STEP: 61/72 -- GLOBAL_STEP: 1020150 | > loss_disc: 2.8363776206970215 (2.7842923109648647) | > loss_disc_real_0: 0.2816372215747833 (0.20863304397121804) | > loss_disc_real_1: 0.22899582982063293 (0.23467733161371263) | > loss_disc_real_2: 0.21895051002502441 (0.24072363166535488) | > loss_disc_real_3: 0.22813977301120758 (0.24122149289631453) | > loss_disc_real_4: 0.22408124804496765 (0.24661400865335933) | > loss_disc_real_5: 0.2559405267238617 (0.24625285800363197) | > loss_0: 2.8363776206970215 (2.7842923109648647) | > grad_norm_0: tensor(14.4701, device='cuda:0') (tensor(14.2870, device='cuda:0')) | > loss_gen: 1.8622361421585083 (1.8397955425450059) | > loss_kl: 2.32146954536438 (2.2189053863775543) | > loss_feat: 2.5215585231781006 (2.5501792665387764) | > loss_mel: 18.210325241088867 (17.699357048409883) | > loss_duration: 1.672136664390564 (1.6084917588312118) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.587724685668945 (25.916728942120663) | > grad_norm_1: tensor(336.3271, device='cuda:0') (tensor(313.5676, device='cuda:0')) | > current_lr_0: 0.0001931448038648791 | > current_lr_1: 0.0001931448038648791 | > step_time: 0.4555 (0.4458445447390197) | > loader_time: 0.011 (0.008095948422541382) --> EVAL PERFORMANCE | > avg_loader_time: 0.008572010926797356 (-8.000790233343366e-05) | > avg_step_time: 0.4532055686896955 (-0.002490224972577171) | > avg_loss_disc: 2.78272934027121 (-0.03148819695056426) | > avg_loss_disc_real_0: 0.21030714142490442 (-0.009133260959470757) | > avg_loss_disc_real_1: 0.2338487686405719 (-0.009368952311260614) | > avg_loss_disc_real_2: 0.23962916202948126 (+0.0006274899966280112) | > avg_loss_disc_real_3: 0.24094131903749116 (+0.0021550651167480117) | > avg_loss_disc_real_4: 0.24581484475605925 (+6.351105763879872e-05) | > avg_loss_disc_real_5: 0.24576473173121333 (-0.0029921265135348096) | > avg_loss_0: 2.78272934027121 (-0.03148819695056426) | > avg_grad_norm_0: tensor(14.2225, device='cuda:0') (+tensor(0.8251, device='cuda:0')) | > avg_loss_gen: 1.8351348974335362 (+0.0077650463077383325) | > avg_loss_kl: 2.243099535015268 (-0.03801803689607386) | > avg_loss_feat: 2.5344409573246054 (+0.10722718104510198) | > avg_loss_mel: 17.77532472744794 (+0.10627126022123434) | > avg_loss_duration: 1.6175462444063642 (-0.0034817339668811176) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.005546301183568 (+0.17976384767344555) | > avg_grad_norm_1: tensor(308.9021, device='cuda:0') (tensor(-55.4898, device='cuda:0'))  > EPOCH: 280/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:04:27)   --> TIME: 2025-02-25 11:04:47 -- STEP: 39/72 -- GLOBAL_STEP: 1020200 | > loss_disc: 2.819833993911743 (2.8119270434746375) | > loss_disc_real_0: 0.23893409967422485 (0.21401299287875494) | > loss_disc_real_1: 0.24042974412441254 (0.23158390246904814) | > loss_disc_real_2: 0.2617775797843933 (0.2527343581120172) | > loss_disc_real_3: 0.27249112725257874 (0.24564991738551703) | > loss_disc_real_4: 0.2620982825756073 (0.24931404414849404) | > loss_disc_real_5: 0.24435940384864807 (0.248603228193063) | > loss_0: 2.819833993911743 (2.8119270434746375) | > grad_norm_0: tensor(5.9576, device='cuda:0') (tensor(12.4545, device='cuda:0')) | > loss_gen: 1.746408462524414 (1.8406270161653175) | > loss_kl: 2.3919780254364014 (2.2822324648881573) | > loss_feat: 2.7147397994995117 (2.545242725274502) | > loss_mel: 19.01177215576172 (17.955237951034167) | > loss_duration: 1.6107196807861328 (1.5923172235488892) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.475618362426758 (26.215657552083332) | > grad_norm_1: tensor(476.7455, device='cuda:0') (tensor(240.3348, device='cuda:0')) | > current_lr_0: 0.00019312066076439597 | > current_lr_1: 0.00019312066076439597 | > step_time: 0.4573 (0.4434802715594952) | > loader_time: 0.0085 (0.0073355161226712744) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626269622587824 (+5.425869579046838e-05) | > avg_step_time: 0.45895081506648533 (+0.005745246376789803) | > avg_loss_disc: 2.807354943853029 (+0.02462560358181909) | > avg_loss_disc_real_0: 0.2175134761442601 (+0.007206334719355667) | > avg_loss_disc_real_1: 0.23019250744665173 (-0.0036562611939201795) | > avg_loss_disc_real_2: 0.2465086518458916 (+0.006879489816410345) | > avg_loss_disc_real_3: 0.24149739847216808 (+0.0005560794346769204) | > avg_loss_disc_real_4: 0.2479971706027716 (+0.002182325846712352) | > avg_loss_disc_real_5: 0.2494219547845949 (+0.003657223053381564) | > avg_loss_0: 2.807354943853029 (+0.02462560358181909) | > avg_grad_norm_0: tensor(13.0812, device='cuda:0') (tensor(-1.1413, device='cuda:0')) | > avg_loss_gen: 1.835390893506332 (+0.0002559960727959165) | > avg_loss_kl: 2.282790561796915 (+0.03969102678164704) | > avg_loss_feat: 2.502540128331788 (-0.03190082899281732) | > avg_loss_mel: 17.763950871749675 (-0.011373855698266766) | > avg_loss_duration: 1.6137793299178003 (-0.0037669144885639216) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.99845203883211 (-0.007094262351458269) | > avg_grad_norm_1: tensor(278.4073, device='cuda:0') (tensor(-30.4948, device='cuda:0'))  > EPOCH: 281/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:05:05)   --> TIME: 2025-02-25 11:05:14 -- STEP: 17/72 -- GLOBAL_STEP: 1020250 | > loss_disc: 2.9045443534851074 (2.801492550793816) | > loss_disc_real_0: 0.20506907999515533 (0.19684391687898076) | > loss_disc_real_1: 0.2951095700263977 (0.2353585546507555) | > loss_disc_real_2: 0.2741856575012207 (0.24116895391660578) | > loss_disc_real_3: 0.24318072199821472 (0.24195413554415984) | > loss_disc_real_4: 0.22229059040546417 (0.2429544531247195) | > loss_disc_real_5: 0.25699034333229065 (0.24504025192821727) | > loss_0: 2.9045443534851074 (2.801492550793816) | > grad_norm_0: tensor(7.9325, device='cuda:0') (tensor(14.3066, device='cuda:0')) | > loss_gen: 1.818892240524292 (1.8063819618786083) | > loss_kl: 2.308643102645874 (2.131818511906792) | > loss_feat: 2.3854730129241943 (2.535334587097168) | > loss_mel: 17.747392654418945 (17.377751294304343) | > loss_duration: 1.6014727354049683 (1.5714533258886898) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.861873626708984 (25.422739814309512) | > grad_norm_1: tensor(560.2899, device='cuda:0') (tensor(453.3899, device='cuda:0')) | > current_lr_0: 0.0001930965206818004 | > current_lr_1: 0.0001930965206818004 | > step_time: 0.4322 (0.44338389003978057) | > loader_time: 0.0061 (0.0065922456629136025)  --> TIME: 2025-02-25 11:05:38 -- STEP: 67/72 -- GLOBAL_STEP: 1020300 | > loss_disc: 2.7623887062072754 (2.8103368033224077) | > loss_disc_real_0: 0.25228071212768555 (0.20534876344808892) | > loss_disc_real_1: 0.20374000072479248 (0.23365911707949283) | > loss_disc_real_2: 0.24424929916858673 (0.24235532354952685) | > loss_disc_real_3: 0.21455013751983643 (0.2428091086113631) | > loss_disc_real_4: 0.2594526708126068 (0.24558592398664844) | > loss_disc_real_5: 0.2523883283138275 (0.2501393592179711) | > loss_0: 2.7623887062072754 (2.8103368033224077) | > grad_norm_0: tensor(13.6044, device='cuda:0') (tensor(13.3946, device='cuda:0')) | > loss_gen: 1.9124383926391602 (1.809305666097954) | > loss_kl: 2.5864038467407227 (2.2506345635029814) | > loss_feat: 2.469252347946167 (2.4944724698564893) | > loss_mel: 19.412635803222656 (17.84968450176182) | > loss_duration: 1.7029858827590942 (1.6090463257547636) | > amp_scaler: 128.0 (128.0) | > loss_1: 28.083715438842773 (26.013143397089262) | > grad_norm_1: tensor(268.0295, device='cuda:0') (tensor(386.2134, device='cuda:0')) | > current_lr_0: 0.0001930965206818004 | > current_lr_1: 0.0001930965206818004 | > step_time: 0.5422 (0.4585699572492002) | > loader_time: 0.0121 (0.008435562475403743) --> EVAL PERFORMANCE | > avg_loader_time: 0.008601420362230758 (-2.4849260357066166e-05) | > avg_step_time: 0.4621923036978278 (+0.003241488631342493) | > avg_loss_disc: 2.8099975451617176 (+0.0026426013086884304) | > avg_loss_disc_real_0: 0.2045639227603523 (-0.012949553383907797) | > avg_loss_disc_real_1: 0.23529886171011857 (+0.005106354263466839) | > avg_loss_disc_real_2: 0.24128786778785813 (-0.005220784058033473) | > avg_loss_disc_real_3: 0.24237343983750947 (+0.0008760413653413901) | > avg_loss_disc_real_4: 0.24532484630463827 (-0.0026723242981333273) | > avg_loss_disc_real_5: 0.24984030803324467 (+0.00041835324864977763) | > avg_loss_0: 2.8099975451617176 (+0.0026426013086884304) | > avg_grad_norm_0: tensor(13.8416, device='cuda:0') (+tensor(0.7604, device='cuda:0')) | > avg_loss_gen: 1.802567720413208 (-0.0328231730931241) | > avg_loss_kl: 2.2627838967551654 (-0.02000666504174964) | > avg_loss_feat: 2.47540391498888 (-0.027136213342908277) | > avg_loss_mel: 17.837515226552185 (+0.07356435480251022) | > avg_loss_duration: 1.6122243270068104 (-0.0015550029109898666) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.990494956432933 (-0.007957082399176585) | > avg_grad_norm_1: tensor(385.5336, device='cuda:0') (+tensor(107.1263, device='cuda:0'))  > EPOCH: 282/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:05:42)   --> TIME: 2025-02-25 11:06:05 -- STEP: 45/72 -- GLOBAL_STEP: 1020350 | > loss_disc: 2.8807685375213623 (2.7886585818396674) | > loss_disc_real_0: 0.16039986908435822 (0.2158219873905182) | > loss_disc_real_1: 0.24586425721645355 (0.2298272838195165) | > loss_disc_real_2: 0.2236216813325882 (0.23983646002080705) | > loss_disc_real_3: 0.26184239983558655 (0.23909691572189332) | > loss_disc_real_4: 0.2710021734237671 (0.24369023872746362) | > loss_disc_real_5: 0.2735334038734436 (0.2477145748005973) | > loss_0: 2.8807685375213623 (2.7886585818396674) | > grad_norm_0: tensor(13.7835, device='cuda:0') (tensor(10.6108, device='cuda:0')) | > loss_gen: 1.79829740524292 (1.8243572367562189) | > loss_kl: 2.2031795978546143 (2.2809609439637923) | > loss_feat: 2.0071346759796143 (2.5082947466108534) | > loss_mel: 18.213844299316406 (17.880350367228193) | > loss_duration: 1.62434983253479 (1.60727858543396) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.846805572509766 (26.10124198065864) | > grad_norm_1: tensor(119.1448, device='cuda:0') (tensor(255.8375, device='cuda:0')) | > current_lr_0: 0.00019307238361671516 | > current_lr_1: 0.00019307238361671516 | > step_time: 0.4813 (0.44823340839809844) | > loader_time: 0.0092 (0.007705365286933052) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754619410340213 (+0.00015319904810945535) | > avg_step_time: 0.46108469828753407 (-0.001107605410293755) | > avg_loss_disc: 2.7959073765177123 (-0.014090168644005363) | > avg_loss_disc_real_0: 0.21464601998597804 (+0.010082097225625752) | > avg_loss_disc_real_1: 0.23255972648170634 (-0.0027391352284122372) | > avg_loss_disc_real_2: 0.2412158234018675 (-7.204438599062168e-05) | > avg_loss_disc_real_3: 0.23977096756579172 (-0.002602472271717754) | > avg_loss_disc_real_4: 0.2442050017941166 (-0.0011198445105216615) | > avg_loss_disc_real_5: 0.24800604141094315 (-0.001834266622301517) | > avg_loss_0: 2.7959073765177123 (-0.014090168644005363) | > avg_grad_norm_0: tensor(12.0250, device='cuda:0') (tensor(-1.8167, device='cuda:0')) | > avg_loss_gen: 1.8213337424775244 (+0.018766022064316434) | > avg_loss_kl: 2.313338546685769 (+0.050554649930603546) | > avg_loss_feat: 2.4581728720329177 (-0.017231042955962117) | > avg_loss_mel: 17.8583200078615 (+0.020804781309315956) | > avg_loss_duration: 1.6216499553599828 (+0.0094256283531724) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.072815183182836 (+0.08232022674990347) | > avg_grad_norm_1: tensor(253.2197, device='cuda:0') (tensor(-132.3139, device='cuda:0'))  > EPOCH: 283/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:06:20)   --> TIME: 2025-02-25 11:06:32 -- STEP: 23/72 -- GLOBAL_STEP: 1020400 | > loss_disc: 2.7806832790374756 (2.8000369486601455) | > loss_disc_real_0: 0.233650341629982 (0.20050343935904297) | > loss_disc_real_1: 0.23621930181980133 (0.2356429864530978) | > loss_disc_real_2: 0.22680526971817017 (0.2391102469485739) | > loss_disc_real_3: 0.2438352406024933 (0.23342829424401987) | > loss_disc_real_4: 0.22913441061973572 (0.2458407859439435) | > loss_disc_real_5: 0.23252244293689728 (0.2440165462701217) | > loss_0: 2.7806832790374756 (2.8000369486601455) | > grad_norm_0: tensor(7.2236, device='cuda:0') (tensor(17.0364, device='cuda:0')) | > loss_gen: 1.8714239597320557 (1.7884825830874236) | > loss_kl: 2.2870030403137207 (2.230852194454359) | > loss_feat: 2.670579671859741 (2.4599291552668032) | > loss_mel: 18.016704559326172 (17.595446379288383) | > loss_duration: 1.6083977222442627 (1.5750274502712747) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.45410919189453 (25.64973781419837) | > grad_norm_1: tensor(192.3038, device='cuda:0') (tensor(290.3983, device='cuda:0')) | > current_lr_0: 0.00019304824956876306 | > current_lr_1: 0.00019304824956876306 | > step_time: 0.4527 (0.4370576506075652) | > loader_time: 0.0066 (0.006694669308869735) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666740336888275 (-8.787907345193802e-05) | > avg_step_time: 0.45466041564941406 (-0.006424282638120005) | > avg_loss_disc: 2.815148152096171 (+0.019240775578458713) | > avg_loss_disc_real_0: 0.206244347905609 (-0.008401672080369044) | > avg_loss_disc_real_1: 0.23739757462286612 (+0.004837848141159784) | > avg_loss_disc_real_2: 0.24103089558406615 (-0.0001849278178013558) | > avg_loss_disc_real_3: 0.2403572762096432 (+0.0005863086438514797) | > avg_loss_disc_real_4: 0.24579673669707608 (+0.0015917349029594685) | > avg_loss_disc_real_5: 0.24674058359273723 (-0.0012654578182059173) | > avg_loss_0: 2.815148152096171 (+0.019240775578458713) | > avg_grad_norm_0: tensor(15.5373, device='cuda:0') (+tensor(3.5123, device='cuda:0')) | > avg_loss_gen: 1.794499869077978 (-0.02683387339954635) | > avg_loss_kl: 2.280741511935919 (-0.032597034749850096) | > avg_loss_feat: 2.4164363045088004 (-0.04173656752411725) | > avg_loss_mel: 17.6119119617301 (-0.24640804613139977) | > avg_loss_duration: 1.6130929228285669 (-0.008557032531415931) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.716682756450815 (-0.35613242673202095) | > avg_grad_norm_1: tensor(294.4210, device='cuda:0') (+tensor(41.2013, device='cuda:0'))  > EPOCH: 284/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:06:57)   --> TIME: 2025-02-25 11:06:59 -- STEP: 1/72 -- GLOBAL_STEP: 1020450 | > loss_disc: 2.8014683723449707 (2.8014683723449707) | > loss_disc_real_0: 0.2588304877281189 (0.2588304877281189) | > loss_disc_real_1: 0.20403538644313812 (0.20403538644313812) | > loss_disc_real_2: 0.21824227273464203 (0.21824227273464203) | > loss_disc_real_3: 0.23391078412532806 (0.23391078412532806) | > loss_disc_real_4: 0.2274361103773117 (0.2274361103773117) | > loss_disc_real_5: 0.2231227606534958 (0.2231227606534958) | > loss_0: 2.8014683723449707 (2.8014683723449707) | > grad_norm_0: tensor(11.1132, device='cuda:0') (tensor(11.1132, device='cuda:0')) | > loss_gen: 1.8736553192138672 (1.8736553192138672) | > loss_kl: 2.273444414138794 (2.273444414138794) | > loss_feat: 2.7396600246429443 (2.7396600246429443) | > loss_mel: 17.966266632080078 (17.966266632080078) | > loss_duration: 1.6285958290100098 (1.6285958290100098) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.48162078857422 (26.48162078857422) | > grad_norm_1: tensor(69.8003, device='cuda:0') (tensor(69.8003, device='cuda:0')) | > current_lr_0: 0.00019302411853756697 | > current_lr_1: 0.00019302411853756697 | > step_time: 0.4427 (0.4426865577697754) | > loader_time: 0.0058 (0.0057830810546875)  --> TIME: 2025-02-25 11:07:22 -- STEP: 51/72 -- GLOBAL_STEP: 1020500 | > loss_disc: 2.6588010787963867 (2.78712770985622) | > loss_disc_real_0: 0.13610345125198364 (0.2092864758828107) | > loss_disc_real_1: 0.23351719975471497 (0.23899483154801762) | > loss_disc_real_2: 0.23289954662322998 (0.24044520217998355) | > loss_disc_real_3: 0.21072790026664734 (0.23794083501778396) | > loss_disc_real_4: 0.21932175755500793 (0.24164045470602372) | > loss_disc_real_5: 0.21773310005664825 (0.2457232618448781) | > loss_0: 2.6588010787963867 (2.78712770985622) | > grad_norm_0: tensor(14.8260, device='cuda:0') (tensor(15.5449, device='cuda:0')) | > loss_gen: 1.8497098684310913 (1.8431756005567663) | > loss_kl: 2.2456753253936768 (2.210519458733353) | > loss_feat: 2.8650994300842285 (2.5790474274579216) | > loss_mel: 16.804224014282227 (17.406280424080645) | > loss_duration: 1.6333255767822266 (1.5966040573868097) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.398035049438477 (25.63562677420822) | > grad_norm_1: tensor(813.7447, device='cuda:0') (tensor(395.3469, device='cuda:0')) | > current_lr_0: 0.00019302411853756697 | > current_lr_1: 0.00019302411853756697 | > step_time: 0.4523 (0.44956716836667526) | > loader_time: 0.0089 (0.007859702203788009) --> EVAL PERFORMANCE | > avg_loader_time: 0.008716240735121176 (+4.9500398232900714e-05) | > avg_step_time: 0.4610876331866627 (+0.0064272175372486195) | > avg_loss_disc: 2.7805378604942645 (-0.03461029160190643) | > avg_loss_disc_real_0: 0.21300296065673022 (+0.006758612751121218) | > avg_loss_disc_real_1: 0.23636110651660974 (-0.0010364681062563774) | > avg_loss_disc_real_2: 0.23908355601236853 (-0.001947339571697626) | > avg_loss_disc_real_3: 0.23795552740634326 (-0.0024017488032999357) | > avg_loss_disc_real_4: 0.24209082021679676 (-0.00370591648027932) | > avg_loss_disc_real_5: 0.24591262638568878 (-0.000827957207048452) | > avg_loss_0: 2.7805378604942645 (-0.03461029160190643) | > avg_grad_norm_0: tensor(14.7912, device='cuda:0') (tensor(-0.7461, device='cuda:0')) | > avg_loss_gen: 1.8440674862391513 (+0.049567617161173194) | > avg_loss_kl: 2.2072850197133884 (-0.07345649222253048) | > avg_loss_feat: 2.5605941356067925 (+0.14415783109799207) | > avg_loss_mel: 17.3657012052939 (-0.24621075643620216) | > avg_loss_duration: 1.6104571063753583 (-0.0026358164532085393) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.58810489278444 (-0.1285778636663757) | > avg_grad_norm_1: tensor(408.6960, device='cuda:0') (+tensor(114.2751, device='cuda:0'))  > EPOCH: 285/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:07:34)   --> TIME: 2025-02-25 11:07:49 -- STEP: 29/72 -- GLOBAL_STEP: 1020550 | > loss_disc: 2.8405089378356934 (2.8161436607097756) | > loss_disc_real_0: 0.13364079594612122 (0.21160851624505272) | > loss_disc_real_1: 0.23961026966571808 (0.2412911586720368) | > loss_disc_real_2: 0.23602533340454102 (0.2414754608581806) | > loss_disc_real_3: 0.26003825664520264 (0.23672419272620102) | > loss_disc_real_4: 0.2557547092437744 (0.2466681214242146) | > loss_disc_real_5: 0.25945624709129333 (0.24440533385194582) | > loss_0: 2.8405089378356934 (2.8161436607097756) | > grad_norm_0: tensor(20.3343, device='cuda:0') (tensor(11.2408, device='cuda:0')) | > loss_gen: 1.674424171447754 (1.8168465844516097) | > loss_kl: 2.264003038406372 (2.19234326790119) | > loss_feat: 2.0849339962005615 (2.5716357148926834) | > loss_mel: 17.579126358032227 (17.726330790026434) | > loss_duration: 1.5938698053359985 (1.5825689044491997) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.19635581970215 (25.88972499452788) | > grad_norm_1: tensor(219.4471, device='cuda:0') (tensor(323.4081, device='cuda:0')) | > current_lr_0: 0.00019299999052274977 | > current_lr_1: 0.00019299999052274977 | > step_time: 0.4368 (0.43726762409867914) | > loader_time: 0.0073 (0.006991633053483634) --> EVAL PERFORMANCE | > avg_loader_time: 0.008561100758297344 (-0.0001551399768238322) | > avg_step_time: 0.4578125913378219 (-0.0032750418488408095) | > avg_loss_disc: 2.8105540342733883 (+0.0300161737791238) | > avg_loss_disc_real_0: 0.21194155388314959 (-0.001061406773580631) | > avg_loss_disc_real_1: 0.2377973624937971 (+0.0014362559771873562) | > avg_loss_disc_real_2: 0.23975791121032877 (+0.0006743551979602469) | > avg_loss_disc_real_3: 0.240624166290525 (+0.002668638884181729) | > avg_loss_disc_real_4: 0.24540249898400104 (+0.0033116787672042847) | > avg_loss_disc_real_5: 0.24749015649439585 (+0.0015775301087070648) | > avg_loss_0: 2.8105540342733883 (+0.0300161737791238) | > avg_grad_norm_0: tensor(12.5675, device='cuda:0') (tensor(-2.2238, device='cuda:0')) | > avg_loss_gen: 1.8218059388684555 (-0.022261547370695745) | > avg_loss_kl: 2.237045147049595 (+0.029760127336206477) | > avg_loss_feat: 2.5235810363796394 (-0.03701309922715312) | > avg_loss_mel: 17.597391853869794 (+0.2316906485758956) | > avg_loss_duration: 1.6137764067717 (+0.003319300396341651) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.793600270445918 (+0.20549537766147807) | > avg_grad_norm_1: tensor(356.1026, device='cuda:0') (tensor(-52.5935, device='cuda:0'))  > EPOCH: 286/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:08:11)   --> TIME: 2025-02-25 11:08:16 -- STEP: 7/72 -- GLOBAL_STEP: 1020600 | > loss_disc: 2.7591915130615234 (2.842125177383423) | > loss_disc_real_0: 0.1881476789712906 (0.22777685948780604) | > loss_disc_real_1: 0.22361969947814941 (0.23570940324238368) | > loss_disc_real_2: 0.2569216787815094 (0.24535381581102098) | > loss_disc_real_3: 0.239326611161232 (0.24375379298414504) | > loss_disc_real_4: 0.2505829334259033 (0.24574083941323416) | > loss_disc_real_5: 0.23662011325359344 (0.24842484721115657) | > loss_0: 2.7591915130615234 (2.842125177383423) | > grad_norm_0: tensor(6.1351, device='cuda:0') (tensor(12.4225, device='cuda:0')) | > loss_gen: 1.8375567197799683 (1.7950923953737532) | > loss_kl: 2.1788442134857178 (2.271514994757516) | > loss_feat: 2.653348684310913 (2.392112527574812) | > loss_mel: 18.566753387451172 (18.52094050816127) | > loss_duration: 1.5903089046478271 (1.5988598380770003) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.826812744140625 (26.57852063860212) | > grad_norm_1: tensor(216.4300, device='cuda:0') (tensor(291.3447, device='cuda:0')) | > current_lr_0: 0.0001929758655239344 | > current_lr_1: 0.0001929758655239344 | > step_time: 0.4444 (0.4366693837302072) | > loader_time: 0.0055 (0.006713458469935826)  --> TIME: 2025-02-25 11:08:39 -- STEP: 57/72 -- GLOBAL_STEP: 1020650 | > loss_disc: 2.8037302494049072 (2.803103626820079) | > loss_disc_real_0: 0.34086596965789795 (0.21526630980926648) | > loss_disc_real_1: 0.2612302601337433 (0.23901592665597013) | > loss_disc_real_2: 0.2578040063381195 (0.2374542849628549) | > loss_disc_real_3: 0.2605208158493042 (0.24034585633821653) | > loss_disc_real_4: 0.24994373321533203 (0.24577482201551137) | > loss_disc_real_5: 0.24578377604484558 (0.24970562756061554) | > loss_0: 2.8037302494049072 (2.803103626820079) | > grad_norm_0: tensor(25.6601, device='cuda:0') (tensor(12.3322, device='cuda:0')) | > loss_gen: 1.9963186979293823 (1.842150746730336) | > loss_kl: 2.3443593978881836 (2.291157090873049) | > loss_feat: 2.4540483951568604 (2.5344668354904436) | > loss_mel: 17.136615753173828 (18.060648667184932) | > loss_duration: 1.6939367055892944 (1.613228910847714) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.625280380249023 (26.34165218420196) | > grad_norm_1: tensor(468.2209, device='cuda:0') (tensor(337.7690, device='cuda:0')) | > current_lr_0: 0.0001929758655239344 | > current_lr_1: 0.0001929758655239344 | > step_time: 0.455 (0.4450340187340452) | > loader_time: 0.0098 (0.008136276613202013) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754226523385922 (+0.00019312576508857815) | > avg_step_time: 0.45538042632626813 (-0.002432165011553744) | > avg_loss_disc: 2.8062547898628343 (-0.004299244410554071) | > avg_loss_disc_real_0: 0.2170380372816408 (+0.005096483398491225) | > avg_loss_disc_real_1: 0.24045623650013562 (+0.0026588740063385186) | > avg_loss_disc_real_2: 0.2400742378872885 (+0.00031632667695971617) | > avg_loss_disc_real_3: 0.2422792913628296 (+0.001655125072304614) | > avg_loss_disc_real_4: 0.24508243547358982 (-0.0003200635104112226) | > avg_loss_disc_real_5: 0.24796513562471095 (+0.0004749791303151063) | > avg_loss_0: 2.8062547898628343 (-0.004299244410554071) | > avg_grad_norm_0: tensor(13.1923, device='cuda:0') (+tensor(0.6249, device='cuda:0')) | > avg_loss_gen: 1.83874421556231 (+0.016938276693854393) | > avg_loss_kl: 2.3047157576386352 (+0.06767061058904034) | > avg_loss_feat: 2.512634230331635 (-0.010946806048004465) | > avg_loss_mel: 18.00564706829232 (+0.40825521442252466) | > avg_loss_duration: 1.623073354573317 (+0.00929694780161694) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.284814619682205 (+0.49121434923628726) | > avg_grad_norm_1: tensor(338.1549, device='cuda:0') (tensor(-17.9477, device='cuda:0'))  > EPOCH: 287/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:08:48)   --> TIME: 2025-02-25 11:09:07 -- STEP: 35/72 -- GLOBAL_STEP: 1020700 | > loss_disc: 2.8853960037231445 (2.7948120525905065) | > loss_disc_real_0: 0.2293282449245453 (0.21990718713828494) | > loss_disc_real_1: 0.21357783675193787 (0.22788896092346736) | > loss_disc_real_2: 0.2518014907836914 (0.24105151551110404) | > loss_disc_real_3: 0.23589210212230682 (0.23863774197442192) | > loss_disc_real_4: 0.2690982222557068 (0.2476675455059324) | > loss_disc_real_5: 0.2577543258666992 (0.24776992542403084) | > loss_0: 2.8853960037231445 (2.7948120525905065) | > grad_norm_0: tensor(5.8019, device='cuda:0') (tensor(14.2305, device='cuda:0')) | > loss_gen: 1.6430617570877075 (1.829705466542925) | > loss_kl: 2.244800090789795 (2.2358912808554514) | > loss_feat: 2.2172203063964844 (2.52594815662929) | > loss_mel: 18.720733642578125 (18.034906441824777) | > loss_duration: 1.636389970779419 (1.5995673043387277) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.462207794189453 (26.226018742152622) | > grad_norm_1: tensor(201.5701, device='cuda:0') (tensor(313.3788, device='cuda:0')) | > current_lr_0: 0.0001929517435407439 | > current_lr_1: 0.0001929517435407439 | > step_time: 0.4423 (0.44104040690830776) | > loader_time: 0.0088 (0.00717942374093192) --> EVAL PERFORMANCE | > avg_loader_time: 0.008765455702660793 (+1.1229179274870743e-05) | > avg_step_time: 0.458877123577494 (+0.0034966972512258665) | > avg_loss_disc: 2.7868419432304274 (-0.019412846632406833) | > avg_loss_disc_real_0: 0.2112805842303894 (-0.0057574530512514155) | > avg_loss_disc_real_1: 0.23375123781217655 (-0.006704998687959068) | > avg_loss_disc_real_2: 0.23909341932182582 (-0.000980818565462671) | > avg_loss_disc_real_3: 0.2380698562507898 (-0.0042094351120398055) | > avg_loss_disc_real_4: 0.24494325757866175 (-0.00013917789492806643) | > avg_loss_disc_real_5: 0.24510678481048262 (-0.002858350814228333) | > avg_loss_0: 2.7868419432304274 (-0.019412846632406833) | > avg_grad_norm_0: tensor(13.7070, device='cuda:0') (+tensor(0.5147, device='cuda:0')) | > avg_loss_gen: 1.8267527768309688 (-0.01199143873134112) | > avg_loss_kl: 2.2501442919314742 (-0.05457146570716098) | > avg_loss_feat: 2.4859844748402984 (-0.026649755491336524) | > avg_loss_mel: 17.7831922987817 (-0.2224547695106196) | > avg_loss_duration: 1.6183087641084697 (-0.004764590464847229) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.964382708912165 (-0.32043191077003996) | > avg_grad_norm_1: tensor(344.0905, device='cuda:0') (+tensor(5.9356, device='cuda:0'))  > EPOCH: 288/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:09:26)   --> TIME: 2025-02-25 11:09:34 -- STEP: 13/72 -- GLOBAL_STEP: 1020750 | > loss_disc: 2.774120569229126 (2.7666532626518836) | > loss_disc_real_0: 0.17831304669380188 (0.19112380765951598) | > loss_disc_real_1: 0.26954934000968933 (0.22569645597384527) | > loss_disc_real_2: 0.25365617871284485 (0.2425821079657628) | > loss_disc_real_3: 0.24003256857395172 (0.2423600393992204) | > loss_disc_real_4: 0.23049208521842957 (0.2458680489888558) | > loss_disc_real_5: 0.21354374289512634 (0.24647497901549706) | > loss_0: 2.774120569229126 (2.7666532626518836) | > grad_norm_0: tensor(8.6249, device='cuda:0') (tensor(7.7928, device='cuda:0')) | > loss_gen: 1.9255040884017944 (1.8390850562315721) | > loss_kl: 2.23016357421875 (2.2493808269500732) | > loss_feat: 2.6341946125030518 (2.6184284687042236) | > loss_mel: 16.99327278137207 (17.635544850276066) | > loss_duration: 1.4975531101226807 (1.584412198800307) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.280689239501953 (25.92685141930213) | > grad_norm_1: tensor(352.5931, device='cuda:0') (tensor(312.7058, device='cuda:0')) | > current_lr_0: 0.00019292762457280132 | > current_lr_1: 0.00019292762457280132 | > step_time: 0.4724 (0.43620564387394833) | > loader_time: 0.0058 (0.006740936866173377)  --> TIME: 2025-02-25 11:09:57 -- STEP: 63/72 -- GLOBAL_STEP: 1020800 | > loss_disc: 2.8252477645874023 (2.7755329571073015) | > loss_disc_real_0: 0.14003831148147583 (0.2035187291247504) | > loss_disc_real_1: 0.23056469857692719 (0.23114258074571217) | > loss_disc_real_2: 0.2334844470024109 (0.2390105648646279) | > loss_disc_real_3: 0.2378670573234558 (0.2392781050432296) | > loss_disc_real_4: 0.23487338423728943 (0.2430654667199604) | > loss_disc_real_5: 0.25305184721946716 (0.24603149720600673) | > loss_0: 2.8252477645874023 (2.7755329571073015) | > grad_norm_0: tensor(22.3242, device='cuda:0') (tensor(11.7707, device='cuda:0')) | > loss_gen: 1.684848666191101 (1.8202816910213895) | > loss_kl: 2.172142744064331 (2.2616374076358854) | > loss_feat: 2.220353364944458 (2.5586391857692177) | > loss_mel: 17.136985778808594 (17.5218139527336) | > loss_duration: 1.5889530181884766 (1.6086600545852903) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.80328369140625 (25.771032242547896) | > grad_norm_1: tensor(41.9863, device='cuda:0') (tensor(365.9814, device='cuda:0')) | > current_lr_0: 0.00019292762457280132 | > current_lr_1: 0.00019292762457280132 | > step_time: 0.4989 (0.4476762574816507) | > loader_time: 0.0113 (0.008307313162183005) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868907780714438 (-7.637789551641176e-05) | > avg_step_time: 0.4552240707504917 (-0.00365305282700229) | > avg_loss_disc: 2.775982803022358 (-0.010859140208069551) | > avg_loss_disc_real_0: 0.2041095921271284 (-0.007170992103260998) | > avg_loss_disc_real_1: 0.23101743656984516 (-0.002733801242331385) | > avg_loss_disc_real_2: 0.2389654750135583 (-0.00012794430826751357) | > avg_loss_disc_real_3: 0.2387340398321689 (+0.0006641835813790964) | > avg_loss_disc_real_4: 0.2433504321625535 (-0.0015928254161082511) | > avg_loss_disc_real_5: 0.24638316559959467 (+0.0012763807891120471) | > avg_loss_0: 2.775982803022358 (-0.010859140208069551) | > avg_grad_norm_0: tensor(11.6770, device='cuda:0') (tensor(-2.0300, device='cuda:0')) | > avg_loss_gen: 1.821994768062108 (-0.004758008768860789) | > avg_loss_kl: 2.2786219791627267 (+0.028477687231252435) | > avg_loss_feat: 2.5562171230853448 (+0.07023264824504638) | > avg_loss_mel: 17.668984265394613 (-0.11420803338708652) | > avg_loss_duration: 1.6156336875029014 (-0.002675076605568316) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.941451811454666 (-0.0229308974574991) | > avg_grad_norm_1: tensor(370.2061, device='cuda:0') (+tensor(26.1156, device='cuda:0'))  > EPOCH: 289/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:10:03)   --> TIME: 2025-02-25 11:10:23 -- STEP: 41/72 -- GLOBAL_STEP: 1020850 | > loss_disc: 2.692415952682495 (2.7742329806816293) | > loss_disc_real_0: 0.25123804807662964 (0.20073144882917404) | > loss_disc_real_1: 0.17810386419296265 (0.23133684549389816) | > loss_disc_real_2: 0.20338313281536102 (0.23828528130926738) | > loss_disc_real_3: 0.22999969124794006 (0.24043437620488609) | > loss_disc_real_4: 0.22575242817401886 (0.24177128803439257) | > loss_disc_real_5: 0.22907079756259918 (0.24457727263613446) | > loss_0: 2.692415952682495 (2.7742329806816293) | > grad_norm_0: tensor(12.5762, device='cuda:0') (tensor(14.0854, device='cuda:0')) | > loss_gen: 1.737446665763855 (1.8326214929906333) | > loss_kl: 2.083160638809204 (2.2937066380570577) | > loss_feat: 3.070840358734131 (2.6455175934768302) | > loss_mel: 18.241376876831055 (17.813827095962154) | > loss_duration: 1.6044777631759644 (1.59369510266839) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.737300872802734 (26.179367995843656) | > grad_norm_1: tensor(200.0924, device='cuda:0') (tensor(305.7795, device='cuda:0')) | > current_lr_0: 0.0001929035086197297 | > current_lr_1: 0.0001929035086197297 | > step_time: 0.4431 (0.43852431018178056) | > loader_time: 0.0096 (0.007341768683456793) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575036492146236 (-0.00011404131499814438) | > avg_step_time: 0.4528415404574972 (-0.0023825302929945114) | > avg_loss_disc: 2.7827835821769615 (+0.006800779154603642) | > avg_loss_disc_real_0: 0.20551264443447892 (+0.0014030523073505219) | > avg_loss_disc_real_1: 0.23213872334487 (+0.001121286775024849) | > avg_loss_disc_real_2: 0.23829193098444335 (-0.0006735440291149586) | > avg_loss_disc_real_3: 0.2397271113496431 (+0.0009930715174742144) | > avg_loss_disc_real_4: 0.2425173930299114 (-0.0008330391326420994) | > avg_loss_disc_real_5: 0.24652913506601898 (+0.00014596946642431163) | > avg_loss_0: 2.7827835821769615 (+0.006800779154603642) | > avg_grad_norm_0: tensor(13.8438, device='cuda:0') (+tensor(2.1668, device='cuda:0')) | > avg_loss_gen: 1.8279977667499596 (+0.006002998687851591) | > avg_loss_kl: 2.322693401659038 (+0.04407142249631146) | > avg_loss_feat: 2.5926898969730856 (+0.03647277388774084) | > avg_loss_mel: 17.92910503333723 (+0.26012076794261674) | > avg_loss_duration: 1.6292610537837928 (+0.01362736628089145) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.301747093737966 (+0.36029528228329966) | > avg_grad_norm_1: tensor(315.3889, device='cuda:0') (tensor(-54.8172, device='cuda:0'))  > EPOCH: 290/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:10:40)   --> TIME: 2025-02-25 11:10:51 -- STEP: 19/72 -- GLOBAL_STEP: 1020900 | > loss_disc: 2.7419495582580566 (2.747938168676276) | > loss_disc_real_0: 0.26932865381240845 (0.20483940839767456) | > loss_disc_real_1: 0.2565634846687317 (0.23132916030130887) | > loss_disc_real_2: 0.2382000982761383 (0.23879809599173696) | > loss_disc_real_3: 0.23622648417949677 (0.23424271141227923) | > loss_disc_real_4: 0.229324609041214 (0.24360467101398267) | > loss_disc_real_5: 0.2317724972963333 (0.24321220972036062) | > loss_0: 2.7419495582580566 (2.747938168676276) | > grad_norm_0: tensor(14.8035, device='cuda:0') (tensor(10.2642, device='cuda:0')) | > loss_gen: 1.9311171770095825 (1.877505202042429) | > loss_kl: 2.3086650371551514 (2.2713303189528617) | > loss_feat: 2.696974754333496 (2.746026039123535) | > loss_mel: 17.79428482055664 (18.148075605693617) | > loss_duration: 1.597336769104004 (1.6137739859129254) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.328380584716797 (26.65671147798237) | > grad_norm_1: tensor(388.0723, device='cuda:0') (tensor(344.9936, device='cuda:0')) | > current_lr_0: 0.00019287939568115225 | > current_lr_1: 0.00019287939568115225 | > step_time: 0.4419 (0.46367084352593674) | > loader_time: 0.0061 (0.006724056444670025)  --> TIME: 2025-02-25 11:11:15 -- STEP: 69/72 -- GLOBAL_STEP: 1020950 | > loss_disc: 2.8026537895202637 (2.7826359237449756) | > loss_disc_real_0: 0.28708457946777344 (0.21007985718872235) | > loss_disc_real_1: 0.21701565384864807 (0.23254653703475345) | > loss_disc_real_2: 0.261233925819397 (0.24135694814764935) | > loss_disc_real_3: 0.23512360453605652 (0.24094521502653757) | > loss_disc_real_4: 0.2516523599624634 (0.24610811407151428) | > loss_disc_real_5: 0.24796882271766663 (0.24551995444125024) | > loss_0: 2.8026537895202637 (2.7826359237449756) | > grad_norm_0: tensor(21.0753, device='cuda:0') (tensor(13.8215, device='cuda:0')) | > loss_gen: 1.7928717136383057 (1.8559033317842346) | > loss_kl: 2.3015127182006836 (2.2545862094215723) | > loss_feat: 2.4134130477905273 (2.627634316250898) | > loss_mel: 16.781627655029297 (17.823882835498747) | > loss_duration: 1.7054595947265625 (1.6217195849487747) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.994884490966797 (26.183726407479547) | > grad_norm_1: tensor(191.5558, device='cuda:0') (tensor(340.3997, device='cuda:0')) | > current_lr_0: 0.00019287939568115225 | > current_lr_1: 0.00019287939568115225 | > step_time: 0.5343 (0.46295143901437946) | > loader_time: 0.0126 (0.008630711099375847) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680991723503863 (+0.00010595523135762709) | > avg_step_time: 0.46450209617614746 (+0.011660555718650267) | > avg_loss_disc: 2.7826436841991584 (-0.0001398979778031162) | > avg_loss_disc_real_0: 0.20957291168226322 (+0.004060267247784299) | > avg_loss_disc_real_1: 0.23381284026192947 (+0.0016741169170594594) | > avg_loss_disc_real_2: 0.2413175299134053 (+0.0030255989289619534) | > avg_loss_disc_real_3: 0.24068187554957163 (+0.000954764199928515) | > avg_loss_disc_real_4: 0.24574684760939908 (+0.0032294545794876728) | > avg_loss_disc_real_5: 0.24524688678728024 (-0.0012822482787387401) | > avg_loss_0: 2.7826436841991584 (-0.0001398979778031162) | > avg_grad_norm_0: tensor(13.7715, device='cuda:0') (tensor(-0.0723, device='cuda:0')) | > avg_loss_gen: 1.854152048137826 (+0.02615428138786635) | > avg_loss_kl: 2.2535788324517263 (-0.06911456920731185) | > avg_loss_feat: 2.622332853330693 (+0.029642956357607275) | > avg_loss_mel: 17.79938356641312 (-0.12972146692411002) | > avg_loss_duration: 1.6216237897604284 (-0.0076372640233643985) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.15107119922906 (-0.15067589450890395) | > avg_grad_norm_1: tensor(342.6021, device='cuda:0') (+tensor(27.2132, device='cuda:0'))  > EPOCH: 291/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:11:17)   --> TIME: 2025-02-25 11:11:41 -- STEP: 47/72 -- GLOBAL_STEP: 1021000 | > loss_disc: 2.7391908168792725 (2.7949113541461053) | > loss_disc_real_0: 0.17322270572185516 (0.20982786331405032) | > loss_disc_real_1: 0.3319394588470459 (0.23508755228620895) | > loss_disc_real_2: 0.21014292538166046 (0.23710160559796273) | > loss_disc_real_3: 0.24068905413150787 (0.23837257572945128) | > loss_disc_real_4: 0.21944664418697357 (0.2427722637323623) | > loss_disc_real_5: 0.23547202348709106 (0.24507982965479505) | > loss_0: 2.7391908168792725 (2.7949113541461053) | > grad_norm_0: tensor(9.3319, device='cuda:0') (tensor(12.5573, device='cuda:0')) | > loss_gen: 1.8934859037399292 (1.81946470128729) | > loss_kl: 2.1889500617980957 (2.200060086047395) | > loss_feat: 2.5361721515655518 (2.5393654488502664) | > loss_mel: 17.679471969604492 (17.712563169763452) | > loss_duration: 1.6229914426803589 (1.5930710523686513) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.921072006225586 (25.86452439490785) | > grad_norm_1: tensor(202.0369, device='cuda:0') (tensor(320.7709, device='cuda:0')) | > current_lr_0: 0.0001928552857566921 | > current_lr_1: 0.0001928552857566921 | > step_time: 0.4495 (0.4426006611357344) | > loader_time: 0.0088 (0.007732005829506732) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687764825955242 (+6.773102451378826e-06) | > avg_step_time: 0.45598955893180737 (-0.00851253724434009) | > avg_loss_disc: 2.795466920019875 (+0.01282323582071676) | > avg_loss_disc_real_0: 0.21191316507232022 (+0.002340253390057001) | > avg_loss_disc_real_1: 0.23425420740960348 (+0.00044136714767401) | > avg_loss_disc_real_2: 0.23922806075761016 (-0.0020894691557951373) | > avg_loss_disc_real_3: 0.2398313390956798 (-0.0008505364538918259) | > avg_loss_disc_real_4: 0.24426726304309468 (-0.0014795845663043983) | > avg_loss_disc_real_5: 0.2471697561337914 (+0.0019228693465111746) | > avg_loss_0: 2.795466920019875 (+0.01282323582071676) | > avg_grad_norm_0: tensor(12.5147, device='cuda:0') (tensor(-1.2568, device='cuda:0')) | > avg_loss_gen: 1.8213083626518787 (-0.032843685485947294) | > avg_loss_kl: 2.228935790733551 (-0.02464304171817533) | > avg_loss_feat: 2.5341940177998072 (-0.08813883553088564) | > avg_loss_mel: 17.769278915835105 (-0.030104650578014258) | > avg_loss_duration: 1.6102931600221446 (-0.011330629738283804) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.964010158055267 (-0.18706104117379496) | > avg_grad_norm_1: tensor(338.1512, device='cuda:0') (tensor(-4.4509, device='cuda:0'))  > EPOCH: 292/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:11:54)   --> TIME: 2025-02-25 11:12:08 -- STEP: 25/72 -- GLOBAL_STEP: 1021050 | > loss_disc: 2.7481651306152344 (2.795923957824707) | > loss_disc_real_0: 0.1227586567401886 (0.2150883835554123) | > loss_disc_real_1: 0.2133799046278 (0.23114767491817476) | > loss_disc_real_2: 0.22401496767997742 (0.2516678172349931) | > loss_disc_real_3: 0.2496238499879837 (0.24210328936576844) | > loss_disc_real_4: 0.23893575370311737 (0.2475474911928177) | > loss_disc_real_5: 0.26311248540878296 (0.24457404732704163) | > loss_0: 2.7481651306152344 (2.795923957824707) | > grad_norm_0: tensor(22.3586, device='cuda:0') (tensor(13.0558, device='cuda:0')) | > loss_gen: 1.6657655239105225 (1.8226352977752684) | > loss_kl: 2.475942850112915 (2.168741283416748) | > loss_feat: 2.4674313068389893 (2.532182607650757) | > loss_mel: 17.79491424560547 (17.48674499511719) | > loss_duration: 1.5903897285461426 (1.5692506122589112) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.994443893432617 (25.579554977416993) | > grad_norm_1: tensor(112.1570, device='cuda:0') (tensor(296.9352, device='cuda:0')) | > current_lr_0: 0.0001928311788459725 | > current_lr_1: 0.0001928311788459725 | > step_time: 0.4394 (0.43485724449157714) | > loader_time: 0.0073 (0.006815938949584961) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859041952751052 (-9.734529844472248e-05) | > avg_step_time: 0.4539921753843066 (-0.001997383547500786) | > avg_loss_disc: 2.8079741571990535 (+0.012507237179178343) | > avg_loss_disc_real_0: 0.21466774934194457 (+0.002754584269624355) | > avg_loss_disc_real_1: 0.23151771388423276 (-0.0027364935253707257) | > avg_loss_disc_real_2: 0.24659285704854514 (+0.007364796290934977) | > avg_loss_disc_real_3: 0.24268100610081578 (+0.002849667005135975) | > avg_loss_disc_real_4: 0.24649891257286072 (+0.002231649529766039) | > avg_loss_disc_real_5: 0.24722120942364276 (+5.145328985134423e-05) | > avg_loss_0: 2.8079741571990535 (+0.012507237179178343) | > avg_grad_norm_0: tensor(13.8067, device='cuda:0') (+tensor(1.2919, device='cuda:0')) | > avg_loss_gen: 1.8127755967664048 (-0.008532765885473914) | > avg_loss_kl: 2.2561887385140005 (+0.02725294778044951) | > avg_loss_feat: 2.4717756425830677 (-0.06241837521673954) | > avg_loss_mel: 17.538572929274864 (-0.23070598656024188) | > avg_loss_duration: 1.607529792987125 (-0.0027633670350195594) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.686842609459244 (-0.27716754859602233) | > avg_grad_norm_1: tensor(281.2738, device='cuda:0') (tensor(-56.8774, device='cuda:0'))  > EPOCH: 293/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:12:31)   --> TIME: 2025-02-25 11:12:35 -- STEP: 3/72 -- GLOBAL_STEP: 1021100 | > loss_disc: 2.7384305000305176 (2.7406749725341797) | > loss_disc_real_0: 0.164200097322464 (0.21261959771315256) | > loss_disc_real_1: 0.23779454827308655 (0.2195264200369517) | > loss_disc_real_2: 0.24552974104881287 (0.2550784846146901) | > loss_disc_real_3: 0.23964132368564606 (0.23101735611756644) | > loss_disc_real_4: 0.2464427798986435 (0.23906241854031882) | > loss_disc_real_5: 0.25767821073532104 (0.24660820265611014) | > loss_0: 2.7384305000305176 (2.7406749725341797) | > grad_norm_0: tensor(10.2987, device='cuda:0') (tensor(10.3624, device='cuda:0')) | > loss_gen: 1.8675742149353027 (1.8502238988876343) | > loss_kl: 2.2802252769470215 (2.3060153325398765) | > loss_feat: 2.8870229721069336 (2.7339771588643393) | > loss_mel: 17.419015884399414 (17.4303773244222) | > loss_duration: 1.6250296831130981 (1.5967896779378254) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.078868865966797 (25.917383193969727) | > grad_norm_1: tensor(751.7812, device='cuda:0') (tensor(333.4536, device='cuda:0')) | > current_lr_0: 0.00019280707494861676 | > current_lr_1: 0.00019280707494861676 | > step_time: 0.4363 (0.43919118245442706) | > loader_time: 0.0057 (0.007028261820475261)  --> TIME: 2025-02-25 11:12:58 -- STEP: 53/72 -- GLOBAL_STEP: 1021150 | > loss_disc: 2.7635960578918457 (2.7881750340731637) | > loss_disc_real_0: 0.15773075819015503 (0.2046405039024803) | > loss_disc_real_1: 0.28408393263816833 (0.23605304225435797) | > loss_disc_real_2: 0.21724967658519745 (0.23750479013290046) | > loss_disc_real_3: 0.23345668613910675 (0.23924300074577332) | > loss_disc_real_4: 0.2282300740480423 (0.24526322562739533) | > loss_disc_real_5: 0.232083261013031 (0.24678443372249603) | > loss_0: 2.7635960578918457 (2.7881750340731637) | > grad_norm_0: tensor(9.4725, device='cuda:0') (tensor(13.9633, device='cuda:0')) | > loss_gen: 1.9698765277862549 (1.8206757329544931) | > loss_kl: 2.089766025543213 (2.2104753615721218) | > loss_feat: 2.317762613296509 (2.5285235395971335) | > loss_mel: 16.189498901367188 (17.619333267211918) | > loss_duration: 1.6286773681640625 (1.5959316357126776) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.195581436157227 (25.774939698993034) | > grad_norm_1: tensor(381.6253, device='cuda:0') (tensor(376.1987, device='cuda:0')) | > current_lr_0: 0.00019280707494861676 | > current_lr_1: 0.00019280707494861676 | > step_time: 0.4548 (0.45385570346184495) | > loader_time: 0.0095 (0.008048898768874832) --> EVAL PERFORMANCE | > avg_loader_time: 0.008767228730967344 (+0.0001768092034568243) | > avg_step_time: 0.4640649506743525 (+0.01007277529004591) | > avg_loss_disc: 2.7864718437194824 (-0.0215023134795711) | > avg_loss_disc_real_0: 0.20685303263681035 (-0.007814716705134228) | > avg_loss_disc_real_1: 0.23296256930055753 (+0.001444855416324775) | > avg_loss_disc_real_2: 0.23828033704153248 (-0.008312520007012658) | > avg_loss_disc_real_3: 0.2382552485230943 (-0.004425757577721484) | > avg_loss_disc_real_4: 0.2443672561309707 (-0.0021316564418900064) | > avg_loss_disc_real_5: 0.245951169813183 (-0.0012700396104597667) | > avg_loss_0: 2.7864718437194824 (-0.0215023134795711) | > avg_grad_norm_0: tensor(15.4919, device='cuda:0') (+tensor(1.6853, device='cuda:0')) | > avg_loss_gen: 1.8139337502734763 (+0.001158153507071491) | > avg_loss_kl: 2.2162538498220314 (-0.03993488869196904) | > avg_loss_feat: 2.498518488776516 (+0.026742846193448244) | > avg_loss_mel: 17.631615773053237 (+0.09304284377837391) | > avg_loss_duration: 1.608039832450974 (+0.0005100394638488659) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.768361830375564 (+0.08151922091632002) | > avg_grad_norm_1: tensor(380.9554, device='cuda:0') (+tensor(99.6815, device='cuda:0'))  > EPOCH: 294/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:13:09)   --> TIME: 2025-02-25 11:13:25 -- STEP: 31/72 -- GLOBAL_STEP: 1021200 | > loss_disc: 2.6971497535705566 (2.7875762524143344) | > loss_disc_real_0: 0.2809670567512512 (0.20394891115926927) | > loss_disc_real_1: 0.16673791408538818 (0.22427339226968826) | > loss_disc_real_2: 0.23503349721431732 (0.241301029920578) | > loss_disc_real_3: 0.2110079824924469 (0.23883299962166818) | > loss_disc_real_4: 0.24288462102413177 (0.24258407229377377) | > loss_disc_real_5: 0.23270413279533386 (0.24460676408583118) | > loss_0: 2.6971497535705566 (2.7875762524143344) | > grad_norm_0: tensor(17.4822, device='cuda:0') (tensor(12.1155, device='cuda:0')) | > loss_gen: 1.9060466289520264 (1.8214789782800982) | > loss_kl: 2.452218532562256 (2.2107820741591904) | > loss_feat: 2.790193557739258 (2.609645632005507) | > loss_mel: 18.168869018554688 (17.755601483006636) | > loss_duration: 1.6020283699035645 (1.575309657281445) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.919355392456055 (25.97281769783266) | > grad_norm_1: tensor(549.8743, device='cuda:0') (tensor(369.7562, device='cuda:0')) | > current_lr_0: 0.00019278297406424816 | > current_lr_1: 0.00019278297406424816 | > step_time: 0.4542 (0.4397935252035818) | > loader_time: 0.0089 (0.007322488292571037) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687486111278267 (-7.97426196890768e-05) | > avg_step_time: 0.4563924493923993 (-0.007672501281953181) | > avg_loss_disc: 2.8106270642347737 (+0.024155220515291287) | > avg_loss_disc_real_0: 0.21426117693034696 (+0.007408144293536617) | > avg_loss_disc_real_1: 0.23144954401002804 (-0.0015130252905294905) | > avg_loss_disc_real_2: 0.24213799225612426 (+0.0038576552145917764) | > avg_loss_disc_real_3: 0.24294532919433756 (+0.0046900806712432686) | > avg_loss_disc_real_4: 0.24423383008426344 (-0.00013342604670726899) | > avg_loss_disc_real_5: 0.24656464516277043 (+0.0006134753495874445) | > avg_loss_0: 2.8106270642347737 (+0.024155220515291287) | > avg_grad_norm_0: tensor(13.6777, device='cuda:0') (tensor(-1.8142, device='cuda:0')) | > avg_loss_gen: 1.8195355005667244 (+0.005601750293248164) | > avg_loss_kl: 2.223819057706376 (+0.007565207884344449) | > avg_loss_feat: 2.5244287453906646 (+0.025910256614148697) | > avg_loss_mel: 17.509414202730422 (-0.1222015703228152) | > avg_loss_duration: 1.608456702299521 (+0.00041686984854716336) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.685654143212545 (-0.08270768716301902) | > avg_grad_norm_1: tensor(343.3386, device='cuda:0') (tensor(-37.6167, device='cuda:0'))  > EPOCH: 295/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:13:46)   --> TIME: 2025-02-25 11:13:52 -- STEP: 9/72 -- GLOBAL_STEP: 1021250 | > loss_disc: 2.7585558891296387 (2.722807354397244) | > loss_disc_real_0: 0.193155437707901 (0.20315244959460366) | > loss_disc_real_1: 0.3174368143081665 (0.23433383968141344) | > loss_disc_real_2: 0.24825657904148102 (0.2342583073510064) | > loss_disc_real_3: 0.23869909346103668 (0.23082383473714194) | > loss_disc_real_4: 0.23716504871845245 (0.23803733620378706) | > loss_disc_real_5: 0.23078632354736328 (0.2278715173403422) | > loss_0: 2.7585558891296387 (2.722807354397244) | > grad_norm_0: tensor(8.8402, device='cuda:0') (tensor(9.1582, device='cuda:0')) | > loss_gen: 1.813661813735962 (1.8500349521636963) | > loss_kl: 2.314002752304077 (2.3074067168765597) | > loss_feat: 2.322643756866455 (2.528359439637926) | > loss_mel: 16.88968849182129 (17.77740414937337) | > loss_duration: 1.580065131187439 (1.58518566025628) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.920061111450195 (26.04839113023546) | > grad_norm_1: tensor(267.1307, device='cuda:0') (tensor(331.2539, device='cuda:0')) | > current_lr_0: 0.00019275887619249013 | > current_lr_1: 0.00019275887619249013 | > step_time: 0.4252 (0.4279181162516276) | > loader_time: 0.0057 (0.00614166259765625)  --> TIME: 2025-02-25 11:14:15 -- STEP: 59/72 -- GLOBAL_STEP: 1021300 | > loss_disc: 2.90568208694458 (2.77502963098429) | > loss_disc_real_0: 0.24836242198944092 (0.20760210426682132) | > loss_disc_real_1: 0.351448655128479 (0.23071306640818967) | > loss_disc_real_2: 0.2832905054092407 (0.24073527374510037) | > loss_disc_real_3: 0.28268489241600037 (0.23937759687334803) | > loss_disc_real_4: 0.2677217125892639 (0.2458147813202971) | > loss_disc_real_5: 0.2808499336242676 (0.24668753475455915) | > loss_0: 2.90568208694458 (2.77502963098429) | > grad_norm_0: tensor(15.4890, device='cuda:0') (tensor(12.8085, device='cuda:0')) | > loss_gen: 1.8363885879516602 (1.8477263632467238) | > loss_kl: 2.399010181427002 (2.244680188469968) | > loss_feat: 2.5397653579711914 (2.580090437905263) | > loss_mel: 18.35344886779785 (17.620308989185403) | > loss_duration: 1.6475780010223389 (1.6003178236848217) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.77618980407715 (25.893123820676642) | > grad_norm_1: tensor(687.2808, device='cuda:0') (tensor(385.8569, device='cuda:0')) | > current_lr_0: 0.00019275887619249013 | > current_lr_1: 0.00019275887619249013 | > step_time: 0.466 (0.4448096509707176) | > loader_time: 0.0098 (0.008122832088147179) --> EVAL PERFORMANCE | > avg_loader_time: 0.008656545424125564 (-3.094068715270293e-05) | > avg_step_time: 0.45188268473450566 (-0.004509764657893656) | > avg_loss_disc: 2.782333689676204 (-0.028293374558569617) | > avg_loss_disc_real_0: 0.21159005070656117 (-0.0026711262237857913) | > avg_loss_disc_real_1: 0.23327399349548447 (+0.0018244494854564308) | > avg_loss_disc_real_2: 0.2392001361914084 (-0.0029378560647158602) | > avg_loss_disc_real_3: 0.2386401780474354 (-0.004305151146902164) | > avg_loss_disc_real_4: 0.2453108658673058 (+0.0010770357830423571) | > avg_loss_disc_real_5: 0.24771474196877277 (+0.0011500968060023375) | > avg_loss_0: 2.782333689676204 (-0.028293374558569617) | > avg_grad_norm_0: tensor(13.6725, device='cuda:0') (tensor(-0.0052, device='cuda:0')) | > avg_loss_gen: 1.8402678278130544 (+0.020732327246330007) | > avg_loss_kl: 2.2627914942486185 (+0.03897243654224258) | > avg_loss_feat: 2.55137626553925 (+0.02694752014858537) | > avg_loss_mel: 17.6472341644932 (+0.13781996176277644) | > avg_loss_duration: 1.6094491431410884 (+0.0009924408415673547) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.91111895064233 (+0.22546480742978403) | > avg_grad_norm_1: tensor(387.0500, device='cuda:0') (+tensor(43.7114, device='cuda:0'))  > EPOCH: 296/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:14:23)   --> TIME: 2025-02-25 11:14:42 -- STEP: 37/72 -- GLOBAL_STEP: 1021350 | > loss_disc: 2.822155714035034 (2.780892056387824) | > loss_disc_real_0: 0.16992910206317902 (0.21100993917600527) | > loss_disc_real_1: 0.23529988527297974 (0.2281758926204733) | > loss_disc_real_2: 0.2683175504207611 (0.23678470020358627) | > loss_disc_real_3: 0.22082272171974182 (0.23393348584303986) | > loss_disc_real_4: 0.22973214089870453 (0.24386786971543287) | > loss_disc_real_5: 0.24902848899364471 (0.24783572997595812) | > loss_0: 2.822155714035034 (2.780892056387824) | > grad_norm_0: tensor(12.6586, device='cuda:0') (tensor(13.7807, device='cuda:0')) | > loss_gen: 1.9123148918151855 (1.844339631699227) | > loss_kl: 2.3156604766845703 (2.2929557658530575) | > loss_feat: 2.5275919437408447 (2.6223285842586206) | > loss_mel: 17.381147384643555 (17.959258517703496) | > loss_duration: 1.643115758895874 (1.5998824834823608) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.779830932617188 (26.318764970109267) | > grad_norm_1: tensor(137.7098, device='cuda:0') (tensor(346.8022, device='cuda:0')) | > current_lr_0: 0.00019273478133296606 | > current_lr_1: 0.00019273478133296606 | > step_time: 0.4536 (0.45234490085292506) | > loader_time: 0.0082 (0.007269524239205025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008601390140157348 (-5.515528396821644e-05) | > avg_step_time: 0.4639237497893857 (+0.012041065054880062) | > avg_loss_disc: 2.7895768729733743 (+0.007243183297170219) | > avg_loss_disc_real_0: 0.21254122802908992 (+0.0009511773225287512) | > avg_loss_disc_real_1: 0.23177681422569382 (-0.0014971792697906494) | > avg_loss_disc_real_2: 0.2391699581918582 (-3.0177999550184964e-05) | > avg_loss_disc_real_3: 0.23784088081037494 (-0.000799297237060459) | > avg_loss_disc_real_4: 0.24536717609620431 (+5.631022889851511e-05) | > avg_loss_disc_real_5: 0.24914652444946933 (+0.0014317824806965584) | > avg_loss_0: 2.7895768729733743 (+0.007243183297170219) | > avg_grad_norm_0: tensor(14.3638, device='cuda:0') (+tensor(0.6913, device='cuda:0')) | > avg_loss_gen: 1.835570107043629 (-0.004697720769425429) | > avg_loss_kl: 2.286817517079098 (+0.024026022830479388) | > avg_loss_feat: 2.522055859297095 (-0.029320406242154817) | > avg_loss_mel: 17.710770781611053 (+0.06353661711785463) | > avg_loss_duration: 1.6183364492067152 (+0.00888730606562671) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.973550769644724 (+0.06243181900239492) | > avg_grad_norm_1: tensor(394.7888, device='cuda:0') (+tensor(7.7387, device='cuda:0'))  > EPOCH: 297/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:15:00)   --> TIME: 2025-02-25 11:15:09 -- STEP: 15/72 -- GLOBAL_STEP: 1021400 | > loss_disc: 2.710153818130493 (2.7700983683268228) | > loss_disc_real_0: 0.1412394642829895 (0.20422704617182413) | > loss_disc_real_1: 0.2084638476371765 (0.22475090424219768) | > loss_disc_real_2: 0.2343026101589203 (0.2323381245136261) | > loss_disc_real_3: 0.23226453363895416 (0.24920358757177988) | > loss_disc_real_4: 0.2422455996274948 (0.24139668941497802) | > loss_disc_real_5: 0.24050943553447723 (0.24750755727291107) | > loss_0: 2.710153818130493 (2.7700983683268228) | > grad_norm_0: tensor(23.6342, device='cuda:0') (tensor(18.1636, device='cuda:0')) | > loss_gen: 1.8636667728424072 (1.8182794411977132) | > loss_kl: 2.1396853923797607 (2.1015677849451704) | > loss_feat: 2.378437042236328 (2.5055249373118085) | > loss_mel: 17.00088119506836 (17.42752850850423) | > loss_duration: 1.5572303533554077 (1.5639584938685098) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.93990135192871 (25.416859436035157) | > grad_norm_1: tensor(98.8699, device='cuda:0') (tensor(332.4207, device='cuda:0')) | > current_lr_0: 0.00019271068948529944 | > current_lr_1: 0.00019271068948529944 | > step_time: 0.4467 (0.4418521722157796) | > loader_time: 0.006 (0.007193740208943685)  --> TIME: 2025-02-25 11:15:33 -- STEP: 65/72 -- GLOBAL_STEP: 1021450 | > loss_disc: 2.834270715713501 (2.7847549658555253) | > loss_disc_real_0: 0.1564573347568512 (0.2045432481628198) | > loss_disc_real_1: 0.22721539437770844 (0.2348859172600966) | > loss_disc_real_2: 0.21139861643314362 (0.23698174426188837) | > loss_disc_real_3: 0.2322634756565094 (0.24172171308444096) | > loss_disc_real_4: 0.23010483384132385 (0.24276830760332255) | > loss_disc_real_5: 0.23289793729782104 (0.24726074521358196) | > loss_0: 2.834270715713501 (2.7847549658555253) | > grad_norm_0: tensor(17.2441, device='cuda:0') (tensor(13.5258, device='cuda:0')) | > loss_gen: 1.7300292253494263 (1.8242945139224713) | > loss_kl: 2.4004287719726562 (2.2312010489977325) | > loss_feat: 2.2792437076568604 (2.5588877787956825) | > loss_mel: 17.498432159423828 (17.67613830566406) | > loss_duration: 1.6641618013381958 (1.6058415706341083) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.572296142578125 (25.896363302377555) | > grad_norm_1: tensor(298.3706, device='cuda:0') (tensor(331.6840, device='cuda:0')) | > current_lr_0: 0.00019271068948529944 | > current_lr_1: 0.00019271068948529944 | > step_time: 0.4971 (0.4516574089343731) | > loader_time: 0.0121 (0.00865161969111516) --> EVAL PERFORMANCE | > avg_loader_time: 0.008919695733298719 (+0.0003183055931413707) | > avg_step_time: 0.45682747934905577 (-0.007096270440329955) | > avg_loss_disc: 2.7823613663794284 (-0.007215506593945875) | > avg_loss_disc_real_0: 0.20580759151300915 (-0.0067336365160807765) | > avg_loss_disc_real_1: 0.23481827302717825 (+0.0030414588014844313) | > avg_loss_disc_real_2: 0.23656502093227816 (-0.0026049372595800535) | > avg_loss_disc_real_3: 0.23971581815833776 (+0.0018749373479628184) | > avg_loss_disc_real_4: 0.24201500080001187 (-0.0033521752961924445) | > avg_loss_disc_real_5: 0.24628869244750118 (-0.0028578320019681525) | > avg_loss_0: 2.7823613663794284 (-0.007215506593945875) | > avg_grad_norm_0: tensor(13.3241, device='cuda:0') (tensor(-1.0398, device='cuda:0')) | > avg_loss_gen: 1.823300282720109 (-0.012269824323519973) | > avg_loss_kl: 2.23566734790802 (-0.05115016917107784) | > avg_loss_feat: 2.55275040277293 (+0.030694543475835) | > avg_loss_mel: 17.671597467341893 (-0.03917331426915993) | > avg_loss_duration: 1.6103601422108393 (-0.007976306995875815) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.893675656385824 (-0.0798751132589004) | > avg_grad_norm_1: tensor(337.2585, device='cuda:0') (tensor(-57.5303, device='cuda:0'))  > EPOCH: 298/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:15:37)   --> TIME: 2025-02-25 11:15:59 -- STEP: 43/72 -- GLOBAL_STEP: 1021500 | > loss_disc: 2.7538909912109375 (2.777101810588393) | > loss_disc_real_0: 0.2075444608926773 (0.21165003540904023) | > loss_disc_real_1: 0.20826277136802673 (0.23374144386413487) | > loss_disc_real_2: 0.2637295424938202 (0.24302128061305645) | > loss_disc_real_3: 0.22950813174247742 (0.2380685293397238) | > loss_disc_real_4: 0.2565784156322479 (0.24408290136692135) | > loss_disc_real_5: 0.266681969165802 (0.24736926236818002) | > loss_0: 2.7538909912109375 (2.777101810588393) | > grad_norm_0: tensor(4.9474, device='cuda:0') (tensor(14.9776, device='cuda:0')) | > loss_gen: 1.85499906539917 (1.8537396913351014) | > loss_kl: 2.1410577297210693 (2.299940189649893) | > loss_feat: 2.452409029006958 (2.6465402148490726) | > loss_mel: 16.716278076171875 (17.97123057343239) | > loss_duration: 1.6056764125823975 (1.6018185698708822) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.770421981811523 (26.373269192008085) | > grad_norm_1: tensor(144.8308, device='cuda:0') (tensor(393.0786, device='cuda:0')) | > current_lr_0: 0.00019268660064911377 | > current_lr_1: 0.00019268660064911377 | > step_time: 0.4902 (0.441716238509777) | > loader_time: 0.0096 (0.007597363272378611) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865445674305231 (-0.0002652389902464084) | > avg_step_time: 0.45575879325329416 (-0.0010686860957616062) | > avg_loss_disc: 2.7805081992082195 (-0.0018531671712089803) | > avg_loss_disc_real_0: 0.20971154138235978 (+0.003903949869350637) | > avg_loss_disc_real_1: 0.23421806243943497 (-0.0006002105877432884) | > avg_loss_disc_real_2: 0.24282134544681494 (+0.006256324514536782) | > avg_loss_disc_real_3: 0.2387819443370255 (-0.0009338738213122699) | > avg_loss_disc_real_4: 0.24425604120946268 (+0.0022410404094508063) | > avg_loss_disc_real_5: 0.2464609251055919 (+0.0001722326580907152) | > avg_loss_0: 2.7805081992082195 (-0.0018531671712089803) | > avg_grad_norm_0: tensor(14.1674, device='cuda:0') (+tensor(0.8433, device='cuda:0')) | > avg_loss_gen: 1.8406353987438577 (+0.01733511602374871) | > avg_loss_kl: 2.296795813130661 (+0.061128465222640926) | > avg_loss_feat: 2.5938674967053914 (+0.04111709393246121) | > avg_loss_mel: 17.82184719031965 (+0.15024972297775818) | > avg_loss_duration: 1.6229359599905955 (+0.012575817779756138) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.17608193948235 (+0.2824062830965275) | > avg_grad_norm_1: tensor(419.9863, device='cuda:0') (+tensor(82.7279, device='cuda:0'))  > EPOCH: 299/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:16:15)   --> TIME: 2025-02-25 11:16:26 -- STEP: 21/72 -- GLOBAL_STEP: 1021550 | > loss_disc: 2.748600721359253 (2.7978494621458507) | > loss_disc_real_0: 0.24738594889640808 (0.23088648986248744) | > loss_disc_real_1: 0.23476408421993256 (0.2253681882506325) | > loss_disc_real_2: 0.22173355519771576 (0.23227137681983767) | > loss_disc_real_3: 0.23668839037418365 (0.23527707230477107) | > loss_disc_real_4: 0.2259458750486374 (0.24473135882899874) | > loss_disc_real_5: 0.2404385507106781 (0.2437441647052765) | > loss_0: 2.748600721359253 (2.7978494621458507) | > grad_norm_0: tensor(6.8645, device='cuda:0') (tensor(16.3516, device='cuda:0')) | > loss_gen: 1.7703032493591309 (1.8348751124881564) | > loss_kl: 2.180509090423584 (2.224400213786534) | > loss_feat: 2.549211263656616 (2.5777365139552524) | > loss_mel: 18.241313934326172 (18.016022091820123) | > loss_duration: 1.616066813468933 (1.594229817390442) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.357404708862305 (26.247263953799294) | > grad_norm_1: tensor(580.1517, device='cuda:0') (tensor(242.2604, device='cuda:0')) | > current_lr_0: 0.00019266251482403263 | > current_lr_1: 0.00019266251482403263 | > step_time: 0.4422 (0.4361092930748349) | > loader_time: 0.0081 (0.006979874202183315)  --> TIME: 2025-02-25 11:16:50 -- STEP: 71/72 -- GLOBAL_STEP: 1021600 | > loss_disc: 2.769923448562622 (2.7774724120825107) | > loss_disc_real_0: 0.09369152039289474 (0.21441808500340287) | > loss_disc_real_1: 0.22454634308815002 (0.22908637095505083) | > loss_disc_real_2: 0.2255910038948059 (0.23734192466232137) | > loss_disc_real_3: 0.24356448650360107 (0.2369333503531738) | > loss_disc_real_4: 0.2686478793621063 (0.24631005000900213) | > loss_disc_real_5: 0.2566763460636139 (0.24596874562787338) | > loss_0: 2.769923448562622 (2.7774724120825107) | > grad_norm_0: tensor(29.9850, device='cuda:0') (tensor(14.6884, device='cuda:0')) | > loss_gen: 1.866000771522522 (1.8424511610622136) | > loss_kl: 2.16019606590271 (2.2698543709768364) | > loss_feat: 2.1025736331939697 (2.5619059314190493) | > loss_mel: 16.9631404876709 (17.67680702746754) | > loss_duration: 1.6051082611083984 (1.618995918354518) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.697019577026367 (25.97001449155136) | > grad_norm_1: tensor(152.0178, device='cuda:0') (tensor(336.0650, device='cuda:0')) | > current_lr_0: 0.00019266251482403263 | > current_lr_1: 0.00019266251482403263 | > step_time: 0.5012 (0.4584967720676476) | > loader_time: 0.008 (0.00871741939598406) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871741939598406 (+6.296265293175007e-05) | > avg_step_time: 0.4584967720676476 (+0.002737978814353448) | > avg_loss_disc: 2.7774724120825107 (-0.003035787125708733) | > avg_loss_disc_real_0: 0.21441808500340287 (+0.0047065436210430855) | > avg_loss_disc_real_1: 0.22908637095505083 (-0.005131691484384138) | > avg_loss_disc_real_2: 0.23734192466232137 (-0.005479420784493566) | > avg_loss_disc_real_3: 0.2369333503531738 (-0.0018485939838516763) | > avg_loss_disc_real_4: 0.24631005000900213 (+0.0020540087995394585) | > avg_loss_disc_real_5: 0.24596874562787338 (-0.0004921794777185129) | > avg_loss_0: 2.7774724120825107 (-0.003035787125708733) | > avg_grad_norm_0: tensor(14.6884, device='cuda:0') (+tensor(0.5211, device='cuda:0')) | > avg_loss_gen: 1.8424511610622136 (+0.0018157623183558957) | > avg_loss_kl: 2.2698543709768364 (-0.026941442153824546) | > avg_loss_feat: 2.5619059314190493 (-0.031961565286342086) | > avg_loss_mel: 17.67680702746754 (-0.14504016285211208) | > avg_loss_duration: 1.618995918354518 (-0.003940041636077529) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.97001449155136 (-0.20606744793099097) | > avg_grad_norm_1: tensor(336.0650, device='cuda:0') (tensor(-83.9213, device='cuda:0'))  > EPOCH: 300/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:16:52)   --> TIME: 2025-02-25 11:17:17 -- STEP: 49/72 -- GLOBAL_STEP: 1021650 | > loss_disc: 2.815166473388672 (2.793517375478939) | > loss_disc_real_0: 0.18407535552978516 (0.21012320901666368) | > loss_disc_real_1: 0.19512349367141724 (0.23144482806020852) | > loss_disc_real_2: 0.19062697887420654 (0.2421936356291479) | > loss_disc_real_3: 0.22047965228557587 (0.2418305937732969) | > loss_disc_real_4: 0.2369689643383026 (0.24820408377112174) | > loss_disc_real_5: 0.25051867961883545 (0.2504360572415955) | > loss_0: 2.815166473388672 (2.793517375478939) | > grad_norm_0: tensor(8.5118, device='cuda:0') (tensor(15.2082, device='cuda:0')) | > loss_gen: 1.7329201698303223 (1.8447420718718548) | > loss_kl: 2.222728967666626 (2.3077006291370004) | > loss_feat: 2.6296653747558594 (2.6042898187831964) | > loss_mel: 17.883026123046875 (18.410153797694615) | > loss_duration: 1.6908211708068848 (1.6257749966212682) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.159160614013672 (26.792661355466258) | > grad_norm_1: tensor(66.2910, device='cuda:0') (tensor(371.0172, device='cuda:0')) | > current_lr_0: 0.00019263843200967962 | > current_lr_1: 0.00019263843200967962 | > step_time: 0.4468 (0.44716489071748694) | > loader_time: 0.0099 (0.007980672680601781) --> EVAL PERFORMANCE | > avg_loader_time: 0.008825080495485114 (+0.00010766109950105344) | > avg_step_time: 0.45867652624425753 (+0.00017975417660992532) | > avg_loss_disc: 2.8020648016056544 (+0.024592389523143687) | > avg_loss_disc_real_0: 0.21207300206305277 (-0.0023450829403501017) | > avg_loss_disc_real_1: 0.233847600263609 (+0.004761229308558185) | > avg_loss_disc_real_2: 0.2421167446693904 (+0.004774820007069025) | > avg_loss_disc_real_3: 0.24213206075446705 (+0.005198710401293244) | > avg_loss_disc_real_4: 0.24728226116005803 (+0.0009722111510558906) | > avg_loss_disc_real_5: 0.24908930587936456 (+0.0031205602514911834) | > avg_loss_0: 2.8020648016056544 (+0.024592389523143687) | > avg_grad_norm_0: tensor(14.2420, device='cuda:0') (tensor(-0.4464, device='cuda:0')) | > avg_loss_gen: 1.8373105458810295 (-0.005140615181184138) | > avg_loss_kl: 2.3375214556573143 (+0.06766708468047788) | > avg_loss_feat: 2.5714634270735197 (+0.009557495654470394) | > avg_loss_mel: 18.32065566156951 (+0.6438486341019711) | > avg_loss_duration: 1.6365740080954323 (+0.01757808974091435) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.703525086523783 (+0.7335105949724223) | > avg_grad_norm_1: tensor(381.3863, device='cuda:0') (+tensor(45.3213, device='cuda:0'))  > EPOCH: 301/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:17:29)   --> TIME: 2025-02-25 11:17:44 -- STEP: 27/72 -- GLOBAL_STEP: 1021700 | > loss_disc: 2.7634177207946777 (2.7660426210474083) | > loss_disc_real_0: 0.18589994311332703 (0.22169363609066717) | > loss_disc_real_1: 0.2159254550933838 (0.2302617105068984) | > loss_disc_real_2: 0.22579741477966309 (0.23415744966930813) | > loss_disc_real_3: 0.2286125272512436 (0.236109490747805) | > loss_disc_real_4: 0.22048698365688324 (0.24255720719143195) | > loss_disc_real_5: 0.2713836133480072 (0.24786226120260027) | > loss_0: 2.7634177207946777 (2.7660426210474083) | > grad_norm_0: tensor(9.0884, device='cuda:0') (tensor(13.6448, device='cuda:0')) | > loss_gen: 1.8991080522537231 (1.8698038966567427) | > loss_kl: 2.2461636066436768 (2.2437620472024986) | > loss_feat: 2.7787230014801025 (2.709553904003567) | > loss_mel: 17.68017578125 (17.83831165455006) | > loss_duration: 1.5424693822860718 (1.5823847276193124) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.14664077758789 (26.243816446374964) | > grad_norm_1: tensor(764.8039, device='cuda:0') (tensor(281.6048, device='cuda:0')) | > current_lr_0: 0.0001926143522056784 | > current_lr_1: 0.0001926143522056784 | > step_time: 0.4407 (0.4395915490609628) | > loader_time: 0.008 (0.007064139401471174) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679279139344121 (-0.00014580135614099266) | > avg_step_time: 0.45781008290572905 (-0.0008664433385284775) | > avg_loss_disc: 2.7772393159463373 (-0.02482548565931708) | > avg_loss_disc_real_0: 0.21769026729842306 (+0.005617265235370289) | > avg_loss_disc_real_1: 0.2297946917758861 (-0.004052908487722912) | > avg_loss_disc_real_2: 0.23752860095299466 (-0.004588143716395737) | > avg_loss_disc_real_3: 0.23594145539780736 (-0.00619060535665969) | > avg_loss_disc_real_4: 0.24370626727460135 (-0.0035759938854566797) | > avg_loss_disc_real_5: 0.24620261326642104 (-0.0028866926129435255) | > avg_loss_0: 2.7772393159463373 (-0.02482548565931708) | > avg_grad_norm_0: tensor(14.8655, device='cuda:0') (+tensor(0.6235, device='cuda:0')) | > avg_loss_gen: 1.8391397738120925 (+0.0018292279310629933) | > avg_loss_kl: 2.254036569259537 (-0.08348488639777729) | > avg_loss_feat: 2.579909529484493 (+0.00844610241097321) | > avg_loss_mel: 17.679709528533504 (-0.6409461330360067) | > avg_loss_duration: 1.6120291562147544 (-0.024544851880677898) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.964824757105866 (-0.7387003294179166) | > avg_grad_norm_1: tensor(364.5600, device='cuda:0') (tensor(-16.8263, device='cuda:0'))  > EPOCH: 302/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:18:07)   --> TIME: 2025-02-25 11:18:11 -- STEP: 5/72 -- GLOBAL_STEP: 1021750 | > loss_disc: 2.8842716217041016 (2.8382942199707033) | > loss_disc_real_0: 0.20059847831726074 (0.21132732033729554) | > loss_disc_real_1: 0.22834423184394836 (0.23219535052776336) | > loss_disc_real_2: 0.2658550441265106 (0.2661381721496582) | > loss_disc_real_3: 0.27071115374565125 (0.26716736555099485) | > loss_disc_real_4: 0.2666778266429901 (0.24826813936233522) | > loss_disc_real_5: 0.2640680968761444 (0.2455129951238632) | > loss_0: 2.8842716217041016 (2.8382942199707033) | > grad_norm_0: tensor(4.0908, device='cuda:0') (tensor(10.8691, device='cuda:0')) | > loss_gen: 1.8544377088546753 (1.8404773712158202) | > loss_kl: 2.275449752807617 (2.251264238357544) | > loss_feat: 2.670358896255493 (2.620777893066406) | > loss_mel: 17.694578170776367 (18.027771759033204) | > loss_duration: 1.5516639947891235 (1.5803472757339478) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.04648780822754 (26.320638275146486) | > grad_norm_1: tensor(624.9313, device='cuda:0') (tensor(495.9346, device='cuda:0')) | > current_lr_0: 0.00019259027541165268 | > current_lr_1: 0.00019259027541165268 | > step_time: 0.4313 (0.43436808586120607) | > loader_time: 0.0082 (0.0069200992584228516)  --> TIME: 2025-02-25 11:18:34 -- STEP: 55/72 -- GLOBAL_STEP: 1021800 | > loss_disc: 2.809201717376709 (2.7916265227577903) | > loss_disc_real_0: 0.23900122940540314 (0.20542287108573046) | > loss_disc_real_1: 0.2759450078010559 (0.23381593173200435) | > loss_disc_real_2: 0.2499486654996872 (0.2418813022700223) | > loss_disc_real_3: 0.25884905457496643 (0.2418780727819963) | > loss_disc_real_4: 0.2542605698108673 (0.24399384260177612) | > loss_disc_real_5: 0.24717064201831818 (0.2479375267570669) | > loss_0: 2.809201717376709 (2.7916265227577903) | > grad_norm_0: tensor(12.1064, device='cuda:0') (tensor(12.6063, device='cuda:0')) | > loss_gen: 1.8935608863830566 (1.8351883866570213) | > loss_kl: 2.326779842376709 (2.2496099472045885) | > loss_feat: 2.4349112510681152 (2.590674766627224) | > loss_mel: 16.93731117248535 (17.658741222728388) | > loss_duration: 1.6477038860321045 (1.5995215134187177) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.240264892578125 (25.933735899491744) | > grad_norm_1: tensor(74.8637, device='cuda:0') (tensor(336.1021, device='cuda:0')) | > current_lr_0: 0.00019259027541165268 | > current_lr_1: 0.00019259027541165268 | > step_time: 0.4628 (0.44399095015092327) | > loader_time: 0.0092 (0.007960167798128996) --> EVAL PERFORMANCE | > avg_loader_time: 0.008681821151518486 (+2.5420121743651575e-06) | > avg_step_time: 0.45407174674557965 (-0.0037383361601494025) | > avg_loss_disc: 2.7951016761887244 (+0.017862360242387076) | > avg_loss_disc_real_0: 0.20627665446257928 (-0.011413612835843773) | > avg_loss_disc_real_1: 0.23067047394497295 (+0.0008757821690868473) | > avg_loss_disc_real_2: 0.24054497585330212 (+0.003016374900307456) | > avg_loss_disc_real_3: 0.2419708186471966 (+0.006029363249389241) | > avg_loss_disc_real_4: 0.24322668683360998 (-0.00047958044099136177) | > avg_loss_disc_real_5: 0.24717372466980572 (+0.0009711114033846835) | > avg_loss_0: 2.7951016761887244 (+0.017862360242387076) | > avg_grad_norm_0: tensor(13.1343, device='cuda:0') (tensor(-1.7313, device='cuda:0')) | > avg_loss_gen: 1.817357597216754 (-0.021782176595338543) | > avg_loss_kl: 2.2674187942289956 (+0.013382224969458623) | > avg_loss_feat: 2.5422418587644326 (-0.037667670720060276) | > avg_loss_mel: 17.583641280590655 (-0.09606824794284918) | > avg_loss_duration: 1.6102962090935506 (-0.001732947121203754) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.8209557869065 (-0.1438689701993674) | > avg_grad_norm_1: tensor(349.4848, device='cuda:0') (tensor(-15.0751, device='cuda:0'))  > EPOCH: 303/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:18:43)   --> TIME: 2025-02-25 11:19:01 -- STEP: 33/72 -- GLOBAL_STEP: 1021850 | > loss_disc: 2.825521469116211 (2.774527499170014) | > loss_disc_real_0: 0.2411937713623047 (0.20200977497028583) | > loss_disc_real_1: 0.236207976937294 (0.23362487102999832) | > loss_disc_real_2: 0.2014303058385849 (0.2410663136027076) | > loss_disc_real_3: 0.2214880734682083 (0.23896263268860904) | > loss_disc_real_4: 0.22648470103740692 (0.24484675671115066) | > loss_disc_real_5: 0.23762740194797516 (0.24824047991723724) | > loss_0: 2.825521469116211 (2.774527499170014) | > grad_norm_0: tensor(7.8001, device='cuda:0') (tensor(13.7019, device='cuda:0')) | > loss_gen: 1.7568612098693848 (1.8366492878306995) | > loss_kl: 2.229407787322998 (2.196628870386066) | > loss_feat: 2.45489764213562 (2.6204299637765596) | > loss_mel: 18.054203033447266 (17.73479276714903) | > loss_duration: 1.5751688480377197 (1.5752092238628503) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.070537567138672 (25.963710206927676) | > grad_norm_1: tensor(404.3659, device='cuda:0') (tensor(285.4925, device='cuda:0')) | > current_lr_0: 0.00019256620162722623 | > current_lr_1: 0.00019256620162722623 | > step_time: 0.4551 (0.4423549897742994) | > loader_time: 0.0082 (0.007215940591060754) --> EVAL PERFORMANCE | > avg_loader_time: 0.008690901205573285 (+9.080054054798684e-06) | > avg_step_time: 0.45743699140951666 (+0.003365244663937006) | > avg_loss_disc: 2.794860561129073 (-0.00024111505965151991) | > avg_loss_disc_real_0: 0.2081973760480612 (+0.001920721585481927) | > avg_loss_disc_real_1: 0.2397542849812709 (+0.009083811036297962) | > avg_loss_disc_real_2: 0.24191214941756825 (+0.0013671735642661331) | > avg_loss_disc_real_3: 0.24099404060504806 (-0.0009767780421485461) | > avg_loss_disc_real_4: 0.2448386289704014 (+0.0016119421367914166) | > avg_loss_disc_real_5: 0.24801439130809946 (+0.0008406666382937411) | > avg_loss_0: 2.794860561129073 (-0.00024111505965151991) | > avg_grad_norm_0: tensor(14.2680, device='cuda:0') (+tensor(1.1338, device='cuda:0')) | > avg_loss_gen: 1.830616425460493 (+0.01325882824373914) | > avg_loss_kl: 2.219374571048039 (-0.0480442231809568) | > avg_loss_feat: 2.535183317224744 (-0.007058541539688434) | > avg_loss_mel: 17.564323586477364 (-0.01931769411329043) | > avg_loss_duration: 1.607881139701521 (-0.002415069392029734) | > avg_amp_scaler: 183.88732394366193 (+55.887323943661926) | > avg_loss_1: 25.75737896771498 (-0.06357681919151759) | > avg_grad_norm_1: tensor(349.2706, device='cuda:0') (tensor(-0.2142, device='cuda:0'))  > EPOCH: 304/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:19:21)   --> TIME: 2025-02-25 11:19:28 -- STEP: 11/72 -- GLOBAL_STEP: 1021900 | > loss_disc: 2.83392596244812 (2.7237291119315405) | > loss_disc_real_0: 0.20136260986328125 (0.21432938900860873) | > loss_disc_real_1: 0.2272775024175644 (0.2269249910658056) | > loss_disc_real_2: 0.2584737241268158 (0.23309727013111115) | > loss_disc_real_3: 0.26230329275131226 (0.22951553626493973) | > loss_disc_real_4: 0.2541424036026001 (0.2365712266076695) | > loss_disc_real_5: 0.25282278656959534 (0.24269994686950336) | > loss_0: 2.83392596244812 (2.7237291119315405) | > grad_norm_0: tensor(9.1960, device='cuda:0') (tensor(12.8717, device='cuda:0')) | > loss_gen: 1.8208656311035156 (1.8890960108150134) | > loss_kl: 2.0669329166412354 (2.0766762386668813) | > loss_feat: 2.1930325031280518 (2.6188925829800693) | > loss_mel: 16.492326736450195 (17.47806791825728) | > loss_duration: 1.5653859376907349 (1.5841760960492222) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.1385440826416 (25.646909366954457) | > grad_norm_1: tensor(205.1776, device='cuda:0') (tensor(315.6954, device='cuda:0')) | > current_lr_0: 0.00019254213085202282 | > current_lr_1: 0.00019254213085202282 | > step_time: 0.445 (0.43289310281926935) | > loader_time: 0.0058 (0.007023442875255238)  --> TIME: 2025-02-25 11:19:51 -- STEP: 61/72 -- GLOBAL_STEP: 1021950 | > loss_disc: 2.858975648880005 (2.7939799535469927) | > loss_disc_real_0: 0.19381791353225708 (0.21317634181898149) | > loss_disc_real_1: 0.2141634076833725 (0.22867131990487458) | > loss_disc_real_2: 0.25042030215263367 (0.24057151450485478) | > loss_disc_real_3: 0.24357783794403076 (0.2378000839323294) | > loss_disc_real_4: 0.2518913447856903 (0.24410057947283886) | > loss_disc_real_5: 0.2644118070602417 (0.24750529034215896) | > loss_0: 2.858975648880005 (2.7939799535469927) | > grad_norm_0: tensor(12.5530, device='cuda:0') (tensor(13.5797, device='cuda:0')) | > loss_gen: 1.7593330144882202 (1.8338084650821374) | > loss_kl: 2.284163475036621 (2.199576053463045) | > loss_feat: 2.431107521057129 (2.5721065841737345) | > loss_mel: 16.48512840270996 (17.468689715276007) | > loss_duration: 1.6453498601913452 (1.599859478043728) | > amp_scaler: 128.0 (199.34426229508196) | > loss_1: 24.60508155822754 (25.674040372254417) | > grad_norm_1: tensor(340.1075, device='cuda:0') (tensor(345.7668, device='cuda:0')) | > current_lr_0: 0.00019254213085202282 | > current_lr_1: 0.00019254213085202282 | > step_time: 0.4663 (0.446649254345503) | > loader_time: 0.0108 (0.008311564805077729) --> EVAL PERFORMANCE | > avg_loader_time: 0.008764005043137244 (+7.310383756395887e-05) | > avg_step_time: 0.4553162380003593 (-0.00212075340915735) | > avg_loss_disc: 2.793761619379822 (-0.001098941749250848) | > avg_loss_disc_real_0: 0.21110661096975836 (+0.002909234921697146) | > avg_loss_disc_real_1: 0.23097049329482333 (-0.008783791686447578) | > avg_loss_disc_real_2: 0.23966993412501375 (-0.0022422152925545002) | > avg_loss_disc_real_3: 0.23874043641795575 (-0.0022536041870923063) | > avg_loss_disc_real_4: 0.24428246285713895 (-0.0005561661132624518) | > avg_loss_disc_real_5: 0.24806485125716304 (+5.045994906358042e-05) | > avg_loss_0: 2.793761619379822 (-0.001098941749250848) | > avg_grad_norm_0: tensor(13.4463, device='cuda:0') (tensor(-0.8218, device='cuda:0')) | > avg_loss_gen: 1.830019477387549 (-0.0005969480729439702) | > avg_loss_kl: 2.2077442760198887 (-0.011630295028150162) | > avg_loss_feat: 2.549644658263301 (+0.014461341038556785) | > avg_loss_mel: 17.440798396795575 (-0.12352518968178927) | > avg_loss_duration: 1.6076338694129193 (-0.0002472702886016087) | > avg_amp_scaler: 189.29577464788733 (+5.408450704225402) | > avg_loss_1: 25.635840751755406 (-0.12153821595957481) | > avg_grad_norm_1: tensor(363.5209, device='cuda:0') (+tensor(14.2503, device='cuda:0'))  > EPOCH: 305/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:19:58)   --> TIME: 2025-02-25 11:20:18 -- STEP: 39/72 -- GLOBAL_STEP: 1022000 | > loss_disc: 2.7166244983673096 (2.7849270747258115) | > loss_disc_real_0: 0.22504165768623352 (0.20089152589058265) | > loss_disc_real_1: 0.2417709231376648 (0.22912066181500754) | > loss_disc_real_2: 0.19717860221862793 (0.2385991402925589) | > loss_disc_real_3: 0.18630057573318481 (0.24059665164886376) | > loss_disc_real_4: 0.24801194667816162 (0.2451918465204728) | > loss_disc_real_5: 0.24191969633102417 (0.24817870595516303) | > loss_0: 2.7166244983673096 (2.7849270747258115) | > grad_norm_0: tensor(11.9012, device='cuda:0') (tensor(14.5242, device='cuda:0')) | > loss_gen: 1.8408498764038086 (1.8271986979704637) | > loss_kl: 2.4335341453552246 (2.2452923334561863) | > loss_feat: 2.7084009647369385 (2.652381823613093) | > loss_mel: 18.20085906982422 (17.79315146421775) | > loss_duration: 1.6091971397399902 (1.5905956488389235) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.7928409576416 (26.108619934473282) | > grad_norm_1: tensor(447.4376, device='cuda:0') (tensor(345.8621, device='cuda:0')) | > current_lr_0: 0.0001925180630856663 | > current_lr_1: 0.0001925180630856663 | > step_time: 0.4818 (0.44212370041089183) | > loader_time: 0.0084 (0.007390132317176232) --> EVAL PERFORMANCE | > avg_loader_time: 0.008683483365555885 (-8.052167758135914e-05) | > avg_step_time: 0.4592793122143813 (+0.003963074214021978) | > avg_loss_disc: 2.796190641295742 (+0.002429021915919982) | > avg_loss_disc_real_0: 0.2137983437155334 (+0.002691732745775055) | > avg_loss_disc_real_1: 0.2315946448017174 (+0.0006241515068940717) | > avg_loss_disc_real_2: 0.24136013044437893 (+0.0016901963193651781) | > avg_loss_disc_real_3: 0.24242206043760542 (+0.0036816240196496652) | > avg_loss_disc_real_4: 0.2443544818481929 (+7.201899105394038e-05) | > avg_loss_disc_real_5: 0.24775062228592348 (-0.00031422897123956073) | > avg_loss_0: 2.796190641295742 (+0.002429021915919982) | > avg_grad_norm_0: tensor(15.2991, device='cuda:0') (+tensor(1.8528, device='cuda:0')) | > avg_loss_gen: 1.8376089754238936 (+0.007589498036344455) | > avg_loss_kl: 2.287173106636799 (+0.07942883061691042) | > avg_loss_feat: 2.5661214153531566 (+0.01647675708985563) | > avg_loss_mel: 17.775790093650276 (+0.33499169685470065) | > avg_loss_duration: 1.614523880918261 (+0.006890011505341631) | > avg_amp_scaler: 128.0 (-61.29577464788733) | > avg_loss_1: 26.081217403143224 (+0.4453766513878179) | > avg_grad_norm_1: tensor(336.7895, device='cuda:0') (tensor(-26.7314, device='cuda:0'))  > EPOCH: 306/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:20:35)   --> TIME: 2025-02-25 11:20:45 -- STEP: 17/72 -- GLOBAL_STEP: 1022050 | > loss_disc: 2.714566230773926 (2.789771963568295) | > loss_disc_real_0: 0.22383320331573486 (0.2014906248625587) | > loss_disc_real_1: 0.24817675352096558 (0.23882916394401998) | > loss_disc_real_2: 0.22834812104701996 (0.23443563720759222) | > loss_disc_real_3: 0.24388152360916138 (0.24037092573502483) | > loss_disc_real_4: 0.239724263548851 (0.24569116971072028) | > loss_disc_real_5: 0.2346046417951584 (0.24614364434691036) | > loss_0: 2.714566230773926 (2.789771963568295) | > grad_norm_0: tensor(13.0948, device='cuda:0') (tensor(11.3878, device='cuda:0')) | > loss_gen: 1.9302815198898315 (1.8224847456988167) | > loss_kl: 2.4050145149230957 (2.233414678012623) | > loss_feat: 2.5019519329071045 (2.557906557531918) | > loss_mel: 17.659563064575195 (18.00323194615981) | > loss_duration: 1.6036936044692993 (1.564373072455911) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.100502014160156 (26.181410845588236) | > grad_norm_1: tensor(486.7065, device='cuda:0') (tensor(381.5222, device='cuda:0')) | > current_lr_0: 0.0001924939983277806 | > current_lr_1: 0.0001924939983277806 | > step_time: 0.4308 (0.4635729649487664) | > loader_time: 0.0062 (0.0067102909088134766)  --> TIME: 2025-02-25 11:21:09 -- STEP: 67/72 -- GLOBAL_STEP: 1022100 | > loss_disc: 2.7971878051757812 (2.7960564093803293) | > loss_disc_real_0: 0.12305481731891632 (0.20521344444644984) | > loss_disc_real_1: 0.23755639791488647 (0.23598421020294302) | > loss_disc_real_2: 0.261913925409317 (0.23675109935340596) | > loss_disc_real_3: 0.2592277526855469 (0.23836056443292702) | > loss_disc_real_4: 0.2398139387369156 (0.2438629798924745) | > loss_disc_real_5: 0.24216532707214355 (0.24589013853179875) | > loss_0: 2.7971878051757812 (2.7960564093803293) | > grad_norm_0: tensor(25.9857, device='cuda:0') (tensor(16.0469, device='cuda:0')) | > loss_gen: 1.8236453533172607 (1.8130733753318218) | > loss_kl: 2.4927828311920166 (2.2875883543669286) | > loss_feat: 2.635488986968994 (2.536227228036567) | > loss_mel: 19.173730850219727 (17.871062905041132) | > loss_duration: 1.7070777416229248 (1.6061001766973466) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.83272361755371 (26.114052103526557) | > grad_norm_1: tensor(466.7992, device='cuda:0') (tensor(353.2534, device='cuda:0')) | > current_lr_0: 0.0001924939983277806 | > current_lr_1: 0.0001924939983277806 | > step_time: 0.502 (0.45970170177630526) | > loader_time: 0.0126 (0.008404610761955603) --> EVAL PERFORMANCE | > avg_loader_time: 0.008581984211021744 (-0.00010149915453414038) | > avg_step_time: 0.4630339044920156 (+0.0037545922776343055) | > avg_loss_disc: 2.7969978762344576 (+0.0008072349387155953) | > avg_loss_disc_real_0: 0.20738749017178174 (-0.006410853543751677) | > avg_loss_disc_real_1: 0.23609684176847967 (+0.0045021969667622674) | > avg_loss_disc_real_2: 0.23841803825237382 (-0.0029420921920051135) | > avg_loss_disc_real_3: 0.23996631969028795 (-0.0024557407473174697) | > avg_loss_disc_real_4: 0.24479890835117285 (+0.000444426502979961) | > avg_loss_disc_real_5: 0.2461996124663823 (-0.001551009819541177) | > avg_loss_0: 2.7969978762344576 (+0.0008072349387155953) | > avg_grad_norm_0: tensor(16.2209, device='cuda:0') (+tensor(0.9219, device='cuda:0')) | > avg_loss_gen: 1.8204525591621936 (-0.017156416261699947) | > avg_loss_kl: 2.3044681314011695 (+0.017295024764370392) | > avg_loss_feat: 2.537417672049832 (-0.028703743303324813) | > avg_loss_mel: 17.969856759192233 (+0.19406666554195695) | > avg_loss_duration: 1.6094542741775513 (-0.0050696067407096645) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.24164943963709 (+0.16043203649386584) | > avg_grad_norm_1: tensor(369.2565, device='cuda:0') (+tensor(32.4670, device='cuda:0'))  > EPOCH: 307/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:21:13)   --> TIME: 2025-02-25 11:21:35 -- STEP: 45/72 -- GLOBAL_STEP: 1022150 | > loss_disc: 2.8718981742858887 (2.782336807250977) | > loss_disc_real_0: 0.15277135372161865 (0.19962487005525165) | > loss_disc_real_1: 0.28001850843429565 (0.23069090743859608) | > loss_disc_real_2: 0.24481381475925446 (0.23787817590766483) | > loss_disc_real_3: 0.27926725149154663 (0.23830068873034582) | > loss_disc_real_4: 0.2709594666957855 (0.24546574585967593) | > loss_disc_real_5: 0.2736271917819977 (0.24425609310468038) | > loss_0: 2.8718981742858887 (2.782336807250977) | > grad_norm_0: tensor(16.9217, device='cuda:0') (tensor(13.7737, device='cuda:0')) | > loss_gen: 1.8294312953948975 (1.8105141056908503) | > loss_kl: 2.269564628601074 (2.22943164507548) | > loss_feat: 2.4366295337677 (2.5651179207695853) | > loss_mel: 18.008766174316406 (17.743730290730795) | > loss_duration: 1.6262654066085815 (1.585448760456509) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.170656204223633 (25.93424284193251) | > grad_norm_1: tensor(377.4037, device='cuda:0') (tensor(365.1352, device='cuda:0')) | > current_lr_0: 0.0001924699365779896 | > current_lr_1: 0.0001924699365779896 | > step_time: 0.4657 (0.4445762899186876) | > loader_time: 0.0085 (0.00764365726047092) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870594172410562 (+0.00012395751308387493) | > avg_step_time: 0.45712445823239606 (-0.005909446259619533) | > avg_loss_disc: 2.7855447715436914 (-0.011453104690766214) | > avg_loss_disc_real_0: 0.20907404252760847 (+0.001686552355826737) | > avg_loss_disc_real_1: 0.23082560531690088 (-0.005271236451578787) | > avg_loss_disc_real_2: 0.2392749593291484 (+0.0008569210767745972) | > avg_loss_disc_real_3: 0.23848650300166976 (-0.0014798166886181852) | > avg_loss_disc_real_4: 0.2446151337992977 (-0.00018377455187515435) | > avg_loss_disc_real_5: 0.24618305538741636 (-1.655707896594527e-05) | > avg_loss_0: 2.7855447715436914 (-0.011453104690766214) | > avg_grad_norm_0: tensor(14.5943, device='cuda:0') (tensor(-1.6266, device='cuda:0')) | > avg_loss_gen: 1.8208402512778699 (+0.0003876921156762414) | > avg_loss_kl: 2.251798817809198 (-0.052669313591971356) | > avg_loss_feat: 2.54391046980737 (+0.006492797757537971) | > avg_loss_mel: 17.582447266914468 (-0.3874094922777651) | > avg_loss_duration: 1.6055558953486695 (-0.0038983788288817767) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.804552776712768 (-0.43709666292432203) | > avg_grad_norm_1: tensor(348.9596, device='cuda:0') (tensor(-20.2969, device='cuda:0'))  > EPOCH: 308/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:21:50)   --> TIME: 2025-02-25 11:22:02 -- STEP: 23/72 -- GLOBAL_STEP: 1022200 | > loss_disc: 2.8224430084228516 (2.789994343467381) | > loss_disc_real_0: 0.20535843074321747 (0.2019353968941647) | > loss_disc_real_1: 0.24426808953285217 (0.24246458838815274) | > loss_disc_real_2: 0.3326278626918793 (0.24643159819685895) | > loss_disc_real_3: 0.25878363847732544 (0.24287556241387906) | > loss_disc_real_4: 0.24861139059066772 (0.24231047798757968) | > loss_disc_real_5: 0.2555208206176758 (0.24651320149069247) | > loss_0: 2.8224430084228516 (2.789994343467381) | > grad_norm_0: tensor(6.3283, device='cuda:0') (tensor(11.7727, device='cuda:0')) | > loss_gen: 1.8374205827713013 (1.8440016404442165) | > loss_kl: 2.5303685665130615 (2.3629869481791617) | > loss_feat: 2.56553316116333 (2.6489595226619556) | > loss_mel: 18.076509475708008 (18.19158371635105) | > loss_duration: 1.607241153717041 (1.5863427804863972) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.61707305908203 (26.633874893188477) | > grad_norm_1: tensor(474.5434, device='cuda:0') (tensor(365.4866, device='cuda:0')) | > current_lr_0: 0.00019244587783591734 | > current_lr_1: 0.00019244587783591734 | > step_time: 0.4317 (0.4331400187119194) | > loader_time: 0.0066 (0.0065601908642312756) --> EVAL PERFORMANCE | > avg_loader_time: 0.008516271349409936 (-0.00018967037469568362) | > avg_step_time: 0.4515335425524644 (-0.005590915679931641) | > avg_loss_disc: 2.783342290932025 (-0.002202480611666502) | > avg_loss_disc_real_0: 0.20283795956154943 (-0.006236082966059042) | > avg_loss_disc_real_1: 0.23302291354662935 (+0.0021973082297284674) | > avg_loss_disc_real_2: 0.23869466424827845 (-0.0005802950808699581) | > avg_loss_disc_real_3: 0.24055337821933584 (+0.0020668752176660754) | > avg_loss_disc_real_4: 0.2440665473400707 (-0.0005485864592270073) | > avg_loss_disc_real_5: 0.24480130512949447 (-0.0013817502579218932) | > avg_loss_0: 2.783342290932025 (-0.002202480611666502) | > avg_grad_norm_0: tensor(14.4163, device='cuda:0') (tensor(-0.1781, device='cuda:0')) | > avg_loss_gen: 1.8282865940685002 (+0.007446342790630389) | > avg_loss_kl: 2.3188960535425536 (+0.06709723573335546) | > avg_loss_feat: 2.590196327424385 (+0.046285857617015314) | > avg_loss_mel: 17.722730179907565 (+0.14028291299309714) | > avg_loss_duration: 1.612007072274114 (+0.006451176925444502) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.072116314525335 (+0.2675635378125669) | > avg_grad_norm_1: tensor(353.6737, device='cuda:0') (+tensor(4.7141, device='cuda:0'))  > EPOCH: 309/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:22:26)   --> TIME: 2025-02-25 11:22:29 -- STEP: 1/72 -- GLOBAL_STEP: 1022250 | > loss_disc: 2.812352180480957 (2.812352180480957) | > loss_disc_real_0: 0.23671990633010864 (0.23671990633010864) | > loss_disc_real_1: 0.27631303668022156 (0.27631303668022156) | > loss_disc_real_2: 0.24260514974594116 (0.24260514974594116) | > loss_disc_real_3: 0.22649961709976196 (0.22649961709976196) | > loss_disc_real_4: 0.23564523458480835 (0.23564523458480835) | > loss_disc_real_5: 0.24797473847866058 (0.24797473847866058) | > loss_0: 2.812352180480957 (2.812352180480957) | > grad_norm_0: tensor(6.8810, device='cuda:0') (tensor(6.8810, device='cuda:0')) | > loss_gen: 1.7475473880767822 (1.7475473880767822) | > loss_kl: 2.3705313205718994 (2.3705313205718994) | > loss_feat: 2.7592246532440186 (2.7592246532440186) | > loss_mel: 18.68450355529785 (18.68450355529785) | > loss_duration: 1.6119458675384521 (1.6119458675384521) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.173751831054688 (27.173751831054688) | > grad_norm_1: tensor(172.6877, device='cuda:0') (tensor(172.6877, device='cuda:0')) | > current_lr_0: 0.00019242182210118784 | > current_lr_1: 0.00019242182210118784 | > step_time: 0.8906 (0.8906130790710449) | > loader_time: 0.0058 (0.005829334259033203)  --> TIME: 2025-02-25 11:22:52 -- STEP: 51/72 -- GLOBAL_STEP: 1022300 | > loss_disc: 2.8341901302337646 (2.8059458545610014) | > loss_disc_real_0: 0.21020697057247162 (0.20541890958944956) | > loss_disc_real_1: 0.21220605075359344 (0.23484079861173443) | > loss_disc_real_2: 0.23227429389953613 (0.23938163998080234) | > loss_disc_real_3: 0.20155632495880127 (0.23792997411653108) | > loss_disc_real_4: 0.2409362494945526 (0.24613984571952446) | > loss_disc_real_5: 0.24561864137649536 (0.25104947651133824) | > loss_0: 2.8341901302337646 (2.8059458545610014) | > grad_norm_0: tensor(7.7698, device='cuda:0') (tensor(16.7024, device='cuda:0')) | > loss_gen: 1.811939001083374 (1.8181760474747302) | > loss_kl: 1.9874277114868164 (2.291738238989138) | > loss_feat: 2.6557750701904297 (2.567039321450627) | > loss_mel: 17.500699996948242 (17.913858301499314) | > loss_duration: 1.6246654987335205 (1.5968877591338813) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.580507278442383 (26.187699486227597) | > grad_norm_1: tensor(750.5444, device='cuda:0') (tensor(288.2148, device='cuda:0')) | > current_lr_0: 0.00019242182210118784 | > current_lr_1: 0.00019242182210118784 | > step_time: 0.4519 (0.4569060054479861) | > loader_time: 0.0092 (0.007845565384509517) --> EVAL PERFORMANCE | > avg_loader_time: 0.008658019589706205 (+0.00014174824029626956) | > avg_step_time: 0.46533447252192967 (+0.013800929969465248) | > avg_loss_disc: 2.7927455599878876 (+0.00940326905586275) | > avg_loss_disc_real_0: 0.20417562158594668 (+0.0013376620243972515) | > avg_loss_disc_real_1: 0.23136161435657823 (-0.0016612991900511187) | > avg_loss_disc_real_2: 0.23802723011500398 (-0.0006674341332744693) | > avg_loss_disc_real_3: 0.23804715086876507 (-0.0025062273505707666) | > avg_loss_disc_real_4: 0.2441389881808993 (+7.244084082860813e-05) | > avg_loss_disc_real_5: 0.248522592682234 (+0.0037212875527395384) | > avg_loss_0: 2.7927455599878876 (+0.00940326905586275) | > avg_grad_norm_0: tensor(14.9849, device='cuda:0') (+tensor(0.5686, device='cuda:0')) | > avg_loss_gen: 1.8164964964692019 (-0.011790097599298388) | > avg_loss_kl: 2.2982084986189726 (-0.020687554923580986) | > avg_loss_feat: 2.5475108757824967 (-0.04268545164188842) | > avg_loss_mel: 17.836148517232544 (+0.11341833732497975) | > avg_loss_duration: 1.6109911035483993 (-0.0010159687257147443) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.109355308640172 (+0.037238994114837) | > avg_grad_norm_1: tensor(289.7894, device='cuda:0') (tensor(-63.8843, device='cuda:0'))  > EPOCH: 310/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:23:04)   --> TIME: 2025-02-25 11:23:19 -- STEP: 29/72 -- GLOBAL_STEP: 1022350 | > loss_disc: 2.7297897338867188 (2.759036154582583) | > loss_disc_real_0: 0.10761744529008865 (0.20069367130254878) | > loss_disc_real_1: 0.2021998167037964 (0.23644366809006395) | > loss_disc_real_2: 0.25036072731018066 (0.24209378762491818) | > loss_disc_real_3: 0.23943480849266052 (0.24175568233276234) | > loss_disc_real_4: 0.25040891766548157 (0.24434713916531925) | > loss_disc_real_5: 0.25482621788978577 (0.24580288704099326) | > loss_0: 2.7297897338867188 (2.759036154582583) | > grad_norm_0: tensor(28.1336, device='cuda:0') (tensor(12.4611, device='cuda:0')) | > loss_gen: 1.67729651927948 (1.8538562922642148) | > loss_kl: 2.2517764568328857 (2.1947122483417907) | > loss_feat: 2.5599355697631836 (2.6582064546387776) | > loss_mel: 17.852956771850586 (17.60286449563914) | > loss_duration: 1.5760774612426758 (1.57023981110803) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.918041229248047 (25.879878932032092) | > grad_norm_1: tensor(782.9058, device='cuda:0') (tensor(377.2680, device='cuda:0')) | > current_lr_0: 0.00019239776937342518 | > current_lr_1: 0.00019239776937342518 | > step_time: 0.4403 (0.4408299347449993) | > loader_time: 0.0074 (0.006871872934801826) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542732453682051 (-0.00011528713602415412) | > avg_step_time: 0.4575589206856741 (-0.0077755518362555565) | > avg_loss_disc: 2.7637681625258756 (-0.028977397462012) | > avg_loss_disc_real_0: 0.1994083563626652 (-0.004767265223281469) | > avg_loss_disc_real_1: 0.23541070499890288 (+0.004049090642324643) | > avg_loss_disc_real_2: 0.24085180188568545 (+0.002824571770681461) | > avg_loss_disc_real_3: 0.2409902422780722 (+0.0029430914093071248) | > avg_loss_disc_real_4: 0.24400408272172364 (-0.00013490545917566044) | > avg_loss_disc_real_5: 0.247088406077573 (-0.0014341866046610197) | > avg_loss_0: 2.7637681625258756 (-0.028977397462012) | > avg_grad_norm_0: tensor(13.4452, device='cuda:0') (tensor(-1.5397, device='cuda:0')) | > avg_loss_gen: 1.852630422148906 (+0.03613392567970419) | > avg_loss_kl: 2.206682389890644 (-0.09152610872832856) | > avg_loss_feat: 2.630823592065086 (+0.08331271628258952) | > avg_loss_mel: 17.465188281636845 (-0.3709602355956996) | > avg_loss_duration: 1.6044941603297918 (-0.006496943218607498) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.75981868152887 (-0.34953662711130207) | > avg_grad_norm_1: tensor(378.6428, device='cuda:0') (+tensor(88.8534, device='cuda:0'))  > EPOCH: 311/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:23:41)   --> TIME: 2025-02-25 11:23:46 -- STEP: 7/72 -- GLOBAL_STEP: 1022400 | > loss_disc: 2.8228402137756348 (2.8046316419328963) | > loss_disc_real_0: 0.2399444282054901 (0.21583811513015203) | > loss_disc_real_1: 0.2232147604227066 (0.23335739757333482) | > loss_disc_real_2: 0.2369728982448578 (0.23503967268126352) | > loss_disc_real_3: 0.25717008113861084 (0.22925718554428645) | > loss_disc_real_4: 0.2334223836660385 (0.2407088897057942) | > loss_disc_real_5: 0.2287445366382599 (0.24637488169329508) | > loss_0: 2.8228402137756348 (2.8046316419328963) | > grad_norm_0: tensor(12.2771, device='cuda:0') (tensor(14.3972, device='cuda:0')) | > loss_gen: 1.7181864976882935 (1.8117273535047258) | > loss_kl: 2.0271384716033936 (2.116034814289638) | > loss_feat: 2.6417698860168457 (2.613539559500558) | > loss_mel: 17.4798641204834 (17.679351534162247) | > loss_duration: 1.5742747783660889 (1.5832408666610718) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.441234588623047 (25.803893770490372) | > grad_norm_1: tensor(454.8232, device='cuda:0') (tensor(529.7851, device='cuda:0')) | > current_lr_0: 0.0001923737196522535 | > current_lr_1: 0.0001923737196522535 | > step_time: 0.4325 (0.4345123767852783) | > loader_time: 0.0071 (0.0062307630266462055)  --> TIME: 2025-02-25 11:24:09 -- STEP: 57/72 -- GLOBAL_STEP: 1022450 | > loss_disc: 2.767796039581299 (2.780438096899736) | > loss_disc_real_0: 0.2100738286972046 (0.20647316726676204) | > loss_disc_real_1: 0.22184212505817413 (0.23640079404178418) | > loss_disc_real_2: 0.2645101845264435 (0.238970880707105) | > loss_disc_real_3: 0.23244325816631317 (0.23658172681666256) | > loss_disc_real_4: 0.23750707507133484 (0.2433506919626604) | > loss_disc_real_5: 0.24218949675559998 (0.24737932310815444) | > loss_0: 2.767796039581299 (2.780438096899736) | > grad_norm_0: tensor(7.4406, device='cuda:0') (tensor(11.6798, device='cuda:0')) | > loss_gen: 1.8771140575408936 (1.8419056461568464) | > loss_kl: 2.2977921962738037 (2.2104467279032667) | > loss_feat: 2.4987053871154785 (2.6448110362939667) | > loss_mel: 16.160301208496094 (17.763907666791958) | > loss_duration: 1.664053201675415 (1.601574655164752) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.49796485900879 (26.062645761590254) | > grad_norm_1: tensor(502.9589, device='cuda:0') (tensor(352.3177, device='cuda:0')) | > current_lr_0: 0.0001923737196522535 | > current_lr_1: 0.0001923737196522535 | > step_time: 0.4633 (0.4480940919173391) | > loader_time: 0.0097 (0.007954860988416168) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599482791524535 (+5.67503378424844e-05) | > avg_step_time: 0.4591891799174564 (+0.0016302592317822828) | > avg_loss_disc: 2.7880536603256014 (+0.024285497799725775) | > avg_loss_disc_real_0: 0.21009777675212268 (+0.010689420389457471) | > avg_loss_disc_real_1: 0.23620411480816317 (+0.0007934098092602926) | > avg_loss_disc_real_2: 0.24234761948316869 (+0.00149581759748324) | > avg_loss_disc_real_3: 0.23797412363576217 (-0.003016118642310023) | > avg_loss_disc_real_4: 0.24375234670202497 (-0.00025173601969866177) | > avg_loss_disc_real_5: 0.24778619219719525 (+0.0006977861196222623) | > avg_loss_0: 2.7880536603256014 (+0.024285497799725775) | > avg_grad_norm_0: tensor(12.4570, device='cuda:0') (tensor(-0.9882, device='cuda:0')) | > avg_loss_gen: 1.843741002217145 (-0.008889419931761067) | > avg_loss_kl: 2.2419354529447966 (+0.035253063054152545) | > avg_loss_feat: 2.606372833251953 (-0.024450758813133078) | > avg_loss_mel: 17.761826031644585 (+0.29663775000773995) | > avg_loss_duration: 1.6093127324547567 (+0.004818572124964948) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.06318817675953 (+0.3033694952306618) | > avg_grad_norm_1: tensor(343.6844, device='cuda:0') (tensor(-34.9584, device='cuda:0'))  > EPOCH: 312/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:24:18)   --> TIME: 2025-02-25 11:24:37 -- STEP: 35/72 -- GLOBAL_STEP: 1022500 | > loss_disc: 2.8525450229644775 (2.7776399203709192) | > loss_disc_real_0: 0.17049747705459595 (0.2034930889095579) | > loss_disc_real_1: 0.24959266185760498 (0.23477255531719754) | > loss_disc_real_2: 0.2602807581424713 (0.23858139770371573) | > loss_disc_real_3: 0.2918750047683716 (0.23794974769864763) | > loss_disc_real_4: 0.2721237242221832 (0.2449649180684771) | > loss_disc_real_5: 0.26282384991645813 (0.24819377958774566) | > loss_0: 2.8525450229644775 (2.7776399203709192) | > grad_norm_0: tensor(13.3771, device='cuda:0') (tensor(12.7554, device='cuda:0')) | > loss_gen: 1.9419198036193848 (1.8406255892344883) | > loss_kl: 2.2660422325134277 (2.165844099862235) | > loss_feat: 2.3236382007598877 (2.616005277633667) | > loss_mel: 17.862173080444336 (17.421124867030557) | > loss_duration: 1.6234259605407715 (1.5706582648413523) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.017200469970703 (25.614258193969725) | > grad_norm_1: tensor(525.2466, device='cuda:0') (tensor(450.8859, device='cuda:0')) | > current_lr_0: 0.00019234967293729698 | > current_lr_1: 0.00019234967293729698 | > step_time: 0.4422 (0.45567690304347447) | > loader_time: 0.0083 (0.007480239868164063) --> EVAL PERFORMANCE | > avg_loader_time: 0.008746456092512105 (+0.00014697330098756983) | > avg_step_time: 0.46413307794382874 (+0.004943898026372351) | > avg_loss_disc: 2.785013971194415 (-0.0030396891311865737) | > avg_loss_disc_real_0: 0.20916823077369745 (-0.0009295459784252336) | > avg_loss_disc_real_1: 0.23385634178846654 (-0.0023477730196966307) | > avg_loss_disc_real_2: 0.23720374661432186 (-0.005143872868846822) | > avg_loss_disc_real_3: 0.2388136626129419 (+0.0008395389771797268) | > avg_loss_disc_real_4: 0.24689506372095835 (+0.003142717018933372) | > avg_loss_disc_real_5: 0.24746857142784226 (-0.00031762076935298866) | > avg_loss_0: 2.785013971194415 (-0.0030396891311865737) | > avg_grad_norm_0: tensor(14.2993, device='cuda:0') (+tensor(1.8422, device='cuda:0')) | > avg_loss_gen: 1.8321640457905515 (-0.01157695642659351) | > avg_loss_kl: 2.2626914961237303 (+0.02075604317893376) | > avg_loss_feat: 2.562951363308329 (-0.04342146994362395) | > avg_loss_mel: 17.632562449280645 (-0.1292635823639401) | > avg_loss_duration: 1.601891792995829 (-0.007420939458927744) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.892261182758165 (-0.17092699400136624) | > avg_grad_norm_1: tensor(440.4575, device='cuda:0') (+tensor(96.7731, device='cuda:0'))  > EPOCH: 313/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:24:56)   --> TIME: 2025-02-25 11:25:04 -- STEP: 13/72 -- GLOBAL_STEP: 1022550 | > loss_disc: 2.6874871253967285 (2.751759254015409) | > loss_disc_real_0: 0.21335364878177643 (0.21570993272157815) | > loss_disc_real_1: 0.24163590371608734 (0.22422070915882403) | > loss_disc_real_2: 0.234480619430542 (0.24558360645404229) | > loss_disc_real_3: 0.22947457432746887 (0.24012672557280615) | > loss_disc_real_4: 0.24823987483978271 (0.24222592092477357) | > loss_disc_real_5: 0.2428431361913681 (0.24732059698838454) | > loss_0: 2.6874871253967285 (2.751759254015409) | > grad_norm_0: tensor(10.7215, device='cuda:0') (tensor(15.0561, device='cuda:0')) | > loss_gen: 1.9351800680160522 (1.8977920917364268) | > loss_kl: 2.314706802368164 (2.089987268814674) | > loss_feat: 2.877817392349243 (2.725639141522921) | > loss_mel: 17.283340454101562 (17.576361876267654) | > loss_duration: 1.4844591617584229 (1.565911577298091) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.895503997802734 (25.855691763070915) | > grad_norm_1: tensor(70.4852, device='cuda:0') (tensor(530.4708, device='cuda:0')) | > current_lr_0: 0.0001923256292281798 | > current_lr_1: 0.0001923256292281798 | > step_time: 0.4587 (0.4358382041637714) | > loader_time: 0.0059 (0.006476347263042743)  --> TIME: 2025-02-25 11:25:27 -- STEP: 63/72 -- GLOBAL_STEP: 1022600 | > loss_disc: 2.7790751457214355 (2.7917571635473344) | > loss_disc_real_0: 0.13725298643112183 (0.21813612644161498) | > loss_disc_real_1: 0.21417880058288574 (0.23265256295128475) | > loss_disc_real_2: 0.225236713886261 (0.24331075591700418) | > loss_disc_real_3: 0.2544078230857849 (0.24037623973119826) | > loss_disc_real_4: 0.2466750144958496 (0.24577408177512033) | > loss_disc_real_5: 0.25058630108833313 (0.24790337397938683) | > loss_0: 2.7790751457214355 (2.7917571635473344) | > grad_norm_0: tensor(14.6459, device='cuda:0') (tensor(16.1472, device='cuda:0')) | > loss_gen: 1.7506158351898193 (1.8492124364489602) | > loss_kl: 2.2509119510650635 (2.223680064791723) | > loss_feat: 2.308478593826294 (2.5726437076689703) | > loss_mel: 17.905010223388672 (17.728233761257595) | > loss_duration: 1.6055538654327393 (1.601340201165941) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.820571899414062 (25.97511012970455) | > grad_norm_1: tensor(279.3955, device='cuda:0') (tensor(376.2566, device='cuda:0')) | > current_lr_0: 0.0001923256292281798 | > current_lr_1: 0.0001923256292281798 | > step_time: 0.5254 (0.45099812083774143) | > loader_time: 0.0112 (0.008352120717366539) --> EVAL PERFORMANCE | > avg_loader_time: 0.008715320640886336 (-3.113545162576961e-05) | > avg_step_time: 0.4589999326517884 (-0.005133145292040342) | > avg_loss_disc: 2.7900328401108863 (+0.005018868916471408) | > avg_loss_disc_real_0: 0.21896164213687602 (+0.009793411363178572) | > avg_loss_disc_real_1: 0.23110103103476512 (-0.0027553107537014176) | > avg_loss_disc_real_2: 0.2422886954646715 (+0.00508494885034963) | > avg_loss_disc_real_3: 0.23932080348612558 (+0.0005071408731836813) | > avg_loss_disc_real_4: 0.24347793888038313 (-0.003417124840575214) | > avg_loss_disc_real_5: 0.24582923843827045 (-0.0016393329895718067) | > avg_loss_0: 2.7900328401108863 (+0.005018868916471408) | > avg_grad_norm_0: tensor(15.5405, device='cuda:0') (+tensor(1.2412, device='cuda:0')) | > avg_loss_gen: 1.8414539535280685 (+0.009289907737517034) | > avg_loss_kl: 2.2357156981884576 (-0.026975797935272716) | > avg_loss_feat: 2.5694078559606846 (+0.006456492652355372) | > avg_loss_mel: 17.758153378123968 (+0.125590928843323) | > avg_loss_duration: 1.6092047926405786 (+0.007312999644749629) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.013935626392634 (+0.12167444363446833) | > avg_grad_norm_1: tensor(378.0887, device='cuda:0') (tensor(-62.3687, device='cuda:0'))  > EPOCH: 314/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:25:33)   --> TIME: 2025-02-25 11:25:54 -- STEP: 41/72 -- GLOBAL_STEP: 1022650 | > loss_disc: 2.790095806121826 (2.783950910335633) | > loss_disc_real_0: 0.3054192364215851 (0.21943728971045193) | > loss_disc_real_1: 0.3105177879333496 (0.230414620259913) | > loss_disc_real_2: 0.20020298659801483 (0.23650725403936898) | > loss_disc_real_3: 0.23589962720870972 (0.23831732389403554) | > loss_disc_real_4: 0.24423132836818695 (0.24841563294573524) | > loss_disc_real_5: 0.24875158071517944 (0.2512252069101102) | > loss_0: 2.790095806121826 (2.783950910335633) | > grad_norm_0: tensor(19.4921, device='cuda:0') (tensor(16.2697, device='cuda:0')) | > loss_gen: 1.8181064128875732 (1.8530196998177506) | > loss_kl: 2.0300610065460205 (2.1996765253020505) | > loss_feat: 2.2508175373077393 (2.6030016817697663) | > loss_mel: 16.786014556884766 (17.504034554086086) | > loss_duration: 1.591118574142456 (1.579061894882016) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.476116180419922 (25.738794140699433) | > grad_norm_1: tensor(447.9107, device='cuda:0') (tensor(443.8242, device='cuda:0')) | > current_lr_0: 0.0001923015885245263 | > current_lr_1: 0.0001923015885245263 | > step_time: 0.4788 (0.4470879450076964) | > loader_time: 0.0095 (0.0073886150267066025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630799575590755 (-8.45210652955803e-05) | > avg_step_time: 0.46052056299129 (+0.0015206303395016163) | > avg_loss_disc: 2.782291375415426 (-0.00774146469546011) | > avg_loss_disc_real_0: 0.2114067663189391 (-0.007554875817936929) | > avg_loss_disc_real_1: 0.23012756398865875 (-0.0009734670461063744) | > avg_loss_disc_real_2: 0.2385863229422502 (-0.0037023725224212822) | > avg_loss_disc_real_3: 0.23704502888968293 (-0.0022757745964426535) | > avg_loss_disc_real_4: 0.24700547020200272 (+0.0035275313216195925) | > avg_loss_disc_real_5: 0.2504458870266525 (+0.0046166485883820385) | > avg_loss_0: 2.782291375415426 (-0.00774146469546011) | > avg_grad_norm_0: tensor(15.2625, device='cuda:0') (tensor(-0.2780, device='cuda:0')) | > avg_loss_gen: 1.8401792368418735 (-0.0012747166861950543) | > avg_loss_kl: 2.2587792671902087 (+0.023063569001751105) | > avg_loss_feat: 2.580895677418776 (+0.011487821458091574) | > avg_loss_mel: 17.606704389545282 (-0.15144898857868583) | > avg_loss_duration: 1.6035820816604185 (-0.005622710980160095) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.8901405065832 (-0.12379511980943292) | > avg_grad_norm_1: tensor(443.3312, device='cuda:0') (+tensor(65.2425, device='cuda:0'))  > EPOCH: 315/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:26:11)   --> TIME: 2025-02-25 11:26:21 -- STEP: 19/72 -- GLOBAL_STEP: 1022700 | > loss_disc: 2.837282657623291 (2.7716238247720817) | > loss_disc_real_0: 0.18485233187675476 (0.19220486989146784) | > loss_disc_real_1: 0.2200591266155243 (0.23353879545864306) | > loss_disc_real_2: 0.20492741465568542 (0.2365555951469823) | > loss_disc_real_3: 0.19700069725513458 (0.2412188194299999) | > loss_disc_real_4: 0.21854855120182037 (0.24392804111305036) | > loss_disc_real_5: 0.2332029491662979 (0.24898290869436765) | > loss_0: 2.837282657623291 (2.7716238247720817) | > grad_norm_0: tensor(10.5198, device='cuda:0') (tensor(13.7218, device='cuda:0')) | > loss_gen: 1.6524503231048584 (1.8494116130628084) | > loss_kl: 2.266411304473877 (2.0959244841023494) | > loss_feat: 2.7607576847076416 (2.7298731803894043) | > loss_mel: 18.46727752685547 (17.955183229948346) | > loss_duration: 1.550018072128296 (1.5795595834129734) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.696914672851562 (26.209952203850996) | > grad_norm_1: tensor(508.0487, device='cuda:0') (tensor(557.9354, device='cuda:0')) | > current_lr_0: 0.00019227755082596072 | > current_lr_1: 0.00019227755082596072 | > step_time: 0.4266 (0.4354476050326699) | > loader_time: 0.0079 (0.006967042621813323)  --> TIME: 2025-02-25 11:26:45 -- STEP: 69/72 -- GLOBAL_STEP: 1022750 | > loss_disc: 2.7645487785339355 (2.7851667784262393) | > loss_disc_real_0: 0.15366999804973602 (0.2064027413725853) | > loss_disc_real_1: 0.21616791188716888 (0.2310313947390819) | > loss_disc_real_2: 0.24094842374324799 (0.24020977620629297) | > loss_disc_real_3: 0.2509194016456604 (0.2411775731522104) | > loss_disc_real_4: 0.253634512424469 (0.24400610962639685) | > loss_disc_real_5: 0.25944119691848755 (0.24637274465699127) | > loss_0: 2.7645487785339355 (2.7851667784262393) | > grad_norm_0: tensor(8.4900, device='cuda:0') (tensor(15.6507, device='cuda:0')) | > loss_gen: 1.8719959259033203 (1.8413482783497244) | > loss_kl: 2.3658576011657715 (2.29356560672539) | > loss_feat: 2.5529465675354004 (2.648102528807046) | > loss_mel: 17.46446418762207 (18.00870809693268) | > loss_duration: 1.7066450119018555 (1.6133325220882029) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.961910247802734 (26.405057077822477) | > grad_norm_1: tensor(269.9420, device='cuda:0') (tensor(396.7402, device='cuda:0')) | > current_lr_0: 0.00019227755082596072 | > current_lr_1: 0.00019227755082596072 | > step_time: 0.5525 (0.45655598502228223) | > loader_time: 0.0127 (0.008735159169072693) --> EVAL PERFORMANCE | > avg_loader_time: 0.008790553455621427 (+0.00015975388003067192) | > avg_step_time: 0.4588407194110709 (-0.0016798435802191092) | > avg_loss_disc: 2.7840065586734823 (+0.0017151832580561965) | > avg_loss_disc_real_0: 0.20728670322979 (-0.004120063089149084) | > avg_loss_disc_real_1: 0.23076689432204608 (+0.000639330333387339) | > avg_loss_disc_real_2: 0.23967185889331388 (+0.0010855359510636708) | > avg_loss_disc_real_3: 0.24063911929096973 (+0.0035940904012868036) | > avg_loss_disc_real_4: 0.24404392821688048 (-0.0029615419851222458) | > avg_loss_disc_real_5: 0.24602472509296847 (-0.004421161933684026) | > avg_loss_0: 2.7840065586734823 (+0.0017151832580561965) | > avg_grad_norm_0: tensor(15.5022, device='cuda:0') (+tensor(0.2397, device='cuda:0')) | > avg_loss_gen: 1.8448467170688467 (+0.004667480226973275) | > avg_loss_kl: 2.297926038083897 (+0.03914677089368812) | > avg_loss_feat: 2.6509540618305474 (+0.07005838441177126) | > avg_loss_mel: 18.010208317931276 (+0.40350392838599447) | > avg_loss_duration: 1.6134294892700625 (+0.009847407609643977) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.417364657764704 (+0.5272241511815032) | > avg_grad_norm_1: tensor(395.9730, device='cuda:0') (tensor(-47.3582, device='cuda:0'))  > EPOCH: 316/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:26:48)   --> TIME: 2025-02-25 11:27:12 -- STEP: 47/72 -- GLOBAL_STEP: 1022800 | > loss_disc: 2.904625415802002 (2.7793366148116743) | > loss_disc_real_0: 0.28804856538772583 (0.2049920156281045) | > loss_disc_real_1: 0.25367847084999084 (0.23226521243440343) | > loss_disc_real_2: 0.25717881321907043 (0.23797661256282887) | > loss_disc_real_3: 0.2401430606842041 (0.23673837869725328) | > loss_disc_real_4: 0.25674623250961304 (0.24347661847763874) | > loss_disc_real_5: 0.2572498619556427 (0.2477680076944067) | > loss_0: 2.904625415802002 (2.7793366148116743) | > grad_norm_0: tensor(17.9464, device='cuda:0') (tensor(11.9471, device='cuda:0')) | > loss_gen: 1.8157353401184082 (1.8495850537685639) | > loss_kl: 2.2145416736602783 (2.2970665439646294) | > loss_feat: 2.5422821044921875 (2.6858449641694415) | > loss_mel: 17.8907413482666 (18.06758239421438) | > loss_duration: 1.6173975467681885 (1.596843219817953) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.08069610595703 (26.49692202628927) | > grad_norm_1: tensor(220.8182, device='cuda:0') (tensor(365.3202, device='cuda:0')) | > current_lr_0: 0.00019225351613210746 | > current_lr_1: 0.00019225351613210746 | > step_time: 0.4537 (0.4449069043423267) | > loader_time: 0.0089 (0.007682211855624585) --> EVAL PERFORMANCE | > avg_loader_time: 0.008642911911010739 (-0.00014764154461068858) | > avg_step_time: 0.46102953292954135 (+0.002188813518470445) | > avg_loss_disc: 2.779606177773274 (-0.004400380900208223) | > avg_loss_disc_real_0: 0.20780218675942488 (+0.0005154835296348748) | > avg_loss_disc_real_1: 0.23561171452764054 (+0.004844820205594458) | > avg_loss_disc_real_2: 0.24115746793612627 (+0.0014856090428123914) | > avg_loss_disc_real_3: 0.23790251558095638 (-0.0027366037100133578) | > avg_loss_disc_real_4: 0.24351691381192545 (-0.000527014404955034) | > avg_loss_disc_real_5: 0.2469558214217844 (+0.0009310963288159391) | > avg_loss_0: 2.779606177773274 (-0.004400380900208223) | > avg_grad_norm_0: tensor(12.4110, device='cuda:0') (tensor(-3.0913, device='cuda:0')) | > avg_loss_gen: 1.8573254178947127 (+0.012478700825865996) | > avg_loss_kl: 2.305942778855982 (+0.00801674077208503) | > avg_loss_feat: 2.6520592427589524 (+0.0011051809284050584) | > avg_loss_mel: 18.013431401319906 (+0.003223083388629533) | > avg_loss_duration: 1.6125473757864723 (-0.0008821134835901301) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.44130607390068 (+0.023941416135976112) | > avg_grad_norm_1: tensor(376.4263, device='cuda:0') (tensor(-19.5467, device='cuda:0'))  > EPOCH: 317/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:27:26)   --> TIME: 2025-02-25 11:27:39 -- STEP: 25/72 -- GLOBAL_STEP: 1022850 | > loss_disc: 2.7661731243133545 (2.757040891647339) | > loss_disc_real_0: 0.22170142829418182 (0.20249655932188035) | > loss_disc_real_1: 0.2540704011917114 (0.22084976196289063) | > loss_disc_real_2: 0.2428264170885086 (0.23698048293590546) | > loss_disc_real_3: 0.3022371828556061 (0.24423931181430816) | > loss_disc_real_4: 0.24272803962230682 (0.24613149464130402) | > loss_disc_real_5: 0.24235360324382782 (0.24700202405452729) | > loss_0: 2.7661731243133545 (2.757040891647339) | > grad_norm_0: tensor(5.8028, device='cuda:0') (tensor(11.1300, device='cuda:0')) | > loss_gen: 2.1949071884155273 (1.888682017326355) | > loss_kl: 2.2943015098571777 (2.1846083164215084) | > loss_feat: 3.3663947582244873 (2.8281348896026612) | > loss_mel: 18.17684555053711 (17.70336624145508) | > loss_duration: 1.5730565786361694 (1.5674787425994874) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.605504989624023 (26.172270126342774) | > grad_norm_1: tensor(75.7149, device='cuda:0') (tensor(387.1569, device='cuda:0')) | > current_lr_0: 0.00019222948444259095 | > current_lr_1: 0.00019222948444259095 | > step_time: 0.4308 (0.4363151168823242) | > loader_time: 0.0073 (0.006951828002929688) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653523216784836 (+1.0611305774097662e-05) | > avg_step_time: 0.4548551532584177 (-0.006174379671123664) | > avg_loss_disc: 2.789113568588042 (+0.009507390814767813) | > avg_loss_disc_real_0: 0.2086542079776106 (+0.0008520212181857278) | > avg_loss_disc_real_1: 0.23052104971778226 (-0.005090664809858286) | > avg_loss_disc_real_2: 0.24066268369345598 (-0.0004947842426702986) | > avg_loss_disc_real_3: 0.24409837097349302 (+0.006195855392536642) | > avg_loss_disc_real_4: 0.24635058452545758 (+0.0028336707135321326) | > avg_loss_disc_real_5: 0.24676287237187505 (-0.00019294904990935224) | > avg_loss_0: 2.789113568588042 (+0.009507390814767813) | > avg_grad_norm_0: tensor(13.3002, device='cuda:0') (+tensor(0.8893, device='cuda:0')) | > avg_loss_gen: 1.853462902592941 (-0.0038625153017717295) | > avg_loss_kl: 2.221045687165059 (-0.08489709169092308) | > avg_loss_feat: 2.6735078851941605 (+0.021448642435208054) | > avg_loss_mel: 17.743966424968885 (-0.2694649763510206) | > avg_loss_duration: 1.6055138396545194 (-0.0070335361319529) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.09749667745241 (-0.3438093964482718) | > avg_grad_norm_1: tensor(339.6494, device='cuda:0') (tensor(-36.7769, device='cuda:0'))  > EPOCH: 318/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:28:02)   --> TIME: 2025-02-25 11:28:06 -- STEP: 3/72 -- GLOBAL_STEP: 1022900 | > loss_disc: 2.743772268295288 (2.7935713132222495) | > loss_disc_real_0: 0.24956576526165009 (0.23351066807905832) | > loss_disc_real_1: 0.18129076063632965 (0.23788436750570932) | > loss_disc_real_2: 0.20877639949321747 (0.22335556149482727) | > loss_disc_real_3: 0.20213861763477325 (0.20906653503576914) | > loss_disc_real_4: 0.20718999207019806 (0.22800355156262717) | > loss_disc_real_5: 0.21512967348098755 (0.22850152850151062) | > loss_0: 2.743772268295288 (2.7935713132222495) | > grad_norm_0: tensor(12.0309, device='cuda:0') (tensor(11.2019, device='cuda:0')) | > loss_gen: 1.6802173852920532 (1.7879093090693157) | > loss_kl: 2.3058419227600098 (2.1244812409083047) | > loss_feat: 3.0254344940185547 (2.768112818400065) | > loss_mel: 18.89609146118164 (18.21324094136556) | > loss_duration: 1.6370877027511597 (1.5930333137512207) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.5446720123291 (26.486777623494465) | > grad_norm_1: tensor(388.5904, device='cuda:0') (tensor(490.3870, device='cuda:0')) | > current_lr_0: 0.0001922054557570356 | > current_lr_1: 0.0001922054557570356 | > step_time: 0.4244 (0.42385490735371906) | > loader_time: 0.0053 (0.005951484044392903)  --> TIME: 2025-02-25 11:28:28 -- STEP: 53/72 -- GLOBAL_STEP: 1022950 | > loss_disc: 2.7487149238586426 (2.7791649845411195) | > loss_disc_real_0: 0.2137656807899475 (0.20488544862787678) | > loss_disc_real_1: 0.26173582673072815 (0.2349802736403807) | > loss_disc_real_2: 0.22307312488555908 (0.23722511276883898) | > loss_disc_real_3: 0.21637952327728271 (0.2380110447136861) | > loss_disc_real_4: 0.23656433820724487 (0.24352458091276996) | > loss_disc_real_5: 0.23641568422317505 (0.24633484882003856) | > loss_0: 2.7487149238586426 (2.7791649845411195) | > grad_norm_0: tensor(5.5128, device='cuda:0') (tensor(11.3359, device='cuda:0')) | > loss_gen: 1.8264001607894897 (1.8333867540899313) | > loss_kl: 2.2065773010253906 (2.218934227835457) | > loss_feat: 2.5773775577545166 (2.6055256555665216) | > loss_mel: 17.10980987548828 (17.86325254980123) | > loss_duration: 1.641462802886963 (1.604388918516771) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.36162757873535 (26.12548824526229) | > grad_norm_1: tensor(551.9975, device='cuda:0') (tensor(362.8147, device='cuda:0')) | > current_lr_0: 0.0001922054557570356 | > current_lr_1: 0.0001922054557570356 | > step_time: 0.4436 (0.44112019718817946) | > loader_time: 0.0093 (0.007878015626151607) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864213956913478 (-1.1383647650057074e-05) | > avg_step_time: 0.46095791333158254 (+0.006102760073164848) | > avg_loss_disc: 2.7909636665397968 (+0.0018500979517548366) | > avg_loss_disc_real_0: 0.20550148241536717 (-0.0031527255622434414) | > avg_loss_disc_real_1: 0.2365852527215447 (+0.006064203003762453) | > avg_loss_disc_real_2: 0.24041238154324007 (-0.0002503021502159031) | > avg_loss_disc_real_3: 0.23974024221091203 (-0.004358128762580987) | > avg_loss_disc_real_4: 0.24399413693119104 (-0.0023564475942665364) | > avg_loss_disc_real_5: 0.24616407697469417 (-0.0005987953971808846) | > avg_loss_0: 2.7909636665397968 (+0.0018500979517548366) | > avg_grad_norm_0: tensor(12.0616, device='cuda:0') (tensor(-1.2386, device='cuda:0')) | > avg_loss_gen: 1.8224233103470064 (-0.031039592245934555) | > avg_loss_kl: 2.239214368269477 (+0.018168681104418383) | > avg_loss_feat: 2.5584026054597238 (-0.11510527973443674) | > avg_loss_mel: 17.8136984328149 (+0.06973200784601374) | > avg_loss_duration: 1.6142821395900886 (+0.008768299935569157) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.048020913567342 (-0.04947576388506647) | > avg_grad_norm_1: tensor(347.9763, device='cuda:0') (+tensor(8.3269, device='cuda:0'))  > EPOCH: 319/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:28:40)   --> TIME: 2025-02-25 11:28:56 -- STEP: 31/72 -- GLOBAL_STEP: 1023000 | > loss_disc: 2.7277729511260986 (2.773282251050395) | > loss_disc_real_0: 0.17535381019115448 (0.20788884691653714) | > loss_disc_real_1: 0.21491268277168274 (0.23975956920654543) | > loss_disc_real_2: 0.23868660628795624 (0.24039875547732076) | > loss_disc_real_3: 0.22409188747406006 (0.23518251074898627) | > loss_disc_real_4: 0.23516109585762024 (0.24264190321968448) | > loss_disc_real_5: 0.22330576181411743 (0.24354863551355177) | > loss_0: 2.7277729511260986 (2.773282251050395) | > grad_norm_0: tensor(13.6265, device='cuda:0') (tensor(14.3201, device='cuda:0')) | > loss_gen: 1.7362711429595947 (1.8565101546625937) | > loss_kl: 2.0153751373291016 (2.119688287858041) | > loss_feat: 2.443049907684326 (2.6982962162263933) | > loss_mel: 17.134244918823242 (17.54381093671245) | > loss_duration: 1.6097970008850098 (1.5650988624941917) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.938737869262695 (25.783404319517075) | > grad_norm_1: tensor(427.8983, device='cuda:0') (tensor(442.1512, device='cuda:0')) | > current_lr_0: 0.00019218143007506598 | > current_lr_1: 0.00019218143007506598 | > step_time: 0.4366 (0.43554645969021705) | > loader_time: 0.0085 (0.006942556750389838) --> EVAL PERFORMANCE | > avg_loader_time: 0.008525526019888865 (-0.00011661354924591416) | > avg_step_time: 0.454416402628724 (-0.006541510702858522) | > avg_loss_disc: 2.7724531670691266 (-0.018510499470670183) | > avg_loss_disc_real_0: 0.20922732500123306 (+0.0037258425858658906) | > avg_loss_disc_real_1: 0.23214007893078764 (-0.004445173790757068) | > avg_loss_disc_real_2: 0.2390287463933649 (-0.001383635149875162) | > avg_loss_disc_real_3: 0.2355831586559054 (-0.00415708355500663) | > avg_loss_disc_real_4: 0.24468088611750535 (+0.0006867491863143116) | > avg_loss_disc_real_5: 0.2463838905096054 (+0.00021981353491123956) | > avg_loss_0: 2.7724531670691266 (-0.018510499470670183) | > avg_grad_norm_0: tensor(13.9260, device='cuda:0') (+tensor(1.8644, device='cuda:0')) | > avg_loss_gen: 1.8415668715893383 (+0.019143561242331852) | > avg_loss_kl: 2.258398260868771 (+0.019183892599293983) | > avg_loss_feat: 2.625008240552016 (+0.06660563509229211) | > avg_loss_mel: 17.656174619432914 (-0.15752381338198518) | > avg_loss_duration: 1.6029465702218069 (-0.011335569368281728) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.984094485430653 (-0.06392642813668914) | > avg_grad_norm_1: tensor(414.1891, device='cuda:0') (+tensor(66.2128, device='cuda:0'))  > EPOCH: 320/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:29:17)   --> TIME: 2025-02-25 11:29:23 -- STEP: 9/72 -- GLOBAL_STEP: 1023050 | > loss_disc: 2.6722497940063477 (2.738232480155097) | > loss_disc_real_0: 0.1556437909603119 (0.18406126317050722) | > loss_disc_real_1: 0.21558678150177002 (0.24200325045320722) | > loss_disc_real_2: 0.21982313692569733 (0.23581115735901725) | > loss_disc_real_3: 0.20403632521629333 (0.2344473765956031) | > loss_disc_real_4: 0.22703124582767487 (0.2437435289223989) | > loss_disc_real_5: 0.24100765585899353 (0.2526532593700621) | > loss_0: 2.6722497940063477 (2.738232480155097) | > grad_norm_0: tensor(6.6504, device='cuda:0') (tensor(8.3227, device='cuda:0')) | > loss_gen: 1.9591073989868164 (1.875636061032613) | > loss_kl: 2.168430805206299 (2.2339691850874157) | > loss_feat: 2.766615390777588 (2.75093494521247) | > loss_mel: 17.568681716918945 (17.788226021660698) | > loss_duration: 1.575273871421814 (1.5796232885784574) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.03810691833496 (26.228389316134983) | > grad_norm_1: tensor(893.0025, device='cuda:0') (tensor(464.9237, device='cuda:0')) | > current_lr_0: 0.0001921574073963066 | > current_lr_1: 0.0001921574073963066 | > step_time: 0.4328 (0.4328562418619792) | > loader_time: 0.0057 (0.006429910659790039)  --> TIME: 2025-02-25 11:29:46 -- STEP: 59/72 -- GLOBAL_STEP: 1023100 | > loss_disc: 2.863588809967041 (2.768580743822001) | > loss_disc_real_0: 0.1272251456975937 (0.19821581610683667) | > loss_disc_real_1: 0.25862234830856323 (0.23418968764402098) | > loss_disc_real_2: 0.2628992199897766 (0.24068921854940512) | > loss_disc_real_3: 0.2600052058696747 (0.23872799135870854) | > loss_disc_real_4: 0.2512909770011902 (0.244485192379709) | > loss_disc_real_5: 0.2474665641784668 (0.24711891124814248) | > loss_0: 2.863588809967041 (2.768580743822001) | > grad_norm_0: tensor(30.0289, device='cuda:0') (tensor(15.2253, device='cuda:0')) | > loss_gen: 1.8585851192474365 (1.8481824014146448) | > loss_kl: 2.1270275115966797 (2.2804002640611034) | > loss_feat: 2.7527413368225098 (2.657886448553053) | > loss_mel: 17.550888061523438 (17.84734735650532) | > loss_duration: 1.629617691040039 (1.6051066770391949) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.918859481811523 (26.238923315274512) | > grad_norm_1: tensor(115.1683, device='cuda:0') (tensor(500.7340, device='cuda:0')) | > current_lr_0: 0.0001921574073963066 | > current_lr_1: 0.0001921574073963066 | > step_time: 0.4688 (0.4445765220512778) | > loader_time: 0.0096 (0.008047718112751588) --> EVAL PERFORMANCE | > avg_loader_time: 0.008608294204926824 (+8.276818503795934e-05) | > avg_step_time: 0.4539372551609093 (-0.000479147467814689) | > avg_loss_disc: 2.777532490206436 (+0.005079323137309633) | > avg_loss_disc_real_0: 0.2029011080592451 (-0.0063262169419879555) | > avg_loss_disc_real_1: 0.2329693354771171 (+0.0008292565463294543) | > avg_loss_disc_real_2: 0.2399052148553687 (+0.0008764684620037877) | > avg_loss_disc_real_3: 0.23861193195195265 (+0.0030287732960472546) | > avg_loss_disc_real_4: 0.24422380580029018 (-0.0004570803172151694) | > avg_loss_disc_real_5: 0.2466268344244487 (+0.00024294391484327993) | > avg_loss_0: 2.777532490206436 (+0.005079323137309633) | > avg_grad_norm_0: tensor(15.5817, device='cuda:0') (+tensor(1.6558, device='cuda:0')) | > avg_loss_gen: 1.8367768022376048 (-0.004790069351733539) | > avg_loss_kl: 2.2822136207365658 (+0.023815359867794594) | > avg_loss_feat: 2.5958960324945584 (-0.02911220805745751) | > avg_loss_mel: 17.770798481686022 (+0.11462386225310794) | > avg_loss_duration: 1.6131647254379702 (+0.010218155216163316) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.098849847283162 (+0.11475536185250945) | > avg_grad_norm_1: tensor(467.1154, device='cuda:0') (+tensor(52.9263, device='cuda:0'))  > EPOCH: 321/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:29:54)   --> TIME: 2025-02-25 11:30:13 -- STEP: 37/72 -- GLOBAL_STEP: 1023150 | > loss_disc: 2.7975900173187256 (2.7583934036461084) | > loss_disc_real_0: 0.20056916773319244 (0.20642083259047686) | > loss_disc_real_1: 0.2799517512321472 (0.23415625095367432) | > loss_disc_real_2: 0.24369537830352783 (0.23825159588375608) | > loss_disc_real_3: 0.24526746571063995 (0.24337546406565486) | > loss_disc_real_4: 0.2381749153137207 (0.2479844194006275) | > loss_disc_real_5: 0.2592611610889435 (0.24702762872786135) | > loss_0: 2.7975900173187256 (2.7583934036461084) | > grad_norm_0: tensor(12.9834, device='cuda:0') (tensor(14.8241, device='cuda:0')) | > loss_gen: 1.9402523040771484 (1.8985151406880971) | > loss_kl: 2.1607778072357178 (2.2207243603629037) | > loss_feat: 2.432279348373413 (2.7627335174663648) | > loss_mel: 16.86301040649414 (18.070195481583884) | > loss_duration: 1.6044859886169434 (1.574258701221363) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.00080680847168 (26.526427140107025) | > grad_norm_1: tensor(109.3203, device='cuda:0') (tensor(482.7805, device='cuda:0')) | > current_lr_0: 0.00019213338772038204 | > current_lr_1: 0.00019213338772038204 | > step_time: 0.4527 (0.4520904695665514) | > loader_time: 0.0084 (0.007306028056789089) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864964135935609 (+4.134715442926512e-05) | > avg_step_time: 0.46287816007372357 (+0.00894090491281424) | > avg_loss_disc: 2.771811172995769 (-0.005721317210667376) | > avg_loss_disc_real_0: 0.21000830157541892 (+0.00710719351617381) | > avg_loss_disc_real_1: 0.2346831197050256 (+0.0017137842279084936) | > avg_loss_disc_real_2: 0.24005337395298648 (+0.00014815909761778379) | > avg_loss_disc_real_3: 0.24219297103478876 (+0.003581039082836107) | > avg_loss_disc_real_4: 0.24547623349747189 (+0.0012524276971817017) | > avg_loss_disc_real_5: 0.24593326806182592 (-0.0006935663626227717) | > avg_loss_0: 2.771811172995769 (-0.005721317210667376) | > avg_grad_norm_0: tensor(17.0071, device='cuda:0') (+tensor(1.4254, device='cuda:0')) | > avg_loss_gen: 1.8727169708466866 (+0.03594016860908189) | > avg_loss_kl: 2.2485790403795916 (-0.0336345803569742) | > avg_loss_feat: 2.6992308522613957 (+0.10333481976683734) | > avg_loss_mel: 17.903045358792156 (+0.1322468771061338) | > avg_loss_duration: 1.6047297481080176 (-0.008434977329952575) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.328301899869675 (+0.22945205258651313) | > avg_grad_norm_1: tensor(475.4398, device='cuda:0') (+tensor(8.3244, device='cuda:0'))  > EPOCH: 322/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:30:31)   --> TIME: 2025-02-25 11:30:40 -- STEP: 15/72 -- GLOBAL_STEP: 1023200 | > loss_disc: 2.82546067237854 (2.7360420068105062) | > loss_disc_real_0: 0.16216106712818146 (0.18047077755133312) | > loss_disc_real_1: 0.14938704669475555 (0.22079528669516246) | > loss_disc_real_2: 0.252958208322525 (0.2355884701013565) | > loss_disc_real_3: 0.24280284345149994 (0.23281334439913431) | > loss_disc_real_4: 0.2552134692668915 (0.2455912361542384) | > loss_disc_real_5: 0.24232658743858337 (0.24200453162193297) | > loss_0: 2.82546067237854 (2.7360420068105062) | > grad_norm_0: tensor(21.8714, device='cuda:0') (tensor(15.2587, device='cuda:0')) | > loss_gen: 1.6914637088775635 (1.8348719040552774) | > loss_kl: 2.04337215423584 (2.280581839879354) | > loss_feat: 2.564086675643921 (2.7578791618347167) | > loss_mel: 18.997581481933594 (17.917246627807618) | > loss_duration: 1.5167200565338135 (1.5631138801574707) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.81322479248047 (26.35369364420573) | > grad_norm_1: tensor(628.5726, device='cuda:0') (tensor(557.7348, device='cuda:0')) | > current_lr_0: 0.000192109371046917 | > current_lr_1: 0.000192109371046917 | > step_time: 0.4341 (0.43468874295552573) | > loader_time: 0.006 (0.006904220581054688)  --> TIME: 2025-02-25 11:31:03 -- STEP: 65/72 -- GLOBAL_STEP: 1023250 | > loss_disc: 2.7945098876953125 (2.7768371838789725) | > loss_disc_real_0: 0.20740875601768494 (0.20314339307638316) | > loss_disc_real_1: 0.24580799043178558 (0.22792142835947182) | > loss_disc_real_2: 0.22843702137470245 (0.240837260392996) | > loss_disc_real_3: 0.23958945274353027 (0.24075574622704432) | > loss_disc_real_4: 0.27483096718788147 (0.24582218115146343) | > loss_disc_real_5: 0.2546406686306 (0.24607065595113314) | > loss_0: 2.7945098876953125 (2.7768371838789725) | > grad_norm_0: tensor(4.7383, device='cuda:0') (tensor(15.1079, device='cuda:0')) | > loss_gen: 1.6860582828521729 (1.8419574058972872) | > loss_kl: 2.4080324172973633 (2.248874367200411) | > loss_feat: 2.354464530944824 (2.6535466340871956) | > loss_mel: 17.98332977294922 (17.46971340179443) | > loss_duration: 1.647486686706543 (1.6004624678538395) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.07937240600586 (25.814554126446065) | > grad_norm_1: tensor(158.3282, device='cuda:0') (tensor(430.5621, device='cuda:0')) | > current_lr_0: 0.000192109371046917 | > current_lr_1: 0.000192109371046917 | > step_time: 0.5171 (0.4504938785846417) | > loader_time: 0.0116 (0.008441294156588041) --> EVAL PERFORMANCE | > avg_loader_time: 0.008712036508909416 (+6.239514955332613e-05) | > avg_step_time: 0.4572539195208482 (-0.005624240552875359) | > avg_loss_disc: 2.772516875199869 (+0.0007057022041001559) | > avg_loss_disc_real_0: 0.20192606600237564 (-0.008082235573043273) | > avg_loss_disc_real_1: 0.22745636609238637 (-0.0072267536126392196) | > avg_loss_disc_real_2: 0.23976320406080973 (-0.00029016989217675593) | > avg_loss_disc_real_3: 0.24021098395468485 (-0.001981987080103914) | > avg_loss_disc_real_4: 0.244124002439875 (-0.001352231057596881) | > avg_loss_disc_real_5: 0.24509573990190534 (-0.0008375281599205808) | > avg_loss_0: 2.772516875199869 (+0.0007057022041001559) | > avg_grad_norm_0: tensor(14.9596, device='cuda:0') (tensor(-2.0476, device='cuda:0')) | > avg_loss_gen: 1.8428639475728426 (-0.029853023273844093) | > avg_loss_kl: 2.2614425034590164 (+0.012863463079424875) | > avg_loss_feat: 2.658350229263305 (-0.04088062299809048) | > avg_loss_mel: 17.480468602247637 (-0.4225767565445189) | > avg_loss_duration: 1.6057340477553892 (+0.001004299647371587) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.848859222841934 (-0.4794426770277411) | > avg_grad_norm_1: tensor(425.7599, device='cuda:0') (tensor(-49.6798, device='cuda:0'))  > EPOCH: 323/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:31:08)   --> TIME: 2025-02-25 11:31:30 -- STEP: 43/72 -- GLOBAL_STEP: 1023300 | > loss_disc: 2.7909698486328125 (2.777664977450704) | > loss_disc_real_0: 0.1942751407623291 (0.20647455474665) | > loss_disc_real_1: 0.2816780209541321 (0.23526995611745258) | > loss_disc_real_2: 0.2318073809146881 (0.2405106678258541) | > loss_disc_real_3: 0.26437056064605713 (0.23782615502213322) | > loss_disc_real_4: 0.2390582412481308 (0.24621922609417937) | > loss_disc_real_5: 0.2510027289390564 (0.24971826374530792) | > loss_0: 2.7909698486328125 (2.777664977450704) | > grad_norm_0: tensor(10.1097, device='cuda:0') (tensor(13.7277, device='cuda:0')) | > loss_gen: 1.8773353099822998 (1.8735001558481261) | > loss_kl: 2.065718650817871 (2.3043746366057287) | > loss_feat: 2.372922897338867 (2.7468799380368965) | > loss_mel: 16.12017822265625 (17.8742165676383) | > loss_duration: 1.6143810749053955 (1.6134441037510716) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.050535202026367 (26.41241543792015) | > grad_norm_1: tensor(496.6516, device='cuda:0') (tensor(374.2165, device='cuda:0')) | > current_lr_0: 0.00019208535737553613 | > current_lr_1: 0.00019208535737553613 | > step_time: 0.4516 (0.4433867931365967) | > loader_time: 0.0098 (0.007714765016422715) --> EVAL PERFORMANCE | > avg_loader_time: 0.00876120782234299 (+4.9171313433574865e-05) | > avg_step_time: 0.45452750568658534 (-0.002726413834262864) | > avg_loss_disc: 2.7915267171994067 (+0.019009841999537702) | > avg_loss_disc_real_0: 0.2083580191706268 (+0.006431953168251153) | > avg_loss_disc_real_1: 0.23407109816309432 (+0.0066147320707079516) | > avg_loss_disc_real_2: 0.242776869468286 (+0.0030136654074762736) | > avg_loss_disc_real_3: 0.2417503488315663 (+0.0015393648768814439) | > avg_loss_disc_real_4: 0.24412021670542972 (-3.7857344452885133e-06) | > avg_loss_disc_real_5: 0.24807513515714189 (+0.00297939525523655) | > avg_loss_0: 2.7915267171994067 (+0.019009841999537702) | > avg_grad_norm_0: tensor(13.5660, device='cuda:0') (tensor(-1.3936, device='cuda:0')) | > avg_loss_gen: 1.8505172427271452 (+0.007653295154302686) | > avg_loss_kl: 2.2918978795199325 (+0.030455376060916084) | > avg_loss_feat: 2.667388640658956 (+0.009038411395650758) | > avg_loss_mel: 17.763027110569915 (+0.28255850832227836) | > avg_loss_duration: 1.624788527757349 (+0.01905448000195986) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.19761950533155 (+0.3487602824896143) | > avg_grad_norm_1: tensor(350.7177, device='cuda:0') (tensor(-75.0422, device='cuda:0'))  > EPOCH: 324/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:31:45)   --> TIME: 2025-02-25 11:31:57 -- STEP: 21/72 -- GLOBAL_STEP: 1023350 | > loss_disc: 2.876026153564453 (2.794273092633202) | > loss_disc_real_0: 0.15610207617282867 (0.22088780786309922) | > loss_disc_real_1: 0.2303772121667862 (0.22727827373005094) | > loss_disc_real_2: 0.2305554747581482 (0.2417658581620171) | > loss_disc_real_3: 0.2578716576099396 (0.24442073064191) | > loss_disc_real_4: 0.25912177562713623 (0.2488997614099866) | > loss_disc_real_5: 0.26021498441696167 (0.24228626844428836) | > loss_0: 2.876026153564453 (2.794273092633202) | > grad_norm_0: tensor(14.9174, device='cuda:0') (tensor(17.0274, device='cuda:0')) | > loss_gen: 1.7371753454208374 (1.8408263751438685) | > loss_kl: 2.291836977005005 (2.1931331214450656) | > loss_feat: 2.499211549758911 (2.5513377870832175) | > loss_mel: 17.948375701904297 (17.496686481294176) | > loss_duration: 1.583204984664917 (1.561380454472133) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.059803009033203 (25.64336395263672) | > grad_norm_1: tensor(598.9221, device='cuda:0') (tensor(481.5382, device='cuda:0')) | > current_lr_0: 0.00019206134670586418 | > current_lr_1: 0.00019206134670586418 | > step_time: 0.4464 (0.45830707322983516) | > loader_time: 0.0065 (0.006811925343104771)  --> TIME: 2025-02-25 11:32:21 -- STEP: 71/72 -- GLOBAL_STEP: 1023400 | > loss_disc: 2.7020716667175293 (2.796086250896186) | > loss_disc_real_0: 0.24681270122528076 (0.2186120926914081) | > loss_disc_real_1: 0.26288336515426636 (0.2338313533806465) | > loss_disc_real_2: 0.24495454132556915 (0.2427507907571927) | > loss_disc_real_3: 0.23996050655841827 (0.244325394571667) | > loss_disc_real_4: 0.24519142508506775 (0.24662703275680542) | > loss_disc_real_5: 0.2578970789909363 (0.24660041970266422) | > loss_0: 2.7020716667175293 (2.796086250896186) | > grad_norm_0: tensor(14.0550, device='cuda:0') (tensor(15.8147, device='cuda:0')) | > loss_gen: 1.9598488807678223 (1.8557338042997977) | > loss_kl: 2.3495497703552246 (2.2489669255807376) | > loss_feat: 2.3159327507019043 (2.589236232596384) | > loss_mel: 17.997695922851562 (17.5954900258024) | > loss_duration: 1.6204750537872314 (1.6015585761674693) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.243501663208008 (25.890985569483796) | > grad_norm_1: tensor(254.4808, device='cuda:0') (tensor(405.9389, device='cuda:0')) | > current_lr_0: 0.00019206134670586418 | > current_lr_1: 0.00019206134670586418 | > step_time: 0.5091 (0.4632773063552212) | > loader_time: 0.0079 (0.008663321884585098) --> EVAL PERFORMANCE | > avg_loader_time: 0.008663321884585098 (-9.788593775789285e-05) | > avg_step_time: 0.4632773063552212 (+0.008749800668635843) | > avg_loss_disc: 2.796086250896186 (+0.004559533696779194) | > avg_loss_disc_real_0: 0.2186120926914081 (+0.01025407352078131) | > avg_loss_disc_real_1: 0.2338313533806465 (-0.00023974478244781494) | > avg_loss_disc_real_2: 0.2427507907571927 (-2.607871109330806e-05) | > avg_loss_disc_real_3: 0.244325394571667 (+0.002575045740100701) | > avg_loss_disc_real_4: 0.24662703275680542 (+0.0025068160513757043) | > avg_loss_disc_real_5: 0.24660041970266422 (-0.0014747154544776653) | > avg_loss_0: 2.796086250896186 (+0.004559533696779194) | > avg_grad_norm_0: tensor(15.8147, device='cuda:0') (+tensor(2.2487, device='cuda:0')) | > avg_loss_gen: 1.8557338042997977 (+0.005216561572652489) | > avg_loss_kl: 2.2489669255807376 (-0.04293095393919488) | > avg_loss_feat: 2.589236232596384 (-0.07815240806257195) | > avg_loss_mel: 17.5954900258024 (-0.16753708476751683) | > avg_loss_duration: 1.6015585761674693 (-0.023229951589879727) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.890985569483796 (-0.3066339358477528) | > avg_grad_norm_1: tensor(405.9389, device='cuda:0') (+tensor(55.2212, device='cuda:0'))  > EPOCH: 325/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:32:22)   --> TIME: 2025-02-25 11:32:47 -- STEP: 49/72 -- GLOBAL_STEP: 1023450 | > loss_disc: 2.8411266803741455 (2.7800024090980995) | > loss_disc_real_0: 0.2208004742860794 (0.20606798876305019) | > loss_disc_real_1: 0.28643766045570374 (0.2333206421866709) | > loss_disc_real_2: 0.25240588188171387 (0.23923231722140798) | > loss_disc_real_3: 0.258991003036499 (0.23887958848962978) | > loss_disc_real_4: 0.23030269145965576 (0.24261769439492906) | > loss_disc_real_5: 0.2634386420249939 (0.24699390907676852) | > loss_0: 2.8411266803741455 (2.7800024090980995) | > grad_norm_0: tensor(7.5317, device='cuda:0') (tensor(15.0175, device='cuda:0')) | > loss_gen: 1.681659460067749 (1.8400287895786518) | > loss_kl: 2.2524216175079346 (2.280781960000797) | > loss_feat: 2.1997554302215576 (2.6286551903705213) | > loss_mel: 16.18074607849121 (17.9914645370172) | > loss_duration: 1.6688573360443115 (1.6018560394948842) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.983440399169922 (26.3427866332385) | > grad_norm_1: tensor(60.0379, device='cuda:0') (tensor(358.2918, device='cuda:0')) | > current_lr_0: 0.00019203733903752594 | > current_lr_1: 0.00019203733903752594 | > step_time: 0.4389 (0.4419492799408582) | > loader_time: 0.0097 (0.007609683640149175) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556023449964926 (-0.00010729843462017191) | > avg_step_time: 0.45420758489152074 (-0.009069721463700442) | > avg_loss_disc: 2.7908155817381095 (-0.005270669158076391) | > avg_loss_disc_real_0: 0.20501575597994767 (-0.013596336711460433) | > avg_loss_disc_real_1: 0.23371737796655842 (-0.00011397541408808554) | > avg_loss_disc_real_2: 0.2400585910384084 (-0.0026921997187842883) | > avg_loss_disc_real_3: 0.2400571773169746 (-0.004268217254692397) | > avg_loss_disc_real_4: 0.24398926802923981 (-0.0026377647275656058) | > avg_loss_disc_real_5: 0.24800419408670613 (+0.001403774384041906) | > avg_loss_0: 2.7908155817381095 (-0.005270669158076391) | > avg_grad_norm_0: tensor(13.7129, device='cuda:0') (tensor(-2.1018, device='cuda:0')) | > avg_loss_gen: 1.8309013726006091 (-0.024832431699188584) | > avg_loss_kl: 2.298835358149568 (+0.049868432568830556) | > avg_loss_feat: 2.593496953937369 (+0.004260721340985096) | > avg_loss_mel: 17.915102166189282 (+0.3196121403868837) | > avg_loss_duration: 1.6156775632374722 (+0.01411898707000292) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.254013410756286 (+0.36302784127249055) | > avg_grad_norm_1: tensor(376.6978, device='cuda:0') (tensor(-29.2412, device='cuda:0'))  > EPOCH: 326/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:32:59)   --> TIME: 2025-02-25 11:33:13 -- STEP: 27/72 -- GLOBAL_STEP: 1023500 | > loss_disc: 2.8434526920318604 (2.804752155586525) | > loss_disc_real_0: 0.28791558742523193 (0.20960362835062873) | > loss_disc_real_1: 0.21637120842933655 (0.23467264627968823) | > loss_disc_real_2: 0.2957988679409027 (0.24373922800576245) | > loss_disc_real_3: 0.2885797619819641 (0.2455284865917983) | > loss_disc_real_4: 0.28699854016304016 (0.25184283322758144) | > loss_disc_real_5: 0.30753177404403687 (0.25069552273662005) | > loss_0: 2.8434526920318604 (2.804752155586525) | > grad_norm_0: tensor(24.9769, device='cuda:0') (tensor(13.2907, device='cuda:0')) | > loss_gen: 1.931338906288147 (1.864127141458017) | > loss_kl: 2.1946849822998047 (2.190485958699827) | > loss_feat: 2.3875784873962402 (2.694010231229994) | > loss_mel: 16.85139274597168 (17.71609864411531) | > loss_duration: 1.5616791248321533 (1.5689318974812825) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.92667579650879 (26.033653683132595) | > grad_norm_1: tensor(412.8069, device='cuda:0') (tensor(329.1646, device='cuda:0')) | > current_lr_0: 0.00019201333437014623 | > current_lr_1: 0.00019201333437014623 | > step_time: 0.4524 (0.43616908567923085) | > loader_time: 0.0083 (0.006961495788009079) --> EVAL PERFORMANCE | > avg_loader_time: 0.008699333164053904 (+0.00014330971408897837) | > avg_step_time: 0.4546000520947953 (+0.0003924672032745713) | > avg_loss_disc: 2.793490769157947 (+0.0026751874198374104) | > avg_loss_disc_real_0: 0.21067857196633244 (+0.005662815986384767) | > avg_loss_disc_real_1: 0.23468270226263663 (+0.0009653242960782071) | > avg_loss_disc_real_2: 0.23773474172807077 (-0.0023238493103376334) | > avg_loss_disc_real_3: 0.23870613100663038 (-0.0013510463103442172) | > avg_loss_disc_real_4: 0.24524878846927428 (+0.0012595204400344673) | > avg_loss_disc_real_5: 0.24958234731580176 (+0.0015781532290956302) | > avg_loss_0: 2.793490769157947 (+0.0026751874198374104) | > avg_grad_norm_0: tensor(15.9254, device='cuda:0') (+tensor(2.2125, device='cuda:0')) | > avg_loss_gen: 1.8248929557665972 (-0.006008416834011987) | > avg_loss_kl: 2.2791740743207254 (-0.0196612838288428) | > avg_loss_feat: 2.565719419801739 (-0.027777534135630333) | > avg_loss_mel: 17.799458906684123 (-0.11564325950515908) | > avg_loss_duration: 1.6053050252753245 (-0.010372537962147765) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.074550306293325 (-0.1794631044629611) | > avg_grad_norm_1: tensor(357.0924, device='cuda:0') (tensor(-19.6054, device='cuda:0'))  > EPOCH: 327/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:33:36)   --> TIME: 2025-02-25 11:33:40 -- STEP: 5/72 -- GLOBAL_STEP: 1023550 | > loss_disc: 2.6890106201171875 (2.7059763431549073) | > loss_disc_real_0: 0.16720739006996155 (0.20890824794769286) | > loss_disc_real_1: 0.2179228961467743 (0.23833104074001313) | > loss_disc_real_2: 0.24166637659072876 (0.23038403689861298) | > loss_disc_real_3: 0.2595908045768738 (0.23618977069854735) | > loss_disc_real_4: 0.23566924035549164 (0.2399183690547943) | > loss_disc_real_5: 0.23080192506313324 (0.24209744930267335) | > loss_0: 2.6890106201171875 (2.7059763431549073) | > grad_norm_0: tensor(2.9822, device='cuda:0') (tensor(10.3571, device='cuda:0')) | > loss_gen: 1.8947391510009766 (1.8961907386779786) | > loss_kl: 2.015329599380493 (2.203285026550293) | > loss_feat: 2.702343702316284 (2.708975648880005) | > loss_mel: 17.394533157348633 (17.97577781677246) | > loss_duration: 1.5431864261627197 (1.5856178998947144) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.55013084411621 (26.369846725463866) | > grad_norm_1: tensor(73.4226, device='cuda:0') (tensor(280.7840, device='cuda:0')) | > current_lr_0: 0.00019198933270334994 | > current_lr_1: 0.00019198933270334994 | > step_time: 0.4477 (0.43260812759399414) | > loader_time: 0.007 (0.006673955917358398)  --> TIME: 2025-02-25 11:34:04 -- STEP: 55/72 -- GLOBAL_STEP: 1023600 | > loss_disc: 2.710791826248169 (2.781059637936679) | > loss_disc_real_0: 0.11665713042020798 (0.20301967019384562) | > loss_disc_real_1: 0.21780090034008026 (0.23436268378387798) | > loss_disc_real_2: 0.2486129105091095 (0.2396153981035406) | > loss_disc_real_3: 0.24529975652694702 (0.2401665137572722) | > loss_disc_real_4: 0.23995602130889893 (0.24479934058406136) | > loss_disc_real_5: 0.2163545787334442 (0.2437588464130055) | > loss_0: 2.710791826248169 (2.781059637936679) | > grad_norm_0: tensor(28.6374, device='cuda:0') (tensor(12.9401, device='cuda:0')) | > loss_gen: 1.7915329933166504 (1.8366778785532172) | > loss_kl: 2.315180540084839 (2.2397242589430375) | > loss_feat: 2.8771517276763916 (2.6482123071497137) | > loss_mel: 18.398094177246094 (17.9674653486772) | > loss_duration: 1.6322925090789795 (1.59645657972856) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.014251708984375 (26.28853634920987) | > grad_norm_1: tensor(807.1691, device='cuda:0') (tensor(395.7739, device='cuda:0')) | > current_lr_0: 0.00019198933270334994 | > current_lr_1: 0.00019198933270334994 | > step_time: 0.4742 (0.4537100011652166) | > loader_time: 0.0093 (0.007865060459483755) --> EVAL PERFORMANCE | > avg_loader_time: 0.008578407932335223 (-0.0001209252317186816) | > avg_step_time: 0.4639159625684711 (+0.009315910473675804) | > avg_loss_disc: 2.770369825228839 (-0.023120943929107796) | > avg_loss_disc_real_0: 0.20143837513218465 (-0.009240196834147785) | > avg_loss_disc_real_1: 0.2320997412775604 (-0.002582960985076238) | > avg_loss_disc_real_2: 0.24039716158114688 (+0.002662419853076109) | > avg_loss_disc_real_3: 0.24009212830536802 (+0.0013859972987376457) | > avg_loss_disc_real_4: 0.24370402223627333 (-0.0015447662330009548) | > avg_loss_disc_real_5: 0.24302905273269598 (-0.006553294583105773) | > avg_loss_0: 2.770369825228839 (-0.023120943929107796) | > avg_grad_norm_0: tensor(13.5133, device='cuda:0') (tensor(-2.4121, device='cuda:0')) | > avg_loss_gen: 1.8418857346118336 (+0.016992778845236423) | > avg_loss_kl: 2.2512782627428076 (-0.027895811577917762) | > avg_loss_feat: 2.642999951268586 (+0.07728053146684699) | > avg_loss_mel: 17.907358384468186 (+0.10789947778406273) | > avg_loss_duration: 1.6077224872481655 (+0.002417461972840984) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.2512447464634 (+0.17669444017007407) | > avg_grad_norm_1: tensor(431.5023, device='cuda:0') (+tensor(74.4099, device='cuda:0'))  > EPOCH: 328/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:34:14)   --> TIME: 2025-02-25 11:34:31 -- STEP: 33/72 -- GLOBAL_STEP: 1023650 | > loss_disc: 2.8405346870422363 (2.7926455772284307) | > loss_disc_real_0: 0.22031500935554504 (0.2191732735796408) | > loss_disc_real_1: 0.1933470368385315 (0.22921613897338058) | > loss_disc_real_2: 0.2318425327539444 (0.2351326233509815) | > loss_disc_real_3: 0.22433628141880035 (0.2413346108162042) | > loss_disc_real_4: 0.21489329636096954 (0.24415268049095618) | > loss_disc_real_5: 0.2167346179485321 (0.24654665273247342) | > loss_0: 2.8405346870422363 (2.7926455772284307) | > grad_norm_0: tensor(6.3949, device='cuda:0') (tensor(16.0522, device='cuda:0')) | > loss_gen: 1.7595329284667969 (1.8488507270812988) | > loss_kl: 2.3971714973449707 (2.1925773981845733) | > loss_feat: 2.6597554683685303 (2.653493678931034) | > loss_mel: 17.611848831176758 (17.58945222334429) | > loss_duration: 1.6695575714111328 (1.5807745962431936) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.09786605834961 (25.865148833303742) | > grad_norm_1: tensor(219.0686, device='cuda:0') (tensor(422.7577, device='cuda:0')) | > current_lr_0: 0.000191965334036762 | > current_lr_1: 0.000191965334036762 | > step_time: 0.4326 (0.43951684778386896) | > loader_time: 0.0079 (0.007092743208914092) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651572214045994 (+7.316428171077159e-05) | > avg_step_time: 0.45779593225935816 (-0.006120030309112956) | > avg_loss_disc: 2.7830258859714996 (+0.012656060742660458) | > avg_loss_disc_real_0: 0.21585192373940643 (+0.014413548607221771) | > avg_loss_disc_real_1: 0.23021917087091526 (-0.0018805704066451323) | > avg_loss_disc_real_2: 0.2368684169692053 (-0.003528744611941581) | > avg_loss_disc_real_3: 0.2398079323097014 (-0.00028419599566661957) | > avg_loss_disc_real_4: 0.24521773507897282 (+0.001513712842699494) | > avg_loss_disc_real_5: 0.24665792769109698 (+0.0036288749584010016) | > avg_loss_0: 2.7830258859714996 (+0.012656060742660458) | > avg_grad_norm_0: tensor(16.6382, device='cuda:0') (+tensor(3.1249, device='cuda:0')) | > avg_loss_gen: 1.8484837438019228 (+0.006598009190089238) | > avg_loss_kl: 2.2358977123045585 (-0.015380550438249152) | > avg_loss_feat: 2.604999229941569 (-0.038000721327016596) | > avg_loss_mel: 17.49810174485328 (-0.409256639614906) | > avg_loss_duration: 1.6143379379326188 (+0.00661545068445335) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.801820459500167 (-0.44942428696323233) | > avg_grad_norm_1: tensor(422.6317, device='cuda:0') (tensor(-8.8706, device='cuda:0'))  > EPOCH: 329/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:34:51)   --> TIME: 2025-02-25 11:34:58 -- STEP: 11/72 -- GLOBAL_STEP: 1023700 | > loss_disc: 2.7794644832611084 (2.7884197018363257) | > loss_disc_real_0: 0.14223670959472656 (0.1885128149932081) | > loss_disc_real_1: 0.2351212352514267 (0.23726508427749982) | > loss_disc_real_2: 0.20108240842819214 (0.23438942432403564) | > loss_disc_real_3: 0.20813584327697754 (0.23729777065190402) | > loss_disc_real_4: 0.22525286674499512 (0.24398281357505106) | > loss_disc_real_5: 0.2206393927335739 (0.2488533922217109) | > loss_0: 2.7794644832611084 (2.7884197018363257) | > grad_norm_0: tensor(16.2141, device='cuda:0') (tensor(8.3863, device='cuda:0')) | > loss_gen: 1.80528724193573 (1.8291530067270452) | > loss_kl: 2.364265203475952 (2.1156954115087334) | > loss_feat: 2.5387637615203857 (2.7726316235282202) | > loss_mel: 18.811168670654297 (17.947845458984375) | > loss_duration: 1.5741039514541626 (1.5827101360667835) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.09358787536621 (26.248035084117543) | > grad_norm_1: tensor(335.3255, device='cuda:0') (tensor(319.6273, device='cuda:0')) | > current_lr_0: 0.0001919413383700074 | > current_lr_1: 0.0001919413383700074 | > step_time: 0.4276 (0.4304455410350453) | > loader_time: 0.0057 (0.0062322833321311255)  --> TIME: 2025-02-25 11:35:21 -- STEP: 61/72 -- GLOBAL_STEP: 1023750 | > loss_disc: 2.744180679321289 (2.779654964071805) | > loss_disc_real_0: 0.1808249056339264 (0.20751270077756193) | > loss_disc_real_1: 0.20026426017284393 (0.23988507724687702) | > loss_disc_real_2: 0.2647702991962433 (0.2407298901530563) | > loss_disc_real_3: 0.25676992535591125 (0.23677602312603935) | > loss_disc_real_4: 0.27003350853919983 (0.2415696209082838) | > loss_disc_real_5: 0.26993370056152344 (0.24363219689150326) | > loss_0: 2.744180679321289 (2.779654964071805) | > grad_norm_0: tensor(4.7992, device='cuda:0') (tensor(13.6085, device='cuda:0')) | > loss_gen: 1.976757287979126 (1.8634559815047218) | > loss_kl: 2.323197364807129 (2.2371430377491177) | > loss_feat: 2.129798650741577 (2.7195980978793783) | > loss_mel: 16.65061378479004 (17.623868739018683) | > loss_duration: 1.6628414392471313 (1.5956514702468623) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.743209838867188 (26.03971718960121) | > grad_norm_1: tensor(528.3564, device='cuda:0') (tensor(462.9377, device='cuda:0')) | > current_lr_0: 0.0001919413383700074 | > current_lr_1: 0.0001919413383700074 | > step_time: 0.4542 (0.4456142441171115) | > loader_time: 0.0106 (0.008095381689853354) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584035954005278 (-6.753626004071638e-05) | > avg_step_time: 0.45534362255687444 (-0.0024523097024837237) | > avg_loss_disc: 2.7845190041501753 (+0.0014931181786757008) | > avg_loss_disc_real_0: 0.21158081492487812 (-0.004271108814528302) | > avg_loss_disc_real_1: 0.2408117317187954 (+0.010592560847880156) | > avg_loss_disc_real_2: 0.24027009937964694 (+0.003401682410441642) | > avg_loss_disc_real_3: 0.23738562724959683 (-0.0024223050601045737) | > avg_loss_disc_real_4: 0.24104765792128066 (-0.004170077157692159) | > avg_loss_disc_real_5: 0.24285740331864694 (-0.0038005243724500426) | > avg_loss_0: 2.7845190041501753 (+0.0014931181786757008) | > avg_grad_norm_0: tensor(13.9896, device='cuda:0') (tensor(-2.6486, device='cuda:0')) | > avg_loss_gen: 1.8577309527867276 (+0.0092472089848048) | > avg_loss_kl: 2.2759737615853957 (+0.04007604928083719) | > avg_loss_feat: 2.689543322778084 (+0.08454409283651465) | > avg_loss_mel: 17.696239377411324 (+0.19813763255804417) | > avg_loss_duration: 1.6060786633424355 (-0.008259274590183274) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.12556589153451 (+0.32374543203434314) | > avg_grad_norm_1: tensor(442.4414, device='cuda:0') (+tensor(19.8098, device='cuda:0'))  > EPOCH: 330/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:35:28)   --> TIME: 2025-02-25 11:35:48 -- STEP: 39/72 -- GLOBAL_STEP: 1023800 | > loss_disc: 2.686159372329712 (2.771645674338708) | > loss_disc_real_0: 0.15549048781394958 (0.20674781004587808) | > loss_disc_real_1: 0.23636934161186218 (0.23028216033409804) | > loss_disc_real_2: 0.21084539592266083 (0.2395837100652548) | > loss_disc_real_3: 0.22178378701210022 (0.23915376686132872) | > loss_disc_real_4: 0.23045869171619415 (0.24552639783957067) | > loss_disc_real_5: 0.23052121698856354 (0.24733946606134766) | > loss_0: 2.686159372329712 (2.771645674338708) | > grad_norm_0: tensor(20.9562, device='cuda:0') (tensor(12.6029, device='cuda:0')) | > loss_gen: 1.7918457984924316 (1.8479496668546627) | > loss_kl: 2.227278709411621 (2.3379294627752056) | > loss_feat: 2.7907893657684326 (2.661883250260964) | > loss_mel: 17.574251174926758 (18.046221072857197) | > loss_duration: 1.6149905920028687 (1.587987951743297) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.999155044555664 (26.481971349471653) | > grad_norm_1: tensor(593.8808, device='cuda:0') (tensor(303.9530, device='cuda:0')) | > current_lr_0: 0.00019191734570271117 | > current_lr_1: 0.00019191734570271117 | > step_time: 0.4471 (0.4520675585820125) | > loader_time: 0.0082 (0.0074370396442902395) --> EVAL PERFORMANCE | > avg_loader_time: 0.008811171625701471 (+0.0002271356716961933) | > avg_step_time: 0.46513283420616475 (+0.009789211649290308) | > avg_loss_disc: 2.789869993505344 (+0.005350989355168867) | > avg_loss_disc_real_0: 0.21439045646660765 (+0.002809641541729524) | > avg_loss_disc_real_1: 0.23365327904761676 (-0.00715845267117865) | > avg_loss_disc_real_2: 0.2407941178117 (+0.0005240184320530472) | > avg_loss_disc_real_3: 0.23988666815657012 (+0.002501040906973284) | > avg_loss_disc_real_4: 0.2454895173579874 (+0.0044418594367067465) | > avg_loss_disc_real_5: 0.2486881168795304 (+0.005830713560883466) | > avg_loss_0: 2.789869993505344 (+0.005350989355168867) | > avg_grad_norm_0: tensor(14.5057, device='cuda:0') (+tensor(0.5161, device='cuda:0')) | > avg_loss_gen: 1.8516096497925234 (-0.006121302994204214) | > avg_loss_kl: 2.3318485474922284 (+0.0558747859068327) | > avg_loss_feat: 2.638735213749845 (-0.050808109028238846) | > avg_loss_mel: 18.001785708145356 (+0.3055463307340318) | > avg_loss_duration: 1.611208418725242 (+0.005129755382806467) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.43518751439914 (+0.30962162286462913) | > avg_grad_norm_1: tensor(299.3093, device='cuda:0') (tensor(-143.1321, device='cuda:0'))  > EPOCH: 331/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:36:05)   --> TIME: 2025-02-25 11:36:15 -- STEP: 17/72 -- GLOBAL_STEP: 1023850 | > loss_disc: 2.8185667991638184 (2.7787247265086457) | > loss_disc_real_0: 0.16910836100578308 (0.21274384067339056) | > loss_disc_real_1: 0.17000296711921692 (0.23231380651978886) | > loss_disc_real_2: 0.25313934683799744 (0.24913722977918737) | > loss_disc_real_3: 0.27212050557136536 (0.2414144663249745) | > loss_disc_real_4: 0.2844898998737335 (0.24515199485947103) | > loss_disc_real_5: 0.26075229048728943 (0.24450207282515132) | > loss_0: 2.8185667991638184 (2.7787247265086457) | > grad_norm_0: tensor(9.8444, device='cuda:0') (tensor(11.0986, device='cuda:0')) | > loss_gen: 1.864850401878357 (1.8614836861105526) | > loss_kl: 2.53837251663208 (2.1514562438516047) | > loss_feat: 2.387856960296631 (2.633612022680395) | > loss_mel: 18.39322280883789 (17.601136375876035) | > loss_duration: 1.6455116271972656 (1.5672703490537756) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.829814910888672 (25.814958347993738) | > grad_norm_1: tensor(362.9413, device='cuda:0') (tensor(313.4448, device='cuda:0')) | > current_lr_0: 0.00019189335603449832 | > current_lr_1: 0.00019189335603449832 | > step_time: 0.4375 (0.4369090725393856) | > loader_time: 0.008 (0.007063458947574391)  --> TIME: 2025-02-25 11:36:39 -- STEP: 67/72 -- GLOBAL_STEP: 1023900 | > loss_disc: 2.845153570175171 (2.8053189106841585) | > loss_disc_real_0: 0.34240132570266724 (0.21519189917329531) | > loss_disc_real_1: 0.3358325958251953 (0.24240937295244702) | > loss_disc_real_2: 0.2650180160999298 (0.24567984311438318) | > loss_disc_real_3: 0.2419353723526001 (0.2416153481202339) | > loss_disc_real_4: 0.2259208709001541 (0.24486606290091328) | > loss_disc_real_5: 0.23583483695983887 (0.2460816700511904) | > loss_0: 2.845153570175171 (2.8053189106841585) | > grad_norm_0: tensor(27.7512, device='cuda:0') (tensor(15.7195, device='cuda:0')) | > loss_gen: 1.9073817729949951 (1.8543618561616584) | > loss_kl: 2.2102036476135254 (2.2112672044270076) | > loss_feat: 2.682612895965576 (2.6078208186733183) | > loss_mel: 17.861906051635742 (17.664324148377382) | > loss_duration: 1.683097243309021 (1.6054211570255792) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.34520149230957 (25.94319520067813) | > grad_norm_1: tensor(517.5181, device='cuda:0') (tensor(356.9597, device='cuda:0')) | > current_lr_0: 0.00019189335603449832 | > current_lr_1: 0.00019189335603449832 | > step_time: 0.5184 (0.4535365816372544) | > loader_time: 0.0123 (0.008631300570359868) --> EVAL PERFORMANCE | > avg_loader_time: 0.008801131181313957 (-1.0040444387513989e-05) | > avg_step_time: 0.45743747832069936 (-0.0076953558854653825) | > avg_loss_disc: 2.8054860779936885 (+0.015616084488344395) | > avg_loss_disc_real_0: 0.21244365228733547 (-0.0019468041792721769) | > avg_loss_disc_real_1: 0.24326540235902222 (+0.009612123311405457) | > avg_loss_disc_real_2: 0.24592668405720886 (+0.005132566245508868) | > avg_loss_disc_real_3: 0.24218074851472612 (+0.0022940803581560087) | > avg_loss_disc_real_4: 0.2452173048341778 (-0.0002722125238096007) | > avg_loss_disc_real_5: 0.2466602581487575 (-0.002027858730772908) | > avg_loss_0: 2.8054860779936885 (+0.015616084488344395) | > avg_grad_norm_0: tensor(15.6153, device='cuda:0') (+tensor(1.1096, device='cuda:0')) | > avg_loss_gen: 1.8496609758323348 (-0.0019486739601886072) | > avg_loss_kl: 2.2156501790167584 (-0.11619836847546994) | > avg_loss_feat: 2.596631673020376 (-0.0421035407294692) | > avg_loss_mel: 17.65057722279724 (-0.351208485348117) | > avg_loss_duration: 1.6077012209825112 (-0.0035071977427307832) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.920221275007222 (-0.5149662393919172) | > avg_grad_norm_1: tensor(357.1768, device='cuda:0') (+tensor(57.8675, device='cuda:0'))  > EPOCH: 332/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:36:43)   --> TIME: 2025-02-25 11:37:05 -- STEP: 45/72 -- GLOBAL_STEP: 1023950 | > loss_disc: 2.811007499694824 (2.7808791107601585) | > loss_disc_real_0: 0.13525056838989258 (0.20539629343483184) | > loss_disc_real_1: 0.217128723859787 (0.2371901469098197) | > loss_disc_real_2: 0.22721293568611145 (0.23814462853802576) | > loss_disc_real_3: 0.248512864112854 (0.23691307140721216) | > loss_disc_real_4: 0.2479783594608307 (0.24368064403533934) | > loss_disc_real_5: 0.25904667377471924 (0.2479201141330931) | > loss_0: 2.811007499694824 (2.7808791107601585) | > grad_norm_0: tensor(19.0614, device='cuda:0') (tensor(12.0390, device='cuda:0')) | > loss_gen: 1.8022375106811523 (1.8496580600738526) | > loss_kl: 2.2429287433624268 (2.1704038328594644) | > loss_feat: 2.7964999675750732 (2.6825532807244197) | > loss_mel: 17.186256408691406 (17.43484348720975) | > loss_duration: 1.661615014076233 (1.584846059481303) | > amp_scaler: 128.0 (156.44444444444446) | > loss_1: 25.689538955688477 (25.722304831610785) | > grad_norm_1: tensor(650.2018, device='cuda:0') (tensor(464.6487, device='cuda:0')) | > current_lr_0: 0.000191869369364994 | > current_lr_1: 0.000191869369364994 | > step_time: 0.4634 (0.4418734232584635) | > loader_time: 0.0086 (0.007956123352050782) --> EVAL PERFORMANCE | > avg_loader_time: 0.008950408075896784 (+0.0001492768945828265) | > avg_step_time: 0.45571234192646726 (-0.0017251363942321074) | > avg_loss_disc: 2.7794254094782005 (-0.02606066851548805) | > avg_loss_disc_real_0: 0.20489056649762144 (-0.007553085789714031) | > avg_loss_disc_real_1: 0.23714692365955298 (-0.006118478699469243) | > avg_loss_disc_real_2: 0.23884712936173022 (-0.007079554695478635) | > avg_loss_disc_real_3: 0.23909109851843874 (-0.003089649996287386) | > avg_loss_disc_real_4: 0.24403918377110656 (-0.0011781210630712469) | > avg_loss_disc_real_5: 0.2478287889504097 (+0.00116853080165219) | > avg_loss_0: 2.7794254094782005 (-0.02606066851548805) | > avg_grad_norm_0: tensor(11.8465, device='cuda:0') (tensor(-3.7688, device='cuda:0')) | > avg_loss_gen: 1.8467223342035857 (-0.002938641628749128) | > avg_loss_kl: 2.2088799526993657 (-0.006770226317392769) | > avg_loss_feat: 2.6310762183766965 (+0.03444454535632069) | > avg_loss_mel: 17.517856100915182 (-0.13272112188205654) | > avg_loss_duration: 1.6073338112360995 (-0.00036740974641169544) | > avg_amp_scaler: 146.0281690140845 (+18.02816901408451) | > avg_loss_1: 25.811868694466604 (-0.10835258054061825) | > avg_grad_norm_1: tensor(426.7996, device='cuda:0') (+tensor(69.6228, device='cuda:0'))  > EPOCH: 333/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:37:20)   --> TIME: 2025-02-25 11:37:32 -- STEP: 23/72 -- GLOBAL_STEP: 1024000 | > loss_disc: 2.7307894229888916 (2.803217535433562) | > loss_disc_real_0: 0.20957612991333008 (0.20960323318191196) | > loss_disc_real_1: 0.21292515099048615 (0.22583295858424643) | > loss_disc_real_2: 0.24317413568496704 (0.23939772857272107) | > loss_disc_real_3: 0.22672942280769348 (0.2370498653339303) | > loss_disc_real_4: 0.22135235369205475 (0.24678645444952924) | > loss_disc_real_5: 0.22329580783843994 (0.24873033230719346) | > loss_0: 2.7307894229888916 (2.803217535433562) | > grad_norm_0: tensor(5.2701, device='cuda:0') (tensor(15.5727, device='cuda:0')) | > loss_gen: 1.7673017978668213 (1.806100124898164) | > loss_kl: 2.1687421798706055 (2.221986397452976) | > loss_feat: 2.7577011585235596 (2.5993529610011885) | > loss_mel: 17.78746795654297 (17.752333516659945) | > loss_duration: 1.6365996599197388 (1.5796313907789148) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.117813110351562 (25.959404489268426) | > grad_norm_1: tensor(363.4551, device='cuda:0') (tensor(315.8454, device='cuda:0')) | > current_lr_0: 0.00019184538569382337 | > current_lr_1: 0.00019184538569382337 | > step_time: 0.4379 (0.4346160992332127) | > loader_time: 0.0062 (0.006756202034328295) --> EVAL PERFORMANCE | > avg_loader_time: 0.008814603510037275 (-0.00013580456585950834) | > avg_step_time: 0.4647509883826887 (+0.009038646456221433) | > avg_loss_disc: 2.7815311928869972 (+0.0021057834087967464) | > avg_loss_disc_real_0: 0.20491516695056164 (+2.4600452940198858e-05) | > avg_loss_disc_real_1: 0.2315958257712109 (-0.005551097888342071) | > avg_loss_disc_real_2: 0.23811044449537572 (-0.0007366848663545034) | > avg_loss_disc_real_3: 0.2375102206854753 (-0.0015808778329634288) | > avg_loss_disc_real_4: 0.24552742024542581 (+0.0014882364743192544) | > avg_loss_disc_real_5: 0.24790129422301974 (+7.250527261004613e-05) | > avg_loss_0: 2.7815311928869972 (+0.0021057834087967464) | > avg_grad_norm_0: tensor(14.9558, device='cuda:0') (+tensor(3.1093, device='cuda:0')) | > avg_loss_gen: 1.8307195196689015 (-0.016002814534684218) | > avg_loss_kl: 2.274150027355678 (+0.06527007465631218) | > avg_loss_feat: 2.6477166934752128 (+0.016640475098516294) | > avg_loss_mel: 17.635574757213316 (+0.11771865629813405) | > avg_loss_duration: 1.609649399636497 (+0.0023155884003973615) | > avg_amp_scaler: 128.0 (-18.02816901408451) | > avg_loss_1: 25.997810444361726 (+0.1859417498951217) | > avg_grad_norm_1: tensor(427.9822, device='cuda:0') (+tensor(1.1827, device='cuda:0'))  > EPOCH: 334/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:37:57)   --> TIME: 2025-02-25 11:38:00 -- STEP: 1/72 -- GLOBAL_STEP: 1024050 | > loss_disc: 2.7223544120788574 (2.7223544120788574) | > loss_disc_real_0: 0.13082964718341827 (0.13082964718341827) | > loss_disc_real_1: 0.2353677898645401 (0.2353677898645401) | > loss_disc_real_2: 0.23780760169029236 (0.23780760169029236) | > loss_disc_real_3: 0.22560058534145355 (0.22560058534145355) | > loss_disc_real_4: 0.23470842838287354 (0.23470842838287354) | > loss_disc_real_5: 0.23513896763324738 (0.23513896763324738) | > loss_0: 2.7223544120788574 (2.7223544120788574) | > grad_norm_0: tensor(26.0347, device='cuda:0') (tensor(26.0347, device='cuda:0')) | > loss_gen: 1.9416205883026123 (1.9416205883026123) | > loss_kl: 2.1163992881774902 (2.1163992881774902) | > loss_feat: 2.8756103515625 (2.8756103515625) | > loss_mel: 17.94223403930664 (17.94223403930664) | > loss_duration: 1.6126255989074707 (1.6126255989074707) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.488489151000977 (26.488489151000977) | > grad_norm_1: tensor(840.2051, device='cuda:0') (tensor(840.2051, device='cuda:0')) | > current_lr_0: 0.00019182140502061165 | > current_lr_1: 0.00019182140502061165 | > step_time: 0.4288 (0.428760290145874) | > loader_time: 0.0057 (0.005735635757446289)  --> TIME: 2025-02-25 11:38:22 -- STEP: 51/72 -- GLOBAL_STEP: 1024100 | > loss_disc: 2.7162654399871826 (2.7543302657557467) | > loss_disc_real_0: 0.16165557503700256 (0.2015334795795235) | > loss_disc_real_1: 0.19988976418972015 (0.23292884289049634) | > loss_disc_real_2: 0.24320314824581146 (0.2391931581146577) | > loss_disc_real_3: 0.24645695090293884 (0.23913878932887433) | > loss_disc_real_4: 0.2582099139690399 (0.24505317065061308) | > loss_disc_real_5: 0.23877108097076416 (0.24488001594356462) | > loss_0: 2.7162654399871826 (2.7543302657557467) | > grad_norm_0: tensor(19.9985, device='cuda:0') (tensor(12.6370, device='cuda:0')) | > loss_gen: 1.8019465208053589 (1.872637895976796) | > loss_kl: 2.0547914505004883 (2.1911598397236243) | > loss_feat: 2.6595194339752197 (2.7168293279760025) | > loss_mel: 16.445344924926758 (17.34449467004514) | > loss_duration: 1.6292791366577148 (1.5853965446060778) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.59088134765625 (25.71051810769474) | > grad_norm_1: tensor(154.7814, device='cuda:0') (tensor(464.7581, device='cuda:0')) | > current_lr_0: 0.00019182140502061165 | > current_lr_1: 0.00019182140502061165 | > step_time: 0.4459 (0.44255318828657564) | > loader_time: 0.0094 (0.0077344670015222884) --> EVAL PERFORMANCE | > avg_loader_time: 0.008594351755061623 (-0.0002202517549756526) | > avg_step_time: 0.4564017678650332 (-0.008349220517655465) | > avg_loss_disc: 2.767119283407507 (-0.014411909479490337) | > avg_loss_disc_real_0: 0.20327603302791084 (-0.001639133922650804) | > avg_loss_disc_real_1: 0.23171398904122098 (+0.00011816327001007432) | > avg_loss_disc_real_2: 0.24112274277378137 (+0.0030122982784056562) | > avg_loss_disc_real_3: 0.23968015925984987 (+0.002169938574374558) | > avg_loss_disc_real_4: 0.24580738682981948 (+0.00027996658439366495) | > avg_loss_disc_real_5: 0.24574397210503968 (-0.0021573221179800595) | > avg_loss_0: 2.767119283407507 (-0.014411909479490337) | > avg_grad_norm_0: tensor(12.0043, device='cuda:0') (tensor(-2.9514, device='cuda:0')) | > avg_loss_gen: 1.8633428526596285 (+0.03262333299072706) | > avg_loss_kl: 2.2360663497951667 (-0.03808367756051112) | > avg_loss_feat: 2.680024697747029 (+0.03230800427181624) | > avg_loss_mel: 17.497277098642268 (-0.13829765857104803) | > avg_loss_duration: 1.6004797404920552 (-0.009169659144441678) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.877190603336818 (-0.12061984102490797) | > avg_grad_norm_1: tensor(432.5217, device='cuda:0') (+tensor(4.5395, device='cuda:0'))  > EPOCH: 335/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:38:34)   --> TIME: 2025-02-25 11:38:49 -- STEP: 29/72 -- GLOBAL_STEP: 1024150 | > loss_disc: 2.731383800506592 (2.762985657001364) | > loss_disc_real_0: 0.2067318856716156 (0.2107922748245042) | > loss_disc_real_1: 0.19918952882289886 (0.22452117913755879) | > loss_disc_real_2: 0.29743069410324097 (0.24010604568596544) | > loss_disc_real_3: 0.24946343898773193 (0.23976996695173197) | > loss_disc_real_4: 0.27024897933006287 (0.2428477730216651) | > loss_disc_real_5: 0.2672901153564453 (0.24503252989259258) | > loss_0: 2.731383800506592 (2.762985657001364) | > grad_norm_0: tensor(8.5907, device='cuda:0') (tensor(12.4242, device='cuda:0')) | > loss_gen: 1.8437554836273193 (1.8639379452014793) | > loss_kl: 2.2502074241638184 (2.3778066059638716) | > loss_feat: 2.831799030303955 (2.7539476772834517) | > loss_mel: 18.465707778930664 (17.861172511659824) | > loss_duration: 1.5798046588897705 (1.5885380259875594) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.971275329589844 (26.445402605780238) | > grad_norm_1: tensor(98.5372, device='cuda:0') (tensor(369.5061, device='cuda:0')) | > current_lr_0: 0.00019179742734498406 | > current_lr_1: 0.00019179742734498406 | > step_time: 0.4433 (0.4345004805203142) | > loader_time: 0.0088 (0.007001408215226798) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865319077397736 (+5.88390189157368e-05) | > avg_step_time: 0.45394808473721354 (-0.0024536831278196836) | > avg_loss_disc: 2.778941402972584 (+0.01182211956507695) | > avg_loss_disc_real_0: 0.21047867393829453 (+0.007202640910383695) | > avg_loss_disc_real_1: 0.23381964610496037 (+0.0021056570637393857) | > avg_loss_disc_real_2: 0.23937347279468052 (-0.0017492699791008537) | > avg_loss_disc_real_3: 0.2388770989548992 (-0.0008030603049506702) | > avg_loss_disc_real_4: 0.2445792790869592 (-0.0012281077428602793) | > avg_loss_disc_real_5: 0.24922099352722438 (+0.0034770214221847007) | > avg_loss_0: 2.778941402972584 (+0.01182211956507695) | > avg_grad_norm_0: tensor(12.6670, device='cuda:0') (+tensor(0.6627, device='cuda:0')) | > avg_loss_gen: 1.8559813130069787 (-0.007361539652649851) | > avg_loss_kl: 2.331481120955776 (+0.09541477116060948) | > avg_loss_feat: 2.6846536414724 (+0.0046289437253710375) | > avg_loss_mel: 17.735622862694967 (+0.23834576405269914) | > avg_loss_duration: 1.6141052313253914 (+0.013625490833336196) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.221844149307465 (+0.34465354597064746) | > avg_grad_norm_1: tensor(372.7707, device='cuda:0') (tensor(-59.7510, device='cuda:0'))  > EPOCH: 336/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:39:11)   --> TIME: 2025-02-25 11:39:16 -- STEP: 7/72 -- GLOBAL_STEP: 1024200 | > loss_disc: 2.77201247215271 (2.754518985748291) | > loss_disc_real_0: 0.22918760776519775 (0.21448932800974166) | > loss_disc_real_1: 0.18957430124282837 (0.23287409756864821) | > loss_disc_real_2: 0.23801355063915253 (0.2293270911489214) | > loss_disc_real_3: 0.21142731606960297 (0.23630874071802413) | > loss_disc_real_4: 0.2414788156747818 (0.23679918476513453) | > loss_disc_real_5: 0.26090946793556213 (0.25289283054215567) | > loss_0: 2.77201247215271 (2.754518985748291) | > grad_norm_0: tensor(10.3931, device='cuda:0') (tensor(12.2764, device='cuda:0')) | > loss_gen: 1.777898907661438 (1.8622122492109026) | > loss_kl: 2.175567150115967 (2.1493413959230696) | > loss_feat: 3.0127265453338623 (2.8091086660112654) | > loss_mel: 19.208444595336914 (17.712617874145508) | > loss_duration: 1.559133768081665 (1.5691619770867484) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.7337703704834 (26.10244233267648) | > grad_norm_1: tensor(642.8749, device='cuda:0') (tensor(343.0163, device='cuda:0')) | > current_lr_0: 0.00019177345266656591 | > current_lr_1: 0.00019177345266656591 | > step_time: 0.4271 (0.43170176233564106) | > loader_time: 0.007 (0.006911822727748326)  --> TIME: 2025-02-25 11:39:39 -- STEP: 57/72 -- GLOBAL_STEP: 1024250 | > loss_disc: 2.78558611869812 (2.7623365092695806) | > loss_disc_real_0: 0.20939156413078308 (0.19664034051330467) | > loss_disc_real_1: 0.20701080560684204 (0.23073494460499078) | > loss_disc_real_2: 0.218520849943161 (0.23759332246947706) | > loss_disc_real_3: 0.23873811960220337 (0.2385005404551824) | > loss_disc_real_4: 0.2568803131580353 (0.24178851408916607) | > loss_disc_real_5: 0.2501019239425659 (0.24645354219695978) | > loss_0: 2.78558611869812 (2.7623365092695806) | > grad_norm_0: tensor(8.6361, device='cuda:0') (tensor(12.0862, device='cuda:0')) | > loss_gen: 1.688024878501892 (1.853420012875607) | > loss_kl: 2.213766574859619 (2.282196837559081) | > loss_feat: 2.446272850036621 (2.8044291755609345) | > loss_mel: 17.021032333374023 (18.017540781121514) | > loss_duration: 1.6451224088668823 (1.5953649240627623) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.014219284057617 (26.552951846206398) | > grad_norm_1: tensor(265.6552, device='cuda:0') (tensor(375.5442, device='cuda:0')) | > current_lr_0: 0.00019177345266656591 | > current_lr_1: 0.00019177345266656591 | > step_time: 0.4743 (0.4436035449044746) | > loader_time: 0.0102 (0.008106570494802375) --> EVAL PERFORMANCE | > avg_loader_time: 0.008747664975448393 (+9.44742014710332e-05) | > avg_step_time: 0.45423397883562977 (+0.00028589409841622837) | > avg_loss_disc: 2.7610447138128147 (-0.017896689159769164) | > avg_loss_disc_real_0: 0.19834197648394275 (-0.01213669745435178) | > avg_loss_disc_real_1: 0.22995960985271025 (-0.003860036252250121) | > avg_loss_disc_real_2: 0.23732202783436843 (-0.0020514449603120932) | > avg_loss_disc_real_3: 0.23783907630074191 (-0.0010380226541572835) | > avg_loss_disc_real_4: 0.24130259496225437 (-0.0032766841247048317) | > avg_loss_disc_real_5: 0.2451769235268445 (-0.0040440700003798735) | > avg_loss_0: 2.7610447138128147 (-0.017896689159769164) | > avg_grad_norm_0: tensor(12.3572, device='cuda:0') (tensor(-0.3099, device='cuda:0')) | > avg_loss_gen: 1.8497891442876466 (-0.006192168719332036) | > avg_loss_kl: 2.2811018967292678 (-0.05037922422650842) | > avg_loss_feat: 2.7784304518095206 (+0.0937768103371206) | > avg_loss_mel: 17.914320583074872 (+0.17869772037990472) | > avg_loss_duration: 1.6050987478712915 (-0.009006483454099934) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.42874091779682 (+0.20689676848935434) | > avg_grad_norm_1: tensor(414.2616, device='cuda:0') (+tensor(41.4909, device='cuda:0'))  > EPOCH: 337/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:39:48)   --> TIME: 2025-02-25 11:40:06 -- STEP: 35/72 -- GLOBAL_STEP: 1024300 | > loss_disc: 2.821631908416748 (2.762571082796369) | > loss_disc_real_0: 0.19930294156074524 (0.204035379418305) | > loss_disc_real_1: 0.2681197226047516 (0.23366308978625705) | > loss_disc_real_2: 0.267105370759964 (0.23678534626960754) | > loss_disc_real_3: 0.27133119106292725 (0.2350063932793481) | > loss_disc_real_4: 0.2491380125284195 (0.2464324721268245) | > loss_disc_real_5: 0.24609176814556122 (0.2474703861134393) | > loss_0: 2.821631908416748 (2.762571082796369) | > grad_norm_0: tensor(6.3192, device='cuda:0') (tensor(15.5655, device='cuda:0')) | > loss_gen: 1.8735756874084473 (1.8585037401744298) | > loss_kl: 2.404520034790039 (2.2587507043566015) | > loss_feat: 2.3397409915924072 (2.776417684555053) | > loss_mel: 18.36402130126953 (17.851512908935547) | > loss_duration: 1.619382381439209 (1.6009090457643782) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.601240158081055 (26.346094403948104) | > grad_norm_1: tensor(69.0051, device='cuda:0') (tensor(401.4847, device='cuda:0')) | > current_lr_0: 0.0001917494809849826 | > current_lr_1: 0.0001917494809849826 | > step_time: 0.4642 (0.439569970539638) | > loader_time: 0.0083 (0.007133347647530692) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862106806795362 (-0.00012659690749477343) | > avg_step_time: 0.4581405310563638 (+0.00390655222073405) | > avg_loss_disc: 2.77764504728183 (+0.01660033346901546) | > avg_loss_disc_real_0: 0.20069854118874375 (+0.0023565647048010008) | > avg_loss_disc_real_1: 0.2309343583986793 (+0.0009747485459690453) | > avg_loss_disc_real_2: 0.23823215885901114 (+0.000910131024642713) | > avg_loss_disc_real_3: 0.2377637423260111 (-7.533397473080683e-05) | > avg_loss_disc_real_4: 0.24469432360689405 (+0.0033917286446396855) | > avg_loss_disc_real_5: 0.24669669736439073 (+0.0015197738375462289) | > avg_loss_0: 2.77764504728183 (+0.01660033346901546) | > avg_grad_norm_0: tensor(15.7219, device='cuda:0') (+tensor(3.3647, device='cuda:0')) | > avg_loss_gen: 1.8326884676033341 (-0.017100676684312477) | > avg_loss_kl: 2.2835966768399087 (+0.0024947801106409706) | > avg_loss_feat: 2.696733206090793 (-0.08169724571872772) | > avg_loss_mel: 17.792937668276505 (-0.12138291479836738) | > avg_loss_duration: 1.6172910559345302 (+0.012192308063238677) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.22324731316365 (-0.205493604633169) | > avg_grad_norm_1: tensor(353.7790, device='cuda:0') (tensor(-60.4826, device='cuda:0'))  > EPOCH: 338/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:40:26)   --> TIME: 2025-02-25 11:40:33 -- STEP: 13/72 -- GLOBAL_STEP: 1024350 | > loss_disc: 2.8110852241516113 (2.721093232815082) | > loss_disc_real_0: 0.09133903682231903 (0.19343753904104233) | > loss_disc_real_1: 0.21086585521697998 (0.23587498183433825) | > loss_disc_real_2: 0.21618294715881348 (0.2272202315238806) | > loss_disc_real_3: 0.22144943475723267 (0.23530652202092683) | > loss_disc_real_4: 0.25182870030403137 (0.24325622274325445) | > loss_disc_real_5: 0.2305678278207779 (0.2409830357019718) | > loss_0: 2.8110852241516113 (2.721093232815082) | > grad_norm_0: tensor(30.7186, device='cuda:0') (tensor(15.2199, device='cuda:0')) | > loss_gen: 1.675154685974121 (1.8778133575732892) | > loss_kl: 2.196413516998291 (2.1931617626777062) | > loss_feat: 3.0535876750946045 (2.8963021681858945) | > loss_mel: 18.73097038269043 (18.31959739098182) | > loss_duration: 1.4824323654174805 (1.5737342742773204) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.138561248779297 (26.860609788161057) | > grad_norm_1: tensor(155.3965, device='cuda:0') (tensor(475.5852, device='cuda:0')) | > current_lr_0: 0.00019172551229985947 | > current_lr_1: 0.00019172551229985947 | > step_time: 0.4566 (0.4391194673684927) | > loader_time: 0.0057 (0.006861393268291767)  --> TIME: 2025-02-25 11:40:57 -- STEP: 63/72 -- GLOBAL_STEP: 1024400 | > loss_disc: 2.692021608352661 (2.771229441203768) | > loss_disc_real_0: 0.21034052968025208 (0.20475029637889255) | > loss_disc_real_1: 0.2772007882595062 (0.23353058691062625) | > loss_disc_real_2: 0.2072775810956955 (0.24164158744471415) | > loss_disc_real_3: 0.24138396978378296 (0.2394008645935664) | > loss_disc_real_4: 0.24787212908267975 (0.24412185190215943) | > loss_disc_real_5: 0.2692258358001709 (0.24632523835651457) | > loss_0: 2.692021608352661 (2.771229441203768) | > grad_norm_0: tensor(6.0344, device='cuda:0') (tensor(15.7888, device='cuda:0')) | > loss_gen: 1.7370071411132812 (1.8687671120204623) | > loss_kl: 2.266709089279175 (2.260972019225832) | > loss_feat: 2.7209134101867676 (2.7498567331404913) | > loss_mel: 17.809646606445312 (17.944376809256422) | > loss_duration: 1.5937607288360596 (1.6113046313089034) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.128036499023438 (26.435277454436772) | > grad_norm_1: tensor(686.1877, device='cuda:0') (tensor(429.4844, device='cuda:0')) | > current_lr_0: 0.00019172551229985947 | > current_lr_1: 0.00019172551229985947 | > step_time: 0.5033 (0.45275072824387325) | > loader_time: 0.0115 (0.008294487756396098) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868344306945801 (+6.237500150439029e-05) | > avg_step_time: 0.4612115033915345 (+0.003070972335170674) | > avg_loss_disc: 2.770314528908528 (-0.0073305183733021195) | > avg_loss_disc_real_0: 0.20248056486458846 (+0.0017820236758447072) | > avg_loss_disc_real_1: 0.22980571338828182 (-0.0011286450103974721) | > avg_loss_disc_real_2: 0.24089843118694468 (+0.0026662723279335387) | > avg_loss_disc_real_3: 0.2384388201253515 (+0.0006750777993403956) | > avg_loss_disc_real_4: 0.24296938428576564 (-0.0017249393211284103) | > avg_loss_disc_real_5: 0.24587457679526906 (-0.000822120569121676) | > avg_loss_0: 2.770314528908528 (-0.0073305183733021195) | > avg_grad_norm_0: tensor(15.5647, device='cuda:0') (tensor(-0.1572, device='cuda:0')) | > avg_loss_gen: 1.857104758141746 (+0.024416290538411767) | > avg_loss_kl: 2.25984573028457 (-0.023750946555338714) | > avg_loss_feat: 2.7313763003953744 (+0.03464309430458146) | > avg_loss_mel: 17.901702921155476 (+0.1087652528789711) | > avg_loss_duration: 1.6163162513517995 (-0.0009748045827306928) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.366346198068538 (+0.1430988849048873) | > avg_grad_norm_1: tensor(431.5967, device='cuda:0') (+tensor(77.8177, device='cuda:0'))  > EPOCH: 339/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:41:03)   --> TIME: 2025-02-25 11:41:24 -- STEP: 41/72 -- GLOBAL_STEP: 1024450 | > loss_disc: 2.5534098148345947 (2.7830274163222892) | > loss_disc_real_0: 0.1199774220585823 (0.20994108733607503) | > loss_disc_real_1: 0.17094272375106812 (0.2332073333786755) | > loss_disc_real_2: 0.21582072973251343 (0.24337289355150082) | > loss_disc_real_3: 0.2288529872894287 (0.2405529425638478) | > loss_disc_real_4: 0.19892284274101257 (0.24534590469627846) | > loss_disc_real_5: 0.2230353206396103 (0.24618469542119562) | > loss_0: 2.5534098148345947 (2.7830274163222892) | > grad_norm_0: tensor(14.8064, device='cuda:0') (tensor(14.2175, device='cuda:0')) | > loss_gen: 1.8334245681762695 (1.851066191021989) | > loss_kl: 2.056283712387085 (2.1860458269351875) | > loss_feat: 3.437021017074585 (2.7016038138691973) | > loss_mel: 17.910106658935547 (17.857178548487223) | > loss_duration: 1.600210189819336 (1.5808720297929717) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.837045669555664 (26.176766279267103) | > grad_norm_1: tensor(583.1843, device='cuda:0') (tensor(355.3609, device='cuda:0')) | > current_lr_0: 0.000191701546610822 | > current_lr_1: 0.000191701546610822 | > step_time: 0.4588 (0.4407083522982714) | > loader_time: 0.0098 (0.007593259578797876) --> EVAL PERFORMANCE | > avg_loader_time: 0.008770573307091084 (+8.713023763307418e-05) | > avg_step_time: 0.4570384697175362 (-0.004173033673998294) | > avg_loss_disc: 2.782369220760507 (+0.012054691851978827) | > avg_loss_disc_real_0: 0.21287832545562527 (+0.010397760591036809) | > avg_loss_disc_real_1: 0.23074994750425848 (+0.000944234115976661) | > avg_loss_disc_real_2: 0.23958636022789379 (-0.0013120709590508928) | > avg_loss_disc_real_3: 0.238403642681283 (-3.517744406850154e-05) | > avg_loss_disc_real_4: 0.24422612324566909 (+0.0012567389599034418) | > avg_loss_disc_real_5: 0.24630058357413387 (+0.00042600677886481253) | > avg_loss_0: 2.782369220760507 (+0.012054691851978827) | > avg_grad_norm_0: tensor(13.8836, device='cuda:0') (tensor(-1.6811, device='cuda:0')) | > avg_loss_gen: 1.8462418200264514 (-0.010862938115294485) | > avg_loss_kl: 2.2198096194737396 (-0.04003611081083047) | > avg_loss_feat: 2.6590293897709376 (-0.07234691062443677) | > avg_loss_mel: 17.711222621756534 (-0.19048029939894207) | > avg_loss_duration: 1.6049798810985727 (-0.011336370253226757) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.0412831910899 (-0.3250630069786382) | > avg_grad_norm_1: tensor(368.3313, device='cuda:0') (tensor(-63.2654, device='cuda:0'))  > EPOCH: 340/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:41:40)   --> TIME: 2025-02-25 11:41:51 -- STEP: 19/72 -- GLOBAL_STEP: 1024500 | > loss_disc: 2.7170863151550293 (2.7418229956375924) | > loss_disc_real_0: 0.15291611850261688 (0.2022274685533423) | > loss_disc_real_1: 0.22426408529281616 (0.23417902777069494) | > loss_disc_real_2: 0.23100344836711884 (0.2385249177092) | > loss_disc_real_3: 0.22292986512184143 (0.23884842976143486) | > loss_disc_real_4: 0.23229138553142548 (0.24272713300428891) | > loss_disc_real_5: 0.23505829274654388 (0.24275042273496328) | > loss_0: 2.7170863151550293 (2.7418229956375924) | > grad_norm_0: tensor(9.0203, device='cuda:0') (tensor(11.4354, device='cuda:0')) | > loss_gen: 1.9490807056427002 (1.8791017595090365) | > loss_kl: 2.058237314224243 (2.0529258753124036) | > loss_feat: 3.0303080081939697 (2.767774820327759) | > loss_mel: 17.795188903808594 (17.532308377717673) | > loss_duration: 1.5409891605377197 (1.561439269467404) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.373804092407227 (25.793550089785928) | > grad_norm_1: tensor(442.5049, device='cuda:0') (tensor(484.2926, device='cuda:0')) | > current_lr_0: 0.00019167758391749564 | > current_lr_1: 0.00019167758391749564 | > step_time: 0.4395 (0.4334276600887901) | > loader_time: 0.0063 (0.006809046393946598)  --> TIME: 2025-02-25 11:42:14 -- STEP: 69/72 -- GLOBAL_STEP: 1024550 | > loss_disc: 2.7457025051116943 (2.7806525195854297) | > loss_disc_real_0: 0.2527731657028198 (0.21010299912397412) | > loss_disc_real_1: 0.22357377409934998 (0.23366352591825568) | > loss_disc_real_2: 0.22164379060268402 (0.23944871473139612) | > loss_disc_real_3: 0.2579897940158844 (0.2400938939789067) | > loss_disc_real_4: 0.23547422885894775 (0.24587449129076971) | > loss_disc_real_5: 0.2468458116054535 (0.247674605552701) | > loss_0: 2.7457025051116943 (2.7806525195854297) | > grad_norm_0: tensor(6.1512, device='cuda:0') (tensor(14.0373, device='cuda:0')) | > loss_gen: 1.8244261741638184 (1.8498492240905762) | > loss_kl: 2.2946054935455322 (2.2341805817424385) | > loss_feat: 2.6361868381500244 (2.668662541154502) | > loss_mel: 18.128986358642578 (17.522863609203394) | > loss_duration: 1.6924241781234741 (1.6023737002110137) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.576629638671875 (25.877929576929066) | > grad_norm_1: tensor(188.6838, device='cuda:0') (tensor(368.0576, device='cuda:0')) | > current_lr_0: 0.00019167758391749564 | > current_lr_1: 0.00019167758391749564 | > step_time: 0.5394 (0.4537789821624756) | > loader_time: 0.0125 (0.008548498153686523) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612568949309873 (-0.0001580043577812107) | > avg_step_time: 0.4556042946560282 (-0.0014341750615080029) | > avg_loss_disc: 2.778336165656506 (-0.004033055104000649) | > avg_loss_disc_real_0: 0.20936486020054615 (-0.003513465255079118) | > avg_loss_disc_real_1: 0.233129524219204 (+0.002379576714945514) | > avg_loss_disc_real_2: 0.23954316868748463 (-4.3191540409159845e-05) | > avg_loss_disc_real_3: 0.23975511231053043 (+0.001351469629247426) | > avg_loss_disc_real_4: 0.2455708342538753 (+0.001344711008206212) | > avg_loss_disc_real_5: 0.24741236571694764 (+0.0011117821428137664) | > avg_loss_0: 2.778336165656506 (-0.004033055104000649) | > avg_grad_norm_0: tensor(13.9219, device='cuda:0') (+tensor(0.0383, device='cuda:0')) | > avg_loss_gen: 1.849205282372488 (+0.002963462346036616) | > avg_loss_kl: 2.237951980510228 (+0.018142361036488364) | > avg_loss_feat: 2.665110487333486 (+0.0060810975625482655) | > avg_loss_mel: 17.525720838090063 (-0.18550178366647074) | > avg_loss_duration: 1.6026710708376388 (-0.00230881026093388) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.880659613810796 (-0.1606235772791038) | > avg_grad_norm_1: tensor(364.3101, device='cuda:0') (tensor(-4.0212, device='cuda:0'))  > EPOCH: 341/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:42:18)   --> TIME: 2025-02-25 11:42:41 -- STEP: 47/72 -- GLOBAL_STEP: 1024600 | > loss_disc: 2.7556326389312744 (2.7800674590658634) | > loss_disc_real_0: 0.14272871613502502 (0.1985557952776868) | > loss_disc_real_1: 0.23042923212051392 (0.23262633700320062) | > loss_disc_real_2: 0.27123886346817017 (0.24173349078665388) | > loss_disc_real_3: 0.21651363372802734 (0.24162298155591846) | > loss_disc_real_4: 0.23707255721092224 (0.24692695793953348) | > loss_disc_real_5: 0.2611254155635834 (0.24971514971966438) | > loss_0: 2.7556326389312744 (2.7800674590658634) | > grad_norm_0: tensor(14.0643, device='cuda:0') (tensor(11.8506, device='cuda:0')) | > loss_gen: 1.7966152429580688 (1.8744995594024658) | > loss_kl: 2.2596495151519775 (2.259827202938972) | > loss_feat: 2.4545018672943115 (2.753677013072562) | > loss_mel: 16.594921112060547 (18.00406744125042) | > loss_duration: 1.6180862188339233 (1.597537192892521) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.723773956298828 (26.489608399411466) | > grad_norm_1: tensor(648.1989, device='cuda:0') (tensor(355.6132, device='cuda:0')) | > current_lr_0: 0.00019165362421950594 | > current_lr_1: 0.00019165362421950594 | > step_time: 0.4556 (0.4433681660510124) | > loader_time: 0.0089 (0.007817207498753325) --> EVAL PERFORMANCE | > avg_loader_time: 0.008768357021707885 (+0.00015578807239801165) | > avg_step_time: 0.45897321633889643 (+0.003368921682868231) | > avg_loss_disc: 2.778233094954155 (-0.00010307070235127114) | > avg_loss_disc_real_0: 0.20055641695647172 (-0.008808443244074426) | > avg_loss_disc_real_1: 0.23392855029710583 (+0.0007990260779018321) | > avg_loss_disc_real_2: 0.2400244541571174 (+0.0004812854696327751) | > avg_loss_disc_real_3: 0.24019915725983365 (+0.0004440449493032239) | > avg_loss_disc_real_4: 0.2456092809287595 (+3.8446674884201615e-05) | > avg_loss_disc_real_5: 0.24840773726013346 (+0.0009953715431858245) | > avg_loss_0: 2.778233094954155 (-0.00010307070235127114) | > avg_grad_norm_0: tensor(12.1398, device='cuda:0') (tensor(-1.7822, device='cuda:0')) | > avg_loss_gen: 1.8637695010279265 (+0.014564218655438443) | > avg_loss_kl: 2.2560914738077495 (+0.018139493297521625) | > avg_loss_feat: 2.6984204977331028 (+0.033310010399616896) | > avg_loss_mel: 17.74993732613577 (+0.22421648804570538) | > avg_loss_duration: 1.6115159182481362 (+0.008844847410497403) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.17973456584232 (+0.2990749520315248) | > avg_grad_norm_1: tensor(383.5894, device='cuda:0') (+tensor(19.2793, device='cuda:0'))  > EPOCH: 342/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:42:55)   --> TIME: 2025-02-25 11:43:08 -- STEP: 25/72 -- GLOBAL_STEP: 1024650 | > loss_disc: 2.679237127304077 (2.766007661819458) | > loss_disc_real_0: 0.20850399136543274 (0.20234226763248445) | > loss_disc_real_1: 0.2096254825592041 (0.22615087628364564) | > loss_disc_real_2: 0.21087799966335297 (0.23550785899162294) | > loss_disc_real_3: 0.24075447022914886 (0.23747365176677704) | > loss_disc_real_4: 0.23305973410606384 (0.24766736090183258) | > loss_disc_real_5: 0.23788656294345856 (0.2504255485534669) | > loss_0: 2.679237127304077 (2.766007661819458) | > grad_norm_0: tensor(6.3830, device='cuda:0') (tensor(16.5400, device='cuda:0')) | > loss_gen: 1.9641990661621094 (1.8552240705490113) | > loss_kl: 2.4753024578094482 (2.256365103721619) | > loss_feat: 3.0779740810394287 (2.7173796844482423) | > loss_mel: 17.980884552001953 (17.822227325439457) | > loss_duration: 1.5606882572174072 (1.570195574760437) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.05904769897461 (26.221391677856445) | > grad_norm_1: tensor(768.5757, device='cuda:0') (tensor(464.4311, device='cuda:0')) | > current_lr_0: 0.0001916296675164785 | > current_lr_1: 0.0001916296675164785 | > step_time: 0.4356 (0.4379125118255615) | > loader_time: 0.0081 (0.007186574935913086) --> EVAL PERFORMANCE | > avg_loader_time: 0.008755589874697404 (-1.2767147010480512e-05) | > avg_step_time: 0.45749910448638487 (-0.0014741118525115615) | > avg_loss_disc: 2.764393097917798 (-0.013839997036356966) | > avg_loss_disc_real_0: 0.19821124723259828 (-0.0023451697238734415) | > avg_loss_disc_real_1: 0.22757866307043692 (-0.006349887226668904) | > avg_loss_disc_real_2: 0.23843861360784988 (-0.0015858405492675254) | > avg_loss_disc_real_3: 0.23644758739941557 (-0.0037515698604180803) | > avg_loss_disc_real_4: 0.24429643259082043 (-0.001312848337939071) | > avg_loss_disc_real_5: 0.24760796171678623 (-0.0007997755433472309) | > avg_loss_0: 2.764393097917798 (-0.013839997036356966) | > avg_grad_norm_0: tensor(15.9786, device='cuda:0') (+tensor(3.8389, device='cuda:0')) | > avg_loss_gen: 1.8475374067333383 (-0.016232094294588162) | > avg_loss_kl: 2.2644836583607626 (+0.008392184553013049) | > avg_loss_feat: 2.6906781095853995 (-0.0077423881477032275) | > avg_loss_mel: 17.597460236347903 (-0.15247708978786534) | > avg_loss_duration: 1.604172857714371 (-0.007343060533765211) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.004332179754552 (-0.17540238608776804) | > avg_grad_norm_1: tensor(516.9057, device='cuda:0') (+tensor(133.3163, device='cuda:0'))  > EPOCH: 343/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:43:32)   --> TIME: 2025-02-25 11:43:35 -- STEP: 3/72 -- GLOBAL_STEP: 1024700 | > loss_disc: 2.7508749961853027 (2.7454257806142173) | > loss_disc_real_0: 0.20609474182128906 (0.2590149740378062) | > loss_disc_real_1: 0.250429630279541 (0.27940760056177777) | > loss_disc_real_2: 0.2460862398147583 (0.2363086094458898) | > loss_disc_real_3: 0.26321569085121155 (0.2661873797575633) | > loss_disc_real_4: 0.25859761238098145 (0.2417277842760086) | > loss_disc_real_5: 0.2641143798828125 (0.25038554271062213) | > loss_0: 2.7508749961853027 (2.7454257806142173) | > grad_norm_0: tensor(27.8898, device='cuda:0') (tensor(27.3881, device='cuda:0')) | > loss_gen: 1.8467206954956055 (1.9694456259409587) | > loss_kl: 2.229449987411499 (2.140941063563029) | > loss_feat: 2.672842025756836 (2.7570842107137046) | > loss_mel: 17.644428253173828 (17.613501866658527) | > loss_duration: 1.614339828491211 (1.5865588585535686) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.007780075073242 (26.06753158569336) | > grad_norm_1: tensor(327.4259, device='cuda:0') (tensor(408.3044, device='cuda:0')) | > current_lr_0: 0.00019160571380803893 | > current_lr_1: 0.00019160571380803893 | > step_time: 0.445 (0.44304243723551434) | > loader_time: 0.0063 (0.005994399388631185)  --> TIME: 2025-02-25 11:43:58 -- STEP: 53/72 -- GLOBAL_STEP: 1024750 | > loss_disc: 2.8368639945983887 (2.764427373994072) | > loss_disc_real_0: 0.12800666689872742 (0.21072600135263406) | > loss_disc_real_1: 0.22189664840698242 (0.23089021837936258) | > loss_disc_real_2: 0.2079685628414154 (0.23667580873336433) | > loss_disc_real_3: 0.19566915929317474 (0.2388369674390217) | > loss_disc_real_4: 0.20612496137619019 (0.24461875518537918) | > loss_disc_real_5: 0.20225906372070312 (0.2449434981593546) | > loss_0: 2.8368639945983887 (2.764427373994072) | > grad_norm_0: tensor(20.8779, device='cuda:0') (tensor(15.6008, device='cuda:0')) | > loss_gen: 1.6853899955749512 (1.8619446731963247) | > loss_kl: 2.3353703022003174 (2.2339761257171626) | > loss_feat: 3.220353364944458 (2.7447863704753366) | > loss_mel: 18.437192916870117 (17.705105691585896) | > loss_duration: 1.6514942646026611 (1.5988836513375335) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.329801559448242 (26.144696721490824) | > grad_norm_1: tensor(594.1792, device='cuda:0') (tensor(416.9824, device='cuda:0')) | > current_lr_0: 0.00019160571380803893 | > current_lr_1: 0.00019160571380803893 | > step_time: 0.453 (0.44681679977560945) | > loader_time: 0.0093 (0.007937624769390752) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871938047274737 (-3.620940195003461e-05) | > avg_step_time: 0.46234596950907103 (+0.004846865022686164) | > avg_loss_disc: 2.768814033185933 (+0.004420935268135029) | > avg_loss_disc_real_0: 0.20771056193281226 (+0.009499314700213979) | > avg_loss_disc_real_1: 0.23281942454861923 (+0.00524076147818231) | > avg_loss_disc_real_2: 0.23733070933483016 (-0.0011079042730197108) | > avg_loss_disc_real_3: 0.23864408467017428 (+0.0021964972707587127) | > avg_loss_disc_real_4: 0.24421026841015883 (-8.616418066159648e-05) | > avg_loss_disc_real_5: 0.24542659101351885 (-0.002181370703267377) | > avg_loss_0: 2.768814033185933 (+0.004420935268135029) | > avg_grad_norm_0: tensor(15.1466, device='cuda:0') (tensor(-0.8321, device='cuda:0')) | > avg_loss_gen: 1.856765820946492 (+0.009228414213153568) | > avg_loss_kl: 2.242032366739192 (-0.022451291621570668) | > avg_loss_feat: 2.7136854655306104 (+0.02300735594521086) | > avg_loss_mel: 17.714986613098997 (+0.11752637675109412) | > avg_loss_duration: 1.6104935592329 (+0.006320701518528926) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.137963899424378 (+0.13363171966982534) | > avg_grad_norm_1: tensor(427.3527, device='cuda:0') (tensor(-89.5530, device='cuda:0'))  > EPOCH: 344/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:44:09)   --> TIME: 2025-02-25 11:44:25 -- STEP: 31/72 -- GLOBAL_STEP: 1024800 | > loss_disc: 2.802572727203369 (2.756264063619798) | > loss_disc_real_0: 0.21837389469146729 (0.19999358226214686) | > loss_disc_real_1: 0.27547675371170044 (0.23599449088496546) | > loss_disc_real_2: 0.2516060769557953 (0.2441203954719728) | > loss_disc_real_3: 0.27163082361221313 (0.2402564791902419) | > loss_disc_real_4: 0.25508689880371094 (0.24525682387813444) | > loss_disc_real_5: 0.23009057343006134 (0.2480777790469508) | > loss_0: 2.802572727203369 (2.756264063619798) | > grad_norm_0: tensor(5.6294, device='cuda:0') (tensor(14.5017, device='cuda:0')) | > loss_gen: 1.8074127435684204 (1.9044469864137712) | > loss_kl: 2.3223516941070557 (2.2407786077068703) | > loss_feat: 2.738311290740967 (2.824730788507769) | > loss_mel: 18.030292510986328 (18.203717508623676) | > loss_duration: 1.6082487106323242 (1.5711252304815477) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.50661849975586 (26.744799275552072) | > grad_norm_1: tensor(67.5031, device='cuda:0') (tensor(421.2195, device='cuda:0')) | > current_lr_0: 0.0001915817630938129 | > current_lr_1: 0.0001915817630938129 | > step_time: 0.4661 (0.4373046275108091) | > loader_time: 0.0087 (0.007182567350326046) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866042392354616 (-5.895654920120945e-05) | > avg_step_time: 0.4545576874638947 (-0.007788282045176342) | > avg_loss_disc: 2.804643046688026 (+0.03582901350209289) | > avg_loss_disc_real_0: 0.21338535383553572 (+0.005674791902723464) | > avg_loss_disc_real_1: 0.23652597611219112 (+0.003706551563571886) | > avg_loss_disc_real_2: 0.24470632260953876 (+0.007375613274708592) | > avg_loss_disc_real_3: 0.24511874603553557 (+0.0064746613653612894) | > avg_loss_disc_real_4: 0.24421370071424564 (+3.432304086808946e-06) | > avg_loss_disc_real_5: 0.24615583600292743 (+0.0007292449894085729) | > avg_loss_0: 2.804643046688026 (+0.03582901350209289) | > avg_grad_norm_0: tensor(16.1382, device='cuda:0') (+tensor(0.9916, device='cuda:0')) | > avg_loss_gen: 1.86702698888913 (+0.010261167942638094) | > avg_loss_kl: 2.2574013924934495 (+0.015369025754257581) | > avg_loss_feat: 2.677622071454223 (-0.03606339407638748) | > avg_loss_mel: 18.019746914715835 (+0.3047603016168381) | > avg_loss_duration: 1.6032112081285934 (-0.007282351104306528) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.425008693211513 (+0.28704479378713543) | > avg_grad_norm_1: tensor(363.6257, device='cuda:0') (tensor(-63.7271, device='cuda:0'))  > EPOCH: 345/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:44:47)   --> TIME: 2025-02-25 11:44:52 -- STEP: 9/72 -- GLOBAL_STEP: 1024850 | > loss_disc: 2.7070631980895996 (2.7560438050164118) | > loss_disc_real_0: 0.18525445461273193 (0.19849326544337803) | > loss_disc_real_1: 0.214814692735672 (0.22601781454351214) | > loss_disc_real_2: 0.2243269383907318 (0.2331209811899397) | > loss_disc_real_3: 0.2410653680562973 (0.24071992437044779) | > loss_disc_real_4: 0.2721828520298004 (0.25028179585933685) | > loss_disc_real_5: 0.2496299296617508 (0.2528854724433687) | > loss_0: 2.7070631980895996 (2.7560438050164118) | > grad_norm_0: tensor(11.3593, device='cuda:0') (tensor(12.7614, device='cuda:0')) | > loss_gen: 1.8176913261413574 (1.8733042743470933) | > loss_kl: 2.4226579666137695 (2.2023905648125544) | > loss_feat: 2.813821315765381 (2.7266618145836725) | > loss_mel: 18.718708038330078 (18.187036726209854) | > loss_duration: 1.607080340385437 (1.5806823836432562) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.379959106445312 (26.57007556491428) | > grad_norm_1: tensor(74.6756, device='cuda:0') (tensor(348.9991, device='cuda:0')) | > current_lr_0: 0.00019155781537342616 | > current_lr_1: 0.00019155781537342616 | > step_time: 0.4332 (0.4372092088063558) | > loader_time: 0.0079 (0.006639215681287978)  --> TIME: 2025-02-25 11:45:16 -- STEP: 59/72 -- GLOBAL_STEP: 1024900 | > loss_disc: 2.7623491287231445 (2.7655531713518045) | > loss_disc_real_0: 0.19535663723945618 (0.20399101457353366) | > loss_disc_real_1: 0.1970084011554718 (0.23085459410134007) | > loss_disc_real_2: 0.21969138085842133 (0.2373375667858932) | > loss_disc_real_3: 0.26611441373825073 (0.24086813805466992) | > loss_disc_real_4: 0.2687889337539673 (0.2449895590543747) | > loss_disc_real_5: 0.3050488531589508 (0.2479406519461486) | > loss_0: 2.7623491287231445 (2.7655531713518045) | > grad_norm_0: tensor(18.7926, device='cuda:0') (tensor(14.0378, device='cuda:0')) | > loss_gen: 1.8647136688232422 (1.8704628823167186) | > loss_kl: 2.2376232147216797 (2.277882877042739) | > loss_feat: 2.7738494873046875 (2.6988378419714447) | > loss_mel: 17.398651123046875 (17.969343993623387) | > loss_duration: 1.6281423568725586 (1.6051760851326635) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.90298080444336 (26.421703564918648) | > grad_norm_1: tensor(800.8626, device='cuda:0') (tensor(429.2485, device='cuda:0')) | > current_lr_0: 0.00019155781537342616 | > current_lr_1: 0.00019155781537342616 | > step_time: 0.4779 (0.4487969188366906) | > loader_time: 0.0102 (0.008175009388034626) --> EVAL PERFORMANCE | > avg_loader_time: 0.008801883375140984 (+0.00014145945159482406) | > avg_step_time: 0.45953293249640664 (+0.004975245032511955) | > avg_loss_disc: 2.7709781989245346 (-0.03366484776349132) | > avg_loss_disc_real_0: 0.20627066576984568 (-0.007114688065690045) | > avg_loss_disc_real_1: 0.23191241479255784 (-0.00461356131963328) | > avg_loss_disc_real_2: 0.23743835133565983 (-0.0072679712738789315) | > avg_loss_disc_real_3: 0.2402940852121568 (-0.0048246608233787625) | > avg_loss_disc_real_4: 0.24418816180296346 (-2.553891128218022e-05) | > avg_loss_disc_real_5: 0.24808193638291157 (+0.0019261003799841414) | > avg_loss_0: 2.7709781989245346 (-0.03366484776349132) | > avg_grad_norm_0: tensor(14.1911, device='cuda:0') (tensor(-1.9471, device='cuda:0')) | > avg_loss_gen: 1.8633326090557474 (-0.0036943798333826106) | > avg_loss_kl: 2.277852029867576 (+0.020450637374126668) | > avg_loss_feat: 2.6817017239584042 (+0.004079652504181297) | > avg_loss_mel: 17.907403556393913 (-0.11234335832192244) | > avg_loss_duration: 1.612208492319349 (+0.008997284190755517) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.342498349471832 (-0.08251034373968125) | > avg_grad_norm_1: tensor(444.7371, device='cuda:0') (+tensor(81.1114, device='cuda:0'))  > EPOCH: 346/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:45:24)   --> TIME: 2025-02-25 11:45:43 -- STEP: 37/72 -- GLOBAL_STEP: 1024950 | > loss_disc: 2.93411922454834 (2.7788318041208626) | > loss_disc_real_0: 0.11396066844463348 (0.2067531047237886) | > loss_disc_real_1: 0.24959206581115723 (0.23343611649564794) | > loss_disc_real_2: 0.31365275382995605 (0.24334995408315915) | > loss_disc_real_3: 0.2883811593055725 (0.2426179695773769) | > loss_disc_real_4: 0.2654661238193512 (0.24699411400266597) | > loss_disc_real_5: 0.24426256120204926 (0.24789150180043396) | > loss_0: 2.93411922454834 (2.7788318041208626) | > grad_norm_0: tensor(19.4603, device='cuda:0') (tensor(12.9318, device='cuda:0')) | > loss_gen: 1.7978562116622925 (1.8508650644405469) | > loss_kl: 2.1763954162597656 (2.2132512975383447) | > loss_feat: 2.690103769302368 (2.722386108862387) | > loss_mel: 18.681900024414062 (17.948971361727327) | > loss_duration: 1.621951937675476 (1.583604471103565) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.96820640563965 (26.319078239234717) | > grad_norm_1: tensor(668.5732, device='cuda:0') (tensor(383.0865, device='cuda:0')) | > current_lr_0: 0.0001915338706465045 | > current_lr_1: 0.0001915338706465045 | > step_time: 0.4531 (0.44129860723340836) | > loader_time: 0.0082 (0.007246616724375132) --> EVAL PERFORMANCE | > avg_loader_time: 0.0085970079395133 (-0.00020487543562768343) | > avg_step_time: 0.4576895069068586 (-0.0018434255895480312) | > avg_loss_disc: 2.7660064730845715 (-0.0049717258399630815) | > avg_loss_disc_real_0: 0.2013726727643483 (-0.00489799300549737) | > avg_loss_disc_real_1: 0.2330339681094801 (+0.0011215533169222636) | > avg_loss_disc_real_2: 0.24096389723495698 (+0.0035255458992971556) | > avg_loss_disc_real_3: 0.24068227809079937 (+0.000388192878642557) | > avg_loss_disc_real_4: 0.24517644941806793 (+0.0009882876151044717) | > avg_loss_disc_real_5: 0.2465147124209874 (-0.001567223961924158) | > avg_loss_0: 2.7660064730845715 (-0.0049717258399630815) | > avg_grad_norm_0: tensor(11.9290, device='cuda:0') (tensor(-2.2621, device='cuda:0')) | > avg_loss_gen: 1.8498404613682922 (-0.013492147687455214) | > avg_loss_kl: 2.254337869899374 (-0.023514159968202275) | > avg_loss_feat: 2.6810148064519317 (-0.0006869175064725397) | > avg_loss_mel: 17.75788917004222 (-0.14951438635169367) | > avg_loss_duration: 1.6118932626616786 (-0.00031522965767027955) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.1549755687445 (-0.1875227807273312) | > avg_grad_norm_1: tensor(432.6490, device='cuda:0') (tensor(-12.0880, device='cuda:0'))  > EPOCH: 347/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:46:01)   --> TIME: 2025-02-25 11:46:10 -- STEP: 15/72 -- GLOBAL_STEP: 1025000 | > loss_disc: 2.764984130859375 (2.7679739316304524) | > loss_disc_real_0: 0.1589186191558838 (0.20148492157459258) | > loss_disc_real_1: 0.21710994839668274 (0.2298071672519048) | > loss_disc_real_2: 0.2515468895435333 (0.24514694114526112) | > loss_disc_real_3: 0.2569648027420044 (0.24167538384596507) | > loss_disc_real_4: 0.2551281750202179 (0.2484677533308665) | > loss_disc_real_5: 0.24716982245445251 (0.25209065973758704) | > loss_0: 2.764984130859375 (2.7679739316304524) | > grad_norm_0: tensor(11.4505, device='cuda:0') (tensor(12.4816, device='cuda:0')) | > loss_gen: 1.85492742061615 (1.8579243739446005) | > loss_kl: 2.2706596851348877 (2.1449902296066283) | > loss_feat: 2.518234968185425 (2.688746388753255) | > loss_mel: 16.808534622192383 (17.343936665852862) | > loss_duration: 1.5363059043884277 (1.5568994601567587) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.988662719726562 (25.59249750773112) | > grad_norm_1: tensor(293.5550, device='cuda:0') (tensor(509.7667, device='cuda:0')) | > current_lr_0: 0.00019150992891267367 | > current_lr_1: 0.00019150992891267367 | > step_time: 0.4269 (0.4387556234995524) | > loader_time: 0.006 (0.006807883580525716) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1025000.pth  --> TIME: 2025-02-25 11:46:37 -- STEP: 65/72 -- GLOBAL_STEP: 1025050 | > loss_disc: 2.7581100463867188 (2.775465811215914) | > loss_disc_real_0: 0.1779930293560028 (0.20185008243872568) | > loss_disc_real_1: 0.21319803595542908 (0.23675613953517033) | > loss_disc_real_2: 0.21913276612758636 (0.24017551174530616) | > loss_disc_real_3: 0.24429520964622498 (0.2389082459303049) | > loss_disc_real_4: 0.24440701305866241 (0.24356364218088297) | > loss_disc_real_5: 0.252983421087265 (0.2476929905322882) | > loss_0: 2.7581100463867188 (2.775465811215914) | > grad_norm_0: tensor(6.9801, device='cuda:0') (tensor(13.5236, device='cuda:0')) | > loss_gen: 1.8259128332138062 (1.8475525874357956) | > loss_kl: 2.351064443588257 (2.1876371273627657) | > loss_feat: 2.8544747829437256 (2.705815821427566) | > loss_mel: 18.28821563720703 (17.555562034020053) | > loss_duration: 1.645421028137207 (1.5997818653400129) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.965087890625 (25.896349628155047) | > grad_norm_1: tensor(709.8112, device='cuda:0') (tensor(505.1755, device='cuda:0')) | > current_lr_0: 0.00019150992891267367 | > current_lr_1: 0.00019150992891267367 | > step_time: 0.5093 (0.4608872230236347) | > loader_time: 0.0117 (0.008309423006497894) --> EVAL PERFORMANCE | > avg_loader_time: 0.008583629635018358 (-1.3378304494942803e-05) | > avg_step_time: 0.4665436173828555 (+0.00885411047599688) | > avg_loss_disc: 2.771637668072338 (+0.005631194987766541) | > avg_loss_disc_real_0: 0.20143818131215135 (+6.550854780304483e-05) | > avg_loss_disc_real_1: 0.23644881265264162 (+0.003414844543161516) | > avg_loss_disc_real_2: 0.23958892322761913 (-0.0013749740073378536) | > avg_loss_disc_real_3: 0.23847731486172744 (-0.0022049632290719323) | > avg_loss_disc_real_4: 0.2426257162866458 (-0.0025507331314221227) | > avg_loss_disc_real_5: 0.2463575026518862 (-0.00015720976910121864) | > avg_loss_0: 2.771637668072338 (+0.005631194987766541) | > avg_grad_norm_0: tensor(13.5230, device='cuda:0') (+tensor(1.5940, device='cuda:0')) | > avg_loss_gen: 1.8451575393408117 (-0.004682922027480441) | > avg_loss_kl: 2.195874036197932 (-0.058463833701441814) | > avg_loss_feat: 2.6914232112991985 (+0.010408404847266794) | > avg_loss_mel: 17.516841995883993 (-0.24104717415822563) | > avg_loss_duration: 1.6051449255204537 (-0.006748337141224914) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.85444184424172 (-0.30053372450278104) | > avg_grad_norm_1: tensor(508.4277, device='cuda:0') (+tensor(75.7787, device='cuda:0'))  > EPOCH: 348/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:46:42)   --> TIME: 2025-02-25 11:47:03 -- STEP: 43/72 -- GLOBAL_STEP: 1025100 | > loss_disc: 2.800535202026367 (2.768302180046258) | > loss_disc_real_0: 0.3162044286727905 (0.20915108520624248) | > loss_disc_real_1: 0.24203619360923767 (0.23901623106280037) | > loss_disc_real_2: 0.25126662850379944 (0.2377929645915364) | > loss_disc_real_3: 0.25208988785743713 (0.23781627693841623) | > loss_disc_real_4: 0.2577507495880127 (0.24283132026361864) | > loss_disc_real_5: 0.2507037818431854 (0.24590577843577363) | > loss_0: 2.800535202026367 (2.768302180046258) | > grad_norm_0: tensor(21.5992, device='cuda:0') (tensor(15.1681, device='cuda:0')) | > loss_gen: 1.8285861015319824 (1.868117656818656) | > loss_kl: 2.260808229446411 (2.2244794923205706) | > loss_feat: 2.14560866355896 (2.7676590098891145) | > loss_mel: 15.602537155151367 (17.779817581176758) | > loss_duration: 1.6117992401123047 (1.5807699380918991) | > amp_scaler: 128.0 (128.0) | > loss_1: 23.449338912963867 (26.220843692158542) | > grad_norm_1: tensor(748.4953, device='cuda:0') (tensor(439.7516, device='cuda:0')) | > current_lr_0: 0.00019148599017155957 | > current_lr_1: 0.00019148599017155957 | > step_time: 0.4565 (0.44391100351200546) | > loader_time: 0.0094 (0.007686154786930527) --> EVAL PERFORMANCE | > avg_loader_time: 0.008709830297550689 (+0.000126200662532331) | > avg_step_time: 0.45758176185715366 (-0.008961855525701834) | > avg_loss_disc: 2.774765078450592 (+0.0031274103782541474) | > avg_loss_disc_real_0: 0.20599916755733355 (+0.004560986245182197) | > avg_loss_disc_real_1: 0.23591407965606367 (-0.0005347329965779413) | > avg_loss_disc_real_2: 0.2377788890415514 (-0.0018100341860677327) | > avg_loss_disc_real_3: 0.2394911915483609 (+0.0010138766866334692) | > avg_loss_disc_real_4: 0.24451514595830945 (+0.0018894296716636394) | > avg_loss_disc_real_5: 0.2479027742651147 (+0.0015452716132285105) | > avg_loss_0: 2.774765078450592 (+0.0031274103782541474) | > avg_grad_norm_0: tensor(15.8001, device='cuda:0') (+tensor(2.2771, device='cuda:0')) | > avg_loss_gen: 1.852243720645636 (+0.007086181304824191) | > avg_loss_kl: 2.248168874794328 (+0.05229483859639572) | > avg_loss_feat: 2.684738432857352 (-0.006684778441846451) | > avg_loss_mel: 17.65612702974132 (+0.13928503385732682) | > avg_loss_duration: 1.6036757381869033 (-0.0014691873335503924) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.044953870101715 (+0.19051202585999505) | > avg_grad_norm_1: tensor(508.7035, device='cuda:0') (+tensor(0.2758, device='cuda:0'))  > EPOCH: 349/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:47:19)   --> TIME: 2025-02-25 11:47:30 -- STEP: 21/72 -- GLOBAL_STEP: 1025150 | > loss_disc: 2.886080265045166 (2.7639276640755788) | > loss_disc_real_0: 0.14588671922683716 (0.19169083876269205) | > loss_disc_real_1: 0.2662756145000458 (0.23274502087207066) | > loss_disc_real_2: 0.238755464553833 (0.23763399322827658) | > loss_disc_real_3: 0.2675183415412903 (0.239035821386746) | > loss_disc_real_4: 0.23964938521385193 (0.24470174596423194) | > loss_disc_real_5: 0.24123793840408325 (0.25254848812307623) | > loss_0: 2.886080265045166 (2.7639276640755788) | > grad_norm_0: tensor(18.9512, device='cuda:0') (tensor(12.9949, device='cuda:0')) | > loss_gen: 1.8350542783737183 (1.8504422846294584) | > loss_kl: 2.1477012634277344 (2.138088209288461) | > loss_feat: 2.677654981613159 (2.8110911505562917) | > loss_mel: 18.343521118164062 (17.745227904546834) | > loss_duration: 1.5991573333740234 (1.5615652061644054) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.60308837890625 (26.106415067400253) | > grad_norm_1: tensor(403.6653, device='cuda:0') (tensor(525.3699, device='cuda:0')) | > current_lr_0: 0.00019146205442278813 | > current_lr_1: 0.00019146205442278813 | > step_time: 0.4343 (0.43712611425490605) | > loader_time: 0.0065 (0.00710478283110119)  --> TIME: 2025-02-25 11:47:54 -- STEP: 71/72 -- GLOBAL_STEP: 1025200 | > loss_disc: 2.741666316986084 (2.7849939406757622) | > loss_disc_real_0: 0.2680114209651947 (0.20498877448934907) | > loss_disc_real_1: 0.21226151287555695 (0.23100235281695783) | > loss_disc_real_2: 0.23504109680652618 (0.23720441926533067) | > loss_disc_real_3: 0.22451072931289673 (0.23939376565772044) | > loss_disc_real_4: 0.22516106069087982 (0.24275572870818662) | > loss_disc_real_5: 0.24203574657440186 (0.2457518031899358) | > loss_0: 2.741666316986084 (2.7849939406757622) | > grad_norm_0: tensor(12.6948, device='cuda:0') (tensor(15.6117, device='cuda:0')) | > loss_gen: 1.8021882772445679 (1.8246293722743718) | > loss_kl: 2.1266026496887207 (2.230839341459141) | > loss_feat: 2.7758395671844482 (2.690024647914188) | > loss_mel: 17.035934448242188 (17.822132513556685) | > loss_duration: 1.6242890357971191 (1.6048740299654678) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.36485481262207 (26.172499911885865) | > grad_norm_1: tensor(354.0565, device='cuda:0') (tensor(415.2769, device='cuda:0')) | > current_lr_0: 0.00019146205442278813 | > current_lr_1: 0.00019146205442278813 | > step_time: 0.4676 (0.45464913610001684) | > loader_time: 0.0078 (0.008744777088433924) --> EVAL PERFORMANCE | > avg_loader_time: 0.008744777088433924 (+3.494679088323488e-05) | > avg_step_time: 0.45464913610001684 (-0.0029326257571368197) | > avg_loss_disc: 2.7849939406757622 (+0.010228862225170055) | > avg_loss_disc_real_0: 0.20498877448934907 (-0.0010103930679844775) | > avg_loss_disc_real_1: 0.23100235281695783 (-0.0049117268391058455) | > avg_loss_disc_real_2: 0.23720441926533067 (-0.0005744697762207207) | > avg_loss_disc_real_3: 0.23939376565772044 (-9.742589064046236e-05) | > avg_loss_disc_real_4: 0.24275572870818662 (-0.0017594172501228245) | > avg_loss_disc_real_5: 0.2457518031899358 (-0.0021509710751788924) | > avg_loss_0: 2.7849939406757622 (+0.010228862225170055) | > avg_grad_norm_0: tensor(15.6117, device='cuda:0') (tensor(-0.1883, device='cuda:0')) | > avg_loss_gen: 1.8246293722743718 (-0.027614348371264086) | > avg_loss_kl: 2.230839341459141 (-0.01732953333518683) | > avg_loss_feat: 2.690024647914188 (+0.005286215056835797) | > avg_loss_mel: 17.822132513556685 (+0.16600548381536484) | > avg_loss_duration: 1.6048740299654678 (+0.0011982917785644531) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.172499911885865 (+0.12754604178415008) | > avg_grad_norm_1: tensor(415.2769, device='cuda:0') (tensor(-93.4265, device='cuda:0'))  > EPOCH: 350/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:47:56)   --> TIME: 2025-02-25 11:48:20 -- STEP: 49/72 -- GLOBAL_STEP: 1025250 | > loss_disc: 2.7200849056243896 (2.7827477795737123) | > loss_disc_real_0: 0.21681419014930725 (0.20568480783579302) | > loss_disc_real_1: 0.15355060994625092 (0.22988879254886083) | > loss_disc_real_2: 0.19971448183059692 (0.23862077539064447) | > loss_disc_real_3: 0.2047065645456314 (0.24084161100338916) | > loss_disc_real_4: 0.21916046738624573 (0.24603593805614782) | > loss_disc_real_5: 0.23065365850925446 (0.24958839009002776) | > loss_0: 2.7200849056243896 (2.7827477795737123) | > grad_norm_0: tensor(8.3312, device='cuda:0') (tensor(12.7242, device='cuda:0')) | > loss_gen: 1.8593544960021973 (1.8516137672930348) | > loss_kl: 2.134587287902832 (2.2635526048893824) | > loss_feat: 2.7364702224731445 (2.6911239088798067) | > loss_mel: 15.927244186401367 (17.544268705406967) | > loss_duration: 1.661836862564087 (1.5893101570557575) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.31949234008789 (25.9398690437784) | > grad_norm_1: tensor(107.6319, device='cuda:0') (tensor(382.7550, device='cuda:0')) | > current_lr_0: 0.00019143812166598528 | > current_lr_1: 0.00019143812166598528 | > step_time: 0.44 (0.4416072125337562) | > loader_time: 0.0093 (0.007741134993884028) --> EVAL PERFORMANCE | > avg_loader_time: 0.008645769575951805 (-9.900751248211916e-05) | > avg_step_time: 0.4567191567219479 (+0.002070020621931068) | > avg_loss_disc: 2.7872699683820694 (+0.002276027706307193) | > avg_loss_disc_real_0: 0.21320801091865754 (+0.008219236429308469) | > avg_loss_disc_real_1: 0.2309160931429393 (-8.625967401854062e-05) | > avg_loss_disc_real_2: 0.23852521978633504 (+0.0013208005210043616) | > avg_loss_disc_real_3: 0.23920214868767162 (-0.0001916169700488246) | > avg_loss_disc_real_4: 0.24429588838362357 (+0.0015401596754369418) | > avg_loss_disc_real_5: 0.2472994990751778 (+0.0015476958852419787) | > avg_loss_0: 2.7872699683820694 (+0.002276027706307193) | > avg_grad_norm_0: tensor(13.9653, device='cuda:0') (tensor(-1.6465, device='cuda:0')) | > avg_loss_gen: 1.8483382379504996 (+0.023708865676127733) | > avg_loss_kl: 2.266046895107752 (+0.03520755364861117) | > avg_loss_feat: 2.6694239629826084 (-0.020600684931579405) | > avg_loss_mel: 17.456705375456472 (-0.36542713810021255) | > avg_loss_duration: 1.6032326389366471 (-0.0016413910288206601) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.843747125545015 (-0.3287527863408499) | > avg_grad_norm_1: tensor(397.3721, device='cuda:0') (tensor(-17.9048, device='cuda:0'))  > EPOCH: 351/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:48:33)   --> TIME: 2025-02-25 11:48:47 -- STEP: 27/72 -- GLOBAL_STEP: 1025300 | > loss_disc: 2.7707600593566895 (2.790854842574508) | > loss_disc_real_0: 0.17257331311702728 (0.21450129786023386) | > loss_disc_real_1: 0.21312548220157623 (0.23232084347142112) | > loss_disc_real_2: 0.2595277726650238 (0.24745729234483507) | > loss_disc_real_3: 0.2495763897895813 (0.24843071235550773) | > loss_disc_real_4: 0.2682707905769348 (0.2471251079329738) | > loss_disc_real_5: 0.2574615776538849 (0.24889481840310274) | > loss_0: 2.7707600593566895 (2.790854842574508) | > grad_norm_0: tensor(7.3598, device='cuda:0') (tensor(15.2745, device='cuda:0')) | > loss_gen: 1.791304111480713 (1.867902980910407) | > loss_kl: 2.3700525760650635 (2.1361637027175324) | > loss_feat: 2.6221189498901367 (2.7195568879445395) | > loss_mel: 17.895442962646484 (17.568791495429146) | > loss_duration: 1.5569908618927002 (1.5560275801905876) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.235910415649414 (25.848442713419598) | > grad_norm_1: tensor(97.6829, device='cuda:0') (tensor(441.7404, device='cuda:0')) | > current_lr_0: 0.00019141419190077703 | > current_lr_1: 0.00019141419190077703 | > step_time: 0.4563 (0.4581553847701461) | > loader_time: 0.0084 (0.007147577073838975) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737238360122896 (+9.146878417109171e-05) | > avg_step_time: 0.4667842858274218 (+0.010065129105473913) | > avg_loss_disc: 2.7974621477261397 (+0.01019217934407024) | > avg_loss_disc_real_0: 0.21662290604181692 (+0.0034148951231593727) | > avg_loss_disc_real_1: 0.234243856139586 (+0.0033277629966467215) | > avg_loss_disc_real_2: 0.23968411961071928 (+0.0011588998243842463) | > avg_loss_disc_real_3: 0.24523325180503683 (+0.006031103117365211) | > avg_loss_disc_real_4: 0.24514492800537968 (+0.0008490396217561147) | > avg_loss_disc_real_5: 0.2465753672828137 (-0.00072413179236408) | > avg_loss_0: 2.7974621477261397 (+0.01019217934407024) | > avg_grad_norm_0: tensor(16.3898, device='cuda:0') (+tensor(2.4245, device='cuda:0')) | > avg_loss_gen: 1.8485950298712288 (+0.00025679192072924906) | > avg_loss_kl: 2.260382742948936 (-0.005664152158816105) | > avg_loss_feat: 2.670405854641552 (+0.0009818916589434679) | > avg_loss_mel: 17.606981868475263 (+0.1502764930187901) | > avg_loss_duration: 1.6000209979608024 (-0.0032116409758446984) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.986386634934117 (+0.14263950938910241) | > avg_grad_norm_1: tensor(402.6792, device='cuda:0') (+tensor(5.3071, device='cuda:0'))  > EPOCH: 352/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:49:10)   --> TIME: 2025-02-25 11:49:15 -- STEP: 5/72 -- GLOBAL_STEP: 1025350 | > loss_disc: 2.835960626602173 (2.820356512069702) | > loss_disc_real_0: 0.27884674072265625 (0.1995557576417923) | > loss_disc_real_1: 0.226924866437912 (0.23939984142780305) | > loss_disc_real_2: 0.2331005185842514 (0.2353278249502182) | > loss_disc_real_3: 0.25792306661605835 (0.24293369054794312) | > loss_disc_real_4: 0.24211639165878296 (0.25630994141101837) | > loss_disc_real_5: 0.2579580545425415 (0.2612315833568573) | > loss_0: 2.835960626602173 (2.820356512069702) | > grad_norm_0: tensor(16.6841, device='cuda:0') (tensor(10.0969, device='cuda:0')) | > loss_gen: 1.8311808109283447 (1.841788911819458) | > loss_kl: 2.441474437713623 (2.3130854606628417) | > loss_feat: 2.3609488010406494 (2.5973558902740477) | > loss_mel: 17.31635093688965 (17.963545989990234) | > loss_duration: 1.538155198097229 (1.5840752601623536) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.488109588623047 (26.299850845336913) | > grad_norm_1: tensor(155.0645, device='cuda:0') (tensor(163.5070, device='cuda:0')) | > current_lr_0: 0.00019139026512678941 | > current_lr_1: 0.00019139026512678941 | > step_time: 0.4357 (0.4395118713378906) | > loader_time: 0.0065 (0.00710911750793457)  --> TIME: 2025-02-25 11:49:38 -- STEP: 55/72 -- GLOBAL_STEP: 1025400 | > loss_disc: 2.773514747619629 (2.7784766413948754) | > loss_disc_real_0: 0.14558735489845276 (0.20170108879154378) | > loss_disc_real_1: 0.1992027908563614 (0.23058463335037233) | > loss_disc_real_2: 0.25388476252555847 (0.2394357583739541) | > loss_disc_real_3: 0.25158172845840454 (0.24071211841973392) | > loss_disc_real_4: 0.2494508922100067 (0.2427299361337315) | > loss_disc_real_5: 0.25036829710006714 (0.24610438671979037) | > loss_0: 2.773514747619629 (2.7784766413948754) | > grad_norm_0: tensor(11.9842, device='cuda:0') (tensor(13.4703, device='cuda:0')) | > loss_gen: 1.863765001296997 (1.845153284072876) | > loss_kl: 2.38508939743042 (2.2088685772635723) | > loss_feat: 2.9402873516082764 (2.729520173506303) | > loss_mel: 16.54863166809082 (17.54761907404119) | > loss_duration: 1.6346906423568726 (1.5870021473277698) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.372465133666992 (25.91816336891868) | > grad_norm_1: tensor(818.6760, device='cuda:0') (tensor(486.9520, device='cuda:0')) | > current_lr_0: 0.00019139026512678941 | > current_lr_1: 0.00019139026512678941 | > step_time: 0.4685 (0.44951164505698465) | > loader_time: 0.0093 (0.008040350133722477) --> EVAL PERFORMANCE | > avg_loader_time: 0.008724269732623033 (-1.2968627499863189e-05) | > avg_step_time: 0.46088901372023033 (-0.005895272107191485) | > avg_loss_disc: 2.7792974357873623 (-0.018164711938777334) | > avg_loss_disc_real_0: 0.20460113998450025 (-0.012021766057316668) | > avg_loss_disc_real_1: 0.23234231644113298 (-0.0019015396984530253) | > avg_loss_disc_real_2: 0.2379061762295978 (-0.0017779433811214718) | > avg_loss_disc_real_3: 0.24081542836108677 (-0.0044178234439500585) | > avg_loss_disc_real_4: 0.241257658936608 (-0.0038872690687716815) | > avg_loss_disc_real_5: 0.24451530231556423 (-0.0020600649672494753) | > avg_loss_0: 2.7792974357873623 (-0.018164711938777334) | > avg_grad_norm_0: tensor(13.6024, device='cuda:0') (tensor(-2.7874, device='cuda:0')) | > avg_loss_gen: 1.8368257049103858 (-0.011769324960843042) | > avg_loss_kl: 2.24271372338416 (-0.01766901956477618) | > avg_loss_feat: 2.689047763045405 (+0.018641908403853158) | > avg_loss_mel: 17.638758538474494 (+0.0317766699992319) | > avg_loss_duration: 1.6015887747348194 (+0.0015677767740169823) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.00893466573366 (+0.022548030799541152) | > avg_grad_norm_1: tensor(487.6574, device='cuda:0') (+tensor(84.9783, device='cuda:0'))  > EPOCH: 353/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:49:48)   --> TIME: 2025-02-25 11:50:05 -- STEP: 33/72 -- GLOBAL_STEP: 1025450 | > loss_disc: 2.6956262588500977 (2.7676276582660098) | > loss_disc_real_0: 0.11527267098426819 (0.19426130006710687) | > loss_disc_real_1: 0.1996278017759323 (0.22879047059651578) | > loss_disc_real_2: 0.2743041217327118 (0.24312937846689514) | > loss_disc_real_3: 0.2204320877790451 (0.2417469507817066) | > loss_disc_real_4: 0.2466987818479538 (0.24457886137745596) | > loss_disc_real_5: 0.27875253558158875 (0.2472390609257149) | > loss_0: 2.6956262588500977 (2.7676276582660098) | > grad_norm_0: tensor(21.6671, device='cuda:0') (tensor(12.8245, device='cuda:0')) | > loss_gen: 1.8930857181549072 (1.8802990624398896) | > loss_kl: 2.462252378463745 (2.3651462468233975) | > loss_feat: 3.007262706756592 (2.8587869875358813) | > loss_mel: 18.18356704711914 (18.523083195541837) | > loss_duration: 1.589948058128357 (1.5945415352330063) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.13611602783203 (27.221856955325965) | > grad_norm_1: tensor(447.6099, device='cuda:0') (tensor(497.7155, device='cuda:0')) | > current_lr_0: 0.00019136634134364855 | > current_lr_1: 0.00019136634134364855 | > step_time: 0.4415 (0.4373770627108487) | > loader_time: 0.0083 (0.007060787894509055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596161721457895 (-0.0001281080111651383) | > avg_step_time: 0.45338296554457974 (-0.007506048175650593) | > avg_loss_disc: 2.7789983212108345 (-0.0002991145765278702) | > avg_loss_disc_real_0: 0.20730205950602676 (+0.0027009195215265125) | > avg_loss_disc_real_1: 0.23168871604220967 (-0.0006536003989233152) | > avg_loss_disc_real_2: 0.23960967340939482 (+0.0017034971797970089) | > avg_loss_disc_real_3: 0.23985604270243308 (-0.0009593856586536942) | > avg_loss_disc_real_4: 0.24582119537910946 (+0.004563536442501459) | > avg_loss_disc_real_5: 0.24707594618830883 (+0.0025606438727446) | > avg_loss_0: 2.7789983212108345 (-0.0002991145765278702) | > avg_grad_norm_0: tensor(16.2466, device='cuda:0') (+tensor(2.6442, device='cuda:0')) | > avg_loss_gen: 1.8723305295890487 (+0.035504824678662894) | > avg_loss_kl: 2.313845154265283 (+0.07113143088112306) | > avg_loss_feat: 2.718025180655467 (+0.028977417610061895) | > avg_loss_mel: 18.0380421759377 (+0.3992836374632063) | > avg_loss_duration: 1.6144052357740806 (+0.012816461039261151) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.556648281258596 (+0.5477136155249376) | > avg_grad_norm_1: tensor(501.4431, device='cuda:0') (+tensor(13.7856, device='cuda:0'))  > EPOCH: 354/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:50:24)   --> TIME: 2025-02-25 11:50:31 -- STEP: 11/72 -- GLOBAL_STEP: 1025500 | > loss_disc: 2.8036465644836426 (2.73512263731523) | > loss_disc_real_0: 0.2669092118740082 (0.2020798298445615) | > loss_disc_real_1: 0.22617009282112122 (0.23295999250628732) | > loss_disc_real_2: 0.2257908135652542 (0.22834805331446908) | > loss_disc_real_3: 0.2428111433982849 (0.23277921161868356) | > loss_disc_real_4: 0.24749615788459778 (0.2373525689948689) | > loss_disc_real_5: 0.24881157279014587 (0.24679123000665146) | > loss_0: 2.8036465644836426 (2.73512263731523) | > grad_norm_0: tensor(19.6802, device='cuda:0') (tensor(19.3472, device='cuda:0')) | > loss_gen: 1.7174341678619385 (1.842908501625061) | > loss_kl: 2.0947375297546387 (2.061773722822016) | > loss_feat: 2.5858609676361084 (2.7228761586275967) | > loss_mel: 16.985614776611328 (17.55541532689875) | > loss_duration: 1.56109619140625 (1.5611134767532349) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.944744110107422 (25.744087392633613) | > grad_norm_1: tensor(258.6945, device='cuda:0') (tensor(594.8587, device='cuda:0')) | > current_lr_0: 0.0001913424205509806 | > current_lr_1: 0.0001913424205509806 | > step_time: 0.4432 (0.4322984435341575) | > loader_time: 0.0056 (0.006510279395363547)  --> TIME: 2025-02-25 11:50:55 -- STEP: 61/72 -- GLOBAL_STEP: 1025550 | > loss_disc: 2.7437186241149902 (2.7812622140665515) | > loss_disc_real_0: 0.21751666069030762 (0.20545525177091847) | > loss_disc_real_1: 0.19419419765472412 (0.23234088108187817) | > loss_disc_real_2: 0.2749776542186737 (0.23682701416680071) | > loss_disc_real_3: 0.22991733253002167 (0.2365366880033837) | > loss_disc_real_4: 0.231603741645813 (0.24515687149079118) | > loss_disc_real_5: 0.22624844312667847 (0.24663391929180895) | > loss_0: 2.7437186241149902 (2.7812622140665515) | > grad_norm_0: tensor(14.5731, device='cuda:0') (tensor(14.5126, device='cuda:0')) | > loss_gen: 1.8902522325515747 (1.8332588008192718) | > loss_kl: 2.321031093597412 (2.2238910178669165) | > loss_feat: 2.707463264465332 (2.6672823390022655) | > loss_mel: 18.11626625061035 (17.797331497317455) | > loss_duration: 1.6499816179275513 (1.5994879984464803) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.684995651245117 (26.121251841060452) | > grad_norm_1: tensor(792.0179, device='cuda:0') (tensor(400.5428, device='cuda:0')) | > current_lr_0: 0.0001913424205509806 | > current_lr_1: 0.0001913424205509806 | > step_time: 0.4613 (0.44870843652819026) | > loader_time: 0.0107 (0.008272413347588206) --> EVAL PERFORMANCE | > avg_loader_time: 0.008736479450279553 (+0.00014031772882165845) | > avg_step_time: 0.45573522675205286 (+0.0023522612074731164) | > avg_loss_disc: 2.7786584169092303 (-0.00033990430160413254) | > avg_loss_disc_real_0: 0.2055302114343979 (-0.001771848071628862) | > avg_loss_disc_real_1: 0.23185600990980443 (+0.00016729386759475884) | > avg_loss_disc_real_2: 0.2376281287468655 (-0.001981544662529311) | > avg_loss_disc_real_3: 0.2345851932612943 (-0.005270849441138786) | > avg_loss_disc_real_4: 0.2435818138256879 (-0.002239381553421571) | > avg_loss_disc_real_5: 0.2464642730397238 (-0.0006116731485850402) | > avg_loss_0: 2.7786584169092303 (-0.00033990430160413254) | > avg_grad_norm_0: tensor(13.9636, device='cuda:0') (tensor(-2.2830, device='cuda:0')) | > avg_loss_gen: 1.829057386223699 (-0.04327314336534971) | > avg_loss_kl: 2.2287050922152023 (-0.08514006205008062) | > avg_loss_feat: 2.6387635385486443 (-0.07926164210682263) | > avg_loss_mel: 17.690946592411525 (-0.3470955835261762) | > avg_loss_duration: 1.6070457629754509 (-0.007359472798629696) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.994518548669948 (-0.5621297325886481) | > avg_grad_norm_1: tensor(407.8014, device='cuda:0') (tensor(-93.6416, device='cuda:0'))  > EPOCH: 355/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:51:02)   --> TIME: 2025-02-25 11:51:21 -- STEP: 39/72 -- GLOBAL_STEP: 1025600 | > loss_disc: 2.7521538734436035 (2.7574842832027335) | > loss_disc_real_0: 0.11302147805690765 (0.19292767040240458) | > loss_disc_real_1: 0.20117582380771637 (0.22888208123353812) | > loss_disc_real_2: 0.2332654893398285 (0.23870796461900076) | > loss_disc_real_3: 0.24064059555530548 (0.23703944912323585) | > loss_disc_real_4: 0.2204563021659851 (0.2421100296271153) | > loss_disc_real_5: 0.2382131814956665 (0.24423065781593323) | > loss_0: 2.7521538734436035 (2.7574842832027335) | > grad_norm_0: tensor(20.5808, device='cuda:0') (tensor(10.8241, device='cuda:0')) | > loss_gen: 1.796630859375 (1.8663727106192174) | > loss_kl: 2.3269643783569336 (2.3067900889959083) | > loss_feat: 2.9017765522003174 (2.8476681281358767) | > loss_mel: 17.89454460144043 (18.215616128383537) | > loss_duration: 1.5992951393127441 (1.590935419767331) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.519210815429688 (26.82738260122446) | > grad_norm_1: tensor(491.5270, device='cuda:0') (tensor(361.0310, device='cuda:0')) | > current_lr_0: 0.00019131850274841172 | > current_lr_1: 0.00019131850274841172 | > step_time: 0.4553 (0.4396691444592598) | > loader_time: 0.0082 (0.007291488158397186) --> EVAL PERFORMANCE | > avg_loader_time: 0.008533165488444584 (-0.00020331396183496941) | > avg_step_time: 0.45622309160904145 (+0.0004878648569885957) | > avg_loss_disc: 2.7712526757952194 (-0.007405741114010933) | > avg_loss_disc_real_0: 0.19810163775380227 (-0.007428573680595629) | > avg_loss_disc_real_1: 0.232313957432626 (+0.0004579475228215779) | > avg_loss_disc_real_2: 0.24013081025069868 (+0.0025026815038331762) | > avg_loss_disc_real_3: 0.2404631921942805 (+0.0058779989329862115) | > avg_loss_disc_real_4: 0.24384563447723925 (+0.00026382065155136636) | > avg_loss_disc_real_5: 0.24774686656367612 (+0.0012825935239523245) | > avg_loss_0: 2.7712526757952194 (-0.007405741114010933) | > avg_grad_norm_0: tensor(11.6544, device='cuda:0') (tensor(-2.3092, device='cuda:0')) | > avg_loss_gen: 1.8520453127337173 (+0.022987926510018353) | > avg_loss_kl: 2.3412570852628893 (+0.11255199304768704) | > avg_loss_feat: 2.71145228265037 (+0.07268874410172588) | > avg_loss_mel: 18.11922667060099 (+0.42828007818946645) | > avg_loss_duration: 1.6182826841381235 (+0.011236921162672608) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.64226413780535 (+0.6477455891354005) | > avg_grad_norm_1: tensor(390.3249, device='cuda:0') (tensor(-17.4766, device='cuda:0'))  > EPOCH: 356/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:51:39)   --> TIME: 2025-02-25 11:51:48 -- STEP: 17/72 -- GLOBAL_STEP: 1025650 | > loss_disc: 2.7540221214294434 (2.7601529009201946) | > loss_disc_real_0: 0.2175784707069397 (0.1972428569022347) | > loss_disc_real_1: 0.2493288516998291 (0.22763838838128483) | > loss_disc_real_2: 0.20802223682403564 (0.23726857672719395) | > loss_disc_real_3: 0.2167872041463852 (0.2349726750570185) | > loss_disc_real_4: 0.229505255818367 (0.24196705222129822) | > loss_disc_real_5: 0.23242627084255219 (0.24473394015256097) | > loss_0: 2.7540221214294434 (2.7601529009201946) | > grad_norm_0: tensor(12.9192, device='cuda:0') (tensor(14.3695, device='cuda:0')) | > loss_gen: 1.807093858718872 (1.8545503966948564) | > loss_kl: 2.855586528778076 (2.2423982900731705) | > loss_feat: 2.9286983013153076 (2.7810882680556355) | > loss_mel: 19.155433654785156 (18.064196081722486) | > loss_duration: 1.6249942779541016 (1.5741522873149199) | > amp_scaler: 128.0 (128.0) | > loss_1: 28.371807098388672 (26.516385246725644) | > grad_norm_1: tensor(622.5677, device='cuda:0') (tensor(404.1414, device='cuda:0')) | > current_lr_0: 0.00019129458793556815 | > current_lr_1: 0.00019129458793556815 | > step_time: 0.4382 (0.4365548526539522) | > loader_time: 0.0061 (0.006966100019567153)  --> TIME: 2025-02-25 11:52:12 -- STEP: 67/72 -- GLOBAL_STEP: 1025700 | > loss_disc: 2.8097550868988037 (2.770127517073902) | > loss_disc_real_0: 0.19117021560668945 (0.19873455756190997) | > loss_disc_real_1: 0.23251304030418396 (0.23175665958603817) | > loss_disc_real_2: 0.28717243671417236 (0.24121822794871545) | > loss_disc_real_3: 0.25176510214805603 (0.2400271165281979) | > loss_disc_real_4: 0.26068398356437683 (0.24389813729186557) | > loss_disc_real_5: 0.2624560296535492 (0.2458268895522872) | > loss_0: 2.8097550868988037 (2.770127517073902) | > grad_norm_0: tensor(17.1042, device='cuda:0') (tensor(13.6006, device='cuda:0')) | > loss_gen: 1.8119131326675415 (1.8588201412514074) | > loss_kl: 2.3129124641418457 (2.3004064310842485) | > loss_feat: 2.439412832260132 (2.764563813138364) | > loss_mel: 16.958608627319336 (17.858920125818965) | > loss_duration: 1.6723499298095703 (1.6018018544609867) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.1951961517334 (26.384512431585968) | > grad_norm_1: tensor(489.8694, device='cuda:0') (tensor(426.7100, device='cuda:0')) | > current_lr_0: 0.00019129458793556815 | > current_lr_1: 0.00019129458793556815 | > step_time: 0.4886 (0.451587940329936) | > loader_time: 0.0123 (0.00851463559848159) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688506945757799 (+0.0001553414573132149) | > avg_step_time: 0.45429232758535465 (-0.001930764023686804) | > avg_loss_disc: 2.769049432915701 (-0.002203242879518452) | > avg_loss_disc_real_0: 0.19830049857707088 (+0.0001988608232686151) | > avg_loss_disc_real_1: 0.23115079692551788 (-0.0011631605071081297) | > avg_loss_disc_real_2: 0.24137493716159336 (+0.0012441269108946773) | > avg_loss_disc_real_3: 0.2393383734243017 (-0.0011248187699788026) | > avg_loss_disc_real_4: 0.24359974810774898 (-0.00024588636949027243) | > avg_loss_disc_real_5: 0.24620257066169254 (-0.0015442959019835723) | > avg_loss_0: 2.769049432915701 (-0.002203242879518452) | > avg_grad_norm_0: tensor(13.6906, device='cuda:0') (+tensor(2.0362, device='cuda:0')) | > avg_loss_gen: 1.8543562754778795 (+0.002310962744162204) | > avg_loss_kl: 2.30218850055211 (-0.03906858471077923) | > avg_loss_feat: 2.7533830085270843 (+0.04193072587671409) | > avg_loss_mel: 17.82058981774559 (-0.2986368528554024) | > avg_loss_duration: 1.6040024992445825 (-0.014280184893540993) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.334520151917367 (-0.30774398588798135) | > avg_grad_norm_1: tensor(435.8304, device='cuda:0') (+tensor(45.5056, device='cuda:0'))  > EPOCH: 357/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:52:16)   --> TIME: 2025-02-25 11:52:38 -- STEP: 45/72 -- GLOBAL_STEP: 1025750 | > loss_disc: 2.930382251739502 (2.7433580239613855) | > loss_disc_real_0: 0.19997623562812805 (0.19599189079470106) | > loss_disc_real_1: 0.27675485610961914 (0.22481727798779805) | > loss_disc_real_2: 0.27812841534614563 (0.23876418206426833) | > loss_disc_real_3: 0.2728739380836487 (0.23501532706949446) | > loss_disc_real_4: 0.28368642926216125 (0.24314425786336263) | > loss_disc_real_5: 0.2671186923980713 (0.24365608990192414) | > loss_0: 2.930382251739502 (2.7433580239613855) | > grad_norm_0: tensor(9.6067, device='cuda:0') (tensor(14.0351, device='cuda:0')) | > loss_gen: 1.7259571552276611 (1.8641183296839396) | > loss_kl: 2.2156758308410645 (2.1891530487272464) | > loss_feat: 2.2246766090393066 (2.832660055160523) | > loss_mel: 17.68710708618164 (17.647308476765954) | > loss_duration: 1.6303589344024658 (1.5996155129538643) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.48377799987793 (26.132855309380425) | > grad_norm_1: tensor(706.9260, device='cuda:0') (tensor(557.3570, device='cuda:0')) | > current_lr_0: 0.0001912706761120762 | > current_lr_1: 0.0001912706761120762 | > step_time: 0.4612 (0.4392899619208442) | > loader_time: 0.009 (0.007532045576307509) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860968442030356 (-7.882252545423837e-05) | > avg_step_time: 0.45635500088543957 (+0.0020626733000849162) | > avg_loss_disc: 2.761232812639694 (-0.007816620276007047) | > avg_loss_disc_real_0: 0.2045591103149132 (+0.006258611737842312) | > avg_loss_disc_real_1: 0.22499558728345684 (-0.006155209642061038) | > avg_loss_disc_real_2: 0.2386851004311736 (-0.0026898367304197535) | > avg_loss_disc_real_3: 0.23886850266389445 (-0.0004698707604072483) | > avg_loss_disc_real_4: 0.24478657879460025 (+0.0011868306868512701) | > avg_loss_disc_real_5: 0.2466340616975032 (+0.00043149103581066606) | > avg_loss_0: 2.761232812639694 (-0.007816620276007047) | > avg_grad_norm_0: tensor(15.1946, device='cuda:0') (+tensor(1.5040, device='cuda:0')) | > avg_loss_gen: 1.8608852977484045 (+0.006529022270524987) | > avg_loss_kl: 2.2194723125914444 (-0.08271618796066571) | > avg_loss_feat: 2.7913216402832894 (+0.037938631756205154) | > avg_loss_mel: 17.76987301463813 (-0.050716803107459185) | > avg_loss_duration: 1.6152195275669368 (+0.011217028322354272) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.25677157119966 (-0.07774858071770652) | > avg_grad_norm_1: tensor(512.7669, device='cuda:0') (+tensor(76.9365, device='cuda:0'))  > EPOCH: 358/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:52:53)   --> TIME: 2025-02-25 11:53:05 -- STEP: 23/72 -- GLOBAL_STEP: 1025800 | > loss_disc: 2.7838449478149414 (2.7958780889925747) | > loss_disc_real_0: 0.2830111086368561 (0.19432451349237692) | > loss_disc_real_1: 0.24676208198070526 (0.24176116935584857) | > loss_disc_real_2: 0.2168525755405426 (0.24669481230818707) | > loss_disc_real_3: 0.23485900461673737 (0.24570539917634882) | > loss_disc_real_4: 0.25060930848121643 (0.2504562605982241) | > loss_disc_real_5: 0.24720732867717743 (0.25485574486462975) | > loss_0: 2.7838449478149414 (2.7958780889925747) | > grad_norm_0: tensor(14.1872, device='cuda:0') (tensor(10.2496, device='cuda:0')) | > loss_gen: 1.7902404069900513 (1.8786546821179597) | > loss_kl: 2.210033416748047 (2.298691277918608) | > loss_feat: 2.67526912689209 (2.7486808714659317) | > loss_mel: 17.59912109375 (18.19288444519043) | > loss_duration: 1.622795581817627 (1.5807411463364311) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.8974609375 (26.69965287913447) | > grad_norm_1: tensor(285.0424, device='cuda:0') (tensor(472.8467, device='cuda:0')) | > current_lr_0: 0.00019124676727756216 | > current_lr_1: 0.00019124676727756216 | > step_time: 0.4344 (0.43522963316544244) | > loader_time: 0.0066 (0.0068531140037204905) --> EVAL PERFORMANCE | > avg_loader_time: 0.008590479971657338 (-1.92044486462227e-05) | > avg_step_time: 0.45669387427853864 (+0.0003388733930990706) | > avg_loss_disc: 2.7838869229168957 (+0.022654110277201767) | > avg_loss_disc_real_0: 0.19887194205337846 (-0.005687168261534731) | > avg_loss_disc_real_1: 0.23643202093285573 (+0.011436433649398892) | > avg_loss_disc_real_2: 0.2418381540288388 (+0.003153053597665184) | > avg_loss_disc_real_3: 0.24171514809131622 (+0.0028466454274217734) | > avg_loss_disc_real_4: 0.243815444095034 (-0.0009711346995662506) | > avg_loss_disc_real_5: 0.24796490979866243 (+0.0013308481011592155) | > avg_loss_0: 2.7838869229168957 (+0.022654110277201767) | > avg_grad_norm_0: tensor(13.3649, device='cuda:0') (tensor(-1.8297, device='cuda:0')) | > avg_loss_gen: 1.8577861802678712 (-0.0030991174805332644) | > avg_loss_kl: 2.316179942077313 (+0.09670762948586864) | > avg_loss_feat: 2.689431475921416 (-0.10189016436187348) | > avg_loss_mel: 18.162196119066696 (+0.39232310442856644) | > avg_loss_duration: 1.607840444000674 (-0.007379083566262823) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.63343445683869 (+0.3766628856390284) | > avg_grad_norm_1: tensor(485.5574, device='cuda:0') (tensor(-27.2095, device='cuda:0'))  > EPOCH: 359/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:53:30)   --> TIME: 2025-02-25 11:53:33 -- STEP: 1/72 -- GLOBAL_STEP: 1025850 | > loss_disc: 2.892669677734375 (2.892669677734375) | > loss_disc_real_0: 0.1720891296863556 (0.1720891296863556) | > loss_disc_real_1: 0.2563384175300598 (0.2563384175300598) | > loss_disc_real_2: 0.273491770029068 (0.273491770029068) | > loss_disc_real_3: 0.2757036089897156 (0.2757036089897156) | > loss_disc_real_4: 0.27971020340919495 (0.27971020340919495) | > loss_disc_real_5: 0.26214468479156494 (0.26214468479156494) | > loss_0: 2.892669677734375 (2.892669677734375) | > grad_norm_0: tensor(8.2091, device='cuda:0') (tensor(8.2091, device='cuda:0')) | > loss_gen: 1.8776249885559082 (1.8776249885559082) | > loss_kl: 2.182997465133667 (2.182997465133667) | > loss_feat: 2.5965511798858643 (2.5965511798858643) | > loss_mel: 17.765932083129883 (17.765932083129883) | > loss_duration: 1.610757827758789 (1.610757827758789) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.033863067626953 (26.033863067626953) | > grad_norm_1: tensor(456.4623, device='cuda:0') (tensor(456.4623, device='cuda:0')) | > current_lr_0: 0.00019122286143165244 | > current_lr_1: 0.00019122286143165244 | > step_time: 0.439 (0.43904924392700195) | > loader_time: 0.0079 (0.007923364639282227)  --> TIME: 2025-02-25 11:53:55 -- STEP: 51/72 -- GLOBAL_STEP: 1025900 | > loss_disc: 2.8332607746124268 (2.770031068839278) | > loss_disc_real_0: 0.19224108755588531 (0.1968183575891981) | > loss_disc_real_1: 0.24585647881031036 (0.23198344485432493) | > loss_disc_real_2: 0.22931838035583496 (0.2405177129834306) | > loss_disc_real_3: 0.23007039725780487 (0.23904323811624564) | > loss_disc_real_4: 0.23632079362869263 (0.24502004186312357) | > loss_disc_real_5: 0.2502477467060089 (0.24572642410502715) | > loss_0: 2.8332607746124268 (2.770031068839278) | > grad_norm_0: tensor(8.9070, device='cuda:0') (tensor(14.1512, device='cuda:0')) | > loss_gen: 1.8764331340789795 (1.8442872271818274) | > loss_kl: 2.0176949501037598 (2.1126652675516464) | > loss_feat: 2.5215110778808594 (2.7297732876796355) | > loss_mel: 16.92457389831543 (17.528074545018804) | > loss_duration: 1.5977227687835693 (1.5852807246002498) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.93793487548828 (25.800081215652764) | > grad_norm_1: tensor(55.4678, device='cuda:0') (tensor(419.2360, device='cuda:0')) | > current_lr_0: 0.00019122286143165244 | > current_lr_1: 0.00019122286143165244 | > step_time: 0.4419 (0.4422045595505658) | > loader_time: 0.0092 (0.007906923107072418) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719531583114408 (+0.00012905161145707056) | > avg_step_time: 0.45398071114446076 (-0.0027131631340778783) | > avg_loss_disc: 2.7627134423860356 (-0.0211734805308601) | > avg_loss_disc_real_0: 0.20117158348291692 (+0.0022996414295384515) | > avg_loss_disc_real_1: 0.23077694861821726 (-0.005655072314638465) | > avg_loss_disc_real_2: 0.239803896193773 (-0.002034257835065806) | > avg_loss_disc_real_3: 0.23886702409092808 (-0.0028481240003881414) | > avg_loss_disc_real_4: 0.2441178064950755 (+0.0003023624000415004) | > avg_loss_disc_real_5: 0.24476818656417684 (-0.0031967232344855823) | > avg_loss_0: 2.7627134423860356 (-0.0211734805308601) | > avg_grad_norm_0: tensor(15.3689, device='cuda:0') (+tensor(2.0040, device='cuda:0')) | > avg_loss_gen: 1.840114479333582 (-0.01767170093428927) | > avg_loss_kl: 2.1769496974810747 (-0.13923024459623834) | > avg_loss_feat: 2.6877013394530396 (-0.0017301364683763865) | > avg_loss_mel: 17.550028424867435 (-0.6121676941992611) | > avg_loss_duration: 1.6012033072995469 (-0.006637136701127089) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.85599730048381 (-0.7774371563548783) | > avg_grad_norm_1: tensor(478.1534, device='cuda:0') (tensor(-7.4040, device='cuda:0'))  > EPOCH: 360/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:54:07)   --> TIME: 2025-02-25 11:54:22 -- STEP: 29/72 -- GLOBAL_STEP: 1025950 | > loss_disc: 2.7773752212524414 (2.7976901366792877) | > loss_disc_real_0: 0.08400280773639679 (0.2091082154676832) | > loss_disc_real_1: 0.23599550127983093 (0.24626872919756793) | > loss_disc_real_2: 0.2626488208770752 (0.24409981020565694) | > loss_disc_real_3: 0.26425039768218994 (0.24522975652382292) | > loss_disc_real_4: 0.2411300390958786 (0.2475015084291326) | > loss_disc_real_5: 0.2544519901275635 (0.24946909217998905) | > loss_0: 2.7773752212524414 (2.7976901366792877) | > grad_norm_0: tensor(31.6427, device='cuda:0') (tensor(19.7383, device='cuda:0')) | > loss_gen: 1.7236196994781494 (1.893113004750219) | > loss_kl: 2.2467310428619385 (2.167052881471042) | > loss_feat: 2.8419716358184814 (2.7522033329667717) | > loss_mel: 17.29266357421875 (17.957904947215113) | > loss_duration: 1.579225778579712 (1.570166776920187) | > amp_scaler: 128.0 (158.89655172413794) | > loss_1: 25.68421173095703 (26.34044101320464) | > grad_norm_1: tensor(62.4506, device='cuda:0') (tensor(460.3359, device='cuda:0')) | > current_lr_0: 0.0001911989585739735 | > current_lr_1: 0.0001911989585739735 | > step_time: 0.4388 (0.43505792782224456) | > loader_time: 0.0078 (0.0067870123633022965) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847823183301469 (-0.00024129975009971882) | > avg_step_time: 0.4512151966632252 (-0.002765514481235576) | > avg_loss_disc: 2.790800292726973 (+0.028086850340937275) | > avg_loss_disc_real_0: 0.21003445111949678 (+0.008862867636579869) | > avg_loss_disc_real_1: 0.2392235555699174 (+0.00844660695170013) | > avg_loss_disc_real_2: 0.24405314469001663 (+0.0042492484962436405) | > avg_loss_disc_real_3: 0.24453800832721548 (+0.005670984236287396) | > avg_loss_disc_real_4: 0.2463378243043389 (+0.002220017809263397) | > avg_loss_disc_real_5: 0.24707947923264034 (+0.0023112926684634993) | > avg_loss_0: 2.790800292726973 (+0.028086850340937275) | > avg_grad_norm_0: tensor(18.8710, device='cuda:0') (+tensor(3.5021, device='cuda:0')) | > avg_loss_gen: 1.872732056698329 (+0.03261757736474702) | > avg_loss_kl: 2.236206392167319 (+0.059256694686244504) | > avg_loss_feat: 2.7174809818536465 (+0.02977964240060693) | > avg_loss_mel: 17.880520834049705 (+0.33049240918226985) | > avg_loss_duration: 1.6044849425974026 (+0.003281635297855745) | > avg_amp_scaler: 140.61971830985914 (+12.619718309859138) | > avg_loss_1: 26.311425115021184 (+0.4554278145373729) | > avg_grad_norm_1: tensor(525.5225, device='cuda:0') (+tensor(47.3691, device='cuda:0'))  > EPOCH: 361/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:54:44)   --> TIME: 2025-02-25 11:54:49 -- STEP: 7/72 -- GLOBAL_STEP: 1026000 | > loss_disc: 2.847836494445801 (2.7748402186802457) | > loss_disc_real_0: 0.33884701132774353 (0.22297484747001103) | > loss_disc_real_1: 0.19665955007076263 (0.22392708914620535) | > loss_disc_real_2: 0.23725374042987823 (0.2114777671439307) | > loss_disc_real_3: 0.2444738894701004 (0.22329541827951158) | > loss_disc_real_4: 0.237458735704422 (0.23882412697587693) | > loss_disc_real_5: 0.23828615248203278 (0.25722433839525494) | > loss_0: 2.847836494445801 (2.7748402186802457) | > grad_norm_0: tensor(19.1024, device='cuda:0') (tensor(17.8957, device='cuda:0')) | > loss_gen: 1.9422836303710938 (1.8419823135648454) | > loss_kl: 2.2353761196136475 (2.1420871189662387) | > loss_feat: 2.4595818519592285 (2.5392277921949113) | > loss_mel: 18.264358520507812 (17.773724147251674) | > loss_duration: 1.591740608215332 (1.5763218402862549) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.49333953857422 (25.873342786516464) | > grad_norm_1: tensor(602.7507, device='cuda:0') (tensor(428.3558, device='cuda:0')) | > current_lr_0: 0.00019117505870415173 | > current_lr_1: 0.00019117505870415173 | > step_time: 0.43 (0.42626963342939106) | > loader_time: 0.0068 (0.006359883717128209)  --> TIME: 2025-02-25 11:55:12 -- STEP: 57/72 -- GLOBAL_STEP: 1026050 | > loss_disc: 2.6728806495666504 (2.769787395209597) | > loss_disc_real_0: 0.1508534550666809 (0.20640383673864499) | > loss_disc_real_1: 0.24637958407402039 (0.23210173923718302) | > loss_disc_real_2: 0.25011900067329407 (0.23740090978773018) | > loss_disc_real_3: 0.22813533246517181 (0.23818784716882205) | > loss_disc_real_4: 0.2699032723903656 (0.24439919700748042) | > loss_disc_real_5: 0.24870920181274414 (0.2480472321050209) | > loss_0: 2.6728806495666504 (2.769787395209597) | > grad_norm_0: tensor(13.7479, device='cuda:0') (tensor(16.6360, device='cuda:0')) | > loss_gen: 1.8089784383773804 (1.86431408346745) | > loss_kl: 2.347593069076538 (2.285029871421947) | > loss_feat: 2.6600825786590576 (2.7770273309004936) | > loss_mel: 17.171415328979492 (18.12567811263235) | > loss_duration: 1.65709388256073 (1.5995053659405625) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.64516258239746 (26.651554643062124) | > grad_norm_1: tensor(557.0458, device='cuda:0') (tensor(485.0920, device='cuda:0')) | > current_lr_0: 0.00019117505870415173 | > current_lr_1: 0.00019117505870415173 | > step_time: 0.4651 (0.4499694631810774) | > loader_time: 0.0096 (0.007875266828035054) --> EVAL PERFORMANCE | > avg_loader_time: 0.008530008960777607 (+5.177712776291768e-05) | > avg_step_time: 0.4593153201358419 (+0.008100123472616738) | > avg_loss_disc: 2.777761053031599 (-0.013039239695373794) | > avg_loss_disc_real_0: 0.20613998143186033 (-0.0038944696876364593) | > avg_loss_disc_real_1: 0.23411855815162122 (-0.0051049974182961755) | > avg_loss_disc_real_2: 0.23745453336708983 (-0.006598611322926801) | > avg_loss_disc_real_3: 0.23961320099696307 (-0.004924807330252412) | > avg_loss_disc_real_4: 0.2447040125098027 (-0.0016338117945361996) | > avg_loss_disc_real_5: 0.24763888225589 (+0.0005594030232496661) | > avg_loss_0: 2.777761053031599 (-0.013039239695373794) | > avg_grad_norm_0: tensor(16.9547, device='cuda:0') (tensor(-1.9163, device='cuda:0')) | > avg_loss_gen: 1.8638849829284239 (-0.008847073769905123) | > avg_loss_kl: 2.3112886724337716 (+0.07508228026645236) | > avg_loss_feat: 2.757740057690043 (+0.04025907583639654) | > avg_loss_mel: 18.074393285831935 (+0.19387245178223012) | > avg_loss_duration: 1.6176573138841441 (+0.013172371286741535) | > avg_amp_scaler: 128.0 (-12.619718309859138) | > avg_loss_1: 26.624964243929153 (+0.3135391289079692) | > avg_grad_norm_1: tensor(468.6326, device='cuda:0') (tensor(-56.8899, device='cuda:0'))  > EPOCH: 362/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:55:21)   --> TIME: 2025-02-25 11:55:39 -- STEP: 35/72 -- GLOBAL_STEP: 1026100 | > loss_disc: 2.8038058280944824 (2.767965357644217) | > loss_disc_real_0: 0.23071491718292236 (0.2002876000744956) | > loss_disc_real_1: 0.23520547151565552 (0.23567539921828679) | > loss_disc_real_2: 0.26635560393333435 (0.23139583553586687) | > loss_disc_real_3: 0.2504371702671051 (0.2357577179159437) | > loss_disc_real_4: 0.26052308082580566 (0.24179873934813909) | > loss_disc_real_5: 0.259928822517395 (0.24403463772365025) | > loss_0: 2.8038058280944824 (2.767965357644217) | > grad_norm_0: tensor(10.0571, device='cuda:0') (tensor(13.0167, device='cuda:0')) | > loss_gen: 1.773719072341919 (1.842745988709586) | > loss_kl: 2.5335278511047363 (2.2585268088749473) | > loss_feat: 2.4335241317749023 (2.811500072479248) | > loss_mel: 18.24858283996582 (18.271473257882253) | > loss_duration: 1.6260600090026855 (1.578817013331822) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.615415573120117 (26.76306321280343) | > grad_norm_1: tensor(942.3916, device='cuda:0') (tensor(585.7910, device='cuda:0')) | > current_lr_0: 0.0001911511618218137 | > current_lr_1: 0.0001911511618218137 | > step_time: 0.4465 (0.4369713715144566) | > loader_time: 0.0085 (0.007209934507097517) --> EVAL PERFORMANCE | > avg_loader_time: 0.008587632380740744 (+5.7623419963136885e-05) | > avg_step_time: 0.4570197085259666 (-0.0022956116098753476) | > avg_loss_disc: 2.7666385073057356 (-0.011122545725863464) | > avg_loss_disc_real_0: 0.20101334574356886 (-0.00512663568829147) | > avg_loss_disc_real_1: 0.23080085322890484 (-0.00331770492271638) | > avg_loss_disc_real_2: 0.23433600093277407 (-0.0031185324343157572) | > avg_loss_disc_real_3: 0.23754972849093692 (-0.0020634725060261483) | > avg_loss_disc_real_4: 0.24143075145466228 (-0.0032732610551404195) | > avg_loss_disc_real_5: 0.24407037987675465 (-0.003568502379135363) | > avg_loss_0: 2.7666385073057356 (-0.011122545725863464) | > avg_grad_norm_0: tensor(13.7417, device='cuda:0') (tensor(-3.2129, device='cuda:0')) | > avg_loss_gen: 1.8378520666713445 (-0.026032916257079375) | > avg_loss_kl: 2.254741705639262 (-0.05654696679450977) | > avg_loss_feat: 2.7426727959807495 (-0.015067261709293511) | > avg_loss_mel: 17.98645664268815 (-0.08793664314378447) | > avg_loss_duration: 1.6041796324958264 (-0.01347768138831773) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.425902836759324 (-0.1990614071698289) | > avg_grad_norm_1: tensor(537.5877, device='cuda:0') (+tensor(68.9551, device='cuda:0'))  > EPOCH: 363/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:55:58)   --> TIME: 2025-02-25 11:56:06 -- STEP: 13/72 -- GLOBAL_STEP: 1026150 | > loss_disc: 2.8092403411865234 (2.731232423048753) | > loss_disc_real_0: 0.21093237400054932 (0.20908824640970963) | > loss_disc_real_1: 0.20235185325145721 (0.22538828276670897) | > loss_disc_real_2: 0.22789481282234192 (0.22345053805754736) | > loss_disc_real_3: 0.2443489134311676 (0.23414576626740968) | > loss_disc_real_4: 0.26678138971328735 (0.25017000047060156) | > loss_disc_real_5: 0.25319918990135193 (0.24931082817224356) | > loss_0: 2.8092403411865234 (2.731232423048753) | > grad_norm_0: tensor(8.5251, device='cuda:0') (tensor(16.3021, device='cuda:0')) | > loss_gen: 1.9032783508300781 (1.9167656073203454) | > loss_kl: 2.2882344722747803 (2.120257863631615) | > loss_feat: 2.820934772491455 (2.884569626588088) | > loss_mel: 17.116008758544922 (17.38985824584961) | > loss_duration: 1.479534387588501 (1.5513805150985718) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.607990264892578 (25.86283199603741) | > grad_norm_1: tensor(607.3874, device='cuda:0') (tensor(409.3767, device='cuda:0')) | > current_lr_0: 0.00019112726792658598 | > current_lr_1: 0.00019112726792658598 | > step_time: 0.4511 (0.43854262278630185) | > loader_time: 0.006 (0.006531275235689604)  --> TIME: 2025-02-25 11:56:29 -- STEP: 63/72 -- GLOBAL_STEP: 1026200 | > loss_disc: 2.813441514968872 (2.7637989407493952) | > loss_disc_real_0: 0.17277973890304565 (0.19967024146564424) | > loss_disc_real_1: 0.26388242840766907 (0.22852427264054617) | > loss_disc_real_2: 0.21707668900489807 (0.23580080081546118) | > loss_disc_real_3: 0.25199955701828003 (0.23645184295518057) | > loss_disc_real_4: 0.22852100431919098 (0.24530950164984142) | > loss_disc_real_5: 0.24103392660617828 (0.24914971916448497) | > loss_0: 2.813441514968872 (2.7637989407493952) | > grad_norm_0: tensor(9.5194, device='cuda:0') (tensor(13.7806, device='cuda:0')) | > loss_gen: 1.8616825342178345 (1.8548794057634141) | > loss_kl: 2.3664510250091553 (2.241698210201566) | > loss_feat: 2.5113277435302734 (2.7618080320812406) | > loss_mel: 17.612821578979492 (17.604037330264152) | > loss_duration: 1.585856556892395 (1.5899166360734) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.938140869140625 (26.05233985658676) | > grad_norm_1: tensor(159.5553, device='cuda:0') (tensor(461.2796, device='cuda:0')) | > current_lr_0: 0.00019112726792658598 | > current_lr_1: 0.00019112726792658598 | > step_time: 0.4804 (0.45105999235122923) | > loader_time: 0.0112 (0.008340082471332853) --> EVAL PERFORMANCE | > avg_loader_time: 0.008707795344607932 (+0.0001201629638671875) | > avg_step_time: 0.45839977264404297 (+0.0013800641180763962) | > avg_loss_disc: 2.7668824632402877 (+0.0002439559345521758) | > avg_loss_disc_real_0: 0.2008193848838269 (-0.00019396085974196509) | > avg_loss_disc_real_1: 0.22856179614302138 (-0.0022390570858834558) | > avg_loss_disc_real_2: 0.23737124301178356 (+0.003035242079009487) | > avg_loss_disc_real_3: 0.2377821385440692 (+0.00023241005313229257) | > avg_loss_disc_real_4: 0.2460786672964902 (+0.004647915841827915) | > avg_loss_disc_real_5: 0.24813377521407431 (+0.00406339533731967) | > avg_loss_0: 2.7668824632402877 (+0.0002439559345521758) | > avg_grad_norm_0: tensor(13.9829, device='cuda:0') (+tensor(0.2412, device='cuda:0')) | > avg_loss_gen: 1.8527852112138774 (+0.014933144542532917) | > avg_loss_kl: 2.258349668811744 (+0.0036079631724823535) | > avg_loss_feat: 2.748384247363453 (+0.0057114513827034585) | > avg_loss_mel: 17.664342208647405 (-0.3221144340407456) | > avg_loss_duration: 1.5981293967072392 (-0.0060502357885872105) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.121990982915314 (-0.30391185384401) | > avg_grad_norm_1: tensor(466.8764, device='cuda:0') (tensor(-70.7113, device='cuda:0'))  > EPOCH: 364/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:56:35)   --> TIME: 2025-02-25 11:56:56 -- STEP: 41/72 -- GLOBAL_STEP: 1026250 | > loss_disc: 2.8499412536621094 (2.791606705363203) | > loss_disc_real_0: 0.2077595293521881 (0.20922104269266129) | > loss_disc_real_1: 0.24083733558654785 (0.23217057382188191) | > loss_disc_real_2: 0.2619504928588867 (0.2349591604093226) | > loss_disc_real_3: 0.28567299246788025 (0.23790075502744534) | > loss_disc_real_4: 0.24360613524913788 (0.24374445182521168) | > loss_disc_real_5: 0.23827184736728668 (0.246747736887234) | > loss_0: 2.8499412536621094 (2.791606705363203) | > grad_norm_0: tensor(16.7970, device='cuda:0') (tensor(19.6003, device='cuda:0')) | > loss_gen: 1.850804328918457 (1.8573817334524014) | > loss_kl: 2.03663969039917 (2.2671149009611544) | > loss_feat: 2.58135986328125 (2.77205682382351) | > loss_mel: 17.8514347076416 (18.23757581013005) | > loss_duration: 1.5979645252227783 (1.582620803902789) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.91820526123047 (26.716750075177448) | > grad_norm_1: tensor(357.7218, device='cuda:0') (tensor(494.1721, device='cuda:0')) | > current_lr_0: 0.00019110337701809513 | > current_lr_1: 0.00019110337701809513 | > step_time: 0.45 (0.4490053421113549) | > loader_time: 0.0109 (0.00737906665336795) --> EVAL PERFORMANCE | > avg_loader_time: 0.008631830484094754 (-7.596486051317762e-05) | > avg_step_time: 0.4610715147475122 (+0.00267174210346921) | > avg_loss_disc: 2.7892396617943134 (+0.022357198554025626) | > avg_loss_disc_real_0: 0.21292578407996138 (+0.012106399196134487) | > avg_loss_disc_real_1: 0.23226407967822651 (+0.003702283535205131) | > avg_loss_disc_real_2: 0.23781748640705164 (+0.0004462433952680811) | > avg_loss_disc_real_3: 0.23974417233970804 (+0.0019620337956388345) | > avg_loss_disc_real_4: 0.2448882949604115 (-0.0011903723360786878) | > avg_loss_disc_real_5: 0.24798359207704035 (-0.00015018313703396569) | > avg_loss_0: 2.7892396617943134 (+0.022357198554025626) | > avg_grad_norm_0: tensor(18.6424, device='cuda:0') (+tensor(4.6595, device='cuda:0')) | > avg_loss_gen: 1.861934356286492 (+0.009149145072614662) | > avg_loss_kl: 2.2753673607194926 (+0.01701769190774849) | > avg_loss_feat: 2.7365101861282133 (-0.011874061235239708) | > avg_loss_mel: 18.137058822202004 (+0.47271661355459926) | > avg_loss_duration: 1.6052767357356115 (+0.007147339028372279) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.61614751144194 (+0.49415652852662717) | > avg_grad_norm_1: tensor(494.4828, device='cuda:0') (+tensor(27.6064, device='cuda:0'))  > EPOCH: 365/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:57:12)   --> TIME: 2025-02-25 11:57:23 -- STEP: 19/72 -- GLOBAL_STEP: 1026300 | > loss_disc: 2.7968173027038574 (2.758080394644486) | > loss_disc_real_0: 0.18414369225502014 (0.19065217673778534) | > loss_disc_real_1: 0.19496943056583405 (0.24453523284510562) | > loss_disc_real_2: 0.20607605576515198 (0.23832556526911886) | > loss_disc_real_3: 0.2250467836856842 (0.24300252999130048) | > loss_disc_real_4: 0.2241741418838501 (0.24172937556316978) | > loss_disc_real_5: 0.24152880907058716 (0.24821944142642774) | > loss_0: 2.7968173027038574 (2.758080394644486) | > grad_norm_0: tensor(11.0051, device='cuda:0') (tensor(12.8295, device='cuda:0')) | > loss_gen: 1.7452832460403442 (1.88902579483233) | > loss_kl: 2.0521011352539062 (2.130706184788754) | > loss_feat: 2.340139865875244 (2.829609695233797) | > loss_mel: 17.276586532592773 (17.716209813168174) | > loss_duration: 1.55665123462677 (1.563062780781796) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.970762252807617 (26.12861452604595) | > grad_norm_1: tensor(71.0763, device='cuda:0') (tensor(422.6837, device='cuda:0')) | > current_lr_0: 0.00019107948909596786 | > current_lr_1: 0.00019107948909596786 | > step_time: 0.4435 (0.43670847541407537) | > loader_time: 0.0076 (0.006767574109529194)  --> TIME: 2025-02-25 11:57:47 -- STEP: 69/72 -- GLOBAL_STEP: 1026350 | > loss_disc: 2.7608957290649414 (2.7618749107139697) | > loss_disc_real_0: 0.2239205688238144 (0.1931326576117156) | > loss_disc_real_1: 0.18930724263191223 (0.22953638499197754) | > loss_disc_real_2: 0.20196963846683502 (0.2376903770626455) | > loss_disc_real_3: 0.23722553253173828 (0.2405537302079408) | > loss_disc_real_4: 0.2428308129310608 (0.2443664605202882) | > loss_disc_real_5: 0.24958297610282898 (0.2470835520737413) | > loss_0: 2.7608957290649414 (2.7618749107139697) | > grad_norm_0: tensor(9.5129, device='cuda:0') (tensor(13.3818, device='cuda:0')) | > loss_gen: 1.7449599504470825 (1.8487113748771558) | > loss_kl: 2.2715604305267334 (2.1877894885298135) | > loss_feat: 2.3145878314971924 (2.7509232431218242) | > loss_mel: 17.56612205505371 (17.511255402495895) | > loss_duration: 1.702131748199463 (1.6060702489769978) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.599363327026367 (25.90474993940713) | > grad_norm_1: tensor(429.0121, device='cuda:0') (tensor(505.6957, device='cuda:0')) | > current_lr_0: 0.00019107948909596786 | > current_lr_1: 0.00019107948909596786 | > step_time: 0.5314 (0.45517420768737793) | > loader_time: 0.0125 (0.008523982504139773) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857522454060299 (-5.6605943491763386e-05) | > avg_step_time: 0.45688174811887067 (-0.004189766628641511) | > avg_loss_disc: 2.762212618975572 (-0.027027042818741354) | > avg_loss_disc_real_0: 0.1919842657278961 (-0.020941518352065286) | > avg_loss_disc_real_1: 0.230766226917925 (-0.0014978527603015102) | > avg_loss_disc_real_2: 0.23797919590708236 (+0.00016170950003072115) | > avg_loss_disc_real_3: 0.2409952274510558 (+0.0012510551113477653) | > avg_loss_disc_real_4: 0.24432490787035982 (-0.0005633870900516869) | > avg_loss_disc_real_5: 0.24718739113337557 (-0.0007962009436647821) | > avg_loss_0: 2.762212618975572 (-0.027027042818741354) | > avg_grad_norm_0: tensor(13.5700, device='cuda:0') (tensor(-5.0724, device='cuda:0')) | > avg_loss_gen: 1.8493244429709206 (-0.012609913315571486) | > avg_loss_kl: 2.190512294500646 (-0.08485506621884653) | > avg_loss_feat: 2.7361809837986044 (-0.00032920232960886864) | > avg_loss_mel: 17.471852557759888 (-0.6652062644421157) | > avg_loss_duration: 1.6061100623977018 (+0.0008333266620903501) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.8539805613773 (-0.7621669500646426) | > avg_grad_norm_1: tensor(511.5913, device='cuda:0') (+tensor(17.1085, device='cuda:0'))  > EPOCH: 366/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:57:50)   --> TIME: 2025-02-25 11:58:13 -- STEP: 47/72 -- GLOBAL_STEP: 1026400 | > loss_disc: 2.830528497695923 (2.7683879263857576) | > loss_disc_real_0: 0.31630080938339233 (0.19942743917728992) | > loss_disc_real_1: 0.25210824608802795 (0.23314213499109795) | > loss_disc_real_2: 0.2737233638763428 (0.23904164262274477) | > loss_disc_real_3: 0.24424518644809723 (0.2391827179396406) | > loss_disc_real_4: 0.2614688277244568 (0.24282687902450562) | > loss_disc_real_5: 0.2433769255876541 (0.24485815934678343) | > loss_0: 2.830528497695923 (2.7683879263857576) | > grad_norm_0: tensor(29.3713, device='cuda:0') (tensor(14.5349, device='cuda:0')) | > loss_gen: 1.9681512117385864 (1.8585592051769824) | > loss_kl: 2.2135424613952637 (2.2251813614622074) | > loss_feat: 2.4455392360687256 (2.7807056650202324) | > loss_mel: 16.806865692138672 (17.416419455345643) | > loss_duration: 1.6206562519073486 (1.5804251838237682) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.05475425720215 (25.861290830246947) | > grad_norm_1: tensor(483.4352, device='cuda:0') (tensor(446.5801, device='cuda:0')) | > current_lr_0: 0.00019105560415983087 | > current_lr_1: 0.00019105560415983087 | > step_time: 0.4545 (0.442895899427698) | > loader_time: 0.009 (0.007677610884321497) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866493372850015 (+8.970918789715995e-05) | > avg_step_time: 0.45588569238152304 (-0.000996055737347623) | > avg_loss_disc: 2.774096156509829 (+0.011883537534257194) | > avg_loss_disc_real_0: 0.19994927145225902 (+0.007965005724362928) | > avg_loss_disc_real_1: 0.23167679406387706 (+0.000910567145952057) | > avg_loss_disc_real_2: 0.24043912681895244 (+0.0024599309118700785) | > avg_loss_disc_real_3: 0.24014431594962804 (-0.0008509115014277702) | > avg_loss_disc_real_4: 0.24496648785933642 (+0.0006415799889765983) | > avg_loss_disc_real_5: 0.24574418680768617 (-0.0014432043256893956) | > avg_loss_0: 2.774096156509829 (+0.011883537534257194) | > avg_grad_norm_0: tensor(15.9279, device='cuda:0') (+tensor(2.3578, device='cuda:0')) | > avg_loss_gen: 1.8495144894425297 (+0.0001900464716091399) | > avg_loss_kl: 2.2564942366640333 (+0.06598194216338715) | > avg_loss_feat: 2.731327352389483 (-0.004853631409121206) | > avg_loss_mel: 17.47887428713517 (+0.007021729375281893) | > avg_loss_duration: 1.6003223254647054 (-0.00578773693299639) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.916532570207625 (+0.06255200883032686) | > avg_grad_norm_1: tensor(431.5141, device='cuda:0') (tensor(-80.0772, device='cuda:0'))  > EPOCH: 367/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:58:27)   --> TIME: 2025-02-25 11:58:40 -- STEP: 25/72 -- GLOBAL_STEP: 1026450 | > loss_disc: 2.703732490539551 (2.7426041507720953) | > loss_disc_real_0: 0.18528306484222412 (0.19253939986228943) | > loss_disc_real_1: 0.18697290122509003 (0.23088061094284057) | > loss_disc_real_2: 0.22589010000228882 (0.23637183666229247) | > loss_disc_real_3: 0.23644188046455383 (0.23653294265270233) | > loss_disc_real_4: 0.25127702951431274 (0.24237469017505645) | > loss_disc_real_5: 0.2533537447452545 (0.24683753490448) | > loss_0: 2.703732490539551 (2.7426041507720953) | > grad_norm_0: tensor(12.3875, device='cuda:0') (tensor(17.5404, device='cuda:0')) | > loss_gen: 1.875633716583252 (1.8721435260772705) | > loss_kl: 1.9997150897979736 (2.0759288930892943) | > loss_feat: 2.533463716506958 (2.881111297607422) | > loss_mel: 17.076906204223633 (17.65207321166992) | > loss_duration: 1.5787379741668701 (1.5573972368240356) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.064455032348633 (26.038654022216797) | > grad_norm_1: tensor(694.5914, device='cuda:0') (tensor(604.5700, device='cuda:0')) | > current_lr_0: 0.0001910317222093109 | > current_lr_1: 0.0001910317222093109 | > step_time: 0.4337 (0.4569245910644531) | > loader_time: 0.0074 (0.006694765090942383) --> EVAL PERFORMANCE | > avg_loader_time: 0.008558169217176842 (-0.00010676451132330886) | > avg_step_time: 0.46203963521500707 (+0.006153942833484027) | > avg_loss_disc: 2.7623267946108974 (-0.011769361898931852) | > avg_loss_disc_real_0: 0.19980320533816243 (-0.00014606611409659354) | > avg_loss_disc_real_1: 0.2295449934375118 (-0.0021318006263652545) | > avg_loss_disc_real_2: 0.23804526723606487 (-0.0023938595828875697) | > avg_loss_disc_real_3: 0.23625910093247052 (-0.003885215017157523) | > avg_loss_disc_real_4: 0.2441904448287588 (-0.0007760430305776278) | > avg_loss_disc_real_5: 0.24762309772867552 (+0.0018789109209893518) | > avg_loss_0: 2.7623267946108974 (-0.011769361898931852) | > avg_grad_norm_0: tensor(18.7718, device='cuda:0') (+tensor(2.8440, device='cuda:0')) | > avg_loss_gen: 1.8635361345720962 (+0.014021645129566496) | > avg_loss_kl: 2.199154922659967 (-0.05733931400406611) | > avg_loss_feat: 2.7940733029808795 (+0.06274595059139632) | > avg_loss_mel: 17.760254362939108 (+0.28138007580393776) | > avg_loss_duration: 1.5982689521682094 (-0.0020533732964960194) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.21528757122201 (+0.2987550010143849) | > avg_grad_norm_1: tensor(512.5425, device='cuda:0') (+tensor(81.0284, device='cuda:0'))  > EPOCH: 368/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:59:04)   --> TIME: 2025-02-25 11:59:07 -- STEP: 3/72 -- GLOBAL_STEP: 1026500 | > loss_disc: 2.567540168762207 (2.5926076571146646) | > loss_disc_real_0: 0.22786301374435425 (0.2065888742605845) | > loss_disc_real_1: 0.2108864188194275 (0.2367035299539566) | > loss_disc_real_2: 0.22054390609264374 (0.227936382095019) | > loss_disc_real_3: 0.19878046214580536 (0.21127342681090036) | > loss_disc_real_4: 0.2292378693819046 (0.2282191962003708) | > loss_disc_real_5: 0.22096529603004456 (0.23947851856549582) | > loss_0: 2.567540168762207 (2.5926076571146646) | > grad_norm_0: tensor(9.3279, device='cuda:0') (tensor(8.5860, device='cuda:0')) | > loss_gen: 1.8377344608306885 (1.955176870028178) | > loss_kl: 2.129551887512207 (2.2326317628224692) | > loss_feat: 3.2435803413391113 (3.0844054222106934) | > loss_mel: 17.063804626464844 (17.711884180704754) | > loss_duration: 1.6254242658615112 (1.5902217626571655) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.900094985961914 (26.57431983947754) | > grad_norm_1: tensor(582.7442, device='cuda:0') (tensor(723.5290, device='cuda:0')) | > current_lr_0: 0.00019100784324403472 | > current_lr_1: 0.00019100784324403472 | > step_time: 0.4394 (0.4347817897796631) | > loader_time: 0.0052 (0.006560087203979492)  --> TIME: 2025-02-25 11:59:30 -- STEP: 53/72 -- GLOBAL_STEP: 1026550 | > loss_disc: 2.7642822265625 (2.766084464091175) | > loss_disc_real_0: 0.27135705947875977 (0.203964408979101) | > loss_disc_real_1: 0.2302817702293396 (0.23179178417853588) | > loss_disc_real_2: 0.24567408859729767 (0.2356012798142883) | > loss_disc_real_3: 0.24634677171707153 (0.24246736208222947) | > loss_disc_real_4: 0.2422037273645401 (0.24561670984861986) | > loss_disc_real_5: 0.24435564875602722 (0.24846805294729632) | > loss_0: 2.7642822265625 (2.766084464091175) | > grad_norm_0: tensor(24.3529, device='cuda:0') (tensor(15.5568, device='cuda:0')) | > loss_gen: 1.8586547374725342 (1.8871163512175937) | > loss_kl: 2.224234104156494 (2.1875345122139414) | > loss_feat: 2.4561574459075928 (2.8120598748045147) | > loss_mel: 16.82748031616211 (17.62236553767942) | > loss_duration: 1.6892828941345215 (1.588785947493787) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.055809020996094 (26.097862207664633) | > grad_norm_1: tensor(762.3193, device='cuda:0') (tensor(459.3712, device='cuda:0')) | > current_lr_0: 0.00019100784324403472 | > current_lr_1: 0.00019100784324403472 | > step_time: 0.4538 (0.4433057263212384) | > loader_time: 0.0095 (0.007720821308639814) --> EVAL PERFORMANCE | > avg_loader_time: 0.008513151759832678 (-4.50174573441639e-05) | > avg_step_time: 0.45471631305318483 (-0.007323322161822243) | > avg_loss_disc: 2.779976055655681 (+0.017649261044783593) | > avg_loss_disc_real_0: 0.20554562804984375 (+0.005742422711681322) | > avg_loss_disc_real_1: 0.23299897556573573 (+0.003453982128223926) | > avg_loss_disc_real_2: 0.2375058605217598 (-0.0005394067143050796) | > avg_loss_disc_real_3: 0.24249606149297365 (+0.00623696056050313) | > avg_loss_disc_real_4: 0.24608678171332454 (+0.0018963368845657524) | > avg_loss_disc_real_5: 0.2485239289596047 (+0.0009008312309291833) | > avg_loss_0: 2.779976055655681 (+0.017649261044783593) | > avg_grad_norm_0: tensor(15.2778, device='cuda:0') (tensor(-3.4940, device='cuda:0')) | > avg_loss_gen: 1.8726744635004393 (+0.009138328928343098) | > avg_loss_kl: 2.257603816583124 (+0.058448893923156664) | > avg_loss_feat: 2.7660294485763766 (-0.028043854404502966) | > avg_loss_mel: 17.865333382512482 (+0.10507901957337396) | > avg_loss_duration: 1.6127940130905367 (+0.014525060922327304) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.374435102435903 (+0.15914753121389325) | > avg_grad_norm_1: tensor(475.7597, device='cuda:0') (tensor(-36.7829, device='cuda:0'))  > EPOCH: 369/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 11:59:41)   --> TIME: 2025-02-25 11:59:57 -- STEP: 31/72 -- GLOBAL_STEP: 1026600 | > loss_disc: 2.805575132369995 (2.790791473081035) | > loss_disc_real_0: 0.285721093416214 (0.20822176481446913) | > loss_disc_real_1: 0.22296324372291565 (0.23633192287337396) | > loss_disc_real_2: 0.23189297318458557 (0.24006628605627245) | > loss_disc_real_3: 0.3006308674812317 (0.24523102708401218) | > loss_disc_real_4: 0.22404839098453522 (0.2415782150722319) | > loss_disc_real_5: 0.22360007464885712 (0.24309010803699493) | > loss_0: 2.805575132369995 (2.790791473081035) | > grad_norm_0: tensor(15.5691, device='cuda:0') (tensor(12.4802, device='cuda:0')) | > loss_gen: 1.82016921043396 (1.861218310171558) | > loss_kl: 2.0673882961273193 (2.2776234726752005) | > loss_feat: 2.8814399242401123 (2.8004425341083157) | > loss_mel: 19.117244720458984 (18.14248608004662) | > loss_duration: 1.6042685508728027 (1.5932667639947706) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.490510940551758 (26.675037199451076) | > grad_norm_1: tensor(856.9381, device='cuda:0') (tensor(317.7480, device='cuda:0')) | > current_lr_0: 0.0001909839672636292 | > current_lr_1: 0.0001909839672636292 | > step_time: 0.4446 (0.43695772078729445) | > loader_time: 0.0093 (0.007025034196915165) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865558167578469 (+0.00014242991595201163) | > avg_step_time: 0.45660304687392544 (+0.001886733820740616) | > avg_loss_disc: 2.7843962118659222 (+0.004420156210241277) | > avg_loss_disc_real_0: 0.20909886146095438 (+0.0035532334111106345) | > avg_loss_disc_real_1: 0.23335046562510478 (+0.0003514900593690473) | > avg_loss_disc_real_2: 0.2431675174706419 (+0.005661656948882099) | > avg_loss_disc_real_3: 0.2413576325060616 (-0.001138428986912049) | > avg_loss_disc_real_4: 0.24195366468228086 (-0.004133117031043687) | > avg_loss_disc_real_5: 0.24364847532460387 (-0.0048754536350008315) | > avg_loss_0: 2.7843962118659222 (+0.004420156210241277) | > avg_grad_norm_0: tensor(14.9228, device='cuda:0') (tensor(-0.3550, device='cuda:0')) | > avg_loss_gen: 1.8531215627428512 (-0.019552900757588088) | > avg_loss_kl: 2.2556249742776577 (-0.0019788423054660775) | > avg_loss_feat: 2.711464371479733 (-0.05456507709664349) | > avg_loss_mel: 17.82526991400921 (-0.04006346850327347) | > avg_loss_duration: 1.6114997158587818 (-0.0012942972317548929) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.256980546763245 (-0.1174545556726585) | > avg_grad_norm_1: tensor(410.3885, device='cuda:0') (tensor(-65.3712, device='cuda:0'))  > EPOCH: 370/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:00:18)   --> TIME: 2025-02-25 12:00:24 -- STEP: 9/72 -- GLOBAL_STEP: 1026650 | > loss_disc: 2.6324081420898438 (2.720231983396742) | > loss_disc_real_0: 0.161184623837471 (0.16084308591153887) | > loss_disc_real_1: 0.20546184480190277 (0.22856615318192375) | > loss_disc_real_2: 0.23496758937835693 (0.23756270276175606) | > loss_disc_real_3: 0.21688924729824066 (0.23791087336010402) | > loss_disc_real_4: 0.23720087110996246 (0.2586848388115565) | > loss_disc_real_5: 0.24791903793811798 (0.25355173647403717) | > loss_0: 2.6324081420898438 (2.720231983396742) | > grad_norm_0: tensor(7.6391, device='cuda:0') (tensor(17.9301, device='cuda:0')) | > loss_gen: 1.8707271814346313 (1.8887610567940607) | > loss_kl: 2.076326370239258 (2.114822268486023) | > loss_feat: 2.77821683883667 (2.769578907224867) | > loss_mel: 17.126567840576172 (17.434957292344833) | > loss_duration: 1.578942894935608 (1.5725064675013225) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.430782318115234 (25.78062629699707) | > grad_norm_1: tensor(443.3471, device='cuda:0') (tensor(759.5646, device='cuda:0')) | > current_lr_0: 0.00019096009426772123 | > current_lr_1: 0.00019096009426772123 | > step_time: 0.4394 (0.4303717878129747) | > loader_time: 0.0054 (0.006163544125027127)  --> TIME: 2025-02-25 12:00:47 -- STEP: 59/72 -- GLOBAL_STEP: 1026700 | > loss_disc: 2.7168333530426025 (2.7515967498391363) | > loss_disc_real_0: 0.23335447907447815 (0.19259236235234697) | > loss_disc_real_1: 0.29326918721199036 (0.23081659083649264) | > loss_disc_real_2: 0.2322208136320114 (0.2364493186696101) | > loss_disc_real_3: 0.2397117018699646 (0.2373474270610486) | > loss_disc_real_4: 0.23628224432468414 (0.24636930151511047) | > loss_disc_real_5: 0.23606015741825104 (0.24752760867951287) | > loss_0: 2.7168333530426025 (2.7515967498391363) | > grad_norm_0: tensor(10.2461, device='cuda:0') (tensor(15.3984, device='cuda:0')) | > loss_gen: 1.8445477485656738 (1.865556072380583) | > loss_kl: 2.585095167160034 (2.2026746838779774) | > loss_feat: 2.978532314300537 (2.8000780546059043) | > loss_mel: 16.78684425354004 (17.602739883681476) | > loss_duration: 1.629190444946289 (1.58955754465976) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.824209213256836 (26.060606390742933) | > grad_norm_1: tensor(854.5385, device='cuda:0') (tensor(603.5091, device='cuda:0')) | > current_lr_0: 0.00019096009426772123 | > current_lr_1: 0.00019096009426772123 | > step_time: 0.4633 (0.4448104914972338) | > loader_time: 0.0099 (0.007959677001177255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008529538839635715 (-0.00012604283614897453) | > avg_step_time: 0.4554101514144683 (-0.0011928954594571661) | > avg_loss_disc: 2.7539126134254563 (-0.030483598440465887) | > avg_loss_disc_real_0: 0.1934198140048645 (-0.01567904745608989) | > avg_loss_disc_real_1: 0.23110352267681714 (-0.0022469429482876446) | > avg_loss_disc_real_2: 0.23658404467811048 (-0.0065834727925314085) | > avg_loss_disc_real_3: 0.23777849573484608 (-0.0035791367712155187) | > avg_loss_disc_real_4: 0.24525166397363368 (+0.0032979992913528267) | > avg_loss_disc_real_5: 0.2475838142801339 (+0.003935338955530027) | > avg_loss_0: 2.7539126134254563 (-0.030483598440465887) | > avg_grad_norm_0: tensor(14.5099, device='cuda:0') (tensor(-0.4128, device='cuda:0')) | > avg_loss_gen: 1.862441813442069 (+0.009320250699217825) | > avg_loss_kl: 2.2536185660832366 (-0.0020064081944211054) | > avg_loss_feat: 2.7778529466038018 (+0.0663885751240687) | > avg_loss_mel: 17.640837790260854 (-0.18443212374835483) | > avg_loss_duration: 1.6007500702226665 (-0.010749645636115357) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.13550132429096 (-0.12147922247228493) | > avg_grad_norm_1: tensor(586.5414, device='cuda:0') (+tensor(176.1530, device='cuda:0'))  > EPOCH: 371/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:00:55)   --> TIME: 2025-02-25 12:01:14 -- STEP: 37/72 -- GLOBAL_STEP: 1026750 | > loss_disc: 2.7904272079467773 (2.7773149593456368) | > loss_disc_real_0: 0.18062224984169006 (0.19899363916467977) | > loss_disc_real_1: 0.23998430371284485 (0.23119489123692383) | > loss_disc_real_2: 0.2786249816417694 (0.2407295599177077) | > loss_disc_real_3: 0.25698503851890564 (0.2373969877088392) | > loss_disc_real_4: 0.2590728998184204 (0.24449568866072474) | > loss_disc_real_5: 0.2679177522659302 (0.24565130552730044) | > loss_0: 2.7904272079467773 (2.7773149593456368) | > grad_norm_0: tensor(5.4643, device='cuda:0') (tensor(13.5576, device='cuda:0')) | > loss_gen: 1.9414629936218262 (1.8538045399897807) | > loss_kl: 2.226778745651245 (2.3571909311655412) | > loss_feat: 2.860595703125 (2.7677264277999467) | > loss_mel: 18.007471084594727 (18.379039661304372) | > loss_duration: 1.646247148513794 (1.595241082681192) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.68255615234375 (26.95300262038772) | > grad_norm_1: tensor(250.8492, device='cuda:0') (tensor(416.7657, device='cuda:0')) | > current_lr_0: 0.00019093622425593777 | > current_lr_1: 0.00019093622425593777 | > step_time: 0.4521 (0.44391061164237355) | > loader_time: 0.0083 (0.007422872491784998) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868641826468454 (+0.00015687942504882466) | > avg_step_time: 0.45857344546788176 (+0.003163294053413479) | > avg_loss_disc: 2.777496250582413 (+0.02358363715695644) | > avg_loss_disc_real_0: 0.2008867041325905 (+0.007466890127726) | > avg_loss_disc_real_1: 0.23182822952807788 (+0.000724706851260748) | > avg_loss_disc_real_2: 0.23813826076581446 (+0.0015542160877039801) | > avg_loss_disc_real_3: 0.23747078549694006 (-0.00030771023790601704) | > avg_loss_disc_real_4: 0.2449429387777624 (-0.00030872519587127334) | > avg_loss_disc_real_5: 0.24618656282693568 (-0.0013972514531982172) | > avg_loss_0: 2.777496250582413 (+0.02358363715695644) | > avg_grad_norm_0: tensor(13.2618, device='cuda:0') (tensor(-1.2481, device='cuda:0')) | > avg_loss_gen: 1.8492138100342013 (-0.013228003407867783) | > avg_loss_kl: 2.3466391496255365 (+0.09302058354229992) | > avg_loss_feat: 2.7483337597108224 (-0.029519186892979388) | > avg_loss_mel: 18.251649507334538 (+0.610811717073684) | > avg_loss_duration: 1.615747398054096 (+0.014997327831429486) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.81158355927803 (+0.6760822349870708) | > avg_grad_norm_1: tensor(403.2223, device='cuda:0') (tensor(-183.3192, device='cuda:0'))  > EPOCH: 372/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:01:33)   --> TIME: 2025-02-25 12:01:41 -- STEP: 15/72 -- GLOBAL_STEP: 1026800 | > loss_disc: 2.749521255493164 (2.7953977584838867) | > loss_disc_real_0: 0.15875259041786194 (0.18338682154814404) | > loss_disc_real_1: 0.24978354573249817 (0.23979407250881196) | > loss_disc_real_2: 0.24039354920387268 (0.24739602009455364) | > loss_disc_real_3: 0.24817585945129395 (0.25672857065995536) | > loss_disc_real_4: 0.24451594054698944 (0.25005902846654254) | > loss_disc_real_5: 0.25158360600471497 (0.2470642884572347) | > loss_0: 2.749521255493164 (2.7953977584838867) | > grad_norm_0: tensor(16.6226, device='cuda:0') (tensor(18.5781, device='cuda:0')) | > loss_gen: 2.063807487487793 (1.8525143305460612) | > loss_kl: 2.0585885047912598 (2.1628906249999997) | > loss_feat: 2.6557724475860596 (2.72983988126119) | > loss_mel: 17.84307289123535 (17.650541178385417) | > loss_duration: 1.5504519939422607 (1.5663420677185058) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.171693801879883 (25.96212819417318) | > grad_norm_1: tensor(293.4822, device='cuda:0') (tensor(441.2842, device='cuda:0')) | > current_lr_0: 0.00019091235722790576 | > current_lr_1: 0.00019091235722790576 | > step_time: 0.4327 (0.43857107162475584) | > loader_time: 0.0059 (0.0065755049387613935)  --> TIME: 2025-02-25 12:02:05 -- STEP: 65/72 -- GLOBAL_STEP: 1026850 | > loss_disc: 2.8972983360290527 (2.7979436214153575) | > loss_disc_real_0: 0.41549164056777954 (0.20320228223617257) | > loss_disc_real_1: 0.139517143368721 (0.2399054751946376) | > loss_disc_real_2: 0.26537519693374634 (0.2417083061658419) | > loss_disc_real_3: 0.24223436415195465 (0.24404345338161176) | > loss_disc_real_4: 0.24518975615501404 (0.244634682398576) | > loss_disc_real_5: 0.24200531840324402 (0.24591553715559153) | > loss_0: 2.8972983360290527 (2.7979436214153575) | > grad_norm_0: tensor(46.4816, device='cuda:0') (tensor(18.3400, device='cuda:0')) | > loss_gen: 1.8552637100219727 (1.8481096451099102) | > loss_kl: 2.4064245223999023 (2.166595853292026) | > loss_feat: 2.327000379562378 (2.7338016693408673) | > loss_mel: 18.193187713623047 (17.752436916644758) | > loss_duration: 1.661856770515442 (1.599612650504479) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.44373321533203 (26.100556799081655) | > grad_norm_1: tensor(130.7349, device='cuda:0') (tensor(420.3300, device='cuda:0')) | > current_lr_0: 0.00019091235722790576 | > current_lr_1: 0.00019091235722790576 | > step_time: 0.4957 (0.45182351699242224) | > loader_time: 0.0117 (0.008278740369356594) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555805179434762 (-0.00013061308524977763) | > avg_step_time: 0.45796819807777944 (-0.0006052473901023148) | > avg_loss_disc: 2.800138379486513 (+0.02264212890410011) | > avg_loss_disc_real_0: 0.2042724250907629 (+0.0033857209581724113) | > avg_loss_disc_real_1: 0.23810876381229346 (+0.006280534284215572) | > avg_loss_disc_real_2: 0.24276282690780263 (+0.004624566141988168) | > avg_loss_disc_real_3: 0.24305668767069427 (+0.005585902173754209) | > avg_loss_disc_real_4: 0.24462395759535507 (-0.0003189811824073352) | > avg_loss_disc_real_5: 0.24530648324691073 (-0.0008800795800249506) | > avg_loss_0: 2.800138379486513 (+0.02264212890410011) | > avg_grad_norm_0: tensor(18.3550, device='cuda:0') (+tensor(5.0932, device='cuda:0')) | > avg_loss_gen: 1.8403757491581876 (-0.008838060876013687) | > avg_loss_kl: 2.174573782464149 (-0.17206536716138743) | > avg_loss_feat: 2.6895309585920524 (-0.05880280111877001) | > avg_loss_mel: 17.67086900791652 (-0.5807804994180188) | > avg_loss_duration: 1.6046262039265162 (-0.011121194127579725) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.97997576753858 (-0.8316077917394509) | > avg_grad_norm_1: tensor(413.1288, device='cuda:0') (+tensor(9.9066, device='cuda:0'))  > EPOCH: 373/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:02:10)   --> TIME: 2025-02-25 12:02:31 -- STEP: 43/72 -- GLOBAL_STEP: 1026900 | > loss_disc: 2.975069522857666 (2.782785698424938) | > loss_disc_real_0: 0.24864453077316284 (0.20537001587623774) | > loss_disc_real_1: 0.19705422222614288 (0.22810766343460526) | > loss_disc_real_2: 0.20866738259792328 (0.23625618011452432) | > loss_disc_real_3: 0.26562657952308655 (0.23796062102151472) | > loss_disc_real_4: 0.23800350725650787 (0.24299308969530947) | > loss_disc_real_5: 0.2399536818265915 (0.24700726395429567) | > loss_0: 2.975069522857666 (2.782785698424938) | > grad_norm_0: tensor(7.1552, device='cuda:0') (tensor(14.5647, device='cuda:0')) | > loss_gen: 1.7234517335891724 (1.846771198649739) | > loss_kl: 2.2020483016967773 (2.1683281161064336) | > loss_feat: 2.7176895141601562 (2.7894591120786445) | > loss_mel: 17.90317153930664 (17.749065709668535) | > loss_duration: 1.5980188846588135 (1.5806795358657837) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.144380569458008 (26.13430351434752) | > grad_norm_1: tensor(472.1105, device='cuda:0') (tensor(452.4904, device='cuda:0')) | > current_lr_0: 0.00019088849318325227 | > current_lr_1: 0.00019088849318325227 | > step_time: 0.4543 (0.4404349826103033) | > loader_time: 0.0094 (0.007393071817797284) --> EVAL PERFORMANCE | > avg_loader_time: 0.008587803639156717 (+3.1998459721955044e-05) | > avg_step_time: 0.4556635467099472 (-0.0023046513678322555) | > avg_loss_disc: 2.7757315434200662 (-0.02440683606644667) | > avg_loss_disc_real_0: 0.20422765916921723 (-4.476592154567527e-05) | > avg_loss_disc_real_1: 0.23102596612043783 (-0.00708279769185563) | > avg_loss_disc_real_2: 0.23708511595155152 (-0.0056777109562511086) | > avg_loss_disc_real_3: 0.23664161667857372 (-0.006415070992120547) | > avg_loss_disc_real_4: 0.24314639035245061 (-0.0014775672429044595) | > avg_loss_disc_real_5: 0.2476310096156429 (+0.002324526368732177) | > avg_loss_0: 2.7757315434200662 (-0.02440683606644667) | > avg_grad_norm_0: tensor(15.0736, device='cuda:0') (tensor(-3.2814, device='cuda:0')) | > avg_loss_gen: 1.8518149869542726 (+0.011439237796085022) | > avg_loss_kl: 2.1982060704432755 (+0.02363228797912642) | > avg_loss_feat: 2.72995203649494 (+0.040421077902887514) | > avg_loss_mel: 17.6653633654957 (-0.005505642420818191) | > avg_loss_duration: 1.605017022347786 (+0.0003908184212697563) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.050353466624944 (+0.07037769908636449) | > avg_grad_norm_1: tensor(459.8314, device='cuda:0') (+tensor(46.7026, device='cuda:0'))  > EPOCH: 374/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:02:47)   --> TIME: 2025-02-25 12:02:58 -- STEP: 21/72 -- GLOBAL_STEP: 1026950 | > loss_disc: 2.871242046356201 (2.7593341781979515) | > loss_disc_real_0: 0.2363227903842926 (0.1975552574509666) | > loss_disc_real_1: 0.24849043786525726 (0.22513938092050098) | > loss_disc_real_2: 0.22858449816703796 (0.2365817633413133) | > loss_disc_real_3: 0.22870084643363953 (0.24590109785397843) | > loss_disc_real_4: 0.23109298944473267 (0.24769119279725216) | > loss_disc_real_5: 0.24082741141319275 (0.24494847087633043) | > loss_0: 2.871242046356201 (2.7593341781979515) | > grad_norm_0: tensor(18.1653, device='cuda:0') (tensor(13.8039, device='cuda:0')) | > loss_gen: 1.865058422088623 (1.8846556402388073) | > loss_kl: 2.0901308059692383 (2.2134161449614025) | > loss_feat: 3.0540833473205566 (2.8945972465333485) | > loss_mel: 17.43560218811035 (17.81898998078846) | > loss_duration: 1.5883092880249023 (1.5797818445024037) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.033184051513672 (26.39144043695359) | > grad_norm_1: tensor(197.5160, device='cuda:0') (tensor(555.4767, device='cuda:0')) | > current_lr_0: 0.00019086463212160436 | > current_lr_1: 0.00019086463212160436 | > step_time: 0.444 (0.46052186829703196) | > loader_time: 0.007 (0.006837254478817894)  --> TIME: 2025-02-25 12:03:23 -- STEP: 71/72 -- GLOBAL_STEP: 1027000 | > loss_disc: 2.5406031608581543 (2.7745241615134226) | > loss_disc_real_0: 0.17404550313949585 (0.1987709483840096) | > loss_disc_real_1: 0.18376120924949646 (0.23103133158784517) | > loss_disc_real_2: 0.21409083902835846 (0.2408840494676375) | > loss_disc_real_3: 0.20106898248195648 (0.24288627799128143) | > loss_disc_real_4: 0.22706113755702972 (0.24384373006686358) | > loss_disc_real_5: 0.23835663497447968 (0.24654307478750256) | > loss_0: 2.5406031608581543 (2.7745241615134226) | > grad_norm_0: tensor(6.0130, device='cuda:0') (tensor(15.2951, device='cuda:0')) | > loss_gen: 2.2029378414154053 (1.8550858329719222) | > loss_kl: 2.226459503173828 (2.249759180445067) | > loss_feat: 3.515106678009033 (2.799007482931648) | > loss_mel: 18.353153228759766 (17.71800923683273) | > loss_duration: 1.624474287033081 (1.605726955642163) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.922130584716797 (26.22758843865193) | > grad_norm_1: tensor(998.7385, device='cuda:0') (tensor(420.1066, device='cuda:0')) | > current_lr_0: 0.00019086463212160436 | > current_lr_1: 0.00019086463212160436 | > step_time: 0.4913 (0.46352135631400093) | > loader_time: 0.0083 (0.008696784435863227) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696784435863227 (+0.00010898079670650963) | > avg_step_time: 0.46352135631400093 (+0.007857809604053745) | > avg_loss_disc: 2.7745241615134226 (-0.001207381906643601) | > avg_loss_disc_real_0: 0.1987709483840096 (-0.005456710785207625) | > avg_loss_disc_real_1: 0.23103133158784517 (+5.365467407342228e-06) | > avg_loss_disc_real_2: 0.2408840494676375 (+0.003798933516085984) | > avg_loss_disc_real_3: 0.24288627799128143 (+0.0062446613127077055) | > avg_loss_disc_real_4: 0.24384373006686358 (+0.0006973397144129645) | > avg_loss_disc_real_5: 0.24654307478750256 (-0.0010879348281403467) | > avg_loss_0: 2.7745241615134226 (-0.001207381906643601) | > avg_grad_norm_0: tensor(15.2951, device='cuda:0') (+tensor(0.2215, device='cuda:0')) | > avg_loss_gen: 1.8550858329719222 (+0.003270846017649598) | > avg_loss_kl: 2.249759180445067 (+0.05155311000179141) | > avg_loss_feat: 2.799007482931648 (+0.06905544643670813) | > avg_loss_mel: 17.71800923683273 (+0.052645871337031025) | > avg_loss_duration: 1.605726955642163 (+0.0007099332943769632) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.22758843865193 (+0.17723497202698724) | > avg_grad_norm_1: tensor(420.1066, device='cuda:0') (tensor(-39.7248, device='cuda:0'))  > EPOCH: 375/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:03:24)   --> TIME: 2025-02-25 12:03:49 -- STEP: 49/72 -- GLOBAL_STEP: 1027050 | > loss_disc: 2.6911206245422363 (2.738797075894414) | > loss_disc_real_0: 0.11331869661808014 (0.19316059046862077) | > loss_disc_real_1: 0.24894528090953827 (0.22974280708906603) | > loss_disc_real_2: 0.23852351307868958 (0.23916639570070772) | > loss_disc_real_3: 0.25741514563560486 (0.23590707961393861) | > loss_disc_real_4: 0.23751692473888397 (0.24200727805799369) | > loss_disc_real_5: 0.23782594501972198 (0.24281579620984137) | > loss_0: 2.6911206245422363 (2.738797075894414) | > grad_norm_0: tensor(16.3519, device='cuda:0') (tensor(17.1249, device='cuda:0')) | > loss_gen: 1.8453755378723145 (1.886767282777903) | > loss_kl: 2.115933895111084 (2.1465761564215846) | > loss_feat: 2.5743534564971924 (2.9190464214402803) | > loss_mel: 16.927377700805664 (17.737001925098657) | > loss_duration: 1.6697134971618652 (1.5858525013437077) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.132753372192383 (26.275244226261062) | > grad_norm_1: tensor(569.2737, device='cuda:0') (tensor(821.1597, device='cuda:0')) | > current_lr_0: 0.00019084077404258915 | > current_lr_1: 0.00019084077404258915 | > step_time: 0.4443 (0.4454193796430315) | > loader_time: 0.0098 (0.007695694358981386) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624362273955011 (-7.242216190821507e-05) | > avg_step_time: 0.45822931007600165 (-0.005292046237999282) | > avg_loss_disc: 2.7398565890083844 (-0.03466757250503827) | > avg_loss_disc_real_0: 0.19446983408760016 (-0.004301114296409447) | > avg_loss_disc_real_1: 0.22932684400551756 (-0.0017044875823276073) | > avg_loss_disc_real_2: 0.23695186116326022 (-0.0039321883043772765) | > avg_loss_disc_real_3: 0.2360052680465537 (-0.006881009944727734) | > avg_loss_disc_real_4: 0.24232238153336752 (-0.0015213485334960586) | > avg_loss_disc_real_5: 0.24415219165909458 (-0.002390883128407989) | > avg_loss_0: 2.7398565890083844 (-0.03466757250503827) | > avg_grad_norm_0: tensor(16.5460, device='cuda:0') (+tensor(1.2509, device='cuda:0')) | > avg_loss_gen: 1.8753950394375223 (+0.020309206465600083) | > avg_loss_kl: 2.168025792484553 (-0.08173338796051377) | > avg_loss_feat: 2.865298996508961 (+0.06629151357731278) | > avg_loss_mel: 17.60856308735593 (-0.10944614947680265) | > avg_loss_duration: 1.6002696483907566 (-0.0054573072514063625) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.117552555782694 (-0.11003588286923716) | > avg_grad_norm_1: tensor(844.9101, device='cuda:0') (+tensor(424.8035, device='cuda:0'))  > EPOCH: 376/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:04:01)   --> TIME: 2025-02-25 12:04:15 -- STEP: 27/72 -- GLOBAL_STEP: 1027100 | > loss_disc: 2.774385452270508 (2.746923570279722) | > loss_disc_real_0: 0.19015517830848694 (0.18885417558528758) | > loss_disc_real_1: 0.24204343557357788 (0.23696301601551198) | > loss_disc_real_2: 0.2550460696220398 (0.23336232039663526) | > loss_disc_real_3: 0.2286483198404312 (0.23773603251686803) | > loss_disc_real_4: 0.24496756494045258 (0.24294015489242696) | > loss_disc_real_5: 0.2432001531124115 (0.24458016675931435) | > loss_0: 2.774385452270508 (2.746923570279722) | > grad_norm_0: tensor(5.7204, device='cuda:0') (tensor(17.7839, device='cuda:0')) | > loss_gen: 1.7811554670333862 (1.8751325386541862) | > loss_kl: 2.061375379562378 (2.1434925220630796) | > loss_feat: 3.1204915046691895 (2.9246289200252957) | > loss_mel: 16.61398696899414 (17.514026535881893) | > loss_duration: 1.5439395904541016 (1.5682041291837339) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.120946884155273 (26.025484438295717) | > grad_norm_1: tensor(347.7872, device='cuda:0') (tensor(671.8519, device='cuda:0')) | > current_lr_0: 0.00019081691894583383 | > current_lr_1: 0.00019081691894583383 | > step_time: 0.4349 (0.43567754604198317) | > loader_time: 0.0081 (0.006854852040608724) --> EVAL PERFORMANCE | > avg_loader_time: 0.00856354538823517 (-6.0816885719841215e-05) | > avg_step_time: 0.45594950125251016 (-0.0022798088234914915) | > avg_loss_disc: 2.7624961087401485 (+0.022639519731764146) | > avg_loss_disc_real_0: 0.19850341388037507 (+0.004033579792774911) | > avg_loss_disc_real_1: 0.23413110975648316 (+0.004804265750965597) | > avg_loss_disc_real_2: 0.2413357974777759 (+0.0043839363145156685) | > avg_loss_disc_real_3: 0.24090933652830795 (+0.004904068481754259) | > avg_loss_disc_real_4: 0.24558240888823926 (+0.0032600273548717418) | > avg_loss_disc_real_5: 0.24771937329164692 (+0.0035671816325523464) | > avg_loss_0: 2.7624961087401485 (+0.022639519731764146) | > avg_grad_norm_0: tensor(17.5519, device='cuda:0') (+tensor(1.0059, device='cuda:0')) | > avg_loss_gen: 1.888292260572944 (+0.012897221135421688) | > avg_loss_kl: 2.2544826957541453 (+0.0864569032695921) | > avg_loss_feat: 2.86151441721849 (-0.003784579290470713) | > avg_loss_mel: 17.891855307028326 (+0.28329221967239704) | > avg_loss_duration: 1.6085830423193919 (+0.008313393928635282) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.504727591930983 (+0.38717503614828885) | > avg_grad_norm_1: tensor(644.9463, device='cuda:0') (tensor(-199.9638, device='cuda:0'))  > EPOCH: 377/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:04:39)   --> TIME: 2025-02-25 12:04:43 -- STEP: 5/72 -- GLOBAL_STEP: 1027150 | > loss_disc: 2.8688864707946777 (2.7101228713989256) | > loss_disc_real_0: 0.14455199241638184 (0.1923386424779892) | > loss_disc_real_1: 0.22257214784622192 (0.23273456394672393) | > loss_disc_real_2: 0.22261731326580048 (0.22055187225341796) | > loss_disc_real_3: 0.2406126707792282 (0.2442850261926651) | > loss_disc_real_4: 0.2473587691783905 (0.2410336285829544) | > loss_disc_real_5: 0.24384678900241852 (0.23765983581542968) | > loss_0: 2.8688864707946777 (2.7101228713989256) | > grad_norm_0: tensor(15.3144, device='cuda:0') (tensor(11.1090, device='cuda:0')) | > loss_gen: 1.685598373413086 (1.8504669427871705) | > loss_kl: 2.305065870285034 (2.2777114868164063) | > loss_feat: 2.507067918777466 (2.8446232795715334) | > loss_mel: 17.643850326538086 (18.325911712646484) | > loss_duration: 1.5588922500610352 (1.5858527421951294) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.70047378540039 (26.88456573486328) | > grad_norm_1: tensor(226.8243, device='cuda:0') (tensor(446.4536, device='cuda:0')) | > current_lr_0: 0.0001907930668309656 | > current_lr_1: 0.0001907930668309656 | > step_time: 0.4207 (0.42880797386169434) | > loader_time: 0.0083 (0.007281446456909179)  --> TIME: 2025-02-25 12:05:06 -- STEP: 55/72 -- GLOBAL_STEP: 1027200 | > loss_disc: 2.7427632808685303 (2.748002039302479) | > loss_disc_real_0: 0.1293134093284607 (0.1860092279585925) | > loss_disc_real_1: 0.2791105806827545 (0.22788415524092587) | > loss_disc_real_2: 0.24384795129299164 (0.23405494689941406) | > loss_disc_real_3: 0.23685134947299957 (0.2357502064921639) | > loss_disc_real_4: 0.2344973087310791 (0.24395583759654652) | > loss_disc_real_5: 0.25841468572616577 (0.24695187481966885) | > loss_0: 2.7427632808685303 (2.748002039302479) | > grad_norm_0: tensor(23.7330, device='cuda:0') (tensor(17.6050, device='cuda:0')) | > loss_gen: 1.8450946807861328 (1.8592360496520997) | > loss_kl: 2.1387181282043457 (2.252630840648305) | > loss_feat: 2.8045976161956787 (2.8512917735359884) | > loss_mel: 16.94862174987793 (18.159029943292794) | > loss_duration: 1.6512956619262695 (1.6068778796629473) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.388328552246094 (26.729066397927024) | > grad_norm_1: tensor(676.3633, device='cuda:0') (tensor(852.1914, device='cuda:0')) | > current_lr_0: 0.0001907930668309656 | > current_lr_1: 0.0001907930668309656 | > step_time: 0.4735 (0.44771966500715776) | > loader_time: 0.0092 (0.008076555078679866) --> EVAL PERFORMANCE | > avg_loader_time: 0.008767719000158176 (+0.00020417361192300552) | > avg_step_time: 0.45859937264885703 (+0.00264987139634687) | > avg_loss_disc: 2.7472373062456157 (-0.01525880249453282) | > avg_loss_disc_real_0: 0.18784437905734694 (-0.01065903482302813) | > avg_loss_disc_real_1: 0.22677684501862863 (-0.007354264737854527) | > avg_loss_disc_real_2: 0.2344941408701346 (-0.0068416566076413) | > avg_loss_disc_real_3: 0.23517336937743175 (-0.005735967150876209) | > avg_loss_disc_real_4: 0.24303545666412568 (-0.00254695222411358) | > avg_loss_disc_real_5: 0.24716192903652998 (-0.0005574442551169456) | > avg_loss_0: 2.7472373062456157 (-0.01525880249453282) | > avg_grad_norm_0: tensor(18.9133, device='cuda:0') (+tensor(1.3614, device='cuda:0')) | > avg_loss_gen: 1.8528371898221299 (-0.03545507075081411) | > avg_loss_kl: 2.253706888413766 (-0.0007758073403794086) | > avg_loss_feat: 2.823849251572515 (-0.03766516564597522) | > avg_loss_mel: 18.088307340380172 (+0.19645203335184647) | > avg_loss_duration: 1.6169967248406212 (+0.008413682521229315) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.63569737823916 (+0.13096978630817802) | > avg_grad_norm_1: tensor(871.5582, device='cuda:0') (+tensor(226.6119, device='cuda:0'))  > EPOCH: 378/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:05:16)   --> TIME: 2025-02-25 12:05:33 -- STEP: 33/72 -- GLOBAL_STEP: 1027250 | > loss_disc: 2.9790451526641846 (2.7262720772714326) | > loss_disc_real_0: 0.05672699213027954 (0.1919323867469123) | > loss_disc_real_1: 0.19184610247612 (0.22181580102804935) | > loss_disc_real_2: 0.24971170723438263 (0.23566446204980215) | > loss_disc_real_3: 0.1981523334980011 (0.23474178079402808) | > loss_disc_real_4: 0.24008595943450928 (0.2417161211822972) | > loss_disc_real_5: 0.22682198882102966 (0.24308523309953284) | > loss_0: 2.9790451526641846 (2.7262720772714326) | > grad_norm_0: tensor(54.2046, device='cuda:0') (tensor(17.6666, device='cuda:0')) | > loss_gen: 1.5540717840194702 (1.9040471640500156) | > loss_kl: 2.1873867511749268 (2.211464589292353) | > loss_feat: 3.181462287902832 (3.034307848323475) | > loss_mel: 18.890296936035156 (18.14877680576209) | > loss_duration: 1.5831791162490845 (1.5850113485798691) | > amp_scaler: 64.0 (100.84848484848484) | > loss_1: 27.39639663696289 (26.88360763318611) | > grad_norm_1: tensor(974.9280, device='cuda:0') (tensor(591.0721, device='cuda:0')) | > current_lr_0: 0.0001907692176976117 | > current_lr_1: 0.0001907692176976117 | > step_time: 0.4377 (0.43588905623464874) | > loader_time: 0.0077 (0.007154334675181995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624852543145838 (-0.00014286645701233787) | > avg_step_time: 0.45310247448128715 (-0.005496898167569875) | > avg_loss_disc: 2.777209752042528 (+0.02997244579691216) | > avg_loss_disc_real_0: 0.20208546970511826 (+0.01424109064777132) | > avg_loss_disc_real_1: 0.2334207810566459 (+0.006643936038017273) | > avg_loss_disc_real_2: 0.2437308516720651 (+0.009236710801930503) | > avg_loss_disc_real_3: 0.243818933695135 (+0.008645564317703247) | > avg_loss_disc_real_4: 0.24343707087174268 (+0.0004016142076170015) | > avg_loss_disc_real_5: 0.2442986165553751 (-0.0028633124811548727) | > avg_loss_0: 2.777209752042528 (+0.02997244579691216) | > avg_grad_norm_0: tensor(17.4988, device='cuda:0') (tensor(-1.4145, device='cuda:0')) | > avg_loss_gen: 1.8844391359409816 (+0.0316019461188517) | > avg_loss_kl: 2.265330370043366 (+0.011623481629599919) | > avg_loss_feat: 2.8564297917863013 (+0.032580540213786424) | > avg_loss_mel: 18.066007949936555 (-0.022299390443617284) | > avg_loss_duration: 1.6206226936528383 (+0.0036259688122171063) | > avg_amp_scaler: 81.12676056338027 (-46.87323943661973) | > avg_loss_1: 26.692830045458297 (+0.057132667219136124) | > avg_grad_norm_1: tensor(512.0201, device='cuda:0') (tensor(-359.5381, device='cuda:0'))  > EPOCH: 379/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:05:53)   --> TIME: 2025-02-25 12:05:59 -- STEP: 11/72 -- GLOBAL_STEP: 1027300 | > loss_disc: 2.7999684810638428 (2.6430051543495874) | > loss_disc_real_0: 0.21883223950862885 (0.196177752180533) | > loss_disc_real_1: 0.3732362687587738 (0.22255655174905603) | > loss_disc_real_2: 0.22583581507205963 (0.23658001964742487) | > loss_disc_real_3: 0.24802318215370178 (0.21235116909850726) | > loss_disc_real_4: 0.2461967170238495 (0.24678047678687356) | > loss_disc_real_5: 0.26055771112442017 (0.23271815207871524) | > loss_0: 2.7999684810638428 (2.6430051543495874) | > grad_norm_0: tensor(17.9598, device='cuda:0') (tensor(15.7307, device='cuda:0')) | > loss_gen: 2.0643434524536133 (2.032077756795016) | > loss_kl: 1.8782179355621338 (2.165652719410983) | > loss_feat: 2.755150079727173 (3.1427740400487725) | > loss_mel: 16.69243621826172 (18.19420623779297) | > loss_duration: 1.56662118434906 (1.5816296989267522) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.956769943237305 (27.11634029041637) | > grad_norm_1: tensor(790.6725, device='cuda:0') (tensor(662.5451, device='cuda:0')) | > current_lr_0: 0.00019074537154539949 | > current_lr_1: 0.00019074537154539949 | > step_time: 0.4262 (0.43101965297352185) | > loader_time: 0.0058 (0.006557594646107067)  --> TIME: 2025-02-25 12:06:23 -- STEP: 61/72 -- GLOBAL_STEP: 1027350 | > loss_disc: 2.874744176864624 (2.775028326472298) | > loss_disc_real_0: 0.13057732582092285 (0.20797076344978613) | > loss_disc_real_1: 0.31265750527381897 (0.2348985268932874) | > loss_disc_real_2: 0.24674153327941895 (0.23724519888885687) | > loss_disc_real_3: 0.24039900302886963 (0.23751699558047) | > loss_disc_real_4: 0.26419520378112793 (0.2469356145526542) | > loss_disc_real_5: 0.26886802911758423 (0.24706369467446063) | > loss_0: 2.874744176864624 (2.775028326472298) | > grad_norm_0: tensor(40.4146, device='cuda:0') (tensor(16.4239, device='cuda:0')) | > loss_gen: 1.7763338088989258 (1.8908689647424417) | > loss_kl: 2.353013515472412 (2.208077327149815) | > loss_feat: 2.511840581893921 (2.847070002164997) | > loss_mel: 17.14641761779785 (17.704537594904668) | > loss_duration: 1.6631680727005005 (1.6001210251792533) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.450773239135742 (26.25067473239586) | > grad_norm_1: tensor(125.4211, device='cuda:0') (tensor(391.2676, device='cuda:0')) | > current_lr_0: 0.00019074537154539949 | > current_lr_1: 0.00019074537154539949 | > step_time: 0.4561 (0.4450650645084068) | > loader_time: 0.0106 (0.008208638331929191) --> EVAL PERFORMANCE | > avg_loader_time: 0.008657180087667117 (+3.232754452127916e-05) | > avg_step_time: 0.4513645675820364 (-0.0017379068992507407) | > avg_loss_disc: 2.7784084400660554 (+0.0011986880235275166) | > avg_loss_disc_real_0: 0.208271265554596 (+0.006185795849477732) | > avg_loss_disc_real_1: 0.23506890090418534 (+0.001648119847539431) | > avg_loss_disc_real_2: 0.2380170077085495 (-0.005713843963515597) | > avg_loss_disc_real_3: 0.23787148905472016 (-0.0059474446404148285) | > avg_loss_disc_real_4: 0.24633279085998805 (+0.0028957199882453655) | > avg_loss_disc_real_5: 0.2468581846062566 (+0.0025595680508815055) | > avg_loss_0: 2.7784084400660554 (+0.0011986880235275166) | > avg_grad_norm_0: tensor(16.7274, device='cuda:0') (tensor(-0.7714, device='cuda:0')) | > avg_loss_gen: 1.8781587238043127 (-0.006280412136668856) | > avg_loss_kl: 2.226133479198941 (-0.0391968908444249) | > avg_loss_feat: 2.807161824803957 (-0.04926796698234437) | > avg_loss_mel: 17.683121828965742 (-0.382886120970813) | > avg_loss_duration: 1.6077367453507974 (-0.012885948302040928) | > avg_amp_scaler: 64.0 (-17.126760563380273) | > avg_loss_1: 26.202312469482422 (-0.49051757597587553) | > avg_grad_norm_1: tensor(409.1980, device='cuda:0') (tensor(-102.8221, device='cuda:0'))  > EPOCH: 380/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:06:30)   --> TIME: 2025-02-25 12:06:49 -- STEP: 39/72 -- GLOBAL_STEP: 1027400 | > loss_disc: 2.6527984142303467 (2.761226611259656) | > loss_disc_real_0: 0.15202395617961884 (0.20460550467937422) | > loss_disc_real_1: 0.23192180693149567 (0.23515920761304024) | > loss_disc_real_2: 0.25537675619125366 (0.23848335444927216) | > loss_disc_real_3: 0.2236909121274948 (0.23755478209409958) | > loss_disc_real_4: 0.22356876730918884 (0.24562778916114417) | > loss_disc_real_5: 0.2414848655462265 (0.24856191109388304) | > loss_0: 2.6527984142303467 (2.761226611259656) | > grad_norm_0: tensor(11.3553, device='cuda:0') (tensor(19.8404, device='cuda:0')) | > loss_gen: 1.8862981796264648 (1.8837770009652162) | > loss_kl: 2.1895432472229004 (2.15942810743283) | > loss_feat: 3.0193674564361572 (2.77902837900015) | > loss_mel: 17.551603317260742 (17.56419370113275) | > loss_duration: 1.595815896987915 (1.5776033890553007) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.24262809753418 (25.964030583699543) | > grad_norm_1: tensor(244.2298, device='cuda:0') (tensor(518.0694, device='cuda:0')) | > current_lr_0: 0.0001907215283739563 | > current_lr_1: 0.0001907215283739563 | > step_time: 0.4562 (0.43828507570120007) | > loader_time: 0.0084 (0.0071964325048984625) --> EVAL PERFORMANCE | > avg_loader_time: 0.008491153448400362 (-0.00016602663926675552) | > avg_step_time: 0.45604762896685536 (+0.004683061384818943) | > avg_loss_disc: 2.769507713720832 (-0.008900726345223209) | > avg_loss_disc_real_0: 0.20691106300538695 (-0.0013602025492090442) | > avg_loss_disc_real_1: 0.23470793076803986 (-0.000360970136145472) | > avg_loss_disc_real_2: 0.23878813524481277 (+0.0007711275362632664) | > avg_loss_disc_real_3: 0.2382709543469926 (+0.000399465292272444) | > avg_loss_disc_real_4: 0.24543079754836122 (-0.0009019933116268253) | > avg_loss_disc_real_5: 0.24709844589233398 (+0.00024026128607737562) | > avg_loss_0: 2.769507713720832 (-0.008900726345223209) | > avg_grad_norm_0: tensor(17.9826, device='cuda:0') (+tensor(1.2552, device='cuda:0')) | > avg_loss_gen: 1.8755952371677882 (-0.0025634866365245035) | > avg_loss_kl: 2.192243542469723 (-0.03388993672921803) | > avg_loss_feat: 2.7528139369588502 (-0.05434788784510669) | > avg_loss_mel: 17.59189671529851 (-0.09122511366723174) | > avg_loss_duration: 1.6032057295382862 (-0.004531015812511141) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.015755183260207 (-0.18655728622221446) | > avg_grad_norm_1: tensor(483.9825, device='cuda:0') (+tensor(74.7845, device='cuda:0'))  > EPOCH: 381/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:07:07)   --> TIME: 2025-02-25 12:07:16 -- STEP: 17/72 -- GLOBAL_STEP: 1027450 | > loss_disc: 2.7322280406951904 (2.7628718123716465) | > loss_disc_real_0: 0.21729601919651031 (0.19536296234411352) | > loss_disc_real_1: 0.25262659788131714 (0.22523290006553426) | > loss_disc_real_2: 0.24435420334339142 (0.2346108310362872) | > loss_disc_real_3: 0.23906613886356354 (0.23497841463369482) | > loss_disc_real_4: 0.23714664578437805 (0.24197865058394039) | > loss_disc_real_5: 0.24346007406711578 (0.24515188090941487) | > loss_0: 2.7322280406951904 (2.7628718123716465) | > grad_norm_0: tensor(10.5431, device='cuda:0') (tensor(13.2013, device='cuda:0')) | > loss_gen: 1.8506712913513184 (1.8516523066688986) | > loss_kl: 2.464487075805664 (2.1741863348904777) | > loss_feat: 2.976253032684326 (2.9453822865205654) | > loss_mel: 19.797651290893555 (18.531782038071576) | > loss_duration: 1.6037451028823853 (1.577401371563182) | > amp_scaler: 64.0 (64.0) | > loss_1: 28.692808151245117 (27.080404730404126) | > grad_norm_1: tensor(721.1539, device='cuda:0') (tensor(647.6884, device='cuda:0')) | > current_lr_0: 0.00019069768818290954 | > current_lr_1: 0.00019069768818290954 | > step_time: 0.4357 (0.4338699088377111) | > loader_time: 0.0064 (0.006760513081270105)  --> TIME: 2025-02-25 12:07:40 -- STEP: 67/72 -- GLOBAL_STEP: 1027500 | > loss_disc: 2.7104833126068115 (2.763533496144992) | > loss_disc_real_0: 0.2812838554382324 (0.20056907643577945) | > loss_disc_real_1: 0.18158817291259766 (0.22385809147980676) | > loss_disc_real_2: 0.2768811583518982 (0.236445666471524) | > loss_disc_real_3: 0.24579203128814697 (0.23635255934587165) | > loss_disc_real_4: 0.23488889634609222 (0.24355337624229603) | > loss_disc_real_5: 0.23563529551029205 (0.2452289230787932) | > loss_0: 2.7104833126068115 (2.763533496144992) | > grad_norm_0: tensor(21.8155, device='cuda:0') (tensor(15.8235, device='cuda:0')) | > loss_gen: 1.9924418926239014 (1.8600503971327598) | > loss_kl: 2.0624189376831055 (2.174296208282015) | > loss_feat: 2.7346062660217285 (2.862042544493035) | > loss_mel: 17.247169494628906 (17.98847174288621) | > loss_duration: 1.6739028692245483 (1.6070212250325218) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.710538864135742 (26.49188212494352) | > grad_norm_1: tensor(807.7459, device='cuda:0') (tensor(472.6703, device='cuda:0')) | > current_lr_0: 0.00019069768818290954 | > current_lr_1: 0.00019069768818290954 | > step_time: 0.5247 (0.45327763059246007) | > loader_time: 0.0116 (0.00838275809786213) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555583550896443 (+6.44301024960818e-05) | > avg_step_time: 0.45725331172137196 (+0.0012056827545166016) | > avg_loss_disc: 2.7628299148989397 (-0.0066777988218924555) | > avg_loss_disc_real_0: 0.2007311340159094 (-0.006179928989477557) | > avg_loss_disc_real_1: 0.22420900545909372 (-0.010498925308946139) | > avg_loss_disc_real_2: 0.23638547180404126 (-0.002402663440771502) | > avg_loss_disc_real_3: 0.23647206538999585 (-0.001798888956996758) | > avg_loss_disc_real_4: 0.2430915419064777 (-0.002339255641883531) | > avg_loss_disc_real_5: 0.24557729716032323 (-0.0015211487320107575) | > avg_loss_0: 2.7628299148989397 (-0.0066777988218924555) | > avg_grad_norm_0: tensor(15.7525, device='cuda:0') (tensor(-2.2301, device='cuda:0')) | > avg_loss_gen: 1.858479380607605 (-0.017115856560183218) | > avg_loss_kl: 2.185704530124933 (-0.0065390123447897075) | > avg_loss_feat: 2.857135091029422 (+0.10432115407057196) | > avg_loss_mel: 17.997599709201864 (+0.40570299390335407) | > avg_loss_duration: 1.609846617134524 (+0.006640887596237821) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.508765341530385 (+0.49301015827017736) | > avg_grad_norm_1: tensor(486.1531, device='cuda:0') (+tensor(2.1706, device='cuda:0'))  > EPOCH: 382/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:07:44)   --> TIME: 2025-02-25 12:08:06 -- STEP: 45/72 -- GLOBAL_STEP: 1027550 | > loss_disc: 2.699223279953003 (2.7209864881303583) | > loss_disc_real_0: 0.21416673064231873 (0.18811294453011618) | > loss_disc_real_1: 0.21000447869300842 (0.2279477149248123) | > loss_disc_real_2: 0.22964246571063995 (0.23791402611467574) | > loss_disc_real_3: 0.23478834331035614 (0.23615272210703955) | > loss_disc_real_4: 0.2378930300474167 (0.2446394513050715) | > loss_disc_real_5: 0.22951968014240265 (0.2437747511598799) | > loss_0: 2.699223279953003 (2.7209864881303583) | > grad_norm_0: tensor(8.0235, device='cuda:0') (tensor(14.6255, device='cuda:0')) | > loss_gen: 1.8134846687316895 (1.900615898768107) | > loss_kl: 2.3868343830108643 (2.0795552253723146) | > loss_feat: 2.9536216259002686 (2.9755506727430556) | > loss_mel: 18.29752540588379 (17.788621881273063) | > loss_duration: 1.618748426437378 (1.5766718758477105) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.070213317871094 (26.321015379163953) | > grad_norm_1: tensor(411.3672, device='cuda:0') (tensor(715.4913, device='cuda:0')) | > current_lr_0: 0.00019067385097188666 | > current_lr_1: 0.00019067385097188666 | > step_time: 0.459 (0.441812006632487) | > loader_time: 0.009 (0.007513025071885851) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647633270478584 (+9.20497195821407e-05) | > avg_step_time: 0.45826681566910005 (+0.0010135039477280894) | > avg_loss_disc: 2.748560435335402 (-0.014269479563537857) | > avg_loss_disc_real_0: 0.19492237763085835 (-0.005808756385051045) | > avg_loss_disc_real_1: 0.2297568938262026 (+0.005547888367108872) | > avg_loss_disc_real_2: 0.24089273033847272 (+0.0045072585344314575) | > avg_loss_disc_real_3: 0.23844915586458126 (+0.0019770904745854134) | > avg_loss_disc_real_4: 0.24471480808627438 (+0.0016232661797966896) | > avg_loss_disc_real_5: 0.24568674828804715 (+0.00010945112772392518) | > avg_loss_0: 2.748560435335402 (-0.014269479563537857) | > avg_grad_norm_0: tensor(16.5508, device='cuda:0') (+tensor(0.7983, device='cuda:0')) | > avg_loss_gen: 1.8817515255699695 (+0.023272144962364516) | > avg_loss_kl: 2.134963268965064 (-0.05074126115986921) | > avg_loss_feat: 2.8680050708878206 (+0.010869979858398438) | > avg_loss_mel: 17.656548580653237 (-0.34105112854862796) | > avg_loss_duration: 1.5983615609961497 (-0.011485056138374361) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.139629793838722 (-0.3691355476916627) | > avg_grad_norm_1: tensor(635.7538, device='cuda:0') (+tensor(149.6007, device='cuda:0'))  > EPOCH: 383/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:08:21)   --> TIME: 2025-02-25 12:08:33 -- STEP: 23/72 -- GLOBAL_STEP: 1027600 | > loss_disc: 2.7364120483398438 (2.7584653004356054) | > loss_disc_real_0: 0.17223620414733887 (0.19801551427530206) | > loss_disc_real_1: 0.16828320920467377 (0.23037887332232102) | > loss_disc_real_2: 0.2656661570072174 (0.23730264798454617) | > loss_disc_real_3: 0.2550831437110901 (0.2415312638749247) | > loss_disc_real_4: 0.25159952044487 (0.24411942323912744) | > loss_disc_real_5: 0.25158342719078064 (0.24636376098446225) | > loss_0: 2.7364120483398438 (2.7584653004356054) | > grad_norm_0: tensor(10.8444, device='cuda:0') (tensor(12.6139, device='cuda:0')) | > loss_gen: 1.9504408836364746 (1.853722888490428) | > loss_kl: 2.2276153564453125 (2.148813952570376) | > loss_feat: 3.014517307281494 (2.8265160581339965) | > loss_mel: 18.93634033203125 (17.454027258831523) | > loss_duration: 1.6108298301696777 (1.5646957055382107) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.739744186401367 (25.847775832466457) | > grad_norm_1: tensor(382.1258, device='cuda:0') (tensor(537.5704, device='cuda:0')) | > current_lr_0: 0.00019065001674051517 | > current_lr_1: 0.00019065001674051517 | > step_time: 0.437 (0.43471614174220874) | > loader_time: 0.0066 (0.006800983263098676) --> EVAL PERFORMANCE | > avg_loader_time: 0.008595446465720592 (-5.218680475799209e-05) | > avg_step_time: 0.4609984579220624 (+0.002731642252962352) | > avg_loss_disc: 2.7729902502516626 (+0.024429814916260728) | > avg_loss_disc_real_0: 0.2042051103543228 (+0.009282732723464443) | > avg_loss_disc_real_1: 0.23304276105383753 (+0.0032858672276349365) | > avg_loss_disc_real_2: 0.24001100533445116 (-0.0008817250040215607) | > avg_loss_disc_real_3: 0.23879826971342866 (+0.00034911384884739327) | > avg_loss_disc_real_4: 0.24585565588843655 (+0.0011408478021621704) | > avg_loss_disc_real_5: 0.24640633375711843 (+0.0007195854690712766) | > avg_loss_0: 2.7729902502516626 (+0.024429814916260728) | > avg_grad_norm_0: tensor(14.6362, device='cuda:0') (tensor(-1.9146, device='cuda:0')) | > avg_loss_gen: 1.8476045736124818 (-0.03414695195748774) | > avg_loss_kl: 2.2601521888249354 (+0.12518891985987146) | > avg_loss_feat: 2.7347855332871562 (-0.1332195376006644) | > avg_loss_mel: 17.611755183045293 (-0.04479339760794332) | > avg_loss_duration: 1.604394064822667 (+0.0060325038265174324) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.058691293420924 (-0.0809385004177976) | > avg_grad_norm_1: tensor(407.8476, device='cuda:0') (tensor(-227.9062, device='cuda:0'))  > EPOCH: 384/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:08:58)   --> TIME: 2025-02-25 12:09:01 -- STEP: 1/72 -- GLOBAL_STEP: 1027650 | > loss_disc: 2.752640724182129 (2.752640724182129) | > loss_disc_real_0: 0.1791629195213318 (0.1791629195213318) | > loss_disc_real_1: 0.22321447730064392 (0.22321447730064392) | > loss_disc_real_2: 0.23812919855117798 (0.23812919855117798) | > loss_disc_real_3: 0.23094937205314636 (0.23094937205314636) | > loss_disc_real_4: 0.23071320354938507 (0.23071320354938507) | > loss_disc_real_5: 0.2374504655599594 (0.2374504655599594) | > loss_0: 2.752640724182129 (2.752640724182129) | > grad_norm_0: tensor(4.8643, device='cuda:0') (tensor(4.8643, device='cuda:0')) | > loss_gen: 1.9587420225143433 (1.9587420225143433) | > loss_kl: 2.123889207839966 (2.123889207839966) | > loss_feat: 2.9843199253082275 (2.9843199253082275) | > loss_mel: 19.20526885986328 (19.20526885986328) | > loss_duration: 1.6160629987716675 (1.6160629987716675) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.888282775878906 (27.888282775878906) | > grad_norm_1: tensor(793.9358, device='cuda:0') (tensor(793.9358, device='cuda:0')) | > current_lr_0: 0.0001906261854884226 | > current_lr_1: 0.0001906261854884226 | > step_time: 0.4426 (0.44256019592285156) | > loader_time: 0.0058 (0.005788087844848633)  --> TIME: 2025-02-25 12:09:24 -- STEP: 51/72 -- GLOBAL_STEP: 1027700 | > loss_disc: 2.786341667175293 (2.7453789664249793) | > loss_disc_real_0: 0.15722966194152832 (0.19754150466007345) | > loss_disc_real_1: 0.21277432143688202 (0.22875288976173774) | > loss_disc_real_2: 0.2379409223794937 (0.23521166630819731) | > loss_disc_real_3: 0.22155988216400146 (0.23849852879842123) | > loss_disc_real_4: 0.23994439840316772 (0.24605580959834306) | > loss_disc_real_5: 0.23514695465564728 (0.2501703725725997) | > loss_0: 2.786341667175293 (2.7453789664249793) | > grad_norm_0: tensor(8.2661, device='cuda:0') (tensor(17.4782, device='cuda:0')) | > loss_gen: 1.9979757070541382 (1.8991828736136942) | > loss_kl: 2.2849388122558594 (2.172649680399427) | > loss_feat: 2.626028060913086 (2.890437878814398) | > loss_mel: 17.681915283203125 (17.760549881879026) | > loss_duration: 1.6452929973602295 (1.5900387366612752) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.23615074157715 (26.312859179926853) | > grad_norm_1: tensor(769.7640, device='cuda:0') (tensor(734.1884, device='cuda:0')) | > current_lr_0: 0.0001906261854884226 | > current_lr_1: 0.0001906261854884226 | > step_time: 0.4434 (0.44467199082468073) | > loader_time: 0.0091 (0.00786437707788804) --> EVAL PERFORMANCE | > avg_loader_time: 0.008713309194000674 (+0.0001178627282800819) | > avg_step_time: 0.45702574286662356 (-0.003972715055438836) | > avg_loss_disc: 2.7600090100731647 (-0.012981240178497888) | > avg_loss_disc_real_0: 0.20217646584964136 (-0.0020286445046814316) | > avg_loss_disc_real_1: 0.2360255816872691 (+0.0029828206334315543) | > avg_loss_disc_real_2: 0.23637945320404752 (-0.0036315521304036424) | > avg_loss_disc_real_3: 0.23861500264053614 (-0.00018326707289251232) | > avg_loss_disc_real_4: 0.24493627963771283 (-0.0009193762507237191) | > avg_loss_disc_real_5: 0.24887016352633357 (+0.0024638297692151367) | > avg_loss_0: 2.7600090100731647 (-0.012981240178497888) | > avg_grad_norm_0: tensor(18.7946, device='cuda:0') (+tensor(4.1583, device='cuda:0')) | > avg_loss_gen: 1.8870584427470891 (+0.039453869134607356) | > avg_loss_kl: 2.1944641311403723 (-0.0656880576845631) | > avg_loss_feat: 2.799784374908662 (+0.06499884162150593) | > avg_loss_mel: 17.697540014562477 (+0.0857848315171843) | > avg_loss_duration: 1.6027232039142663 (-0.0016708609084008508) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.181570294877172 (+0.12287900145624775) | > avg_grad_norm_1: tensor(745.7590, device='cuda:0') (+tensor(337.9114, device='cuda:0'))  > EPOCH: 385/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:09:36)   --> TIME: 2025-02-25 12:09:50 -- STEP: 29/72 -- GLOBAL_STEP: 1027750 | > loss_disc: 2.7077488899230957 (2.7832757522319924) | > loss_disc_real_0: 0.16015781462192535 (0.2039564829962007) | > loss_disc_real_1: 0.1103692501783371 (0.22820099006439076) | > loss_disc_real_2: 0.22321581840515137 (0.256681995659039) | > loss_disc_real_3: 0.22253118455410004 (0.2429431435363046) | > loss_disc_real_4: 0.2253725230693817 (0.24758415900427713) | > loss_disc_real_5: 0.24749664962291718 (0.2492494665343186) | > loss_0: 2.7077488899230957 (2.7832757522319924) | > grad_norm_0: tensor(16.7769, device='cuda:0') (tensor(18.7437, device='cuda:0')) | > loss_gen: 1.6843552589416504 (1.9083773349893505) | > loss_kl: 2.1543421745300293 (2.158785125304912) | > loss_feat: 3.0490665435791016 (2.8534104330786345) | > loss_mel: 19.68569564819336 (17.88159873567778) | > loss_duration: 1.634041666984558 (1.5884250320237259) | > amp_scaler: 64.0 (64.0) | > loss_1: 28.207500457763672 (26.39059678439436) | > grad_norm_1: tensor(912.4866, device='cuda:0') (tensor(562.1730, device='cuda:0')) | > current_lr_0: 0.00019060235721523656 | > current_lr_1: 0.00019060235721523656 | > step_time: 0.4379 (0.4356487948319008) | > loader_time: 0.0075 (0.007067450161637931) --> EVAL PERFORMANCE | > avg_loader_time: 0.008731113353245695 (+1.780415924502142e-05) | > avg_step_time: 0.4546516411741015 (-0.002374101692522057) | > avg_loss_disc: 2.7750902243063487 (+0.015081214233183982) | > avg_loss_disc_real_0: 0.20182261752410674 (-0.000353848325534617) | > avg_loss_disc_real_1: 0.2300169589653821 (-0.006008622721886986) | > avg_loss_disc_real_2: 0.24570445422555354 (+0.009325001021506019) | > avg_loss_disc_real_3: 0.24029702976555892 (+0.0016820271250227725) | > avg_loss_disc_real_4: 0.24489800778912826 (-3.827184858457011e-05) | > avg_loss_disc_real_5: 0.24802512560092227 (-0.0008450379254112961) | > avg_loss_0: 2.7750902243063487 (+0.015081214233183982) | > avg_grad_norm_0: tensor(18.7401, device='cuda:0') (tensor(-0.0545, device='cuda:0')) | > avg_loss_gen: 1.8733354840480105 (-0.01372295869907858) | > avg_loss_kl: 2.242921501817836 (+0.04845737067746381) | > avg_loss_feat: 2.764170421680934 (-0.03561395322772798) | > avg_loss_mel: 17.87088653403269 (+0.1733465194702113) | > avg_loss_duration: 1.6228880647202615 (+0.02016486080599522) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.37420208353392 (+0.19263178865674746) | > avg_grad_norm_1: tensor(707.3350, device='cuda:0') (tensor(-38.4240, device='cuda:0'))  > EPOCH: 386/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:10:12)   --> TIME: 2025-02-25 12:10:17 -- STEP: 7/72 -- GLOBAL_STEP: 1027800 | > loss_disc: 2.7510852813720703 (2.7315310069492886) | > loss_disc_real_0: 0.19271057844161987 (0.20092090112822397) | > loss_disc_real_1: 0.16235733032226562 (0.22301353514194489) | > loss_disc_real_2: 0.25581103563308716 (0.21702427310603006) | > loss_disc_real_3: 0.264119029045105 (0.2310851365327835) | > loss_disc_real_4: 0.23042182624340057 (0.23854078778198787) | > loss_disc_real_5: 0.21569626033306122 (0.2335001208952495) | > loss_0: 2.7510852813720703 (2.7315310069492886) | > grad_norm_0: tensor(8.2835, device='cuda:0') (tensor(15.5690, device='cuda:0')) | > loss_gen: 1.9310698509216309 (1.8883989708764213) | > loss_kl: 2.3314945697784424 (2.255706105913435) | > loss_feat: 3.1005804538726807 (2.884458541870117) | > loss_mel: 18.263702392578125 (18.237825666155135) | > loss_duration: 1.5708410739898682 (1.580651777131217) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.197689056396484 (26.847041266305105) | > grad_norm_1: tensor(338.2233, device='cuda:0') (tensor(366.6618, device='cuda:0')) | > current_lr_0: 0.00019057853192058465 | > current_lr_1: 0.00019057853192058465 | > step_time: 0.439 (0.4336944988795689) | > loader_time: 0.0054 (0.006469590323311942)  --> TIME: 2025-02-25 12:10:40 -- STEP: 57/72 -- GLOBAL_STEP: 1027850 | > loss_disc: 2.7445671558380127 (2.7603980072757652) | > loss_disc_real_0: 0.24538759887218475 (0.20753374761133864) | > loss_disc_real_1: 0.2299884408712387 (0.2284624960861708) | > loss_disc_real_2: 0.23721149563789368 (0.232627500306096) | > loss_disc_real_3: 0.27130356431007385 (0.23821380446877397) | > loss_disc_real_4: 0.2561630308628082 (0.24426605670075668) | > loss_disc_real_5: 0.267119824886322 (0.2453437173052838) | > loss_0: 2.7445671558380127 (2.7603980072757652) | > grad_norm_0: tensor(23.9664, device='cuda:0') (tensor(19.6431, device='cuda:0')) | > loss_gen: 1.9521427154541016 (1.876045547033611) | > loss_kl: 2.3262743949890137 (2.2213268321857123) | > loss_feat: 2.673288106918335 (2.8233115212959157) | > loss_mel: 16.977943420410156 (17.61325352652031) | > loss_duration: 1.6273882389068604 (1.5916640298408375) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.557035446166992 (26.125601316753187) | > grad_norm_1: tensor(591.9962, device='cuda:0') (tensor(613.3073, device='cuda:0')) | > current_lr_0: 0.00019057853192058465 | > current_lr_1: 0.00019057853192058465 | > step_time: 0.452 (0.4433135233427349) | > loader_time: 0.0096 (0.007962917026720551) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626837125966248 (-0.00010427622727944737) | > avg_step_time: 0.4553178867823641 (+0.0006662456082626167) | > avg_loss_disc: 2.769146392043208 (-0.00594383226314088) | > avg_loss_disc_real_0: 0.20438122067233205 (+0.0025586031482253135) | > avg_loss_disc_real_1: 0.2288934074237313 (-0.001123551541650808) | > avg_loss_disc_real_2: 0.23404602160756016 (-0.011658432617993375) | > avg_loss_disc_real_3: 0.24010762572288513 (-0.0001894040426737853) | > avg_loss_disc_real_4: 0.24444744242748745 (-0.0004505653616408145) | > avg_loss_disc_real_5: 0.24580256284122737 (-0.0022225627596949016) | > avg_loss_0: 2.769146392043208 (-0.00594383226314088) | > avg_grad_norm_0: tensor(18.1529, device='cuda:0') (tensor(-0.5871, device='cuda:0')) | > avg_loss_gen: 1.8626109495968886 (-0.010724534451121981) | > avg_loss_kl: 2.24111127181792 (-0.001810229999916313) | > avg_loss_feat: 2.7837995542606837 (+0.01962913257974952) | > avg_loss_mel: 17.60336126408107 (-0.2675252699516193) | > avg_loss_duration: 1.601873730269956 (-0.02101433445030554) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.092756620595157 (-0.2814454629387626) | > avg_grad_norm_1: tensor(564.2960, device='cuda:0') (tensor(-143.0390, device='cuda:0'))  > EPOCH: 387/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:10:49)   --> TIME: 2025-02-25 12:11:07 -- STEP: 35/72 -- GLOBAL_STEP: 1027900 | > loss_disc: 2.7779507637023926 (2.773520994186402) | > loss_disc_real_0: 0.2150585800409317 (0.20213190700326647) | > loss_disc_real_1: 0.18306714296340942 (0.2313457352774484) | > loss_disc_real_2: 0.2257293313741684 (0.24210184855120523) | > loss_disc_real_3: 0.23704862594604492 (0.24013641944953373) | > loss_disc_real_4: 0.2530384361743927 (0.24373050587517875) | > loss_disc_real_5: 0.2717036306858063 (0.24605843552521298) | > loss_0: 2.7779507637023926 (2.773520994186402) | > grad_norm_0: tensor(7.9838, device='cuda:0') (tensor(16.9004, device='cuda:0')) | > loss_gen: 1.814239501953125 (1.8474367243903025) | > loss_kl: 2.391911745071411 (2.267009745325361) | > loss_feat: 2.678894519805908 (2.786631883893694) | > loss_mel: 17.138294219970703 (17.72393052237374) | > loss_duration: 1.613201379776001 (1.5686464616230555) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.63654136657715 (26.193655286516464) | > grad_norm_1: tensor(756.6682, device='cuda:0') (tensor(479.9103, device='cuda:0')) | > current_lr_0: 0.00019055470960409457 | > current_lr_1: 0.00019055470960409457 | > step_time: 0.4549 (0.4511821610586984) | > loader_time: 0.0102 (0.007361861637660436) --> EVAL PERFORMANCE | > avg_loader_time: 0.008702953096846459 (+7.611597088021116e-05) | > avg_step_time: 0.4621544723779383 (+0.006836585595574152) | > avg_loss_disc: 2.778791612302754 (+0.009645220259546061) | > avg_loss_disc_real_0: 0.20809290159336277 (+0.003711680921030719) | > avg_loss_disc_real_1: 0.23320449573892943 (+0.004311088315198136) | > avg_loss_disc_real_2: 0.23915115061780096 (+0.005105129010240794) | > avg_loss_disc_real_3: 0.23719801302527038 (-0.0029096126976147496) | > avg_loss_disc_real_4: 0.24445612560695326 (+8.683179465812652e-06) | > avg_loss_disc_real_5: 0.24736884865962283 (+0.001566285818395463) | > avg_loss_0: 2.778791612302754 (+0.009645220259546061) | > avg_grad_norm_0: tensor(17.8664, device='cuda:0') (tensor(-0.2865, device='cuda:0')) | > avg_loss_gen: 1.8541803477515637 (-0.008430601845324848) | > avg_loss_kl: 2.242272405557228 (+0.0011611337393082621) | > avg_loss_feat: 2.7305090024437706 (-0.05329055181691311) | > avg_loss_mel: 17.63945360586677 (+0.0360923417857002) | > avg_loss_duration: 1.5985980436835487 (-0.0032756865864072005) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.065013321352676 (-0.027743299242480646) | > avg_grad_norm_1: tensor(477.4248, device='cuda:0') (tensor(-86.8712, device='cuda:0'))  > EPOCH: 388/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:11:27)   --> TIME: 2025-02-25 12:11:34 -- STEP: 13/72 -- GLOBAL_STEP: 1027950 | > loss_disc: 2.83575439453125 (2.8222307425278883) | > loss_disc_real_0: 0.16569197177886963 (0.21379695316919914) | > loss_disc_real_1: 0.16127964854240417 (0.2493764952971385) | > loss_disc_real_2: 0.2555895447731018 (0.24999297696810502) | > loss_disc_real_3: 0.27728337049484253 (0.2511541327604881) | > loss_disc_real_4: 0.27238929271698 (0.24149069304649645) | > loss_disc_real_5: 0.2626533508300781 (0.24406856298446655) | > loss_0: 2.83575439453125 (2.8222307425278883) | > grad_norm_0: tensor(14.4546, device='cuda:0') (tensor(17.9519, device='cuda:0')) | > loss_gen: 1.6969542503356934 (1.8655914159921498) | > loss_kl: 2.1703505516052246 (2.1513326901655927) | > loss_feat: 2.4232075214385986 (2.834361718251155) | > loss_mel: 17.652151107788086 (18.05144221966083) | > loss_duration: 1.5505659580230713 (1.5752732845453115) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.493228912353516 (26.478001814622147) | > grad_norm_1: tensor(287.1566, device='cuda:0') (tensor(393.4210, device='cuda:0')) | > current_lr_0: 0.00019053089026539405 | > current_lr_1: 0.00019053089026539405 | > step_time: 0.4626 (0.4362942622258113) | > loader_time: 0.0072 (0.006491569372323843)  --> TIME: 2025-02-25 12:11:58 -- STEP: 63/72 -- GLOBAL_STEP: 1028000 | > loss_disc: 2.7298080921173096 (2.789272667869689) | > loss_disc_real_0: 0.243761345744133 (0.20017341879152115) | > loss_disc_real_1: 0.25500914454460144 (0.23502543024600497) | > loss_disc_real_2: 0.2212781459093094 (0.2405357483833555) | > loss_disc_real_3: 0.24891558289527893 (0.24569398637801884) | > loss_disc_real_4: 0.27233192324638367 (0.24678507896642837) | > loss_disc_real_5: 0.27850472927093506 (0.24696625698180424) | > loss_0: 2.7298080921173096 (2.789272667869689) | > grad_norm_0: tensor(13.1055, device='cuda:0') (tensor(17.8173, device='cuda:0')) | > loss_gen: 2.0637834072113037 (1.8469709460697477) | > loss_kl: 2.230212926864624 (2.215530306573898) | > loss_feat: 2.9590818881988525 (2.740873034038241) | > loss_mel: 18.32938003540039 (17.78205320570204) | > loss_duration: 1.6037551164627075 (1.601796719763014) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.186214447021484 (26.187224463810995) | > grad_norm_1: tensor(842.0876, device='cuda:0') (tensor(485.8955, device='cuda:0')) | > current_lr_0: 0.00019053089026539405 | > current_lr_1: 0.00019053089026539405 | > step_time: 0.4916 (0.4504910575018989) | > loader_time: 0.0119 (0.008184928742666093) --> EVAL PERFORMANCE | > avg_loader_time: 0.008565197528248102 (-0.00013775556859835746) | > avg_step_time: 0.45830728974140866 (-0.0038471826365296113) | > avg_loss_disc: 2.791043348715339 (+0.012251736412585146) | > avg_loss_disc_real_0: 0.20258361699295716 (-0.005509284600405617) | > avg_loss_disc_real_1: 0.2354454173588417 (+0.0022409216199122672) | > avg_loss_disc_real_2: 0.24194922770412874 (+0.002798077086327788) | > avg_loss_disc_real_3: 0.2434289663190573 (+0.006230953293786923) | > avg_loss_disc_real_4: 0.2466213799698252 (+0.0021652543628719323) | > avg_loss_disc_real_5: 0.2475983047149551 (+0.00022945605533225555) | > avg_loss_0: 2.791043348715339 (+0.012251736412585146) | > avg_grad_norm_0: tensor(18.2303, device='cuda:0') (+tensor(0.3639, device='cuda:0')) | > avg_loss_gen: 1.843190505471028 (-0.010989842280535678) | > avg_loss_kl: 2.2100815773010254 (-0.032190828256202675) | > avg_loss_feat: 2.724797225334275 (-0.005711777109495664) | > avg_loss_mel: 17.768548025211818 (+0.12909441934504784) | > avg_loss_duration: 1.6075856450577857 (+0.008987601374236931) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.154203226868535 (+0.08918990551585892) | > avg_grad_norm_1: tensor(494.1810, device='cuda:0') (+tensor(16.7562, device='cuda:0'))  > EPOCH: 389/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:12:04)   --> TIME: 2025-02-25 12:12:24 -- STEP: 41/72 -- GLOBAL_STEP: 1028050 | > loss_disc: 2.8110318183898926 (2.7812621186419237) | > loss_disc_real_0: 0.2672543525695801 (0.20153142857115444) | > loss_disc_real_1: 0.2890724837779999 (0.22504410402077) | > loss_disc_real_2: 0.22658251225948334 (0.24019675174864327) | > loss_disc_real_3: 0.2550163269042969 (0.2460158085677682) | > loss_disc_real_4: 0.25971975922584534 (0.24358057212538836) | > loss_disc_real_5: 0.262458860874176 (0.24361563055980495) | > loss_0: 2.8110318183898926 (2.7812621186419237) | > grad_norm_0: tensor(21.6136, device='cuda:0') (tensor(17.5462, device='cuda:0')) | > loss_gen: 2.0387067794799805 (1.8567894261057785) | > loss_kl: 2.0790584087371826 (2.171043454146967) | > loss_feat: 2.6333906650543213 (2.772738317163979) | > loss_mel: 17.465187072753906 (17.60197099825231) | > loss_duration: 1.5963956117630005 (1.5801686135733999) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.8127384185791 (25.982711001140316) | > grad_norm_1: tensor(59.7564, device='cuda:0') (tensor(442.5674, device='cuda:0')) | > current_lr_0: 0.00019050707390411086 | > current_lr_1: 0.00019050707390411086 | > step_time: 0.4461 (0.4401902512806218) | > loader_time: 0.0091 (0.00729967326652713) --> EVAL PERFORMANCE | > avg_loader_time: 0.008480441402381574 (-8.475612586652734e-05) | > avg_step_time: 0.4556161282767712 (-0.002691161464637437) | > avg_loss_disc: 2.7892905557659318 (-0.0017527929494072225) | > avg_loss_disc_real_0: 0.20637432160511823 (+0.0037907046121610777) | > avg_loss_disc_real_1: 0.22793472463815984 (-0.007510692720681861) | > avg_loss_disc_real_2: 0.23931242357677138 (-0.0026368041273573672) | > avg_loss_disc_real_3: 0.24393107962440436 (+0.0005021133053470517) | > avg_loss_disc_real_4: 0.24567109661203035 (-0.0009502833577948455) | > avg_loss_disc_real_5: 0.2462318929987894 (-0.0013664117161656941) | > avg_loss_0: 2.7892905557659318 (-0.0017527929494072225) | > avg_grad_norm_0: tensor(16.2457, device='cuda:0') (tensor(-1.9846, device='cuda:0')) | > avg_loss_gen: 1.8473319873003893 (+0.004141481829361249) | > avg_loss_kl: 2.2522322493539724 (+0.04215067205294698) | > avg_loss_feat: 2.710122454334313 (-0.01467477099996195) | > avg_loss_mel: 17.77061289129123 (+0.002064866079411587) | > avg_loss_duration: 1.6085923272119442 (+0.0010066821541585558) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.18889198840504 (+0.03468876153650413) | > avg_grad_norm_1: tensor(398.7130, device='cuda:0') (tensor(-95.4680, device='cuda:0'))  > EPOCH: 390/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:12:41)   --> TIME: 2025-02-25 12:12:51 -- STEP: 19/72 -- GLOBAL_STEP: 1028100 | > loss_disc: 2.8224682807922363 (2.75738253091511) | > loss_disc_real_0: 0.2745896577835083 (0.20587877380220512) | > loss_disc_real_1: 0.2192455679178238 (0.22563656458729192) | > loss_disc_real_2: 0.23967379331588745 (0.23935401988656899) | > loss_disc_real_3: 0.2306336760520935 (0.23531845055128398) | > loss_disc_real_4: 0.2475551813840866 (0.2393569240444585) | > loss_disc_real_5: 0.23731942474842072 (0.2421373873949051) | > loss_0: 2.8224682807922363 (2.75738253091511) | > grad_norm_0: tensor(10.2952, device='cuda:0') (tensor(14.2718, device='cuda:0')) | > loss_gen: 1.8738653659820557 (1.8778113377721686) | > loss_kl: 2.1939938068389893 (2.2000661711943774) | > loss_feat: 3.06510591506958 (2.815644063447651) | > loss_mel: 18.5113582611084 (18.017817447060033) | > loss_duration: 1.5486739873886108 (1.561391460268121) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.192996978759766 (26.472730435823138) | > grad_norm_1: tensor(873.2371, device='cuda:0') (tensor(439.9842, device='cuda:0')) | > current_lr_0: 0.00019048326051987284 | > current_lr_1: 0.00019048326051987284 | > step_time: 0.4242 (0.4345288527639289) | > loader_time: 0.0062 (0.0069282431351511105)  --> TIME: 2025-02-25 12:13:15 -- STEP: 69/72 -- GLOBAL_STEP: 1028150 | > loss_disc: 2.838514804840088 (2.7653595191844995) | > loss_disc_real_0: 0.20172935724258423 (0.20241829925689145) | > loss_disc_real_1: 0.26292920112609863 (0.23329102820244388) | > loss_disc_real_2: 0.25887641310691833 (0.2424810420775759) | > loss_disc_real_3: 0.2839720845222473 (0.2390867560238078) | > loss_disc_real_4: 0.27194973826408386 (0.24279331966586734) | > loss_disc_real_5: 0.2884376347064972 (0.24762142957120703) | > loss_0: 2.838514804840088 (2.7653595191844995) | > grad_norm_0: tensor(5.0076, device='cuda:0') (tensor(14.3518, device='cuda:0')) | > loss_gen: 1.9347363710403442 (1.875273094660994) | > loss_kl: 2.1760432720184326 (2.21899554176607) | > loss_feat: 2.5858500003814697 (2.7707082326861388) | > loss_mel: 17.2382869720459 (17.685982234236114) | > loss_duration: 1.7060825824737549 (1.5986405870188838) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.64099884033203 (26.149599655814793) | > grad_norm_1: tensor(617.8770, device='cuda:0') (tensor(488.3093, device='cuda:0')) | > current_lr_0: 0.00019048326051987284 | > current_lr_1: 0.00019048326051987284 | > step_time: 0.5192 (0.45164503221926483) | > loader_time: 0.0126 (0.008620431457740673) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680326837888906 (+0.00019988543550733193) | > avg_step_time: 0.4533338110211869 (-0.0022823172555843096) | > avg_loss_disc: 2.767852789919141 (-0.02143776584679058) | > avg_loss_disc_real_0: 0.20410112889719684 (-0.0022731927079213954) | > avg_loss_disc_real_1: 0.232099475575165 (+0.004164750937005163) | > avg_loss_disc_real_2: 0.2413511076863383 (+0.002038684109566924) | > avg_loss_disc_real_3: 0.2388533996024602 (-0.00507768002194417) | > avg_loss_disc_real_4: 0.24288794482258005 (-0.0027831517894502944) | > avg_loss_disc_real_5: 0.24807992577552795 (+0.0018480327767385618) | > avg_loss_0: 2.767852789919141 (-0.02143776584679058) | > avg_grad_norm_0: tensor(14.4245, device='cuda:0') (tensor(-1.8212, device='cuda:0')) | > avg_loss_gen: 1.8685171856007106 (+0.021185198300321284) | > avg_loss_kl: 2.22163510826272 (-0.030597141091252222) | > avg_loss_feat: 2.760330737476617 (+0.050208283142303856) | > avg_loss_mel: 17.667935317670786 (-0.1026775736204435) | > avg_loss_duration: 1.5991750149659707 (-0.00941731224597353) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.11759330856968 (-0.07129867983535831) | > avg_grad_norm_1: tensor(482.7948, device='cuda:0') (+tensor(84.0818, device='cuda:0'))  > EPOCH: 391/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:13:18)   --> TIME: 2025-02-25 12:13:41 -- STEP: 47/72 -- GLOBAL_STEP: 1028200 | > loss_disc: 2.7616848945617676 (2.750219309583624) | > loss_disc_real_0: 0.18429791927337646 (0.19654232374531158) | > loss_disc_real_1: 0.1836777925491333 (0.23559868208905485) | > loss_disc_real_2: 0.27524420619010925 (0.23827794986836454) | > loss_disc_real_3: 0.2625228464603424 (0.23986577734034112) | > loss_disc_real_4: 0.26968374848365784 (0.24363724284983695) | > loss_disc_real_5: 0.2576315402984619 (0.24384774743242466) | > loss_0: 2.7616848945617676 (2.750219309583624) | > grad_norm_0: tensor(8.1746, device='cuda:0') (tensor(14.8507, device='cuda:0')) | > loss_gen: 1.8122026920318604 (1.8714281066935112) | > loss_kl: 2.2220513820648193 (2.17975247667191) | > loss_feat: 2.8065476417541504 (2.7811003948779818) | > loss_mel: 17.280168533325195 (17.63692052313622) | > loss_duration: 1.6080684661865234 (1.600069299657294) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.72903823852539 (26.069270600663856) | > grad_norm_1: tensor(316.0818, device='cuda:0') (tensor(580.4747, device='cuda:0')) | > current_lr_0: 0.00019045945011230786 | > current_lr_1: 0.00019045945011230786 | > step_time: 0.4506 (0.4536429313903159) | > loader_time: 0.0088 (0.007703147035963992) --> EVAL PERFORMANCE | > avg_loader_time: 0.008790157210658971 (+0.00010983037277006481) | > avg_step_time: 0.4634893947923687 (+0.010155583771181809) | > avg_loss_disc: 2.754476359192755 (-0.013376430726386168) | > avg_loss_disc_real_0: 0.19859448442576635 (-0.005506644471430483) | > avg_loss_disc_real_1: 0.23885826556615428 (+0.006758789990989278) | > avg_loss_disc_real_2: 0.23794133499474593 (-0.003409772691592372) | > avg_loss_disc_real_3: 0.23943710411098643 (+0.0005837045085262393) | > avg_loss_disc_real_4: 0.24254269709049817 (-0.0003452477320818881) | > avg_loss_disc_real_5: 0.24304471847037196 (-0.005035207305155998) | > avg_loss_0: 2.754476359192755 (-0.013376430726386168) | > avg_grad_norm_0: tensor(14.5567, device='cuda:0') (+tensor(0.1322, device='cuda:0')) | > avg_loss_gen: 1.86666833682799 (-0.0018488487727206238) | > avg_loss_kl: 2.1937478260255205 (-0.027887282237199607) | > avg_loss_feat: 2.764950322433257 (+0.004619584956639944) | > avg_loss_mel: 17.66120582902935 (-0.006729488641436632) | > avg_loss_duration: 1.6121111755639734 (+0.012936160598002733) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.098683263214543 (-0.018910045355138294) | > avg_grad_norm_1: tensor(556.2432, device='cuda:0') (+tensor(73.4485, device='cuda:0'))  > EPOCH: 392/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:13:55)   --> TIME: 2025-02-25 12:14:08 -- STEP: 25/72 -- GLOBAL_STEP: 1028250 | > loss_disc: 2.702505588531494 (2.7442469787597656) | > loss_disc_real_0: 0.24191278219223022 (0.20744444787502286) | > loss_disc_real_1: 0.16020239889621735 (0.23536423742771148) | > loss_disc_real_2: 0.23738279938697815 (0.23715599417686462) | > loss_disc_real_3: 0.23541082441806793 (0.23829565405845643) | > loss_disc_real_4: 0.2374470829963684 (0.24499162256717683) | > loss_disc_real_5: 0.2453988492488861 (0.2514096516370773) | > loss_0: 2.702505588531494 (2.7442469787597656) | > grad_norm_0: tensor(26.4260, device='cuda:0') (tensor(18.8242, device='cuda:0')) | > loss_gen: 1.7251789569854736 (1.9339370918273926) | > loss_kl: 2.1136186122894287 (2.089987778663635) | > loss_feat: 2.8956596851348877 (2.8598701095581056) | > loss_mel: 16.28373146057129 (17.758509902954103) | > loss_duration: 1.5724971294403076 (1.5621986722946166) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.590686798095703 (26.20450378417969) | > grad_norm_1: tensor(766.8719, device='cuda:0') (tensor(796.2161, device='cuda:0')) | > current_lr_0: 0.0001904356426810438 | > current_lr_1: 0.0001904356426810438 | > step_time: 0.4329 (0.43305856704711915) | > loader_time: 0.0075 (0.006766748428344726) --> EVAL PERFORMANCE | > avg_loader_time: 0.008682724455712542 (-0.00010743275494642934) | > avg_step_time: 0.45574769167832924 (-0.007741703114039489) | > avg_loss_disc: 2.761283558858953 (+0.006807199666198027) | > avg_loss_disc_real_0: 0.20393260872699845 (+0.0053381243012320945) | > avg_loss_disc_real_1: 0.23429864078340396 (-0.004559624782750321) | > avg_loss_disc_real_2: 0.23828529702945495 (+0.00034396203470901665) | > avg_loss_disc_real_3: 0.2380226793843256 (-0.00141442472666084) | > avg_loss_disc_real_4: 0.2458351947052378 (+0.0032924976147396257) | > avg_loss_disc_real_5: 0.24906135109108937 (+0.006016632620717416) | > avg_loss_0: 2.761283558858953 (+0.006807199666198027) | > avg_grad_norm_0: tensor(17.1684, device='cuda:0') (+tensor(2.6117, device='cuda:0')) | > avg_loss_gen: 1.8863708234169114 (+0.019702486588921486) | > avg_loss_kl: 2.152449178024076 (-0.04129864800144434) | > avg_loss_feat: 2.7796884355410723 (+0.014738113107815476) | > avg_loss_mel: 17.685930695332278 (+0.024724866302928916) | > avg_loss_duration: 1.5990033233669443 (-0.013107852197029146) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.103442689062845 (+0.004759425848302357) | > avg_grad_norm_1: tensor(657.0504, device='cuda:0') (+tensor(100.8072, device='cuda:0'))  > EPOCH: 393/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:14:32)   --> TIME: 2025-02-25 12:14:35 -- STEP: 3/72 -- GLOBAL_STEP: 1028300 | > loss_disc: 2.771240234375 (2.697619597117106) | > loss_disc_real_0: 0.3270067274570465 (0.2237582728266716) | > loss_disc_real_1: 0.27338504791259766 (0.24697594344615936) | > loss_disc_real_2: 0.25894540548324585 (0.2383043517669042) | > loss_disc_real_3: 0.2891314923763275 (0.26763025919596356) | > loss_disc_real_4: 0.2523593008518219 (0.2399737536907196) | > loss_disc_real_5: 0.25225722789764404 (0.23790623247623444) | > loss_0: 2.771240234375 (2.697619597117106) | > grad_norm_0: tensor(32.0700, device='cuda:0') (tensor(26.5412, device='cuda:0')) | > loss_gen: 2.172217845916748 (2.1709068616231284) | > loss_kl: 2.105435609817505 (2.1902664502461753) | > loss_feat: 2.9848034381866455 (3.105153958002726) | > loss_mel: 19.015209197998047 (18.277611414591473) | > loss_duration: 1.6238569021224976 (1.6147221326828003) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.90152359008789 (27.358660380045574) | > grad_norm_1: tensor(451.7320, device='cuda:0') (tensor(562.2889, device='cuda:0')) | > current_lr_0: 0.00019041183822570866 | > current_lr_1: 0.00019041183822570866 | > step_time: 0.423 (0.42728304862976074) | > loader_time: 0.0057 (0.006300052007039388)  --> TIME: 2025-02-25 12:14:58 -- STEP: 53/72 -- GLOBAL_STEP: 1028350 | > loss_disc: 2.74713397026062 (2.768503868354941) | > loss_disc_real_0: 0.27431291341781616 (0.1984008234345688) | > loss_disc_real_1: 0.2178012579679489 (0.2327125997475858) | > loss_disc_real_2: 0.18511295318603516 (0.24143922863141545) | > loss_disc_real_3: 0.22765056788921356 (0.2425583061182274) | > loss_disc_real_4: 0.20924612879753113 (0.2457156611501046) | > loss_disc_real_5: 0.2137216329574585 (0.2505991231158095) | > loss_0: 2.74713397026062 (2.768503868354941) | > grad_norm_0: tensor(17.9638, device='cuda:0') (tensor(17.1975, device='cuda:0')) | > loss_gen: 1.8340723514556885 (1.8782715864901274) | > loss_kl: 2.1116771697998047 (2.230615971223363) | > loss_feat: 2.9739997386932373 (2.758406693080685) | > loss_mel: 17.53084373474121 (17.851183693363982) | > loss_duration: 1.6339269876480103 (1.5902721544481673) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.084518432617188 (26.308750224563312) | > grad_norm_1: tensor(522.8270, device='cuda:0') (tensor(569.7396, device='cuda:0')) | > current_lr_0: 0.00019041183822570866 | > current_lr_1: 0.00019041183822570866 | > step_time: 0.4523 (0.4445077068400833) | > loader_time: 0.009 (0.007817196396161925) --> EVAL PERFORMANCE | > avg_loader_time: 0.008827770260018361 (+0.00014504580430581976) | > avg_step_time: 0.4566107434286198 (+0.0008630517502905555) | > avg_loss_disc: 2.7668704281390553 (+0.0055868692801022135) | > avg_loss_disc_real_0: 0.19989924353193228 (-0.004033365195066169) | > avg_loss_disc_real_1: 0.23041315313795924 (-0.0038854876454447185) | > avg_loss_disc_real_2: 0.24169934740368748 (+0.0034140503742325357) | > avg_loss_disc_real_3: 0.2421774148521289 (+0.004154735467803317) | > avg_loss_disc_real_4: 0.24380567585918264 (-0.0020295188460551505) | > avg_loss_disc_real_5: 0.2469316563555892 (-0.002129694735500176) | > avg_loss_0: 2.7668704281390553 (+0.0055868692801022135) | > avg_grad_norm_0: tensor(16.8738, device='cuda:0') (tensor(-0.2946, device='cuda:0')) | > avg_loss_gen: 1.8706808342060572 (-0.015689989210854183) | > avg_loss_kl: 2.2294575664359075 (+0.07700838841183133) | > avg_loss_feat: 2.751128495579034 (-0.028559939962038428) | > avg_loss_mel: 17.729439050378932 (+0.0435083550466544) | > avg_loss_duration: 1.602322988107171 (+0.003319664740226713) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.1830289330281 (+0.07958624396525593) | > avg_grad_norm_1: tensor(606.6677, device='cuda:0') (tensor(-50.3828, device='cuda:0'))  > EPOCH: 394/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:15:09)   --> TIME: 2025-02-25 12:15:25 -- STEP: 31/72 -- GLOBAL_STEP: 1028400 | > loss_disc: 2.7376701831817627 (2.7643832468217417) | > loss_disc_real_0: 0.22741350531578064 (0.18718589217432083) | > loss_disc_real_1: 0.29011502861976624 (0.2385560788454548) | > loss_disc_real_2: 0.24514277279376984 (0.23793949715552792) | > loss_disc_real_3: 0.24109654128551483 (0.2442544525669467) | > loss_disc_real_4: 0.25663456320762634 (0.24900167122963937) | > loss_disc_real_5: 0.26844522356987 (0.25054566177629656) | > loss_0: 2.7376701831817627 (2.7643832468217417) | > grad_norm_0: tensor(11.1345, device='cuda:0') (tensor(16.8515, device='cuda:0')) | > loss_gen: 2.0680131912231445 (1.9290207355253157) | > loss_kl: 2.408155918121338 (2.2749888550850663) | > loss_feat: 3.0372672080993652 (3.0090048697686966) | > loss_mel: 18.468236923217773 (18.16047957635695) | > loss_duration: 1.6265888214111328 (1.6174866422530143) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.60826301574707 (26.99098064053443) | > grad_norm_1: tensor(1686.2510, device='cuda:0') (tensor(945.2708, device='cuda:0')) | > current_lr_0: 0.00019038803674593043 | > current_lr_1: 0.00019038803674593043 | > step_time: 0.4371 (0.4379086417536582) | > loader_time: 0.0086 (0.0071622863892586) --> EVAL PERFORMANCE | > avg_loader_time: 0.008689124819258568 (-0.000138645440759793) | > avg_step_time: 0.461851566610202 (+0.005240823181582199) | > avg_loss_disc: 2.767606026689771 (+0.0007355985507158103) | > avg_loss_disc_real_0: 0.19164530948643954 (-0.008253934045492739) | > avg_loss_disc_real_1: 0.23356251376615444 (+0.003149360628195208) | > avg_loss_disc_real_2: 0.23582415807415064 (-0.005875189329536845) | > avg_loss_disc_real_3: 0.24101772547607692 (-0.0011596893760519866) | > avg_loss_disc_real_4: 0.24597716352469484 (+0.0021714876655122006) | > avg_loss_disc_real_5: 0.2490917421142821 (+0.0021600857586928923) | > avg_loss_0: 2.767606026689771 (+0.0007355985507158103) | > avg_grad_norm_0: tensor(21.4345, device='cuda:0') (+tensor(4.5607, device='cuda:0')) | > avg_loss_gen: 1.8727998095498959 (+0.002118975343838647) | > avg_loss_kl: 2.283611839925739 (+0.05415427348983126) | > avg_loss_feat: 2.7986678338386635 (+0.04753933825962964) | > avg_loss_mel: 18.11269725208551 (+0.3832582017065782) | > avg_loss_duration: 1.625895882996035 (+0.02357289488886405) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.693672771185216 (+0.5106438381571152) | > avg_grad_norm_1: tensor(864.8355, device='cuda:0') (+tensor(258.1678, device='cuda:0'))  > EPOCH: 395/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:15:46)   --> TIME: 2025-02-25 12:15:52 -- STEP: 9/72 -- GLOBAL_STEP: 1028450 | > loss_disc: 2.635713815689087 (2.701876746283637) | > loss_disc_real_0: 0.13940905034542084 (0.20050109757317436) | > loss_disc_real_1: 0.20115172863006592 (0.2246058699157503) | > loss_disc_real_2: 0.21693351864814758 (0.22832942836814457) | > loss_disc_real_3: 0.23008641600608826 (0.23761885033713448) | > loss_disc_real_4: 0.24478009343147278 (0.23751768635378945) | > loss_disc_real_5: 0.22888223826885223 (0.23305804861916435) | > loss_0: 2.635713815689087 (2.701876746283637) | > grad_norm_0: tensor(10.5021, device='cuda:0') (tensor(17.4699, device='cuda:0')) | > loss_gen: 1.9331037998199463 (1.9093186325497098) | > loss_kl: 2.108647346496582 (2.1735247241126165) | > loss_feat: 3.4276835918426514 (2.9463548130459256) | > loss_mel: 18.662935256958008 (17.969920476277668) | > loss_duration: 1.5683491230010986 (1.5800436867607965) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.700719833374023 (26.57916196187337) | > grad_norm_1: tensor(720.4506, device='cuda:0') (tensor(805.6602, device='cuda:0')) | > current_lr_0: 0.0001903642382413372 | > current_lr_1: 0.0001903642382413372 | > step_time: 0.4267 (0.4316415521833632) | > loader_time: 0.0059 (0.006246328353881836)  --> TIME: 2025-02-25 12:16:15 -- STEP: 59/72 -- GLOBAL_STEP: 1028500 | > loss_disc: 2.8279433250427246 (2.7480671324972374) | > loss_disc_real_0: 0.2003224641084671 (0.19867580245106908) | > loss_disc_real_1: 0.2250787764787674 (0.2292247553498058) | > loss_disc_real_2: 0.2706526219844818 (0.2354947636693211) | > loss_disc_real_3: 0.25454750657081604 (0.23757473543538885) | > loss_disc_real_4: 0.24369804561138153 (0.2442655101165933) | > loss_disc_real_5: 0.2518797218799591 (0.24606075200994135) | > loss_0: 2.8279433250427246 (2.7480671324972374) | > grad_norm_0: tensor(6.2112, device='cuda:0') (tensor(15.3441, device='cuda:0')) | > loss_gen: 1.7950758934020996 (1.8821665816387887) | > loss_kl: 2.0331239700317383 (2.2927975169682906) | > loss_feat: 2.612445831298828 (2.8380195245904436) | > loss_mel: 16.4759578704834 (18.025520486346746) | > loss_duration: 1.630035400390625 (1.6278693958864374) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.546640396118164 (26.666373527656166) | > grad_norm_1: tensor(436.4156, device='cuda:0') (tensor(697.1761, device='cuda:0')) | > current_lr_0: 0.0001903642382413372 | > current_lr_1: 0.0001903642382413372 | > step_time: 0.4689 (0.44639239876957265) | > loader_time: 0.0097 (0.00799891504190736) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870137483301297 (+1.225001375440224e-05) | > avg_step_time: 0.45716547630202603 (-0.004686090308175961) | > avg_loss_disc: 2.7495138678752196 (-0.018092158814551507) | > avg_loss_disc_real_0: 0.19779713577787641 (+0.006151826291436874) | > avg_loss_disc_real_1: 0.22874736177249694 (-0.004815151993657507) | > avg_loss_disc_real_2: 0.23509530156431063 (-0.0007288565098400035) | > avg_loss_disc_real_3: 0.2371998452384707 (-0.003817880237606208) | > avg_loss_disc_real_4: 0.24448606791630598 (-0.001491095608388865) | > avg_loss_disc_real_5: 0.24614795622691302 (-0.0029437858873690714) | > avg_loss_0: 2.7495138678752196 (-0.018092158814551507) | > avg_grad_norm_0: tensor(16.3320, device='cuda:0') (tensor(-5.1025, device='cuda:0')) | > avg_loss_gen: 1.8725922208436778 (-0.0002075887062180648) | > avg_loss_kl: 2.290629229075472 (+0.007017389149733155) | > avg_loss_feat: 2.8029928274557623 (+0.004324993617098816) | > avg_loss_mel: 17.932928071895116 (-0.17976918019039445) | > avg_loss_duration: 1.6301349384683959 (+0.004239055472360809) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.529277317960496 (-0.1643954532247207) | > avg_grad_norm_1: tensor(724.2498, device='cuda:0') (tensor(-140.5857, device='cuda:0'))  > EPOCH: 396/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:16:23)   --> TIME: 2025-02-25 12:16:42 -- STEP: 37/72 -- GLOBAL_STEP: 1028550 | > loss_disc: 2.8001747131347656 (2.7384491997796134) | > loss_disc_real_0: 0.2833888530731201 (0.1897447542161555) | > loss_disc_real_1: 0.2529850900173187 (0.22409549959608027) | > loss_disc_real_2: 0.24923379719257355 (0.2423850958411758) | > loss_disc_real_3: 0.2503994405269623 (0.2425090412835817) | > loss_disc_real_4: 0.237808957695961 (0.24701671262045163) | > loss_disc_real_5: 0.24868562817573547 (0.2486271954871513) | > loss_0: 2.8001747131347656 (2.7384491997796134) | > grad_norm_0: tensor(21.6628, device='cuda:0') (tensor(14.4561, device='cuda:0')) | > loss_gen: 1.9392826557159424 (1.9186740340413273) | > loss_kl: 2.072654962539673 (2.185779577976949) | > loss_feat: 3.057695150375366 (2.9652045997413428) | > loss_mel: 17.46080207824707 (17.736746865349836) | > loss_duration: 1.5760633945465088 (1.5791384336110705) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.10649871826172 (26.385543668592298) | > grad_norm_1: tensor(245.4849, device='cuda:0') (tensor(808.9829, device='cuda:0')) | > current_lr_0: 0.00019034044271155703 | > current_lr_1: 0.00019034044271155703 | > step_time: 0.4698 (0.44128710514790304) | > loader_time: 0.0081 (0.007348434345142261) --> EVAL PERFORMANCE | > avg_loader_time: 0.008742104113941458 (+4.0729280928486836e-05) | > avg_step_time: 0.4587592373431568 (+0.0015937610411307923) | > avg_loss_disc: 2.7553640553649044 (+0.005850187489684799) | > avg_loss_disc_real_0: 0.19845916677108952 (+0.000662030993213103) | > avg_loss_disc_real_1: 0.22707886070432798 (-0.0016685010681689594) | > avg_loss_disc_real_2: 0.2422612405998606 (+0.007165939035549962) | > avg_loss_disc_real_3: 0.24141160514153226 (+0.004211759903061552) | > avg_loss_disc_real_4: 0.2460505807063949 (+0.0015645127900889289) | > avg_loss_disc_real_5: 0.24778244424034174 (+0.0016344880134287199) | > avg_loss_0: 2.7553640553649044 (+0.005850187489684799) | > avg_grad_norm_0: tensor(17.6534, device='cuda:0') (+tensor(1.3213, device='cuda:0')) | > avg_loss_gen: 1.8921493174324573 (+0.019557096588779466) | > avg_loss_kl: 2.3143222130520242 (+0.023692983976552284) | > avg_loss_feat: 2.8471996851370367 (+0.044206857681274414) | > avg_loss_mel: 18.100750855996573 (+0.1678227841014568) | > avg_loss_duration: 1.6076547965197496 (-0.022480141948646226) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.762077062902318 (+0.23279974494182198) | > avg_grad_norm_1: tensor(833.6659, device='cuda:0') (+tensor(109.4161, device='cuda:0'))  > EPOCH: 397/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:17:01)   --> TIME: 2025-02-25 12:17:09 -- STEP: 15/72 -- GLOBAL_STEP: 1028600 | > loss_disc: 2.794893741607666 (2.7014445145924886) | > loss_disc_real_0: 0.24077150225639343 (0.19161147971947987) | > loss_disc_real_1: 0.24351146817207336 (0.23307855427265167) | > loss_disc_real_2: 0.23703740537166595 (0.22053962747255962) | > loss_disc_real_3: 0.23969513177871704 (0.2308981458346049) | > loss_disc_real_4: 0.23969992995262146 (0.23089127043883007) | > loss_disc_real_5: 0.25154122710227966 (0.23088309665520987) | > loss_0: 2.794893741607666 (2.7014445145924886) | > grad_norm_0: tensor(10.9048, device='cuda:0') (tensor(17.6505, device='cuda:0')) | > loss_gen: 1.8453915119171143 (1.9099307457605998) | > loss_kl: 2.3314685821533203 (2.365847603480021) | > loss_feat: 3.1682567596435547 (3.008216635386149) | > loss_mel: 18.93022918701172 (18.5381960550944) | > loss_duration: 1.5566078424453735 (1.5796537081400552) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.831954956054688 (27.401844787597657) | > grad_norm_1: tensor(755.3519, device='cuda:0') (tensor(747.3481, device='cuda:0')) | > current_lr_0: 0.0001903166501562181 | > current_lr_1: 0.0001903166501562181 | > step_time: 0.4404 (0.4372785250345866) | > loader_time: 0.0067 (0.006886132558186849)  --> TIME: 2025-02-25 12:17:33 -- STEP: 65/72 -- GLOBAL_STEP: 1028650 | > loss_disc: 2.715268135070801 (2.7352407822242153) | > loss_disc_real_0: 0.1524384468793869 (0.18986831353260922) | > loss_disc_real_1: 0.17737171053886414 (0.2265348255634308) | > loss_disc_real_2: 0.24571435153484344 (0.23115482605420626) | > loss_disc_real_3: 0.22488407790660858 (0.2358901498409418) | > loss_disc_real_4: 0.23438198864459991 (0.24160968042336978) | > loss_disc_real_5: 0.2412216067314148 (0.24621833494076362) | > loss_0: 2.715268135070801 (2.7352407822242153) | > grad_norm_0: tensor(14.6957, device='cuda:0') (tensor(17.6731, device='cuda:0')) | > loss_gen: 1.9188166856765747 (1.8755550164442796) | > loss_kl: 2.3127715587615967 (2.204030873225284) | > loss_feat: 3.03171443939209 (2.8636024365058312) | > loss_mel: 19.11162757873535 (17.82332521585317) | > loss_duration: 1.641094446182251 (1.5957086177972646) | > amp_scaler: 64.0 (64.0) | > loss_1: 28.01602554321289 (26.362222143319936) | > grad_norm_1: tensor(484.7295, device='cuda:0') (tensor(835.2477, device='cuda:0')) | > current_lr_0: 0.0001903166501562181 | > current_lr_1: 0.0001903166501562181 | > step_time: 0.5301 (0.4522125941056472) | > loader_time: 0.0115 (0.008437545482928936) --> EVAL PERFORMANCE | > avg_loader_time: 0.008694608446577904 (-4.749566736355314e-05) | > avg_step_time: 0.4583854675292969 (-0.0003737698138599477) | > avg_loss_disc: 2.741993625399093 (-0.013370429965811503) | > avg_loss_disc_real_0: 0.19356242346931513 (-0.004896743301774392) | > avg_loss_disc_real_1: 0.2265609003288645 (-0.0005179603754634776) | > avg_loss_disc_real_2: 0.23187018888936917 (-0.010391051710491428) | > avg_loss_disc_real_3: 0.23632907741506334 (-0.005082527726468916) | > avg_loss_disc_real_4: 0.24093872014905365 (-0.005111860557341252) | > avg_loss_disc_real_5: 0.2461340498756355 (-0.001648394364706235) | > avg_loss_0: 2.741993625399093 (-0.013370429965811503) | > avg_grad_norm_0: tensor(17.8639, device='cuda:0') (+tensor(0.2105, device='cuda:0')) | > avg_loss_gen: 1.8683055790377334 (-0.023843738394723868) | > avg_loss_kl: 2.207298221722454 (-0.1070239913295703) | > avg_loss_feat: 2.833268256254599 (-0.01393142888243748) | > avg_loss_mel: 17.78215568166383 (-0.3185951743327422) | > avg_loss_duration: 1.600029492042434 (-0.007625304477315575) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.29105721057301 (-0.47101985232930943) | > avg_grad_norm_1: tensor(798.3593, device='cuda:0') (tensor(-35.3066, device='cuda:0'))  > EPOCH: 398/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:17:38)   --> TIME: 2025-02-25 12:18:00 -- STEP: 43/72 -- GLOBAL_STEP: 1028700 | > loss_disc: 2.7909629344940186 (2.7227456957794898) | > loss_disc_real_0: 0.2006250023841858 (0.1843255604075831) | > loss_disc_real_1: 0.14352363348007202 (0.22163487139136293) | > loss_disc_real_2: 0.24592293798923492 (0.2371638711801795) | > loss_disc_real_3: 0.2775923013687134 (0.23659675654976867) | > loss_disc_real_4: 0.2523919641971588 (0.24371353796748227) | > loss_disc_real_5: 0.2641254961490631 (0.2447606772184372) | > loss_0: 2.7909629344940186 (2.7227456957794898) | > grad_norm_0: tensor(9.8374, device='cuda:0') (tensor(18.8200, device='cuda:0')) | > loss_gen: 2.1381421089172363 (1.9046959960183432) | > loss_kl: 2.3000035285949707 (2.1126164336537205) | > loss_feat: 3.024599075317383 (2.9746615165887875) | > loss_mel: 17.875736236572266 (17.30758201244266) | > loss_duration: 1.6173090934753418 (1.5683756761772687) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.95578956604004 (25.867931720822355) | > grad_norm_1: tensor(209.8669, device='cuda:0') (tensor(1017.8925, device='cuda:0')) | > current_lr_0: 0.00019029286057494854 | > current_lr_1: 0.00019029286057494854 | > step_time: 0.4607 (0.4557064134021138) | > loader_time: 0.0099 (0.007648351580597634) --> EVAL PERFORMANCE | > avg_loader_time: 0.008734044894366197 (+3.943644778829289e-05) | > avg_step_time: 0.46385109256690654 (+0.005465625037609667) | > avg_loss_disc: 2.760742103549796 (+0.01874847815070302) | > avg_loss_disc_real_0: 0.19239622658826935 (-0.0011661968810457723) | > avg_loss_disc_real_1: 0.23332374259619645 (+0.006762842267331953) | > avg_loss_disc_real_2: 0.24297302247772753 (+0.011102833588358368) | > avg_loss_disc_real_3: 0.2401215480247014 (+0.0037924706096380545) | > avg_loss_disc_real_4: 0.2449457286529138 (+0.004007008503860154) | > avg_loss_disc_real_5: 0.24774309174275735 (+0.001609041867121852) | > avg_loss_0: 2.760742103549796 (+0.01874847815070302) | > avg_grad_norm_0: tensor(18.0571, device='cuda:0') (+tensor(0.1933, device='cuda:0')) | > avg_loss_gen: 1.9003427314086698 (+0.032037152370936406) | > avg_loss_kl: 2.263140581023525 (+0.05584235930107129) | > avg_loss_feat: 2.9073001700387873 (+0.07403191378418805) | > avg_loss_mel: 17.946691647381858 (+0.1645359657180272) | > avg_loss_duration: 1.618616048718842 (+0.01858655667640785) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.636091312892 (+0.3450341023189907) | > avg_grad_norm_1: tensor(955.2507, device='cuda:0') (+tensor(156.8914, device='cuda:0'))  > EPOCH: 399/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:18:15)   --> TIME: 2025-02-25 12:18:27 -- STEP: 21/72 -- GLOBAL_STEP: 1028750 | > loss_disc: 2.740455389022827 (2.770982083820161) | > loss_disc_real_0: 0.26222819089889526 (0.20245819361436934) | > loss_disc_real_1: 0.2115379422903061 (0.22383476864723933) | > loss_disc_real_2: 0.2254551202058792 (0.23458667880012876) | > loss_disc_real_3: 0.2781999111175537 (0.24569242128304072) | > loss_disc_real_4: 0.2743373215198517 (0.24644153529689425) | > loss_disc_real_5: 0.275523841381073 (0.2464884626013892) | > loss_0: 2.740455389022827 (2.770982083820161) | > grad_norm_0: tensor(26.3160, device='cuda:0') (tensor(21.9440, device='cuda:0')) | > loss_gen: 2.0513482093811035 (1.8626336313429332) | > loss_kl: 2.168984889984131 (2.1441661516825357) | > loss_feat: 2.335103750228882 (2.8075494879768) | > loss_mel: 16.75096321105957 (17.924251102265856) | > loss_duration: 1.6037697792053223 (1.5855938480013894) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.910167694091797 (26.324194499424525) | > grad_norm_1: tensor(1000.9995, device='cuda:0') (tensor(782.0744, device='cuda:0')) | > current_lr_0: 0.00019026907396737667 | > current_lr_1: 0.00019026907396737667 | > step_time: 0.4384 (0.43697995231265113) | > loader_time: 0.0077 (0.006802831377301898)  --> TIME: 2025-02-25 12:18:51 -- STEP: 71/72 -- GLOBAL_STEP: 1028800 | > loss_disc: 2.7673041820526123 (2.756178308540666) | > loss_disc_real_0: 0.090512715280056 (0.19756550866533332) | > loss_disc_real_1: 0.2817372977733612 (0.2304764452954413) | > loss_disc_real_2: 0.2701452374458313 (0.23485496976006198) | > loss_disc_real_3: 0.24643519520759583 (0.23608687331139203) | > loss_disc_real_4: 0.24060480296611786 (0.24447376438429658) | > loss_disc_real_5: 0.2386205494403839 (0.24640506526953737) | > loss_0: 2.7673041820526123 (2.756178308540666) | > grad_norm_0: tensor(24.8815, device='cuda:0') (tensor(20.0618, device='cuda:0')) | > loss_gen: 1.912639856338501 (1.8576565561160234) | > loss_kl: 2.405876636505127 (2.275664316096777) | > loss_feat: 2.9242947101593018 (2.8054577296888326) | > loss_mel: 18.776573181152344 (17.887471131875486) | > loss_duration: 1.6153435707092285 (1.6100170058263858) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.634727478027344 (26.436266831948725) | > grad_norm_1: tensor(539.5962, device='cuda:0') (tensor(790.1307, device='cuda:0')) | > current_lr_0: 0.00019026907396737667 | > current_lr_1: 0.00019026907396737667 | > step_time: 0.51 (0.45684926274796606) | > loader_time: 0.0082 (0.008641944804661711) --> EVAL PERFORMANCE | > avg_loader_time: 0.008641944804661711 (-9.210008970448637e-05) | > avg_step_time: 0.45684926274796606 (-0.007001829818940486) | > avg_loss_disc: 2.756178308540666 (-0.004563795009129734) | > avg_loss_disc_real_0: 0.19756550866533332 (+0.005169282077063964) | > avg_loss_disc_real_1: 0.2304764452954413 (-0.0028472973007551416) | > avg_loss_disc_real_2: 0.23485496976006198 (-0.008118052717665553) | > avg_loss_disc_real_3: 0.23608687331139203 (-0.004034674713309372) | > avg_loss_disc_real_4: 0.24447376438429658 (-0.00047196426861723095) | > avg_loss_disc_real_5: 0.24640506526953737 (-0.0013380264732199831) | > avg_loss_0: 2.756178308540666 (-0.004563795009129734) | > avg_grad_norm_0: tensor(20.0618, device='cuda:0') (+tensor(2.0046, device='cuda:0')) | > avg_loss_gen: 1.8576565561160234 (-0.0426861752926464) | > avg_loss_kl: 2.275664316096777 (+0.012523735073251885) | > avg_loss_feat: 2.8054577296888326 (-0.10184244034995471) | > avg_loss_mel: 17.887471131875486 (-0.05922051550637164) | > avg_loss_duration: 1.6100170058263858 (-0.008599042892456055) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.436266831948725 (-0.1998244809432741) | > avg_grad_norm_1: tensor(790.1307, device='cuda:0') (tensor(-165.1200, device='cuda:0'))  > EPOCH: 400/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:18:52)   --> TIME: 2025-02-25 12:19:17 -- STEP: 49/72 -- GLOBAL_STEP: 1028850 | > loss_disc: 2.829106092453003 (2.773120807141674) | > loss_disc_real_0: 0.3162960708141327 (0.20641074877004234) | > loss_disc_real_1: 0.21901704370975494 (0.228996428908134) | > loss_disc_real_2: 0.23139625787734985 (0.23894170778138296) | > loss_disc_real_3: 0.24457913637161255 (0.23943947012327155) | > loss_disc_real_4: 0.2612384259700775 (0.24348693021706172) | > loss_disc_real_5: 0.24884538352489471 (0.24832160284324567) | > loss_0: 2.829106092453003 (2.773120807141674) | > grad_norm_0: tensor(38.9437, device='cuda:0') (tensor(20.1127, device='cuda:0')) | > loss_gen: 1.8580167293548584 (1.8760103060274709) | > loss_kl: 2.205305337905884 (2.3195381310521332) | > loss_feat: 2.4465701580047607 (2.8115985782778994) | > loss_mel: 18.02034568786621 (18.237638162106883) | > loss_duration: 1.6751952171325684 (1.6022089719772339) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.205434799194336 (26.846994438949896) | > grad_norm_1: tensor(1029.9521, device='cuda:0') (tensor(698.5907, device='cuda:0')) | > current_lr_0: 0.00019024529033313074 | > current_lr_1: 0.00019024529033313074 | > step_time: 0.4419 (0.44164849787342303) | > loader_time: 0.0096 (0.007686415497137576) --> EVAL PERFORMANCE | > avg_loader_time: 0.008775150272208201 (+0.0001332054675464902) | > avg_step_time: 0.45355043612735374 (-0.0032988266206123162) | > avg_loss_disc: 2.768919971627249 (+0.012741663086582644) | > avg_loss_disc_real_0: 0.19670779310481648 (-0.0008577155605168363) | > avg_loss_disc_real_1: 0.23344405771980822 (+0.002967612424366911) | > avg_loss_disc_real_2: 0.23676130742254392 (+0.0019063376624819384) | > avg_loss_disc_real_3: 0.23915900982601543 (+0.0030721365146234025) | > avg_loss_disc_real_4: 0.2435826023279781 (-0.0008911620563184786) | > avg_loss_disc_real_5: 0.24704830432441874 (+0.0006432390548813671) | > avg_loss_0: 2.768919971627249 (+0.012741663086582644) | > avg_grad_norm_0: tensor(19.3976, device='cuda:0') (tensor(-0.6642, device='cuda:0')) | > avg_loss_gen: 1.8629600397298034 (+0.005303483613779969) | > avg_loss_kl: 2.3190437639263313 (+0.04337944782955416) | > avg_loss_feat: 2.802757162443349 (-0.0027005672454833984) | > avg_loss_mel: 18.118059991111217 (+0.23058885923573058) | > avg_loss_duration: 1.613668681870044 (+0.0036516760436582363) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.716489791870117 (+0.28022295992139234) | > avg_grad_norm_1: tensor(708.5859, device='cuda:0') (tensor(-81.5449, device='cuda:0'))  > EPOCH: 401/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:19:29)   --> TIME: 2025-02-25 12:19:43 -- STEP: 27/72 -- GLOBAL_STEP: 1028900 | > loss_disc: 2.8092823028564453 (2.786786264843411) | > loss_disc_real_0: 0.12833236157894135 (0.21046119486844098) | > loss_disc_real_1: 0.23471404612064362 (0.2277337654873177) | > loss_disc_real_2: 0.2466365098953247 (0.24933730010633115) | > loss_disc_real_3: 0.2850101888179779 (0.24531318744023642) | > loss_disc_real_4: 0.25799354910850525 (0.24559312010252918) | > loss_disc_real_5: 0.2730998694896698 (0.24713962111208174) | > loss_0: 2.8092823028564453 (2.786786264843411) | > grad_norm_0: tensor(16.9095, device='cuda:0') (tensor(14.6250, device='cuda:0')) | > loss_gen: 1.8357019424438477 (1.901222652859158) | > loss_kl: 2.2092299461364746 (2.14826120712139) | > loss_feat: 2.5321645736694336 (2.9717399809095593) | > loss_mel: 17.49210548400879 (17.87161367910879) | > loss_duration: 1.5490481853485107 (1.5606604682074652) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.618249893188477 (26.45349813390661) | > grad_norm_1: tensor(239.1036, device='cuda:0') (tensor(501.8353, device='cuda:0')) | > current_lr_0: 0.0001902215096718391 | > current_lr_1: 0.0001902215096718391 | > step_time: 0.443 (0.4342939588758681) | > loader_time: 0.0082 (0.0070889437640154805) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719262942461899 (-5.5887329746302436e-05) | > avg_step_time: 0.4518272104397626 (-0.0017232256875911456) | > avg_loss_disc: 2.7845637160287775 (+0.015643744401528714) | > avg_loss_disc_real_0: 0.20984770011314205 (+0.013139907008325569) | > avg_loss_disc_real_1: 0.23089470871737305 (-0.0025493490024351695) | > avg_loss_disc_real_2: 0.2425390404294914 (+0.0057777330069474775) | > avg_loss_disc_real_3: 0.24285474566506668 (+0.0036957358390512507) | > avg_loss_disc_real_4: 0.24568829507055417 (+0.0021056927425760763) | > avg_loss_disc_real_5: 0.24638336246282283 (-0.000664941861595908) | > avg_loss_0: 2.7845637160287775 (+0.015643744401528714) | > avg_grad_norm_0: tensor(20.2701, device='cuda:0') (+tensor(0.8725, device='cuda:0')) | > avg_loss_gen: 1.8761328307675644 (+0.013172791037761034) | > avg_loss_kl: 2.2334463781034435 (-0.08559738582288778) | > avg_loss_feat: 2.8078918624931664 (+0.005134700049817198) | > avg_loss_mel: 17.676448190715945 (-0.4416118003952718) | > avg_loss_duration: 1.603045920251121 (-0.010622761618923171) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.19696536534269 (-0.5195244265274255) | > avg_grad_norm_1: tensor(709.8871, device='cuda:0') (+tensor(1.3012, device='cuda:0'))  > EPOCH: 402/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:20:06)   --> TIME: 2025-02-25 12:20:10 -- STEP: 5/72 -- GLOBAL_STEP: 1028950 | > loss_disc: 2.7427334785461426 (2.774624061584473) | > loss_disc_real_0: 0.2062261700630188 (0.22561128586530685) | > loss_disc_real_1: 0.24485230445861816 (0.21751203536987304) | > loss_disc_real_2: 0.22308237850666046 (0.22152828574180602) | > loss_disc_real_3: 0.2429037243127823 (0.24253274500370026) | > loss_disc_real_4: 0.24790214002132416 (0.23200669884681702) | > loss_disc_real_5: 0.2531149685382843 (0.24434859156608582) | > loss_0: 2.7427334785461426 (2.774624061584473) | > grad_norm_0: tensor(13.7688, device='cuda:0') (tensor(19.8356, device='cuda:0')) | > loss_gen: 1.7351300716400146 (1.865553331375122) | > loss_kl: 2.160564422607422 (2.1154003143310547) | > loss_feat: 2.817138671875 (2.8153789043426514) | > loss_mel: 16.957426071166992 (17.65874900817871) | > loss_duration: 1.5482161045074463 (1.5756614446640014) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.218475341796875 (26.03074378967285) | > grad_norm_1: tensor(717.3550, device='cuda:0') (tensor(853.1884, device='cuda:0')) | > current_lr_0: 0.0001901977319831301 | > current_lr_1: 0.0001901977319831301 | > step_time: 0.4303 (0.5298165798187255) | > loader_time: 0.0063 (0.006632280349731445)  --> TIME: 2025-02-25 12:20:33 -- STEP: 55/72 -- GLOBAL_STEP: 1029000 | > loss_disc: 2.811711311340332 (2.7546301278201017) | > loss_disc_real_0: 0.18279165029525757 (0.1914878782900897) | > loss_disc_real_1: 0.2547515630722046 (0.23001383962956343) | > loss_disc_real_2: 0.2378109246492386 (0.2362426459789276) | > loss_disc_real_3: 0.230906143784523 (0.23666014562953602) | > loss_disc_real_4: 0.23852896690368652 (0.24247984859076413) | > loss_disc_real_5: 0.2435431331396103 (0.2452946587042375) | > loss_0: 2.811711311340332 (2.7546301278201017) | > grad_norm_0: tensor(8.0580, device='cuda:0') (tensor(18.3171, device='cuda:0')) | > loss_gen: 1.8025877475738525 (1.8712750738317316) | > loss_kl: 2.294625759124756 (2.213761652599682) | > loss_feat: 2.4740610122680664 (2.887491295554421) | > loss_mel: 17.5118350982666 (17.67699988972057) | > loss_duration: 1.6622848510742188 (1.593503678928722) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.745393753051758 (26.243031623146752) | > grad_norm_1: tensor(1022.4221, device='cuda:0') (tensor(843.3573, device='cuda:0')) | > current_lr_0: 0.0001901977319831301 | > current_lr_1: 0.0001901977319831301 | > step_time: 0.4659 (0.45308314236727626) | > loader_time: 0.0094 (0.007839415290138938) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666874657214526 (-5.238828524737303e-05) | > avg_step_time: 0.46244921482784646 (+0.010622004388083861) | > avg_loss_disc: 2.7540409229171106 (-0.030522793111666946) | > avg_loss_disc_real_0: 0.1932419324424905 (-0.016605767670651556) | > avg_loss_disc_real_1: 0.22788382321596146 (-0.0030108855014115976) | > avg_loss_disc_real_2: 0.2353403692094373 (-0.007198671220054104) | > avg_loss_disc_real_3: 0.2366060473969285 (-0.006248698268138175) | > avg_loss_disc_real_4: 0.24271106090344174 (-0.0029772341671124303) | > avg_loss_disc_real_5: 0.24563948117511372 (-0.0007438812877091083) | > avg_loss_0: 2.7540409229171106 (-0.030522793111666946) | > avg_grad_norm_0: tensor(19.5226, device='cuda:0') (tensor(-0.7475, device='cuda:0')) | > avg_loss_gen: 1.867142917404712 (-0.008989913362852375) | > avg_loss_kl: 2.232231551492718 (-0.0012148266107256944) | > avg_loss_feat: 2.866019504171023 (+0.058127641677856445) | > avg_loss_mel: 17.705667938984615 (+0.029219748268669576) | > avg_loss_duration: 1.6047631871532386 (+0.001717266902117709) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.275825043799173 (+0.07885967845648167) | > avg_grad_norm_1: tensor(919.8495, device='cuda:0') (+tensor(209.9625, device='cuda:0'))  > EPOCH: 403/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:20:43)   --> TIME: 2025-02-25 12:21:00 -- STEP: 33/72 -- GLOBAL_STEP: 1029050 | > loss_disc: 2.78548264503479 (2.744433764255408) | > loss_disc_real_0: 0.25931769609451294 (0.19084344307581583) | > loss_disc_real_1: 0.18184533715248108 (0.2285433610280355) | > loss_disc_real_2: 0.22844457626342773 (0.23580674691633743) | > loss_disc_real_3: 0.19713760912418365 (0.23453106798908926) | > loss_disc_real_4: 0.20059481263160706 (0.24245169803951727) | > loss_disc_real_5: 0.20965616405010223 (0.24391879096175684) | > loss_0: 2.78548264503479 (2.744433764255408) | > grad_norm_0: tensor(13.2103, device='cuda:0') (tensor(18.1007, device='cuda:0')) | > loss_gen: 1.8758761882781982 (1.8714962041739263) | > loss_kl: 2.4216530323028564 (2.2708439682469233) | > loss_feat: 3.115588426589966 (2.9140081766879926) | > loss_mel: 18.001750946044922 (18.107687921235062) | > loss_duration: 1.578532099723816 (1.5785754882928096) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.993398666381836 (26.742611624977805) | > grad_norm_1: tensor(1213.7930, device='cuda:0') (tensor(816.7701, device='cuda:0')) | > current_lr_0: 0.00019017395726663222 | > current_lr_1: 0.00019017395726663222 | > step_time: 0.4511 (0.4397107832359545) | > loader_time: 0.0098 (0.007367885474002723) --> EVAL PERFORMANCE | > avg_loader_time: 0.008934115020322129 (+0.0002672403631076032) | > avg_step_time: 0.4569988788013727 (-0.005450336026473768) | > avg_loss_disc: 2.7576292770009645 (+0.0035883540838539396) | > avg_loss_disc_real_0: 0.19912830414906357 (+0.005886371706573079) | > avg_loss_disc_real_1: 0.2289626995022868 (+0.001078876286325331) | > avg_loss_disc_real_2: 0.23765689281510635 (+0.0023165236056690575) | > avg_loss_disc_real_3: 0.23832522080817692 (+0.0017191734112484147) | > avg_loss_disc_real_4: 0.2442795071803348 (+0.0015684462768930651) | > avg_loss_disc_real_5: 0.24575129864921033 (+0.00011181747409660936) | > avg_loss_0: 2.7576292770009645 (+0.0035883540838539396) | > avg_grad_norm_0: tensor(18.0611, device='cuda:0') (tensor(-1.4614, device='cuda:0')) | > avg_loss_gen: 1.866014322764437 (-0.001128594640275038) | > avg_loss_kl: 2.259557243803857 (+0.02732569231113935) | > avg_loss_feat: 2.837698395823089 (-0.028321108347933777) | > avg_loss_mel: 18.05008189778933 (+0.34441395880471504) | > avg_loss_duration: 1.6106086650364835 (+0.005845477883244854) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.623960548723247 (+0.3481355049240733) | > avg_grad_norm_1: tensor(744.7213, device='cuda:0') (tensor(-175.1282, device='cuda:0'))  > EPOCH: 404/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:21:20)   --> TIME: 2025-02-25 12:21:27 -- STEP: 11/72 -- GLOBAL_STEP: 1029100 | > loss_disc: 2.9274444580078125 (2.7645794695073906) | > loss_disc_real_0: 0.13477833569049835 (0.1804924194108356) | > loss_disc_real_1: 0.2283669412136078 (0.22681481323458932) | > loss_disc_real_2: 0.20422908663749695 (0.25132235342806036) | > loss_disc_real_3: 0.21735744178295135 (0.2351051392880353) | > loss_disc_real_4: 0.2357141524553299 (0.24342172389680689) | > loss_disc_real_5: 0.23967798054218292 (0.2528454783287915) | > loss_0: 2.9274444580078125 (2.7645794695073906) | > grad_norm_0: tensor(27.2268, device='cuda:0') (tensor(19.5661, device='cuda:0')) | > loss_gen: 1.6131794452667236 (1.8794585899873213) | > loss_kl: 2.4770400524139404 (2.265728993849321) | > loss_feat: 2.9796557426452637 (3.0148576606403696) | > loss_mel: 20.437509536743164 (19.887937372381035) | > loss_duration: 1.5636141300201416 (1.5882345221259377) | > amp_scaler: 64.0 (64.0) | > loss_1: 29.070999145507812 (28.63621711730957) | > grad_norm_1: tensor(231.2895, device='cuda:0') (tensor(716.6844, device='cuda:0')) | > current_lr_0: 0.00019015018552197388 | > current_lr_1: 0.00019015018552197388 | > step_time: 0.4299 (0.432992393320257) | > loader_time: 0.0056 (0.00639542666348544)  --> TIME: 2025-02-25 12:21:50 -- STEP: 61/72 -- GLOBAL_STEP: 1029150 | > loss_disc: 2.726222515106201 (2.75128891038113) | > loss_disc_real_0: 0.18657928705215454 (0.18952648427154198) | > loss_disc_real_1: 0.22548028826713562 (0.22811308600863472) | > loss_disc_real_2: 0.24729551374912262 (0.23921717776626836) | > loss_disc_real_3: 0.2230902761220932 (0.23753672251935865) | > loss_disc_real_4: 0.24019195139408112 (0.2451025079019734) | > loss_disc_real_5: 0.2543487846851349 (0.24908464707312036) | > loss_0: 2.726222515106201 (2.75128891038113) | > grad_norm_0: tensor(26.6179, device='cuda:0') (tensor(20.5695, device='cuda:0')) | > loss_gen: 1.7593618631362915 (1.8826734550663682) | > loss_kl: 2.2123444080352783 (2.2716962489925447) | > loss_feat: 2.6415958404541016 (2.914873959588223) | > loss_mel: 16.363061904907227 (18.382792582277393) | > loss_duration: 1.6563910245895386 (1.6022442344759331) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.632755279541016 (27.054280578112994) | > grad_norm_1: tensor(803.9822, device='cuda:0') (tensor(881.8774, device='cuda:0')) | > current_lr_0: 0.00019015018552197388 | > current_lr_1: 0.00019015018552197388 | > step_time: 0.4866 (0.44598217088668074) | > loader_time: 0.0155 (0.008204933072699875) --> EVAL PERFORMANCE | > avg_loader_time: 0.008823351121284592 (-0.00011076389903753696) | > avg_step_time: 0.45679792887727977 (-0.00020094992409291912) | > avg_loss_disc: 2.749642855684522 (-0.007986421316442449) | > avg_loss_disc_real_0: 0.19147665704220113 (-0.007651647106862447) | > avg_loss_disc_real_1: 0.22740001619701655 (-0.001562683305270235) | > avg_loss_disc_real_2: 0.23707767145734437 (-0.0005792213577619776) | > avg_loss_disc_real_3: 0.23628934210454913 (-0.002035878703627786) | > avg_loss_disc_real_4: 0.24327518780466537 (-0.0010043193756694435) | > avg_loss_disc_real_5: 0.2484965204772815 (+0.0027452218280711593) | > avg_loss_0: 2.749642855684522 (-0.007986421316442449) | > avg_grad_norm_0: tensor(21.2758, device='cuda:0') (+tensor(3.2146, device='cuda:0')) | > avg_loss_gen: 1.8766265100156758 (+0.010612187251238803) | > avg_loss_kl: 2.2674069253491678 (+0.007849681545310627) | > avg_loss_feat: 2.8665587633428444 (+0.028860367519755403) | > avg_loss_mel: 18.29384892423388 (+0.2437670264445515) | > avg_loss_duration: 1.6080455041267503 (-0.0025631609097331864) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.912486761388646 (+0.28852621266539913) | > avg_grad_norm_1: tensor(912.4773, device='cuda:0') (+tensor(167.7560, device='cuda:0'))  > EPOCH: 405/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:21:58)   --> TIME: 2025-02-25 12:22:18 -- STEP: 39/72 -- GLOBAL_STEP: 1029200 | > loss_disc: 2.7500555515289307 (2.7526168945508127) | > loss_disc_real_0: 0.17381106317043304 (0.19713128854831058) | > loss_disc_real_1: 0.19601041078567505 (0.22518654702565608) | > loss_disc_real_2: 0.2558760941028595 (0.23826325436433157) | > loss_disc_real_3: 0.2607220411300659 (0.23761984132803404) | > loss_disc_real_4: 0.25781553983688354 (0.24404078187086645) | > loss_disc_real_5: 0.22918547689914703 (0.24591993177548432) | > loss_0: 2.7500555515289307 (2.7526168945508127) | > grad_norm_0: tensor(14.1081, device='cuda:0') (tensor(13.9276, device='cuda:0')) | > loss_gen: 1.8654402494430542 (1.9074737139237232) | > loss_kl: 2.4430172443389893 (2.3271189286158633) | > loss_feat: 2.964261054992676 (3.0176667861449413) | > loss_mel: 18.653736114501953 (18.387354924128605) | > loss_duration: 1.638965129852295 (1.5955155476545677) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.565420150756836 (27.235130065526718) | > grad_norm_1: tensor(1013.3595, device='cuda:0') (tensor(510.7395, device='cuda:0')) | > current_lr_0: 0.00019012641674878364 | > current_lr_1: 0.00019012641674878364 | > step_time: 0.4503 (0.45058735211690265) | > loader_time: 0.008 (0.007290357198470678) --> EVAL PERFORMANCE | > avg_loader_time: 0.008608925510460217 (-0.00021442561082437445) | > avg_step_time: 0.4589430412776034 (+0.002145112400323612) | > avg_loss_disc: 2.7623623122631664 (+0.012719456578644373) | > avg_loss_disc_real_0: 0.19887623726062373 (+0.007399580218422602) | > avg_loss_disc_real_1: 0.23159319833970407 (+0.004193182142687518) | > avg_loss_disc_real_2: 0.2403816841018032 (+0.0033040126444588147) | > avg_loss_disc_real_3: 0.2396268626333962 (+0.00333752052884706) | > avg_loss_disc_real_4: 0.2460573974629523 (+0.0027822096582869327) | > avg_loss_disc_real_5: 0.24775666774998248 (-0.0007398527272990119) | > avg_loss_0: 2.7623623122631664 (+0.012719456578644373) | > avg_grad_norm_0: tensor(17.7459, device='cuda:0') (tensor(-3.5299, device='cuda:0')) | > avg_loss_gen: 1.8990575226259903 (+0.022431012610314438) | > avg_loss_kl: 2.3047053813934326 (+0.03729845604426485) | > avg_loss_feat: 2.8745940235299123 (+0.008035260187067905) | > avg_loss_mel: 18.05911014449429 (-0.23473877973959034) | > avg_loss_duration: 1.620093624356767 (+0.012048120230016668) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.75756072998047 (-0.15492603140817707) | > avg_grad_norm_1: tensor(589.6210, device='cuda:0') (tensor(-322.8563, device='cuda:0'))  > EPOCH: 406/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:22:35)   --> TIME: 2025-02-25 12:22:44 -- STEP: 17/72 -- GLOBAL_STEP: 1029250 | > loss_disc: 2.6168203353881836 (2.711607947069056) | > loss_disc_real_0: 0.20920097827911377 (0.1949999450760729) | > loss_disc_real_1: 0.23519760370254517 (0.22746482491493225) | > loss_disc_real_2: 0.23475439846515656 (0.2346045848201303) | > loss_disc_real_3: 0.20981371402740479 (0.23441842548987446) | > loss_disc_real_4: 0.23546096682548523 (0.24105988968821132) | > loss_disc_real_5: 0.22839918732643127 (0.2432191871544894) | > loss_0: 2.6168203353881836 (2.711607947069056) | > grad_norm_0: tensor(10.1344, device='cuda:0') (tensor(14.5038, device='cuda:0')) | > loss_gen: 1.917002558708191 (1.9079558779211605) | > loss_kl: 2.168586254119873 (2.085764955071842) | > loss_feat: 3.285130023956299 (2.962523670757518) | > loss_mel: 17.833337783813477 (17.544623879825366) | > loss_duration: 1.6048691272735596 (1.5541608824449427) | > amp_scaler: 64.0 (109.17647058823529) | > loss_1: 26.80892562866211 (26.05502936419319) | > grad_norm_1: tensor(1925.8251, device='cuda:0') (tensor(859.8807, device='cuda:0')) | > current_lr_0: 0.00019010265094669004 | > current_lr_1: 0.00019010265094669004 | > step_time: 0.4425 (0.4331971056321088) | > loader_time: 0.0074 (0.006918738870059743)  --> TIME: 2025-02-25 12:23:08 -- STEP: 67/72 -- GLOBAL_STEP: 1029300 | > loss_disc: 2.6814019680023193 (2.7429468347065478) | > loss_disc_real_0: 0.2495398372411728 (0.19771935079079953) | > loss_disc_real_1: 0.23879268765449524 (0.23283221116706507) | > loss_disc_real_2: 0.2117360383272171 (0.2392729289496123) | > loss_disc_real_3: 0.2067902833223343 (0.23757425373169913) | > loss_disc_real_4: 0.2160864621400833 (0.24350257885100238) | > loss_disc_real_5: 0.22041195631027222 (0.2458560989419026) | > loss_0: 2.6814019680023193 (2.7429468347065478) | > grad_norm_0: tensor(29.5328, device='cuda:0') (tensor(20.2420, device='cuda:0')) | > loss_gen: 1.8593350648880005 (1.913088335919736) | > loss_kl: 2.4463541507720947 (2.1371945961197802) | > loss_feat: 2.977126359939575 (2.8905242628126) | > loss_mel: 19.551692962646484 (17.477406245559003) | > loss_duration: 1.7092487812042236 (1.5931939384830531) | > amp_scaler: 64.0 (75.46268656716416) | > loss_1: 28.543758392333984 (26.011407339750832) | > grad_norm_1: tensor(610.3838, device='cuda:0') (tensor(933.6761, device='cuda:0')) | > current_lr_0: 0.00019010265094669004 | > current_lr_1: 0.00019010265094669004 | > step_time: 0.5184 (0.4517431437079586) | > loader_time: 0.0158 (0.008682613942160538) --> EVAL PERFORMANCE | > avg_loader_time: 0.008834617238649181 (+0.00022569172818896407) | > avg_step_time: 0.455582964588219 (-0.0033600766893843725) | > avg_loss_disc: 2.739648076849923 (-0.022714235413243244) | > avg_loss_disc_real_0: 0.1966716798258499 (-0.002204557434773824) | > avg_loss_disc_real_1: 0.2324431446236624 (+0.0008499462839583194) | > avg_loss_disc_real_2: 0.2380422376830813 (-0.0023394464187218855) | > avg_loss_disc_real_3: 0.23644670025563577 (-0.0031801623777604204) | > avg_loss_disc_real_4: 0.2421589306962322 (-0.0038984667667200934) | > avg_loss_disc_real_5: 0.24414901876113784 (-0.003607648988844636) | > avg_loss_0: 2.739648076849923 (-0.022714235413243244) | > avg_grad_norm_0: tensor(20.2220, device='cuda:0') (+tensor(2.4761, device='cuda:0')) | > avg_loss_gen: 1.9151829330014511 (+0.01612541037546089) | > avg_loss_kl: 2.16074690180765 (-0.14395847958578267) | > avg_loss_feat: 2.9078950042455967 (+0.033300980715684325) | > avg_loss_mel: 17.578201602882064 (-0.4809085416122265) | > avg_loss_duration: 1.5980030365393196 (-0.022090587817447327) | > avg_amp_scaler: 74.81690140845068 (+10.816901408450676) | > avg_loss_1: 26.16002947847608 (-0.5975312515043889) | > avg_grad_norm_1: tensor(955.1151, device='cuda:0') (+tensor(365.4941, device='cuda:0'))  > EPOCH: 407/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:23:12)   --> TIME: 2025-02-25 12:23:34 -- STEP: 45/72 -- GLOBAL_STEP: 1029350 | > loss_disc: 2.8298375606536865 (2.7647274865044498) | > loss_disc_real_0: 0.1960580199956894 (0.20166948288679123) | > loss_disc_real_1: 0.21131856739521027 (0.23238510456350114) | > loss_disc_real_2: 0.2318357676267624 (0.24084112180603875) | > loss_disc_real_3: 0.23242977261543274 (0.23799865742524465) | > loss_disc_real_4: 0.22506043314933777 (0.24761109318998117) | > loss_disc_real_5: 0.22916722297668457 (0.24804553621345096) | > loss_0: 2.8298375606536865 (2.7647274865044498) | > grad_norm_0: tensor(7.8579, device='cuda:0') (tensor(19.2061, device='cuda:0')) | > loss_gen: 1.77313232421875 (1.8894525077607898) | > loss_kl: 2.3120367527008057 (2.2166305462519333) | > loss_feat: 2.7991175651550293 (2.895122342639499) | > loss_mel: 17.856042861938477 (18.249524519178603) | > loss_duration: 1.6287388801574707 (1.5957522286309136) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.369068145751953 (26.846482255723743) | > grad_norm_1: tensor(846.4028, device='cuda:0') (tensor(864.2142, device='cuda:0')) | > current_lr_0: 0.0001900788881153217 | > current_lr_1: 0.0001900788881153217 | > step_time: 0.5109 (0.44433351622687445) | > loader_time: 0.0086 (0.00762737062242296) --> EVAL PERFORMANCE | > avg_loader_time: 0.008875148397096444 (+4.0531158447262156e-05) | > avg_step_time: 0.4584937330702661 (+0.002910768482047077) | > avg_loss_disc: 2.7594400828992827 (+0.01979200604935949) | > avg_loss_disc_real_0: 0.2009017522066412 (+0.004230072380791289) | > avg_loss_disc_real_1: 0.23536813636900675 (+0.0029249917453443575) | > avg_loss_disc_real_2: 0.24021126182985977 (+0.00216902414677847) | > avg_loss_disc_real_3: 0.23724563377843777 (+0.0007989335228019978) | > avg_loss_disc_real_4: 0.24572732813761264 (+0.003568397441380433) | > avg_loss_disc_real_5: 0.24785981694577444 (+0.003710798184636599) | > avg_loss_0: 2.7594400828992827 (+0.01979200604935949) | > avg_grad_norm_0: tensor(18.3064, device='cuda:0') (tensor(-1.9156, device='cuda:0')) | > avg_loss_gen: 1.8897053322321933 (-0.02547760076925787) | > avg_loss_kl: 2.250202455990751 (+0.08945555418310125) | > avg_loss_feat: 2.855553039362733 (-0.05234196488286358) | > avg_loss_mel: 18.22622874085332 (+0.6480271379712548) | > avg_loss_duration: 1.6150640803323668 (+0.01706104379304718) | > avg_amp_scaler: 64.0 (-10.816901408450676) | > avg_loss_1: 26.83675376462265 (+0.6767242861465697) | > avg_grad_norm_1: tensor(782.6655, device='cuda:0') (tensor(-172.4495, device='cuda:0'))  > EPOCH: 408/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:23:49)   --> TIME: 2025-02-25 12:24:02 -- STEP: 23/72 -- GLOBAL_STEP: 1029400 | > loss_disc: 2.817441940307617 (2.782029898270317) | > loss_disc_real_0: 0.3000216484069824 (0.1926743174376695) | > loss_disc_real_1: 0.26249268651008606 (0.23739540058633554) | > loss_disc_real_2: 0.204718217253685 (0.2534325226493504) | > loss_disc_real_3: 0.223099023103714 (0.24533545387827832) | > loss_disc_real_4: 0.25352412462234497 (0.24697626803232284) | > loss_disc_real_5: 0.24458235502243042 (0.25148092015929846) | > loss_0: 2.817441940307617 (2.782029898270317) | > grad_norm_0: tensor(19.3289, device='cuda:0') (tensor(18.9709, device='cuda:0')) | > loss_gen: 1.855933666229248 (1.9195457645084546) | > loss_kl: 1.9977242946624756 (2.120953813843105) | > loss_feat: 2.7720210552215576 (2.9144324116084888) | > loss_mel: 17.96796989440918 (18.257040770157527) | > loss_duration: 1.595710039138794 (1.5648501541303552) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.189359664916992 (26.776822712110437) | > grad_norm_1: tensor(596.8052, device='cuda:0') (tensor(1024.4496, device='cuda:0')) | > current_lr_0: 0.00019005512825430728 | > current_lr_1: 0.00019005512825430728 | > step_time: 0.4348 (0.44104197750920837) | > loader_time: 0.0062 (0.00719414586606233) --> EVAL PERFORMANCE | > avg_loader_time: 0.008865040792545806 (-1.01076045506375e-05) | > avg_step_time: 0.4603348214861373 (+0.001841088415871217) | > avg_loss_disc: 2.7751677439246385 (+0.015727661025355832) | > avg_loss_disc_real_0: 0.19528788212739248 (-0.005613870079248712) | > avg_loss_disc_real_1: 0.24344975679693087 (+0.008081620427924124) | > avg_loss_disc_real_2: 0.24481151914092855 (+0.004600257311068778) | > avg_loss_disc_real_3: 0.24260368754326458 (+0.0053580537648268145) | > avg_loss_disc_real_4: 0.24519929373768015 (-0.0005280343999324866) | > avg_loss_disc_real_5: 0.24926294218486453 (+0.0014031252390900861) | > avg_loss_0: 2.7751677439246385 (+0.015727661025355832) | > avg_grad_norm_0: tensor(17.9753, device='cuda:0') (tensor(-0.3311, device='cuda:0')) | > avg_loss_gen: 1.8902385721743946 (+0.0005332399422013356) | > avg_loss_kl: 2.181230911066835 (-0.0689715449239161) | > avg_loss_feat: 2.8463505691205953 (-0.009202470242137828) | > avg_loss_mel: 18.134539281818228 (-0.0916894590350914) | > avg_loss_duration: 1.6170334564128392 (+0.001969376080472429) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.669392599186427 (-0.1673611654362226) | > avg_grad_norm_1: tensor(950.8294, device='cuda:0') (+tensor(168.1639, device='cuda:0'))  > EPOCH: 409/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:24:27)   --> TIME: 2025-02-25 12:24:29 -- STEP: 1/72 -- GLOBAL_STEP: 1029450 | > loss_disc: 2.6779303550720215 (2.6779303550720215) | > loss_disc_real_0: 0.19763852655887604 (0.19763852655887604) | > loss_disc_real_1: 0.22346773743629456 (0.22346773743629456) | > loss_disc_real_2: 0.24284903705120087 (0.24284903705120087) | > loss_disc_real_3: 0.23229007422924042 (0.23229007422924042) | > loss_disc_real_4: 0.25751978158950806 (0.25751978158950806) | > loss_disc_real_5: 0.25329992175102234 (0.25329992175102234) | > loss_0: 2.6779303550720215 (2.6779303550720215) | > grad_norm_0: tensor(12.2392, device='cuda:0') (tensor(12.2392, device='cuda:0')) | > loss_gen: 1.923768162727356 (1.923768162727356) | > loss_kl: 1.9004563093185425 (1.9004563093185425) | > loss_feat: 2.9697041511535645 (2.9697041511535645) | > loss_mel: 18.850467681884766 (18.850467681884766) | > loss_duration: 1.5967750549316406 (1.5967750549316406) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.241172790527344 (27.241172790527344) | > grad_norm_1: tensor(1507.9167, device='cuda:0') (tensor(1507.9167, device='cuda:0')) | > current_lr_0: 0.00019003137136327548 | > current_lr_1: 0.00019003137136327548 | > step_time: 0.4264 (0.42639946937561035) | > loader_time: 0.0076 (0.007598400115966797)  --> TIME: 2025-02-25 12:24:52 -- STEP: 51/72 -- GLOBAL_STEP: 1029500 | > loss_disc: 2.72265362739563 (2.720737859314563) | > loss_disc_real_0: 0.14941784739494324 (0.18846901287050807) | > loss_disc_real_1: 0.21405120193958282 (0.22964235439020045) | > loss_disc_real_2: 0.2492036521434784 (0.2327773436611774) | > loss_disc_real_3: 0.23958152532577515 (0.23570428321174547) | > loss_disc_real_4: 0.24715222418308258 (0.24288192625139274) | > loss_disc_real_5: 0.25481289625167847 (0.24520085459830715) | > loss_0: 2.72265362739563 (2.720737859314563) | > grad_norm_0: tensor(14.1107, device='cuda:0') (tensor(19.9362, device='cuda:0')) | > loss_gen: 1.87046217918396 (1.9073088963826497) | > loss_kl: 2.06207537651062 (2.195724213824553) | > loss_feat: 2.94708251953125 (2.943961970946368) | > loss_mel: 18.29582405090332 (17.901096736683552) | > loss_duration: 1.631946086883545 (1.5896051654628678) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.807390213012695 (26.537696950575885) | > grad_norm_1: tensor(125.0517, device='cuda:0') (tensor(965.7664, device='cuda:0')) | > current_lr_0: 0.00019003137136327548 | > current_lr_1: 0.00019003137136327548 | > step_time: 0.4452 (0.44345750060735967) | > loader_time: 0.0088 (0.007776199602613262) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685783601142987 (-0.00017925719140281932) | > avg_step_time: 0.4551661048136966 (-0.005168716672440676) | > avg_loss_disc: 2.725392620328446 (-0.049775123596192294) | > avg_loss_disc_real_0: 0.18780656862007059 (-0.007481313507321896) | > avg_loss_disc_real_1: 0.2315199316387445 (-0.011929825158186358) | > avg_loss_disc_real_2: 0.23427183611292235 (-0.010539683028006203) | > avg_loss_disc_real_3: 0.23616984108803976 (-0.006433846455224823) | > avg_loss_disc_real_4: 0.2416554328421472 (-0.003543860895532963) | > avg_loss_disc_real_5: 0.2450465973414166 (-0.004216344843447933) | > avg_loss_0: 2.725392620328446 (-0.049775123596192294) | > avg_grad_norm_0: tensor(21.2974, device='cuda:0') (+tensor(3.3221, device='cuda:0')) | > avg_loss_gen: 1.8975426395174484 (+0.007304067343053777) | > avg_loss_kl: 2.198604138804154 (+0.017373227737318864) | > avg_loss_feat: 2.8992955382441132 (+0.05294496912351798) | > avg_loss_mel: 17.79098176284574 (-0.34355751897248865) | > avg_loss_duration: 1.6031976384176334 (-0.013835817995205835) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.38962173461914 (-0.2797708645672863) | > avg_grad_norm_1: tensor(1008.8457, device='cuda:0') (+tensor(58.0163, device='cuda:0'))  > EPOCH: 410/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:25:04)   --> TIME: 2025-02-25 12:25:18 -- STEP: 29/72 -- GLOBAL_STEP: 1029550 | > loss_disc: 2.6597511768341064 (2.7177902254565005) | > loss_disc_real_0: 0.1983446478843689 (0.17885850549771867) | > loss_disc_real_1: 0.21011468768119812 (0.22726258532754307) | > loss_disc_real_2: 0.24799183011054993 (0.2355081490401564) | > loss_disc_real_3: 0.22684036195278168 (0.23424990475177765) | > loss_disc_real_4: 0.23214147984981537 (0.2414857786277245) | > loss_disc_real_5: 0.2473098635673523 (0.24726594470698257) | > loss_0: 2.6597511768341064 (2.7177902254565005) | > grad_norm_0: tensor(7.0017, device='cuda:0') (tensor(20.3344, device='cuda:0')) | > loss_gen: 1.858985185623169 (1.8891731377305656) | > loss_kl: 2.3102197647094727 (2.153492927551269) | > loss_feat: 3.214657783508301 (3.0196788557644547) | > loss_mel: 19.82000160217285 (18.114875530374462) | > loss_duration: 1.6184742450714111 (1.5755165404286877) | > amp_scaler: 64.0 (64.0) | > loss_1: 28.82234001159668 (26.7527369137468) | > grad_norm_1: tensor(143.2970, device='cuda:0') (tensor(922.3105, device='cuda:0')) | > current_lr_0: 0.00019000761744185506 | > current_lr_1: 0.00019000761744185506 | > step_time: 0.4377 (0.43370972830673743) | > loader_time: 0.0083 (0.007057822983840416) --> EVAL PERFORMANCE | > avg_loader_time: 0.008786288785262846 (+0.00010050518411985898) | > avg_step_time: 0.4556289424358959 (+0.0004628376221992858) | > avg_loss_disc: 2.754206281312754 (+0.02881366098430771) | > avg_loss_disc_real_0: 0.19482117670942362 (+0.0070146080893530305) | > avg_loss_disc_real_1: 0.22788452871248754 (-0.00363540292625697) | > avg_loss_disc_real_2: 0.2359634434253397 (+0.0016916073124173536) | > avg_loss_disc_real_3: 0.23800674752450326 (+0.0018369064364635035) | > avg_loss_disc_real_4: 0.2432373367564779 (+0.0015819039143307179) | > avg_loss_disc_real_5: 0.2463584542274475 (+0.0013118568860309154) | > avg_loss_0: 2.754206281312754 (+0.02881366098430771) | > avg_grad_norm_0: tensor(20.6114, device='cuda:0') (tensor(-0.6860, device='cuda:0')) | > avg_loss_gen: 1.8767036048459336 (-0.020839034671514822) | > avg_loss_kl: 2.1942282394624093 (-0.004375899341744649) | > avg_loss_feat: 2.8959528963330765 (-0.0033426419110367256) | > avg_loss_mel: 17.88640231817541 (+0.09542055532967098) | > avg_loss_duration: 1.6081461755322737 (+0.004948537114640272) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.461433222596074 (+0.07181148797693382) | > avg_grad_norm_1: tensor(923.9376, device='cuda:0') (tensor(-84.9081, device='cuda:0'))  > EPOCH: 411/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:25:41)   --> TIME: 2025-02-25 12:25:46 -- STEP: 7/72 -- GLOBAL_STEP: 1029600 | > loss_disc: 2.7674312591552734 (2.6769007614680698) | > loss_disc_real_0: 0.26606208086013794 (0.19588279724121094) | > loss_disc_real_1: 0.21845822036266327 (0.2305036655494145) | > loss_disc_real_2: 0.2264910787343979 (0.22555420015539443) | > loss_disc_real_3: 0.2055288553237915 (0.23810350894927979) | > loss_disc_real_4: 0.22992387413978577 (0.2395043479544776) | > loss_disc_real_5: 0.2316015511751175 (0.24618137095655715) | > loss_0: 2.7674312591552734 (2.6769007614680698) | > grad_norm_0: tensor(18.8422, device='cuda:0') (tensor(20.1650, device='cuda:0')) | > loss_gen: 1.8829891681671143 (1.9393558842795235) | > loss_kl: 1.865954875946045 (1.9497946671077184) | > loss_feat: 3.025827646255493 (2.9344502857753207) | > loss_mel: 17.18147087097168 (17.51327214922224) | > loss_duration: 1.5351200103759766 (1.5602918352399553) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.491363525390625 (25.897165025983536) | > grad_norm_1: tensor(1240.7456, device='cuda:0') (tensor(886.9002, device='cuda:0')) | > current_lr_0: 0.0001899838664896748 | > current_lr_1: 0.0001899838664896748 | > step_time: 0.4385 (0.4335123470851353) | > loader_time: 0.0055 (0.006801775523594448)  --> TIME: 2025-02-25 12:26:09 -- STEP: 57/72 -- GLOBAL_STEP: 1029650 | > loss_disc: 2.7058234214782715 (2.7394795124990896) | > loss_disc_real_0: 0.16479207575321198 (0.19305617872037387) | > loss_disc_real_1: 0.19173428416252136 (0.2281495097436403) | > loss_disc_real_2: 0.22097760438919067 (0.23526862917239205) | > loss_disc_real_3: 0.22618848085403442 (0.23744222312642818) | > loss_disc_real_4: 0.26237744092941284 (0.24494991804424085) | > loss_disc_real_5: 0.2559439539909363 (0.2464088904753066) | > loss_0: 2.7058234214782715 (2.7394795124990896) | > grad_norm_0: tensor(24.0784, device='cuda:0') (tensor(17.7586, device='cuda:0')) | > loss_gen: 1.8051451444625854 (1.8914396700106169) | > loss_kl: 2.328119993209839 (2.32337885153921) | > loss_feat: 2.9787323474884033 (2.9106286701403166) | > loss_mel: 17.747499465942383 (18.364795651352193) | > loss_duration: 1.6546050310134888 (1.6021483362766735) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.514102935791016 (27.092391265066045) | > grad_norm_1: tensor(1222.1857, device='cuda:0') (tensor(744.9784, device='cuda:0')) | > current_lr_0: 0.0001899838664896748 | > current_lr_1: 0.0001899838664896748 | > step_time: 0.4553 (0.44654653365151925) | > loader_time: 0.0096 (0.008032581262421191) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737453272644902 (-4.883551261794343e-05) | > avg_step_time: 0.4565724184815313 (+0.000943476045635383) | > avg_loss_disc: 2.750117926530435 (-0.004088354782318948) | > avg_loss_disc_real_0: 0.1946574481020511 (-0.00016372860737251527) | > avg_loss_disc_real_1: 0.2328354233587292 (+0.004950894646241655) | > avg_loss_disc_real_2: 0.23731397050367276 (+0.0013505270783330525) | > avg_loss_disc_real_3: 0.23646729659866278 (-0.0015394509258404854) | > avg_loss_disc_real_4: 0.2431804785426234 (-5.68582138545104e-05) | > avg_loss_disc_real_5: 0.24484287328283552 (-0.0015155809446119883) | > avg_loss_0: 2.750117926530435 (-0.004088354782318948) | > avg_grad_norm_0: tensor(20.1719, device='cuda:0') (tensor(-0.4395, device='cuda:0')) | > avg_loss_gen: 1.8934960768256388 (+0.016792471979705237) | > avg_loss_kl: 2.320733970319721 (+0.12650573085731187) | > avg_loss_feat: 2.8829902998158627 (-0.012962596517213765) | > avg_loss_mel: 18.398021308469097 (+0.5116189902936874) | > avg_loss_duration: 1.6110570162114963 (+0.0029108406792226482) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 27.106298742159993 (+0.6448655195639184) | > avg_grad_norm_1: tensor(789.6115, device='cuda:0') (tensor(-134.3261, device='cuda:0'))  > EPOCH: 412/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:26:18)   --> TIME: 2025-02-25 12:26:36 -- STEP: 35/72 -- GLOBAL_STEP: 1029700 | > loss_disc: 2.8323018550872803 (2.755449969427926) | > loss_disc_real_0: 0.19359683990478516 (0.19248776010104587) | > loss_disc_real_1: 0.282902330160141 (0.22693729613508498) | > loss_disc_real_2: 0.2338535487651825 (0.2405352222067969) | > loss_disc_real_3: 0.22840406000614166 (0.24121827014854977) | > loss_disc_real_4: 0.2502671778202057 (0.2475384656872068) | > loss_disc_real_5: 0.25385046005249023 (0.25142126509121476) | > loss_0: 2.8323018550872803 (2.755449969427926) | > grad_norm_0: tensor(20.0257, device='cuda:0') (tensor(22.7363, device='cuda:0')) | > loss_gen: 1.7902554273605347 (1.8918798344475882) | > loss_kl: 2.54215145111084 (2.1132518972669323) | > loss_feat: 2.810678243637085 (2.8906518595559256) | > loss_mel: 18.57168197631836 (17.8865359715053) | > loss_duration: 1.6265099048614502 (1.57412497316088) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.341276168823242 (26.356444549560546) | > grad_norm_1: tensor(361.1192, device='cuda:0') (tensor(963.2772, device='cuda:0')) | > current_lr_0: 0.00018996011850636358 | > current_lr_1: 0.00018996011850636358 | > step_time: 0.4434 (0.44110151699611116) | > loader_time: 0.0081 (0.00731370108468192) --> EVAL PERFORMANCE | > avg_loader_time: 0.008724699557667049 (-1.2753714977853728e-05) | > avg_step_time: 0.45632475530597527 (-0.00024766317555602324) | > avg_loss_disc: 2.746245256611997 (-0.003872669918437932) | > avg_loss_disc_real_0: 0.1887344794583992 (-0.00592296864365191) | > avg_loss_disc_real_1: 0.22761396098304804 (-0.005221462375681163) | > avg_loss_disc_real_2: 0.2396893557948126 (+0.002375385291139842) | > avg_loss_disc_real_3: 0.23892857847918927 (+0.0024612818805264947) | > avg_loss_disc_real_4: 0.24690563032324886 (+0.003725151780625463) | > avg_loss_disc_real_5: 0.25016062885103085 (+0.0053177555681953315) | > avg_loss_0: 2.746245256611997 (-0.003872669918437932) | > avg_grad_norm_0: tensor(21.9144, device='cuda:0') (+tensor(1.7425, device='cuda:0')) | > avg_loss_gen: 1.9042553347601017 (+0.010759257934462862) | > avg_loss_kl: 2.189532659423183 (-0.1312013108965382) | > avg_loss_feat: 2.91462259897044 (+0.031632299154577215) | > avg_loss_mel: 18.185104504437515 (-0.21291680403158253) | > avg_loss_duration: 1.6062815525162388 (-0.004775463695257498) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.799796735736685 (-0.30650200642330816) | > avg_grad_norm_1: tensor(1051.1812, device='cuda:0') (+tensor(261.5696, device='cuda:0'))  > EPOCH: 413/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:26:55)   --> TIME: 2025-02-25 12:27:03 -- STEP: 13/72 -- GLOBAL_STEP: 1029750 | > loss_disc: 2.721052885055542 (2.7676664682535024) | > loss_disc_real_0: 0.22330932319164276 (0.20862942188978195) | > loss_disc_real_1: 0.2189059555530548 (0.24384344197236574) | > loss_disc_real_2: 0.2627512812614441 (0.23577176149074847) | > loss_disc_real_3: 0.2262396514415741 (0.23849489024052253) | > loss_disc_real_4: 0.24462927877902985 (0.24445430705180535) | > loss_disc_real_5: 0.26318374276161194 (0.24212732108739707) | > loss_0: 2.721052885055542 (2.7676664682535024) | > grad_norm_0: tensor(16.4969, device='cuda:0') (tensor(19.6684, device='cuda:0')) | > loss_gen: 1.940525770187378 (1.9105369861309345) | > loss_kl: 2.2858052253723145 (2.2561860359632053) | > loss_feat: 2.80830979347229 (2.874929739878728) | > loss_mel: 17.642227172851562 (18.754977299616886) | > loss_duration: 1.5184650421142578 (1.625948740885808) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.19533348083496 (27.42257939852201) | > grad_norm_1: tensor(263.3293, device='cuda:0') (tensor(895.5278, device='cuda:0')) | > current_lr_0: 0.00018993637349155027 | > current_lr_1: 0.00018993637349155027 | > step_time: 0.4567 (0.43473122670100284) | > loader_time: 0.0058 (0.007001693432147686)  --> TIME: 2025-02-25 12:27:26 -- STEP: 63/72 -- GLOBAL_STEP: 1029800 | > loss_disc: 2.6941781044006348 (2.7617012394799128) | > loss_disc_real_0: 0.1909925937652588 (0.20860643010763896) | > loss_disc_real_1: 0.1834694743156433 (0.23379595837895834) | > loss_disc_real_2: 0.25338229537010193 (0.2364154751338656) | > loss_disc_real_3: 0.24710968136787415 (0.2387767999418198) | > loss_disc_real_4: 0.2349628508090973 (0.2439885271920098) | > loss_disc_real_5: 0.23436762392520905 (0.24598208684769887) | > loss_0: 2.6941781044006348 (2.7617012394799128) | > grad_norm_0: tensor(9.5438, device='cuda:0') (tensor(18.8571, device='cuda:0')) | > loss_gen: 1.937735676765442 (1.8874709625092765) | > loss_kl: 2.1691949367523193 (2.2073404732204622) | > loss_feat: 2.7655463218688965 (2.7841358222658674) | > loss_mel: 17.65776252746582 (17.846079145159038) | > loss_duration: 1.603263020515442 (1.6156780171015905) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.133502960205078 (26.34070450919015) | > grad_norm_1: tensor(319.3601, device='cuda:0') (tensor(887.9564, device='cuda:0')) | > current_lr_0: 0.00018993637349155027 | > current_lr_1: 0.00018993637349155027 | > step_time: 0.5159 (0.4499111213381328) | > loader_time: 0.0112 (0.008143228197854664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008546829223632814 (-0.0001778703340342344) | > avg_step_time: 0.45744381152408226 (+0.001119056218106984) | > avg_loss_disc: 2.7610053881792953 (+0.014760131567298274) | > avg_loss_disc_real_0: 0.2075074666612585 (+0.018772987202859304) | > avg_loss_disc_real_1: 0.23303887279520571 (+0.005424911812157679) | > avg_loss_disc_real_2: 0.23633320944409975 (-0.003356146350712852) | > avg_loss_disc_real_3: 0.23794037985130095 (-0.0009881986278883204) | > avg_loss_disc_real_4: 0.24355914517187735 (-0.003346485151371509) | > avg_loss_disc_real_5: 0.2452125221910611 (-0.004948106659969753) | > avg_loss_0: 2.7610053881792953 (+0.014760131567298274) | > avg_grad_norm_0: tensor(18.7921, device='cuda:0') (tensor(-3.1223, device='cuda:0')) | > avg_loss_gen: 1.8797983602738717 (-0.024456974486229965) | > avg_loss_kl: 2.2115134235838774 (+0.021980764160694388) | > avg_loss_feat: 2.778137599918204 (-0.13648499905223588) | > avg_loss_mel: 17.831103875603475 (-0.35400062883404004) | > avg_loss_duration: 1.620770085025841 (+0.014488532509602248) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.321323421639462 (-0.47847331409722216) | > avg_grad_norm_1: tensor(895.9353, device='cuda:0') (tensor(-155.2458, device='cuda:0'))  > EPOCH: 414/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:27:32)   --> TIME: 2025-02-25 12:27:53 -- STEP: 41/72 -- GLOBAL_STEP: 1029850 | > loss_disc: 2.8114728927612305 (2.7349016259356245) | > loss_disc_real_0: 0.10413972288370132 (0.17966252983343312) | > loss_disc_real_1: 0.2653946876525879 (0.22800358003232538) | > loss_disc_real_2: 0.2163657695055008 (0.24236993855092584) | > loss_disc_real_3: 0.23598650097846985 (0.23810814102975333) | > loss_disc_real_4: 0.25209593772888184 (0.24329348562694178) | > loss_disc_real_5: 0.23931804299354553 (0.2492649667873615) | > loss_0: 2.8114728927612305 (2.7349016259356245) | > grad_norm_0: tensor(32.8578, device='cuda:0') (tensor(27.7804, device='cuda:0')) | > loss_gen: 1.7651054859161377 (1.9184106733740829) | > loss_kl: 2.0045318603515625 (2.069410175811954) | > loss_feat: 2.914268732070923 (3.020150894071998) | > loss_mel: 18.284875869750977 (17.53013392192561) | > loss_duration: 1.581303358078003 (1.5890869948922135) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.550085067749023 (26.127192334430973) | > grad_norm_1: tensor(1351.8082, device='cuda:0') (tensor(1417.9117, device='cuda:0')) | > current_lr_0: 0.00018991263144486382 | > current_lr_1: 0.00018991263144486382 | > step_time: 0.4585 (0.4432961882614508) | > loader_time: 0.0091 (0.00782171691336283) --> EVAL PERFORMANCE | > avg_loader_time: 0.008974746919014082 (+0.0004279176953812681) | > avg_step_time: 0.4597086033350985 (+0.0022647918110162424) | > avg_loss_disc: 2.744316134654301 (-0.016689253524994285) | > avg_loss_disc_real_0: 0.19556351992445933 (-0.01194394673679916) | > avg_loss_disc_real_1: 0.2296830864859299 (-0.0033557863092758267) | > avg_loss_disc_real_2: 0.23951431332339704 (+0.0031811038792972923) | > avg_loss_disc_real_3: 0.23893843108499555 (+0.0009980512336945913) | > avg_loss_disc_real_4: 0.2436602491308266 (+0.00010110395894924462) | > avg_loss_disc_real_5: 0.24769231509154951 (+0.0024797929004884145) | > avg_loss_0: 2.744316134654301 (-0.016689253524994285) | > avg_grad_norm_0: tensor(28.0725, device='cuda:0') (+tensor(9.2804, device='cuda:0')) | > avg_loss_gen: 1.9060149545400915 (+0.026216594266219806) | > avg_loss_kl: 2.101668898488434 (-0.1098445250954434) | > avg_loss_feat: 2.8981947227263114 (+0.12005712280810732) | > avg_loss_mel: 17.345499616273692 (-0.4856042593297829) | > avg_loss_duration: 1.6057148449857468 (-0.015055240040094287) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 25.857092924521 (-0.46423049711846076) | > avg_grad_norm_1: tensor(1211.8041, device='cuda:0') (+tensor(315.8688, device='cuda:0'))  > EPOCH: 415/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:28:10)   --> TIME: 2025-02-25 12:28:20 -- STEP: 19/72 -- GLOBAL_STEP: 1029900 | > loss_disc: 2.687626361846924 (2.734089738444278) | > loss_disc_real_0: 0.19749024510383606 (0.18672121276980952) | > loss_disc_real_1: 0.23093090951442719 (0.24750400216955887) | > loss_disc_real_2: 0.24771736562252045 (0.24278871715068817) | > loss_disc_real_3: 0.2530517578125 (0.24556747000468404) | > loss_disc_real_4: 0.2569273114204407 (0.24919370836333224) | > loss_disc_real_5: 0.24646468460559845 (0.24651088761655907) | > loss_0: 2.687626361846924 (2.734089738444278) | > grad_norm_0: tensor(6.0636, device='cuda:0') (tensor(20.4010, device='cuda:0')) | > loss_gen: 1.7061399221420288 (1.9456075743625039) | > loss_kl: 2.5531671047210693 (2.0924009335668465) | > loss_feat: 2.9008359909057617 (3.015274248625103) | > loss_mel: 19.0096435546875 (17.67406604164525) | > loss_duration: 1.6826251745224 (1.5669313230012591) | > amp_scaler: 64.0 (64.0) | > loss_1: 27.8524112701416 (26.294280202765215) | > grad_norm_1: tensor(135.1495, device='cuda:0') (tensor(1123.8109, device='cuda:0')) | > current_lr_0: 0.00018988889236593322 | > current_lr_1: 0.00018988889236593322 | > step_time: 0.4305 (0.43945904781943873) | > loader_time: 0.0076 (0.006797690140573602)  --> TIME: 2025-02-25 12:28:44 -- STEP: 69/72 -- GLOBAL_STEP: 1029950 | > loss_disc: 2.822312355041504 (2.7425684376039365) | > loss_disc_real_0: 0.23622028529644012 (0.18828796260598776) | > loss_disc_real_1: 0.23517407476902008 (0.23610123128130817) | > loss_disc_real_2: 0.23302367329597473 (0.24075886520786563) | > loss_disc_real_3: 0.23677383363246918 (0.23931946227515954) | > loss_disc_real_4: 0.2410152405500412 (0.2442341189885485) | > loss_disc_real_5: 0.24910487234592438 (0.24528534883174344) | > loss_0: 2.822312355041504 (2.7425684376039365) | > grad_norm_0: tensor(9.9518, device='cuda:0') (tensor(20.1804, device='cuda:0')) | > loss_gen: 1.7036263942718506 (1.9318401260652405) | > loss_kl: 2.3970372676849365 (2.3253734992898027) | > loss_feat: 2.608900308609009 (3.0209731191828633) | > loss_mel: 18.319583892822266 (18.383224680803817) | > loss_duration: 1.717474102973938 (1.628477086191592) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.74662208557129 (27.289888879527215) | > grad_norm_1: tensor(1190.8384, device='cuda:0') (tensor(1141.5791, device='cuda:0')) | > current_lr_0: 0.00018988889236593322 | > current_lr_1: 0.00018988889236593322 | > step_time: 0.538 (0.4564501997353374) | > loader_time: 0.0124 (0.008569793424744534) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619798740870512 (-0.0003549481781435701) | > avg_step_time: 0.4582633535626908 (-0.0014452497724076752) | > avg_loss_disc: 2.743481310320572 (-0.0008348243337290562) | > avg_loss_disc_real_0: 0.18706729126648164 (-0.008496228657977695) | > avg_loss_disc_real_1: 0.2349287640880531 (+0.0052456776021232165) | > avg_loss_disc_real_2: 0.24097089175607117 (+0.0014565784326741327) | > avg_loss_disc_real_3: 0.2382210906542523 (-0.0007173404307432574) | > avg_loss_disc_real_4: 0.24379580163619888 (+0.0001355525053722828) | > avg_loss_disc_real_5: 0.24513216841388757 (-0.0025601466776619397) | > avg_loss_0: 2.743481310320572 (-0.0008348243337290562) | > avg_grad_norm_0: tensor(20.1458, device='cuda:0') (tensor(-7.9267, device='cuda:0')) | > avg_loss_gen: 1.928190162484075 (+0.02217520794398342) | > avg_loss_kl: 2.325302263380776 (+0.2236333648923421) | > avg_loss_feat: 3.018309173449664 (+0.12011445072335247) | > avg_loss_mel: 18.385103145115806 (+1.0396035288421146) | > avg_loss_duration: 1.6283365017931226 (+0.022621656807375823) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 27.2852415971353 (+1.4281486726142987) | > avg_grad_norm_1: tensor(1135.7003, device='cuda:0') (tensor(-76.1038, device='cuda:0'))  > EPOCH: 416/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:28:47)   --> TIME: 2025-02-25 12:29:10 -- STEP: 47/72 -- GLOBAL_STEP: 1030000 | > loss_disc: 2.792229652404785 (2.7554461499477947) | > loss_disc_real_0: 0.1960507184267044 (0.19367094448906316) | > loss_disc_real_1: 0.26159989833831787 (0.23525976310385036) | > loss_disc_real_2: 0.251980185508728 (0.23643388456486641) | > loss_disc_real_3: 0.2594345510005951 (0.23765502172581693) | > loss_disc_real_4: 0.27382171154022217 (0.24727732038244285) | > loss_disc_real_5: 0.26586779952049255 (0.25096705365688243) | > loss_0: 2.792229652404785 (2.7554461499477947) | > grad_norm_0: tensor(5.8165, device='cuda:0') (tensor(20.4768, device='cuda:0')) | > loss_gen: 1.7909971475601196 (1.9174163620522682) | > loss_kl: 2.259490728378296 (2.3313425272069077) | > loss_feat: 2.2925829887390137 (2.9965715103961053) | > loss_mel: 17.265789031982422 (18.713065695255366) | > loss_duration: 1.6371502876281738 (1.625951183603165) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.246009826660156 (27.584347177059094) | > grad_norm_1: tensor(148.5451, device='cuda:0') (tensor(876.4487, device='cuda:0')) | > current_lr_0: 0.00018986515625438747 | > current_lr_1: 0.00018986515625438747 | > step_time: 0.4482 (0.4403649391011989) | > loader_time: 0.0087 (0.007509398967661756) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1030000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.00850914229809398 (-0.00011065644277653136) | > avg_step_time: 0.45415785950674137 (-0.004105494055949455) | > avg_loss_disc: 2.76169521707884 (+0.01821390675826784) | > avg_loss_disc_real_0: 0.19807559765980282 (+0.011008306393321177) | > avg_loss_disc_real_1: 0.2316732236617048 (-0.003255540426348291) | > avg_loss_disc_real_2: 0.23747978327979505 (-0.0034911084762761224) | > avg_loss_disc_real_3: 0.23752523350044036 (-0.0006958571538119296) | > avg_loss_disc_real_4: 0.24444622577915728 (+0.0006504241429584057) | > avg_loss_disc_real_5: 0.2500643092142025 (+0.004932140800314899) | > avg_loss_0: 2.76169521707884 (+0.01821390675826784) | > avg_grad_norm_0: tensor(20.0281, device='cuda:0') (tensor(-0.1177, device='cuda:0')) | > avg_loss_gen: 1.8919337719259128 (-0.0362563905581621) | > avg_loss_kl: 2.315015623267268 (-0.010286640113508216) | > avg_loss_feat: 2.90934831323758 (-0.10896086021208395) | > avg_loss_mel: 18.42722371598365 (+0.042120570867844265) | > avg_loss_duration: 1.6321225905082595 (+0.0037860887151368594) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 27.175643894034373 (-0.10959770310092765) | > avg_grad_norm_1: tensor(827.3524, device='cuda:0') (tensor(-308.3480, device='cuda:0'))  > EPOCH: 417/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:29:27)   --> TIME: 2025-02-25 12:29:40 -- STEP: 25/72 -- GLOBAL_STEP: 1030050 | > loss_disc: 2.6564583778381348 (2.751477670669556) | > loss_disc_real_0: 0.1554504632949829 (0.2018487721681595) | > loss_disc_real_1: 0.22963441908359528 (0.24374481797218323) | > loss_disc_real_2: 0.1933276653289795 (0.2365950483083725) | > loss_disc_real_3: 0.22345556318759918 (0.23346566617488862) | > loss_disc_real_4: 0.2416601926088333 (0.24224897146224975) | > loss_disc_real_5: 0.261659175157547 (0.2466048228740692) | > loss_0: 2.6564583778381348 (2.751477670669556) | > grad_norm_0: tensor(13.3506, device='cuda:0') (tensor(22.0890, device='cuda:0')) | > loss_gen: 1.8835904598236084 (1.9107800149917602) | > loss_kl: 2.2103922367095947 (2.1683330154418945) | > loss_feat: 3.1807501316070557 (2.8993553066253663) | > loss_mel: 17.58519744873047 (18.08357192993164) | > loss_duration: 1.5816853046417236 (1.5637723875045777) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.44161605834961 (26.62581283569336) | > grad_norm_1: tensor(106.8098, device='cuda:0') (tensor(672.5111, device='cuda:0')) | > current_lr_0: 0.00018984142310985566 | > current_lr_1: 0.00018984142310985566 | > step_time: 0.4276 (0.4348402786254883) | > loader_time: 0.0071 (0.0068400192260742185) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624197731555348 (+0.00011505543346136682) | > avg_step_time: 0.4536296951938683 (-0.0005281643128730584) | > avg_loss_disc: 2.739201048730125 (-0.022494168348714982) | > avg_loss_disc_real_0: 0.19411628630379557 (-0.003959311356007245) | > avg_loss_disc_real_1: 0.23315156177735666 (+0.001478338115651845) | > avg_loss_disc_real_2: 0.23829583173066798 (+0.000816048450872936) | > avg_loss_disc_real_3: 0.23510403259539267 (-0.002421200905047688) | > avg_loss_disc_real_4: 0.24237329922091794 (-0.0020729265582393464) | > avg_loss_disc_real_5: 0.24438633150617842 (-0.005677977708024057) | > avg_loss_0: 2.739201048730125 (-0.022494168348714982) | > avg_grad_norm_0: tensor(23.2682, device='cuda:0') (+tensor(3.2401, device='cuda:0')) | > avg_loss_gen: 1.9070098332955803 (+0.015076061369667437) | > avg_loss_kl: 2.1393410370383465 (-0.1756745862289213) | > avg_loss_feat: 2.9472677237550977 (+0.03791941051751779) | > avg_loss_mel: 17.796182820494746 (-0.6310408954889049) | > avg_loss_duration: 1.597490325779982 (-0.03463226472827752) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.387291921696193 (-0.7883519723381802) | > avg_grad_norm_1: tensor(1100.7357, device='cuda:0') (+tensor(273.3834, device='cuda:0'))  > EPOCH: 418/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:30:04)   --> TIME: 2025-02-25 12:30:07 -- STEP: 3/72 -- GLOBAL_STEP: 1030100 | > loss_disc: 2.778888702392578 (2.738910516103109) | > loss_disc_real_0: 0.21892264485359192 (0.27887023488680523) | > loss_disc_real_1: 0.27812841534614563 (0.24589181939760843) | > loss_disc_real_2: 0.2503924071788788 (0.2447874148686727) | > loss_disc_real_3: 0.28444811701774597 (0.2661159932613373) | > loss_disc_real_4: 0.28466174006462097 (0.26806263128916424) | > loss_disc_real_5: 0.28529444336891174 (0.2643787016471227) | > loss_0: 2.778888702392578 (2.738910516103109) | > grad_norm_0: tensor(9.0173, device='cuda:0') (tensor(21.5159, device='cuda:0')) | > loss_gen: 1.6987533569335938 (1.9366413752237956) | > loss_kl: 2.0509836673736572 (2.1225143671035767) | > loss_feat: 2.7498459815979004 (2.8774665196736655) | > loss_mel: 17.693620681762695 (18.58893585205078) | > loss_duration: 1.593632459640503 (1.5782878796259563) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.786834716796875 (27.103845596313477) | > grad_norm_1: tensor(91.9656, device='cuda:0') (tensor(939.4270, device='cuda:0')) | > current_lr_0: 0.00018981769293196693 | > current_lr_1: 0.00018981769293196693 | > step_time: 0.4226 (0.43138329188028973) | > loader_time: 0.0064 (0.006087462107340495)  --> TIME: 2025-02-25 12:30:30 -- STEP: 53/72 -- GLOBAL_STEP: 1030150 | > loss_disc: 2.786160707473755 (2.754892214289252) | > loss_disc_real_0: 0.15954208374023438 (0.19202723129177993) | > loss_disc_real_1: 0.21589013934135437 (0.228315384320493) | > loss_disc_real_2: 0.24228504300117493 (0.23835035090176565) | > loss_disc_real_3: 0.26102414727211 (0.23623810483599608) | > loss_disc_real_4: 0.23944327235221863 (0.2428295078142634) | > loss_disc_real_5: 0.2554139792919159 (0.2484350623387211) | > loss_0: 2.786160707473755 (2.754892214289252) | > grad_norm_0: tensor(17.7868, device='cuda:0') (tensor(23.9002, device='cuda:0')) | > loss_gen: 1.858903408050537 (1.8907061995200392) | > loss_kl: 2.3066136837005615 (2.3521016656227838) | > loss_feat: 2.906073570251465 (2.9571409315433144) | > loss_mel: 15.867753982543945 (18.409660231392337) | > loss_duration: 1.6474355459213257 (1.6042554108601699) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.586780548095703 (27.213864380458617) | > grad_norm_1: tensor(127.7267, device='cuda:0') (tensor(896.5645, device='cuda:0')) | > current_lr_0: 0.00018981769293196693 | > current_lr_1: 0.00018981769293196693 | > step_time: 0.4703 (0.44487829928128225) | > loader_time: 0.0094 (0.007985956263992022) --> EVAL PERFORMANCE | > avg_loader_time: 0.008740572862222162 (+0.00011637513066681433) | > avg_step_time: 0.4565365448803969 (+0.002906849686528601) | > avg_loss_disc: 2.734936552987972 (-0.0042644957421527074) | > avg_loss_disc_real_0: 0.19055653373960038 (-0.00355975256419519) | > avg_loss_disc_real_1: 0.2301420741517779 (-0.0030094876255787606) | > avg_loss_disc_real_2: 0.23649355223481083 (-0.0018022794958571509) | > avg_loss_disc_real_3: 0.23391150546745515 (-0.0011925271279375205) | > avg_loss_disc_real_4: 0.24113182786484838 (-0.0012414713560695567) | > avg_loss_disc_real_5: 0.24512028966990995 (+0.0007339581637315351) | > avg_loss_0: 2.734936552987972 (-0.0042644957421527074) | > avg_grad_norm_0: tensor(23.1324, device='cuda:0') (tensor(-0.1358, device='cuda:0')) | > avg_loss_gen: 1.9065591751689641 (-0.0004506581266161547) | > avg_loss_kl: 2.317120053398778 (+0.17777901636043136) | > avg_loss_feat: 2.9642884059691093 (+0.01702068221401154) | > avg_loss_mel: 18.241807131700114 (+0.4456243112053677) | > avg_loss_duration: 1.6139275977309322 (+0.01643727195095024) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 27.043702273301676 (+0.6564103516054836) | > avg_grad_norm_1: tensor(915.1317, device='cuda:0') (tensor(-185.6040, device='cuda:0'))  > EPOCH: 419/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:30:41)   --> TIME: 2025-02-25 12:30:56 -- STEP: 31/72 -- GLOBAL_STEP: 1030200 | > loss_disc: 2.6975033283233643 (2.7356939854160434) | > loss_disc_real_0: 0.15994945168495178 (0.18749643093155277) | > loss_disc_real_1: 0.2719865143299103 (0.24367809920541703) | > loss_disc_real_2: 0.17839941382408142 (0.23525484723429527) | > loss_disc_real_3: 0.21649256348609924 (0.23963321889600447) | > loss_disc_real_4: 0.22761835157871246 (0.24395741570380428) | > loss_disc_real_5: 0.22970223426818848 (0.24855749309062955) | > loss_0: 2.6975033283233643 (2.7356939854160434) | > grad_norm_0: tensor(7.5562, device='cuda:0') (tensor(17.9793, device='cuda:0')) | > loss_gen: 1.927311658859253 (1.9184540433268393) | > loss_kl: 1.9452837705612183 (2.0039079919938114) | > loss_feat: 2.8597426414489746 (2.9730401269851194) | > loss_mel: 17.670564651489258 (17.695889380670355) | > loss_duration: 1.5993103981018066 (1.5574856035171016) | > amp_scaler: 64.0 (64.0) | > loss_1: 26.002212524414062 (26.148777069584018) | > grad_norm_1: tensor(185.7865, device='cuda:0') (tensor(989.0649, device='cuda:0')) | > current_lr_0: 0.00018979396572035043 | > current_lr_1: 0.00018979396572035043 | > step_time: 0.4382 (0.43542715041868146) | > loader_time: 0.0083 (0.007078701449978736) --> EVAL PERFORMANCE | > avg_loader_time: 0.008750126395427006 (+9.553533204843812e-06) | > avg_step_time: 0.4609722856064917 (+0.004435740726094817) | > avg_loss_disc: 2.7392248099958403 (+0.004288257007868168) | > avg_loss_disc_real_0: 0.19295475096769735 (+0.002398217228096966) | > avg_loss_disc_real_1: 0.23450654682139277 (+0.004364472669614872) | > avg_loss_disc_real_2: 0.2367235650898705 (+0.00023001285505966362) | > avg_loss_disc_real_3: 0.23758856826265093 (+0.003677062795195779) | > avg_loss_disc_real_4: 0.24395726348312807 (+0.0028254356182796925) | > avg_loss_disc_real_5: 0.24692252173390186 (+0.0018022320639919098) | > avg_loss_0: 2.7392248099958403 (+0.004288257007868168) | > avg_grad_norm_0: tensor(18.0432, device='cuda:0') (tensor(-5.0892, device='cuda:0')) | > avg_loss_gen: 1.9049388623573411 (-0.0016203128116230037) | > avg_loss_kl: 2.107016180602597 (-0.2101038727961808) | > avg_loss_feat: 2.9434928658982398 (-0.020795540070869478) | > avg_loss_mel: 17.793280520909267 (-0.44852661079084655) | > avg_loss_duration: 1.59965662217476 (-0.014270975556172294) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.34838499149806 (-0.6953172818036144) | > avg_grad_norm_1: tensor(1051.8479, device='cuda:0') (+tensor(136.7162, device='cuda:0'))  > EPOCH: 420/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:31:18)   --> TIME: 2025-02-25 12:31:24 -- STEP: 9/72 -- GLOBAL_STEP: 1030250 | > loss_disc: 2.793301582336426 (2.714543792936537) | > loss_disc_real_0: 0.12324251234531403 (0.18966272638903725) | > loss_disc_real_1: 0.21193550527095795 (0.21518411238988241) | > loss_disc_real_2: 0.22270283102989197 (0.23596335616376665) | > loss_disc_real_3: 0.22574882209300995 (0.2364533527029885) | > loss_disc_real_4: 0.23573656380176544 (0.23868034448888567) | > loss_disc_real_5: 0.25240176916122437 (0.2430752565463384) | > loss_0: 2.793301582336426 (2.714543792936537) | > grad_norm_0: tensor(28.7462, device='cuda:0') (tensor(17.1780, device='cuda:0')) | > loss_gen: 1.677433729171753 (1.8649578359391954) | > loss_kl: 2.0155134201049805 (1.9682290289137099) | > loss_feat: 2.555842399597168 (2.8990892834133573) | > loss_mel: 16.70479965209961 (17.09986220465766) | > loss_duration: 1.548729419708252 (1.5643195576137967) | > amp_scaler: 64.0 (64.0) | > loss_1: 24.502317428588867 (25.39645767211914) | > grad_norm_1: tensor(811.6628, device='cuda:0') (tensor(1274.3596, device='cuda:0')) | > current_lr_0: 0.00018977024147463537 | > current_lr_1: 0.00018977024147463537 | > step_time: 0.4265 (0.42977650960286456) | > loader_time: 0.0056 (0.006374067730373806)  --> TIME: 2025-02-25 12:31:47 -- STEP: 59/72 -- GLOBAL_STEP: 1030300 | > loss_disc: 2.7834413051605225 (2.7343694880857305) | > loss_disc_real_0: 0.15659290552139282 (0.1840797059111676) | > loss_disc_real_1: 0.20680753886699677 (0.23739207700147466) | > loss_disc_real_2: 0.24377572536468506 (0.2353974984864057) | > loss_disc_real_3: 0.21970826387405396 (0.2381216166383129) | > loss_disc_real_4: 0.25377771258354187 (0.2413074932361053) | > loss_disc_real_5: 0.26418793201446533 (0.24590633759054087) | > loss_0: 2.7834413051605225 (2.7343694880857305) | > grad_norm_0: tensor(18.7030, device='cuda:0') (tensor(20.3902, device='cuda:0')) | > loss_gen: 1.9132684469223022 (1.8992656772419558) | > loss_kl: 2.3652842044830322 (2.191042493965667) | > loss_feat: 2.598235845565796 (2.9307677543769453) | > loss_mel: 17.153850555419922 (17.89299007997675) | > loss_duration: 1.6240991353988647 (1.591033721374253) | > amp_scaler: 64.0 (64.0) | > loss_1: 25.654739379882812 (26.50509970066911) | > grad_norm_1: tensor(1374.5356, device='cuda:0') (tensor(1077.8524, device='cuda:0')) | > current_lr_0: 0.00018977024147463537 | > current_lr_1: 0.00018977024147463537 | > step_time: 0.4785 (0.4461572614766784) | > loader_time: 0.0098 (0.00798029414678024) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547843342096033 (-0.0002022830533309726) | > avg_step_time: 0.4555732767346879 (-0.005399008871803834) | > avg_loss_disc: 2.7441184487141355 (+0.004893638718295179) | > avg_loss_disc_real_0: 0.1911122858734198 (-0.0018424650942775456) | > avg_loss_disc_real_1: 0.2353626699514792 (+0.000856123130086428) | > avg_loss_disc_real_2: 0.23718313998739485 (+0.00045957489752435077) | > avg_loss_disc_real_3: 0.23813601656698843 (+0.0005474483043375056) | > avg_loss_disc_real_4: 0.24230866092191616 (-0.0016486025612119093) | > avg_loss_disc_real_5: 0.24605031689287912 (-0.000872204841022739) | > avg_loss_0: 2.7441184487141355 (+0.004893638718295179) | > avg_grad_norm_0: tensor(20.4977, device='cuda:0') (+tensor(2.4545, device='cuda:0')) | > avg_loss_gen: 1.8877134776451219 (-0.01722538471221924) | > avg_loss_kl: 2.2080453466361685 (+0.10102916603357137) | > avg_loss_feat: 2.8827038818681747 (-0.06078898403006505) | > avg_loss_mel: 17.870996851316647 (+0.07771633040738024) | > avg_loss_duration: 1.6000024365707182 (+0.000345814395958266) | > avg_amp_scaler: 64.0 (+0.0) | > avg_loss_1: 26.449462004110845 (+0.10107701261278379) | > avg_grad_norm_1: tensor(1037.0245, device='cuda:0') (tensor(-14.8234, device='cuda:0'))  > EPOCH: 421/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:31:55)   --> TIME: 2025-02-25 12:32:14 -- STEP: 37/72 -- GLOBAL_STEP: 1030350 | > loss_disc: 2.8387022018432617 (2.738438251856211) | > loss_disc_real_0: 0.3746691346168518 (0.20016390951098623) | > loss_disc_real_1: 0.2377353459596634 (0.22735284429949684) | > loss_disc_real_2: 0.26306018233299255 (0.23754588816616987) | > loss_disc_real_3: 0.2721102833747864 (0.23573563872156916) | > loss_disc_real_4: 0.22952373325824738 (0.24240947615455938) | > loss_disc_real_5: 0.2299441248178482 (0.24564830193648468) | > loss_0: 2.8387022018432617 (2.738438251856211) | > grad_norm_0: tensor(42.7936, device='cuda:0') (tensor(19.6589, device='cuda:0')) | > loss_gen: 2.1971302032470703 (1.9399348626265656) | > loss_kl: 2.241025447845459 (2.398680796494355) | > loss_feat: 2.5734059810638428 (3.075473927162789) | > loss_mel: 17.10248565673828 (18.428376120489997) | > loss_duration: 1.6229288578033447 (1.5919005967475273) | > amp_scaler: 32.0 (48.432432432432435) | > loss_1: 25.736974716186523 (27.434366277746253) | > grad_norm_1: tensor(845.6747, device='cuda:0') (tensor(964.0639, device='cuda:0')) | > current_lr_0: 0.00018974652019445104 | > current_lr_1: 0.00018974652019445104 | > step_time: 0.4467 (0.4395477127384495) | > loader_time: 0.0083 (0.007381026809279983) --> EVAL PERFORMANCE | > avg_loader_time: 0.008895138619651254 (+0.00034729527755522106) | > avg_step_time: 0.457696988549031 (+0.002123711814343121) | > avg_loss_disc: 2.7494861945300033 (+0.005367745815867853) | > avg_loss_disc_real_0: 0.19659946460119435 (+0.005487178727774544) | > avg_loss_disc_real_1: 0.22937427440159758 (-0.00598839554988162) | > avg_loss_disc_real_2: 0.24249013460857768 (+0.005306994621182837) | > avg_loss_disc_real_3: 0.23790254517340323 (-0.00023347139358520508) | > avg_loss_disc_real_4: 0.24396661763459865 (+0.0016579567126824846) | > avg_loss_disc_real_5: 0.2475151820501811 (+0.0014648651573019866) | > avg_loss_0: 2.7494861945300033 (+0.005367745815867853) | > avg_grad_norm_0: tensor(21.2812, device='cuda:0') (+tensor(0.7835, device='cuda:0')) | > avg_loss_gen: 1.9220823905837368 (+0.03436891293861488) | > avg_loss_kl: 2.3299621864103943 (+0.12191683977422585) | > avg_loss_feat: 3.0144871019981276 (+0.13178322012995292) | > avg_loss_mel: 18.149768372656588 (+0.2787715213399409) | > avg_loss_duration: 1.6168601697599385 (+0.016857733189220347) | > avg_amp_scaler: 40.56338028169014 (-23.436619718309863) | > avg_loss_1: 27.033160276815924 (+0.5836982727050781) | > avg_grad_norm_1: tensor(1130.5582, device='cuda:0') (+tensor(93.5337, device='cuda:0'))  > EPOCH: 422/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:32:32)   --> TIME: 2025-02-25 12:32:41 -- STEP: 15/72 -- GLOBAL_STEP: 1030400 | > loss_disc: 2.706298828125 (2.722674576441447) | > loss_disc_real_0: 0.16247057914733887 (0.17287077605724335) | > loss_disc_real_1: 0.19502004981040955 (0.23100364605585735) | > loss_disc_real_2: 0.2539990544319153 (0.23820584615071613) | > loss_disc_real_3: 0.2509967088699341 (0.24400104681650797) | > loss_disc_real_4: 0.24995340406894684 (0.244632292787234) | > loss_disc_real_5: 0.24597711861133575 (0.2448122541109721) | > loss_0: 2.706298828125 (2.722674576441447) | > grad_norm_0: tensor(7.0785, device='cuda:0') (tensor(20.2454, device='cuda:0')) | > loss_gen: 1.9701933860778809 (1.9259507735570272) | > loss_kl: 2.1140520572662354 (2.2407628933588666) | > loss_feat: 2.6162314414978027 (3.05717077255249) | > loss_mel: 17.867168426513672 (18.92617390950521) | > loss_duration: 1.5392646789550781 (1.5812820434570312) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.106910705566406 (27.73134053548177) | > grad_norm_1: tensor(260.8233, device='cuda:0') (tensor(1318.0436, device='cuda:0')) | > current_lr_0: 0.00018972280187942673 | > current_lr_1: 0.00018972280187942673 | > step_time: 0.429 (0.4329549312591553) | > loader_time: 0.0088 (0.006839656829833984)  --> TIME: 2025-02-25 12:33:05 -- STEP: 65/72 -- GLOBAL_STEP: 1030450 | > loss_disc: 2.6654610633850098 (2.736569885107187) | > loss_disc_real_0: 0.20130759477615356 (0.18770667944963162) | > loss_disc_real_1: 0.2148798406124115 (0.23738165451930118) | > loss_disc_real_2: 0.257515549659729 (0.23760659832220812) | > loss_disc_real_3: 0.22311878204345703 (0.23989959634267366) | > loss_disc_real_4: 0.2242380827665329 (0.24410395691028008) | > loss_disc_real_5: 0.222848579287529 (0.24504809425427362) | > loss_0: 2.6654610633850098 (2.736569885107187) | > grad_norm_0: tensor(11.4782, device='cuda:0') (tensor(19.9989, device='cuda:0')) | > loss_gen: 1.8301609754562378 (1.9139859969799335) | > loss_kl: 2.303682804107666 (2.2146579430653497) | > loss_feat: 3.18544602394104 (2.9896782911740827) | > loss_mel: 18.373586654663086 (18.17783910311185) | > loss_duration: 1.659635066986084 (1.6080477146001961) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.35251235961914 (26.90420925433819) | > grad_norm_1: tensor(650.8705, device='cuda:0') (tensor(1128.8420, device='cuda:0')) | > current_lr_0: 0.00018972280187942673 | > current_lr_1: 0.00018972280187942673 | > step_time: 0.5105 (0.45854149965139535) | > loader_time: 0.0158 (0.008427572250366212) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696774361838758 (-0.00019836425781249653) | > avg_step_time: 0.4622030191018548 (+0.004506030552823781) | > avg_loss_disc: 2.7359061979911696 (-0.013579996538833683) | > avg_loss_disc_real_0: 0.18626556167720068 (-0.010333902923993665) | > avg_loss_disc_real_1: 0.23682129949751035 (+0.007447025095912774) | > avg_loss_disc_real_2: 0.23676019088483194 (-0.005729943723745745) | > avg_loss_disc_real_3: 0.23961024972754466 (+0.0017077045541414315) | > avg_loss_disc_real_4: 0.24506177218027517 (+0.0010951545456765188) | > avg_loss_disc_real_5: 0.24509478035107465 (-0.002420401699106456) | > avg_loss_0: 2.7359061979911696 (-0.013579996538833683) | > avg_grad_norm_0: tensor(19.6516, device='cuda:0') (tensor(-1.6297, device='cuda:0')) | > avg_loss_gen: 1.9109894104406868 (-0.011092980143049935) | > avg_loss_kl: 2.226079166775018 (-0.10388301963537616) | > avg_loss_feat: 2.9776114913779255 (-0.03687561062020217) | > avg_loss_mel: 18.16533091370488 (+0.015562541048293355) | > avg_loss_duration: 1.6126133270666634 (-0.004246842693275132) | > avg_amp_scaler: 32.0 (-8.563380281690137) | > avg_loss_1: 26.892624519240687 (-0.14053575757523618) | > avg_grad_norm_1: tensor(1113.0374, device='cuda:0') (tensor(-17.5209, device='cuda:0'))  > EPOCH: 423/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:33:10)   --> TIME: 2025-02-25 12:33:31 -- STEP: 43/72 -- GLOBAL_STEP: 1030500 | > loss_disc: 2.902944326400757 (2.7532614220020384) | > loss_disc_real_0: 0.21940621733665466 (0.2099660016423048) | > loss_disc_real_1: 0.22629646956920624 (0.22840635651765867) | > loss_disc_real_2: 0.33008238673210144 (0.24054370751214582) | > loss_disc_real_3: 0.2336587905883789 (0.24530663913072542) | > loss_disc_real_4: 0.25182533264160156 (0.24113628822703695) | > loss_disc_real_5: 0.25505709648132324 (0.2462369536244592) | > loss_0: 2.902944326400757 (2.7532614220020384) | > grad_norm_0: tensor(27.0467, device='cuda:0') (tensor(26.0064, device='cuda:0')) | > loss_gen: 1.7162754535675049 (1.9424155867376993) | > loss_kl: 1.9821162223815918 (2.100233277609181) | > loss_feat: 3.191678047180176 (3.0406853376432905) | > loss_mel: 17.722028732299805 (17.98600081510322) | > loss_duration: 1.6045829057693481 (1.5759376093398694) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.21668243408203 (26.64527254326399) | > grad_norm_1: tensor(1752.8250, device='cuda:0') (tensor(1187.5948, device='cuda:0')) | > current_lr_0: 0.00018969908652919179 | > current_lr_1: 0.00018969908652919179 | > step_time: 0.4522 (0.44714601649794466) | > loader_time: 0.0097 (0.007455287977706554) --> EVAL PERFORMANCE | > avg_loader_time: 0.008726388635769697 (+2.961427393093942e-05) | > avg_step_time: 0.46053059000364494 (-0.00167242909820986) | > avg_loss_disc: 2.763513427385142 (+0.02760722939397242) | > avg_loss_disc_real_0: 0.20293624363314938 (+0.016670681955948702) | > avg_loss_disc_real_1: 0.22980290777246717 (-0.007018391725043177) | > avg_loss_disc_real_2: 0.24269630917361085 (+0.005936118288778908) | > avg_loss_disc_real_3: 0.24200204581442014 (+0.0023917960868754806) | > avg_loss_disc_real_4: 0.24236637353897095 (-0.0026953986413042197) | > avg_loss_disc_real_5: 0.24664312160350907 (+0.0015483412524344153) | > avg_loss_0: 2.763513427385142 (+0.02760722939397242) | > avg_grad_norm_0: tensor(23.0881, device='cuda:0') (+tensor(3.4366, device='cuda:0')) | > avg_loss_gen: 1.9161327439294735 (+0.0051433334887867055) | > avg_loss_kl: 2.200265595610712 (-0.025813571164305937) | > avg_loss_feat: 2.9373728355891267 (-0.040238655788798816) | > avg_loss_mel: 18.34668374397386 (+0.18135283026897753) | > avg_loss_duration: 1.6016409565025653 (-0.010972370564098055) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.002095800050547 (+0.1094712808098599) | > avg_grad_norm_1: tensor(1006.8879, device='cuda:0') (tensor(-106.1494, device='cuda:0'))  > EPOCH: 424/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:33:47)   --> TIME: 2025-02-25 12:33:58 -- STEP: 21/72 -- GLOBAL_STEP: 1030550 | > loss_disc: 2.773681879043579 (2.718682890846616) | > loss_disc_real_0: 0.21244342625141144 (0.19674875622703916) | > loss_disc_real_1: 0.24964995682239532 (0.21581401711418516) | > loss_disc_real_2: 0.22625114023685455 (0.23869319260120392) | > loss_disc_real_3: 0.24789324402809143 (0.232504394082796) | > loss_disc_real_4: 0.2313462495803833 (0.24420387333347685) | > loss_disc_real_5: 0.2406640648841858 (0.24534399310747781) | > loss_0: 2.773681879043579 (2.718682890846616) | > grad_norm_0: tensor(9.8131, device='cuda:0') (tensor(20.2661, device='cuda:0')) | > loss_gen: 1.8061420917510986 (1.9192467417035783) | > loss_kl: 1.7573657035827637 (2.1561758688517982) | > loss_feat: 2.699732542037964 (2.9934951123737155) | > loss_mel: 16.72264289855957 (17.873894827706472) | > loss_duration: 1.5701611042022705 (1.5690478029705228) | > amp_scaler: 32.0 (32.0) | > loss_1: 24.556045532226562 (26.511859893798828) | > grad_norm_1: tensor(102.7019, device='cuda:0') (tensor(1103.6953, device='cuda:0')) | > current_lr_0: 0.00018967537414337564 | > current_lr_1: 0.00018967537414337564 | > step_time: 0.4316 (0.4339576902843657) | > loader_time: 0.0067 (0.006900639761061896)  --> TIME: 2025-02-25 12:34:22 -- STEP: 71/72 -- GLOBAL_STEP: 1030600 | > loss_disc: 2.9059817790985107 (2.7484258933806074) | > loss_disc_real_0: 0.12212985754013062 (0.18911376663706672) | > loss_disc_real_1: 0.22804829478263855 (0.22442140545643552) | > loss_disc_real_2: 0.22484241425991058 (0.23903205613015402) | > loss_disc_real_3: 0.2164914309978485 (0.2376782690555277) | > loss_disc_real_4: 0.25199398398399353 (0.2455224249984177) | > loss_disc_real_5: 0.25553661584854126 (0.25014734184238274) | > loss_0: 2.9059817790985107 (2.7484258933806074) | > grad_norm_0: tensor(26.1800, device='cuda:0') (tensor(23.1768, device='cuda:0')) | > loss_gen: 1.567997694015503 (1.9110253125848904) | > loss_kl: 2.0109496116638184 (2.257207279473963) | > loss_feat: 2.659186840057373 (2.9691010159505926) | > loss_mel: 17.384244918823242 (18.016341625804635) | > loss_duration: 1.626476764678955 (1.6160279008704173) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.248855590820312 (26.769702911376953) | > grad_norm_1: tensor(972.6193, device='cuda:0') (tensor(1282.8838, device='cuda:0')) | > current_lr_0: 0.00018967537414337564 | > current_lr_1: 0.00018967537414337564 | > step_time: 0.4765 (0.454177157979616) | > loader_time: 0.0082 (0.008856608834065178) --> EVAL PERFORMANCE | > avg_loader_time: 0.008856608834065178 (+0.00013022019829548107) | > avg_step_time: 0.454177157979616 (-0.006353432024028938) | > avg_loss_disc: 2.7484258933806074 (-0.015087534004534664) | > avg_loss_disc_real_0: 0.18911376663706672 (-0.013822476996082661) | > avg_loss_disc_real_1: 0.22442140545643552 (-0.0053815023160316555) | > avg_loss_disc_real_2: 0.23903205613015402 (-0.0036642530434568277) | > avg_loss_disc_real_3: 0.2376782690555277 (-0.00432377675889245) | > avg_loss_disc_real_4: 0.2455224249984177 (+0.0031560514594467515) | > avg_loss_disc_real_5: 0.25014734184238274 (+0.003504220238873673) | > avg_loss_0: 2.7484258933806074 (-0.015087534004534664) | > avg_grad_norm_0: tensor(23.1768, device='cuda:0') (+tensor(0.0886, device='cuda:0')) | > avg_loss_gen: 1.9110253125848904 (-0.005107431344583135) | > avg_loss_kl: 2.257207279473963 (+0.0569416838632506) | > avg_loss_feat: 2.9691010159505926 (+0.03172818036146596) | > avg_loss_mel: 18.016341625804635 (-0.33034211816922365) | > avg_loss_duration: 1.6160279008704173 (+0.014386944367851928) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.769702911376953 (-0.2323928886735942) | > avg_grad_norm_1: tensor(1282.8838, device='cuda:0') (+tensor(275.9958, device='cuda:0'))  > EPOCH: 425/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:34:24)   --> TIME: 2025-02-25 12:34:48 -- STEP: 49/72 -- GLOBAL_STEP: 1030650 | > loss_disc: 2.8339667320251465 (2.751458216686638) | > loss_disc_real_0: 0.2849549651145935 (0.1909814407022632) | > loss_disc_real_1: 0.2731529474258423 (0.23442637889969106) | > loss_disc_real_2: 0.30607154965400696 (0.23964638795171464) | > loss_disc_real_3: 0.2731357216835022 (0.24225680682123923) | > loss_disc_real_4: 0.27346354722976685 (0.24445719195871937) | > loss_disc_real_5: 0.2892054617404938 (0.2469703269248106) | > loss_0: 2.8339667320251465 (2.751458216686638) | > grad_norm_0: tensor(24.6388, device='cuda:0') (tensor(21.7419, device='cuda:0')) | > loss_gen: 2.051630973815918 (1.9246280606912107) | > loss_kl: 2.1817452907562256 (2.1553606476102556) | > loss_feat: 2.7261838912963867 (3.0301326294334574) | > loss_mel: 18.276491165161133 (17.920392523006523) | > loss_duration: 1.656322717666626 (1.5966301304953439) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.892375946044922 (26.62714401556521) | > grad_norm_1: tensor(199.3078, device='cuda:0') (tensor(1024.1428, device='cuda:0')) | > current_lr_0: 0.00018965166472160772 | > current_lr_1: 0.00018965166472160772 | > step_time: 0.4488 (0.44412596371709084) | > loader_time: 0.0099 (0.007740371081293846) --> EVAL PERFORMANCE | > avg_loader_time: 0.008650631971762215 (-0.00020597686230296348) | > avg_step_time: 0.4567550605451557 (+0.0025779025655396914) | > avg_loss_disc: 2.736143283441033 (-0.012282609939574307) | > avg_loss_disc_real_0: 0.18742539597229219 (-0.0016883706647745356) | > avg_loss_disc_real_1: 0.22949604962913084 (+0.0050746441726953195) | > avg_loss_disc_real_2: 0.23488005659949612 (-0.004151999530657896) | > avg_loss_disc_real_3: 0.2388427650005045 (+0.0011644959449768066) | > avg_loss_disc_real_4: 0.243392053204523 (-0.002130371793894692) | > avg_loss_disc_real_5: 0.24670819182630996 (-0.003439150016072784) | > avg_loss_0: 2.736143283441033 (-0.012282609939574307) | > avg_grad_norm_0: tensor(22.6063, device='cuda:0') (tensor(-0.5705, device='cuda:0')) | > avg_loss_gen: 1.910723649280172 (-0.0003016633047183692) | > avg_loss_kl: 2.1938212001827395 (-0.06338607929122331) | > avg_loss_feat: 2.999134728606319 (+0.030033712655726408) | > avg_loss_mel: 17.902618085834355 (-0.11372353997028029) | > avg_loss_duration: 1.6081356397816833 (-0.007892261088733976) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.61443323484609 (-0.15526967653086388) | > avg_grad_norm_1: tensor(1189.7816, device='cuda:0') (tensor(-93.1022, device='cuda:0'))  > EPOCH: 426/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:35:01)   --> TIME: 2025-02-25 12:35:15 -- STEP: 27/72 -- GLOBAL_STEP: 1030700 | > loss_disc: 2.717268705368042 (2.707845520090174) | > loss_disc_real_0: 0.22486697137355804 (0.1759362855443248) | > loss_disc_real_1: 0.19776244461536407 (0.23799907167752585) | > loss_disc_real_2: 0.25828200578689575 (0.2288468055151127) | > loss_disc_real_3: 0.25202563405036926 (0.2362912733245779) | > loss_disc_real_4: 0.25134167075157166 (0.2396073987086614) | > loss_disc_real_5: 0.2563384175300598 (0.248486563011452) | > loss_0: 2.717268705368042 (2.707845520090174) | > grad_norm_0: tensor(30.5351, device='cuda:0') (tensor(23.6439, device='cuda:0')) | > loss_gen: 1.9142823219299316 (1.940010185594912) | > loss_kl: 2.150984287261963 (2.0745679140090947) | > loss_feat: 3.007631301879883 (3.0878070990244546) | > loss_mel: 16.983579635620117 (17.74678816618743) | > loss_duration: 1.5610525608062744 (1.568812440942835) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.617528915405273 (26.417985562924986) | > grad_norm_1: tensor(2216.9133, device='cuda:0') (tensor(1583.7086, device='cuda:0')) | > current_lr_0: 0.00018962795826351752 | > current_lr_1: 0.00018962795826351752 | > step_time: 0.4367 (0.4373541408114963) | > loader_time: 0.0083 (0.0067481376506664135) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611605200968998 (-3.9026770793216745e-05) | > avg_step_time: 0.45732045509445834 (+0.0005653945493026558) | > avg_loss_disc: 2.7316176488365924 (-0.004525634604440665) | > avg_loss_disc_real_0: 0.18551835333797292 (-0.0019070426343192615) | > avg_loss_disc_real_1: 0.23032647560180072 (+0.0008304259726698848) | > avg_loss_disc_real_2: 0.23757932828345768 (+0.002699271683961557) | > avg_loss_disc_real_3: 0.23837892018573384 (-0.00046384481477065864) | > avg_loss_disc_real_4: 0.24343229620389537 (+4.0242999372358534e-05) | > avg_loss_disc_real_5: 0.24748160700562974 (+0.0007734151793197808) | > avg_loss_0: 2.7316176488365924 (-0.004525634604440665) | > avg_grad_norm_0: tensor(24.6689, device='cuda:0') (+tensor(2.0626, device='cuda:0')) | > avg_loss_gen: 1.9195141171065855 (+0.008790467826413462) | > avg_loss_kl: 2.1619368388619216 (-0.031884361320817955) | > avg_loss_feat: 3.006880038221118 (+0.007745309614799023) | > avg_loss_mel: 17.823312826559576 (-0.07930525927477916) | > avg_loss_duration: 1.6029652860802663 (-0.005170353701416941) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.51460889359595 (-0.09982434125014095) | > avg_grad_norm_1: tensor(1440.0688, device='cuda:0') (+tensor(250.2872, device='cuda:0'))  > EPOCH: 427/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:35:38)   --> TIME: 2025-02-25 12:35:43 -- STEP: 5/72 -- GLOBAL_STEP: 1030750 | > loss_disc: 2.7035975456237793 (2.7261680126190186) | > loss_disc_real_0: 0.10633808374404907 (0.21805413365364074) | > loss_disc_real_1: 0.23628848791122437 (0.22154224514961243) | > loss_disc_real_2: 0.22092898190021515 (0.2260401338338852) | > loss_disc_real_3: 0.22471307218074799 (0.2341970056295395) | > loss_disc_real_4: 0.23997966945171356 (0.2365011006593704) | > loss_disc_real_5: 0.23196108639240265 (0.23427045941352845) | > loss_0: 2.7035975456237793 (2.7261680126190186) | > grad_norm_0: tensor(29.9458, device='cuda:0') (tensor(23.1130, device='cuda:0')) | > loss_gen: 1.8351712226867676 (1.890858817100525) | > loss_kl: 2.3662736415863037 (2.1905452728271486) | > loss_feat: 2.84224271774292 (3.0033638954162596) | > loss_mel: 17.88018035888672 (18.294239807128907) | > loss_duration: 1.5378005504608154 (1.58369038105011) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.461666107177734 (26.96269760131836) | > grad_norm_1: tensor(1011.9721, device='cuda:0') (tensor(1106.0294, device='cuda:0')) | > current_lr_0: 0.00018960425476873457 | > current_lr_1: 0.00018960425476873457 | > step_time: 0.419 (0.42600479125976565) | > loader_time: 0.0062 (0.006273078918457031)  --> TIME: 2025-02-25 12:36:06 -- STEP: 55/72 -- GLOBAL_STEP: 1030800 | > loss_disc: 2.836965560913086 (2.7127873897552495) | > loss_disc_real_0: 0.08586098253726959 (0.1913403106006709) | > loss_disc_real_1: 0.2860465943813324 (0.21818819940090178) | > loss_disc_real_2: 0.24866041541099548 (0.23241536265069787) | > loss_disc_real_3: 0.2351822704076767 (0.23327070718461818) | > loss_disc_real_4: 0.20727302134037018 (0.24290839271111922) | > loss_disc_real_5: 0.23486222326755524 (0.24461534050377934) | > loss_0: 2.836965560913086 (2.7127873897552495) | > grad_norm_0: tensor(23.8982, device='cuda:0') (tensor(21.8677, device='cuda:0')) | > loss_gen: 1.7986884117126465 (1.922675440528176) | > loss_kl: 2.444612741470337 (2.1459121227264406) | > loss_feat: 2.8855085372924805 (3.0936247608878396) | > loss_mel: 16.615137100219727 (17.643466810746624) | > loss_duration: 1.6421751976013184 (1.585077014836398) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.386123657226562 (26.39075587879528) | > grad_norm_1: tensor(121.4458, device='cuda:0') (tensor(1289.7308, device='cuda:0')) | > current_lr_0: 0.00018960425476873457 | > current_lr_1: 0.00018960425476873457 | > step_time: 0.4679 (0.4435580730438232) | > loader_time: 0.0096 (0.007819916985251687) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853962629613742 (-7.197890483157804e-05) | > avg_step_time: 0.4567681500609492 (-0.0005523050335091706) | > avg_loss_disc: 2.7350028165629214 (+0.003385167726329019) | > avg_loss_disc_real_0: 0.1923778795860183 (+0.006859526248045367) | > avg_loss_disc_real_1: 0.22544835696757679 (-0.004878118634223938) | > avg_loss_disc_real_2: 0.23628277111221369 (-0.001296557171243995) | > avg_loss_disc_real_3: 0.2369515104612834 (-0.0014274097244504347) | > avg_loss_disc_real_4: 0.2441249347068894 (+0.0006926385029940307) | > avg_loss_disc_real_5: 0.24568574508311045 (-0.0017958619225192929) | > avg_loss_0: 2.7350028165629214 (+0.003385167726329019) | > avg_grad_norm_0: tensor(22.1860, device='cuda:0') (tensor(-2.4829, device='cuda:0')) | > avg_loss_gen: 1.9195843280201228 (+7.021091353731634e-05) | > avg_loss_kl: 2.243110391455637 (+0.08117355259371539) | > avg_loss_feat: 3.03813400402875 (+0.03125396580763207) | > avg_loss_mel: 17.895781960285884 (+0.07246913372630814) | > avg_loss_duration: 1.607674563434762 (+0.004709277354495667) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.704285017201602 (+0.1896761236056541) | > avg_grad_norm_1: tensor(1305.2084, device='cuda:0') (tensor(-134.8605, device='cuda:0'))  > EPOCH: 428/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:36:16)   --> TIME: 2025-02-25 12:36:32 -- STEP: 33/72 -- GLOBAL_STEP: 1030850 | > loss_disc: 2.645642042160034 (2.724820787256414) | > loss_disc_real_0: 0.1331121325492859 (0.1920049763990171) | > loss_disc_real_1: 0.24325868487358093 (0.2302033630284396) | > loss_disc_real_2: 0.21443912386894226 (0.2390029714866118) | > loss_disc_real_3: 0.247304767370224 (0.23316975344311108) | > loss_disc_real_4: 0.2453993707895279 (0.24027505652471023) | > loss_disc_real_5: 0.251191109418869 (0.24162544219782858) | > loss_0: 2.645642042160034 (2.724820787256414) | > grad_norm_0: tensor(32.0152, device='cuda:0') (tensor(21.4347, device='cuda:0')) | > loss_gen: 1.9153032302856445 (1.925604892499519) | > loss_kl: 2.278027296066284 (2.278256037018515) | > loss_feat: 3.1409316062927246 (3.058305668108391) | > loss_mel: 18.10281753540039 (18.17358924403335) | > loss_duration: 1.583947777748108 (1.572440454454133) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.021028518676758 (27.00819645505963) | > grad_norm_1: tensor(2025.4341, device='cuda:0') (tensor(1278.7330, device='cuda:0')) | > current_lr_0: 0.00018958055423688846 | > current_lr_1: 0.00018958055423688846 | > step_time: 0.4419 (0.4364209897590406) | > loader_time: 0.0076 (0.00704745812849565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575923006299519 (+3.629671016209916e-05) | > avg_step_time: 0.4558893694004542 (-0.00087878066049496) | > avg_loss_disc: 2.7321098186600374 (-0.002892997902883998) | > avg_loss_disc_real_0: 0.191977509625361 (-0.00040036996065728747) | > avg_loss_disc_real_1: 0.22671397953805789 (+0.0012656225704811008) | > avg_loss_disc_real_2: 0.23678702095025023 (+0.0005042498380365412) | > avg_loss_disc_real_3: 0.23844367244713743 (+0.0014921619858540291) | > avg_loss_disc_real_4: 0.24322248260739823 (-0.0009024520994911633) | > avg_loss_disc_real_5: 0.24408804698729178 (-0.001597698095818667) | > avg_loss_0: 2.7321098186600374 (-0.002892997902883998) | > avg_grad_norm_0: tensor(24.9085, device='cuda:0') (+tensor(2.7226, device='cuda:0')) | > avg_loss_gen: 1.9242438279407126 (+0.0046594999205897825) | > avg_loss_kl: 2.403405607586176 (+0.16029521613053888) | > avg_loss_feat: 3.0179853002789994 (-0.02014870374975075) | > avg_loss_mel: 18.479507526881257 (+0.5837255665953727) | > avg_loss_duration: 1.611768625151943 (+0.0040940617171809635) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.436910978505306 (+0.7326259613037038) | > avg_grad_norm_1: tensor(1414.0941, device='cuda:0') (+tensor(108.8857, device='cuda:0'))  > EPOCH: 429/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:36:53)   --> TIME: 2025-02-25 12:37:00 -- STEP: 11/72 -- GLOBAL_STEP: 1030900 | > loss_disc: 2.8085482120513916 (2.665595054626465) | > loss_disc_real_0: 0.24896515905857086 (0.1687261394479058) | > loss_disc_real_1: 0.28339043259620667 (0.227134261618961) | > loss_disc_real_2: 0.26527807116508484 (0.22893175347284836) | > loss_disc_real_3: 0.2942516505718231 (0.2351058545437726) | > loss_disc_real_4: 0.26453131437301636 (0.23054224930026315) | > loss_disc_real_5: 0.28780627250671387 (0.23348182168873874) | > loss_0: 2.8085482120513916 (2.665595054626465) | > grad_norm_0: tensor(15.9782, device='cuda:0') (tensor(18.2039, device='cuda:0')) | > loss_gen: 2.08137583732605 (2.011309103532271) | > loss_kl: 2.2401907444000244 (2.317946889183738) | > loss_feat: 3.051344871520996 (3.1792179237712515) | > loss_mel: 18.109039306640625 (19.38129268993031) | > loss_duration: 1.5828273296356201 (1.590918941931291) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.064775466918945 (28.48068445379084) | > grad_norm_1: tensor(75.3240, device='cuda:0') (tensor(1140.4391, device='cuda:0')) | > current_lr_0: 0.00018955685666760885 | > current_lr_1: 0.00018955685666760885 | > step_time: 0.4364 (0.48119753057306464) | > loader_time: 0.0057 (0.006314841183749112)  --> TIME: 2025-02-25 12:37:23 -- STEP: 61/72 -- GLOBAL_STEP: 1030950 | > loss_disc: 2.826974868774414 (2.724593322785174) | > loss_disc_real_0: 0.2676458954811096 (0.1831038501419005) | > loss_disc_real_1: 0.219077005982399 (0.2282211531381138) | > loss_disc_real_2: 0.2852643132209778 (0.24146179170882115) | > loss_disc_real_3: 0.26952290534973145 (0.2383325976915047) | > loss_disc_real_4: 0.27675214409828186 (0.24350595376530632) | > loss_disc_real_5: 0.2809427082538605 (0.24781031882176638) | > loss_0: 2.826974868774414 (2.724593322785174) | > grad_norm_0: tensor(13.4074, device='cuda:0') (tensor(19.5485, device='cuda:0')) | > loss_gen: 1.9431973695755005 (1.939393672786775) | > loss_kl: 2.29602313041687 (2.282325138811205) | > loss_feat: 2.49045991897583 (3.0666421475957653) | > loss_mel: 17.671981811523438 (18.33557816802478) | > loss_duration: 1.642000675201416 (1.6208294786390711) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.043663024902344 (27.244768424112287) | > grad_norm_1: tensor(1112.0824, device='cuda:0') (tensor(1169.3882, device='cuda:0')) | > current_lr_0: 0.00018955685666760885 | > current_lr_1: 0.00018955685666760885 | > step_time: 0.4584 (0.4566032886505127) | > loader_time: 0.0111 (0.0080401819260394) --> EVAL PERFORMANCE | > avg_loader_time: 0.008528460919017527 (-4.746208728199225e-05) | > avg_step_time: 0.4648842475783657 (+0.008994878177911503) | > avg_loss_disc: 2.7285398798929132 (-0.0035699387671241922) | > avg_loss_disc_real_0: 0.18299604752953627 (-0.008981462095824733) | > avg_loss_disc_real_1: 0.2291549093706507 (+0.0024409298325928086) | > avg_loss_disc_real_2: 0.23922019567288144 (+0.002433174722631215) | > avg_loss_disc_real_3: 0.23846030885904607 (+1.663641190863263e-05) | > avg_loss_disc_real_4: 0.24367097460887802 (+0.0004484920014797833) | > avg_loss_disc_real_5: 0.24911935291659665 (+0.005031305929304869) | > avg_loss_0: 2.7285398798929132 (-0.0035699387671241922) | > avg_grad_norm_0: tensor(19.2318, device='cuda:0') (tensor(-5.6768, device='cuda:0')) | > avg_loss_gen: 1.9272256585913645 (+0.00298183065065194) | > avg_loss_kl: 2.2670766770000186 (-0.13632893058615725) | > avg_loss_feat: 3.0286498271243674 (+0.010664526845368005) | > avg_loss_mel: 18.268333139553874 (-0.21117438732738236) | > avg_loss_duration: 1.625066792461234 (+0.01329816730929112) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.116351920114436 (-0.3205590583908702) | > avg_grad_norm_1: tensor(1146.1863, device='cuda:0') (tensor(-267.9078, device='cuda:0'))  > EPOCH: 430/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:37:30)   --> TIME: 2025-02-25 12:37:50 -- STEP: 39/72 -- GLOBAL_STEP: 1031000 | > loss_disc: 2.5433945655822754 (2.7330068930601463) | > loss_disc_real_0: 0.117989182472229 (0.19020161930567178) | > loss_disc_real_1: 0.1582750529050827 (0.23625095723531184) | > loss_disc_real_2: 0.20652222633361816 (0.23296901774712098) | > loss_disc_real_3: 0.20454633235931396 (0.23734909448868188) | > loss_disc_real_4: 0.1817389726638794 (0.24275565453064749) | > loss_disc_real_5: 0.22525270283222198 (0.24563224460834113) | > loss_0: 2.5433945655822754 (2.7330068930601463) | > grad_norm_0: tensor(13.4334, device='cuda:0') (tensor(18.4867, device='cuda:0')) | > loss_gen: 1.888336420059204 (1.9059541103167412) | > loss_kl: 2.1498799324035645 (2.0461381520980444) | > loss_feat: 3.4479267597198486 (3.051719531034812) | > loss_mel: 18.243648529052734 (17.61302101917756) | > loss_duration: 1.6061525344848633 (1.5657127820528471) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.33594512939453 (26.18254588200496) | > grad_norm_1: tensor(1390.3711, device='cuda:0') (tensor(1227.0757, device='cuda:0')) | > current_lr_0: 0.0001895331620605254 | > current_lr_1: 0.0001895331620605254 | > step_time: 0.4484 (0.4455738801222581) | > loader_time: 0.0083 (0.007451295852661133) --> EVAL PERFORMANCE | > avg_loader_time: 0.008711200364878486 (+0.00018273944586095874) | > avg_step_time: 0.46010489530966314 (-0.004779352268702575) | > avg_loss_disc: 2.737450230289513 (+0.008910350396599931) | > avg_loss_disc_real_0: 0.18916110904283925 (+0.006165061513302983) | > avg_loss_disc_real_1: 0.23444717115079852 (+0.005292261780147828) | > avg_loss_disc_real_2: 0.23641777353387483 (-0.0028024221390066106) | > avg_loss_disc_real_3: 0.2371722783001376 (-0.0012880305589084706) | > avg_loss_disc_real_4: 0.24330684521668394 (-0.0003641293921940736) | > avg_loss_disc_real_5: 0.24574059331920786 (-0.0033787595973887863) | > avg_loss_0: 2.737450230289513 (+0.008910350396599931) | > avg_grad_norm_0: tensor(19.5827, device='cuda:0') (+tensor(0.3509, device='cuda:0')) | > avg_loss_gen: 1.9056506140131346 (-0.021575044578229896) | > avg_loss_kl: 2.1558434493105176 (-0.11123322768950095) | > avg_loss_feat: 3.0089534067771804 (-0.019696420347186994) | > avg_loss_mel: 17.889404431195327 (-0.3789287083585471) | > avg_loss_duration: 1.6023520231246948 (-0.022714769336539264) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.562203958000936 (-0.5541479621135004) | > avg_grad_norm_1: tensor(1287.8113, device='cuda:0') (+tensor(141.6250, device='cuda:0'))  > EPOCH: 431/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:38:07)   --> TIME: 2025-02-25 12:38:17 -- STEP: 17/72 -- GLOBAL_STEP: 1031050 | > loss_disc: 2.676330327987671 (2.6892797105452595) | > loss_disc_real_0: 0.24987483024597168 (0.1933610868804595) | > loss_disc_real_1: 0.2698109447956085 (0.2219930194756564) | > loss_disc_real_2: 0.21883755922317505 (0.23256095367319443) | > loss_disc_real_3: 0.2491958886384964 (0.23120801501414356) | > loss_disc_real_4: 0.259960412979126 (0.2378450614564559) | > loss_disc_real_5: 0.24670784175395966 (0.2399986132102854) | > loss_0: 2.676330327987671 (2.6892797105452595) | > grad_norm_0: tensor(24.1537, device='cuda:0') (tensor(31.0698, device='cuda:0')) | > loss_gen: 1.9180580377578735 (1.9730714980293722) | > loss_kl: 2.1975698471069336 (2.057954143075382) | > loss_feat: 3.0607662200927734 (3.2018199808457317) | > loss_mel: 17.141071319580078 (17.882516299977027) | > loss_duration: 1.5983737707138062 (1.5624249261968277) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.91583824157715 (26.67778710757985) | > grad_norm_1: tensor(1206.6893, device='cuda:0') (tensor(1792.4171, device='cuda:0')) | > current_lr_0: 0.00018950947041526783 | > current_lr_1: 0.00018950947041526783 | > step_time: 0.4411 (0.43981190288768096) | > loader_time: 0.0097 (0.006865655674653894)  --> TIME: 2025-02-25 12:38:41 -- STEP: 67/72 -- GLOBAL_STEP: 1031100 | > loss_disc: 2.8058669567108154 (2.707033146673174) | > loss_disc_real_0: 0.27663570642471313 (0.1899644403315302) | > loss_disc_real_1: 0.21217720210552216 (0.22367395171478613) | > loss_disc_real_2: 0.24479223787784576 (0.23234282211581272) | > loss_disc_real_3: 0.21513697504997253 (0.2329875395369174) | > loss_disc_real_4: 0.22639653086662292 (0.24114815579421484) | > loss_disc_real_5: 0.2419668287038803 (0.24526026222243238) | > loss_0: 2.8058669567108154 (2.707033146673174) | > grad_norm_0: tensor(35.2332, device='cuda:0') (tensor(29.8934, device='cuda:0')) | > loss_gen: 1.9784590005874634 (1.936622518212048) | > loss_kl: 2.2495481967926025 (2.097412794383603) | > loss_feat: 3.3541343212127686 (3.050132822634569) | > loss_mel: 18.102752685546875 (17.601380718288144) | > loss_duration: 1.6830110549926758 (1.5900896271662925) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.367904663085938 (26.275638551854374) | > grad_norm_1: tensor(2004.5319, device='cuda:0') (tensor(1771.7476, device='cuda:0')) | > current_lr_0: 0.00018950947041526783 | > current_lr_1: 0.00018950947041526783 | > step_time: 0.5277 (0.45627926356756865) | > loader_time: 0.0124 (0.008500981686720208) --> EVAL PERFORMANCE | > avg_loader_time: 0.008678557167590504 (-3.264319728798169e-05) | > avg_step_time: 0.4603178232488498 (+0.00021292793918664676) | > avg_loss_disc: 2.708551353132221 (-0.028898877157292002) | > avg_loss_disc_real_0: 0.1902212765015347 (+0.0010601674586954435) | > avg_loss_disc_real_1: 0.224395032709753 (-0.010052138441045522) | > avg_loss_disc_real_2: 0.2321799831910872 (-0.004237790342787623) | > avg_loss_disc_real_3: 0.2336894691410199 (-0.0034828091591176946) | > avg_loss_disc_real_4: 0.2412473439750537 (-0.002059501241630235) | > avg_loss_disc_real_5: 0.24537020543931234 (-0.00037038787989551736) | > avg_loss_0: 2.708551353132221 (-0.028898877157292002) | > avg_grad_norm_0: tensor(29.5125, device='cuda:0') (+tensor(9.9298, device='cuda:0')) | > avg_loss_gen: 1.933875325699927 (+0.028224711686792414) | > avg_loss_kl: 2.1022622199125687 (-0.05358122939794896) | > avg_loss_feat: 3.042877674102783 (+0.03392426732560283) | > avg_loss_mel: 17.604815671141726 (-0.2845887600536017) | > avg_loss_duration: 1.5933540841223488 (-0.008997939002346023) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.277185090830628 (-0.285018867170308) | > avg_grad_norm_1: tensor(1749.4061, device='cuda:0') (+tensor(461.5948, device='cuda:0'))  > EPOCH: 432/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:38:45)   --> TIME: 2025-02-25 12:39:08 -- STEP: 45/72 -- GLOBAL_STEP: 1031150 | > loss_disc: 2.782205104827881 (2.7444105095333517) | > loss_disc_real_0: 0.13567329943180084 (0.18944642179542118) | > loss_disc_real_1: 0.22049416601657867 (0.23640329539775848) | > loss_disc_real_2: 0.18323743343353271 (0.23498995469676123) | > loss_disc_real_3: 0.22155658900737762 (0.24134264787038168) | > loss_disc_real_4: 0.2792995274066925 (0.24597087734275394) | > loss_disc_real_5: 0.2671458423137665 (0.24810938305324978) | > loss_0: 2.782205104827881 (2.7444105095333517) | > grad_norm_0: tensor(29.3755, device='cuda:0') (tensor(24.4628, device='cuda:0')) | > loss_gen: 1.8806102275848389 (1.9516722520192464) | > loss_kl: 1.9163572788238525 (2.2835880438486735) | > loss_feat: 2.870025396347046 (3.0767747667100696) | > loss_mel: 17.246185302734375 (18.37788670857747) | > loss_duration: 1.5939844846725464 (1.596961834695604) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.50716209411621 (27.286883544921874) | > grad_norm_1: tensor(1023.8788, device='cuda:0') (tensor(1592.9622, device='cuda:0')) | > current_lr_0: 0.00018948578173146592 | > current_lr_1: 0.00018948578173146592 | > step_time: 0.4701 (0.45515287717183434) | > loader_time: 0.0091 (0.0076284037695990665) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680857403177611 (+2.3002355871073327e-06) | > avg_step_time: 0.46556194063643336 (+0.005244117387583569) | > avg_loss_disc: 2.7390161500850185 (+0.030464796952797357) | > avg_loss_disc_real_0: 0.19168692837718507 (+0.00146565187565037) | > avg_loss_disc_real_1: 0.23359360871180682 (+0.009198576002053815) | > avg_loss_disc_real_2: 0.23750184749213743 (+0.005321864301050222) | > avg_loss_disc_real_3: 0.23901198636478102 (+0.00532251722376112) | > avg_loss_disc_real_4: 0.24308253824710846 (+0.001835194272054752) | > avg_loss_disc_real_5: 0.24776837565529514 (+0.0023981702159827922) | > avg_loss_0: 2.7390161500850185 (+0.030464796952797357) | > avg_grad_norm_0: tensor(25.6204, device='cuda:0') (tensor(-3.8920, device='cuda:0')) | > avg_loss_gen: 1.9377504402483012 (+0.0038751145483741922) | > avg_loss_kl: 2.2796542476600323 (+0.17739202774746365) | > avg_loss_feat: 3.031185109850386 (-0.011692564252397286) | > avg_loss_mel: 18.141566854127692 (+0.5367511829859666) | > avg_loss_duration: 1.614186748652391 (+0.02083266453004229) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.004343247749436 (+0.7271581569188079) | > avg_grad_norm_1: tensor(1567.5790, device='cuda:0') (tensor(-181.8271, device='cuda:0'))  > EPOCH: 433/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:39:22)   --> TIME: 2025-02-25 12:39:35 -- STEP: 23/72 -- GLOBAL_STEP: 1031200 | > loss_disc: 2.7200100421905518 (2.7411540487538213) | > loss_disc_real_0: 0.13398398458957672 (0.18751474923413733) | > loss_disc_real_1: 0.1848643273115158 (0.24307142975537674) | > loss_disc_real_2: 0.23979124426841736 (0.2308237092650455) | > loss_disc_real_3: 0.2580428123474121 (0.23713306693927103) | > loss_disc_real_4: 0.21742139756679535 (0.24193945332713748) | > loss_disc_real_5: 0.23241077363491058 (0.24479033247284268) | > loss_0: 2.7200100421905518 (2.7411540487538213) | > grad_norm_0: tensor(30.7208, device='cuda:0') (tensor(19.3972, device='cuda:0')) | > loss_gen: 1.7804980278015137 (1.92415930395541) | > loss_kl: 2.2050840854644775 (2.3078520298004155) | > loss_feat: 3.064518690109253 (3.016314174817956) | > loss_mel: 17.457988739013672 (17.99278856360394) | > loss_duration: 1.5841872692108154 (1.5731057291445525) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.09227752685547 (26.81421943332838) | > grad_norm_1: tensor(801.7239, device='cuda:0') (tensor(1148.5677, device='cuda:0')) | > current_lr_0: 0.00018946209600874948 | > current_lr_1: 0.00018946209600874948 | > step_time: 0.4505 (0.44287575846133026) | > loader_time: 0.0065 (0.006963937178902004) --> EVAL PERFORMANCE | > avg_loader_time: 0.008689118103242252 (+8.260700064641191e-06) | > avg_step_time: 0.4607709696595098 (-0.0047909709769235476) | > avg_loss_disc: 2.7390352000652896 (+1.9049980271024225e-05) | > avg_loss_disc_real_0: 0.18999518674444144 (-0.0016917416327436319) | > avg_loss_disc_real_1: 0.23380415309482896 (+0.00021054438302214873) | > avg_loss_disc_real_2: 0.23464991320186937 (-0.00285193429026806) | > avg_loss_disc_real_3: 0.23942017051535594 (+0.0004081841505749195) | > avg_loss_disc_real_4: 0.24270039314115552 (-0.00038214510595294127) | > avg_loss_disc_real_5: 0.24629813663556543 (-0.0014702390197297022) | > avg_loss_0: 2.7390352000652896 (+1.9049980271024225e-05) | > avg_grad_norm_0: tensor(25.3099, device='cuda:0') (tensor(-0.3106, device='cuda:0')) | > avg_loss_gen: 1.922855360407225 (-0.014895079841076297) | > avg_loss_kl: 2.2580292090563705 (-0.021625038603661828) | > avg_loss_feat: 3.0003490817378946 (-0.03083602811249131) | > avg_loss_mel: 17.98584116008919 (-0.15572569403850167) | > avg_loss_duration: 1.6032516905959224 (-0.01093505805646866) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.770326318875163 (-0.23401692887427217) | > avg_grad_norm_1: tensor(1457.6854, device='cuda:0') (tensor(-109.8936, device='cuda:0'))  > EPOCH: 434/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:40:00)   --> TIME: 2025-02-25 12:40:02 -- STEP: 1/72 -- GLOBAL_STEP: 1031250 | > loss_disc: 2.6511476039886475 (2.6511476039886475) | > loss_disc_real_0: 0.12558254599571228 (0.12558254599571228) | > loss_disc_real_1: 0.23039457201957703 (0.23039457201957703) | > loss_disc_real_2: 0.2262270450592041 (0.2262270450592041) | > loss_disc_real_3: 0.2213277518749237 (0.2213277518749237) | > loss_disc_real_4: 0.2345227301120758 (0.2345227301120758) | > loss_disc_real_5: 0.24286454916000366 (0.24286454916000366) | > loss_0: 2.6511476039886475 (2.6511476039886475) | > grad_norm_0: tensor(8.5689, device='cuda:0') (tensor(8.5689, device='cuda:0')) | > loss_gen: 2.2325847148895264 (2.2325847148895264) | > loss_kl: 2.167800188064575 (2.167800188064575) | > loss_feat: 3.2332968711853027 (3.2332968711853027) | > loss_mel: 18.182483673095703 (18.182483673095703) | > loss_duration: 1.6028265953063965 (1.6028265953063965) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.418991088867188 (27.418991088867188) | > grad_norm_1: tensor(1428.4336, device='cuda:0') (tensor(1428.4336, device='cuda:0')) | > current_lr_0: 0.00018943841324674837 | > current_lr_1: 0.00018943841324674837 | > step_time: 0.4336 (0.4335653781890869) | > loader_time: 0.0063 (0.00634312629699707)  --> TIME: 2025-02-25 12:40:25 -- STEP: 51/72 -- GLOBAL_STEP: 1031300 | > loss_disc: 2.723041296005249 (2.762279608670403) | > loss_disc_real_0: 0.18497812747955322 (0.19442185847198262) | > loss_disc_real_1: 0.2453612983226776 (0.2429524242293601) | > loss_disc_real_2: 0.25987106561660767 (0.2474335382382075) | > loss_disc_real_3: 0.24814054369926453 (0.24034230761668263) | > loss_disc_real_4: 0.24522212147712708 (0.2464633949247061) | > loss_disc_real_5: 0.286397248506546 (0.24963800229278266) | > loss_0: 2.723041296005249 (2.762279608670403) | > grad_norm_0: tensor(38.1828, device='cuda:0') (tensor(26.2799, device='cuda:0')) | > loss_gen: 1.8862957954406738 (1.9816412481607175) | > loss_kl: 2.0812487602233887 (2.3693373530518778) | > loss_feat: 2.849592924118042 (3.1567995314504587) | > loss_mel: 16.981714248657227 (19.021841273588286) | > loss_duration: 1.633885383605957 (1.6739545639823465) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.432735443115234 (28.2035737879136) | > grad_norm_1: tensor(1909.5774, device='cuda:0') (tensor(1445.7024, device='cuda:0')) | > current_lr_0: 0.00018943841324674837 | > current_lr_1: 0.00018943841324674837 | > step_time: 0.4433 (0.4454400960136862) | > loader_time: 0.0089 (0.007876905740476121) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870489738356899 (+1.5779280326738004e-05) | > avg_step_time: 0.4574807496138022 (-0.0032902200457076147) | > avg_loss_disc: 2.7556543316639646 (+0.016619131598675008) | > avg_loss_disc_real_0: 0.19694888140534011 (+0.006953694660898679) | > avg_loss_disc_real_1: 0.23781755062895762 (+0.00401339753412866) | > avg_loss_disc_real_2: 0.2453591330370433 (+0.010709219835173922) | > avg_loss_disc_real_3: 0.2392527700310022 (-0.00016740048435373578) | > avg_loss_disc_real_4: 0.24394377583349255 (+0.0012433826923370361) | > avg_loss_disc_real_5: 0.24822563562594668 (+0.001927498990381249) | > avg_loss_0: 2.7556543316639646 (+0.016619131598675008) | > avg_grad_norm_0: tensor(26.9750, device='cuda:0') (+tensor(1.6651, device='cuda:0')) | > avg_loss_gen: 1.9570626221912009 (+0.03420726178397593) | > avg_loss_kl: 2.3166506794137014 (+0.058621470357330896) | > avg_loss_feat: 3.0900918362845835 (+0.0897427545466889) | > avg_loss_mel: 18.670011708434192 (+0.6841705483450014) | > avg_loss_duration: 1.6655749099355348 (+0.06232321933961238) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.699391647123953 (+0.9290653282487895) | > avg_grad_norm_1: tensor(1412.2274, device='cuda:0') (tensor(-45.4580, device='cuda:0'))  > EPOCH: 435/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:40:37)   --> TIME: 2025-02-25 12:40:52 -- STEP: 29/72 -- GLOBAL_STEP: 1031350 | > loss_disc: 2.756795883178711 (2.6957145066096864) | > loss_disc_real_0: 0.30777066946029663 (0.1764872801201097) | > loss_disc_real_1: 0.21245908737182617 (0.2242109672776584) | > loss_disc_real_2: 0.29319852590560913 (0.2357065317959621) | > loss_disc_real_3: 0.27844002842903137 (0.24065237929081096) | > loss_disc_real_4: 0.2461475282907486 (0.24599915126274371) | > loss_disc_real_5: 0.2560158967971802 (0.24726135001100347) | > loss_0: 2.756795883178711 (2.6957145066096864) | > grad_norm_0: tensor(31.7132, device='cuda:0') (tensor(28.3254, device='cuda:0')) | > loss_gen: 2.156399965286255 (1.9764147544729298) | > loss_kl: 2.208702802658081 (2.3299741744995117) | > loss_feat: 2.9650511741638184 (3.1348486686575003) | > loss_mel: 18.006555557250977 (18.054803453642748) | > loss_duration: 1.5786290168762207 (1.5806493841368576) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.91533851623535 (27.076690410745556) | > grad_norm_1: tensor(1532.8353, device='cuda:0') (tensor(1801.0820, device='cuda:0')) | > current_lr_0: 0.00018941473344509253 | > current_lr_1: 0.00018941473344509253 | > step_time: 0.4373 (0.4352207266051194) | > loader_time: 0.0074 (0.006962225354951003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008649355928662797 (-5.554145490619354e-05) | > avg_step_time: 0.46045866146893566 (+0.002977911855133464) | > avg_loss_disc: 2.7092987416495737 (-0.046355590014390824) | > avg_loss_disc_real_0: 0.18739525470095622 (-0.009553626704383894) | > avg_loss_disc_real_1: 0.22618342180487136 (-0.011634128824086265) | > avg_loss_disc_real_2: 0.2356588989496231 (-0.009700234087420184) | > avg_loss_disc_real_3: 0.23741023242473602 (-0.0018425376062661813) | > avg_loss_disc_real_4: 0.24378242186257537 (-0.00016135397091718295) | > avg_loss_disc_real_5: 0.2460471809330121 (-0.002178454692934595) | > avg_loss_0: 2.7092987416495737 (-0.046355590014390824) | > avg_grad_norm_0: tensor(28.3350, device='cuda:0') (+tensor(1.3600, device='cuda:0')) | > avg_loss_gen: 1.9557528260728003 (-0.0013097961184005413) | > avg_loss_kl: 2.276056251055757 (-0.040594428357944246) | > avg_loss_feat: 3.0534191400232458 (-0.036672696261337734) | > avg_loss_mel: 17.852946375457332 (-0.8170653329768598) | > avg_loss_duration: 1.607684333559493 (-0.05789057637604178) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.745858904341574 (-0.9535327427823788) | > avg_grad_norm_1: tensor(1680.5771, device='cuda:0') (+tensor(268.3497, device='cuda:0'))  > EPOCH: 436/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:41:14)   --> TIME: 2025-02-25 12:41:19 -- STEP: 7/72 -- GLOBAL_STEP: 1031400 | > loss_disc: 2.6944286823272705 (2.6979379653930664) | > loss_disc_real_0: 0.1601938009262085 (0.17501973892961228) | > loss_disc_real_1: 0.24458134174346924 (0.2299780058009284) | > loss_disc_real_2: 0.2673814296722412 (0.24691322871616908) | > loss_disc_real_3: 0.21803219616413116 (0.23137534303324564) | > loss_disc_real_4: 0.22866296768188477 (0.24304988980293274) | > loss_disc_real_5: 0.2335868775844574 (0.2399322518280574) | > loss_0: 2.6944286823272705 (2.6979379653930664) | > grad_norm_0: tensor(8.4998, device='cuda:0') (tensor(15.2077, device='cuda:0')) | > loss_gen: 2.097557544708252 (1.959011435508728) | > loss_kl: 2.712244987487793 (2.3657849516187395) | > loss_feat: 3.3416035175323486 (3.2446560519082204) | > loss_mel: 20.090883255004883 (18.97358022417341) | > loss_duration: 1.6288139820098877 (1.596935442515782) | > amp_scaler: 32.0 (32.0) | > loss_1: 29.871103286743164 (28.139968054635183) | > grad_norm_1: tensor(949.8759, device='cuda:0') (tensor(827.4825, device='cuda:0')) | > current_lr_0: 0.0001893910566034119 | > current_lr_1: 0.0001893910566034119 | > step_time: 0.4433 (0.4346425533294678) | > loader_time: 0.0056 (0.006417478833879743)  --> TIME: 2025-02-25 12:41:42 -- STEP: 57/72 -- GLOBAL_STEP: 1031450 | > loss_disc: 2.7351090908050537 (2.723706191046196) | > loss_disc_real_0: 0.1844945251941681 (0.18399127415920558) | > loss_disc_real_1: 0.2827157974243164 (0.23306674052748763) | > loss_disc_real_2: 0.2466432750225067 (0.23738828562853628) | > loss_disc_real_3: 0.2577391564846039 (0.24188669731742457) | > loss_disc_real_4: 0.2605334520339966 (0.24636726238225637) | > loss_disc_real_5: 0.26211073994636536 (0.24759310248651004) | > loss_0: 2.7351090908050537 (2.723706191046196) | > grad_norm_0: tensor(24.1727, device='cuda:0') (tensor(23.4078, device='cuda:0')) | > loss_gen: 2.099182367324829 (1.9600118189527278) | > loss_kl: 2.225285530090332 (2.227844093975269) | > loss_feat: 3.1031763553619385 (3.113055103703549) | > loss_mel: 17.924480438232422 (18.0200648391456) | > loss_duration: 1.6366796493530273 (1.5937639684007878) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.98880386352539 (26.914740043773985) | > grad_norm_1: tensor(1849.1290, device='cuda:0') (tensor(1465.6041, device='cuda:0')) | > current_lr_0: 0.0001893910566034119 | > current_lr_1: 0.0001893910566034119 | > step_time: 0.4661 (0.44864342087193537) | > loader_time: 0.0095 (0.007977753354792009) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086071726302026 (-4.2183298460196986e-05) | > avg_step_time: 0.4590305879082478 (-0.0014280735606878592) | > avg_loss_disc: 2.728918797533277 (+0.01962005588370319) | > avg_loss_disc_real_0: 0.1853764812711259 (-0.0020187734298303117) | > avg_loss_disc_real_1: 0.2338802850162479 (+0.007696863211376553) | > avg_loss_disc_real_2: 0.23783204421191148 (+0.002173145262288373) | > avg_loss_disc_real_3: 0.2414847066704656 (+0.00407447424572957) | > avg_loss_disc_real_4: 0.2466414190933738 (+0.002858997230798438) | > avg_loss_disc_real_5: 0.2490275498007385 (+0.0029803688677264017) | > avg_loss_0: 2.728918797533277 (+0.01962005588370319) | > avg_grad_norm_0: tensor(23.1952, device='cuda:0') (tensor(-5.1398, device='cuda:0')) | > avg_loss_gen: 1.9428873431514686 (-0.012865482921331717) | > avg_loss_kl: 2.2659054289401426 (-0.010150822115614577) | > avg_loss_feat: 3.056156547976212 (+0.0027374079529662865) | > avg_loss_mel: 18.10001080472705 (+0.2470644292697166) | > avg_loss_duration: 1.606570431883906 (-0.0011139016755870035) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.97153069939412 (+0.22567179505254487) | > avg_grad_norm_1: tensor(1478.7393, device='cuda:0') (tensor(-201.8379, device='cuda:0'))  > EPOCH: 437/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:41:51)   --> TIME: 2025-02-25 12:42:09 -- STEP: 35/72 -- GLOBAL_STEP: 1031500 | > loss_disc: 2.7390804290771484 (2.739050728934152) | > loss_disc_real_0: 0.16425387561321259 (0.18456130496093204) | > loss_disc_real_1: 0.2214205414056778 (0.22122860678604672) | > loss_disc_real_2: 0.2759725749492645 (0.2431079374892371) | > loss_disc_real_3: 0.2546088993549347 (0.2382357209920883) | > loss_disc_real_4: 0.27143144607543945 (0.24452592091900963) | > loss_disc_real_5: 0.26618075370788574 (0.24569159703595297) | > loss_0: 2.7390804290771484 (2.739050728934152) | > grad_norm_0: tensor(34.5664, device='cuda:0') (tensor(26.6988, device='cuda:0')) | > loss_gen: 1.8806591033935547 (1.9846560614449638) | > loss_kl: 2.1670591831207275 (2.175680698667253) | > loss_feat: 3.0469186305999756 (3.209262657165527) | > loss_mel: 18.28917121887207 (18.751708466666088) | > loss_duration: 1.6022166013717651 (1.5773509195872715) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.986024856567383 (27.69865869794573) | > grad_norm_1: tensor(2131.5066, device='cuda:0') (tensor(1728.6812, device='cuda:0')) | > current_lr_0: 0.00018936738272133647 | > current_lr_1: 0.00018936738272133647 | > step_time: 0.4684 (0.43631758689880373) | > loader_time: 0.0084 (0.0070638111659458705) --> EVAL PERFORMANCE | > avg_loader_time: 0.008524172742601848 (-8.299988760075185e-05) | > avg_step_time: 0.4534573622152839 (-0.0055732256929639234) | > avg_loss_disc: 2.728873064820196 (-4.573271308094817e-05) | > avg_loss_disc_real_0: 0.1864950159485911 (+0.0011185346774651994) | > avg_loss_disc_real_1: 0.22490650793196448 (-0.00897377708428343) | > avg_loss_disc_real_2: 0.2369498266300685 (-0.0008822175818429812) | > avg_loss_disc_real_3: 0.23579578521386 (-0.0056889214566056) | > avg_loss_disc_real_4: 0.24278302553673864 (-0.00385839355663517) | > avg_loss_disc_real_5: 0.24379006531876576 (-0.005237484481972726) | > avg_loss_0: 2.728873064820196 (-4.573271308094817e-05) | > avg_grad_norm_0: tensor(23.9372, device='cuda:0') (+tensor(0.7420, device='cuda:0')) | > avg_loss_gen: 1.9429669967839416 (+7.965363247297574e-05) | > avg_loss_kl: 2.2377341767431975 (-0.02817125219694505) | > avg_loss_feat: 3.087233459445792 (+0.031076911469579915) | > avg_loss_mel: 18.524200560341427 (+0.4241897556143783) | > avg_loss_duration: 1.6077792678080813 (+0.001208835924175311) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.39991443257936 (+0.4283837331852425) | > avg_grad_norm_1: tensor(1523.5695, device='cuda:0') (+tensor(44.8302, device='cuda:0'))  > EPOCH: 438/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:42:28)   --> TIME: 2025-02-25 12:42:36 -- STEP: 13/72 -- GLOBAL_STEP: 1031550 | > loss_disc: 2.5989816188812256 (2.668147380535419) | > loss_disc_real_0: 0.2092791199684143 (0.16034462704108313) | > loss_disc_real_1: 0.2137252688407898 (0.22120303374070388) | > loss_disc_real_2: 0.22458966076374054 (0.22887745270362267) | > loss_disc_real_3: 0.23287031054496765 (0.23193009426960579) | > loss_disc_real_4: 0.22895285487174988 (0.23933911896668947) | > loss_disc_real_5: 0.24973323941230774 (0.24130862951278687) | > loss_0: 2.5989816188812256 (2.668147380535419) | > grad_norm_0: tensor(28.0495, device='cuda:0') (tensor(20.9038, device='cuda:0')) | > loss_gen: 2.0169506072998047 (1.9379919675680308) | > loss_kl: 1.9812424182891846 (2.0384457203058095) | > loss_feat: 3.1918461322784424 (3.202110235507672) | > loss_mel: 16.693384170532227 (17.803611902090218) | > loss_duration: 1.4893803596496582 (1.5604153688137348) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.372802734375 (26.542575396024265) | > grad_norm_1: tensor(2122.8469, device='cuda:0') (tensor(1458.1626, device='cuda:0')) | > current_lr_0: 0.0001893437117984963 | > current_lr_1: 0.0001893437117984963 | > step_time: 0.4464 (0.43533649811377895) | > loader_time: 0.0116 (0.006740698447594276)  --> TIME: 2025-02-25 12:43:00 -- STEP: 63/72 -- GLOBAL_STEP: 1031600 | > loss_disc: 2.7761893272399902 (2.7017672478206576) | > loss_disc_real_0: 0.12263455241918564 (0.17794755148509192) | > loss_disc_real_1: 0.24674947559833527 (0.2255198016526207) | > loss_disc_real_2: 0.2209712415933609 (0.23293706822016882) | > loss_disc_real_3: 0.23751679062843323 (0.2368371912411281) | > loss_disc_real_4: 0.24707907438278198 (0.2411542371624992) | > loss_disc_real_5: 0.26307621598243713 (0.24761803400894952) | > loss_0: 2.7761893272399902 (2.7017672478206576) | > grad_norm_0: tensor(26.6783, device='cuda:0') (tensor(23.1533, device='cuda:0')) | > loss_gen: 1.7104144096374512 (1.9105651832762218) | > loss_kl: 2.0847842693328857 (2.0739932590060746) | > loss_feat: 2.7692794799804688 (3.0991881953345404) | > loss_mel: 17.45932388305664 (17.454735120137535) | > loss_duration: 1.592437505722046 (1.5866103266912794) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.616241455078125 (26.125092067415753) | > grad_norm_1: tensor(874.1679, device='cuda:0') (tensor(1446.7233, device='cuda:0')) | > current_lr_0: 0.0001893437117984963 | > current_lr_1: 0.0001893437117984963 | > step_time: 0.4869 (0.45895687360612175) | > loader_time: 0.0115 (0.008390937532697401) --> EVAL PERFORMANCE | > avg_loader_time: 0.008756043205798508 (+0.0002318704631966602) | > avg_step_time: 0.46519560545263156 (+0.011738243237347679) | > avg_loss_disc: 2.705832800395052 (-0.023040264425143953) | > avg_loss_disc_real_0: 0.18121196623419372 (-0.0052830497143973865) | > avg_loss_disc_real_1: 0.22504407524223058 (+0.0001375673102660957) | > avg_loss_disc_real_2: 0.23443044596154924 (-0.0025193806685192555) | > avg_loss_disc_real_3: 0.2363401519160875 (+0.0005443667022275167) | > avg_loss_disc_real_4: 0.2414460578854655 (-0.001336967651273141) | > avg_loss_disc_real_5: 0.24688994653627905 (+0.003099881217513284) | > avg_loss_0: 2.705832800395052 (-0.023040264425143953) | > avg_grad_norm_0: tensor(22.7371, device='cuda:0') (tensor(-1.2002, device='cuda:0')) | > avg_loss_gen: 1.912136198769153 (-0.030830798014788607) | > avg_loss_kl: 2.089549410511069 (-0.1481847662321285) | > avg_loss_feat: 3.084249076708941 (-0.0029843827368507547) | > avg_loss_mel: 17.469864710955555 (-1.0543358493858719) | > avg_loss_duration: 1.5940486595664225 (-0.013730608241658837) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.149848024610062 (-1.2500664079692996) | > avg_grad_norm_1: tensor(1444.9218, device='cuda:0') (tensor(-78.6477, device='cuda:0'))  > EPOCH: 439/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:43:06)   --> TIME: 2025-02-25 12:43:26 -- STEP: 41/72 -- GLOBAL_STEP: 1031650 | > loss_disc: 2.784010648727417 (2.6972725100633577) | > loss_disc_real_0: 0.31515684723854065 (0.18477561161285494) | > loss_disc_real_1: 0.27185434103012085 (0.22711171682288006) | > loss_disc_real_2: 0.20092669129371643 (0.23383623506964707) | > loss_disc_real_3: 0.1944570392370224 (0.23622376090142785) | > loss_disc_real_4: 0.2551835775375366 (0.24125927432281216) | > loss_disc_real_5: 0.2568269968032837 (0.24531194785746133) | > loss_0: 2.784010648727417 (2.6972725100633577) | > grad_norm_0: tensor(25.7191, device='cuda:0') (tensor(29.3531, device='cuda:0')) | > loss_gen: 2.056715965270996 (2.006406185103626) | > loss_kl: 2.0672075748443604 (2.39435992589811) | > loss_feat: 3.0223023891448975 (3.206100812772425) | > loss_mel: 19.605213165283203 (18.62239886493218) | > loss_duration: 1.6454523801803589 (1.6655561894905277) | > amp_scaler: 32.0 (32.0) | > loss_1: 28.39689064025879 (27.89482195784406) | > grad_norm_1: tensor(712.8428, device='cuda:0') (tensor(1917.2446, device='cuda:0')) | > current_lr_0: 0.00018932004383452147 | > current_lr_1: 0.00018932004383452147 | > step_time: 0.453 (0.43960032230470236) | > loader_time: 0.0093 (0.007320805293757741) --> EVAL PERFORMANCE | > avg_loader_time: 0.008510733993960098 (-0.0002453092118384105) | > avg_step_time: 0.45424775002707896 (-0.0109478554255526) | > avg_loss_disc: 2.7257529178135833 (+0.01992011741853128) | > avg_loss_disc_real_0: 0.18861444388896645 (+0.007402477654772727) | > avg_loss_disc_real_1: 0.23115301027264393 (+0.006108935030413348) | > avg_loss_disc_real_2: 0.23473385144287431 (+0.0003034054813250697) | > avg_loss_disc_real_3: 0.23653645540626955 (+0.00019630349018204263) | > avg_loss_disc_real_4: 0.24286238219536527 (+0.001416324309899769) | > avg_loss_disc_real_5: 0.24599853283922438 (-0.0008914136970546715) | > avg_loss_0: 2.7257529178135833 (+0.01992011741853128) | > avg_grad_norm_0: tensor(27.9159, device='cuda:0') (+tensor(5.1789, device='cuda:0')) | > avg_loss_gen: 1.9586645045750577 (+0.046528305805904724) | > avg_loss_kl: 2.4057752750289274 (+0.3162258645178584) | > avg_loss_feat: 3.0845884437292392 (+0.00033936702029802035) | > avg_loss_mel: 18.66140435447156 (+1.1915396435160055) | > avg_loss_duration: 1.659549980096414 (+0.06550132052999147) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.769982458839955 (+1.6201344342298931) | > avg_grad_norm_1: tensor(1739.9171, device='cuda:0') (+tensor(294.9954, device='cuda:0'))  > EPOCH: 440/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:43:43)   --> TIME: 2025-02-25 12:43:53 -- STEP: 19/72 -- GLOBAL_STEP: 1031700 | > loss_disc: 2.7909903526306152 (2.7010082445646586) | > loss_disc_real_0: 0.15570753812789917 (0.18330822649754977) | > loss_disc_real_1: 0.22579196095466614 (0.2262685189121648) | > loss_disc_real_2: 0.20284077525138855 (0.23485026861491956) | > loss_disc_real_3: 0.2325325459241867 (0.2357688640293322) | > loss_disc_real_4: 0.22887177765369415 (0.23383201188162753) | > loss_disc_real_5: 0.21997666358947754 (0.23730688581341192) | > loss_0: 2.7909903526306152 (2.7010082445646586) | > grad_norm_0: tensor(15.7607, device='cuda:0') (tensor(31.0177, device='cuda:0')) | > loss_gen: 1.868965744972229 (2.02360863434641) | > loss_kl: 2.0935311317443848 (2.398189005098845) | > loss_feat: 3.021019458770752 (3.2636972979495398) | > loss_mel: 17.092199325561523 (20.176004710950348) | > loss_duration: 1.5363802909851074 (1.5731293966895656) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.61209487915039 (29.434629038760537) | > grad_norm_1: tensor(1597.4941, device='cuda:0') (tensor(1993.9366, device='cuda:0')) | > current_lr_0: 0.00018929637882904215 | > current_lr_1: 0.00018929637882904215 | > step_time: 0.4269 (0.43424886151363973) | > loader_time: 0.0062 (0.006609339463083367)  --> TIME: 2025-02-25 12:44:17 -- STEP: 69/72 -- GLOBAL_STEP: 1031750 | > loss_disc: 2.83711576461792 (2.756559302841408) | > loss_disc_real_0: 0.3323759436607361 (0.19273047678280567) | > loss_disc_real_1: 0.22521847486495972 (0.23632455368836722) | > loss_disc_real_2: 0.23442010581493378 (0.2420771782812865) | > loss_disc_real_3: 0.24207842350006104 (0.24645027950190115) | > loss_disc_real_4: 0.27611058950424194 (0.24259375662043475) | > loss_disc_real_5: 0.28288331627845764 (0.24323751680229022) | > loss_0: 2.83711576461792 (2.756559302841408) | > grad_norm_0: tensor(25.0015, device='cuda:0') (tensor(24.9481, device='cuda:0')) | > loss_gen: 1.9371169805526733 (1.9586175507393435) | > loss_kl: 2.4670422077178955 (2.3347213907518265) | > loss_feat: 2.7160544395446777 (3.102254701697308) | > loss_mel: 20.070987701416016 (19.07838909867881) | > loss_duration: 1.7411513328552246 (1.6200001222499902) | > amp_scaler: 32.0 (32.0) | > loss_1: 28.932353973388672 (28.093982779461406) | > grad_norm_1: tensor(272.8938, device='cuda:0') (tensor(1173.5359, device='cuda:0')) | > current_lr_0: 0.00018929637882904215 | > current_lr_1: 0.00018929637882904215 | > step_time: 0.5605 (0.4527812867924787) | > loader_time: 0.0126 (0.008475935977438221) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853400834849183 (+2.3274354531732308e-05) | > avg_step_time: 0.45408662943772865 (-0.00016112058935030538) | > avg_loss_disc: 2.7566724629469324 (+0.030919545133349047) | > avg_loss_disc_real_0: 0.19293145740955647 (+0.004317013520590024) | > avg_loss_disc_real_1: 0.23711546898727687 (+0.00596245871463294) | > avg_loss_disc_real_2: 0.24129170011466658 (+0.006557848671792266) | > avg_loss_disc_real_3: 0.24593970998072287 (+0.009403254574453318) | > avg_loss_disc_real_4: 0.2430396663471007 (+0.00017728415173542955) | > avg_loss_disc_real_5: 0.2437361799915072 (-0.0022623528477171695) | > avg_loss_0: 2.7566724629469324 (+0.030919545133349047) | > avg_grad_norm_0: tensor(25.0378, device='cuda:0') (tensor(-2.8781, device='cuda:0')) | > avg_loss_gen: 1.952108685399445 (-0.006555819175612765) | > avg_loss_kl: 2.33644442659029 (-0.06933084843863746) | > avg_loss_feat: 3.0875903418366337 (+0.003001898107394485) | > avg_loss_mel: 19.04830427572761 (+0.3868999212560489) | > avg_loss_duration: 1.6208873987197876 (-0.03866258137662637) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 28.045335044323558 (+0.27535258548360275) | > avg_grad_norm_1: tensor(1152.3737, device='cuda:0') (tensor(-587.5435, device='cuda:0'))  > EPOCH: 441/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:44:20)   --> TIME: 2025-02-25 12:44:43 -- STEP: 47/72 -- GLOBAL_STEP: 1031800 | > loss_disc: 2.61165714263916 (2.704301823960973) | > loss_disc_real_0: 0.19725725054740906 (0.18711469369999906) | > loss_disc_real_1: 0.2831175625324249 (0.2225890911005913) | > loss_disc_real_2: 0.26194095611572266 (0.23573911602192738) | > loss_disc_real_3: 0.2399817258119583 (0.23519526517137568) | > loss_disc_real_4: 0.2502046525478363 (0.24473512679972548) | > loss_disc_real_5: 0.2614107131958008 (0.2459510904043279) | > loss_0: 2.61165714263916 (2.704301823960973) | > grad_norm_0: tensor(15.8958, device='cuda:0') (tensor(21.9747, device='cuda:0')) | > loss_gen: 2.0286476612091064 (1.9450625891381121) | > loss_kl: 2.0267553329467773 (2.2131643726470633) | > loss_feat: 3.2014405727386475 (3.0942018590074905) | > loss_mel: 18.20682716369629 (18.06887618531573) | > loss_duration: 1.6252758502960205 (1.5904550273367701) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.08894920349121 (26.911760208454538) | > grad_norm_1: tensor(2383.8196, device='cuda:0') (tensor(1403.5593, device='cuda:0')) | > current_lr_0: 0.00018927271678168851 | > current_lr_1: 0.00018927271678168851 | > step_time: 0.4603 (0.4436539284726407) | > loader_time: 0.0092 (0.0077329950129732174) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870247961769641 (+0.0001684712692045804) | > avg_step_time: 0.4635604670349981 (+0.009473837597269441) | > avg_loss_disc: 2.711179236291159 (-0.04549322665577327) | > avg_loss_disc_real_0: 0.18276147208583188 (-0.010169985323724595) | > avg_loss_disc_real_1: 0.22395835669947345 (-0.013157112287803419) | > avg_loss_disc_real_2: 0.23989389123211444 (-0.0013978088825521429) | > avg_loss_disc_real_3: 0.23724216516588775 (-0.008697544814835118) | > avg_loss_disc_real_4: 0.24338322793933706 (+0.0003435615922363633) | > avg_loss_disc_real_5: 0.24815419819993031 (+0.004418018208423108) | > avg_loss_0: 2.711179236291159 (-0.04549322665577327) | > avg_grad_norm_0: tensor(24.4203, device='cuda:0') (tensor(-0.6175, device='cuda:0')) | > avg_loss_gen: 1.9515638015639614 (-0.0005448838354835672) | > avg_loss_kl: 2.2899838249448328 (-0.04646060164545718) | > avg_loss_feat: 3.0925727495005435 (+0.0049824076639097825) | > avg_loss_mel: 18.285926227838225 (-0.7623780478893849) | > avg_loss_duration: 1.6151283341394347 (-0.005759064580352913) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.235175092455368 (-0.8101599518681901) | > avg_grad_norm_1: tensor(1725.3098, device='cuda:0') (+tensor(572.9362, device='cuda:0'))  > EPOCH: 442/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:44:57)   --> TIME: 2025-02-25 12:45:10 -- STEP: 25/72 -- GLOBAL_STEP: 1031850 | > loss_disc: 2.5982043743133545 (2.712224397659302) | > loss_disc_real_0: 0.2580830454826355 (0.1948126202821732) | > loss_disc_real_1: 0.21942465007305145 (0.22964373230934143) | > loss_disc_real_2: 0.2306775599718094 (0.23504059731960297) | > loss_disc_real_3: 0.23324330151081085 (0.23357367277145386) | > loss_disc_real_4: 0.2331663966178894 (0.24520228505134584) | > loss_disc_real_5: 0.241728276014328 (0.24044387817382812) | > loss_0: 2.5982043743133545 (2.712224397659302) | > grad_norm_0: tensor(24.4587, device='cuda:0') (tensor(21.4165, device='cuda:0')) | > loss_gen: 2.1023683547973633 (1.9531834268569945) | > loss_kl: 2.148730993270874 (2.217409200668335) | > loss_feat: 3.383312225341797 (3.115689296722412) | > loss_mel: 18.7269344329834 (18.41821792602539) | > loss_duration: 1.5712206363677979 (1.57883882522583) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.932567596435547 (27.283338623046873) | > grad_norm_1: tensor(2863.2830, device='cuda:0') (tensor(1249.5796, device='cuda:0')) | > current_lr_0: 0.0001892490576920908 | > current_lr_1: 0.0001892490576920908 | > step_time: 0.4359 (0.438294095993042) | > loader_time: 0.0075 (0.0070089054107666015) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865347620467065 (-4.9003413025760015e-05) | > avg_step_time: 0.4579503569804447 (-0.005610110054553419) | > avg_loss_disc: 2.7303682750379537 (+0.019189038746794562) | > avg_loss_disc_real_0: 0.1901915955921294 (+0.007430123506297515) | > avg_loss_disc_real_1: 0.22950190872373716 (+0.005543552024263709) | > avg_loss_disc_real_2: 0.2348137489506896 (-0.005080142281424838) | > avg_loss_disc_real_3: 0.23465966296867585 (-0.0025825021972119) | > avg_loss_disc_real_4: 0.24417272378021562 (+0.0007894958408785624) | > avg_loss_disc_real_5: 0.24641075730323792 (-0.0017434408966923998) | > avg_loss_0: 2.7303682750379537 (+0.019189038746794562) | > avg_grad_norm_0: tensor(23.0112, device='cuda:0') (tensor(-1.4091, device='cuda:0')) | > avg_loss_gen: 1.9209171315314064 (-0.030646670032554946) | > avg_loss_kl: 2.294325737886026 (+0.004341912941193016) | > avg_loss_feat: 3.0516698394023196 (-0.04090291009822389) | > avg_loss_mel: 18.588301618334263 (+0.3023753904960387) | > avg_loss_duration: 1.6260145432512525 (+0.010886209111817813) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.48122881499815 (+0.2460537225427828) | > avg_grad_norm_1: tensor(1492.9070, device='cuda:0') (tensor(-232.4028, device='cuda:0'))  > EPOCH: 443/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:45:34)   --> TIME: 2025-02-25 12:45:38 -- STEP: 3/72 -- GLOBAL_STEP: 1031900 | > loss_disc: 2.659041404724121 (2.6914618015289307) | > loss_disc_real_0: 0.19622308015823364 (0.1569004307190577) | > loss_disc_real_1: 0.17942215502262115 (0.22551550964514414) | > loss_disc_real_2: 0.22365884482860565 (0.22065613170464835) | > loss_disc_real_3: 0.2515052855014801 (0.22778897484143576) | > loss_disc_real_4: 0.25186479091644287 (0.24986637135346731) | > loss_disc_real_5: 0.24374446272850037 (0.2397354394197464) | > loss_0: 2.659041404724121 (2.6914618015289307) | > grad_norm_0: tensor(23.3549, device='cuda:0') (tensor(26.1882, device='cuda:0')) | > loss_gen: 2.063136339187622 (1.9250023365020752) | > loss_kl: 2.388624429702759 (2.2602012157440186) | > loss_feat: 3.1404647827148438 (3.2293776671091714) | > loss_mel: 18.74005889892578 (18.777467091878254) | > loss_duration: 1.633604884147644 (1.6040362119674683) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.96588897705078 (27.796084721883137) | > grad_norm_1: tensor(3071.4897, device='cuda:0') (tensor(2677.7090, device='cuda:0')) | > current_lr_0: 0.0001892254015598793 | > current_lr_1: 0.0001892254015598793 | > step_time: 0.4335 (0.4290131727854411) | > loader_time: 0.0087 (0.0069043636322021484)  --> TIME: 2025-02-25 12:46:00 -- STEP: 53/72 -- GLOBAL_STEP: 1031950 | > loss_disc: 2.7998032569885254 (2.7410096987238477) | > loss_disc_real_0: 0.27995386719703674 (0.18107165515703974) | > loss_disc_real_1: 0.26510724425315857 (0.23256288496953137) | > loss_disc_real_2: 0.22534538805484772 (0.23642910342171508) | > loss_disc_real_3: 0.25974220037460327 (0.24340210292699202) | > loss_disc_real_4: 0.2290199249982834 (0.24843870243936214) | > loss_disc_real_5: 0.2374621331691742 (0.25228594102949475) | > loss_0: 2.7998032569885254 (2.7410096987238477) | > grad_norm_0: tensor(19.7510, device='cuda:0') (tensor(27.6547, device='cuda:0')) | > loss_gen: 1.8556982278823853 (1.9563246223161805) | > loss_kl: 2.2624082565307617 (2.280414938926697) | > loss_feat: 2.231370449066162 (3.130257251127711) | > loss_mel: 16.32398796081543 (18.635297307428328) | > loss_duration: 1.650926947593689 (1.6290656125770424) | > amp_scaler: 32.0 (32.0) | > loss_1: 24.324390411376953 (27.63135967614516) | > grad_norm_1: tensor(750.2278, device='cuda:0') (tensor(1916.4282, device='cuda:0')) | > current_lr_0: 0.0001892254015598793 | > current_lr_1: 0.0001892254015598793 | > step_time: 0.4437 (0.44276994129396835) | > loader_time: 0.0091 (0.007858969130606021) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652216951612016 (-1.2592530586347933e-06) | > avg_step_time: 0.45574943112655425 (-0.002200925853890423) | > avg_loss_disc: 2.7316172693816716 (+0.0012489943437179463) | > avg_loss_disc_real_0: 0.18334986379658672 (-0.006841731795542677) | > avg_loss_disc_real_1: 0.23608415479391393 (+0.006582246070176767) | > avg_loss_disc_real_2: 0.23581084713969433 (+0.0009970981890047304) | > avg_loss_disc_real_3: 0.2423499388174272 (+0.007690275848751343) | > avg_loss_disc_real_4: 0.2452190402947681 (+0.0010463165145524755) | > avg_loss_disc_real_5: 0.24861845835833488 (+0.002207701055096961) | > avg_loss_0: 2.7316172693816716 (+0.0012489943437179463) | > avg_grad_norm_0: tensor(25.7316, device='cuda:0') (+tensor(2.7204, device='cuda:0')) | > avg_loss_gen: 1.9577935715796242 (+0.03687644004821777) | > avg_loss_kl: 2.3418175922313216 (+0.04749185434529579) | > avg_loss_feat: 3.084916930803111 (+0.033247091400791184) | > avg_loss_mel: 18.861391846562785 (+0.27309022822852214) | > avg_loss_duration: 1.637929296829331 (+0.011914753578078585) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.883849157413966 (+0.40262034241581546) | > avg_grad_norm_1: tensor(1860.1714, device='cuda:0') (+tensor(367.2644, device='cuda:0'))  > EPOCH: 444/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:46:11)   --> TIME: 2025-02-25 12:46:27 -- STEP: 31/72 -- GLOBAL_STEP: 1032000 | > loss_disc: 2.7701292037963867 (2.737950663412771) | > loss_disc_real_0: 0.21806541085243225 (0.18726658845140087) | > loss_disc_real_1: 0.26481893658638 (0.22780157577606938) | > loss_disc_real_2: 0.26037585735321045 (0.23485428431341726) | > loss_disc_real_3: 0.22998572885990143 (0.23780258288306574) | > loss_disc_real_4: 0.2308153212070465 (0.24838623260298076) | > loss_disc_real_5: 0.22930023074150085 (0.2541047161625277) | > loss_0: 2.7701292037963867 (2.737950663412771) | > grad_norm_0: tensor(12.2203, device='cuda:0') (tensor(27.5792, device='cuda:0')) | > loss_gen: 1.944582223892212 (1.9852952841789502) | > loss_kl: 2.3665008544921875 (2.4821917664620186) | > loss_feat: 3.30100417137146 (3.2248882016827984) | > loss_mel: 20.25649642944336 (20.01353891434208) | > loss_duration: 1.6171281337738037 (1.6155209272138533) | > amp_scaler: 32.0 (32.0) | > loss_1: 29.4857120513916 (29.321435066961474) | > grad_norm_1: tensor(933.8074, device='cuda:0') (tensor(2059.1340, device='cuda:0')) | > current_lr_0: 0.0001892017483846843 | > current_lr_1: 0.0001892017483846843 | > step_time: 0.4407 (0.4355443446866928) | > loader_time: 0.0085 (0.007041854243124685) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614768444652289 (-3.7448506959726624e-05) | > avg_step_time: 0.4617965557205845 (+0.006047124594030229) | > avg_loss_disc: 2.7081706926856244 (-0.023446576696047217) | > avg_loss_disc_real_0: 0.18475223112274225 (+0.0014023673261555347) | > avg_loss_disc_real_1: 0.22630309324029466 (-0.009781061553619269) | > avg_loss_disc_real_2: 0.23085114024054837 (-0.00495970689914596) | > avg_loss_disc_real_3: 0.23463451463571736 (-0.00771542418170984) | > avg_loss_disc_real_4: 0.24256588665532394 (-0.0026531536394441557) | > avg_loss_disc_real_5: 0.24717611305310688 (-0.0014423453052279933) | > avg_loss_0: 2.7081706926856244 (-0.023446576696047217) | > avg_grad_norm_0: tensor(30.7768, device='cuda:0') (+tensor(5.0452, device='cuda:0')) | > avg_loss_gen: 1.9675597372189377 (+0.00976616563931354) | > avg_loss_kl: 2.3277810358665367 (-0.014036556364784847) | > avg_loss_feat: 3.11288132466061 (+0.02796439385749938) | > avg_loss_mel: 18.745462874291647 (-0.11592897227113852) | > avg_loss_duration: 1.6246560022864542 (-0.013273294542876846) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.778340890373983 (-0.1055082670399834) | > avg_grad_norm_1: tensor(2066.6785, device='cuda:0') (+tensor(206.5071, device='cuda:0'))  > EPOCH: 445/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:46:49)   --> TIME: 2025-02-25 12:46:55 -- STEP: 9/72 -- GLOBAL_STEP: 1032050 | > loss_disc: 2.815631628036499 (2.685669263203939) | > loss_disc_real_0: 0.1674407422542572 (0.1572501692507002) | > loss_disc_real_1: 0.19659721851348877 (0.22660896844334072) | > loss_disc_real_2: 0.19685395061969757 (0.24780556890699598) | > loss_disc_real_3: 0.23656147718429565 (0.23089312513669333) | > loss_disc_real_4: 0.26665595173835754 (0.2411464403072993) | > loss_disc_real_5: 0.24413402378559113 (0.24592793318960401) | > loss_0: 2.815631628036499 (2.685669263203939) | > grad_norm_0: tensor(15.8181, device='cuda:0') (tensor(29.6376, device='cuda:0')) | > loss_gen: 1.783190369606018 (1.9659129513634577) | > loss_kl: 2.0837700366973877 (2.030831045574612) | > loss_feat: 3.021178960800171 (3.1888111432393393) | > loss_mel: 18.064743041992188 (17.558951059977215) | > loss_duration: 1.5523911714553833 (1.5773000717163086) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.505273818969727 (26.32180616590712) | > grad_norm_1: tensor(1001.8293, device='cuda:0') (tensor(1977.3114, device='cuda:0')) | > current_lr_0: 0.00018917809816613621 | > current_lr_1: 0.00018917809816613621 | > step_time: 0.4213 (0.4295407666100396) | > loader_time: 0.0057 (0.0064697265625)  --> TIME: 2025-02-25 12:47:18 -- STEP: 59/72 -- GLOBAL_STEP: 1032100 | > loss_disc: 2.729572057723999 (2.7226123405715166) | > loss_disc_real_0: 0.2280907928943634 (0.17276395200672795) | > loss_disc_real_1: 0.19969502091407776 (0.2287377063500679) | > loss_disc_real_2: 0.241248220205307 (0.2568852100837028) | > loss_disc_real_3: 0.24096520245075226 (0.244583053103948) | > loss_disc_real_4: 0.25209587812423706 (0.2452710836620654) | > loss_disc_real_5: 0.2630055546760559 (0.2486519773127669) | > loss_0: 2.729572057723999 (2.7226123405715166) | > grad_norm_0: tensor(94.2231, device='cuda:0') (tensor(25.3028, device='cuda:0')) | > loss_gen: 1.9774086475372314 (1.9662701618873462) | > loss_kl: 1.995725154876709 (2.1327876579963556) | > loss_feat: 3.003711700439453 (3.137603359707332) | > loss_mel: 15.2775297164917 (17.71958714824612) | > loss_duration: 1.6402342319488525 (1.5953820277068576) | > amp_scaler: 32.0 (32.0) | > loss_1: 23.894609451293945 (26.551630408076917) | > grad_norm_1: tensor(2557.2930, device='cuda:0') (tensor(1625.3915, device='cuda:0')) | > current_lr_0: 0.00018917809816613621 | > current_lr_1: 0.00018917809816613621 | > step_time: 0.4692 (0.44735479758957686) | > loader_time: 0.0096 (0.008007591053590937) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580533551498197 (-3.423489315409166e-05) | > avg_step_time: 0.4574235056487607 (-0.004373050071823759) | > avg_loss_disc: 2.7292289465246067 (+0.0210582538389823) | > avg_loss_disc_real_0: 0.17663865483982463 (-0.008113576282917617) | > avg_loss_disc_real_1: 0.22836071735536548 (+0.002057624115070822) | > avg_loss_disc_real_2: 0.2543388036355166 (+0.023487663394968217) | > avg_loss_disc_real_3: 0.24123888935001803 (+0.00660437471430067) | > avg_loss_disc_real_4: 0.24507472632636487 (+0.00250883967104093) | > avg_loss_disc_real_5: 0.24858676610698163 (+0.0014106530538747497) | > avg_loss_0: 2.7292289465246067 (+0.0210582538389823) | > avg_grad_norm_0: tensor(25.0452, device='cuda:0') (tensor(-5.7316, device='cuda:0')) | > avg_loss_gen: 1.9534474157951243 (-0.014112321423813468) | > avg_loss_kl: 2.1500470722225353 (-0.1777339636440014) | > avg_loss_feat: 3.0912527332843194 (-0.021628591376290807) | > avg_loss_mel: 17.755073775707835 (-0.9903890985838117) | > avg_loss_duration: 1.6034473566941811 (-0.02120864559227309) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.553268378889058 (-1.2250725114849246) | > avg_grad_norm_1: tensor(1617.2866, device='cuda:0') (tensor(-449.3918, device='cuda:0'))  > EPOCH: 446/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:47:26)   --> TIME: 2025-02-25 12:47:45 -- STEP: 37/72 -- GLOBAL_STEP: 1032150 | > loss_disc: 2.633140802383423 (2.7470116873045227) | > loss_disc_real_0: 0.14743247628211975 (0.19996636240063487) | > loss_disc_real_1: 0.2875222861766815 (0.22432868102112333) | > loss_disc_real_2: 0.2257590889930725 (0.23749483396878113) | > loss_disc_real_3: 0.233768031001091 (0.2395900024755581) | > loss_disc_real_4: 0.23570841550827026 (0.24448905885219574) | > loss_disc_real_5: 0.2258242815732956 (0.2474610028234688) | > loss_0: 2.633140802383423 (2.7470116873045227) | > grad_norm_0: tensor(16.4698, device='cuda:0') (tensor(22.3905, device='cuda:0')) | > loss_gen: 1.987788200378418 (1.9127448023976505) | > loss_kl: 2.8382294178009033 (2.2955106077967464) | > loss_feat: 3.24726939201355 (3.0672205525475578) | > loss_mel: 18.9400691986084 (18.34812071516707) | > loss_duration: 1.6575558185577393 (1.5828306288332552) | > amp_scaler: 32.0 (32.0) | > loss_1: 28.67091178894043 (27.206427393732845) | > grad_norm_1: tensor(640.1058, device='cuda:0') (tensor(1070.7444, device='cuda:0')) | > current_lr_0: 0.00018915445090386543 | > current_lr_1: 0.00018915445090386543 | > step_time: 0.4502 (0.4384408126006255) | > loader_time: 0.0088 (0.007460503964810758) --> EVAL PERFORMANCE | > avg_loader_time: 0.008846403847278004 (+0.00026587029577980655) | > avg_step_time: 0.45623453570083833 (-0.0011889699479223914) | > avg_loss_disc: 2.735382996814351 (+0.006154050289744362) | > avg_loss_disc_real_0: 0.1974821954335965 (+0.02084354059377186) | > avg_loss_disc_real_1: 0.23058343542293763 (+0.0022227180675721547) | > avg_loss_disc_real_2: 0.23715633741566833 (-0.017182466219848258) | > avg_loss_disc_real_3: 0.23663384540819785 (-0.0046050439418201805) | > avg_loss_disc_real_4: 0.24326002157070267 (-0.0018147047556621998) | > avg_loss_disc_real_5: 0.2466393419554536 (-0.0019474241515280433) | > avg_loss_0: 2.735382996814351 (+0.006154050289744362) | > avg_grad_norm_0: tensor(22.0897, device='cuda:0') (tensor(-2.9554, device='cuda:0')) | > avg_loss_gen: 1.929517191900334 (-0.02393022389479027) | > avg_loss_kl: 2.3841090403812037 (+0.23406196815866842) | > avg_loss_feat: 3.0777931146218744 (-0.013459618662444939) | > avg_loss_mel: 18.605711117596687 (+0.8506373418888522) | > avg_loss_duration: 1.616396106464762 (+0.012948749770580914) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.613526599507935 (+1.060258220618877) | > avg_grad_norm_1: tensor(1040.4905, device='cuda:0') (tensor(-576.7961, device='cuda:0'))  > EPOCH: 447/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:48:03)   --> TIME: 2025-02-25 12:48:12 -- STEP: 15/72 -- GLOBAL_STEP: 1032200 | > loss_disc: 2.809201240539551 (2.7495803674062094) | > loss_disc_real_0: 0.20847177505493164 (0.17888412922620772) | > loss_disc_real_1: 0.2506861686706543 (0.2325602501630783) | > loss_disc_real_2: 0.2517614960670471 (0.23887540896733603) | > loss_disc_real_3: 0.25999850034713745 (0.24140475193659464) | > loss_disc_real_4: 0.2810167074203491 (0.25285478035608927) | > loss_disc_real_5: 0.24002833664417267 (0.265273051460584) | > loss_0: 2.809201240539551 (2.7495803674062094) | > grad_norm_0: tensor(11.9324, device='cuda:0') (tensor(16.3076, device='cuda:0')) | > loss_gen: 1.9500694274902344 (2.0070122798283894) | > loss_kl: 2.123572587966919 (2.3957059144973756) | > loss_feat: 2.7670676708221436 (3.2750881195068358) | > loss_mel: 18.864887237548828 (19.244984690348307) | > loss_duration: 1.5717748403549194 (1.582950194676717) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.277372360229492 (28.505741119384766) | > grad_norm_1: tensor(64.8249, device='cuda:0') (tensor(976.2531, device='cuda:0')) | > current_lr_0: 0.00018913080659750244 | > current_lr_1: 0.00018913080659750244 | > step_time: 0.4272 (0.43295270601908364) | > loader_time: 0.0061 (0.006671841939290365)  --> TIME: 2025-02-25 12:48:36 -- STEP: 65/72 -- GLOBAL_STEP: 1032250 | > loss_disc: 2.8512511253356934 (2.7129665448115428) | > loss_disc_real_0: 0.11336616426706314 (0.17730516516245334) | > loss_disc_real_1: 0.21892288327217102 (0.23101301926832932) | > loss_disc_real_2: 0.2535494267940521 (0.23881820623691266) | > loss_disc_real_3: 0.22287960350513458 (0.2388237258562675) | > loss_disc_real_4: 0.22904308140277863 (0.244775239779399) | > loss_disc_real_5: 0.22852279245853424 (0.24735431602367994) | > loss_0: 2.8512511253356934 (2.7129665448115428) | > grad_norm_0: tensor(27.0271, device='cuda:0') (tensor(23.8488, device='cuda:0')) | > loss_gen: 1.9018585681915283 (1.9647399297127357) | > loss_kl: 2.6384129524230957 (2.323732583339398) | > loss_feat: 3.1168246269226074 (3.142093581419725) | > loss_mel: 20.66447639465332 (18.79777374267578) | > loss_duration: 1.7068359851837158 (1.6147561220022348) | > amp_scaler: 32.0 (32.0) | > loss_1: 30.028409957885742 (27.843095955481896) | > grad_norm_1: tensor(88.5725, device='cuda:0') (tensor(1545.0237, device='cuda:0')) | > current_lr_0: 0.00018913080659750244 | > current_lr_1: 0.00018913080659750244 | > step_time: 0.5254 (0.45910173196059006) | > loader_time: 0.0121 (0.008392730126014124) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866858052535796 (-0.00017782332192004327) | > avg_step_time: 0.4659975448124845 (+0.009763009111646181) | > avg_loss_disc: 2.710947520296339 (-0.024435476518012145) | > avg_loss_disc_real_0: 0.1772860082941996 (-0.020196187139396887) | > avg_loss_disc_real_1: 0.23090954628628743 (+0.0003261108633497989) | > avg_loss_disc_real_2: 0.23905369114707892 (+0.0018973537314105893) | > avg_loss_disc_real_3: 0.2388234342068014 (+0.0021895887986035645) | > avg_loss_disc_real_4: 0.24413403994600538 (+0.0008740183753027098) | > avg_loss_disc_real_5: 0.24675771069358776 (+0.00011836873813417248) | > avg_loss_0: 2.710947520296339 (-0.024435476518012145) | > avg_grad_norm_0: tensor(23.8592, device='cuda:0') (+tensor(1.7695, device='cuda:0')) | > avg_loss_gen: 1.9598769503580014 (+0.030359758457667363) | > avg_loss_kl: 2.3371364818492406 (-0.04697255853196314) | > avg_loss_feat: 3.123038825854449 (+0.04524571123257459) | > avg_loss_mel: 18.797734972456812 (+0.19202385486012474) | > avg_loss_duration: 1.619659699184794 (+0.0032635927200319603) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.837446911234252 (+0.22392031172631732) | > avg_grad_norm_1: tensor(1509.9915, device='cuda:0') (+tensor(469.5010, device='cuda:0'))  > EPOCH: 448/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:48:41)   --> TIME: 2025-02-25 12:49:02 -- STEP: 43/72 -- GLOBAL_STEP: 1032300 | > loss_disc: 2.704102039337158 (2.700611996096234) | > loss_disc_real_0: 0.19334635138511658 (0.18092418054855147) | > loss_disc_real_1: 0.25106826424598694 (0.22587342387021975) | > loss_disc_real_2: 0.22189927101135254 (0.23681802458541337) | > loss_disc_real_3: 0.22421860694885254 (0.23575395553611045) | > loss_disc_real_4: 0.237024188041687 (0.2424858070390169) | > loss_disc_real_5: 0.23358426988124847 (0.2448806405760521) | > loss_0: 2.704102039337158 (2.700611996096234) | > grad_norm_0: tensor(12.8486, device='cuda:0') (tensor(30.2421, device='cuda:0')) | > loss_gen: 1.9300627708435059 (2.0019464076951494) | > loss_kl: 1.914747953414917 (2.1922724579655855) | > loss_feat: 3.334803342819214 (3.3208811560342477) | > loss_mel: 17.6729679107666 (18.608379142228948) | > loss_duration: 1.606445550918579 (1.5948117001112116) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.459028244018555 (27.718290639478106) | > grad_norm_1: tensor(2591.2188, device='cuda:0') (tensor(1886.0754, device='cuda:0')) | > current_lr_0: 0.00018910716524667774 | > current_lr_1: 0.00018910716524667774 | > step_time: 0.4532 (0.4431670100189919) | > loader_time: 0.0095 (0.0074693546738735465) --> EVAL PERFORMANCE | > avg_loader_time: 0.008644157732036754 (-2.4422793321206976e-05) | > avg_step_time: 0.4584681148260412 (-0.007529429986443292) | > avg_loss_disc: 2.7063863848296688 (-0.004561135466670141) | > avg_loss_disc_real_0: 0.1778163529512748 (+0.000530344657075188) | > avg_loss_disc_real_1: 0.22651877978318175 (-0.004390766503105686) | > avg_loss_disc_real_2: 0.2412098677645267 (+0.0021561766174477692) | > avg_loss_disc_real_3: 0.23687286591026144 (-0.0019505682965399729) | > avg_loss_disc_real_4: 0.24245922078549023 (-0.0016748191605151508) | > avg_loss_disc_real_5: 0.2457126878516775 (-0.0010450228419102703) | > avg_loss_0: 2.7063863848296688 (-0.004561135466670141) | > avg_grad_norm_0: tensor(30.6277, device='cuda:0') (+tensor(6.7685, device='cuda:0')) | > avg_loss_gen: 1.9831807546212639 (+0.023303804263262506) | > avg_loss_kl: 2.178197635731227 (-0.15893884611801345) | > avg_loss_feat: 3.244391337246962 (+0.12135251139251313) | > avg_loss_mel: 18.357713457564227 (-0.44002151489258523) | > avg_loss_duration: 1.6079788023317363 (-0.011680896853057732) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.371461922014262 (-0.4659849892199901) | > avg_grad_norm_1: tensor(1894.6810, device='cuda:0') (+tensor(384.6896, device='cuda:0'))  > EPOCH: 449/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:49:18)   --> TIME: 2025-02-25 12:49:29 -- STEP: 21/72 -- GLOBAL_STEP: 1032350 | > loss_disc: 2.7434327602386475 (2.704950275875273) | > loss_disc_real_0: 0.1538403034210205 (0.1892942651396706) | > loss_disc_real_1: 0.18262851238250732 (0.23279855648676553) | > loss_disc_real_2: 0.25560832023620605 (0.2384876168909527) | > loss_disc_real_3: 0.2503291964530945 (0.2351554369642621) | > loss_disc_real_4: 0.25834041833877563 (0.2438561731860751) | > loss_disc_real_5: 0.26351460814476013 (0.24947099600519454) | > loss_0: 2.7434327602386475 (2.704950275875273) | > grad_norm_0: tensor(29.8462, device='cuda:0') (tensor(23.0333, device='cuda:0')) | > loss_gen: 1.8588312864303589 (1.9621436709449405) | > loss_kl: 2.0888805389404297 (2.0686230886550176) | > loss_feat: 3.140101432800293 (3.2560974870409285) | > loss_mel: 17.935462951660156 (18.085083552769248) | > loss_duration: 1.6099580526351929 (1.5683028981799172) | > amp_scaler: 64.0 (59.42857142857143) | > loss_1: 26.63323402404785 (26.940251032511394) | > grad_norm_1: tensor(1893.3082, device='cuda:0') (tensor(1476.0049, device='cuda:0')) | > current_lr_0: 0.0001890835268510219 | > current_lr_1: 0.0001890835268510219 | > step_time: 0.4347 (0.4408422424679711) | > loader_time: 0.0067 (0.006804409481230236)  --> TIME: 2025-02-25 12:49:54 -- STEP: 71/72 -- GLOBAL_STEP: 1032400 | > loss_disc: 2.9092297554016113 (2.725347492056833) | > loss_disc_real_0: 0.14026443660259247 (0.18672280643187777) | > loss_disc_real_1: 0.2390090674161911 (0.2314214007535451) | > loss_disc_real_2: 0.1941322386264801 (0.23923538604252775) | > loss_disc_real_3: 0.27081298828125 (0.23720074224639948) | > loss_disc_real_4: 0.22308914363384247 (0.24670466465849272) | > loss_disc_real_5: 0.20780356228351593 (0.24821531919526393) | > loss_0: 2.9092297554016113 (2.725347492056833) | > grad_norm_0: tensor(24.7987, device='cuda:0') (tensor(25.3669, device='cuda:0')) | > loss_gen: 1.6744799613952637 (1.9669586487219368) | > loss_kl: 2.140718936920166 (2.2002115686174855) | > loss_feat: 2.928446054458618 (3.185688938893063) | > loss_mel: 17.929136276245117 (18.20042505398603) | > loss_duration: 1.6106047630310059 (1.6073172495398722) | > amp_scaler: 32.0 (51.83098591549296) | > loss_1: 26.283388137817383 (27.1606015487456) | > grad_norm_1: tensor(1017.8151, device='cuda:0') (tensor(1524.7854, device='cuda:0')) | > current_lr_0: 0.0001890835268510219 | > current_lr_1: 0.0001890835268510219 | > step_time: 0.5065 (0.4618289907213668) | > loader_time: 0.0082 (0.008772457149666799) --> EVAL PERFORMANCE | > avg_loader_time: 0.008772457149666799 (+0.00012829941763004525) | > avg_step_time: 0.4618289907213668 (+0.003360875895325577) | > avg_loss_disc: 2.725347492056833 (+0.018961107227164042) | > avg_loss_disc_real_0: 0.18672280643187777 (+0.008906453480602983) | > avg_loss_disc_real_1: 0.2314214007535451 (+0.004902620970363342) | > avg_loss_disc_real_2: 0.23923538604252775 (-0.001974481721998933) | > avg_loss_disc_real_3: 0.23720074224639948 (+0.0003278763361380388) | > avg_loss_disc_real_4: 0.24670466465849272 (+0.004245443873002491) | > avg_loss_disc_real_5: 0.24821531919526393 (+0.0025026313435864422) | > avg_loss_0: 2.725347492056833 (+0.018961107227164042) | > avg_grad_norm_0: tensor(25.3669, device='cuda:0') (tensor(-5.2608, device='cuda:0')) | > avg_loss_gen: 1.9669586487219368 (-0.016222105899327044) | > avg_loss_kl: 2.2002115686174855 (+0.022013932886258303) | > avg_loss_feat: 3.185688938893063 (-0.05870239835389901) | > avg_loss_mel: 18.20042505398603 (-0.15728840357819607) | > avg_loss_duration: 1.6073172495398722 (-0.000661552791864084) | > avg_amp_scaler: 51.83098591549296 (+19.83098591549296) | > avg_loss_1: 27.1606015487456 (-0.21086037326866247) | > avg_grad_norm_1: tensor(1524.7854, device='cuda:0') (tensor(-369.8956, device='cuda:0'))  > EPOCH: 450/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:49:55)   --> TIME: 2025-02-25 12:50:20 -- STEP: 49/72 -- GLOBAL_STEP: 1032450 | > loss_disc: 2.648885726928711 (2.7136741122420958) | > loss_disc_real_0: 0.1945495903491974 (0.19009512602066508) | > loss_disc_real_1: 0.21457578241825104 (0.23514278993314627) | > loss_disc_real_2: 0.23340608179569244 (0.24010601700568684) | > loss_disc_real_3: 0.22082261741161346 (0.23320664313374734) | > loss_disc_real_4: 0.24582131206989288 (0.24495773534385526) | > loss_disc_real_5: 0.24273896217346191 (0.24761458258239583) | > loss_0: 2.648885726928711 (2.7136741122420958) | > grad_norm_0: tensor(13.1577, device='cuda:0') (tensor(25.5805, device='cuda:0')) | > loss_gen: 1.9509007930755615 (1.9500833652457412) | > loss_kl: 2.0164005756378174 (2.1294365926664702) | > loss_feat: 2.6034674644470215 (3.028825107885867) | > loss_mel: 17.85549545288086 (17.690125660020474) | > loss_duration: 1.659407377243042 (1.577275774916824) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.08567237854004 (26.375746707527004) | > grad_norm_1: tensor(866.0035, device='cuda:0') (tensor(1318.4028, device='cuda:0')) | > current_lr_0: 0.0001890598914101655 | > current_lr_1: 0.0001890598914101655 | > step_time: 0.4485 (0.4573901332154566) | > loader_time: 0.0104 (0.007931923379703444) --> EVAL PERFORMANCE | > avg_loader_time: 0.008870259137220787 (+9.780198755398803e-05) | > avg_step_time: 0.46549165752572075 (+0.0036626668043539534) | > avg_loss_disc: 2.71461833362848 (-0.010729158428352648) | > avg_loss_disc_real_0: 0.18808813791879467 (+0.001365331486916893) | > avg_loss_disc_real_1: 0.23465053737163544 (+0.0032291366180903502) | > avg_loss_disc_real_2: 0.24142395358690075 (+0.0021885675443729935) | > avg_loss_disc_real_3: 0.23491337139841537 (-0.0022873708479841104) | > avg_loss_disc_real_4: 0.24397347468725392 (-0.0027311899712388066) | > avg_loss_disc_real_5: 0.2471138952483593 (-0.0011014239469046283) | > avg_loss_0: 2.71461833362848 (-0.010729158428352648) | > avg_grad_norm_0: tensor(27.1412, device='cuda:0') (+tensor(1.7743, device='cuda:0')) | > avg_loss_gen: 1.9507865888971678 (-0.01617205982476899) | > avg_loss_kl: 2.1394809484481807 (-0.06073062016930475) | > avg_loss_feat: 3.020250075299975 (-0.16543886359308813) | > avg_loss_mel: 17.63921134572633 (-0.5612137082597002) | > avg_loss_duration: 1.5936681686992376 (-0.013649080840634653) | > avg_amp_scaler: 32.0 (-19.83098591549296) | > avg_loss_1: 26.34339735541545 (-0.8172041933301486) | > avg_grad_norm_1: tensor(1469.0767, device='cuda:0') (tensor(-55.7087, device='cuda:0'))  > EPOCH: 451/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:50:33)   --> TIME: 2025-02-25 12:50:47 -- STEP: 27/72 -- GLOBAL_STEP: 1032500 | > loss_disc: 2.5781185626983643 (2.703607691658868) | > loss_disc_real_0: 0.2252279818058014 (0.1831629174175086) | > loss_disc_real_1: 0.23811273276805878 (0.22429909308751425) | > loss_disc_real_2: 0.21640413999557495 (0.23153401083416408) | > loss_disc_real_3: 0.2339017540216446 (0.23140643426665553) | > loss_disc_real_4: 0.22740943729877472 (0.243317860144156) | > loss_disc_real_5: 0.23257693648338318 (0.2466718388928307) | > loss_0: 2.5781185626983643 (2.703607691658868) | > grad_norm_0: tensor(53.5066, device='cuda:0') (tensor(33.2201, device='cuda:0')) | > loss_gen: 2.2857847213745117 (1.9884140756395128) | > loss_kl: 2.208378553390503 (2.3096144817493576) | > loss_feat: 3.5703682899475098 (3.238270468182034) | > loss_mel: 18.607969284057617 (18.35855349787959) | > loss_duration: 1.566025733947754 (1.595096270243327) | > amp_scaler: 32.0 (32.0) | > loss_1: 28.238525390625 (27.4899489791305) | > grad_norm_1: tensor(3202.6790, device='cuda:0') (tensor(2082.8035, device='cuda:0')) | > current_lr_0: 0.00018903625892373923 | > current_lr_1: 0.00018903625892373923 | > step_time: 0.4481 (0.4398198922475179) | > loader_time: 0.0084 (0.006675702554208261) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614301681518555 (-0.00025595745570223226) | > avg_step_time: 0.46185455523746116 (-0.003637102288259586) | > avg_loss_disc: 2.709254046561013 (-0.005364287067467277) | > avg_loss_disc_real_0: 0.18073296746317769 (-0.007355170455616983) | > avg_loss_disc_real_1: 0.23163891289855393 (-0.003011624473081509) | > avg_loss_disc_real_2: 0.2315267670742223 (-0.009897186512678458) | > avg_loss_disc_real_3: 0.23316057467124832 (-0.0017527967271670497) | > avg_loss_disc_real_4: 0.2439532261079466 (-2.024857930732482e-05) | > avg_loss_disc_real_5: 0.2445742345192063 (-0.002539660729153015) | > avg_loss_0: 2.709254046561013 (-0.005364287067467277) | > avg_grad_norm_0: tensor(33.0581, device='cuda:0') (+tensor(5.9169, device='cuda:0')) | > avg_loss_gen: 1.9653889780313196 (+0.014602389134151794) | > avg_loss_kl: 2.250087439174384 (+0.11060649072620343) | > avg_loss_feat: 3.1528873074222616 (+0.13263723212228662) | > avg_loss_mel: 18.009500127443133 (+0.3702887817168019) | > avg_loss_duration: 1.6091417174943736 (+0.015473548795136072) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.98700566358969 (+0.6436083081742403) | > avg_grad_norm_1: tensor(1948.0291, device='cuda:0') (+tensor(478.9524, device='cuda:0'))  > EPOCH: 452/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:51:11)   --> TIME: 2025-02-25 12:51:15 -- STEP: 5/72 -- GLOBAL_STEP: 1032550 | > loss_disc: 2.768366813659668 (2.689890909194946) | > loss_disc_real_0: 0.28737664222717285 (0.20010044425725937) | > loss_disc_real_1: 0.24888306856155396 (0.22560709416866304) | > loss_disc_real_2: 0.23356257379055023 (0.2537838727235794) | > loss_disc_real_3: 0.2342224419116974 (0.23729958832263948) | > loss_disc_real_4: 0.2474955916404724 (0.23992661833763124) | > loss_disc_real_5: 0.24965326488018036 (0.25455511808395387) | > loss_0: 2.768366813659668 (2.689890909194946) | > grad_norm_0: tensor(61.4862, device='cuda:0') (tensor(38.7983, device='cuda:0')) | > loss_gen: 2.0903987884521484 (2.0012518167495728) | > loss_kl: 1.970893383026123 (2.0250523328781127) | > loss_feat: 2.9455089569091797 (2.9282209873199463) | > loss_mel: 17.54387855529785 (17.56604766845703) | > loss_duration: 1.525963544845581 (1.5644986391067506) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.076642990112305 (26.085071563720703) | > grad_norm_1: tensor(2038.4064, device='cuda:0') (tensor(1952.2670, device='cuda:0')) | > current_lr_0: 0.00018901262939137375 | > current_lr_1: 0.00018901262939137375 | > step_time: 0.4271 (0.43328080177307127) | > loader_time: 0.0076 (0.007237768173217774)  --> TIME: 2025-02-25 12:51:38 -- STEP: 55/72 -- GLOBAL_STEP: 1032600 | > loss_disc: 2.6324877738952637 (2.719248290495438) | > loss_disc_real_0: 0.16634313762187958 (0.17973989939147775) | > loss_disc_real_1: 0.23704810440540314 (0.23536242978139357) | > loss_disc_real_2: 0.2578984498977661 (0.235701924291524) | > loss_disc_real_3: 0.22695374488830566 (0.23588591841134157) | > loss_disc_real_4: 0.23865295946598053 (0.24564307765527205) | > loss_disc_real_5: 0.23547784984111786 (0.24682613855058497) | > loss_0: 2.6324877738952637 (2.719248290495438) | > grad_norm_0: tensor(21.9446, device='cuda:0') (tensor(30.0026, device='cuda:0')) | > loss_gen: 2.0317130088806152 (2.0064742175015535) | > loss_kl: 2.3402862548828125 (2.342347166754983) | > loss_feat: 3.315469741821289 (3.22384239543568) | > loss_mel: 18.0591983795166 (18.676562916148786) | > loss_duration: 1.6426327228546143 (1.6003095084970647) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.389301300048828 (27.849536236849698) | > grad_norm_1: tensor(1269.9358, device='cuda:0') (tensor(2011.6086, device='cuda:0')) | > current_lr_0: 0.00018901262939137375 | > current_lr_1: 0.00018901262939137375 | > step_time: 0.4632 (0.44751578244295986) | > loader_time: 0.0092 (0.008041850003329189) --> EVAL PERFORMANCE | > avg_loader_time: 0.008792534680433672 (+0.00017823299891511768) | > avg_step_time: 0.4592838085873026 (-0.0025707466501585508) | > avg_loss_disc: 2.7064064321383614 (-0.0028476144226514677) | > avg_loss_disc_real_0: 0.17904462801738524 (-0.0016883394457924417) | > avg_loss_disc_real_1: 0.23270557547958803 (+0.0010666625810341057) | > avg_loss_disc_real_2: 0.23344276152865986 (+0.001915994454437575) | > avg_loss_disc_real_3: 0.23629229148508799 (+0.003131716813839669) | > avg_loss_disc_real_4: 0.24397707006461183 (+2.3843956665242638e-05) | > avg_loss_disc_real_5: 0.246249291048923 (+0.001675056529716723) | > avg_loss_0: 2.7064064321383614 (-0.0028476144226514677) | > avg_grad_norm_0: tensor(31.2390, device='cuda:0') (tensor(-1.8191, device='cuda:0')) | > avg_loss_gen: 2.0014171533181635 (+0.03602817528684388) | > avg_loss_kl: 2.327782385785815 (+0.07769494661143073) | > avg_loss_feat: 3.182412254978234 (+0.029524947555972236) | > avg_loss_mel: 18.508588428228666 (+0.4990883007855338) | > avg_loss_duration: 1.6088450139677022 (-0.00029670352667143796) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.629045325265803 (+0.6420396616761117) | > avg_grad_norm_1: tensor(2129.9805, device='cuda:0') (+tensor(181.9514, device='cuda:0'))  > EPOCH: 453/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:51:48)   --> TIME: 2025-02-25 12:52:05 -- STEP: 33/72 -- GLOBAL_STEP: 1032650 | > loss_disc: 2.7117650508880615 (2.6641837972583193) | > loss_disc_real_0: 0.23293377459049225 (0.16875654639619764) | > loss_disc_real_1: 0.20906054973602295 (0.22166981480338357) | > loss_disc_real_2: 0.2317596971988678 (0.22830352593551984) | > loss_disc_real_3: 0.23848551511764526 (0.23233651527852722) | > loss_disc_real_4: 0.23951786756515503 (0.23971258916638113) | > loss_disc_real_5: 0.24091702699661255 (0.24221831424669785) | > loss_0: 2.7117650508880615 (2.6641837972583193) | > grad_norm_0: tensor(9.5822, device='cuda:0') (tensor(33.4519, device='cuda:0')) | > loss_gen: 1.8666318655014038 (1.9912212653593584) | > loss_kl: 2.00960111618042 (2.0662612770542945) | > loss_feat: 3.2508294582366943 (3.2609362240993613) | > loss_mel: 17.55573272705078 (17.744785193241) | > loss_duration: 1.5587146282196045 (1.5573827642382998) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.24150848388672 (26.62058685765122) | > grad_norm_1: tensor(835.4659, device='cuda:0') (tensor(2305.2124, device='cuda:0')) | > current_lr_0: 0.0001889890028126998 | > current_lr_1: 0.0001889890028126998 | > step_time: 0.4647 (0.4444912491422711) | > loader_time: 0.008 (0.007320078936490146) --> EVAL PERFORMANCE | > avg_loader_time: 0.008821819869565293 (+2.928518913162051e-05) | > avg_step_time: 0.4620287216884989 (+0.002744913101196289) | > avg_loss_disc: 2.6907490978778243 (-0.01565733426053706) | > avg_loss_disc_real_0: 0.17149074627480032 (-0.0075538817425849225) | > avg_loss_disc_real_1: 0.22493345678692134 (-0.007772118692666696) | > avg_loss_disc_real_2: 0.23074200749397278 (-0.002700754034687086) | > avg_loss_disc_real_3: 0.23406517799471466 (-0.002227113490373328) | > avg_loss_disc_real_4: 0.241793680988567 (-0.002183389076044845) | > avg_loss_disc_real_5: 0.24443152001206303 (-0.001817771036859983) | > avg_loss_0: 2.6907490978778243 (-0.01565733426053706) | > avg_grad_norm_0: tensor(36.5738, device='cuda:0') (+tensor(5.3348, device='cuda:0')) | > avg_loss_gen: 1.9943734659275538 (-0.007043687390609676) | > avg_loss_kl: 2.2108665721517204 (-0.11691581363409442) | > avg_loss_feat: 3.2368116949645565 (+0.05439943998632257) | > avg_loss_mel: 18.275511056604522 (-0.23307737162414455) | > avg_loss_duration: 1.603499797028555 (-0.005345216939147246) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.321062705886195 (-0.307982619379608) | > avg_grad_norm_1: tensor(2498.2119, device='cuda:0') (+tensor(368.2314, device='cuda:0'))  > EPOCH: 454/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:52:26)   --> TIME: 2025-02-25 12:52:33 -- STEP: 11/72 -- GLOBAL_STEP: 1032700 | > loss_disc: 2.599212884902954 (2.6205251000144263) | > loss_disc_real_0: 0.10284820944070816 (0.1579344387758862) | > loss_disc_real_1: 0.23519055545330048 (0.20839166776700455) | > loss_disc_real_2: 0.19550380110740662 (0.23544169014150446) | > loss_disc_real_3: 0.22803905606269836 (0.23352566090497104) | > loss_disc_real_4: 0.23389217257499695 (0.24356089938770642) | > loss_disc_real_5: 0.24994869530200958 (0.2478061860257929) | > loss_0: 2.599212884902954 (2.6205251000144263) | > grad_norm_0: tensor(20.2750, device='cuda:0') (tensor(27.4890, device='cuda:0')) | > loss_gen: 2.188180923461914 (2.0353355299342764) | > loss_kl: 2.369683027267456 (2.1093762679533525) | > loss_feat: 3.4976038932800293 (3.3620513569224966) | > loss_mel: 20.723777770996094 (18.613089301369406) | > loss_duration: 1.636004090309143 (1.5847505872899836) | > amp_scaler: 32.0 (32.0) | > loss_1: 30.41524887084961 (27.704603021795098) | > grad_norm_1: tensor(2833.7683, device='cuda:0') (tensor(2480.7051, device='cuda:0')) | > current_lr_0: 0.00018896537918734822 | > current_lr_1: 0.00018896537918734822 | > step_time: 0.4381 (0.44243706356395374) | > loader_time: 0.0074 (0.007015228271484375)  --> TIME: 2025-02-25 12:52:56 -- STEP: 61/72 -- GLOBAL_STEP: 1032750 | > loss_disc: 2.7227511405944824 (2.706954084458898) | > loss_disc_real_0: 0.22228875756263733 (0.18593010990346065) | > loss_disc_real_1: 0.2137790471315384 (0.22993415471960288) | > loss_disc_real_2: 0.22579030692577362 (0.23644899320406992) | > loss_disc_real_3: 0.20100025832653046 (0.24162493377435404) | > loss_disc_real_4: 0.22370681166648865 (0.24076300615169963) | > loss_disc_real_5: 0.23762139678001404 (0.24363470712646113) | > loss_0: 2.7227511405944824 (2.706954084458898) | > grad_norm_0: tensor(12.9970, device='cuda:0') (tensor(25.6362, device='cuda:0')) | > loss_gen: 1.781152367591858 (1.9882583872216648) | > loss_kl: 2.2303426265716553 (2.2145029619091847) | > loss_feat: 2.823957920074463 (3.1641048876965634) | > loss_mel: 17.005294799804688 (18.324883507900545) | > loss_duration: 1.6375700235366821 (1.6061110945998645) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.478317260742188 (27.29786078656306) | > grad_norm_1: tensor(1459.9406, device='cuda:0') (tensor(1829.8468, device='cuda:0')) | > current_lr_0: 0.00018896537918734822 | > current_lr_1: 0.00018896537918734822 | > step_time: 0.4649 (0.45202402990372453) | > loader_time: 0.0109 (0.008220058972718287) --> EVAL PERFORMANCE | > avg_loader_time: 0.008686001871673156 (-0.00013581799789213686) | > avg_step_time: 0.4609739981906515 (-0.0010547234978474096) | > avg_loss_disc: 2.70602097309811 (+0.015271875220285835) | > avg_loss_disc_real_0: 0.19124494849796025 (+0.019754202223159933) | > avg_loss_disc_real_1: 0.22879065825066097 (+0.0038572014637396346) | > avg_loss_disc_real_2: 0.23331494209632067 (+0.0025729346023478927) | > avg_loss_disc_real_3: 0.24077479919077646 (+0.006709621196061805) | > avg_loss_disc_real_4: 0.23984016247198614 (-0.001953518516580849) | > avg_loss_disc_real_5: 0.24214961956924116 (-0.002281900442821866) | > avg_loss_0: 2.70602097309811 (+0.015271875220285835) | > avg_grad_norm_0: tensor(26.4725, device='cuda:0') (tensor(-10.1013, device='cuda:0')) | > avg_loss_gen: 1.9809483004287936 (-0.013425165498760272) | > avg_loss_kl: 2.226314240777996 (+0.015447668626275668) | > avg_loss_feat: 3.1285199447416923 (-0.10829175022286419) | > avg_loss_mel: 18.381033883967863 (+0.10552282736334107) | > avg_loss_duration: 1.6124049122904387 (+0.008905115261883756) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.329221215046626 (+0.008158509160431038) | > avg_grad_norm_1: tensor(1766.0773, device='cuda:0') (tensor(-732.1346, device='cuda:0'))  > EPOCH: 455/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:53:03)   --> TIME: 2025-02-25 12:53:23 -- STEP: 39/72 -- GLOBAL_STEP: 1032800 | > loss_disc: 2.7610065937042236 (2.7450850560114937) | > loss_disc_real_0: 0.19012495875358582 (0.18927887712533656) | > loss_disc_real_1: 0.11248377710580826 (0.22474817817027753) | > loss_disc_real_2: 0.24482659995555878 (0.2407751698524524) | > loss_disc_real_3: 0.21083785593509674 (0.23785607784222335) | > loss_disc_real_4: 0.23623542487621307 (0.24561864748979226) | > loss_disc_real_5: 0.25164923071861267 (0.24807314001596892) | > loss_0: 2.7610065937042236 (2.7450850560114937) | > grad_norm_0: tensor(10.7281, device='cuda:0') (tensor(26.8960, device='cuda:0')) | > loss_gen: 1.9055631160736084 (1.9480071373474903) | > loss_kl: 2.122401475906372 (2.112283596625695) | > loss_feat: 2.789787769317627 (3.105499823888143) | > loss_mel: 17.262331008911133 (18.070365954668095) | > loss_duration: 1.5841338634490967 (1.5681781218602107) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.664216995239258 (26.804334689409306) | > grad_norm_1: tensor(527.2330, device='cuda:0') (tensor(1749.0800, device='cuda:0')) | > current_lr_0: 0.0001889417585149498 | > current_lr_1: 0.0001889417585149498 | > step_time: 0.4513 (0.43952153890560836) | > loader_time: 0.0085 (0.0073205507718599756) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653979905894105 (-3.2021965779050615e-05) | > avg_step_time: 0.456316632284245 (-0.004657365906406474) | > avg_loss_disc: 2.7428728996867866 (+0.03685192658867642) | > avg_loss_disc_real_0: 0.19056493106862188 (-0.0006800174293383754) | > avg_loss_disc_real_1: 0.23198619778727142 (+0.0031955395366104478) | > avg_loss_disc_real_2: 0.24046824809531092 (+0.00715330599899025) | > avg_loss_disc_real_3: 0.2382208312481222 (-0.0025539679426542505) | > avg_loss_disc_real_4: 0.24409099930608777 (+0.004250836834101629) | > avg_loss_disc_real_5: 0.24682177246456416 (+0.004672152895322995) | > avg_loss_0: 2.7428728996867866 (+0.03685192658867642) | > avg_grad_norm_0: tensor(26.9444, device='cuda:0') (+tensor(0.4720, device='cuda:0')) | > avg_loss_gen: 1.9363181104122753 (-0.044630190016518245) | > avg_loss_kl: 2.1319024579625734 (-0.09441178281542273) | > avg_loss_feat: 3.0516235559759 (-0.07689638876579208) | > avg_loss_mel: 17.833258239316265 (-0.5477756446515976) | > avg_loss_duration: 1.5947326287417347 (-0.017672283548703982) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.54783493364361 (-0.7813862814030159) | > avg_grad_norm_1: tensor(1724.8408, device='cuda:0') (tensor(-41.2365, device='cuda:0'))  > EPOCH: 456/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:53:40)   --> TIME: 2025-02-25 12:53:49 -- STEP: 17/72 -- GLOBAL_STEP: 1032850 | > loss_disc: 2.6871635913848877 (2.7234275761772606) | > loss_disc_real_0: 0.18662109971046448 (0.1859774878796409) | > loss_disc_real_1: 0.2460513859987259 (0.23487619003828833) | > loss_disc_real_2: 0.2476406991481781 (0.23522237995091608) | > loss_disc_real_3: 0.22941362857818604 (0.23482736331575058) | > loss_disc_real_4: 0.2400100827217102 (0.250890125246609) | > loss_disc_real_5: 0.26264017820358276 (0.24411863877492793) | > loss_0: 2.6871635913848877 (2.7234275761772606) | > grad_norm_0: tensor(5.9818, device='cuda:0') (tensor(32.9863, device='cuda:0')) | > loss_gen: 1.9241909980773926 (1.9799067062490128) | > loss_kl: 2.1723124980926514 (2.1044722585117115) | > loss_feat: 2.815505266189575 (3.1503236714531395) | > loss_mel: 17.458433151245117 (17.924070358276367) | > loss_duration: 1.5897963047027588 (1.557462117251228) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.960237503051758 (26.716235216926126) | > grad_norm_1: tensor(163.4235, device='cuda:0') (tensor(2161.2200, device='cuda:0')) | > current_lr_0: 0.00018891814079513542 | > current_lr_1: 0.00018891814079513542 | > step_time: 0.4293 (0.4312600388246424) | > loader_time: 0.0084 (0.00676885773153866)  --> TIME: 2025-02-25 12:54:13 -- STEP: 67/72 -- GLOBAL_STEP: 1032900 | > loss_disc: 2.8009326457977295 (2.748890510245935) | > loss_disc_real_0: 0.20842793583869934 (0.19513501010056752) | > loss_disc_real_1: 0.18877370655536652 (0.2333614225707837) | > loss_disc_real_2: 0.2160429209470749 (0.2370965516389306) | > loss_disc_real_3: 0.22309061884880066 (0.23698606353197524) | > loss_disc_real_4: 0.23078759014606476 (0.24677484271241656) | > loss_disc_real_5: 0.24083548784255981 (0.24597974922229995) | > loss_0: 2.8009326457977295 (2.748890510245935) | > grad_norm_0: tensor(29.3006, device='cuda:0') (tensor(32.8967, device='cuda:0')) | > loss_gen: 1.8059587478637695 (1.9478402369057954) | > loss_kl: 2.4320924282073975 (2.3653502304162553) | > loss_feat: 2.764761447906494 (3.0826813142691085) | > loss_mel: 18.06719207763672 (18.746803198287736) | > loss_duration: 1.6711342334747314 (1.605905701865011) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.741138458251953 (27.748580733341957) | > grad_norm_1: tensor(1891.8224, device='cuda:0') (tensor(1742.5175, device='cuda:0')) | > current_lr_0: 0.00018891814079513542 | > current_lr_1: 0.00018891814079513542 | > step_time: 0.5228 (0.45144947251277184) | > loader_time: 0.0128 (0.008510603833554396) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870649579545142 (+5.251588955731447e-05) | > avg_step_time: 0.4555808725491376 (-0.0007357597351074219) | > avg_loss_disc: 2.7484939467739053 (+0.005621047087118747) | > avg_loss_disc_real_0: 0.1955422803654637 (+0.004977349296841821) | > avg_loss_disc_real_1: 0.23423231928281382 (+0.0022461214955424025) | > avg_loss_disc_real_2: 0.2368829653296672 (-0.0035852827656437103) | > avg_loss_disc_real_3: 0.23632288419864547 (-0.0018979470494767392) | > avg_loss_disc_real_4: 0.24564440107681382 (+0.0015534017707260483) | > avg_loss_disc_real_5: 0.24577492034771073 (-0.0010468521168534306) | > avg_loss_0: 2.7484939467739053 (+0.005621047087118747) | > avg_grad_norm_0: tensor(32.2998, device='cuda:0') (+tensor(5.3553, device='cuda:0')) | > avg_loss_gen: 1.9464203992360074 (+0.010102288823732097) | > avg_loss_kl: 2.362100794281758 (+0.23019833631918463) | > avg_loss_feat: 3.0646668722931767 (+0.01304331631727651) | > avg_loss_mel: 18.70564968485228 (+0.8723914455360138) | > avg_loss_duration: 1.6084592711757606 (+0.013726642434025882) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.687297068851095 (+1.1394621352074843) | > avg_grad_norm_1: tensor(1739.7168, device='cuda:0') (+tensor(14.8760, device='cuda:0'))  > EPOCH: 457/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:54:17)   --> TIME: 2025-02-25 12:54:40 -- STEP: 45/72 -- GLOBAL_STEP: 1032950 | > loss_disc: 2.7602977752685547 (2.7644214577145045) | > loss_disc_real_0: 0.13281723856925964 (0.20842591143316694) | > loss_disc_real_1: 0.19182699918746948 (0.23202239639229244) | > loss_disc_real_2: 0.3047315776348114 (0.24785345130496556) | > loss_disc_real_3: 0.25157761573791504 (0.24097596506277721) | > loss_disc_real_4: 0.2598029375076294 (0.2452852295504676) | > loss_disc_real_5: 0.24556894600391388 (0.2470116866959466) | > loss_0: 2.7602977752685547 (2.7644214577145045) | > grad_norm_0: tensor(25.9410, device='cuda:0') (tensor(33.5545, device='cuda:0')) | > loss_gen: 1.9830751419067383 (1.9503878090116715) | > loss_kl: 2.207385301589966 (2.3238197724024445) | > loss_feat: 3.0895819664001465 (3.0519974019792344) | > loss_mel: 18.26386070251465 (19.014719178941515) | > loss_duration: 1.645817518234253 (1.6058334350585937) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.189720153808594 (27.946757507324218) | > grad_norm_1: tensor(1678.8083, device='cuda:0') (tensor(1668.8851, device='cuda:0')) | > current_lr_0: 0.000188894526027536 | > current_lr_1: 0.000188894526027536 | > step_time: 0.4554 (0.4421721988254123) | > loader_time: 0.0085 (0.00780323346455892) --> EVAL PERFORMANCE | > avg_loader_time: 0.00882028861784599 (+0.00011379282239457056) | > avg_step_time: 0.45500225752172335 (-0.0005786150274142421) | > avg_loss_disc: 2.7549410840155373 (+0.006447137241631928) | > avg_loss_disc_real_0: 0.2032666737344903 (+0.007724393369026611) | > avg_loss_disc_real_1: 0.23231153169148405 (-0.00192078759132977) | > avg_loss_disc_real_2: 0.2417012914805345 (+0.0048183261508673025) | > avg_loss_disc_real_3: 0.2411500226024171 (+0.004827138403771636) | > avg_loss_disc_real_4: 0.24281781953825077 (-0.00282658153856305) | > avg_loss_disc_real_5: 0.24791056715266802 (+0.0021356468049572863) | > avg_loss_0: 2.7549410840155373 (+0.006447137241631928) | > avg_grad_norm_0: tensor(31.8272, device='cuda:0') (tensor(-0.4726, device='cuda:0')) | > avg_loss_gen: 1.9364102306500288 (-0.010010168585978585) | > avg_loss_kl: 2.29143991100956 (-0.07066088327219822) | > avg_loss_feat: 3.0105307773805 (-0.05413609491267657) | > avg_loss_mel: 18.566621834123644 (-0.13902785072863466) | > avg_loss_duration: 1.6161631614389553 (+0.007703890263194735) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.421165869269572 (-0.26613119958152254) | > avg_grad_norm_1: tensor(1651.2018, device='cuda:0') (tensor(-88.5150, device='cuda:0'))  > EPOCH: 458/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:54:54)   --> TIME: 2025-02-25 12:55:06 -- STEP: 23/72 -- GLOBAL_STEP: 1033000 | > loss_disc: 2.781862735748291 (2.6921505824379297) | > loss_disc_real_0: 0.148493692278862 (0.16518280428388846) | > loss_disc_real_1: 0.2298082560300827 (0.22842487239319345) | > loss_disc_real_2: 0.2271183729171753 (0.23078941262286642) | > loss_disc_real_3: 0.21307329833507538 (0.23278643255648407) | > loss_disc_real_4: 0.18983398377895355 (0.23812631550042526) | > loss_disc_real_5: 0.24068720638751984 (0.24686441019825314) | > loss_0: 2.781862735748291 (2.6921505824379297) | > grad_norm_0: tensor(28.8113, device='cuda:0') (tensor(26.9183, device='cuda:0')) | > loss_gen: 1.958867073059082 (1.9555756734765095) | > loss_kl: 2.123781442642212 (2.3404791717943936) | > loss_feat: 3.1611504554748535 (3.259187366651452) | > loss_mel: 16.953659057617188 (18.668775641399883) | > loss_duration: 1.574826717376709 (1.5915352779885996) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.77228355407715 (27.815552835879117) | > grad_norm_1: tensor(1357.7410, device='cuda:0') (tensor(1575.9449, device='cuda:0')) | > current_lr_0: 0.00018887091421178256 | > current_lr_1: 0.00018887091421178256 | > step_time: 0.4334 (0.43523953271948773) | > loader_time: 0.0066 (0.00678259393443232) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612951762239697 (-0.00020733685560629309) | > avg_step_time: 0.4555214525948108 (+0.0005191950730874528) | > avg_loss_disc: 2.7280826770084015 (-0.02685840700713582) | > avg_loss_disc_real_0: 0.1877973497333661 (-0.015469324001124218) | > avg_loss_disc_real_1: 0.2396915160434347 (+0.007379984351950641) | > avg_loss_disc_real_2: 0.23357893841367372 (-0.008122353066860794) | > avg_loss_disc_real_3: 0.23812784548376648 (-0.003022177118650632) | > avg_loss_disc_real_4: 0.24134322119430757 (-0.0014745983439432009) | > avg_loss_disc_real_5: 0.24874623308719043 (+0.0008356659345224171) | > avg_loss_0: 2.7280826770084015 (-0.02685840700713582) | > avg_grad_norm_0: tensor(31.5205, device='cuda:0') (tensor(-0.3067, device='cuda:0')) | > avg_loss_gen: 1.9591356280823828 (+0.022725397432354022) | > avg_loss_kl: 2.2145675958042403 (-0.07687231520531945) | > avg_loss_feat: 3.0843307871214103 (+0.07380000974091017) | > avg_loss_mel: 17.918737330906836 (-0.6478845032168081) | > avg_loss_duration: 1.6081734475955156 (-0.007989713843439716) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.784944829806477 (-0.6362210394630949) | > avg_grad_norm_1: tensor(1736.2445, device='cuda:0') (+tensor(85.0427, device='cuda:0'))  > EPOCH: 459/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:55:31)   --> TIME: 2025-02-25 12:55:34 -- STEP: 1/72 -- GLOBAL_STEP: 1033050 | > loss_disc: 2.713623523712158 (2.713623523712158) | > loss_disc_real_0: 0.1318434327840805 (0.1318434327840805) | > loss_disc_real_1: 0.19566644728183746 (0.19566644728183746) | > loss_disc_real_2: 0.2537071406841278 (0.2537071406841278) | > loss_disc_real_3: 0.24884478747844696 (0.24884478747844696) | > loss_disc_real_4: 0.23566381633281708 (0.23566381633281708) | > loss_disc_real_5: 0.23882952332496643 (0.23882952332496643) | > loss_0: 2.713623523712158 (2.713623523712158) | > grad_norm_0: tensor(35.5280, device='cuda:0') (tensor(35.5280, device='cuda:0')) | > loss_gen: 1.9387760162353516 (1.9387760162353516) | > loss_kl: 2.193869113922119 (2.193869113922119) | > loss_feat: 3.0921502113342285 (3.0921502113342285) | > loss_mel: 18.153079986572266 (18.153079986572266) | > loss_duration: 1.605709195137024 (1.605709195137024) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.983583450317383 (26.983583450317383) | > grad_norm_1: tensor(2976.2520, device='cuda:0') (tensor(2976.2520, device='cuda:0')) | > current_lr_0: 0.00018884730534750609 | > current_lr_1: 0.00018884730534750609 | > step_time: 0.423 (0.4230062961578369) | > loader_time: 0.0049 (0.004888772964477539)  --> TIME: 2025-02-25 12:55:56 -- STEP: 51/72 -- GLOBAL_STEP: 1033100 | > loss_disc: 2.6794204711914062 (2.687228408514284) | > loss_disc_real_0: 0.23332959413528442 (0.16477606555118274) | > loss_disc_real_1: 0.25379934906959534 (0.22089291086383894) | > loss_disc_real_2: 0.25137269496917725 (0.23386966802325904) | > loss_disc_real_3: 0.20439265668392181 (0.23453395682222702) | > loss_disc_real_4: 0.20609258115291595 (0.2393688425129535) | > loss_disc_real_5: 0.22952665388584137 (0.24449900610774172) | > loss_0: 2.6794204711914062 (2.687228408514284) | > grad_norm_0: tensor(50.0572, device='cuda:0') (tensor(39.3182, device='cuda:0')) | > loss_gen: 2.122586250305176 (1.9325583308350807) | > loss_kl: 1.8478734493255615 (2.0697374858108217) | > loss_feat: 3.20884108543396 (3.100635327544867) | > loss_mel: 16.435548782348633 (17.321144384496357) | > loss_duration: 1.6151262521743774 (1.5783641057855942) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.2299747467041 (26.002439760694315) | > grad_norm_1: tensor(2773.2046, device='cuda:0') (tensor(2558.9565, device='cuda:0')) | > current_lr_0: 0.00018884730534750609 | > current_lr_1: 0.00018884730534750609 | > step_time: 0.4456 (0.4425303655512193) | > loader_time: 0.0089 (0.007781183018403894) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629738445013342 (+1.678668277364445e-05) | > avg_step_time: 0.45457647887753766 (-0.0009449737172731454) | > avg_loss_disc: 2.7105974378720132 (-0.01748523913638822) | > avg_loss_disc_real_0: 0.17553094370474273 (-0.012266406028623361) | > avg_loss_disc_real_1: 0.22596976496803928 (-0.013721751075395416) | > avg_loss_disc_real_2: 0.23762075707946026 (+0.00404181866578654) | > avg_loss_disc_real_3: 0.2374458306692016 (-0.0006820148145648686) | > avg_loss_disc_real_4: 0.24148163115474539 (+0.0001384099604378186) | > avg_loss_disc_real_5: 0.24474377552388418 (-0.00400245756330625) | > avg_loss_0: 2.7105974378720132 (-0.01748523913638822) | > avg_grad_norm_0: tensor(35.5981, device='cuda:0') (+tensor(4.0776, device='cuda:0')) | > avg_loss_gen: 1.9184484582551768 (-0.04068716982720599) | > avg_loss_kl: 2.1269177148039913 (-0.08764988100024906) | > avg_loss_feat: 3.0211898676106625 (-0.06314091951074774) | > avg_loss_mel: 17.457606664845645 (-0.4611306660611909) | > avg_loss_duration: 1.5938475064828361 (-0.014325941112679486) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.118010292590505 (-0.6669345372159725) | > avg_grad_norm_1: tensor(2208.0127, device='cuda:0') (+tensor(471.7682, device='cuda:0'))  > EPOCH: 460/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:56:08)   --> TIME: 2025-02-25 12:56:23 -- STEP: 29/72 -- GLOBAL_STEP: 1033150 | > loss_disc: 2.646904468536377 (2.7230073665750436) | > loss_disc_real_0: 0.09002930670976639 (0.17282353007587895) | > loss_disc_real_1: 0.2514033317565918 (0.23052008013273106) | > loss_disc_real_2: 0.22989818453788757 (0.23848499049400462) | > loss_disc_real_3: 0.22654573619365692 (0.24108944113912253) | > loss_disc_real_4: 0.26399725675582886 (0.24901165427832767) | > loss_disc_real_5: 0.27223631739616394 (0.24753651999194046) | > loss_0: 2.646904468536377 (2.7230073665750436) | > grad_norm_0: tensor(19.9202, device='cuda:0') (tensor(31.3546, device='cuda:0')) | > loss_gen: 2.2078778743743896 (1.9977726155313953) | > loss_kl: 2.481975555419922 (2.2375053825049553) | > loss_feat: 3.820972442626953 (3.2678587683315934) | > loss_mel: 20.685230255126953 (18.41903305053711) | > loss_duration: 1.591355562210083 (1.5877187375364632) | > amp_scaler: 32.0 (32.0) | > loss_1: 30.787412643432617 (27.509888616101495) | > grad_norm_1: tensor(1376.8683, device='cuda:0') (tensor(2510.7300, device='cuda:0')) | > current_lr_0: 0.00018882369943433764 | > current_lr_1: 0.00018882369943433764 | > step_time: 0.4394 (0.43507245491290913) | > loader_time: 0.0074 (0.007143990746859847) --> EVAL PERFORMANCE | > avg_loader_time: 0.008919652079192686 (+0.00028991363417934374) | > avg_step_time: 0.4598505094017781 (+0.005274030524240414) | > avg_loss_disc: 2.6954743156970387 (-0.01512312217497458) | > avg_loss_disc_real_0: 0.17452206284227506 (-0.0010088808624676704) | > avg_loss_disc_real_1: 0.22473817444603209 (-0.001231590522007192) | > avg_loss_disc_real_2: 0.23242061175930667 (-0.0052001453201535874) | > avg_loss_disc_real_3: 0.23701372406852078 (-0.00043210660068082607) | > avg_loss_disc_real_4: 0.245811735870133 (+0.0043301047153876115) | > avg_loss_disc_real_5: 0.24770008657180087 (+0.0029563110479166876) | > avg_loss_0: 2.6954743156970387 (-0.01512312217497458) | > avg_grad_norm_0: tensor(38.4226, device='cuda:0') (+tensor(2.8245, device='cuda:0')) | > avg_loss_gen: 1.9925794131319288 (+0.07413095487675192) | > avg_loss_kl: 2.2042963907752235 (+0.07737867597123227) | > avg_loss_feat: 3.2209162644937006 (+0.19972639688303806) | > avg_loss_mel: 18.0054792484767 (+0.5478725836310545) | > avg_loss_duration: 1.6090732561030856 (+0.015225749620249474) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 27.032344549474583 (+0.9143342568840787) | > avg_grad_norm_1: tensor(2933.1121, device='cuda:0') (+tensor(725.0994, device='cuda:0'))  > EPOCH: 461/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:56:45)   --> TIME: 2025-02-25 12:56:51 -- STEP: 7/72 -- GLOBAL_STEP: 1033200 | > loss_disc: 2.5401535034179688 (2.6526710305895125) | > loss_disc_real_0: 0.14186230301856995 (0.20953951137406485) | > loss_disc_real_1: 0.20819054543972015 (0.21282048736299788) | > loss_disc_real_2: 0.18429245054721832 (0.19846431698117936) | > loss_disc_real_3: 0.22666746377944946 (0.24039064986365183) | > loss_disc_real_4: 0.2440388947725296 (0.2355112028973443) | > loss_disc_real_5: 0.23138783872127533 (0.23825813829898834) | > loss_0: 2.5401535034179688 (2.6526710305895125) | > grad_norm_0: tensor(30.7792, device='cuda:0') (tensor(42.8597, device='cuda:0')) | > loss_gen: 2.0680623054504395 (1.9428714684077673) | > loss_kl: 1.945768117904663 (2.0229352031435286) | > loss_feat: 3.4467551708221436 (3.1493596008845737) | > loss_mel: 17.726543426513672 (17.49021475655692) | > loss_duration: 1.563102126121521 (1.5700911283493042) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.75023078918457 (26.175471714564733) | > grad_norm_1: tensor(3638.3076, device='cuda:0') (tensor(3098.7534, device='cuda:0')) | > current_lr_0: 0.00018880009647190835 | > current_lr_1: 0.00018880009647190835 | > step_time: 0.4448 (0.4378352165222168) | > loader_time: 0.0057 (0.0063909803118024555)  --> TIME: 2025-02-25 12:57:14 -- STEP: 57/72 -- GLOBAL_STEP: 1033250 | > loss_disc: 2.765613555908203 (2.6883224060660913) | > loss_disc_real_0: 0.25762149691581726 (0.18432030584989934) | > loss_disc_real_1: 0.20586475729942322 (0.2259810600887265) | > loss_disc_real_2: 0.2038755714893341 (0.23317570775224453) | > loss_disc_real_3: 0.21177659928798676 (0.23793619452861317) | > loss_disc_real_4: 0.2424304336309433 (0.2433749173294034) | > loss_disc_real_5: 0.24487647414207458 (0.24618985909118987) | > loss_0: 2.765613555908203 (2.6883224060660913) | > grad_norm_0: tensor(39.3631, device='cuda:0') (tensor(39.7254, device='cuda:0')) | > loss_gen: 1.808204174041748 (1.9879883067649708) | > loss_kl: 1.948705792427063 (2.058181250304507) | > loss_feat: 3.0846846103668213 (3.129578285049974) | > loss_mel: 16.79351234436035 (17.13878288603666) | > loss_duration: 1.6290459632873535 (1.5793779147298712) | > amp_scaler: 32.0 (32.0) | > loss_1: 25.26415252685547 (25.89390835009123) | > grad_norm_1: tensor(1522.0437, device='cuda:0') (tensor(3030.5439, device='cuda:0')) | > current_lr_0: 0.00018880009647190835 | > current_lr_1: 0.00018880009647190835 | > step_time: 0.4521 (0.4498358400244462) | > loader_time: 0.0096 (0.00799837028771116) --> EVAL PERFORMANCE | > avg_loader_time: 0.00877009646993288 (-0.00014955560925980493) | > avg_step_time: 0.459470486976731 (-0.0003800224250470885) | > avg_loss_disc: 2.6911310209354884 (-0.004343294761550265) | > avg_loss_disc_real_0: 0.1865566028150874 (+0.012034539972812341) | > avg_loss_disc_real_1: 0.22597356770240085 (+0.0012353932563687609) | > avg_loss_disc_real_2: 0.23312866624812006 (+0.0007080544888133922) | > avg_loss_disc_real_3: 0.236710485857977 (-0.0003032382105437881) | > avg_loss_disc_real_4: 0.24205429705095963 (-0.00375743881917337) | > avg_loss_disc_real_5: 0.24646963745775358 (-0.001230449114047294) | > avg_loss_0: 2.6911310209354884 (-0.004343294761550265) | > avg_grad_norm_0: tensor(40.3748, device='cuda:0') (+tensor(1.9521, device='cuda:0')) | > avg_loss_gen: 1.985838000203522 (-0.006741412928406687) | > avg_loss_kl: 2.0807513032161014 (-0.12354508755912219) | > avg_loss_feat: 3.1230306826846697 (-0.09788558180903095) | > avg_loss_mel: 17.13904003358223 (-0.86643921489447) | > avg_loss_duration: 1.5895625302489376 (-0.019510725854148037) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 25.918222239319707 (-1.114122310154876) | > avg_grad_norm_1: tensor(3003.5854, device='cuda:0') (+tensor(70.4734, device='cuda:0'))  > EPOCH: 462/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:57:23)   --> TIME: 2025-02-25 12:57:41 -- STEP: 35/72 -- GLOBAL_STEP: 1033300 | > loss_disc: 2.7037253379821777 (2.6845333780561176) | > loss_disc_real_0: 0.30524742603302 (0.18061615973711015) | > loss_disc_real_1: 0.21614022552967072 (0.2212081477046013) | > loss_disc_real_2: 0.21039561927318573 (0.22953868763787405) | > loss_disc_real_3: 0.2375205010175705 (0.2332064415727343) | > loss_disc_real_4: 0.25091633200645447 (0.24261703618935176) | > loss_disc_real_5: 0.2562112510204315 (0.24875899382999966) | > loss_0: 2.7037253379821777 (2.6845333780561176) | > grad_norm_0: tensor(40.3110, device='cuda:0') (tensor(37.5606, device='cuda:0')) | > loss_gen: 2.20198655128479 (1.9872031654630389) | > loss_kl: 2.0371286869049072 (2.0051301343100403) | > loss_feat: 3.2510173320770264 (3.1805075781685965) | > loss_mel: 18.329233169555664 (17.172799055916922) | > loss_duration: 1.6038339138031006 (1.5533521277563913) | > amp_scaler: 32.0 (32.0) | > loss_1: 27.423198699951172 (25.898991884504046) | > grad_norm_1: tensor(3898.4766, device='cuda:0') (tensor(2920.7322, device='cuda:0')) | > current_lr_0: 0.00018877649645984934 | > current_lr_1: 0.00018877649645984934 | > step_time: 0.4447 (0.4431152275630406) | > loader_time: 0.0085 (0.007190166200910296) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630114541926857 (-0.00013998192800602355) | > avg_step_time: 0.460532416760082 (+0.0010619297833510122) | > avg_loss_disc: 2.690965286442932 (-0.00016573449255652406) | > avg_loss_disc_real_0: 0.17530673931182272 (-0.011249863503264679) | > avg_loss_disc_real_1: 0.2272586773189021 (+0.0012851096165012454) | > avg_loss_disc_real_2: 0.23048259275899807 (-0.0026460734891219917) | > avg_loss_disc_real_3: 0.2349641971604925 (-0.0017462886974844993) | > avg_loss_disc_real_4: 0.24268666623343885 (+0.0006323691824792199) | > avg_loss_disc_real_5: 0.2488241991106893 (+0.0023545616529357105) | > avg_loss_0: 2.690965286442932 (-0.00016573449255652406) | > avg_grad_norm_0: tensor(40.1685, device='cuda:0') (tensor(-0.2063, device='cuda:0')) | > avg_loss_gen: 1.9681607283337015 (-0.0176772718698206) | > avg_loss_kl: 2.0789542735462447 (-0.0017970296698566202) | > avg_loss_feat: 3.1499258195850213 (+0.026895136900351613) | > avg_loss_mel: 17.279140324659743 (+0.14010029107751265) | > avg_loss_duration: 1.5880233838524616 (-0.00153914639647601) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.06420441748391 (+0.14598217816420345) | > avg_grad_norm_1: tensor(2947.5869, device='cuda:0') (tensor(-55.9985, device='cuda:0'))  > EPOCH: 463/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:58:00)   --> TIME: 2025-02-25 12:58:08 -- STEP: 13/72 -- GLOBAL_STEP: 1033350 | > loss_disc: 2.6602249145507812 (2.6530170807471642) | > loss_disc_real_0: 0.1419149935245514 (0.1674711268681746) | > loss_disc_real_1: 0.1982155442237854 (0.22166333060998183) | > loss_disc_real_2: 0.204558327794075 (0.22936665094815767) | > loss_disc_real_3: 0.20967945456504822 (0.23126499698712275) | > loss_disc_real_4: 0.2331438809633255 (0.2401613570176638) | > loss_disc_real_5: 0.24913328886032104 (0.24363989325670096) | > loss_0: 2.6602249145507812 (2.6530170807471642) | > grad_norm_0: tensor(21.8901, device='cuda:0') (tensor(37.3314, device='cuda:0')) | > loss_gen: 1.9197386503219604 (1.992061550800617) | > loss_kl: 2.090148448944092 (1.9882464959071233) | > loss_feat: 3.493443489074707 (3.2239354573763332) | > loss_mel: 17.336729049682617 (17.0420397244967) | > loss_duration: 1.4802114963531494 (1.548734949185298) | > amp_scaler: 32.0 (32.0) | > loss_1: 26.320270538330078 (25.795018709622898) | > grad_norm_1: tensor(2587.7129, device='cuda:0') (tensor(3003.2344, device='cuda:0')) | > current_lr_0: 0.00018875289939779185 | > current_lr_1: 0.00018875289939779185 | > step_time: 0.4532 (0.4350762734046349) | > loader_time: 0.0061 (0.006803530913132887)  --> TIME: 2025-02-25 12:58:32 -- STEP: 63/72 -- GLOBAL_STEP: 1033400 | > loss_disc: 2.797609806060791 (2.6791866317627924) | > loss_disc_real_0: 0.22142760455608368 (0.17534895063865752) | > loss_disc_real_1: 0.1928284615278244 (0.2229891535308626) | > loss_disc_real_2: 0.23625408113002777 (0.23261820513104636) | > loss_disc_real_3: 0.2072308510541916 (0.2353754322680216) | > loss_disc_real_4: 0.2265750765800476 (0.24147385760905252) | > loss_disc_real_5: 0.24410322308540344 (0.24528638356261784) | > loss_0: 2.797609806060791 (2.6791866317627924) | > grad_norm_0: tensor(38.2063, device='cuda:0') (tensor(42.6855, device='cuda:0')) | > loss_gen: 1.857790470123291 (1.989815971207997) | > loss_kl: 2.093576431274414 (2.058985010025993) | > loss_feat: 2.6513662338256836 (3.220033607785664) | > loss_mel: 16.383676528930664 (17.10870888119652) | > loss_duration: 1.5939280986785889 (1.5782429131250533) | > amp_scaler: 32.0 (32.0) | > loss_1: 24.580337524414062 (25.955786599053276) | > grad_norm_1: tensor(2094.7495, device='cuda:0') (tensor(3246.3594, device='cuda:0')) | > current_lr_0: 0.00018875289939779185 | > current_lr_1: 0.00018875289939779185 | > step_time: 0.4934 (0.4555196232265896) | > loader_time: 0.0116 (0.00825975811670697) --> EVAL PERFORMANCE | > avg_loader_time: 0.008674171608938297 (+4.4057067011439924e-05) | > avg_step_time: 0.46245484284951655 (+0.001922426089434548) | > avg_loss_disc: 2.6920490231312497 (+0.001083736688317849) | > avg_loss_disc_real_0: 0.1785067478326005 (+0.003200008520777786) | > avg_loss_disc_real_1: 0.22371810128991032 (-0.003540576028991771) | > avg_loss_disc_real_2: 0.23433471775390732 (+0.0038521249949092506) | > avg_loss_disc_real_3: 0.2354786601704611 (+0.0005144630099686021) | > avg_loss_disc_real_4: 0.24119391659615744 (-0.0014927496372814097) | > avg_loss_disc_real_5: 0.24537622487880814 (-0.0034479742318811457) | > avg_loss_0: 2.6920490231312497 (+0.001083736688317849) | > avg_grad_norm_0: tensor(40.7575, device='cuda:0') (+tensor(0.5890, device='cuda:0')) | > avg_loss_gen: 1.9743719268852555 (+0.0062111985515540535) | > avg_loss_kl: 2.0846678404740886 (+0.0057135669278438606) | > avg_loss_feat: 3.178579582294948 (+0.02865376270992659) | > avg_loss_mel: 17.199434898269008 (-0.07970542639073486) | > avg_loss_duration: 1.5878080012093128 (-0.00021538264314879818) | > avg_amp_scaler: 32.0 (+0.0) | > avg_loss_1: 26.024862450613103 (-0.03934196687080771) | > avg_grad_norm_1: tensor(3050.9194, device='cuda:0') (+tensor(103.3325, device='cuda:0'))  > EPOCH: 464/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:58:38)   --> TIME: 2025-02-25 12:58:59 -- STEP: 41/72 -- GLOBAL_STEP: 1033450 | > loss_disc: 2.5199999809265137 (2.7383772512761557) | > loss_disc_real_0: 0.17076683044433594 (0.19536695429464665) | > loss_disc_real_1: 0.2074883133172989 (0.2306890903813083) | > loss_disc_real_2: 0.16474434733390808 (0.23180832077817218) | > loss_disc_real_3: 0.2137341946363449 (0.24058259151330808) | > loss_disc_real_4: 0.24780848622322083 (0.24589152488766647) | > loss_disc_real_5: 0.25959673523902893 (0.24540259634576192) | > loss_0: 2.5199999809265137 (2.7383772512761557) | > grad_norm_0: tensor(27.4826, device='cuda:0') (tensor(44.5644, device='cuda:0')) | > loss_gen: 2.026348114013672 (2.0143855141430365) | > loss_kl: 2.3719565868377686 (2.7164031790523993) | > loss_feat: 3.6076059341430664 (3.278290899788461) | > loss_mel: 19.226329803466797 (19.63416820619165) | > loss_duration: 1.6029856204986572 (1.6534902787790067) | > amp_scaler: 32.0 (32.0) | > loss_1: 28.83522605895996 (29.296738182626118) | > grad_norm_1: tensor(3425.3875, device='cuda:0') (tensor(3015.4465, device='cuda:0')) | > current_lr_0: 0.0001887293052853671 | > current_lr_1: 0.0001887293052853671 | > step_time: 0.4649 (0.4445128906063917) | > loader_time: 0.009 (0.007438624777444979) --> EVAL PERFORMANCE | > avg_loader_time: 0.00883225655891526 (+0.00015808494997696272) | > avg_step_time: 0.45897873690430546 (-0.003476105945211083) | > avg_loss_disc: 2.7543479388868306 (+0.062298915755580886) | > avg_loss_disc_real_0: 0.18980399681858615 (+0.011297248985985636) | > avg_loss_disc_real_1: 0.23081129464045377 (+0.007093193350543453) | > avg_loss_disc_real_2: 0.23638410530459714 (+0.002049387550689813) | > avg_loss_disc_real_3: 0.2405062777475572 (+0.005027617577096105) | > avg_loss_disc_real_4: 0.24973325851097913 (+0.008539341914821696) | > avg_loss_disc_real_5: 0.2517368195342346 (+0.00636059465542646) | > avg_loss_0: 2.7543479388868306 (+0.062298915755580886) | > avg_grad_norm_0: tensor(42.5947, device='cuda:0') (+tensor(1.8372, device='cuda:0')) | > avg_loss_gen: 2.0173549165188427 (+0.042982989633587154) | > avg_loss_kl: 2.842598859692963 (+0.7579310192188746) | > avg_loss_feat: 3.2451429434225587 (+0.06656336112761085) | > avg_loss_mel: 19.99339812910053 (+2.7939632308315225) | > avg_loss_duration: 1.6654255927448542 (+0.07761759153554149) | > avg_amp_scaler: 27.943661971830984 (-4.056338028169016) | > avg_loss_1: 29.763920394467636 (+3.739057943854533) | > avg_grad_norm_1: tensor(3023.9578, device='cuda:0') (tensor(-26.9617, device='cuda:0'))  > EPOCH: 465/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:59:15)   --> TIME: 2025-02-25 12:59:25 -- STEP: 19/72 -- GLOBAL_STEP: 1033500 | > loss_disc: 2.7642369270324707 (2.7795332732953524) | > loss_disc_real_0: 0.3765782415866852 (0.22387952553598503) | > loss_disc_real_1: 0.2654779851436615 (0.22462540472808637) | > loss_disc_real_2: 0.25677168369293213 (0.24483060366228998) | > loss_disc_real_3: 0.24864792823791504 (0.24153114384726473) | > loss_disc_real_4: 0.22437073290348053 (0.2403881981184608) | > loss_disc_real_5: 0.24902960658073425 (0.2393150933479008) | > loss_0: 2.7642369270324707 (2.7795332732953524) | > grad_norm_0: tensor(49.6444, device='cuda:0') (tensor(32.3034, device='cuda:0')) | > loss_gen: 1.9854073524475098 (1.9080526640540676) | > loss_kl: 2.832395553588867 (2.564989403674477) | > loss_feat: 2.6501898765563965 (2.88137065736871) | > loss_mel: 18.80064582824707 (19.15885895176938) | > loss_duration: 1.6259117126464844 (1.6080825266085172) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.894550323486328 (28.121354253668535) | > grad_norm_1: tensor(1337.7047, device='cuda:0') (tensor(1119.1796, device='cuda:0')) | > current_lr_0: 0.00018870571412220644 | > current_lr_1: 0.00018870571412220644 | > step_time: 0.4297 (0.4341194880636115) | > loader_time: 0.0063 (0.0068485611363461145)  --> TIME: 2025-02-25 12:59:49 -- STEP: 69/72 -- GLOBAL_STEP: 1033550 | > loss_disc: 2.7939655780792236 (2.6911719156348184) | > loss_disc_real_0: 0.053157832473516464 (0.17120575429736704) | > loss_disc_real_1: 0.14491945505142212 (0.2248223940099495) | > loss_disc_real_2: 0.26983481645584106 (0.2348315363776856) | > loss_disc_real_3: 0.284607857465744 (0.23623444701450458) | > loss_disc_real_4: 0.2208549976348877 (0.23873839106248773) | > loss_disc_real_5: 0.20702789723873138 (0.24454071081202963) | > loss_0: 2.7939655780792236 (2.6911719156348184) | > grad_norm_0: tensor(90.2921, device='cuda:0') (tensor(38.2403, device='cuda:0')) | > loss_gen: 1.7056206464767456 (2.0136107137237778) | > loss_kl: 2.41074275970459 (2.539781041767286) | > loss_feat: 3.3988237380981445 (3.2269958102184795) | > loss_mel: 18.93935203552246 (19.38091402468474) | > loss_duration: 1.7033288478851318 (1.6441333760385928) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.157867431640625 (28.80543507009313) | > grad_norm_1: tensor(3049.3545, device='cuda:0') (tensor(2575.4387, device='cuda:0')) | > current_lr_0: 0.00018870571412220644 | > current_lr_1: 0.00018870571412220644 | > step_time: 0.5272 (0.4541451620019) | > loader_time: 0.0124 (0.008813350096992825) --> EVAL PERFORMANCE | > avg_loader_time: 0.008863170382002709 (+3.091382308744936e-05) | > avg_step_time: 0.4555962858065753 (-0.003382451097730166) | > avg_loss_disc: 2.6938027899030224 (-0.06054514898380825) | > avg_loss_disc_real_0: 0.1698502843648615 (-0.019953712453724654) | > avg_loss_disc_real_1: 0.2229722072960625 (-0.00783908734439126) | > avg_loss_disc_real_2: 0.23595461186388844 (-0.0004294934407086948) | > avg_loss_disc_real_3: 0.23730586511148533 (-0.0032004126360718754) | > avg_loss_disc_real_4: 0.23857453087685812 (-0.011158727634121013) | > avg_loss_disc_real_5: 0.24431294181817015 (-0.007423877716064453) | > avg_loss_0: 2.6938027899030224 (-0.06054514898380825) | > avg_grad_norm_0: tensor(38.2918, device='cuda:0') (tensor(-4.3029, device='cuda:0')) | > avg_loss_gen: 2.012236472586512 (-0.005118443932330852) | > avg_loss_kl: 2.534129844584935 (-0.30846901510802827) | > avg_loss_feat: 3.221819289973084 (-0.023323653449474513) | > avg_loss_mel: 19.35781629320601 (-0.63558183589452) | > avg_loss_duration: 1.6434635696276811 (-0.021962023117173102) | > avg_amp_scaler: 16.0 (-11.943661971830984) | > avg_loss_1: 28.769465567360463 (-0.9944548271071731) | > avg_grad_norm_1: tensor(2567.4214, device='cuda:0') (tensor(-456.5364, device='cuda:0'))  > EPOCH: 466/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 12:59:52)   --> TIME: 2025-02-25 13:00:15 -- STEP: 47/72 -- GLOBAL_STEP: 1033600 | > loss_disc: 2.5360074043273926 (2.7212215281547376) | > loss_disc_real_0: 0.10992535948753357 (0.18008461594581604) | > loss_disc_real_1: 0.23915201425552368 (0.23349527507386308) | > loss_disc_real_2: 0.22063063085079193 (0.23578942900008343) | > loss_disc_real_3: 0.22414736449718475 (0.2423570431927417) | > loss_disc_real_4: 0.23594911396503448 (0.2501814165647994) | > loss_disc_real_5: 0.22741182148456573 (0.2477574583063735) | > loss_0: 2.5360074043273926 (2.7212215281547376) | > grad_norm_0: tensor(35.4232, device='cuda:0') (tensor(29.2009, device='cuda:0')) | > loss_gen: 2.1747541427612305 (1.9920197776023383) | > loss_kl: 2.083218812942505 (2.5598239695772205) | > loss_feat: 3.4972336292266846 (3.1715116805218635) | > loss_mel: 18.582515716552734 (19.63004773728391) | > loss_duration: 1.6456043720245361 (1.6454221517481702) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.983327865600586 (28.99882543847916) | > grad_norm_1: tensor(4134.2759, device='cuda:0') (tensor(2342.3662, device='cuda:0')) | > current_lr_0: 0.00018868212590794116 | > current_lr_1: 0.00018868212590794116 | > step_time: 0.4506 (0.4426927109982105) | > loader_time: 0.009 (0.007646621541774019) --> EVAL PERFORMANCE | > avg_loader_time: 0.008649819333788375 (-0.00021335104821433408) | > avg_step_time: 0.45633119596561916 (+0.0007349101590438667) | > avg_loss_disc: 2.7268465270458804 (+0.03304373714285802) | > avg_loss_disc_real_0: 0.18584237919307092 (+0.01599209482820943) | > avg_loss_disc_real_1: 0.22939418424183214 (+0.006421976945769625) | > avg_loss_disc_real_2: 0.23656707183575967 (+0.000612459971871232) | > avg_loss_disc_real_3: 0.24035804666263957 (+0.0030521815511542483) | > avg_loss_disc_real_4: 0.24761873525632938 (+0.00904420437947126) | > avg_loss_disc_real_5: 0.24748072342973365 (+0.0031677816115635027) | > avg_loss_0: 2.7268465270458804 (+0.03304373714285802) | > avg_grad_norm_0: tensor(30.8178, device='cuda:0') (tensor(-7.4740, device='cuda:0')) | > avg_loss_gen: 1.9716631929639359 (-0.04057327962257595) | > avg_loss_kl: 2.4446119187583384 (-0.08951792582659657) | > avg_loss_feat: 3.101247787475586 (-0.12057150249749826) | > avg_loss_mel: 19.07494925109434 (-0.2828670421116719) | > avg_loss_duration: 1.6463180377449786 (+0.002854468117297504) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.238790162852112 (-0.5306754045083508) | > avg_grad_norm_1: tensor(2261.1770, device='cuda:0') (tensor(-306.2444, device='cuda:0'))  > EPOCH: 467/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:00:29)   --> TIME: 2025-02-25 13:00:43 -- STEP: 25/72 -- GLOBAL_STEP: 1033650 | > loss_disc: 2.680234670639038 (2.7255826950073248) | > loss_disc_real_0: 0.27064526081085205 (0.19254853695631027) | > loss_disc_real_1: 0.19079968333244324 (0.23626519054174422) | > loss_disc_real_2: 0.20641043782234192 (0.2308662438392639) | > loss_disc_real_3: 0.18290013074874878 (0.23033299267292023) | > loss_disc_real_4: 0.2033875733613968 (0.237656392455101) | > loss_disc_real_5: 0.20090799033641815 (0.24114384472370148) | > loss_0: 2.680234670639038 (2.7255826950073248) | > grad_norm_0: tensor(20.8705, device='cuda:0') (tensor(26.3423, device='cuda:0')) | > loss_gen: 1.8002104759216309 (1.9469951438903808) | > loss_kl: 2.30478835105896 (2.1462802839279167) | > loss_feat: 3.2435414791107178 (3.1851183128356935) | > loss_mel: 19.222461700439453 (18.640632705688475) | > loss_duration: 1.5812938213348389 (1.5749952602386474) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.15229606628418 (27.494021759033203) | > grad_norm_1: tensor(1651.0928, device='cuda:0') (tensor(1515.8478, device='cuda:0')) | > current_lr_0: 0.00018865854064220266 | > current_lr_1: 0.00018865854064220266 | > step_time: 0.436 (0.45413076400756835) | > loader_time: 0.0074 (0.006877651214599609) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614200941273862 (-3.561839251451336e-05) | > avg_step_time: 0.4626323572346862 (+0.006301161269067046) | > avg_loss_disc: 2.734627542361407 (+0.007781015315526663) | > avg_loss_disc_real_0: 0.18332894010023332 (-0.002513439092837605) | > avg_loss_disc_real_1: 0.2312183473521555 (+0.0018241631103233513) | > avg_loss_disc_real_2: 0.2395054515398724 (+0.0029383797041127313) | > avg_loss_disc_real_3: 0.23690368570912052 (-0.003454360953519059) | > avg_loss_disc_real_4: 0.2418571735771609 (-0.00576156167916847) | > avg_loss_disc_real_5: 0.24493262087795095 (-0.0025481025517826994) | > avg_loss_0: 2.734627542361407 (+0.007781015315526663) | > avg_grad_norm_0: tensor(24.8570, device='cuda:0') (tensor(-5.9608, device='cuda:0')) | > avg_loss_gen: 1.9177569990426722 (-0.05390619392126372) | > avg_loss_kl: 2.156648432704764 (-0.28796348605357425) | > avg_loss_feat: 3.095414846715793 (-0.005832940759793015) | > avg_loss_mel: 18.14306154385419 (-0.931887707240147) | > avg_loss_duration: 1.6064733266830444 (-0.03984471106193421) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.919355231271663 (-1.3194349315804494) | > avg_grad_norm_1: tensor(1649.0660, device='cuda:0') (tensor(-612.1110, device='cuda:0'))  > EPOCH: 468/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:01:07)   --> TIME: 2025-02-25 13:01:10 -- STEP: 3/72 -- GLOBAL_STEP: 1033700 | > loss_disc: 2.775155782699585 (2.682078202565511) | > loss_disc_real_0: 0.09062223136425018 (0.12784903248151144) | > loss_disc_real_1: 0.198399156332016 (0.2256598820288976) | > loss_disc_real_2: 0.253886878490448 (0.2561410367488861) | > loss_disc_real_3: 0.24653659760951996 (0.2439109832048416) | > loss_disc_real_4: 0.22391252219676971 (0.23042729993661246) | > loss_disc_real_5: 0.22679473459720612 (0.23406517505645752) | > loss_0: 2.775155782699585 (2.682078202565511) | > grad_norm_0: tensor(34.6138, device='cuda:0') (tensor(38.3082, device='cuda:0')) | > loss_gen: 1.802350640296936 (1.9318161408106487) | > loss_kl: 2.2471044063568115 (2.0299319426218667) | > loss_feat: 3.4901037216186523 (3.351023515065511) | > loss_mel: 19.32782554626465 (18.40391794840495) | > loss_duration: 1.6151788234710693 (1.5955942074457805) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.482563018798828 (27.312283198038738) | > grad_norm_1: tensor(1292.0518, device='cuda:0') (tensor(2964.6326, device='cuda:0')) | > current_lr_0: 0.00018863495832462237 | > current_lr_1: 0.00018863495832462237 | > step_time: 0.4334 (0.4391063054402669) | > loader_time: 0.0081 (0.007132053375244141)  --> TIME: 2025-02-25 13:01:33 -- STEP: 53/72 -- GLOBAL_STEP: 1033750 | > loss_disc: 2.5816164016723633 (2.679595173529859) | > loss_disc_real_0: 0.13610117137432098 (0.17096205041655954) | > loss_disc_real_1: 0.22898106276988983 (0.22543775107500688) | > loss_disc_real_2: 0.21450568735599518 (0.23115631572480472) | > loss_disc_real_3: 0.23134039342403412 (0.23357176836931481) | > loss_disc_real_4: 0.23902836441993713 (0.24135303469199054) | > loss_disc_real_5: 0.2429770678281784 (0.2435081570215945) | > loss_0: 2.5816164016723633 (2.679595173529859) | > grad_norm_0: tensor(13.1598, device='cuda:0') (tensor(29.6240, device='cuda:0')) | > loss_gen: 2.1365513801574707 (1.9357366291981823) | > loss_kl: 2.042902946472168 (2.0995891544054137) | > loss_feat: 3.3706870079040527 (3.1574141259463326) | > loss_mel: 16.86294937133789 (17.84946227523515) | > loss_duration: 1.61867356300354 (1.584864465695507) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.031763076782227 (26.627066594249797) | > grad_norm_1: tensor(3469.9841, device='cuda:0') (tensor(2288.5073, device='cuda:0')) | > current_lr_0: 0.00018863495832462237 | > current_lr_1: 0.00018863495832462237 | > step_time: 0.4454 (0.44272523556115495) | > loader_time: 0.0091 (0.007777128579481592) --> EVAL PERFORMANCE | > avg_loader_time: 0.008572299715498803 (-4.190122577505881e-05) | > avg_step_time: 0.4555855099584015 (-0.0070468472762846845) | > avg_loss_disc: 2.6842426582121512 (-0.05038488414925579) | > avg_loss_disc_real_0: 0.17199717650950794 (-0.01133176359072538) | > avg_loss_disc_real_1: 0.22507770254578388 (-0.006140644806371609) | > avg_loss_disc_real_2: 0.230174881261839 (-0.009330570278033412) | > avg_loss_disc_real_3: 0.2338505616070519 (-0.0030531241020686217) | > avg_loss_disc_real_4: 0.2419431744746759 (+8.600089751498174e-05) | > avg_loss_disc_real_5: 0.24394024824592428 (-0.0009923726320266724) | > avg_loss_0: 2.6842426582121512 (-0.05038488414925579) | > avg_grad_norm_0: tensor(30.7077, device='cuda:0') (+tensor(5.8507, device='cuda:0')) | > avg_loss_gen: 1.9318737849383287 (+0.01411678589565657) | > avg_loss_kl: 2.1068627666419664 (-0.04978566606279777) | > avg_loss_feat: 3.15824405240341 (+0.06282920568761696) | > avg_loss_mel: 17.78710337088141 (-0.3559581729727803) | > avg_loss_duration: 1.59750647612021 (-0.008966850562834416) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.58159038382517 (-0.3377648474464934) | > avg_grad_norm_1: tensor(2417.5823, device='cuda:0') (+tensor(768.5162, device='cuda:0'))  > EPOCH: 469/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:01:44)   --> TIME: 2025-02-25 13:01:59 -- STEP: 31/72 -- GLOBAL_STEP: 1033800 | > loss_disc: 2.7180161476135254 (2.689119777371806) | > loss_disc_real_0: 0.12253625690937042 (0.17169980440408952) | > loss_disc_real_1: 0.27622172236442566 (0.22740140076606505) | > loss_disc_real_2: 0.25086650252342224 (0.24214278305730513) | > loss_disc_real_3: 0.3064141571521759 (0.23685899905620084) | > loss_disc_real_4: 0.2597690522670746 (0.24393271342400583) | > loss_disc_real_5: 0.2344711273908615 (0.245133317766651) | > loss_0: 2.7180161476135254 (2.689119777371806) | > grad_norm_0: tensor(24.3829, device='cuda:0') (tensor(29.8021, device='cuda:0')) | > loss_gen: 2.0871825218200684 (1.9784433880159933) | > loss_kl: 2.4627912044525146 (2.5394050651980984) | > loss_feat: 3.346438407897949 (3.304656974730953) | > loss_mel: 19.486194610595703 (18.956817442370998) | > loss_duration: 1.6348105669021606 (1.6152196122754006) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.017419815063477 (28.394542817146547) | > grad_norm_1: tensor(1227.7142, device='cuda:0') (tensor(1887.8735, device='cuda:0')) | > current_lr_0: 0.00018861137895483178 | > current_lr_1: 0.00018861137895483178 | > step_time: 0.4427 (0.4355534968837615) | > loader_time: 0.0092 (0.0071407210442327685) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869954807657591 (+0.0001272483610771074) | > avg_step_time: 0.45542196152915415 (-0.00016354842924737323) | > avg_loss_disc: 2.7336575984954834 (+0.049414940283332154) | > avg_loss_disc_real_0: 0.1826482485507576 (+0.010651072041249654) | > avg_loss_disc_real_1: 0.2354875966696672 (+0.010409894123883323) | > avg_loss_disc_real_2: 0.24064989199101086 (+0.010475010729171869) | > avg_loss_disc_real_3: 0.24314821144224893 (+0.009297649835197042) | > avg_loss_disc_real_4: 0.2462099320451978 (+0.004266757570521917) | > avg_loss_disc_real_5: 0.2508220070264708 (+0.006881758780546504) | > avg_loss_0: 2.7336575984954834 (+0.049414940283332154) | > avg_grad_norm_0: tensor(32.6315, device='cuda:0') (+tensor(1.9238, device='cuda:0')) | > avg_loss_gen: 2.003183232226842 (+0.0713094472885134) | > avg_loss_kl: 2.5111717590143985 (+0.4043089923724321) | > avg_loss_feat: 3.2467558484681893 (+0.08851179606477944) | > avg_loss_mel: 18.98820834092691 (+1.201104970045499) | > avg_loss_duration: 1.631126106624872 (+0.03361963050466188) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.380445372890417 (+1.7988549890652479) | > avg_grad_norm_1: tensor(2028.9760, device='cuda:0') (tensor(-388.6063, device='cuda:0'))  > EPOCH: 470/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:02:20)   --> TIME: 2025-02-25 13:02:26 -- STEP: 9/72 -- GLOBAL_STEP: 1033850 | > loss_disc: 2.720308542251587 (2.6796051131354437) | > loss_disc_real_0: 0.1630185842514038 (0.2049946983655294) | > loss_disc_real_1: 0.25020578503608704 (0.22332017785973018) | > loss_disc_real_2: 0.209601491689682 (0.21008545656998953) | > loss_disc_real_3: 0.2796308100223541 (0.2331557720899582) | > loss_disc_real_4: 0.2387978583574295 (0.22965063485834333) | > loss_disc_real_5: 0.23047678172588348 (0.23029228217071956) | > loss_0: 2.720308542251587 (2.6796051131354437) | > grad_norm_0: tensor(12.4232, device='cuda:0') (tensor(37.8871, device='cuda:0')) | > loss_gen: 1.8146193027496338 (1.9900919993718464) | > loss_kl: 2.012881278991699 (2.153593831592136) | > loss_feat: 3.133126974105835 (3.2131153212653265) | > loss_mel: 19.861427307128906 (18.89351929558648) | > loss_duration: 1.6008639335632324 (1.5905740393532648) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.42291831970215 (27.84089512295193) | > grad_norm_1: tensor(1517.5437, device='cuda:0') (tensor(2707.3369, device='cuda:0')) | > current_lr_0: 0.0001885878025324624 | > current_lr_1: 0.0001885878025324624 | > step_time: 0.4264 (0.43049703703986275) | > loader_time: 0.0065 (0.006625837749905056)  --> TIME: 2025-02-25 13:02:50 -- STEP: 59/72 -- GLOBAL_STEP: 1033900 | > loss_disc: 2.6847565174102783 (2.686725402282457) | > loss_disc_real_0: 0.20728284120559692 (0.17292383717278303) | > loss_disc_real_1: 0.1634775847196579 (0.22746074376469952) | > loss_disc_real_2: 0.29249754548072815 (0.23248872868085313) | > loss_disc_real_3: 0.23571939766407013 (0.2347389752581968) | > loss_disc_real_4: 0.23580606281757355 (0.24098037309565787) | > loss_disc_real_5: 0.2537789046764374 (0.2427560128902985) | > loss_0: 2.6847565174102783 (2.686725402282457) | > grad_norm_0: tensor(8.3433, device='cuda:0') (tensor(33.9121, device='cuda:0')) | > loss_gen: 1.8712971210479736 (1.986582273143833) | > loss_kl: 2.1404685974121094 (2.21470896268295) | > loss_feat: 3.062653064727783 (3.2209573963941156) | > loss_mel: 17.150653839111328 (18.23956085463701) | > loss_duration: 1.630040168762207 (1.598043272050761) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.855113983154297 (27.25985303975768) | > grad_norm_1: tensor(445.4003, device='cuda:0') (tensor(2657.1150, device='cuda:0')) | > current_lr_0: 0.0001885878025324624 | > current_lr_1: 0.0001885878025324624 | > step_time: 0.4709 (0.4534022201926021) | > loader_time: 0.0102 (0.008131867748195841) --> EVAL PERFORMANCE | > avg_loader_time: 0.008736462660238777 (+3.691458366286704e-05) | > avg_step_time: 0.4596226383262957 (+0.004200676797141534) | > avg_loss_disc: 2.7011533723750594 (-0.03250422612042403) | > avg_loss_disc_real_0: 0.17443907103488143 (-0.00820917751587616) | > avg_loss_disc_real_1: 0.23061015068645208 (-0.0048774459832151285) | > avg_loss_disc_real_2: 0.234529939545712 (-0.006119952445298865) | > avg_loss_disc_real_3: 0.23715610802173615 (-0.00599210342051279) | > avg_loss_disc_real_4: 0.2410024064527431 (-0.005207525592454715) | > avg_loss_disc_real_5: 0.24339178561324804 (-0.007430221413222748) | > avg_loss_0: 2.7011533723750594 (-0.03250422612042403) | > avg_grad_norm_0: tensor(34.1883, device='cuda:0') (+tensor(1.5568, device='cuda:0')) | > avg_loss_gen: 1.9862618429560057 (-0.016921389270836418) | > avg_loss_kl: 2.223159924359389 (-0.28801183465500957) | > avg_loss_feat: 3.1890931431676295 (-0.057662705300559836) | > avg_loss_mel: 18.23189461399131 (-0.7563137269355984) | > avg_loss_duration: 1.6060920651529877 (-0.02503404147188415) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.236501854909978 (-1.1439435179804391) | > avg_grad_norm_1: tensor(2693.2744, device='cuda:0') (+tensor(664.2985, device='cuda:0'))  > EPOCH: 471/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:02:58)   --> TIME: 2025-02-25 13:03:16 -- STEP: 37/72 -- GLOBAL_STEP: 1033950 | > loss_disc: 2.701188087463379 (2.7203282601124537) | > loss_disc_real_0: 0.14491689205169678 (0.17242286897994377) | > loss_disc_real_1: 0.18934018909931183 (0.24361780789252874) | > loss_disc_real_2: 0.2043883502483368 (0.23138845611262965) | > loss_disc_real_3: 0.21163585782051086 (0.23770649610338984) | > loss_disc_real_4: 0.25293707847595215 (0.24302988479266296) | > loss_disc_real_5: 0.26516982913017273 (0.2513839074888745) | > loss_0: 2.701188087463379 (2.7203282601124537) | > grad_norm_0: tensor(17.6487, device='cuda:0') (tensor(26.3089, device='cuda:0')) | > loss_gen: 1.9568841457366943 (1.9635380248765688) | > loss_kl: 2.1679978370666504 (2.1750570406784884) | > loss_feat: 2.7425339221954346 (3.1699425465351827) | > loss_mel: 18.24968910217285 (18.83621236440298) | > loss_duration: 1.6147778034210205 (1.6017188542598002) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.731884002685547 (27.74646913683092) | > grad_norm_1: tensor(2672.6443, device='cuda:0') (tensor(2050.1946, device='cuda:0')) | > current_lr_0: 0.00018856422905714584 | > current_lr_1: 0.00018856422905714584 | > step_time: 0.4509 (0.4378740594193742) | > loader_time: 0.0082 (0.007125165011431719) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855131216452155 (-0.00018515049571722793) | > avg_step_time: 0.45665059962742766 (-0.002972038698868029) | > avg_loss_disc: 2.7323352451055825 (+0.031181872730523086) | > avg_loss_disc_real_0: 0.18380498434876053 (+0.009365913313879093) | > avg_loss_disc_real_1: 0.23402061754129302 (+0.003410466854840949) | > avg_loss_disc_real_2: 0.2322028934116095 (-0.002327046134102506) | > avg_loss_disc_real_3: 0.23404592464507465 (-0.003110183376661496) | > avg_loss_disc_real_4: 0.24226412974612813 (+0.0012617232933850309) | > avg_loss_disc_real_5: 0.24552864633815388 (+0.002136860724905848) | > avg_loss_0: 2.7323352451055825 (+0.031181872730523086) | > avg_grad_norm_0: tensor(29.7920, device='cuda:0') (tensor(-4.3963, device='cuda:0')) | > avg_loss_gen: 1.9368140831799574 (-0.04944775977604832) | > avg_loss_kl: 2.174970673843168 (-0.048189250516220916) | > avg_loss_feat: 3.0760340522712384 (-0.1130590908963911) | > avg_loss_mel: 18.22779467408086 (-0.00409993991044999) | > avg_loss_duration: 1.616787527648496 (+0.010695462495508234) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.03240134010852 (-0.20410051480145697) | > avg_grad_norm_1: tensor(2055.9292, device='cuda:0') (tensor(-637.3452, device='cuda:0'))  > EPOCH: 472/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:03:35)   --> TIME: 2025-02-25 13:03:43 -- STEP: 15/72 -- GLOBAL_STEP: 1034000 | > loss_disc: 2.777212619781494 (2.718137280146281) | > loss_disc_real_0: 0.09388155490159988 (0.16598263780275982) | > loss_disc_real_1: 0.1966303139925003 (0.23870172103246054) | > loss_disc_real_2: 0.24814027547836304 (0.2442825625340144) | > loss_disc_real_3: 0.1895613968372345 (0.23867762982845306) | > loss_disc_real_4: 0.2537758946418762 (0.2478082686662674) | > loss_disc_real_5: 0.24969033896923065 (0.2525623877843221) | > loss_0: 2.777212619781494 (2.718137280146281) | > grad_norm_0: tensor(56.5905, device='cuda:0') (tensor(28.2728, device='cuda:0')) | > loss_gen: 1.7338122129440308 (1.959443179766337) | > loss_kl: 1.9181551933288574 (2.0161215305328373) | > loss_feat: 3.574467897415161 (3.1997457186381024) | > loss_mel: 19.02790069580078 (17.81153132120768) | > loss_duration: 1.556321382522583 (1.5522266944249472) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.810657501220703 (26.539068603515624) | > grad_norm_1: tensor(2389.2080, device='cuda:0') (tensor(1880.6750, device='cuda:0')) | > current_lr_0: 0.00018854065852851368 | > current_lr_1: 0.00018854065852851368 | > step_time: 0.4289 (0.43262952168782554) | > loader_time: 0.0059 (0.0065724849700927734)  --> TIME: 2025-02-25 13:04:07 -- STEP: 65/72 -- GLOBAL_STEP: 1034050 | > loss_disc: 2.904768705368042 (2.6955584195943976) | > loss_disc_real_0: 0.06714395433664322 (0.16580263525247574) | > loss_disc_real_1: 0.22824105620384216 (0.2269847491612801) | > loss_disc_real_2: 0.22328227758407593 (0.23430555967184213) | > loss_disc_real_3: 0.2521836757659912 (0.2349698481651453) | > loss_disc_real_4: 0.24392327666282654 (0.2465070481483753) | > loss_disc_real_5: 0.24451152980327606 (0.24937690565219303) | > loss_0: 2.904768705368042 (2.6955584195943976) | > grad_norm_0: tensor(65.9078, device='cuda:0') (tensor(39.0116, device='cuda:0')) | > loss_gen: 1.6367920637130737 (1.963143223982591) | > loss_kl: 2.5648179054260254 (2.109699375812824) | > loss_feat: 2.8385751247406006 (3.2176908309643086) | > loss_mel: 17.883535385131836 (17.82176026564378) | > loss_duration: 1.6436793804168701 (1.5888402792123648) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.567398071289062 (26.701133962777945) | > grad_norm_1: tensor(1623.5380, device='cuda:0') (tensor(2698.5801, device='cuda:0')) | > current_lr_0: 0.00018854065852851368 | > current_lr_1: 0.00018854065852851368 | > step_time: 0.5234 (0.44648854182316705) | > loader_time: 0.0116 (0.00827639286334698) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855987508532027 (+8.562920798720411e-06) | > avg_step_time: 0.4529010443620279 (-0.0037495552653997732) | > avg_loss_disc: 2.7014624971739005 (-0.030872747931681932) | > avg_loss_disc_real_0: 0.16578302500953138 (-0.01802195933922915) | > avg_loss_disc_real_1: 0.2260953897321728 (-0.007925227809120222) | > avg_loss_disc_real_2: 0.23216779215235106 (-3.5101259258429884e-05) | > avg_loss_disc_real_3: 0.23633230509052813 (+0.002286380445453484) | > avg_loss_disc_real_4: 0.24573681304152584 (+0.0034726832953977105) | > avg_loss_disc_real_5: 0.2498048485164912 (+0.004276202178337324) | > avg_loss_0: 2.7014624971739005 (-0.030872747931681932) | > avg_grad_norm_0: tensor(39.6376, device='cuda:0') (+tensor(9.8456, device='cuda:0')) | > avg_loss_gen: 1.960831623681834 (+0.024017540501876544) | > avg_loss_kl: 2.1273200394402085 (-0.04765063440295947) | > avg_loss_feat: 3.213957853720222 (+0.13792380144898342) | > avg_loss_mel: 17.929629661667512 (-0.2981650124133495) | > avg_loss_duration: 1.594814421425403 (-0.02197310622309301) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.82655358650315 (-0.20584775360537222) | > avg_grad_norm_1: tensor(2700.9141, device='cuda:0') (+tensor(644.9849, device='cuda:0'))  > EPOCH: 473/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:04:12)   --> TIME: 2025-02-25 13:04:33 -- STEP: 43/72 -- GLOBAL_STEP: 1034100 | > loss_disc: 2.5049543380737305 (2.667844140252401) | > loss_disc_real_0: 0.13593792915344238 (0.16528736280147416) | > loss_disc_real_1: 0.2520296275615692 (0.2240575784167578) | > loss_disc_real_2: 0.24356122314929962 (0.24530980614728706) | > loss_disc_real_3: 0.23890534043312073 (0.2299386415370675) | > loss_disc_real_4: 0.20877906680107117 (0.23352727259314338) | > loss_disc_real_5: 0.190835639834404 (0.23724790367969248) | > loss_0: 2.5049543380737305 (2.667844140252401) | > grad_norm_0: tensor(33.3848, device='cuda:0') (tensor(38.9332, device='cuda:0')) | > loss_gen: 2.1172571182250977 (2.002828453862389) | > loss_kl: 2.2731103897094727 (2.157064010930616) | > loss_feat: 3.6627955436706543 (3.3288026576818424) | > loss_mel: 17.791889190673828 (18.00887085670649) | > loss_duration: 1.596065878868103 (1.5800281596738235) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.441116333007812 (27.077594180439792) | > grad_norm_1: tensor(5173.9932, device='cuda:0') (tensor(2947.4128, device='cuda:0')) | > current_lr_0: 0.00018851709094619762 | > current_lr_1: 0.00018851709094619762 | > step_time: 0.4457 (0.4384953698446584) | > loader_time: 0.0098 (0.007373892983724904) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556523793180225 (-3.3512921400451895e-06) | > avg_step_time: 0.45248578635739606 (-0.0004152580046318244) | > avg_loss_disc: 2.7118262640187436 (+0.01036376684484308) | > avg_loss_disc_real_0: 0.16824243668939023 (+0.0024594116798588583) | > avg_loss_disc_real_1: 0.22646072758755215 (+0.00036533785537934405) | > avg_loss_disc_real_2: 0.24220649205463032 (+0.010038699902279263) | > avg_loss_disc_real_3: 0.2401835066331944 (+0.0038512015426662716) | > avg_loss_disc_real_4: 0.24220396872137634 (-0.0035328443201494975) | > avg_loss_disc_real_5: 0.24434505151191227 (-0.005459797004578937) | > avg_loss_0: 2.7118262640187436 (+0.01036376684484308) | > avg_grad_norm_0: tensor(40.7317, device='cuda:0') (+tensor(1.0941, device='cuda:0')) | > avg_loss_gen: 1.988963990144326 (+0.028132366462491953) | > avg_loss_kl: 2.242256705190095 (+0.11493666574988648) | > avg_loss_feat: 3.2441821131907718 (+0.030224259470549963) | > avg_loss_mel: 18.734973988062897 (+0.8053443263953852) | > avg_loss_duration: 1.6044226968792121 (+0.009608275453809156) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.814799617713607 (+0.9882460312104584) | > avg_grad_norm_1: tensor(3086.6189, device='cuda:0') (+tensor(385.7048, device='cuda:0'))  > EPOCH: 474/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:04:49)   --> TIME: 2025-02-25 13:05:00 -- STEP: 21/72 -- GLOBAL_STEP: 1034150 | > loss_disc: 2.6593565940856934 (2.7307129360380626) | > loss_disc_real_0: 0.2372298538684845 (0.16669077035926638) | > loss_disc_real_1: 0.1955498456954956 (0.24150790132227398) | > loss_disc_real_2: 0.1873379945755005 (0.2263310785804476) | > loss_disc_real_3: 0.22776921093463898 (0.23578975243227823) | > loss_disc_real_4: 0.21081772446632385 (0.2508920856884547) | > loss_disc_real_5: 0.23457182943820953 (0.25412806158974055) | > loss_0: 2.6593565940856934 (2.7307129360380626) | > grad_norm_0: tensor(26.0037, device='cuda:0') (tensor(33.3941, device='cuda:0')) | > loss_gen: 1.8620997667312622 (1.9780909674508231) | > loss_kl: 2.1338634490966797 (2.480931827000209) | > loss_feat: 3.4832699298858643 (3.3516311872573126) | > loss_mel: 19.586566925048828 (21.996172859555198) | > loss_duration: 1.5797832012176514 (1.583465990566072) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.645584106445312 (31.39029303051176) | > grad_norm_1: tensor(4171.8525, device='cuda:0') (tensor(3013.1423, device='cuda:0')) | > current_lr_0: 0.00018849352630982935 | > current_lr_1: 0.00018849352630982935 | > step_time: 0.4526 (0.4329196157909575) | > loader_time: 0.0064 (0.006543091365269252)  --> TIME: 2025-02-25 13:05:24 -- STEP: 71/72 -- GLOBAL_STEP: 1034200 | > loss_disc: 2.712869167327881 (2.670475063189654) | > loss_disc_real_0: 0.18494001030921936 (0.16354864636357402) | > loss_disc_real_1: 0.21331873536109924 (0.2272018122840935) | > loss_disc_real_2: 0.24096091091632843 (0.22635051608085632) | > loss_disc_real_3: 0.2505791485309601 (0.23499110760823103) | > loss_disc_real_4: 0.21825765073299408 (0.24149184327729992) | > loss_disc_real_5: 0.24688097834587097 (0.24631548767358485) | > loss_0: 2.712869167327881 (2.670475063189654) | > grad_norm_0: tensor(29.4898, device='cuda:0') (tensor(38.5385, device='cuda:0')) | > loss_gen: 1.934650182723999 (1.9625537244366928) | > loss_kl: 2.0773990154266357 (2.2368951968743773) | > loss_feat: 3.0306642055511475 (3.2096676054135176) | > loss_mel: 17.330013275146484 (19.117951836384517) | > loss_duration: 1.6079351902008057 (1.6019801556224553) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.980661392211914 (28.12904852208957) | > grad_norm_1: tensor(3616.1892, device='cuda:0') (tensor(3573.7166, device='cuda:0')) | > current_lr_0: 0.00018849352630982935 | > current_lr_1: 0.00018849352630982935 | > step_time: 0.5166 (0.457378303500968) | > loader_time: 0.0079 (0.008493289141587809) --> EVAL PERFORMANCE | > avg_loader_time: 0.008493289141587809 (-6.3234651592416e-05) | > avg_step_time: 0.457378303500968 (+0.004892517143571917) | > avg_loss_disc: 2.670475063189654 (-0.041351200829089496) | > avg_loss_disc_real_0: 0.16354864636357402 (-0.004693790325816211) | > avg_loss_disc_real_1: 0.2272018122840935 (+0.0007410846965413553) | > avg_loss_disc_real_2: 0.22635051608085632 (-0.015855975973774) | > avg_loss_disc_real_3: 0.23499110760823103 (-0.005192399024963379) | > avg_loss_disc_real_4: 0.24149184327729992 (-0.0007121254440764224) | > avg_loss_disc_real_5: 0.24631548767358485 (+0.001970436161672584) | > avg_loss_0: 2.670475063189654 (-0.041351200829089496) | > avg_grad_norm_0: tensor(38.5385, device='cuda:0') (tensor(-2.1932, device='cuda:0')) | > avg_loss_gen: 1.9625537244366928 (-0.02641026570763305) | > avg_loss_kl: 2.2368951968743773 (-0.005361508315717689) | > avg_loss_feat: 3.2096676054135176 (-0.03451450777725418) | > avg_loss_mel: 19.117951836384517 (+0.3829778483216195) | > avg_loss_duration: 1.6019801556224553 (-0.0024425412567568028) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.12904852208957 (+0.31424890437596176) | > avg_grad_norm_1: tensor(3573.7166, device='cuda:0') (+tensor(487.0977, device='cuda:0'))  > EPOCH: 475/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:05:26)   --> TIME: 2025-02-25 13:05:50 -- STEP: 49/72 -- GLOBAL_STEP: 1034250 | > loss_disc: 2.760529041290283 (2.652058776544065) | > loss_disc_real_0: 0.22037486732006073 (0.16253334025339206) | > loss_disc_real_1: 0.18965639173984528 (0.22257679335925043) | > loss_disc_real_2: 0.27204954624176025 (0.2321741924602158) | > loss_disc_real_3: 0.20243166387081146 (0.23493773170879909) | > loss_disc_real_4: 0.22854731976985931 (0.24253117703661628) | > loss_disc_real_5: 0.23793703317642212 (0.2447241049031822) | > loss_0: 2.760529041290283 (2.652058776544065) | > grad_norm_0: tensor(63.2019, device='cuda:0') (tensor(40.4580, device='cuda:0')) | > loss_gen: 1.8741216659545898 (1.9860853273041394) | > loss_kl: 1.9504623413085938 (2.026404239693466) | > loss_feat: 3.380272150039673 (3.286456127556003) | > loss_mel: 17.730266571044922 (17.421427512655452) | > loss_duration: 1.6458854675292969 (1.5715849350909799) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.581008911132812 (26.291958322330398) | > grad_norm_1: tensor(3209.1077, device='cuda:0') (tensor(3915.2246, device='cuda:0')) | > current_lr_0: 0.00018846996461904062 | > current_lr_1: 0.00018846996461904062 | > step_time: 0.4501 (0.4432424087913669) | > loader_time: 0.0104 (0.00780418940952846) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871683510256485 (+0.00022354596097704134) | > avg_step_time: 0.4553203582763672 (-0.002057945224600788) | > avg_loss_disc: 2.6680668206282063 (-0.002408242561447782) | > avg_loss_disc_real_0: 0.16753540450418497 (+0.0039867581406109465) | > avg_loss_disc_real_1: 0.22179940762654157 (-0.005402404657551929) | > avg_loss_disc_real_2: 0.22971742358845723 (+0.003366907507600908) | > avg_loss_disc_real_3: 0.23496597060855007 (-2.513699968095806e-05) | > avg_loss_disc_real_4: 0.24426698600742178 (+0.00277514273012186) | > avg_loss_disc_real_5: 0.2466135753292433 (+0.00029808765565844664) | > avg_loss_0: 2.6680668206282063 (-0.002408242561447782) | > avg_grad_norm_0: tensor(41.1384, device='cuda:0') (+tensor(2.5998, device='cuda:0')) | > avg_loss_gen: 1.9683454809054521 (+0.00579175646875929) | > avg_loss_kl: 2.0530587216498137 (-0.1838364752245636) | > avg_loss_feat: 3.2276669858207163 (+0.01799938040719873) | > avg_loss_mel: 17.32130832403479 (-1.7966435123497284) | > avg_loss_duration: 1.5881527830177629 (-0.013827372604692467) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.15853239784778 (-1.9705161242417901) | > avg_grad_norm_1: tensor(3940.2544, device='cuda:0') (+tensor(366.5378, device='cuda:0'))  > EPOCH: 476/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:06:03)   --> TIME: 2025-02-25 13:06:17 -- STEP: 27/72 -- GLOBAL_STEP: 1034300 | > loss_disc: 2.7085626125335693 (2.699707843639232) | > loss_disc_real_0: 0.16333959996700287 (0.16985318423421294) | > loss_disc_real_1: 0.2594533860683441 (0.22550170951419407) | > loss_disc_real_2: 0.26622477173805237 (0.2354634580788789) | > loss_disc_real_3: 0.23817826807498932 (0.2364383312287154) | > loss_disc_real_4: 0.23528797924518585 (0.23909450736310747) | > loss_disc_real_5: 0.23662663996219635 (0.24202874192485102) | > loss_0: 2.7085626125335693 (2.699707843639232) | > grad_norm_0: tensor(27.3876, device='cuda:0') (tensor(43.8349, device='cuda:0')) | > loss_gen: 1.9461015462875366 (1.9437696359775685) | > loss_kl: 2.0047364234924316 (2.026703905176233) | > loss_feat: 3.185800075531006 (3.2382283917179815) | > loss_mel: 17.245820999145508 (17.185914993286133) | > loss_duration: 1.5319089889526367 (1.544879092110528) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.91436767578125 (25.939496216950594) | > grad_norm_1: tensor(2137.8513, device='cuda:0') (tensor(3560.4082, device='cuda:0')) | > current_lr_0: 0.00018844640587346325 | > current_lr_1: 0.00018844640587346325 | > step_time: 0.439 (0.4389970832400852) | > loader_time: 0.0086 (0.00720258112306948) --> EVAL PERFORMANCE | > avg_loader_time: 0.008751593844991334 (+3.475874242648419e-05) | > avg_step_time: 0.45725424860564756 (+0.0019338903292803744) | > avg_loss_disc: 2.6987937470556984 (+0.030726926427492085) | > avg_loss_disc_real_0: 0.17309440263140372 (+0.005558998127218745) | > avg_loss_disc_real_1: 0.22442545836240474 (+0.002626050735863167) | > avg_loss_disc_real_2: 0.2354450758913873 (+0.00572765230293007) | > avg_loss_disc_real_3: 0.2377677343680825 (+0.002801763759532422) | > avg_loss_disc_real_4: 0.24202941148213938 (-0.002237574525282393) | > avg_loss_disc_real_5: 0.24463777474954096 (-0.0019758005797023415) | > avg_loss_0: 2.6987937470556984 (+0.030726926427492085) | > avg_grad_norm_0: tensor(44.6467, device='cuda:0') (+tensor(3.5084, device='cuda:0')) | > avg_loss_gen: 1.9443474302829151 (-0.023998050622537015) | > avg_loss_kl: 2.0587848173060896 (+0.005726095656275909) | > avg_loss_feat: 3.1584237897899787 (-0.06924319603073759) | > avg_loss_mel: 17.225618644499445 (-0.09568967953534369) | > avg_loss_duration: 1.5843747300161442 (-0.0037780530016187086) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.971549531103864 (-0.1869828667439144) | > avg_grad_norm_1: tensor(3546.1287, device='cuda:0') (tensor(-394.1257, device='cuda:0'))  > EPOCH: 477/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:06:40)   --> TIME: 2025-02-25 13:06:44 -- STEP: 5/72 -- GLOBAL_STEP: 1034350 | > loss_disc: 2.7040348052978516 (2.7052287101745605) | > loss_disc_real_0: 0.11567521095275879 (0.16773432195186616) | > loss_disc_real_1: 0.24454408884048462 (0.23910679519176484) | > loss_disc_real_2: 0.20421966910362244 (0.22855159938335418) | > loss_disc_real_3: 0.2229568064212799 (0.22900034487247467) | > loss_disc_real_4: 0.24662360548973083 (0.2464209347963333) | > loss_disc_real_5: 0.24289390444755554 (0.23533000648021699) | > loss_0: 2.7040348052978516 (2.7052287101745605) | > grad_norm_0: tensor(35.2545, device='cuda:0') (tensor(38.1926, device='cuda:0')) | > loss_gen: 1.9572343826293945 (1.9301296710968017) | > loss_kl: 1.8931092023849487 (1.9915008544921875) | > loss_feat: 3.4481348991394043 (3.1320606231689454) | > loss_mel: 17.664749145507812 (17.104000091552734) | > loss_duration: 1.5264482498168945 (1.5642336845397948) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.489673614501953 (25.721924591064454) | > grad_norm_1: tensor(3023.4146, device='cuda:0') (tensor(3047.5396, device='cuda:0')) | > current_lr_0: 0.00018842285007272905 | > current_lr_1: 0.00018842285007272905 | > step_time: 0.4383 (0.428834867477417) | > loader_time: 0.0073 (0.006348133087158203)  --> TIME: 2025-02-25 13:07:08 -- STEP: 55/72 -- GLOBAL_STEP: 1034400 | > loss_disc: 2.669299364089966 (2.7008808872916488) | > loss_disc_real_0: 0.15226933360099792 (0.17100200402465737) | > loss_disc_real_1: 0.24131745100021362 (0.22997517694126476) | > loss_disc_real_2: 0.2529541254043579 (0.22957477894696324) | > loss_disc_real_3: 0.21571363508701324 (0.23395209881392393) | > loss_disc_real_4: 0.24883492290973663 (0.24162309982559899) | > loss_disc_real_5: 0.25362953543663025 (0.24409044601700522) | > loss_0: 2.669299364089966 (2.7008808872916488) | > grad_norm_0: tensor(34.7126, device='cuda:0') (tensor(43.8130, device='cuda:0')) | > loss_gen: 1.8959636688232422 (1.9381044366142965) | > loss_kl: 2.142235279083252 (2.077487542412498) | > loss_feat: 3.173718214035034 (3.1642583370208737) | > loss_mel: 17.216053009033203 (17.211564289439814) | > loss_duration: 1.6250061988830566 (1.5748142285780473) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.052974700927734 (25.966228623823685) | > grad_norm_1: tensor(2433.3694, device='cuda:0') (tensor(3335.0173, device='cuda:0')) | > current_lr_0: 0.00018842285007272905 | > current_lr_1: 0.00018842285007272905 | > step_time: 0.4687 (0.4458815574645996) | > loader_time: 0.0095 (0.007842974229292437) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585986956744127 (-0.00016560688824720712) | > avg_step_time: 0.45529339682887976 (-0.0019608517767678024) | > avg_loss_disc: 2.7049156041212488 (+0.006121857065550351) | > avg_loss_disc_real_0: 0.1736912279796433 (+0.0005968253482395713) | > avg_loss_disc_real_1: 0.23031320832145047 (+0.005887749959045729) | > avg_loss_disc_real_2: 0.22983816916673955 (-0.005606906724647753) | > avg_loss_disc_real_3: 0.2353456975288794 (-0.0024220368392030844) | > avg_loss_disc_real_4: 0.2413996243980569 (-0.0006297870840824837) | > avg_loss_disc_real_5: 0.24424775553421235 (-0.0003900192153286075) | > avg_loss_0: 2.7049156041212488 (+0.006121857065550351) | > avg_grad_norm_0: tensor(43.7294, device='cuda:0') (tensor(-0.9173, device='cuda:0')) | > avg_loss_gen: 1.9331198729259866 (-0.011227557356928486) | > avg_loss_kl: 2.101902296845342 (+0.04311747953925238) | > avg_loss_feat: 3.129898360077764 (-0.028525429712214834) | > avg_loss_mel: 17.17691995056583 (-0.048698693933616255) | > avg_loss_duration: 1.5874581756726118 (+0.003083445656467676) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.92929850833517 (-0.042251022768695634) | > avg_grad_norm_1: tensor(3317.1799, device='cuda:0') (tensor(-228.9487, device='cuda:0'))  > EPOCH: 478/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:07:17)   --> TIME: 2025-02-25 13:07:34 -- STEP: 33/72 -- GLOBAL_STEP: 1034450 | > loss_disc: 2.695228338241577 (2.7240609833688447) | > loss_disc_real_0: 0.1775868833065033 (0.16137118095701392) | > loss_disc_real_1: 0.26212045550346375 (0.21679182757030835) | > loss_disc_real_2: 0.23922386765480042 (0.24097005300449598) | > loss_disc_real_3: 0.20565171539783478 (0.23412135469191003) | > loss_disc_real_4: 0.21853791177272797 (0.2397928951364575) | > loss_disc_real_5: 0.2286141812801361 (0.244807528275432) | > loss_0: 2.695228338241577 (2.7240609833688447) | > grad_norm_0: tensor(55.2071, device='cuda:0') (tensor(46.4019, device='cuda:0')) | > loss_gen: 1.944033145904541 (1.8920514186223347) | > loss_kl: 2.119065523147583 (2.099878173885923) | > loss_feat: 3.220957040786743 (3.1737791191447866) | > loss_mel: 17.47399139404297 (17.408295515811812) | > loss_duration: 1.5525593757629395 (1.5553220980095142) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.310606002807617 (26.129326097892992) | > grad_norm_1: tensor(2388.7292, device='cuda:0') (tensor(3242.8511, device='cuda:0')) | > current_lr_0: 0.00018839929721646995 | > current_lr_1: 0.00018839929721646995 | > step_time: 0.4463 (0.4371902581417199) | > loader_time: 0.0079 (0.007261745857469963) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710300418692573 (+0.00012431346194844567) | > avg_step_time: 0.45645617767119073 (+0.0011627808423109731) | > avg_loss_disc: 2.7127952542103504 (+0.00787965008910163) | > avg_loss_disc_real_0: 0.16577353024146924 (-0.007917697738174051) | > avg_loss_disc_real_1: 0.22020873091590237 (-0.010104477405548096) | > avg_loss_disc_real_2: 0.23652703913164805 (+0.0066888699649085) | > avg_loss_disc_real_3: 0.23542446443732357 (+7.876690844416112e-05) | > avg_loss_disc_real_4: 0.24075912883583928 (-0.0006404955622176245) | > avg_loss_disc_real_5: 0.24366565153632366 (-0.0005821039978886888) | > avg_loss_0: 2.7127952542103504 (+0.00787965008910163) | > avg_grad_norm_0: tensor(45.4260, device='cuda:0') (+tensor(1.6965, device='cuda:0')) | > avg_loss_gen: 1.8987728642745756 (-0.03434700865141105) | > avg_loss_kl: 2.1130423126086373 (+0.011140015763295263) | > avg_loss_feat: 3.089192272911609 (-0.04070608716615487) | > avg_loss_mel: 17.208097887710785 (+0.031177937144956758) | > avg_loss_duration: 1.589258378660175 (+0.0018002029875632264) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.89836373127682 (-0.030934777058348573) | > avg_grad_norm_1: tensor(3205.6238, device='cuda:0') (tensor(-111.5562, device='cuda:0'))  > EPOCH: 479/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:07:54)   --> TIME: 2025-02-25 13:08:01 -- STEP: 11/72 -- GLOBAL_STEP: 1034500 | > loss_disc: 2.753302574157715 (2.7225994847037573) | > loss_disc_real_0: 0.2138833999633789 (0.16672595861283215) | > loss_disc_real_1: 0.21059870719909668 (0.2260131984949112) | > loss_disc_real_2: 0.25975900888442993 (0.22746564989740198) | > loss_disc_real_3: 0.2479470819234848 (0.24261253665794025) | > loss_disc_real_4: 0.2611324191093445 (0.24954832142049616) | > loss_disc_real_5: 0.2608094811439514 (0.25520687076178467) | > loss_0: 2.753302574157715 (2.7225994847037573) | > grad_norm_0: tensor(33.1981, device='cuda:0') (tensor(43.7145, device='cuda:0')) | > loss_gen: 1.9106534719467163 (1.9273154735565186) | > loss_kl: 2.1193974018096924 (2.039570082317699) | > loss_feat: 2.6066129207611084 (3.09196138381958) | > loss_mel: 17.034494400024414 (17.260580322959207) | > loss_duration: 1.5310412645339966 (1.551753900267861) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.20220184326172 (25.8711820082231) | > grad_norm_1: tensor(1828.6158, device='cuda:0') (tensor(3335.4133, device='cuda:0')) | > current_lr_0: 0.00018837574730431789 | > current_lr_1: 0.00018837574730431789 | > step_time: 0.4292 (0.43439579010009766) | > loader_time: 0.0063 (0.00680221210826527)  --> TIME: 2025-02-25 13:08:24 -- STEP: 61/72 -- GLOBAL_STEP: 1034550 | > loss_disc: 2.6718664169311523 (2.697375469520444) | > loss_disc_real_0: 0.19385656714439392 (0.17313428797194216) | > loss_disc_real_1: 0.21876230835914612 (0.22852462994270636) | > loss_disc_real_2: 0.24380643665790558 (0.22901813295043882) | > loss_disc_real_3: 0.22434957325458527 (0.23713181790758353) | > loss_disc_real_4: 0.2618142366409302 (0.24367324617065367) | > loss_disc_real_5: 0.2649829089641571 (0.24542990668875272) | > loss_0: 2.6718664169311523 (2.697375469520444) | > grad_norm_0: tensor(49.0448, device='cuda:0') (tensor(46.1315, device='cuda:0')) | > loss_gen: 1.9101543426513672 (1.946563959121704) | > loss_kl: 2.254182815551758 (2.088883759545499) | > loss_feat: 3.235517978668213 (3.1698829072420716) | > loss_mel: 17.421770095825195 (17.287093834798846) | > loss_duration: 1.6164013147354126 (1.577653302521002) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.438026428222656 (26.070077833582143) | > grad_norm_1: tensor(3723.7700, device='cuda:0') (tensor(3649.3479, device='cuda:0')) | > current_lr_0: 0.00018837574730431789 | > current_lr_1: 0.00018837574730431789 | > step_time: 0.4733 (0.44845349280560604) | > loader_time: 0.0115 (0.008320186958938347) --> EVAL PERFORMANCE | > avg_loader_time: 0.008818230158846141 (+0.00010792974015356831) | > avg_step_time: 0.45862337233315054 (+0.0021671946619598037) | > avg_loss_disc: 2.7001580621155217 (-0.01263719209482872) | > avg_loss_disc_real_0: 0.1731695302985084 (+0.007396000057039165) | > avg_loss_disc_real_1: 0.2270175470432765 (+0.006808816127374134) | > avg_loss_disc_real_2: 0.2277106974326389 (-0.008816341699009161) | > avg_loss_disc_real_3: 0.23648262989353125 (+0.0010581654562076825) | > avg_loss_disc_real_4: 0.24256820032294368 (+0.0018090714871044078) | > avg_loss_disc_real_5: 0.2449417361910914 (+0.0012760846547677507) | > avg_loss_0: 2.7001580621155217 (-0.01263719209482872) | > avg_grad_norm_0: tensor(46.1040, device='cuda:0') (+tensor(0.6780, device='cuda:0')) | > avg_loss_gen: 1.9398134543862142 (+0.04104059011163863) | > avg_loss_kl: 2.102548391046659 (-0.010493921561978148) | > avg_loss_feat: 3.1559749858480104 (+0.06678271293640137) | > avg_loss_mel: 17.311753071529765 (+0.10365518381897942) | > avg_loss_duration: 1.5859149593702504 (-0.003343419289924654) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.096004942773092 (+0.197641211496272) | > avg_grad_norm_1: tensor(3669.6411, device='cuda:0') (+tensor(464.0173, device='cuda:0'))  > EPOCH: 480/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:08:31)   --> TIME: 2025-02-25 13:08:51 -- STEP: 39/72 -- GLOBAL_STEP: 1034600 | > loss_disc: 2.6341469287872314 (2.7105807891258826) | > loss_disc_real_0: 0.13807114958763123 (0.1886670277095758) | > loss_disc_real_1: 0.22902652621269226 (0.2246746546947039) | > loss_disc_real_2: 0.18622484803199768 (0.23606148744240785) | > loss_disc_real_3: 0.2296571582555771 (0.23445443770824334) | > loss_disc_real_4: 0.2347811907529831 (0.24401010458285993) | > loss_disc_real_5: 0.23254334926605225 (0.2466837526896061) | > loss_0: 2.6341469287872314 (2.7105807891258826) | > grad_norm_0: tensor(38.2977, device='cuda:0') (tensor(47.9070, device='cuda:0')) | > loss_gen: 1.8789831399917603 (1.958450870636182) | > loss_kl: 2.04683518409729 (2.063225058408884) | > loss_feat: 3.811234712600708 (3.1781872602609487) | > loss_mel: 17.708967208862305 (17.274245384411934) | > loss_duration: 1.6070685386657715 (1.5568750485395773) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.053089141845703 (26.030983509161533) | > grad_norm_1: tensor(5514.0815, device='cuda:0') (tensor(3761.2720, device='cuda:0')) | > current_lr_0: 0.00018835220033590485 | > current_lr_1: 0.00018835220033590485 | > step_time: 0.4574 (0.45301189178075546) | > loader_time: 0.0083 (0.007110473437186999) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514333778703717 (-0.00030389638014242416) | > avg_step_time: 0.46321440078842807 (+0.004591028455277535) | > avg_loss_disc: 2.7306296825408927 (+0.030471620425370993) | > avg_loss_disc_real_0: 0.18128551269920778 (+0.008115982400699384) | > avg_loss_disc_real_1: 0.22852129793502915 (+0.001503750891752642) | > avg_loss_disc_real_2: 0.243926808867656 (+0.01621611143501711) | > avg_loss_disc_real_3: 0.2364620134024553 (-2.0616491075958754e-05) | > avg_loss_disc_real_4: 0.24413798245745646 (+0.0015697821345127816) | > avg_loss_disc_real_5: 0.24585641794641253 (+0.0009146817553211117) | > avg_loss_0: 2.7306296825408927 (+0.030471620425370993) | > avg_grad_norm_0: tensor(49.1880, device='cuda:0') (+tensor(3.0840, device='cuda:0')) | > avg_loss_gen: 1.9427616831282495 (+0.0029482287420352637) | > avg_loss_kl: 2.110766417543654 (+0.008218026496994657) | > avg_loss_feat: 3.1442785598862337 (-0.011696425961776669) | > avg_loss_mel: 17.28494581034486 (-0.026807261184906395) | > avg_loss_duration: 1.5852795940049935 (-0.0006353653652568791) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.06803209009305 (-0.027972852680044014) | > avg_grad_norm_1: tensor(3610.5217, device='cuda:0') (tensor(-59.1194, device='cuda:0'))  > EPOCH: 481/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:09:09)   --> TIME: 2025-02-25 13:09:18 -- STEP: 17/72 -- GLOBAL_STEP: 1034650 | > loss_disc: 2.679673910140991 (2.7052318629096535) | > loss_disc_real_0: 0.19992667436599731 (0.17777219195576274) | > loss_disc_real_1: 0.24802786111831665 (0.2218904328696868) | > loss_disc_real_2: 0.2146715372800827 (0.2268006600001279) | > loss_disc_real_3: 0.24296632409095764 (0.24015355723745682) | > loss_disc_real_4: 0.2533913552761078 (0.24287237840540268) | > loss_disc_real_5: 0.2514737844467163 (0.2445333950659808) | > loss_0: 2.679673910140991 (2.7052318629096535) | > grad_norm_0: tensor(24.5603, device='cuda:0') (tensor(45.0798, device='cuda:0')) | > loss_gen: 2.0347671508789062 (1.9415380323634428) | > loss_kl: 2.2889716625213623 (2.0733673151801613) | > loss_feat: 3.4579553604125977 (3.201284689061782) | > loss_mel: 18.490938186645508 (17.79475251366111) | > loss_duration: 1.5850228071212769 (1.548418402671814) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.857654571533203 (26.55936117733226) | > grad_norm_1: tensor(2921.7144, device='cuda:0') (tensor(3220.8975, device='cuda:0')) | > current_lr_0: 0.00018832865631086286 | > current_lr_1: 0.00018832865631086286 | > step_time: 0.4358 (0.4355054041918586) | > loader_time: 0.0075 (0.006701427347519819)  --> TIME: 2025-02-25 13:09:42 -- STEP: 67/72 -- GLOBAL_STEP: 1034700 | > loss_disc: 2.81850528717041 (2.7231184404287765) | > loss_disc_real_0: 0.3391370177268982 (0.18505955598692392) | > loss_disc_real_1: 0.3108205795288086 (0.23300659434119267) | > loss_disc_real_2: 0.2507930099964142 (0.23248385715840467) | > loss_disc_real_3: 0.24622713029384613 (0.2421015741220161) | > loss_disc_real_4: 0.25944897532463074 (0.24446285149054742) | > loss_disc_real_5: 0.2588353753089905 (0.2466766175939076) | > loss_0: 2.81850528717041 (2.7231184404287765) | > grad_norm_0: tensor(69.4526, device='cuda:0') (tensor(50.5393, device='cuda:0')) | > loss_gen: 2.2510435581207275 (1.9510704883888585) | > loss_kl: 2.2451047897338867 (2.1608042450093503) | > loss_feat: 2.962393045425415 (3.1039107664307553) | > loss_mel: 17.373872756958008 (17.37429610295083) | > loss_duration: 1.6600207090377808 (1.5915562359254751) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.492435455322266 (26.181637778210995) | > grad_norm_1: tensor(4003.3945, device='cuda:0') (tensor(3529.6572, device='cuda:0')) | > current_lr_0: 0.00018832865631086286 | > current_lr_1: 0.00018832865631086286 | > step_time: 0.5206 (0.45376602215553397) | > loader_time: 0.012 (0.008485292320820823) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653173983936578 (+0.0001388402052328614) | > avg_step_time: 0.4575046552738673 (-0.005709745514560771) | > avg_loss_disc: 2.7256448134570053 (-0.0049848690838874) | > avg_loss_disc_real_0: 0.18438396970151172 (+0.0030984570023039337) | > avg_loss_disc_real_1: 0.23388507731363806 (+0.005363779378608907) | > avg_loss_disc_real_2: 0.23280534148216248 (-0.011121467385493522) | > avg_loss_disc_real_3: 0.2420061278930852 (+0.0055441144906299145) | > avg_loss_disc_real_4: 0.2449084841449496 (+0.0007705016874931248) | > avg_loss_disc_real_5: 0.24710038661117284 (+0.0012439686647603143) | > avg_loss_0: 2.7256448134570053 (-0.0049848690838874) | > avg_grad_norm_0: tensor(50.0696, device='cuda:0') (+tensor(0.8816, device='cuda:0')) | > avg_loss_gen: 1.9437701366317104 (+0.0010084535034609043) | > avg_loss_kl: 2.1664433261038543 (+0.055676908560200467) | > avg_loss_feat: 3.08431124351394 (-0.05996731637229358) | > avg_loss_mel: 17.335598824729388 (+0.05065301438452963) | > avg_loss_duration: 1.5938372276198696 (+0.008557633614876048) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.12396076363577 (+0.05592867354272002) | > avg_grad_norm_1: tensor(3493.1169, device='cuda:0') (tensor(-117.4048, device='cuda:0'))  > EPOCH: 482/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:09:46)   --> TIME: 2025-02-25 13:10:08 -- STEP: 45/72 -- GLOBAL_STEP: 1034750 | > loss_disc: 2.697819232940674 (2.7056527773539227) | > loss_disc_real_0: 0.1485813558101654 (0.1774773442082935) | > loss_disc_real_1: 0.21329738199710846 (0.22947799795203738) | > loss_disc_real_2: 0.23730741441249847 (0.23235757582717473) | > loss_disc_real_3: 0.241257444024086 (0.23315269218550788) | > loss_disc_real_4: 0.2531522214412689 (0.2420378393597073) | > loss_disc_real_5: 0.22058063745498657 (0.24346950749556223) | > loss_0: 2.697819232940674 (2.7056527773539227) | > grad_norm_0: tensor(61.1638, device='cuda:0') (tensor(46.9109, device='cuda:0')) | > loss_gen: 1.8181880712509155 (1.9637634038925171) | > loss_kl: 2.151106119155884 (2.354033822483486) | > loss_feat: 2.8607723712921143 (3.180878851148817) | > loss_mel: 17.83107566833496 (18.42240715026856) | > loss_duration: 1.6037931442260742 (1.6048636489444308) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.264934539794922 (27.525946553548177) | > grad_norm_1: tensor(2730.2539, device='cuda:0') (tensor(3559.3831, device='cuda:0')) | > current_lr_0: 0.000188305115228824 | > current_lr_1: 0.000188305115228824 | > step_time: 0.4704 (0.44417782359653046) | > loader_time: 0.0089 (0.0077364444732666016) --> EVAL PERFORMANCE | > avg_loader_time: 0.008765153481926717 (+0.00011197949799013861) | > avg_step_time: 0.45868938069948006 (+0.0011847254256127648) | > avg_loss_disc: 2.720720378445908 (-0.0049244350110972945) | > avg_loss_disc_real_0: 0.1797760020889027 (-0.00460796761260901) | > avg_loss_disc_real_1: 0.22874652227045786 (-0.0051385550431801985) | > avg_loss_disc_real_2: 0.23161483566525956 (-0.0011905058169029148) | > avg_loss_disc_real_3: 0.2334949640740811 (-0.008511163819004103) | > avg_loss_disc_real_4: 0.23992983346254054 (-0.004978650682409047) | > avg_loss_disc_real_5: 0.24295753366510633 (-0.004142852946066505) | > avg_loss_0: 2.720720378445908 (-0.0049244350110972945) | > avg_grad_norm_0: tensor(49.6628, device='cuda:0') (tensor(-0.4068, device='cuda:0')) | > avg_loss_gen: 1.934819090534264 (-0.008951046097446458) | > avg_loss_kl: 2.311571587978954 (+0.14512826187509953) | > avg_loss_feat: 3.1300370491726297 (+0.045725805658689556) | > avg_loss_mel: 18.111084414200036 (+0.775485589470648) | > avg_loss_duration: 1.6134235271265809 (+0.019586299506711313) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.10093546585298 (+0.9769747022172126) | > avg_grad_norm_1: tensor(3549.4624, device='cuda:0') (+tensor(56.3455, device='cuda:0'))  > EPOCH: 483/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:10:23)   --> TIME: 2025-02-25 13:10:36 -- STEP: 23/72 -- GLOBAL_STEP: 1034800 | > loss_disc: 2.7012476921081543 (2.709869892700859) | > loss_disc_real_0: 0.11317415535449982 (0.17815781741038614) | > loss_disc_real_1: 0.275814950466156 (0.2262153087750725) | > loss_disc_real_2: 0.25181618332862854 (0.2410465375236843) | > loss_disc_real_3: 0.23080097138881683 (0.23628010244473166) | > loss_disc_real_4: 0.2443579137325287 (0.24653148456760074) | > loss_disc_real_5: 0.26000291109085083 (0.24826275719248733) | > loss_0: 2.7012476921081543 (2.709869892700859) | > grad_norm_0: tensor(36.5939, device='cuda:0') (tensor(45.2487, device='cuda:0')) | > loss_gen: 1.8476417064666748 (1.9510633271673452) | > loss_kl: 2.2187352180480957 (2.0815388119739033) | > loss_feat: 2.6353046894073486 (3.1728054958841074) | > loss_mel: 16.55550193786621 (17.48035613350246) | > loss_duration: 1.562015414237976 (1.5462586465089216) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.819198608398438 (26.232021829356317) | > grad_norm_1: tensor(3469.9932, device='cuda:0') (tensor(3597.5508, device='cuda:0')) | > current_lr_0: 0.0001882815770894204 | > current_lr_1: 0.0001882815770894204 | > step_time: 0.4303 (0.43688340809034265) | > loader_time: 0.0063 (0.006668743879898735) --> EVAL PERFORMANCE | > avg_loader_time: 0.008587639096757058 (-0.00017751438516965846) | > avg_step_time: 0.4573683671548333 (-0.0013210135446467541) | > avg_loss_disc: 2.7242405179520732 (+0.003520139506165254) | > avg_loss_disc_real_0: 0.18710185067964272 (+0.007325848590740008) | > avg_loss_disc_real_1: 0.22172421049064314 (-0.007022311779814716) | > avg_loss_disc_real_2: 0.23687264323234558 (+0.00525780756708602) | > avg_loss_disc_real_3: 0.23510141754654093 (+0.001606453472459829) | > avg_loss_disc_real_4: 0.24432353591415243 (+0.004393702451611886) | > avg_loss_disc_real_5: 0.24606070426148427 (+0.0031031705963779355) | > avg_loss_0: 2.7242405179520732 (+0.003520139506165254) | > avg_grad_norm_0: tensor(46.5112, device='cuda:0') (tensor(-3.1516, device='cuda:0')) | > avg_loss_gen: 1.9338569456422832 (-0.0009621448919807651) | > avg_loss_kl: 2.1846611583736575 (-0.1269104296052963) | > avg_loss_feat: 3.0908355142029245 (-0.03920153496970524) | > avg_loss_mel: 17.527635493748623 (-0.5834489204514135) | > avg_loss_duration: 1.589154297197369 (-0.02426922992921199) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.326143184178314 (-0.7747922816746673) | > avg_grad_norm_1: tensor(3498.2334, device='cuda:0') (tensor(-51.2290, device='cuda:0'))  > EPOCH: 484/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:11:01)   --> TIME: 2025-02-25 13:11:03 -- STEP: 1/72 -- GLOBAL_STEP: 1034850 | > loss_disc: 2.7288217544555664 (2.7288217544555664) | > loss_disc_real_0: 0.09645717591047287 (0.09645717591047287) | > loss_disc_real_1: 0.23865258693695068 (0.23865258693695068) | > loss_disc_real_2: 0.22265315055847168 (0.22265315055847168) | > loss_disc_real_3: 0.258417546749115 (0.258417546749115) | > loss_disc_real_4: 0.24281606078147888 (0.24281606078147888) | > loss_disc_real_5: 0.24259734153747559 (0.24259734153747559) | > loss_0: 2.7288217544555664 (2.7288217544555664) | > grad_norm_0: tensor(75.9884, device='cuda:0') (tensor(75.9884, device='cuda:0')) | > loss_gen: 1.9100624322891235 (1.9100624322891235) | > loss_kl: 2.392927408218384 (2.392927408218384) | > loss_feat: 3.4111106395721436 (3.4111106395721436) | > loss_mel: 19.98802375793457 (19.98802375793457) | > loss_duration: 1.585418701171875 (1.585418701171875) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.28754234313965 (29.28754234313965) | > grad_norm_1: tensor(4241.3687, device='cuda:0') (tensor(4241.3687, device='cuda:0')) | > current_lr_0: 0.00018825804189228422 | > current_lr_1: 0.00018825804189228422 | > step_time: 0.4277 (0.4276883602142334) | > loader_time: 0.0061 (0.006108999252319336)  --> TIME: 2025-02-25 13:11:26 -- STEP: 51/72 -- GLOBAL_STEP: 1034900 | > loss_disc: 2.756932497024536 (2.70758028123893) | > loss_disc_real_0: 0.20963311195373535 (0.17356052278887987) | > loss_disc_real_1: 0.2537323236465454 (0.232722587445203) | > loss_disc_real_2: 0.2556535303592682 (0.22984268501693128) | > loss_disc_real_3: 0.24001169204711914 (0.2357174555460612) | > loss_disc_real_4: 0.25839465856552124 (0.2433172356264264) | > loss_disc_real_5: 0.26364433765411377 (0.24905998011430105) | > loss_0: 2.756932497024536 (2.70758028123893) | > grad_norm_0: tensor(66.7488, device='cuda:0') (tensor(50.9062, device='cuda:0')) | > loss_gen: 1.8878313302993774 (1.931161321845709) | > loss_kl: 2.092249631881714 (2.1718412520838726) | > loss_feat: 2.6858508586883545 (3.0830356700747625) | > loss_mel: 16.241344451904297 (17.606570486928895) | > loss_duration: 1.6078484058380127 (1.5732748742197074) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.515125274658203 (26.36588373371199) | > grad_norm_1: tensor(3295.0964, device='cuda:0') (tensor(3862.9824, device='cuda:0')) | > current_lr_0: 0.00018825804189228422 | > current_lr_1: 0.00018825804189228422 | > step_time: 0.4525 (0.44632405860751284) | > loader_time: 0.0089 (0.007824149786257277) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652297543807768 (+6.465844705070938e-05) | > avg_step_time: 0.4584857745909355 (+0.001117407436102169) | > avg_loss_disc: 2.7145036207118505 (-0.009736897240222753) | > avg_loss_disc_real_0: 0.17548945210349387 (-0.011612398576148847) | > avg_loss_disc_real_1: 0.23093019341918783 (+0.009205982928544687) | > avg_loss_disc_real_2: 0.2322039996654215 (-0.004668643566924091) | > avg_loss_disc_real_3: 0.23610032864019903 (+0.0009989110936581003) | > avg_loss_disc_real_4: 0.24371372541071665 (-0.0006098105034357815) | > avg_loss_disc_real_5: 0.24882713023205877 (+0.0027664259705744987) | > avg_loss_0: 2.7145036207118505 (-0.009736897240222753) | > avg_grad_norm_0: tensor(50.7002, device='cuda:0') (+tensor(4.1890, device='cuda:0')) | > avg_loss_gen: 1.9197575559078808 (-0.01409938973440239) | > avg_loss_kl: 2.187306615668284 (+0.0026454572946263966) | > avg_loss_feat: 3.041991838267152 (-0.048843675935772435) | > avg_loss_mel: 17.47922435276945 (-0.048411140979172984) | > avg_loss_duration: 1.586859194325729 (-0.002295102871639809) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.215139738270935 (-0.11100344590737876) | > avg_grad_norm_1: tensor(3812.5254, device='cuda:0') (+tensor(314.2920, device='cuda:0'))  > EPOCH: 485/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:11:38)   --> TIME: 2025-02-25 13:11:53 -- STEP: 29/72 -- GLOBAL_STEP: 1034950 | > loss_disc: 2.7900590896606445 (2.7012720847951956) | > loss_disc_real_0: 0.07641254365444183 (0.17953722744152464) | > loss_disc_real_1: 0.2214535027742386 (0.22398584156200804) | > loss_disc_real_2: 0.28420963883399963 (0.24090043892120494) | > loss_disc_real_3: 0.198907732963562 (0.23903218682469993) | > loss_disc_real_4: 0.24730022251605988 (0.24566484679435863) | > loss_disc_real_5: 0.2543468177318573 (0.24620849716252294) | > loss_0: 2.7900590896606445 (2.7012720847951956) | > grad_norm_0: tensor(56.0864, device='cuda:0') (tensor(54.0717, device='cuda:0')) | > loss_gen: 1.8932121992111206 (1.9798328712068756) | > loss_kl: 2.1513283252716064 (2.082078744625223) | > loss_feat: 3.0247950553894043 (3.2674818532220247) | > loss_mel: 17.08245086669922 (17.20001503516888) | > loss_duration: 1.5227646827697754 (1.5429374513954952) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.674551010131836 (26.07234579941322) | > grad_norm_1: tensor(3515.7727, device='cuda:0') (tensor(4145.7603, device='cuda:0')) | > current_lr_0: 0.0001882345096370477 | > current_lr_1: 0.0001882345096370477 | > step_time: 0.4562 (0.4380974194099163) | > loader_time: 0.0077 (0.007128805949770171) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869888319096095 (+4.658564715318177e-05) | > avg_step_time: 0.45461466950429996 (-0.003871105086635518) | > avg_loss_disc: 2.723877409814109 (+0.009373789102258723) | > avg_loss_disc_real_0: 0.182067231917885 (+0.006577779814391116) | > avg_loss_disc_real_1: 0.22660907137561853 (-0.004321122043569298) | > avg_loss_disc_real_2: 0.24006632096330885 (+0.007862321297887359) | > avg_loss_disc_real_3: 0.23857564153805585 (+0.002475312897856824) | > avg_loss_disc_real_4: 0.24425825371708668 (+0.000544528306370029) | > avg_loss_disc_real_5: 0.24695462806963583 (-0.0018725021624229343) | > avg_loss_0: 2.723877409814109 (+0.009373789102258723) | > avg_grad_norm_0: tensor(52.3607, device='cuda:0') (+tensor(1.6604, device='cuda:0')) | > avg_loss_gen: 1.9367712030948048 (+0.017013647186923997) | > avg_loss_kl: 2.134100489213433 (-0.0532061264548509) | > avg_loss_feat: 3.122265422847909 (+0.08027358458075717) | > avg_loss_mel: 17.16374034612952 (-0.3154840066399309) | > avg_loss_duration: 1.5838492504307922 (-0.0030099438949369173) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.940726589149154 (-0.2744131491217807) | > avg_grad_norm_1: tensor(3855.8406, device='cuda:0') (+tensor(43.3152, device='cuda:0'))  > EPOCH: 486/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:12:15)   --> TIME: 2025-02-25 13:12:20 -- STEP: 7/72 -- GLOBAL_STEP: 1035000 | > loss_disc: 2.6513988971710205 (2.7013256890433177) | > loss_disc_real_0: 0.16315315663814545 (0.17031231735433852) | > loss_disc_real_1: 0.1976705640554428 (0.22943400698048727) | > loss_disc_real_2: 0.2265607863664627 (0.22724564799240657) | > loss_disc_real_3: 0.2252809852361679 (0.23717460674898966) | > loss_disc_real_4: 0.2501647472381592 (0.2484005242586136) | > loss_disc_real_5: 0.2483064979314804 (0.2469481932265418) | > loss_0: 2.6513988971710205 (2.7013256890433177) | > grad_norm_0: tensor(59.3685, device='cuda:0') (tensor(64.6907, device='cuda:0')) | > loss_gen: 2.0767340660095215 (1.9331933430262975) | > loss_kl: 2.4462552070617676 (2.09466814994812) | > loss_feat: 3.332688570022583 (3.0669102328164235) | > loss_mel: 17.967317581176758 (17.289095742361887) | > loss_duration: 1.5503382682800293 (1.5567737477166312) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.373334884643555 (25.940640585763113) | > grad_norm_1: tensor(4702.0327, device='cuda:0') (tensor(3789.2358, device='cuda:0')) | > current_lr_0: 0.00018821098032334305 | > current_lr_1: 0.00018821098032334305 | > step_time: 0.4336 (0.43684407642909456) | > loader_time: 0.0055 (0.007125684193202427) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1035000.pth  --> TIME: 2025-02-25 13:12:46 -- STEP: 57/72 -- GLOBAL_STEP: 1035050 | > loss_disc: 2.7950053215026855 (2.7217832740984464) | > loss_disc_real_0: 0.07412004470825195 (0.18064904513589122) | > loss_disc_real_1: 0.23253339529037476 (0.22606752682150455) | > loss_disc_real_2: 0.2603835165500641 (0.23484293123086294) | > loss_disc_real_3: 0.26291000843048096 (0.23669159360099257) | > loss_disc_real_4: 0.23354549705982208 (0.24388741584200607) | > loss_disc_real_5: 0.23198522627353668 (0.2452270163778673) | > loss_0: 2.7950053215026855 (2.7217832740984464) | > grad_norm_0: tensor(70.3674, device='cuda:0') (tensor(56.0802, device='cuda:0')) | > loss_gen: 1.827797770500183 (1.936914374953822) | > loss_kl: 2.0987987518310547 (2.149346224048681) | > loss_feat: 2.9846081733703613 (3.1792208939267876) | > loss_mel: 16.39251136779785 (17.332201137877345) | > loss_duration: 1.6428844928741455 (1.5758072953475148) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.94659996032715 (26.173489788122346) | > grad_norm_1: tensor(2244.3589, device='cuda:0') (tensor(3674.0259, device='cuda:0')) | > current_lr_0: 0.00018821098032334305 | > current_lr_1: 0.00018821098032334305 | > step_time: 0.4644 (0.4491207641467713) | > loader_time: 0.0096 (0.007998499953955934) --> EVAL PERFORMANCE | > avg_loader_time: 0.008634241533951021 (-6.464165700992806e-05) | > avg_step_time: 0.45819452782751807 (+0.0035798583232181103) | > avg_loss_disc: 2.720875421040495 (-0.0030019887736143502) | > avg_loss_disc_real_0: 0.17999639970735765 (-0.0020708322105273402) | > avg_loss_disc_real_1: 0.22651357151253124 (-9.549986308729497e-05) | > avg_loss_disc_real_2: 0.2332051285135914 (-0.006861192449717446) | > avg_loss_disc_real_3: 0.23756235271272524 (-0.0010132888253306105) | > avg_loss_disc_real_4: 0.2434408834702532 (-0.000817370246833482) | > avg_loss_disc_real_5: 0.24407683753631484 (-0.0028777905333209897) | > avg_loss_0: 2.720875421040495 (-0.0030019887736143502) | > avg_grad_norm_0: tensor(56.2226, device='cuda:0') (+tensor(3.8619, device='cuda:0')) | > avg_loss_gen: 1.930780548444936 (-0.0059906546498686986) | > avg_loss_kl: 2.1768129627469555 (+0.04271247353352248) | > avg_loss_feat: 3.143116383485391 (+0.02085096063748182) | > avg_loss_mel: 17.3157313843848 (+0.15199103825528226) | > avg_loss_duration: 1.5876281781935355 (+0.003778927762743356) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.15406933636733 (+0.21334274721817437) | > avg_grad_norm_1: tensor(3663.9170, device='cuda:0') (tensor(-191.9236, device='cuda:0'))  > EPOCH: 487/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:12:55)   --> TIME: 2025-02-25 13:13:13 -- STEP: 35/72 -- GLOBAL_STEP: 1035100 | > loss_disc: 2.8854434490203857 (2.70924277986799) | > loss_disc_real_0: 0.2996808588504791 (0.1825112393924168) | > loss_disc_real_1: 0.27938446402549744 (0.23082535692623685) | > loss_disc_real_2: 0.21544349193572998 (0.23427573144435881) | > loss_disc_real_3: 0.2455051988363266 (0.23539227843284607) | > loss_disc_real_4: 0.23522329330444336 (0.2435627622263772) | > loss_disc_real_5: 0.21953661739826202 (0.2474532391343798) | > loss_0: 2.8854434490203857 (2.70924277986799) | > grad_norm_0: tensor(47.4043, device='cuda:0') (tensor(52.6149, device='cuda:0')) | > loss_gen: 1.8095308542251587 (1.9626135349273681) | > loss_kl: 2.1379764080047607 (2.1435193061828604) | > loss_feat: 2.8522801399230957 (3.193195343017578) | > loss_mel: 17.861772537231445 (17.30051362173898) | > loss_duration: 1.6161119937896729 (1.5567188910075596) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.27766990661621 (26.1565605708531) | > grad_norm_1: tensor(3802.4812, device='cuda:0') (tensor(3936.5925, device='cuda:0')) | > current_lr_0: 0.00018818745395080263 | > current_lr_1: 0.00018818745395080263 | > step_time: 0.4497 (0.4355790002005441) | > loader_time: 0.0082 (0.007197713851928711) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857759193635323 (-5.6649597597791326e-05) | > avg_step_time: 0.45274483989661846 (-0.005449687930899616) | > avg_loss_disc: 2.7189248817067746 (-0.001950539333720247) | > avg_loss_disc_real_0: 0.18001668076490016 (+2.0281057542509418e-05) | > avg_loss_disc_real_1: 0.2299116033483559 (+0.0033980318358246553) | > avg_loss_disc_real_2: 0.23415016939102765 (+0.0009450408774362429) | > avg_loss_disc_real_3: 0.23792198909000611 (+0.0003596363772808697) | > avg_loss_disc_real_4: 0.242483054668131 (-0.0009578288021221959) | > avg_loss_disc_real_5: 0.24622887540870989 (+0.0021520378723950406) | > avg_loss_0: 2.7189248817067746 (-0.001950539333720247) | > avg_grad_norm_0: tensor(53.0480, device='cuda:0') (tensor(-3.1745, device='cuda:0')) | > avg_loss_gen: 1.9401589548084097 (+0.009378406363473646) | > avg_loss_kl: 2.1706450119824483 (-0.006167950764507157) | > avg_loss_feat: 3.094657619234542 (-0.04845876425084894) | > avg_loss_mel: 17.19502161590147 (-0.12070976848333004) | > avg_loss_duration: 1.5884142660758864 (+0.0007860878823509054) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.98889737733653 (-0.16517195903079696) | > avg_grad_norm_1: tensor(3849.0171, device='cuda:0') (+tensor(185.1001, device='cuda:0'))  > EPOCH: 488/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:13:32)   --> TIME: 2025-02-25 13:13:40 -- STEP: 13/72 -- GLOBAL_STEP: 1035150 | > loss_disc: 2.859541893005371 (2.7315777448507457) | > loss_disc_real_0: 0.2795775234699249 (0.16619923137701476) | > loss_disc_real_1: 0.23711541295051575 (0.2324862663562481) | > loss_disc_real_2: 0.2579612135887146 (0.225367193038647) | > loss_disc_real_3: 0.23894037306308746 (0.23020544533546156) | > loss_disc_real_4: 0.2522146999835968 (0.2385524052840013) | > loss_disc_real_5: 0.2682107388973236 (0.23525074124336243) | > loss_0: 2.859541893005371 (2.7315777448507457) | > grad_norm_0: tensor(60.9848, device='cuda:0') (tensor(53.1193, device='cuda:0')) | > loss_gen: 1.9906995296478271 (1.9067991421772883) | > loss_kl: 1.9637540578842163 (2.077954805814303) | > loss_feat: 2.49839448928833 (3.054472263042743) | > loss_mel: 16.263914108276367 (17.33848469073956) | > loss_duration: 1.4703786373138428 (1.5444361705046434) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.18714141845703 (25.922147017258865) | > grad_norm_1: tensor(3646.2288, device='cuda:0') (tensor(3762.4939, device='cuda:0')) | > current_lr_0: 0.00018816393051905876 | > current_lr_1: 0.00018816393051905876 | > step_time: 0.4649 (0.43927539311922514) | > loader_time: 0.0073 (0.006912194765531099)  --> TIME: 2025-02-25 13:14:03 -- STEP: 63/72 -- GLOBAL_STEP: 1035200 | > loss_disc: 2.645897150039673 (2.7150804277450318) | > loss_disc_real_0: 0.19244886934757233 (0.1746606101828908) | > loss_disc_real_1: 0.22912578284740448 (0.22712672726502495) | > loss_disc_real_2: 0.2355872541666031 (0.23110553409372056) | > loss_disc_real_3: 0.25355833768844604 (0.2332066193459526) | > loss_disc_real_4: 0.2519437372684479 (0.2436779581365131) | > loss_disc_real_5: 0.2618291676044464 (0.24566555188761818) | > loss_0: 2.645897150039673 (2.7150804277450318) | > grad_norm_0: tensor(42.8979, device='cuda:0') (tensor(55.4571, device='cuda:0')) | > loss_gen: 2.0242300033569336 (1.9271535911257305) | > loss_kl: 2.306910514831543 (2.147490034027706) | > loss_feat: 3.1420786380767822 (3.0896310730585976) | > loss_mel: 17.528907775878906 (17.27285721188499) | > loss_duration: 1.6278126239776611 (1.5825847708989704) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.629941940307617 (26.019716565571134) | > grad_norm_1: tensor(6058.3672, device='cuda:0') (tensor(4163.4253, device='cuda:0')) | > current_lr_0: 0.00018816393051905876 | > current_lr_1: 0.00018816393051905876 | > step_time: 0.4838 (0.4499929367549836) | > loader_time: 0.0113 (0.008340018136160714) --> EVAL PERFORMANCE | > avg_loader_time: 0.008704296300109004 (+0.00012670436375577382) | > avg_step_time: 0.45511946543841286 (+0.002374625541794406) | > avg_loss_disc: 2.711771887792668 (-0.00715299391410662) | > avg_loss_disc_real_0: 0.17435091838870248 (-0.005665762376197675) | > avg_loss_disc_real_1: 0.22819518497292424 (-0.0017164183754316553) | > avg_loss_disc_real_2: 0.22859475499307605 (-0.005555414397951597) | > avg_loss_disc_real_3: 0.23380841650593448 (-0.004113572584071634) | > avg_loss_disc_real_4: 0.24266416443065858 (+0.00018110976252758149) | > avg_loss_disc_real_5: 0.24551161284178075 (-0.0007172625669291388) | > avg_loss_0: 2.711771887792668 (-0.00715299391410662) | > avg_grad_norm_0: tensor(56.2243, device='cuda:0') (+tensor(3.1763, device='cuda:0')) | > avg_loss_gen: 1.9321872432466964 (-0.00797171156171328) | > avg_loss_kl: 2.1716280000310557 (+0.0009829880486074138) | > avg_loss_feat: 3.0999365692407315 (+0.005278950006189387) | > avg_loss_mel: 17.35568575791909 (+0.16066414201761958) | > avg_loss_duration: 1.5908110242494395 (+0.002396758173553115) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.150248514094823 (+0.1613511367582916) | > avg_grad_norm_1: tensor(4232.2915, device='cuda:0') (+tensor(383.2744, device='cuda:0'))  > EPOCH: 489/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:14:09)   --> TIME: 2025-02-25 13:14:30 -- STEP: 41/72 -- GLOBAL_STEP: 1035250 | > loss_disc: 2.9796037673950195 (2.7238335202379926) | > loss_disc_real_0: 0.061980120837688446 (0.16399910755273772) | > loss_disc_real_1: 0.22761158645153046 (0.22837241502796732) | > loss_disc_real_2: 0.2794945538043976 (0.23829787533457686) | > loss_disc_real_3: 0.28439638018608093 (0.2356086089843657) | > loss_disc_real_4: 0.25674131512641907 (0.23848777718660308) | > loss_disc_real_5: 0.24465660750865936 (0.24355398663660374) | > loss_0: 2.9796037673950195 (2.7238335202379926) | > grad_norm_0: tensor(50.7642, device='cuda:0') (tensor(57.1665, device='cuda:0')) | > loss_gen: 1.9030011892318726 (1.950092135406122) | > loss_kl: 2.11763596534729 (2.1444793677911527) | > loss_feat: 2.9501168727874756 (3.266458395050793) | > loss_mel: 17.24211883544922 (17.60026254886534) | > loss_duration: 1.5674049854278564 (1.5601432556059303) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.7802791595459 (26.52143562130812) | > grad_norm_1: tensor(2357.1794, device='cuda:0') (tensor(4334.1338, device='cuda:0')) | > current_lr_0: 0.00018814041002774388 | > current_lr_1: 0.00018814041002774388 | > step_time: 0.4607 (0.4530607374703012) | > loader_time: 0.0113 (0.007330783983556236) --> EVAL PERFORMANCE | > avg_loader_time: 0.008631434239132303 (-7.286206097670105e-05) | > avg_step_time: 0.46750947119484487 (+0.01239000575643201) | > avg_loss_disc: 2.7581494727604827 (+0.046377584967814744) | > avg_loss_disc_real_0: 0.18397270069575647 (+0.009621782307053989) | > avg_loss_disc_real_1: 0.2315910435478452 (+0.0033958585749209735) | > avg_loss_disc_real_2: 0.23747239503222453 (+0.008877640039148482) | > avg_loss_disc_real_3: 0.23743445142893724 (+0.0036260349230027578) | > avg_loss_disc_real_4: 0.24285860653494445 (+0.0001944421042858746) | > avg_loss_disc_real_5: 0.24670891778569826 (+0.0011973049439175099) | > avg_loss_0: 2.7581494727604827 (+0.046377584967814744) | > avg_grad_norm_0: tensor(55.6852, device='cuda:0') (tensor(-0.5391, device='cuda:0')) | > avg_loss_gen: 1.9344702757580179 (+0.002283032511321448) | > avg_loss_kl: 2.208827858239832 (+0.03719985820877625) | > avg_loss_feat: 3.145569478961783 (+0.04563290972105172) | > avg_loss_mel: 17.630941914840488 (+0.2752561569213974) | > avg_loss_duration: 1.5913804165074523 (+0.0005693922580127797) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.51118985028334 (+0.3609413361885174) | > avg_grad_norm_1: tensor(3767.6323, device='cuda:0') (tensor(-464.6592, device='cuda:0'))  > EPOCH: 490/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:14:47)   --> TIME: 2025-02-25 13:14:57 -- STEP: 19/72 -- GLOBAL_STEP: 1035300 | > loss_disc: 2.952440023422241 (2.839170957866468) | > loss_disc_real_0: 0.36196169257164 (0.18769217870737376) | > loss_disc_real_1: 0.2766750454902649 (0.311076572458995) | > loss_disc_real_2: 0.3356981575489044 (0.240664530741541) | > loss_disc_real_3: 0.26689115166664124 (0.24239821810471385) | > loss_disc_real_4: 0.2590155303478241 (0.2480905377551129) | > loss_disc_real_5: 0.2544189393520355 (0.24763012402936033) | > loss_0: 2.952440023422241 (2.839170957866468) | > grad_norm_0: tensor(27.1044, device='cuda:0') (tensor(46.4284, device='cuda:0')) | > loss_gen: 1.9572004079818726 (2.013419358353866) | > loss_kl: 3.083822727203369 (2.220572540634557) | > loss_feat: 2.8349084854125977 (3.108514484606291) | > loss_mel: 20.18511199951172 (18.059763657419307) | > loss_duration: 1.8426761627197266 (1.573518106811925) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.90372085571289 (26.97578781529477) | > grad_norm_1: tensor(572.3993, device='cuda:0') (tensor(2640.2795, device='cuda:0')) | > current_lr_0: 0.00018811689247649042 | > current_lr_1: 0.00018811689247649042 | > step_time: 0.4262 (0.4349988510734157) | > loader_time: 0.0062 (0.006671867872539319)  --> TIME: 2025-02-25 13:15:21 -- STEP: 69/72 -- GLOBAL_STEP: 1035350 | > loss_disc: 2.7578353881835938 (2.7783049120419268) | > loss_disc_real_0: 0.21807709336280823 (0.18271405886912692) | > loss_disc_real_1: 0.18683552742004395 (0.254570946637271) | > loss_disc_real_2: 0.26354798674583435 (0.2453337931546612) | > loss_disc_real_3: 0.22075535356998444 (0.23932844897111258) | > loss_disc_real_4: 0.2741711437702179 (0.2451711785102236) | > loss_disc_real_5: 0.25725579261779785 (0.25014734160209046) | > loss_0: 2.7578353881835938 (2.7783049120419268) | > grad_norm_0: tensor(89.4275, device='cuda:0') (tensor(47.5151, device='cuda:0')) | > loss_gen: 1.97459077835083 (2.0053992444190425) | > loss_kl: 2.3036575317382812 (2.485098515731701) | > loss_feat: 2.6032044887542725 (3.0800098923669346) | > loss_mel: 17.67635726928711 (18.432725740515668) | > loss_duration: 1.705974817276001 (1.630390384922857) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.263784408569336 (27.63362351016722) | > grad_norm_1: tensor(2653.1428, device='cuda:0') (tensor(2799.7756, device='cuda:0')) | > current_lr_0: 0.00018811689247649042 | > current_lr_1: 0.00018811689247649042 | > step_time: 0.5163 (0.4542762507563052) | > loader_time: 0.0128 (0.008705685104148975) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754988791237414 (+0.0001235545521051113) | > avg_step_time: 0.4561038185173357 (-0.011405652677509148) | > avg_loss_disc: 2.779563537785705 (+0.02141406502522214) | > avg_loss_disc_real_0: 0.186425807190613 (+0.0024531064948565273) | > avg_loss_disc_real_1: 0.25288138685511863 (+0.02129034330727342) | > avg_loss_disc_real_2: 0.24540678988879835 (+0.007934394856573823) | > avg_loss_disc_real_3: 0.2375675576253676 (+0.00013310619643036592) | > avg_loss_disc_real_4: 0.2451280458292491 (+0.002269439294304648) | > avg_loss_disc_real_5: 0.2498848406781613 (+0.0031759228924630456) | > avg_loss_0: 2.779563537785705 (+0.02141406502522214) | > avg_grad_norm_0: tensor(48.6655, device='cuda:0') (tensor(-7.0197, device='cuda:0')) | > avg_loss_gen: 2.003983218904952 (+0.0695129431469339) | > avg_loss_kl: 2.479166009056736 (+0.27033815081690404) | > avg_loss_feat: 3.0732030499149374 (-0.07236642904684576) | > avg_loss_mel: 18.401647218516175 (+0.7707053036756868) | > avg_loss_duration: 1.6301422068770504 (+0.03876179036959804) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.588141427913182 (+1.076951577629842) | > avg_grad_norm_1: tensor(2808.0759, device='cuda:0') (tensor(-959.5564, device='cuda:0'))  > EPOCH: 491/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:15:24)   --> TIME: 2025-02-25 13:15:47 -- STEP: 47/72 -- GLOBAL_STEP: 1035400 | > loss_disc: 2.980727195739746 (2.767446340398585) | > loss_disc_real_0: 0.07025447487831116 (0.19353786459628572) | > loss_disc_real_1: 0.20886850357055664 (0.24204295509039087) | > loss_disc_real_2: 0.254242479801178 (0.2466799829234468) | > loss_disc_real_3: 0.22510065138339996 (0.24608295268200814) | > loss_disc_real_4: 0.2511466443538666 (0.24401162152594708) | > loss_disc_real_5: 0.25000667572021484 (0.24867530547557992) | > loss_0: 2.980727195739746 (2.767446340398585) | > grad_norm_0: tensor(29.1082, device='cuda:0') (tensor(42.6308, device='cuda:0')) | > loss_gen: 1.6364725828170776 (1.9549146839912905) | > loss_kl: 2.157524585723877 (2.4621446031205196) | > loss_feat: 2.912355661392212 (3.0642441282881068) | > loss_mel: 18.606874465942383 (19.511827063053214) | > loss_duration: 1.638664960861206 (1.6118435707498104) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.95189094543457 (28.604974097393928) | > grad_norm_1: tensor(58.4427, device='cuda:0') (tensor(2671.7705, device='cuda:0')) | > current_lr_0: 0.00018809337786493085 | > current_lr_1: 0.00018809337786493085 | > step_time: 0.4603 (0.44327969753995855) | > loader_time: 0.0089 (0.007880449295043945) --> EVAL PERFORMANCE | > avg_loader_time: 0.008814140104911695 (+5.9151313674281336e-05) | > avg_step_time: 0.4580861078181737 (+0.0019822893008379916) | > avg_loss_disc: 2.733185694251262 (-0.04637784353444285) | > avg_loss_disc_real_0: 0.19676553468469163 (+0.010339727494078632) | > avg_loss_disc_real_1: 0.2346761240295961 (-0.01820526282552254) | > avg_loss_disc_real_2: 0.23657434276292022 (-0.008832447125878135) | > avg_loss_disc_real_3: 0.24237429801846894 (+0.004806740393101333) | > avg_loss_disc_real_4: 0.2407524929919713 (-0.004375552837277807) | > avg_loss_disc_real_5: 0.24553973783909436 (-0.004345102839066944) | > avg_loss_0: 2.733185694251262 (-0.04637784353444285) | > avg_grad_norm_0: tensor(42.0440, device='cuda:0') (tensor(-6.6215, device='cuda:0')) | > avg_loss_gen: 1.9499918121687123 (-0.05399140673623948) | > avg_loss_kl: 2.505615640693987 (+0.026449631637250892) | > avg_loss_feat: 3.0328039384224046 (-0.040399111492532835) | > avg_loss_mel: 19.44479622639402 (+1.0431490078778438) | > avg_loss_duration: 1.6300642104216025 (-7.79964554478152e-05) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.56327177773059 (+0.9751303498174089) | > avg_grad_norm_1: tensor(2743.5513, device='cuda:0') (tensor(-64.5247, device='cuda:0'))  > EPOCH: 492/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:16:01)   --> TIME: 2025-02-25 13:16:15 -- STEP: 25/72 -- GLOBAL_STEP: 1035450 | > loss_disc: 2.5884408950805664 (2.613144006729126) | > loss_disc_real_0: 0.13625283539295197 (0.15963444501161575) | > loss_disc_real_1: 0.18360191583633423 (0.2179123878479004) | > loss_disc_real_2: 0.1993086338043213 (0.2341972440481186) | > loss_disc_real_3: 0.21899059414863586 (0.2351132196187973) | > loss_disc_real_4: 0.22308215498924255 (0.23987055838108062) | > loss_disc_real_5: 0.2173590213060379 (0.24357934057712555) | > loss_0: 2.5884408950805664 (2.613144006729126) | > grad_norm_0: tensor(31.8333, device='cuda:0') (tensor(34.7124, device='cuda:0')) | > loss_gen: 1.9830702543258667 (2.086552076339722) | > loss_kl: 2.3984415531158447 (2.174381818771362) | > loss_feat: 3.7520010471343994 (3.613617115020752) | > loss_mel: 18.772705078125 (18.24618545532227) | > loss_duration: 1.5830966234207153 (1.5639094448089599) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.489315032958984 (27.684645690917968) | > grad_norm_1: tensor(4175.4443, device='cuda:0') (tensor(3097.1714, device='cuda:0')) | > current_lr_0: 0.00018806986619269772 | > current_lr_1: 0.00018806986619269772 | > step_time: 0.4334 (0.44277143478393555) | > loader_time: 0.009 (0.007192182540893555) --> EVAL PERFORMANCE | > avg_loader_time: 0.008781695030104949 (-3.2445074806746604e-05) | > avg_step_time: 0.45832523493699623 (+0.00023912711882251303) | > avg_loss_disc: 2.6991975441784932 (-0.033988150072768786) | > avg_loss_disc_real_0: 0.17647036374874517 (-0.02029517093594646) | > avg_loss_disc_real_1: 0.2241766333160266 (-0.010499490713569481) | > avg_loss_disc_real_2: 0.23905309489075566 (+0.0024787521278354374) | > avg_loss_disc_real_3: 0.23764850912799299 (-0.0047257888904759515) | > avg_loss_disc_real_4: 0.2474709348359578 (+0.006718441843986511) | > avg_loss_disc_real_5: 0.24477472397643077 (-0.0007650138626635883) | > avg_loss_0: 2.6991975441784932 (-0.033988150072768786) | > avg_grad_norm_0: tensor(38.0391, device='cuda:0') (tensor(-4.0049, device='cuda:0')) | > avg_loss_gen: 2.0111041572731994 (+0.06111234510448704) | > avg_loss_kl: 2.278483044933265 (-0.22713259576072176) | > avg_loss_feat: 3.240650227372076 (+0.20784628894967128) | > avg_loss_mel: 18.229643244138906 (-1.2151529822551126) | > avg_loss_duration: 1.605615347204074 (-0.024448863217528594) | > avg_amp_scaler: 23.661971830985916 (+7.661971830985916) | > avg_loss_1: 27.365496030995544 (-1.1977757467350472) | > avg_grad_norm_1: tensor(2784.3889, device='cuda:0') (+tensor(40.8376, device='cuda:0'))  > EPOCH: 493/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:16:38)   --> TIME: 2025-02-25 13:16:42 -- STEP: 3/72 -- GLOBAL_STEP: 1035500 | > loss_disc: 2.308131694793701 (2.433718283971151) | > loss_disc_real_0: 0.13965335488319397 (0.13576838870843252) | > loss_disc_real_1: 0.22843457758426666 (0.21409127116203308) | > loss_disc_real_2: 0.17849449813365936 (0.20214327176411948) | > loss_disc_real_3: 0.24535125494003296 (0.24503183861573538) | > loss_disc_real_4: 0.2317938357591629 (0.23963500062624613) | > loss_disc_real_5: 0.2567211389541626 (0.24843748907248178) | > loss_0: 2.308131694793701 (2.433718283971151) | > grad_norm_0: tensor(14.2710, device='cuda:0') (tensor(24.0009, device='cuda:0')) | > loss_gen: 2.601252555847168 (2.465897003809611) | > loss_kl: 2.080345869064331 (2.0846662521362305) | > loss_feat: 4.4350762367248535 (4.022692839304606) | > loss_mel: 18.451641082763672 (18.129549662272137) | > loss_duration: 1.6237640380859375 (1.5897815227508545) | > amp_scaler: 16.0 (21.333333333333332) | > loss_1: 29.192081451416016 (28.292587916056316) | > grad_norm_1: tensor(7221.5225, device='cuda:0') (tensor(3827.9304, device='cuda:0')) | > current_lr_0: 0.00018804635745942361 | > current_lr_1: 0.00018804635745942361 | > step_time: 0.4224 (0.42548100153605145) | > loader_time: 0.0059 (0.006677389144897461)  --> TIME: 2025-02-25 13:17:05 -- STEP: 53/72 -- GLOBAL_STEP: 1035550 | > loss_disc: 2.7185511589050293 (2.6696561912320695) | > loss_disc_real_0: 0.1341974139213562 (0.16550135907699473) | > loss_disc_real_1: 0.22207358479499817 (0.2252268136109946) | > loss_disc_real_2: 0.31817007064819336 (0.23362863738581818) | > loss_disc_real_3: 0.23287945985794067 (0.23670846729908349) | > loss_disc_real_4: 0.2445286065340042 (0.24149773705680416) | > loss_disc_real_5: 0.2477341592311859 (0.24638046436714675) | > loss_0: 2.7185511589050293 (2.6696561912320695) | > grad_norm_0: tensor(73.4620, device='cuda:0') (tensor(48.2202, device='cuda:0')) | > loss_gen: 2.057436466217041 (2.0222648742063996) | > loss_kl: 2.002711772918701 (2.073528543958124) | > loss_feat: 3.4728970527648926 (3.271664426011859) | > loss_mel: 17.228214263916016 (17.468143697054884) | > loss_duration: 1.6240531206130981 (1.5771212555327505) | > amp_scaler: 16.0 (16.301886792452837) | > loss_1: 26.385313034057617 (26.412722893480986) | > grad_norm_1: tensor(4042.2473, device='cuda:0') (tensor(3504.4150, device='cuda:0')) | > current_lr_0: 0.00018804635745942361 | > current_lr_1: 0.00018804635745942361 | > step_time: 0.4471 (0.4452652886228741) | > loader_time: 0.0092 (0.007989361601055794) --> EVAL PERFORMANCE | > avg_loader_time: 0.008886313774216345 (+0.00010461874411139653) | > avg_step_time: 0.4571183701636086 (-0.001206864773387606) | > avg_loss_disc: 2.670098731215571 (-0.02909881296292216) | > avg_loss_disc_real_0: 0.1651011220586132 (-0.011369241690131981) | > avg_loss_disc_real_1: 0.22567550754043417 (+0.0014988742244075592) | > avg_loss_disc_real_2: 0.23214795744754899 (-0.0069051374432066714) | > avg_loss_disc_real_3: 0.2377202227501802 (+7.171362218721944e-05) | > avg_loss_disc_real_4: 0.24129857038947897 (-0.00617236444647884) | > avg_loss_disc_real_5: 0.24637242543025756 (+0.0015977014538267886) | > avg_loss_0: 2.670098731215571 (-0.02909881296292216) | > avg_grad_norm_0: tensor(49.6113, device='cuda:0') (+tensor(11.5723, device='cuda:0')) | > avg_loss_gen: 2.009599877075411 (-0.0015042801977882547) | > avg_loss_kl: 2.0867723276917363 (-0.19171071724152888) | > avg_loss_feat: 3.217185587950156 (-0.02346463942192001) | > avg_loss_mel: 17.413299614275005 (-0.8163436298639013) | > avg_loss_duration: 1.5898655266828938 (-0.015749820521180125) | > avg_amp_scaler: 16.225352112676056 (-7.43661971830986) | > avg_loss_1: 26.31672295046524 (-1.048773080530303) | > avg_grad_norm_1: tensor(3560.7710, device='cuda:0') (+tensor(776.3821, device='cuda:0'))  > EPOCH: 494/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:17:15)   --> TIME: 2025-02-25 13:17:31 -- STEP: 31/72 -- GLOBAL_STEP: 1035600 | > loss_disc: 2.7700231075286865 (2.6456985550542034) | > loss_disc_real_0: 0.19563928246498108 (0.16917602501569257) | > loss_disc_real_1: 0.1948556751012802 (0.22300887011712597) | > loss_disc_real_2: 0.18909670412540436 (0.22825333283793542) | > loss_disc_real_3: 0.2627520263195038 (0.2342001449677252) | > loss_disc_real_4: 0.2550138831138611 (0.242987968267933) | > loss_disc_real_5: 0.24850204586982727 (0.24875597751909692) | > loss_0: 2.7700231075286865 (2.6456985550542034) | > grad_norm_0: tensor(36.5124, device='cuda:0') (tensor(50.1328, device='cuda:0')) | > loss_gen: 1.7979011535644531 (2.087302646329325) | > loss_kl: 2.1698057651519775 (1.9917881681073097) | > loss_feat: 3.1260387897491455 (3.4493408510761876) | > loss_mel: 17.44618034362793 (17.394922564106604) | > loss_duration: 1.595617413520813 (1.5493641399568128) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.135543823242188 (26.47271857723113) | > grad_norm_1: tensor(1954.0914, device='cuda:0') (tensor(4471.1577, device='cuda:0')) | > current_lr_0: 0.00018802285166474118 | > current_lr_1: 0.00018802285166474118 | > step_time: 0.4448 (0.43741765329914706) | > loader_time: 0.0086 (0.007110980249220325) --> EVAL PERFORMANCE | > avg_loader_time: 0.008609231089202451 (-0.00027708268501389424) | > avg_step_time: 0.45638113626292054 (-0.0007372339006880835) | > avg_loss_disc: 2.6462927704126065 (-0.0238059608029646) | > avg_loss_disc_real_0: 0.16314613367889969 (-0.0019549883797135015) | > avg_loss_disc_real_1: 0.2231791743510206 (-0.0024963331894135576) | > avg_loss_disc_real_2: 0.2228709725846707 (-0.009276984862878285) | > avg_loss_disc_real_3: 0.23314658101175872 (-0.00457364173842148) | > avg_loss_disc_real_4: 0.24214036888639692 (+0.0008417984969179559) | > avg_loss_disc_real_5: 0.24688633814663954 (+0.0005139127163819868) | > avg_loss_0: 2.6462927704126065 (-0.0238059608029646) | > avg_grad_norm_0: tensor(56.3823, device='cuda:0') (+tensor(6.7710, device='cuda:0')) | > avg_loss_gen: 2.0469439667715146 (+0.03734408969610348) | > avg_loss_kl: 2.0448012738160677 (-0.041971053875668574) | > avg_loss_feat: 3.3650030619661573 (+0.14781747401600143) | > avg_loss_mel: 17.262630959631686 (-0.1506686546433187) | > avg_loss_duration: 1.584505492532757 (-0.00536003415013675) | > avg_amp_scaler: 16.0 (-0.22535211267605604) | > avg_loss_1: 26.30388482859437 (-0.012838121870871078) | > avg_grad_norm_1: tensor(4503.6533, device='cuda:0') (+tensor(942.8823, device='cuda:0'))  > EPOCH: 495/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:17:53)   --> TIME: 2025-02-25 13:17:58 -- STEP: 9/72 -- GLOBAL_STEP: 1035650 | > loss_disc: 2.6061859130859375 (2.6262111398908825) | > loss_disc_real_0: 0.27522096037864685 (0.16201767408185536) | > loss_disc_real_1: 0.19611313939094543 (0.22167214254538217) | > loss_disc_real_2: 0.19223806262016296 (0.2527269803815418) | > loss_disc_real_3: 0.257892906665802 (0.2336399091614617) | > loss_disc_real_4: 0.24326056241989136 (0.2445691857073042) | > loss_disc_real_5: 0.24743658304214478 (0.24998118811183506) | > loss_0: 2.6061859130859375 (2.6262111398908825) | > grad_norm_0: tensor(93.4707, device='cuda:0') (tensor(70.1504, device='cuda:0')) | > loss_gen: 2.2009544372558594 (2.0970303350024753) | > loss_kl: 2.013579845428467 (1.9777889781528049) | > loss_feat: 3.3338260650634766 (3.513055271572537) | > loss_mel: 17.53383445739746 (17.295427534315323) | > loss_duration: 1.542614459991455 (1.556853559282091) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.62480926513672 (26.440155453152126) | > grad_norm_1: tensor(5678.0630, device='cuda:0') (tensor(4920.7280, device='cuda:0')) | > current_lr_0: 0.0001879993488082831 | > current_lr_1: 0.0001879993488082831 | > step_time: 0.4243 (0.4344144927130805) | > loader_time: 0.007 (0.007062885496351454)  --> TIME: 2025-02-25 13:18:21 -- STEP: 59/72 -- GLOBAL_STEP: 1035700 | > loss_disc: 2.8149166107177734 (2.641350996696343) | > loss_disc_real_0: 0.10183635354042053 (0.1625791581000312) | > loss_disc_real_1: 0.2345922589302063 (0.22343666422165046) | > loss_disc_real_2: 0.23272737860679626 (0.2345344131275759) | > loss_disc_real_3: 0.243202805519104 (0.2323795011488058) | > loss_disc_real_4: 0.2728074789047241 (0.24212330554501485) | > loss_disc_real_5: 0.27847978472709656 (0.24512227586770463) | > loss_0: 2.8149166107177734 (2.641350996696343) | > grad_norm_0: tensor(51.1571, device='cuda:0') (tensor(63.3779, device='cuda:0')) | > loss_gen: 1.8910521268844604 (2.051005185660669) | > loss_kl: 1.7983490228652954 (2.0106912750308794) | > loss_feat: 2.669124126434326 (3.3431177583791443) | > loss_mel: 15.644669532775879 (16.992612968056886) | > loss_duration: 1.6068717241287231 (1.5699609053336967) | > amp_scaler: 16.0 (16.0) | > loss_1: 23.61006736755371 (25.96738808842029) | > grad_norm_1: tensor(2746.4158, device='cuda:0') (tensor(4815.9180, device='cuda:0')) | > current_lr_0: 0.0001879993488082831 | > current_lr_1: 0.0001879993488082831 | > step_time: 0.4685 (0.4446005295898955) | > loader_time: 0.0097 (0.008077872001518634) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874605984754965 (+0.00013682875834719957) | > avg_step_time: 0.454388934122005 (-0.0019922021409155555) | > avg_loss_disc: 2.6442865317975968 (-0.0020062386150097034) | > avg_loss_disc_real_0: 0.16363416034990635 (+0.0004880266710066694) | > avg_loss_disc_real_1: 0.2220899651587849 (-0.0010892091922357072) | > avg_loss_disc_real_2: 0.23308701099644244 (+0.010216038411771738) | > avg_loss_disc_real_3: 0.23268541420849276 (-0.0004611668032659666) | > avg_loss_disc_real_4: 0.2425259572099632 (+0.00038558832356627715) | > avg_loss_disc_real_5: 0.24622965068884298 (-0.0006566874577965676) | > avg_loss_0: 2.6442865317975968 (-0.0020062386150097034) | > avg_grad_norm_0: tensor(64.2028, device='cuda:0') (+tensor(7.8205, device='cuda:0')) | > avg_loss_gen: 2.040841428326889 (-0.006102538444625605) | > avg_loss_kl: 2.0387459271390678 (-0.006055346676999918) | > avg_loss_feat: 3.304426972295197 (-0.060576089670960176) | > avg_loss_mel: 16.990063627001266 (-0.27256733263041966) | > avg_loss_duration: 1.5800293465735207 (-0.0044761459592363995) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.9541072845459 (-0.3497775440484716) | > avg_grad_norm_1: tensor(4812.3340, device='cuda:0') (+tensor(308.6807, device='cuda:0'))  > EPOCH: 496/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:18:30)   --> TIME: 2025-02-25 13:18:48 -- STEP: 37/72 -- GLOBAL_STEP: 1035750 | > loss_disc: 2.5886147022247314 (2.642925668407131) | > loss_disc_real_0: 0.21752020716667175 (0.16435870226170565) | > loss_disc_real_1: 0.13107708096504211 (0.21706066018826253) | > loss_disc_real_2: 0.2137356847524643 (0.22980508047181206) | > loss_disc_real_3: 0.200277179479599 (0.23118337143111872) | > loss_disc_real_4: 0.21882319450378418 (0.2404928936346157) | > loss_disc_real_5: 0.2252323031425476 (0.24414336802186193) | > loss_0: 2.5886147022247314 (2.642925668407131) | > grad_norm_0: tensor(71.6037, device='cuda:0') (tensor(61.8380, device='cuda:0')) | > loss_gen: 2.252974510192871 (2.048195436194136) | > loss_kl: 1.8341912031173706 (2.009808459797421) | > loss_feat: 3.8272101879119873 (3.3675825144793534) | > loss_mel: 18.289457321166992 (17.133024447673076) | > loss_duration: 1.5738630294799805 (1.5449606309065949) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.777698516845703 (26.10357140206002) | > grad_norm_1: tensor(6399.6606, device='cuda:0') (tensor(4774.0083, device='cuda:0')) | > current_lr_0: 0.00018797584888968205 | > current_lr_1: 0.00018797584888968205 | > step_time: 0.4569 (0.4412715757215345) | > loader_time: 0.0078 (0.0073015174350223025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008761731671615383 (+1.5671824065732406e-05) | > avg_step_time: 0.45761294096288546 (+0.003224006840880478) | > avg_loss_disc: 2.657155507047411 (+0.01286897524981434) | > avg_loss_disc_real_0: 0.1661689911090152 (+0.002534830759108847) | > avg_loss_disc_real_1: 0.22513569030963201 (+0.0030457251508471117) | > avg_loss_disc_real_2: 0.22797713229354 (-0.005109878702902443) | > avg_loss_disc_real_3: 0.2340515520371182 (+0.0013661378286254355) | > avg_loss_disc_real_4: 0.2403639601150029 (-0.0021619970949602885) | > avg_loss_disc_real_5: 0.24270387476598712 (-0.003525775922855856) | > avg_loss_0: 2.657155507047411 (+0.01286897524981434) | > avg_grad_norm_0: tensor(65.5440, device='cuda:0') (+tensor(1.3411, device='cuda:0')) | > avg_loss_gen: 2.033476188149249 (-0.0073652401776400644) | > avg_loss_kl: 2.048032077265458 (+0.009286150126390336) | > avg_loss_feat: 3.3005349434597386 (-0.003892028835458472) | > avg_loss_mel: 17.042251801826573 (+0.05218817482530724) | > avg_loss_duration: 1.5768253366712113 (-0.0032040099023094015) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.00112031211316 (+0.047013027567260224) | > avg_grad_norm_1: tensor(4969.1948, device='cuda:0') (+tensor(156.8608, device='cuda:0'))  > EPOCH: 497/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:19:07)   --> TIME: 2025-02-25 13:19:15 -- STEP: 15/72 -- GLOBAL_STEP: 1035800 | > loss_disc: 2.739474296569824 (2.607699712117513) | > loss_disc_real_0: 0.1671695113182068 (0.1532060171167056) | > loss_disc_real_1: 0.22402311861515045 (0.22114962736765545) | > loss_disc_real_2: 0.19382664561271667 (0.21622583170731863) | > loss_disc_real_3: 0.23273010551929474 (0.23756738503774008) | > loss_disc_real_4: 0.22775469720363617 (0.2429018348455429) | > loss_disc_real_5: 0.2422638088464737 (0.24773781696955363) | > loss_0: 2.739474296569824 (2.607699712117513) | > grad_norm_0: tensor(49.4314, device='cuda:0') (tensor(65.9704, device='cuda:0')) | > loss_gen: 2.016300916671753 (2.099904481569926) | > loss_kl: 2.0066449642181396 (1.9593955278396606) | > loss_feat: 3.3368892669677734 (3.558469867706299) | > loss_mel: 17.935657501220703 (17.22533473968506) | > loss_duration: 1.5135818719863892 (1.5257078647613525) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.80907440185547 (26.36881294250488) | > grad_norm_1: tensor(4764.7065, device='cuda:0') (tensor(5609.5498, device='cuda:0')) | > current_lr_0: 0.00018795235190857083 | > current_lr_1: 0.00018795235190857083 | > step_time: 0.4372 (0.4353111426035563) | > loader_time: 0.006 (0.0068898836771647135)  --> TIME: 2025-02-25 13:19:39 -- STEP: 65/72 -- GLOBAL_STEP: 1035850 | > loss_disc: 2.6312432289123535 (2.6457120455228367) | > loss_disc_real_0: 0.18330708146095276 (0.160427103134302) | > loss_disc_real_1: 0.20898059010505676 (0.22303749781388502) | > loss_disc_real_2: 0.2845260202884674 (0.22565994698267716) | > loss_disc_real_3: 0.25028282403945923 (0.23642374529288365) | > loss_disc_real_4: 0.2647147476673126 (0.2434454218699382) | > loss_disc_real_5: 0.23387454450130463 (0.24630294556801136) | > loss_0: 2.6312432289123535 (2.6457120455228367) | > grad_norm_0: tensor(116.1067, device='cuda:0') (tensor(70.5842, device='cuda:0')) | > loss_gen: 2.1985268592834473 (2.074953910020682) | > loss_kl: 2.258775472640991 (2.0573897141676674) | > loss_feat: 3.379410743713379 (3.4220166169680084) | > loss_mel: 17.095413208007812 (17.051538643470177) | > loss_duration: 1.6378945112228394 (1.57124589406527) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.57002067565918 (26.17714494558481) | > grad_norm_1: tensor(6337.8828, device='cuda:0') (tensor(5572.2793, device='cuda:0')) | > current_lr_0: 0.00018795235190857083 | > current_lr_1: 0.00018795235190857083 | > step_time: 0.4945 (0.4514312927539532) | > loader_time: 0.0121 (0.008862363375150242) --> EVAL PERFORMANCE | > avg_loader_time: 0.009130017858156016 (+0.00036828618654063255) | > avg_step_time: 0.4555099211948019 (-0.0021030197680835805) | > avg_loss_disc: 2.6495489939837387 (-0.007606513063672438) | > avg_loss_disc_real_0: 0.16075027366759073 (-0.00541871744142447) | > avg_loss_disc_real_1: 0.22400484622364314 (-0.0011308440859888746) | > avg_loss_disc_real_2: 0.22567260055474833 (-0.0023045317387916653) | > avg_loss_disc_real_3: 0.23695212098914134 (+0.002900568952023147) | > avg_loss_disc_real_4: 0.24300986234570893 (+0.0026459022307060154) | > avg_loss_disc_real_5: 0.24619368159435165 (+0.003489806828364528) | > avg_loss_0: 2.6495489939837387 (-0.007606513063672438) | > avg_grad_norm_0: tensor(69.3915, device='cuda:0') (+tensor(3.8476, device='cuda:0')) | > avg_loss_gen: 2.065326455613258 (+0.0318502674640091) | > avg_loss_kl: 2.0721715772655642 (+0.024139500000106118) | > avg_loss_feat: 3.392431833374668 (+0.0918968899149295) | > avg_loss_mel: 17.089166587507222 (+0.04691478568064866) | > avg_loss_duration: 1.5769206738807786 (+9.53372095673366e-05) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.19601730561592 (+0.19489699350276268) | > avg_grad_norm_1: tensor(5464.9771, device='cuda:0') (+tensor(495.7822, device='cuda:0'))  > EPOCH: 498/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:19:44)   --> TIME: 2025-02-25 13:20:05 -- STEP: 43/72 -- GLOBAL_STEP: 1035900 | > loss_disc: 2.75007700920105 (2.6983144615971764) | > loss_disc_real_0: 0.13591349124908447 (0.1634842762073805) | > loss_disc_real_1: 0.21052438020706177 (0.23107504220895989) | > loss_disc_real_2: 0.23398306965827942 (0.23534524821957878) | > loss_disc_real_3: 0.22435463964939117 (0.23961547918097917) | > loss_disc_real_4: 0.2308148741722107 (0.24304437706636828) | > loss_disc_real_5: 0.22191409766674042 (0.2456867916639461) | > loss_0: 2.75007700920105 (2.6983144615971764) | > grad_norm_0: tensor(43.5233, device='cuda:0') (tensor(66.5921, device='cuda:0')) | > loss_gen: 1.912874698638916 (2.0571702081103656) | > loss_kl: 2.0155951976776123 (2.086393192757008) | > loss_feat: 3.1423110961914062 (3.4143538752267526) | > loss_mel: 16.243038177490234 (17.220460270726402) | > loss_duration: 1.5845253467559814 (1.5533440944760344) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.898344039916992 (26.331721549810364) | > grad_norm_1: tensor(2658.9802, device='cuda:0') (tensor(5320.1299, device='cuda:0')) | > current_lr_0: 0.00018792885786458223 | > current_lr_1: 0.00018792885786458223 | > step_time: 0.4509 (0.43997615437174953) | > loader_time: 0.0098 (0.0076325216958689135) --> EVAL PERFORMANCE | > avg_loader_time: 0.008691693695498184 (-0.000438324162657832) | > avg_step_time: 0.45398490529664803 (-0.0015250158981538475) | > avg_loss_disc: 2.6961483149461345 (+0.04659932096239583) | > avg_loss_disc_real_0: 0.17123033851385117 (+0.010480064846260434) | > avg_loss_disc_real_1: 0.2300236160066766 (+0.006018769783033451) | > avg_loss_disc_real_2: 0.23768997654108934 (+0.012017375986341006) | > avg_loss_disc_real_3: 0.24227573292356142 (+0.005323611934420075) | > avg_loss_disc_real_4: 0.2445575479470508 (+0.001547685601341886) | > avg_loss_disc_real_5: 0.24724140511432163 (+0.0010477235199699841) | > avg_loss_0: 2.6961483149461345 (+0.04659932096239583) | > avg_grad_norm_0: tensor(67.1925, device='cuda:0') (tensor(-2.1990, device='cuda:0')) | > avg_loss_gen: 2.0608174347541697 (-0.004509020859088331) | > avg_loss_kl: 2.1055233310645733 (+0.03335175379900912) | > avg_loss_feat: 3.3562021927094796 (-0.03622964066518852) | > avg_loss_mel: 17.180213659582005 (+0.09104707207478313) | > avg_loss_duration: 1.577650550385596 (+0.0007298765048173728) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.280407193680883 (+0.08438988806496184) | > avg_grad_norm_1: tensor(5164.0732, device='cuda:0') (tensor(-300.9038, device='cuda:0'))  > EPOCH: 499/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:20:21)   --> TIME: 2025-02-25 13:20:32 -- STEP: 21/72 -- GLOBAL_STEP: 1035950 | > loss_disc: 2.8032422065734863 (2.6499237105959934) | > loss_disc_real_0: 0.41181790828704834 (0.17997182231573833) | > loss_disc_real_1: 0.36555129289627075 (0.23454627323718297) | > loss_disc_real_2: 0.25568512082099915 (0.21667711011001042) | > loss_disc_real_3: 0.2421177476644516 (0.23248965115774245) | > loss_disc_real_4: 0.231906458735466 (0.24515803371156966) | > loss_disc_real_5: 0.24264834821224213 (0.24914832626070296) | > loss_0: 2.8032422065734863 (2.6499237105959934) | > grad_norm_0: tensor(82.1497, device='cuda:0') (tensor(74.8108, device='cuda:0')) | > loss_gen: 2.588878631591797 (2.1023766597112026) | > loss_kl: 2.0641236305236816 (2.0941841034662154) | > loss_feat: 3.6334900856018066 (3.3343566712879) | > loss_mel: 16.22787094116211 (17.37006105695452) | > loss_duration: 1.569596290588379 (1.5409439348039173) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.08395767211914 (26.441922505696613) | > grad_norm_1: tensor(6980.6665, device='cuda:0') (tensor(5541.2573, device='cuda:0')) | > current_lr_0: 0.00018790536675734917 | > current_lr_1: 0.00018790536675734917 | > step_time: 0.4417 (0.4575649556659517) | > loader_time: 0.0063 (0.006734609603881836)  --> TIME: 2025-02-25 13:20:57 -- STEP: 71/72 -- GLOBAL_STEP: 1036000 | > loss_disc: 2.7453718185424805 (2.706028505110405) | > loss_disc_real_0: 0.144819974899292 (0.17927902643109705) | > loss_disc_real_1: 0.22324314713478088 (0.2351981781318154) | > loss_disc_real_2: 0.19801738858222961 (0.23168339569803695) | > loss_disc_real_3: 0.19257405400276184 (0.2346383488514054) | > loss_disc_real_4: 0.21196717023849487 (0.24288627043576308) | > loss_disc_real_5: 0.20666107535362244 (0.24759435212947942) | > loss_0: 2.7453718185424805 (2.706028505110405) | > grad_norm_0: tensor(55.5961, device='cuda:0') (tensor(67.4869, device='cuda:0')) | > loss_gen: 1.6885473728179932 (2.0163259153634723) | > loss_kl: 2.3026883602142334 (2.242575343226043) | > loss_feat: 3.022275686264038 (3.215821578469075) | > loss_mel: 18.715600967407227 (17.86405316205091) | > loss_duration: 1.624631404876709 (1.604296301452207) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.353744506835938 (26.943072332462794) | > grad_norm_1: tensor(4314.4438, device='cuda:0') (tensor(4822.6689, device='cuda:0')) | > current_lr_0: 0.00018790536675734917 | > current_lr_1: 0.00018790536675734917 | > step_time: 0.486 (0.46637598561569) | > loader_time: 0.008 (0.008848784675060867) --> EVAL PERFORMANCE | > avg_loader_time: 0.008848784675060867 (+0.00015709097956268306) | > avg_step_time: 0.46637598561569 (+0.012391080319041958) | > avg_loss_disc: 2.706028505110405 (+0.009880190164270441) | > avg_loss_disc_real_0: 0.17927902643109705 (+0.00804868791724589) | > avg_loss_disc_real_1: 0.2351981781318154 (+0.00517456212513881) | > avg_loss_disc_real_2: 0.23168339569803695 (-0.006006580843052389) | > avg_loss_disc_real_3: 0.2346383488514054 (-0.007637384072156028) | > avg_loss_disc_real_4: 0.24288627043576308 (-0.0016712775112877332) | > avg_loss_disc_real_5: 0.24759435212947942 (+0.00035294701515778404) | > avg_loss_0: 2.706028505110405 (+0.009880190164270441) | > avg_grad_norm_0: tensor(67.4869, device='cuda:0') (+tensor(0.2944, device='cuda:0')) | > avg_loss_gen: 2.0163259153634723 (-0.0444915193906974) | > avg_loss_kl: 2.242575343226043 (+0.13705201216146978) | > avg_loss_feat: 3.215821578469075 (-0.1403806142404047) | > avg_loss_mel: 17.86405316205091 (+0.6838395024689063) | > avg_loss_duration: 1.604296301452207 (+0.026645751066610934) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.943072332462794 (+0.6626651387819109) | > avg_grad_norm_1: tensor(4822.6689, device='cuda:0') (tensor(-341.4043, device='cuda:0'))  > EPOCH: 500/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:20:58)   --> TIME: 2025-02-25 13:21:23 -- STEP: 49/72 -- GLOBAL_STEP: 1036050 | > loss_disc: 2.7457337379455566 (2.7437863641855667) | > loss_disc_real_0: 0.24138596653938293 (0.18513159104147736) | > loss_disc_real_1: 0.20339491963386536 (0.23322033182698854) | > loss_disc_real_2: 0.18063631653785706 (0.2358255574897844) | > loss_disc_real_3: 0.23834313452243805 (0.24444991684689812) | > loss_disc_real_4: 0.22694605588912964 (0.2448078223637172) | > loss_disc_real_5: 0.2197311520576477 (0.24763285840044214) | > loss_0: 2.7457337379455566 (2.7437863641855667) | > grad_norm_0: tensor(43.3922, device='cuda:0') (tensor(49.1199, device='cuda:0')) | > loss_gen: 1.8136361837387085 (1.9202849329734335) | > loss_kl: 2.1794238090515137 (2.1371167119668444) | > loss_feat: 2.9043617248535156 (3.0577558546650168) | > loss_mel: 17.856943130493164 (17.53552417365872) | > loss_duration: 1.6537278890609741 (1.5723145008087158) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.40809440612793 (26.22299626408791) | > grad_norm_1: tensor(1573.3621, device='cuda:0') (tensor(3543.8931, device='cuda:0')) | > current_lr_0: 0.00018788187858650448 | > current_lr_1: 0.00018788187858650448 | > step_time: 0.4386 (0.4464623733442657) | > loader_time: 0.0097 (0.00778774339325574) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669312571136048 (-0.00017947210392481837) | > avg_step_time: 0.45750252965470434 (-0.008873455960985654) | > avg_loss_disc: 2.725757793641426 (+0.019729288531021183) | > avg_loss_disc_real_0: 0.1828502285963213 (+0.0035712021652242465) | > avg_loss_disc_real_1: 0.23337908802737653 (-0.0018190901044388696) | > avg_loss_disc_real_2: 0.23467352566584734 (+0.0029901299678103954) | > avg_loss_disc_real_3: 0.24181307579430056 (+0.007174726942895171) | > avg_loss_disc_real_4: 0.24463809250106275 (+0.0017518220652996686) | > avg_loss_disc_real_5: 0.24734760044326246 (-0.00024675168621696164) | > avg_loss_0: 2.725757793641426 (+0.019729288531021183) | > avg_grad_norm_0: tensor(53.7245, device='cuda:0') (tensor(-13.7624, device='cuda:0')) | > avg_loss_gen: 1.962963539110103 (-0.05336237625336926) | > avg_loss_kl: 2.152500392685473 (-0.09007495054057024) | > avg_loss_feat: 3.101268160511071 (-0.11455341795800411) | > avg_loss_mel: 17.478793775531614 (-0.3852593865192979) | > avg_loss_duration: 1.5876269978536683 (-0.016669303598538576) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.283152996654245 (-0.6599193358085493) | > avg_grad_norm_1: tensor(3916.7593, device='cuda:0') (tensor(-905.9097, device='cuda:0'))  > EPOCH: 501/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:21:36)   --> TIME: 2025-02-25 13:21:49 -- STEP: 27/72 -- GLOBAL_STEP: 1036100 | > loss_disc: 2.7456324100494385 (2.6809924002046936) | > loss_disc_real_0: 0.15342408418655396 (0.16343781365840523) | > loss_disc_real_1: 0.2382955700159073 (0.2256042786218502) | > loss_disc_real_2: 0.3345591425895691 (0.22942103114393023) | > loss_disc_real_3: 0.2509967088699341 (0.23298707769976723) | > loss_disc_real_4: 0.25655096769332886 (0.24157289388003172) | > loss_disc_real_5: 0.272696316242218 (0.24447206214622216) | > loss_0: 2.7456324100494385 (2.6809924002046936) | > grad_norm_0: tensor(88.1214, device='cuda:0') (tensor(63.9852, device='cuda:0')) | > loss_gen: 1.9491935968399048 (1.9935486802348383) | > loss_kl: 2.3016250133514404 (2.0841328567928743) | > loss_feat: 2.844940185546875 (3.30127955366064) | > loss_mel: 15.712679862976074 (17.063183749163592) | > loss_duration: 1.5449671745300293 (1.5435387072739777) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.35340690612793 (25.98568379437482) | > grad_norm_1: tensor(4158.1685, device='cuda:0') (tensor(4948.1777, device='cuda:0')) | > current_lr_0: 0.00018785839335168115 | > current_lr_1: 0.00018785839335168115 | > step_time: 0.4347 (0.43544853175127946) | > loader_time: 0.0081 (0.0067857106526692705) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624886123227402 (-4.4426447908646124e-05) | > avg_step_time: 0.456044056046177 (-0.0014584736085273553) | > avg_loss_disc: 2.6836782106211485 (-0.04207958302027759) | > avg_loss_disc_real_0: 0.15929499536123068 (-0.023555233235090622) | > avg_loss_disc_real_1: 0.2269542093428088 (-0.006424878684567731) | > avg_loss_disc_real_2: 0.2301610099597716 (-0.004512515706075748) | > avg_loss_disc_real_3: 0.2390551680410412 (-0.0027579077532593477) | > avg_loss_disc_real_4: 0.24373666690268986 (-0.0009014255983728903) | > avg_loss_disc_real_5: 0.2468278857184128 (-0.000519714724849657) | > avg_loss_0: 2.6836782106211485 (-0.04207958302027759) | > avg_grad_norm_0: tensor(68.4048, device='cuda:0') (+tensor(14.6803, device='cuda:0')) | > avg_loss_gen: 2.006186450031442 (+0.0432229109213389) | > avg_loss_kl: 2.116186061375578 (-0.03631433130989503) | > avg_loss_feat: 3.2441541913529517 (+0.1428860308418809) | > avg_loss_mel: 17.118841345881073 (-0.35995242965054075) | > avg_loss_duration: 1.5823357138835208 (-0.005291283970147553) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.06770399926414 (-0.21544899739010503) | > avg_grad_norm_1: tensor(5041.5078, device='cuda:0') (+tensor(1124.7485, device='cuda:0'))  > EPOCH: 502/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:22:12)   --> TIME: 2025-02-25 13:22:17 -- STEP: 5/72 -- GLOBAL_STEP: 1036150 | > loss_disc: 2.7512965202331543 (2.5778048992156983) | > loss_disc_real_0: 0.08097301423549652 (0.13962790369987488) | > loss_disc_real_1: 0.2158278375864029 (0.22968721091747285) | > loss_disc_real_2: 0.23483838140964508 (0.20837628245353698) | > loss_disc_real_3: 0.24108974635601044 (0.24058058857917786) | > loss_disc_real_4: 0.24790284037590027 (0.2403905898332596) | > loss_disc_real_5: 0.23563311994075775 (0.23266348242759705) | > loss_0: 2.7512965202331543 (2.5778048992156983) | > grad_norm_0: tensor(74.7560, device='cuda:0') (tensor(51.7042, device='cuda:0')) | > loss_gen: 1.908107876777649 (2.177027869224548) | > loss_kl: 2.014662981033325 (2.006639909744263) | > loss_feat: 3.2870264053344727 (3.680359363555908) | > loss_mel: 17.4012451171875 (17.64307975769043) | > loss_duration: 1.5230557918548584 (1.550624942779541) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.134098052978516 (27.05773162841797) | > grad_norm_1: tensor(6291.0132, device='cuda:0') (tensor(6451.9224, device='cuda:0')) | > current_lr_0: 0.0001878349110525122 | > current_lr_1: 0.0001878349110525122 | > step_time: 0.4372 (0.4358072280883789) | > loader_time: 0.0061 (0.006589698791503906)  --> TIME: 2025-02-25 13:22:39 -- STEP: 55/72 -- GLOBAL_STEP: 1036200 | > loss_disc: 2.9862923622131348 (2.7618407899683173) | > loss_disc_real_0: 0.38402771949768066 (0.17480940168554132) | > loss_disc_real_1: 0.2695470452308655 (0.24056600020690397) | > loss_disc_real_2: 0.2742858827114105 (0.2418739996173165) | > loss_disc_real_3: 0.27350953221321106 (0.2345746782692996) | > loss_disc_real_4: 0.2660330832004547 (0.24623538228598507) | > loss_disc_real_5: 0.24811583757400513 (0.24909739087928426) | > loss_0: 2.9862923622131348 (2.7618407899683173) | > grad_norm_0: tensor(31.9400, device='cuda:0') (tensor(54.4418, device='cuda:0')) | > loss_gen: 1.8185930252075195 (2.0549109328876836) | > loss_kl: 2.349436044692993 (2.6063198891553006) | > loss_feat: 2.5248680114746094 (3.374472678791393) | > loss_mel: 18.72574234008789 (19.278382492065425) | > loss_duration: 1.6574987173080444 (1.604109438982877) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.076139450073242 (28.918195377696644) | > grad_norm_1: tensor(463.7617, device='cuda:0') (tensor(4397.2554, device='cuda:0')) | > current_lr_0: 0.0001878349110525122 | > current_lr_1: 0.0001878349110525122 | > step_time: 0.4673 (0.44176379550587047) | > loader_time: 0.0092 (0.007923919504339043) --> EVAL PERFORMANCE | > avg_loader_time: 0.008821353106431559 (+0.0001964669832041563) | > avg_step_time: 0.4535336863826698 (-0.002510369663507206) | > avg_loss_disc: 2.761321423758923 (+0.07764321313777467) | > avg_loss_disc_real_0: 0.1726180270943843 (+0.013323031733153634) | > avg_loss_disc_real_1: 0.2396860688295163 (+0.012731859486707492) | > avg_loss_disc_real_2: 0.24008720231727815 (+0.009926192357506552) | > avg_loss_disc_real_3: 0.2351691483611792 (-0.0038860196798620184) | > avg_loss_disc_real_4: 0.24553244676388486 (+0.0017957798611950015) | > avg_loss_disc_real_5: 0.24912360751293075 (+0.002295721794517952) | > avg_loss_0: 2.761321423758923 (+0.07764321313777467) | > avg_grad_norm_0: tensor(52.7993, device='cuda:0') (tensor(-15.6055, device='cuda:0')) | > avg_loss_gen: 2.0042498397155546 (-0.0019366103158873216) | > avg_loss_kl: 2.8006831944828297 (+0.6844971331072518) | > avg_loss_feat: 3.238995632655184 (-0.005158558697767823) | > avg_loss_mel: 19.77359873811963 (+2.654757392238558) | > avg_loss_duration: 1.7008299323874463 (+0.1184942185039255) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.518357236620407 (+3.450653237356267) | > avg_grad_norm_1: tensor(4256.6338, device='cuda:0') (tensor(-784.8740, device='cuda:0'))  > EPOCH: 503/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:22:50)   --> TIME: 2025-02-25 13:23:07 -- STEP: 33/72 -- GLOBAL_STEP: 1036250 | > loss_disc: 2.6313838958740234 (2.5981473200248955) | > loss_disc_real_0: 0.18021836876869202 (0.15424152305631927) | > loss_disc_real_1: 0.2405846118927002 (0.22517621878421668) | > loss_disc_real_2: 0.19000138342380524 (0.2284955486203685) | > loss_disc_real_3: 0.1902243196964264 (0.22595721167145352) | > loss_disc_real_4: 0.24054674804210663 (0.23953108895908704) | > loss_disc_real_5: 0.24410460889339447 (0.23607187469800314) | > loss_0: 2.6313838958740234 (2.5981473200248955) | > grad_norm_0: tensor(42.8211, device='cuda:0') (tensor(54.4332, device='cuda:0')) | > loss_gen: 1.856947422027588 (2.0647048480582955) | > loss_kl: 2.209975481033325 (2.4670717788465097) | > loss_feat: 3.612846612930298 (3.4989709492885703) | > loss_mel: 17.91753578186035 (19.233295036084723) | > loss_duration: 1.660897970199585 (1.7034299879363088) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.258203506469727 (28.967472712198894) | > grad_norm_1: tensor(5225.0381, device='cuda:0') (tensor(5315.5342, device='cuda:0')) | > current_lr_0: 0.0001878114316886306 | > current_lr_1: 0.0001878114316886306 | > step_time: 0.4521 (0.4394517811861905) | > loader_time: 0.0078 (0.007083560481215968) --> EVAL PERFORMANCE | > avg_loader_time: 0.008766859350070148 (-5.449375636141021e-05) | > avg_step_time: 0.45767784118652344 (+0.004144154803853661) | > avg_loss_disc: 2.6415928081727365 (-0.11972861558618675) | > avg_loss_disc_real_0: 0.16059469232256984 (-0.01202333477181447) | > avg_loss_disc_real_1: 0.22121560615552982 (-0.01847046267398647) | > avg_loss_disc_real_2: 0.230854890715908 (-0.009232311601370141) | > avg_loss_disc_real_3: 0.23459213699253512 (-0.0005770113686440759) | > avg_loss_disc_real_4: 0.2433674001777676 (-0.0021650465861172696) | > avg_loss_disc_real_5: 0.24256200374851764 (-0.006561603764413115) | > avg_loss_0: 2.6415928081727365 (-0.11972861558618675) | > avg_grad_norm_0: tensor(58.9216, device='cuda:0') (+tensor(6.1223, device='cuda:0')) | > avg_loss_gen: 2.01853085403711 (+0.014281014321555485) | > avg_loss_kl: 2.3503616930733267 (-0.450321501409503) | > avg_loss_feat: 3.3004100725684364 (+0.06141443991325257) | > avg_loss_mel: 18.438006172717458 (-1.3355925654021732) | > avg_loss_duration: 1.6898500230950368 (-0.01097990929240944) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.797158953169703 (-1.721198283450704) | > avg_grad_norm_1: tensor(5164.8569, device='cuda:0') (+tensor(908.2231, device='cuda:0'))  > EPOCH: 504/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:23:27)   --> TIME: 2025-02-25 13:23:34 -- STEP: 11/72 -- GLOBAL_STEP: 1036300 | > loss_disc: 2.6858630180358887 (2.6968279751864346) | > loss_disc_real_0: 0.1812180131673813 (0.14661071788180957) | > loss_disc_real_1: 0.25618308782577515 (0.23013996400616385) | > loss_disc_real_2: 0.1816318929195404 (0.20391875979575244) | > loss_disc_real_3: 0.22115270793437958 (0.2368693541396748) | > loss_disc_real_4: 0.24718594551086426 (0.254495080221783) | > loss_disc_real_5: 0.2602085769176483 (0.25842311030084436) | > loss_0: 2.6858630180358887 (2.6968279751864346) | > grad_norm_0: tensor(12.7189, device='cuda:0') (tensor(63.3347, device='cuda:0')) | > loss_gen: 2.0446863174438477 (1.9618694132024592) | > loss_kl: 2.004136323928833 (2.0893734151666816) | > loss_feat: 2.9656009674072266 (3.349363175305453) | > loss_mel: 17.5452880859375 (17.715088757601652) | > loss_duration: 1.576144814491272 (1.5991924459284002) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.13585662841797 (26.714887445623223) | > grad_norm_1: tensor(5119.9072, device='cuda:0') (tensor(5717.8936, device='cuda:0')) | > current_lr_0: 0.0001877879552596695 | > current_lr_1: 0.0001877879552596695 | > step_time: 0.4344 (0.43505900556390936) | > loader_time: 0.0066 (0.006408214569091797)  --> TIME: 2025-02-25 13:23:57 -- STEP: 61/72 -- GLOBAL_STEP: 1036350 | > loss_disc: 2.7315571308135986 (2.7067555716780367) | > loss_disc_real_0: 0.16134610772132874 (0.16106374354147518) | > loss_disc_real_1: 0.22398552298545837 (0.22411540370495592) | > loss_disc_real_2: 0.22112342715263367 (0.24506863829542383) | > loss_disc_real_3: 0.19773025810718536 (0.2331804420615806) | > loss_disc_real_4: 0.2683866620063782 (0.24469260822554104) | > loss_disc_real_5: 0.2567635476589203 (0.24555902647190406) | > loss_0: 2.7315571308135986 (2.7067555716780367) | > grad_norm_0: tensor(92.8517, device='cuda:0') (tensor(62.8070, device='cuda:0')) | > loss_gen: 1.9642068147659302 (1.9671532970960024) | > loss_kl: 2.1181180477142334 (2.120719454327568) | > loss_feat: 2.8677287101745605 (3.223891402854294) | > loss_mel: 16.787607192993164 (17.589644385165855) | > loss_duration: 1.6300103664398193 (1.6025386657871186) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.3676700592041 (26.503947023485527) | > grad_norm_1: tensor(5264.7788, device='cuda:0') (tensor(4954.4819, device='cuda:0')) | > current_lr_0: 0.0001877879552596695 | > current_lr_1: 0.0001877879552596695 | > step_time: 0.4708 (0.44919113252983717) | > loader_time: 0.0112 (0.008284865832719647) --> EVAL PERFORMANCE | > avg_loader_time: 0.008858546404771402 (+9.168705470125396e-05) | > avg_step_time: 0.4575258476633421 (-0.0001519935231813485) | > avg_loss_disc: 2.7231123648898703 (+0.08151955671713385) | > avg_loss_disc_real_0: 0.16474517613229614 (+0.004150483809726296) | > avg_loss_disc_real_1: 0.23051649436984264 (+0.009300888214312825) | > avg_loss_disc_real_2: 0.2444226133151793 (+0.013567722599271304) | > avg_loss_disc_real_3: 0.23402032088225996 (-0.0005718161102751529) | > avg_loss_disc_real_4: 0.24514658875028852 (+0.0017791885725209344) | > avg_loss_disc_real_5: 0.24683473311679463 (+0.004272729368276995) | > avg_loss_0: 2.7231123648898703 (+0.08151955671713385) | > avg_grad_norm_0: tensor(61.2224, device='cuda:0') (+tensor(2.3008, device='cuda:0')) | > avg_loss_gen: 1.9627635361443103 (-0.05576731789279976) | > avg_loss_kl: 2.1287546762278384 (-0.22160701684548822) | > avg_loss_feat: 3.1746436105647557 (-0.1257664620036807) | > avg_loss_mel: 17.57163045104121 (-0.8663757216762491) | > avg_loss_duration: 1.608725190162659 (-0.08112483293237793) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.446517299598373 (-1.35064165357133) | > avg_grad_norm_1: tensor(4793.1753, device='cuda:0') (tensor(-371.6816, device='cuda:0'))  > EPOCH: 505/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:24:04)   --> TIME: 2025-02-25 13:24:24 -- STEP: 39/72 -- GLOBAL_STEP: 1036400 | > loss_disc: 2.5458993911743164 (2.709573690707867) | > loss_disc_real_0: 0.12672710418701172 (0.16010639214744934) | > loss_disc_real_1: 0.2095137983560562 (0.21917770745662543) | > loss_disc_real_2: 0.21807903051376343 (0.24629668662181267) | > loss_disc_real_3: 0.23433025181293488 (0.23659520959242797) | > loss_disc_real_4: 0.24754633009433746 (0.2424321323633194) | > loss_disc_real_5: 0.2534787058830261 (0.2459361725128614) | > loss_0: 2.5458993911743164 (2.709573690707867) | > grad_norm_0: tensor(82.6008, device='cuda:0') (tensor(60.4555, device='cuda:0')) | > loss_gen: 2.177736520767212 (1.9643094356243427) | > loss_kl: 2.620847225189209 (2.073773686702434) | > loss_feat: 3.804802894592285 (3.2838227320940065) | > loss_mel: 18.958576202392578 (17.568805694580078) | > loss_duration: 1.610535740852356 (1.5672880319448619) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.17249870300293 (26.457999449509842) | > grad_norm_1: tensor(8850.7686, device='cuda:0') (tensor(4722.7930, device='cuda:0')) | > current_lr_0: 0.00018776448176526206 | > current_lr_1: 0.00018776448176526206 | > step_time: 0.4674 (0.43985803922017414) | > loader_time: 0.0086 (0.007324909552549705) --> EVAL PERFORMANCE | > avg_loader_time: 0.008716351549390336 (-0.00014219485538106633) | > avg_step_time: 0.45807405256889233 (+0.0005482049055502425) | > avg_loss_disc: 2.7193814432117303 (-0.0037309216781400423) | > avg_loss_disc_real_0: 0.1669767186780211 (+0.0022315425457249682) | > avg_loss_disc_real_1: 0.22249188819821453 (-0.008024606171628118) | > avg_loss_disc_real_2: 0.23584687415982636 (-0.008575739155352952) | > avg_loss_disc_real_3: 0.23748583147223568 (+0.003465510589975712) | > avg_loss_disc_real_4: 0.24449413364202205 (-0.0006524551082664753) | > avg_loss_disc_real_5: 0.24671427296920562 (-0.00012046014758901324) | > avg_loss_0: 2.7193814432117303 (-0.0037309216781400423) | > avg_grad_norm_0: tensor(58.3375, device='cuda:0') (tensor(-2.8848, device='cuda:0')) | > avg_loss_gen: 1.9607248742815475 (-0.002038661862762803) | > avg_loss_kl: 2.258601553003552 (+0.1298468767757135) | > avg_loss_feat: 3.203176285179568 (+0.028532674614812237) | > avg_loss_mel: 18.16763112242792 (+0.5960006713867116) | > avg_loss_duration: 1.6023637828692583 (-0.006361407293400623) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.19249752205862 (+0.745980222460247) | > avg_grad_norm_1: tensor(4638.3672, device='cuda:0') (tensor(-154.8081, device='cuda:0'))  > EPOCH: 506/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:24:42)   --> TIME: 2025-02-25 13:24:51 -- STEP: 17/72 -- GLOBAL_STEP: 1036450 | > loss_disc: 2.6388330459594727 (2.6898293214685776) | > loss_disc_real_0: 0.1516064703464508 (0.16469654660014546) | > loss_disc_real_1: 0.251224547624588 (0.22360725525547476) | > loss_disc_real_2: 0.2865827679634094 (0.24360153254340677) | > loss_disc_real_3: 0.24036511778831482 (0.2380849464851267) | > loss_disc_real_4: 0.24935601651668549 (0.24815747141838074) | > loss_disc_real_5: 0.26101088523864746 (0.24501251823761883) | > loss_0: 2.6388330459594727 (2.6898293214685776) | > grad_norm_0: tensor(17.8993, device='cuda:0') (tensor(28.9077, device='cuda:0')) | > loss_gen: 2.078519821166992 (1.966832210035885) | > loss_kl: 3.479494094848633 (2.5123207569122314) | > loss_feat: 3.069525957107544 (3.177427095525405) | > loss_mel: 20.32747459411621 (19.363070992862475) | > loss_duration: 1.7148962020874023 (1.62531176034142) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.669910430908203 (28.644962647381952) | > grad_norm_1: tensor(3002.4656, device='cuda:0') (tensor(2819.1218, device='cuda:0')) | > current_lr_0: 0.0001877410112050414 | > current_lr_1: 0.0001877410112050414 | > step_time: 0.4315 (0.43430520506466136) | > loader_time: 0.0059 (0.006899707457598518)  --> TIME: 2025-02-25 13:25:15 -- STEP: 67/72 -- GLOBAL_STEP: 1036500 | > loss_disc: 2.8391225337982178 (2.7312647940507575) | > loss_disc_real_0: 0.15412801504135132 (0.17615489872978696) | > loss_disc_real_1: 0.2021467089653015 (0.22819020610247084) | > loss_disc_real_2: 0.17322751879692078 (0.24130434309368703) | > loss_disc_real_3: 0.2326308935880661 (0.23988971158639708) | > loss_disc_real_4: 0.241585835814476 (0.24492957298435383) | > loss_disc_real_5: 0.23039989173412323 (0.24611891895087798) | > loss_0: 2.8391225337982178 (2.7312647940507575) | > grad_norm_0: tensor(14.7733, device='cuda:0') (tensor(44.5274, device='cuda:0')) | > loss_gen: 1.831295371055603 (1.9646756257583846) | > loss_kl: 2.7059717178344727 (2.6160453646930297) | > loss_feat: 2.8713362216949463 (3.187264851669767) | > loss_mel: 20.14821434020996 (19.527123465466854) | > loss_duration: 1.6996791362762451 (1.6489840087605947) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.256494522094727 (28.9440932202695) | > grad_norm_1: tensor(1323.8505, device='cuda:0') (tensor(3333.4109, device='cuda:0')) | > current_lr_0: 0.0001877410112050414 | > current_lr_1: 0.0001877410112050414 | > step_time: 0.5125 (0.452541429605057) | > loader_time: 0.0119 (0.008702388450280945) --> EVAL PERFORMANCE | > avg_loader_time: 0.008865265779092277 (+0.00014891422970194149) | > avg_step_time: 0.4561438862706574 (-0.0019301662982349477) | > avg_loss_disc: 2.7318295525832914 (+0.012448109371561156) | > avg_loss_disc_real_0: 0.17708710161313204 (+0.010110382935110934) | > avg_loss_disc_real_1: 0.22924732658225047 (+0.006755438384035944) | > avg_loss_disc_real_2: 0.2427538729889292 (+0.0069069988291028395) | > avg_loss_disc_real_3: 0.2400554250663435 (+0.0025695935941078274) | > avg_loss_disc_real_4: 0.24460795269885532 (+0.00011381905683327531) | > avg_loss_disc_real_5: 0.24549057240217503 (-0.0012237005670305834) | > avg_loss_0: 2.7318295525832914 (+0.012448109371561156) | > avg_grad_norm_0: tensor(43.4334, device='cuda:0') (tensor(-14.9041, device='cuda:0')) | > avg_loss_gen: 1.9647425043750817 (+0.004017630093534175) | > avg_loss_kl: 2.609797282957695 (+0.35119572995414305) | > avg_loss_feat: 3.1746413808473397 (-0.028534904332228272) | > avg_loss_mel: 19.527601242065426 (+1.3599701196375058) | > avg_loss_duration: 1.6509746094824562 (+0.04861082661319793) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.927756967678874 (+1.7352594456202546) | > avg_grad_norm_1: tensor(3301.1428, device='cuda:0') (tensor(-1337.2244, device='cuda:0'))  > EPOCH: 507/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:25:19)   --> TIME: 2025-02-25 13:25:41 -- STEP: 45/72 -- GLOBAL_STEP: 1036550 | > loss_disc: 2.8272149562835693 (2.7477282577090785) | > loss_disc_real_0: 0.14149339497089386 (0.18760748025443819) | > loss_disc_real_1: 0.16680654883384705 (0.23080198764801024) | > loss_disc_real_2: 0.13882185518741608 (0.2643472870190938) | > loss_disc_real_3: 0.24308733642101288 (0.23826249175601535) | > loss_disc_real_4: 0.23305661976337433 (0.24349857138262854) | > loss_disc_real_5: 0.25468266010284424 (0.2512104968229929) | > loss_0: 2.8272149562835693 (2.7477282577090785) | > grad_norm_0: tensor(14.2983, device='cuda:0') (tensor(51.8320, device='cuda:0')) | > loss_gen: 1.8204684257507324 (2.013374813397726) | > loss_kl: 2.123224973678589 (2.1220679018232547) | > loss_feat: 2.91947078704834 (3.271342330508762) | > loss_mel: 17.95419692993164 (18.132969241672093) | > loss_duration: 1.6240661144256592 (1.5840719487931993) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.44142723083496 (27.123826387193468) | > grad_norm_1: tensor(377.2333, device='cuda:0') (tensor(3576.4795, device='cuda:0')) | > current_lr_0: 0.00018771754357864076 | > current_lr_1: 0.00018771754357864076 | > step_time: 0.4706 (0.4449864811367459) | > loader_time: 0.0088 (0.007704485787285699) --> EVAL PERFORMANCE | > avg_loader_time: 0.008791235131277164 (-7.403064781511329e-05) | > avg_step_time: 0.45771036013750965 (+0.0015664738668522693) | > avg_loss_disc: 2.7129857338650116 (-0.01884381871827978) | > avg_loss_disc_real_0: 0.17435769999111203 (-0.0027294016220200112) | > avg_loss_disc_real_1: 0.2253412689331552 (-0.003906057649095268) | > avg_loss_disc_real_2: 0.2504515717147101 (+0.00769769872578091) | > avg_loss_disc_real_3: 0.236922659714457 (-0.0031327653518865017) | > avg_loss_disc_real_4: 0.24224884766088406 (-0.0023591050379712653) | > avg_loss_disc_real_5: 0.24773141028175882 (+0.00224083787958379) | > avg_loss_0: 2.7129857338650116 (-0.01884381871827978) | > avg_grad_norm_0: tensor(55.6934, device='cuda:0') (+tensor(12.2600, device='cuda:0')) | > avg_loss_gen: 2.0066046227871532 (+0.04186211841207155) | > avg_loss_kl: 2.1381591978207437 (-0.47163808513695127) | > avg_loss_feat: 3.249118999696113 (+0.07447761884877346) | > avg_loss_mel: 17.97221882242552 (-1.5553824196399049) | > avg_loss_duration: 1.6019363235419906 (-0.049038285940465665) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.96803804854272 (-1.9597189191361544) | > avg_grad_norm_1: tensor(4030.1887, device='cuda:0') (+tensor(729.0459, device='cuda:0'))  > EPOCH: 508/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:25:56)   --> TIME: 2025-02-25 13:26:08 -- STEP: 23/72 -- GLOBAL_STEP: 1036600 | > loss_disc: 2.708120822906494 (2.651053895121035) | > loss_disc_real_0: 0.2333892434835434 (0.15041096087383188) | > loss_disc_real_1: 0.1985275149345398 (0.2203809547683467) | > loss_disc_real_2: 0.2574000060558319 (0.23120536428430807) | > loss_disc_real_3: 0.25095126032829285 (0.2330955085547074) | > loss_disc_real_4: 0.2800099849700928 (0.2409001394458439) | > loss_disc_real_5: 0.2562412619590759 (0.2430777433125869) | > loss_0: 2.708120822906494 (2.651053895121035) | > grad_norm_0: tensor(83.3189, device='cuda:0') (tensor(58.9823, device='cuda:0')) | > loss_gen: 2.1135308742523193 (2.00235376669013) | > loss_kl: 1.6260751485824585 (1.9796489166176838) | > loss_feat: 2.9599485397338867 (3.2541895845661992) | > loss_mel: 16.83551025390625 (17.308168991752297) | > loss_duration: 1.5761477947235107 (1.5567011885021045) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.1112117767334 (26.1010624429454) | > grad_norm_1: tensor(4347.9712, device='cuda:0') (tensor(3777.0850, device='cuda:0')) | > current_lr_0: 0.00018769407888569344 | > current_lr_1: 0.00018769407888569344 | > step_time: 0.4378 (0.4341511415398639) | > loader_time: 0.0063 (0.006794867308243462) --> EVAL PERFORMANCE | > avg_loader_time: 0.008771973596492283 (-1.9261534784880893e-05) | > avg_step_time: 0.45270291516478633 (-0.005007444972723318) | > avg_loss_disc: 2.6393059307420756 (-0.07367980312293598) | > avg_loss_disc_real_0: 0.1509751649179929 (-0.02338253507311913) | > avg_loss_disc_real_1: 0.21954174570634333 (-0.005799523226811876) | > avg_loss_disc_real_2: 0.22896803306861663 (-0.02148353864609348) | > avg_loss_disc_real_3: 0.23547116698513568 (-0.0014514927293213242) | > avg_loss_disc_real_4: 0.24177092859442806 (-0.0004779190664560007) | > avg_loss_disc_real_5: 0.24605773368351896 (-0.0016736765982398594) | > avg_loss_0: 2.6393059307420756 (-0.07367980312293598) | > avg_grad_norm_0: tensor(61.8443, device='cuda:0') (+tensor(6.1509, device='cuda:0')) | > avg_loss_gen: 2.012863730041074 (+0.00625910725392087) | > avg_loss_kl: 2.0349142568212146 (-0.10324494099952908) | > avg_loss_feat: 3.2598414857622604 (+0.01072248606614723) | > avg_loss_mel: 17.240575682949018 (-0.7316431394765033) | > avg_loss_duration: 1.5905965583425172 (-0.011339765199473328) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.138791608138824 (-0.829246440403896) | > avg_grad_norm_1: tensor(4386.9097, device='cuda:0') (+tensor(356.7209, device='cuda:0'))  > EPOCH: 509/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:26:33)   --> TIME: 2025-02-25 13:26:35 -- STEP: 1/72 -- GLOBAL_STEP: 1036650 | > loss_disc: 2.7101523876190186 (2.7101523876190186) | > loss_disc_real_0: 0.20167405903339386 (0.20167405903339386) | > loss_disc_real_1: 0.19348683953285217 (0.19348683953285217) | > loss_disc_real_2: 0.19109490513801575 (0.19109490513801575) | > loss_disc_real_3: 0.23409947752952576 (0.23409947752952576) | > loss_disc_real_4: 0.24446259438991547 (0.24446259438991547) | > loss_disc_real_5: 0.23865652084350586 (0.23865652084350586) | > loss_0: 2.7101523876190186 (2.7101523876190186) | > grad_norm_0: tensor(28.4538, device='cuda:0') (tensor(28.4538, device='cuda:0')) | > loss_gen: 1.974891185760498 (1.974891185760498) | > loss_kl: 1.9664576053619385 (1.9664576053619385) | > loss_feat: 3.206784248352051 (3.206784248352051) | > loss_mel: 17.1473331451416 (17.1473331451416) | > loss_duration: 1.577592372894287 (1.577592372894287) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.87306022644043 (25.87306022644043) | > grad_norm_1: tensor(2875.3076, device='cuda:0') (tensor(2875.3076, device='cuda:0')) | > current_lr_0: 0.00018767061712583272 | > current_lr_1: 0.00018767061712583272 | > step_time: 0.4237 (0.4236781597137451) | > loader_time: 0.0058 (0.0057833194732666016)  --> TIME: 2025-02-25 13:26:58 -- STEP: 51/72 -- GLOBAL_STEP: 1036700 | > loss_disc: 2.4683151245117188 (2.607521239448997) | > loss_disc_real_0: 0.1213846206665039 (0.14510966457572638) | > loss_disc_real_1: 0.2100691795349121 (0.22144251097651088) | > loss_disc_real_2: 0.21693404018878937 (0.2275562286376953) | > loss_disc_real_3: 0.2263147234916687 (0.2343095868241553) | > loss_disc_real_4: 0.23767460882663727 (0.24077840616890028) | > loss_disc_real_5: 0.22873367369174957 (0.24401518731724983) | > loss_0: 2.4683151245117188 (2.607521239448997) | > grad_norm_0: tensor(61.5781, device='cuda:0') (tensor(66.3425, device='cuda:0')) | > loss_gen: 2.2333507537841797 (2.0882521073023477) | > loss_kl: 2.0240893363952637 (1.9826320433149152) | > loss_feat: 3.996018886566162 (3.5011639080795587) | > loss_mel: 16.140512466430664 (17.2629592184927) | > loss_duration: 1.5928125381469727 (1.5668328135621312) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.986785888671875 (26.401840471753886) | > grad_norm_1: tensor(7487.3843, device='cuda:0') (tensor(5455.3652, device='cuda:0')) | > current_lr_0: 0.00018767061712583272 | > current_lr_1: 0.00018767061712583272 | > step_time: 0.4492 (0.4417368757958506) | > loader_time: 0.0091 (0.007757135466033337) --> EVAL PERFORMANCE | > avg_loader_time: 0.008742570877075195 (-2.9402719417087955e-05) | > avg_step_time: 0.46340872872043665 (+0.010705813555650312) | > avg_loss_disc: 2.610419414412808 (-0.028886516329267753) | > avg_loss_disc_real_0: 0.14768988250846593 (-0.0032852824095269684) | > avg_loss_disc_real_1: 0.22079127670173915 (+0.00124953099539582) | > avg_loss_disc_real_2: 0.22851470679464475 (-0.00045332627397187686) | > avg_loss_disc_real_3: 0.23558096083956706 (+0.00010979385443138368) | > avg_loss_disc_real_4: 0.2399988503943027 (-0.001772078200125371) | > avg_loss_disc_real_5: 0.24343534821355847 (-0.002622385469960492) | > avg_loss_0: 2.610419414412808 (-0.028886516329267753) | > avg_grad_norm_0: tensor(68.7278, device='cuda:0') (+tensor(6.8835, device='cuda:0')) | > avg_loss_gen: 2.0910000935406754 (+0.07813636349960129) | > avg_loss_kl: 2.015958624826351 (-0.01895563199486361) | > avg_loss_feat: 3.474752489949616 (+0.21491100418735565) | > avg_loss_mel: 17.229320351506622 (-0.011255331442395544) | > avg_loss_duration: 1.5832567231755859 (-0.007339835166931374) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.394288479442327 (+0.25549687130350307) | > avg_grad_norm_1: tensor(5589.9023, device='cuda:0') (+tensor(1202.9927, device='cuda:0'))  > EPOCH: 510/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:27:10)   --> TIME: 2025-02-25 13:27:25 -- STEP: 29/72 -- GLOBAL_STEP: 1036750 | > loss_disc: 2.6858842372894287 (2.6307420319524306) | > loss_disc_real_0: 0.15934675931930542 (0.15144981880640163) | > loss_disc_real_1: 0.18601003289222717 (0.2188754297535995) | > loss_disc_real_2: 0.2211170494556427 (0.2315650274013651) | > loss_disc_real_3: 0.23104670643806458 (0.2345115794190045) | > loss_disc_real_4: 0.22323819994926453 (0.24203235990014568) | > loss_disc_real_5: 0.2459907978773117 (0.24371041974117016) | > loss_0: 2.6858842372894287 (2.6307420319524306) | > grad_norm_0: tensor(44.4142, device='cuda:0') (tensor(73.0427, device='cuda:0')) | > loss_gen: 1.8957874774932861 (2.0687357927190853) | > loss_kl: 1.9743257761001587 (1.9726718047569538) | > loss_feat: 3.173546314239502 (3.472561737586712) | > loss_mel: 17.04486656188965 (17.21222818308864) | > loss_duration: 1.5402799844741821 (1.5416535673470333) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.62880516052246 (26.26785107316642) | > grad_norm_1: tensor(3164.9009, device='cuda:0') (tensor(5447.9331, device='cuda:0')) | > current_lr_0: 0.00018764715829869199 | > current_lr_1: 0.00018764715829869199 | > step_time: 0.4438 (0.4392090583669728) | > loader_time: 0.0076 (0.007077118446087015) --> EVAL PERFORMANCE | > avg_loader_time: 0.008855718961903748 (+0.00011314808482855258) | > avg_step_time: 0.4586301555096264 (-0.004778573210810266) | > avg_loss_disc: 2.635421366758751 (+0.025001952345943046) | > avg_loss_disc_real_0: 0.15115080028772354 (+0.0034609177792576107) | > avg_loss_disc_real_1: 0.22287330451145979 (+0.002082027809720638) | > avg_loss_disc_real_2: 0.22822941377968856 (-0.00028529301495619497) | > avg_loss_disc_real_3: 0.23390306322507456 (-0.0016778976144924962) | > avg_loss_disc_real_4: 0.24160821064257285 (+0.0016093602482701586) | > avg_loss_disc_real_5: 0.24258352270428563 (-0.0008518255092728466) | > avg_loss_0: 2.635421366758751 (+0.025001952345943046) | > avg_grad_norm_0: tensor(74.6923, device='cuda:0') (+tensor(5.9645, device='cuda:0')) | > avg_loss_gen: 2.0599852028027392 (-0.031014890737936174) | > avg_loss_kl: 2.028492185431466 (+0.012533560605115124) | > avg_loss_feat: 3.4209249926285006 (-0.05382749732111547) | > avg_loss_mel: 17.108660885985483 (-0.12065946552113971) | > avg_loss_duration: 1.5829308486320603 (-0.00032587454352550616) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.200994115480235 (-0.19329436396209232) | > avg_grad_norm_1: tensor(5665.9966, device='cuda:0') (+tensor(76.0942, device='cuda:0'))  > EPOCH: 511/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:27:47)   --> TIME: 2025-02-25 13:27:52 -- STEP: 7/72 -- GLOBAL_STEP: 1036800 | > loss_disc: 2.5721099376678467 (2.6204031876155307) | > loss_disc_real_0: 0.11510379612445831 (0.14190624334982463) | > loss_disc_real_1: 0.20088154077529907 (0.22711048594542912) | > loss_disc_real_2: 0.21595388650894165 (0.20765264545168197) | > loss_disc_real_3: 0.26151105761528015 (0.2515410568032946) | > loss_disc_real_4: 0.24907180666923523 (0.2531452625989914) | > loss_disc_real_5: 0.25414496660232544 (0.25306456855365206) | > loss_0: 2.5721099376678467 (2.6204031876155307) | > grad_norm_0: tensor(78.2384, device='cuda:0') (tensor(74.6777, device='cuda:0')) | > loss_gen: 2.1748158931732178 (2.081060562814985) | > loss_kl: 1.8638131618499756 (1.9595585720879691) | > loss_feat: 3.494509220123291 (3.4605769089290073) | > loss_mel: 17.34465789794922 (17.232609612601145) | > loss_duration: 1.5363826751708984 (1.549414106777736) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.4141788482666 (26.283219473702566) | > grad_norm_1: tensor(6875.6255, device='cuda:0') (tensor(6213.5747, device='cuda:0')) | > current_lr_0: 0.00018762370240390464 | > current_lr_1: 0.00018762370240390464 | > step_time: 0.4336 (0.43054403577532085) | > loader_time: 0.0055 (0.006654296602521624)  --> TIME: 2025-02-25 13:28:15 -- STEP: 57/72 -- GLOBAL_STEP: 1036850 | > loss_disc: 2.752189874649048 (2.6381944355211755) | > loss_disc_real_0: 0.11060595512390137 (0.14865595677442714) | > loss_disc_real_1: 0.23720641434192657 (0.22478241434222773) | > loss_disc_real_2: 0.20051339268684387 (0.22692846755186716) | > loss_disc_real_3: 0.2437739074230194 (0.23861045325011537) | > loss_disc_real_4: 0.2488655298948288 (0.24489697082000866) | > loss_disc_real_5: 0.2487213909626007 (0.24757532566262966) | > loss_0: 2.752189874649048 (2.6381944355211755) | > grad_norm_0: tensor(57.1938, device='cuda:0') (tensor(74.0995, device='cuda:0')) | > loss_gen: 1.8860576152801514 (2.0710691548230353) | > loss_kl: 1.9714057445526123 (2.026539566224081) | > loss_feat: 2.922409772872925 (3.341853744105289) | > loss_mel: 15.896476745605469 (16.849298945644446) | > loss_duration: 1.6245416402816772 (1.5684160705198322) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.300891876220703 (25.857177299365663) | > grad_norm_1: tensor(4192.3511, device='cuda:0') (tensor(5980.6753, device='cuda:0')) | > current_lr_0: 0.00018762370240390464 | > current_lr_1: 0.00018762370240390464 | > step_time: 0.4605 (0.4431567484872383) | > loader_time: 0.0094 (0.007906620962577953) --> EVAL PERFORMANCE | > avg_loader_time: 0.008635974266159701 (-0.0002197446957440466) | > avg_step_time: 0.4541196117938404 (-0.004510543715785964) | > avg_loss_disc: 2.6443233389249987 (+0.008901972166247774) | > avg_loss_disc_real_0: 0.1524091878197562 (+0.0012583875320326676) | > avg_loss_disc_real_1: 0.22504287055680447 (+0.002169566045344684) | > avg_loss_disc_real_2: 0.228749660748831 (+0.000520246969142435) | > avg_loss_disc_real_3: 0.23854338891909155 (+0.0046403256940169835) | > avg_loss_disc_real_4: 0.2434838745795505 (+0.0018756639369776618) | > avg_loss_disc_real_5: 0.2470865747038747 (+0.004503051999589086) | > avg_loss_0: 2.6443233389249987 (+0.008901972166247774) | > avg_grad_norm_0: tensor(74.5695, device='cuda:0') (tensor(-0.1227, device='cuda:0')) | > avg_loss_gen: 2.0722079478519064 (+0.012222745049167205) | > avg_loss_kl: 2.0607430112194 (+0.03225082578793392) | > avg_loss_feat: 3.3176182290198097 (-0.10330676360869084) | > avg_loss_mel: 16.878229221827546 (-0.23043166415793692) | > avg_loss_duration: 1.5801623740666348 (-0.002768474565425505) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.90896066477601 (-0.29203345070422415) | > avg_grad_norm_1: tensor(6030.3647, device='cuda:0') (+tensor(364.3682, device='cuda:0'))  > EPOCH: 512/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:28:24)   --> TIME: 2025-02-25 13:28:42 -- STEP: 35/72 -- GLOBAL_STEP: 1036900 | > loss_disc: 2.645904064178467 (2.6786039556775774) | > loss_disc_real_0: 0.14420008659362793 (0.16248673253825732) | > loss_disc_real_1: 0.24264532327651978 (0.22423226322446552) | > loss_disc_real_2: 0.24155662953853607 (0.22909263691731863) | > loss_disc_real_3: 0.25252753496170044 (0.2362769297191075) | > loss_disc_real_4: 0.24650375545024872 (0.24078229708330973) | > loss_disc_real_5: 0.2573649287223816 (0.24279050273554667) | > loss_0: 2.645904064178467 (2.6786039556775774) | > grad_norm_0: tensor(106.0100, device='cuda:0') (tensor(77.7071, device='cuda:0')) | > loss_gen: 2.1066741943359375 (2.065950788770403) | > loss_kl: 2.3304762840270996 (2.0055903026035855) | > loss_feat: 2.9963605403900146 (3.5285825048174178) | > loss_mel: 16.76161766052246 (17.284732982090542) | > loss_duration: 1.6035950183868408 (1.54942239012037) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.798725128173828 (26.434279142107282) | > grad_norm_1: tensor(5032.6343, device='cuda:0') (tensor(5728.0366, device='cuda:0')) | > current_lr_0: 0.00018760024944110414 | > current_lr_1: 0.00018760024944110414 | > step_time: 0.4507 (0.43932488305228096) | > loader_time: 0.0086 (0.007159941537039621) --> EVAL PERFORMANCE | > avg_loader_time: 0.008560526538902605 (-7.544772725709588e-05) | > avg_step_time: 0.46280220528723487 (+0.008682593493394453) | > avg_loss_disc: 2.6641836535762735 (+0.019860314651274802) | > avg_loss_disc_real_0: 0.16238408457015602 (+0.009974896750399809) | > avg_loss_disc_real_1: 0.22474670703981964 (-0.000296163516984832) | > avg_loss_disc_real_2: 0.223761212552937 (-0.004988448195894002) | > avg_loss_disc_real_3: 0.23610896459767516 (-0.002434424321416384) | > avg_loss_disc_real_4: 0.24189212897294005 (-0.0015917456066104618) | > avg_loss_disc_real_5: 0.24327973244895398 (-0.003806842254920728) | > avg_loss_0: 2.6641836535762735 (+0.019860314651274802) | > avg_grad_norm_0: tensor(77.6748, device='cuda:0') (+tensor(3.1053, device='cuda:0')) | > avg_loss_gen: 2.0569377113396015 (-0.015270236512304969) | > avg_loss_kl: 2.0747393990906184 (+0.013996387871218374) | > avg_loss_feat: 3.4334164236632874 (+0.11579819464347763) | > avg_loss_mel: 17.192800441258385 (+0.31457121943083877) | > avg_loss_duration: 1.58458957201998 (+0.00442719795334523) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.342483547371877 (+0.4335228825958666) | > avg_grad_norm_1: tensor(5836.6704, device='cuda:0') (tensor(-193.6943, device='cuda:0'))  > EPOCH: 513/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:29:01)   --> TIME: 2025-02-25 13:29:09 -- STEP: 13/72 -- GLOBAL_STEP: 1036950 | > loss_disc: 2.7774224281311035 (2.657977434305044) | > loss_disc_real_0: 0.1734675168991089 (0.143472739137136) | > loss_disc_real_1: 0.2243807017803192 (0.22841575283270615) | > loss_disc_real_2: 0.2066095769405365 (0.230376015488918) | > loss_disc_real_3: 0.21940019726753235 (0.23360470166573158) | > loss_disc_real_4: 0.23999854922294617 (0.23904767976357386) | > loss_disc_real_5: 0.2454315572977066 (0.24215455467884356) | > loss_0: 2.7774224281311035 (2.657977434305044) | > grad_norm_0: tensor(53.8073, device='cuda:0') (tensor(79.9177, device='cuda:0')) | > loss_gen: 1.930069923400879 (2.0530984401702876) | > loss_kl: 2.0949461460113525 (2.0542926329832816) | > loss_feat: 3.068697690963745 (3.46507714344905) | > loss_mel: 16.183151245117188 (17.187725360576923) | > loss_duration: 1.4672257900238037 (1.5405038686899037) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.744089126586914 (26.300697033221905) | > grad_norm_1: tensor(4631.3066, device='cuda:0') (tensor(5793.3120, device='cuda:0')) | > current_lr_0: 0.000187576799409924 | > current_lr_1: 0.000187576799409924 | > step_time: 0.4846 (0.4454278028928317) | > loader_time: 0.0094 (0.007258782019981971)  --> TIME: 2025-02-25 13:29:33 -- STEP: 63/72 -- GLOBAL_STEP: 1037000 | > loss_disc: 2.7371668815612793 (2.696196105745104) | > loss_disc_real_0: 0.14403048157691956 (0.15700148708290523) | > loss_disc_real_1: 0.28842002153396606 (0.23034440643257564) | > loss_disc_real_2: 0.11681005358695984 (0.24109768039650387) | > loss_disc_real_3: 0.22762393951416016 (0.2392796159736694) | > loss_disc_real_4: 0.2415197193622589 (0.2447342345165828) | > loss_disc_real_5: 0.22571846842765808 (0.24480314245299686) | > loss_0: 2.7371668815612793 (2.696196105745104) | > grad_norm_0: tensor(110.2302, device='cuda:0') (tensor(75.4255, device='cuda:0')) | > loss_gen: 1.839814305305481 (2.073282338324048) | > loss_kl: 2.274517297744751 (2.174516416731333) | > loss_feat: 2.8377256393432617 (3.44701409718347) | > loss_mel: 18.266462326049805 (17.604889067392502) | > loss_duration: 1.6008023023605347 (1.5812545825564672) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.819320678710938 (26.880956407577273) | > grad_norm_1: tensor(5470.5405, device='cuda:0') (tensor(5468.0522, device='cuda:0')) | > current_lr_0: 0.000187576799409924 | > current_lr_1: 0.000187576799409924 | > step_time: 0.5043 (0.4523551312703935) | > loader_time: 0.0156 (0.008391637650747148) --> EVAL PERFORMANCE | > avg_loader_time: 0.008870749406411613 (+0.00031022286750900796) | > avg_step_time: 0.4599799404681568 (-0.0028222648190780464) | > avg_loss_disc: 2.7123890863337987 (+0.0482054327575252) | > avg_loss_disc_real_0: 0.1501530492809457 (-0.012231035289210324) | > avg_loss_disc_real_1: 0.23235423506145747 (+0.007607528021637833) | > avg_loss_disc_real_2: 0.24195684712957327 (+0.018195634576636283) | > avg_loss_disc_real_3: 0.24171122257978145 (+0.0056022579821062846) | > avg_loss_disc_real_4: 0.24590538169296694 (+0.004013252720026894) | > avg_loss_disc_real_5: 0.2458521373255152 (+0.002572404876561213) | > avg_loss_0: 2.7123890863337987 (+0.0482054327575252) | > avg_grad_norm_0: tensor(75.8985, device='cuda:0') (tensor(-1.7764, device='cuda:0')) | > avg_loss_gen: 2.066097808555819 (+0.009160097216217356) | > avg_loss_kl: 2.257347990089737 (+0.18260859099911864) | > avg_loss_feat: 3.450589888532397 (+0.017173464869109534) | > avg_loss_mel: 17.931869829204718 (+0.7390693879463335) | > avg_loss_duration: 1.594196381703229 (+0.009606809683248896) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.300101804061676 (+0.9576182566897984) | > avg_grad_norm_1: tensor(5509.7012, device='cuda:0') (tensor(-326.9692, device='cuda:0'))  > EPOCH: 514/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:29:39)   --> TIME: 2025-02-25 13:29:59 -- STEP: 41/72 -- GLOBAL_STEP: 1037050 | > loss_disc: 2.9851016998291016 (2.758783392789887) | > loss_disc_real_0: 0.29993683099746704 (0.181866013058802) | > loss_disc_real_1: 0.1584353744983673 (0.23758640812664497) | > loss_disc_real_2: 0.09195656329393387 (0.21468483638472674) | > loss_disc_real_3: 0.27173736691474915 (0.2410335031951346) | > loss_disc_real_4: 0.27009645104408264 (0.24665189734319362) | > loss_disc_real_5: 0.24824172258377075 (0.2467709332704544) | > loss_0: 2.9851016998291016 (2.758783392789887) | > grad_norm_0: tensor(89.8939, device='cuda:0') (tensor(86.8415, device='cuda:0')) | > loss_gen: 1.7044072151184082 (2.0640451152150225) | > loss_kl: 2.671642303466797 (2.3448235348957343) | > loss_feat: 3.198498010635376 (3.351236575987281) | > loss_mel: 21.607412338256836 (18.817776098483947) | > loss_duration: 1.7683916091918945 (1.6011312007904053) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.95035171508789 (28.179012484666778) | > grad_norm_1: tensor(3158.2488, device='cuda:0') (tensor(5219.0195, device='cuda:0')) | > current_lr_0: 0.00018755335230999777 | > current_lr_1: 0.00018755335230999777 | > step_time: 0.4471 (0.440628889130383) | > loader_time: 0.0093 (0.0074094039637868) --> EVAL PERFORMANCE | > avg_loader_time: 0.008770526294976893 (-0.0001002231114347208) | > avg_step_time: 0.45689081138288473 (-0.003089129085272091) | > avg_loss_disc: 2.746187683562158 (+0.033798597228359206) | > avg_loss_disc_real_0: 0.17182732541376441 (+0.02167427613281872) | > avg_loss_disc_real_1: 0.23030270762007002 (-0.002051527441387452) | > avg_loss_disc_real_2: 0.22498701754170405 (-0.016969829587869217) | > avg_loss_disc_real_3: 0.24059683021525263 (-0.0011143923645288156) | > avg_loss_disc_real_4: 0.25196590410991454 (+0.0060605224169476) | > avg_loss_disc_real_5: 0.25625571629530947 (+0.01040357896979427) | > avg_loss_0: 2.746187683562158 (+0.033798597228359206) | > avg_grad_norm_0: tensor(73.0238, device='cuda:0') (tensor(-2.8747, device='cuda:0')) | > avg_loss_gen: 2.093833817562586 (+0.027736009006767315) | > avg_loss_kl: 2.8106518362609427 (+0.5533038461712056) | > avg_loss_feat: 3.4184806917754695 (-0.03210919675692736) | > avg_loss_mel: 20.45837149821536 (+2.526501669010642) | > avg_loss_duration: 1.6614530758118966 (+0.06725669410866764) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 30.442791011971487 (+3.142689207909811) | > avg_grad_norm_1: tensor(5093.0845, device='cuda:0') (tensor(-416.6167, device='cuda:0'))  > EPOCH: 515/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:30:16)   --> TIME: 2025-02-25 13:30:26 -- STEP: 19/72 -- GLOBAL_STEP: 1037100 | > loss_disc: 2.723555326461792 (2.7649614559976676) | > loss_disc_real_0: 0.24764740467071533 (0.17824555423698926) | > loss_disc_real_1: 0.22383931279182434 (0.2574097929816497) | > loss_disc_real_2: 0.20334094762802124 (0.23054306287514537) | > loss_disc_real_3: 0.19868384301662445 (0.24142737686634064) | > loss_disc_real_4: 0.25800782442092896 (0.24202873910728254) | > loss_disc_real_5: 0.25281012058258057 (0.2394292597707949) | > loss_0: 2.723555326461792 (2.7649614559976676) | > grad_norm_0: tensor(30.6476, device='cuda:0') (tensor(46.2477, device='cuda:0')) | > loss_gen: 1.7504392862319946 (1.969251325255946) | > loss_kl: 2.364166021347046 (2.4878416563335217) | > loss_feat: 3.012605667114258 (3.1666772742020455) | > loss_mel: 18.60506820678711 (19.792288730019017) | > loss_duration: 1.5424766540527344 (1.5939094392876876) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.274755477905273 (29.009968205502158) | > grad_norm_1: tensor(72.7975, device='cuda:0') (tensor(3014.8997, device='cuda:0')) | > current_lr_0: 0.00018752990814095903 | > current_lr_1: 0.00018752990814095903 | > step_time: 0.4319 (0.4420508961928518) | > loader_time: 0.0061 (0.006660938262939453)  --> TIME: 2025-02-25 13:30:50 -- STEP: 69/72 -- GLOBAL_STEP: 1037150 | > loss_disc: 2.6837944984436035 (2.742895458055579) | > loss_disc_real_0: 0.15991204977035522 (0.1812275027235349) | > loss_disc_real_1: 0.2695291340351105 (0.23515122200267902) | > loss_disc_real_2: 0.22606191039085388 (0.23432525992393494) | > loss_disc_real_3: 0.22780686616897583 (0.23759836822316266) | > loss_disc_real_4: 0.2534574568271637 (0.24683408525542935) | > loss_disc_real_5: 0.2510126233100891 (0.24799887663212375) | > loss_0: 2.6837944984436035 (2.742895458055579) | > grad_norm_0: tensor(63.8566, device='cuda:0') (tensor(56.4681, device='cuda:0')) | > loss_gen: 1.8659782409667969 (1.945633065873298) | > loss_kl: 2.4247238636016846 (2.48904830476512) | > loss_feat: 2.612253427505493 (3.0530752202738887) | > loss_mel: 17.615848541259766 (19.890883763631184) | > loss_duration: 1.7160263061523438 (1.6407346587250198) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.234830856323242 (29.019374930340312) | > grad_norm_1: tensor(3138.2129, device='cuda:0') (tensor(3787.5684, device='cuda:0')) | > current_lr_0: 0.00018752990814095903 | > current_lr_1: 0.00018752990814095903 | > step_time: 0.5516 (0.4582185261491416) | > loader_time: 0.0126 (0.00869719878486965) --> EVAL PERFORMANCE | > avg_loader_time: 0.00884445956055547 (+7.393326557857821e-05) | > avg_step_time: 0.4602905662966446 (+0.003399754913759878) | > avg_loss_disc: 2.7386541937438533 (-0.007533489818304595) | > avg_loss_disc_real_0: 0.1795437302387936 (+0.007716404825029194) | > avg_loss_disc_real_1: 0.23510396018834182 (+0.004801252568271797) | > avg_loss_disc_real_2: 0.23295231394364802 (+0.007965296401943961) | > avg_loss_disc_real_3: 0.23732108297482343 (-0.0032757472404292) | > avg_loss_disc_real_4: 0.24632310510521205 (-0.005642799004702492) | > avg_loss_disc_real_5: 0.24750143877217468 (-0.00875427752313479) | > avg_loss_0: 2.7386541937438533 (-0.007533489818304595) | > avg_grad_norm_0: tensor(56.7425, device='cuda:0') (tensor(-16.2813, device='cuda:0')) | > avg_loss_gen: 1.946218188379852 (-0.1476156291827342) | > avg_loss_kl: 2.486247385051888 (-0.3244044512090549) | > avg_loss_feat: 3.0603705224856523 (-0.3581101692898172) | > avg_loss_mel: 19.85252388430313 (-0.6058476139122284) | > avg_loss_duration: 1.6403828389208082 (-0.021070236891088445) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.98574273015412 (-1.457048281817368) | > avg_grad_norm_1: tensor(3848.6509, device='cuda:0') (tensor(-1244.4336, device='cuda:0'))  > EPOCH: 516/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:30:54)   --> TIME: 2025-02-25 13:31:17 -- STEP: 47/72 -- GLOBAL_STEP: 1037200 | > loss_disc: 2.769320249557495 (2.6556208996062582) | > loss_disc_real_0: 0.24309378862380981 (0.1639509508584408) | > loss_disc_real_1: 0.26299402117729187 (0.2156903455232052) | > loss_disc_real_2: 0.17618998885154724 (0.24050862199448525) | > loss_disc_real_3: 0.24613314867019653 (0.23618458782104737) | > loss_disc_real_4: 0.2644219994544983 (0.2416655069335978) | > loss_disc_real_5: 0.2799685001373291 (0.24356913186134177) | > loss_0: 2.769320249557495 (2.6556208996062582) | > grad_norm_0: tensor(25.1315, device='cuda:0') (tensor(62.8635, device='cuda:0')) | > loss_gen: 1.912947416305542 (2.004564221869124) | > loss_kl: 2.024233102798462 (2.0704098584804136) | > loss_feat: 3.0868782997131348 (3.3097132621927465) | > loss_mel: 17.206783294677734 (17.879383330649514) | > loss_duration: 1.6069350242614746 (1.580957737374813) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.83777618408203 (26.845028410566613) | > grad_norm_1: tensor(5172.3965, device='cuda:0') (tensor(4778.9341, device='cuda:0')) | > current_lr_0: 0.0001875064669024414 | > current_lr_1: 0.0001875064669024414 | > step_time: 0.4526 (0.44094198815366054) | > loader_time: 0.0088 (0.007615753944883955) --> EVAL PERFORMANCE | > avg_loader_time: 0.008772611618041992 (-7.18479425134786e-05) | > avg_step_time: 0.4585632774191843 (-0.0017272888774603201) | > avg_loss_disc: 2.657373700343387 (-0.08128049340046628) | > avg_loss_disc_real_0: 0.1642208358561489 (-0.015322894382644697) | > avg_loss_disc_real_1: 0.21670857323727138 (-0.01839538695107043) | > avg_loss_disc_real_2: 0.23283490076870986 (-0.00011741317493815795) | > avg_loss_disc_real_3: 0.23423917297746094 (-0.003081909997362492) | > avg_loss_disc_real_4: 0.24065703888174514 (-0.005666066223466909) | > avg_loss_disc_real_5: 0.24468231893761058 (-0.0028191198345640933) | > avg_loss_0: 2.657373700343387 (-0.08128049340046628) | > avg_grad_norm_0: tensor(67.4486, device='cuda:0') (+tensor(10.7061, device='cuda:0')) | > avg_loss_gen: 1.9783019499039989 (+0.032083761524146936) | > avg_loss_kl: 2.082414803370623 (-0.4038325816812649) | > avg_loss_feat: 3.2364829493240572 (+0.1761124268384049) | > avg_loss_mel: 17.698715881562563 (-2.1538080027405684) | > avg_loss_duration: 1.5979042288283227 (-0.042478610092485436) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.593819739113393 (-2.391922991040726) | > avg_grad_norm_1: tensor(4906.9502, device='cuda:0') (+tensor(1058.2993, device='cuda:0'))  > EPOCH: 517/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:31:31)   --> TIME: 2025-02-25 13:31:44 -- STEP: 25/72 -- GLOBAL_STEP: 1037250 | > loss_disc: 2.5597941875457764 (2.6040510749816894) | > loss_disc_real_0: 0.16464564204216003 (0.15346282929182054) | > loss_disc_real_1: 0.20278100669384003 (0.2225211066007614) | > loss_disc_real_2: 0.23542924225330353 (0.22052002012729643) | > loss_disc_real_3: 0.2339661419391632 (0.22987862348556518) | > loss_disc_real_4: 0.22371158003807068 (0.23959233462810517) | > loss_disc_real_5: 0.23812702298164368 (0.24111688137054443) | > loss_0: 2.5597941875457764 (2.6040510749816894) | > grad_norm_0: tensor(93.1445, device='cuda:0') (tensor(80.6149, device='cuda:0')) | > loss_gen: 2.149944305419922 (2.0494421339035034) | > loss_kl: 2.06040096282959 (1.9504410696029664) | > loss_feat: 3.7073333263397217 (3.475414581298828) | > loss_mel: 17.85862922668457 (17.54554817199707) | > loss_duration: 1.55706787109375 (1.5538102197647095) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.333375930786133 (26.57465660095215) | > grad_norm_1: tensor(7861.6382, device='cuda:0') (tensor(6203.5054, device='cuda:0')) | > current_lr_0: 0.00018748302859407858 | > current_lr_1: 0.00018748302859407858 | > step_time: 0.4358 (0.440419921875) | > loader_time: 0.0074 (0.006950912475585938) --> EVAL PERFORMANCE | > avg_loader_time: 0.008841350045002683 (+6.873842696069124e-05) | > avg_step_time: 0.45996673678008604 (+0.0014034593609017487) | > avg_loss_disc: 2.639966366996228 (-0.01740733334715916) | > avg_loss_disc_real_0: 0.1558942617454999 (-0.008326574110649021) | > avg_loss_disc_real_1: 0.2218958559170575 (+0.005187282679786104) | > avg_loss_disc_real_2: 0.23194351876285715 (-0.0008913820058527033) | > avg_loss_disc_real_3: 0.23263801424436167 (-0.0016011587330992672) | > avg_loss_disc_real_4: 0.240634378622955 (-2.266025879013589e-05) | > avg_loss_disc_real_5: 0.2437136043125475 (-0.000968714625063094) | > avg_loss_0: 2.639966366996228 (-0.01740733334715916) | > avg_grad_norm_0: tensor(80.0131, device='cuda:0') (+tensor(12.5645, device='cuda:0')) | > avg_loss_gen: 2.0221701255986386 (+0.043868175694639744) | > avg_loss_kl: 2.011620291521851 (-0.07079451184877206) | > avg_loss_feat: 3.3731157141672052 (+0.136632764843148) | > avg_loss_mel: 17.36169606866971 (-0.337019812892855) | > avg_loss_duration: 1.5895223869404322 (-0.008381841887890484) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.35812471953916 (-0.23569501957423356) | > avg_grad_norm_1: tensor(6178.6841, device='cuda:0') (+tensor(1271.7339, device='cuda:0'))  > EPOCH: 518/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:32:08)   --> TIME: 2025-02-25 13:32:11 -- STEP: 3/72 -- GLOBAL_STEP: 1037300 | > loss_disc: 2.6200714111328125 (2.712892214457194) | > loss_disc_real_0: 0.24079641699790955 (0.19029268125693002) | > loss_disc_real_1: 0.20532186329364777 (0.21839626133441925) | > loss_disc_real_2: 0.32510411739349365 (0.32010573148727417) | > loss_disc_real_3: 0.2642209231853485 (0.2538686990737915) | > loss_disc_real_4: 0.2675285041332245 (0.2692537804444631) | > loss_disc_real_5: 0.2593339681625366 (0.2599913279215495) | > loss_0: 2.6200714111328125 (2.712892214457194) | > grad_norm_0: tensor(132.3722, device='cuda:0') (tensor(83.7550, device='cuda:0')) | > loss_gen: 2.336357355117798 (2.206033229827881) | > loss_kl: 2.1021196842193604 (2.063511848449707) | > loss_feat: 3.142279863357544 (3.2589115301767984) | > loss_mel: 18.024200439453125 (17.42546780904134) | > loss_duration: 1.591876745223999 (1.5776995023091633) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.196834564208984 (26.53162447611491) | > grad_norm_1: tensor(6388.9165, device='cuda:0') (tensor(5605.0210, device='cuda:0')) | > current_lr_0: 0.00018745959321550432 | > current_lr_1: 0.00018745959321550432 | > step_time: 0.4217 (0.4343356291453044) | > loader_time: 0.0057 (0.006147940953572591)  --> TIME: 2025-02-25 13:32:34 -- STEP: 53/72 -- GLOBAL_STEP: 1037350 | > loss_disc: 2.8521857261657715 (2.714521214647113) | > loss_disc_real_0: 0.159712553024292 (0.1843247336317909) | > loss_disc_real_1: 0.23771339654922485 (0.2230968823972738) | > loss_disc_real_2: 0.11402139812707901 (0.232519305680158) | > loss_disc_real_3: 0.2153591364622116 (0.23726920333673368) | > loss_disc_real_4: 0.21520070731639862 (0.24333169578381306) | > loss_disc_real_5: 0.2363327592611313 (0.24637060373459221) | > loss_0: 2.8521857261657715 (2.714521214647113) | > grad_norm_0: tensor(40.2028, device='cuda:0') (tensor(82.7211, device='cuda:0')) | > loss_gen: 1.6083040237426758 (2.013313965977363) | > loss_kl: 2.1345324516296387 (2.0642757145863655) | > loss_feat: 3.1456878185272217 (3.330523927256746) | > loss_mel: 17.479276657104492 (17.36999648472048) | > loss_duration: 1.6385538578033447 (1.5867486922246106) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.00635528564453 (26.364858735282468) | > grad_norm_1: tensor(688.0828, device='cuda:0') (tensor(5069.2534, device='cuda:0')) | > current_lr_0: 0.00018745959321550432 | > current_lr_1: 0.00018745959321550432 | > step_time: 0.4567 (0.4411654697274262) | > loader_time: 0.009 (0.007869742951303157) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867096806915713 (-0.00017038197584555263) | > avg_step_time: 0.4546580650437046 (-0.005308671736381443) | > avg_loss_disc: 2.732908769392631 (+0.09294240239640317) | > avg_loss_disc_real_0: 0.17811095457471599 (+0.022216692829216095) | > avg_loss_disc_real_1: 0.22402132596348373 (+0.002125470046426242) | > avg_loss_disc_real_2: 0.24361144688347697 (+0.011667928120619814) | > avg_loss_disc_real_3: 0.23662210623143423 (+0.003984091987072558) | > avg_loss_disc_real_4: 0.2411793741122098 (+0.0005449954892547959) | > avg_loss_disc_real_5: 0.24570045420821285 (+0.00198684989566536) | > avg_loss_0: 2.732908769392631 (+0.09294240239640317) | > avg_grad_norm_0: tensor(75.6266, device='cuda:0') (tensor(-4.3865, device='cuda:0')) | > avg_loss_gen: 1.9802023642499682 (-0.04196776134867042) | > avg_loss_kl: 2.116814087814008 (+0.10519379629215697) | > avg_loss_feat: 3.2458750630768254 (-0.12724065109037985) | > avg_loss_mel: 17.47588000499027 (+0.11418393632056123) | > avg_loss_duration: 1.5996753128481582 (+0.010152925907725985) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.418446876633336 (+0.06032215709417699) | > avg_grad_norm_1: tensor(4695.1060, device='cuda:0') (tensor(-1483.5781, device='cuda:0'))  > EPOCH: 519/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:32:45)   --> TIME: 2025-02-25 13:33:01 -- STEP: 31/72 -- GLOBAL_STEP: 1037400 | > loss_disc: 2.7468700408935547 (2.6866667578297276) | > loss_disc_real_0: 0.17396505177021027 (0.16395629942417145) | > loss_disc_real_1: 0.20085391402244568 (0.23190955673494645) | > loss_disc_real_2: 0.2831648886203766 (0.24125724982830785) | > loss_disc_real_3: 0.26066845655441284 (0.23940799169002042) | > loss_disc_real_4: 0.26991385221481323 (0.24526422831319994) | > loss_disc_real_5: 0.26834627985954285 (0.24602088332176208) | > loss_0: 2.7468700408935547 (2.6866667578297276) | > grad_norm_0: tensor(19.2988, device='cuda:0') (tensor(66.3070, device='cuda:0')) | > loss_gen: 2.0023207664489746 (1.974465908542756) | > loss_kl: 1.8099701404571533 (2.019729983422064) | > loss_feat: 2.9074909687042236 (3.181239535731654) | > loss_mel: 17.678199768066406 (17.428101170447565) | > loss_duration: 1.5993871688842773 (1.5506987110260995) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.99736785888672 (26.154235224569998) | > grad_norm_1: tensor(3837.4297, device='cuda:0') (tensor(4483.4756, device='cuda:0')) | > current_lr_0: 0.00018743616076635238 | > current_lr_1: 0.00018743616076635238 | > step_time: 0.4388 (0.4335552876995456) | > loader_time: 0.0087 (0.006985064475767074) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611752953327877 (-5.921511582925379e-05) | > avg_step_time: 0.46077986838112417 (+0.0061218033374195735) | > avg_loss_disc: 2.6852812599128404 (-0.047627509479790664) | > avg_loss_disc_real_0: 0.1686156455060126 (-0.009495309068703384) | > avg_loss_disc_real_1: 0.23004060668844573 (+0.006019280724961995) | > avg_loss_disc_real_2: 0.23383527952180783 (-0.009776167361669141) | > avg_loss_disc_real_3: 0.23729469196897157 (+0.0006725857375373401) | > avg_loss_disc_real_4: 0.24216496209863206 (+0.0009855879864222594) | > avg_loss_disc_real_5: 0.24447530151253016 (-0.0012251526956826853) | > avg_loss_0: 2.6852812599128404 (-0.047627509479790664) | > avg_grad_norm_0: tensor(70.5474, device='cuda:0') (tensor(-5.0792, device='cuda:0')) | > avg_loss_gen: 1.9556395016925436 (-0.02456286255742457) | > avg_loss_kl: 2.053782132309928 (-0.06303195550407992) | > avg_loss_feat: 3.141418873424261 (-0.1044561896525642) | > avg_loss_mel: 17.16624452026797 (-0.30963548472229974) | > avg_loss_duration: 1.5859125852584839 (-0.01376272758967434) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.902997567620076 (-0.5154493090132597) | > avg_grad_norm_1: tensor(4718.9175, device='cuda:0') (+tensor(23.8115, device='cuda:0'))  > EPOCH: 520/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:33:22)   --> TIME: 2025-02-25 13:33:28 -- STEP: 9/72 -- GLOBAL_STEP: 1037450 | > loss_disc: 2.4545199871063232 (2.697681294547187) | > loss_disc_real_0: 0.12863487005233765 (0.1616816164718734) | > loss_disc_real_1: 0.20830225944519043 (0.22116769519117144) | > loss_disc_real_2: 0.18456989526748657 (0.251471565829383) | > loss_disc_real_3: 0.21175213158130646 (0.24327345026863945) | > loss_disc_real_4: 0.2440849244594574 (0.24614357617166308) | > loss_disc_real_5: 0.24281762540340424 (0.23940903279516432) | > loss_0: 2.4545199871063232 (2.697681294547187) | > grad_norm_0: tensor(41.9616, device='cuda:0') (tensor(71.9344, device='cuda:0')) | > loss_gen: 2.26054310798645 (1.9976947969860501) | > loss_kl: 1.7584953308105469 (1.938445462120904) | > loss_feat: 4.220902442932129 (3.368057621849908) | > loss_mel: 18.48770523071289 (17.350490358140732) | > loss_duration: 1.5492480993270874 (1.5532877577675714) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.276893615722656 (26.207975811428494) | > grad_norm_1: tensor(7489.5210, device='cuda:0') (tensor(5000.3462, device='cuda:0')) | > current_lr_0: 0.00018741273124625658 | > current_lr_1: 0.00018741273124625658 | > step_time: 0.4437 (0.43436113993326825) | > loader_time: 0.0056 (0.006274223327636719)  --> TIME: 2025-02-25 13:33:51 -- STEP: 59/72 -- GLOBAL_STEP: 1037500 | > loss_disc: 2.684439182281494 (2.683924048633899) | > loss_disc_real_0: 0.09009695798158646 (0.15798375149399554) | > loss_disc_real_1: 0.22765259444713593 (0.2242988342955961) | > loss_disc_real_2: 0.23513613641262054 (0.2294299051923267) | > loss_disc_real_3: 0.2624996602535248 (0.2380028028609389) | > loss_disc_real_4: 0.2827974259853363 (0.24529738572694487) | > loss_disc_real_5: 0.25402653217315674 (0.24574480420452052) | > loss_0: 2.684439182281494 (2.683924048633899) | > grad_norm_0: tensor(82.3983, device='cuda:0') (tensor(75.8705, device='cuda:0')) | > loss_gen: 1.8880478143692017 (1.9714328232458083) | > loss_kl: 2.1749958992004395 (2.018178796364089) | > loss_feat: 3.299090623855591 (3.2505935450731696) | > loss_mel: 17.1645565032959 (17.229543847552808) | > loss_duration: 1.6106090545654297 (1.5732750589564695) | > amp_scaler: 32.0 (16.54237288135593) | > loss_1: 26.137300491333008 (26.043024095438298) | > grad_norm_1: tensor(4069.5366, device='cuda:0') (tensor(5457.3560, device='cuda:0')) | > current_lr_0: 0.00018741273124625658 | > current_lr_1: 0.00018741273124625658 | > step_time: 0.4772 (0.4485494848025047) | > loader_time: 0.0103 (0.008160700232295665) --> EVAL PERFORMANCE | > avg_loader_time: 0.008835084001782914 (+0.00022333104845503708) | > avg_step_time: 0.45801805778288507 (-0.0027618105982391006) | > avg_loss_disc: 2.688270924796521 (+0.002989664883680465) | > avg_loss_disc_real_0: 0.15974217280745512 (-0.008873472698557483) | > avg_loss_disc_real_1: 0.22425561912462744 (-0.005784987563818289) | > avg_loss_disc_real_2: 0.23189874811911246 (-0.0019365314026953684) | > avg_loss_disc_real_3: 0.2370725821861079 (-0.0002221097828636609) | > avg_loss_disc_real_4: 0.2445302534271294 (+0.002365291328497332) | > avg_loss_disc_real_5: 0.24681126777554901 (+0.0023359662630188516) | > avg_loss_0: 2.688270924796521 (+0.002989664883680465) | > avg_grad_norm_0: tensor(75.6354, device='cuda:0') (+tensor(5.0881, device='cuda:0')) | > avg_loss_gen: 1.9711683719930515 (+0.015528870300507869) | > avg_loss_kl: 2.0552792095802195 (+0.0014970772702915802) | > avg_loss_feat: 3.231365485930107 (+0.0899466125058459) | > avg_loss_mel: 17.208767608857496 (+0.042523088589526026) | > avg_loss_duration: 1.5851744144735203 (-0.0007381707849636321) | > avg_amp_scaler: 17.12676056338028 (+1.1267605633802802) | > avg_loss_1: 26.05175507236535 (+0.14875750474527294) | > avg_grad_norm_1: tensor(5388.6528, device='cuda:0') (+tensor(669.7354, device='cuda:0'))  > EPOCH: 521/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:33:59)   --> TIME: 2025-02-25 13:34:18 -- STEP: 37/72 -- GLOBAL_STEP: 1037550 | > loss_disc: 2.860213279724121 (2.7244042125908106) | > loss_disc_real_0: 0.2149679809808731 (0.22017385327332728) | > loss_disc_real_1: 0.27453553676605225 (0.2247671394332035) | > loss_disc_real_2: 0.22702592611312866 (0.2354147321469075) | > loss_disc_real_3: 0.2026340514421463 (0.23159759068811261) | > loss_disc_real_4: 0.2850174903869629 (0.2350037842988968) | > loss_disc_real_5: 0.2535860538482666 (0.23546036071068532) | > loss_0: 2.860213279724121 (2.7244042125908106) | > grad_norm_0: tensor(93.4551, device='cuda:0') (tensor(64.7270, device='cuda:0')) | > loss_gen: 1.9248912334442139 (2.1201111432668327) | > loss_kl: 2.5673604011535645 (2.6840680998724866) | > loss_feat: 3.292320489883423 (3.4314895385020487) | > loss_mel: 20.272315979003906 (19.719300166980645) | > loss_duration: 1.63142991065979 (1.6335005341349422) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.688316345214844 (29.588469634184968) | > grad_norm_1: tensor(5275.6758, device='cuda:0') (tensor(3823.3645, device='cuda:0')) | > current_lr_0: 0.0001873893046548508 | > current_lr_1: 0.0001873893046548508 | > step_time: 0.4582 (0.440895093453897) | > loader_time: 0.0083 (0.007106394381136508) --> EVAL PERFORMANCE | > avg_loader_time: 0.008529629505855936 (-0.00030545449592697844) | > avg_step_time: 0.4556850916902784 (-0.002332966092606692) | > avg_loss_disc: 2.7532864859406376 (+0.06501556114411677) | > avg_loss_disc_real_0: 0.18878519346176734 (+0.029043020654312218) | > avg_loss_disc_real_1: 0.2378572778592647 (+0.013601658734637256) | > avg_loss_disc_real_2: 0.2422537323034985 (+0.010354984184386046) | > avg_loss_disc_real_3: 0.23628451937521008 (-0.0007880628108978271) | > avg_loss_disc_real_4: 0.24494324771451279 (+0.0004129942873833947) | > avg_loss_disc_real_5: 0.24341276141119675 (-0.003398506364352266) | > avg_loss_0: 2.7532864859406376 (+0.06501556114411677) | > avg_grad_norm_0: tensor(63.4886, device='cuda:0') (tensor(-12.1468, device='cuda:0')) | > avg_loss_gen: 2.067383257436081 (+0.09621488544302936) | > avg_loss_kl: 2.669777719067856 (+0.6144985094876367) | > avg_loss_feat: 3.323611165436221 (+0.09224567950611373) | > avg_loss_mel: 19.53394163158578 (+2.3251740227282838) | > avg_loss_duration: 1.6542826068233434 (+0.06910819234982313) | > avg_amp_scaler: 16.0 (-1.1267605633802802) | > avg_loss_1: 29.248996412250357 (+3.1972413398850072) | > avg_grad_norm_1: tensor(3900.8894, device='cuda:0') (tensor(-1487.7634, device='cuda:0'))  > EPOCH: 522/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:34:36)   --> TIME: 2025-02-25 13:34:45 -- STEP: 15/72 -- GLOBAL_STEP: 1037600 | > loss_disc: 2.774492025375366 (2.685695695877075) | > loss_disc_real_0: 0.0989004448056221 (0.12502156024177868) | > loss_disc_real_1: 0.2125813066959381 (0.2348404735326767) | > loss_disc_real_2: 0.19231903553009033 (0.23209451039632162) | > loss_disc_real_3: 0.22799065709114075 (0.23978420694669086) | > loss_disc_real_4: 0.21660540997982025 (0.2615913957357407) | > loss_disc_real_5: 0.2690316438674927 (0.27705840865770975) | > loss_0: 2.774492025375366 (2.685695695877075) | > grad_norm_0: tensor(23.4733, device='cuda:0') (tensor(59.0021, device='cuda:0')) | > loss_gen: 2.024512767791748 (2.3008801380793256) | > loss_kl: 2.9928016662597656 (3.7246396541595463) | > loss_feat: 3.2463200092315674 (3.983312718073526) | > loss_mel: 21.79526138305664 (23.56632754007975) | > loss_duration: 1.623152256011963 (1.680936320622762) | > amp_scaler: 16.0 (16.0) | > loss_1: 31.68204689025879 (35.256096267700194) | > grad_norm_1: tensor(4617.8369, device='cuda:0') (tensor(7084.3848, device='cuda:0')) | > current_lr_0: 0.00018736588099176892 | > current_lr_1: 0.00018736588099176892 | > step_time: 0.4377 (0.4359930992126465) | > loader_time: 0.0059 (0.006859699885050456)  --> TIME: 2025-02-25 13:35:09 -- STEP: 65/72 -- GLOBAL_STEP: 1037650 | > loss_disc: 2.528757333755493 (2.68437243608328) | > loss_disc_real_0: 0.22102870047092438 (0.15454606774907845) | > loss_disc_real_1: 0.27988675236701965 (0.23204290247880496) | > loss_disc_real_2: 0.21423733234405518 (0.2313070746568533) | > loss_disc_real_3: 0.20843034982681274 (0.23314297726521124) | > loss_disc_real_4: 0.2317861020565033 (0.24969849219689003) | > loss_disc_real_5: 0.24136947095394135 (0.25119312015863554) | > loss_0: 2.528757333755493 (2.68437243608328) | > grad_norm_0: tensor(41.6704, device='cuda:0') (tensor(65.9809, device='cuda:0')) | > loss_gen: 2.4669618606567383 (2.0984398988577038) | > loss_kl: 2.252811908721924 (2.6109908012243417) | > loss_feat: 3.5809617042541504 (3.3932967369373026) | > loss_mel: 18.401165008544922 (19.626495126577524) | > loss_duration: 1.6387755870819092 (1.632607784638038) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.340675354003906 (29.361830403254583) | > grad_norm_1: tensor(6888.2349, device='cuda:0') (tensor(5462.3320, device='cuda:0')) | > current_lr_0: 0.00018736588099176892 | > current_lr_1: 0.00018736588099176892 | > step_time: 0.5136 (0.4509713172912598) | > loader_time: 0.0117 (0.008457099474393402) --> EVAL PERFORMANCE | > avg_loader_time: 0.008729535089412203 (+0.00019990558355626777) | > avg_step_time: 0.46355630646289236 (+0.007871214772613988) | > avg_loss_disc: 2.6809866226894754 (-0.07229986325116222) | > avg_loss_disc_real_0: 0.15510414766383843 (-0.03368104579792891) | > avg_loss_disc_real_1: 0.23084841689593355 (-0.007008860963331143) | > avg_loss_disc_real_2: 0.2320153576807237 (-0.010238374622774798) | > avg_loss_disc_real_3: 0.23286434357434932 (-0.003420175800860764) | > avg_loss_disc_real_4: 0.2500159524276223 (+0.005072704713109527) | > avg_loss_disc_real_5: 0.25105351671366605 (+0.007640755302469299) | > avg_loss_0: 2.6809866226894754 (-0.07229986325116222) | > avg_grad_norm_0: tensor(66.6539, device='cuda:0') (+tensor(3.1652, device='cuda:0')) | > avg_loss_gen: 2.092075878465679 (+0.02469262102959835) | > avg_loss_kl: 2.5772734880447388 (-0.09250423102311744) | > avg_loss_feat: 3.388024934580628 (+0.06441376914440733) | > avg_loss_mel: 19.475609040596115 (-0.05833259098966437) | > avg_loss_duration: 1.6340146182288586 (-0.020267988594484754) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.16699801700216 (-0.08199839524819552) | > avg_grad_norm_1: tensor(5486.3188, device='cuda:0') (+tensor(1585.4294, device='cuda:0'))  > EPOCH: 523/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:35:14)   --> TIME: 2025-02-25 13:35:35 -- STEP: 43/72 -- GLOBAL_STEP: 1037700 | > loss_disc: 2.7108089923858643 (2.7294297772784564) | > loss_disc_real_0: 0.21672503650188446 (0.17839394388503807) | > loss_disc_real_1: 0.14547491073608398 (0.23330682689367338) | > loss_disc_real_2: 0.2765713632106781 (0.24730060734721118) | > loss_disc_real_3: 0.21855391561985016 (0.2504807974016944) | > loss_disc_real_4: 0.239979550242424 (0.24764241901941078) | > loss_disc_real_5: 0.2373979389667511 (0.2561403904543367) | > loss_0: 2.7108089923858643 (2.7294297772784564) | > grad_norm_0: tensor(41.6813, device='cuda:0') (tensor(56.0624, device='cuda:0')) | > loss_gen: 1.7765343189239502 (2.169752328894859) | > loss_kl: 2.4112894535064697 (3.0765309222908908) | > loss_feat: 3.010382652282715 (3.5495728670164595) | > loss_mel: 19.35187339782715 (21.177684340366095) | > loss_duration: 1.695896029472351 (1.8049040062482966) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.245975494384766 (31.778444334518074) | > grad_norm_1: tensor(2910.7344, device='cuda:0') (tensor(4371.3696, device='cuda:0')) | > current_lr_0: 0.00018734246025664495 | > current_lr_1: 0.00018734246025664495 | > step_time: 0.4607 (0.4428063381549924) | > loader_time: 0.0097 (0.007438110750775004) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640551231276822 (-8.898385813538127e-05) | > avg_step_time: 0.4571542269747022 (-0.00640207948819016) | > avg_loss_disc: 2.723973042528394 (+0.04298641983891871) | > avg_loss_disc_real_0: 0.17862836701769225 (+0.02352421935385382) | > avg_loss_disc_real_1: 0.2337056641847315 (+0.002857247288797937) | > avg_loss_disc_real_2: 0.24409676133327082 (+0.012081403652547118) | > avg_loss_disc_real_3: 0.2435056556698302 (+0.010641312095480887) | > avg_loss_disc_real_4: 0.24672902868667118 (-0.003286923740951131) | > avg_loss_disc_real_5: 0.24927942748640625 (-0.0017740892272598008) | > avg_loss_0: 2.723973042528394 (+0.04298641983891871) | > avg_grad_norm_0: tensor(51.2854, device='cuda:0') (tensor(-15.3685, device='cuda:0')) | > avg_loss_gen: 2.073973518022348 (-0.018102360443331023) | > avg_loss_kl: 2.769183615563621 (+0.19191012751888215) | > avg_loss_feat: 3.30724582201998 (-0.08077911256064807) | > avg_loss_mel: 20.1021298690581 (+0.6265208284619845) | > avg_loss_duration: 1.7560752277642908 (+0.12206060953543219) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 30.008607944972077 (+0.8416099279699161) | > avg_grad_norm_1: tensor(3875.1360, device='cuda:0') (tensor(-1611.1829, device='cuda:0'))  > EPOCH: 524/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:35:51)   --> TIME: 2025-02-25 13:36:02 -- STEP: 21/72 -- GLOBAL_STEP: 1037750 | > loss_disc: 2.6826517581939697 (2.7136409396216985) | > loss_disc_real_0: 0.10703655332326889 (0.1854506049837385) | > loss_disc_real_1: 0.18409572541713715 (0.2239099322330384) | > loss_disc_real_2: 0.2014739215373993 (0.23695809074810573) | > loss_disc_real_3: 0.18545174598693848 (0.23877823991434916) | > loss_disc_real_4: 0.2200872004032135 (0.24215344587961832) | > loss_disc_real_5: 0.23179663717746735 (0.24508404093129293) | > loss_0: 2.6826517581939697 (2.7136409396216985) | > grad_norm_0: tensor(23.3397, device='cuda:0') (tensor(49.6530, device='cuda:0')) | > loss_gen: 1.8359442949295044 (1.9751063471748715) | > loss_kl: 2.030579090118408 (2.0812137751352213) | > loss_feat: 3.558016300201416 (3.236802271434239) | > loss_mel: 17.6391658782959 (17.927216484433128) | > loss_duration: 1.612442135810852 (1.5983023302895683) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.6761474609375 (26.818641480945406) | > grad_norm_1: tensor(2341.8621, device='cuda:0') (tensor(3493.2961, device='cuda:0')) | > current_lr_0: 0.00018731904244911286 | > current_lr_1: 0.00018731904244911286 | > step_time: 0.4447 (0.4329809120723179) | > loader_time: 0.0073 (0.006689003535679409)  --> TIME: 2025-02-25 13:36:26 -- STEP: 71/72 -- GLOBAL_STEP: 1037800 | > loss_disc: 2.538752555847168 (2.709521992105834) | > loss_disc_real_0: 0.0854097306728363 (0.17948496918862974) | > loss_disc_real_1: 0.19306620955467224 (0.22493713842311375) | > loss_disc_real_2: 0.20390623807907104 (0.24038533656529978) | > loss_disc_real_3: 0.23343946039676666 (0.23535309060358664) | > loss_disc_real_4: 0.22711513936519623 (0.24153526673014736) | > loss_disc_real_5: 0.2200235277414322 (0.24310247998842052) | > loss_0: 2.538752555847168 (2.709521992105834) | > grad_norm_0: tensor(73.9348, device='cuda:0') (tensor(49.4715, device='cuda:0')) | > loss_gen: 1.8810460567474365 (1.9757393682506723) | > loss_kl: 2.1175804138183594 (2.1553798796425396) | > loss_feat: 3.5378262996673584 (3.1768981940309766) | > loss_mel: 16.977216720581055 (17.94997333472884) | > loss_duration: 1.5986101627349854 (1.6162300378503933) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.112279891967773 (26.874220888379597) | > grad_norm_1: tensor(5457.2979, device='cuda:0') (tensor(3966.9194, device='cuda:0')) | > current_lr_0: 0.00018731904244911286 | > current_lr_1: 0.00018731904244911286 | > step_time: 0.5057 (0.45690682236577423) | > loader_time: 0.0081 (0.008717513420212434) --> EVAL PERFORMANCE | > avg_loader_time: 0.008717513420212434 (+7.696218893561181e-05) | > avg_step_time: 0.45690682236577423 (-0.0002474046089279702) | > avg_loss_disc: 2.709521992105834 (-0.014451050422560119) | > avg_loss_disc_real_0: 0.17948496918862974 (+0.0008566021709374982) | > avg_loss_disc_real_1: 0.22493713842311375 (-0.00876852576161774) | > avg_loss_disc_real_2: 0.24038533656529978 (-0.003711424767971039) | > avg_loss_disc_real_3: 0.23535309060358664 (-0.008152565066243567) | > avg_loss_disc_real_4: 0.24153526673014736 (-0.0051937619565238236) | > avg_loss_disc_real_5: 0.24310247998842052 (-0.006176947497985724) | > avg_loss_0: 2.709521992105834 (-0.014451050422560119) | > avg_grad_norm_0: tensor(49.4715, device='cuda:0') (tensor(-1.8139, device='cuda:0')) | > avg_loss_gen: 1.9757393682506723 (-0.09823414977167588) | > avg_loss_kl: 2.1553798796425396 (-0.6138037359210813) | > avg_loss_feat: 3.1768981940309766 (-0.13034762798900346) | > avg_loss_mel: 17.94997333472884 (-2.1521565343292615) | > avg_loss_duration: 1.6162300378503933 (-0.1398451899138975) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.874220888379597 (-3.1343870565924803) | > avg_grad_norm_1: tensor(3966.9194, device='cuda:0') (+tensor(91.7834, device='cuda:0'))  > EPOCH: 525/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:36:28)   --> TIME: 2025-02-25 13:36:53 -- STEP: 49/72 -- GLOBAL_STEP: 1037850 | > loss_disc: 2.863210678100586 (2.7210880785572287) | > loss_disc_real_0: 0.16566546261310577 (0.15457311090157955) | > loss_disc_real_1: 0.18222591280937195 (0.22370499037966438) | > loss_disc_real_2: 0.3051412105560303 (0.24186926623996424) | > loss_disc_real_3: 0.24075792729854584 (0.24358229430354372) | > loss_disc_real_4: 0.2515072524547577 (0.24443255334484335) | > loss_disc_real_5: 0.25928181409835815 (0.248147045167125) | > loss_0: 2.863210678100586 (2.7210880785572287) | > grad_norm_0: tensor(55.6168, device='cuda:0') (tensor(56.2991, device='cuda:0')) | > loss_gen: 1.8763408660888672 (1.9737813472747803) | > loss_kl: 2.2356913089752197 (2.2555386776826816) | > loss_feat: 2.8967723846435547 (3.19781147217264) | > loss_mel: 18.920612335205078 (18.68161684153031) | > loss_duration: 1.640720248222351 (1.599647312748189) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.57013702392578 (27.70839566600566) | > grad_norm_1: tensor(2258.6035, device='cuda:0') (tensor(3985.1489, device='cuda:0')) | > current_lr_0: 0.0001872956275688067 | > current_lr_1: 0.0001872956275688067 | > step_time: 0.4508 (0.4559697375005605) | > loader_time: 0.0098 (0.007889591917699697) --> EVAL PERFORMANCE | > avg_loader_time: 0.008842612656069473 (+0.0001250992358570388) | > avg_step_time: 0.4670800188897361 (+0.01017319652396187) | > avg_loss_disc: 2.716261306279142 (+0.006739314173307864) | > avg_loss_disc_real_0: 0.1614947329524537 (-0.017990236236176044) | > avg_loss_disc_real_1: 0.22706199720711776 (+0.002124858784004008) | > avg_loss_disc_real_2: 0.2445994877899197 (+0.004214151224619905) | > avg_loss_disc_real_3: 0.2395887274137685 (+0.004235636810181853) | > avg_loss_disc_real_4: 0.2417040467262268 (+0.00016877999607944894) | > avg_loss_disc_real_5: 0.24500755462008464 (+0.0019050746316641165) | > avg_loss_0: 2.716261306279142 (+0.006739314173307864) | > avg_grad_norm_0: tensor(55.7409, device='cuda:0') (+tensor(6.2694, device='cuda:0')) | > avg_loss_gen: 1.9670134279089915 (-0.008725940341680838) | > avg_loss_kl: 2.2795536853897738 (+0.12417380574723413) | > avg_loss_feat: 3.1516674840953987 (-0.025230709935577966) | > avg_loss_mel: 18.64073142199449 (+0.6907580872656531) | > avg_loss_duration: 1.6115391204054927 (-0.0046909174449005775) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.65050506591797 (+0.776284177538372) | > avg_grad_norm_1: tensor(3769.9316, device='cuda:0') (tensor(-196.9878, device='cuda:0'))  > EPOCH: 526/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:37:06)   --> TIME: 2025-02-25 13:37:20 -- STEP: 27/72 -- GLOBAL_STEP: 1037900 | > loss_disc: 2.6930205821990967 (2.6628438543390343) | > loss_disc_real_0: 0.12754172086715698 (0.1532377288849265) | > loss_disc_real_1: 0.1980111002922058 (0.21956945293479496) | > loss_disc_real_2: 0.21631760895252228 (0.22764120554482495) | > loss_disc_real_3: 0.2232099324464798 (0.23112820088863373) | > loss_disc_real_4: 0.20978336036205292 (0.2405757274892595) | > loss_disc_real_5: 0.22148151695728302 (0.2413745743257028) | > loss_0: 2.6930205821990967 (2.6628438543390343) | > grad_norm_0: tensor(68.2388, device='cuda:0') (tensor(58.5828, device='cuda:0')) | > loss_gen: 1.7626338005065918 (1.9586150204693829) | > loss_kl: 2.201371908187866 (2.0930640697479244) | > loss_feat: 3.6082639694213867 (3.272907018661499) | > loss_mel: 18.810781478881836 (18.036614594636138) | > loss_duration: 1.589150309562683 (1.5633761485417683) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.97220230102539 (26.924576512089484) | > grad_norm_1: tensor(3577.7927, device='cuda:0') (tensor(3792.4463, device='cuda:0')) | > current_lr_0: 0.0001872722156153606 | > current_lr_1: 0.0001872722156153606 | > step_time: 0.4526 (0.43995175538239656) | > loader_time: 0.0076 (0.006979801036693432) --> EVAL PERFORMANCE | > avg_loader_time: 0.008699155189621614 (-0.00014345746644785912) | > avg_step_time: 0.45806341775706116 (-0.00901660113267494) | > avg_loss_disc: 2.6994346766404704 (-0.01682662963867143) | > avg_loss_disc_real_0: 0.17350311419913467 (+0.012008381246680966) | > avg_loss_disc_real_1: 0.22409235791001522 (-0.002969639297102533) | > avg_loss_disc_real_2: 0.23201748917640094 (-0.012581998613518747) | > avg_loss_disc_real_3: 0.23763706167818796 (-0.0019516657355805322) | > avg_loss_disc_real_4: 0.24232944069613874 (+0.0006253939699119304) | > avg_loss_disc_real_5: 0.24428943618082663 (-0.0007181184392580087) | > avg_loss_0: 2.6994346766404704 (-0.01682662963867143) | > avg_grad_norm_0: tensor(56.3434, device='cuda:0') (+tensor(0.6024, device='cuda:0')) | > avg_loss_gen: 1.9454204901843004 (-0.021592937724691064) | > avg_loss_kl: 2.1322756367669986 (-0.14727804862277516) | > avg_loss_feat: 3.1334628320076097 (-0.018204652087788986) | > avg_loss_mel: 17.776296615600582 (-0.8644348063939091) | > avg_loss_duration: 1.5973630354438029 (-0.014176084961689872) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.584818477361974 (-1.0656865885559945) | > avg_grad_norm_1: tensor(3705.9771, device='cuda:0') (tensor(-63.9546, device='cuda:0'))  > EPOCH: 527/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:37:43)   --> TIME: 2025-02-25 13:37:47 -- STEP: 5/72 -- GLOBAL_STEP: 1037950 | > loss_disc: 2.7485029697418213 (2.692521810531616) | > loss_disc_real_0: 0.24329575896263123 (0.1653006374835968) | > loss_disc_real_1: 0.23597823083400726 (0.21331863701343537) | > loss_disc_real_2: 0.23059049248695374 (0.2065921813249588) | > loss_disc_real_3: 0.2369912713766098 (0.2307147353887558) | > loss_disc_real_4: 0.2540167272090912 (0.2549133777618408) | > loss_disc_real_5: 0.23775091767311096 (0.23206427693367004) | > loss_0: 2.7485029697418213 (2.692521810531616) | > grad_norm_0: tensor(36.8019, device='cuda:0') (tensor(51.7408, device='cuda:0')) | > loss_gen: 1.9072911739349365 (1.9814751148223877) | > loss_kl: 1.922973394393921 (2.0096344232559202) | > loss_feat: 2.966299057006836 (3.2860436916351317) | > loss_mel: 16.972810745239258 (17.300378036499023) | > loss_duration: 1.5185422897338867 (1.5720484018325807) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.287918090820312 (26.149580001831055) | > grad_norm_1: tensor(1156.4155, device='cuda:0') (tensor(3405.2351, device='cuda:0')) | > current_lr_0: 0.00018724880658840866 | > current_lr_1: 0.00018724880658840866 | > step_time: 0.4253 (0.43591651916503904) | > loader_time: 0.006 (0.006487846374511719)  --> TIME: 2025-02-25 13:38:11 -- STEP: 55/72 -- GLOBAL_STEP: 1038000 | > loss_disc: 2.5887961387634277 (2.7067813049663196) | > loss_disc_real_0: 0.17806504666805267 (0.1478151719678532) | > loss_disc_real_1: 0.1966175138950348 (0.22288881811228664) | > loss_disc_real_2: 0.23566748201847076 (0.23284282657233152) | > loss_disc_real_3: 0.23379310965538025 (0.23519974567673424) | > loss_disc_real_4: 0.22105179727077484 (0.24185373457995327) | > loss_disc_real_5: 0.19280429184436798 (0.2454500751061873) | > loss_0: 2.5887961387634277 (2.7067813049663196) | > grad_norm_0: tensor(66.4506, device='cuda:0') (tensor(64.4259, device='cuda:0')) | > loss_gen: 2.0919926166534424 (1.9061688488179986) | > loss_kl: 2.2046332359313965 (2.237281799316406) | > loss_feat: 3.490199565887451 (3.1466768741607667) | > loss_mel: 17.940662384033203 (17.793615930730642) | > loss_duration: 1.6338238716125488 (1.58931114023382) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.361310958862305 (26.673054504394532) | > grad_norm_1: tensor(6216.6323, device='cuda:0') (tensor(4169.6270, device='cuda:0')) | > current_lr_0: 0.00018724880658840866 | > current_lr_1: 0.00018724880658840866 | > step_time: 0.4599 (0.44896502494812013) | > loader_time: 0.0094 (0.008092156323519624) --> EVAL PERFORMANCE | > avg_loader_time: 0.008846108342560248 (+0.000146953152938634) | > avg_step_time: 0.4596904096469073 (+0.0016269918898461633) | > avg_loss_disc: 2.704688371067316 (+0.005253694426845534) | > avg_loss_disc_real_0: 0.1509297064072649 (-0.022573407791869754) | > avg_loss_disc_real_1: 0.22305010280139007 (-0.0010422551086251564) | > avg_loss_disc_real_2: 0.23474105101236156 (+0.0027235618359606195) | > avg_loss_disc_real_3: 0.23485224452656758 (-0.002784817151620378) | > avg_loss_disc_real_4: 0.24228115044009518 (-4.829025604355386e-05) | > avg_loss_disc_real_5: 0.24653810535518217 (+0.0022486691743555387) | > avg_loss_0: 2.704688371067316 (+0.005253694426845534) | > avg_grad_norm_0: tensor(67.6841, device='cuda:0') (+tensor(11.3408, device='cuda:0')) | > avg_loss_gen: 1.909289198861995 (-0.0361312913223053) | > avg_loss_kl: 2.213513822622702 (+0.0812381858557032) | > avg_loss_feat: 3.1139529322234676 (-0.019509899784142082) | > avg_loss_mel: 17.61575453046342 (-0.16054208513716262) | > avg_loss_duration: 1.5987856522412367 (+0.0014226167974338289) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.451295986981464 (-0.13352249038051056) | > avg_grad_norm_1: tensor(4173.7446, device='cuda:0') (+tensor(467.7676, device='cuda:0'))  > EPOCH: 528/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:38:21)   --> TIME: 2025-02-25 13:38:38 -- STEP: 33/72 -- GLOBAL_STEP: 1038050 | > loss_disc: 2.5631020069122314 (2.65650057070183) | > loss_disc_real_0: 0.16227802634239197 (0.15806012352307636) | > loss_disc_real_1: 0.2492842823266983 (0.22211059295769894) | > loss_disc_real_2: 0.19025762379169464 (0.22782692087419104) | > loss_disc_real_3: 0.20817957818508148 (0.2315026024977366) | > loss_disc_real_4: 0.25074753165245056 (0.24369709103396445) | > loss_disc_real_5: 0.26114872097969055 (0.2426453625613993) | > loss_0: 2.5631020069122314 (2.65650057070183) | > grad_norm_0: tensor(56.6592, device='cuda:0') (tensor(71.0046, device='cuda:0')) | > loss_gen: 2.04978084564209 (1.9542570475376013) | > loss_kl: 2.1904711723327637 (1.9962915434981838) | > loss_feat: 3.2192842960357666 (3.2644132339593135) | > loss_mel: 17.11485481262207 (17.320228027574952) | > loss_duration: 1.561091661453247 (1.5543437401453655) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.135482788085938 (26.08953366135106) | > grad_norm_1: tensor(6170.1538, device='cuda:0') (tensor(4629.0083, device='cuda:0')) | > current_lr_0: 0.0001872254004875851 | > current_lr_1: 0.0001872254004875851 | > step_time: 0.4426 (0.45251072536815296) | > loader_time: 0.008 (0.007069587707519531) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598055638058084 (-0.00024805270450216355) | > avg_step_time: 0.4617541037814718 (+0.0020636941345644755) | > avg_loss_disc: 2.6704641698111957 (-0.034224201256120246) | > avg_loss_disc_real_0: 0.1578772160368906 (+0.0069475096296256755) | > avg_loss_disc_real_1: 0.2205257340216301 (-0.002524368779759978) | > avg_loss_disc_real_2: 0.22887574262182478 (-0.005865308390536783) | > avg_loss_disc_real_3: 0.23181802852892539 (-0.003034215997642198) | > avg_loss_disc_real_4: 0.24147225307746673 (-0.0008088973626284579) | > avg_loss_disc_real_5: 0.24447825131282 (-0.002059854042362169) | > avg_loss_0: 2.6704641698111957 (-0.034224201256120246) | > avg_grad_norm_0: tensor(74.4042, device='cuda:0') (+tensor(6.7201, device='cuda:0')) | > avg_loss_gen: 1.944203215585628 (+0.03491401672363281) | > avg_loss_kl: 2.0338014884733826 (-0.1797123341493192) | > avg_loss_feat: 3.207113927518818 (+0.09316099529535027) | > avg_loss_mel: 17.12282945068789 (-0.492925079775528) | > avg_loss_duration: 1.5865285195095435 (-0.012257132731693154) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.894476662219407 (-0.5568193247620563) | > avg_grad_norm_1: tensor(5014.6445, device='cuda:0') (+tensor(840.8999, device='cuda:0'))  > EPOCH: 529/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:38:58)   --> TIME: 2025-02-25 13:39:05 -- STEP: 11/72 -- GLOBAL_STEP: 1038100 | > loss_disc: 2.587730884552002 (2.652915607799183) | > loss_disc_real_0: 0.11363518238067627 (0.15899154272946445) | > loss_disc_real_1: 0.22684504091739655 (0.2181126353415576) | > loss_disc_real_2: 0.17940378189086914 (0.24374928935007614) | > loss_disc_real_3: 0.21959660947322845 (0.24417253922332416) | > loss_disc_real_4: 0.2502613961696625 (0.25078210505572235) | > loss_disc_real_5: 0.26355108618736267 (0.24741402945735239) | > loss_0: 2.587730884552002 (2.652915607799183) | > grad_norm_0: tensor(57.3083, device='cuda:0') (tensor(73.3684, device='cuda:0')) | > loss_gen: 1.9629122018814087 (2.004914305426858) | > loss_kl: 1.8612810373306274 (1.96292029727589) | > loss_feat: 3.169433116912842 (3.2420567599209873) | > loss_mel: 16.807273864746094 (17.015165849165484) | > loss_duration: 1.5187162160873413 (1.546097831292586) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.319616317749023 (25.771154577081855) | > grad_norm_1: tensor(5554.4624, device='cuda:0') (tensor(5673.7993, device='cuda:0')) | > current_lr_0: 0.00018720199731252416 | > current_lr_1: 0.00018720199731252416 | > step_time: 0.4363 (0.4364560300653631) | > loader_time: 0.0071 (0.005956606431440873)  --> TIME: 2025-02-25 13:39:28 -- STEP: 61/72 -- GLOBAL_STEP: 1038150 | > loss_disc: 2.683565378189087 (2.670450065956741) | > loss_disc_real_0: 0.13300850987434387 (0.162424342798405) | > loss_disc_real_1: 0.24315355718135834 (0.22263352182067808) | > loss_disc_real_2: 0.2785069942474365 (0.23486049741995138) | > loss_disc_real_3: 0.24217642843723297 (0.23543413887258435) | > loss_disc_real_4: 0.25671905279159546 (0.24248272033988452) | > loss_disc_real_5: 0.24535951018333435 (0.24500332308597253) | > loss_0: 2.683565378189087 (2.670450065956741) | > grad_norm_0: tensor(116.7793, device='cuda:0') (tensor(82.1637, device='cuda:0')) | > loss_gen: 1.983144760131836 (1.9782048561533943) | > loss_kl: 2.1022865772247314 (2.007626062533894) | > loss_feat: 2.846790075302124 (3.2278986172597923) | > loss_mel: 16.585176467895508 (17.129877606376276) | > loss_duration: 1.622036099433899 (1.57343617423636) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.139432907104492 (25.917043060552878) | > grad_norm_1: tensor(7479.8105, device='cuda:0') (tensor(5491.0693, device='cuda:0')) | > current_lr_0: 0.00018720199731252416 | > current_lr_1: 0.00018720199731252416 | > step_time: 0.4679 (0.45222231599151114) | > loader_time: 0.0111 (0.008250557008336804) --> EVAL PERFORMANCE | > avg_loader_time: 0.008847606014197982 (+0.00024955037613989817) | > avg_step_time: 0.461458360645133 (-0.00029574313633878857) | > avg_loss_disc: 2.6678990075286007 (-0.002565162282595068) | > avg_loss_disc_real_0: 0.16145044869520295 (+0.0035732326583123586) | > avg_loss_disc_real_1: 0.22155469551052845 (+0.0010289614888983611) | > avg_loss_disc_real_2: 0.235395191211096 (+0.006519448589271226) | > avg_loss_disc_real_3: 0.2346448988561899 (+0.0028268703272645024) | > avg_loss_disc_real_4: 0.24153229426330244 (+6.00411858357186e-05) | > avg_loss_disc_real_5: 0.24428752274580404 (-0.00019072856701596308) | > avg_loss_0: 2.6678990075286007 (-0.002565162282595068) | > avg_grad_norm_0: tensor(82.4332, device='cuda:0') (+tensor(8.0290, device='cuda:0')) | > avg_loss_gen: 1.9775897314850712 (+0.0333865158994433) | > avg_loss_kl: 2.0200222797796754 (-0.01377920869370719) | > avg_loss_feat: 3.215184403137422 (+0.008070475618604078) | > avg_loss_mel: 17.141763028964192 (+0.018933578276300267) | > avg_loss_duration: 1.582077810462092 (-0.004450709047451484) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.936637072496012 (+0.042160410276604665) | > avg_grad_norm_1: tensor(5589.5703, device='cuda:0') (+tensor(574.9258, device='cuda:0'))  > EPOCH: 530/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:39:35)   --> TIME: 2025-02-25 13:39:55 -- STEP: 39/72 -- GLOBAL_STEP: 1038200 | > loss_disc: 2.616844415664673 (2.6703087305411315) | > loss_disc_real_0: 0.20670700073242188 (0.16642176512724313) | > loss_disc_real_1: 0.23292295634746552 (0.2259550086962871) | > loss_disc_real_2: 0.22850024700164795 (0.23060764830846053) | > loss_disc_real_3: 0.22942806780338287 (0.23448793972149873) | > loss_disc_real_4: 0.2526751756668091 (0.24266681227928552) | > loss_disc_real_5: 0.26577991247177124 (0.24895212780206633) | > loss_0: 2.616844415664673 (2.6703087305411315) | > grad_norm_0: tensor(38.6412, device='cuda:0') (tensor(88.5692, device='cuda:0')) | > loss_gen: 2.0066654682159424 (2.0039675541413136) | > loss_kl: 2.1506941318511963 (2.0134527347026734) | > loss_feat: 3.396303176879883 (3.3007685037759633) | > loss_mel: 17.969402313232422 (17.085048210926537) | > loss_duration: 1.5911287069320679 (1.554126718105414) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.114194869995117 (25.957363715538612) | > grad_norm_1: tensor(6185.2534, device='cuda:0') (tensor(5804.8892, device='cuda:0')) | > current_lr_0: 0.0001871785970628601 | > current_lr_1: 0.0001871785970628601 | > step_time: 0.4578 (0.44242438903221715) | > loader_time: 0.0083 (0.0075524648030598955) --> EVAL PERFORMANCE | > avg_loader_time: 0.00890138451482209 (+5.377850062410726e-05) | > avg_step_time: 0.4581754442671655 (-0.0032829163779675152) | > avg_loss_disc: 2.6842023553982592 (+0.01630334786965859) | > avg_loss_disc_real_0: 0.16283428212496598 (+0.0013838334297630317) | > avg_loss_disc_real_1: 0.22409490065675386 (+0.0025402051462254105) | > avg_loss_disc_real_2: 0.23881418373383267 (+0.003418992522736669) | > avg_loss_disc_real_3: 0.2358161294963998 (+0.0011712306402099082) | > avg_loss_disc_real_4: 0.24277366802725994 (+0.0012413737639574984) | > avg_loss_disc_real_5: 0.24637222583864776 (+0.002084703092843726) | > avg_loss_0: 2.6842023553982592 (+0.01630334786965859) | > avg_grad_norm_0: tensor(86.0431, device='cuda:0') (+tensor(3.6099, device='cuda:0')) | > avg_loss_gen: 1.9815087855701716 (+0.003919054085100404) | > avg_loss_kl: 2.059411968983395 (+0.03938968920371977) | > avg_loss_feat: 3.24524269977086 (+0.030058296633438175) | > avg_loss_mel: 16.972349945928002 (-0.16941308303618996) | > avg_loss_duration: 1.5813133363992395 (-0.000764474062852516) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.839826745046697 (-0.09681032744931528) | > avg_grad_norm_1: tensor(5633.9976, device='cuda:0') (+tensor(44.4272, device='cuda:0'))  > EPOCH: 531/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:40:13)   --> TIME: 2025-02-25 13:40:22 -- STEP: 17/72 -- GLOBAL_STEP: 1038250 | > loss_disc: 2.768634080886841 (2.751911864561193) | > loss_disc_real_0: 0.07708938419818878 (0.19989020596532261) | > loss_disc_real_1: 0.2039794623851776 (0.22034299724242268) | > loss_disc_real_2: 0.2401309311389923 (0.2477199952392017) | > loss_disc_real_3: 0.22873854637145996 (0.23502621668226578) | > loss_disc_real_4: 0.24064350128173828 (0.24386340116753297) | > loss_disc_real_5: 0.25001925230026245 (0.2503247138331918) | > loss_0: 2.768634080886841 (2.751911864561193) | > grad_norm_0: tensor(72.7858, device='cuda:0') (tensor(73.7997, device='cuda:0')) | > loss_gen: 1.799736738204956 (1.954122830839718) | > loss_kl: 2.371307611465454 (2.031916106448454) | > loss_feat: 3.0041372776031494 (3.1401695083169376) | > loss_mel: 18.110919952392578 (17.223993750179517) | > loss_duration: 1.5955876111984253 (1.5350884479634903) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.88168716430664 (25.88529093125287) | > grad_norm_1: tensor(4311.7715, device='cuda:0') (tensor(4212.5557, device='cuda:0')) | > current_lr_0: 0.00018715519973822723 | > current_lr_1: 0.00018715519973822723 | > step_time: 0.4389 (0.43516095946816835) | > loader_time: 0.0065 (0.006974416620591108)  --> TIME: 2025-02-25 13:40:46 -- STEP: 67/72 -- GLOBAL_STEP: 1038300 | > loss_disc: 2.7072722911834717 (2.7497551868210968) | > loss_disc_real_0: 0.12800922989845276 (0.16879717193877525) | > loss_disc_real_1: 0.19936293363571167 (0.225410787249679) | > loss_disc_real_2: 0.18761086463928223 (0.24663365954783426) | > loss_disc_real_3: 0.21591445803642273 (0.2374332991998587) | > loss_disc_real_4: 0.22938337922096252 (0.2451253862523321) | > loss_disc_real_5: 0.24561376869678497 (0.24651870887670943) | > loss_0: 2.7072722911834717 (2.7497551868210968) | > grad_norm_0: tensor(89.9301, device='cuda:0') (tensor(86.1947, device='cuda:0')) | > loss_gen: 1.5224993228912354 (1.9093049818010472) | > loss_kl: 2.27836012840271 (2.090638568152242) | > loss_feat: 2.656136989593506 (3.1277322733580175) | > loss_mel: 15.978878021240234 (16.967490352801416) | > loss_duration: 1.657064437866211 (1.5792615075609577) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.092939376831055 (25.674427829571624) | > grad_norm_1: tensor(3745.3708, device='cuda:0') (tensor(4767.3716, device='cuda:0')) | > current_lr_0: 0.00018715519973822723 | > current_lr_1: 0.00018715519973822723 | > step_time: 0.528 (0.4602419582765494) | > loader_time: 0.0123 (0.00852077754575815) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693721932424629 (-0.00020766258239746094) | > avg_step_time: 0.4637084880345304 (+0.005533043767364909) | > avg_loss_disc: 2.750578578089324 (+0.06637622269106469) | > avg_loss_disc_real_0: 0.16228651753823528 (-0.0005477645867306979) | > avg_loss_disc_real_1: 0.2256277281213814 (+0.0015328274646275453) | > avg_loss_disc_real_2: 0.24472400168297995 (+0.005909817949147272) | > avg_loss_disc_real_3: 0.2378932858856631 (+0.0020771563892633127) | > avg_loss_disc_real_4: 0.24557853710483496 (+0.0028048690775750174) | > avg_loss_disc_real_5: 0.24583481126268145 (-0.0005374145759663163) | > avg_loss_0: 2.750578578089324 (+0.06637622269106469) | > avg_grad_norm_0: tensor(86.1629, device='cuda:0') (+tensor(0.1199, device='cuda:0')) | > avg_loss_gen: 1.9036985148846264 (-0.07781027068554525) | > avg_loss_kl: 2.1010305394589053 (+0.0416185704755101) | > avg_loss_feat: 3.1269110760218664 (-0.11833162374899375) | > avg_loss_mel: 16.983825737321872 (+0.011475791393870338) | > avg_loss_duration: 1.5821716886171153 (+0.0008583522178757796) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.697637692303726 (-0.142189052742971) | > avg_grad_norm_1: tensor(4751.1558, device='cuda:0') (tensor(-882.8418, device='cuda:0'))  > EPOCH: 532/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:40:50)   --> TIME: 2025-02-25 13:41:13 -- STEP: 45/72 -- GLOBAL_STEP: 1038350 | > loss_disc: 2.6041066646575928 (2.7388660589853924) | > loss_disc_real_0: 0.14783228933811188 (0.17970810416671965) | > loss_disc_real_1: 0.16530296206474304 (0.2373354739613003) | > loss_disc_real_2: 0.22572080790996552 (0.2332602196269565) | > loss_disc_real_3: 0.3359106183052063 (0.2404563385579321) | > loss_disc_real_4: 0.21704141795635223 (0.2404792884985606) | > loss_disc_real_5: 0.23310834169387817 (0.2423394438293245) | > loss_0: 2.6041066646575928 (2.7388660589853924) | > grad_norm_0: tensor(99.1533, device='cuda:0') (tensor(64.5512, device='cuda:0')) | > loss_gen: 2.051565170288086 (1.9998002290725703) | > loss_kl: 2.4195938110351562 (2.600585749414232) | > loss_feat: 3.3932790756225586 (3.1544148763020834) | > loss_mel: 17.589778900146484 (18.690886221991647) | > loss_duration: 1.6444969177246094 (1.6454269329706828) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.09871482849121 (28.091113959418404) | > grad_norm_1: tensor(6949.4404, device='cuda:0') (tensor(4357.1460, device='cuda:0')) | > current_lr_0: 0.00018713180533825994 | > current_lr_1: 0.00018713180533825994 | > step_time: 0.4944 (0.4465596146053738) | > loader_time: 0.0088 (0.007687446806165907) --> EVAL PERFORMANCE | > avg_loader_time: 0.008691814583791813 (-1.9073486328159694e-06) | > avg_step_time: 0.4607401733667078 (-0.0029683146678226024) | > avg_loss_disc: 2.7228665049646943 (-0.027712073124629644) | > avg_loss_disc_real_0: 0.17403061189491983 (+0.011744094356684553) | > avg_loss_disc_real_1: 0.22954747290678426 (+0.003919744785402857) | > avg_loss_disc_real_2: 0.23606682642244955 (-0.008657175260530392) | > avg_loss_disc_real_3: 0.24070205035763728 (+0.0028087644719741733) | > avg_loss_disc_real_4: 0.24091833997780168 (-0.004660197127033278) | > avg_loss_disc_real_5: 0.2438509619571793 (-0.0019838493055021333) | > avg_loss_0: 2.7228665049646943 (-0.027712073124629644) | > avg_grad_norm_0: tensor(70.9495, device='cuda:0') (tensor(-15.2134, device='cuda:0')) | > avg_loss_gen: 1.9781615180028993 (+0.07446300311827292) | > avg_loss_kl: 2.5045389272797274 (+0.4035083878208221) | > avg_loss_feat: 3.1196699008135726 (-0.007241175208293704) | > avg_loss_mel: 18.194755943728165 (+1.2109302064062923) | > avg_loss_duration: 1.6485384467621924 (+0.06636675814507709) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.445664768487635 (+1.7480270761839094) | > avg_grad_norm_1: tensor(4542.4487, device='cuda:0') (tensor(-208.7070, device='cuda:0'))  > EPOCH: 533/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:41:28)   --> TIME: 2025-02-25 13:41:40 -- STEP: 23/72 -- GLOBAL_STEP: 1038400 | > loss_disc: 2.5761685371398926 (2.6989252878272016) | > loss_disc_real_0: 0.16744540631771088 (0.13928052588649417) | > loss_disc_real_1: 0.24027155339717865 (0.2339279820089755) | > loss_disc_real_2: 0.24712537229061127 (0.23273029534713083) | > loss_disc_real_3: 0.24839816987514496 (0.23484010152194812) | > loss_disc_real_4: 0.25549420714378357 (0.24532404865907587) | > loss_disc_real_5: 0.2600061297416687 (0.24594107723754385) | > loss_0: 2.5761685371398926 (2.6989252878272016) | > grad_norm_0: tensor(77.1888, device='cuda:0') (tensor(86.5899, device='cuda:0')) | > loss_gen: 2.0630805492401123 (1.9509024049924768) | > loss_kl: 1.8396012783050537 (2.085777946140454) | > loss_feat: 3.170639991760254 (3.309311462485272) | > loss_mel: 17.131319046020508 (17.37548707879108) | > loss_duration: 1.5767498016357422 (1.5597371484922327) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.781389236450195 (26.28121583358101) | > grad_norm_1: tensor(5856.6670, device='cuda:0') (tensor(5507.7363, device='cuda:0')) | > current_lr_0: 0.00018710841386259265 | > current_lr_1: 0.00018710841386259265 | > step_time: 0.4367 (0.4360689702241317) | > loader_time: 0.0065 (0.0068240269370701) --> EVAL PERFORMANCE | > avg_loader_time: 0.008683103910634213 (-8.710673157599422e-06) | > avg_step_time: 0.4572038549772451 (-0.0035363183894627226) | > avg_loss_disc: 2.6833255123084694 (-0.039540992656224905) | > avg_loss_disc_real_0: 0.15255152622998594 (-0.021479085664933895) | > avg_loss_disc_real_1: 0.22554598916584337 (-0.004001483740940892) | > avg_loss_disc_real_2: 0.2302335687506367 (-0.005833257671812864) | > avg_loss_disc_real_3: 0.23535501033487455 (-0.0053470400227627335) | > avg_loss_disc_real_4: 0.24453090928809743 (+0.0036125693102957435) | > avg_loss_disc_real_5: 0.24829104500757138 (+0.004440083050392063) | > avg_loss_0: 2.6833255123084694 (-0.039540992656224905) | > avg_grad_norm_0: tensor(86.8270, device='cuda:0') (+tensor(15.8775, device='cuda:0')) | > avg_loss_gen: 1.9561234276059647 (-0.022038090396934606) | > avg_loss_kl: 2.1160730257840217 (-0.3884659014957057) | > avg_loss_feat: 3.2194530997477786 (+0.09978319893420595) | > avg_loss_mel: 17.20012173182527 (-0.9946342119028948) | > avg_loss_duration: 1.5898707097684834 (-0.058667736993708974) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.081641774781993 (-1.3640229937056425) | > avg_grad_norm_1: tensor(5623.1436, device='cuda:0') (+tensor(1080.6948, device='cuda:0'))  > EPOCH: 534/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:42:05)   --> TIME: 2025-02-25 13:42:07 -- STEP: 1/72 -- GLOBAL_STEP: 1038450 | > loss_disc: 2.6085362434387207 (2.6085362434387207) | > loss_disc_real_0: 0.150857612490654 (0.150857612490654) | > loss_disc_real_1: 0.2032848447561264 (0.2032848447561264) | > loss_disc_real_2: 0.20767422020435333 (0.20767422020435333) | > loss_disc_real_3: 0.2169860154390335 (0.2169860154390335) | > loss_disc_real_4: 0.2264358252286911 (0.2264358252286911) | > loss_disc_real_5: 0.2506885528564453 (0.2506885528564453) | > loss_0: 2.6085362434387207 (2.6085362434387207) | > grad_norm_0: tensor(51.7084, device='cuda:0') (tensor(51.7084, device='cuda:0')) | > loss_gen: 2.119128704071045 (2.119128704071045) | > loss_kl: 1.9353163242340088 (1.9353163242340088) | > loss_feat: 3.101712226867676 (3.101712226867676) | > loss_mel: 17.505355834960938 (17.505355834960938) | > loss_duration: 1.5803505182266235 (1.5803505182266235) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.241865158081055 (26.241865158081055) | > grad_norm_1: tensor(7842.2134, device='cuda:0') (tensor(7842.2134, device='cuda:0')) | > current_lr_0: 0.0001870850253108598 | > current_lr_1: 0.0001870850253108598 | > step_time: 0.4334 (0.43343162536621094) | > loader_time: 0.0078 (0.007846593856811523)  --> TIME: 2025-02-25 13:42:30 -- STEP: 51/72 -- GLOBAL_STEP: 1038500 | > loss_disc: 2.817199230194092 (2.6709868674184762) | > loss_disc_real_0: 0.10604678094387054 (0.15052511981304953) | > loss_disc_real_1: 0.2138528674840927 (0.2213860522298252) | > loss_disc_real_2: 0.19956892728805542 (0.23287916241907605) | > loss_disc_real_3: 0.2213873416185379 (0.2345589627237881) | > loss_disc_real_4: 0.2304665595293045 (0.24153394383542678) | > loss_disc_real_5: 0.23497915267944336 (0.24597742656866708) | > loss_0: 2.817199230194092 (2.6709868674184762) | > grad_norm_0: tensor(108.6045, device='cuda:0') (tensor(91.0449, device='cuda:0')) | > loss_gen: 1.6982977390289307 (1.9796103239059448) | > loss_kl: 1.9215373992919922 (2.057314573549757) | > loss_feat: 2.8202333450317383 (3.3126399937797997) | > loss_mel: 14.542412757873535 (17.01426515392229) | > loss_duration: 1.5865733623504639 (1.5651132803337247) | > amp_scaler: 16.0 (16.0) | > loss_1: 22.569053649902344 (25.92894348443723) | > grad_norm_1: tensor(4754.7983, device='cuda:0') (tensor(6403.8062, device='cuda:0')) | > current_lr_0: 0.0001870850253108598 | > current_lr_1: 0.0001870850253108598 | > step_time: 0.4409 (0.4430118682337742) | > loader_time: 0.009 (0.00768887295442469) --> EVAL PERFORMANCE | > avg_loader_time: 0.008694272645762271 (+1.1168735128058022e-05) | > avg_step_time: 0.4542277228664344 (-0.0029761321108106586) | > avg_loss_disc: 2.6703821168818944 (-0.012943395426574966) | > avg_loss_disc_real_0: 0.15211009905791611 (-0.00044142717206982485) | > avg_loss_disc_real_1: 0.2249484662438782 (-0.0005975229219651601) | > avg_loss_disc_real_2: 0.23180658065936935 (+0.0015730119087326577) | > avg_loss_disc_real_3: 0.23473830605056925 (-0.0006167042843052972) | > avg_loss_disc_real_4: 0.24186098806455103 (-0.0026699212235463954) | > avg_loss_disc_real_5: 0.24555277656501448 (-0.002738268442556896) | > avg_loss_0: 2.6703821168818944 (-0.012943395426574966) | > avg_grad_norm_0: tensor(89.2870, device='cuda:0') (+tensor(2.4600, device='cuda:0')) | > avg_loss_gen: 1.9794933006797037 (+0.023369873073739056) | > avg_loss_kl: 2.0872294902801523 (-0.028843535503869422) | > avg_loss_feat: 3.2514472142071797 (+0.031994114459401146) | > avg_loss_mel: 17.002374192358744 (-0.19774753946652623) | > avg_loss_duration: 1.5810887360237014 (-0.008781973744782068) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.901633114881918 (-0.18000865990007497) | > avg_grad_norm_1: tensor(6218.2827, device='cuda:0') (+tensor(595.1392, device='cuda:0'))  > EPOCH: 535/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:42:42)   --> TIME: 2025-02-25 13:42:57 -- STEP: 29/72 -- GLOBAL_STEP: 1038550 | > loss_disc: 2.800908088684082 (2.707465755528417) | > loss_disc_real_0: 0.20532390475273132 (0.16651051337349004) | > loss_disc_real_1: 0.1996045857667923 (0.22222217609142436) | > loss_disc_real_2: 0.28436601161956787 (0.23973629649343162) | > loss_disc_real_3: 0.21587227284908295 (0.23489766439487195) | > loss_disc_real_4: 0.23268438875675201 (0.2425737422088097) | > loss_disc_real_5: 0.23595701158046722 (0.24501077368341642) | > loss_0: 2.800908088684082 (2.707465755528417) | > grad_norm_0: tensor(114.2894, device='cuda:0') (tensor(92.7515, device='cuda:0')) | > loss_gen: 1.914875864982605 (1.9882544196885206) | > loss_kl: 2.56738018989563 (2.1027946431061317) | > loss_feat: 3.058734893798828 (3.3033062014086494) | > loss_mel: 19.24718475341797 (17.468095779418945) | > loss_duration: 1.573671579360962 (1.549009372448099) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.361845016479492 (26.411460218758418) | > grad_norm_1: tensor(4744.4844, device='cuda:0') (tensor(5692.3647, device='cuda:0')) | > current_lr_0: 0.00018706163968269592 | > current_lr_1: 0.00018706163968269592 | > step_time: 0.4462 (0.44010434479549015) | > loader_time: 0.0075 (0.007038371316317855) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087266807824793 (+3.2408136717029454e-05) | > avg_step_time: 0.45923103413111727 (+0.0050033112646828415) | > avg_loss_disc: 2.687422493813743 (+0.017040376931848566) | > avg_loss_disc_real_0: 0.16316802925626997 (+0.011057930198353855) | > avg_loss_disc_real_1: 0.22494784060498357 (-6.256388946357383e-07) | > avg_loss_disc_real_2: 0.2357177690301143 (+0.003911188370744945) | > avg_loss_disc_real_3: 0.2332650838603436 (-0.001473222190225637) | > avg_loss_disc_real_4: 0.23939153048354136 (-0.0024694575810096653) | > avg_loss_disc_real_5: 0.24365196219632324 (-0.0019008143686912449) | > avg_loss_0: 2.687422493813743 (+0.017040376931848566) | > avg_grad_norm_0: tensor(76.4466, device='cuda:0') (tensor(-12.8404, device='cuda:0')) | > avg_loss_gen: 2.022518487043783 (+0.04302518636407915) | > avg_loss_kl: 2.462946299096229 (+0.37571680881607694) | > avg_loss_feat: 3.325777896693055 (+0.07433068248587515) | > avg_loss_mel: 18.91244727121272 (+1.9100730788539764) | > avg_loss_duration: 1.6416676581745415 (+0.06057892215084015) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.365357493011043 (+2.4637243781291254) | > avg_grad_norm_1: tensor(5888.1274, device='cuda:0') (tensor(-330.1553, device='cuda:0'))  > EPOCH: 536/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:43:19)   --> TIME: 2025-02-25 13:43:24 -- STEP: 7/72 -- GLOBAL_STEP: 1038600 | > loss_disc: 2.635265350341797 (2.637405123029436) | > loss_disc_real_0: 0.15325620770454407 (0.15144156238862447) | > loss_disc_real_1: 0.21788352727890015 (0.2116252098764692) | > loss_disc_real_2: 0.15167176723480225 (0.24146456377846853) | > loss_disc_real_3: 0.2503926753997803 (0.2225577618394579) | > loss_disc_real_4: 0.23627160489559174 (0.2461244080747877) | > loss_disc_real_5: 0.2497148960828781 (0.24620949583394186) | > loss_0: 2.635265350341797 (2.637405123029436) | > grad_norm_0: tensor(22.1426, device='cuda:0') (tensor(101.1585, device='cuda:0')) | > loss_gen: 1.9585919380187988 (1.955432449068342) | > loss_kl: 2.33923602104187 (2.178887265069144) | > loss_feat: 3.5274195671081543 (3.3716284206935336) | > loss_mel: 17.59830093383789 (18.119578497750418) | > loss_duration: 1.6044923067092896 (1.6178333418709892) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.028039932250977 (27.243359429495676) | > grad_norm_1: tensor(8951.3125, device='cuda:0') (tensor(6701.8247, device='cuda:0')) | > current_lr_0: 0.00018703825697773558 | > current_lr_1: 0.00018703825697773558 | > step_time: 0.4367 (0.4289149216243199) | > loader_time: 0.0076 (0.006186246871948242)  --> TIME: 2025-02-25 13:43:47 -- STEP: 57/72 -- GLOBAL_STEP: 1038650 | > loss_disc: 2.505115509033203 (2.717200852276986) | > loss_disc_real_0: 0.2151922732591629 (0.1616155526747829) | > loss_disc_real_1: 0.21516476571559906 (0.2323092610428208) | > loss_disc_real_2: 0.2037351131439209 (0.22984607726858372) | > loss_disc_real_3: 0.2148871123790741 (0.24167488332380327) | > loss_disc_real_4: 0.22312870621681213 (0.24919308225313816) | > loss_disc_real_5: 0.2175232321023941 (0.24952339080342079) | > loss_0: 2.505115509033203 (2.717200852276986) | > grad_norm_0: tensor(68.6888, device='cuda:0') (tensor(74.3626, device='cuda:0')) | > loss_gen: 2.284086227416992 (1.987733339008532) | > loss_kl: 2.157330274581909 (2.15186147522508) | > loss_feat: 3.543038845062256 (3.1830811207754564) | > loss_mel: 17.089086532592773 (17.49543400814659) | > loss_duration: 1.63035249710083 (1.6023721297581992) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.703893661499023 (26.42048210010194) | > grad_norm_1: tensor(8555.2139, device='cuda:0') (tensor(5837.7192, device='cuda:0')) | > current_lr_0: 0.00018703825697773558 | > current_lr_1: 0.00018703825697773558 | > step_time: 0.4534 (0.4480779254645632) | > loader_time: 0.0096 (0.008029862454063013) --> EVAL PERFORMANCE | > avg_loader_time: 0.008783646032843791 (+5.696525036449039e-05) | > avg_step_time: 0.4597997934045926 (+0.0005687592734753277) | > avg_loss_disc: 2.7132651503656953 (+0.02584265655195228) | > avg_loss_disc_real_0: 0.16660191627665305 (+0.0034338870203830807) | > avg_loss_disc_real_1: 0.22945820435251987 (+0.0045103637475363) | > avg_loss_disc_real_2: 0.2296967323816998 (-0.006021036648414496) | > avg_loss_disc_real_3: 0.24056006086544252 (+0.007294977005098902) | > avg_loss_disc_real_4: 0.2484607339744836 (+0.00906920349094223) | > avg_loss_disc_real_5: 0.24954630875251674 (+0.005894346556193503) | > avg_loss_0: 2.7132651503656953 (+0.02584265655195228) | > avg_grad_norm_0: tensor(74.7559, device='cuda:0') (tensor(-1.6907, device='cuda:0')) | > avg_loss_gen: 1.9863407998017861 (-0.03617768724199677) | > avg_loss_kl: 2.154915297535103 (-0.30803100156112606) | > avg_loss_feat: 3.1702839690194997 (-0.1554939276735552) | > avg_loss_mel: 17.493824085719147 (-1.4186231854935727) | > avg_loss_duration: 1.6092472731227607 (-0.0324203850517808) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.4146115477656 (-1.950745945245444) | > avg_grad_norm_1: tensor(5762.5820, device='cuda:0') (tensor(-125.5454, device='cuda:0'))  > EPOCH: 537/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:43:56)   --> TIME: 2025-02-25 13:44:14 -- STEP: 35/72 -- GLOBAL_STEP: 1038700 | > loss_disc: 2.8400309085845947 (2.7033889702388216) | > loss_disc_real_0: 0.43031615018844604 (0.17485495401280268) | > loss_disc_real_1: 0.2642190456390381 (0.22664095546518054) | > loss_disc_real_2: 0.25723305344581604 (0.22354991010257175) | > loss_disc_real_3: 0.28510376811027527 (0.23391494495528084) | > loss_disc_real_4: 0.2814067006111145 (0.2392479406935828) | > loss_disc_real_5: 0.2504206597805023 (0.24905543625354767) | > loss_0: 2.8400309085845947 (2.7033889702388216) | > grad_norm_0: tensor(47.0876, device='cuda:0') (tensor(65.6690, device='cuda:0')) | > loss_gen: 2.9278969764709473 (2.0010712862014772) | > loss_kl: 2.366537094116211 (2.191053761754717) | > loss_feat: 3.79939603805542 (3.3156550679888044) | > loss_mel: 18.16242027282715 (17.87279292515346) | > loss_duration: 1.6037945747375488 (1.5671291283198765) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.860044479370117 (26.947702189854212) | > grad_norm_1: tensor(7657.2344, device='cuda:0') (tensor(4593.0664, device='cuda:0')) | > current_lr_0: 0.00018701487719561334 | > current_lr_1: 0.00018701487719561334 | > step_time: 0.4511 (0.43863331249782017) | > loader_time: 0.0083 (0.007378278459821429) --> EVAL PERFORMANCE | > avg_loader_time: 0.008744572249936387 (-3.907378290740442e-05) | > avg_step_time: 0.4556076560221927 (-0.004192137382399874) | > avg_loss_disc: 2.762081223474422 (+0.0488160731087266) | > avg_loss_disc_real_0: 0.1785276589364233 (+0.011925742659770255) | > avg_loss_disc_real_1: 0.23547465679511217 (+0.006016452442592302) | > avg_loss_disc_real_2: 0.2349692167530597 (+0.005272484371359909) | > avg_loss_disc_real_3: 0.23757098111468303 (-0.002989079750759488) | > avg_loss_disc_real_4: 0.24449884135958175 (-0.003961892614901846) | > avg_loss_disc_real_5: 0.24850866009651776 (-0.0010376486559989795) | > avg_loss_0: 2.762081223474422 (+0.0488160731087266) | > avg_grad_norm_0: tensor(62.9126, device='cuda:0') (tensor(-11.8434, device='cuda:0')) | > avg_loss_gen: 1.9691454373614885 (-0.017195362440297624) | > avg_loss_kl: 2.4519129017709007 (+0.2969976042357976) | > avg_loss_feat: 3.2215529932102687 (+0.05126902419076895) | > avg_loss_mel: 19.63483627749161 (+2.141012191772461) | > avg_loss_duration: 1.6102556812931115 (+0.0010084081703507497) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.887703317991445 (+2.473091770225846) | > avg_grad_norm_1: tensor(4048.5427, device='cuda:0') (tensor(-1714.0393, device='cuda:0'))  > EPOCH: 538/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:44:33)   --> TIME: 2025-02-25 13:44:41 -- STEP: 13/72 -- GLOBAL_STEP: 1038750 | > loss_disc: 3.001558542251587 (2.776131795002864) | > loss_disc_real_0: 0.08787715435028076 (0.09413397355148426) | > loss_disc_real_1: 0.2431849092245102 (0.24824335139531356) | > loss_disc_real_2: 0.15199387073516846 (0.2626963108778) | > loss_disc_real_3: 0.2620151937007904 (0.2762197038302055) | > loss_disc_real_4: 0.28580009937286377 (0.268438200537975) | > loss_disc_real_5: 0.2904888093471527 (0.24287289381027222) | > loss_0: 3.001558542251587 (2.776131795002864) | > grad_norm_0: tensor(64.8651, device='cuda:0') (tensor(53.3325, device='cuda:0')) | > loss_gen: 1.640167474746704 (2.1591350115262546) | > loss_kl: 2.293625831604004 (2.4241570876194882) | > loss_feat: 2.8140177726745605 (3.559282266176664) | > loss_mel: 18.74544334411621 (19.015707162710335) | > loss_duration: 1.4945704936981201 (1.5661905270356398) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.987823486328125 (28.724471899179314) | > grad_norm_1: tensor(3346.3877, device='cuda:0') (tensor(4714.0615, device='cuda:0')) | > current_lr_0: 0.0001869915003359639 | > current_lr_1: 0.0001869915003359639 | > step_time: 0.4563 (0.4350640590374286) | > loader_time: 0.0058 (0.006758506481464093)  --> TIME: 2025-02-25 13:45:04 -- STEP: 63/72 -- GLOBAL_STEP: 1038800 | > loss_disc: 2.691707134246826 (2.717202099542769) | > loss_disc_real_0: 0.13414108753204346 (0.13790878697875003) | > loss_disc_real_1: 0.22552794218063354 (0.22807215083213078) | > loss_disc_real_2: 0.2277679741382599 (0.22981267954621995) | > loss_disc_real_3: 0.2224680632352829 (0.24476313614656056) | > loss_disc_real_4: 0.2651729881763458 (0.24967774464970544) | > loss_disc_real_5: 0.28135183453559875 (0.24628133527816287) | > loss_0: 2.691707134246826 (2.717202099542769) | > grad_norm_0: tensor(20.5390, device='cuda:0') (tensor(65.0561, device='cuda:0')) | > loss_gen: 2.14792799949646 (1.971532963571094) | > loss_kl: 3.292893171310425 (2.539138406042069) | > loss_feat: 3.2164838314056396 (3.262094463620867) | > loss_mel: 22.809532165527344 (19.017398092481827) | > loss_duration: 1.7194093465805054 (1.620685164890592) | > amp_scaler: 16.0 (16.0) | > loss_1: 33.18624496459961 (28.410849041408962) | > grad_norm_1: tensor(4377.5889, device='cuda:0') (tensor(4148.1675, device='cuda:0')) | > current_lr_0: 0.0001869915003359639 | > current_lr_1: 0.0001869915003359639 | > step_time: 0.5018 (0.448672680627732) | > loader_time: 0.0113 (0.008323839732578822) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872311121980909 (-2.1461030127296887e-05) | > avg_step_time: 0.4586067669828173 (+0.002999110960624607) | > avg_loss_disc: 2.7112815413676516 (-0.050799682106770216) | > avg_loss_disc_real_0: 0.14674015949204777 (-0.03178749944437553) | > avg_loss_disc_real_1: 0.226903648023874 (-0.008571008771238176) | > avg_loss_disc_real_2: 0.23108222283108135 (-0.00388699392197836) | > avg_loss_disc_real_3: 0.24278209977586504 (+0.005211118661182013) | > avg_loss_disc_real_4: 0.24792228702088476 (+0.0034234456613030095) | > avg_loss_disc_real_5: 0.2457341516101864 (-0.002774508486331345) | > avg_loss_0: 2.7112815413676516 (-0.050799682106770216) | > avg_grad_norm_0: tensor(65.8435, device='cuda:0') (+tensor(2.9310, device='cuda:0')) | > avg_loss_gen: 1.9814023669336882 (+0.012256929572199704) | > avg_loss_kl: 2.6047826568845296 (+0.15286975511362888) | > avg_loss_feat: 3.256742037517924 (+0.03518904430765524) | > avg_loss_mel: 19.30920360457729 (-0.32563267291431686) | > avg_loss_duration: 1.6339706132109737 (+0.023714931917862225) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.786101220359264 (-0.1016020976321812) | > avg_grad_norm_1: tensor(4156.7847, device='cuda:0') (+tensor(108.2419, device='cuda:0'))  > EPOCH: 539/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:45:11)   --> TIME: 2025-02-25 13:45:32 -- STEP: 41/72 -- GLOBAL_STEP: 1038850 | > loss_disc: 2.5828568935394287 (2.7099363687561784) | > loss_disc_real_0: 0.1813824474811554 (0.1833636732544841) | > loss_disc_real_1: 0.19783450663089752 (0.22268482570241138) | > loss_disc_real_2: 0.295415997505188 (0.2354440565516309) | > loss_disc_real_3: 0.22997041046619415 (0.23169722353539815) | > loss_disc_real_4: 0.22210857272148132 (0.24070395747335946) | > loss_disc_real_5: 0.21646259725093842 (0.23967322888897685) | > loss_0: 2.5828568935394287 (2.7099363687561784) | > grad_norm_0: tensor(38.0776, device='cuda:0') (tensor(57.1087, device='cuda:0')) | > loss_gen: 2.032092809677124 (1.9879569164136568) | > loss_kl: 2.186357021331787 (2.4405338706039803) | > loss_feat: 3.418239116668701 (3.3500441225563606) | > loss_mel: 18.251405715942383 (19.580915730173995) | > loss_duration: 1.5947716236114502 (1.6103520858578566) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.482868194580078 (28.96980271688322) | > grad_norm_1: tensor(5801.2041, device='cuda:0') (tensor(3774.0249, device='cuda:0')) | > current_lr_0: 0.0001869681263984219 | > current_lr_1: 0.0001869681263984219 | > step_time: 0.4448 (0.4410149237004722) | > loader_time: 0.009 (0.006784125072200124) --> EVAL PERFORMANCE | > avg_loader_time: 0.008283047608926262 (-0.0004400636108828279) | > avg_step_time: 0.45680221705369545 (-0.0018045499291218792) | > avg_loss_disc: 2.699303543063957 (-0.011977998303694726) | > avg_loss_disc_real_0: 0.1751005216698412 (+0.02836036217779342) | > avg_loss_disc_real_1: 0.2233803626936926 (-0.003523285330181397) | > avg_loss_disc_real_2: 0.2345913864777122 (+0.003509163646630842) | > avg_loss_disc_real_3: 0.23458918950087587 (-0.008192910274989168) | > avg_loss_disc_real_4: 0.24439894922182592 (-0.0035233377990588344) | > avg_loss_disc_real_5: 0.2453908773375229 (-0.0003432742726635074) | > avg_loss_0: 2.699303543063957 (-0.011977998303694726) | > avg_grad_norm_0: tensor(64.6551, device='cuda:0') (tensor(-1.1884, device='cuda:0')) | > avg_loss_gen: 1.9830511120003718 (+0.0016487450666835546) | > avg_loss_kl: 2.3325383327376676 (-0.27224432414686195) | > avg_loss_feat: 3.3022650194839693 (+0.04552298196604543) | > avg_loss_mel: 18.940365777888765 (-0.3688378266885266) | > avg_loss_duration: 1.6204702585515842 (-0.01350035465938948) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.178690440218215 (-0.6074107801410484) | > avg_grad_norm_1: tensor(4110.6963, device='cuda:0') (tensor(-46.0884, device='cuda:0'))  > EPOCH: 540/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:45:49)   --> TIME: 2025-02-25 13:45:59 -- STEP: 19/72 -- GLOBAL_STEP: 1038900 | > loss_disc: 2.6824142932891846 (2.6449014011182284) | > loss_disc_real_0: 0.10708943009376526 (0.14013226722416125) | > loss_disc_real_1: 0.2283622771501541 (0.2356231385155728) | > loss_disc_real_2: 0.20975853502750397 (0.22913382241600438) | > loss_disc_real_3: 0.23004761338233948 (0.23459719356737638) | > loss_disc_real_4: 0.2448652982711792 (0.24340247951055827) | > loss_disc_real_5: 0.26098909974098206 (0.25161985817708465) | > loss_0: 2.6824142932891846 (2.6449014011182284) | > grad_norm_0: tensor(102.8913, device='cuda:0') (tensor(74.6494, device='cuda:0')) | > loss_gen: 1.7209380865097046 (1.976458700079667) | > loss_kl: 2.175649404525757 (2.0817172025379382) | > loss_feat: 3.1369476318359375 (3.394989565799111) | > loss_mel: 19.02105712890625 (18.174245432803506) | > loss_duration: 1.5596345663070679 (1.5649478435516357) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.614227294921875 (27.192358619288395) | > grad_norm_1: tensor(3244.7861, device='cuda:0') (tensor(4584.4751, device='cuda:0')) | > current_lr_0: 0.0001869447553826221 | > current_lr_1: 0.0001869447553826221 | > step_time: 0.4319 (0.4363338068911904) | > loader_time: 0.0061 (0.00699397137290553)  --> TIME: 2025-02-25 13:46:23 -- STEP: 69/72 -- GLOBAL_STEP: 1038950 | > loss_disc: 2.7193541526794434 (2.682921692944955) | > loss_disc_real_0: 0.0858665183186531 (0.1438835318321767) | > loss_disc_real_1: 0.2074144035577774 (0.23106348212214484) | > loss_disc_real_2: 0.2783185541629791 (0.23327895437461743) | > loss_disc_real_3: 0.27299976348876953 (0.23637083539928216) | > loss_disc_real_4: 0.25361368060112 (0.24403857774492624) | > loss_disc_real_5: 0.2480759173631668 (0.24809023090030835) | > loss_0: 2.7193541526794434 (2.682921692944955) | > grad_norm_0: tensor(90.3727, device='cuda:0') (tensor(78.0763, device='cuda:0')) | > loss_gen: 1.9267419576644897 (1.9543183005374412) | > loss_kl: 2.288965940475464 (2.2826607227325435) | > loss_feat: 3.2898638248443604 (3.2842525641123452) | > loss_mel: 18.898529052734375 (18.69817489126454) | > loss_duration: 1.7104978561401367 (1.6380174298217332) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.114601135253906 (27.857423865276836) | > grad_norm_1: tensor(5109.3677, device='cuda:0') (tensor(4624.5732, device='cuda:0')) | > current_lr_0: 0.0001869447553826221 | > current_lr_1: 0.0001869447553826221 | > step_time: 0.5409 (0.4563340035037718) | > loader_time: 0.0125 (0.008803568024566211) --> EVAL PERFORMANCE | > avg_loader_time: 0.008856199157070109 (+0.0005731515481438472) | > avg_step_time: 0.45799802054821603 (+0.0011958034945205864) | > avg_loss_disc: 2.6860008004685523 (-0.013302742595404649) | > avg_loss_disc_real_0: 0.14260169926663518 (-0.03249882240320601) | > avg_loss_disc_real_1: 0.22975021558748165 (+0.006369852893789052) | > avg_loss_disc_real_2: 0.2335282759347432 (-0.0010631105429689813) | > avg_loss_disc_real_3: 0.2365426861904037 (+0.0019534966895278227) | > avg_loss_disc_real_4: 0.24377466507360968 (-0.0006242841482162476) | > avg_loss_disc_real_5: 0.24744171428848322 (+0.002050836950960311) | > avg_loss_0: 2.6860008004685523 (-0.013302742595404649) | > avg_grad_norm_0: tensor(79.6377, device='cuda:0') (+tensor(14.9826, device='cuda:0')) | > avg_loss_gen: 1.9492088589869754 (-0.033842253013396384) | > avg_loss_kl: 2.2816943282812407 (-0.050844004456426894) | > avg_loss_feat: 3.2850258484692643 (-0.017239171014705068) | > avg_loss_mel: 18.681517627877245 (-0.25884815001152006) | > avg_loss_duration: 1.6371457157000688 (+0.016675457148484618) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.83459233566069 (-0.3440981045575242) | > avg_grad_norm_1: tensor(4652.5718, device='cuda:0') (+tensor(541.8755, device='cuda:0'))  > EPOCH: 541/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:46:26)   --> TIME: 2025-02-25 13:46:49 -- STEP: 47/72 -- GLOBAL_STEP: 1039000 | > loss_disc: 2.6047427654266357 (2.6647868917343462) | > loss_disc_real_0: 0.1620146632194519 (0.14690463546108695) | > loss_disc_real_1: 0.18609967827796936 (0.21988228977994717) | > loss_disc_real_2: 0.2366315871477127 (0.2320435440286677) | > loss_disc_real_3: 0.2764335572719574 (0.23216690123081207) | > loss_disc_real_4: 0.23831747472286224 (0.2392406993089838) | > loss_disc_real_5: 0.24072368443012238 (0.24402841862211835) | > loss_0: 2.6047427654266357 (2.6647868917343462) | > grad_norm_0: tensor(13.0811, device='cuda:0') (tensor(85.8165, device='cuda:0')) | > loss_gen: 1.9072285890579224 (1.9525469947368541) | > loss_kl: 2.173943281173706 (2.0293229691525725) | > loss_feat: 3.466111898422241 (3.3836989402770996) | > loss_mel: 17.66901206970215 (17.590103555232922) | > loss_duration: 1.6299290657043457 (1.5743933789273525) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.846223831176758 (26.530066023481652) | > grad_norm_1: tensor(6622.3940, device='cuda:0') (tensor(5137.3042, device='cuda:0')) | > current_lr_0: 0.00018692138728819925 | > current_lr_1: 0.00018692138728819925 | > step_time: 0.4496 (0.44325283740429167) | > loader_time: 0.0089 (0.007733025449387571) --> EVAL PERFORMANCE | > avg_loader_time: 0.00877674532608247 (-7.945383098763824e-05) | > avg_step_time: 0.45684561930911644 (-0.0011524012390995986) | > avg_loss_disc: 2.693294377394125 (+0.007293576925572864) | > avg_loss_disc_real_0: 0.14930251051842333 (+0.006700811251788147) | > avg_loss_disc_real_1: 0.2317240704113329 (+0.0019738548238512343) | > avg_loss_disc_real_2: 0.23628694784473364 (+0.0027586719099904344) | > avg_loss_disc_real_3: 0.23558601401221585 (-0.0009566721781878451) | > avg_loss_disc_real_4: 0.2415518366115194 (-0.002222828462090265) | > avg_loss_disc_real_5: 0.24668672387029084 (-0.0007549904181923806) | > avg_loss_0: 2.693294377394125 (+0.007293576925572864) | > avg_grad_norm_0: tensor(89.0221, device='cuda:0') (+tensor(9.3845, device='cuda:0')) | > avg_loss_gen: 1.959266464475175 (+0.010057605488199561) | > avg_loss_kl: 2.05509982982152 (-0.22659449845972057) | > avg_loss_feat: 3.2832503990388253 (-0.0017754494304389468) | > avg_loss_mel: 17.513076325537455 (-1.1684413023397902) | > avg_loss_duration: 1.5916171040333493 (-0.04552861166671951) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.402310169918437 (-1.4322821657422544) | > avg_grad_norm_1: tensor(5158.8496, device='cuda:0') (+tensor(506.2778, device='cuda:0'))  > EPOCH: 542/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:47:03)   --> TIME: 2025-02-25 13:47:16 -- STEP: 25/72 -- GLOBAL_STEP: 1039050 | > loss_disc: 2.9381752014160156 (2.723803129196167) | > loss_disc_real_0: 0.3524079918861389 (0.20421266853809356) | > loss_disc_real_1: 0.22507019340991974 (0.2184281998872757) | > loss_disc_real_2: 0.20783913135528564 (0.22896980822086335) | > loss_disc_real_3: 0.25952088832855225 (0.23511236250400544) | > loss_disc_real_4: 0.24462175369262695 (0.24233827471733094) | > loss_disc_real_5: 0.2613368630409241 (0.24617773413658142) | > loss_0: 2.9381752014160156 (2.723803129196167) | > grad_norm_0: tensor(101.6936, device='cuda:0') (tensor(94.3659, device='cuda:0')) | > loss_gen: 1.8299022912979126 (1.939199652671814) | > loss_kl: 2.036802291870117 (2.1603954648971557) | > loss_feat: 3.1227636337280273 (3.1799717426300047) | > loss_mel: 19.395118713378906 (17.751714630126948) | > loss_duration: 1.5544822216033936 (1.557835669517517) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.939067840576172 (26.589116668701173) | > grad_norm_1: tensor(1466.0188, device='cuda:0') (tensor(4458.0654, device='cuda:0')) | > current_lr_0: 0.00018689802211478823 | > current_lr_1: 0.00018689802211478823 | > step_time: 0.4296 (0.4391485691070557) | > loader_time: 0.0078 (0.007211570739746094) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738017418015174 (-3.8727908067297256e-05) | > avg_step_time: 0.4591597167538925 (+0.0023140974447760643) | > avg_loss_disc: 2.69600949488895 (+0.0027151174948247814) | > avg_loss_disc_real_0: 0.1749437465004518 (+0.02564123598202847) | > avg_loss_disc_real_1: 0.2256804771826301 (-0.006043593228702793) | > avg_loss_disc_real_2: 0.2263644627282317 (-0.009922485116501928) | > avg_loss_disc_real_3: 0.23592693096315356 (+0.0003409169509377141) | > avg_loss_disc_real_4: 0.24569543986253337 (+0.004143603251013955) | > avg_loss_disc_real_5: 0.24987040523072354 (+0.0031836813604327052) | > avg_loss_0: 2.69600949488895 (+0.0027151174948247814) | > avg_grad_norm_0: tensor(75.7560, device='cuda:0') (tensor(-13.2661, device='cuda:0')) | > avg_loss_gen: 2.040487034220091 (+0.08122056974491598) | > avg_loss_kl: 2.4479329938619907 (+0.39283316404047053) | > avg_loss_feat: 3.293398343341451 (+0.010147944302625778) | > avg_loss_mel: 19.28096252763775 (+1.7678862021002963) | > avg_loss_duration: 1.624937974231344 (+0.03332087019799457) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.687718646627076 (+2.285408476708639) | > avg_grad_norm_1: tensor(4277.6084, device='cuda:0') (tensor(-881.2412, device='cuda:0'))  > EPOCH: 543/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:47:41)   --> TIME: 2025-02-25 13:47:44 -- STEP: 3/72 -- GLOBAL_STEP: 1039100 | > loss_disc: 2.7410781383514404 (2.732766071955363) | > loss_disc_real_0: 0.12545031309127808 (0.11193055162827174) | > loss_disc_real_1: 0.21577559411525726 (0.20512616634368896) | > loss_disc_real_2: 0.2199033796787262 (0.2403807838757833) | > loss_disc_real_3: 0.2094961553812027 (0.21621842682361603) | > loss_disc_real_4: 0.258029043674469 (0.2492532730102539) | > loss_disc_real_5: 0.25246965885162354 (0.23792597154776254) | > loss_0: 2.7410781383514404 (2.732766071955363) | > grad_norm_0: tensor(156.4256, device='cuda:0') (tensor(103.4348, device='cuda:0')) | > loss_gen: 1.7283366918563843 (1.9305835167566936) | > loss_kl: 2.185455083847046 (2.138745148976644) | > loss_feat: 3.0998034477233887 (3.630667765935262) | > loss_mel: 19.180530548095703 (19.682133356730144) | > loss_duration: 1.6129324436187744 (1.6043029228846233) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.807058334350586 (28.986432393391926) | > grad_norm_1: tensor(5060.4219, device='cuda:0') (tensor(6941.8438, device='cuda:0')) | > current_lr_0: 0.00018687465986202387 | > current_lr_1: 0.00018687465986202387 | > step_time: 0.4318 (0.43897517522176105) | > loader_time: 0.0057 (0.00667572021484375)  --> TIME: 2025-02-25 13:48:07 -- STEP: 53/72 -- GLOBAL_STEP: 1039150 | > loss_disc: 2.5635905265808105 (2.731063865265756) | > loss_disc_real_0: 0.14269806444644928 (0.17148757107415288) | > loss_disc_real_1: 0.24146229028701782 (0.22388338902086583) | > loss_disc_real_2: 0.25656718015670776 (0.25421652422761015) | > loss_disc_real_3: 0.23250119388103485 (0.234357532184079) | > loss_disc_real_4: 0.23990264534950256 (0.24400415932232478) | > loss_disc_real_5: 0.2486720234155655 (0.2439719849037674) | > loss_0: 2.5635905265808105 (2.731063865265756) | > grad_norm_0: tensor(27.1358, device='cuda:0') (tensor(67.7386, device='cuda:0')) | > loss_gen: 2.199354648590088 (2.001692479511477) | > loss_kl: 2.0646378993988037 (2.230645535127171) | > loss_feat: 3.583195924758911 (3.202568418574783) | > loss_mel: 18.500646591186523 (18.765162701876662) | > loss_duration: 1.6432554721832275 (1.6010218656288002) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.9910888671875 (27.801090780294167) | > grad_norm_1: tensor(4625.8545, device='cuda:0') (tensor(4017.5623, device='cuda:0')) | > current_lr_0: 0.00018687465986202387 | > current_lr_1: 0.00018687465986202387 | > step_time: 0.4496 (0.4462133488565121) | > loader_time: 0.0093 (0.007899410319778153) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666125821395657 (-7.189159661951695e-05) | > avg_step_time: 0.45805330007848605 (-0.001106416675406452) | > avg_loss_disc: 2.7411456813274975 (+0.045136186438547554) | > avg_loss_disc_real_0: 0.17011330213765025 (-0.004830444362801556) | > avg_loss_disc_real_1: 0.23409287075341587 (+0.00841239357078577) | > avg_loss_disc_real_2: 0.24941888577501534 (+0.023054423046783623) | > avg_loss_disc_real_3: 0.23605265227002156 (+0.00012572130686799854) | > avg_loss_disc_real_4: 0.24660344862602127 (+0.0009080087634878997) | > avg_loss_disc_real_5: 0.2475997356461807 (-0.0022706695845428337) | > avg_loss_0: 2.7411456813274975 (+0.045136186438547554) | > avg_grad_norm_0: tensor(64.7897, device='cuda:0') (tensor(-10.9663, device='cuda:0')) | > avg_loss_gen: 1.9929757386865745 (-0.04751129553351641) | > avg_loss_kl: 2.2292571806571853 (-0.21867581320480545) | > avg_loss_feat: 3.1622502065040696 (-0.13114813683738147) | > avg_loss_mel: 18.820430970527763 (-0.46053155710998794) | > avg_loss_duration: 1.6121402894946892 (-0.012797684736654658) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.81705423811792 (-0.8706644085091568) | > avg_grad_norm_1: tensor(4151.1577, device='cuda:0') (tensor(-126.4507, device='cuda:0'))  > EPOCH: 544/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:48:18)   --> TIME: 2025-02-25 13:48:34 -- STEP: 31/72 -- GLOBAL_STEP: 1039200 | > loss_disc: 2.574880361557007 (2.68517441134299) | > loss_disc_real_0: 0.1234566941857338 (0.1502924982097841) | > loss_disc_real_1: 0.24432168900966644 (0.22332418685959232) | > loss_disc_real_2: 0.24165378510951996 (0.22601567881722603) | > loss_disc_real_3: 0.24770759046077728 (0.2377501779025601) | > loss_disc_real_4: 0.2579847276210785 (0.23539948607644728) | > loss_disc_real_5: 0.2771089971065521 (0.2400246868210454) | > loss_0: 2.574880361557007 (2.68517441134299) | > grad_norm_0: tensor(52.8485, device='cuda:0') (tensor(80.4109, device='cuda:0')) | > loss_gen: 1.8829693794250488 (1.8782576668647029) | > loss_kl: 3.214097023010254 (2.155343040343253) | > loss_feat: 3.3277835845947266 (3.0835879156666417) | > loss_mel: 20.8780460357666 (18.014333294283958) | > loss_duration: 1.6453791856765747 (1.5611487165574105) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.948274612426758 (26.692670452979304) | > grad_norm_1: tensor(7659.2383, device='cuda:0') (tensor(4608.4097, device='cuda:0')) | > current_lr_0: 0.00018685130052954112 | > current_lr_1: 0.00018685130052954112 | > step_time: 0.4517 (0.437620509055353) | > loader_time: 0.0088 (0.007011190537483462) --> EVAL PERFORMANCE | > avg_loader_time: 0.008600339083604414 (-6.5786737791243e-05) | > avg_step_time: 0.45600611391201823 (-0.002047186166467818) | > avg_loss_disc: 2.698097524508624 (-0.04304815681887364) | > avg_loss_disc_real_0: 0.15526810897068238 (-0.014845193166967863) | > avg_loss_disc_real_1: 0.22581209641107372 (-0.008280774342342145) | > avg_loss_disc_real_2: 0.23636672110624715 (-0.013052164668768185) | > avg_loss_disc_real_3: 0.23646050020002982 (+0.0004078479300082538) | > avg_loss_disc_real_4: 0.242203896524201 (-0.0043995521018202755) | > avg_loss_disc_real_5: 0.24087289290528902 (-0.006726842740891692) | > avg_loss_0: 2.698097524508624 (-0.04304815681887364) | > avg_grad_norm_0: tensor(70.6400, device='cuda:0') (+tensor(5.8503, device='cuda:0')) | > avg_loss_gen: 1.9275067322690722 (-0.0654690064175023) | > avg_loss_kl: 2.3114314952366772 (+0.08217431457949198) | > avg_loss_feat: 3.1193467193925883 (-0.04290348711148129) | > avg_loss_mel: 18.348621260951937 (-0.471809709575826) | > avg_loss_duration: 1.6070588038001261 (-0.005081485694563126) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.31396484375 (-0.5030893943679189) | > avg_grad_norm_1: tensor(4523.8003, device='cuda:0') (+tensor(372.6426, device='cuda:0'))  > EPOCH: 545/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:48:55)   --> TIME: 2025-02-25 13:49:01 -- STEP: 9/72 -- GLOBAL_STEP: 1039250 | > loss_disc: 2.629206657409668 (2.574562178717719) | > loss_disc_real_0: 0.06083998456597328 (0.11506636771890852) | > loss_disc_real_1: 0.19666454195976257 (0.2153624693552653) | > loss_disc_real_2: 0.24187861382961273 (0.22890067597230276) | > loss_disc_real_3: 0.20634600520133972 (0.22167650361855826) | > loss_disc_real_4: 0.2280268669128418 (0.24046424859099919) | > loss_disc_real_5: 0.23615288734436035 (0.22704475455813938) | > loss_0: 2.629206657409668 (2.574562178717719) | > grad_norm_0: tensor(71.7305, device='cuda:0') (tensor(56.8625, device='cuda:0')) | > loss_gen: 1.9236526489257812 (2.1357007688946195) | > loss_kl: 2.2123515605926514 (2.222792042626275) | > loss_feat: 3.799006223678589 (3.786220656500922) | > loss_mel: 19.06665802001953 (18.928532706366646) | > loss_duration: 1.588917851448059 (1.5805874930487738) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.590587615966797 (28.653833389282227) | > grad_norm_1: tensor(4082.5110, device='cuda:0') (tensor(5988.4097, device='cuda:0')) | > current_lr_0: 0.00018682794411697493 | > current_lr_1: 0.00018682794411697493 | > step_time: 0.433 (0.436876032087538) | > loader_time: 0.0067 (0.006719748179117839)  --> TIME: 2025-02-25 13:49:24 -- STEP: 59/72 -- GLOBAL_STEP: 1039300 | > loss_disc: 2.5700523853302 (2.6894974789376986) | > loss_disc_real_0: 0.05801549553871155 (0.17924835034093614) | > loss_disc_real_1: 0.19114045798778534 (0.22523726699715954) | > loss_disc_real_2: 0.17684845626354218 (0.24027689230644098) | > loss_disc_real_3: 0.20992910861968994 (0.23839261900570433) | > loss_disc_real_4: 0.2413690984249115 (0.2479789613162057) | > loss_disc_real_5: 0.2576158046722412 (0.249000151531171) | > loss_0: 2.5700523853302 (2.6894974789376986) | > grad_norm_0: tensor(72.0846, device='cuda:0') (tensor(66.8405, device='cuda:0')) | > loss_gen: 1.8620285987854004 (2.0681393469794322) | > loss_kl: 2.4342474937438965 (2.2020162566233488) | > loss_feat: 3.2685751914978027 (3.356306758977599) | > loss_mel: 18.571170806884766 (18.13530844348973) | > loss_duration: 1.6842873096466064 (1.6049970893536585) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.820310592651367 (27.36676798028461) | > grad_norm_1: tensor(4044.7932, device='cuda:0') (tensor(4326.2949, device='cuda:0')) | > current_lr_0: 0.00018682794411697493 | > current_lr_1: 0.00018682794411697493 | > step_time: 0.4788 (0.4490431850239382) | > loader_time: 0.0098 (0.008122367373967576) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722097101345869 (+0.00012175801774145546) | > avg_step_time: 0.4582087691401092 (+0.0022026552280909883) | > avg_loss_disc: 2.6804624409742757 (-0.017635083534348173) | > avg_loss_disc_real_0: 0.1805618740618229 (+0.025293765091140508) | > avg_loss_disc_real_1: 0.22694019385626618 (+0.0011280974451924608) | > avg_loss_disc_real_2: 0.23565840847055677 (-0.0007083126356903779) | > avg_loss_disc_real_3: 0.23586130457025178 (-0.0005991956297780321) | > avg_loss_disc_real_4: 0.2438800444065685 (+0.0016761478823675013) | > avg_loss_disc_real_5: 0.24522894432007428 (+0.004356051414785261) | > avg_loss_0: 2.6804624409742757 (-0.017635083534348173) | > avg_grad_norm_0: tensor(67.2499, device='cuda:0') (tensor(-3.3901, device='cuda:0')) | > avg_loss_gen: 2.071535709878089 (+0.1440289776090169) | > avg_loss_kl: 2.2147045135498047 (-0.09672698168687255) | > avg_loss_feat: 3.3328686566420003 (+0.21352193724941193) | > avg_loss_mel: 18.109131208607856 (-0.23949005234408105) | > avg_loss_duration: 1.6126603828349584 (+0.005601579034832271) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.34090058232697 (+0.02693573857697018) | > avg_grad_norm_1: tensor(4308.5181, device='cuda:0') (tensor(-215.2822, device='cuda:0'))  > EPOCH: 546/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:49:32)   --> TIME: 2025-02-25 13:49:51 -- STEP: 37/72 -- GLOBAL_STEP: 1039350 | > loss_disc: 2.825230598449707 (2.73142705092559) | > loss_disc_real_0: 0.21268397569656372 (0.14511394047656578) | > loss_disc_real_1: 0.2758321166038513 (0.2183693333252056) | > loss_disc_real_2: 0.1772373914718628 (0.24488172780823064) | > loss_disc_real_3: 0.26682811975479126 (0.2473803460597992) | > loss_disc_real_4: 0.24721230566501617 (0.2485244608408696) | > loss_disc_real_5: 0.23574624955654144 (0.25110961053822484) | > loss_0: 2.825230598449707 (2.73142705092559) | > grad_norm_0: tensor(50.7899, device='cuda:0') (tensor(67.1524, device='cuda:0')) | > loss_gen: 1.931204080581665 (1.97628138194213) | > loss_kl: 1.988722324371338 (2.071433354068447) | > loss_feat: 2.7362537384033203 (3.269671208149678) | > loss_mel: 15.973282814025879 (17.721977182336754) | > loss_duration: 1.5990266799926758 (1.5613179625691593) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.22848892211914 (26.600680892531937) | > grad_norm_1: tensor(3056.8706, device='cuda:0') (tensor(4426.4414, device='cuda:0')) | > current_lr_0: 0.0001868045906239603 | > current_lr_1: 0.0001868045906239603 | > step_time: 0.4568 (0.4561382499901024) | > loader_time: 0.0085 (0.007352319923607078) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872886348778093 (+6.7663864350610975e-06) | > avg_step_time: 0.4642610012645453 (+0.006052232124436063) | > avg_loss_disc: 2.7621321006559985 (+0.0816696596817228) | > avg_loss_disc_real_0: 0.1617884375679661 (-0.018773436493856793) | > avg_loss_disc_real_1: 0.22015727993468165 (-0.0067829139215845324) | > avg_loss_disc_real_2: 0.24162489364684467 (+0.0059664851762878945) | > avg_loss_disc_real_3: 0.24194696986339462 (+0.0060856652931428334) | > avg_loss_disc_real_4: 0.2437844838894589 (-9.556051710959967e-05) | > avg_loss_disc_real_5: 0.24647983111126312 (+0.0012508867911888433) | > avg_loss_0: 2.7621321006559985 (+0.0816696596817228) | > avg_grad_norm_0: tensor(67.5400, device='cuda:0') (+tensor(0.2901, device='cuda:0')) | > avg_loss_gen: 1.9163545521212295 (-0.1551811577568596) | > avg_loss_kl: 2.244119719720224 (+0.02941520617041915) | > avg_loss_feat: 3.125482206613245 (-0.20738645002875522) | > avg_loss_mel: 18.128339015262227 (+0.019207806654371495) | > avg_loss_duration: 1.600942507596083 (-0.011717875238875353) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.015237888819737 (-0.3256626935072333) | > avg_grad_norm_1: tensor(3862.8384, device='cuda:0') (tensor(-445.6797, device='cuda:0'))  > EPOCH: 547/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:50:10)   --> TIME: 2025-02-25 13:50:18 -- STEP: 15/72 -- GLOBAL_STEP: 1039400 | > loss_disc: 2.7861504554748535 (2.7785104592641194) | > loss_disc_real_0: 0.18612249195575714 (0.20528806646664938) | > loss_disc_real_1: 0.16329482197761536 (0.2420812060435613) | > loss_disc_real_2: 0.23717431724071503 (0.2652969380219778) | > loss_disc_real_3: 0.2551913261413574 (0.24038884143034617) | > loss_disc_real_4: 0.26364728808403015 (0.2565002053976059) | > loss_disc_real_5: 0.27269047498703003 (0.26611931622028345) | > loss_0: 2.7861504554748535 (2.7785104592641194) | > grad_norm_0: tensor(52.0073, device='cuda:0') (tensor(48.7498, device='cuda:0')) | > loss_gen: 1.7305457592010498 (1.9445534944534302) | > loss_kl: 2.1082327365875244 (2.0546745936075848) | > loss_feat: 2.593071937561035 (2.8498475710550943) | > loss_mel: 18.215486526489258 (17.633734893798824) | > loss_duration: 1.5607373714447021 (1.558502435684204) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.208072662353516 (26.041312789916994) | > grad_norm_1: tensor(506.2882, device='cuda:0') (tensor(2410.6870, device='cuda:0')) | > current_lr_0: 0.0001867812400501323 | > current_lr_1: 0.0001867812400501323 | > step_time: 0.4374 (0.43962726593017576) | > loader_time: 0.0063 (0.006557289759318034)  --> TIME: 2025-02-25 13:50:42 -- STEP: 65/72 -- GLOBAL_STEP: 1039450 | > loss_disc: 2.532043218612671 (2.6948706810290997) | > loss_disc_real_0: 0.1891096830368042 (0.16859720876583686) | > loss_disc_real_1: 0.19626235961914062 (0.23139939033068144) | > loss_disc_real_2: 0.23233015835285187 (0.2363452517069303) | > loss_disc_real_3: 0.21877162158489227 (0.23094532100053933) | > loss_disc_real_4: 0.23672570288181305 (0.24597408290092765) | > loss_disc_real_5: 0.23407834768295288 (0.2536950054077002) | > loss_0: 2.532043218612671 (2.6948706810290997) | > grad_norm_0: tensor(94.4118, device='cuda:0') (tensor(61.2932, device='cuda:0')) | > loss_gen: 2.1388189792633057 (1.9363389968872071) | > loss_kl: 2.2482879161834717 (2.150834604410024) | > loss_feat: 3.200146436691284 (3.0428547015556924) | > loss_mel: 17.137331008911133 (17.804946385897125) | > loss_duration: 1.6551706790924072 (1.5932669401168824) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.3797550201416 (26.528241583017202) | > grad_norm_1: tensor(7641.1460, device='cuda:0') (tensor(3855.0854, device='cuda:0')) | > current_lr_0: 0.0001867812400501323 | > current_lr_1: 0.0001867812400501323 | > step_time: 0.4941 (0.4521624161646916) | > loader_time: 0.0116 (0.008310695794912485) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577659096516354 (-0.00015120439126457656) | > avg_step_time: 0.457586063465602 (-0.006674937798943292) | > avg_loss_disc: 2.6887328053863957 (-0.07339929526960276) | > avg_loss_disc_real_0: 0.16992612434944637 (+0.008137686781480274) | > avg_loss_disc_real_1: 0.2311095935358128 (+0.010952313601131164) | > avg_loss_disc_real_2: 0.23652660259058778 (-0.005098291056256893) | > avg_loss_disc_real_3: 0.23179707266915012 (-0.0101498971942445) | > avg_loss_disc_real_4: 0.24590898924310442 (+0.0021245053536455283) | > avg_loss_disc_real_5: 0.2532127155384547 (+0.006732884427191582) | > avg_loss_0: 2.6887328053863957 (-0.07339929526960276) | > avg_grad_norm_0: tensor(63.3598, device='cuda:0') (tensor(-4.1802, device='cuda:0')) | > avg_loss_gen: 1.9465704568674866 (+0.030215904746257127) | > avg_loss_kl: 2.1535878953799386 (-0.09053182434028528) | > avg_loss_feat: 3.050021077545596 (-0.07546112906764924) | > avg_loss_mel: 17.80782359754536 (-0.32051541771686587) | > avg_loss_duration: 1.5974521670542972 (-0.0034903405417858746) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.555455086936412 (-0.45978280188332477) | > avg_grad_norm_1: tensor(3960.7068, device='cuda:0') (+tensor(97.8684, device='cuda:0'))  > EPOCH: 548/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:50:47)   --> TIME: 2025-02-25 13:51:08 -- STEP: 43/72 -- GLOBAL_STEP: 1039500 | > loss_disc: 2.7141096591949463 (2.652719187182049) | > loss_disc_real_0: 0.14474263787269592 (0.1596060289546501) | > loss_disc_real_1: 0.2368069887161255 (0.22223201498042705) | > loss_disc_real_2: 0.25407642126083374 (0.22865740712298904) | > loss_disc_real_3: 0.2493068128824234 (0.23742794366770012) | > loss_disc_real_4: 0.25777387619018555 (0.24150027681228725) | > loss_disc_real_5: 0.2664608359336853 (0.24230011743168497) | > loss_0: 2.7141096591949463 (2.652719187182049) | > grad_norm_0: tensor(90.7063, device='cuda:0') (tensor(66.3059, device='cuda:0')) | > loss_gen: 2.120145559310913 (2.049444786337919) | > loss_kl: 2.275751829147339 (2.2183683500733484) | > loss_feat: 3.5126373767852783 (3.392724075982737) | > loss_mel: 17.334096908569336 (18.090366141740667) | > loss_duration: 1.624514102935791 (1.586635287417922) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.867145538330078 (27.337538519570995) | > grad_norm_1: tensor(7885.4976, device='cuda:0') (tensor(5287.9087, device='cuda:0')) | > current_lr_0: 0.00018675789239512602 | > current_lr_1: 0.00018675789239512602 | > step_time: 0.462 (0.44480418604473737) | > loader_time: 0.01 (0.007575694904770962) --> EVAL PERFORMANCE | > avg_loader_time: 0.008709309806286447 (+0.0001316507097700939) | > avg_step_time: 0.45868915235492547 (+0.001103088889323478) | > avg_loss_disc: 2.6840346027427997 (-0.0046982026435959945) | > avg_loss_disc_real_0: 0.1656855773967756 (-0.0042405469526707595) | > avg_loss_disc_real_1: 0.21861777695971477 (-0.012491816576098047) | > avg_loss_disc_real_2: 0.2270957267200443 (-0.009430875870543476) | > avg_loss_disc_real_3: 0.235963052427265 (+0.0041659797581148905) | > avg_loss_disc_real_4: 0.24246183310596037 (-0.003447156137144053) | > avg_loss_disc_real_5: 0.24552523900925274 (-0.007687476529201959) | > avg_loss_0: 2.6840346027427997 (-0.0046982026435959945) | > avg_grad_norm_0: tensor(76.8864, device='cuda:0') (+tensor(13.5266, device='cuda:0')) | > avg_loss_gen: 1.9873436706166867 (+0.040773213749200066) | > avg_loss_kl: 2.2020803857857065 (+0.048492490405767974) | > avg_loss_feat: 3.26218001943239 (+0.2121589418867944) | > avg_loss_mel: 17.831936554169992 (+0.024112956624630755) | > avg_loss_duration: 1.605953874722333 (+0.008501707668035818) | > avg_amp_scaler: 16.225352112676056 (+0.22535211267605604) | > avg_loss_1: 26.889494586998307 (+0.33403950006189476) | > avg_grad_norm_1: tensor(4943.9556, device='cuda:0') (+tensor(983.2488, device='cuda:0'))  > EPOCH: 549/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:51:24)   --> TIME: 2025-02-25 13:51:36 -- STEP: 21/72 -- GLOBAL_STEP: 1039550 | > loss_disc: 2.8270509243011475 (2.676448867434548) | > loss_disc_real_0: 0.085476815700531 (0.15478322264694033) | > loss_disc_real_1: 0.26385268568992615 (0.2274573097626368) | > loss_disc_real_2: 0.27268025279045105 (0.24163482373669035) | > loss_disc_real_3: 0.253187358379364 (0.23453534713813237) | > loss_disc_real_4: 0.2908420264720917 (0.24433003720783053) | > loss_disc_real_5: 0.2792051136493683 (0.2478679780449186) | > loss_0: 2.8270509243011475 (2.676448867434548) | > grad_norm_0: tensor(87.4488, device='cuda:0') (tensor(70.7371, device='cuda:0')) | > loss_gen: 1.8280055522918701 (2.1168316262108937) | > loss_kl: 2.0112361907958984 (2.0195995966593423) | > loss_feat: 2.9545555114746094 (3.490757170177641) | > loss_mel: 17.82896614074707 (17.674375170753116) | > loss_duration: 1.5817110538482666 (1.5613778148378645) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.2044734954834 (26.86294128781273) | > grad_norm_1: tensor(3219.7659, device='cuda:0') (tensor(4275.0908, device='cuda:0')) | > current_lr_0: 0.00018673454765857661 | > current_lr_1: 0.00018673454765857661 | > step_time: 0.437 (0.46255695252191453) | > loader_time: 0.0071 (0.0069804078056698755)  --> TIME: 2025-02-25 13:52:00 -- STEP: 71/72 -- GLOBAL_STEP: 1039600 | > loss_disc: 2.5546507835388184 (2.668697431053914) | > loss_disc_real_0: 0.15140071511268616 (0.15676593654592277) | > loss_disc_real_1: 0.17924918234348297 (0.22279573599217642) | > loss_disc_real_2: 0.21925431489944458 (0.23502230035587096) | > loss_disc_real_3: 0.22909480333328247 (0.23326150443352445) | > loss_disc_real_4: 0.22134503722190857 (0.24379170318724405) | > loss_disc_real_5: 0.23012687265872955 (0.24906199393977582) | > loss_0: 2.5546507835388184 (2.668697431053914) | > grad_norm_0: tensor(66.4870, device='cuda:0') (tensor(76.9199, device='cuda:0')) | > loss_gen: 2.096850872039795 (1.9989902435893743) | > loss_kl: 1.9827289581298828 (2.0573010847602085) | > loss_feat: 3.374962568283081 (3.241134435358182) | > loss_mel: 16.726255416870117 (17.32128491200192) | > loss_duration: 1.5886423587799072 (1.5907415269126355) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.769439697265625 (26.20945207837602) | > grad_norm_1: tensor(5766.9370, device='cuda:0') (tensor(4885.2993, device='cuda:0')) | > current_lr_0: 0.00018673454765857661 | > current_lr_1: 0.00018673454765857661 | > step_time: 0.4975 (0.4639675482897691) | > loader_time: 0.0082 (0.008776940090555541) --> EVAL PERFORMANCE | > avg_loader_time: 0.008776940090555541 (+6.763028426909345e-05) | > avg_step_time: 0.4639675482897691 (+0.005278395934843638) | > avg_loss_disc: 2.668697431053914 (-0.015337171688885487) | > avg_loss_disc_real_0: 0.15676593654592277 (-0.008919640850852839) | > avg_loss_disc_real_1: 0.22279573599217642 (+0.004177959032461653) | > avg_loss_disc_real_2: 0.23502230035587096 (+0.007926573635826661) | > avg_loss_disc_real_3: 0.23326150443352445 (-0.0027015479937405606) | > avg_loss_disc_real_4: 0.24379170318724405 (+0.001329870081283685) | > avg_loss_disc_real_5: 0.24906199393977582 (+0.0035367549305230783) | > avg_loss_0: 2.668697431053914 (-0.015337171688885487) | > avg_grad_norm_0: tensor(76.9199, device='cuda:0') (+tensor(0.0335, device='cuda:0')) | > avg_loss_gen: 1.9989902435893743 (+0.011646572972687563) | > avg_loss_kl: 2.0573010847602085 (-0.14477930102549808) | > avg_loss_feat: 3.241134435358182 (-0.02104558407420809) | > avg_loss_mel: 17.32128491200192 (-0.5106516421680709) | > avg_loss_duration: 1.5907415269126355 (-0.01521234780969749) | > avg_amp_scaler: 16.0 (-0.22535211267605604) | > avg_loss_1: 26.20945207837602 (-0.6800425086222859) | > avg_grad_norm_1: tensor(4885.2993, device='cuda:0') (tensor(-58.6562, device='cuda:0'))  > EPOCH: 550/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:52:02)   --> TIME: 2025-02-25 13:52:26 -- STEP: 49/72 -- GLOBAL_STEP: 1039650 | > loss_disc: 2.6770176887512207 (2.6206664260552865) | > loss_disc_real_0: 0.1316896229982376 (0.1459136257062153) | > loss_disc_real_1: 0.22690026462078094 (0.22437094182384257) | > loss_disc_real_2: 0.23173905909061432 (0.23222703562707317) | > loss_disc_real_3: 0.2113027423620224 (0.2312210579307712) | > loss_disc_real_4: 0.22510352730751038 (0.24108006394639306) | > loss_disc_real_5: 0.2350403517484665 (0.24401386964077854) | > loss_0: 2.6770176887512207 (2.6206664260552865) | > grad_norm_0: tensor(84.6390, device='cuda:0') (tensor(93.4728, device='cuda:0')) | > loss_gen: 1.7755017280578613 (2.030649703376148) | > loss_kl: 2.062906265258789 (1.9514448010191625) | > loss_feat: 2.785611152648926 (3.387373535000548) | > loss_mel: 15.82719612121582 (17.111919675554542) | > loss_duration: 1.6325185298919678 (1.5647517150762134) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.08373260498047 (26.046139503011897) | > grad_norm_1: tensor(3975.2671, device='cuda:0') (tensor(6066.8926, device='cuda:0')) | > current_lr_0: 0.00018671120584011928 | > current_lr_1: 0.00018671120584011928 | > step_time: 0.4499 (0.44620491047294775) | > loader_time: 0.0097 (0.007688731563334562) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611101499745545 (-0.00016583859080999616) | > avg_step_time: 0.45950830486458794 (-0.00445924342518117) | > avg_loss_disc: 2.629573929477746 (-0.039123501576168174) | > avg_loss_disc_real_0: 0.1519381551885269 (-0.004827781357395866) | > avg_loss_disc_real_1: 0.2240283103056357 (+0.0012325743134592726) | > avg_loss_disc_real_2: 0.23119199108070052 (-0.003830309275170446) | > avg_loss_disc_real_3: 0.231986800759611 (-0.001274703673913441) | > avg_loss_disc_real_4: 0.2419154769937757 (-0.001876226193468361) | > avg_loss_disc_real_5: 0.24457985141747435 (-0.004482142522301474) | > avg_loss_0: 2.629573929477746 (-0.039123501576168174) | > avg_grad_norm_0: tensor(97.3199, device='cuda:0') (+tensor(20.3999, device='cuda:0')) | > avg_loss_gen: 2.0218978599763253 (+0.022907616386951046) | > avg_loss_kl: 1.9895193560022704 (-0.06778172875793809) | > avg_loss_feat: 3.3223719899083526 (+0.08123755455017045) | > avg_loss_mel: 17.063216424324136 (-0.2580684876777859) | > avg_loss_duration: 1.5823895158902022 (-0.008352011022433325) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.979395181360378 (-0.23005689701564336) | > avg_grad_norm_1: tensor(6145.3945, device='cuda:0') (+tensor(1260.0952, device='cuda:0'))  > EPOCH: 551/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:52:39)   --> TIME: 2025-02-25 13:52:53 -- STEP: 27/72 -- GLOBAL_STEP: 1039700 | > loss_disc: 2.5811214447021484 (2.746545164673417) | > loss_disc_real_0: 0.27128928899765015 (0.19049867932443265) | > loss_disc_real_1: 0.28399592638015747 (0.23614832758903503) | > loss_disc_real_2: 0.2824857234954834 (0.23210684568793682) | > loss_disc_real_3: 0.2661951780319214 (0.23233923647138807) | > loss_disc_real_4: 0.2048771232366562 (0.24076643089453378) | > loss_disc_real_5: 0.2167413830757141 (0.24184610501483636) | > loss_0: 2.5811214447021484 (2.746545164673417) | > grad_norm_0: tensor(45.3266, device='cuda:0') (tensor(85.7425, device='cuda:0')) | > loss_gen: 2.2753243446350098 (1.9843554540916726) | > loss_kl: 3.070590019226074 (2.628985056170711) | > loss_feat: 3.4074838161468506 (3.3060534441912615) | > loss_mel: 22.647689819335938 (20.78552588710079) | > loss_duration: 1.645227074623108 (1.6139420933193631) | > amp_scaler: 16.0 (16.0) | > loss_1: 33.04631423950195 (30.318862137971102) | > grad_norm_1: tensor(1027.3102, device='cuda:0') (tensor(4088.5830, device='cuda:0')) | > current_lr_0: 0.00018668786693938925 | > current_lr_1: 0.00018668786693938925 | > step_time: 0.4452 (0.4376670166298195) | > loader_time: 0.0084 (0.006996349052146629) --> EVAL PERFORMANCE | > avg_loader_time: 0.008692425741276275 (+8.132424153073012e-05) | > avg_step_time: 0.4578655101883579 (-0.0016427946762300194) | > avg_loss_disc: 2.7529823511419154 (+0.12340842166416932) | > avg_loss_disc_real_0: 0.19169991022684207 (+0.039761755038315166) | > avg_loss_disc_real_1: 0.22995642897948412 (+0.0059281186738484315) | > avg_loss_disc_real_2: 0.2441098003740042 (+0.01291780929330369) | > avg_loss_disc_real_3: 0.23285271898961404 (+0.0008659182300030377) | > avg_loss_disc_real_4: 0.24481857221731 (+0.0029030952235342966) | > avg_loss_disc_real_5: 0.2442021991165591 (-0.00037765230091524327) | > avg_loss_0: 2.7529823511419154 (+0.12340842166416932) | > avg_grad_norm_0: tensor(89.0574, device='cuda:0') (tensor(-8.2625, device='cuda:0')) | > avg_loss_gen: 2.000367940311701 (-0.021529919664624142) | > avg_loss_kl: 2.5039563061485834 (+0.5144369501463131) | > avg_loss_feat: 3.264079446523962 (-0.058292543384390605) | > avg_loss_mel: 19.83908939361572 (+2.7758729692915836) | > avg_loss_duration: 1.6394797983303877 (+0.05709028244018555) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.24697314517599 (+3.2675779638156115) | > avg_grad_norm_1: tensor(4579.1108, device='cuda:0') (tensor(-1566.2837, device='cuda:0'))  > EPOCH: 552/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:53:16)   --> TIME: 2025-02-25 13:53:20 -- STEP: 5/72 -- GLOBAL_STEP: 1039750 | > loss_disc: 2.514014959335327 (2.6221478462219237) | > loss_disc_real_0: 0.13108429312705994 (0.16031161248683928) | > loss_disc_real_1: 0.22709740698337555 (0.22024360299110413) | > loss_disc_real_2: 0.2208813577890396 (0.2299651801586151) | > loss_disc_real_3: 0.21261604130268097 (0.23942252695560456) | > loss_disc_real_4: 0.22779938578605652 (0.24723616242408752) | > loss_disc_real_5: 0.22698451578617096 (0.24958700239658355) | > loss_0: 2.514014959335327 (2.6221478462219237) | > grad_norm_0: tensor(8.8503, device='cuda:0') (tensor(28.8929, device='cuda:0')) | > loss_gen: 2.1931238174438477 (2.047690773010254) | > loss_kl: 2.1834933757781982 (2.154478907585144) | > loss_feat: 3.650038957595825 (3.275482654571533) | > loss_mel: 18.79954719543457 (18.23108787536621) | > loss_duration: 1.5414721965789795 (1.5856810808181763) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.367673873901367 (27.294421005249024) | > grad_norm_1: tensor(5511.0957, device='cuda:0') (tensor(3985.6931, device='cuda:0')) | > current_lr_0: 0.0001866645309560218 | > current_lr_1: 0.0001866645309560218 | > step_time: 0.4496 (0.4353615760803223) | > loader_time: 0.006 (0.006524324417114258)  --> TIME: 2025-02-25 13:53:44 -- STEP: 55/72 -- GLOBAL_STEP: 1039800 | > loss_disc: 2.7724459171295166 (2.726674461364745) | > loss_disc_real_0: 0.14045670628547668 (0.15481059530919253) | > loss_disc_real_1: 0.23378846049308777 (0.23011401891708375) | > loss_disc_real_2: 0.24886056780815125 (0.23944070637226106) | > loss_disc_real_3: 0.24134346842765808 (0.2352913721041246) | > loss_disc_real_4: 0.2399834841489792 (0.24403595219958912) | > loss_disc_real_5: 0.2439318299293518 (0.24658518785780126) | > loss_0: 2.7724459171295166 (2.726674461364745) | > grad_norm_0: tensor(84.5289, device='cuda:0') (tensor(70.8028, device='cuda:0')) | > loss_gen: 1.7241880893707275 (1.9658235723322088) | > loss_kl: 2.2081351280212402 (2.1368346886201333) | > loss_feat: 3.2565364837646484 (3.2983618606220593) | > loss_mel: 18.727895736694336 (17.86025421836159) | > loss_duration: 1.6595916748046875 (1.5921399615027687) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.57634735107422 (26.853413980657404) | > grad_norm_1: tensor(4369.9292, device='cuda:0') (tensor(4369.4009, device='cuda:0')) | > current_lr_0: 0.0001866645309560218 | > current_lr_1: 0.0001866645309560218 | > step_time: 0.4691 (0.45553743622519755) | > loader_time: 0.0093 (0.00797287334095348) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685098567479092 (-7.327173797182923e-06) | > avg_step_time: 0.46549667438990633 (+0.007631164201548413) | > avg_loss_disc: 2.71548138537877 (-0.037500965763145544) | > avg_loss_disc_real_0: 0.14683230305221723 (-0.044867607174624846) | > avg_loss_disc_real_1: 0.22970050930137365 (-0.0002559196781104778) | > avg_loss_disc_real_2: 0.24278776137761668 (-0.0013220389963875256) | > avg_loss_disc_real_3: 0.23857494580074096 (+0.0057222268111269126) | > avg_loss_disc_real_4: 0.2464281093906349 (+0.0016095371733249042) | > avg_loss_disc_real_5: 0.25097828893594337 (+0.006776089819384268) | > avg_loss_0: 2.71548138537877 (-0.037500965763145544) | > avg_grad_norm_0: tensor(72.4697, device='cuda:0') (tensor(-16.5877, device='cuda:0')) | > avg_loss_gen: 2.0086778268008163 (+0.008309886489115126) | > avg_loss_kl: 2.1985565927666673 (-0.3053997133819162) | > avg_loss_feat: 3.3690151899633274 (+0.1049357434393654) | > avg_loss_mel: 18.33896876053071 (-1.5001206330850074) | > avg_loss_duration: 1.60832331382053 (-0.03115648450985775) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.52354143706846 (-1.7234317081075297) | > avg_grad_norm_1: tensor(4998.3389, device='cuda:0') (+tensor(419.2280, device='cuda:0'))  > EPOCH: 553/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:53:54)   --> TIME: 2025-02-25 13:54:11 -- STEP: 33/72 -- GLOBAL_STEP: 1039850 | > loss_disc: 2.6613311767578125 (2.576093565333973) | > loss_disc_real_0: 0.048343248665332794 (0.10695827707196727) | > loss_disc_real_1: 0.20246993005275726 (0.21773094390377853) | > loss_disc_real_2: 0.24030248820781708 (0.24146212276184198) | > loss_disc_real_3: 0.24277076125144958 (0.23274074720613885) | > loss_disc_real_4: 0.2545303404331207 (0.24032368262608847) | > loss_disc_real_5: 0.26553165912628174 (0.24058359725908798) | > loss_0: 2.6613311767578125 (2.576093565333973) | > grad_norm_0: tensor(90.0958, device='cuda:0') (tensor(72.3564, device='cuda:0')) | > loss_gen: 1.9487522840499878 (2.1406251047596796) | > loss_kl: 1.9625380039215088 (2.068285736170682) | > loss_feat: 3.7316596508026123 (3.7726117480884898) | > loss_mel: 17.797361373901367 (18.097651568326082) | > loss_duration: 1.5688972473144531 (1.5650974945588545) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.009206771850586 (27.644271792787492) | > grad_norm_1: tensor(5302.3955, device='cuda:0') (tensor(5670.0620, device='cuda:0')) | > current_lr_0: 0.00018664119788965231 | > current_lr_1: 0.00018664119788965231 | > step_time: 0.4611 (0.4396265347798665) | > loader_time: 0.008 (0.007256139408458363) --> EVAL PERFORMANCE | > avg_loader_time: 0.008742940257972395 (+5.784169049330261e-05) | > avg_step_time: 0.4576558898872053 (-0.007840784502701037) | > avg_loss_disc: 2.5865410146578935 (-0.12894037072087627) | > avg_loss_disc_real_0: 0.12010654807090759 (-0.026725754981309635) | > avg_loss_disc_real_1: 0.22189943870188486 (-0.007801070599488785) | > avg_loss_disc_real_2: 0.23254407564519156 (-0.010243685732425123) | > avg_loss_disc_real_3: 0.23485319379349828 (-0.0037217520072426735) | > avg_loss_disc_real_4: 0.23949571730385363 (-0.006932392086781258) | > avg_loss_disc_real_5: 0.2418027700672687 (-0.009175518868674681) | > avg_loss_0: 2.5865410146578935 (-0.12894037072087627) | > avg_grad_norm_0: tensor(79.4902, device='cuda:0') (+tensor(7.0205, device='cuda:0')) | > avg_loss_gen: 2.1184419779710373 (+0.10976415117022098) | > avg_loss_kl: 2.0639983868934744 (-0.1345582058731929) | > avg_loss_feat: 3.594911387268926 (+0.22589619730559862) | > avg_loss_mel: 17.616546550267177 (-0.7224222102635345) | > avg_loss_duration: 1.594059190280001 (-0.014264123540528972) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.987957672334055 (-0.5355837647344046) | > avg_grad_norm_1: tensor(5611.6558, device='cuda:0') (+tensor(613.3169, device='cuda:0'))  > EPOCH: 554/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:54:31)   --> TIME: 2025-02-25 13:54:38 -- STEP: 11/72 -- GLOBAL_STEP: 1039900 | > loss_disc: 2.606752634048462 (2.594015750018033) | > loss_disc_real_0: 0.07086893916130066 (0.10663111507892609) | > loss_disc_real_1: 0.22107715904712677 (0.22728034447539935) | > loss_disc_real_2: 0.22168877720832825 (0.22705127163366837) | > loss_disc_real_3: 0.17931438982486725 (0.2329127625985579) | > loss_disc_real_4: 0.22084060311317444 (0.24281907352534207) | > loss_disc_real_5: 0.2346942126750946 (0.24519782039252194) | > loss_0: 2.606752634048462 (2.594015750018033) | > grad_norm_0: tensor(93.4007, device='cuda:0') (tensor(82.1606, device='cuda:0')) | > loss_gen: 1.7444961071014404 (2.000191710212014) | > loss_kl: 2.2046780586242676 (1.9764126105742021) | > loss_feat: 3.5203866958618164 (3.4825419946150347) | > loss_mel: 17.790956497192383 (17.26830716566606) | > loss_duration: 1.557778000831604 (1.557039195841009) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.818294525146484 (26.28449283946644) | > grad_norm_1: tensor(4795.9761, device='cuda:0') (tensor(5050.9180, device='cuda:0')) | > current_lr_0: 0.0001866178677399161 | > current_lr_1: 0.0001866178677399161 | > step_time: 0.4275 (0.4352172071283514) | > loader_time: 0.0059 (0.006907311352816495)  --> TIME: 2025-02-25 13:55:01 -- STEP: 61/72 -- GLOBAL_STEP: 1039950 | > loss_disc: 2.4678025245666504 (2.58176402185784) | > loss_disc_real_0: 0.20491212606430054 (0.12116609367190814) | > loss_disc_real_1: 0.2365926206111908 (0.21902204146150683) | > loss_disc_real_2: 0.16350986063480377 (0.22769690414921182) | > loss_disc_real_3: 0.22633886337280273 (0.23213368922960562) | > loss_disc_real_4: 0.2415010780096054 (0.24027000343213317) | > loss_disc_real_5: 0.24115541577339172 (0.24332222049353552) | > loss_0: 2.4678025245666504 (2.58176402185784) | > grad_norm_0: tensor(124.0162, device='cuda:0') (tensor(90.0749, device='cuda:0')) | > loss_gen: 2.195904493331909 (2.0736865430581757) | > loss_kl: 2.069392442703247 (2.0030809758139436) | > loss_feat: 3.5363903045654297 (3.5189395029036725) | > loss_mel: 17.157318115234375 (17.160360914761903) | > loss_duration: 1.6194803714752197 (1.576106565897582) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.57848358154297 (26.33217445748751) | > grad_norm_1: tensor(6919.9814, device='cuda:0') (tensor(6027.5342, device='cuda:0')) | > current_lr_0: 0.0001866178677399161 | > current_lr_1: 0.0001866178677399161 | > step_time: 0.4726 (0.44643345426340575) | > loader_time: 0.0104 (0.008193848563022304) --> EVAL PERFORMANCE | > avg_loader_time: 0.008649574199192964 (-9.336605877943022e-05) | > avg_step_time: 0.4556627374299815 (-0.0019931524572238124) | > avg_loss_disc: 2.5908357559795117 (+0.004294741321618112) | > avg_loss_disc_real_0: 0.12531558602628573 (+0.005209037955378137) | > avg_loss_disc_real_1: 0.21862479939427173 (-0.0032746393076131253) | > avg_loss_disc_real_2: 0.2280854517725152 (-0.004458623872676343) | > avg_loss_disc_real_3: 0.23270688594227107 (-0.002146307851227214) | > avg_loss_disc_real_4: 0.240727069722095 (+0.001231352418241377) | > avg_loss_disc_real_5: 0.24360363961945117 (+0.001800869552182477) | > avg_loss_0: 2.5908357559795117 (+0.004294741321618112) | > avg_grad_norm_0: tensor(90.7872, device='cuda:0') (+tensor(11.2970, device='cuda:0')) | > avg_loss_gen: 2.0683102658097163 (-0.05013171216132095) | > avg_loss_kl: 2.015286193767064 (-0.04871219312641051) | > avg_loss_feat: 3.471261659138639 (-0.12364972813028707) | > avg_loss_mel: 17.151754338976364 (-0.4647922112908134) | > avg_loss_duration: 1.5845920787730687 (-0.009467111506932246) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.29120455997091 (-0.6967531123631439) | > avg_grad_norm_1: tensor(6002.2026, device='cuda:0') (+tensor(390.5469, device='cuda:0'))  > EPOCH: 555/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:55:08)   --> TIME: 2025-02-25 13:55:28 -- STEP: 39/72 -- GLOBAL_STEP: 1040000 | > loss_disc: 2.5031325817108154 (2.5918737130287357) | > loss_disc_real_0: 0.08783246576786041 (0.12606070353052554) | > loss_disc_real_1: 0.22186967730522156 (0.22329791501546517) | > loss_disc_real_2: 0.21709252893924713 (0.22589641618422973) | > loss_disc_real_3: 0.20821824669837952 (0.23080433905124664) | > loss_disc_real_4: 0.22667698562145233 (0.23972885616314718) | > loss_disc_real_5: 0.21563532948493958 (0.24309714214924055) | > loss_0: 2.5031325817108154 (2.5918737130287357) | > grad_norm_0: tensor(106.9615, device='cuda:0') (tensor(95.6182, device='cuda:0')) | > loss_gen: 1.9362597465515137 (2.105395347644121) | > loss_kl: 2.1471059322357178 (1.9608422731741881) | > loss_feat: 3.7124431133270264 (3.572419362190442) | > loss_mel: 17.429094314575195 (17.191550792791897) | > loss_duration: 1.5828018188476562 (1.5491471473987286) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.80770492553711 (26.379354623647835) | > grad_norm_1: tensor(8570.6309, device='cuda:0') (tensor(6770.6514, device='cuda:0')) | > current_lr_0: 0.0001865945405064486 | > current_lr_1: 0.0001865945405064486 | > step_time: 0.4734 (0.4543285064208202) | > loader_time: 0.0084 (0.007400482128827999) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1040000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008625389824450855 (-2.4184374742108883e-05) | > avg_step_time: 0.46656762714117345 (+0.010904889711191967) | > avg_loss_disc: 2.5980351273442652 (+0.007199371364753571) | > avg_loss_disc_real_0: 0.12886044188914159 (+0.0035448558628558557) | > avg_loss_disc_real_1: 0.22226181122618663 (+0.0036370118319148936) | > avg_loss_disc_real_2: 0.23009260250648983 (+0.0020071507339746164) | > avg_loss_disc_real_3: 0.23328289893311513 (+0.0005760129908440603) | > avg_loss_disc_real_4: 0.2404613159072231 (-0.00026575381487189964) | > avg_loss_disc_real_5: 0.24439198135490148 (+0.0007883417354503097) | > avg_loss_0: 2.5980351273442652 (+0.007199371364753571) | > avg_grad_norm_0: tensor(96.8275, device='cuda:0') (+tensor(6.0404, device='cuda:0')) | > avg_loss_gen: 2.1147797561027644 (+0.04646949029304803) | > avg_loss_kl: 2.00654105401375 (-0.008745139753314035) | > avg_loss_feat: 3.5431296892569097 (+0.07186803011827081) | > avg_loss_mel: 17.124643688470538 (-0.027110650505825618) | > avg_loss_duration: 1.5797844755817467 (-0.004807603191322007) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.368878539179413 (+0.07767397920850172) | > avg_grad_norm_1: tensor(6715.9429, device='cuda:0') (+tensor(713.7402, device='cuda:0'))  > EPOCH: 556/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:55:48)   --> TIME: 2025-02-25 13:55:58 -- STEP: 17/72 -- GLOBAL_STEP: 1040050 | > loss_disc: 2.3168535232543945 (2.5655491211835075) | > loss_disc_real_0: 0.0575200580060482 (0.13427793672856161) | > loss_disc_real_1: 0.25628969073295593 (0.2299015110029894) | > loss_disc_real_2: 0.18057742714881897 (0.218106606427361) | > loss_disc_real_3: 0.20421339571475983 (0.23177193017566905) | > loss_disc_real_4: 0.23737895488739014 (0.23898181143928976) | > loss_disc_real_5: 0.23027348518371582 (0.2442699109806734) | > loss_0: 2.3168535232543945 (2.5655491211835075) | > grad_norm_0: tensor(135.8854, device='cuda:0') (tensor(94.6482, device='cuda:0')) | > loss_gen: 2.242964506149292 (2.1560901333304012) | > loss_kl: 2.091851234436035 (1.962840753443101) | > loss_feat: 4.523303985595703 (3.7193806592155907) | > loss_mel: 18.11083221435547 (17.38253559785731) | > loss_duration: 1.5908894538879395 (1.5344071247998405) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.559839248657227 (26.75525396010455) | > grad_norm_1: tensor(9753.4424, device='cuda:0') (tensor(6674.1826, device='cuda:0')) | > current_lr_0: 0.00018657121618888527 | > current_lr_1: 0.00018657121618888527 | > step_time: 0.4349 (0.43932140574735756) | > loader_time: 0.0088 (0.006786290337057675)  --> TIME: 2025-02-25 13:56:22 -- STEP: 67/72 -- GLOBAL_STEP: 1040100 | > loss_disc: 2.4316983222961426 (2.6438677346528467) | > loss_disc_real_0: 0.11294073611497879 (0.14166435410282505) | > loss_disc_real_1: 0.2666502594947815 (0.23854900006927662) | > loss_disc_real_2: 0.2031548023223877 (0.22856653982134006) | > loss_disc_real_3: 0.23491829633712769 (0.24119381806743678) | > loss_disc_real_4: 0.2408759444952011 (0.2411694682356137) | > loss_disc_real_5: 0.24471743404865265 (0.2445611030752979) | > loss_0: 2.4316983222961426 (2.6438677346528467) | > grad_norm_0: tensor(40.7132, device='cuda:0') (tensor(91.6479, device='cuda:0')) | > loss_gen: 2.2122817039489746 (2.1257098379419808) | > loss_kl: 2.31122088432312 (2.139357926240608) | > loss_feat: 3.721813678741455 (3.515030636716245) | > loss_mel: 19.130693435668945 (17.70080953569555) | > loss_duration: 1.6895127296447754 (1.6008827259291465) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.065521240234375 (27.081790354714464) | > grad_norm_1: tensor(7945.7979, device='cuda:0') (tensor(6435.1792, device='cuda:0')) | > current_lr_0: 0.00018657121618888527 | > current_lr_1: 0.00018657121618888527 | > step_time: 0.5231 (0.45684635105417737) | > loader_time: 0.0121 (0.008399547036014383) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584704197628395 (-4.068562682246063e-05) | > avg_step_time: 0.4604328081641399 (-0.0061348189770335515) | > avg_loss_disc: 2.650059119076796 (+0.052023991732530916) | > avg_loss_disc_real_0: 0.13853159305495275 (+0.00967115116581116) | > avg_loss_disc_real_1: 0.2378845901136667 (+0.015622778887480065) | > avg_loss_disc_real_2: 0.23313424725767593 (+0.0030416447511860956) | > avg_loss_disc_real_3: 0.24024633160779174 (+0.006963432674676612) | > avg_loss_disc_real_4: 0.2404071608899345 (-5.415501728861516e-05) | > avg_loss_disc_real_5: 0.24388612114207844 (-0.0005058602128230338) | > avg_loss_0: 2.650059119076796 (+0.052023991732530916) | > avg_grad_norm_0: tensor(91.8171, device='cuda:0') (tensor(-5.0104, device='cuda:0')) | > avg_loss_gen: 2.117573343532186 (+0.0027935874294215957) | > avg_loss_kl: 2.1472146578238043 (+0.14067360381005445) | > avg_loss_feat: 3.5079474482737796 (-0.035182240983130164) | > avg_loss_mel: 17.68220141236212 (+0.5575577238915805) | > avg_loss_duration: 1.6034865127482885 (+0.023702037166541734) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.05842308259346 (+0.6895445434140477) | > avg_grad_norm_1: tensor(6446.6084, device='cuda:0') (tensor(-269.3345, device='cuda:0'))  > EPOCH: 557/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:56:26)   --> TIME: 2025-02-25 13:56:48 -- STEP: 45/72 -- GLOBAL_STEP: 1040150 | > loss_disc: 2.7026243209838867 (2.6132467799716523) | > loss_disc_real_0: 0.12434360384941101 (0.12147550673948394) | > loss_disc_real_1: 0.24159543216228485 (0.22203650242752498) | > loss_disc_real_2: 0.2904074788093567 (0.22475035985310873) | > loss_disc_real_3: 0.23855990171432495 (0.2389352560043335) | > loss_disc_real_4: 0.23907224833965302 (0.24387826455963982) | > loss_disc_real_5: 0.256702721118927 (0.2479789545138677) | > loss_0: 2.7026243209838867 (2.6132467799716523) | > grad_norm_0: tensor(130.1908, device='cuda:0') (tensor(101.9276, device='cuda:0')) | > loss_gen: 2.085374355316162 (2.1163809140523275) | > loss_kl: 2.310192108154297 (2.0195588694678412) | > loss_feat: 3.5380020141601562 (3.6601661364237468) | > loss_mel: 16.997316360473633 (17.540383699205186) | > loss_duration: 1.586796522140503 (1.5646963463889227) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.517681121826172 (26.901185692681207) | > grad_norm_1: tensor(7105.8359, device='cuda:0') (tensor(7156.4854, device='cuda:0')) | > current_lr_0: 0.00018654789478686164 | > current_lr_1: 0.00018654789478686164 | > step_time: 0.4562 (0.4396558867560493) | > loader_time: 0.0091 (0.00779730478922526) --> EVAL PERFORMANCE | > avg_loader_time: 0.008794767755857654 (+0.0002100635582292596) | > avg_step_time: 0.4548848716306015 (-0.005547936533538411) | > avg_loss_disc: 2.6036457444580505 (-0.04641337461874562) | > avg_loss_disc_real_0: 0.12571495923567821 (-0.012816633819274531) | > avg_loss_disc_real_1: 0.22113769344041045 (-0.01674689667325624) | > avg_loss_disc_real_2: 0.2242493230692098 (-0.00888492418846612) | > avg_loss_disc_real_3: 0.23618353294654632 (-0.004062798661245426) | > avg_loss_disc_real_4: 0.24232251333518767 (+0.0019153524452531767) | > avg_loss_disc_real_5: 0.2477655551383193 (+0.0038794339962408553) | > avg_loss_0: 2.6036457444580505 (-0.04641337461874562) | > avg_grad_norm_0: tensor(101.1991, device='cuda:0') (+tensor(9.3820, device='cuda:0')) | > avg_loss_gen: 2.1089571983041897 (-0.008616145227996252) | > avg_loss_kl: 2.053729280619554 (-0.09348537720425032) | > avg_loss_feat: 3.5630427917964016 (+0.05509534352262202) | > avg_loss_mel: 17.37352087799932 (-0.3086805343628001) | > avg_loss_duration: 1.5853843302793906 (-0.01810218246889783) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.684634383295624 (-0.37378869929783676) | > avg_grad_norm_1: tensor(6908.0908, device='cuda:0') (+tensor(461.4824, device='cuda:0'))  > EPOCH: 558/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:57:03)   --> TIME: 2025-02-25 13:57:15 -- STEP: 23/72 -- GLOBAL_STEP: 1040200 | > loss_disc: 2.6710329055786133 (2.5848392610964566) | > loss_disc_real_0: 0.1328597068786621 (0.12399058416485786) | > loss_disc_real_1: 0.22095781564712524 (0.222480283483215) | > loss_disc_real_2: 0.25448712706565857 (0.2194181862084762) | > loss_disc_real_3: 0.23377934098243713 (0.23175127026827438) | > loss_disc_real_4: 0.25294554233551025 (0.23811975121498108) | > loss_disc_real_5: 0.25649604201316833 (0.24113994059355362) | > loss_0: 2.6710329055786133 (2.5848392610964566) | > grad_norm_0: tensor(125.2641, device='cuda:0') (tensor(101.1684, device='cuda:0')) | > loss_gen: 2.0252294540405273 (2.124373943909355) | > loss_kl: 1.892266869544983 (1.9695583581924438) | > loss_feat: 3.2291207313537598 (3.652469427689262) | > loss_mel: 17.019458770751953 (17.415749093760613) | > loss_duration: 1.5659689903259277 (1.537183367687723) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.732046127319336 (26.699334683625594) | > grad_norm_1: tensor(6213.6499, device='cuda:0') (tensor(6873.9551, device='cuda:0')) | > current_lr_0: 0.00018652457630001328 | > current_lr_1: 0.00018652457630001328 | > step_time: 0.4409 (0.4349935054779053) | > loader_time: 0.0063 (0.006572681924571161) --> EVAL PERFORMANCE | > avg_loader_time: 0.008513165191865305 (-0.00028160256399234994) | > avg_step_time: 0.4535269065642021 (-0.0013579650663994025) | > avg_loss_disc: 2.6158625911658917 (+0.012216846707841178) | > avg_loss_disc_real_0: 0.1347111611508988 (+0.008996201915220575) | > avg_loss_disc_real_1: 0.22359544605436463 (+0.0024577526139541728) | > avg_loss_disc_real_2: 0.23171330474212137 (+0.007463981672911568) | > avg_loss_disc_real_3: 0.23731807902665206 (+0.0011345460801057417) | > avg_loss_disc_real_4: 0.24188834953476007 (-0.0004341638004276005) | > avg_loss_disc_real_5: 0.24509763864564224 (-0.002667916492677058) | > avg_loss_0: 2.6158625911658917 (+0.012216846707841178) | > avg_grad_norm_0: tensor(101.7312, device='cuda:0') (+tensor(0.5321, device='cuda:0')) | > avg_loss_gen: 2.1325669792336486 (+0.023609780929458868) | > avg_loss_kl: 2.047226707700273 (-0.0065025729192811355) | > avg_loss_feat: 3.589705722432741 (+0.02666293063633951) | > avg_loss_mel: 17.30875418219768 (-0.0647666958016373) | > avg_loss_duration: 1.5799548222985067 (-0.00542950798088393) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.658208578405247 (-0.02642580489037627) | > avg_grad_norm_1: tensor(7117.0942, device='cuda:0') (+tensor(209.0034, device='cuda:0'))  > EPOCH: 559/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:57:40)   --> TIME: 2025-02-25 13:57:42 -- STEP: 1/72 -- GLOBAL_STEP: 1040250 | > loss_disc: 2.353529930114746 (2.353529930114746) | > loss_disc_real_0: 0.12673325836658478 (0.12673325836658478) | > loss_disc_real_1: 0.28215935826301575 (0.28215935826301575) | > loss_disc_real_2: 0.17811207473278046 (0.17811207473278046) | > loss_disc_real_3: 0.22019970417022705 (0.22019970417022705) | > loss_disc_real_4: 0.22586438059806824 (0.22586438059806824) | > loss_disc_real_5: 0.23886315524578094 (0.23886315524578094) | > loss_0: 2.353529930114746 (2.353529930114746) | > grad_norm_0: tensor(51.7924, device='cuda:0') (tensor(51.7924, device='cuda:0')) | > loss_gen: 2.599820375442505 (2.599820375442505) | > loss_kl: 1.9326539039611816 (1.9326539039611816) | > loss_feat: 4.445007801055908 (4.445007801055908) | > loss_mel: 17.98518943786621 (17.98518943786621) | > loss_duration: 1.545281171798706 (1.545281171798706) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.507951736450195 (28.507951736450195) | > grad_norm_1: tensor(9041.9902, device='cuda:0') (tensor(9041.9902, device='cuda:0')) | > current_lr_0: 0.00018650126072797577 | > current_lr_1: 0.00018650126072797577 | > step_time: 0.4482 (0.44818711280822754) | > loader_time: 0.0069 (0.006942272186279297)  --> TIME: 2025-02-25 13:58:05 -- STEP: 51/72 -- GLOBAL_STEP: 1040300 | > loss_disc: 2.524542808532715 (2.7492129475462663) | > loss_disc_real_0: 0.08541852235794067 (0.19471807777881622) | > loss_disc_real_1: 0.24573063850402832 (0.24061684894795513) | > loss_disc_real_2: 0.22635133564472198 (0.23173653436642067) | > loss_disc_real_3: 0.2454167604446411 (0.23602150789662904) | > loss_disc_real_4: 0.23976722359657288 (0.24676856398582458) | > loss_disc_real_5: 0.2346092015504837 (0.24556771242151074) | > loss_0: 2.524542808532715 (2.7492129475462663) | > grad_norm_0: tensor(49.4599, device='cuda:0') (tensor(78.8007, device='cuda:0')) | > loss_gen: 2.228196382522583 (1.9932307893154668) | > loss_kl: 2.024099588394165 (2.1831073527242624) | > loss_feat: 3.533020257949829 (3.2611911343593225) | > loss_mel: 17.180782318115234 (17.610908545699775) | > loss_duration: 1.615673542022705 (1.5728199037851072) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.581771850585938 (26.621257968977385) | > grad_norm_1: tensor(7652.7393, device='cuda:0') (tensor(4644.7148, device='cuda:0')) | > current_lr_0: 0.00018650126072797577 | > current_lr_1: 0.00018650126072797577 | > step_time: 0.4825 (0.442893107732137) | > loader_time: 0.009 (0.007850862016864852) --> EVAL PERFORMANCE | > avg_loader_time: 0.008683402773360133 (+0.00017023758149482815) | > avg_step_time: 0.45535872016154544 (+0.001831813597343357) | > avg_loss_disc: 2.7205786604276843 (+0.10471606926179255) | > avg_loss_disc_real_0: 0.18804339392923972 (+0.05333223277834093) | > avg_loss_disc_real_1: 0.2373126512262183 (+0.01371720517185368) | > avg_loss_disc_real_2: 0.22859920036624853 (-0.0031141043758728393) | > avg_loss_disc_real_3: 0.23685122228844066 (-0.0004668567382113964) | > avg_loss_disc_real_4: 0.24652831340339823 (+0.004639963868638158) | > avg_loss_disc_real_5: 0.2473391053122534 (+0.0022414666666111527) | > avg_loss_0: 2.7205786604276843 (+0.10471606926179255) | > avg_grad_norm_0: tensor(83.9713, device='cuda:0') (tensor(-17.7599, device='cuda:0')) | > avg_loss_gen: 2.017634131539036 (-0.11493284769461276) | > avg_loss_kl: 2.1846344252707244 (+0.13740771757045156) | > avg_loss_feat: 3.241491307674999 (-0.34821441475774195) | > avg_loss_mel: 17.479627716709192 (+0.17087353451151088) | > avg_loss_duration: 1.5880712875178162 (+0.008116465219309532) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.51145916253748 (-0.14674941586776669) | > avg_grad_norm_1: tensor(4927.5503, device='cuda:0') (tensor(-2189.5439, device='cuda:0'))  > EPOCH: 560/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:58:17)   --> TIME: 2025-02-25 13:58:32 -- STEP: 29/72 -- GLOBAL_STEP: 1040350 | > loss_disc: 2.5301263332366943 (2.6087528754924905) | > loss_disc_real_0: 0.21515928208827972 (0.15079653211708727) | > loss_disc_real_1: 0.1322891116142273 (0.2179540010361836) | > loss_disc_real_2: 0.2545034885406494 (0.2196100271981338) | > loss_disc_real_3: 0.22334876656532288 (0.2284726901301022) | > loss_disc_real_4: 0.21883858740329742 (0.23948457580188226) | > loss_disc_real_5: 0.23618942499160767 (0.24536348827953997) | > loss_0: 2.5301263332366943 (2.6087528754924905) | > grad_norm_0: tensor(52.4890, device='cuda:0') (tensor(94.1584, device='cuda:0')) | > loss_gen: 2.831113815307617 (2.1762500425864904) | > loss_kl: 1.9767848253250122 (2.0417304984454456) | > loss_feat: 4.48444938659668 (3.642811076394443) | > loss_mel: 19.236597061157227 (17.287569901038864) | > loss_duration: 1.574460506439209 (1.5400740195964944) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.103404998779297 (26.688435324307147) | > grad_norm_1: tensor(10073.7754, device='cuda:0') (tensor(7223.8042, device='cuda:0')) | > current_lr_0: 0.00018647794807038476 | > current_lr_1: 0.00018647794807038476 | > step_time: 0.4357 (0.43691118010159197) | > loader_time: 0.0073 (0.007172855837591763) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869725119899696 (+1.3848425636828199e-05) | > avg_step_time: 0.4586068408589968 (+0.00324812069745134) | > avg_loss_disc: 2.6899002437860196 (-0.030678416641664707) | > avg_loss_disc_real_0: 0.151242384503425 (-0.03680100942581471) | > avg_loss_disc_real_1: 0.2245714507052596 (-0.012741200520958701) | > avg_loss_disc_real_2: 0.21917025912815416 (-0.009428941238094374) | > avg_loss_disc_real_3: 0.23439083762571844 (-0.0024603846627222203) | > avg_loss_disc_real_4: 0.24425967184590622 (-0.0022686415574920127) | > avg_loss_disc_real_5: 0.2479564566427553 (+0.0006173513305019196) | > avg_loss_0: 2.6899002437860196 (-0.030678416641664707) | > avg_grad_norm_0: tensor(87.4320, device='cuda:0') (+tensor(3.4608, device='cuda:0')) | > avg_loss_gen: 2.0286894734476655 (+0.011055341908629668) | > avg_loss_kl: 2.1550751890934694 (-0.029559236177254977) | > avg_loss_feat: 3.3665852815332546 (+0.1250939738582555) | > avg_loss_mel: 17.43356363538286 (-0.046064081326331774) | > avg_loss_duration: 1.5910812968939123 (+0.003010009376096079) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.574994852845094 (+0.06353569030761363) | > avg_grad_norm_1: tensor(5814.0820, device='cuda:0') (+tensor(886.5317, device='cuda:0'))  > EPOCH: 561/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:58:54)   --> TIME: 2025-02-25 13:58:59 -- STEP: 7/72 -- GLOBAL_STEP: 1040400 | > loss_disc: 2.9427502155303955 (2.667597770690918) | > loss_disc_real_0: 0.08627188205718994 (0.1552655271121434) | > loss_disc_real_1: 0.25409677624702454 (0.22435300052165985) | > loss_disc_real_2: 0.26072412729263306 (0.26023302333695547) | > loss_disc_real_3: 0.2205737978219986 (0.24388574702399118) | > loss_disc_real_4: 0.2521169185638428 (0.24982126695769175) | > loss_disc_real_5: 0.2587641477584839 (0.24570524905409133) | > loss_0: 2.9427502155303955 (2.667597770690918) | > grad_norm_0: tensor(124.6408, device='cuda:0') (tensor(63.5824, device='cuda:0')) | > loss_gen: 1.5748188495635986 (2.325052091053554) | > loss_kl: 2.412534236907959 (2.113796148981367) | > loss_feat: 3.269099235534668 (3.708882978984288) | > loss_mel: 18.54463005065918 (18.031593322753906) | > loss_duration: 1.5436811447143555 (1.5594077280589513) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.344764709472656 (27.7387330191476) | > grad_norm_1: tensor(3311.3777, device='cuda:0') (tensor(6274.1841, device='cuda:0')) | > current_lr_0: 0.00018645463832687594 | > current_lr_1: 0.00018645463832687594 | > step_time: 0.4372 (0.42968484333583284) | > loader_time: 0.0055 (0.006507328578404018)  --> TIME: 2025-02-25 13:59:22 -- STEP: 57/72 -- GLOBAL_STEP: 1040450 | > loss_disc: 2.5853021144866943 (2.6368998435505655) | > loss_disc_real_0: 0.1388746052980423 (0.1376852315376725) | > loss_disc_real_1: 0.21336999535560608 (0.22926703827422962) | > loss_disc_real_2: 0.20098160207271576 (0.22509768762086568) | > loss_disc_real_3: 0.24933119118213654 (0.23239902210863014) | > loss_disc_real_4: 0.23843494057655334 (0.247199668696052) | > loss_disc_real_5: 0.22960500419139862 (0.24483895327961236) | > loss_0: 2.5853021144866943 (2.6368998435505655) | > grad_norm_0: tensor(119.7547, device='cuda:0') (tensor(81.8148, device='cuda:0')) | > loss_gen: 2.119809627532959 (2.175786447106747) | > loss_kl: 2.289315938949585 (2.640528676802652) | > loss_feat: 3.6666030883789062 (3.63562742450781) | > loss_mel: 18.000463485717773 (19.315303802490238) | > loss_duration: 1.6568270921707153 (1.6288988004650986) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.733020782470703 (29.396145000792387) | > grad_norm_1: tensor(6454.6982, device='cuda:0') (tensor(6877.5728, device='cuda:0')) | > current_lr_0: 0.00018645463832687594 | > current_lr_1: 0.00018645463832687594 | > step_time: 0.4562 (0.4467766828704299) | > loader_time: 0.0096 (0.008012838530958745) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866906743654063 (-2.8183762456330044e-05) | > avg_step_time: 0.45786760222743933 (-0.0007392386315574484) | > avg_loss_disc: 2.626334569823575 (-0.06356567396244461) | > avg_loss_disc_real_0: 0.13496446415362223 (-0.01627792034980277) | > avg_loss_disc_real_1: 0.22737496923392927 (+0.0028035185286696684) | > avg_loss_disc_real_2: 0.22277394944513346 (+0.0036036903169792966) | > avg_loss_disc_real_3: 0.23208718673444131 (-0.002303650891277126) | > avg_loss_disc_real_4: 0.2453343217641534 (+0.0010746499182471836) | > avg_loss_disc_real_5: 0.2450850207201192 (-0.002871435922636112) | > avg_loss_0: 2.626334569823575 (-0.06356567396244461) | > avg_grad_norm_0: tensor(83.0711, device='cuda:0') (tensor(-4.3609, device='cuda:0')) | > avg_loss_gen: 2.1648308945373764 (+0.13614142108971095) | > avg_loss_kl: 2.5710936784744263 (+0.41601848938095687) | > avg_loss_feat: 3.5833691173875835 (+0.2167838358543288) | > avg_loss_mel: 18.998701928367087 (+1.5651382929842264) | > avg_loss_duration: 1.6326380649083096 (+0.04155676801439734) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.95063346540424 (+2.375638612559147) | > avg_grad_norm_1: tensor(7006.3965, device='cuda:0') (+tensor(1192.3145, device='cuda:0'))  > EPOCH: 562/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 13:59:31)   --> TIME: 2025-02-25 13:59:49 -- STEP: 35/72 -- GLOBAL_STEP: 1040500 | > loss_disc: 2.665388822555542 (2.7026550361088346) | > loss_disc_real_0: 0.0868765339255333 (0.1785354036305632) | > loss_disc_real_1: 0.2696695923805237 (0.23658016749790736) | > loss_disc_real_2: 0.21704110503196716 (0.22810272489275252) | > loss_disc_real_3: 0.2318042516708374 (0.23511963954993656) | > loss_disc_real_4: 0.2662995457649231 (0.2427082576922008) | > loss_disc_real_5: 0.24956992268562317 (0.2463257644857679) | > loss_0: 2.665388822555542 (2.7026550361088346) | > grad_norm_0: tensor(69.9141, device='cuda:0') (tensor(103.8613, device='cuda:0')) | > loss_gen: 1.8036352396011353 (2.019575657163347) | > loss_kl: 2.4305009841918945 (2.159785168511527) | > loss_feat: 3.2569637298583984 (3.3551182065691267) | > loss_mel: 18.860267639160156 (18.14887706211635) | > loss_duration: 1.6228383779525757 (1.566339087486267) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.974206924438477 (27.249695042201452) | > grad_norm_1: tensor(7627.8213, device='cuda:0') (tensor(5983.5146, device='cuda:0')) | > current_lr_0: 0.00018643133149708508 | > current_lr_1: 0.00018643133149708508 | > step_time: 0.4414 (0.4399206842694964) | > loader_time: 0.0081 (0.007075636727469308) --> EVAL PERFORMANCE | > avg_loader_time: 0.008532903563808389 (-0.00013616387273224229) | > avg_step_time: 0.4578394654770972 (-2.81367503421337e-05) | > avg_loss_disc: 2.709325733319135 (+0.08299116349556002) | > avg_loss_disc_real_0: 0.16290020727565593 (+0.0279357431220337) | > avg_loss_disc_real_1: 0.2345800796444987 (+0.007205110410569426) | > avg_loss_disc_real_2: 0.24483775127101956 (+0.022063801825886104) | > avg_loss_disc_real_3: 0.2357035260385191 (+0.003616339304077787) | > avg_loss_disc_real_4: 0.24650894441235224 (+0.001174622648198842) | > avg_loss_disc_real_5: 0.24726025193509926 (+0.0021752312149800612) | > avg_loss_0: 2.709325733319135 (+0.08299116349556002) | > avg_grad_norm_0: tensor(85.1916, device='cuda:0') (+tensor(2.1205, device='cuda:0')) | > avg_loss_gen: 2.0722222546456557 (-0.09260863989172075) | > avg_loss_kl: 2.499374080711687 (-0.07171959776273917) | > avg_loss_feat: 3.349300035288636 (-0.2340690820989475) | > avg_loss_mel: 19.60688773679062 (+0.6081858084235314) | > avg_loss_duration: 1.6318734414140945 (-0.0007646234942151331) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.159657464900487 (+0.20902399949624595) | > avg_grad_norm_1: tensor(5407.5806, device='cuda:0') (tensor(-1598.8159, device='cuda:0'))  > EPOCH: 563/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:00:08)   --> TIME: 2025-02-25 14:00:16 -- STEP: 13/72 -- GLOBAL_STEP: 1040550 | > loss_disc: 2.4604902267456055 (2.608992411540105) | > loss_disc_real_0: 0.13971784710884094 (0.15476758434222296) | > loss_disc_real_1: 0.17499186098575592 (0.23653552853144133) | > loss_disc_real_2: 0.20162643492221832 (0.2156371886913593) | > loss_disc_real_3: 0.21461687982082367 (0.22097012859124404) | > loss_disc_real_4: 0.2293606847524643 (0.23509560754665962) | > loss_disc_real_5: 0.23724232614040375 (0.23509494501810807) | > loss_0: 2.4604902267456055 (2.608992411540105) | > grad_norm_0: tensor(82.5170, device='cuda:0') (tensor(81.6421, device='cuda:0')) | > loss_gen: 2.212101936340332 (2.040708688589243) | > loss_kl: 2.332914352416992 (2.1682927425091085) | > loss_feat: 3.823455572128296 (3.3789829657627988) | > loss_mel: 18.57421112060547 (18.85700401893029) | > loss_duration: 1.4928544759750366 (1.5723793781720674) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.43553924560547 (28.01736802321214) | > grad_norm_1: tensor(7212.9453, device='cuda:0') (tensor(5531.8643, device='cuda:0')) | > current_lr_0: 0.00018640802758064794 | > current_lr_1: 0.00018640802758064794 | > step_time: 0.4768 (0.4376886807955228) | > loader_time: 0.0067 (0.00653675886300894)  --> TIME: 2025-02-25 14:00:39 -- STEP: 63/72 -- GLOBAL_STEP: 1040600 | > loss_disc: 2.713151693344116 (2.6984803184630377) | > loss_disc_real_0: 0.08403299748897552 (0.1772404572083837) | > loss_disc_real_1: 0.3436969518661499 (0.23693929033146965) | > loss_disc_real_2: 0.20964613556861877 (0.2324566703940195) | > loss_disc_real_3: 0.2121596485376358 (0.23399504569787827) | > loss_disc_real_4: 0.2433767467737198 (0.2438751875408112) | > loss_disc_real_5: 0.21606482565402985 (0.24633385548515926) | > loss_0: 2.713151693344116 (2.6984803184630377) | > grad_norm_0: tensor(126.2744, device='cuda:0') (tensor(79.5510, device='cuda:0')) | > loss_gen: 1.8888386487960815 (2.0031545692019987) | > loss_kl: 2.747945547103882 (2.137932240016876) | > loss_feat: 3.4533135890960693 (3.2093334954882424) | > loss_mel: 19.987045288085938 (18.01628806099059) | > loss_duration: 1.6882554292678833 (1.5953683796383085) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.765398025512695 (26.962076883467418) | > grad_norm_1: tensor(6406.4409, device='cuda:0') (tensor(4485.1392, device='cuda:0')) | > current_lr_0: 0.00018640802758064794 | > current_lr_1: 0.00018640802758064794 | > step_time: 0.4978 (0.4452953300778828) | > loader_time: 0.011 (0.008118224522424123) --> EVAL PERFORMANCE | > avg_loader_time: 0.008502704996458242 (-3.0198567350146407e-05) | > avg_step_time: 0.45117982004729795 (-0.0066596454297992436) | > avg_loss_disc: 2.7001002439310855 (-0.009225489388049457) | > avg_loss_disc_real_0: 0.17397211292679882 (+0.011071905651142883) | > avg_loss_disc_real_1: 0.23970729018181142 (+0.005127210537312715) | > avg_loss_disc_real_2: 0.23190606878676884 (-0.012931682484250723) | > avg_loss_disc_real_3: 0.23555409194717944 (-0.0001494340913396619) | > avg_loss_disc_real_4: 0.24429638557870623 (-0.0022125588336460167) | > avg_loss_disc_real_5: 0.24511161110770535 (-0.0021486408273939106) | > avg_loss_0: 2.7001002439310855 (-0.009225489388049457) | > avg_grad_norm_0: tensor(78.7938, device='cuda:0') (tensor(-6.3978, device='cuda:0')) | > avg_loss_gen: 2.00480932752851 (-0.06741292711714575) | > avg_loss_kl: 2.228700238214411 (-0.2706738424972759) | > avg_loss_feat: 3.1952347453211396 (-0.15406528996749635) | > avg_loss_mel: 18.2849942059584 (-1.3218935308322166) | > avg_loss_duration: 1.6086606928999996 (-0.02321274851409494) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.322399300588692 (-1.8372581643117947) | > avg_grad_norm_1: tensor(4568.6479, device='cuda:0') (tensor(-838.9326, device='cuda:0'))  > EPOCH: 564/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:00:45)   --> TIME: 2025-02-25 14:01:06 -- STEP: 41/72 -- GLOBAL_STEP: 1040650 | > loss_disc: 2.925771713256836 (2.678426562286005) | > loss_disc_real_0: 0.04465766251087189 (0.15442696849747403) | > loss_disc_real_1: 0.22508253157138824 (0.2228070582921912) | > loss_disc_real_2: 0.3859562873840332 (0.23116788522499362) | > loss_disc_real_3: 0.18245314061641693 (0.23003056863459145) | > loss_disc_real_4: 0.25549808144569397 (0.24514387693347) | > loss_disc_real_5: 0.24998734891414642 (0.24761401334913766) | > loss_0: 2.925771713256836 (2.678426562286005) | > grad_norm_0: tensor(121.2459, device='cuda:0') (tensor(84.1979, device='cuda:0')) | > loss_gen: 1.709940791130066 (1.996147798328865) | > loss_kl: 2.026770830154419 (2.3030650092334284) | > loss_feat: 2.994492769241333 (3.2985600378455184) | > loss_mel: 18.367759704589844 (18.916018323200504) | > loss_duration: 1.5959253311157227 (1.590298007174236) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.694889068603516 (28.10408964389708) | > grad_norm_1: tensor(2997.7529, device='cuda:0') (tensor(5425.7104, device='cuda:0')) | > current_lr_0: 0.00018638472657720034 | > current_lr_1: 0.00018638472657720034 | > step_time: 0.4466 (0.4398998923417998) | > loader_time: 0.0089 (0.007257205684010576) --> EVAL PERFORMANCE | > avg_loader_time: 0.00849088816575601 (-1.1816830702232259e-05) | > avg_step_time: 0.4551812393564573 (+0.004001419309159371) | > avg_loss_disc: 2.6950394603568064 (-0.005060783574279082) | > avg_loss_disc_real_0: 0.16397027057451266 (-0.010001842352286155) | > avg_loss_disc_real_1: 0.23078335464840205 (-0.008923935533409366) | > avg_loss_disc_real_2: 0.23665023311762742 (+0.004744164330858586) | > avg_loss_disc_real_3: 0.23245936107467596 (-0.003094730872503476) | > avg_loss_disc_real_4: 0.24347923591103351 (-0.0008171496676727119) | > avg_loss_disc_real_5: 0.24598298861946857 (+0.0008713775117632216) | > avg_loss_0: 2.6950394603568064 (-0.005060783574279082) | > avg_grad_norm_0: tensor(78.1255, device='cuda:0') (tensor(-0.6682, device='cuda:0')) | > avg_loss_gen: 2.04044956892309 (+0.03564024139458022) | > avg_loss_kl: 2.223402387659315 (-0.00529785055509624) | > avg_loss_feat: 3.2781156385448615 (+0.08288089322372194) | > avg_loss_mel: 18.41817278257558 (+0.1331785766171798) | > avg_loss_duration: 1.6076825219140927 (-0.000978170985906912) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.567823221985723 (+0.2454239213970304) | > avg_grad_norm_1: tensor(5060.2637, device='cuda:0') (+tensor(491.6157, device='cuda:0'))  > EPOCH: 565/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:01:22)   --> TIME: 2025-02-25 14:01:33 -- STEP: 19/72 -- GLOBAL_STEP: 1040700 | > loss_disc: 2.545780658721924 (2.685417100002891) | > loss_disc_real_0: 0.19909650087356567 (0.18494605978852824) | > loss_disc_real_1: 0.22690695524215698 (0.21967440608300662) | > loss_disc_real_2: 0.19641008973121643 (0.2236103156679555) | > loss_disc_real_3: 0.2597150504589081 (0.24616698920726776) | > loss_disc_real_4: 0.22001533210277557 (0.24362022625772575) | > loss_disc_real_5: 0.2092140167951584 (0.24294363432808927) | > loss_0: 2.545780658721924 (2.685417100002891) | > grad_norm_0: tensor(23.4822, device='cuda:0') (tensor(81.0616, device='cuda:0')) | > loss_gen: 2.3132832050323486 (2.122709958176864) | > loss_kl: 1.9412040710449219 (1.931846399056284) | > loss_feat: 3.830850124359131 (3.4023015122664604) | > loss_mel: 17.162790298461914 (17.78955519826789) | > loss_duration: 1.5467530488967896 (1.5456392388594777) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.79488182067871 (26.79205282110917) | > grad_norm_1: tensor(7935.0586, device='cuda:0') (tensor(5273.3306, device='cuda:0')) | > current_lr_0: 0.00018636142848637817 | > current_lr_1: 0.00018636142848637817 | > step_time: 0.4376 (0.43444588309840154) | > loader_time: 0.0063 (0.006597732242785002)  --> TIME: 2025-02-25 14:01:57 -- STEP: 69/72 -- GLOBAL_STEP: 1040750 | > loss_disc: 2.8673715591430664 (2.7037072147148242) | > loss_disc_real_0: 0.1704866886138916 (0.16349202912786734) | > loss_disc_real_1: 0.23385758697986603 (0.2219898486914842) | > loss_disc_real_2: 0.2621636986732483 (0.2315389777439228) | > loss_disc_real_3: 0.27566930651664734 (0.24157904779565506) | > loss_disc_real_4: 0.2434699684381485 (0.24661542669586514) | > loss_disc_real_5: 0.24015234410762787 (0.24699882212756347) | > loss_0: 2.8673715591430664 (2.7037072147148242) | > grad_norm_0: tensor(23.1422, device='cuda:0') (tensor(77.8210, device='cuda:0')) | > loss_gen: 1.848168134689331 (2.030733393586201) | > loss_kl: 2.189035177230835 (2.1368172756139785) | > loss_feat: 2.4563729763031006 (3.3151731249214946) | > loss_mel: 17.621816635131836 (18.050043949182484) | > loss_duration: 1.7127771377563477 (1.6062490439069443) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.828170776367188 (27.139017022174336) | > grad_norm_1: tensor(450.4930, device='cuda:0') (tensor(4771.2764, device='cuda:0')) | > current_lr_0: 0.00018636142848637817 | > current_lr_1: 0.00018636142848637817 | > step_time: 0.5215 (0.45495309345964074) | > loader_time: 0.0139 (0.008752162905706873) --> EVAL PERFORMANCE | > avg_loader_time: 0.008814308005319514 (+0.0003234198395635039) | > avg_step_time: 0.45630896595162407 (+0.0011277265951667448) | > avg_loss_disc: 2.706291054336118 (+0.011251593979311636) | > avg_loss_disc_real_0: 0.16629412098669671 (+0.002323850412184053) | > avg_loss_disc_real_1: 0.2220484303756499 (-0.00873492427275216) | > avg_loss_disc_real_2: 0.2320219330804449 (-0.004628300037182537) | > avg_loss_disc_real_3: 0.24167465156232806 (+0.0092152904876521) | > avg_loss_disc_real_4: 0.24644645628794817 (+0.002967220376914659) | > avg_loss_disc_real_5: 0.2470133688668131 (+0.0010303802473445312) | > avg_loss_0: 2.706291054336118 (+0.011251593979311636) | > avg_grad_norm_0: tensor(77.4192, device='cuda:0') (tensor(-0.7063, device='cuda:0')) | > avg_loss_gen: 2.0269966209438492 (-0.013452947979240903) | > avg_loss_kl: 2.1369987104980037 (-0.08640367716131125) | > avg_loss_feat: 3.3016516625041694 (+0.023536023959307872) | > avg_loss_mel: 18.047941785463145 (-0.37023099711243646) | > avg_loss_duration: 1.606493204412326 (-0.0011893175017667623) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.12008221048705 (-0.4477410114986746) | > avg_grad_norm_1: tensor(4677.0146, device='cuda:0') (tensor(-383.2490, device='cuda:0'))  > EPOCH: 566/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:01:59)   --> TIME: 2025-02-25 14:02:22 -- STEP: 47/72 -- GLOBAL_STEP: 1040800 | > loss_disc: 2.6198105812072754 (2.7175175788554737) | > loss_disc_real_0: 0.22607247531414032 (0.1708986350196474) | > loss_disc_real_1: 0.2374301552772522 (0.2264375810293441) | > loss_disc_real_2: 0.21131938695907593 (0.2386411301633145) | > loss_disc_real_3: 0.20981210470199585 (0.23594288147510367) | > loss_disc_real_4: 0.23019595444202423 (0.24596928028350182) | > loss_disc_real_5: 0.23269964754581451 (0.2463191398914824) | > loss_0: 2.6198105812072754 (2.7175175788554737) | > grad_norm_0: tensor(76.3700, device='cuda:0') (tensor(63.0495, device='cuda:0')) | > loss_gen: 2.466625690460205 (1.9743773125587625) | > loss_kl: 2.2353756427764893 (2.1114259882176176) | > loss_feat: 3.3246660232543945 (3.203889542437614) | > loss_mel: 16.673521041870117 (18.010268475147008) | > loss_duration: 1.6076436042785645 (1.5889664203562635) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.307830810546875 (26.88892774379) | > grad_norm_1: tensor(5360.9761, device='cuda:0') (tensor(4140.9648, device='cuda:0')) | > current_lr_0: 0.00018633813330781737 | > current_lr_1: 0.00018633813330781737 | > step_time: 0.4437 (0.4391995440138147) | > loader_time: 0.0088 (0.007617113438058406) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619221163467623 (-0.00019508684185189085) | > avg_step_time: 0.4501905139063445 (-0.006118452045279554) | > avg_loss_disc: 2.7201397989837215 (+0.01384874464760344) | > avg_loss_disc_real_0: 0.1814322934394152 (+0.015138172452718479) | > avg_loss_disc_real_1: 0.22444416393696423 (+0.0023957335613143393) | > avg_loss_disc_real_2: 0.23894784064360067 (+0.006925907563155781) | > avg_loss_disc_real_3: 0.23651016598016444 (-0.005164485582163619) | > avg_loss_disc_real_4: 0.24399160814117377 (-0.0024548481467744077) | > avg_loss_disc_real_5: 0.24623942417158207 (-0.000773944695231038) | > avg_loss_0: 2.7201397989837215 (+0.01384874464760344) | > avg_grad_norm_0: tensor(65.5922, device='cuda:0') (tensor(-11.8270, device='cuda:0')) | > avg_loss_gen: 1.9663853796435073 (-0.0606112413003419) | > avg_loss_kl: 2.114266743122691 (-0.02273196737531258) | > avg_loss_feat: 3.1215465471778114 (-0.18010511532635798) | > avg_loss_mel: 17.729166715917454 (-0.31877506954569057) | > avg_loss_duration: 1.6046980602640502 (-0.0017951441482757513) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.536063422619456 (-0.584018787867592) | > avg_grad_norm_1: tensor(4356.3550, device='cuda:0') (tensor(-320.6597, device='cuda:0'))  > EPOCH: 567/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:02:36)   --> TIME: 2025-02-25 14:02:49 -- STEP: 25/72 -- GLOBAL_STEP: 1040850 | > loss_disc: 2.6731648445129395 (2.6480347824096686) | > loss_disc_real_0: 0.20694267749786377 (0.13785656347870823) | > loss_disc_real_1: 0.30753466486930847 (0.22402537107467652) | > loss_disc_real_2: 0.25411680340766907 (0.23893517911434173) | > loss_disc_real_3: 0.2274652123451233 (0.2312723344564438) | > loss_disc_real_4: 0.2553216218948364 (0.24573850631713867) | > loss_disc_real_5: 0.2514985501766205 (0.24647968173027038) | > loss_0: 2.6731648445129395 (2.6480347824096686) | > grad_norm_0: tensor(59.3800, device='cuda:0') (tensor(75.6940, device='cuda:0')) | > loss_gen: 2.1560943126678467 (2.0792041540145867) | > loss_kl: 2.729299306869507 (2.129219193458557) | > loss_feat: 3.3391366004943848 (3.521621141433716) | > loss_mel: 20.878211975097656 (18.27189842224121) | > loss_duration: 1.7930595874786377 (1.5760578441619877) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.895801544189453 (27.578000793457033) | > grad_norm_1: tensor(8241.7793, device='cuda:0') (tensor(4726.1226, device='cuda:0')) | > current_lr_0: 0.00018631484104115388 | > current_lr_1: 0.00018631484104115388 | > step_time: 0.4351 (0.4346997928619385) | > loader_time: 0.0072 (0.0068854808807373045) --> EVAL PERFORMANCE | > avg_loader_time: 0.008690541898700553 (+7.132073523292981e-05) | > avg_step_time: 0.4569104020024689 (+0.006719888096124393) | > avg_loss_disc: 2.691739498729437 (-0.028400300254284527) | > avg_loss_disc_real_0: 0.1589807757819202 (-0.02245151765749498) | > avg_loss_disc_real_1: 0.22554778674958456 (+0.0011036228126203307) | > avg_loss_disc_real_2: 0.22969359096506953 (-0.00925424967853114) | > avg_loss_disc_real_3: 0.2341161120525548 (-0.002394053927609635) | > avg_loss_disc_real_4: 0.24284029384733927 (-0.001151314293834499) | > avg_loss_disc_real_5: 0.24464148954606393 (-0.0015979346255181326) | > avg_loss_0: 2.691739498729437 (-0.028400300254284527) | > avg_grad_norm_0: tensor(71.7303, device='cuda:0') (+tensor(6.1381, device='cuda:0')) | > avg_loss_gen: 1.9905129671096793 (+0.02412758746617194) | > avg_loss_kl: 2.3268234074955254 (+0.21255666437283427) | > avg_loss_feat: 3.3463297427540097 (+0.2247831955761983) | > avg_loss_mel: 18.628173693804666 (+0.8990069778872112) | > avg_loss_duration: 1.679467817427407 (+0.07476975716335676) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.9713077545166 (+1.4352443318971453) | > avg_grad_norm_1: tensor(4413.4614, device='cuda:0') (+tensor(57.1064, device='cuda:0'))  > EPOCH: 568/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:03:13)   --> TIME: 2025-02-25 14:03:16 -- STEP: 3/72 -- GLOBAL_STEP: 1040900 | > loss_disc: 2.6614508628845215 (2.5582168102264404) | > loss_disc_real_0: 0.2925553023815155 (0.25295745333035785) | > loss_disc_real_1: 0.2702856659889221 (0.2507592439651489) | > loss_disc_real_2: 0.22470176219940186 (0.22964412967363992) | > loss_disc_real_3: 0.22311830520629883 (0.20960657795270285) | > loss_disc_real_4: 0.244660884141922 (0.2330619990825653) | > loss_disc_real_5: 0.2359645813703537 (0.23465260366598764) | > loss_0: 2.6614508628845215 (2.5582168102264404) | > grad_norm_0: tensor(124.3697, device='cuda:0') (tensor(113.0308, device='cuda:0')) | > loss_gen: 2.250833034515381 (2.391395409901937) | > loss_kl: 2.1289680004119873 (2.2055994669596353) | > loss_feat: 3.1255040168762207 (3.5542333126068115) | > loss_mel: 17.93660545349121 (18.779115676879883) | > loss_duration: 1.6095170974731445 (1.5971660614013672) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.051429748535156 (28.52751096089681) | > grad_norm_1: tensor(4991.4126, device='cuda:0') (tensor(5718.4053, device='cuda:0')) | > current_lr_0: 0.00018629155168602374 | > current_lr_1: 0.00018629155168602374 | > step_time: 0.423 (0.42723361651102704) | > loader_time: 0.007 (0.007192134857177734)  --> TIME: 2025-02-25 14:03:39 -- STEP: 53/72 -- GLOBAL_STEP: 1040950 | > loss_disc: 2.5420427322387695 (2.6699047673423335) | > loss_disc_real_0: 0.11496414244174957 (0.1453307221801775) | > loss_disc_real_1: 0.22943894565105438 (0.23505438161346148) | > loss_disc_real_2: 0.2511591911315918 (0.23885943192356038) | > loss_disc_real_3: 0.24221058189868927 (0.2376469888214795) | > loss_disc_real_4: 0.2690453827381134 (0.24526527214725063) | > loss_disc_real_5: 0.26041269302368164 (0.24476072810730845) | > loss_0: 2.5420427322387695 (2.6699047673423335) | > grad_norm_0: tensor(14.9463, device='cuda:0') (tensor(85.1773, device='cuda:0')) | > loss_gen: 2.013317584991455 (2.0040472206079727) | > loss_kl: 2.548192262649536 (2.3349823771782634) | > loss_feat: 3.9162042140960693 (3.331843542602827) | > loss_mel: 20.17335319519043 (19.33591684305442) | > loss_duration: 1.6509475708007812 (1.5959695802544647) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.302013397216797 (28.602759415248656) | > grad_norm_1: tensor(10464.7988, device='cuda:0') (tensor(5858.2354, device='cuda:0')) | > current_lr_0: 0.00018629155168602374 | > current_lr_1: 0.00018629155168602374 | > step_time: 0.449 (0.4441238619246573) | > loader_time: 0.0089 (0.0077720723062191365) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534176248899647 (-0.00015636564980090611) | > avg_step_time: 0.4569321719693466 (+2.1769966877693836e-05) | > avg_loss_disc: 2.669678416050656 (-0.02206108267878104) | > avg_loss_disc_real_0: 0.13467893450402876 (-0.02430184127789145) | > avg_loss_disc_real_1: 0.23263310462656156 (+0.007085317876976999) | > avg_loss_disc_real_2: 0.23915788762166468 (+0.00946429665659515) | > avg_loss_disc_real_3: 0.2367543057656624 (+0.002638193713107584) | > avg_loss_disc_real_4: 0.24316278456802098 (+0.00032249072068171736) | > avg_loss_disc_real_5: 0.2451346281548621 (+0.0004931386087981549) | > avg_loss_0: 2.669678416050656 (-0.02206108267878104) | > avg_grad_norm_0: tensor(85.9710, device='cuda:0') (+tensor(14.2407, device='cuda:0')) | > avg_loss_gen: 1.9830069155760206 (-0.007506051533658686) | > avg_loss_kl: 2.3471299460236446 (+0.020306538528119145) | > avg_loss_feat: 3.269142083718743 (-0.0771876590352667) | > avg_loss_mel: 19.10444995718942 (+0.4762762633847544) | > avg_loss_duration: 1.6096740809964463 (-0.06979373643096065) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.313402928097144 (+0.3420951735805424) | > avg_grad_norm_1: tensor(6012.8286, device='cuda:0') (+tensor(1599.3672, device='cuda:0'))  > EPOCH: 569/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:03:50)   --> TIME: 2025-02-25 14:04:06 -- STEP: 31/72 -- GLOBAL_STEP: 1041000 | > loss_disc: 2.6016011238098145 (2.557998049643732) | > loss_disc_real_0: 0.13891386985778809 (0.11708835728706853) | > loss_disc_real_1: 0.24160446226596832 (0.21562370369511266) | > loss_disc_real_2: 0.24934737384319305 (0.22424860779316194) | > loss_disc_real_3: 0.27835679054260254 (0.23297461771195935) | > loss_disc_real_4: 0.2541278600692749 (0.24040184097905312) | > loss_disc_real_5: 0.24908286333084106 (0.2446206433157767) | > loss_0: 2.6016011238098145 (2.557998049643732) | > grad_norm_0: tensor(182.4703, device='cuda:0') (tensor(99.2625, device='cuda:0')) | > loss_gen: 2.0270607471466064 (2.0581135903635337) | > loss_kl: 2.1362802982330322 (2.028761752190129) | > loss_feat: 3.1982970237731934 (3.498714885404033) | > loss_mel: 17.95449447631836 (17.63133301273469) | > loss_duration: 1.603402018547058 (1.5558749552695983) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.91953468322754 (26.77279835362588) | > grad_norm_1: tensor(8069.6187, device='cuda:0') (tensor(8062.6973, device='cuda:0')) | > current_lr_0: 0.00018626826524206297 | > current_lr_1: 0.00018626826524206297 | > step_time: 0.4408 (0.43955640639028243) | > loader_time: 0.0084 (0.007315035789243637) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710421306986205 (+0.00017624505808655835) | > avg_step_time: 0.45696665199709613 (+3.44800277495283e-05) | > avg_loss_disc: 2.6039268097407375 (-0.06575160630991839) | > avg_loss_disc_real_0: 0.11384934575205118 (-0.020829588751977587) | > avg_loss_disc_real_1: 0.2192573759337546 (-0.013375728692806954) | > avg_loss_disc_real_2: 0.23768597421511797 (-0.0014719134065467043) | > avg_loss_disc_real_3: 0.23617676152309902 (-0.0005775442425633714) | > avg_loss_disc_real_4: 0.2429672250865211 (-0.00019555948149987956) | > avg_loss_disc_real_5: 0.24668249991578114 (+0.0015478717609190562) | > avg_loss_0: 2.6039268097407375 (-0.06575160630991839) | > avg_grad_norm_0: tensor(111.1264, device='cuda:0') (+tensor(25.1553, device='cuda:0')) | > avg_loss_gen: 2.035964408390959 (+0.052957492814938556) | > avg_loss_kl: 2.052173532230754 (-0.2949564137928906) | > avg_loss_feat: 3.4650442700990487 (+0.19590218638030565) | > avg_loss_mel: 17.492370753221113 (-1.6120792039683067) | > avg_loss_duration: 1.590446710586548 (-0.019227370409898192) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.63599984074982 (-1.6774030873473222) | > avg_grad_norm_1: tensor(8118.9595, device='cuda:0') (+tensor(2106.1309, device='cuda:0'))  > EPOCH: 570/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:04:27)   --> TIME: 2025-02-25 14:04:33 -- STEP: 9/72 -- GLOBAL_STEP: 1041050 | > loss_disc: 2.612441062927246 (2.6056607829199896) | > loss_disc_real_0: 0.08729815483093262 (0.1334573949376742) | > loss_disc_real_1: 0.18722248077392578 (0.2325939734776815) | > loss_disc_real_2: 0.23772981762886047 (0.24049780600600773) | > loss_disc_real_3: 0.21644090116024017 (0.24105238086647457) | > loss_disc_real_4: 0.2361377626657486 (0.2346977624628279) | > loss_disc_real_5: 0.24535445868968964 (0.23003844751252067) | > loss_0: 2.612441062927246 (2.6056607829199896) | > grad_norm_0: tensor(89.1061, device='cuda:0') (tensor(104.2675, device='cuda:0')) | > loss_gen: 1.8500484228134155 (2.0484370125664606) | > loss_kl: 1.8666810989379883 (1.9495780732896593) | > loss_feat: 3.638125419616699 (3.44297374619378) | > loss_mel: 18.20502471923828 (17.17020098368327) | > loss_duration: 1.5455174446105957 (1.561756796307034) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.105396270751953 (26.17294650607639) | > grad_norm_1: tensor(6646.6509, device='cuda:0') (tensor(8177.3472, device='cuda:0')) | > current_lr_0: 0.0001862449817089077 | > current_lr_1: 0.0001862449817089077 | > step_time: 0.4252 (0.43310154808892143) | > loader_time: 0.0056 (0.006790055168999566)  --> TIME: 2025-02-25 14:04:57 -- STEP: 59/72 -- GLOBAL_STEP: 1041100 | > loss_disc: 2.7096457481384277 (2.6038121328515524) | > loss_disc_real_0: 0.0660993903875351 (0.12411014489450695) | > loss_disc_real_1: 0.14525426924228668 (0.21501573768712706) | > loss_disc_real_2: 0.2811453342437744 (0.22636976585549823) | > loss_disc_real_3: 0.24170124530792236 (0.23281457489829951) | > loss_disc_real_4: 0.20970956981182098 (0.237954346557795) | > loss_disc_real_5: 0.21864178776741028 (0.2410713767601272) | > loss_0: 2.7096457481384277 (2.6038121328515524) | > grad_norm_0: tensor(176.9343, device='cuda:0') (tensor(118.7447, device='cuda:0')) | > loss_gen: 1.8420499563217163 (2.017968832436254) | > loss_kl: 1.9306222200393677 (1.965990060466831) | > loss_feat: 3.7333319187164307 (3.487519147032398) | > loss_mel: 16.4317626953125 (17.1707043728586) | > loss_duration: 1.6118440628051758 (1.576189790741872) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.549610137939453 (26.218372183330988) | > grad_norm_1: tensor(8741.8926, device='cuda:0') (tensor(8081.3125, device='cuda:0')) | > current_lr_0: 0.0001862449817089077 | > current_lr_1: 0.0001862449817089077 | > step_time: 0.4649 (0.4474616010310286) | > loader_time: 0.01 (0.008153353707265044) --> EVAL PERFORMANCE | > avg_loader_time: 0.008709887383689344 (-5.339232968613189e-07) | > avg_step_time: 0.4569739959609341 (+7.3439638379468875e-06) | > avg_loss_disc: 2.6124013108266917 (+0.008474501085954156) | > avg_loss_disc_real_0: 0.12753958905666643 (+0.013690243304615254) | > avg_loss_disc_real_1: 0.21914436258900333 (-0.00011301334475127822) | > avg_loss_disc_real_2: 0.22861384463981843 (-0.009072129575299542) | > avg_loss_disc_real_3: 0.2339496799338032 (-0.0022270815892958262) | > avg_loss_disc_real_4: 0.23734045490412645 (-0.005626770182394658) | > avg_loss_disc_real_5: 0.24044123019131136 (-0.0062412697244697835) | > avg_loss_0: 2.6124013108266917 (+0.008474501085954156) | > avg_grad_norm_0: tensor(120.1824, device='cuda:0') (+tensor(9.0561, device='cuda:0')) | > avg_loss_gen: 2.0252694244116123 (-0.010694983979346873) | > avg_loss_kl: 1.990541248254373 (-0.061632283976380986) | > avg_loss_feat: 3.4599099595781784 (-0.005134310520870322) | > avg_loss_mel: 17.22955776268328 (-0.2628129905378316) | > avg_loss_duration: 1.5854434295439384 (-0.00500328104260972) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.290721785854284 (-0.3452780548955374) | > avg_grad_norm_1: tensor(8047.9233, device='cuda:0') (tensor(-71.0361, device='cuda:0'))  > EPOCH: 571/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:05:04)   --> TIME: 2025-02-25 14:05:23 -- STEP: 37/72 -- GLOBAL_STEP: 1041150 | > loss_disc: 2.5718233585357666 (2.77227166536692) | > loss_disc_real_0: 0.08058059215545654 (0.1661257756924307) | > loss_disc_real_1: 0.1627928465604782 (0.2551906549850026) | > loss_disc_real_2: 0.20917104184627533 (0.23435667721000877) | > loss_disc_real_3: 0.20245887339115143 (0.2302245980984456) | > loss_disc_real_4: 0.23393607139587402 (0.2485366342840968) | > loss_disc_real_5: 0.2284298837184906 (0.2490757876956785) | > loss_0: 2.5718233585357666 (2.77227166536692) | > grad_norm_0: tensor(105.9788, device='cuda:0') (tensor(109.8425, device='cuda:0')) | > loss_gen: 1.8714476823806763 (1.9637229249284074) | > loss_kl: 2.4251716136932373 (2.156723399420043) | > loss_feat: 3.4055287837982178 (3.3149288860527246) | > loss_mel: 18.557037353515625 (17.920550578349346) | > loss_duration: 1.6099029779434204 (1.5655758219796259) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.869089126586914 (26.921501623617637) | > grad_norm_1: tensor(6248.2852, device='cuda:0') (tensor(5535.9243, device='cuda:0')) | > current_lr_0: 0.00018622170108619407 | > current_lr_1: 0.00018622170108619407 | > step_time: 0.4678 (0.4384628437660836) | > loader_time: 0.0079 (0.007140146719442832) --> EVAL PERFORMANCE | > avg_loader_time: 0.008490021799651668 (-0.0002198655840376755) | > avg_step_time: 0.45411448747339384 (-0.002859508487540241) | > avg_loss_disc: 2.784192545313231 (+0.17179123448653932) | > avg_loss_disc_real_0: 0.19744217999174563 (+0.0699025909350792) | > avg_loss_disc_real_1: 0.24144133180379873 (+0.022296969214795404) | > avg_loss_disc_real_2: 0.23728823535878893 (+0.008674390718970498) | > avg_loss_disc_real_3: 0.23720669368623007 (+0.003257013752426874) | > avg_loss_disc_real_4: 0.24666081515836044 (+0.009320360254233995) | > avg_loss_disc_real_5: 0.24392299815802507 (+0.003481767966713706) | > avg_loss_0: 2.784192545313231 (+0.17179123448653932) | > avg_grad_norm_0: tensor(91.8184, device='cuda:0') (tensor(-28.3641, device='cuda:0')) | > avg_loss_gen: 1.9632163232480977 (-0.06205310116351459) | > avg_loss_kl: 2.2366146587989704 (+0.24607341054459742) | > avg_loss_feat: 3.1203962547678343 (-0.33951370481034404) | > avg_loss_mel: 18.113059755782007 (+0.8835019930987258) | > avg_loss_duration: 1.5990193931149765 (+0.013575963571038185) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.032306456230057 (+0.7415846703757722) | > avg_grad_norm_1: tensor(4720.4990, device='cuda:0') (tensor(-3327.4243, device='cuda:0'))  > EPOCH: 572/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:05:42)   --> TIME: 2025-02-25 14:05:50 -- STEP: 15/72 -- GLOBAL_STEP: 1041200 | > loss_disc: 2.7057199478149414 (2.669850746790568) | > loss_disc_real_0: 0.19310547411441803 (0.18892850478490195) | > loss_disc_real_1: 0.24984689056873322 (0.22940617899099985) | > loss_disc_real_2: 0.2377278208732605 (0.22860058148701987) | > loss_disc_real_3: 0.2648887038230896 (0.23692739605903626) | > loss_disc_real_4: 0.2364567369222641 (0.2396407425403595) | > loss_disc_real_5: 0.21924445033073425 (0.24060461421807608) | > loss_0: 2.7057199478149414 (2.669850746790568) | > grad_norm_0: tensor(69.6123, device='cuda:0') (tensor(74.8587, device='cuda:0')) | > loss_gen: 2.0485246181488037 (1.9501548131306967) | > loss_kl: 2.198591709136963 (2.05112890402476) | > loss_feat: 3.581979513168335 (3.033712100982666) | > loss_mel: 18.24970054626465 (17.59691098531088) | > loss_duration: 1.5293009281158447 (1.5460721810658773) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.608097076416016 (26.17797876993815) | > grad_norm_1: tensor(6779.6577, device='cuda:0') (tensor(3891.0361, device='cuda:0')) | > current_lr_0: 0.0001861984233735583 | > current_lr_1: 0.0001861984233735583 | > step_time: 0.427 (0.4320167382558187) | > loader_time: 0.006 (0.007105477650960286)  --> TIME: 2025-02-25 14:06:14 -- STEP: 65/72 -- GLOBAL_STEP: 1041250 | > loss_disc: 2.563807487487793 (2.6536310562720673) | > loss_disc_real_0: 0.18247950077056885 (0.1621094054900683) | > loss_disc_real_1: 0.183196023106575 (0.2216855615377426) | > loss_disc_real_2: 0.18858005106449127 (0.23284912476172814) | > loss_disc_real_3: 0.22265370190143585 (0.23391398168527164) | > loss_disc_real_4: 0.2510576844215393 (0.24372712465432975) | > loss_disc_real_5: 0.24247024953365326 (0.2462315545632289) | > loss_0: 2.563807487487793 (2.6536310562720673) | > grad_norm_0: tensor(76.3109, device='cuda:0') (tensor(90.8945, device='cuda:0')) | > loss_gen: 2.082801342010498 (1.9601672374285184) | > loss_kl: 2.359232187271118 (2.056838277670053) | > loss_feat: 3.427676200866699 (3.1198395545666036) | > loss_mel: 17.943161010742188 (17.279587349524864) | > loss_duration: 1.6288328170776367 (1.5795673241982093) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.44170379638672 (25.995999615009016) | > grad_norm_1: tensor(7334.9180, device='cuda:0') (tensor(5184.1416, device='cuda:0')) | > current_lr_0: 0.0001861984233735583 | > current_lr_1: 0.0001861984233735583 | > step_time: 0.5302 (0.45056603871859036) | > loader_time: 0.0118 (0.008525063441349911) --> EVAL PERFORMANCE | > avg_loader_time: 0.008775912540059695 (+0.000285890740408027) | > avg_step_time: 0.4573970445444886 (+0.0032825570710947605) | > avg_loss_disc: 2.658086299896241 (-0.1261062454169899) | > avg_loss_disc_real_0: 0.15899565805431828 (-0.03844652193742734) | > avg_loss_disc_real_1: 0.2224172832260669 (-0.019024048577731822) | > avg_loss_disc_real_2: 0.23582576446130243 (-0.0014624708974864953) | > avg_loss_disc_real_3: 0.23459823282671646 (-0.002608460859513606) | > avg_loss_disc_real_4: 0.2431441478326287 (-0.0035166673257317482) | > avg_loss_disc_real_5: 0.24569012161711573 (+0.0017671234590906637) | > avg_loss_0: 2.658086299896241 (-0.1261062454169899) | > avg_grad_norm_0: tensor(92.6329, device='cuda:0') (+tensor(0.8145, device='cuda:0')) | > avg_loss_gen: 1.9576465327974777 (-0.005569790450620005) | > avg_loss_kl: 2.069007843313083 (-0.1676068154858874) | > avg_loss_feat: 3.1170045322095845 (-0.003391722558249821) | > avg_loss_mel: 17.28303003982759 (-0.8300297159544172) | > avg_loss_duration: 1.5845454864098991 (-0.014473906705077422) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.011234283447266 (-1.021072172782791) | > avg_grad_norm_1: tensor(5266.7993, device='cuda:0') (+tensor(546.3003, device='cuda:0'))  > EPOCH: 573/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:06:19)   --> TIME: 2025-02-25 14:06:40 -- STEP: 43/72 -- GLOBAL_STEP: 1041300 | > loss_disc: 2.6944031715393066 (2.6098719530327377) | > loss_disc_real_0: 0.11835076659917831 (0.14462837126365927) | > loss_disc_real_1: 0.2149955928325653 (0.21562524139881134) | > loss_disc_real_2: 0.26304200291633606 (0.22631816496682722) | > loss_disc_real_3: 0.22688394784927368 (0.23415024959763817) | > loss_disc_real_4: 0.25462350249290466 (0.24219245792821395) | > loss_disc_real_5: 0.2500087022781372 (0.24407340031723643) | > loss_0: 2.6944031715393066 (2.6098719530327377) | > grad_norm_0: tensor(132.7375, device='cuda:0') (tensor(115.6415, device='cuda:0')) | > loss_gen: 1.926467776298523 (2.0315963029861446) | > loss_kl: 1.973716139793396 (1.9914716814839564) | > loss_feat: 3.1283113956451416 (3.430689063183097) | > loss_mel: 15.833660125732422 (17.165278833965928) | > loss_duration: 1.567528486251831 (1.552674645601317) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.429683685302734 (26.171710258306458) | > grad_norm_1: tensor(6681.2119, device='cuda:0') (tensor(7142.2007, device='cuda:0')) | > current_lr_0: 0.0001861751485706366 | > current_lr_1: 0.0001861751485706366 | > step_time: 0.4677 (0.44371685316396314) | > loader_time: 0.0095 (0.007531082907388377) --> EVAL PERFORMANCE | > avg_loader_time: 0.008672385148599113 (-0.00010352739146058179) | > avg_step_time: 0.4574740537455384 (+7.700920104980469e-05) | > avg_loss_disc: 2.6345683991069526 (-0.023517900789288504) | > avg_loss_disc_real_0: 0.1405564466307701 (-0.01843921142354818) | > avg_loss_disc_real_1: 0.2190590758978481 (-0.003358207328218815) | > avg_loss_disc_real_2: 0.22704848982918424 (-0.008777274632118193) | > avg_loss_disc_real_3: 0.2393294866655914 (+0.0047312538388749326) | > avg_loss_disc_real_4: 0.24448885401369821 (+0.0013447061810695216) | > avg_loss_disc_real_5: 0.24645646147325007 (+0.0007663398561343349) | > avg_loss_0: 2.6345683991069526 (-0.023517900789288504) | > avg_grad_norm_0: tensor(115.1755, device='cuda:0') (+tensor(22.5426, device='cuda:0')) | > avg_loss_gen: 2.006238079406846 (+0.048591546609368486) | > avg_loss_kl: 2.018781581395109 (-0.05022626191797386) | > avg_loss_feat: 3.342780717661683 (+0.22577618545209832) | > avg_loss_mel: 17.04970197274652 (-0.23332806708107157) | > avg_loss_duration: 1.5779096744429895 (-0.006635811966909655) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.995411886295805 (-0.015822397151460166) | > avg_grad_norm_1: tensor(6918.7495, device='cuda:0') (+tensor(1651.9502, device='cuda:0'))  > EPOCH: 574/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:06:56)   --> TIME: 2025-02-25 14:07:07 -- STEP: 21/72 -- GLOBAL_STEP: 1041350 | > loss_disc: 2.7013769149780273 (2.8183342048100064) | > loss_disc_real_0: 0.16587047278881073 (0.16830312247787202) | > loss_disc_real_1: 0.23373323678970337 (0.22043146122069585) | > loss_disc_real_2: 0.4952666759490967 (0.32013738013449167) | > loss_disc_real_3: 0.2192070484161377 (0.2357058780533927) | > loss_disc_real_4: 0.23630784451961517 (0.2500124779485521) | > loss_disc_real_5: 0.2513563632965088 (0.24790324696472713) | > loss_0: 2.7013769149780273 (2.8183342048100064) | > grad_norm_0: tensor(40.5688, device='cuda:0') (tensor(118.3244, device='cuda:0')) | > loss_gen: 2.083585500717163 (2.0524456955137707) | > loss_kl: 1.9089574813842773 (2.1401557354699996) | > loss_feat: 3.1142303943634033 (3.411343324752081) | > loss_mel: 16.94392204284668 (17.93905044737316) | > loss_duration: 1.5575958490371704 (1.5691415752683366) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.608291625976562 (27.11213629586356) | > grad_norm_1: tensor(4921.5493, device='cuda:0') (tensor(6062.3169, device='cuda:0')) | > current_lr_0: 0.00018615187667706527 | > current_lr_1: 0.00018615187667706527 | > step_time: 0.4365 (0.43834662437438965) | > loader_time: 0.0063 (0.007066068195161365)  --> TIME: 2025-02-25 14:07:31 -- STEP: 71/72 -- GLOBAL_STEP: 1041400 | > loss_disc: 2.4611456394195557 (2.7140029249056963) | > loss_disc_real_0: 0.13141348958015442 (0.14559114010820934) | > loss_disc_real_1: 0.2320333570241928 (0.22044568708245182) | > loss_disc_real_2: 0.22300297021865845 (0.2574517661836785) | > loss_disc_real_3: 0.20940141379833221 (0.23591680488955807) | > loss_disc_real_4: 0.20100238919258118 (0.2424088930579978) | > loss_disc_real_5: 0.2119206339120865 (0.24293088157412032) | > loss_0: 2.4611456394195557 (2.7140029249056963) | > grad_norm_0: tensor(95.1259, device='cuda:0') (tensor(119.6157, device='cuda:0')) | > loss_gen: 1.998915195465088 (1.9778698111923643) | > loss_kl: 2.065077066421509 (2.116201622385375) | > loss_feat: 3.744607925415039 (3.3036591133601227) | > loss_mel: 16.80581283569336 (17.42132502542414) | > loss_duration: 1.5978806018829346 (1.5926064974825147) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.21229362487793 (26.41166192712918) | > grad_norm_1: tensor(8711.5947, device='cuda:0') (tensor(6435.7759, device='cuda:0')) | > current_lr_0: 0.00018615187667706527 | > current_lr_1: 0.00018615187667706527 | > step_time: 0.4673 (0.45709865193971444) | > loader_time: 0.008 (0.008997322807849295) --> EVAL PERFORMANCE | > avg_loader_time: 0.008997322807849295 (+0.0003249376592501813) | > avg_step_time: 0.45709865193971444 (-0.00037540180582396054) | > avg_loss_disc: 2.7140029249056963 (+0.07943452579874366) | > avg_loss_disc_real_0: 0.14559114010820934 (+0.005034693477439234) | > avg_loss_disc_real_1: 0.22044568708245182 (+0.001386611184603731) | > avg_loss_disc_real_2: 0.2574517661836785 (+0.030403276354494246) | > avg_loss_disc_real_3: 0.23591680488955807 (-0.0034126817760333217) | > avg_loss_disc_real_4: 0.2424088930579978 (-0.0020799609557004273) | > avg_loss_disc_real_5: 0.24293088157412032 (-0.003525579899129744) | > avg_loss_0: 2.7140029249056963 (+0.07943452579874366) | > avg_grad_norm_0: tensor(119.6157, device='cuda:0') (+tensor(4.4402, device='cuda:0')) | > avg_loss_gen: 1.9778698111923643 (-0.028368268214481906) | > avg_loss_kl: 2.116201622385375 (+0.09742004099026591) | > avg_loss_feat: 3.3036591133601227 (-0.03912160430156009) | > avg_loss_mel: 17.42132502542414 (+0.3716230526776201) | > avg_loss_duration: 1.5926064974825147 (+0.014696823039525242) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.41166192712918 (+0.4162500408333756) | > avg_grad_norm_1: tensor(6435.7759, device='cuda:0') (tensor(-482.9736, device='cuda:0'))  > EPOCH: 575/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:07:33)   --> TIME: 2025-02-25 14:07:58 -- STEP: 49/72 -- GLOBAL_STEP: 1041450 | > loss_disc: 2.691480875015259 (2.623279887802746) | > loss_disc_real_0: 0.1159839779138565 (0.1357661476548837) | > loss_disc_real_1: 0.21975310146808624 (0.22274443780889316) | > loss_disc_real_2: 0.2424929440021515 (0.22636731729215506) | > loss_disc_real_3: 0.23427428305149078 (0.23260633647441864) | > loss_disc_real_4: 0.24521256983280182 (0.2436681730406625) | > loss_disc_real_5: 0.2368609756231308 (0.24494156515111729) | > loss_0: 2.691480875015259 (2.623279887802746) | > grad_norm_0: tensor(120.2964, device='cuda:0') (tensor(125.3846, device='cuda:0')) | > loss_gen: 1.914361834526062 (2.015896334940074) | > loss_kl: 2.1360344886779785 (1.9898339874890385) | > loss_feat: 3.6613073348999023 (3.3740366040443885) | > loss_mel: 17.173185348510742 (16.951086238938935) | > loss_duration: 1.6370770931243896 (1.5588257264117806) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.5219669342041 (25.889678838301677) | > grad_norm_1: tensor(8247.8613, device='cuda:0') (tensor(7643.0166, device='cuda:0')) | > current_lr_0: 0.00018612860769248062 | > current_lr_1: 0.00018612860769248062 | > step_time: 0.4427 (0.4546480130176155) | > loader_time: 0.0098 (0.00764507176924725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579062743925713 (-0.00041826006392358187) | > avg_step_time: 0.46464126210817147 (+0.007542610168457031) | > avg_loss_disc: 2.636769707773772 (-0.0772332171319241) | > avg_loss_disc_real_0: 0.13497340810340894 (-0.010617732004800401) | > avg_loss_disc_real_1: 0.22161544754471577 (+0.0011697604622639413) | > avg_loss_disc_real_2: 0.22859957184590085 (-0.028852194337777637) | > avg_loss_disc_real_3: 0.23552342346856292 (-0.0003933814209951536) | > avg_loss_disc_real_4: 0.24444794990647006 (+0.002039056848472276) | > avg_loss_disc_real_5: 0.2450764454166654 (+0.0021455638425450785) | > avg_loss_0: 2.636769707773772 (-0.0772332171319241) | > avg_grad_norm_0: tensor(124.5697, device='cuda:0') (+tensor(4.9540, device='cuda:0')) | > avg_loss_gen: 2.0098781317052716 (+0.03200832051290736) | > avg_loss_kl: 2.0325651840424874 (-0.08363643834288759) | > avg_loss_feat: 3.385623357665371 (+0.08196424430524818) | > avg_loss_mel: 16.9620753274837 (-0.45924969794043946) | > avg_loss_duration: 1.577305985168672 (-0.015300512313842773) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.967448086805746 (-0.44421384032343525) | > avg_grad_norm_1: tensor(7532.6768, device='cuda:0') (+tensor(1096.9009, device='cuda:0'))  > EPOCH: 576/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:08:10)   --> TIME: 2025-02-25 14:08:24 -- STEP: 27/72 -- GLOBAL_STEP: 1041500 | > loss_disc: 2.6666781902313232 (2.7490453631789595) | > loss_disc_real_0: 0.16227784752845764 (0.151493513473758) | > loss_disc_real_1: 0.21400609612464905 (0.2155998674807725) | > loss_disc_real_2: 0.2209222912788391 (0.228125246586623) | > loss_disc_real_3: 0.23580674827098846 (0.23695443405045402) | > loss_disc_real_4: 0.24928945302963257 (0.2329844324677079) | > loss_disc_real_5: 0.25062960386276245 (0.23707968438113178) | > loss_0: 2.6666781902313232 (2.7490453631789595) | > grad_norm_0: tensor(175.9749, device='cuda:0') (tensor(122.7409, device='cuda:0')) | > loss_gen: 1.9976651668548584 (1.9211201800240412) | > loss_kl: 2.056514263153076 (1.991424282391866) | > loss_feat: 3.3169639110565186 (3.3122073191183583) | > loss_mel: 15.812105178833008 (17.041002520808462) | > loss_duration: 1.5349948406219482 (1.5285792748133342) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.718242645263672 (25.79433356391059) | > grad_norm_1: tensor(8120.7231, device='cuda:0') (tensor(6559.4434, device='cuda:0')) | > current_lr_0: 0.00018610534161651906 | > current_lr_1: 0.00018610534161651906 | > step_time: 0.4375 (0.43711465376394765) | > loader_time: 0.0072 (0.006828908567075376) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522849687388244 (-5.621305653746855e-05) | > avg_step_time: 0.4551812628625144 (-0.009459999245657047) | > avg_loss_disc: 2.75689710026056 (+0.12012739248678761) | > avg_loss_disc_real_0: 0.14939057942427383 (+0.014417171320864897) | > avg_loss_disc_real_1: 0.23528392046270236 (+0.01366847291798659) | > avg_loss_disc_real_2: 0.240021073902157 (+0.011421502056256139) | > avg_loss_disc_real_3: 0.23712773726020062 (+0.001604313791637696) | > avg_loss_disc_real_4: 0.23767531631698072 (-0.006772633589489346) | > avg_loss_disc_real_5: 0.24178992841445224 (-0.003286517002213163) | > avg_loss_0: 2.75689710026056 (+0.12012739248678761) | > avg_grad_norm_0: tensor(125.0570, device='cuda:0') (+tensor(0.4873, device='cuda:0')) | > avg_loss_gen: 1.9339378921078965 (-0.07594023959737517) | > avg_loss_kl: 2.1104310583060895 (+0.07786587426360203) | > avg_loss_feat: 3.234783837493037 (-0.15083952017233404) | > avg_loss_mel: 17.14201130665523 (+0.1799359791715318) | > avg_loss_duration: 1.5767701226221005 (-0.0005358625465714528) | > avg_amp_scaler: 16.225352112676056 (+0.22535211267605604) | > avg_loss_1: 25.997934207110337 (+0.03048612030459097) | > avg_grad_norm_1: tensor(6156.4023, device='cuda:0') (tensor(-1376.2744, device='cuda:0'))  > EPOCH: 577/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:08:47)   --> TIME: 2025-02-25 14:08:52 -- STEP: 5/72 -- GLOBAL_STEP: 1041550 | > loss_disc: 2.9023804664611816 (2.7596013069152834) | > loss_disc_real_0: 0.06435755640268326 (0.10704362392425537) | > loss_disc_real_1: 0.17365987598896027 (0.1882103979587555) | > loss_disc_real_2: 0.21459008753299713 (0.24145095348358153) | > loss_disc_real_3: 0.25339949131011963 (0.2315530449151993) | > loss_disc_real_4: 0.23664924502372742 (0.24005118310451506) | > loss_disc_real_5: 0.2630491554737091 (0.24419143199920654) | > loss_0: 2.9023804664611816 (2.7596013069152834) | > grad_norm_0: tensor(170.9927, device='cuda:0') (tensor(120.1431, device='cuda:0')) | > loss_gen: 1.5767526626586914 (1.739115834236145) | > loss_kl: 2.377701997756958 (2.4259835720062255) | > loss_feat: 2.822993040084839 (3.2336297035217285) | > loss_mel: 19.234094619750977 (20.370323944091798) | > loss_duration: 1.567704677581787 (1.624332809448242) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.579248428344727 (29.393386840820312) | > grad_norm_1: tensor(5641.0283, device='cuda:0') (tensor(6541.8335, device='cuda:0')) | > current_lr_0: 0.00018608207844881698 | > current_lr_1: 0.00018608207844881698 | > step_time: 0.4247 (0.4402920722961426) | > loader_time: 0.0063 (0.007224893569946289)  --> TIME: 2025-02-25 14:09:15 -- STEP: 55/72 -- GLOBAL_STEP: 1041600 | > loss_disc: 2.508249521255493 (2.6381573070179334) | > loss_disc_real_0: 0.19149154424667358 (0.13908562396060337) | > loss_disc_real_1: 0.2342667430639267 (0.2202793286605315) | > loss_disc_real_2: 0.22298428416252136 (0.231355067003857) | > loss_disc_real_3: 0.21068193018436432 (0.23384126262231306) | > loss_disc_real_4: 0.23773612082004547 (0.24157773798162288) | > loss_disc_real_5: 0.25297290086746216 (0.2437945940277793) | > loss_0: 2.508249521255493 (2.6381573070179334) | > grad_norm_0: tensor(92.8684, device='cuda:0') (tensor(121.4281, device='cuda:0')) | > loss_gen: 2.3394775390625 (1.9875642408024181) | > loss_kl: 2.087921380996704 (2.152248311042785) | > loss_feat: 3.3910648822784424 (3.277283369411122) | > loss_mel: 15.665724754333496 (17.663094832680443) | > loss_duration: 1.6228981018066406 (1.5761059262535788) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.107086181640625 (26.656296747381038) | > grad_norm_1: tensor(9013.4004, device='cuda:0') (tensor(7334.5859, device='cuda:0')) | > current_lr_0: 0.00018608207844881698 | > current_lr_1: 0.00018608207844881698 | > step_time: 0.4605 (0.4459486701271751) | > loader_time: 0.0093 (0.007910026203502308) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619429359973316 (+9.657967258507212e-05) | > avg_step_time: 0.4571775281932992 (+0.0019962653307847855) | > avg_loss_disc: 2.645910551850225 (-0.11098654841033495) | > avg_loss_disc_real_0: 0.1397941262054611 (-0.009596453218812723) | > avg_loss_disc_real_1: 0.22207746938080855 (-0.013206451081893805) | > avg_loss_disc_real_2: 0.23189536492589494 (-0.008125708976262053) | > avg_loss_disc_real_3: 0.23634680559937382 (-0.0007809316608267947) | > avg_loss_disc_real_4: 0.24250225051188132 (+0.004826934194900601) | > avg_loss_disc_real_5: 0.24355098857006557 (+0.0017610601556133365) | > avg_loss_0: 2.645910551850225 (-0.11098654841033495) | > avg_grad_norm_0: tensor(124.1281, device='cuda:0') (tensor(-0.9289, device='cuda:0')) | > avg_loss_gen: 1.9871902952731495 (+0.05325240316525304) | > avg_loss_kl: 2.1429072265893643 (+0.03247616828327482) | > avg_loss_feat: 3.2441182237275887 (+0.00933438623455185) | > avg_loss_mel: 17.5231572943674 (+0.3811459877121699) | > avg_loss_duration: 1.58644872148272 (+0.009678598860619614) | > avg_amp_scaler: 16.0 (-0.22535211267605604) | > avg_loss_1: 26.483821761440222 (+0.48588755432988506) | > avg_grad_norm_1: tensor(7422.4204, device='cuda:0') (+tensor(1266.0181, device='cuda:0'))  > EPOCH: 578/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:09:24)   --> TIME: 2025-02-25 14:09:42 -- STEP: 33/72 -- GLOBAL_STEP: 1041650 | > loss_disc: 2.694129228591919 (2.657729669050737) | > loss_disc_real_0: 0.18843898177146912 (0.13203455756107965) | > loss_disc_real_1: 0.24616365134716034 (0.21070779769709616) | > loss_disc_real_2: 0.203894704580307 (0.24791927884022388) | > loss_disc_real_3: 0.22517575323581696 (0.2359017517530557) | > loss_disc_real_4: 0.23768669366836548 (0.24287047033960169) | > loss_disc_real_5: 0.24955657124519348 (0.24475992809642444) | > loss_0: 2.694129228591919 (2.657729669050737) | > grad_norm_0: tensor(111.4083, device='cuda:0') (tensor(140.3313, device='cuda:0')) | > loss_gen: 1.9175734519958496 (2.038585435260425) | > loss_kl: 2.156254291534424 (2.0088020201885337) | > loss_feat: 2.913418769836426 (3.502989675059463) | > loss_mel: 16.608123779296875 (17.277854688239813) | > loss_duration: 1.557005524635315 (1.537470402139606) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.152376174926758 (26.365702311197918) | > grad_norm_1: tensor(6241.9917, device='cuda:0') (tensor(8181.2437, device='cuda:0')) | > current_lr_0: 0.00018605881818901086 | > current_lr_1: 0.00018605881818901086 | > step_time: 0.4394 (0.4516377015547319) | > loader_time: 0.008 (0.0072255568070845174) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671280363915675 (+5.185100394235892e-05) | > avg_step_time: 0.46238882105115436 (+0.005211292857855154) | > avg_loss_disc: 2.6494736973668487 (+0.0035631455166238446) | > avg_loss_disc_real_0: 0.13976610861193967 (-2.8017593521445505e-05) | > avg_loss_disc_real_1: 0.2206245417326269 (-0.00145292764818164) | > avg_loss_disc_real_2: 0.2371510163369313 (+0.005255651411036372) | > avg_loss_disc_real_3: 0.2354118395859087 (-0.0009349660134651272) | > avg_loss_disc_real_4: 0.2428320864556541 (+0.0003298359437727827) | > avg_loss_disc_real_5: 0.2438925326290265 (+0.00034154405896091866) | > avg_loss_0: 2.6494736973668487 (+0.0035631455166238446) | > avg_grad_norm_0: tensor(135.9916, device='cuda:0') (+tensor(11.8635, device='cuda:0')) | > avg_loss_gen: 2.034900230421146 (+0.047709935147996596) | > avg_loss_kl: 2.0678419479182075 (-0.07506527867115675) | > avg_loss_feat: 3.4174099774427815 (+0.1732917537151928) | > avg_loss_mel: 17.197856956804294 (-0.3253003375631067) | > avg_loss_duration: 1.5762115784094368 (-0.010237143073283272) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.294220696032887 (-0.18960106540733435) | > avg_grad_norm_1: tensor(8059.6372, device='cuda:0') (+tensor(637.2168, device='cuda:0'))  > EPOCH: 579/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:10:02)   --> TIME: 2025-02-25 14:10:09 -- STEP: 11/72 -- GLOBAL_STEP: 1041700 | > loss_disc: 2.7479395866394043 (2.747720328244296) | > loss_disc_real_0: 0.1298612356185913 (0.13159327818588776) | > loss_disc_real_1: 0.26934880018234253 (0.20446409420533615) | > loss_disc_real_2: 0.21151456236839294 (0.22553663362156262) | > loss_disc_real_3: 0.22032560408115387 (0.231613889336586) | > loss_disc_real_4: 0.23819108307361603 (0.24253983795642853) | > loss_disc_real_5: 0.220100536942482 (0.24758041988719592) | > loss_0: 2.7479395866394043 (2.747720328244296) | > grad_norm_0: tensor(37.9181, device='cuda:0') (tensor(144.5377, device='cuda:0')) | > loss_gen: 2.088270902633667 (1.9147154851393267) | > loss_kl: 2.2950551509857178 (2.049727960066362) | > loss_feat: 3.4901626110076904 (3.399337313391946) | > loss_mel: 17.751482009887695 (17.27394624189897) | > loss_duration: 1.530622959136963 (1.538370739329945) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.15559196472168 (26.176097869873047) | > grad_norm_1: tensor(6900.2837, device='cuda:0') (tensor(6908.7495, device='cuda:0')) | > current_lr_0: 0.00018603556083673722 | > current_lr_1: 0.00018603556083673722 | > step_time: 0.4253 (0.4323397766460072) | > loader_time: 0.0059 (0.006541425531560724)  --> TIME: 2025-02-25 14:10:32 -- STEP: 61/72 -- GLOBAL_STEP: 1041750 | > loss_disc: 2.4669883251190186 (2.696691853101136) | > loss_disc_real_0: 0.151568204164505 (0.14537171188925138) | > loss_disc_real_1: 0.20595908164978027 (0.2239135811563398) | > loss_disc_real_2: 0.2598138153553009 (0.24421067413736564) | > loss_disc_real_3: 0.21720847487449646 (0.24043678480093597) | > loss_disc_real_4: 0.24277174472808838 (0.24011580963603785) | > loss_disc_real_5: 0.243711918592453 (0.24382596695032277) | > loss_0: 2.4669883251190186 (2.696691853101136) | > grad_norm_0: tensor(142.7321, device='cuda:0') (tensor(137.5160, device='cuda:0')) | > loss_gen: 2.3738818168640137 (2.0006479454822226) | > loss_kl: 2.163801431655884 (2.1255298364357866) | > loss_feat: 4.017035007476807 (3.371337867173992) | > loss_mel: 17.550296783447266 (17.052631706487933) | > loss_duration: 1.623795509338379 (1.569785635979449) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.728809356689453 (26.119933018918896) | > grad_norm_1: tensor(10618.2119, device='cuda:0') (tensor(7424.0928, device='cuda:0')) | > current_lr_0: 0.00018603556083673722 | > current_lr_1: 0.00018603556083673722 | > step_time: 0.4579 (0.4489642049445481) | > loader_time: 0.0106 (0.008196209297805533) --> EVAL PERFORMANCE | > avg_loader_time: 0.008650148418587696 (-2.1131945327979712e-05) | > avg_step_time: 0.45869468635236715 (-0.0036941346987872103) | > avg_loss_disc: 2.697563728816072 (+0.048090031449223414) | > avg_loss_disc_real_0: 0.14547465741634374 (+0.005708548804404079) | > avg_loss_disc_real_1: 0.22598633044202562 (+0.005361788709398713) | > avg_loss_disc_real_2: 0.2441478560508137 (+0.006996839713882402) | > avg_loss_disc_real_3: 0.24028160013783145 (+0.004869760551922758) | > avg_loss_disc_real_4: 0.24077694936537405 (-0.002055137090280046) | > avg_loss_disc_real_5: 0.24371889464452234 (-0.00017363798450414913) | > avg_loss_0: 2.697563728816072 (+0.048090031449223414) | > avg_grad_norm_0: tensor(137.8030, device='cuda:0') (+tensor(1.8114, device='cuda:0')) | > avg_loss_gen: 1.998266253672855 (-0.03663397674829105) | > avg_loss_kl: 2.1352855420448402 (+0.0674435941266327) | > avg_loss_feat: 3.3324410042292634 (-0.08496897321351815) | > avg_loss_mel: 17.044485213051377 (-0.15337174375291696) | > avg_loss_duration: 1.5776878712882458 (+0.0014762928788090157) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.088165954804758 (-0.2060547412281295) | > avg_grad_norm_1: tensor(7536.6460, device='cuda:0') (tensor(-522.9912, device='cuda:0'))  > EPOCH: 580/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:10:39)   --> TIME: 2025-02-25 14:10:59 -- STEP: 39/72 -- GLOBAL_STEP: 1041800 | > loss_disc: 2.68326735496521 (2.768664146080995) | > loss_disc_real_0: 0.2704097032546997 (0.18942758880364588) | > loss_disc_real_1: 0.3059210777282715 (0.22964450277579138) | > loss_disc_real_2: 0.26916319131851196 (0.2419835111269584) | > loss_disc_real_3: 0.21653136610984802 (0.23228539946751717) | > loss_disc_real_4: 0.2717197835445404 (0.24215208643522018) | > loss_disc_real_5: 0.2541455626487732 (0.24192264828926477) | > loss_0: 2.68326735496521 (2.768664146080995) | > grad_norm_0: tensor(90.8269, device='cuda:0') (tensor(127.8258, device='cuda:0')) | > loss_gen: 2.4640467166900635 (2.04095013325031) | > loss_kl: 2.8920059204101562 (2.3712309018159528) | > loss_feat: 3.9368984699249268 (3.4983378862723327) | > loss_mel: 20.957561492919922 (18.470346621977978) | > loss_duration: 1.6249635219573975 (1.5808880023467236) | > amp_scaler: 16.0 (16.0) | > loss_1: 31.87547492980957 (27.961753551776592) | > grad_norm_1: tensor(4924.1655, device='cuda:0') (tensor(6339.9644, device='cuda:0')) | > current_lr_0: 0.00018601230639163263 | > current_lr_1: 0.00018601230639163263 | > step_time: 0.441 (0.43959396313398313) | > loader_time: 0.0086 (0.007547714771368565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008798454848813341 (+0.00014830643022564587) | > avg_step_time: 0.45340324455583597 (-0.005291441796531182) | > avg_loss_disc: 2.7716150518874048 (+0.07405132307133266) | > avg_loss_disc_real_0: 0.1804502813119284 (+0.034975623895584645) | > avg_loss_disc_real_1: 0.23525432707138463 (+0.00926799662935901) | > avg_loss_disc_real_2: 0.23511460790751684 (-0.009033248143296868) | > avg_loss_disc_real_3: 0.23415115286766644 (-0.0061304472701650126) | > avg_loss_disc_real_4: 0.24344869167871877 (+0.002671742313344716) | > avg_loss_disc_real_5: 0.24389616767285574 (+0.00017727302833339675) | > avg_loss_0: 2.7716150518874048 (+0.07405132307133266) | > avg_grad_norm_0: tensor(121.0651, device='cuda:0') (tensor(-16.7379, device='cuda:0')) | > avg_loss_gen: 1.9728246826521103 (-0.025441571020744735) | > avg_loss_kl: 2.3504123620583988 (+0.21512682001355854) | > avg_loss_feat: 3.268784979699363 (-0.06365602452990027) | > avg_loss_mel: 18.225398117387797 (+1.1809129043364202) | > avg_loss_duration: 1.6033087834505968 (+0.02562091216235096) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.420728790927942 (+1.3325628361231843) | > avg_grad_norm_1: tensor(5869.4243, device='cuda:0') (tensor(-1667.2217, device='cuda:0'))  > EPOCH: 581/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:11:16)   --> TIME: 2025-02-25 14:11:25 -- STEP: 17/72 -- GLOBAL_STEP: 1041850 | > loss_disc: 2.622293472290039 (2.623614283169017) | > loss_disc_real_0: 0.08167117834091187 (0.10412480068557403) | > loss_disc_real_1: 0.15015959739685059 (0.23164577168576858) | > loss_disc_real_2: 0.19746153056621552 (0.23407713630620172) | > loss_disc_real_3: 0.2272946536540985 (0.22889643381623662) | > loss_disc_real_4: 0.24734006822109222 (0.23704962169422822) | > loss_disc_real_5: 0.2506348192691803 (0.2416654439533458) | > loss_0: 2.622293472290039 (2.623614283169017) | > grad_norm_0: tensor(161.0441, device='cuda:0') (tensor(122.3069, device='cuda:0')) | > loss_gen: 1.7029588222503662 (1.9505493290284102) | > loss_kl: 2.1589365005493164 (2.0476089715957637) | > loss_feat: 3.3660728931427 (3.478588945725385) | > loss_mel: 18.45134735107422 (17.46023402494543) | > loss_duration: 1.5826137065887451 (1.5466501081691069) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.26192855834961 (26.483631246230182) | > grad_norm_1: tensor(6764.7476, device='cuda:0') (tensor(7050.5571, device='cuda:0')) | > current_lr_0: 0.00018598905485333366 | > current_lr_1: 0.00018598905485333366 | > step_time: 0.43 (0.43229205468121695) | > loader_time: 0.0061 (0.00668953446780934)  --> TIME: 2025-02-25 14:11:49 -- STEP: 67/72 -- GLOBAL_STEP: 1041900 | > loss_disc: 2.649038076400757 (2.6919752697446455) | > loss_disc_real_0: 0.08695211261510849 (0.1401377938020585) | > loss_disc_real_1: 0.18049544095993042 (0.22776224019367303) | > loss_disc_real_2: 0.21706432104110718 (0.2327510636689058) | > loss_disc_real_3: 0.2664303183555603 (0.2329042960903538) | > loss_disc_real_4: 0.21741518378257751 (0.23983587947354387) | > loss_disc_real_5: 0.2128068059682846 (0.24125459456621712) | > loss_0: 2.649038076400757 (2.6919752697446455) | > grad_norm_0: tensor(109.9225, device='cuda:0') (tensor(107.8320, device='cuda:0')) | > loss_gen: 1.791890025138855 (2.007042877709688) | > loss_kl: 2.1997599601745605 (2.7007422358242423) | > loss_feat: 3.5102477073669434 (3.447648397132532) | > loss_mel: 17.760604858398438 (19.377641478581218) | > loss_duration: 1.6827642917633057 (1.6084787187291616) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.94526481628418 (29.14155373644473) | > grad_norm_1: tensor(5989.8706, device='cuda:0') (tensor(5724.2588, device='cuda:0')) | > current_lr_0: 0.00018598905485333366 | > current_lr_1: 0.00018598905485333366 | > step_time: 0.5176 (0.4501323735536034) | > loader_time: 0.0123 (0.008386821889165622) --> EVAL PERFORMANCE | > avg_loader_time: 0.008574589877061441 (-0.00022386497175190014) | > avg_step_time: 0.4603789423553037 (+0.006975697799467717) | > avg_loss_disc: 2.6871077443512394 (-0.08450730753616531) | > avg_loss_disc_real_0: 0.14213084319317845 (-0.03831943811874994) | > avg_loss_disc_real_1: 0.22719790282803523 (-0.008056424243349403) | > avg_loss_disc_real_2: 0.2323255910420082 (-0.0027890168655086423) | > avg_loss_disc_real_3: 0.23297550057021665 (-0.0011756522974497863) | > avg_loss_disc_real_4: 0.2396226621849436 (-0.003826029493775168) | > avg_loss_disc_real_5: 0.24104136543374666 (-0.0028548022391090833) | > avg_loss_0: 2.6871077443512394 (-0.08450730753616531) | > avg_grad_norm_0: tensor(107.5658, device='cuda:0') (tensor(-13.4993, device='cuda:0')) | > avg_loss_gen: 2.0206009770782902 (+0.04777629442617992) | > avg_loss_kl: 2.682800677460683 (+0.3323883154022842) | > avg_loss_feat: 3.4421067741555227 (+0.17332179445615958) | > avg_loss_mel: 19.320759679230168 (+1.0953615618423704) | > avg_loss_duration: 1.6104979699766133 (+0.007189186526016522) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.076766081259283 (+1.6560372903313407) | > avg_grad_norm_1: tensor(5667.8125, device='cuda:0') (tensor(-201.6118, device='cuda:0'))  > EPOCH: 582/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:11:53)   --> TIME: 2025-02-25 14:12:16 -- STEP: 45/72 -- GLOBAL_STEP: 1041950 | > loss_disc: 2.5654783248901367 (2.670761903127034) | > loss_disc_real_0: 0.107570119202137 (0.1523159059385459) | > loss_disc_real_1: 0.21637727320194244 (0.22708448767662048) | > loss_disc_real_2: 0.25105321407318115 (0.24059394432438744) | > loss_disc_real_3: 0.24511541426181793 (0.23214697473578982) | > loss_disc_real_4: 0.24816805124282837 (0.24473542703522577) | > loss_disc_real_5: 0.25195205211639404 (0.2514686508311166) | > loss_0: 2.5654783248901367 (2.670761903127034) | > grad_norm_0: tensor(71.8914, device='cuda:0') (tensor(88.0524, device='cuda:0')) | > loss_gen: 2.282917022705078 (2.0443427032894554) | > loss_kl: 2.2434866428375244 (2.150104022026062) | > loss_feat: 3.7846274375915527 (3.40238200823466) | > loss_mel: 17.5022029876709 (17.713400014241536) | > loss_duration: 1.590551733970642 (1.5713708215289648) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.403785705566406 (26.88159989251031) | > grad_norm_1: tensor(6714.6001, device='cuda:0') (tensor(5194.1494, device='cuda:0')) | > current_lr_0: 0.000185965806221477 | > current_lr_1: 0.000185965806221477 | > step_time: 0.4672 (0.44670853614807127) | > loader_time: 0.0088 (0.0075625684526231555) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653100107757139 (+7.85102306956973e-05) | > avg_step_time: 0.45902500689869197 (-0.0013539354566117212) | > avg_loss_disc: 2.6990865720829493 (+0.011978827731709885) | > avg_loss_disc_real_0: 0.1783649311204192 (+0.036234087927240755) | > avg_loss_disc_real_1: 0.22909677154581312 (+0.0018988687177778907) | > avg_loss_disc_real_2: 0.23898143352757037 (+0.006655842485562169) | > avg_loss_disc_real_3: 0.232816015540714 (-0.00015948502950266508) | > avg_loss_disc_real_4: 0.2459060635784982 (+0.0062834013935546) | > avg_loss_disc_real_5: 0.2485788111955347 (+0.007537445761788053) | > avg_loss_0: 2.6990865720829493 (+0.011978827731709885) | > avg_grad_norm_0: tensor(87.5305, device='cuda:0') (tensor(-20.0352, device='cuda:0')) | > avg_loss_gen: 2.0545547629745906 (+0.03395378589630038) | > avg_loss_kl: 2.1521083519492348 (-0.5306923255114482) | > avg_loss_feat: 3.293587886111837 (-0.14851888804368585) | > avg_loss_mel: 17.60835502517055 (-1.7124046540596183) | > avg_loss_duration: 1.5914312970470377 (-0.019066672929575645) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.700037459252584 (-2.3767286220066985) | > avg_grad_norm_1: tensor(4908.9868, device='cuda:0') (tensor(-758.8257, device='cuda:0'))  > EPOCH: 583/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:12:30)   --> TIME: 2025-02-25 14:12:43 -- STEP: 23/72 -- GLOBAL_STEP: 1042000 | > loss_disc: 2.736968994140625 (2.723320214644722) | > loss_disc_real_0: 0.1385655403137207 (0.17276848593483798) | > loss_disc_real_1: 0.22724075615406036 (0.27421004260363785) | > loss_disc_real_2: 0.20854595303535461 (0.24178040027618408) | > loss_disc_real_3: 0.1869489699602127 (0.22805483380089636) | > loss_disc_real_4: 0.24189217388629913 (0.2444346475860347) | > loss_disc_real_5: 0.24470198154449463 (0.2466143408547277) | > loss_0: 2.736968994140625 (2.723320214644722) | > grad_norm_0: tensor(5.9114, device='cuda:0') (tensor(72.1395, device='cuda:0')) | > loss_gen: 2.0953280925750732 (2.0629995128382803) | > loss_kl: 2.0111083984375 (2.0236499050389165) | > loss_feat: 3.3812508583068848 (3.2237784655197808) | > loss_mel: 17.867801666259766 (17.414401178774625) | > loss_duration: 1.5683047771453857 (1.5440686526505842) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.923791885375977 (26.2688975126847) | > grad_norm_1: tensor(4273.6001, device='cuda:0') (tensor(4706.3613, device='cuda:0')) | > current_lr_0: 0.0001859425604956993 | > current_lr_1: 0.0001859425604956993 | > step_time: 0.4296 (0.4365739615067192) | > loader_time: 0.0063 (0.006965160369873047) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737392828498088 (+8.429272074094926e-05) | > avg_step_time: 0.4569745433162635 (-0.002050463582428441) | > avg_loss_disc: 2.728330051395255 (+0.02924347931230553) | > avg_loss_disc_real_0: 0.1749693732446348 (-0.0033955578757844107) | > avg_loss_disc_real_1: 0.25021563501844946 (+0.021118863472636334) | > avg_loss_disc_real_2: 0.24074326854356579 (+0.0017618350159954166) | > avg_loss_disc_real_3: 0.2359836414246492 (+0.00316762588393521) | > avg_loss_disc_real_4: 0.2402631900679897 (-0.005642873510508489) | > avg_loss_disc_real_5: 0.2443793623800009 (-0.004199448815533802) | > avg_loss_0: 2.728330051395255 (+0.02924347931230553) | > avg_grad_norm_0: tensor(79.3505, device='cuda:0') (tensor(-8.1800, device='cuda:0')) | > avg_loss_gen: 2.009675679072527 (-0.04487908390206341) | > avg_loss_kl: 2.4279277929117975 (+0.2758194409625627) | > avg_loss_feat: 3.2048345350883376 (-0.08875335102349924) | > avg_loss_mel: 18.774234395631602 (+1.1658793704610524) | > avg_loss_duration: 1.6120330263191545 (+0.020601729272116787) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.028705382011307 (+1.3286679227587221) | > avg_grad_norm_1: tensor(5133.9731, device='cuda:0') (+tensor(224.9863, device='cuda:0'))  > EPOCH: 584/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:13:08)   --> TIME: 2025-02-25 14:13:10 -- STEP: 1/72 -- GLOBAL_STEP: 1042050 | > loss_disc: 2.770709991455078 (2.770709991455078) | > loss_disc_real_0: 0.09311750531196594 (0.09311750531196594) | > loss_disc_real_1: 0.1395217776298523 (0.1395217776298523) | > loss_disc_real_2: 0.20917122066020966 (0.20917122066020966) | > loss_disc_real_3: 0.21591325104236603 (0.21591325104236603) | > loss_disc_real_4: 0.2425781935453415 (0.2425781935453415) | > loss_disc_real_5: 0.264080286026001 (0.264080286026001) | > loss_0: 2.770709991455078 (2.770709991455078) | > grad_norm_0: tensor(128.7235, device='cuda:0') (tensor(128.7235, device='cuda:0')) | > loss_gen: 1.5897767543792725 (1.5897767543792725) | > loss_kl: 2.397355794906616 (2.397355794906616) | > loss_feat: 3.0859267711639404 (3.0859267711639404) | > loss_mel: 20.756637573242188 (20.756637573242188) | > loss_duration: 1.6143735647201538 (1.6143735647201538) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.44407081604004 (29.44407081604004) | > grad_norm_1: tensor(6022.9941, device='cuda:0') (tensor(6022.9941, device='cuda:0')) | > current_lr_0: 0.00018591931767563734 | > current_lr_1: 0.00018591931767563734 | > step_time: 0.4472 (0.4472169876098633) | > loader_time: 0.0063 (0.0062732696533203125)  --> TIME: 2025-02-25 14:13:33 -- STEP: 51/72 -- GLOBAL_STEP: 1042100 | > loss_disc: 2.7162926197052 (2.6542531135035494) | > loss_disc_real_0: 0.10792036354541779 (0.13506215116849135) | > loss_disc_real_1: 0.22824418544769287 (0.22541205161342434) | > loss_disc_real_2: 0.2682848274707794 (0.22523917082478018) | > loss_disc_real_3: 0.22149452567100525 (0.23109215176572986) | > loss_disc_real_4: 0.2484751045703888 (0.23824100050271727) | > loss_disc_real_5: 0.2813413441181183 (0.23390402922443315) | > loss_0: 2.7162926197052 (2.6542531135035494) | > grad_norm_0: tensor(138.5990, device='cuda:0') (tensor(85.3041, device='cuda:0')) | > loss_gen: 2.0065879821777344 (1.9732506392048854) | > loss_kl: 2.3668363094329834 (2.6103203974518103) | > loss_feat: 3.425004243850708 (3.340533724018172) | > loss_mel: 19.10108757019043 (19.609726625330314) | > loss_duration: 1.654164433479309 (1.658717244279151) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.553682327270508 (29.192548452639112) | > grad_norm_1: tensor(7241.5586, device='cuda:0') (tensor(5240.4639, device='cuda:0')) | > current_lr_0: 0.00018591931767563734 | > current_lr_1: 0.00018591931767563734 | > step_time: 0.4391 (0.43967555550967946) | > loader_time: 0.009 (0.0076871245515112785) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859641021406147 (-0.00014098261443661747) | > avg_step_time: 0.4519735054231026 (-0.00500103789316092) | > avg_loss_disc: 2.6634137328241914 (-0.0649163185710635) | > avg_loss_disc_real_0: 0.13481088292220952 (-0.04015849032242527) | > avg_loss_disc_real_1: 0.22437290752857503 (-0.02584272748987443) | > avg_loss_disc_real_2: 0.23401045820242922 (-0.006732810341136569) | > avg_loss_disc_real_3: 0.2352720559902594 (-0.0007115854343897887) | > avg_loss_disc_real_4: 0.2448164980176469 (+0.004553307949657193) | > avg_loss_disc_real_5: 0.24373693386433828 (-0.0006424285156626242) | > avg_loss_0: 2.6634137328241914 (-0.0649163185710635) | > avg_grad_norm_0: tensor(91.6925, device='cuda:0') (+tensor(12.3421, device='cuda:0')) | > avg_loss_gen: 1.9962301472542991 (-0.013445531818228051) | > avg_loss_kl: 2.5398927188255405 (+0.11196492591374296) | > avg_loss_feat: 3.334403031308886 (+0.12956849622054856) | > avg_loss_mel: 19.26744281070334 (+0.49320841507173796) | > avg_loss_duration: 1.6577906608581543 (+0.045757634538999836) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.79575922791387 (+0.7670538459025629) | > avg_grad_norm_1: tensor(6010.0391, device='cuda:0') (+tensor(876.0659, device='cuda:0'))  > EPOCH: 585/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:13:44)   --> TIME: 2025-02-25 14:13:59 -- STEP: 29/72 -- GLOBAL_STEP: 1042150 | > loss_disc: 2.4938125610351562 (2.5665087535463535) | > loss_disc_real_0: 0.0903783068060875 (0.12218504399061203) | > loss_disc_real_1: 0.23344974219799042 (0.2142893754202744) | > loss_disc_real_2: 0.2542767822742462 (0.22881490043525038) | > loss_disc_real_3: 0.24673646688461304 (0.23189814029068784) | > loss_disc_real_4: 0.25151461362838745 (0.24161424051071034) | > loss_disc_real_5: 0.25803959369659424 (0.2452249675989151) | > loss_0: 2.4938125610351562 (2.5665087535463535) | > grad_norm_0: tensor(64.5232, device='cuda:0') (tensor(116.1194, device='cuda:0')) | > loss_gen: 2.318981885910034 (2.0857274121251597) | > loss_kl: 2.119933843612671 (2.112796914988551) | > loss_feat: 3.906221389770508 (3.5026569037601867) | > loss_mel: 17.7863712310791 (17.765662160413015) | > loss_duration: 1.568331003189087 (1.5652436922336448) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.699838638305664 (27.032087391820447) | > grad_norm_1: tensor(10072.5684, device='cuda:0') (tensor(8399.8672, device='cuda:0')) | > current_lr_0: 0.00018589607776092786 | > current_lr_1: 0.00018589607776092786 | > step_time: 0.4444 (0.4330243077771417) | > loader_time: 0.0075 (0.007092500555104223) --> EVAL PERFORMANCE | > avg_loader_time: 0.008843112999284771 (+0.000246702785223301) | > avg_step_time: 0.4610403725798701 (+0.009066867156767466) | > avg_loss_disc: 2.581146787589706 (-0.08226694523448552) | > avg_loss_disc_real_0: 0.13073294725216608 (-0.004077935670043442) | > avg_loss_disc_real_1: 0.21518583763653124 (-0.009187069892043792) | > avg_loss_disc_real_2: 0.23069494291090628 (-0.0033155152915229358) | > avg_loss_disc_real_3: 0.2332317482837489 (-0.002040307706510508) | > avg_loss_disc_real_4: 0.24373720313461733 (-0.0010792948830295745) | > avg_loss_disc_real_5: 0.24829173696712709 (+0.004554803102788801) | > avg_loss_0: 2.581146787589706 (-0.08226694523448552) | > avg_grad_norm_0: tensor(123.0778, device='cuda:0') (+tensor(31.3853, device='cuda:0')) | > avg_loss_gen: 2.0819853107694164 (+0.08575516351511725) | > avg_loss_kl: 2.1151106475104746 (-0.42478207131506585) | > avg_loss_feat: 3.4297251130493596 (+0.09532208174047341) | > avg_loss_mel: 17.55797971806056 (-1.7094630926427783) | > avg_loss_duration: 1.59777518729089 (-0.06001547356726422) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.782576037124855 (-2.0131831907890145) | > avg_grad_norm_1: tensor(8215.9766, device='cuda:0') (+tensor(2205.9375, device='cuda:0'))  > EPOCH: 586/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:14:22)   --> TIME: 2025-02-25 14:14:27 -- STEP: 7/72 -- GLOBAL_STEP: 1042200 | > loss_disc: 2.5144283771514893 (2.5528995990753174) | > loss_disc_real_0: 0.12741167843341827 (0.12330884273563113) | > loss_disc_real_1: 0.18886037170886993 (0.21674580233437674) | > loss_disc_real_2: 0.24408367276191711 (0.23134487867355347) | > loss_disc_real_3: 0.20589973032474518 (0.22970985727650778) | > loss_disc_real_4: 0.23348714411258698 (0.2371042243071965) | > loss_disc_real_5: 0.23660990595817566 (0.2418229537350791) | > loss_0: 2.5144283771514893 (2.5528995990753174) | > grad_norm_0: tensor(138.6628, device='cuda:0') (tensor(138.7028, device='cuda:0')) | > loss_gen: 2.1044297218322754 (2.154139416558402) | > loss_kl: 2.170619249343872 (2.0520381927490234) | > loss_feat: 3.5265376567840576 (3.6217682702200755) | > loss_mel: 17.557636260986328 (17.595057351248606) | > loss_duration: 1.5404328107833862 (1.5629334449768066) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.899654388427734 (26.98593657357352) | > grad_norm_1: tensor(9819.2012, device='cuda:0') (tensor(9236.4082, device='cuda:0')) | > current_lr_0: 0.00018587284075120773 | > current_lr_1: 0.00018587284075120773 | > step_time: 0.4297 (0.43439882142203196) | > loader_time: 0.0074 (0.006762027740478516)  --> TIME: 2025-02-25 14:14:50 -- STEP: 57/72 -- GLOBAL_STEP: 1042250 | > loss_disc: 2.712047576904297 (2.6991069860625685) | > loss_disc_real_0: 0.1480102241039276 (0.1662768512441401) | > loss_disc_real_1: 0.24041935801506042 (0.2376278419243662) | > loss_disc_real_2: 0.21916928887367249 (0.24617407876148556) | > loss_disc_real_3: 0.20400983095169067 (0.23356531455851437) | > loss_disc_real_4: 0.23392271995544434 (0.24085499764534465) | > loss_disc_real_5: 0.21360404789447784 (0.2432572776288317) | > loss_0: 2.712047576904297 (2.6991069860625685) | > grad_norm_0: tensor(100.0579, device='cuda:0') (tensor(118.4049, device='cuda:0')) | > loss_gen: 1.8257514238357544 (2.08468364832694) | > loss_kl: 2.373919725418091 (2.053261867740698) | > loss_feat: 3.0210585594177246 (3.3951577094563268) | > loss_mel: 17.390623092651367 (17.480241340503362) | > loss_duration: 1.6673035621643066 (1.5813468652859068) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.278657913208008 (26.594691560979474) | > grad_norm_1: tensor(4280.2490, device='cuda:0') (tensor(6791.7134, device='cuda:0')) | > current_lr_0: 0.00018587284075120773 | > current_lr_1: 0.00018587284075120773 | > step_time: 0.4555 (0.44782945984288264) | > loader_time: 0.0096 (0.008059593669155186) --> EVAL PERFORMANCE | > avg_loader_time: 0.008678224724783024 (-0.0001648882745017479) | > avg_step_time: 0.45901030218097527 (-0.002030070398894801) | > avg_loss_disc: 2.697350045325051 (+0.11620325773534512) | > avg_loss_disc_real_0: 0.1652271523022316 (+0.03449420505006551) | > avg_loss_disc_real_1: 0.23651673655274888 (+0.021330898916217644) | > avg_loss_disc_real_2: 0.24367313548712663 (+0.012978192576220349) | > avg_loss_disc_real_3: 0.23480980874786914 (+0.0015780604641202345) | > avg_loss_disc_real_4: 0.24028534725518294 (-0.00345185587943439) | > avg_loss_disc_real_5: 0.24420574853118038 (-0.004085988435946708) | > avg_loss_0: 2.697350045325051 (+0.11620325773534512) | > avg_grad_norm_0: tensor(112.9369, device='cuda:0') (tensor(-10.1409, device='cuda:0')) | > avg_loss_gen: 2.049390646773324 (-0.03259466399609234) | > avg_loss_kl: 2.095029610983083 (-0.02008103652739157) | > avg_loss_feat: 3.306335643983223 (-0.12338946906613657) | > avg_loss_mel: 17.566342528437225 (+0.008362810376663532) | > avg_loss_duration: 1.599244367908424 (+0.0014691806175339828) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.616342813196315 (-0.1662332239285398) | > avg_grad_norm_1: tensor(6482.8838, device='cuda:0') (tensor(-1733.0928, device='cuda:0'))  > EPOCH: 587/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:14:59)   --> TIME: 2025-02-25 14:15:16 -- STEP: 35/72 -- GLOBAL_STEP: 1042300 | > loss_disc: 2.3943493366241455 (2.604893514088222) | > loss_disc_real_0: 0.11962233483791351 (0.1367476680449077) | > loss_disc_real_1: 0.18714991211891174 (0.2141439689057214) | > loss_disc_real_2: 0.19942207634449005 (0.22895331425326212) | > loss_disc_real_3: 0.22582602500915527 (0.23049190512725284) | > loss_disc_real_4: 0.22323884069919586 (0.2437806955405644) | > loss_disc_real_5: 0.2602081000804901 (0.2450388537985938) | > loss_0: 2.3943493366241455 (2.604893514088222) | > grad_norm_0: tensor(86.6225, device='cuda:0') (tensor(126.3332, device='cuda:0')) | > loss_gen: 2.1422667503356934 (1.999057446207319) | > loss_kl: 2.204531192779541 (1.9575161252702986) | > loss_feat: 3.76310658454895 (3.3594814096178327) | > loss_mel: 18.441234588623047 (17.365964862278535) | > loss_duration: 1.604080319404602 (1.5522505283355712) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.15522003173828 (26.234270531790596) | > grad_norm_1: tensor(9133.6152, device='cuda:0') (tensor(6958.7500, device='cuda:0')) | > current_lr_0: 0.00018584960664611383 | > current_lr_1: 0.00018584960664611383 | > step_time: 0.4345 (0.4370006152561733) | > loader_time: 0.0081 (0.007170193535940988) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696948978262889 (+1.8724253479865047e-05) | > avg_step_time: 0.45474652505256763 (-0.004263777128407642) | > avg_loss_disc: 2.619628694695486 (-0.07772135062956487) | > avg_loss_disc_real_0: 0.13644970009024715 (-0.02877745221198444) | > avg_loss_disc_real_1: 0.21859306285918598 (-0.017923673693562903) | > avg_loss_disc_real_2: 0.22876623230920712 (-0.014906903177919512) | > avg_loss_disc_real_3: 0.23216190514430193 (-0.0026479036035672032) | > avg_loss_disc_real_4: 0.24250437424216473 (+0.002219026986981787) | > avg_loss_disc_real_5: 0.24547844999273058 (+0.001272701461550202) | > avg_loss_0: 2.619628694695486 (-0.07772135062956487) | > avg_grad_norm_0: tensor(131.7652, device='cuda:0') (+tensor(18.8284, device='cuda:0')) | > avg_loss_gen: 1.99935906705722 (-0.05003157971610395) | > avg_loss_kl: 1.9992065564007826 (-0.09582305458230045) | > avg_loss_feat: 3.3332217579156582 (+0.02688611393243523) | > avg_loss_mel: 17.20838004098812 (-0.35796248744910386) | > avg_loss_duration: 1.5835225850763455 (-0.01572178283207859) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.1236900813143 (-0.4926527318820142) | > avg_grad_norm_1: tensor(7221.7144, device='cuda:0') (+tensor(738.8306, device='cuda:0'))  > EPOCH: 588/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:15:36)   --> TIME: 2025-02-25 14:15:43 -- STEP: 13/72 -- GLOBAL_STEP: 1042350 | > loss_disc: 2.892878770828247 (2.651743063559899) | > loss_disc_real_0: 0.06280434131622314 (0.1494577074280152) | > loss_disc_real_1: 0.23583362996578217 (0.21350969144931206) | > loss_disc_real_2: 0.2292417734861374 (0.23062724333543044) | > loss_disc_real_3: 0.23097749054431915 (0.23748372839047358) | > loss_disc_real_4: 0.2346225529909134 (0.2420292290357443) | > loss_disc_real_5: 0.24087880551815033 (0.2452885485612429) | > loss_0: 2.892878770828247 (2.651743063559899) | > grad_norm_0: tensor(56.9005, device='cuda:0') (tensor(133.8027, device='cuda:0')) | > loss_gen: 1.8036588430404663 (2.002382617730361) | > loss_kl: 1.7771685123443604 (1.9053252110114465) | > loss_feat: 3.239680528640747 (3.4176643811739407) | > loss_mel: 16.638227462768555 (17.23995436154879) | > loss_duration: 1.4754058122634888 (1.5383901687768788) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.93414306640625 (26.1037167769212) | > grad_norm_1: tensor(2416.2197, device='cuda:0') (tensor(7785.1436, device='cuda:0')) | > current_lr_0: 0.00018582637544528306 | > current_lr_1: 0.00018582637544528306 | > step_time: 0.4679 (0.43274807929992676) | > loader_time: 0.0068 (0.006411424049964318)  --> TIME: 2025-02-25 14:16:06 -- STEP: 63/72 -- GLOBAL_STEP: 1042400 | > loss_disc: 2.6090474128723145 (2.6965363706861223) | > loss_disc_real_0: 0.04816638305783272 (0.15321960595865097) | > loss_disc_real_1: 0.1945139318704605 (0.21780966387854683) | > loss_disc_real_2: 0.23750180006027222 (0.24163976098809922) | > loss_disc_real_3: 0.24152234196662903 (0.2372753137633914) | > loss_disc_real_4: 0.26653313636779785 (0.24220495399028535) | > loss_disc_real_5: 0.2410721480846405 (0.24557152272216856) | > loss_0: 2.6090474128723145 (2.6965363706861223) | > grad_norm_0: tensor(70.1751, device='cuda:0') (tensor(125.9347, device='cuda:0')) | > loss_gen: 1.992911696434021 (1.9702501902504572) | > loss_kl: 2.3134093284606934 (2.0293085707558514) | > loss_feat: 3.379690647125244 (3.354185467674619) | > loss_mel: 16.244834899902344 (17.217930627247647) | > loss_duration: 1.6436336040496826 (1.574635144263979) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.574480056762695 (26.146310140216162) | > grad_norm_1: tensor(6804.0527, device='cuda:0') (tensor(6557.7793, device='cuda:0')) | > current_lr_0: 0.00018582637544528306 | > current_lr_1: 0.00018582637544528306 | > step_time: 0.4783 (0.4433395749046689) | > loader_time: 0.0112 (0.008285431634812131) --> EVAL PERFORMANCE | > avg_loader_time: 0.008646854212586312 (-5.009476567657649e-05) | > avg_step_time: 0.4580334038801596 (+0.003286878827591988) | > avg_loss_disc: 2.701952853672941 (+0.08232415897745504) | > avg_loss_disc_real_0: 0.16171463185422857 (+0.025264931763981424) | > avg_loss_disc_real_1: 0.21878307491121157 (+0.00019001205202559546) | > avg_loss_disc_real_2: 0.238629334829223 (+0.009863102520015876) | > avg_loss_disc_real_3: 0.23726965570953532 (+0.005107750565233382) | > avg_loss_disc_real_4: 0.24303183903996373 (+0.0005274647977989988) | > avg_loss_disc_real_5: 0.2468870716615462 (+0.0014086216688156128) | > avg_loss_0: 2.701952853672941 (+0.08232415897745504) | > avg_grad_norm_0: tensor(126.8203, device='cuda:0') (tensor(-4.9449, device='cuda:0')) | > avg_loss_gen: 1.9887219734594856 (-0.01063709359773446) | > avg_loss_kl: 2.0611870137738504 (+0.06198045737306779) | > avg_loss_feat: 3.344163216335673 (+0.010941458420014705) | > avg_loss_mel: 17.296289255921266 (+0.08790921493314485) | > avg_loss_duration: 1.5846776995860354 (+0.0011551145096899695) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.275039269890584 (+0.1513491885762832) | > avg_grad_norm_1: tensor(6537.9062, device='cuda:0') (tensor(-683.8081, device='cuda:0'))  > EPOCH: 589/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:16:13)   --> TIME: 2025-02-25 14:16:33 -- STEP: 41/72 -- GLOBAL_STEP: 1042450 | > loss_disc: 2.910435199737549 (2.7227503904482213) | > loss_disc_real_0: 0.20519982278347015 (0.2182100029980264) | > loss_disc_real_1: 0.20811806619167328 (0.2204130851640934) | > loss_disc_real_2: 0.2089994102716446 (0.24058156224285684) | > loss_disc_real_3: 0.16968239843845367 (0.2414958466116975) | > loss_disc_real_4: 0.2635849714279175 (0.24443693895165514) | > loss_disc_real_5: 0.2571652829647064 (0.24674003335033975) | > loss_0: 2.910435199737549 (2.7227503904482213) | > grad_norm_0: tensor(152.6888, device='cuda:0') (tensor(87.0865, device='cuda:0')) | > loss_gen: 1.6220729351043701 (2.071383921111502) | > loss_kl: 2.380582809448242 (2.59809395452825) | > loss_feat: 2.9929161071777344 (3.3972897006244196) | > loss_mel: 18.865520477294922 (19.710590734714415) | > loss_duration: 1.6349549293518066 (1.6593782843613043) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.49604606628418 (29.436736734901988) | > grad_norm_1: tensor(580.7853, device='cuda:0') (tensor(4319.8594, device='cuda:0')) | > current_lr_0: 0.0001858031471483524 | > current_lr_1: 0.0001858031471483524 | > step_time: 0.4476 (0.43806271436737804) | > loader_time: 0.0094 (0.0075039340228569215) --> EVAL PERFORMANCE | > avg_loader_time: 0.008648768277235434 (+1.914064649121555e-06) | > avg_step_time: 0.45419973051044304 (-0.0038336733697165726) | > avg_loss_disc: 2.741965498722774 (+0.04001264504983304) | > avg_loss_disc_real_0: 0.21762524744574452 (+0.055910615591515944) | > avg_loss_disc_real_1: 0.22645429994018984 (+0.007671225028978268) | > avg_loss_disc_real_2: 0.2376863530404131 (-0.0009429817888098879) | > avg_loss_disc_real_3: 0.23733724695695957 (+6.759124742425349e-05) | > avg_loss_disc_real_4: 0.24648312132962993 (+0.0034512822896662076) | > avg_loss_disc_real_5: 0.24618166601154165 (-0.0007054056500045425) | > avg_loss_0: 2.741965498722774 (+0.04001264504983304) | > avg_grad_norm_0: tensor(87.4366, device='cuda:0') (tensor(-39.3837, device='cuda:0')) | > avg_loss_gen: 2.0353212742738322 (+0.0465993008143466) | > avg_loss_kl: 2.497140036502355 (+0.43595302272850445) | > avg_loss_feat: 3.271176596762429 (-0.07298661957324404) | > avg_loss_mel: 19.11760982996981 (+1.821320574048542) | > avg_loss_duration: 1.6552119255065918 (+0.07053422592055636) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.576459696595098 (+2.3014204267045137) | > avg_grad_norm_1: tensor(4174.3022, device='cuda:0') (tensor(-2363.6040, device='cuda:0'))  > EPOCH: 590/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:16:50)   --> TIME: 2025-02-25 14:17:00 -- STEP: 19/72 -- GLOBAL_STEP: 1042500 | > loss_disc: 2.606344699859619 (2.7872621762125114) | > loss_disc_real_0: 0.06400565057992935 (0.1239215345367005) | > loss_disc_real_1: 0.3489336669445038 (0.37185002863407135) | > loss_disc_real_2: 0.2308865487575531 (0.24020117599713176) | > loss_disc_real_3: 0.2567068636417389 (0.23690901147691826) | > loss_disc_real_4: 0.2864393889904022 (0.2555834062789616) | > loss_disc_real_5: 0.24994418025016785 (0.2510111841716264) | > loss_0: 2.606344699859619 (2.7872621762125114) | > grad_norm_0: tensor(23.1547, device='cuda:0') (tensor(72.2490, device='cuda:0')) | > loss_gen: 2.446415901184082 (2.154251355873911) | > loss_kl: 2.146967649459839 (2.2789870563306307) | > loss_feat: 3.818157196044922 (3.2472838602567973) | > loss_mel: 17.84800148010254 (18.502779308118317) | > loss_duration: 1.5426591634750366 (1.5650863961169594) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.802200317382812 (27.74838808963173) | > grad_norm_1: tensor(5489.9082, device='cuda:0') (tensor(4961.9951, device='cuda:0')) | > current_lr_0: 0.00018577992175495886 | > current_lr_1: 0.00018577992175495886 | > step_time: 0.4284 (0.43449523574427557) | > loader_time: 0.0064 (0.006885101920679996)  --> TIME: 2025-02-25 14:17:24 -- STEP: 69/72 -- GLOBAL_STEP: 1042550 | > loss_disc: 2.7506296634674072 (2.7355227574058203) | > loss_disc_real_0: 0.19413775205612183 (0.1564740351792695) | > loss_disc_real_1: 0.29177427291870117 (0.26630689160547383) | > loss_disc_real_2: 0.287686288356781 (0.23634153150993845) | > loss_disc_real_3: 0.27918851375579834 (0.23657943718675253) | > loss_disc_real_4: 0.2307872325181961 (0.24890041588873102) | > loss_disc_real_5: 0.25532203912734985 (0.24990204430144766) | > loss_0: 2.7506296634674072 (2.7355227574058203) | > grad_norm_0: tensor(22.3812, device='cuda:0') (tensor(87.6065, device='cuda:0')) | > loss_gen: 1.8412132263183594 (1.9809869907904363) | > loss_kl: 2.249579429626465 (2.189976453781129) | > loss_feat: 2.510435104370117 (3.094595518665037) | > loss_mel: 17.800037384033203 (17.80961357337841) | > loss_duration: 1.686805248260498 (1.592873213947683) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.088069915771484 (26.668045790299125) | > grad_norm_1: tensor(573.0677, device='cuda:0') (tensor(4468.7383, device='cuda:0')) | > current_lr_0: 0.00018577992175495886 | > current_lr_1: 0.00018577992175495886 | > step_time: 0.538 (0.4562757326209027) | > loader_time: 0.0128 (0.008601948834847713) --> EVAL PERFORMANCE | > avg_loader_time: 0.008654691803623254 (+5.923526387820177e-06) | > avg_step_time: 0.4576810514423209 (+0.0034813209318778626) | > avg_loss_disc: 2.7361177793690854 (-0.005847719353688774) | > avg_loss_disc_real_0: 0.15723094732408793 (-0.060394300121656586) | > avg_loss_disc_real_1: 0.2643153623376093 (+0.03786106239741949) | > avg_loss_disc_real_2: 0.23739415470143438 (-0.0002921983389787275) | > avg_loss_disc_real_3: 0.2364364312568181 (-0.0009008157001414718) | > avg_loss_disc_real_4: 0.24867769068395587 (+0.002194569354325937) | > avg_loss_disc_real_5: 0.25015456386854945 (+0.003972897857007801) | > avg_loss_0: 2.7361177793690854 (-0.005847719353688774) | > avg_grad_norm_0: tensor(86.7511, device='cuda:0') (tensor(-0.6856, device='cuda:0')) | > avg_loss_gen: 1.9750231823451083 (-0.06029809192872393) | > avg_loss_kl: 2.192596935890091 (-0.30454310061226364) | > avg_loss_feat: 3.0886270160406406 (-0.18254958072178828) | > avg_loss_mel: 17.787651088875784 (-1.3299587410940248) | > avg_loss_duration: 1.5933763040623197 (-0.061835621444272126) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.63727456751004 (-1.9391851290850575) | > avg_grad_norm_1: tensor(4437.2007, device='cuda:0') (+tensor(262.8984, device='cuda:0'))  > EPOCH: 591/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:17:27)   --> TIME: 2025-02-25 14:17:51 -- STEP: 47/72 -- GLOBAL_STEP: 1042600 | > loss_disc: 2.732333183288574 (2.641530265199377) | > loss_disc_real_0: 0.17806476354599 (0.1331780125644613) | > loss_disc_real_1: 0.24097518622875214 (0.22372712099805792) | > loss_disc_real_2: 0.27143287658691406 (0.23168965508329106) | > loss_disc_real_3: 0.2789551615715027 (0.23970358834621755) | > loss_disc_real_4: 0.2188263237476349 (0.24548417868766378) | > loss_disc_real_5: 0.2655307948589325 (0.2444883071361704) | > loss_0: 2.732333183288574 (2.641530265199377) | > grad_norm_0: tensor(35.2689, device='cuda:0') (tensor(74.1988, device='cuda:0')) | > loss_gen: 2.120105028152466 (2.0280269409747835) | > loss_kl: 2.1121268272399902 (2.0892899492953685) | > loss_feat: 3.3429880142211914 (3.4828770363584476) | > loss_mel: 15.921140670776367 (17.671788317091924) | > loss_duration: 1.6087303161621094 (1.5717925036207159) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.105091094970703 (26.843774876696) | > grad_norm_1: tensor(4119.0493, device='cuda:0') (tensor(4815.9873, device='cuda:0')) | > current_lr_0: 0.00018575669926473948 | > current_lr_1: 0.00018575669926473948 | > step_time: 0.4554 (0.4514657842352035) | > loader_time: 0.0109 (0.007741283863148791) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696915398181322 (+4.2223594558068664e-05) | > avg_step_time: 0.46099158072135815 (+0.0033105292790372443) | > avg_loss_disc: 2.6718415878188444 (-0.064276191550241) | > avg_loss_disc_real_0: 0.14147796282466032 (-0.015752984499427608) | > avg_loss_disc_real_1: 0.22943197841375645 (-0.03488338392385287) | > avg_loss_disc_real_2: 0.231295892260444 (-0.006098262440990376) | > avg_loss_disc_real_3: 0.23916961294664463 (+0.0027331816898265304) | > avg_loss_disc_real_4: 0.24105630416265675 (-0.007621386521299123) | > avg_loss_disc_real_5: 0.2429064652869399 (-0.007248098581609547) | > avg_loss_0: 2.6718415878188444 (-0.064276191550241) | > avg_grad_norm_0: tensor(82.9697, device='cuda:0') (tensor(-3.7813, device='cuda:0')) | > avg_loss_gen: 1.9894915265096744 (+0.014468344164566105) | > avg_loss_kl: 2.102067099490636 (-0.09052983639945511) | > avg_loss_feat: 3.3860115971363767 (+0.29738458109573607) | > avg_loss_mel: 17.50371309737084 (-0.28393799150494203) | > avg_loss_duration: 1.5884267864092974 (-0.004949517653022317) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.569710261385207 (-0.067564306124833) | > avg_grad_norm_1: tensor(4666.8828, device='cuda:0') (+tensor(229.6821, device='cuda:0'))  > EPOCH: 592/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:18:04)   --> TIME: 2025-02-25 14:18:17 -- STEP: 25/72 -- GLOBAL_STEP: 1042650 | > loss_disc: 2.7413225173950195 (2.7240020561218263) | > loss_disc_real_0: 0.2129574716091156 (0.18159492880105974) | > loss_disc_real_1: 0.2038002908229828 (0.22262691259384154) | > loss_disc_real_2: 0.21395927667617798 (0.23953433930873871) | > loss_disc_real_3: 0.28294241428375244 (0.23385752230882642) | > loss_disc_real_4: 0.26318642497062683 (0.24313495099544524) | > loss_disc_real_5: 0.23674601316452026 (0.24226493060588836) | > loss_0: 2.7413225173950195 (2.7240020561218263) | > grad_norm_0: tensor(46.5113, device='cuda:0') (tensor(83.7203, device='cuda:0')) | > loss_gen: 1.9886815547943115 (1.9165286016464234) | > loss_kl: 3.135284423828125 (2.1717648029327394) | > loss_feat: 2.902930974960327 (3.100502653121948) | > loss_mel: 21.55248260498047 (17.628172149658198) | > loss_duration: 1.9198381900787354 (1.5590153169631957) | > amp_scaler: 16.0 (16.0) | > loss_1: 31.499217987060547 (26.375983200073243) | > grad_norm_1: tensor(1763.0312, device='cuda:0') (tensor(3380.1536, device='cuda:0')) | > current_lr_0: 0.00018573347967733138 | > current_lr_1: 0.00018573347967733138 | > step_time: 0.4413 (0.4434974193572998) | > loader_time: 0.008 (0.0071105480194091795) --> EVAL PERFORMANCE | > avg_loader_time: 0.008797588482709003 (+0.00010067308452768077) | > avg_step_time: 0.4599234923510484 (-0.00106808837030975) | > avg_loss_disc: 2.734066338606282 (+0.06222475078743761) | > avg_loss_disc_real_0: 0.18796132683334216 (+0.04648336400868183) | > avg_loss_disc_real_1: 0.23527735891476484 (+0.005845380501008385) | > avg_loss_disc_real_2: 0.24166457082184267 (+0.01036867856139867) | > avg_loss_disc_real_3: 0.23662205743537823 (-0.0025475555112663972) | > avg_loss_disc_real_4: 0.2445320123937768 (+0.0034757082311200616) | > avg_loss_disc_real_5: 0.24301371733907243 (+0.00010725205213252265) | > avg_loss_0: 2.734066338606282 (+0.06222475078743761) | > avg_grad_norm_0: tensor(68.6444, device='cuda:0') (tensor(-14.3254, device='cuda:0')) | > avg_loss_gen: 1.9861224822595085 (-0.0033690442501659312) | > avg_loss_kl: 2.5150819227729038 (+0.4130148232822677) | > avg_loss_feat: 3.1711180176533444 (-0.21489357948303223) | > avg_loss_mel: 18.886659756512717 (+1.3829466591418758) | > avg_loss_duration: 1.672580771043267 (+0.08415398463396961) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.231562815921407 (+1.6618525545361997) | > avg_grad_norm_1: tensor(3527.7083, device='cuda:0') (tensor(-1139.1746, device='cuda:0'))  > EPOCH: 593/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:18:41)   --> TIME: 2025-02-25 14:18:45 -- STEP: 3/72 -- GLOBAL_STEP: 1042700 | > loss_disc: 2.451791286468506 (2.5814477602640786) | > loss_disc_real_0: 0.06128475069999695 (0.09549245735009511) | > loss_disc_real_1: 0.2561953663825989 (0.21731909612814584) | > loss_disc_real_2: 0.23230265080928802 (0.19451365868250528) | > loss_disc_real_3: 0.25182655453681946 (0.21763751904169717) | > loss_disc_real_4: 0.23177240788936615 (0.21555674076080322) | > loss_disc_real_5: 0.2511535584926605 (0.2404384265343348) | > loss_0: 2.451791286468506 (2.5814477602640786) | > grad_norm_0: tensor(73.7111, device='cuda:0') (tensor(55.5186, device='cuda:0')) | > loss_gen: 2.326993942260742 (1.9322586854298909) | > loss_kl: 2.237107992172241 (2.0992827812830606) | > loss_feat: 3.8058414459228516 (3.5573047002156577) | > loss_mel: 18.140758514404297 (18.139376322428387) | > loss_duration: 1.6215786933898926 (1.60331396261851) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.132280349731445 (27.331536610921223) | > grad_norm_1: tensor(7509.7104, device='cuda:0') (tensor(4498.0322, device='cuda:0')) | > current_lr_0: 0.0001857102629923717 | > current_lr_1: 0.0001857102629923717 | > step_time: 0.4338 (0.4307490984598796) | > loader_time: 0.0059 (0.005416711171468099)  --> TIME: 2025-02-25 14:19:07 -- STEP: 53/72 -- GLOBAL_STEP: 1042750 | > loss_disc: 3.0369834899902344 (2.739581045114769) | > loss_disc_real_0: 0.03494850546121597 (0.14065815680572444) | > loss_disc_real_1: 0.18767066299915314 (0.23449377209510444) | > loss_disc_real_2: 0.2247980237007141 (0.24616678424601285) | > loss_disc_real_3: 0.23987191915512085 (0.23617294796232907) | > loss_disc_real_4: 0.24355512857437134 (0.250066191520331) | > loss_disc_real_5: 0.23457391560077667 (0.25988480448722845) | > loss_0: 3.0369834899902344 (2.739581045114769) | > grad_norm_0: tensor(41.9868, device='cuda:0') (tensor(83.2900, device='cuda:0')) | > loss_gen: 1.5244519710540771 (2.020814954109912) | > loss_kl: 2.3963232040405273 (2.772152653280294) | > loss_feat: 2.5260658264160156 (3.424779568078383) | > loss_mel: 18.1242618560791 (20.337932802596182) | > loss_duration: 1.673635482788086 (1.653720848965195) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.24473762512207 (30.209400644842184) | > grad_norm_1: tensor(1049.8665, device='cuda:0') (tensor(4110.8896, device='cuda:0')) | > current_lr_0: 0.0001857102629923717 | > current_lr_1: 0.0001857102629923717 | > step_time: 0.4426 (0.4409526654009549) | > loader_time: 0.0093 (0.007775869009629736) --> EVAL PERFORMANCE | > avg_loader_time: 0.008592132111670269 (-0.00020545637103873415) | > avg_step_time: 0.45551371574401855 (-0.004409776607029847) | > avg_loss_disc: 2.707667542175508 (-0.02639879643077414) | > avg_loss_disc_real_0: 0.1411457935112043 (-0.046815533322137864) | > avg_loss_disc_real_1: 0.2339654762140462 (-0.0013118827007186362) | > avg_loss_disc_real_2: 0.2466246332501022 (+0.004960062428259526) | > avg_loss_disc_real_3: 0.23183098015650896 (-0.004791077278869266) | > avg_loss_disc_real_4: 0.24670697580760634 (+0.002174963413829528) | > avg_loss_disc_real_5: 0.2548291263446001 (+0.01181540900552766) | > avg_loss_0: 2.707667542175508 (-0.02639879643077414) | > avg_grad_norm_0: tensor(82.8224, device='cuda:0') (+tensor(14.1780, device='cuda:0')) | > avg_loss_gen: 2.0281297824752165 (+0.04200730021570798) | > avg_loss_kl: 2.666006239367203 (+0.15092431659429906) | > avg_loss_feat: 3.366130973251772 (+0.19501295559842768) | > avg_loss_mel: 19.783986830375564 (+0.8973270738628472) | > avg_loss_duration: 1.6516097414661461 (-0.020971029577120825) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.495863444368606 (+1.2643006284471987) | > avg_grad_norm_1: tensor(4327.3867, device='cuda:0') (+tensor(799.6785, device='cuda:0'))  > EPOCH: 594/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:19:18)   --> TIME: 2025-02-25 14:19:35 -- STEP: 31/72 -- GLOBAL_STEP: 1042800 | > loss_disc: 2.5962281227111816 (2.7338136165372786) | > loss_disc_real_0: 0.2075183093547821 (0.17290494350656385) | > loss_disc_real_1: 0.22835664451122284 (0.2243894886585974) | > loss_disc_real_2: 0.23950546979904175 (0.23020887999765335) | > loss_disc_real_3: 0.25296536087989807 (0.22805350442086497) | > loss_disc_real_4: 0.2337026745080948 (0.2402149736881256) | > loss_disc_real_5: 0.23547938466072083 (0.24225151058166258) | > loss_0: 2.5962281227111816 (2.7338136165372786) | > grad_norm_0: tensor(22.4452, device='cuda:0') (tensor(81.9378, device='cuda:0')) | > loss_gen: 2.3679230213165283 (1.8846545642422092) | > loss_kl: 2.203735589981079 (2.1058940002995143) | > loss_feat: 3.3612048625946045 (3.0758085712309806) | > loss_mel: 17.452974319458008 (17.905316814299557) | > loss_duration: 1.6026078462600708 (1.565430183564463) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.988445281982422 (26.537104391282604) | > grad_norm_1: tensor(4561.5391, device='cuda:0') (tensor(3776.9263, device='cuda:0')) | > current_lr_0: 0.00018568704920949764 | > current_lr_1: 0.00018568704920949764 | > step_time: 0.4359 (0.4505255606866652) | > loader_time: 0.0082 (0.006779401533065304) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534864640571701 (-5.7267471098567876e-05) | > avg_step_time: 0.4606520961707746 (+0.005138380426756073) | > avg_loss_disc: 2.757779668754256 (+0.050112126578748306) | > avg_loss_disc_real_0: 0.19735933126698077 (+0.05621353775577648) | > avg_loss_disc_real_1: 0.22107964638673083 (-0.01288582982731537) | > avg_loss_disc_real_2: 0.2422583327746727 (-0.004366300475429491) | > avg_loss_disc_real_3: 0.23045454688475164 (-0.0013764332717573213) | > avg_loss_disc_real_4: 0.2424146140545187 (-0.004292361753087642) | > avg_loss_disc_real_5: 0.2428422406525679 (-0.011986885692032184) | > avg_loss_0: 2.757779668754256 (+0.050112126578748306) | > avg_grad_norm_0: tensor(89.2446, device='cuda:0') (+tensor(6.4223, device='cuda:0')) | > avg_loss_gen: 1.9370933855083627 (-0.09103639696685373) | > avg_loss_kl: 2.12170156122933 (-0.5443046781378729) | > avg_loss_feat: 3.0632113302257693 (-0.3029196430260028) | > avg_loss_mel: 17.673677484754105 (-2.1103093456214594) | > avg_loss_duration: 1.5944384689062414 (-0.05717127255990473) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.3901221852907 (-3.105741259077906) | > avg_grad_norm_1: tensor(3903.1562, device='cuda:0') (tensor(-424.2305, device='cuda:0'))  > EPOCH: 595/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:19:56)   --> TIME: 2025-02-25 14:20:01 -- STEP: 9/72 -- GLOBAL_STEP: 1042850 | > loss_disc: 2.548373222351074 (2.60148024559021) | > loss_disc_real_0: 0.10505945235490799 (0.1840029690000746) | > loss_disc_real_1: 0.16964757442474365 (0.20747315221362644) | > loss_disc_real_2: 0.22522801160812378 (0.22699170642428929) | > loss_disc_real_3: 0.2297893762588501 (0.2274475246667862) | > loss_disc_real_4: 0.2504406273365021 (0.245983996325069) | > loss_disc_real_5: 0.23564627766609192 (0.23305030332671273) | > loss_0: 2.548373222351074 (2.60148024559021) | > grad_norm_0: tensor(62.4881, device='cuda:0') (tensor(124.4592, device='cuda:0')) | > loss_gen: 1.92075777053833 (2.0572633743286133) | > loss_kl: 1.973114013671875 (2.0535688267813788) | > loss_feat: 3.5326626300811768 (3.3888648880852594) | > loss_mel: 17.7259464263916 (17.557742648654514) | > loss_duration: 1.5563075542449951 (1.5598181751039293) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.70878791809082 (26.61725828382704) | > grad_norm_1: tensor(6310.5244, device='cuda:0') (tensor(5722.3267, device='cuda:0')) | > current_lr_0: 0.00018566383832834644 | > current_lr_1: 0.00018566383832834644 | > step_time: 0.4186 (0.42835338910420734) | > loader_time: 0.0055 (0.00640416145324707)  --> TIME: 2025-02-25 14:20:25 -- STEP: 59/72 -- GLOBAL_STEP: 1042900 | > loss_disc: 2.44476580619812 (2.6436013730905814) | > loss_disc_real_0: 0.17091472446918488 (0.13804525585245273) | > loss_disc_real_1: 0.2543787658214569 (0.2195570931091147) | > loss_disc_real_2: 0.22905407845973969 (0.24242291334321944) | > loss_disc_real_3: 0.225996732711792 (0.22961620304544092) | > loss_disc_real_4: 0.2504113018512726 (0.2478988761618986) | > loss_disc_real_5: 0.27734044194221497 (0.2530141192472587) | > loss_0: 2.44476580619812 (2.6436013730905814) | > grad_norm_0: tensor(78.1992, device='cuda:0') (tensor(86.9819, device='cuda:0')) | > loss_gen: 2.907777786254883 (2.102363750085993) | > loss_kl: 2.410891056060791 (2.3442136170500416) | > loss_feat: 4.474660396575928 (3.520683979583999) | > loss_mel: 20.132137298583984 (19.44195107282218) | > loss_duration: 1.6626543998718262 (1.6372087001800537) | > amp_scaler: 16.0 (16.0) | > loss_1: 31.58812141418457 (29.04642124499305) | > grad_norm_1: tensor(6353.4097, device='cuda:0') (tensor(5372.1851, device='cuda:0')) | > current_lr_0: 0.00018566383832834644 | > current_lr_1: 0.00018566383832834644 | > step_time: 0.4821 (0.44539238234697764) | > loader_time: 0.0098 (0.007979498071185611) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853757791116204 (+2.7132705903380044e-06) | > avg_step_time: 0.45407590731768543 (-0.006576188853089193) | > avg_loss_disc: 2.64206967891102 (-0.1157099898432361) | > avg_loss_disc_real_0: 0.1397067698794351 (-0.05765256138754568) | > avg_loss_disc_real_1: 0.22069571723400708 (-0.0003839291527237554) | > avg_loss_disc_real_2: 0.24038808005796353 (-0.001870252716709181) | > avg_loss_disc_real_3: 0.2292567166224332 (-0.0011978302623184556) | > avg_loss_disc_real_4: 0.24631657461884995 (+0.0039019605643312583) | > avg_loss_disc_real_5: 0.2531617241846004 (+0.01031948353203252) | > avg_loss_0: 2.64206967891102 (-0.1157099898432361) | > avg_grad_norm_0: tensor(88.7891, device='cuda:0') (tensor(-0.4555, device='cuda:0')) | > avg_loss_gen: 2.079685520118391 (+0.1425921346100283) | > avg_loss_kl: 2.3445570888653604 (+0.22285552763603045) | > avg_loss_feat: 3.4347144046299896 (+0.37150307440422026) | > avg_loss_mel: 19.286027881461134 (+1.6123503967070292) | > avg_loss_duration: 1.640265703201294 (+0.04582723429505253) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.785250730917486 (+2.395128545626786) | > avg_grad_norm_1: tensor(5232.5703, device='cuda:0') (+tensor(1329.4141, device='cuda:0'))  > EPOCH: 596/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:20:32)   --> TIME: 2025-02-25 14:20:51 -- STEP: 37/72 -- GLOBAL_STEP: 1042950 | > loss_disc: 2.775513172149658 (2.652660994916349) | > loss_disc_real_0: 0.4041045308113098 (0.16072720218751887) | > loss_disc_real_1: 0.2188994586467743 (0.22374783657692573) | > loss_disc_real_2: 0.23913653194904327 (0.22737974895013346) | > loss_disc_real_3: 0.2376166731119156 (0.23099660833139676) | > loss_disc_real_4: 0.23302769660949707 (0.2367273351630649) | > loss_disc_real_5: 0.23519417643547058 (0.2386009487751368) | > loss_0: 2.775513172149658 (2.652660994916349) | > grad_norm_0: tensor(206.5445, device='cuda:0') (tensor(97.9493, device='cuda:0')) | > loss_gen: 2.4505786895751953 (2.0191014747361877) | > loss_kl: 1.9517616033554077 (2.0705975680737874) | > loss_feat: 2.9124176502227783 (3.3016616911501497) | > loss_mel: 17.01661491394043 (17.69694828342747) | > loss_duration: 1.596963882446289 (1.564064351288048) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.92833709716797 (26.652373545878643) | > grad_norm_1: tensor(6002.4189, device='cuda:0') (tensor(5443.6338, device='cuda:0')) | > current_lr_0: 0.00018564063034855538 | > current_lr_1: 0.00018564063034855538 | > step_time: 0.4427 (0.43784861951260956) | > loader_time: 0.0082 (0.007262816300263276) --> EVAL PERFORMANCE | > avg_loader_time: 0.008597494850695973 (+5.991693953393343e-05) | > avg_step_time: 0.4542701781635553 (+0.00019427084586987053) | > avg_loss_disc: 2.705471606321738 (+0.06340192741071782) | > avg_loss_disc_real_0: 0.20262000195577112 (+0.06291323207633603) | > avg_loss_disc_real_1: 0.22435015450480958 (+0.003654437270802502) | > avg_loss_disc_real_2: 0.2305760723604283 (-0.009812007697535235) | > avg_loss_disc_real_3: 0.23488390781510043 (+0.005627191192667247) | > avg_loss_disc_real_4: 0.23937669669238615 (-0.0069398779264638055) | > avg_loss_disc_real_5: 0.24128713406307598 (-0.011874590121524448) | > avg_loss_0: 2.705471606321738 (+0.06340192741071782) | > avg_grad_norm_0: tensor(104.6866, device='cuda:0') (+tensor(15.8975, device='cuda:0')) | > avg_loss_gen: 1.9900097712664533 (-0.08967574885193774) | > avg_loss_kl: 2.0677495623978084 (-0.276807526467552) | > avg_loss_feat: 3.1575641799980487 (-0.27715022463194083) | > avg_loss_mel: 17.535042507547733 (-1.7509853739134016) | > avg_loss_duration: 1.5920587442290617 (-0.04820695897223226) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.34242471506898 (-2.4428260158485067) | > avg_grad_norm_1: tensor(4865.7690, device='cuda:0') (tensor(-366.8013, device='cuda:0'))  > EPOCH: 597/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:21:09)   --> TIME: 2025-02-25 14:21:18 -- STEP: 15/72 -- GLOBAL_STEP: 1043000 | > loss_disc: 2.473724603652954 (2.709552844365438) | > loss_disc_real_0: 0.17077454924583435 (0.1329925894737244) | > loss_disc_real_1: 0.22936822474002838 (0.21673123637835184) | > loss_disc_real_2: 0.2185428887605667 (0.2593179414669673) | > loss_disc_real_3: 0.23482003808021545 (0.25663018027941387) | > loss_disc_real_4: 0.2634386718273163 (0.2528022954861323) | > loss_disc_real_5: 0.25365737080574036 (0.24768342971801757) | > loss_0: 2.473724603652954 (2.709552844365438) | > grad_norm_0: tensor(94.7368, device='cuda:0') (tensor(98.6474, device='cuda:0')) | > loss_gen: 2.3150646686553955 (2.0595229228337604) | > loss_kl: 2.1577308177948 (2.049712928136189) | > loss_feat: 3.7059149742126465 (3.3959541161855062) | > loss_mel: 19.85895347595215 (17.655325571695965) | > loss_duration: 1.6446218490600586 (1.573051683108012) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.68228530883789 (26.73356704711914) | > grad_norm_1: tensor(6083.6611, device='cuda:0') (tensor(4572.9375, device='cuda:0')) | > current_lr_0: 0.0001856174252697618 | > current_lr_1: 0.0001856174252697618 | > step_time: 0.4228 (0.4321181297302246) | > loader_time: 0.0058 (0.00706793467203776)  --> TIME: 2025-02-25 14:21:41 -- STEP: 65/72 -- GLOBAL_STEP: 1043050 | > loss_disc: 2.645770788192749 (2.6798085616185117) | > loss_disc_real_0: 0.15489903092384338 (0.14139037011907649) | > loss_disc_real_1: 0.1845630705356598 (0.22711259974883152) | > loss_disc_real_2: 0.24067862331867218 (0.2396883643590487) | > loss_disc_real_3: 0.2331009954214096 (0.23911570608615876) | > loss_disc_real_4: 0.22674763202667236 (0.24717917281847734) | > loss_disc_real_5: 0.22042083740234375 (0.24829816657763262) | > loss_0: 2.645770788192749 (2.6798085616185117) | > grad_norm_0: tensor(121.1710, device='cuda:0') (tensor(83.4886, device='cuda:0')) | > loss_gen: 1.7861841917037964 (2.0053230817501366) | > loss_kl: 2.706031322479248 (2.1524041010783264) | > loss_feat: 3.0321028232574463 (3.294135464154757) | > loss_mel: 20.960308074951172 (17.966645167424137) | > loss_duration: 1.6864956617355347 (1.5958050067608174) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.17112159729004 (27.014312920203576) | > grad_norm_1: tensor(6042.4722, device='cuda:0') (tensor(4737.6523, device='cuda:0')) | > current_lr_0: 0.0001856174252697618 | > current_lr_1: 0.0001856174252697618 | > step_time: 0.5204 (0.44711301510150614) | > loader_time: 0.0117 (0.0084197447850154) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688500229741486 (+9.100537904551363e-05) | > avg_step_time: 0.4532470635964837 (-0.0010231145670716035) | > avg_loss_disc: 2.6805963751295923 (-0.02487523119214563) | > avg_loss_disc_real_0: 0.14220850384781056 (-0.060411498107960565) | > avg_loss_disc_real_1: 0.22582391366152696 (+0.0014737591567173802) | > avg_loss_disc_real_2: 0.23904620236913923 (+0.008470130008710941) | > avg_loss_disc_real_3: 0.2372092520686942 (+0.002325344253593764) | > avg_loss_disc_real_4: 0.2465439711658048 (+0.007167274473418639) | > avg_loss_disc_real_5: 0.2472905974153062 (+0.006003463352230232) | > avg_loss_0: 2.6805963751295923 (-0.02487523119214563) | > avg_grad_norm_0: tensor(82.9151, device='cuda:0') (tensor(-21.7715, device='cuda:0')) | > avg_loss_gen: 2.011434877422494 (+0.021425106156040874) | > avg_loss_kl: 2.187399224496223 (+0.11964966209841466) | > avg_loss_feat: 3.282457533017011 (+0.1248933530189622) | > avg_loss_mel: 18.129474129475344 (+0.5944316219276118) | > avg_loss_duration: 1.6021666795435086 (+0.01010793531444687) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.21293247921366 (+0.8705077641446799) | > avg_grad_norm_1: tensor(4708.3711, device='cuda:0') (tensor(-157.3979, device='cuda:0'))  > EPOCH: 598/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:21:46)   --> TIME: 2025-02-25 14:22:08 -- STEP: 43/72 -- GLOBAL_STEP: 1043100 | > loss_disc: 2.7143349647521973 (2.6209515915360564) | > loss_disc_real_0: 0.09364771097898483 (0.10791985331059889) | > loss_disc_real_1: 0.20820021629333496 (0.21397327822308207) | > loss_disc_real_2: 0.27419084310531616 (0.2259433169004529) | > loss_disc_real_3: 0.2475949376821518 (0.24301104178262312) | > loss_disc_real_4: 0.2500443756580353 (0.24602998342624927) | > loss_disc_real_5: 0.24061131477355957 (0.251250871045645) | > loss_0: 2.7143349647521973 (2.6209515915360564) | > grad_norm_0: tensor(110.5252, device='cuda:0') (tensor(93.2577, device='cuda:0')) | > loss_gen: 1.9469412565231323 (2.062846327936927) | > loss_kl: 1.9839040040969849 (2.070415158604466) | > loss_feat: 3.0801584720611572 (3.5996622928353244) | > loss_mel: 16.06427764892578 (17.762065222096993) | > loss_duration: 1.6107045412063599 (1.5676619202591653) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.685985565185547 (27.062650813612827) | > grad_norm_1: tensor(6989.1890, device='cuda:0') (tensor(5817.5459, device='cuda:0')) | > current_lr_0: 0.00018559422309160308 | > current_lr_1: 0.00018559422309160308 | > step_time: 0.4667 (0.4539895168570585) | > loader_time: 0.0096 (0.0075263478035150575) --> EVAL PERFORMANCE | > avg_loader_time: 0.008665558318017234 (-2.294191172425193e-05) | > avg_step_time: 0.46562040020042744 (+0.012373336603943741) | > avg_loss_disc: 2.6316937124225457 (-0.048902662707046574) | > avg_loss_disc_real_0: 0.10857825860066313 (-0.03363024524714743) | > avg_loss_disc_real_1: 0.21786787770163846 (-0.007956035959888502) | > avg_loss_disc_real_2: 0.22913724435886867 (-0.009908958010270558) | > avg_loss_disc_real_3: 0.23857464924664565 (+0.0013653971779514495) | > avg_loss_disc_real_4: 0.24305100739002228 (-0.0034929637757825094) | > avg_loss_disc_real_5: 0.24962734546459897 (+0.002336748049292764) | > avg_loss_0: 2.6316937124225457 (-0.048902662707046574) | > avg_grad_norm_0: tensor(101.1431, device='cuda:0') (+tensor(18.2279, device='cuda:0')) | > avg_loss_gen: 2.0436214242182995 (+0.03218654679580535) | > avg_loss_kl: 2.0734301251424867 (-0.11396909935373634) | > avg_loss_feat: 3.5374227208150946 (+0.25496518779808364) | > avg_loss_mel: 17.50265814552844 (-0.626815983946905) | > avg_loss_duration: 1.5877566236845204 (-0.014410055858988136) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.744888842945368 (-0.46804363626829115) | > avg_grad_norm_1: tensor(5914.6904, device='cuda:0') (+tensor(1206.3193, device='cuda:0'))  > EPOCH: 599/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:22:24)   --> TIME: 2025-02-25 14:22:35 -- STEP: 21/72 -- GLOBAL_STEP: 1043150 | > loss_disc: 2.613964080810547 (2.570262012027559) | > loss_disc_real_0: 0.14148618280887604 (0.11396775678509757) | > loss_disc_real_1: 0.23182988166809082 (0.2241875564768201) | > loss_disc_real_2: 0.2796417474746704 (0.22296944899218424) | > loss_disc_real_3: 0.25642624497413635 (0.2355002143553325) | > loss_disc_real_4: 0.2539466619491577 (0.2412594443275815) | > loss_disc_real_5: 0.2540730834007263 (0.24305575163591475) | > loss_0: 2.613964080810547 (2.570262012027559) | > grad_norm_0: tensor(160.8141, device='cuda:0') (tensor(114.4780, device='cuda:0')) | > loss_gen: 2.1873936653137207 (2.1204977319354095) | > loss_kl: 1.856950044631958 (1.992078111285255) | > loss_feat: 3.414846420288086 (3.642961343129476) | > loss_mel: 17.067293167114258 (17.296805790492467) | > loss_duration: 1.5433045625686646 (1.5384005989347185) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.069786071777344 (26.59074338277181) | > grad_norm_1: tensor(5162.8760, device='cuda:0') (tensor(6935.5024, device='cuda:0')) | > current_lr_0: 0.0001855710238137166 | > current_lr_1: 0.0001855710238137166 | > step_time: 0.4422 (0.43813053766886395) | > loader_time: 0.0064 (0.006707316353207543)  --> TIME: 2025-02-25 14:22:59 -- STEP: 71/72 -- GLOBAL_STEP: 1043200 | > loss_disc: 2.682216167449951 (2.604923328883211) | > loss_disc_real_0: 0.06561253219842911 (0.11550117861217177) | > loss_disc_real_1: 0.23882924020290375 (0.22429425426771943) | > loss_disc_real_2: 0.24580417573451996 (0.22727551447673583) | > loss_disc_real_3: 0.2264062762260437 (0.23533656857383084) | > loss_disc_real_4: 0.2491127997636795 (0.24054258054410907) | > loss_disc_real_5: 0.2381902039051056 (0.24300818292187973) | > loss_0: 2.682216167449951 (2.604923328883211) | > grad_norm_0: tensor(90.1309, device='cuda:0') (tensor(108.2873, device='cuda:0')) | > loss_gen: 1.8466451168060303 (2.081050132361935) | > loss_kl: 1.9013081789016724 (2.041633857807643) | > loss_feat: 3.0930187702178955 (3.527256780946758) | > loss_mel: 17.2819881439209 (17.03674959800612) | > loss_duration: 1.6006509065628052 (1.576326418930376) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.72361183166504 (26.263016794768856) | > grad_norm_1: tensor(5227.1523, device='cuda:0') (tensor(6556.5903, device='cuda:0')) | > current_lr_0: 0.0001855710238137166 | > current_lr_1: 0.0001855710238137166 | > step_time: 0.4902 (0.4575795892258765) | > loader_time: 0.0081 (0.0086039791644459) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086039791644459 (-6.157915357133358e-05) | > avg_step_time: 0.4575795892258765 (-0.00804081097455095) | > avg_loss_disc: 2.604923328883211 (-0.026770383539334475) | > avg_loss_disc_real_0: 0.11550117861217177 (+0.00692292001150864) | > avg_loss_disc_real_1: 0.22429425426771943 (+0.006426376566080971) | > avg_loss_disc_real_2: 0.22727551447673583 (-0.0018617298821328454) | > avg_loss_disc_real_3: 0.23533656857383084 (-0.003238080672814808) | > avg_loss_disc_real_4: 0.24054258054410907 (-0.0025084268459132086) | > avg_loss_disc_real_5: 0.24300818292187973 (-0.006619162542719242) | > avg_loss_0: 2.604923328883211 (-0.026770383539334475) | > avg_grad_norm_0: tensor(108.2873, device='cuda:0') (+tensor(7.1443, device='cuda:0')) | > avg_loss_gen: 2.081050132361935 (+0.037428708143635525) | > avg_loss_kl: 2.041633857807643 (-0.03179626733484353) | > avg_loss_feat: 3.527256780946758 (-0.010165939868336693) | > avg_loss_mel: 17.03674959800612 (-0.4659085475223179) | > avg_loss_duration: 1.576326418930376 (-0.011430204754144357) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.263016794768856 (-0.48187204817651264) | > avg_grad_norm_1: tensor(6556.5903, device='cuda:0') (+tensor(641.8999, device='cuda:0'))  > EPOCH: 600/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:23:01)   --> TIME: 2025-02-25 14:23:25 -- STEP: 49/72 -- GLOBAL_STEP: 1043250 | > loss_disc: 2.5495493412017822 (2.5883505295734004) | > loss_disc_real_0: 0.08521458506584167 (0.10074392348832013) | > loss_disc_real_1: 0.17973844707012177 (0.22181464579640603) | > loss_disc_real_2: 0.2012801170349121 (0.2295973769256047) | > loss_disc_real_3: 0.22245338559150696 (0.2377957567876699) | > loss_disc_real_4: 0.24305173754692078 (0.2455571309036138) | > loss_disc_real_5: 0.250202476978302 (0.24532703568740766) | > loss_0: 2.5495493412017822 (2.5883505295734004) | > grad_norm_0: tensor(99.0082, device='cuda:0') (tensor(119.8296, device='cuda:0')) | > loss_gen: 2.107672929763794 (2.098171968849338) | > loss_kl: 1.9947031736373901 (2.0052736413722134) | > loss_feat: 3.6890361309051514 (3.6437328834922944) | > loss_mel: 16.9591121673584 (16.981564969432593) | > loss_duration: 1.6383017301559448 (1.5569720024965248) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.388826370239258 (26.285715336702307) | > grad_norm_1: tensor(6161.4775, device='cuda:0') (tensor(6892.4023, device='cuda:0')) | > current_lr_0: 0.0001855478274357399 | > current_lr_1: 0.0001855478274357399 | > step_time: 0.446 (0.44234856780694454) | > loader_time: 0.0094 (0.007568899466066944) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514568839274662 (-8.941032517123873e-05) | > avg_step_time: 0.4553578873755227 (-0.0022217018503538077) | > avg_loss_disc: 2.591516776823661 (-0.013406552059550325) | > avg_loss_disc_real_0: 0.10781332509408534 (-0.007687853518086429) | > avg_loss_disc_real_1: 0.22193439808529866 (-0.0023598561824207664) | > avg_loss_disc_real_2: 0.22717148653218444 (-0.00010402794455138809) | > avg_loss_disc_real_3: 0.23541326862825473 (+7.670005442389494e-05) | > avg_loss_disc_real_4: 0.24469993924590902 (+0.004157358701799951) | > avg_loss_disc_real_5: 0.2463180534016918 (+0.0033098704798120715) | > avg_loss_0: 2.591516776823661 (-0.013406552059550325) | > avg_grad_norm_0: tensor(119.7435, device='cuda:0') (+tensor(11.4562, device='cuda:0')) | > avg_loss_gen: 2.089244750184073 (+0.008194617822137928) | > avg_loss_kl: 2.0325186370124286 (-0.00911522079521454) | > avg_loss_feat: 3.5309414225564875 (+0.0036846416097295887) | > avg_loss_mel: 16.98665371747084 (-0.05009588053528091) | > avg_loss_duration: 1.5756252097411894 (-0.0007012091891867023) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.214983684915893 (-0.0480331098529625) | > avg_grad_norm_1: tensor(6758.1265, device='cuda:0') (+tensor(201.5361, device='cuda:0'))  > EPOCH: 601/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:23:38)   --> TIME: 2025-02-25 14:23:52 -- STEP: 27/72 -- GLOBAL_STEP: 1043300 | > loss_disc: 2.718926191329956 (2.5764837441621005) | > loss_disc_real_0: 0.16553576290607452 (0.11657505620408941) | > loss_disc_real_1: 0.23023530840873718 (0.21800454016085025) | > loss_disc_real_2: 0.21593892574310303 (0.2258003663133692) | > loss_disc_real_3: 0.18650425970554352 (0.2292504459619522) | > loss_disc_real_4: 0.21223995089530945 (0.23901315640520165) | > loss_disc_real_5: 0.23644857108592987 (0.24396415054798126) | > loss_0: 2.718926191329956 (2.5764837441621005) | > grad_norm_0: tensor(150.0319, device='cuda:0') (tensor(123.5781, device='cuda:0')) | > loss_gen: 1.8679871559143066 (2.1112021605173745) | > loss_kl: 2.0645062923431396 (1.959990558800874) | > loss_feat: 3.159226655960083 (3.629757598594383) | > loss_mel: 16.86487579345703 (17.024893513432254) | > loss_duration: 1.518040418624878 (1.5326659149593778) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.474634170532227 (26.25850995381673) | > grad_norm_1: tensor(6100.3647, device='cuda:0') (tensor(7109.5718, device='cuda:0')) | > current_lr_0: 0.00018552463395731043 | > current_lr_1: 0.00018552463395731043 | > step_time: 0.4376 (0.43494719046133534) | > loader_time: 0.0081 (0.006830162472195095) --> EVAL PERFORMANCE | > avg_loader_time: 0.008540788166959521 (+2.621932768485935e-05) | > avg_step_time: 0.453963964757785 (-0.0013939226177376862) | > avg_loss_disc: 2.590591941081302 (-0.0009248357423587983) | > avg_loss_disc_real_0: 0.11363039648448917 (+0.005817071390403827) | > avg_loss_disc_real_1: 0.22114646497746587 (-0.0007879331078327889) | > avg_loss_disc_real_2: 0.22875632681477237 (+0.0015848402825879293) | > avg_loss_disc_real_3: 0.2337925291397202 (-0.0016207394885345283) | > avg_loss_disc_real_4: 0.24093438810865644 (-0.0037655511372525763) | > avg_loss_disc_real_5: 0.2454919540126559 (-0.0008260993890359114) | > avg_loss_0: 2.590591941081302 (-0.0009248357423587983) | > avg_grad_norm_0: tensor(123.1533, device='cuda:0') (+tensor(3.4097, device='cuda:0')) | > avg_loss_gen: 2.097105011134081 (+0.007860260950008247) | > avg_loss_kl: 2.0147381903420034 (-0.01778044667042522) | > avg_loss_feat: 3.5355725624191927 (+0.004631139862705247) | > avg_loss_mel: 16.930974490206015 (-0.055679227264825215) | > avg_loss_duration: 1.5748017153269809 (-0.000823494414208481) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.153191982860296 (-0.06179170205559714) | > avg_grad_norm_1: tensor(7146.1885, device='cuda:0') (+tensor(388.0620, device='cuda:0'))  > EPOCH: 602/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:24:15)   --> TIME: 2025-02-25 14:24:19 -- STEP: 5/72 -- GLOBAL_STEP: 1043350 | > loss_disc: 2.6855955123901367 (2.5593098640441894) | > loss_disc_real_0: 0.10639417171478271 (0.08608592301607132) | > loss_disc_real_1: 0.2075924426317215 (0.22319495379924775) | > loss_disc_real_2: 0.3408981263637543 (0.2571606934070587) | > loss_disc_real_3: 0.27654168009757996 (0.24147965312004088) | > loss_disc_real_4: 0.24992188811302185 (0.23623879253864288) | > loss_disc_real_5: 0.23558904230594635 (0.23047060668468475) | > loss_0: 2.6855955123901367 (2.5593098640441894) | > grad_norm_0: tensor(103.5217, device='cuda:0') (tensor(124.0892, device='cuda:0')) | > loss_gen: 2.424701452255249 (2.2911774635314943) | > loss_kl: 2.028446912765503 (1.960344123840332) | > loss_feat: 4.1516642570495605 (3.9738964080810546) | > loss_mel: 17.37746238708496 (17.559546661376952) | > loss_duration: 1.5078457593917847 (1.5461873769760133) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.49011993408203 (27.33115234375) | > grad_norm_1: tensor(7175.9385, device='cuda:0') (tensor(8586.5205, device='cuda:0')) | > current_lr_0: 0.00018550144337806576 | > current_lr_1: 0.00018550144337806576 | > step_time: 0.4367 (0.43185720443725584) | > loader_time: 0.0083 (0.006879472732543945)  --> TIME: 2025-02-25 14:24:42 -- STEP: 55/72 -- GLOBAL_STEP: 1043400 | > loss_disc: 2.767956018447876 (2.6047618172385474) | > loss_disc_real_0: 0.07202955335378647 (0.11205818720839239) | > loss_disc_real_1: 0.19742216169834137 (0.22362815737724304) | > loss_disc_real_2: 0.21325862407684326 (0.23005943487990987) | > loss_disc_real_3: 0.2686195373535156 (0.2350977748632431) | > loss_disc_real_4: 0.26058363914489746 (0.2418334343216636) | > loss_disc_real_5: 0.2621974050998688 (0.24569056332111358) | > loss_0: 2.767956018447876 (2.6047618172385474) | > grad_norm_0: tensor(100.2675, device='cuda:0') (tensor(105.7975, device='cuda:0')) | > loss_gen: 1.8223769664764404 (2.1694357568567444) | > loss_kl: 2.1321940422058105 (2.0803262537175957) | > loss_feat: 2.9153032302856445 (3.7177900314331054) | > loss_mel: 16.8773250579834 (17.18700459220193) | > loss_duration: 1.6105903387069702 (1.567026400566101) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.3577880859375 (26.72158317565918) | > grad_norm_1: tensor(7195.0820, device='cuda:0') (tensor(7416.3491, device='cuda:0')) | > current_lr_0: 0.00018550144337806576 | > current_lr_1: 0.00018550144337806576 | > step_time: 0.4772 (0.4464064771478826) | > loader_time: 0.0093 (0.007965239611538975) --> EVAL PERFORMANCE | > avg_loader_time: 0.008674964098863203 (+0.00013417593190368123) | > avg_step_time: 0.4577395479444047 (+0.003775583186619691) | > avg_loss_disc: 2.6043934553441868 (+0.013801514262884673) | > avg_loss_disc_real_0: 0.11445742530722014 (+0.0008270288227309741) | > avg_loss_disc_real_1: 0.22667677574594255 (+0.005530310768476682) | > avg_loss_disc_real_2: 0.22835244238376617 (-0.0004038844310061962) | > avg_loss_disc_real_3: 0.23463378028130868 (+0.0008412511415884782) | > avg_loss_disc_real_4: 0.24149849297295153 (+0.0005641048642950897) | > avg_loss_disc_real_5: 0.24554797356397334 (+5.60195513174544e-05) | > avg_loss_0: 2.6043934553441868 (+0.013801514262884673) | > avg_grad_norm_0: tensor(111.7028, device='cuda:0') (tensor(-11.4505, device='cuda:0')) | > avg_loss_gen: 2.1472684010653427 (+0.05016338993126146) | > avg_loss_kl: 2.096509874706537 (+0.08177168436453375) | > avg_loss_feat: 3.6334770800362173 (+0.09790451761702457) | > avg_loss_mel: 17.165637768490217 (+0.2346632782842022) | > avg_loss_duration: 1.5804275878718201 (+0.005625872544839261) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.623320754145233 (+0.47012877128493713) | > avg_grad_norm_1: tensor(7259.2231, device='cuda:0') (+tensor(113.0347, device='cuda:0'))  > EPOCH: 603/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:24:52)   --> TIME: 2025-02-25 14:25:09 -- STEP: 33/72 -- GLOBAL_STEP: 1043450 | > loss_disc: 2.429281234741211 (2.5984529292944707) | > loss_disc_real_0: 0.10080039501190186 (0.10790397926713481) | > loss_disc_real_1: 0.2200758457183838 (0.2218100545984326) | > loss_disc_real_2: 0.19618722796440125 (0.22692545119560126) | > loss_disc_real_3: 0.21931174397468567 (0.23344553690968137) | > loss_disc_real_4: 0.2168603390455246 (0.23828570544719696) | > loss_disc_real_5: 0.23833191394805908 (0.24429574473337692) | > loss_0: 2.429281234741211 (2.5984529292944707) | > grad_norm_0: tensor(80.2874, device='cuda:0') (tensor(119.2561, device='cuda:0')) | > loss_gen: 2.317305564880371 (2.0949857198830806) | > loss_kl: 2.0832395553588867 (2.0120767427213266) | > loss_feat: 4.038043022155762 (3.626259796547167) | > loss_mel: 17.287012100219727 (17.12923749287923) | > loss_duration: 1.558106541633606 (1.536462462309635) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.283708572387695 (26.399022304650508) | > grad_norm_1: tensor(8235.8887, device='cuda:0') (tensor(7326.9185, device='cuda:0')) | > current_lr_0: 0.0001854782556976435 | > current_lr_1: 0.0001854782556976435 | > step_time: 0.4381 (0.4370238564231179) | > loader_time: 0.0077 (0.007383035890983812) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871628438922721 (+4.132029036400808e-05) | > avg_step_time: 0.45345299344667245 (-0.004286554497732242) | > avg_loss_disc: 2.60140662461939 (-0.002986830724796885) | > avg_loss_disc_real_0: 0.10778722560531656 (-0.006670199701903579) | > avg_loss_disc_real_1: 0.22336176058775942 (-0.0033150151581831377) | > avg_loss_disc_real_2: 0.22382970238235636 (-0.004522740001409814) | > avg_loss_disc_real_3: 0.234255581884317 (-0.0003781983969916858) | > avg_loss_disc_real_4: 0.2408168271813594 (-0.0006816657915921231) | > avg_loss_disc_real_5: 0.24507852297433666 (-0.00046945058963668296) | > avg_loss_0: 2.60140662461939 (-0.002986830724796885) | > avg_grad_norm_0: tensor(121.3361, device='cuda:0') (+tensor(9.6333, device='cuda:0')) | > avg_loss_gen: 2.1091287639779104 (-0.03813963708743229) | > avg_loss_kl: 2.061259568577082 (-0.03525030612945512) | > avg_loss_feat: 3.6344259423269354 (+0.000948862290718111) | > avg_loss_mel: 17.094197340414556 (-0.07144042807566109) | > avg_loss_duration: 1.5742042904168787 (-0.006223297454941434) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.473216070255763 (-0.15010468388947018) | > avg_grad_norm_1: tensor(7561.1885, device='cuda:0') (+tensor(301.9653, device='cuda:0'))  > EPOCH: 604/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:25:29)   --> TIME: 2025-02-25 14:25:36 -- STEP: 11/72 -- GLOBAL_STEP: 1043500 | > loss_disc: 2.6906638145446777 (2.570440422404896) | > loss_disc_real_0: 0.0589323416352272 (0.13107182627374475) | > loss_disc_real_1: 0.28581497073173523 (0.2329064580527219) | > loss_disc_real_2: 0.2813079357147217 (0.23915374820882623) | > loss_disc_real_3: 0.24960456788539886 (0.23299347947944293) | > loss_disc_real_4: 0.2507968544960022 (0.24821943721987985) | > loss_disc_real_5: 0.2529596984386444 (0.24777473373846573) | > loss_0: 2.6906638145446777 (2.570440422404896) | > grad_norm_0: tensor(144.0126, device='cuda:0') (tensor(126.6229, device='cuda:0')) | > loss_gen: 2.1553778648376465 (2.2927930788560347) | > loss_kl: 1.9641034603118896 (2.0250702879645606) | > loss_feat: 3.529780864715576 (3.86071853204207) | > loss_mel: 17.09574317932129 (17.616609053178266) | > loss_duration: 1.5255376100540161 (1.5420806407928467) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.27054214477539 (27.337271603670988) | > grad_norm_1: tensor(8584.8311, device='cuda:0') (tensor(8302.5264, device='cuda:0')) | > current_lr_0: 0.0001854550709156813 | > current_lr_1: 0.0001854550709156813 | > step_time: 0.4528 (0.4333845702084628) | > loader_time: 0.0057 (0.006442720239812677)  --> TIME: 2025-02-25 14:26:00 -- STEP: 61/72 -- GLOBAL_STEP: 1043550 | > loss_disc: 2.547306537628174 (2.6390723947618837) | > loss_disc_real_0: 0.18844297528266907 (0.11308533295256193) | > loss_disc_real_1: 0.24119621515274048 (0.2291974545013709) | > loss_disc_real_2: 0.24622012674808502 (0.22588578171905924) | > loss_disc_real_3: 0.22650107741355896 (0.23186369655562228) | > loss_disc_real_4: 0.2522847056388855 (0.24823663591361433) | > loss_disc_real_5: 0.254289835691452 (0.2526785582303999) | > loss_0: 2.547306537628174 (2.6390723947618837) | > grad_norm_0: tensor(146.4932, device='cuda:0') (tensor(121.9242, device='cuda:0')) | > loss_gen: 2.2458550930023193 (2.1665417483595553) | > loss_kl: 2.1597938537597656 (2.347151533502047) | > loss_feat: 3.281062364578247 (3.7349862661518034) | > loss_mel: 17.016136169433594 (18.08861299420967) | > loss_duration: 1.6253552436828613 (1.586736837371451) | > amp_scaler: 16.0 (16.262295081967213) | > loss_1: 26.328203201293945 (27.92402961605885) | > grad_norm_1: tensor(9975.2344, device='cuda:0') (tensor(7762.7607, device='cuda:0')) | > current_lr_0: 0.0001854550709156813 | > current_lr_1: 0.0001854550709156813 | > step_time: 0.4566 (0.44957284067497877) | > loader_time: 0.0105 (0.008096694946289062) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555694365165603 (-0.00016059002406160726) | > avg_step_time: 0.45816816746349065 (+0.004715174016818202) | > avg_loss_disc: 2.6360799090963023 (+0.034673284476912425) | > avg_loss_disc_real_0: 0.11684111002045618 (+0.009053884415139615) | > avg_loss_disc_real_1: 0.2288943583696661 (+0.0055325977819066785) | > avg_loss_disc_real_2: 0.22634793051951368 (+0.0025182281371573245) | > avg_loss_disc_real_3: 0.2301422307189082 (-0.004113351165408813) | > avg_loss_disc_real_4: 0.24483604876088425 (+0.004019221579524834) | > avg_loss_disc_real_5: 0.25111906608225576 (+0.0060405431079191) | > avg_loss_0: 2.6360799090963023 (+0.034673284476912425) | > avg_grad_norm_0: tensor(124.3288, device='cuda:0') (+tensor(2.9927, device='cuda:0')) | > avg_loss_gen: 2.1630578595148005 (+0.05392909553689007) | > avg_loss_kl: 2.3214795841297633 (+0.26022001555268126) | > avg_loss_feat: 3.6822344316563136 (+0.0478084893293782) | > avg_loss_mel: 17.969468788361887 (+0.8752714479473305) | > avg_loss_duration: 1.59309059801236 (+0.018886307595481266) | > avg_amp_scaler: 16.225352112676056 (+0.22535211267605604) | > avg_loss_1: 27.729331432933538 (+1.2561153626777752) | > avg_grad_norm_1: tensor(7845.8872, device='cuda:0') (+tensor(284.6987, device='cuda:0'))  > EPOCH: 605/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:26:06)   --> TIME: 2025-02-25 14:26:26 -- STEP: 39/72 -- GLOBAL_STEP: 1043600 | > loss_disc: 2.3933279514312744 (2.5719439738836045) | > loss_disc_real_0: 0.1107570081949234 (0.10719935080179802) | > loss_disc_real_1: 0.23213844001293182 (0.22213718600762197) | > loss_disc_real_2: 0.2198546975851059 (0.213340923954279) | > loss_disc_real_3: 0.22754165530204773 (0.2323801284417128) | > loss_disc_real_4: 0.21665960550308228 (0.23971900993432754) | > loss_disc_real_5: 0.24092167615890503 (0.24276323501880354) | > loss_0: 2.3933279514312744 (2.5719439738836045) | > grad_norm_0: tensor(127.2630, device='cuda:0') (tensor(128.9797, device='cuda:0')) | > loss_gen: 2.559048652648926 (2.1509600908328323) | > loss_kl: 2.1824774742126465 (2.043064973293206) | > loss_feat: 4.432733535766602 (3.689225661448943) | > loss_mel: 17.486434936523438 (17.069938415136093) | > loss_duration: 1.5691328048706055 (1.54895826180776) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.229827880859375 (26.502147136590427) | > grad_norm_1: tensor(9394.7900, device='cuda:0') (tensor(8456.4043, device='cuda:0')) | > current_lr_0: 0.0001854318890318168 | > current_lr_1: 0.0001854318890318168 | > step_time: 0.4448 (0.4362503015078031) | > loader_time: 0.0083 (0.007201952811999199) --> EVAL PERFORMANCE | > avg_loader_time: 0.008486193670353418 (-6.950069481218533e-05) | > avg_step_time: 0.4527782621517987 (-0.005389905311691923) | > avg_loss_disc: 2.5949370659572977 (-0.041142843139004626) | > avg_loss_disc_real_0: 0.11725407461045494 (+0.00041296458999875996) | > avg_loss_disc_real_1: 0.22809156355723528 (-0.0008027948124308126) | > avg_loss_disc_real_2: 0.22040688005131734 (-0.005941050468196346) | > avg_loss_disc_real_3: 0.23704255718580433 (+0.006900326466896145) | > avg_loss_disc_real_4: 0.24287457995011774 (-0.0019614688107665035) | > avg_loss_disc_real_5: 0.2434365021091112 (-0.007682563973144568) | > avg_loss_0: 2.5949370659572977 (-0.041142843139004626) | > avg_grad_norm_0: tensor(131.7628, device='cuda:0') (+tensor(7.4341, device='cuda:0')) | > avg_loss_gen: 2.15916018922564 (-0.0038976702891604376) | > avg_loss_kl: 2.093131501909712 (-0.2283480822200512) | > avg_loss_feat: 3.6215999294334735 (-0.06063450222284006) | > avg_loss_mel: 17.063061244051223 (-0.9064075443106638) | > avg_loss_duration: 1.5783802741010426 (-0.014710323911317325) | > avg_amp_scaler: 16.0 (-0.22535211267605604) | > avg_loss_1: 26.51533298761073 (-1.213998445322808) | > avg_grad_norm_1: tensor(8195.1270, device='cuda:0') (+tensor(349.2397, device='cuda:0'))  > EPOCH: 606/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:26:43)   --> TIME: 2025-02-25 14:26:53 -- STEP: 17/72 -- GLOBAL_STEP: 1043650 | > loss_disc: 2.4518086910247803 (2.5821249625262093) | > loss_disc_real_0: 0.09574533998966217 (0.11231327999164076) | > loss_disc_real_1: 0.21615178883075714 (0.2212018212851356) | > loss_disc_real_2: 0.18059314787387848 (0.2246152136255713) | > loss_disc_real_3: 0.22734184563159943 (0.22664893637685216) | > loss_disc_real_4: 0.2541855275630951 (0.24100210649125717) | > loss_disc_real_5: 0.23533619940280914 (0.24566112634013682) | > loss_0: 2.4518086910247803 (2.5821249625262093) | > grad_norm_0: tensor(111.2283, device='cuda:0') (tensor(122.9959, device='cuda:0')) | > loss_gen: 2.062836170196533 (2.159698016503278) | > loss_kl: 2.2017080783843994 (2.032754337086397) | > loss_feat: 3.6454968452453613 (3.7012325174668255) | > loss_mel: 17.291484832763672 (17.197920518762924) | > loss_duration: 1.5568954944610596 (1.5306269210927628) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.7584228515625 (26.62223221273983) | > grad_norm_1: tensor(10013.3623, device='cuda:0') (tensor(8256.2881, device='cuda:0')) | > current_lr_0: 0.00018540871004568783 | > current_lr_1: 0.00018540871004568783 | > step_time: 0.4292 (0.43537072574391084) | > loader_time: 0.0061 (0.0069055697497199565)  --> TIME: 2025-02-25 14:27:16 -- STEP: 67/72 -- GLOBAL_STEP: 1043700 | > loss_disc: 2.5628557205200195 (2.5788071902830203) | > loss_disc_real_0: 0.09994661062955856 (0.11386805683819216) | > loss_disc_real_1: 0.23432260751724243 (0.22371982126983245) | > loss_disc_real_2: 0.22337482869625092 (0.2213764486472998) | > loss_disc_real_3: 0.24881763756275177 (0.23285922118976934) | > loss_disc_real_4: 0.23444759845733643 (0.24112914391418003) | > loss_disc_real_5: 0.2544699013233185 (0.2452153285492712) | > loss_0: 2.5628557205200195 (2.5788071902830203) | > grad_norm_0: tensor(133.8875, device='cuda:0') (tensor(131.2739, device='cuda:0')) | > loss_gen: 2.18304443359375 (2.164410381174799) | > loss_kl: 2.191997766494751 (2.087968902801399) | > loss_feat: 3.587076425552368 (3.6790754510395565) | > loss_mel: 17.431140899658203 (17.099549250816228) | > loss_duration: 1.6586230993270874 (1.573046924462959) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.051881790161133 (26.60405070746123) | > grad_norm_1: tensor(9506.5322, device='cuda:0') (tensor(8405.9463, device='cuda:0')) | > current_lr_0: 0.00018540871004568783 | > current_lr_1: 0.00018540871004568783 | > step_time: 0.5129 (0.44975129526052904) | > loader_time: 0.012 (0.008487505699271586) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669859926465531 (+0.00018366625611211315) | > avg_step_time: 0.45300946101336415 (+0.00023119886156541813) | > avg_loss_disc: 2.5770171091590126 (-0.017919956798285064) | > avg_loss_disc_real_0: 0.11302716269249648 (-0.004226911917958459) | > avg_loss_disc_real_1: 0.22399802149181636 (-0.004093542065418926) | > avg_loss_disc_real_2: 0.22124316763709967 (+0.0008362875857823304) | > avg_loss_disc_real_3: 0.23219949531723078 (-0.004843061868573556) | > avg_loss_disc_real_4: 0.24072606001101748 (-0.0021485199391002574) | > avg_loss_disc_real_5: 0.24482436604063276 (+0.0013878639315215713) | > avg_loss_0: 2.5770171091590126 (-0.017919956798285064) | > avg_grad_norm_0: tensor(128.4407, device='cuda:0') (tensor(-3.3221, device='cuda:0')) | > avg_loss_gen: 2.1737426267543305 (+0.014582437528690484) | > avg_loss_kl: 2.0932531642242207 (+0.0001216623145086082) | > avg_loss_feat: 3.6968235365101987 (+0.07522360707672515) | > avg_loss_mel: 17.103621751489772 (+0.04056050743854911) | > avg_loss_duration: 1.5758784942223991 (-0.0025017798786435286) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.6433193582884 (+0.1279863706776716) | > avg_grad_norm_1: tensor(8368.5410, device='cuda:0') (+tensor(173.4141, device='cuda:0'))  > EPOCH: 607/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:27:20)   --> TIME: 2025-02-25 14:27:42 -- STEP: 45/72 -- GLOBAL_STEP: 1043750 | > loss_disc: 2.6388297080993652 (2.5876807848612464) | > loss_disc_real_0: 0.10370523482561111 (0.11462660369773706) | > loss_disc_real_1: 0.2191782295703888 (0.22477367023626962) | > loss_disc_real_2: 0.2655611038208008 (0.2203139821688334) | > loss_disc_real_3: 0.21801729500293732 (0.23209293815824722) | > loss_disc_real_4: 0.2588329017162323 (0.24200495746400622) | > loss_disc_real_5: 0.25154799222946167 (0.24448740747239855) | > loss_0: 2.6388297080993652 (2.5876807848612464) | > grad_norm_0: tensor(166.9463, device='cuda:0') (tensor(123.5945, device='cuda:0')) | > loss_gen: 1.9286473989486694 (2.1418117682139073) | > loss_kl: 2.1881697177886963 (2.0793888330459596) | > loss_feat: 3.334381341934204 (3.6901229540507) | > loss_mel: 17.64275360107422 (17.2847777472602) | > loss_duration: 1.5973955392837524 (1.5544264131122165) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.691347122192383 (26.750528038872613) | > grad_norm_1: tensor(6507.3511, device='cuda:0') (tensor(7973.5688, device='cuda:0')) | > current_lr_0: 0.0001853855339569321 | > current_lr_1: 0.0001853855339569321 | > step_time: 0.4519 (0.4381745391421848) | > loader_time: 0.0093 (0.007807774013943143) --> EVAL PERFORMANCE | > avg_loader_time: 0.00896706043834418 (+0.00029720051187864804) | > avg_step_time: 0.4540223168655181 (+0.0010128558521539377) | > avg_loss_disc: 2.6311553632709344 (+0.054138254111921746) | > avg_loss_disc_real_0: 0.13478984492240656 (+0.021762682229910082) | > avg_loss_disc_real_1: 0.22777987617841908 (+0.0037818546866027203) | > avg_loss_disc_real_2: 0.2281143302229089 (+0.006871162585809221) | > avg_loss_disc_real_3: 0.23277006676079523 (+0.0005705714435644549) | > avg_loss_disc_real_4: 0.24052822359011206 (-0.0001978364209054284) | > avg_loss_disc_real_5: 0.24409074787522705 (-0.0007336181654057083) | > avg_loss_0: 2.6311553632709344 (+0.054138254111921746) | > avg_grad_norm_0: tensor(120.0188, device='cuda:0') (tensor(-8.4219, device='cuda:0')) | > avg_loss_gen: 2.1467565398820683 (-0.026986086872262227) | > avg_loss_kl: 2.1493096905694875 (+0.05605652634526681) | > avg_loss_feat: 3.654832326190572 (-0.04199121031962649) | > avg_loss_mel: 17.408852281704753 (+0.30523053021498114) | > avg_loss_duration: 1.582336316645985 (+0.006457822423585835) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.942087388374436 (+0.29876803008603403) | > avg_grad_norm_1: tensor(7714.0903, device='cuda:0') (tensor(-654.4507, device='cuda:0'))  > EPOCH: 608/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:27:57)   --> TIME: 2025-02-25 14:28:09 -- STEP: 23/72 -- GLOBAL_STEP: 1043800 | > loss_disc: 3.2578952312469482 (2.661290707795516) | > loss_disc_real_0: 0.0440295934677124 (0.11374525952598323) | > loss_disc_real_1: 0.21436651051044464 (0.24167776366938715) | > loss_disc_real_2: 0.14799906313419342 (0.20669348343558933) | > loss_disc_real_3: 0.3064274489879608 (0.24833335435908774) | > loss_disc_real_4: 0.2961813509464264 (0.24816692393759024) | > loss_disc_real_5: 0.24564631283283234 (0.24545176780742148) | > loss_0: 3.2578952312469482 (2.661290707795516) | > grad_norm_0: tensor(253.7292, device='cuda:0') (tensor(131.3200, device='cuda:0')) | > loss_gen: 1.465887188911438 (2.047211066536282) | > loss_kl: 2.173506021499634 (2.09628006686335) | > loss_feat: 3.445770740509033 (3.5128313665804654) | > loss_mel: 17.096851348876953 (17.464922697647758) | > loss_duration: 1.5596153736114502 (1.5428290418956592) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.74163246154785 (26.66407469044561) | > grad_norm_1: tensor(5776.7153, device='cuda:0') (tensor(7108.7544, device='cuda:0')) | > current_lr_0: 0.00018536236076518748 | > current_lr_1: 0.00018536236076518748 | > step_time: 0.4361 (0.4357453636501146) | > loader_time: 0.0067 (0.006801055825274923) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666434758146045 (-0.0003006256801981345) | > avg_step_time: 0.4569458289885185 (+0.0029235121230004046) | > avg_loss_disc: 2.6858641899807356 (+0.05470882670980126) | > avg_loss_disc_real_0: 0.12596609789720722 (-0.008823747025199336) | > avg_loss_disc_real_1: 0.24074539815036344 (+0.012965521971944366) | > avg_loss_disc_real_2: 0.21496096358332836 (-0.013153366639580527) | > avg_loss_disc_real_3: 0.24161225179551352 (+0.008842185034718286) | > avg_loss_disc_real_4: 0.2445751172556004 (+0.004046893665488355) | > avg_loss_disc_real_5: 0.2508455212687103 (+0.006754773393483238) | > avg_loss_0: 2.6858641899807356 (+0.05470882670980126) | > avg_grad_norm_0: tensor(116.5817, device='cuda:0') (tensor(-3.4371, device='cuda:0')) | > avg_loss_gen: 2.0455191001086166 (-0.10123743977345168) | > avg_loss_kl: 2.267763601222508 (+0.11845391065302069) | > avg_loss_feat: 3.4419857179614866 (-0.21284660822908563) | > avg_loss_mel: 17.918155119452674 (+0.5093028377479207) | > avg_loss_duration: 1.5896809336165305 (+0.007344616970545559) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.263104693990357 (+0.32101730561592134) | > avg_grad_norm_1: tensor(6737.8975, device='cuda:0') (tensor(-976.1929, device='cuda:0'))  > EPOCH: 609/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:28:34)   --> TIME: 2025-02-25 14:28:37 -- STEP: 1/72 -- GLOBAL_STEP: 1043850 | > loss_disc: 2.5001392364501953 (2.5001392364501953) | > loss_disc_real_0: 0.11101993918418884 (0.11101993918418884) | > loss_disc_real_1: 0.22583158314228058 (0.22583158314228058) | > loss_disc_real_2: 0.1938869059085846 (0.1938869059085846) | > loss_disc_real_3: 0.25458088517189026 (0.25458088517189026) | > loss_disc_real_4: 0.22736506164073944 (0.22736506164073944) | > loss_disc_real_5: 0.24545978009700775 (0.24545978009700775) | > loss_0: 2.5001392364501953 (2.5001392364501953) | > grad_norm_0: tensor(184.0811, device='cuda:0') (tensor(184.0811, device='cuda:0')) | > loss_gen: 1.9958646297454834 (1.9958646297454834) | > loss_kl: 1.9212559461593628 (1.9212559461593628) | > loss_feat: 3.4367012977600098 (3.4367012977600098) | > loss_mel: 17.11579132080078 (17.11579132080078) | > loss_duration: 1.5856529474258423 (1.5856529474258423) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.055267333984375 (26.055267333984375) | > grad_norm_1: tensor(8792.3076, device='cuda:0') (tensor(8792.3076, device='cuda:0')) | > current_lr_0: 0.00018533919047009182 | > current_lr_1: 0.00018533919047009182 | > step_time: 0.4277 (0.4277057647705078) | > loader_time: 0.0059 (0.0059261322021484375)  --> TIME: 2025-02-25 14:28:59 -- STEP: 51/72 -- GLOBAL_STEP: 1043900 | > loss_disc: 2.5915770530700684 (2.5688924041448855) | > loss_disc_real_0: 0.09385000169277191 (0.11451231461821818) | > loss_disc_real_1: 0.23460808396339417 (0.22036057885955362) | > loss_disc_real_2: 0.21646836400032043 (0.21622512036678837) | > loss_disc_real_3: 0.24022036790847778 (0.23571205022288302) | > loss_disc_real_4: 0.24448733031749725 (0.23988480807519427) | > loss_disc_real_5: 0.2634402811527252 (0.2427328272777445) | > loss_0: 2.5915770530700684 (2.5688924041448855) | > grad_norm_0: tensor(108.9097, device='cuda:0') (tensor(126.9006, device='cuda:0')) | > loss_gen: 2.078446388244629 (2.165050275185529) | > loss_kl: 2.0159409046173096 (2.0158937608494476) | > loss_feat: 3.165688991546631 (3.6636464782789644) | > loss_mel: 14.873407363891602 (17.05665139590993) | > loss_duration: 1.5922071933746338 (1.5564385395424039) | > amp_scaler: 16.0 (16.0) | > loss_1: 23.725690841674805 (26.457680533914004) | > grad_norm_1: tensor(7755.4971, device='cuda:0') (tensor(8476.4014, device='cuda:0')) | > current_lr_0: 0.00018533919047009182 | > current_lr_1: 0.00018533919047009182 | > step_time: 0.4351 (0.44019285837809247) | > loader_time: 0.0088 (0.007659584868187998) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855357546201894 (-0.00011285929612710535) | > avg_step_time: 0.4522715152149469 (-0.004674313773571581) | > avg_loss_disc: 2.5669154046286997 (-0.1189487853520359) | > avg_loss_disc_real_0: 0.11694614479029683 (-0.009019953106910394) | > avg_loss_disc_real_1: 0.22269224994619127 (-0.01805314820417217) | > avg_loss_disc_real_2: 0.21806746889168105 (+0.003106505308352686) | > avg_loss_disc_real_3: 0.232736575141759 (-0.008875676653754522) | > avg_loss_disc_real_4: 0.23825979862414615 (-0.006315318631454264) | > avg_loss_disc_real_5: 0.24258074794017093 (-0.008264773328539365) | > avg_loss_0: 2.5669154046286997 (-0.1189487853520359) | > avg_grad_norm_0: tensor(129.2774, device='cuda:0') (+tensor(12.6957, device='cuda:0')) | > avg_loss_gen: 2.162859387800727 (+0.11734028769211058) | > avg_loss_kl: 2.0586310692236456 (-0.20913253199886261) | > avg_loss_feat: 3.641829080984626 (+0.19984336302313954) | > avg_loss_mel: 17.052860824155136 (-0.865294295297538) | > avg_loss_duration: 1.5741888855544615 (-0.015492048062069053) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.490369313199754 (-0.7727353807906034) | > avg_grad_norm_1: tensor(8377.9463, device='cuda:0') (+tensor(1640.0488, device='cuda:0'))  > EPOCH: 610/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:29:11)   --> TIME: 2025-02-25 14:29:26 -- STEP: 29/72 -- GLOBAL_STEP: 1043950 | > loss_disc: 2.6439385414123535 (2.5930802986539643) | > loss_disc_real_0: 0.04288463294506073 (0.1212902076799294) | > loss_disc_real_1: 0.1792120337486267 (0.22279909766953568) | > loss_disc_real_2: 0.17462888360023499 (0.23429691277701278) | > loss_disc_real_3: 0.2326592653989792 (0.23733216267207574) | > loss_disc_real_4: 0.24661023914813995 (0.24173698086163092) | > loss_disc_real_5: 0.23250603675842285 (0.24448252803292767) | > loss_0: 2.6439385414123535 (2.5930802986539643) | > grad_norm_0: tensor(163.1682, device='cuda:0') (tensor(113.7249, device='cuda:0')) | > loss_gen: 1.7380988597869873 (2.2233483914671273) | > loss_kl: 2.0633647441864014 (2.053851694896303) | > loss_feat: 3.66613507270813 (3.795110891605246) | > loss_mel: 18.009048461914062 (17.206571776291426) | > loss_duration: 1.544473648071289 (1.5292102961704648) | > amp_scaler: 8.0 (8.275862068965518) | > loss_1: 27.021120071411133 (26.808093103869208) | > grad_norm_1: tensor(8098.6816, device='cuda:0') (tensor(8058.0742, device='cuda:0')) | > current_lr_0: 0.00018531602307128304 | > current_lr_1: 0.00018531602307128304 | > step_time: 0.4528 (0.4357943370424468) | > loader_time: 0.0079 (0.007116169765077788) --> EVAL PERFORMANCE | > avg_loader_time: 0.008642220161330533 (+8.864469931159358e-05) | > avg_step_time: 0.4563092077282113 (+0.004037692513264413) | > avg_loss_disc: 2.6022507304876625 (+0.03533532585896282) | > avg_loss_disc_real_0: 0.12416327865400785 (+0.00721713386371102) | > avg_loss_disc_real_1: 0.22912891985664904 (+0.006436669910457771) | > avg_loss_disc_real_2: 0.22388064756359852 (+0.005813178671917468) | > avg_loss_disc_real_3: 0.23298694697064412 (+0.00025037182888512644) | > avg_loss_disc_real_4: 0.2416035681962967 (+0.0033437695721505456) | > avg_loss_disc_real_5: 0.2445554731177612 (+0.0019747251775902863) | > avg_loss_0: 2.6022507304876625 (+0.03533532585896282) | > avg_grad_norm_0: tensor(116.9643, device='cuda:0') (tensor(-12.3131, device='cuda:0')) | > avg_loss_gen: 2.172366249729211 (+0.009506861928483712) | > avg_loss_kl: 2.100667285247588 (+0.04203621602394225) | > avg_loss_feat: 3.628192512082382 (-0.013636568902243962) | > avg_loss_mel: 17.07854868660511 (+0.02568786244997412) | > avg_loss_duration: 1.5741130130391725 (-7.5872515288955e-05) | > avg_amp_scaler: 8.112676056338028 (-7.887323943661972) | > avg_loss_1: 26.553887837369675 (+0.06351852416992188) | > avg_grad_norm_1: tensor(7897.0586, device='cuda:0') (tensor(-480.8877, device='cuda:0'))  > EPOCH: 611/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:29:48)   --> TIME: 2025-02-25 14:29:53 -- STEP: 7/72 -- GLOBAL_STEP: 1044000 | > loss_disc: 2.7606654167175293 (2.5965255328587125) | > loss_disc_real_0: 0.09061954915523529 (0.08830666063087327) | > loss_disc_real_1: 0.16148799657821655 (0.21880089385168894) | > loss_disc_real_2: 0.3892037868499756 (0.23493937722274236) | > loss_disc_real_3: 0.21777954697608948 (0.24931299473558152) | > loss_disc_real_4: 0.19388625025749207 (0.2453938126564026) | > loss_disc_real_5: 0.23789791762828827 (0.24238864651748113) | > loss_0: 2.7606654167175293 (2.5965255328587125) | > grad_norm_0: tensor(115.1651, device='cuda:0') (tensor(116.9031, device='cuda:0')) | > loss_gen: 2.117344856262207 (2.197059597287859) | > loss_kl: 2.132117509841919 (2.0375696761267528) | > loss_feat: 4.518100738525391 (3.917175463267735) | > loss_mel: 17.867647171020508 (17.281701632908412) | > loss_duration: 1.502015471458435 (1.535021526472909) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.137224197387695 (26.96852765764509) | > grad_norm_1: tensor(7367.5840, device='cuda:0') (tensor(8898.9062, device='cuda:0')) | > current_lr_0: 0.00018529285856839912 | > current_lr_1: 0.00018529285856839912 | > step_time: 0.4318 (0.43824914523533415) | > loader_time: 0.0063 (0.006948062351771763)  --> TIME: 2025-02-25 14:30:17 -- STEP: 57/72 -- GLOBAL_STEP: 1044050 | > loss_disc: 2.855311155319214 (2.595461004658749) | > loss_disc_real_0: 0.10392545908689499 (0.09955010961806564) | > loss_disc_real_1: 0.2915702164173126 (0.23278375414379857) | > loss_disc_real_2: 0.27769017219543457 (0.2222574993706586) | > loss_disc_real_3: 0.2355203926563263 (0.23122632451224745) | > loss_disc_real_4: 0.23247815668582916 (0.24094875862723902) | > loss_disc_real_5: 0.23023247718811035 (0.24556122223536173) | > loss_0: 2.855311155319214 (2.595461004658749) | > grad_norm_0: tensor(117.5258, device='cuda:0') (tensor(122.4279, device='cuda:0')) | > loss_gen: 1.7730275392532349 (2.1689681170279513) | > loss_kl: 2.1777443885803223 (2.1145592216859788) | > loss_feat: 2.974776029586792 (3.7316853205362954) | > loss_mel: 17.764644622802734 (17.17513716848273) | > loss_duration: 1.6205718517303467 (1.5614827097507946) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.310762405395508 (26.751832393177768) | > grad_norm_1: tensor(4531.9546, device='cuda:0') (tensor(8282.7012, device='cuda:0')) | > current_lr_0: 0.00018529285856839912 | > current_lr_1: 0.00018529285856839912 | > step_time: 0.4654 (0.4502940512540048) | > loader_time: 0.01 (0.008206225278084737) --> EVAL PERFORMANCE | > avg_loader_time: 0.008830859627522213 (+0.00018863946619167987) | > avg_step_time: 0.45919614778438084 (+0.002886940056169518) | > avg_loss_disc: 2.6010912169872875 (-0.00115951350037502) | > avg_loss_disc_real_0: 0.10531479818090585 (-0.018848480473102003) | > avg_loss_disc_real_1: 0.2310353416792104 (+0.0019064218225613438) | > avg_loss_disc_real_2: 0.21868079501978108 (-0.00519985254381744) | > avg_loss_disc_real_3: 0.2337903682614716 (+0.0008034212908274718) | > avg_loss_disc_real_4: 0.24258726793275753 (+0.0009836997364608413) | > avg_loss_disc_real_5: 0.24511256709065235 (+0.0005570939728911373) | > avg_loss_0: 2.6010912169872875 (-0.00115951350037502) | > avg_grad_norm_0: tensor(119.9432, device='cuda:0') (+tensor(2.9789, device='cuda:0')) | > avg_loss_gen: 2.150171207710051 (-0.022195042019159672) | > avg_loss_kl: 2.153122326018105 (+0.052455040770517325) | > avg_loss_feat: 3.6565514920463023 (+0.028358979963920117) | > avg_loss_mel: 17.195812225341793 (+0.11726353873668316) | > avg_loss_duration: 1.574570637353709 (+0.00045762431453644226) | > avg_amp_scaler: 8.0 (-0.11267605633802802) | > avg_loss_1: 26.73022788679096 (+0.17634004942128456) | > avg_grad_norm_1: tensor(8085.8804, device='cuda:0') (+tensor(188.8218, device='cuda:0'))  > EPOCH: 612/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:30:26)   --> TIME: 2025-02-25 14:30:43 -- STEP: 35/72 -- GLOBAL_STEP: 1044100 | > loss_disc: 2.4704883098602295 (2.781289393561227) | > loss_disc_real_0: 0.07081523537635803 (0.12623368757111686) | > loss_disc_real_1: 0.2124805599451065 (0.21283803220306124) | > loss_disc_real_2: 0.40747326612472534 (0.27647399859769006) | > loss_disc_real_3: 0.254606157541275 (0.22948844262531826) | > loss_disc_real_4: 0.2636600136756897 (0.2595727860927582) | > loss_disc_real_5: 0.25878122448921204 (0.2590231878416879) | > loss_0: 2.4704883098602295 (2.781289393561227) | > grad_norm_0: tensor(50.9693, device='cuda:0') (tensor(136.7966, device='cuda:0')) | > loss_gen: 2.366446018218994 (2.0106240374701363) | > loss_kl: 2.4494645595550537 (2.437438971655709) | > loss_feat: 3.799818277359009 (3.507160745348249) | > loss_mel: 18.097801208496094 (18.2817384992327) | > loss_duration: 1.5993595123291016 (1.5983214616775512) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.312889099121094 (27.83528415134975) | > grad_norm_1: tensor(8865.0547, device='cuda:0') (tensor(6185.0088, device='cuda:0')) | > current_lr_0: 0.00018526969696107805 | > current_lr_1: 0.00018526969696107805 | > step_time: 0.437 (0.4406301294054304) | > loader_time: 0.0084 (0.0071027891976492745) --> EVAL PERFORMANCE | > avg_loader_time: 0.008531358880056461 (-0.0002995007474657519) | > avg_step_time: 0.45412654943869146 (-0.0050695983456893745) | > avg_loss_disc: 2.691342679547592 (+0.09025146256030459) | > avg_loss_disc_real_0: 0.11432961951678908 (+0.00901482133588323) | > avg_loss_disc_real_1: 0.21977409290176042 (-0.011261248777449967) | > avg_loss_disc_real_2: 0.24175452199620256 (+0.023073726976421488) | > avg_loss_disc_real_3: 0.23432020108464738 (+0.0005298328231757854) | > avg_loss_disc_real_4: 0.2507746079018418 (+0.008187339969084273) | > avg_loss_disc_real_5: 0.25012777116097196 (+0.005015204070319607) | > avg_loss_0: 2.691342679547592 (+0.09025146256030459) | > avg_grad_norm_0: tensor(140.5997, device='cuda:0') (+tensor(20.6565, device='cuda:0')) | > avg_loss_gen: 2.035710181988461 (-0.11446102572159012) | > avg_loss_kl: 2.315524175133503 (+0.162401849115398) | > avg_loss_feat: 3.504739929252947 (-0.1518115627933554) | > avg_loss_mel: 17.725250781421927 (+0.5294385560801338) | > avg_loss_duration: 1.6065385526334737 (+0.031967915279764725) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.187763966305155 (+0.4575360795141954) | > avg_grad_norm_1: tensor(6835.6260, device='cuda:0') (tensor(-1250.2544, device='cuda:0'))  > EPOCH: 613/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:31:02)   --> TIME: 2025-02-25 14:31:10 -- STEP: 13/72 -- GLOBAL_STEP: 1044150 | > loss_disc: 2.8958959579467773 (2.5638811038090634) | > loss_disc_real_0: 0.06085976958274841 (0.10683858050749852) | > loss_disc_real_1: 0.19926977157592773 (0.21098518256957716) | > loss_disc_real_2: 0.2098868042230606 (0.2006816829626377) | > loss_disc_real_3: 0.2484247386455536 (0.23423192248894617) | > loss_disc_real_4: 0.24837549030780792 (0.23737939045979425) | > loss_disc_real_5: 0.24192816019058228 (0.23957152664661407) | > loss_0: 2.8958959579467773 (2.5638811038090634) | > grad_norm_0: tensor(82.5000, device='cuda:0') (tensor(147.8978, device='cuda:0')) | > loss_gen: 1.7123465538024902 (2.0949068619654727) | > loss_kl: 1.9406323432922363 (2.0023126418773947) | > loss_feat: 3.41072678565979 (3.641029669688298) | > loss_mel: 16.887222290039062 (17.283781051635742) | > loss_duration: 1.4471995830535889 (1.5309219910548284) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.398128509521484 (26.552952106182392) | > grad_norm_1: tensor(5314.8179, device='cuda:0') (tensor(8063.5596, device='cuda:0')) | > current_lr_0: 0.0001852465382489579 | > current_lr_1: 0.0001852465382489579 | > step_time: 0.4479 (0.42911802805387056) | > loader_time: 0.0058 (0.006611805695753831)  --> TIME: 2025-02-25 14:31:33 -- STEP: 63/72 -- GLOBAL_STEP: 1044200 | > loss_disc: 2.6863510608673096 (2.6288559058355903) | > loss_disc_real_0: 0.17312473058700562 (0.11148635889329607) | > loss_disc_real_1: 0.20276889204978943 (0.21819440925878192) | > loss_disc_real_2: 0.29461705684661865 (0.23934287684304373) | > loss_disc_real_3: 0.22815780341625214 (0.2397216357409008) | > loss_disc_real_4: 0.2390020191669464 (0.24308014460026273) | > loss_disc_real_5: 0.26415854692459106 (0.2456365076322404) | > loss_0: 2.6863510608673096 (2.6288559058355903) | > grad_norm_0: tensor(94.5210, device='cuda:0') (tensor(135.1648, device='cuda:0')) | > loss_gen: 2.2547414302825928 (2.0850212025263946) | > loss_kl: 2.289160966873169 (2.0885451994245012) | > loss_feat: 3.482572317123413 (3.5378498993222673) | > loss_mel: 17.48031997680664 (17.090149122571194) | > loss_duration: 1.6127088069915771 (1.5679071838893588) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.119503021240234 (26.36947238256061) | > grad_norm_1: tensor(6155.3428, device='cuda:0') (tensor(7348.5645, device='cuda:0')) | > current_lr_0: 0.0001852465382489579 | > current_lr_1: 0.0001852465382489579 | > step_time: 0.4963 (0.44694803631494917) | > loader_time: 0.0112 (0.008226133528209865) --> EVAL PERFORMANCE | > avg_loader_time: 0.008601205449708747 (+6.984656965228556e-05) | > avg_step_time: 0.4539251461834975 (-0.00020140325519396596) | > avg_loss_disc: 2.629498357504186 (-0.06184432204340595) | > avg_loss_disc_real_0: 0.11103245930772433 (-0.003297160209064745) | > avg_loss_disc_real_1: 0.218931613044 (-0.0008424798577604253) | > avg_loss_disc_real_2: 0.23849796514276048 (-0.0032565568534420886) | > avg_loss_disc_real_3: 0.2390588899733315 (+0.004738688888684117) | > avg_loss_disc_real_4: 0.24148627758865626 (-0.009288330313185544) | > avg_loss_disc_real_5: 0.24542148096460692 (-0.004706290196365037) | > avg_loss_0: 2.629498357504186 (-0.06184432204340595) | > avg_grad_norm_0: tensor(136.8398, device='cuda:0') (tensor(-3.7598, device='cuda:0')) | > avg_loss_gen: 2.0713329499875988 (+0.03562276799913766) | > avg_loss_kl: 2.102357050062905 (-0.2131671250705982) | > avg_loss_feat: 3.5011905616437886 (-0.0035493676091582493) | > avg_loss_mel: 17.130318480478213 (-0.5949323009437144) | > avg_loss_duration: 1.5757141180441414 (-0.03082443458933226) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.380912941946114 (-0.806851024359041) | > avg_grad_norm_1: tensor(7240.2666, device='cuda:0') (+tensor(404.6406, device='cuda:0'))  > EPOCH: 614/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:31:39)   --> TIME: 2025-02-25 14:32:00 -- STEP: 41/72 -- GLOBAL_STEP: 1044250 | > loss_disc: 2.8061680793762207 (2.609928346261745) | > loss_disc_real_0: 0.19021524488925934 (0.10905876564906865) | > loss_disc_real_1: 0.22640715539455414 (0.2225041498498219) | > loss_disc_real_2: 0.20115400850772858 (0.20225316722218584) | > loss_disc_real_3: 0.21997469663619995 (0.23920173761321278) | > loss_disc_real_4: 0.22807316482067108 (0.24595214717271852) | > loss_disc_real_5: 0.2384735345840454 (0.2448071499423283) | > loss_0: 2.8061680793762207 (2.609928346261745) | > grad_norm_0: tensor(82.6693, device='cuda:0') (tensor(133.8705, device='cuda:0')) | > loss_gen: 1.8346283435821533 (2.068754382249786) | > loss_kl: 1.957661509513855 (2.0309474497306645) | > loss_feat: 3.147075891494751 (3.618259656720045) | > loss_mel: 16.644275665283203 (17.166907077882353) | > loss_duration: 1.5686918497085571 (1.5436123783995466) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.152332305908203 (26.428480892646604) | > grad_norm_1: tensor(7325.8599, device='cuda:0') (tensor(7666.4722, device='cuda:0')) | > current_lr_0: 0.00018522338243167678 | > current_lr_1: 0.00018522338243167678 | > step_time: 0.4572 (0.4486711955652004) | > loader_time: 0.01 (0.007126116171115782) --> EVAL PERFORMANCE | > avg_loader_time: 0.008440984806544344 (-0.0001602206431644028) | > avg_step_time: 0.4600869198920022 (+0.006161773708504681) | > avg_loss_disc: 2.592554193147471 (-0.03694416435671499) | > avg_loss_disc_real_0: 0.11196485681223198 (+0.0009323975045076455) | > avg_loss_disc_real_1: 0.2238719530508552 (+0.004940340006855198) | > avg_loss_disc_real_2: 0.20986295543925862 (-0.028635009703501857) | > avg_loss_disc_real_3: 0.23613309671341534 (-0.002925793259916154) | > avg_loss_disc_real_4: 0.2430760081804974 (+0.001589730591841143) | > avg_loss_disc_real_5: 0.24473154376929915 (-0.0006899371953077715) | > avg_loss_0: 2.592554193147471 (-0.03694416435671499) | > avg_grad_norm_0: tensor(135.1529, device='cuda:0') (tensor(-1.6869, device='cuda:0')) | > avg_loss_gen: 2.0853887662081663 (+0.014055816220567507) | > avg_loss_kl: 2.0771755786009236 (-0.025181471461981353) | > avg_loss_feat: 3.5420271544389323 (+0.040836592795143734) | > avg_loss_mel: 17.024034432961916 (-0.10628404751629716) | > avg_loss_duration: 1.5730216671043717 (-0.002692450939769664) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.301647696696538 (-0.07926524524957657) | > avg_grad_norm_1: tensor(7857.2876, device='cuda:0') (+tensor(617.0210, device='cuda:0'))  > EPOCH: 615/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:32:17)   --> TIME: 2025-02-25 14:32:27 -- STEP: 19/72 -- GLOBAL_STEP: 1044300 | > loss_disc: 2.486823320388794 (2.557063428979171) | > loss_disc_real_0: 0.1434045433998108 (0.10066910558625271) | > loss_disc_real_1: 0.2518578767776489 (0.22176509781887657) | > loss_disc_real_2: 0.20449407398700714 (0.22268816279737572) | > loss_disc_real_3: 0.23303750157356262 (0.23729718986310458) | > loss_disc_real_4: 0.2312818020582199 (0.2457556104973743) | > loss_disc_real_5: 0.2361309975385666 (0.2440266091572611) | > loss_0: 2.486823320388794 (2.557063428979171) | > grad_norm_0: tensor(153.9134, device='cuda:0') (tensor(132.9186, device='cuda:0')) | > loss_gen: 2.4386138916015625 (2.2015446926418103) | > loss_kl: 2.0714707374572754 (2.022235079815513) | > loss_feat: 4.247035980224609 (3.8372563186444735) | > loss_mel: 17.12153434753418 (17.20561539499383) | > loss_duration: 1.517181158065796 (1.5259937173441838) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.395835876464844 (26.792645203439815) | > grad_norm_1: tensor(9061.0889, device='cuda:0') (tensor(8634.4756, device='cuda:0')) | > current_lr_0: 0.00018520022950887282 | > current_lr_1: 0.00018520022950887282 | > step_time: 0.4277 (0.4360435134486148) | > loader_time: 0.006 (0.0068454742431640625)  --> TIME: 2025-02-25 14:32:51 -- STEP: 69/72 -- GLOBAL_STEP: 1044350 | > loss_disc: 2.803842067718506 (2.6701147279877593) | > loss_disc_real_0: 0.15230943262577057 (0.13575146185315173) | > loss_disc_real_1: 0.3131325840950012 (0.24649907130262125) | > loss_disc_real_2: 0.2673130929470062 (0.23119595841221188) | > loss_disc_real_3: 0.24196426570415497 (0.2367516393246858) | > loss_disc_real_4: 0.24078868329524994 (0.24475793626861295) | > loss_disc_real_5: 0.23264425992965698 (0.2471781075000763) | > loss_0: 2.803842067718506 (2.6701147279877593) | > grad_norm_0: tensor(83.9529, device='cuda:0') (tensor(129.6179, device='cuda:0')) | > loss_gen: 1.8132601976394653 (2.0877639072528775) | > loss_kl: 2.3360345363616943 (2.1324466985204955) | > loss_feat: 2.7510087490081787 (3.415469843408336) | > loss_mel: 16.967998504638672 (17.08620891018191) | > loss_duration: 1.6797173023223877 (1.575880299443784) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.54802131652832 (26.29776954650879) | > grad_norm_1: tensor(6768.9741, device='cuda:0') (tensor(7480.0425, device='cuda:0')) | > current_lr_0: 0.00018520022950887282 | > current_lr_1: 0.00018520022950887282 | > step_time: 0.5557 (0.455464508222497) | > loader_time: 0.013 (0.008668979008992515) --> EVAL PERFORMANCE | > avg_loader_time: 0.008715898218289228 (+0.0002749134117448846) | > avg_step_time: 0.4568520599687603 (-0.0032348599232418507) | > avg_loss_disc: 2.6729873838559004 (+0.08043319070842925) | > avg_loss_disc_real_0: 0.13594846889166765 (+0.023983612079435668) | > avg_loss_disc_real_1: 0.24534140967986953 (+0.02146945662901434) | > avg_loss_disc_real_2: 0.23072532469957646 (+0.020862369260317842) | > avg_loss_disc_real_3: 0.23603108987002305 (-0.00010200684339228827) | > avg_loss_disc_real_4: 0.24421115849219577 (+0.001135150311698363) | > avg_loss_disc_real_5: 0.2466165754996555 (+0.0018850317303563402) | > avg_loss_0: 2.6729873838559004 (+0.08043319070842925) | > avg_grad_norm_0: tensor(126.6228, device='cuda:0') (tensor(-8.5301, device='cuda:0')) | > avg_loss_gen: 2.0818398133130134 (-0.0035489528951528904) | > avg_loss_kl: 2.1397084367107344 (+0.0625328581098108) | > avg_loss_feat: 3.410809644511048 (-0.13121750992788428) | > avg_loss_mel: 17.103644129256136 (+0.0796096962942201) | > avg_loss_duration: 1.5765775761134189 (+0.00355590900904712) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.312579490769078 (+0.010931794072540413) | > avg_grad_norm_1: tensor(7383.9258, device='cuda:0') (tensor(-473.3618, device='cuda:0'))  > EPOCH: 616/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:32:54)   --> TIME: 2025-02-25 14:33:17 -- STEP: 47/72 -- GLOBAL_STEP: 1044400 | > loss_disc: 2.8241660594940186 (2.72290430677698) | > loss_disc_real_0: 0.3376765847206116 (0.1417750689260503) | > loss_disc_real_1: 0.24946655333042145 (0.2406050077778228) | > loss_disc_real_2: 0.41011467576026917 (0.271162712510596) | > loss_disc_real_3: 0.2777063548564911 (0.2403058149713151) | > loss_disc_real_4: 0.25394898653030396 (0.23525753395354493) | > loss_disc_real_5: 0.26117756962776184 (0.24373734631436936) | > loss_0: 2.8241660594940186 (2.72290430677698) | > grad_norm_0: tensor(79.6011, device='cuda:0') (tensor(120.8971, device='cuda:0')) | > loss_gen: 2.512511730194092 (2.1784040293794997) | > loss_kl: 2.356154203414917 (3.287133105257724) | > loss_feat: 3.060662031173706 (3.6663309208890227) | > loss_mel: 18.45062828063965 (21.5639245459374) | > loss_duration: 1.6552094221115112 (1.6925790360633364) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.035165786743164 (32.38837177195447) | > grad_norm_1: tensor(5778.5596, device='cuda:0') (tensor(5736.7061, device='cuda:0')) | > current_lr_0: 0.0001851770794801842 | > current_lr_1: 0.0001851770794801842 | > step_time: 0.4508 (0.44236708194651503) | > loader_time: 0.009 (0.0076197766243143285) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614775160668602 (-0.00010112305762062686) | > avg_step_time: 0.4554853237850565 (-0.0013667361837038094) | > avg_loss_disc: 2.7145432519241117 (+0.04155586806821132) | > avg_loss_disc_real_0: 0.14171476489011667 (+0.0057662959984490225) | > avg_loss_disc_real_1: 0.23431040430572672 (-0.01103100537414281) | > avg_loss_disc_real_2: 0.26721545424259885 (+0.03649012954302239) | > avg_loss_disc_real_3: 0.2391136800739127 (+0.0030825902038896513) | > avg_loss_disc_real_4: 0.23786214392789654 (-0.006349014564299232) | > avg_loss_disc_real_5: 0.24294073963668986 (-0.003675835862965632) | > avg_loss_0: 2.7145432519241117 (+0.04155586806821132) | > avg_grad_norm_0: tensor(118.3851, device='cuda:0') (tensor(-8.2377, device='cuda:0')) | > avg_loss_gen: 2.1041187669189876 (+0.022278953605974205) | > avg_loss_kl: 2.9962040370618794 (+0.856495600351145) | > avg_loss_feat: 3.4548481686014525 (+0.04403852409040443) | > avg_loss_mel: 20.400190326529493 (+3.296546197273358) | > avg_loss_duration: 1.6769812543627243 (+0.1004036782493054) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 30.632342540042494 (+4.319763049273416) | > avg_grad_norm_1: tensor(5693.4790, device='cuda:0') (tensor(-1690.4468, device='cuda:0'))  > EPOCH: 617/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:33:31)   --> TIME: 2025-02-25 14:33:44 -- STEP: 25/72 -- GLOBAL_STEP: 1044450 | > loss_disc: 2.480323314666748 (2.7644441699981694) | > loss_disc_real_0: 0.21410630643367767 (0.1409425757825375) | > loss_disc_real_1: 0.1380195915699005 (0.21527686655521394) | > loss_disc_real_2: 0.21117813885211945 (0.26289096236228937) | > loss_disc_real_3: 0.1904800683259964 (0.24680601954460138) | > loss_disc_real_4: 0.19885128736495972 (0.2531220865249634) | > loss_disc_real_5: 0.20253340899944305 (0.24858738481998444) | > loss_0: 2.480323314666748 (2.7644441699981694) | > grad_norm_0: tensor(56.1554, device='cuda:0') (tensor(114.9156, device='cuda:0')) | > loss_gen: 2.72601580619812 (1.9681118392944337) | > loss_kl: 2.2465920448303223 (2.1725316667556758) | > loss_feat: 4.344864368438721 (3.2036435317993166) | > loss_mel: 18.257680892944336 (17.78920379638672) | > loss_duration: 1.564929485321045 (1.5599135971069336) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.14008140563965 (26.693404388427734) | > grad_norm_1: tensor(9811.6816, device='cuda:0') (tensor(5488.4531, device='cuda:0')) | > current_lr_0: 0.00018515393234524918 | > current_lr_1: 0.00018515393234524918 | > step_time: 0.4513 (0.43712021827697756) | > loader_time: 0.0068 (0.00680476188659668) --> EVAL PERFORMANCE | > avg_loader_time: 0.008616726163407449 (+1.9510027388473794e-06) | > avg_step_time: 0.45445718227977483 (-0.0010281415052816878) | > avg_loss_disc: 2.7611348662577884 (+0.046591614333676645) | > avg_loss_disc_real_0: 0.1656241508226999 (+0.023909385932583238) | > avg_loss_disc_real_1: 0.22374509149034258 (-0.010565312815384142) | > avg_loss_disc_real_2: 0.24103324996753472 (-0.026182204275064125) | > avg_loss_disc_real_3: 0.23292892709584304 (-0.006184752978069669) | > avg_loss_disc_real_4: 0.24478385691911403 (+0.0069217129912174935) | > avg_loss_disc_real_5: 0.24517740183313128 (+0.002236662196441419) | > avg_loss_0: 2.7611348662577884 (+0.046591614333676645) | > avg_grad_norm_0: tensor(117.5008, device='cuda:0') (tensor(-0.8843, device='cuda:0')) | > avg_loss_gen: 1.9973651556901528 (-0.10675361122883475) | > avg_loss_kl: 2.232829105686132 (-0.7633749313757474) | > avg_loss_feat: 3.2175185713969485 (-0.23732959720450397) | > avg_loss_mel: 18.086738170032774 (-2.3134521564967194) | > avg_loss_duration: 1.5960890508033863 (-0.08089220355933802) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.130540095584493 (-3.5018024444580007) | > avg_grad_norm_1: tensor(5468.1860, device='cuda:0') (tensor(-225.2930, device='cuda:0'))  > EPOCH: 618/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:34:08)   --> TIME: 2025-02-25 14:34:11 -- STEP: 3/72 -- GLOBAL_STEP: 1044500 | > loss_disc: 2.831562042236328 (2.649932861328125) | > loss_disc_real_0: 0.05482007935643196 (0.09107857073346774) | > loss_disc_real_1: 0.22307613492012024 (0.2730809152126312) | > loss_disc_real_2: 0.23361679911613464 (0.2427845001220703) | > loss_disc_real_3: 0.23488850891590118 (0.26451516648133594) | > loss_disc_real_4: 0.255752831697464 (0.2678023676077525) | > loss_disc_real_5: 0.26768606901168823 (0.26634641488393146) | > loss_0: 2.831562042236328 (2.649932861328125) | > grad_norm_0: tensor(107.4735, device='cuda:0') (tensor(68.5518, device='cuda:0')) | > loss_gen: 1.5499889850616455 (1.8767571051915486) | > loss_kl: 2.6679093837738037 (2.650667587916056) | > loss_feat: 2.948880910873413 (3.2870004971822104) | > loss_mel: 20.82244873046875 (20.904200236002605) | > loss_duration: 1.6656769514083862 (1.6377524137496948) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.654905319213867 (30.356377919514973) | > grad_norm_1: tensor(2231.0818, device='cuda:0') (tensor(5469.0640, device='cuda:0')) | > current_lr_0: 0.000185130788103706 | > current_lr_1: 0.000185130788103706 | > step_time: 0.4471 (0.44185741742451984) | > loader_time: 0.0064 (0.006112972895304362)  --> TIME: 2025-02-25 14:34:34 -- STEP: 53/72 -- GLOBAL_STEP: 1044550 | > loss_disc: 2.794527292251587 (2.7134214617171377) | > loss_disc_real_0: 0.12393754720687866 (0.1547730726031762) | > loss_disc_real_1: 0.24878285825252533 (0.23230338672984321) | > loss_disc_real_2: 0.23577573895454407 (0.23540570038669514) | > loss_disc_real_3: 0.24538415670394897 (0.2286832847122876) | > loss_disc_real_4: 0.24857643246650696 (0.2399192899465561) | > loss_disc_real_5: 0.24280916154384613 (0.24334582995693638) | > loss_0: 2.794527292251587 (2.7134214617171377) | > grad_norm_0: tensor(132.3812, device='cuda:0') (tensor(80.7126, device='cuda:0')) | > loss_gen: 1.812332034111023 (1.9597627036976364) | > loss_kl: 2.398979425430298 (2.264060663727094) | > loss_feat: 3.1489932537078857 (3.248734366218999) | > loss_mel: 17.147130966186523 (18.102038635397854) | > loss_duration: 1.6406499147415161 (1.591556747004671) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.14808464050293 (27.166153457929504) | > grad_norm_1: tensor(5067.1548, device='cuda:0') (tensor(4768.6509, device='cuda:0')) | > current_lr_0: 0.000185130788103706 | > current_lr_1: 0.000185130788103706 | > step_time: 0.4552 (0.44311188751796504) | > loader_time: 0.0089 (0.007779256352838481) --> EVAL PERFORMANCE | > avg_loader_time: 0.008558545314090353 (-5.818084931709551e-05) | > avg_step_time: 0.45521672007063746 (+0.00075953779086263) | > avg_loss_disc: 2.6861749299815005 (-0.07495993627628783) | > avg_loss_disc_real_0: 0.15438891438321325 (-0.011235236439486651) | > avg_loss_disc_real_1: 0.2322955810478036 (+0.008550489557461016) | > avg_loss_disc_real_2: 0.234289492729684 (-0.006743757237850728) | > avg_loss_disc_real_3: 0.2293435660466342 (-0.0035853610492088406) | > avg_loss_disc_real_4: 0.239809169315956 (-0.004974687603158029) | > avg_loss_disc_real_5: 0.2436685455097279 (-0.0015088563234033625) | > avg_loss_0: 2.6861749299815005 (-0.07495993627628783) | > avg_grad_norm_0: tensor(82.5717, device='cuda:0') (tensor(-34.9290, device='cuda:0')) | > avg_loss_gen: 2.0246153918790144 (+0.027250236188861576) | > avg_loss_kl: 2.2724331164024245 (+0.03960401071629249) | > avg_loss_feat: 3.292746036825046 (+0.07522746542809733) | > avg_loss_mel: 18.080544310556323 (-0.0061938594764505694) | > avg_loss_duration: 1.6034722529666525 (+0.007383202163266267) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.273811313467966 (+0.1432712178834734) | > avg_grad_norm_1: tensor(5058.5566, device='cuda:0') (tensor(-409.6294, device='cuda:0'))  > EPOCH: 619/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:34:45)   --> TIME: 2025-02-25 14:35:01 -- STEP: 31/72 -- GLOBAL_STEP: 1044600 | > loss_disc: 2.497958183288574 (2.6710435113599225) | > loss_disc_real_0: 0.08845040202140808 (0.15941794697315462) | > loss_disc_real_1: 0.20372672379016876 (0.21226807251091925) | > loss_disc_real_2: 0.2729288935661316 (0.23301640393272524) | > loss_disc_real_3: 0.24406129121780396 (0.2377973953562398) | > loss_disc_real_4: 0.24599941074848175 (0.2334958665793942) | > loss_disc_real_5: 0.2404436618089676 (0.24058248823688877) | > loss_0: 2.497958183288574 (2.6710435113599225) | > grad_norm_0: tensor(86.5368, device='cuda:0') (tensor(83.0271, device='cuda:0')) | > loss_gen: 2.3495047092437744 (2.0786581385520195) | > loss_kl: 2.5554754734039307 (2.4705911297951975) | > loss_feat: 4.066005706787109 (3.499201120868806) | > loss_mel: 21.15078353881836 (19.581923884730188) | > loss_duration: 1.6110150814056396 (1.6267113608698691) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.732784271240234 (29.25708573864352) | > grad_norm_1: tensor(8776.7490, device='cuda:0') (tensor(4228.4922, device='cuda:0')) | > current_lr_0: 0.00018510764675519302 | > current_lr_1: 0.00018510764675519302 | > step_time: 0.4536 (0.43530871022132134) | > loader_time: 0.0111 (0.00704100824171497) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630500712864838 (+7.19553987744842e-05) | > avg_step_time: 0.45418591566488775 (-0.0010308044057497079) | > avg_loss_disc: 2.7562412745516065 (+0.07006634457010597) | > avg_loss_disc_real_0: 0.13789114639372888 (-0.01649776798948438) | > avg_loss_disc_real_1: 0.22766854215255924 (-0.004627038895244351) | > avg_loss_disc_real_2: 0.2649182484183513 (+0.030628755688667325) | > avg_loss_disc_real_3: 0.2418099580936029 (+0.0124663920469687) | > avg_loss_disc_real_4: 0.24482359831601802 (+0.005014429000062021) | > avg_loss_disc_real_5: 0.24869371593838008 (+0.005025170428652165) | > avg_loss_0: 2.7562412745516065 (+0.07006634457010597) | > avg_grad_norm_0: tensor(96.4677, device='cuda:0') (+tensor(13.8959, device='cuda:0')) | > avg_loss_gen: 1.9857479484987934 (-0.03886744338022097) | > avg_loss_kl: 2.668839357268643 (+0.39640624086621834) | > avg_loss_feat: 3.2952293778809025 (+0.0024833410558566804) | > avg_loss_mel: 19.986037106581136 (+1.905492796024813) | > avg_loss_duration: 1.6822359377229716 (+0.07876368475631912) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.618089863951777 (+2.3442785504838106) | > avg_grad_norm_1: tensor(4649.8701, device='cuda:0') (tensor(-408.6865, device='cuda:0'))  > EPOCH: 620/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:35:22)   --> TIME: 2025-02-25 14:35:27 -- STEP: 9/72 -- GLOBAL_STEP: 1044650 | > loss_disc: 2.721561908721924 (2.6053307056427) | > loss_disc_real_0: 0.06224372610449791 (0.12383015495207575) | > loss_disc_real_1: 0.28983160853385925 (0.23148543139298758) | > loss_disc_real_2: 0.2662184536457062 (0.23199307752980125) | > loss_disc_real_3: 0.2551877200603485 (0.23227240476343367) | > loss_disc_real_4: 0.26486653089523315 (0.23409065273072985) | > loss_disc_real_5: 0.26273104548454285 (0.24790607558356392) | > loss_0: 2.721561908721924 (2.6053307056427) | > grad_norm_0: tensor(161.6937, device='cuda:0') (tensor(114.4745, device='cuda:0')) | > loss_gen: 1.8446524143218994 (1.9898248381084866) | > loss_kl: 2.375471353530884 (2.4153041309780545) | > loss_feat: 2.870176315307617 (3.254667599995931) | > loss_mel: 19.91663932800293 (19.644348356458877) | > loss_duration: 1.6040091514587402 (1.6228637562857733) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.61094856262207 (28.92700873480903) | > grad_norm_1: tensor(4130.1802, device='cuda:0') (tensor(5338.5396, device='cuda:0')) | > current_lr_0: 0.00018508450829934862 | > current_lr_1: 0.00018508450829934862 | > step_time: 0.4234 (0.43042633268568253) | > loader_time: 0.0055 (0.006465461519029405)  --> TIME: 2025-02-25 14:35:50 -- STEP: 59/72 -- GLOBAL_STEP: 1044700 | > loss_disc: 2.3621246814727783 (2.7141691789788713) | > loss_disc_real_0: 0.12048420310020447 (0.1550934069111186) | > loss_disc_real_1: 0.17121069133281708 (0.23483786012156535) | > loss_disc_real_2: 0.2392202764749527 (0.24559120909642365) | > loss_disc_real_3: 0.23272237181663513 (0.24010737619157566) | > loss_disc_real_4: 0.23083624243736267 (0.250686802096286) | > loss_disc_real_5: 0.24387775361537933 (0.25036482992818804) | > loss_0: 2.3621246814727783 (2.7141691789788713) | > grad_norm_0: tensor(21.9122, device='cuda:0') (tensor(102.1516, device='cuda:0')) | > loss_gen: 2.260685920715332 (1.9683471938311048) | > loss_kl: 2.6771280765533447 (2.6705980341313245) | > loss_feat: 3.973501682281494 (3.1581802529803777) | > loss_mel: 20.3089656829834 (20.03412030106884) | > loss_duration: 1.6480138301849365 (1.6227813595432345) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.86829376220703 (29.454027111247434) | > grad_norm_1: tensor(8407.9355, device='cuda:0') (tensor(5141.2007, device='cuda:0')) | > current_lr_0: 0.00018508450829934862 | > current_lr_1: 0.00018508450829934862 | > step_time: 0.4938 (0.4426585819761632) | > loader_time: 0.01 (0.008114830922272248) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669658445976154 (+3.915773311131618e-05) | > avg_step_time: 0.4533711252078204 (-0.0008147904570673381) | > avg_loss_disc: 2.6894578127793864 (-0.06678346177222005) | > avg_loss_disc_real_0: 0.14600139567759676 (+0.008110249283867887) | > avg_loss_disc_real_1: 0.2282672367045577 (+0.0005986945519984577) | > avg_loss_disc_real_2: 0.2415014960816209 (-0.023416752336730434) | > avg_loss_disc_real_3: 0.23543735583063582 (-0.00637260226296707) | > avg_loss_disc_real_4: 0.24839299665370448 (+0.0035693983376864635) | > avg_loss_disc_real_5: 0.24800837270810572 (-0.0006853432302743601) | > avg_loss_0: 2.6894578127793864 (-0.06678346177222005) | > avg_grad_norm_0: tensor(108.5348, device='cuda:0') (+tensor(12.0671, device='cuda:0')) | > avg_loss_gen: 1.9557959848726303 (-0.029951963626163147) | > avg_loss_kl: 2.6482842908778657 (-0.0205550663907772) | > avg_loss_feat: 3.1624971409918556 (-0.1327322368890469) | > avg_loss_mel: 19.892402326556997 (-0.09363478002413927) | > avg_loss_duration: 1.6284935457605711 (-0.05374239196240049) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.287473248763824 (-0.33061661518795304) | > avg_grad_norm_1: tensor(5472.0630, device='cuda:0') (+tensor(822.1929, device='cuda:0'))  > EPOCH: 621/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:35:58)   --> TIME: 2025-02-25 14:36:17 -- STEP: 37/72 -- GLOBAL_STEP: 1044750 | > loss_disc: 2.6046738624572754 (2.5787559586602287) | > loss_disc_real_0: 0.18020664155483246 (0.11360965367104556) | > loss_disc_real_1: 0.21260763704776764 (0.2083129000824851) | > loss_disc_real_2: 0.20233306288719177 (0.26046411250088664) | > loss_disc_real_3: 0.2164231538772583 (0.2424326506820885) | > loss_disc_real_4: 0.22603553533554077 (0.24285028230499578) | > loss_disc_real_5: 0.2229098230600357 (0.24378533419725057) | > loss_0: 2.6046738624572754 (2.5787559586602287) | > grad_norm_0: tensor(94.7536, device='cuda:0') (tensor(131.3151, device='cuda:0')) | > loss_gen: 1.9724642038345337 (2.102717302940987) | > loss_kl: 2.057399272918701 (2.1467646747022053) | > loss_feat: 3.21598744392395 (3.482448384568498) | > loss_mel: 17.93256378173828 (18.359329172082845) | > loss_duration: 1.5963661670684814 (1.5710951830889728) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.774782180786133 (27.662354907474004) | > grad_norm_1: tensor(8436.2197, device='cuda:0') (tensor(8546.9170, device='cuda:0')) | > current_lr_0: 0.0001850613727358112 | > current_lr_1: 0.0001850613727358112 | > step_time: 0.4455 (0.4384290656527957) | > loader_time: 0.0081 (0.007259433333938186) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860549698413258 (-6.416146184357388e-05) | > avg_step_time: 0.46242235412060373 (+0.009051228912783316) | > avg_loss_disc: 2.584565864482396 (-0.10489194829699056) | > avg_loss_disc_real_0: 0.11684354562574709 (-0.029157850051849676) | > avg_loss_disc_real_1: 0.2129371594375288 (-0.015330077267028897) | > avg_loss_disc_real_2: 0.2474295950271715 (+0.005928098945550603) | > avg_loss_disc_real_3: 0.2413565115609639 (+0.005919155730328085) | > avg_loss_disc_real_4: 0.24541027151362996 (-0.002982725140074527) | > avg_loss_disc_real_5: 0.24694859267960131 (-0.0010597800285044034) | > avg_loss_0: 2.584565864482396 (-0.10489194829699056) | > avg_grad_norm_0: tensor(136.9835, device='cuda:0') (+tensor(28.4488, device='cuda:0')) | > avg_loss_gen: 2.0871280297427104 (+0.1313320448700801) | > avg_loss_kl: 2.1549482211260726 (-0.493336069751793) | > avg_loss_feat: 3.424981906380452 (+0.2624847653885962) | > avg_loss_mel: 17.99150616014507 (-1.9008961664119255) | > avg_loss_duration: 1.596147700094841 (-0.03234584566573018) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.254712171957525 (-2.032761076806299) | > avg_grad_norm_1: tensor(8364.5078, device='cuda:0') (+tensor(2892.4448, device='cuda:0'))  > EPOCH: 622/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:36:36)   --> TIME: 2025-02-25 14:36:44 -- STEP: 15/72 -- GLOBAL_STEP: 1044800 | > loss_disc: 2.6386241912841797 (2.551622994740804) | > loss_disc_real_0: 0.07021008431911469 (0.11259596794843674) | > loss_disc_real_1: 0.19534608721733093 (0.20891358057657877) | > loss_disc_real_2: 0.22888746857643127 (0.22763644059499105) | > loss_disc_real_3: 0.23602508008480072 (0.22862394650777182) | > loss_disc_real_4: 0.2410789430141449 (0.235585821668307) | > loss_disc_real_5: 0.26361283659935 (0.24194636444250742) | > loss_0: 2.6386241912841797 (2.551622994740804) | > grad_norm_0: tensor(183.3963, device='cuda:0') (tensor(148.0975, device='cuda:0')) | > loss_gen: 1.885011911392212 (2.051599502563477) | > loss_kl: 1.9541491270065308 (2.037633220354716) | > loss_feat: 3.2300844192504883 (3.505498695373535) | > loss_mel: 18.2006893157959 (17.64879264831543) | > loss_duration: 1.551576018333435 (1.5485915740331013) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.821510314941406 (26.792115529378254) | > grad_norm_1: tensor(8665.7373, device='cuda:0') (tensor(8362.4795, device='cuda:0')) | > current_lr_0: 0.0001850382400642192 | > current_lr_1: 0.0001850382400642192 | > step_time: 0.4236 (0.43280669848124187) | > loader_time: 0.0081 (0.006566445032755534)  --> TIME: 2025-02-25 14:37:08 -- STEP: 65/72 -- GLOBAL_STEP: 1044850 | > loss_disc: 2.5352017879486084 (2.5704751454866845) | > loss_disc_real_0: 0.1279117465019226 (0.11423491210891651) | > loss_disc_real_1: 0.2880113422870636 (0.21494507491588594) | > loss_disc_real_2: 0.23418034613132477 (0.23257835438618293) | > loss_disc_real_3: 0.2436537742614746 (0.23374572556752424) | > loss_disc_real_4: 0.24854403734207153 (0.24155976680608895) | > loss_disc_real_5: 0.2487935572862625 (0.2474042404156465) | > loss_0: 2.5352017879486084 (2.5704751454866845) | > grad_norm_0: tensor(130.1490, device='cuda:0') (tensor(154.2174, device='cuda:0')) | > loss_gen: 2.4345157146453857 (2.094631020839399) | > loss_kl: 2.1897225379943848 (2.0452928781509407) | > loss_feat: 3.958606004714966 (3.5618868350982664) | > loss_mel: 18.06389808654785 (17.458337695782006) | > loss_duration: 1.624661922454834 (1.5807563121502215) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.271404266357422 (26.740904705341045) | > grad_norm_1: tensor(9805.4229, device='cuda:0') (tensor(9099.0322, device='cuda:0')) | > current_lr_0: 0.0001850382400642192 | > current_lr_1: 0.0001850382400642192 | > step_time: 0.5248 (0.4498557347517747) | > loader_time: 0.0115 (0.00825852980980506) --> EVAL PERFORMANCE | > avg_loader_time: 0.008531865939288069 (-7.363104484451113e-05) | > avg_step_time: 0.4547110745604609 (-0.007711279560142836) | > avg_loss_disc: 2.56925683290186 (-0.015309031580535759) | > avg_loss_disc_real_0: 0.11495681678954985 (-0.0018867288361972406) | > avg_loss_disc_real_1: 0.21397878607393991 (+0.0010416266364111082) | > avg_loss_disc_real_2: 0.23111453841269855 (-0.01631505661447294) | > avg_loss_disc_real_3: 0.23232132049513535 (-0.009035191065828563) | > avg_loss_disc_real_4: 0.24081147682498877 (-0.004598794688641189) | > avg_loss_disc_real_5: 0.2469341943801289 (-1.4398299472417886e-05) | > avg_loss_0: 2.56925683290186 (-0.015309031580535759) | > avg_grad_norm_0: tensor(152.7058, device='cuda:0') (+tensor(15.7223, device='cuda:0')) | > avg_loss_gen: 2.089897404254323 (+0.0027693745116126323) | > avg_loss_kl: 2.051049894010518 (-0.10389832711555469) | > avg_loss_feat: 3.548908441839084 (+0.12392653545863208) | > avg_loss_mel: 17.42597573240039 (-0.5655304277446831) | > avg_loss_duration: 1.585281919425642 (-0.010865780669198966) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.701113338201818 (-0.5535988337557072) | > avg_grad_norm_1: tensor(8992.3516, device='cuda:0') (+tensor(627.8438, device='cuda:0'))  > EPOCH: 623/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:37:13)   --> TIME: 2025-02-25 14:37:34 -- STEP: 43/72 -- GLOBAL_STEP: 1044900 | > loss_disc: 2.57317852973938 (2.5661508538002185) | > loss_disc_real_0: 0.09187595546245575 (0.1146294672128766) | > loss_disc_real_1: 0.19899211823940277 (0.2194293588399887) | > loss_disc_real_2: 0.204748272895813 (0.22821976487026657) | > loss_disc_real_3: 0.23988290131092072 (0.23362962243168853) | > loss_disc_real_4: 0.25416678190231323 (0.24370814444020736) | > loss_disc_real_5: 0.2549057900905609 (0.2466585819804391) | > loss_0: 2.57317852973938 (2.5661508538002185) | > grad_norm_0: tensor(264.9072, device='cuda:0') (tensor(166.3612, device='cuda:0')) | > loss_gen: 1.8760840892791748 (2.109676660493363) | > loss_kl: 1.868831753730774 (1.9711731450502263) | > loss_feat: 3.1157572269439697 (3.607813923857933) | > loss_mel: 16.495084762573242 (17.338703288588416) | > loss_duration: 1.5978546142578125 (1.558486403420914) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.953611373901367 (26.585853399232377) | > grad_norm_1: tensor(10276.6162, device='cuda:0') (tensor(9489.6094, device='cuda:0')) | > current_lr_0: 0.00018501511028421117 | > current_lr_1: 0.00018501511028421117 | > step_time: 0.4509 (0.4383462528849757) | > loader_time: 0.0096 (0.007420290348141692) --> EVAL PERFORMANCE | > avg_loader_time: 0.008597004581505144 (+6.51386422170757e-05) | > avg_step_time: 0.4540374043961646 (-0.0006736701642963139) | > avg_loss_disc: 2.576254240224059 (+0.006997407322198779) | > avg_loss_disc_real_0: 0.11823223898528328 (+0.0032754221957334295) | > avg_loss_disc_real_1: 0.2194034154985992 (+0.005424629424659294) | > avg_loss_disc_real_2: 0.22980142353286206 (-0.0013131148798364856) | > avg_loss_disc_real_3: 0.23444566382488735 (+0.0021243433297520042) | > avg_loss_disc_real_4: 0.24412071075237973 (+0.0033092339273909643) | > avg_loss_disc_real_5: 0.247482202842202 (+0.0005480084620730907) | > avg_loss_0: 2.576254240224059 (+0.006997407322198779) | > avg_grad_norm_0: tensor(168.3774, device='cuda:0') (+tensor(15.6715, device='cuda:0')) | > avg_loss_gen: 2.109355698169117 (+0.019458293914794034) | > avg_loss_kl: 2.0013356947563063 (-0.04971419925421161) | > avg_loss_feat: 3.544701206852013 (-0.004207234987070763) | > avg_loss_mel: 17.215291412783344 (-0.21068431961704448) | > avg_loss_duration: 1.5808212522050025 (-0.004460667220639536) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.4515052311857 (-0.2496081070161189) | > avg_grad_norm_1: tensor(9542.3545, device='cuda:0') (+tensor(550.0029, device='cuda:0'))  > EPOCH: 624/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:37:49)   --> TIME: 2025-02-25 14:38:01 -- STEP: 21/72 -- GLOBAL_STEP: 1044950 | > loss_disc: 2.7426271438598633 (2.595754657472883) | > loss_disc_real_0: 0.17350202798843384 (0.12873207813217527) | > loss_disc_real_1: 0.2251717895269394 (0.2147181041183926) | > loss_disc_real_2: 0.17377999424934387 (0.23123376071453094) | > loss_disc_real_3: 0.2391607165336609 (0.23029085993766785) | > loss_disc_real_4: 0.2524603307247162 (0.24113390062536513) | > loss_disc_real_5: 0.24859267473220825 (0.24335490947677976) | > loss_0: 2.7426271438598633 (2.595754657472883) | > grad_norm_0: tensor(218.5101, device='cuda:0') (tensor(180.4102, device='cuda:0')) | > loss_gen: 1.960544228553772 (2.113040021487645) | > loss_kl: 1.7740139961242676 (1.9145437933149791) | > loss_feat: 3.255307912826538 (3.6809369382404147) | > loss_mel: 17.30482292175293 (17.155788330804732) | > loss_duration: 1.5384080410003662 (1.5400277376174927) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.833097457885742 (26.40433711097354) | > grad_norm_1: tensor(10047.9199, device='cuda:0') (tensor(10188.0371, device='cuda:0')) | > current_lr_0: 0.00018499198339542563 | > current_lr_1: 0.00018499198339542563 | > step_time: 0.4467 (0.4314552488781157) | > loader_time: 0.0065 (0.006464504060291108)  --> TIME: 2025-02-25 14:38:25 -- STEP: 71/72 -- GLOBAL_STEP: 1045000 | > loss_disc: 2.391334056854248 (2.6875970262876696) | > loss_disc_real_0: 0.11124548316001892 (0.12586132452731402) | > loss_disc_real_1: 0.1907341033220291 (0.2530294026077633) | > loss_disc_real_2: 0.31848588585853577 (0.25952746477765093) | > loss_disc_real_3: 0.23334680497646332 (0.2388970434245929) | > loss_disc_real_4: 0.21748150885105133 (0.24957243930286085) | > loss_disc_real_5: 0.24303101003170013 (0.25017323964078664) | > loss_0: 2.391334056854248 (2.6875970262876696) | > grad_norm_0: tensor(110.8981, device='cuda:0') (tensor(175.9737, device='cuda:0')) | > loss_gen: 2.358006000518799 (2.108701430575948) | > loss_kl: 2.0552871227264404 (2.004991887320935) | > loss_feat: 3.873061418533325 (3.4836533271091086) | > loss_mel: 16.61939811706543 (17.117241201266438) | > loss_duration: 1.573264241218567 (1.579144278042753) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.47901725769043 (26.29373228046256) | > grad_norm_1: tensor(10650.5332, device='cuda:0') (tensor(9250.9863, device='cuda:0')) | > current_lr_0: 0.00018499198339542563 | > current_lr_1: 0.00018499198339542563 | > step_time: 0.4784 (0.45356421739282743) | > loader_time: 0.008 (0.008517910057390238) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1045000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008517910057390238 (-7.909452411490603e-05) | > avg_step_time: 0.45356421739282743 (-0.0004731870033371499) | > avg_loss_disc: 2.6875970262876696 (+0.1113427860636107) | > avg_loss_disc_real_0: 0.12586132452731402 (+0.007629085542030747) | > avg_loss_disc_real_1: 0.2530294026077633 (+0.03362598710916409) | > avg_loss_disc_real_2: 0.25952746477765093 (+0.029726041244788864) | > avg_loss_disc_real_3: 0.2388970434245929 (+0.0044513795997055405) | > avg_loss_disc_real_4: 0.24957243930286085 (+0.005451728550481122) | > avg_loss_disc_real_5: 0.25017323964078664 (+0.0026910367985846506) | > avg_loss_0: 2.6875970262876696 (+0.1113427860636107) | > avg_grad_norm_0: tensor(175.9737, device='cuda:0') (+tensor(7.5963, device='cuda:0')) | > avg_loss_gen: 2.108701430575948 (-0.000654267593168889) | > avg_loss_kl: 2.004991887320935 (+0.0036561925646285687) | > avg_loss_feat: 3.4836533271091086 (-0.0610478797429046) | > avg_loss_mel: 17.117241201266438 (-0.09805021151690596) | > avg_loss_duration: 1.579144278042753 (-0.0016769741622495449) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.29373228046256 (-0.1577729507231389) | > avg_grad_norm_1: tensor(9250.9863, device='cuda:0') (tensor(-291.3682, device='cuda:0'))  > EPOCH: 625/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:38:30)   --> TIME: 2025-02-25 14:38:54 -- STEP: 49/72 -- GLOBAL_STEP: 1045050 | > loss_disc: 2.8521108627319336 (2.621195063299063) | > loss_disc_real_0: 0.12099900096654892 (0.11627681507748) | > loss_disc_real_1: 0.18721768260002136 (0.22097964432774758) | > loss_disc_real_2: 0.11806683242321014 (0.22181731340836505) | > loss_disc_real_3: 0.17338602244853973 (0.23141024581023625) | > loss_disc_real_4: 0.2230880856513977 (0.24452442538981534) | > loss_disc_real_5: 0.24308450520038605 (0.24375406181325718) | > loss_0: 2.8521108627319336 (2.621195063299063) | > grad_norm_0: tensor(282.6756, device='cuda:0') (tensor(167.7242, device='cuda:0')) | > loss_gen: 1.4048283100128174 (2.0054067373275766) | > loss_kl: 2.107328176498413 (1.986880754937931) | > loss_feat: 2.9178240299224854 (3.4300349926461977) | > loss_mel: 16.968215942382812 (17.020397264130267) | > loss_duration: 1.6269915103912354 (1.555794093073631) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.025188446044922 (25.998513747234735) | > grad_norm_1: tensor(2748.1882, device='cuda:0') (tensor(9213.4873, device='cuda:0')) | > current_lr_0: 0.0001849688593975012 | > current_lr_1: 0.0001849688593975012 | > step_time: 0.4391 (0.4455051616746552) | > loader_time: 0.0096 (0.0077580529816296635) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696488931145465 (+0.00017857887375522657) | > avg_step_time: 0.4575458076638235 (+0.003981590270996094) | > avg_loss_disc: 2.637683556113445 (-0.04991347017422454) | > avg_loss_disc_real_0: 0.11784481661210597 (-0.008016507915208054) | > avg_loss_disc_real_1: 0.21658042472013286 (-0.036448977887630435) | > avg_loss_disc_real_2: 0.22721370970699148 (-0.03231375507065945) | > avg_loss_disc_real_3: 0.23201047429736232 (-0.0068865691272305685) | > avg_loss_disc_real_4: 0.24178401412258685 (-0.007788425180274006) | > avg_loss_disc_real_5: 0.24329286205096984 (-0.0068803775898168) | > avg_loss_0: 2.637683556113445 (-0.04991347017422454) | > avg_grad_norm_0: tensor(165.2964, device='cuda:0') (tensor(-10.6773, device='cuda:0')) | > avg_loss_gen: 1.9996243950346835 (-0.10907703554126469) | > avg_loss_kl: 2.0331736248983465 (+0.028181737577411603) | > avg_loss_feat: 3.41657361178331 (-0.0670797153257987) | > avg_loss_mel: 17.030877140206353 (-0.08636406106008465) | > avg_loss_duration: 1.574546973470231 (-0.004597304572521832) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.05479570845483 (-0.23893657200773077) | > avg_grad_norm_1: tensor(8816.2178, device='cuda:0') (tensor(-434.7686, device='cuda:0'))  > EPOCH: 626/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:39:07)   --> TIME: 2025-02-25 14:39:21 -- STEP: 27/72 -- GLOBAL_STEP: 1045100 | > loss_disc: 2.7895772457122803 (2.767475137004146) | > loss_disc_real_0: 0.06905726343393326 (0.13630762420318748) | > loss_disc_real_1: 0.24398617446422577 (0.24383834390728562) | > loss_disc_real_2: 0.24146941304206848 (0.2673892494704988) | > loss_disc_real_3: 0.21065331995487213 (0.24683334981953656) | > loss_disc_real_4: 0.23127761483192444 (0.25429872064678744) | > loss_disc_real_5: 0.230152428150177 (0.24265331968113227) | > loss_0: 2.7895772457122803 (2.767475137004146) | > grad_norm_0: tensor(133.8107, device='cuda:0') (tensor(175.4345, device='cuda:0')) | > loss_gen: 1.7096275091171265 (1.941553862006576) | > loss_kl: 2.329118490219116 (1.9726075640431158) | > loss_feat: 2.929945230484009 (3.2898378637101917) | > loss_mel: 17.98438262939453 (17.073587276317454) | > loss_duration: 1.599400520324707 (1.5363815978721336) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.552474975585938 (25.813968163949472) | > grad_norm_1: tensor(4453.6626, device='cuda:0') (tensor(6662.5288, device='cuda:0')) | > current_lr_0: 0.0001849457382900765 | > current_lr_1: 0.0001849457382900765 | > step_time: 0.4512 (0.43521341571101435) | > loader_time: 0.0077 (0.006870499363651982) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556906606110049 (-0.0001395823250354162) | > avg_step_time: 0.4529902229846363 (-0.004555584679187208) | > avg_loss_disc: 2.7205062483398006 (+0.08282269222635552) | > avg_loss_disc_real_0: 0.12884721621661113 (+0.011002399604505164) | > avg_loss_disc_real_1: 0.23450277808686376 (+0.017922353366730898) | > avg_loss_disc_real_2: 0.2393482661163303 (+0.012134556409338831) | > avg_loss_disc_real_3: 0.24276437138167906 (+0.010753897084316733) | > avg_loss_disc_real_4: 0.24314379545164774 (+0.001359781329060894) | > avg_loss_disc_real_5: 0.24477932885499068 (+0.0014864668040208417) | > avg_loss_0: 2.7205062483398006 (+0.08282269222635552) | > avg_grad_norm_0: tensor(168.5326, device='cuda:0') (+tensor(3.2363, device='cuda:0')) | > avg_loss_gen: 1.9605638426794132 (-0.039060552355270284) | > avg_loss_kl: 2.06144070961106 (+0.02826708471271333) | > avg_loss_feat: 3.297001418933062 (-0.119572192850248) | > avg_loss_mel: 17.048503526499566 (+0.017626386293212448) | > avg_loss_duration: 1.5798328980593614 (+0.005285924589130353) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.947342456226618 (-0.10745325222821123) | > avg_grad_norm_1: tensor(7396.0410, device='cuda:0') (tensor(-1420.1768, device='cuda:0'))  > EPOCH: 627/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:39:43)   --> TIME: 2025-02-25 14:39:47 -- STEP: 5/72 -- GLOBAL_STEP: 1045150 | > loss_disc: 2.5552244186401367 (2.657560396194458) | > loss_disc_real_0: 0.18213143944740295 (0.20680301487445832) | > loss_disc_real_1: 0.22811822593212128 (0.29099243581295015) | > loss_disc_real_2: 0.2433985024690628 (0.2504881739616394) | > loss_disc_real_3: 0.2609974145889282 (0.1935621529817581) | > loss_disc_real_4: 0.2554841935634613 (0.2525342881679535) | > loss_disc_real_5: 0.2660013437271118 (0.2617291361093521) | > loss_0: 2.5552244186401367 (2.657560396194458) | > grad_norm_0: tensor(202.2027, device='cuda:0') (tensor(191.4679, device='cuda:0')) | > loss_gen: 2.8057169914245605 (2.3562822580337524) | > loss_kl: 2.9140703678131104 (2.7290058612823485) | > loss_feat: 3.9350392818450928 (3.595237398147583) | > loss_mel: 20.33222007751465 (19.391855239868164) | > loss_duration: 1.6055450439453125 (1.7287079572677613) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.592592239379883 (29.801089096069337) | > grad_norm_1: tensor(10714.6855, device='cuda:0') (tensor(8499.9580, device='cuda:0')) | > current_lr_0: 0.00018492262007279025 | > current_lr_1: 0.00018492262007279025 | > step_time: 0.434 (0.42775874137878417) | > loader_time: 0.0061 (0.007004594802856446)  --> TIME: 2025-02-25 14:40:11 -- STEP: 55/72 -- GLOBAL_STEP: 1045200 | > loss_disc: 3.093346118927002 (2.745017992366444) | > loss_disc_real_0: 0.1893739402294159 (0.11636139980771323) | > loss_disc_real_1: 0.20158268511295319 (0.22682024186307734) | > loss_disc_real_2: 0.12567050755023956 (0.25392651801759547) | > loss_disc_real_3: 0.17853038012981415 (0.23471285836263137) | > loss_disc_real_4: 0.21274222433567047 (0.23292930153283206) | > loss_disc_real_5: 0.2506413161754608 (0.2442804756489667) | > loss_0: 3.093346118927002 (2.745017992366444) | > grad_norm_0: tensor(345.9108, device='cuda:0') (tensor(190.2748, device='cuda:0')) | > loss_gen: 1.4927310943603516 (1.9730365341359921) | > loss_kl: 2.2038848400115967 (2.3457712867043234) | > loss_feat: 3.119448661804199 (3.522596497969194) | > loss_mel: 17.60173797607422 (17.982107318531376) | > loss_duration: 1.6275513172149658 (1.6043921947479247) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.04535484313965 (27.427903747558595) | > grad_norm_1: tensor(3428.8857, device='cuda:0') (tensor(8291.6836, device='cuda:0')) | > current_lr_0: 0.00018492262007279025 | > current_lr_1: 0.00018492262007279025 | > step_time: 0.4573 (0.4490168658169833) | > loader_time: 0.0092 (0.007885872233997691) --> EVAL PERFORMANCE | > avg_loader_time: 0.008610967179419293 (+5.40605733092437e-05) | > avg_step_time: 0.45775475636334484 (+0.004764533378708524) | > avg_loss_disc: 2.7528280070130253 (+0.03232175867322473) | > avg_loss_disc_real_0: 0.1243225345519227 (-0.004524681664688432) | > avg_loss_disc_real_1: 0.2299301349361178 (-0.004572643150745959) | > avg_loss_disc_real_2: 0.24952480375347003 (+0.010176537637139715) | > avg_loss_disc_real_3: 0.236088966819602 (-0.0066754045620770475) | > avg_loss_disc_real_4: 0.23522877882064228 (-0.007915016631005467) | > avg_loss_disc_real_5: 0.24485442692964848 (+7.509807465780338e-05) | > avg_loss_0: 2.7528280070130253 (+0.03232175867322473) | > avg_grad_norm_0: tensor(185.1176, device='cuda:0') (+tensor(16.5849, device='cuda:0')) | > avg_loss_gen: 1.962518661794528 (+0.0019548191151148586) | > avg_loss_kl: 2.3097409429684492 (+0.24830023335738938) | > avg_loss_feat: 3.4206933773739236 (+0.12369195844086178) | > avg_loss_mel: 17.778022685521083 (+0.7295191590215175) | > avg_loss_duration: 1.6095942947226511 (+0.029761396663289696) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.080569898578478 (+1.1332274423518598) | > avg_grad_norm_1: tensor(8084.3711, device='cuda:0') (+tensor(688.3301, device='cuda:0'))  > EPOCH: 628/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:40:20)   --> TIME: 2025-02-25 14:40:37 -- STEP: 33/72 -- GLOBAL_STEP: 1045250 | > loss_disc: 2.7571704387664795 (2.6178497617894956) | > loss_disc_real_0: 0.07890589535236359 (0.10652973525451892) | > loss_disc_real_1: 0.25563544034957886 (0.2245840665065881) | > loss_disc_real_2: 0.22221344709396362 (0.2227612855759534) | > loss_disc_real_3: 0.23105250298976898 (0.22930121150883762) | > loss_disc_real_4: 0.2252834290266037 (0.23874363032254306) | > loss_disc_real_5: 0.233753502368927 (0.23987552433302908) | > loss_0: 2.7571704387664795 (2.6178497617894956) | > grad_norm_0: tensor(196.4238, device='cuda:0') (tensor(176.6199, device='cuda:0')) | > loss_gen: 1.7916537523269653 (1.985525167349613) | > loss_kl: 1.8634463548660278 (2.003562808036804) | > loss_feat: 3.287712812423706 (3.400921409780329) | > loss_mel: 17.334840774536133 (17.184659553296637) | > loss_duration: 1.5610771179199219 (1.5407603401126284) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.83873176574707 (26.115429271351207) | > grad_norm_1: tensor(10022.7783, device='cuda:0') (tensor(9127.1230, device='cuda:0')) | > current_lr_0: 0.00018489950474528114 | > current_lr_1: 0.00018489950474528114 | > step_time: 0.4365 (0.43691473296194366) | > loader_time: 0.0081 (0.007171695882623846) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628694104476715 (+1.7726925057422183e-05) | > avg_step_time: 0.4566751097289609 (-0.001079646634383924) | > avg_loss_disc: 2.6212348266386654 (-0.13159318037435996) | > avg_loss_disc_real_0: 0.11758046786130315 (-0.0067420666906195525) | > avg_loss_disc_real_1: 0.22297952691434134 (-0.0069506080217764665) | > avg_loss_disc_real_2: 0.22862407166353413 (-0.020900732089935897) | > avg_loss_disc_real_3: 0.23357189079405558 (-0.002517076025546433) | > avg_loss_disc_real_4: 0.24081601391375904 (+0.00558723509311676) | > avg_loss_disc_real_5: 0.24447418707357327 (-0.00038023985607521515) | > avg_loss_0: 2.6212348266386654 (-0.13159318037435996) | > avg_grad_norm_0: tensor(182.5350, device='cuda:0') (tensor(-2.5826, device='cuda:0')) | > avg_loss_gen: 2.0213820229113937 (+0.05886336111686563) | > avg_loss_kl: 2.04393789801799 (-0.26580304495045937) | > avg_loss_feat: 3.4050266239005076 (-0.015666753473416062) | > avg_loss_mel: 17.12933978228501 (-0.6486829032360717) | > avg_loss_duration: 1.5745481706001387 (-0.0350461241225124) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.174234578307246 (-0.9063353202712321) | > avg_grad_norm_1: tensor(9419.7314, device='cuda:0') (+tensor(1335.3604, device='cuda:0'))  > EPOCH: 629/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:40:57)   --> TIME: 2025-02-25 14:41:04 -- STEP: 11/72 -- GLOBAL_STEP: 1045300 | > loss_disc: 2.359585762023926 (2.5955637368288906) | > loss_disc_real_0: 0.09574519097805023 (0.10819343646818941) | > loss_disc_real_1: 0.21420502662658691 (0.2199081006375226) | > loss_disc_real_2: 0.21282242238521576 (0.23855485699393533) | > loss_disc_real_3: 0.22586144506931305 (0.2293921709060669) | > loss_disc_real_4: 0.2510768473148346 (0.2436629073186354) | > loss_disc_real_5: 0.2576070725917816 (0.2434426166794517) | > loss_0: 2.359585762023926 (2.5955637368288906) | > grad_norm_0: tensor(67.7455, device='cuda:0') (tensor(176.9195, device='cuda:0')) | > loss_gen: 2.6406357288360596 (2.0994805964556607) | > loss_kl: 1.7703592777252197 (1.9580464904958552) | > loss_feat: 4.734888553619385 (3.6317191774194892) | > loss_mel: 17.955875396728516 (17.07240728898482) | > loss_duration: 1.5260882377624512 (1.542968294837258) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.627849578857422 (26.304622129960492) | > grad_norm_1: tensor(13095.9668, device='cuda:0') (tensor(10146.7354, device='cuda:0')) | > current_lr_0: 0.00018487639230718798 | > current_lr_1: 0.00018487639230718798 | > step_time: 0.4271 (0.4280013821341775) | > loader_time: 0.0057 (0.006363391876220703)  --> TIME: 2025-02-25 14:41:27 -- STEP: 61/72 -- GLOBAL_STEP: 1045350 | > loss_disc: 2.496593952178955 (2.7190597682702737) | > loss_disc_real_0: 0.0760335624217987 (0.13403529795955438) | > loss_disc_real_1: 0.2151390165090561 (0.23455704113499062) | > loss_disc_real_2: 0.2659059166908264 (0.2533454938990171) | > loss_disc_real_3: 0.14898651838302612 (0.22162423380574242) | > loss_disc_real_4: 0.1937689334154129 (0.23939811865814398) | > loss_disc_real_5: 0.23979522287845612 (0.24518802615462756) | > loss_0: 2.496593952178955 (2.7190597682702737) | > grad_norm_0: tensor(89.0652, device='cuda:0') (tensor(171.4820, device='cuda:0')) | > loss_gen: 2.2815842628479004 (1.991978946279307) | > loss_kl: 2.075796127319336 (2.0899107358494735) | > loss_feat: 3.9817447662353516 (3.3323783170981485) | > loss_mel: 17.133237838745117 (17.084728913229018) | > loss_duration: 1.620702862739563 (1.5686176839422008) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.09306526184082 (26.06761447718886) | > grad_norm_1: tensor(10561.5146, device='cuda:0') (tensor(8081.5059, device='cuda:0')) | > current_lr_0: 0.00018487639230718798 | > current_lr_1: 0.00018487639230718798 | > step_time: 0.4533 (0.4433973421815966) | > loader_time: 0.0106 (0.008144816414254614) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626256190555199 (-2.437913921515822e-06) | > avg_step_time: 0.4524062217121393 (-0.004268888016821626) | > avg_loss_disc: 2.715513565170933 (+0.09427873853226743) | > avg_loss_disc_real_0: 0.13277868784859148 (+0.015198219987288333) | > avg_loss_disc_real_1: 0.23331129466983633 (+0.010331767755494992) | > avg_loss_disc_real_2: 0.2552962042915989 (+0.026672132628064754) | > avg_loss_disc_real_3: 0.2258098657492181 (-0.007762025044837473) | > avg_loss_disc_real_4: 0.2395654669110204 (-0.0012505470027386334) | > avg_loss_disc_real_5: 0.2462488677300198 (+0.0017746806564465367) | > avg_loss_0: 2.715513565170933 (+0.09427873853226743) | > avg_grad_norm_0: tensor(175.3586, device='cuda:0') (tensor(-7.1764, device='cuda:0')) | > avg_loss_gen: 1.984897878808035 (-0.03648414410335876) | > avg_loss_kl: 2.1035803774712782 (+0.05964247945328838) | > avg_loss_feat: 3.3021455247637252 (-0.10288109913678234) | > avg_loss_mel: 17.138619852737634 (+0.009280070452621914) | > avg_loss_duration: 1.5763896166438787 (+0.001841446043739925) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.105633104351202 (-0.06860147395604344) | > avg_grad_norm_1: tensor(8134.9277, device='cuda:0') (tensor(-1284.8037, device='cuda:0'))  > EPOCH: 630/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:41:34)   --> TIME: 2025-02-25 14:41:54 -- STEP: 39/72 -- GLOBAL_STEP: 1045400 | > loss_disc: 2.6056411266326904 (2.732951439343966) | > loss_disc_real_0: 0.07215312123298645 (0.11015704312385657) | > loss_disc_real_1: 0.22335180640220642 (0.23687490897300917) | > loss_disc_real_2: 0.23078811168670654 (0.24386899708173213) | > loss_disc_real_3: 0.21705728769302368 (0.239376873924182) | > loss_disc_real_4: 0.18460702896118164 (0.2510814200609158) | > loss_disc_real_5: 0.22461803257465363 (0.23742270087584472) | > loss_0: 2.6056411266326904 (2.732951439343966) | > grad_norm_0: tensor(10.3338, device='cuda:0') (tensor(173.1667, device='cuda:0')) | > loss_gen: 2.0480854511260986 (1.9819190441033776) | > loss_kl: 3.1375458240509033 (2.224421929090451) | > loss_feat: 4.006298065185547 (3.3248612513908973) | > loss_mel: 21.38827133178711 (17.58418176113031) | > loss_duration: 1.6603009700775146 (1.557676810484666) | > amp_scaler: 8.0 (8.0) | > loss_1: 32.240501403808594 (26.673061028504982) | > grad_norm_1: tensor(8769.4072, device='cuda:0') (tensor(7217.2886, device='cuda:0')) | > current_lr_0: 0.00018485328275814956 | > current_lr_1: 0.00018485328275814956 | > step_time: 0.4465 (0.44905851437495303) | > loader_time: 0.0082 (0.007210872112176357) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522567614703115 (-0.00010368857585208412) | > avg_step_time: 0.4595143492792694 (+0.007108127567130085) | > avg_loss_disc: 2.6913089920097684 (-0.024204573161164422) | > avg_loss_disc_real_0: 0.11343906788339078 (-0.0193396199652007) | > avg_loss_disc_real_1: 0.23075429045818222 (-0.0025570042116541125) | > avg_loss_disc_real_2: 0.24061959583154865 (-0.014676608460050228) | > avg_loss_disc_real_3: 0.24172437421872583 (+0.015914508469507732) | > avg_loss_disc_real_4: 0.2554467259578302 (+0.015881259046809787) | > avg_loss_disc_real_5: 0.24222550656594022 (-0.004023361164079586) | > avg_loss_0: 2.6913089920097684 (-0.024204573161164422) | > avg_grad_norm_0: tensor(162.3156, device='cuda:0') (tensor(-13.0430, device='cuda:0')) | > avg_loss_gen: 2.0126080647320808 (+0.027710185924045838) | > avg_loss_kl: 2.361451921328692 (+0.25787154385741395) | > avg_loss_feat: 3.3570385986650493 (+0.05489307390132403) | > avg_loss_mel: 17.95645493520817 (+0.8178350824705376) | > avg_loss_duration: 1.5978768644198564 (+0.02148724777597777) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.28543053210621 (+1.179797427755009) | > avg_grad_norm_1: tensor(7390.1602, device='cuda:0') (tensor(-744.7676, device='cuda:0'))  > EPOCH: 631/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:42:11)   --> TIME: 2025-02-25 14:42:21 -- STEP: 17/72 -- GLOBAL_STEP: 1045450 | > loss_disc: 2.874513626098633 (2.695752564598532) | > loss_disc_real_0: 0.06321655213832855 (0.1113062470274813) | > loss_disc_real_1: 0.2017250508069992 (0.22818771530600154) | > loss_disc_real_2: 0.26122885942459106 (0.23391035374473124) | > loss_disc_real_3: 0.2723233699798584 (0.24326789554427652) | > loss_disc_real_4: 0.26277756690979004 (0.24518845449475682) | > loss_disc_real_5: 0.2513069212436676 (0.24156673953813665) | > loss_0: 2.874513626098633 (2.695752564598532) | > grad_norm_0: tensor(356.5797, device='cuda:0') (tensor(171.3689, device='cuda:0')) | > loss_gen: 1.8626446723937988 (1.9350480121724747) | > loss_kl: 2.4821622371673584 (2.327621067271513) | > loss_feat: 3.572021722793579 (3.3721420344184425) | > loss_mel: 19.657493591308594 (18.611154556274414) | > loss_duration: 1.5809261798858643 (1.5902203672072466) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.155248641967773 (27.83618590411018) | > grad_norm_1: tensor(10108.0020, device='cuda:0') (tensor(7088.5864, device='cuda:0')) | > current_lr_0: 0.0001848301760978048 | > current_lr_1: 0.0001848301760978048 | > step_time: 0.4373 (0.4411264868343578) | > loader_time: 0.0082 (0.00693204823662253)  --> TIME: 2025-02-25 14:42:45 -- STEP: 67/72 -- GLOBAL_STEP: 1045500 | > loss_disc: 2.923679828643799 (2.716377301002616) | > loss_disc_real_0: 0.20897622406482697 (0.16682665967451993) | > loss_disc_real_1: 0.18223300576210022 (0.23223736659804387) | > loss_disc_real_2: 0.22140440344810486 (0.23161620390948964) | > loss_disc_real_3: 0.2290693074464798 (0.2347960189652087) | > loss_disc_real_4: 0.2133796066045761 (0.24087220430374146) | > loss_disc_real_5: 0.19852037727832794 (0.24149590641704957) | > loss_0: 2.923679828643799 (2.716377301002616) | > grad_norm_0: tensor(186.0176, device='cuda:0') (tensor(160.5785, device='cuda:0')) | > loss_gen: 1.6991037130355835 (1.9621827335500006) | > loss_kl: 2.704958915710449 (2.2435567290035645) | > loss_feat: 3.224722146987915 (3.2062481303713213) | > loss_mel: 20.35426139831543 (17.834386170800055) | > loss_duration: 1.732619047164917 (1.5978219402370168) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.715665817260742 (26.84419572175439) | > grad_norm_1: tensor(8342.9951, device='cuda:0') (tensor(6680.4907, device='cuda:0')) | > current_lr_0: 0.0001848301760978048 | > current_lr_1: 0.0001848301760978048 | > step_time: 0.4946 (0.45306233861553136) | > loader_time: 0.0121 (0.008438832724272316) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086162526842574 (+9.368506955428554e-05) | > avg_step_time: 0.4574068700763541 (-0.0021074792029152634) | > avg_loss_disc: 2.715909477690576 (+0.024600485680807527) | > avg_loss_disc_real_0: 0.16418624695547868 (+0.050747179072087895) | > avg_loss_disc_real_1: 0.2324002746125342 (+0.0016459841543519926) | > avg_loss_disc_real_2: 0.2313337848639824 (-0.009285810967566255) | > avg_loss_disc_real_3: 0.23480986856238942 (-0.006914505656336417) | > avg_loss_disc_real_4: 0.2399370053284605 (-0.0155097206293697) | > avg_loss_disc_real_5: 0.2401516252840069 (-0.0020738812819333097) | > avg_loss_0: 2.715909477690576 (+0.024600485680807527) | > avg_grad_norm_0: tensor(157.3401, device='cuda:0') (tensor(-4.9755, device='cuda:0')) | > avg_loss_gen: 1.9588585151752955 (-0.05374954955678524) | > avg_loss_kl: 2.264747510493641 (-0.09670441083505121) | > avg_loss_feat: 3.2126101574427643 (-0.14442844122228493) | > avg_loss_mel: 17.92749652056627 (-0.028958414641902408) | > avg_loss_duration: 1.6015878445665601 (+0.0037109801467036796) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.965300600293656 (-0.32012993181255567) | > avg_grad_norm_1: tensor(6686.1245, device='cuda:0') (tensor(-704.0356, device='cuda:0'))  > EPOCH: 632/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:42:48)   --> TIME: 2025-02-25 14:43:11 -- STEP: 45/72 -- GLOBAL_STEP: 1045550 | > loss_disc: 2.7010135650634766 (2.6880149364471437) | > loss_disc_real_0: 0.12578077614307404 (0.13453431054949763) | > loss_disc_real_1: 0.2140742689371109 (0.247578861978319) | > loss_disc_real_2: 0.2299565076828003 (0.24446938700146145) | > loss_disc_real_3: 0.25679659843444824 (0.24815635085105897) | > loss_disc_real_4: 0.25319531559944153 (0.24789417021804386) | > loss_disc_real_5: 0.24359636008739471 (0.2457430250114864) | > loss_0: 2.7010135650634766 (2.6880149364471437) | > grad_norm_0: tensor(41.8726, device='cuda:0') (tensor(108.7651, device='cuda:0')) | > loss_gen: 2.282783269882202 (2.1133911238776313) | > loss_kl: 2.5447843074798584 (2.5311003473069937) | > loss_feat: 3.957923650741577 (3.4089449458652075) | > loss_mel: 18.157684326171875 (18.934823015001086) | > loss_duration: 1.614145040512085 (1.5942530737982856) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.557321548461914 (28.582512622409396) | > grad_norm_1: tensor(5140.0469, device='cuda:0') (tensor(5622.7612, device='cuda:0')) | > current_lr_0: 0.00018480707232579257 | > current_lr_1: 0.00018480707232579257 | > step_time: 0.4614 (0.43819917572869194) | > loader_time: 0.009 (0.007564444012112087) --> EVAL PERFORMANCE | > avg_loader_time: 0.008632068902673855 (+1.5816218416455155e-05) | > avg_step_time: 0.45268997675935985 (-0.004716893316994264) | > avg_loss_disc: 2.6701699344205183 (-0.04573954327005758) | > avg_loss_disc_real_0: 0.12158640804634963 (-0.04259983890912905) | > avg_loss_disc_real_1: 0.24944310918660242 (+0.017042834574068216) | > avg_loss_disc_real_2: 0.24276912233359377 (+0.011435337469611367) | > avg_loss_disc_real_3: 0.24610171288671628 (+0.011291844324326866) | > avg_loss_disc_real_4: 0.2456994610773006 (+0.005762455748840101) | > avg_loss_disc_real_5: 0.24519570864422216 (+0.005044083360215251) | > avg_loss_0: 2.6701699344205183 (-0.04573954327005758) | > avg_grad_norm_0: tensor(111.5863, device='cuda:0') (tensor(-45.7537, device='cuda:0')) | > avg_loss_gen: 2.0853251218795776 (+0.12646660670428211) | > avg_loss_kl: 2.441294287292051 (+0.17654677679841013) | > avg_loss_feat: 3.3395764659827862 (+0.1269663085400219) | > avg_loss_mel: 18.436934175625655 (+0.5094376550593864) | > avg_loss_duration: 1.6103404088759083 (+0.008752564309348232) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.913470630914393 (+0.9481700306207372) | > avg_grad_norm_1: tensor(5649.6919, device='cuda:0') (tensor(-1036.4326, device='cuda:0'))  > EPOCH: 633/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:43:25)   --> TIME: 2025-02-25 14:43:37 -- STEP: 23/72 -- GLOBAL_STEP: 1045600 | > loss_disc: 2.6154890060424805 (2.563242881194405) | > loss_disc_real_0: 0.10366390645503998 (0.09911518462974092) | > loss_disc_real_1: 0.20680826902389526 (0.22812104160371033) | > loss_disc_real_2: 0.2335490882396698 (0.23490143210991568) | > loss_disc_real_3: 0.23940229415893555 (0.2333886163390201) | > loss_disc_real_4: 0.24381783604621887 (0.24537802649580914) | > loss_disc_real_5: 0.24946124851703644 (0.24769300675910452) | > loss_0: 2.6154890060424805 (2.563242881194405) | > grad_norm_0: tensor(181.8262, device='cuda:0') (tensor(132.8131, device='cuda:0')) | > loss_gen: 2.0054025650024414 (2.1400440516679184) | > loss_kl: 1.9596481323242188 (2.017652698185133) | > loss_feat: 3.3435134887695312 (3.52409203156181) | > loss_mel: 16.815475463867188 (17.30438485352889) | > loss_duration: 1.5666074752807617 (1.5471321551696113) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.69064712524414 (26.53330595596977) | > grad_norm_1: tensor(7677.8672, device='cuda:0') (tensor(6697.9922, device='cuda:0')) | > current_lr_0: 0.00018478397144175182 | > current_lr_1: 0.00018478397144175182 | > step_time: 0.4434 (0.4323255601136581) | > loader_time: 0.0062 (0.006633240243662958) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857246089989031 (-5.960800278354515e-05) | > avg_step_time: 0.45115484318263094 (-0.0015351335767289087) | > avg_loss_disc: 2.614020317373141 (-0.0561496170473772) | > avg_loss_disc_real_0: 0.11716689211381993 (-0.004419515932529691) | > avg_loss_disc_real_1: 0.24637027242234055 (-0.0030728367642618715) | > avg_loss_disc_real_2: 0.23698496944467787 (-0.0057841528889159) | > avg_loss_disc_real_3: 0.23156312045077204 (-0.014538592435944242) | > avg_loss_disc_real_4: 0.24806971302334696 (+0.00237025194604637) | > avg_loss_disc_real_5: 0.2534895436444752 (+0.00829383500025302) | > avg_loss_0: 2.614020317373141 (-0.0561496170473772) | > avg_grad_norm_0: tensor(130.5723, device='cuda:0') (+tensor(18.9859, device='cuda:0')) | > avg_loss_gen: 2.184109712990237 (+0.09878459111065929) | > avg_loss_kl: 2.591498598246507 (+0.15020431095445597) | > avg_loss_feat: 3.644598836630163 (+0.3050223706473769) | > avg_loss_mel: 19.07631886173302 (+0.6393846861073662) | > avg_loss_duration: 1.6137580502201134 (+0.00341764134420508) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.11028397251183 (+1.1968133415974371) | > avg_grad_norm_1: tensor(7197.8042, device='cuda:0') (+tensor(1548.1123, device='cuda:0'))  > EPOCH: 634/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:44:02)   --> TIME: 2025-02-25 14:44:04 -- STEP: 1/72 -- GLOBAL_STEP: 1045650 | > loss_disc: 2.624952793121338 (2.624952793121338) | > loss_disc_real_0: 0.029359927400946617 (0.029359927400946617) | > loss_disc_real_1: 0.18886622786521912 (0.18886622786521912) | > loss_disc_real_2: 0.31401532888412476 (0.31401532888412476) | > loss_disc_real_3: 0.19248512387275696 (0.19248512387275696) | > loss_disc_real_4: 0.22623437643051147 (0.22623437643051147) | > loss_disc_real_5: 0.2351972609758377 (0.2351972609758377) | > loss_0: 2.624952793121338 (2.624952793121338) | > grad_norm_0: tensor(97.0725, device='cuda:0') (tensor(97.0725, device='cuda:0')) | > loss_gen: 1.9556035995483398 (1.9556035995483398) | > loss_kl: 2.30993390083313 (2.30993390083313) | > loss_feat: 3.6539835929870605 (3.6539835929870605) | > loss_mel: 18.81277084350586 (18.81277084350586) | > loss_duration: 1.6325589418411255 (1.6325589418411255) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.364850997924805 (28.364850997924805) | > grad_norm_1: tensor(6210.4868, device='cuda:0') (tensor(6210.4868, device='cuda:0')) | > current_lr_0: 0.0001847608734453216 | > current_lr_1: 0.0001847608734453216 | > step_time: 0.4213 (0.4212961196899414) | > loader_time: 0.005 (0.00498509407043457)  --> TIME: 2025-02-25 14:44:27 -- STEP: 51/72 -- GLOBAL_STEP: 1045700 | > loss_disc: 2.8931474685668945 (2.6791246638578525) | > loss_disc_real_0: 0.12695668637752533 (0.12982834375226035) | > loss_disc_real_1: 0.26395195722579956 (0.22764213003364264) | > loss_disc_real_2: 0.2772620618343353 (0.24860283995375895) | > loss_disc_real_3: 0.24011845886707306 (0.23588908069274006) | > loss_disc_real_4: 0.24324025213718414 (0.24119666744680965) | > loss_disc_real_5: 0.23860816657543182 (0.24718861457179575) | > loss_0: 2.8931474685668945 (2.6791246638578525) | > grad_norm_0: tensor(158.8359, device='cuda:0') (tensor(123.4447, device='cuda:0')) | > loss_gen: 1.8039406538009644 (2.029181912833569) | > loss_kl: 2.1974527835845947 (2.476008102005603) | > loss_feat: 2.8593344688415527 (3.3271809886483585) | > loss_mel: 16.896831512451172 (18.759063234516216) | > loss_duration: 1.6263346672058105 (1.6300296222462374) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.383893966674805 (28.221463895311544) | > grad_norm_1: tensor(6824.4526, device='cuda:0') (tensor(6864.0542, device='cuda:0')) | > current_lr_0: 0.0001847608734453216 | > current_lr_1: 0.0001847608734453216 | > step_time: 0.4398 (0.45174463589986164) | > loader_time: 0.009 (0.007833083470662435) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651971817016603 (+7.951091712629296e-05) | > avg_step_time: 0.45964494221647023 (+0.008490099033839293) | > avg_loss_disc: 2.6852045327844754 (+0.07118421541133424) | > avg_loss_disc_real_0: 0.13430159517996754 (+0.017134703066147602) | > avg_loss_disc_real_1: 0.23079840985822006 (-0.015571862564120492) | > avg_loss_disc_real_2: 0.24196976968939865 (+0.004984800244720783) | > avg_loss_disc_real_3: 0.23304785662133928 (+0.0014847361705672413) | > avg_loss_disc_real_4: 0.24058296185144237 (-0.007486751171904588) | > avg_loss_disc_real_5: 0.2468980116323686 (-0.006591532012106577) | > avg_loss_0: 2.6852045327844754 (+0.07118421541133424) | > avg_grad_norm_0: tensor(132.3125, device='cuda:0') (+tensor(1.7402, device='cuda:0')) | > avg_loss_gen: 2.0086504053062115 (-0.1754593076840254) | > avg_loss_kl: 2.4272502207420237 (-0.1642483775044834) | > avg_loss_feat: 3.290746141487444 (-0.3538526951427192) | > avg_loss_mel: 18.56298443968867 (-0.5133344220443519) | > avg_loss_duration: 1.6326076917245356 (+0.018849641504422143) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.922238927491954 (-1.1880450450198765) | > avg_grad_norm_1: tensor(6910.3643, device='cuda:0') (tensor(-287.4399, device='cuda:0'))  > EPOCH: 635/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:44:39)   --> TIME: 2025-02-25 14:44:54 -- STEP: 29/72 -- GLOBAL_STEP: 1045750 | > loss_disc: 2.6548519134521484 (2.653293954915014) | > loss_disc_real_0: 0.09251469373703003 (0.13540608153260988) | > loss_disc_real_1: 0.2419990599155426 (0.22114249190379834) | > loss_disc_real_2: 0.2642877697944641 (0.23588528766714292) | > loss_disc_real_3: 0.23627352714538574 (0.23423523327399945) | > loss_disc_real_4: 0.23971077799797058 (0.23989708927170983) | > loss_disc_real_5: 0.24419951438903809 (0.24222771309573074) | > loss_0: 2.6548519134521484 (2.653293954915014) | > grad_norm_0: tensor(144.2686, device='cuda:0') (tensor(131.6144, device='cuda:0')) | > loss_gen: 1.915309190750122 (2.03636937305845) | > loss_kl: 2.1250104904174805 (2.187042672058632) | > loss_feat: 3.257418155670166 (3.505367065298146) | > loss_mel: 17.437660217285156 (18.149525083344557) | > loss_duration: 1.5450870990753174 (1.5597989805813492) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.280487060546875 (27.438103314103753) | > grad_norm_1: tensor(8171.0498, device='cuda:0') (tensor(6770.7368, device='cuda:0')) | > current_lr_0: 0.0001847377783361409 | > current_lr_1: 0.0001847377783361409 | > step_time: 0.4719 (0.43679668985564135) | > loader_time: 0.0074 (0.006787826275003368) --> EVAL PERFORMANCE | > avg_loader_time: 0.008500952116200621 (-0.00015101970081598214) | > avg_step_time: 0.45521751927657866 (-0.004427422939891568) | > avg_loss_disc: 2.688760361201326 (+0.003555828416850648) | > avg_loss_disc_real_0: 0.12944191195805313 (-0.004859683221914407) | > avg_loss_disc_real_1: 0.21963761222194617 (-0.011160797636273895) | > avg_loss_disc_real_2: 0.2441187752804286 (+0.0021490055910299566) | > avg_loss_disc_real_3: 0.23191819014683576 (-0.0011296664745035212) | > avg_loss_disc_real_4: 0.23545825691290304 (-0.005124704938539337) | > avg_loss_disc_real_5: 0.24351999142640074 (-0.0033780202059678632) | > avg_loss_0: 2.688760361201326 (+0.003555828416850648) | > avg_grad_norm_0: tensor(160.4542, device='cuda:0') (+tensor(28.1417, device='cuda:0')) | > avg_loss_gen: 1.9715521218071512 (-0.0370982834990603) | > avg_loss_kl: 2.1684708242685025 (-0.2587793964735212) | > avg_loss_feat: 3.3784566798680267 (+0.08771053838058274) | > avg_loss_mel: 17.60192375451746 (-0.9610606851712085) | > avg_loss_duration: 1.5904975441140186 (-0.04211014761051701) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.710901152919714 (-1.2113377745722396) | > avg_grad_norm_1: tensor(7096.1792, device='cuda:0') (+tensor(185.8149, device='cuda:0'))  > EPOCH: 636/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:45:16)   --> TIME: 2025-02-25 14:45:21 -- STEP: 7/72 -- GLOBAL_STEP: 1045800 | > loss_disc: 2.525108814239502 (2.557789053235735) | > loss_disc_real_0: 0.1392466425895691 (0.0997463943702834) | > loss_disc_real_1: 0.23193101584911346 (0.22300140346799577) | > loss_disc_real_2: 0.2545183598995209 (0.23866660254342215) | > loss_disc_real_3: 0.24114137887954712 (0.22637054111276353) | > loss_disc_real_4: 0.238439679145813 (0.24158182527337754) | > loss_disc_real_5: 0.24019108712673187 (0.24051270527499063) | > loss_0: 2.525108814239502 (2.557789053235735) | > grad_norm_0: tensor(113.6262, device='cuda:0') (tensor(169.6541, device='cuda:0')) | > loss_gen: 2.199450969696045 (2.066094858305795) | > loss_kl: 1.8875212669372559 (1.9729772295270647) | > loss_feat: 3.5872962474823 (3.497397150312151) | > loss_mel: 18.4616756439209 (17.534546170915878) | > loss_duration: 1.5218234062194824 (1.5474731922149658) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.65776824951172 (26.61848885672433) | > grad_norm_1: tensor(6928.6719, device='cuda:0') (tensor(8610.7188, device='cuda:0')) | > current_lr_0: 0.0001847146861138489 | > current_lr_1: 0.0001847146861138489 | > step_time: 0.4408 (0.4301201616014753) | > loader_time: 0.0066 (0.006738696779523577)  --> TIME: 2025-02-25 14:45:44 -- STEP: 57/72 -- GLOBAL_STEP: 1045850 | > loss_disc: 2.753037691116333 (2.6511486622325156) | > loss_disc_real_0: 0.17917805910110474 (0.11190357438304968) | > loss_disc_real_1: 0.2175341248512268 (0.21364592983011613) | > loss_disc_real_2: 0.16199299693107605 (0.24156717459360758) | > loss_disc_real_3: 0.29531046748161316 (0.2273974175515928) | > loss_disc_real_4: 0.22780351340770721 (0.23658863389701174) | > loss_disc_real_5: 0.2299184948205948 (0.2444529039295096) | > loss_0: 2.753037691116333 (2.6511486622325156) | > grad_norm_0: tensor(329.6024, device='cuda:0') (tensor(181.4144, device='cuda:0')) | > loss_gen: 1.773334264755249 (1.9993198670839003) | > loss_kl: 2.038114547729492 (2.0034605892081006) | > loss_feat: 3.132503032684326 (3.565385843578138) | > loss_mel: 15.825310707092285 (17.136004414474755) | > loss_duration: 1.6222870349884033 (1.5686263954430295) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.391550064086914 (26.27279719971774) | > grad_norm_1: tensor(6032.4736, device='cuda:0') (tensor(8744.0859, device='cuda:0')) | > current_lr_0: 0.0001847146861138489 | > current_lr_1: 0.0001847146861138489 | > step_time: 0.4801 (0.44454976550319736) | > loader_time: 0.0096 (0.008100965566802442) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871663362207547 (+0.0002156815058748495) | > avg_step_time: 0.45389420885435294 (-0.0013233104222257208) | > avg_loss_disc: 2.652397320304118 (-0.03636304089720799) | > avg_loss_disc_real_0: 0.11167059733834066 (-0.017771314619712472) | > avg_loss_disc_real_1: 0.2135929712946986 (-0.006044640927247574) | > avg_loss_disc_real_2: 0.2354422262017156 (-0.008676549078713014) | > avg_loss_disc_real_3: 0.2274215324244029 (-0.004496657722432851) | > avg_loss_disc_real_4: 0.23597606554837294 (+0.0005178086354699074) | > avg_loss_disc_real_5: 0.24347822955796417 (-4.1761868436573923e-05) | > avg_loss_0: 2.652397320304118 (-0.03636304089720799) | > avg_grad_norm_0: tensor(175.1632, device='cuda:0') (+tensor(14.7090, device='cuda:0')) | > avg_loss_gen: 1.9963322642823336 (+0.024780142475182343) | > avg_loss_kl: 2.0286454986518536 (-0.13982532561664884) | > avg_loss_feat: 3.530852828227298 (+0.1523961483592715) | > avg_loss_mel: 17.109830305609904 (-0.492093448907557) | > avg_loss_duration: 1.5793331072364054 (-0.011164436877613149) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.244994150081148 (-0.46590700283856634) | > avg_grad_norm_1: tensor(8569.5391, device='cuda:0') (+tensor(1473.3599, device='cuda:0'))  > EPOCH: 637/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:45:53)   --> TIME: 2025-02-25 14:46:11 -- STEP: 35/72 -- GLOBAL_STEP: 1045900 | > loss_disc: 2.538313865661621 (2.6382380281175895) | > loss_disc_real_0: 0.16135112941265106 (0.11194625156266348) | > loss_disc_real_1: 0.23398366570472717 (0.22271353091512408) | > loss_disc_real_2: 0.1877579540014267 (0.2212004902107375) | > loss_disc_real_3: 0.24184775352478027 (0.24248340215001787) | > loss_disc_real_4: 0.23122382164001465 (0.24816760207925523) | > loss_disc_real_5: 0.2735673189163208 (0.257239145892007) | > loss_0: 2.538313865661621 (2.6382380281175895) | > grad_norm_0: tensor(177.3445, device='cuda:0') (tensor(191.2319, device='cuda:0')) | > loss_gen: 2.340379476547241 (2.061412463869368) | > loss_kl: 2.1562576293945312 (1.9741755315235683) | > loss_feat: 3.6759798526763916 (3.6085643223353796) | > loss_mel: 16.904937744140625 (17.05909870692662) | > loss_duration: 1.5724456310272217 (1.5419227225439889) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.649999618530273 (26.24517375401088) | > grad_norm_1: tensor(13209.1650, device='cuda:0') (tensor(9528.1455, device='cuda:0')) | > current_lr_0: 0.00018469159677808464 | > current_lr_1: 0.00018469159677808464 | > step_time: 0.4376 (0.4359177385057722) | > loader_time: 0.0084 (0.0071165425436837335) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647734010723275 (-6.88996113521953e-05) | > avg_step_time: 0.4567587207740461 (+0.002864511919693169) | > avg_loss_disc: 2.6913387607520742 (+0.03894144044795622) | > avg_loss_disc_real_0: 0.11854167062212044 (+0.006871073283779783) | > avg_loss_disc_real_1: 0.22380096433867872 (+0.010207993043980124) | > avg_loss_disc_real_2: 0.22743796032499258 (-0.00800426587672301) | > avg_loss_disc_real_3: 0.25251578088377563 (+0.02509424845937272) | > avg_loss_disc_real_4: 0.2508479195161604 (+0.01487185396778748) | > avg_loss_disc_real_5: 0.25338578601958034 (+0.009907556461616174) | > avg_loss_0: 2.6913387607520742 (+0.03894144044795622) | > avg_grad_norm_0: tensor(191.3157, device='cuda:0') (+tensor(16.1526, device='cuda:0')) | > avg_loss_gen: 2.0188452844888394 (+0.022513020206505807) | > avg_loss_kl: 2.017957128269572 (-0.010688370382281764) | > avg_loss_feat: 3.462539253100543 (-0.0683135751267554) | > avg_loss_mel: 16.926230444035053 (-0.18359986157485153) | > avg_loss_duration: 1.5754062407453295 (-0.0039268664910758755) | > avg_amp_scaler: 9.577464788732392 (+1.5774647887323923) | > avg_loss_1: 26.000978469848633 (-0.24401568023251485) | > avg_grad_norm_1: tensor(8902.0312, device='cuda:0') (+tensor(332.4922, device='cuda:0'))  > EPOCH: 638/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:46:30)   --> TIME: 2025-02-25 14:46:38 -- STEP: 13/72 -- GLOBAL_STEP: 1045950 | > loss_disc: 2.679173707962036 (2.593551654082078) | > loss_disc_real_0: 0.13388697803020477 (0.11866564017075759) | > loss_disc_real_1: 0.22283755242824554 (0.22297886472481948) | > loss_disc_real_2: 0.21951615810394287 (0.2333130549926024) | > loss_disc_real_3: 0.2313220500946045 (0.2276342511177063) | > loss_disc_real_4: 0.2320372462272644 (0.23366200236173776) | > loss_disc_real_5: 0.2510365843772888 (0.2364984957071451) | > loss_0: 2.679173707962036 (2.593551654082078) | > grad_norm_0: tensor(244.2917, device='cuda:0') (tensor(194.0948, device='cuda:0')) | > loss_gen: 2.009800672531128 (2.02722495335799) | > loss_kl: 1.9730323553085327 (1.9848961279942439) | > loss_feat: 3.6207194328308105 (3.4941854660327616) | > loss_mel: 17.34088706970215 (17.03303043658917) | > loss_duration: 1.4624378681182861 (1.5276892460309541) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.406877517700195 (26.067026138305664) | > grad_norm_1: tensor(9535.2539, device='cuda:0') (tensor(8681.5264, device='cuda:0')) | > current_lr_0: 0.00018466851032848737 | > current_lr_1: 0.00018466851032848737 | > step_time: 0.4725 (0.43497397349430966) | > loader_time: 0.0057 (0.006973486680250902)  --> TIME: 2025-02-25 14:47:01 -- STEP: 63/72 -- GLOBAL_STEP: 1046000 | > loss_disc: 2.3714284896850586 (2.6071607574583995) | > loss_disc_real_0: 0.08443246781826019 (0.11132753025444728) | > loss_disc_real_1: 0.20810328423976898 (0.21967381995821755) | > loss_disc_real_2: 0.21700583398342133 (0.24058452177615391) | > loss_disc_real_3: 0.1922508329153061 (0.23050920688916768) | > loss_disc_real_4: 0.22545288503170013 (0.23833180774771978) | > loss_disc_real_5: 0.26537302136421204 (0.2445266991853714) | > loss_0: 2.3714284896850586 (2.6071607574583995) | > grad_norm_0: tensor(121.1691, device='cuda:0') (tensor(192.0080, device='cuda:0')) | > loss_gen: 2.2014667987823486 (2.0448992422648837) | > loss_kl: 2.0793490409851074 (2.0054821249038457) | > loss_feat: 3.464073419570923 (3.4786249493795727) | > loss_mel: 16.944889068603516 (16.81851146334693) | > loss_duration: 1.5838755369186401 (1.565166607735649) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.27365493774414 (25.912684425475106) | > grad_norm_1: tensor(11241.0312, device='cuda:0') (tensor(9281.9326, device='cuda:0')) | > current_lr_0: 0.00018466851032848737 | > current_lr_1: 0.00018466851032848737 | > step_time: 0.4804 (0.4483656126355368) | > loader_time: 0.012 (0.008300788818843784) --> EVAL PERFORMANCE | > avg_loader_time: 0.008674295855240086 (+2.656184451681025e-05) | > avg_step_time: 0.4554375796250894 (-0.001321141148956706) | > avg_loss_disc: 2.610729469379909 (-0.08060929137216544) | > avg_loss_disc_real_0: 0.11514384792724126 (-0.0033978226948791768) | > avg_loss_disc_real_1: 0.22092332029846354 (-0.002877644040215177) | > avg_loss_disc_real_2: 0.24092737826663005 (+0.013489417941637466) | > avg_loss_disc_real_3: 0.23302130694960205 (-0.019494473934173584) | > avg_loss_disc_real_4: 0.23945642525041608 (-0.011391494265744345) | > avg_loss_disc_real_5: 0.24547351477011828 (-0.00791227124946206) | > avg_loss_0: 2.610729469379909 (-0.08060929137216544) | > avg_grad_norm_0: tensor(193.0249, device='cuda:0') (+tensor(1.7091, device='cuda:0')) | > avg_loss_gen: 2.0554234293145193 (+0.03657814482567989) | > avg_loss_kl: 2.0212654160781645 (+0.0033082878085926026) | > avg_loss_feat: 3.464572772173814 (+0.002033519073271428) | > avg_loss_mel: 16.85852027946794 (-0.06771016456711365) | > avg_loss_duration: 1.5731372178440364 (-0.0022690229012931606) | > avg_amp_scaler: 16.0 (+6.422535211267608) | > avg_loss_1: 25.97291916860661 (-0.02805930124202405) | > avg_grad_norm_1: tensor(9318.4912, device='cuda:0') (+tensor(416.4600, device='cuda:0'))  > EPOCH: 639/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:47:07)   --> TIME: 2025-02-25 14:47:28 -- STEP: 41/72 -- GLOBAL_STEP: 1046050 | > loss_disc: 2.6967906951904297 (2.5919288425910763) | > loss_disc_real_0: 0.09754475951194763 (0.11256647055468909) | > loss_disc_real_1: 0.2308308184146881 (0.21931678747258535) | > loss_disc_real_2: 0.23128052055835724 (0.2370180218684964) | > loss_disc_real_3: 0.2240825891494751 (0.2305833250284195) | > loss_disc_real_4: 0.2446468323469162 (0.2393989061437002) | > loss_disc_real_5: 0.24705442786216736 (0.24340902777706705) | > loss_0: 2.6967906951904297 (2.5919288425910763) | > grad_norm_0: tensor(140.1290, device='cuda:0') (tensor(188.3216, device='cuda:0')) | > loss_gen: 1.8971596956253052 (2.0734979990052014) | > loss_kl: 1.8526089191436768 (1.951507981230573) | > loss_feat: 3.0316834449768066 (3.5033312657984292) | > loss_mel: 16.430404663085938 (16.85178310115163) | > loss_duration: 1.5658377408981323 (1.5426423026294245) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.777694702148438 (25.92276275448683) | > grad_norm_1: tensor(8237.4629, device='cuda:0') (tensor(9937.6582, device='cuda:0')) | > current_lr_0: 0.0001846454267646963 | > current_lr_1: 0.0001846454267646963 | > step_time: 0.4588 (0.4388639112798179) | > loader_time: 0.0089 (0.007550297713861233) --> EVAL PERFORMANCE | > avg_loader_time: 0.008721395277641188 (+4.709942240110204e-05) | > avg_step_time: 0.4543775974864691 (-0.0010599821386202968) | > avg_loss_disc: 2.6199509056521135 (+0.00922143627220473) | > avg_loss_disc_real_0: 0.11609128621262564 (+0.0009474382853843777) | > avg_loss_disc_real_1: 0.22422182392066634 (+0.0032985036222027975) | > avg_loss_disc_real_2: 0.23813323802511457 (-0.002794140241515475) | > avg_loss_disc_real_3: 0.2291526863692512 (-0.00386862058035084) | > avg_loss_disc_real_4: 0.2378874998277342 (-0.0015689254226818883) | > avg_loss_disc_real_5: 0.2454813280155961 (+7.813245477827913e-06) | > avg_loss_0: 2.6199509056521135 (+0.00922143627220473) | > avg_grad_norm_0: tensor(198.2911, device='cuda:0') (+tensor(5.2662, device='cuda:0')) | > avg_loss_gen: 2.056843062521704 (+0.001419633207184745) | > avg_loss_kl: 2.005695599905202 (-0.01556981617296227) | > avg_loss_feat: 3.4667302689082184 (+0.0021574967344042406) | > avg_loss_mel: 16.849927203755982 (-0.008593075711956999) | > avg_loss_duration: 1.5714831402604008 (-0.0016540775836355603) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 25.950679322363627 (-0.022239846242982253) | > avg_grad_norm_1: tensor(9921.1533, device='cuda:0') (+tensor(602.6621, device='cuda:0'))  > EPOCH: 640/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:47:44)   --> TIME: 2025-02-25 14:47:55 -- STEP: 19/72 -- GLOBAL_STEP: 1046100 | > loss_disc: 2.610483407974243 (2.6014095607556795) | > loss_disc_real_0: 0.10079213976860046 (0.1219173756084944) | > loss_disc_real_1: 0.23067355155944824 (0.22611776778572484) | > loss_disc_real_2: 0.24617062509059906 (0.23491203863369792) | > loss_disc_real_3: 0.28518301248550415 (0.23663376585433357) | > loss_disc_real_4: 0.2781316936016083 (0.24234077883394142) | > loss_disc_real_5: 0.24487172067165375 (0.24077062622496956) | > loss_0: 2.610483407974243 (2.6014095607556795) | > grad_norm_0: tensor(237.9870, device='cuda:0') (tensor(193.3432, device='cuda:0')) | > loss_gen: 2.1294302940368652 (2.096002277575041) | > loss_kl: 2.0705716609954834 (2.0071385847894767) | > loss_feat: 3.279493570327759 (3.483840013805189) | > loss_mel: 15.844315528869629 (16.99047941910593) | > loss_duration: 1.526604413986206 (1.5288666235773187) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.85041618347168 (26.106327157271537) | > grad_norm_1: tensor(9415.5000, device='cuda:0') (tensor(9668.0996, device='cuda:0')) | > current_lr_0: 0.0001846223460863507 | > current_lr_1: 0.0001846223460863507 | > step_time: 0.4426 (0.43415532614055435) | > loader_time: 0.0065 (0.0064883106633236536)  --> TIME: 2025-02-25 14:48:18 -- STEP: 69/72 -- GLOBAL_STEP: 1046150 | > loss_disc: 2.9196810722351074 (2.617171684900919) | > loss_disc_real_0: 0.21801185607910156 (0.11678860055795615) | > loss_disc_real_1: 0.3387495279312134 (0.22040389240651892) | > loss_disc_real_2: 0.21933594346046448 (0.23562240384627078) | > loss_disc_real_3: 0.20378829538822174 (0.2328862023094426) | > loss_disc_real_4: 0.24029934406280518 (0.24324815493562948) | > loss_disc_real_5: 0.22778621315956116 (0.2438891476047212) | > loss_0: 2.9196810722351074 (2.617171684900919) | > grad_norm_0: tensor(640.6350, device='cuda:0') (tensor(203.1778, device='cuda:0')) | > loss_gen: 1.8786187171936035 (2.081819294155508) | > loss_kl: 2.1943747997283936 (2.0666491346082823) | > loss_feat: 3.099578857421875 (3.564599303231723) | > loss_mel: 17.506975173950195 (16.91770823105522) | > loss_duration: 1.6704936027526855 (1.5688749759093574) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.35004234313965 (26.199651054714035) | > grad_norm_1: tensor(4151.1665, device='cuda:0') (tensor(9974.5693, device='cuda:0')) | > current_lr_0: 0.0001846223460863507 | > current_lr_1: 0.0001846223460863507 | > step_time: 0.5378 (0.4539986278699792) | > loader_time: 0.0131 (0.008633634318476135) --> EVAL PERFORMANCE | > avg_loader_time: 0.008695666219147155 (-2.572905849403291e-05) | > avg_step_time: 0.45546930944415887 (+0.001091711957689756) | > avg_loss_disc: 2.624308700292883 (+0.004357794640769352) | > avg_loss_disc_real_0: 0.11940815950363455 (+0.0033168732910089094) | > avg_loss_disc_real_1: 0.22160849289994844 (-0.002613331020717896) | > avg_loss_disc_real_2: 0.2354023813361853 (-0.0027308566889292785) | > avg_loss_disc_real_3: 0.2306565973960178 (+0.0015039110267665856) | > avg_loss_disc_real_4: 0.24197123650933655 (+0.004083736681602362) | > avg_loss_disc_real_5: 0.24312426653546346 (-0.0023570614801326495) | > avg_loss_0: 2.624308700292883 (+0.004357794640769352) | > avg_grad_norm_0: tensor(210.4427, device='cuda:0') (+tensor(12.1516, device='cuda:0')) | > avg_loss_gen: 2.07573289938376 (+0.01888983686205581) | > avg_loss_kl: 2.0664730155971682 (+0.06077741569196604) | > avg_loss_feat: 3.562083103287388 (+0.09535283437916942) | > avg_loss_mel: 16.912223587573415 (+0.062296383817432854) | > avg_loss_duration: 1.5694683017865034 (-0.0020148384738973846) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.185981025158515 (+0.2353017027948887) | > avg_grad_norm_1: tensor(9968.2510, device='cuda:0') (+tensor(47.0977, device='cuda:0'))  > EPOCH: 641/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:48:21)   --> TIME: 2025-02-25 14:48:44 -- STEP: 47/72 -- GLOBAL_STEP: 1046200 | > loss_disc: 2.8958523273468018 (2.7075127794387495) | > loss_disc_real_0: 0.22502294182777405 (0.12685634552481326) | > loss_disc_real_1: 0.24400240182876587 (0.24024403570814334) | > loss_disc_real_2: 0.23857389390468597 (0.2393827609559323) | > loss_disc_real_3: 0.2282477766275406 (0.24043686656241722) | > loss_disc_real_4: 0.22096528112888336 (0.24004390677239032) | > loss_disc_real_5: 0.24135342240333557 (0.24396769226865567) | > loss_0: 2.8958523273468018 (2.7075127794387495) | > grad_norm_0: tensor(279.4837, device='cuda:0') (tensor(194.7154, device='cuda:0')) | > loss_gen: 1.6305140256881714 (1.9689130022170696) | > loss_kl: 2.163742780685425 (2.090464130361029) | > loss_feat: 2.6934142112731934 (3.3077104497463146) | > loss_mel: 16.975326538085938 (17.043200959550575) | > loss_duration: 1.6200414896011353 (1.5555272457447458) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.083038330078125 (25.965815605001247) | > grad_norm_1: tensor(1591.9224, device='cuda:0') (tensor(7992.5225, device='cuda:0')) | > current_lr_0: 0.0001845992682930899 | > current_lr_1: 0.0001845992682930899 | > step_time: 0.4478 (0.4409047745643778) | > loader_time: 0.0092 (0.007601778557960024) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630285800342828 (-6.538041880432659e-05) | > avg_step_time: 0.4553225208336199 (-0.00014678861053896153) | > avg_loss_disc: 2.7201584426450056 (+0.09584974235212274) | > avg_loss_disc_real_0: 0.160469197418908 (+0.04106103791527345) | > avg_loss_disc_real_1: 0.23505376814536647 (+0.013445275245418026) | > avg_loss_disc_real_2: 0.2360272575432146 (+0.0006248762070293068) | > avg_loss_disc_real_3: 0.23812270710166072 (+0.0074661097056429315) | > avg_loss_disc_real_4: 0.2415709846036535 (-0.0004002519056830467) | > avg_loss_disc_real_5: 0.24121065479768833 (-0.0019136117377751283) | > avg_loss_0: 2.7201584426450056 (+0.09584974235212274) | > avg_grad_norm_0: tensor(165.1349, device='cuda:0') (tensor(-45.3078, device='cuda:0')) | > avg_loss_gen: 2.0017237915119654 (-0.07400910787179438) | > avg_loss_kl: 2.630926075116009 (+0.5644530595188408) | > avg_loss_feat: 3.3174796540972213 (-0.24460344919016652) | > avg_loss_mel: 19.05126595832932 (+2.139042370755906) | > avg_loss_duration: 1.7141333465844812 (+0.14466504479797782) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.715528729935766 (+2.5295477047772508) | > avg_grad_norm_1: tensor(6871.3174, device='cuda:0') (tensor(-3096.9336, device='cuda:0'))  > EPOCH: 642/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:48:58)   --> TIME: 2025-02-25 14:49:11 -- STEP: 25/72 -- GLOBAL_STEP: 1046250 | > loss_disc: 2.601736307144165 (2.7165966606140137) | > loss_disc_real_0: 0.16042762994766235 (0.16463658899068837) | > loss_disc_real_1: 0.2564895451068878 (0.23486449897289277) | > loss_disc_real_2: 0.2684110701084137 (0.2344297683238983) | > loss_disc_real_3: 0.2716844379901886 (0.2219282341003418) | > loss_disc_real_4: 0.25828802585601807 (0.2284777718782425) | > loss_disc_real_5: 0.23137222230434418 (0.23218595445156098) | > loss_0: 2.601736307144165 (2.7165966606140137) | > grad_norm_0: tensor(38.5852, device='cuda:0') (tensor(120.6419, device='cuda:0')) | > loss_gen: 2.6752727031707764 (2.2216118574142456) | > loss_kl: 2.5074656009674072 (2.70773889541626) | > loss_feat: 4.164865016937256 (3.6543990707397462) | > loss_mel: 19.381080627441406 (20.88423957824707) | > loss_duration: 1.646371841430664 (1.6894846200942992) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.37505531311035 (31.15747398376465) | > grad_norm_1: tensor(5212.8643, device='cuda:0') (tensor(5747.0703, device='cuda:0')) | > current_lr_0: 0.00018457619338455326 | > current_lr_1: 0.00018457619338455326 | > step_time: 0.4287 (0.43479427337646487) | > loader_time: 0.0086 (0.006913108825683594) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614002818792635 (-1.6282981550192963e-05) | > avg_step_time: 0.4566053806895941 (+0.0012828598559742055) | > avg_loss_disc: 2.7057063545979245 (-0.01445208804708109) | > avg_loss_disc_real_0: 0.14552852128383145 (-0.014940676135076547) | > avg_loss_disc_real_1: 0.22203925692699325 (-0.013014511218373215) | > avg_loss_disc_real_2: 0.2328431975673622 (-0.0031840599758524157) | > avg_loss_disc_real_3: 0.22588858050359806 (-0.012234126598062667) | > avg_loss_disc_real_4: 0.2345514631187412 (-0.0070195214849123055) | > avg_loss_disc_real_5: 0.2378812551918164 (-0.003329399605871919) | > avg_loss_0: 2.7057063545979245 (-0.01445208804708109) | > avg_grad_norm_0: tensor(120.6674, device='cuda:0') (tensor(-44.4675, device='cuda:0')) | > avg_loss_gen: 2.0151849515001548 (+0.013461159988189308) | > avg_loss_kl: 2.414515310609844 (-0.21641076450616525) | > avg_loss_feat: 3.345033904196511 (+0.02755425009928958) | > avg_loss_mel: 18.803918408675933 (-0.24734754965338723) | > avg_loss_duration: 1.6762779128383583 (-0.03785543374612299) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.254930603672083 (-0.4605981262636831) | > avg_grad_norm_1: tensor(5044.6558, device='cuda:0') (tensor(-1826.6616, device='cuda:0'))  > EPOCH: 643/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:49:35)   --> TIME: 2025-02-25 14:49:39 -- STEP: 3/72 -- GLOBAL_STEP: 1046300 | > loss_disc: 2.8229820728302 (2.765604257583618) | > loss_disc_real_0: 0.30874860286712646 (0.2878531217575073) | > loss_disc_real_1: 0.21957671642303467 (0.24387413760026297) | > loss_disc_real_2: 0.2843703627586365 (0.2817627688248952) | > loss_disc_real_3: 0.2264215052127838 (0.2230179806550344) | > loss_disc_real_4: 0.2589111328125 (0.2554344932238261) | > loss_disc_real_5: 0.2559863328933716 (0.25307125846544903) | > loss_0: 2.8229820728302 (2.765604257583618) | > grad_norm_0: tensor(187.0129, device='cuda:0') (tensor(133.6163, device='cuda:0')) | > loss_gen: 1.9065730571746826 (2.0492393573125205) | > loss_kl: 2.083723783493042 (2.084932049115499) | > loss_feat: 3.1130995750427246 (3.0119218826293945) | > loss_mel: 17.757020950317383 (17.900259017944336) | > loss_duration: 1.6191283464431763 (1.6024495760599773) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.47954559326172 (26.648802439371746) | > grad_norm_1: tensor(830.6513, device='cuda:0') (tensor(2890.1064, device='cuda:0')) | > current_lr_0: 0.0001845531213603802 | > current_lr_1: 0.0001845531213603802 | > step_time: 0.428 (0.4320102532704671) | > loader_time: 0.0128 (0.00822885831197103)  --> TIME: 2025-02-25 14:50:02 -- STEP: 53/72 -- GLOBAL_STEP: 1046350 | > loss_disc: 2.5871975421905518 (2.8114053888140984) | > loss_disc_real_0: 0.06323893368244171 (0.17139991042467784) | > loss_disc_real_1: 0.22487789392471313 (0.22393486487134448) | > loss_disc_real_2: 0.2661898732185364 (0.2631193143579196) | > loss_disc_real_3: 0.23088611662387848 (0.23616065506665213) | > loss_disc_real_4: 0.24299632012844086 (0.2503236239811159) | > loss_disc_real_5: 0.2470073252916336 (0.24957000140873892) | > loss_0: 2.5871975421905518 (2.8114053888140984) | > grad_norm_0: tensor(151.2951, device='cuda:0') (tensor(131.6628, device='cuda:0')) | > loss_gen: 1.9161384105682373 (1.969756639228677) | > loss_kl: 2.2541544437408447 (2.1001288283546007) | > loss_feat: 3.431741952896118 (3.201548104016286) | > loss_mel: 17.608491897583008 (17.751197850929113) | > loss_duration: 1.6415027379989624 (1.5961297278134328) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.852027893066406 (26.6187608107081) | > grad_norm_1: tensor(7984.4751, device='cuda:0') (tensor(5338.7070, device='cuda:0')) | > current_lr_0: 0.0001845531213603802 | > current_lr_1: 0.0001845531213603802 | > step_time: 0.4491 (0.44766125139200463) | > loader_time: 0.0093 (0.008045812822737784) --> EVAL PERFORMANCE | > avg_loader_time: 0.008804442177356129 (+0.0001904393585634937) | > avg_step_time: 0.46065435946827205 (+0.004048978778677936) | > avg_loss_disc: 2.796581137348229 (+0.09087478275030447) | > avg_loss_disc_real_0: 0.18960714083112465 (+0.044078619547293196) | > avg_loss_disc_real_1: 0.22208246709385387 (+4.32101668606244e-05) | > avg_loss_disc_real_2: 0.2559705983165285 (+0.023127400749166332) | > avg_loss_disc_real_3: 0.23493298111667096 (+0.009044400613072906) | > avg_loss_disc_real_4: 0.24750409978376298 (+0.012952636665021777) | > avg_loss_disc_real_5: 0.2480310197447388 (+0.01014976455292238) | > avg_loss_0: 2.796581137348229 (+0.09087478275030447) | > avg_grad_norm_0: tensor(128.7192, device='cuda:0') (+tensor(8.0518, device='cuda:0')) | > avg_loss_gen: 1.9711094809250094 (-0.04407547057514538) | > avg_loss_kl: 2.113548832879939 (-0.3009664777299048) | > avg_loss_feat: 3.1377858981280258 (-0.20724800606848515) | > avg_loss_mel: 17.700771922796537 (-1.103146485879396) | > avg_loss_duration: 1.6068065485484164 (-0.06947136428994183) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.53002250026649 (-1.7249081034055926) | > avg_grad_norm_1: tensor(5163.2891, device='cuda:0') (+tensor(118.6333, device='cuda:0'))  > EPOCH: 644/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:50:13)   --> TIME: 2025-02-25 14:50:28 -- STEP: 31/72 -- GLOBAL_STEP: 1046400 | > loss_disc: 2.6961705684661865 (2.691770653570852) | > loss_disc_real_0: 0.1484750658273697 (0.16386615088389767) | > loss_disc_real_1: 0.2835942208766937 (0.21783218749107852) | > loss_disc_real_2: 0.2593860626220703 (0.24039148899816698) | > loss_disc_real_3: 0.2823962867259979 (0.2495059524813006) | > loss_disc_real_4: 0.2963065505027771 (0.255444731443159) | > loss_disc_real_5: 0.2604215145111084 (0.2500552467761501) | > loss_0: 2.6961705684661865 (2.691770653570852) | > grad_norm_0: tensor(250.2336, device='cuda:0') (tensor(84.8882, device='cuda:0')) | > loss_gen: 1.9238033294677734 (2.0197356285587436) | > loss_kl: 2.414574384689331 (2.5053230831699986) | > loss_feat: 2.85192608833313 (3.2792841619060886) | > loss_mel: 17.814861297607422 (18.68497245542465) | > loss_duration: 1.6014330387115479 (1.585877526190973) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.606597900390625 (28.07519297445974) | > grad_norm_1: tensor(7418.4585, device='cuda:0') (tensor(5222.0586, device='cuda:0')) | > current_lr_0: 0.00018453005222021014 | > current_lr_1: 0.00018453005222021014 | > step_time: 0.4462 (0.43449996363732124) | > loader_time: 0.0085 (0.007137152456468151) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651884508804537 (-0.0001525576685515919) | > avg_step_time: 0.45280379980382784 (-0.007850559664444201) | > avg_loss_disc: 2.6877528076440518 (-0.10882832970417722) | > avg_loss_disc_real_0: 0.16107126309628214 (-0.028535877734842507) | > avg_loss_disc_real_1: 0.24056829323231335 (+0.018485826138459477) | > avg_loss_disc_real_2: 0.23257897563383612 (-0.023391622682692403) | > avg_loss_disc_real_3: 0.23844143916183794 (+0.0035084580451669733) | > avg_loss_disc_real_4: 0.24386360129000437 (-0.0036404984937586093) | > avg_loss_disc_real_5: 0.24583339460299047 (-0.002197625141748316) | > avg_loss_0: 2.6877528076440518 (-0.10882832970417722) | > avg_grad_norm_0: tensor(110.0667, device='cuda:0') (tensor(-18.6525, device='cuda:0')) | > avg_loss_gen: 2.0042060529681995 (+0.033096572043190164) | > avg_loss_kl: 2.396524424284277 (+0.2829755914043379) | > avg_loss_feat: 3.241694211959839 (+0.10390831383181309) | > avg_loss_mel: 18.25252863386987 (+0.551756711073331) | > avg_loss_duration: 1.6103116441780412 (+0.003505095629624755) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.505264926964127 (+0.9752424266976369) | > avg_grad_norm_1: tensor(5801.0723, device='cuda:0') (+tensor(637.7832, device='cuda:0'))  > EPOCH: 645/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:50:49)   --> TIME: 2025-02-25 14:50:55 -- STEP: 9/72 -- GLOBAL_STEP: 1046450 | > loss_disc: 2.691039562225342 (2.647487931781345) | > loss_disc_real_0: 0.09908903390169144 (0.10796371433469984) | > loss_disc_real_1: 0.22060999274253845 (0.225949141714308) | > loss_disc_real_2: 0.2821149230003357 (0.2678453259997898) | > loss_disc_real_3: 0.2438391149044037 (0.24140655828846824) | > loss_disc_real_4: 0.2586336135864258 (0.2461738967233234) | > loss_disc_real_5: 0.2412518560886383 (0.24621262484126621) | > loss_0: 2.691039562225342 (2.647487931781345) | > grad_norm_0: tensor(30.0170, device='cuda:0') (tensor(97.9908, device='cuda:0')) | > loss_gen: 1.9481114149093628 (2.054455426004198) | > loss_kl: 1.7426717281341553 (1.9495468139648438) | > loss_feat: 3.2907943725585938 (3.4353089332580566) | > loss_mel: 18.093812942504883 (17.574588351779514) | > loss_duration: 1.5662819147109985 (1.5682101514604356) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.641672134399414 (26.582109451293945) | > grad_norm_1: tensor(969.3800, device='cuda:0') (tensor(5524.5186, device='cuda:0')) | > current_lr_0: 0.0001845069859636826 | > current_lr_1: 0.0001845069859636826 | > step_time: 0.4235 (0.4274277687072754) | > loader_time: 0.0064 (0.006591240564982097)  --> TIME: 2025-02-25 14:51:19 -- STEP: 59/72 -- GLOBAL_STEP: 1046500 | > loss_disc: 2.556720018386841 (2.6899627103643904) | > loss_disc_real_0: 0.15217456221580505 (0.14581057303032635) | > loss_disc_real_1: 0.2142123132944107 (0.226672584475097) | > loss_disc_real_2: 0.1977107971906662 (0.239776116559061) | > loss_disc_real_3: 0.20927320420742035 (0.23870060075137575) | > loss_disc_real_4: 0.24017347395420074 (0.24970154994625163) | > loss_disc_real_5: 0.23517774045467377 (0.24824089190717472) | > loss_0: 2.556720018386841 (2.6899627103643904) | > grad_norm_0: tensor(77.7139, device='cuda:0') (tensor(106.8663, device='cuda:0')) | > loss_gen: 2.2188963890075684 (2.018467577837282) | > loss_kl: 2.2746450901031494 (2.1164173958665224) | > loss_feat: 3.946767807006836 (3.31439263941878) | > loss_mel: 17.111854553222656 (17.446600429082313) | > loss_duration: 1.6271483898162842 (1.5803191197120536) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.179311752319336 (26.476196870965474) | > grad_norm_1: tensor(6948.8584, device='cuda:0') (tensor(5472.0200, device='cuda:0')) | > current_lr_0: 0.0001845069859636826 | > current_lr_1: 0.0001845069859636826 | > step_time: 0.4874 (0.45451880713640636) | > loader_time: 0.0098 (0.008093825841354112) --> EVAL PERFORMANCE | > avg_loader_time: 0.008665779946555556 (+1.389543775101934e-05) | > avg_step_time: 0.4605737336924378 (+0.00776993388860997) | > avg_loss_disc: 2.6870942686645076 (-0.0006585389795441543) | > avg_loss_disc_real_0: 0.13913250997872426 (-0.021938753117557885) | > avg_loss_disc_real_1: 0.22955783299157317 (-0.011010460240740177) | > avg_loss_disc_real_2: 0.2434599363887814 (+0.010880960754945296) | > avg_loss_disc_real_3: 0.237602464959655 (-0.0008389742021829294) | > avg_loss_disc_real_4: 0.2480021950224756 (+0.004138593732471246) | > avg_loss_disc_real_5: 0.24891258881125652 (+0.0030791942082660506) | > avg_loss_0: 2.6870942686645076 (-0.0006585389795441543) | > avg_grad_norm_0: tensor(116.9854, device='cuda:0') (+tensor(6.9187, device='cuda:0')) | > avg_loss_gen: 2.003614912570363 (-0.000591140397836476) | > avg_loss_kl: 2.138804831974942 (-0.2577195923093347) | > avg_loss_feat: 3.2744522564847705 (+0.032758044524931584) | > avg_loss_mel: 17.4249291890104 (-0.8275994448594695) | > avg_loss_duration: 1.5897199889303932 (-0.020591655247647944) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.431520891861176 (-1.0737440351029512) | > avg_grad_norm_1: tensor(5652.8486, device='cuda:0') (tensor(-148.2236, device='cuda:0'))  > EPOCH: 646/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:51:27)   --> TIME: 2025-02-25 14:51:45 -- STEP: 37/72 -- GLOBAL_STEP: 1046550 | > loss_disc: 2.6813974380493164 (2.645373937245961) | > loss_disc_real_0: 0.20013798773288727 (0.12797669049453092) | > loss_disc_real_1: 0.22086212038993835 (0.22789701013951688) | > loss_disc_real_2: 0.30350154638290405 (0.24971809419425758) | > loss_disc_real_3: 0.2209852635860443 (0.23871098861501022) | > loss_disc_real_4: 0.24122057855129242 (0.24630053542755745) | > loss_disc_real_5: 0.23755113780498505 (0.24553017398795565) | > loss_0: 2.6813974380493164 (2.645373937245961) | > grad_norm_0: tensor(73.9774, device='cuda:0') (tensor(102.1732, device='cuda:0')) | > loss_gen: 2.0839486122131348 (2.1097315227663187) | > loss_kl: 2.0268192291259766 (2.22688107554977) | > loss_feat: 3.2471864223480225 (3.5095859476037927) | > loss_mel: 16.857555389404297 (17.91479543737463) | > loss_duration: 1.5867009162902832 (1.5603744371517285) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.802209854125977 (27.321368294793206) | > grad_norm_1: tensor(7686.0474, device='cuda:0') (tensor(4690.0918, device='cuda:0')) | > current_lr_0: 0.00018448392259043714 | > current_lr_1: 0.00018448392259043714 | > step_time: 0.4625 (0.4383588288281415) | > loader_time: 0.0081 (0.0072239605156151025) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861331778512874 (-5.2462161426816006e-05) | > avg_step_time: 0.4567587980082337 (-0.0038149356842041016) | > avg_loss_disc: 2.6641461580571995 (-0.02294811060730817) | > avg_loss_disc_real_0: 0.13570656328344008 (-0.0034259466952841766) | > avg_loss_disc_real_1: 0.22453406663008138 (-0.005023766361491794) | > avg_loss_disc_real_2: 0.25070818692026 (+0.007248250531478567) | > avg_loss_disc_real_3: 0.24007146878981253 (+0.0024690038301575234) | > avg_loss_disc_real_4: 0.24533480657658108 (-0.0026673884458945363) | > avg_loss_disc_real_5: 0.2458506230737122 (-0.003061965737544331) | > avg_loss_0: 2.6641461580571995 (-0.02294811060730817) | > avg_grad_norm_0: tensor(110.5971, device='cuda:0') (tensor(-6.3882, device='cuda:0')) | > avg_loss_gen: 2.0608140884990416 (+0.057199175928678514) | > avg_loss_kl: 2.2549576944028824 (+0.1161528624279402) | > avg_loss_feat: 3.3564806891159273 (+0.08202843263115689) | > avg_loss_mel: 17.83445143363846 (+0.40952224462806086) | > avg_loss_duration: 1.5924956294852244 (+0.002775640554831149) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.099199429364273 (+0.6676785375030967) | > avg_grad_norm_1: tensor(5059.2363, device='cuda:0') (tensor(-593.6123, device='cuda:0'))  > EPOCH: 647/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:52:04)   --> TIME: 2025-02-25 14:52:12 -- STEP: 15/72 -- GLOBAL_STEP: 1046600 | > loss_disc: 2.782738208770752 (2.6176631768544514) | > loss_disc_real_0: 0.33440732955932617 (0.14374220867951712) | > loss_disc_real_1: 0.1508524864912033 (0.22373956739902495) | > loss_disc_real_2: 0.18117041885852814 (0.2377851108709971) | > loss_disc_real_3: 0.2038591504096985 (0.22338927884896595) | > loss_disc_real_4: 0.23086272180080414 (0.24236569702625274) | > loss_disc_real_5: 0.2315957397222519 (0.23341471056143442) | > loss_0: 2.782738208770752 (2.6176631768544514) | > grad_norm_0: tensor(181.6717, device='cuda:0') (tensor(125.8244, device='cuda:0')) | > loss_gen: 1.8213917016983032 (2.0702506701151533) | > loss_kl: 1.9975351095199585 (2.8131944576899213) | > loss_feat: 3.2910256385803223 (3.4192647139231362) | > loss_mel: 19.1026668548584 (20.02776247660319) | > loss_duration: 1.5567396879196167 (1.6197364489237467) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.769359588623047 (29.950209172566733) | > grad_norm_1: tensor(143.3983, device='cuda:0') (tensor(5314.2900, device='cuda:0')) | > current_lr_0: 0.00018446086210011334 | > current_lr_1: 0.00018446086210011334 | > step_time: 0.4387 (0.44160639444986977) | > loader_time: 0.011 (0.006756877899169922)  --> TIME: 2025-02-25 14:52:36 -- STEP: 65/72 -- GLOBAL_STEP: 1046650 | > loss_disc: 2.6060690879821777 (2.707017179635855) | > loss_disc_real_0: 0.07890981435775757 (0.1706432351699242) | > loss_disc_real_1: 0.23134157061576843 (0.25134485960006714) | > loss_disc_real_2: 0.17512603104114532 (0.2378299002464001) | > loss_disc_real_3: 0.2336682379245758 (0.22825835736898276) | > loss_disc_real_4: 0.2630084455013275 (0.24748931458363166) | > loss_disc_real_5: 0.29112470149993896 (0.24588564726022574) | > loss_0: 2.6060690879821777 (2.707017179635855) | > grad_norm_0: tensor(166.4885, device='cuda:0') (tensor(115.1969, device='cuda:0')) | > loss_gen: 2.0278637409210205 (2.1064015663587132) | > loss_kl: 2.3498878479003906 (2.7161955558336697) | > loss_feat: 3.399564504623413 (3.4311095237731934) | > loss_mel: 18.964021682739258 (19.6665891794058) | > loss_duration: 1.670480489730835 (1.678375506401062) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.411819458007812 (29.59867128225473) | > grad_norm_1: tensor(6462.9292, device='cuda:0') (tensor(6076.5776, device='cuda:0')) | > current_lr_0: 0.00018446086210011334 | > current_lr_1: 0.00018446086210011334 | > step_time: 0.5189 (0.4539712869204008) | > loader_time: 0.0118 (0.008358130088219276) --> EVAL PERFORMANCE | > avg_loader_time: 0.008648368674264828 (+3.5050889136087685e-05) | > avg_step_time: 0.45969638354341746 (+0.002937585535183751) | > avg_loss_disc: 2.7108845139893005 (+0.04673835593210107) | > avg_loss_disc_real_0: 0.17080464360999387 (+0.03509808032655379) | > avg_loss_disc_real_1: 0.24878820692989187 (+0.02425414029981049) | > avg_loss_disc_real_2: 0.23474704925443085 (-0.015961137665829128) | > avg_loss_disc_real_3: 0.22823848208071482 (-0.011832986709097715) | > avg_loss_disc_real_4: 0.24663101115696867 (+0.0012962045803875943) | > avg_loss_disc_real_5: 0.24625236992265137 (+0.00040174684893917734) | > avg_loss_0: 2.7108845139893005 (+0.04673835593210107) | > avg_grad_norm_0: tensor(115.2785, device='cuda:0') (+tensor(4.6814, device='cuda:0')) | > avg_loss_gen: 2.087585952919974 (+0.026771864420932268) | > avg_loss_kl: 2.6800830616077906 (+0.42512536720490823) | > avg_loss_feat: 3.3869437096824107 (+0.030463020566483312) | > avg_loss_mel: 19.54567326290507 (+1.7112218292666093) | > avg_loss_duration: 1.6767183626201791 (+0.08422273313495476) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 29.377004274180237 (+2.2778048448159645) | > avg_grad_norm_1: tensor(5972.3989, device='cuda:0') (+tensor(913.1626, device='cuda:0'))  > EPOCH: 648/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:52:41)   --> TIME: 2025-02-25 14:53:02 -- STEP: 43/72 -- GLOBAL_STEP: 1046700 | > loss_disc: 2.7437736988067627 (2.700723099154095) | > loss_disc_real_0: 0.3125351071357727 (0.14926397064050964) | > loss_disc_real_1: 0.15435482561588287 (0.21234885187343108) | > loss_disc_real_2: 0.241438627243042 (0.24592467241509017) | > loss_disc_real_3: 0.3211870491504669 (0.23997896771098293) | > loss_disc_real_4: 0.24719181656837463 (0.24757503146348997) | > loss_disc_real_5: 0.2674923241138458 (0.24508210427539293) | > loss_0: 2.7437736988067627 (2.700723099154095) | > grad_norm_0: tensor(21.9062, device='cuda:0') (tensor(102.7517, device='cuda:0')) | > loss_gen: 2.139484405517578 (1.9783621638320212) | > loss_kl: 2.032689094543457 (2.211192022922427) | > loss_feat: 3.1120433807373047 (3.329732567764992) | > loss_mel: 17.0805606842041 (18.446009436319038) | > loss_duration: 1.593462347984314 (1.580483170442803) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.95823860168457 (27.545779383459756) | > grad_norm_1: tensor(7755.2788, device='cuda:0') (tensor(6118.1934, device='cuda:0')) | > current_lr_0: 0.00018443780449235083 | > current_lr_1: 0.00018443780449235083 | > step_time: 0.449 (0.44059919202050496) | > loader_time: 0.0095 (0.007568947104520576) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661263425585253 (+1.2894751320425418e-05) | > avg_step_time: 0.4547036903005251 (-0.004992693242892365) | > avg_loss_disc: 2.685765837279843 (-0.025118676709457333) | > avg_loss_disc_real_0: 0.13889261576491344 (-0.03191202784508043) | > avg_loss_disc_real_1: 0.22007362324167307 (-0.028714583688218803) | > avg_loss_disc_real_2: 0.23805052881509486 (+0.0033034795606640133) | > avg_loss_disc_real_3: 0.23262883575869278 (+0.004390353677977965) | > avg_loss_disc_real_4: 0.24490861007025544 (-0.0017224010867132322) | > avg_loss_disc_real_5: 0.24461638066970126 (-0.0016359892529501097) | > avg_loss_0: 2.685765837279843 (-0.025118676709457333) | > avg_grad_norm_0: tensor(103.7945, device='cuda:0') (tensor(-11.4840, device='cuda:0')) | > avg_loss_gen: 1.9753348558721409 (-0.11225109704783298) | > avg_loss_kl: 2.1810190963073515 (-0.49906396530043917) | > avg_loss_feat: 3.3195422266570613 (-0.06740148302534932) | > avg_loss_mel: 18.159288594420534 (-1.386384668484535) | > avg_loss_duration: 1.599327434956188 (-0.07739092766399125) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.23451224850937 (-2.142492025670869) | > avg_grad_norm_1: tensor(5817.0234, device='cuda:0') (tensor(-155.3755, device='cuda:0'))  > EPOCH: 649/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:53:18)   --> TIME: 2025-02-25 14:53:29 -- STEP: 21/72 -- GLOBAL_STEP: 1046750 | > loss_disc: 2.939216136932373 (2.8179946399870373) | > loss_disc_real_0: 0.1819688230752945 (0.16068926373762743) | > loss_disc_real_1: 0.41526827216148376 (0.23017050840315365) | > loss_disc_real_2: 0.2623784840106964 (0.3303605644475846) | > loss_disc_real_3: 0.2631489038467407 (0.2633720366727738) | > loss_disc_real_4: 0.25340643525123596 (0.24933016442117234) | > loss_disc_real_5: 0.24880537390708923 (0.24645311704703735) | > loss_0: 2.939216136932373 (2.8179946399870373) | > grad_norm_0: tensor(163.3454, device='cuda:0') (tensor(134.5247, device='cuda:0')) | > loss_gen: 1.8718234300613403 (2.0366645824341547) | > loss_kl: 1.9958436489105225 (2.0400118316922864) | > loss_feat: 2.4199724197387695 (3.2279792740231468) | > loss_mel: 16.954586029052734 (17.96732502891904) | > loss_duration: 1.5518440008163452 (1.5505315178916566) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.794071197509766 (26.82251212710426) | > grad_norm_1: tensor(1695.4991, device='cuda:0') (tensor(4997.0591, device='cuda:0')) | > current_lr_0: 0.00018441474976678928 | > current_lr_1: 0.00018441474976678928 | > step_time: 0.4303 (0.42976810818626765) | > loader_time: 0.0061 (0.006834370749337333)  --> TIME: 2025-02-25 14:53:54 -- STEP: 71/72 -- GLOBAL_STEP: 1046800 | > loss_disc: 2.7292885780334473 (2.7831521773002517) | > loss_disc_real_0: 0.17868943512439728 (0.18223266243200062) | > loss_disc_real_1: 0.21704551577568054 (0.2447814551562491) | > loss_disc_real_2: 0.2307298630475998 (0.26266340050898807) | > loss_disc_real_3: 0.23459647595882416 (0.2484760213066154) | > loss_disc_real_4: 0.24390745162963867 (0.250031529597833) | > loss_disc_real_5: 0.23478640615940094 (0.24626223826912091) | > loss_0: 2.7292885780334473 (2.7831521773002517) | > grad_norm_0: tensor(18.4684, device='cuda:0') (tensor(123.6419, device='cuda:0')) | > loss_gen: 1.9228549003601074 (1.960563911518581) | > loss_kl: 1.9908478260040283 (2.0693448849127316) | > loss_feat: 2.9214792251586914 (3.0956672883369554) | > loss_mel: 17.078006744384766 (17.602379543680545) | > loss_duration: 1.5982387065887451 (1.5841883689584864) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.51142692565918 (26.3121439168151) | > grad_norm_1: tensor(5304.2378, device='cuda:0') (tensor(4481.9702, device='cuda:0')) | > current_lr_0: 0.00018441474976678928 | > current_lr_1: 0.00018441474976678928 | > step_time: 0.4909 (0.46249477628251195) | > loader_time: 0.008 (0.008770865453800689) --> EVAL PERFORMANCE | > avg_loader_time: 0.008770865453800689 (+0.00010960202821543551) | > avg_step_time: 0.46249477628251195 (+0.007791085981986856) | > avg_loss_disc: 2.7831521773002517 (+0.09738634002040847) | > avg_loss_disc_real_0: 0.18223266243200062 (+0.043340046667087184) | > avg_loss_disc_real_1: 0.2447814551562491 (+0.02470783191457604) | > avg_loss_disc_real_2: 0.26266340050898807 (+0.0246128716938932) | > avg_loss_disc_real_3: 0.2484760213066154 (+0.01584718554792261) | > avg_loss_disc_real_4: 0.250031529597833 (+0.005122919527577541) | > avg_loss_disc_real_5: 0.24626223826912091 (+0.0016458575994196534) | > avg_loss_0: 2.7831521773002517 (+0.09738634002040847) | > avg_grad_norm_0: tensor(123.6419, device='cuda:0') (+tensor(19.8473, device='cuda:0')) | > avg_loss_gen: 1.960563911518581 (-0.014770944353559967) | > avg_loss_kl: 2.0693448849127316 (-0.11167421139461986) | > avg_loss_feat: 3.0956672883369554 (-0.22387493832010597) | > avg_loss_mel: 17.602379543680545 (-0.5569090507399892) | > avg_loss_duration: 1.5841883689584864 (-0.01513906599770154) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.3121439168151 (-0.922368331694269) | > avg_grad_norm_1: tensor(4481.9702, device='cuda:0') (tensor(-1335.0532, device='cuda:0'))  > EPOCH: 650/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:53:55)   --> TIME: 2025-02-25 14:54:20 -- STEP: 49/72 -- GLOBAL_STEP: 1046850 | > loss_disc: 2.6780877113342285 (2.6655732417593203) | > loss_disc_real_0: 0.2029862254858017 (0.14146418390529497) | > loss_disc_real_1: 0.24985140562057495 (0.22311885411642035) | > loss_disc_real_2: 0.25554004311561584 (0.23523813029941248) | > loss_disc_real_3: 0.2602701485157013 (0.2306338427018146) | > loss_disc_real_4: 0.24067388474941254 (0.24076278568530568) | > loss_disc_real_5: 0.26396986842155457 (0.24614546311144925) | > loss_0: 2.6780877113342285 (2.6655732417593203) | > grad_norm_0: tensor(101.7073, device='cuda:0') (tensor(99.4466, device='cuda:0')) | > loss_gen: 2.1192715167999268 (1.9873720091216418) | > loss_kl: 2.9301600456237793 (2.1480006198493804) | > loss_feat: 2.7532799243927 (3.4058951115121645) | > loss_mel: 18.73441505432129 (17.763414246695383) | > loss_duration: 1.6999609470367432 (1.5654563271269506) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.23708724975586 (26.870138401887854) | > grad_norm_1: tensor(7624.5366, device='cuda:0') (tensor(4747.2051, device='cuda:0')) | > current_lr_0: 0.0001843916979230684 | > current_lr_1: 0.0001843916979230684 | > step_time: 0.4416 (0.44488700068726833) | > loader_time: 0.0095 (0.007698511590763014) --> EVAL PERFORMANCE | > avg_loader_time: 0.008659937012363485 (-0.00011092844143720422) | > avg_step_time: 0.45832739749424894 (-0.00416737878826301) | > avg_loss_disc: 2.6433589256985086 (-0.13979325160174305) | > avg_loss_disc_real_0: 0.14660217922548172 (-0.035630483206518904) | > avg_loss_disc_real_1: 0.22211659458321584 (-0.02266486057303327) | > avg_loss_disc_real_2: 0.23110604412119154 (-0.031557356387796526) | > avg_loss_disc_real_3: 0.22716498815677535 (-0.021311033149840036) | > avg_loss_disc_real_4: 0.2396725166431615 (-0.010359012954671482) | > avg_loss_disc_real_5: 0.24606009877063859 (-0.00020213949848232882) | > avg_loss_0: 2.6433589256985086 (-0.13979325160174305) | > avg_grad_norm_0: tensor(108.1598, device='cuda:0') (tensor(-15.4821, device='cuda:0')) | > avg_loss_gen: 1.9867697601587 (+0.0262058486401191) | > avg_loss_kl: 2.2687146898726342 (+0.19936980495990264) | > avg_loss_feat: 3.3644873424315116 (+0.2688200540945562) | > avg_loss_mel: 18.22911805166325 (+0.6267385079827044) | > avg_loss_duration: 1.5929897553484205 (+0.008801386389934107) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.442079597795512 (+1.129935680980413) | > avg_grad_norm_1: tensor(5274.7139, device='cuda:0') (+tensor(792.7437, device='cuda:0'))  > EPOCH: 651/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:54:32)   --> TIME: 2025-02-25 14:54:46 -- STEP: 27/72 -- GLOBAL_STEP: 1046900 | > loss_disc: 2.6549036502838135 (2.6549010894916676) | > loss_disc_real_0: 0.16653619706630707 (0.1513972635622378) | > loss_disc_real_1: 0.21173825860023499 (0.21978539614765732) | > loss_disc_real_2: 0.24655139446258545 (0.24015472718962916) | > loss_disc_real_3: 0.2492341250181198 (0.23625289179660655) | > loss_disc_real_4: 0.2424747794866562 (0.2407655522779182) | > loss_disc_real_5: 0.24898195266723633 (0.24482562089407886) | > loss_0: 2.6549036502838135 (2.6549010894916676) | > grad_norm_0: tensor(248.3668, device='cuda:0') (tensor(151.9448, device='cuda:0')) | > loss_gen: 1.9347378015518188 (1.950832658343845) | > loss_kl: 2.0418312549591064 (2.0615874705491244) | > loss_feat: 3.232402801513672 (3.2580147319369845) | > loss_mel: 17.662338256835938 (17.742324687816478) | > loss_duration: 1.5386263132095337 (1.5488563555258292) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.409936904907227 (26.561615837944878) | > grad_norm_1: tensor(7168.8154, device='cuda:0') (tensor(7038.4531, device='cuda:0')) | > current_lr_0: 0.00018436864896082802 | > current_lr_1: 0.00018436864896082802 | > step_time: 0.4579 (0.4368167011826127) | > loader_time: 0.009 (0.006941927803887261) --> EVAL PERFORMANCE | > avg_loader_time: 0.008643207415728502 (-1.6729596634982788e-05) | > avg_step_time: 0.457383115526656 (-0.0009442819675929415) | > avg_loss_disc: 2.6584884952491437 (+0.015129569550635047) | > avg_loss_disc_real_0: 0.15753784188082515 (+0.010935662655343431) | > avg_loss_disc_real_1: 0.21933720984929045 (-0.0027793847339253885) | > avg_loss_disc_real_2: 0.23602791550293775 (+0.004921871381746212) | > avg_loss_disc_real_3: 0.23889960516506517 (+0.011734617008289816) | > avg_loss_disc_real_4: 0.2432646205727483 (+0.0035921039295868096) | > avg_loss_disc_real_5: 0.2466704870613528 (+0.00061038829071422) | > avg_loss_0: 2.6584884952491437 (+0.015129569550635047) | > avg_grad_norm_0: tensor(158.0196, device='cuda:0') (+tensor(49.8598, device='cuda:0')) | > avg_loss_gen: 1.9531697605697202 (-0.033599999588979745) | > avg_loss_kl: 2.060887348483985 (-0.20782734138864933) | > avg_loss_feat: 3.1547889239351514 (-0.20969841849636017) | > avg_loss_mel: 17.357701234414545 (-0.8714168172487042) | > avg_loss_duration: 1.5807060540561946 (-0.01228370129222589) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.107253383582748 (-1.3348262142127645) | > avg_grad_norm_1: tensor(7056.5679, device='cuda:0') (+tensor(1781.8540, device='cuda:0'))  > EPOCH: 652/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:55:10)   --> TIME: 2025-02-25 14:55:14 -- STEP: 5/72 -- GLOBAL_STEP: 1046950 | > loss_disc: 2.6091010570526123 (2.621004247665405) | > loss_disc_real_0: 0.15214520692825317 (0.12990417778491975) | > loss_disc_real_1: 0.22901244461536407 (0.21159321963787078) | > loss_disc_real_2: 0.21490463614463806 (0.22680832743644713) | > loss_disc_real_3: 0.23610325157642365 (0.23095151484012605) | > loss_disc_real_4: 0.2578754723072052 (0.23976109623908998) | > loss_disc_real_5: 0.25591975450515747 (0.2414096862077713) | > loss_0: 2.6091010570526123 (2.621004247665405) | > grad_norm_0: tensor(136.8364, device='cuda:0') (tensor(148.9853, device='cuda:0')) | > loss_gen: 2.0480899810791016 (2.0106889247894286) | > loss_kl: 2.0348544120788574 (1.8806422233581543) | > loss_feat: 3.1606597900390625 (3.3151553153991697) | > loss_mel: 17.41563606262207 (17.37946472167969) | > loss_duration: 1.5222043991088867 (1.549797821044922) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.181446075439453 (26.13574981689453) | > grad_norm_1: tensor(7625.1255, device='cuda:0') (tensor(9163.7236, device='cuda:0')) | > current_lr_0: 0.00018434560287970792 | > current_lr_1: 0.00018434560287970792 | > step_time: 0.4214 (0.43316121101379396) | > loader_time: 0.0064 (0.007332658767700196)  --> TIME: 2025-02-25 14:55:37 -- STEP: 55/72 -- GLOBAL_STEP: 1047000 | > loss_disc: 2.522199869155884 (2.6244550011374734) | > loss_disc_real_0: 0.10689696669578552 (0.14348536187952218) | > loss_disc_real_1: 0.2227887660264969 (0.2167689336971803) | > loss_disc_real_2: 0.21810872852802277 (0.23328188955783843) | > loss_disc_real_3: 0.21509042382240295 (0.2307534629648382) | > loss_disc_real_4: 0.24259568750858307 (0.24077702110463922) | > loss_disc_real_5: 0.23931893706321716 (0.24816979684612966) | > loss_0: 2.522199869155884 (2.6244550011374734) | > grad_norm_0: tensor(91.3889, device='cuda:0') (tensor(178.8070, device='cuda:0')) | > loss_gen: 2.062920093536377 (2.0059584465893834) | > loss_kl: 1.8748128414154053 (1.9787591804157605) | > loss_feat: 3.574068546295166 (3.3665639443831012) | > loss_mel: 17.072092056274414 (17.230176440152256) | > loss_duration: 1.6245474815368652 (1.5611491680145264) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.20844078063965 (26.14260718605735) | > grad_norm_1: tensor(8967.5518, device='cuda:0') (tensor(9351.5342, device='cuda:0')) | > current_lr_0: 0.00018434560287970792 | > current_lr_1: 0.00018434560287970792 | > step_time: 0.4685 (0.44691618139093575) | > loader_time: 0.0093 (0.007892508940263228) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612733491709535 (-3.0473924018966855e-05) | > avg_step_time: 0.45955656279980295 (+0.0021734472731469445) | > avg_loss_disc: 2.6317709231040847 (-0.026717572145058988) | > avg_loss_disc_real_0: 0.14949845165853773 (-0.008039390222287418) | > avg_loss_disc_real_1: 0.2169088666707697 (-0.002428343178520753) | > avg_loss_disc_real_2: 0.2342440387312795 (-0.0017838767716582549) | > avg_loss_disc_real_3: 0.23109389526743285 (-0.007805709897632324) | > avg_loss_disc_real_4: 0.24115311134029443 (-0.002111509232453873) | > avg_loss_disc_real_5: 0.24781150276392278 (+0.0011410157025699696) | > avg_loss_0: 2.6317709231040847 (-0.026717572145058988) | > avg_grad_norm_0: tensor(184.4901, device='cuda:0') (+tensor(26.4706, device='cuda:0')) | > avg_loss_gen: 2.00262238274158 (+0.04945262217185964) | > avg_loss_kl: 1.9955822027904886 (-0.06530514569349632) | > avg_loss_feat: 3.313032224144734 (+0.15824330020958266) | > avg_loss_mel: 17.180659750817526 (-0.17704148359701932) | > avg_loss_duration: 1.574124576340259 (-0.006581477715935646) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.066021234216823 (-0.0412321493659249) | > avg_grad_norm_1: tensor(9260.3936, device='cuda:0') (+tensor(2203.8257, device='cuda:0'))  > EPOCH: 653/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:55:47)   --> TIME: 2025-02-25 14:56:04 -- STEP: 33/72 -- GLOBAL_STEP: 1047050 | > loss_disc: 2.662898063659668 (2.6396634723200942) | > loss_disc_real_0: 0.11944468319416046 (0.15085881677540866) | > loss_disc_real_1: 0.2197745144367218 (0.2196905224612265) | > loss_disc_real_2: 0.2336639016866684 (0.23250315044865463) | > loss_disc_real_3: 0.228183314204216 (0.23584685632676788) | > loss_disc_real_4: 0.23907792568206787 (0.24282252291838327) | > loss_disc_real_5: 0.24959442019462585 (0.24934162109187155) | > loss_0: 2.662898063659668 (2.6396634723200942) | > grad_norm_0: tensor(268.7335, device='cuda:0') (tensor(197.1393, device='cuda:0')) | > loss_gen: 1.9077256917953491 (2.022706277442701) | > loss_kl: 1.793013572692871 (1.9460850014831081) | > loss_feat: 3.186114549636841 (3.4641501253301445) | > loss_mel: 17.25156021118164 (17.365688092780836) | > loss_duration: 1.5428699254989624 (1.5337704347841667) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.681283950805664 (26.332400293061227) | > grad_norm_1: tensor(10379.6201, device='cuda:0') (tensor(9719.2871, device='cuda:0')) | > current_lr_0: 0.00018432255967934795 | > current_lr_1: 0.00018432255967934795 | > step_time: 0.4426 (0.43573578921231354) | > loader_time: 0.0079 (0.0070096723961107655) --> EVAL PERFORMANCE | > avg_loader_time: 0.008513957681790203 (-9.877580991933185e-05) | > avg_step_time: 0.45380501679971186 (-0.00575154600009109) | > avg_loss_disc: 2.661826079999897 (+0.030055156895812285) | > avg_loss_disc_real_0: 0.15898597717914784 (+0.009487525520610107) | > avg_loss_disc_real_1: 0.22757630721783978 (+0.010667440547070084) | > avg_loss_disc_real_2: 0.23918555320148738 (+0.004941514470207881) | > avg_loss_disc_real_3: 0.23629295049418866 (+0.0051990552267558166) | > avg_loss_disc_real_4: 0.2416243160694418 (+0.00047120472914735645) | > avg_loss_disc_real_5: 0.24801009998355114 (+0.00019859721962836585) | > avg_loss_0: 2.661826079999897 (+0.030055156895812285) | > avg_grad_norm_0: tensor(179.8734, device='cuda:0') (tensor(-4.6167, device='cuda:0')) | > avg_loss_gen: 2.0800882359625588 (+0.07746585322097888) | > avg_loss_kl: 2.073922311755972 (+0.0783401089654836) | > avg_loss_feat: 3.448872532643063 (+0.13584030849832907) | > avg_loss_mel: 17.396997075685313 (+0.21633732486778712) | > avg_loss_duration: 1.5773514727471583 (+0.0032268964068993267) | > avg_amp_scaler: 12.619718309859154 (-3.380281690140846) | > avg_loss_1: 26.577231796694473 (+0.5112105624776504) | > avg_grad_norm_1: tensor(8083.2842, device='cuda:0') (tensor(-1177.1094, device='cuda:0'))  > EPOCH: 654/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:56:24)   --> TIME: 2025-02-25 14:56:30 -- STEP: 11/72 -- GLOBAL_STEP: 1047100 | > loss_disc: 2.5545337200164795 (2.578638943758878) | > loss_disc_real_0: 0.0698033794760704 (0.08647282929583029) | > loss_disc_real_1: 0.2969202995300293 (0.22102912854064594) | > loss_disc_real_2: 0.23989181220531464 (0.23168355361981827) | > loss_disc_real_3: 0.22711610794067383 (0.22823217104781757) | > loss_disc_real_4: 0.2101624310016632 (0.24181549792939966) | > loss_disc_real_5: 0.2152990996837616 (0.24914706566117026) | > loss_0: 2.5545337200164795 (2.578638943758878) | > grad_norm_0: tensor(139.9381, device='cuda:0') (tensor(158.3231, device='cuda:0')) | > loss_gen: 1.9821139574050903 (2.0954957658594306) | > loss_kl: 1.8589770793914795 (1.9720904935490002) | > loss_feat: 3.5346062183380127 (3.666892571882768) | > loss_mel: 17.153297424316406 (17.342680844393644) | > loss_duration: 1.5142152309417725 (1.542479699308222) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.043210983276367 (26.619639656760476) | > grad_norm_1: tensor(5903.2925, device='cuda:0') (tensor(6890.0015, device='cuda:0')) | > current_lr_0: 0.00018429951935938803 | > current_lr_1: 0.00018429951935938803 | > step_time: 0.4241 (0.4312370040199973) | > loader_time: 0.0056 (0.006625500592318448)  --> TIME: 2025-02-25 14:56:53 -- STEP: 61/72 -- GLOBAL_STEP: 1047150 | > loss_disc: 2.5635738372802734 (2.602234152496838) | > loss_disc_real_0: 0.09412257373332977 (0.11622828568835727) | > loss_disc_real_1: 0.22201327979564667 (0.22516703605651855) | > loss_disc_real_2: 0.23588970303535461 (0.23544014428482682) | > loss_disc_real_3: 0.2384341061115265 (0.2358826144796903) | > loss_disc_real_4: 0.26849645376205444 (0.2432617916435492) | > loss_disc_real_5: 0.2583678066730499 (0.24686467207846094) | > loss_0: 2.5635738372802734 (2.602234152496838) | > grad_norm_0: tensor(149.6012, device='cuda:0') (tensor(146.7396, device='cuda:0')) | > loss_gen: 1.9924404621124268 (2.138036419133671) | > loss_kl: 2.0122992992401123 (2.009145982929917) | > loss_feat: 3.3515625 (3.5768658919412584) | > loss_mel: 16.996755599975586 (17.075764405922808) | > loss_duration: 1.611248254776001 (1.5637862389204933) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.96430778503418 (26.363599276933513) | > grad_norm_1: tensor(6335.5059, device='cuda:0') (tensor(6710.7168, device='cuda:0')) | > current_lr_0: 0.00018429951935938803 | > current_lr_1: 0.00018429951935938803 | > step_time: 0.4565 (0.44290108758895125) | > loader_time: 0.0107 (0.00816007520331711) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619862543025486 (+0.00010590486123528316) | > avg_step_time: 0.4526332062734684 (-0.001171810526243433) | > avg_loss_disc: 2.602861434641019 (-0.05896464535887791) | > avg_loss_disc_real_0: 0.11520948549601394 (-0.0437764916831339) | > avg_loss_disc_real_1: 0.22577767976572816 (-0.001798627452111623) | > avg_loss_disc_real_2: 0.23385445374838063 (-0.005331099453106752) | > avg_loss_disc_real_3: 0.23417232426959025 (-0.0021206262245984098) | > avg_loss_disc_real_4: 0.242587922744348 (+0.00096360667490622) | > avg_loss_disc_real_5: 0.24577014714899198 (-0.0022399528345591613) | > avg_loss_0: 2.602861434641019 (-0.05896464535887791) | > avg_grad_norm_0: tensor(144.3432, device='cuda:0') (tensor(-35.5303, device='cuda:0')) | > avg_loss_gen: 2.131067541283621 (+0.050979305321062096) | > avg_loss_kl: 2.0253000292979495 (-0.04862228245802269) | > avg_loss_feat: 3.558178965474518 (+0.1093064328314548) | > avg_loss_mel: 17.06467836675509 (-0.33231870893022375) | > avg_loss_duration: 1.5724406376690931 (-0.004910835078065112) | > avg_amp_scaler: 8.0 (-4.619718309859154) | > avg_loss_1: 26.35166579233089 (-0.2255660043635821) | > avg_grad_norm_1: tensor(6685.5332, device='cuda:0') (tensor(-1397.7510, device='cuda:0'))  > EPOCH: 655/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:57:00)   --> TIME: 2025-02-25 14:57:20 -- STEP: 39/72 -- GLOBAL_STEP: 1047200 | > loss_disc: 2.4466869831085205 (2.5716974123930316) | > loss_disc_real_0: 0.16346782445907593 (0.09706592321013793) | > loss_disc_real_1: 0.24636287987232208 (0.22161283439550644) | > loss_disc_real_2: 0.2336902767419815 (0.23640940548517766) | > loss_disc_real_3: 0.22855035960674286 (0.2337399965677506) | > loss_disc_real_4: 0.24536867439746857 (0.23893906672795615) | > loss_disc_real_5: 0.2527257800102234 (0.2451044955314734) | > loss_0: 2.4466869831085205 (2.5716974123930316) | > grad_norm_0: tensor(110.6920, device='cuda:0') (tensor(144.9384, device='cuda:0')) | > loss_gen: 2.373162269592285 (2.1682171485362915) | > loss_kl: 2.1378846168518066 (2.005431878261077) | > loss_feat: 3.4735991954803467 (3.713550555400359) | > loss_mel: 16.457555770874023 (17.020595623896668) | > loss_duration: 1.5796151161193848 (1.538759274360461) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.021814346313477 (26.446554477398212) | > grad_norm_1: tensor(6642.1963, device='cuda:0') (tensor(7387.0938, device='cuda:0')) | > current_lr_0: 0.0001842764819194681 | > current_lr_1: 0.0001842764819194681 | > step_time: 0.4468 (0.4430224345280574) | > loader_time: 0.0082 (0.00752404408577161) --> EVAL PERFORMANCE | > avg_loader_time: 0.008830315630200886 (+0.00021045308717539991) | > avg_step_time: 0.4569196902530294 (+0.0042864839795609955) | > avg_loss_disc: 2.5999287618717677 (-0.0029326727692513543) | > avg_loss_disc_real_0: 0.12279546559906342 (+0.007585980103049478) | > avg_loss_disc_real_1: 0.221412245236652 (-0.004365434529076145) | > avg_loss_disc_real_2: 0.2330626022647804 (-0.0007918514836002255) | > avg_loss_disc_real_3: 0.2353783086152144 (+0.0012059843456241337) | > avg_loss_disc_real_4: 0.24083253321513323 (-0.0017553895292147792) | > avg_loss_disc_real_5: 0.24542924992635218 (-0.00034089722263980216) | > avg_loss_0: 2.5999287618717677 (-0.0029326727692513543) | > avg_grad_norm_0: tensor(131.5788, device='cuda:0') (tensor(-12.7644, device='cuda:0')) | > avg_loss_gen: 2.1547934673201867 (+0.02372592603656587) | > avg_loss_kl: 2.045326511624833 (+0.02002648232688342) | > avg_loss_feat: 3.5961625273798554 (+0.03798356190533747) | > avg_loss_mel: 17.045205734145473 (-0.019472632609616625) | > avg_loss_duration: 1.5697992486013492 (-0.0026413890677439156) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.41128746892365 (+0.05962167659275863) | > avg_grad_norm_1: tensor(7007.4307, device='cuda:0') (+tensor(321.8975, device='cuda:0'))  > EPOCH: 656/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:57:38)   --> TIME: 2025-02-25 14:57:48 -- STEP: 17/72 -- GLOBAL_STEP: 1047250 | > loss_disc: 2.4499120712280273 (2.56255542530733) | > loss_disc_real_0: 0.0665806382894516 (0.08496308282894247) | > loss_disc_real_1: 0.16645091772079468 (0.21698082983493805) | > loss_disc_real_2: 0.1594872772693634 (0.2162849543725743) | > loss_disc_real_3: 0.2895539402961731 (0.23904715390766368) | > loss_disc_real_4: 0.24911032617092133 (0.24358281054917505) | > loss_disc_real_5: 0.2602075934410095 (0.24149152285912456) | > loss_0: 2.4499120712280273 (2.56255542530733) | > grad_norm_0: tensor(86.7208, device='cuda:0') (tensor(133.4376, device='cuda:0')) | > loss_gen: 2.1139750480651855 (2.159571696730221) | > loss_kl: 2.262972354888916 (1.9692496552186853) | > loss_feat: 4.3454413414001465 (4.029311460607192) | > loss_mel: 17.062570571899414 (17.244928135591394) | > loss_duration: 1.5645184516906738 (1.5299681074479048) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.349477767944336 (26.93302895041073) | > grad_norm_1: tensor(8965.9141, device='cuda:0') (tensor(8300.8105, device='cuda:0')) | > current_lr_0: 0.00018425344735922817 | > current_lr_1: 0.00018425344735922817 | > step_time: 0.4539 (0.44252796734080596) | > loader_time: 0.0093 (0.007164702695958754)  --> TIME: 2025-02-25 14:58:11 -- STEP: 67/72 -- GLOBAL_STEP: 1047300 | > loss_disc: 2.94046950340271 (2.6196273796593963) | > loss_disc_real_0: 0.08168197423219681 (0.10175081662166474) | > loss_disc_real_1: 0.2971438467502594 (0.22552152511788837) | > loss_disc_real_2: 0.3154671788215637 (0.22257844836854224) | > loss_disc_real_3: 0.24420998990535736 (0.24053188024172142) | > loss_disc_real_4: 0.24670018255710602 (0.2421849436724364) | > loss_disc_real_5: 0.22797799110412598 (0.24582506026794662) | > loss_0: 2.94046950340271 (2.6196273796593963) | > grad_norm_0: tensor(182.6371, device='cuda:0') (tensor(136.6200, device='cuda:0')) | > loss_gen: 1.682297706604004 (2.159264107248676) | > loss_kl: 2.1562867164611816 (2.0292402327950327) | > loss_feat: 3.2036375999450684 (3.7916734859124936) | > loss_mel: 16.997825622558594 (17.010061150166525) | > loss_duration: 1.6694962978363037 (1.5682529371176193) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.709543228149414 (26.55849194882521) | > grad_norm_1: tensor(7206.0342, device='cuda:0') (tensor(7779.7617, device='cuda:0')) | > current_lr_0: 0.00018425344735922817 | > current_lr_1: 0.00018425344735922817 | > step_time: 0.5257 (0.45566037875502857) | > loader_time: 0.0127 (0.008762331151250583) --> EVAL PERFORMANCE | > avg_loader_time: 0.008918013371212381 (+8.769774101149465e-05) | > avg_step_time: 0.45975914807386803 (+0.002839457820838609) | > avg_loss_disc: 2.6344875684926206 (+0.03455880662085287) | > avg_loss_disc_real_0: 0.10162634998035262 (-0.021169115618710793) | > avg_loss_disc_real_1: 0.22699046890500565 (+0.0055782236683536435) | > avg_loss_disc_real_2: 0.22437857007476647 (-0.00868403219001393) | > avg_loss_disc_real_3: 0.24178368902542222 (+0.006405380410207828) | > avg_loss_disc_real_4: 0.2421785490193837 (+0.0013460158042504777) | > avg_loss_disc_real_5: 0.2458172121518095 (+0.0003879622254573112) | > avg_loss_0: 2.6344875684926206 (+0.03455880662085287) | > avg_grad_norm_0: tensor(135.3406, device='cuda:0') (+tensor(3.7619, device='cuda:0')) | > avg_loss_gen: 2.144403024458549 (-0.010390442861637705) | > avg_loss_kl: 2.039579033851625 (-0.005747477773208054) | > avg_loss_feat: 3.762327744927205 (+0.16616521754734936) | > avg_loss_mel: 17.046280551964127 (+0.0010748178186545942) | > avg_loss_duration: 1.5719340099415309 (+0.0021347613401816457) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.564524368501047 (+0.15323689957739717) | > avg_grad_norm_1: tensor(7592.4653, device='cuda:0') (+tensor(585.0347, device='cuda:0'))  > EPOCH: 657/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:58:15)   --> TIME: 2025-02-25 14:58:38 -- STEP: 45/72 -- GLOBAL_STEP: 1047350 | > loss_disc: 2.8253860473632812 (2.7002167648739284) | > loss_disc_real_0: 0.10636341571807861 (0.1332051623198721) | > loss_disc_real_1: 0.21437790989875793 (0.25474141910672193) | > loss_disc_real_2: 0.21608129143714905 (0.23173947003152634) | > loss_disc_real_3: 0.23978762328624725 (0.24009082913398738) | > loss_disc_real_4: 0.2478226125240326 (0.2421564151843389) | > loss_disc_real_5: 0.2599627375602722 (0.23626560899946425) | > loss_0: 2.8253860473632812 (2.7002167648739284) | > grad_norm_0: tensor(61.5614, device='cuda:0') (tensor(158.9919, device='cuda:0')) | > loss_gen: 1.6732866764068604 (1.9621692498524983) | > loss_kl: 2.133817672729492 (2.605171706941393) | > loss_feat: 2.6998908519744873 (3.2582635667588975) | > loss_mel: 18.029808044433594 (19.377695465087896) | > loss_duration: 1.5836232900619507 (1.5936584843529595) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.12042808532715 (28.796958499484592) | > grad_norm_1: tensor(3828.3867, device='cuda:0') (tensor(8783.1172, device='cuda:0')) | > current_lr_0: 0.00018423041567830825 | > current_lr_1: 0.00018423041567830825 | > step_time: 0.4627 (0.4428015338049995) | > loader_time: 0.0088 (0.0075975312127007375) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685958217567121 (-0.00023205515364525982) | > avg_step_time: 0.45830548313302055 (-0.0014536649408474767) | > avg_loss_disc: 2.708806706146455 (+0.07431913765383458) | > avg_loss_disc_real_0: 0.13907706475173923 (+0.0374507147713866) | > avg_loss_disc_real_1: 0.2458682232864306 (+0.018877754381424944) | > avg_loss_disc_real_2: 0.23376110175126036 (+0.009382531676493888) | > avg_loss_disc_real_3: 0.23790121980955903 (-0.003882469215863188) | > avg_loss_disc_real_4: 0.24216141793089854 (-1.71310884851672e-05) | > avg_loss_disc_real_5: 0.23800506390316387 (-0.007812148248645623) | > avg_loss_0: 2.708806706146455 (+0.07431913765383458) | > avg_grad_norm_0: tensor(158.2916, device='cuda:0') (+tensor(22.9510, device='cuda:0')) | > avg_loss_gen: 1.9556471461981115 (-0.1887558782604375) | > avg_loss_kl: 2.4762961931631597 (+0.4367171593115349) | > avg_loss_feat: 3.255358699341895 (-0.50696904558531) | > avg_loss_mel: 18.755846507112746 (+1.7095659551486193) | > avg_loss_duration: 1.6050855458622248 (+0.033151535920693975) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.048234079925106 (+1.4837097114240585) | > avg_grad_norm_1: tensor(8666.9600, device='cuda:0') (+tensor(1074.4946, device='cuda:0'))  > EPOCH: 658/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:58:52)   --> TIME: 2025-02-25 14:59:05 -- STEP: 23/72 -- GLOBAL_STEP: 1047400 | > loss_disc: 2.5579073429107666 (2.635085479072903) | > loss_disc_real_0: 0.1528465747833252 (0.1385641885192498) | > loss_disc_real_1: 0.23909126222133636 (0.2219829747210378) | > loss_disc_real_2: 0.2253822535276413 (0.23603211213713107) | > loss_disc_real_3: 0.23654943704605103 (0.23461207164370496) | > loss_disc_real_4: 0.25747543573379517 (0.24764629550602124) | > loss_disc_real_5: 0.26290076971054077 (0.24915368595848914) | > loss_0: 2.5579073429107666 (2.635085479072903) | > grad_norm_0: tensor(75.8455, device='cuda:0') (tensor(161.1005, device='cuda:0')) | > loss_gen: 2.1516430377960205 (1.985175635503686) | > loss_kl: 1.9094996452331543 (2.0230972196744834) | > loss_feat: 3.1009674072265625 (3.304031703783118) | > loss_mel: 16.35182762145996 (17.425075033436652) | > loss_duration: 1.5556361675262451 (1.5347268011258997) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.0695743560791 (26.272106170654297) | > grad_norm_1: tensor(8877.3262, device='cuda:0') (tensor(10038.1758, device='cuda:0')) | > current_lr_0: 0.00018420738687634846 | > current_lr_1: 0.00018420738687634846 | > step_time: 0.4403 (0.43607842403909436) | > loader_time: 0.0068 (0.007023158280745796) --> EVAL PERFORMANCE | > avg_loader_time: 0.008748467539397764 (+6.250932183064252e-05) | > avg_step_time: 0.45719191390024105 (-0.0011135692327794988) | > avg_loss_disc: 2.732617112952219 (+0.02381040680576385) | > avg_loss_disc_real_0: 0.1671971321945459 (+0.02812006744280668) | > avg_loss_disc_real_1: 0.2481657353715158 (+0.0022975120850851904) | > avg_loss_disc_real_2: 0.2352008189953549 (+0.0014397172440945294) | > avg_loss_disc_real_3: 0.2369439803378683 (-0.0009572394716907406) | > avg_loss_disc_real_4: 0.24281250821872497 (+0.0006510902878264224) | > avg_loss_disc_real_5: 0.24394476392739256 (+0.005939700024228695) | > avg_loss_0: 2.732617112952219 (+0.02381040680576385) | > avg_grad_norm_0: tensor(175.4465, device='cuda:0') (+tensor(17.1549, device='cuda:0')) | > avg_loss_gen: 1.9712610798822323 (+0.015613933684120784) | > avg_loss_kl: 2.171498132423615 (-0.30479806073954485) | > avg_loss_feat: 3.1789772376208236 (-0.07638146172107119) | > avg_loss_mel: 17.51930446356116 (-1.2365420435515873) | > avg_loss_duration: 1.5811777081288083 (-0.023907837733416537) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.4222184033461 (-1.626015676579005) | > avg_grad_norm_1: tensor(8289.3096, device='cuda:0') (tensor(-377.6504, device='cuda:0'))  > EPOCH: 659/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 14:59:30)   --> TIME: 2025-02-25 14:59:32 -- STEP: 1/72 -- GLOBAL_STEP: 1047450 | > loss_disc: 3.2601876258850098 (3.2601876258850098) | > loss_disc_real_0: 0.6508680582046509 (0.6508680582046509) | > loss_disc_real_1: 0.1463214010000229 (0.1463214010000229) | > loss_disc_real_2: 0.3313547968864441 (0.3313547968864441) | > loss_disc_real_3: 0.24514883756637573 (0.24514883756637573) | > loss_disc_real_4: 0.3199109137058258 (0.3199109137058258) | > loss_disc_real_5: 0.23788416385650635 (0.23788416385650635) | > loss_0: 3.2601876258850098 (3.2601876258850098) | > grad_norm_0: tensor(475.2249, device='cuda:0') (tensor(475.2249, device='cuda:0')) | > loss_gen: 2.103147506713867 (2.103147506713867) | > loss_kl: 2.295790195465088 (2.295790195465088) | > loss_feat: 3.2362682819366455 (3.2362682819366455) | > loss_mel: 20.43534278869629 (20.43534278869629) | > loss_duration: 1.6034796237945557 (1.6034796237945557) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.674028396606445 (29.674028396606445) | > grad_norm_1: tensor(2300.8804, device='cuda:0') (tensor(2300.8804, device='cuda:0')) | > current_lr_0: 0.0001841843609529889 | > current_lr_1: 0.0001841843609529889 | > step_time: 0.4237 (0.42368149757385254) | > loader_time: 0.0049 (0.00485539436340332)  --> TIME: 2025-02-25 14:59:55 -- STEP: 51/72 -- GLOBAL_STEP: 1047500 | > loss_disc: 2.64046311378479 (2.746059600044699) | > loss_disc_real_0: 0.2004137933254242 (0.163728536745789) | > loss_disc_real_1: 0.21997535228729248 (0.2052971884316089) | > loss_disc_real_2: 0.2270134687423706 (0.24348840000582675) | > loss_disc_real_3: 0.24388647079467773 (0.2336045538677889) | > loss_disc_real_4: 0.24391138553619385 (0.24623887328540578) | > loss_disc_real_5: 0.2561681568622589 (0.2462852220324909) | > loss_0: 2.64046311378479 (2.746059600044699) | > grad_norm_0: tensor(79.4728, device='cuda:0') (tensor(153.2965, device='cuda:0')) | > loss_gen: 2.0027568340301514 (1.8662308127272362) | > loss_kl: 2.0134170055389404 (2.2401808336669324) | > loss_feat: 3.4162423610687256 (3.1769601457259236) | > loss_mel: 17.53052520751953 (18.384878943948195) | > loss_duration: 1.6165884733200073 (1.5789684290979422) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.57952880859375 (27.24721908569336) | > grad_norm_1: tensor(6890.2158, device='cuda:0') (tensor(6350.4922, device='cuda:0')) | > current_lr_0: 0.0001841843609529889 | > current_lr_1: 0.0001841843609529889 | > step_time: 0.4528 (0.45358674666460824) | > loader_time: 0.0088 (0.007750240026735792) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863033952847333 (-0.00011812801092443356) | > avg_step_time: 0.463741359576373 (+0.006549445676131971) | > avg_loss_disc: 2.7239021717662544 (-0.008714941185964609) | > avg_loss_disc_real_0: 0.16578204979674077 (-0.0014150823978051341) | > avg_loss_disc_real_1: 0.20859168847681772 (-0.03957404689469807) | > avg_loss_disc_real_2: 0.2432337380211118 (+0.008032919025756924) | > avg_loss_disc_real_3: 0.23282833670226621 (-0.004115643635602073) | > avg_loss_disc_real_4: 0.24444954558997087 (+0.001637037371245903) | > avg_loss_disc_real_5: 0.2444412498406961 (+0.0004964859133035349) | > avg_loss_0: 2.7239021717662544 (-0.008714941185964609) | > avg_grad_norm_0: tensor(158.3305, device='cuda:0') (tensor(-17.1161, device='cuda:0')) | > avg_loss_gen: 1.8829526464704056 (-0.0883084334118267) | > avg_loss_kl: 2.214110428178814 (+0.04261229575519909) | > avg_loss_feat: 3.1616783343570334 (-0.017298903263790244) | > avg_loss_mel: 18.133729263090753 (+0.6144247995295942) | > avg_loss_duration: 1.5904304192099772 (+0.009252711081168874) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.982900968739685 (+0.5606825653935843) | > avg_grad_norm_1: tensor(6703.6997, device='cuda:0') (tensor(-1585.6099, device='cuda:0'))  > EPOCH: 660/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:00:07)   --> TIME: 2025-02-25 15:00:22 -- STEP: 29/72 -- GLOBAL_STEP: 1047550 | > loss_disc: 2.610653877258301 (2.6900351951862205) | > loss_disc_real_0: 0.1306440234184265 (0.15313116319734474) | > loss_disc_real_1: 0.20932182669639587 (0.21462109684944153) | > loss_disc_real_2: 0.23049134016036987 (0.22613481920340966) | > loss_disc_real_3: 0.24906980991363525 (0.23431328812549854) | > loss_disc_real_4: 0.25559157133102417 (0.23879530923119907) | > loss_disc_real_5: 0.24338676035404205 (0.2428521020659085) | > loss_0: 2.610653877258301 (2.6900351951862205) | > grad_norm_0: tensor(53.9683, device='cuda:0') (tensor(177.5204, device='cuda:0')) | > loss_gen: 1.9698901176452637 (1.9188550875104706) | > loss_kl: 1.8179478645324707 (1.9726533314277386) | > loss_feat: 3.2048301696777344 (3.2945062620886443) | > loss_mel: 16.805875778198242 (17.14076249352817) | > loss_duration: 1.5567415952682495 (1.5333586109095607) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.35528564453125 (25.86013570325128) | > grad_norm_1: tensor(9600.7012, device='cuda:0') (tensor(8628.9141, device='cuda:0')) | > current_lr_0: 0.00018416133790786978 | > current_lr_1: 0.00018416133790786978 | > step_time: 0.4411 (0.43914980723940095) | > loader_time: 0.0071 (0.00677154804098195) --> EVAL PERFORMANCE | > avg_loader_time: 0.008622810874186773 (-7.528654286556927e-06) | > avg_step_time: 0.4603927471268345 (-0.003348612449538546) | > avg_loss_disc: 2.7116518390010778 (-0.01225033276517662) | > avg_loss_disc_real_0: 0.16612949306276484 (+0.0003474432660240667) | > avg_loss_disc_real_1: 0.21369181311046576 (+0.005100124633648034) | > avg_loss_disc_real_2: 0.23350169792981215 (-0.00973204009129966) | > avg_loss_disc_real_3: 0.236680849337242 (+0.0038525126349757965) | > avg_loss_disc_real_4: 0.24027745089900326 (-0.004172094690967604) | > avg_loss_disc_real_5: 0.243151048119639 (-0.0012902017210570937) | > avg_loss_0: 2.7116518390010778 (-0.01225033276517662) | > avg_grad_norm_0: tensor(164.2757, device='cuda:0') (+tensor(5.9453, device='cuda:0')) | > avg_loss_gen: 1.9783669844479628 (+0.09541433797755716) | > avg_loss_kl: 2.269048818400208 (+0.054938390221393796) | > avg_loss_feat: 3.3505829320827 (+0.1889045977256667) | > avg_loss_mel: 18.321622432117735 (+0.1878931690269816) | > avg_loss_duration: 1.6401220442543567 (+0.04969162504437952) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.559743370808345 (+0.5768424020686602) | > avg_grad_norm_1: tensor(6824.6172, device='cuda:0') (+tensor(120.9175, device='cuda:0'))  > EPOCH: 661/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:00:45)   --> TIME: 2025-02-25 15:00:50 -- STEP: 7/72 -- GLOBAL_STEP: 1047600 | > loss_disc: 2.575976610183716 (2.6442818301064626) | > loss_disc_real_0: 0.13772237300872803 (0.12081918652568545) | > loss_disc_real_1: 0.20177596807479858 (0.22032370737620763) | > loss_disc_real_2: 0.2278919368982315 (0.2430582514830998) | > loss_disc_real_3: 0.25034019351005554 (0.2499272440161024) | > loss_disc_real_4: 0.2580932378768921 (0.2793594130447933) | > loss_disc_real_5: 0.2670672535896301 (0.2647692390850612) | > loss_0: 2.575976610183716 (2.6442818301064626) | > grad_norm_0: tensor(235.2414, device='cuda:0') (tensor(111.8951, device='cuda:0')) | > loss_gen: 1.8459079265594482 (2.0996973514556885) | > loss_kl: 2.0056612491607666 (2.171959842954363) | > loss_feat: 2.9900577068328857 (3.3930018629346574) | > loss_mel: 17.892072677612305 (18.21490696498326) | > loss_duration: 1.5814080238342285 (1.5794917855943953) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.315105438232422 (27.4590573992048) | > grad_norm_1: tensor(6392.5884, device='cuda:0') (tensor(6831.7139, device='cuda:0')) | > current_lr_0: 0.00018413831774063128 | > current_lr_1: 0.00018413831774063128 | > step_time: 0.4319 (0.4321305751800537) | > loader_time: 0.0057 (0.006411858967372349)  --> TIME: 2025-02-25 15:01:13 -- STEP: 57/72 -- GLOBAL_STEP: 1047650 | > loss_disc: 2.6518425941467285 (2.676060776961477) | > loss_disc_real_0: 0.16137349605560303 (0.13221782455710984) | > loss_disc_real_1: 0.24271273612976074 (0.2260821221167581) | > loss_disc_real_2: 0.23875835537910461 (0.2373583899778232) | > loss_disc_real_3: 0.2515401244163513 (0.23582450993228377) | > loss_disc_real_4: 0.2315574586391449 (0.2453790139733699) | > loss_disc_real_5: 0.24561138451099396 (0.24583266727757036) | > loss_0: 2.6518425941467285 (2.676060776961477) | > grad_norm_0: tensor(171.5809, device='cuda:0') (tensor(118.6862, device='cuda:0')) | > loss_gen: 1.9562709331512451 (2.018780641388475) | > loss_kl: 1.9698503017425537 (2.1428727852670773) | > loss_feat: 3.154536724090576 (3.3644655671036032) | > loss_mel: 17.059072494506836 (18.028352536653212) | > loss_duration: 1.6135995388031006 (1.5851776850850958) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.75333023071289 (27.139649106745136) | > grad_norm_1: tensor(4458.6494, device='cuda:0') (tensor(5100.6724, device='cuda:0')) | > current_lr_0: 0.00018413831774063128 | > current_lr_1: 0.00018413831774063128 | > step_time: 0.4626 (0.4481347778387237) | > loader_time: 0.0103 (0.007988352524606805) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865200203909001 (+2.919116490323649e-05) | > avg_step_time: 0.4590444195438439 (-0.0013483275829905828) | > avg_loss_disc: 2.6753357967860256 (-0.036316042215052136) | > avg_loss_disc_real_0: 0.1254429082373078 (-0.040686584825457034) | > avg_loss_disc_real_1: 0.23227574980594742 (+0.018583936695481668) | > avg_loss_disc_real_2: 0.2342547314687514 (+0.0007530335389392406) | > avg_loss_disc_real_3: 0.24135634450005813 (+0.004675495162816123) | > avg_loss_disc_real_4: 0.24606236584589514 (+0.005784914946891873) | > avg_loss_disc_real_5: 0.24676481959685473 (+0.003613771477215727) | > avg_loss_0: 2.6753357967860256 (-0.036316042215052136) | > avg_grad_norm_0: tensor(123.1545, device='cuda:0') (tensor(-41.1212, device='cuda:0')) | > avg_loss_gen: 2.021199456402953 (+0.04283247195499018) | > avg_loss_kl: 2.1450639106857947 (-0.12398490771441306) | > avg_loss_feat: 3.394664589787873 (+0.044081657705172805) | > avg_loss_mel: 17.96580876095194 (-0.35581367116579443) | > avg_loss_duration: 1.592897999454552 (-0.04722404479980469) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.11963465515996 (-0.4401087156483854) | > avg_grad_norm_1: tensor(5200.6729, device='cuda:0') (tensor(-1623.9443, device='cuda:0'))  > EPOCH: 662/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:01:22)   --> TIME: 2025-02-25 15:01:40 -- STEP: 35/72 -- GLOBAL_STEP: 1047700 | > loss_disc: 2.8835456371307373 (2.6455512455531527) | > loss_disc_real_0: 0.10560740530490875 (0.11032245968069349) | > loss_disc_real_1: 0.2065168023109436 (0.23223477687154498) | > loss_disc_real_2: 0.40743333101272583 (0.2477525851556233) | > loss_disc_real_3: 0.18486684560775757 (0.22016691480364117) | > loss_disc_real_4: 0.24755391478538513 (0.23694096846239907) | > loss_disc_real_5: 0.25732243061065674 (0.24178075875554766) | > loss_0: 2.8835456371307373 (2.6455512455531527) | > grad_norm_0: tensor(296.8320, device='cuda:0') (tensor(149.7753, device='cuda:0')) | > loss_gen: 1.8005740642547607 (2.059010655539377) | > loss_kl: 2.0039851665496826 (1.9921325819832938) | > loss_feat: 3.029630184173584 (3.6781282901763914) | > loss_mel: 17.124488830566406 (17.150661986214775) | > loss_duration: 1.5816550254821777 (1.5405141932623727) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.540332794189453 (26.42044792175293) | > grad_norm_1: tensor(5799.9600, device='cuda:0') (tensor(6515.3276, device='cuda:0')) | > current_lr_0: 0.0001841153004509137 | > current_lr_1: 0.0001841153004509137 | > step_time: 0.4628 (0.4536275863647461) | > loader_time: 0.0082 (0.007158061436244419) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628025860853598 (-2.3976178236411946e-05) | > avg_step_time: 0.4634189504972646 (+0.004374530953420719) | > avg_loss_disc: 2.642161265225477 (-0.033174531560548726) | > avg_loss_disc_real_0: 0.09904642744173467 (-0.026396480795573135) | > avg_loss_disc_real_1: 0.22984079177110966 (-0.0024349580348377586) | > avg_loss_disc_real_2: 0.24688779730612123 (+0.012633065837369839) | > avg_loss_disc_real_3: 0.2168494935186816 (-0.024506850981376532) | > avg_loss_disc_real_4: 0.2359053803161836 (-0.010156985529711532) | > avg_loss_disc_real_5: 0.24144715091712038 (-0.00531766867973435) | > avg_loss_0: 2.642161265225477 (-0.033174531560548726) | > avg_grad_norm_0: tensor(150.8353, device='cuda:0') (+tensor(27.6809, device='cuda:0')) | > avg_loss_gen: 2.041221029321914 (+0.020021572918961006) | > avg_loss_kl: 2.0056455806947087 (-0.13941832999108605) | > avg_loss_feat: 3.610105447366204 (+0.2154408575783311) | > avg_loss_mel: 16.995070014201424 (-0.9707387467505164) | > avg_loss_duration: 1.571544046133337 (-0.021353953321215036) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.223586176482723 (-0.8960484786772369) | > avg_grad_norm_1: tensor(6578.6021, device='cuda:0') (+tensor(1377.9292, device='cuda:0'))  > EPOCH: 663/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:01:59)   --> TIME: 2025-02-25 15:02:07 -- STEP: 13/72 -- GLOBAL_STEP: 1047750 | > loss_disc: 2.6689062118530273 (2.7773884259737454) | > loss_disc_real_0: 0.11070987582206726 (0.08665428640177616) | > loss_disc_real_1: 0.3231881856918335 (0.23049767315387726) | > loss_disc_real_2: 0.27968940138816833 (0.25968615252238053) | > loss_disc_real_3: 0.20003943145275116 (0.27273877423543197) | > loss_disc_real_4: 0.27330753207206726 (0.26381269555825454) | > loss_disc_real_5: 0.22289811074733734 (0.22659167188864487) | > loss_0: 2.6689062118530273 (2.7773884259737454) | > grad_norm_0: tensor(128.5512, device='cuda:0') (tensor(178.3902, device='cuda:0')) | > loss_gen: 2.244605541229248 (1.9997731447219849) | > loss_kl: 1.8045026063919067 (1.9157903377826397) | > loss_feat: 4.053397178649902 (3.5719745525946984) | > loss_mel: 17.042083740234375 (17.022244673508865) | > loss_duration: 1.472824215888977 (1.527453248317425) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.617412567138672 (26.03723584688627) | > grad_norm_1: tensor(5109.2944, device='cuda:0') (tensor(4941.0186, device='cuda:0')) | > current_lr_0: 0.00018409228603835735 | > current_lr_1: 0.00018409228603835735 | > step_time: 0.4777 (0.43915339616628796) | > loader_time: 0.0068 (0.006750216850867638)  --> TIME: 2025-02-25 15:02:31 -- STEP: 63/72 -- GLOBAL_STEP: 1047800 | > loss_disc: 2.7304816246032715 (2.737865220932733) | > loss_disc_real_0: 0.18472787737846375 (0.15232643062278392) | > loss_disc_real_1: 0.24112659692764282 (0.2268128494421641) | > loss_disc_real_2: 0.25595200061798096 (0.24270665716557277) | > loss_disc_real_3: 0.25437843799591064 (0.24384074953813392) | > loss_disc_real_4: 0.2361467331647873 (0.24676279676338989) | > loss_disc_real_5: 0.26342684030532837 (0.2396320400256959) | > loss_0: 2.7304816246032715 (2.737865220932733) | > grad_norm_0: tensor(256.7827, device='cuda:0') (tensor(129.4256, device='cuda:0')) | > loss_gen: 2.113680124282837 (1.9676768855443074) | > loss_kl: 2.7203848361968994 (2.2354757483043364) | > loss_feat: 3.452479124069214 (3.2802363690875826) | > loss_mel: 21.677841186523438 (17.887115009247303) | > loss_duration: 1.6712682247161865 (1.5827411091516888) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.635652542114258 (26.95324489048549) | > grad_norm_1: tensor(6107.8018, device='cuda:0') (tensor(4839.4111, device='cuda:0')) | > current_lr_0: 0.00018409228603835735 | > current_lr_1: 0.00018409228603835735 | > step_time: 0.5189 (0.45304568608601886) | > loader_time: 0.0113 (0.008279675529116678) --> EVAL PERFORMANCE | > avg_loader_time: 0.008657556184580629 (+2.9530323727031127e-05) | > avg_step_time: 0.4608565552133909 (-0.0025623952838736974) | > avg_loss_disc: 2.7306676616131416 (+0.08850639638766467) | > avg_loss_disc_real_0: 0.15464148568120642 (+0.05559505823947175) | > avg_loss_disc_real_1: 0.2255654366503299 (-0.004275355120779756) | > avg_loss_disc_real_2: 0.24466151271907377 (-0.0022262845870474612) | > avg_loss_disc_real_3: 0.24734820005759378 (+0.03049870653891218) | > avg_loss_disc_real_4: 0.24895374728760253 (+0.013048366971418923) | > avg_loss_disc_real_5: 0.24469986411047653 (+0.003252713193356155) | > avg_loss_0: 2.7306676616131416 (+0.08850639638766467) | > avg_grad_norm_0: tensor(125.2727, device='cuda:0') (tensor(-25.5627, device='cuda:0')) | > avg_loss_gen: 1.9901002494382187 (-0.05112077988369523) | > avg_loss_kl: 2.2701883349620124 (+0.2645427542673038) | > avg_loss_feat: 3.272045380632642 (-0.3380600667335618) | > avg_loss_mel: 17.964909043110588 (+0.9698390289091634) | > avg_loss_duration: 1.5919525455421126 (+0.02040849940877565) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.089195358921106 (+0.8656091824383836) | > avg_grad_norm_1: tensor(4964.5298, device='cuda:0') (tensor(-1614.0723, device='cuda:0'))  > EPOCH: 664/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:02:37)   --> TIME: 2025-02-25 15:02:58 -- STEP: 41/72 -- GLOBAL_STEP: 1047850 | > loss_disc: 2.7068166732788086 (2.625718291212873) | > loss_disc_real_0: 0.07397592067718506 (0.11285112162188786) | > loss_disc_real_1: 0.2756756842136383 (0.23662152050471888) | > loss_disc_real_2: 0.24585050344467163 (0.22827679791101596) | > loss_disc_real_3: 0.2554275393486023 (0.23378550134054044) | > loss_disc_real_4: 0.2731367349624634 (0.23820219461510822) | > loss_disc_real_5: 0.27611735463142395 (0.2399018257856369) | > loss_0: 2.7068166732788086 (2.625718291212873) | > grad_norm_0: tensor(159.3783, device='cuda:0') (tensor(118.5566, device='cuda:0')) | > loss_gen: 1.8466113805770874 (2.079074641553367) | > loss_kl: 2.146716356277466 (2.7280308269872897) | > loss_feat: 2.820858955383301 (3.4638798062394307) | > loss_mel: 19.021928787231445 (20.32434454196837) | > loss_duration: 1.6085646152496338 (1.7104219837886532) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.444679260253906 (30.305751847057806) | > grad_norm_1: tensor(5851.5889, device='cuda:0') (tensor(6232.9653, device='cuda:0')) | > current_lr_0: 0.00018406927450260254 | > current_lr_1: 0.00018406927450260254 | > step_time: 0.4496 (0.4441947704408227) | > loader_time: 0.0091 (0.007665942354900081) --> EVAL PERFORMANCE | > avg_loader_time: 0.008804966026628518 (+0.00014740984204788914) | > avg_step_time: 0.46135358071663013 (+0.0004970255032392124) | > avg_loss_disc: 2.7139711782965863 (-0.016696483316555266) | > avg_loss_disc_real_0: 0.1306983343732189 (-0.02394315130798752) | > avg_loss_disc_real_1: 0.23045917296073806 (+0.004893736310408153) | > avg_loss_disc_real_2: 0.23715530419853373 (-0.007506208520540042) | > avg_loss_disc_real_3: 0.23529803941787128 (-0.012050160639722501) | > avg_loss_disc_real_4: 0.24085765846178564 (-0.008096088825816888) | > avg_loss_disc_real_5: 0.24129104929071077 (-0.0034088148197657653) | > avg_loss_0: 2.7139711782965863 (-0.016696483316555266) | > avg_grad_norm_0: tensor(144.3290, device='cuda:0') (+tensor(19.0564, device='cuda:0')) | > avg_loss_gen: 1.9773194940996837 (-0.012780755338535066) | > avg_loss_kl: 2.5462327977301364 (+0.276044462768124) | > avg_loss_feat: 3.2973640502338677 (+0.02531866960122553) | > avg_loss_mel: 19.27236301798216 (+1.3074539748715708) | > avg_loss_duration: 1.6822263522886896 (+0.09027380674657692) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.775505818111796 (+1.6863104591906897) | > avg_grad_norm_1: tensor(5928.0981, device='cuda:0') (+tensor(963.5684, device='cuda:0'))  > EPOCH: 665/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:03:14)   --> TIME: 2025-02-25 15:03:25 -- STEP: 19/72 -- GLOBAL_STEP: 1047900 | > loss_disc: 2.782639980316162 (2.4938489763360274) | > loss_disc_real_0: 0.07960952073335648 (0.10648057413728614) | > loss_disc_real_1: 0.17413923144340515 (0.19800272191825666) | > loss_disc_real_2: 0.2226775735616684 (0.21711250669077822) | > loss_disc_real_3: 0.2300047129392624 (0.22196765871424423) | > loss_disc_real_4: 0.23586252331733704 (0.22836180501862577) | > loss_disc_real_5: 0.21491748094558716 (0.22689540056805863) | > loss_0: 2.782639980316162 (2.4938489763360274) | > grad_norm_0: tensor(153.9875, device='cuda:0') (tensor(120.1731, device='cuda:0')) | > loss_gen: 1.6843966245651245 (2.0932522071035287) | > loss_kl: 2.2624640464782715 (2.2813376125536466) | > loss_feat: 3.2544164657592773 (3.678870866173192) | > loss_mel: 16.848186492919922 (18.34304096824245) | > loss_duration: 1.563515543937683 (1.5830005909267224) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.612977981567383 (27.979502326563786) | > grad_norm_1: tensor(4361.2437, device='cuda:0') (tensor(5610.9604, device='cuda:0')) | > current_lr_0: 0.0001840462658432897 | > current_lr_1: 0.0001840462658432897 | > step_time: 0.4502 (0.46169386411968033) | > loader_time: 0.0066 (0.007149884575291684)  --> TIME: 2025-02-25 15:03:49 -- STEP: 69/72 -- GLOBAL_STEP: 1047950 | > loss_disc: 2.672017812728882 (2.663123545439347) | > loss_disc_real_0: 0.2777704894542694 (0.1763727493260218) | > loss_disc_real_1: 0.2415217012166977 (0.22138307362362958) | > loss_disc_real_2: 0.24513043463230133 (0.23239310344924097) | > loss_disc_real_3: 0.221711665391922 (0.2332961699668912) | > loss_disc_real_4: 0.2643257677555084 (0.24298580336397974) | > loss_disc_real_5: 0.2666880488395691 (0.24518279040205307) | > loss_0: 2.672017812728882 (2.663123545439347) | > grad_norm_0: tensor(116.7839, device='cuda:0') (tensor(110.6310, device='cuda:0')) | > loss_gen: 2.2755324840545654 (2.021903535594111) | > loss_kl: 2.2526984214782715 (2.145544300908628) | > loss_feat: 3.408597230911255 (3.3006750431613647) | > loss_mel: 18.39252471923828 (17.731272960054703) | > loss_duration: 1.6647388935089111 (1.5943482492281043) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.99409294128418 (26.79374412868334) | > grad_norm_1: tensor(7373.4790, device='cuda:0') (tensor(4946.3062, device='cuda:0')) | > current_lr_0: 0.0001840462658432897 | > current_lr_1: 0.0001840462658432897 | > step_time: 0.5384 (0.4626501919566721) | > loader_time: 0.0125 (0.008694444877513946) --> EVAL PERFORMANCE | > avg_loader_time: 0.008746284834096134 (-5.86811925323838e-05) | > avg_step_time: 0.464486387413992 (+0.0031328066973618784) | > avg_loss_disc: 2.6638589442615777 (-0.05011223403500864) | > avg_loss_disc_real_0: 0.1775066060289531 (+0.0468082716557342) | > avg_loss_disc_real_1: 0.22165717499356874 (-0.008801997967169323) | > avg_loss_disc_real_2: 0.23314292015324176 (-0.004012384045291972) | > avg_loss_disc_real_3: 0.23292770729938023 (-0.002370332118491053) | > avg_loss_disc_real_4: 0.24374323390739064 (+0.0028855754456049987) | > avg_loss_disc_real_5: 0.24560111970968648 (+0.004310070418975714) | > avg_loss_0: 2.6638589442615777 (-0.05011223403500864) | > avg_grad_norm_0: tensor(110.1043, device='cuda:0') (tensor(-34.2248, device='cuda:0')) | > avg_loss_gen: 2.0207613622638543 (+0.043441868164170616) | > avg_loss_kl: 2.144331455230713 (-0.40190134249942355) | > avg_loss_feat: 3.2845720035929076 (-0.01279204664096012) | > avg_loss_mel: 17.691083961809184 (-1.5812790561729742) | > avg_loss_duration: 1.5942895479605232 (-0.08793680432816631) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.73503835436324 (-2.0404674637485556) | > avg_grad_norm_1: tensor(4964.0059, device='cuda:0') (tensor(-964.0923, device='cuda:0'))  > EPOCH: 666/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:03:52)   --> TIME: 2025-02-25 15:04:15 -- STEP: 47/72 -- GLOBAL_STEP: 1048000 | > loss_disc: 2.7099008560180664 (2.7232722525900983) | > loss_disc_real_0: 0.0667521208524704 (0.14554969387802674) | > loss_disc_real_1: 0.20765064656734467 (0.25379581559211645) | > loss_disc_real_2: 0.241295725107193 (0.24412178295723935) | > loss_disc_real_3: 0.23798717558383942 (0.2451879975009472) | > loss_disc_real_4: 0.25062277913093567 (0.24194860965647597) | > loss_disc_real_5: 0.2576967477798462 (0.23995171170285406) | > loss_0: 2.7099008560180664 (2.7232722525900983) | > grad_norm_0: tensor(210.1514, device='cuda:0') (tensor(111.6793, device='cuda:0')) | > loss_gen: 1.8149315118789673 (2.0214646801035454) | > loss_kl: 2.163259267807007 (2.2105825251721316) | > loss_feat: 3.0778353214263916 (3.2981656409324485) | > loss_mel: 17.230703353881836 (18.3143970814157) | > loss_duration: 1.6172475814819336 (1.584570123794231) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.903976440429688 (27.429180104681787) | > grad_norm_1: tensor(6479.2827, device='cuda:0') (tensor(5206.6577, device='cuda:0')) | > current_lr_0: 0.00018402326006005927 | > current_lr_1: 0.00018402326006005927 | > step_time: 0.458 (0.44333542154190386) | > loader_time: 0.0088 (0.007752403299859229) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868422548535844 (-6.20593487376947e-05) | > avg_step_time: 0.4564485180545861 (-0.008037869359405925) | > avg_loss_disc: 2.71504570732654 (+0.05118676306496228) | > avg_loss_disc_real_0: 0.14571527491363007 (-0.031791331115323035) | > avg_loss_disc_real_1: 0.2402025283852093 (+0.018545353391640568) | > avg_loss_disc_real_2: 0.24434072136039464 (+0.011197801207152885) | > avg_loss_disc_real_3: 0.24078256962165026 (+0.007854862322270034) | > avg_loss_disc_real_4: 0.24400520030881317 (+0.00026196640142253647) | > avg_loss_disc_real_5: 0.24520536438680032 (-0.0003957553228861599) | > avg_loss_0: 2.71504570732654 (+0.05118676306496228) | > avg_grad_norm_0: tensor(107.5204, device='cuda:0') (tensor(-2.5838, device='cuda:0')) | > avg_loss_gen: 2.0283895781342407 (+0.007628215870386423) | > avg_loss_kl: 2.271961188652145 (+0.12762973342143225) | > avg_loss_feat: 3.2952749359775595 (+0.01070293238465192) | > avg_loss_mel: 18.502652598098962 (+0.8115686362897776) | > avg_loss_duration: 1.604422260338152 (+0.010132712377628694) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.702700655225296 (+0.967662300862056) | > avg_grad_norm_1: tensor(5350.0693, device='cuda:0') (+tensor(386.0635, device='cuda:0'))  > EPOCH: 667/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:04:29)   --> TIME: 2025-02-25 15:04:42 -- STEP: 25/72 -- GLOBAL_STEP: 1048050 | > loss_disc: 2.686567783355713 (2.725897455215454) | > loss_disc_real_0: 0.12652719020843506 (0.1580647844076157) | > loss_disc_real_1: 0.22770777344703674 (0.2779989111423493) | > loss_disc_real_2: 0.25444912910461426 (0.22992731809616088) | > loss_disc_real_3: 0.2543596029281616 (0.25353988170623787) | > loss_disc_real_4: 0.26217830181121826 (0.25209554433822634) | > loss_disc_real_5: 0.25416210293769836 (0.24483277797698974) | > loss_0: 2.686567783355713 (2.725897455215454) | > grad_norm_0: tensor(37.2230, device='cuda:0') (tensor(109.9136, device='cuda:0')) | > loss_gen: 1.9283263683319092 (2.1058395242691033) | > loss_kl: 2.3557276725769043 (2.0682625913619996) | > loss_feat: 2.9522855281829834 (3.2670402240753176) | > loss_mel: 18.922332763671875 (17.94040267944336) | > loss_duration: 1.5855443477630615 (1.5528417491912843) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.744216918945312 (26.934386749267578) | > grad_norm_1: tensor(366.3586, device='cuda:0') (tensor(4990.9644, device='cuda:0')) | > current_lr_0: 0.00018400025715255175 | > current_lr_1: 0.00018400025715255175 | > step_time: 0.4547 (0.4371539783477783) | > loader_time: 0.0091 (0.007084674835205078) --> EVAL PERFORMANCE | > avg_loader_time: 0.008761271624497963 (+7.704613913952357e-05) | > avg_step_time: 0.4545818181105063 (-0.001866699944079786) | > avg_loss_disc: 2.740849750142702 (+0.02580404281616211) | > avg_loss_disc_real_0: 0.14063456958868134 (-0.0050807053249487255) | > avg_loss_disc_real_1: 0.2532530399695248 (+0.0130505115843155) | > avg_loss_disc_real_2: 0.2361907677751192 (-0.008149953585275455) | > avg_loss_disc_real_3: 0.2408860334208314 (+0.00010346379918113602) | > avg_loss_disc_real_4: 0.2457682328744674 (+0.0017630325656542156) | > avg_loss_disc_real_5: 0.24523679219501118 (+3.142780821085989e-05) | > avg_loss_0: 2.740849750142702 (+0.02580404281616211) | > avg_grad_norm_0: tensor(114.5001, device='cuda:0') (+tensor(6.9796, device='cuda:0')) | > avg_loss_gen: 1.9697351707539084 (-0.05865440738033234) | > avg_loss_kl: 2.1831055527002046 (-0.08885563595194057) | > avg_loss_feat: 3.182665458867248 (-0.11260947711031166) | > avg_loss_mel: 17.92736077644456 (-0.5752918216544032) | > avg_loss_duration: 1.6148768848096822 (+0.010454624471530227) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.877743976216923 (-0.8249566790083733) | > avg_grad_norm_1: tensor(4939.5503, device='cuda:0') (tensor(-410.5190, device='cuda:0'))  > EPOCH: 668/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:05:06)   --> TIME: 2025-02-25 15:05:09 -- STEP: 3/72 -- GLOBAL_STEP: 1048100 | > loss_disc: 2.568851947784424 (2.6314518451690674) | > loss_disc_real_0: 0.26997146010398865 (0.27055591344833374) | > loss_disc_real_1: 0.20630870759487152 (0.22178876399993896) | > loss_disc_real_2: 0.21980218589305878 (0.21522157887617746) | > loss_disc_real_3: 0.21763302385807037 (0.2189646710952123) | > loss_disc_real_4: 0.2217736393213272 (0.2251947969198227) | > loss_disc_real_5: 0.23438617587089539 (0.23315225541591644) | > loss_0: 2.568851947784424 (2.6314518451690674) | > grad_norm_0: tensor(156.2157, device='cuda:0') (tensor(105.2944, device='cuda:0')) | > loss_gen: 2.3276331424713135 (2.17272412776947) | > loss_kl: 2.0293209552764893 (1.957029660542806) | > loss_feat: 3.536993980407715 (3.3119822343190513) | > loss_mel: 17.96971893310547 (17.5350284576416) | > loss_duration: 1.6002483367919922 (1.5830131371815999) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.46391487121582 (26.559778213500977) | > grad_norm_1: tensor(5702.2593, device='cuda:0') (tensor(5605.3882, device='cuda:0')) | > current_lr_0: 0.00018397725712040766 | > current_lr_1: 0.00018397725712040766 | > step_time: 0.4231 (0.43341588973999023) | > loader_time: 0.0073 (0.006200949350992839)  --> TIME: 2025-02-25 15:05:32 -- STEP: 53/72 -- GLOBAL_STEP: 1048150 | > loss_disc: 2.7891430854797363 (2.66661827519255) | > loss_disc_real_0: 0.2526075541973114 (0.16541059765050997) | > loss_disc_real_1: 0.2593195140361786 (0.22652185413072695) | > loss_disc_real_2: 0.23179665207862854 (0.23426712093488225) | > loss_disc_real_3: 0.263685941696167 (0.23260146126432238) | > loss_disc_real_4: 0.250495970249176 (0.23922802760915937) | > loss_disc_real_5: 0.24633602797985077 (0.2438801647919529) | > loss_0: 2.7891430854797363 (2.66661827519255) | > grad_norm_0: tensor(193.9121, device='cuda:0') (tensor(121.0782, device='cuda:0')) | > loss_gen: 1.966621994972229 (1.994379297742304) | > loss_kl: 2.2459166049957275 (2.206669384578489) | > loss_feat: 2.3650801181793213 (3.23400294555808) | > loss_mel: 15.554131507873535 (17.902247932721977) | > loss_duration: 1.6260497570037842 (1.6125285333057617) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.75779914855957 (26.94982812989433) | > grad_norm_1: tensor(1461.9132, device='cuda:0') (tensor(5454.3047, device='cuda:0')) | > current_lr_0: 0.00018397725712040766 | > current_lr_1: 0.00018397725712040766 | > step_time: 0.4509 (0.45181107970903506) | > loader_time: 0.0093 (0.007910998362415243) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661995471363338 (-9.927615313462534e-05) | > avg_step_time: 0.46241557766014424 (+0.007833759549637942) | > avg_loss_disc: 2.677245845257397 (-0.06360390488530498) | > avg_loss_disc_real_0: 0.16369948992636843 (+0.02306492033768709) | > avg_loss_disc_real_1: 0.22937300633376753 (-0.023880033635757275) | > avg_loss_disc_real_2: 0.23424379254730654 (-0.0019469752278126473) | > avg_loss_disc_real_3: 0.2348122174890948 (-0.006073815931736587) | > avg_loss_disc_real_4: 0.24106395097685532 (-0.004704281897612073) | > avg_loss_disc_real_5: 0.24362974897236891 (-0.0016070432226422682) | > avg_loss_0: 2.677245845257397 (-0.06360390488530498) | > avg_grad_norm_0: tensor(112.4203, device='cuda:0') (tensor(-2.0797, device='cuda:0')) | > avg_loss_gen: 1.9957317533627363 (+0.02599658260882798) | > avg_loss_kl: 2.21093097874816 (+0.02782542604795557) | > avg_loss_feat: 3.250889932605582 (+0.06822447373833418) | > avg_loss_mel: 17.908391321209105 (-0.018969455235453125) | > avg_loss_duration: 1.6175271608460113 (+0.002650276036329169) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.983471104796504 (+0.10572712857958066) | > avg_grad_norm_1: tensor(5375.0298, device='cuda:0') (+tensor(435.4795, device='cuda:0'))  > EPOCH: 669/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:05:43)   --> TIME: 2025-02-25 15:05:59 -- STEP: 31/72 -- GLOBAL_STEP: 1048200 | > loss_disc: 2.7684290409088135 (2.745729707902478) | > loss_disc_real_0: 0.10891591757535934 (0.1695022215285609) | > loss_disc_real_1: 0.1971682757139206 (0.2430567236677293) | > loss_disc_real_2: 0.2612457275390625 (0.23853808112682834) | > loss_disc_real_3: 0.2317395657300949 (0.24201363084777708) | > loss_disc_real_4: 0.24022462964057922 (0.2463539949347896) | > loss_disc_real_5: 0.24908654391765594 (0.24854239869502284) | > loss_0: 2.7684290409088135 (2.745729707902478) | > grad_norm_0: tensor(45.6084, device='cuda:0') (tensor(107.9909, device='cuda:0')) | > loss_gen: 1.7478954792022705 (1.9854592046430037) | > loss_kl: 2.220492124557495 (2.147649080522599) | > loss_feat: 2.9658548831939697 (3.233531936522453) | > loss_mel: 19.480398178100586 (17.929463109662457) | > loss_duration: 1.6210448741912842 (1.5878436411580732) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.03568458557129 (26.883947064799646) | > grad_norm_1: tensor(4153.8477, device='cuda:0') (tensor(4416.4478, device='cuda:0')) | > current_lr_0: 0.0001839542599632676 | > current_lr_1: 0.0001839542599632676 | > step_time: 0.4518 (0.4367488507301577) | > loader_time: 0.0086 (0.006964829660231067) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618949164806959 (-4.304630655637895e-05) | > avg_step_time: 0.4563125321562861 (-0.006103045503858151) | > avg_loss_disc: 2.7016835582088414 (+0.024437712951444368) | > avg_loss_disc_real_0: 0.16203795142576724 (-0.0016615385006011896) | > avg_loss_disc_real_1: 0.2352876128025458 (+0.005914606468778255) | > avg_loss_disc_real_2: 0.23940986730683017 (+0.005166074759523631) | > avg_loss_disc_real_3: 0.23841156422252385 (+0.003599346733429043) | > avg_loss_disc_real_4: 0.24670028329735072 (+0.005636332320495402) | > avg_loss_disc_real_5: 0.24922248511247233 (+0.005592736140103416) | > avg_loss_0: 2.7016835582088414 (+0.024437712951444368) | > avg_grad_norm_0: tensor(111.0101, device='cuda:0') (tensor(-1.4102, device='cuda:0')) | > avg_loss_gen: 1.9838131733343636 (-0.01191858002837276) | > avg_loss_kl: 2.150952958724869 (-0.05997802002329111) | > avg_loss_feat: 3.1600808694329063 (-0.09080906317267567) | > avg_loss_mel: 17.636927215146354 (-0.2714641060627514) | > avg_loss_duration: 1.6087998302889541 (-0.008727330557057211) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.540574006631342 (-0.44289709816516165) | > avg_grad_norm_1: tensor(4812.1777, device='cuda:0') (tensor(-562.8521, device='cuda:0'))  > EPOCH: 670/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:06:20)   --> TIME: 2025-02-25 15:06:26 -- STEP: 9/72 -- GLOBAL_STEP: 1048250 | > loss_disc: 2.720524787902832 (2.6961701181199818) | > loss_disc_real_0: 0.09952206909656525 (0.18523375358846453) | > loss_disc_real_1: 0.12036053836345673 (0.1878211937016911) | > loss_disc_real_2: 0.17033617198467255 (0.22459626197814941) | > loss_disc_real_3: 0.2141779661178589 (0.22887302107281154) | > loss_disc_real_4: 0.2591182291507721 (0.2514977372354931) | > loss_disc_real_5: 0.24897950887680054 (0.25363509356975555) | > loss_0: 2.720524787902832 (2.6961701181199818) | > grad_norm_0: tensor(149.0677, device='cuda:0') (tensor(98.5625, device='cuda:0')) | > loss_gen: 1.7241275310516357 (1.9306102991104126) | > loss_kl: 1.9551388025283813 (1.9102565844853718) | > loss_feat: 3.5230295658111572 (3.3793268468644886) | > loss_mel: 16.889158248901367 (17.598094304402668) | > loss_duration: 1.5364593267440796 (1.5452512502670288) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.627912521362305 (26.36353937784831) | > grad_norm_1: tensor(7867.5537, device='cuda:0') (tensor(5336.7671, device='cuda:0')) | > current_lr_0: 0.0001839312656807722 | > current_lr_1: 0.0001839312656807722 | > step_time: 0.426 (0.42961186832851833) | > loader_time: 0.0062 (0.006850825415717231)  --> TIME: 2025-02-25 15:06:49 -- STEP: 59/72 -- GLOBAL_STEP: 1048300 | > loss_disc: 2.6999995708465576 (2.680980148961988) | > loss_disc_real_0: 0.22770729660987854 (0.1406996644149392) | > loss_disc_real_1: 0.22173482179641724 (0.21975628387624935) | > loss_disc_real_2: 0.18676431477069855 (0.23437978630348788) | > loss_disc_real_3: 0.21527379751205444 (0.2357973888263864) | > loss_disc_real_4: 0.23960648477077484 (0.2425434410066928) | > loss_disc_real_5: 0.2807798385620117 (0.24744234170954107) | > loss_0: 2.6999995708465576 (2.680980148961988) | > grad_norm_0: tensor(134.5815, device='cuda:0') (tensor(117.8105, device='cuda:0')) | > loss_gen: 2.052762508392334 (1.9994141469567506) | > loss_kl: 2.4505865573883057 (2.041324645785962) | > loss_feat: 3.2608625888824463 (3.3285205121767723) | > loss_mel: 19.032573699951172 (17.635761357970157) | > loss_duration: 1.6567614078521729 (1.5729793047500868) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.45354652404785 (26.577999793877037) | > grad_norm_1: tensor(6413.8140, device='cuda:0') (tensor(5142.8945, device='cuda:0')) | > current_lr_0: 0.0001839312656807722 | > current_lr_1: 0.0001839312656807722 | > step_time: 0.4719 (0.44394379551127805) | > loader_time: 0.0103 (0.008051484318102821) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629617556719713 (+1.0668391912754122e-05) | > avg_step_time: 0.45491537577669383 (-0.001397156379592257) | > avg_loss_disc: 2.726439479371191 (+0.02475592116234937) | > avg_loss_disc_real_0: 0.13680046471491666 (-0.025237486710850576) | > avg_loss_disc_real_1: 0.2155035856953809 (-0.019784027107164898) | > avg_loss_disc_real_2: 0.24192681153055648 (+0.0025169442237263084) | > avg_loss_disc_real_3: 0.24019930291343744 (+0.0017877386909135873) | > avg_loss_disc_real_4: 0.24329784050793715 (-0.003402442789413568) | > avg_loss_disc_real_5: 0.24544382494100384 (-0.0037786601714684953) | > avg_loss_0: 2.726439479371191 (+0.02475592116234937) | > avg_grad_norm_0: tensor(135.7092, device='cuda:0') (+tensor(24.6991, device='cuda:0')) | > avg_loss_gen: 1.956973200113001 (-0.026839973221362667) | > avg_loss_kl: 2.0852019669304434 (-0.06575099179442567) | > avg_loss_feat: 3.263936768115406 (+0.10385589868249978) | > avg_loss_mel: 17.655714840956136 (+0.018787625809782327) | > avg_loss_duration: 1.5837769206141081 (-0.025022909674845994) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.54560357751981 (+0.0050295708884675605) | > avg_grad_norm_1: tensor(5032.7886, device='cuda:0') (+tensor(220.6108, device='cuda:0'))  > EPOCH: 671/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:06:57)   --> TIME: 2025-02-25 15:07:16 -- STEP: 37/72 -- GLOBAL_STEP: 1048350 | > loss_disc: 2.8059964179992676 (2.6841467522286084) | > loss_disc_real_0: 0.08829975128173828 (0.18228828423732035) | > loss_disc_real_1: 0.14475823938846588 (0.22520943912299904) | > loss_disc_real_2: 0.24419943988323212 (0.23743113150467743) | > loss_disc_real_3: 0.23957687616348267 (0.2359750117804553) | > loss_disc_real_4: 0.2664657235145569 (0.24406250103099927) | > loss_disc_real_5: 0.2706719934940338 (0.2442031377876127) | > loss_0: 2.8059964179992676 (2.6841467522286084) | > grad_norm_0: tensor(41.5314, device='cuda:0') (tensor(107.4297, device='cuda:0')) | > loss_gen: 1.6879067420959473 (2.004716824840855) | > loss_kl: 2.1380817890167236 (2.2070763368864323) | > loss_feat: 2.6816656589508057 (3.2956381746240564) | > loss_mel: 17.29983901977539 (18.14846549162993) | > loss_duration: 1.596367597579956 (1.561252152597582) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.403860092163086 (27.217148961247624) | > grad_norm_1: tensor(2855.0730, device='cuda:0') (tensor(4785.4658, device='cuda:0')) | > current_lr_0: 0.0001839082742725621 | > current_lr_1: 0.0001839082742725621 | > step_time: 0.4481 (0.4376862822352229) | > loader_time: 0.0087 (0.00729634310748126) --> EVAL PERFORMANCE | > avg_loader_time: 0.008646263203150789 (+1.664564643107623e-05) | > avg_step_time: 0.4565362493756791 (+0.001620873598985273) | > avg_loss_disc: 2.684216872067519 (-0.042222607303672) | > avg_loss_disc_real_0: 0.176928628393462 (+0.040128163678545326) | > avg_loss_disc_real_1: 0.22046493980246532 (+0.004961354107084426) | > avg_loss_disc_real_2: 0.23501338043683012 (-0.006913431093726358) | > avg_loss_disc_real_3: 0.23246132320081683 (-0.007737979712620607) | > avg_loss_disc_real_4: 0.24371387253344898 (+0.0004160320255118255) | > avg_loss_disc_real_5: 0.24477353859955156 (-0.0006702863414522753) | > avg_loss_0: 2.684216872067519 (-0.042222607303672) | > avg_grad_norm_0: tensor(112.7452, device='cuda:0') (tensor(-22.9639, device='cuda:0')) | > avg_loss_gen: 2.0002905247916636 (+0.04331732467866267) | > avg_loss_kl: 2.2083546645204795 (+0.12315269759003611) | > avg_loss_feat: 3.2235117092938492 (-0.04042505882155689) | > avg_loss_mel: 17.89591735517475 (+0.2402025142186126) | > avg_loss_duration: 1.5904555522220236 (+0.006678631607915486) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.918529859730896 (+0.37292628221108615) | > avg_grad_norm_1: tensor(4996.8101, device='cuda:0') (tensor(-35.9785, device='cuda:0'))  > EPOCH: 672/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:07:34)   --> TIME: 2025-02-25 15:07:43 -- STEP: 15/72 -- GLOBAL_STEP: 1048400 | > loss_disc: 2.383795738220215 (2.7009071350097655) | > loss_disc_real_0: 0.06532043218612671 (0.11353188877304395) | > loss_disc_real_1: 0.21637797355651855 (0.26809087793032327) | > loss_disc_real_2: 0.2851589024066925 (0.2760657807191213) | > loss_disc_real_3: 0.21454542875289917 (0.2662983139355978) | > loss_disc_real_4: 0.2571340501308441 (0.25174672305583956) | > loss_disc_real_5: 0.22820748388767242 (0.24823082685470582) | > loss_0: 2.383795738220215 (2.7009071350097655) | > grad_norm_0: tensor(61.1855, device='cuda:0') (tensor(109.6866, device='cuda:0')) | > loss_gen: 2.20711088180542 (2.1831483840942383) | > loss_kl: 2.0300710201263428 (2.0000597476959228) | > loss_feat: 4.099348545074463 (3.7646706740061444) | > loss_mel: 17.872196197509766 (17.570657475789393) | > loss_duration: 1.5154526233673096 (1.5337963183720906) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.724180221557617 (27.052332814534505) | > grad_norm_1: tensor(3304.1777, device='cuda:0') (tensor(4069.7517, device='cuda:0')) | > current_lr_0: 0.00018388528573827801 | > current_lr_1: 0.00018388528573827801 | > step_time: 0.443 (0.43682255744934084) | > loader_time: 0.0058 (0.0066931724548339845)  --> TIME: 2025-02-25 15:08:07 -- STEP: 65/72 -- GLOBAL_STEP: 1048450 | > loss_disc: 2.8505332469940186 (2.670055829561674) | > loss_disc_real_0: 0.11036521196365356 (0.14715785309672355) | > loss_disc_real_1: 0.2346523553133011 (0.2289129328269225) | > loss_disc_real_2: 0.25115668773651123 (0.23727585467008444) | > loss_disc_real_3: 0.23864609003067017 (0.23320359725218553) | > loss_disc_real_4: 0.23645661771297455 (0.24481080976816325) | > loss_disc_real_5: 0.2404559701681137 (0.24903178627674397) | > loss_0: 2.8505332469940186 (2.670055829561674) | > grad_norm_0: tensor(30.5062, device='cuda:0') (tensor(107.6743, device='cuda:0')) | > loss_gen: 1.737545132637024 (2.074471448018001) | > loss_kl: 2.3744900226593018 (2.0796505212783805) | > loss_feat: 2.7041056156158447 (3.4325657807863674) | > loss_mel: 17.849084854125977 (17.45266421391413) | > loss_duration: 1.636353611946106 (1.5731694038097674) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.30158042907715 (26.612521450336164) | > grad_norm_1: tensor(1611.8219, device='cuda:0') (tensor(4687.4028, device='cuda:0')) | > current_lr_0: 0.00018388528573827801 | > current_lr_1: 0.00018388528573827801 | > step_time: 0.497 (0.45060902742239145) | > loader_time: 0.0119 (0.008371800642747148) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684433681864136 (+3.8170478713347306e-05) | > avg_step_time: 0.45577367594544316 (-0.0007625734302359466) | > avg_loss_disc: 2.6664730226489866 (-0.017743849418532243) | > avg_loss_disc_real_0: 0.15284357749870126 (-0.024085050894760734) | > avg_loss_disc_real_1: 0.22818668522465396 (+0.007721745422188647) | > avg_loss_disc_real_2: 0.2374917942453438 (+0.0024784138085136853) | > avg_loss_disc_real_3: 0.23380717215403704 (+0.0013458489532202078) | > avg_loss_disc_real_4: 0.24362522699463535 (-8.864553881363091e-05) | > avg_loss_disc_real_5: 0.2479803828706204 (+0.0032068442710688483) | > avg_loss_0: 2.6664730226489866 (-0.017743849418532243) | > avg_grad_norm_0: tensor(107.7682, device='cuda:0') (tensor(-4.9771, device='cuda:0')) | > avg_loss_gen: 2.092589479097178 (+0.0922989543055146) | > avg_loss_kl: 2.087073405024031 (-0.12128125949644852) | > avg_loss_feat: 3.4307316524881712 (+0.20721994319432202) | > avg_loss_mel: 17.453486455997947 (-0.44243089917680223) | > avg_loss_duration: 1.5781411587352483 (-0.012314393486775321) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.64202225376183 (-0.27650760596906565) | > avg_grad_norm_1: tensor(4680.7515, device='cuda:0') (tensor(-316.0586, device='cuda:0'))  > EPOCH: 673/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:08:12)   --> TIME: 2025-02-25 15:08:33 -- STEP: 43/72 -- GLOBAL_STEP: 1048500 | > loss_disc: 2.811391830444336 (2.6818272235781646) | > loss_disc_real_0: 0.09149815142154694 (0.13347431688114655) | > loss_disc_real_1: 0.22359369695186615 (0.2172181737977405) | > loss_disc_real_2: 0.2557760179042816 (0.241190561028414) | > loss_disc_real_3: 0.24052764475345612 (0.23679158646006918) | > loss_disc_real_4: 0.22329185903072357 (0.24242887628632923) | > loss_disc_real_5: 0.22282156348228455 (0.24573812339194986) | > loss_0: 2.811391830444336 (2.6818272235781646) | > grad_norm_0: tensor(58.2846, device='cuda:0') (tensor(105.5603, device='cuda:0')) | > loss_gen: 1.7653987407684326 (1.9651838513307789) | > loss_kl: 2.3858139514923096 (2.060306587884592) | > loss_feat: 3.1794652938842773 (3.356861031332681) | > loss_mel: 17.991409301757812 (17.396283726359524) | > loss_duration: 1.5951848030090332 (1.553787727688634) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.91727066040039 (26.332422788753068) | > grad_norm_1: tensor(3196.8218, device='cuda:0') (tensor(4663.9067, device='cuda:0')) | > current_lr_0: 0.00018386230007756072 | > current_lr_1: 0.00018386230007756072 | > step_time: 0.4631 (0.4422739716463311) | > loader_time: 0.0098 (0.007424515347148097) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630373108554896 (-5.406057330924023e-05) | > avg_step_time: 0.45981827923949337 (+0.004044603294050209) | > avg_loss_disc: 2.7294282275186457 (+0.06295520486965911) | > avg_loss_disc_real_0: 0.1652924541226575 (+0.012448876623956245) | > avg_loss_disc_real_1: 0.22331033930392333 (-0.004876345920730635) | > avg_loss_disc_real_2: 0.2343533189154007 (-0.0031384753299431023) | > avg_loss_disc_real_3: 0.23182310709651088 (-0.0019840650575261576) | > avg_loss_disc_real_4: 0.24410643837821316 (+0.00048121138357781756) | > avg_loss_disc_real_5: 0.24331292992746326 (-0.004667452943157152) | > avg_loss_0: 2.7294282275186457 (+0.06295520486965911) | > avg_grad_norm_0: tensor(88.8626, device='cuda:0') (tensor(-18.9056, device='cuda:0')) | > avg_loss_gen: 1.9242638215212755 (-0.16832565757590268) | > avg_loss_kl: 2.19714331626892 (+0.11006991124488907) | > avg_loss_feat: 3.1838309026100267 (-0.2469007498781446) | > avg_loss_mel: 17.756846239869024 (+0.30335978387107687) | > avg_loss_duration: 1.588867313425306 (+0.010726154690057665) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.65095154668244 (+0.008929292920608844) | > avg_grad_norm_1: tensor(4082.9934, device='cuda:0') (tensor(-597.7581, device='cuda:0'))  > EPOCH: 674/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:08:49)   --> TIME: 2025-02-25 15:09:00 -- STEP: 21/72 -- GLOBAL_STEP: 1048550 | > loss_disc: 2.5805423259735107 (2.6844922474452426) | > loss_disc_real_0: 0.1471221148967743 (0.1437387587059112) | > loss_disc_real_1: 0.22516636550426483 (0.21853986382484436) | > loss_disc_real_2: 0.2203378826379776 (0.24600929944288163) | > loss_disc_real_3: 0.2302866280078888 (0.24382050477323078) | > loss_disc_real_4: 0.23071721196174622 (0.24694749925817763) | > loss_disc_real_5: 0.23454894125461578 (0.25667748139018104) | > loss_0: 2.5805423259735107 (2.6844922474452426) | > grad_norm_0: tensor(35.5646, device='cuda:0') (tensor(108.8748, device='cuda:0')) | > loss_gen: 1.8764708042144775 (1.980004384404137) | > loss_kl: 2.2228598594665527 (2.8611077808198466) | > loss_feat: 3.013369083404541 (3.254277433667864) | > loss_mel: 18.698457717895508 (21.65601884751093) | > loss_duration: 1.5844002962112427 (1.6308994179680234) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.395557403564453 (31.382308142525808) | > grad_norm_1: tensor(6019.4683, device='cuda:0') (tensor(5096.4683, device='cuda:0')) | > current_lr_0: 0.00018383931729005102 | > current_lr_1: 0.00018383931729005102 | > step_time: 0.4439 (0.43603335108075825) | > loader_time: 0.0067 (0.007211844126383464)  --> TIME: 2025-02-25 15:09:24 -- STEP: 71/72 -- GLOBAL_STEP: 1048600 | > loss_disc: 2.5447258949279785 (2.656950389835197) | > loss_disc_real_0: 0.16823381185531616 (0.15269554107331898) | > loss_disc_real_1: 0.22425667941570282 (0.21740174849688168) | > loss_disc_real_2: 0.25906410813331604 (0.23139515267291538) | > loss_disc_real_3: 0.20344272255897522 (0.23646709343916933) | > loss_disc_real_4: 0.2385280877351761 (0.2418510634714449) | > loss_disc_real_5: 0.2593293786048889 (0.2461697410949519) | > loss_0: 2.5447258949279785 (2.656950389835197) | > grad_norm_0: tensor(196.7229, device='cuda:0') (tensor(120.3656, device='cuda:0')) | > loss_gen: 2.065107822418213 (1.9993391003407224) | > loss_kl: 1.9064661264419556 (2.3889192275598017) | > loss_feat: 3.0760960578918457 (3.3367382909210637) | > loss_mel: 17.276472091674805 (19.283012014039798) | > loss_duration: 1.6116857528686523 (1.612789508322595) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.93582534790039 (28.6207983527385) | > grad_norm_1: tensor(6241.9165, device='cuda:0') (tensor(5821.8389, device='cuda:0')) | > current_lr_0: 0.00018383931729005102 | > current_lr_1: 0.00018383931729005102 | > step_time: 0.4907 (0.45472828435226226) | > loader_time: 0.0081 (0.008938134556085288) --> EVAL PERFORMANCE | > avg_loader_time: 0.008938134556085288 (+0.00030776144753039136) | > avg_step_time: 0.45472828435226226 (-0.0050899948872311085) | > avg_loss_disc: 2.656950389835197 (-0.07247783768344851) | > avg_loss_disc_real_0: 0.15269554107331898 (-0.01259691304933852) | > avg_loss_disc_real_1: 0.21740174849688168 (-0.005908590807041647) | > avg_loss_disc_real_2: 0.23139515267291538 (-0.0029581662424853217) | > avg_loss_disc_real_3: 0.23646709343916933 (+0.004643986342658446) | > avg_loss_disc_real_4: 0.2418510634714449 (-0.002255374906768276) | > avg_loss_disc_real_5: 0.2461697410949519 (+0.0028568111674886487) | > avg_loss_0: 2.656950389835197 (-0.07247783768344851) | > avg_grad_norm_0: tensor(120.3656, device='cuda:0') (+tensor(31.5030, device='cuda:0')) | > avg_loss_gen: 1.9993391003407224 (+0.07507527881944687) | > avg_loss_kl: 2.3889192275598017 (+0.19177591129088167) | > avg_loss_feat: 3.3367382909210637 (+0.152907388311037) | > avg_loss_mel: 19.283012014039798 (+1.5261657741707744) | > avg_loss_duration: 1.612789508322595 (+0.023922194897288973) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.6207983527385 (+1.9698468060560614) | > avg_grad_norm_1: tensor(5821.8389, device='cuda:0') (+tensor(1738.8455, device='cuda:0'))  > EPOCH: 675/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:09:26)   --> TIME: 2025-02-25 15:09:50 -- STEP: 49/72 -- GLOBAL_STEP: 1048650 | > loss_disc: 2.442721366882324 (2.678499902997697) | > loss_disc_real_0: 0.07329985499382019 (0.11088275168167085) | > loss_disc_real_1: 0.2774011492729187 (0.21123898226995858) | > loss_disc_real_2: 0.1634417176246643 (0.235444470631833) | > loss_disc_real_3: 0.2654075622558594 (0.2254116367928836) | > loss_disc_real_4: 0.21890808641910553 (0.23581252657637305) | > loss_disc_real_5: 0.20601153373718262 (0.2448969179270219) | > loss_0: 2.442721366882324 (2.678499902997697) | > grad_norm_0: tensor(64.9838, device='cuda:0') (tensor(162.4505, device='cuda:0')) | > loss_gen: 2.3498926162719727 (1.9330851447825526) | > loss_kl: 1.9981945753097534 (2.035112215548145) | > loss_feat: 4.141592025756836 (3.553899254117693) | > loss_mel: 17.19239044189453 (17.4508060922428) | > loss_duration: 1.6550039052963257 (1.5677071469170707) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.337072372436523 (26.540609943623444) | > grad_norm_1: tensor(8349.5566, device='cuda:0') (tensor(5712.1089, device='cuda:0')) | > current_lr_0: 0.00018381633737538975 | > current_lr_1: 0.00018381633737538975 | > step_time: 0.4562 (0.44334716212992764) | > loader_time: 0.0094 (0.007683661519264688) --> EVAL PERFORMANCE | > avg_loader_time: 0.008581953988948338 (-0.0003561805671369496) | > avg_step_time: 0.45315766670334506 (-0.0015706176489171941) | > avg_loss_disc: 2.7268383368639872 (+0.06988794702879009) | > avg_loss_disc_real_0: 0.116929484268939 (-0.035766056804379984) | > avg_loss_disc_real_1: 0.21932243387883818 (+0.0019206853819564995) | > avg_loss_disc_real_2: 0.23755779946354075 (+0.006162646790625365) | > avg_loss_disc_real_3: 0.2399619072675705 (+0.0034948138284011665) | > avg_loss_disc_real_4: 0.24116281304560916 (-0.0006882504258357292) | > avg_loss_disc_real_5: 0.24575177800487463 (-0.0004179630900772724) | > avg_loss_0: 2.7268383368639872 (+0.06988794702879009) | > avg_grad_norm_0: tensor(152.2548, device='cuda:0') (+tensor(31.8892, device='cuda:0')) | > avg_loss_gen: 1.9146465986547336 (-0.08469250168598874) | > avg_loss_kl: 2.074275800879571 (-0.3146434266802305) | > avg_loss_feat: 3.436163778036413 (+0.09942548711534949) | > avg_loss_mel: 17.447840139899455 (-1.835171874140343) | > avg_loss_duration: 1.5836521017719323 (-0.02913740655066266) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.45657853677239 (-2.16421981596611) | > avg_grad_norm_1: tensor(5409.7671, device='cuda:0') (tensor(-412.0718, device='cuda:0'))  > EPOCH: 676/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:10:03)   --> TIME: 2025-02-25 15:10:17 -- STEP: 27/72 -- GLOBAL_STEP: 1048700 | > loss_disc: 2.7153797149658203 (2.65494622124566) | > loss_disc_real_0: 0.22941967844963074 (0.1779351549016105) | > loss_disc_real_1: 0.24601472914218903 (0.21114210177350928) | > loss_disc_real_2: 0.22489675879478455 (0.24464923198576324) | > loss_disc_real_3: 0.26559263467788696 (0.22376640582526172) | > loss_disc_real_4: 0.23754602670669556 (0.2373007811881878) | > loss_disc_real_5: 0.2565968632698059 (0.24678698182106018) | > loss_0: 2.7153797149658203 (2.65494622124566) | > grad_norm_0: tensor(17.4079, device='cuda:0') (tensor(115.0314, device='cuda:0')) | > loss_gen: 2.2751541137695312 (2.0629948377609257) | > loss_kl: 1.955174207687378 (1.9496286621800176) | > loss_feat: 3.8300304412841797 (3.3792468353554055) | > loss_mel: 17.046632766723633 (17.338798558270486) | > loss_duration: 1.5527677536010742 (1.5351735927440502) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.659759521484375 (26.265842367101598) | > grad_norm_1: tensor(5698.1318, device='cuda:0') (tensor(6315.6597, device='cuda:0')) | > current_lr_0: 0.00018379336033321782 | > current_lr_1: 0.00018379336033321782 | > step_time: 0.4391 (0.4370241341767488) | > loader_time: 0.0083 (0.00699660513136122) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625618169005487 (+4.366418005714856e-05) | > avg_step_time: 0.45659139458562287 (+0.003433727882277804) | > avg_loss_disc: 2.7035635491492047 (-0.02327478771478253) | > avg_loss_disc_real_0: 0.1616695385793565 (+0.04474005431041751) | > avg_loss_disc_real_1: 0.21542086630639895 (-0.0039015675724392296) | > avg_loss_disc_real_2: 0.24470977799993165 (+0.007151978536390907) | > avg_loss_disc_real_3: 0.2329843673907535 (-0.006977539876816985) | > avg_loss_disc_real_4: 0.2486518770456314 (+0.00748906400002225) | > avg_loss_disc_real_5: 0.25081486286411825 (+0.005063084859243616) | > avg_loss_0: 2.7035635491492047 (-0.02327478771478253) | > avg_grad_norm_0: tensor(155.1915, device='cuda:0') (+tensor(2.9367, device='cuda:0')) | > avg_loss_gen: 1.9698703473722434 (+0.05522374871750979) | > avg_loss_kl: 2.0080367628957183 (-0.06623903798385289) | > avg_loss_feat: 3.184562525279086 (-0.25160125275732703) | > avg_loss_mel: 17.091589847081146 (-0.35625029281830933) | > avg_loss_duration: 1.5734078800174551 (-0.010244221754477145) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.827467340818593 (-0.6291111959537972) | > avg_grad_norm_1: tensor(5894.7529, device='cuda:0') (+tensor(484.9858, device='cuda:0'))  > EPOCH: 677/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:10:40)   --> TIME: 2025-02-25 15:10:44 -- STEP: 5/72 -- GLOBAL_STEP: 1048750 | > loss_disc: 2.603672742843628 (2.650462579727173) | > loss_disc_real_0: 0.11680875718593597 (0.1624116748571396) | > loss_disc_real_1: 0.44500166177749634 (0.31046694368124006) | > loss_disc_real_2: 0.2116834670305252 (0.23210144340991973) | > loss_disc_real_3: 0.21664345264434814 (0.22361824810504913) | > loss_disc_real_4: 0.23651660978794098 (0.25724999606609344) | > loss_disc_real_5: 0.2303706705570221 (0.2385973274707794) | > loss_0: 2.603672742843628 (2.650462579727173) | > grad_norm_0: tensor(90.9650, device='cuda:0') (tensor(103.0874, device='cuda:0')) | > loss_gen: 2.334533452987671 (2.3034683227539063) | > loss_kl: 1.8974316120147705 (1.9551558256149293) | > loss_feat: 3.819744825363159 (3.4925849437713623) | > loss_mel: 17.74363136291504 (17.295125198364257) | > loss_duration: 1.5100845098495483 (1.5518767595291139) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.3054256439209 (26.598210525512695) | > grad_norm_1: tensor(7175.5132, device='cuda:0') (tensor(6294.1675, device='cuda:0')) | > current_lr_0: 0.00018377038616317615 | > current_lr_1: 0.00018377038616317615 | > step_time: 0.4344 (0.4319037437438965) | > loader_time: 0.009 (0.007410287857055664)  --> TIME: 2025-02-25 15:11:07 -- STEP: 55/72 -- GLOBAL_STEP: 1048800 | > loss_disc: 2.7656478881835938 (2.71669884595004) | > loss_disc_real_0: 0.2608293294906616 (0.15283726822923527) | > loss_disc_real_1: 0.16199155151844025 (0.24213601608168) | > loss_disc_real_2: 0.23264296352863312 (0.25736776183952) | > loss_disc_real_3: 0.19839687645435333 (0.24268754585222765) | > loss_disc_real_4: 0.2531745731830597 (0.24880931052294644) | > loss_disc_real_5: 0.25271332263946533 (0.24662166129459034) | > loss_0: 2.7656478881835938 (2.71669884595004) | > grad_norm_0: tensor(226.4319, device='cuda:0') (tensor(120.3361, device='cuda:0')) | > loss_gen: 1.7966879606246948 (2.0737121820449826) | > loss_kl: 2.1988167762756348 (2.116099758581682) | > loss_feat: 2.770467758178711 (3.387137525731867) | > loss_mel: 19.28505516052246 (17.5618217641657) | > loss_duration: 1.646933674812317 (1.569578511064703) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.697961807250977 (26.708349574695934) | > grad_norm_1: tensor(3864.9482, device='cuda:0') (tensor(4990.4604, device='cuda:0')) | > current_lr_0: 0.00018377038616317615 | > current_lr_1: 0.00018377038616317615 | > step_time: 0.4592 (0.44188639467412777) | > loader_time: 0.0091 (0.00788974328474565) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860419407696791 (-2.1424092037576267e-05) | > avg_step_time: 0.45377175908693124 (-0.0028196354986916305) | > avg_loss_disc: 2.7218753854993363 (+0.01831183635013156) | > avg_loss_disc_real_0: 0.16655183439208587 (+0.004882295812729365) | > avg_loss_disc_real_1: 0.2392533886390673 (+0.02383252233266836) | > avg_loss_disc_real_2: 0.2529803098927083 (+0.008270531892776656) | > avg_loss_disc_real_3: 0.24541243386100714 (+0.01242806647025363) | > avg_loss_disc_real_4: 0.2483640963762579 (-0.0002877806693735163) | > avg_loss_disc_real_5: 0.2481302520338918 (-0.002684610830226447) | > avg_loss_0: 2.7218753854993363 (+0.01831183635013156) | > avg_grad_norm_0: tensor(120.1041, device='cuda:0') (tensor(-35.0873, device='cuda:0')) | > avg_loss_gen: 2.059351984883698 (+0.08948163751145466) | > avg_loss_kl: 2.1375581361878093 (+0.129521373292091) | > avg_loss_feat: 3.302275973306575 (+0.11771344802748906) | > avg_loss_mel: 17.589734775919307 (+0.49814492883816186) | > avg_loss_duration: 1.5842974924705397 (+0.010889612453084618) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67321817639848 (+0.8457508355798851) | > avg_grad_norm_1: tensor(5269.2622, device='cuda:0') (tensor(-625.4907, device='cuda:0'))  > EPOCH: 678/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:11:17)   --> TIME: 2025-02-25 15:11:34 -- STEP: 33/72 -- GLOBAL_STEP: 1048850 | > loss_disc: 2.8292574882507324 (2.774944601636945) | > loss_disc_real_0: 0.2091890275478363 (0.2157094913224379) | > loss_disc_real_1: 0.5247986316680908 (0.2437740485325004) | > loss_disc_real_2: 0.22676198184490204 (0.25210997823512904) | > loss_disc_real_3: 0.23276440799236298 (0.24399358395374182) | > loss_disc_real_4: 0.22125756740570068 (0.2512220410686551) | > loss_disc_real_5: 0.21932637691497803 (0.2529006139798598) | > loss_0: 2.8292574882507324 (2.774944601636945) | > grad_norm_0: tensor(23.4670, device='cuda:0') (tensor(103.9726, device='cuda:0')) | > loss_gen: 2.1451117992401123 (2.0644299550489946) | > loss_kl: 2.1517183780670166 (2.383495742624456) | > loss_feat: 3.544098377227783 (3.229838486873742) | > loss_mel: 18.75311851501465 (19.151053862138227) | > loss_duration: 1.560776710510254 (1.5982605327259412) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.154823303222656 (28.427078593860973) | > grad_norm_1: tensor(6228., device='cuda:0') (tensor(5557.0532, device='cuda:0')) | > current_lr_0: 0.00018374741486490575 | > current_lr_1: 0.00018374741486490575 | > step_time: 0.4555 (0.4517787153070623) | > loader_time: 0.0097 (0.007536490758260091) --> EVAL PERFORMANCE | > avg_loader_time: 0.008938849811822596 (+0.00033465573485468546) | > avg_step_time: 0.4616348541958231 (+0.007863095108891882) | > avg_loss_disc: 2.742349822756271 (+0.020474437256934763) | > avg_loss_disc_real_0: 0.18277152979248007 (+0.016219695400394196) | > avg_loss_disc_real_1: 0.23630987810836712 (-0.0029435105307001885) | > avg_loss_disc_real_2: 0.24771860997441789 (-0.0052616999182904245) | > avg_loss_disc_real_3: 0.23878949670724467 (-0.00662293715376247) | > avg_loss_disc_real_4: 0.24765402897143027 (-0.0007100674048276245) | > avg_loss_disc_real_5: 0.25334631284357784 (+0.005216060809686035) | > avg_loss_0: 2.742349822756271 (+0.020474437256934763) | > avg_grad_norm_0: tensor(111.5083, device='cuda:0') (tensor(-8.5958, device='cuda:0')) | > avg_loss_gen: 2.0005499158107054 (-0.05880206907299268) | > avg_loss_kl: 2.4081639655878844 (+0.27060582940007505) | > avg_loss_feat: 3.1677985191345215 (-0.1344774541720537) | > avg_loss_mel: 19.217983487626192 (+1.6282487117068847) | > avg_loss_duration: 1.6163300678763592 (+0.0320325754058195) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.41082602487483 (+1.737607848476351) | > avg_grad_norm_1: tensor(5763.1870, device='cuda:0') (+tensor(493.9248, device='cuda:0'))  > EPOCH: 679/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:11:54)   --> TIME: 2025-02-25 15:12:01 -- STEP: 11/72 -- GLOBAL_STEP: 1048900 | > loss_disc: 2.5638697147369385 (2.558953653682362) | > loss_disc_real_0: 0.12307660281658173 (0.1609361239454963) | > loss_disc_real_1: 0.20786342024803162 (0.23125779899683865) | > loss_disc_real_2: 0.20681120455265045 (0.22455593130805276) | > loss_disc_real_3: 0.2127085030078888 (0.22400680048899216) | > loss_disc_real_4: 0.23356808722019196 (0.23950523815371774) | > loss_disc_real_5: 0.23917719721794128 (0.2339509969407862) | > loss_0: 2.5638697147369385 (2.558953653682362) | > grad_norm_0: tensor(225.6731, device='cuda:0') (tensor(123.8350, device='cuda:0')) | > loss_gen: 1.9498329162597656 (2.0688468759710137) | > loss_kl: 2.1147546768188477 (2.318389740857211) | > loss_feat: 3.631812334060669 (3.3766922517256304) | > loss_mel: 19.007591247558594 (18.930524132468484) | > loss_duration: 1.5559290647506714 (1.583265109495683) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.259920120239258 (28.277718110518023) | > grad_norm_1: tensor(7538.5083, device='cuda:0') (tensor(6629.7720, device='cuda:0')) | > current_lr_0: 0.00018372444643804763 | > current_lr_1: 0.00018372444643804763 | > step_time: 0.4334 (0.4332009662281383) | > loader_time: 0.0056 (0.006464156237515536)  --> TIME: 2025-02-25 15:12:24 -- STEP: 61/72 -- GLOBAL_STEP: 1048950 | > loss_disc: 2.620509624481201 (2.6646910221850284) | > loss_disc_real_0: 0.14495296776294708 (0.17064104003251576) | > loss_disc_real_1: 0.22248126566410065 (0.22610464692115784) | > loss_disc_real_2: 0.24297571182250977 (0.23430875798717873) | > loss_disc_real_3: 0.27044880390167236 (0.2364178026308779) | > loss_disc_real_4: 0.2597130239009857 (0.24048093694155334) | > loss_disc_real_5: 0.2724641263484955 (0.24216457566276925) | > loss_0: 2.620509624481201 (2.6646910221850284) | > grad_norm_0: tensor(200.0458, device='cuda:0') (tensor(127.1098, device='cuda:0')) | > loss_gen: 2.1193511486053467 (2.016640205852319) | > loss_kl: 2.2247121334075928 (2.2403860717523294) | > loss_feat: 3.2142672538757324 (3.2782544894296617) | > loss_mel: 17.65985107421875 (18.230564211235677) | > loss_duration: 1.622973084449768 (1.5904317605690879) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.841154098510742 (27.356276809192096) | > grad_norm_1: tensor(4343.1387, device='cuda:0') (tensor(5878.6748, device='cuda:0')) | > current_lr_0: 0.00018372444643804763 | > current_lr_1: 0.00018372444643804763 | > step_time: 0.465 (0.4476065166660997) | > loader_time: 0.011 (0.008066079655631647) --> EVAL PERFORMANCE | > avg_loader_time: 0.008554304149788873 (-0.000384545662033723) | > avg_step_time: 0.4567888555392413 (-0.004845998656581807) | > avg_loss_disc: 2.6675439351041557 (-0.07480588765211538) | > avg_loss_disc_real_0: 0.1714163168110478 (-0.011355212981432272) | > avg_loss_disc_real_1: 0.22634238214559957 (-0.009967495962767553) | > avg_loss_disc_real_2: 0.2338552972380544 (-0.013863312736363487) | > avg_loss_disc_real_3: 0.2360620525941043 (-0.0027274441131403815) | > avg_loss_disc_real_4: 0.24044171920124913 (-0.007212309770181141) | > avg_loss_disc_real_5: 0.24303726033425668 (-0.010309052509321154) | > avg_loss_0: 2.6675439351041557 (-0.07480588765211538) | > avg_grad_norm_0: tensor(121.2867, device='cuda:0') (+tensor(9.7784, device='cuda:0')) | > avg_loss_gen: 2.005561214097787 (+0.0050112982870818) | > avg_loss_kl: 2.2384253757100705 (-0.16973858987781387) | > avg_loss_feat: 3.2518847525959274 (+0.08408623346140587) | > avg_loss_mel: 18.188844895698658 (-1.0291385919275342) | > avg_loss_duration: 1.5969412393972908 (-0.01938882847906842) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.28165750100579 (-1.1291685238690405) | > avg_grad_norm_1: tensor(5852.4204, device='cuda:0') (+tensor(89.2334, device='cuda:0'))  > EPOCH: 680/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:12:31)   --> TIME: 2025-02-25 15:12:51 -- STEP: 39/72 -- GLOBAL_STEP: 1049000 | > loss_disc: 2.6416335105895996 (2.6832453287564793) | > loss_disc_real_0: 0.18295630812644958 (0.16396270864284956) | > loss_disc_real_1: 0.2203187495470047 (0.23235623290141424) | > loss_disc_real_2: 0.2553635835647583 (0.24115152542407697) | > loss_disc_real_3: 0.2347339242696762 (0.23806410149121895) | > loss_disc_real_4: 0.25040769577026367 (0.24391608933607736) | > loss_disc_real_5: 0.25332385301589966 (0.24827338793338874) | > loss_0: 2.6416335105895996 (2.6832453287564793) | > grad_norm_0: tensor(76.2277, device='cuda:0') (tensor(127.4151, device='cuda:0')) | > loss_gen: 2.0870628356933594 (2.080175271401038) | > loss_kl: 1.9747122526168823 (1.9990932696904895) | > loss_feat: 2.9687442779541016 (3.3350402269607935) | > loss_mel: 17.162227630615234 (17.326959316547104) | > loss_duration: 1.5739480257034302 (1.5488506647256703) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.766695022583008 (26.290118584266075) | > grad_norm_1: tensor(5829.8623, device='cuda:0') (tensor(6602.7002, device='cuda:0')) | > current_lr_0: 0.00018370148088224286 | > current_lr_1: 0.00018370148088224286 | > step_time: 0.4517 (0.44321661729079026) | > loader_time: 0.0087 (0.007408594473814353) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684447113896765 (+0.00013014296410789224) | > avg_step_time: 0.4566347834090112 (-0.00015407213023010957) | > avg_loss_disc: 2.6590522813125395 (-0.008491653791616205) | > avg_loss_disc_real_0: 0.14612461091347148 (-0.025291705897576316) | > avg_loss_disc_real_1: 0.2270234517858062 (+0.0006810696402066441) | > avg_loss_disc_real_2: 0.23953403343617077 (+0.005678736198116374) | > avg_loss_disc_real_3: 0.23482363702545703 (-0.0012384155686472609) | > avg_loss_disc_real_4: 0.24161991917751205 (+0.0011781999762629225) | > avg_loss_disc_real_5: 0.2454649526888216 (+0.0024276923545649143) | > avg_loss_0: 2.6590522813125395 (-0.008491653791616205) | > avg_grad_norm_0: tensor(119.0013, device='cuda:0') (tensor(-2.2854, device='cuda:0')) | > avg_loss_gen: 2.059975043149063 (+0.0544138290512759) | > avg_loss_kl: 2.1774176980408146 (-0.061007677669255855) | > avg_loss_feat: 3.3712596624669895 (+0.11937490987106214) | > avg_loss_mel: 17.985860582808375 (-0.2029843128902833) | > avg_loss_duration: 1.603221080672573 (+0.00627984127528225) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.197733811929194 (-0.08392368907659531) | > avg_grad_norm_1: tensor(6329.2866, device='cuda:0') (+tensor(476.8662, device='cuda:0'))  > EPOCH: 681/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:13:08)   --> TIME: 2025-02-25 15:13:18 -- STEP: 17/72 -- GLOBAL_STEP: 1049050 | > loss_disc: 2.7373147010803223 (2.694251944037045) | > loss_disc_real_0: 0.1306038349866867 (0.16483458065811327) | > loss_disc_real_1: 0.22102823853492737 (0.21624044811024384) | > loss_disc_real_2: 0.24317438900470734 (0.2418965180130566) | > loss_disc_real_3: 0.24994663894176483 (0.2338925468571046) | > loss_disc_real_4: 0.2646953761577606 (0.25156321771004625) | > loss_disc_real_5: 0.2786206901073456 (0.2544346364105449) | > loss_0: 2.7373147010803223 (2.694251944037045) | > grad_norm_0: tensor(44.7707, device='cuda:0') (tensor(154.3134, device='cuda:0')) | > loss_gen: 2.2225663661956787 (1.998488748774809) | > loss_kl: 2.4773268699645996 (2.033697717330035) | > loss_feat: 2.762941360473633 (3.3036687093622543) | > loss_mel: 19.88311767578125 (17.93286626479205) | > loss_duration: 1.6307487487792969 (1.56166611699497) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.976701736450195 (26.830387676463406) | > grad_norm_1: tensor(6413.4580, device='cuda:0') (tensor(5693.9995, device='cuda:0')) | > current_lr_0: 0.00018367851819713258 | > current_lr_1: 0.00018367851819713258 | > step_time: 0.4233 (0.4308352891136618) | > loader_time: 0.0061 (0.0066986504723044)  --> TIME: 2025-02-25 15:13:42 -- STEP: 67/72 -- GLOBAL_STEP: 1049100 | > loss_disc: 3.072377920150757 (2.703584268911561) | > loss_disc_real_0: 0.06946513801813126 (0.17419871754610713) | > loss_disc_real_1: 0.28116706013679504 (0.21872632105403872) | > loss_disc_real_2: 0.22463402152061462 (0.23758596470996515) | > loss_disc_real_3: 0.20521694421768188 (0.23258226954225283) | > loss_disc_real_4: 0.22669421136379242 (0.24787390899302356) | > loss_disc_real_5: 0.24789656698703766 (0.24866364896297455) | > loss_0: 3.072377920150757 (2.703584268911561) | > grad_norm_0: tensor(119.0534, device='cuda:0') (tensor(141.9802, device='cuda:0')) | > loss_gen: 1.6362898349761963 (2.08056571234518) | > loss_kl: 2.0782477855682373 (2.1011856367338964) | > loss_feat: 2.8908166885375977 (3.4560445315802277) | > loss_mel: 17.834815979003906 (17.724358103168544) | > loss_duration: 1.6684606075286865 (1.588962268473497) | > amp_scaler: 16.0 (13.014925373134329) | > loss_1: 26.108631134033203 (26.951116163339186) | > grad_norm_1: tensor(1760.1449, device='cuda:0') (tensor(4929.3628, device='cuda:0')) | > current_lr_0: 0.00018367851819713258 | > current_lr_1: 0.00018367851819713258 | > step_time: 0.5266 (0.45688813479978646) | > loader_time: 0.0127 (0.00875440284387389) --> EVAL PERFORMANCE | > avg_loader_time: 0.008912653990194828 (+0.00022820687629806352) | > avg_step_time: 0.4608343957175671 (+0.004199612308555922) | > avg_loss_disc: 2.7140692254187355 (+0.055016944106196064) | > avg_loss_disc_real_0: 0.1777993986304377 (+0.03167478771696622) | > avg_loss_disc_real_1: 0.2186311503321352 (-0.008392301453671008) | > avg_loss_disc_real_2: 0.23763480299795178 (-0.0018992304382189906) | > avg_loss_disc_real_3: 0.23288965623983196 (-0.0019339807856250668) | > avg_loss_disc_real_4: 0.24745798110961914 (+0.0058380619321070915) | > avg_loss_disc_real_5: 0.2487935765948094 (+0.0033286239059878153) | > avg_loss_0: 2.7140692254187355 (+0.055016944106196064) | > avg_grad_norm_0: tensor(143.6901, device='cuda:0') (+tensor(24.6889, device='cuda:0')) | > avg_loss_gen: 2.0663371724142157 (+0.006362129265152561) | > avg_loss_kl: 2.101063498308962 (-0.07635419973185265) | > avg_loss_feat: 3.416392228972744 (+0.04513256650575448) | > avg_loss_mel: 17.685076821018274 (-0.30078376179010036) | > avg_loss_duration: 1.5910925109621505 (-0.012128569710422532) | > avg_amp_scaler: 13.183098591549296 (+5.183098591549296) | > avg_loss_1: 26.85996214101012 (-0.337771670919075) | > avg_grad_norm_1: tensor(4795.0278, device='cuda:0') (tensor(-1534.2588, device='cuda:0'))  > EPOCH: 682/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:13:46)   --> TIME: 2025-02-25 15:14:08 -- STEP: 45/72 -- GLOBAL_STEP: 1049150 | > loss_disc: 3.2227139472961426 (2.7325167073143857) | > loss_disc_real_0: 0.0699857696890831 (0.1611509573956331) | > loss_disc_real_1: 0.2116427719593048 (0.2307147353887558) | > loss_disc_real_2: 0.19414876401424408 (0.22950909170839523) | > loss_disc_real_3: 0.2726845443248749 (0.23177009556028577) | > loss_disc_real_4: 0.25928544998168945 (0.2442486223247316) | > loss_disc_real_5: 0.23045404255390167 (0.24328824943966335) | > loss_0: 3.2227139472961426 (2.7325167073143857) | > grad_norm_0: tensor(109.2953, device='cuda:0') (tensor(99.3827, device='cuda:0')) | > loss_gen: 1.4743633270263672 (2.0116599268383446) | > loss_kl: 2.991891384124756 (2.6673481623331705) | > loss_feat: 2.860022783279419 (3.4896166218651667) | > loss_mel: 20.79444694519043 (20.12716272142198) | > loss_duration: 1.9980977773666382 (1.6474094735251532) | > amp_scaler: 16.0 (16.0) | > loss_1: 30.118824005126953 (29.94319665696886) | > grad_norm_1: tensor(4816.5303, device='cuda:0') (tensor(4898.1909, device='cuda:0')) | > current_lr_0: 0.00018365555838235795 | > current_lr_1: 0.00018365555838235795 | > step_time: 0.4537 (0.44245577918158635) | > loader_time: 0.0093 (0.007504733403523763) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573206377701021 (-0.0003394476124938071) | > avg_step_time: 0.4559204645559821 (-0.004913931161585006) | > avg_loss_disc: 2.703681468963623 (-0.010387756455112473) | > avg_loss_disc_real_0: 0.1484183858502919 (-0.029381012780145793) | > avg_loss_disc_real_1: 0.23331392168159215 (+0.014682771349456947) | > avg_loss_disc_real_2: 0.22784122027141948 (-0.009793582726532302) | > avg_loss_disc_real_3: 0.23526892548715564 (+0.0023792692473236787) | > avg_loss_disc_real_4: 0.24406491010121897 (-0.003393071008400167) | > avg_loss_disc_real_5: 0.24358904823450975 (-0.005204528360299665) | > avg_loss_0: 2.703681468963623 (-0.010387756455112473) | > avg_grad_norm_0: tensor(114.6894, device='cuda:0') (tensor(-29.0007, device='cuda:0')) | > avg_loss_gen: 1.9927041614559335 (-0.07363301095828212) | > avg_loss_kl: 2.641860804087679 (+0.540797305778717) | > avg_loss_feat: 3.3754459602732054 (-0.040946268699538546) | > avg_loss_mel: 20.04414797500825 (+2.3590711539899765) | > avg_loss_duration: 1.6868484255293725 (+0.09575591456722199) | > avg_amp_scaler: 16.0 (+2.816901408450704) | > avg_loss_1: 29.741007092972875 (+2.881044951962757) | > avg_grad_norm_1: tensor(5536.5708, device='cuda:0') (+tensor(741.5430, device='cuda:0'))  > EPOCH: 683/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:14:23)   --> TIME: 2025-02-25 15:14:35 -- STEP: 23/72 -- GLOBAL_STEP: 1049200 | > loss_disc: 2.49422287940979 (2.578866419584855) | > loss_disc_real_0: 0.08540868759155273 (0.13951646407013354) | > loss_disc_real_1: 0.20352140069007874 (0.21702824273835059) | > loss_disc_real_2: 0.2414557933807373 (0.2331159328636916) | > loss_disc_real_3: 0.23095692694187164 (0.23832840569641278) | > loss_disc_real_4: 0.24837587773799896 (0.2425031260303829) | > loss_disc_real_5: 0.2580569386482239 (0.2421751611906549) | > loss_0: 2.49422287940979 (2.578866419584855) | > grad_norm_0: tensor(79.8424, device='cuda:0') (tensor(163.1743, device='cuda:0')) | > loss_gen: 2.1388537883758545 (2.052702525387639) | > loss_kl: 2.1838951110839844 (2.096163946649302) | > loss_feat: 3.724661350250244 (3.389043123825737) | > loss_mel: 19.326189041137695 (17.987330561098844) | > loss_duration: 1.595711588859558 (1.5889573615530264) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.969308853149414 (27.114197274912957) | > grad_norm_1: tensor(7158.1001, device='cuda:0') (tensor(7484.9355, device='cuda:0')) | > current_lr_0: 0.00018363260143756013 | > current_lr_1: 0.00018363260143756013 | > step_time: 0.4635 (0.4342121041339377) | > loader_time: 0.0078 (0.006865252619204314) --> EVAL PERFORMANCE | > avg_loader_time: 0.008787423792019697 (+0.00021421741431867578) | > avg_step_time: 0.45597541500145283 (+5.495044547071393e-05) | > avg_loss_disc: 2.6095279270494487 (-0.09415354191417435) | > avg_loss_disc_real_0: 0.12447933653290842 (-0.023939049317383485) | > avg_loss_disc_real_1: 0.22035418618732777 (-0.01295973549426438) | > avg_loss_disc_real_2: 0.23497395951982955 (+0.007132739248410069) | > avg_loss_disc_real_3: 0.22623555467162335 (-0.009033370815532293) | > avg_loss_disc_real_4: 0.2388009061695824 (-0.005264003931636563) | > avg_loss_disc_real_5: 0.24302945275541762 (-0.0005595954790921231) | > avg_loss_0: 2.6095279270494487 (-0.09415354191417435) | > avg_grad_norm_0: tensor(160.4295, device='cuda:0') (+tensor(45.7401, device='cuda:0')) | > avg_loss_gen: 2.010441681029091 (+0.0177375195731575) | > avg_loss_kl: 2.092343865985601 (-0.5495169381020779) | > avg_loss_feat: 3.4087480759956468 (+0.03330211572244135) | > avg_loss_mel: 17.72123839150012 (-2.3229095835081317) | > avg_loss_duration: 1.6051006384298832 (-0.08174778709948938) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.83787254548409 (-2.9031345474887864) | > avg_grad_norm_1: tensor(7091.5825, device='cuda:0') (+tensor(1555.0117, device='cuda:0'))  > EPOCH: 684/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:15:00)   --> TIME: 2025-02-25 15:15:02 -- STEP: 1/72 -- GLOBAL_STEP: 1049250 | > loss_disc: 2.5090415477752686 (2.5090415477752686) | > loss_disc_real_0: 0.1116141602396965 (0.1116141602396965) | > loss_disc_real_1: 0.24215756356716156 (0.24215756356716156) | > loss_disc_real_2: 0.2895904779434204 (0.2895904779434204) | > loss_disc_real_3: 0.22624781727790833 (0.22624781727790833) | > loss_disc_real_4: 0.2526020407676697 (0.2526020407676697) | > loss_disc_real_5: 0.2377605140209198 (0.2377605140209198) | > loss_0: 2.5090415477752686 (2.5090415477752686) | > grad_norm_0: tensor(197.3262, device='cuda:0') (tensor(197.3262, device='cuda:0')) | > loss_gen: 2.175908088684082 (2.175908088684082) | > loss_kl: 1.7732760906219482 (1.7732760906219482) | > loss_feat: 3.469712972640991 (3.469712972640991) | > loss_mel: 18.265567779541016 (18.265567779541016) | > loss_duration: 1.5884711742401123 (1.5884711742401123) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.272933959960938 (27.272933959960938) | > grad_norm_1: tensor(9432.3730, device='cuda:0') (tensor(9432.3730, device='cuda:0')) | > current_lr_0: 0.00018360964736238044 | > current_lr_1: 0.00018360964736238044 | > step_time: 0.4221 (0.4221310615539551) | > loader_time: 0.0063 (0.00633549690246582)  --> TIME: 2025-02-25 15:15:25 -- STEP: 51/72 -- GLOBAL_STEP: 1049300 | > loss_disc: 2.7936153411865234 (2.6655858731737325) | > loss_disc_real_0: 0.17619073390960693 (0.11753435272212122) | > loss_disc_real_1: 0.24660134315490723 (0.21983157828742383) | > loss_disc_real_2: 0.2251613587141037 (0.225394928280045) | > loss_disc_real_3: 0.2348284125328064 (0.234308514232729) | > loss_disc_real_4: 0.29519546031951904 (0.24415209012873032) | > loss_disc_real_5: 0.2613869905471802 (0.24288428063486137) | > loss_0: 2.7936153411865234 (2.6655858731737325) | > grad_norm_0: tensor(19.7074, device='cuda:0') (tensor(184.8791, device='cuda:0')) | > loss_gen: 1.9854954481124878 (1.9604189582899507) | > loss_kl: 2.032092809677124 (1.9598812005099129) | > loss_feat: 2.927814483642578 (3.4792922805337345) | > loss_mel: 17.416015625 (17.365196901209213) | > loss_duration: 1.5998890399932861 (1.5670613588071336) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.961307525634766 (26.331850650263767) | > grad_norm_1: tensor(7434.0957, device='cuda:0') (tensor(7408.3984, device='cuda:0')) | > current_lr_0: 0.00018360964736238044 | > current_lr_1: 0.00018360964736238044 | > step_time: 0.4393 (0.4517076249216117) | > loader_time: 0.0088 (0.007917315352196785) --> EVAL PERFORMANCE | > avg_loader_time: 0.008713520748514524 (-7.390304350517359e-05) | > avg_step_time: 0.4645073783229774 (+0.00853196332152456) | > avg_loss_disc: 2.6639047441348223 (+0.05437681708537356) | > avg_loss_disc_real_0: 0.13262652165033442 (+0.008147185117426) | > avg_loss_disc_real_1: 0.22100546872112112 (+0.0006512825337933459) | > avg_loss_disc_real_2: 0.22891462940565296 (-0.0060593301141765865) | > avg_loss_disc_real_3: 0.2347207747294869 (+0.008485220057863563) | > avg_loss_disc_real_4: 0.24222264407386243 (+0.00342173790428002) | > avg_loss_disc_real_5: 0.24453390169311578 (+0.0015044489376981607) | > avg_loss_0: 2.6639047441348223 (+0.05437681708537356) | > avg_grad_norm_0: tensor(185.2419, device='cuda:0') (+tensor(24.8125, device='cuda:0')) | > avg_loss_gen: 1.970941335382596 (-0.03950034564649507) | > avg_loss_kl: 1.9991604240847305 (-0.09318344190087058) | > avg_loss_feat: 3.3663192702011324 (-0.04242880579451436) | > avg_loss_mel: 17.239840010522116 (-0.48139838097800336) | > avg_loss_duration: 1.5822948066281601 (-0.02280583180172302) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.15855582331268 (-0.6793167221714107) | > avg_grad_norm_1: tensor(7386.3662, device='cuda:0') (+tensor(294.7837, device='cuda:0'))  > EPOCH: 685/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:15:37)   --> TIME: 2025-02-25 15:15:52 -- STEP: 29/72 -- GLOBAL_STEP: 1049350 | > loss_disc: 2.649350881576538 (2.7217087745666504) | > loss_disc_real_0: 0.09232351183891296 (0.1839979979498633) | > loss_disc_real_1: 0.21870918571949005 (0.22365130997937302) | > loss_disc_real_2: 0.24856042861938477 (0.2533112743805195) | > loss_disc_real_3: 0.23115399479866028 (0.2601780100115414) | > loss_disc_real_4: 0.2707817852497101 (0.25722554069140857) | > loss_disc_real_5: 0.26452693343162537 (0.2605981451684031) | > loss_0: 2.649350881576538 (2.7217087745666504) | > grad_norm_0: tensor(210.5652, device='cuda:0') (tensor(159.1295, device='cuda:0')) | > loss_gen: 1.9937772750854492 (2.0755893197552915) | > loss_kl: 2.3559439182281494 (2.06963523091941) | > loss_feat: 3.6120641231536865 (3.315196497686978) | > loss_mel: 18.01078987121582 (17.764042229487977) | > loss_duration: 1.5520048141479492 (1.543792572514764) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.524581909179688 (26.76825635186557) | > grad_norm_1: tensor(9910.1328, device='cuda:0') (tensor(6393.3213, device='cuda:0')) | > current_lr_0: 0.00018358669615646013 | > current_lr_1: 0.00018358669615646013 | > step_time: 0.4565 (0.4369790142980115) | > loader_time: 0.0075 (0.006929767542871936) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555234318048185 (-0.00015828643046633846) | > avg_step_time: 0.4556820963470028 (-0.008825281975974597) | > avg_loss_disc: 2.692276877416691 (+0.028372133281868717) | > avg_loss_disc_real_0: 0.16048672396532246 (+0.027860202314988036) | > avg_loss_disc_real_1: 0.23896748247280927 (+0.017962013751688155) | > avg_loss_disc_real_2: 0.24413677189551608 (+0.015222142489863116) | > avg_loss_disc_real_3: 0.245250394016924 (+0.010529619287437092) | > avg_loss_disc_real_4: 0.25025132571307696 (+0.008028681639214524) | > avg_loss_disc_real_5: 0.2509747856939344 (+0.006440884000818603) | > avg_loss_0: 2.692276877416691 (+0.028372133281868717) | > avg_grad_norm_0: tensor(148.4914, device='cuda:0') (tensor(-36.7505, device='cuda:0')) | > avg_loss_gen: 2.0379653413530807 (+0.06702400597048475) | > avg_loss_kl: 2.1662754424860777 (+0.16711501840134724) | > avg_loss_feat: 3.2899257565887883 (-0.07639351361234414) | > avg_loss_mel: 17.83927988670241 (+0.5994398761802948) | > avg_loss_duration: 1.5934662197677183 (+0.011171413139558162) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.926912952476822 (+0.7683571291641442) | > avg_grad_norm_1: tensor(6518.7227, device='cuda:0') (tensor(-867.6436, device='cuda:0'))  > EPOCH: 686/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:16:14)   --> TIME: 2025-02-25 15:16:19 -- STEP: 7/72 -- GLOBAL_STEP: 1049400 | > loss_disc: 2.763740062713623 (2.85758113861084) | > loss_disc_real_0: 0.058869846165180206 (0.05299465198601995) | > loss_disc_real_1: 0.24418888986110687 (0.25080928419317516) | > loss_disc_real_2: 0.24203233420848846 (0.3059111684560776) | > loss_disc_real_3: 0.22671844065189362 (0.20575889945030212) | > loss_disc_real_4: 0.2424820065498352 (0.23086027375289372) | > loss_disc_real_5: 0.2287246286869049 (0.2292815170117787) | > loss_0: 2.763740062713623 (2.85758113861084) | > grad_norm_0: tensor(230.8039, device='cuda:0') (tensor(256.5238, device='cuda:0')) | > loss_gen: 1.7775713205337524 (1.7721609217779977) | > loss_kl: 1.9239553213119507 (1.992114509854998) | > loss_feat: 3.405609130859375 (3.573089599609375) | > loss_mel: 17.3530216217041 (17.415286745343888) | > loss_duration: 1.5357205867767334 (1.5534861258098058) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.995878219604492 (26.30613817487444) | > grad_norm_1: tensor(7284.3696, device='cuda:0') (tensor(7310.4668, device='cuda:0')) | > current_lr_0: 0.00018356374781944057 | > current_lr_1: 0.00018356374781944057 | > step_time: 0.4313 (0.42874261311122347) | > loader_time: 0.007 (0.006396021161760602)  --> TIME: 2025-02-25 15:16:42 -- STEP: 57/72 -- GLOBAL_STEP: 1049450 | > loss_disc: 3.0456743240356445 (2.6761191267716256) | > loss_disc_real_0: 0.11299437284469604 (0.12009773723650397) | > loss_disc_real_1: 0.21785816550254822 (0.22375775586094773) | > loss_disc_real_2: 0.21554788947105408 (0.24518231446282912) | > loss_disc_real_3: 0.26212385296821594 (0.2257899555720781) | > loss_disc_real_4: 0.23207376897335052 (0.24038497842194742) | > loss_disc_real_5: 0.25323936343193054 (0.24039520218706967) | > loss_0: 3.0456743240356445 (2.6761191267716256) | > grad_norm_0: tensor(223.4861, device='cuda:0') (tensor(169.5397, device='cuda:0')) | > loss_gen: 1.4598898887634277 (1.9828738789809377) | > loss_kl: 2.1042251586914062 (1.9906499929595411) | > loss_feat: 2.7894861698150635 (3.4490696840118944) | > loss_mel: 16.464845657348633 (17.134348735474692) | > loss_duration: 1.6339097023010254 (1.565846468272962) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.452356338500977 (26.12278878061395) | > grad_norm_1: tensor(93.5989, device='cuda:0') (tensor(7263.2783, device='cuda:0')) | > current_lr_0: 0.00018356374781944057 | > current_lr_1: 0.00018356374781944057 | > step_time: 0.4528 (0.44471930202684906) | > loader_time: 0.0096 (0.00788292550204093) --> EVAL PERFORMANCE | > avg_loader_time: 0.008539804270569707 (-1.543004747847805e-05) | > avg_step_time: 0.45414162353730536 (-0.0015404728096974352) | > avg_loss_disc: 2.6686554391619186 (-0.023621438254772364) | > avg_loss_disc_real_0: 0.11306155740585126 (-0.047425166559471196) | > avg_loss_disc_real_1: 0.2217953234491214 (-0.017172159023687877) | > avg_loss_disc_real_2: 0.24856455666078653 (+0.004427784765270448) | > avg_loss_disc_real_3: 0.23358620388407103 (-0.01166419013285297) | > avg_loss_disc_real_4: 0.24129793740494151 (-0.00895338830813544) | > avg_loss_disc_real_5: 0.24237793453142675 (-0.008596851162507635) | > avg_loss_0: 2.6686554391619186 (-0.023621438254772364) | > avg_grad_norm_0: tensor(164.2817, device='cuda:0') (+tensor(15.7903, device='cuda:0')) | > avg_loss_gen: 2.0112725664192523 (-0.026692774933828378) | > avg_loss_kl: 2.3040078525811856 (+0.13773241009510784) | > avg_loss_feat: 3.47201003491039 (+0.1820842783216019) | > avg_loss_mel: 18.095451851965663 (+0.2561719652632526) | > avg_loss_duration: 1.6248989373865261 (+0.03143271761880784) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.50764124158402 (+0.5807282891071992) | > avg_grad_norm_1: tensor(6886.3540, device='cuda:0') (+tensor(367.6313, device='cuda:0'))  > EPOCH: 687/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:16:51)   --> TIME: 2025-02-25 15:17:09 -- STEP: 35/72 -- GLOBAL_STEP: 1049500 | > loss_disc: 2.818775177001953 (2.641200508390154) | > loss_disc_real_0: 0.3420008420944214 (0.1532336216952119) | > loss_disc_real_1: 0.2165861427783966 (0.2251868486404419) | > loss_disc_real_2: 0.20623520016670227 (0.23095461513314927) | > loss_disc_real_3: 0.2664734423160553 (0.23588164150714874) | > loss_disc_real_4: 0.2449864000082016 (0.24154581683022636) | > loss_disc_real_5: 0.2353208065032959 (0.24171306576047624) | > loss_0: 2.818775177001953 (2.641200508390154) | > grad_norm_0: tensor(212.9526, device='cuda:0') (tensor(132.2680, device='cuda:0')) | > loss_gen: 1.9310195446014404 (2.123377295902797) | > loss_kl: 2.2083234786987305 (2.36833712714059) | > loss_feat: 2.7601397037506104 (3.607125561577933) | > loss_mel: 19.231121063232422 (19.224335861206054) | > loss_duration: 1.6044716835021973 (1.6178342853273664) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.735074996948242 (28.9410094669887) | > grad_norm_1: tensor(2454.7727, device='cuda:0') (tensor(5469.0190, device='cuda:0')) | > current_lr_0: 0.00018354080235096314 | > current_lr_1: 0.00018354080235096314 | > step_time: 0.4354 (0.43500166620527) | > loader_time: 0.0081 (0.007136086055210659) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857262544228997 (+3.2821171720263595e-05) | > avg_step_time: 0.46117368886168575 (+0.007032065324380388) | > avg_loss_disc: 2.6488822950443756 (-0.01977314411754305) | > avg_loss_disc_real_0: 0.1446339592442546 (+0.03157240183840333) | > avg_loss_disc_real_1: 0.22049220431018884 (-0.0013031191389325514) | > avg_loss_disc_real_2: 0.23864504464075598 (-0.009919512020030552) | > avg_loss_disc_real_3: 0.23687209272888346 (+0.003285888844812429) | > avg_loss_disc_real_4: 0.24232399946367236 (+0.0010260620587308478) | > avg_loss_disc_real_5: 0.24421573440793534 (+0.001837799876508589) | > avg_loss_0: 2.6488822950443756 (-0.01977314411754305) | > avg_grad_norm_0: tensor(128.4623, device='cuda:0') (tensor(-35.8194, device='cuda:0')) | > avg_loss_gen: 2.0819880592990927 (+0.07071549287984036) | > avg_loss_kl: 2.251729785556525 (-0.05227806702466076) | > avg_loss_feat: 3.5218950963356126 (+0.04988506142522242) | > avg_loss_mel: 18.50827192924392 (+0.4128200772782584) | > avg_loss_duration: 1.620861765364526 (-0.004037172022000046) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.9847463151099 (+0.47710507352587683) | > avg_grad_norm_1: tensor(5536.7822, device='cuda:0') (tensor(-1349.5718, device='cuda:0'))  > EPOCH: 688/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:17:29)   --> TIME: 2025-02-25 15:17:36 -- STEP: 13/72 -- GLOBAL_STEP: 1049550 | > loss_disc: 2.738131284713745 (2.684680003386277) | > loss_disc_real_0: 0.15069633722305298 (0.1653802773127189) | > loss_disc_real_1: 0.20717009902000427 (0.22498560410279494) | > loss_disc_real_2: 0.21933792531490326 (0.23815972988422102) | > loss_disc_real_3: 0.25372371077537537 (0.24507063397994408) | > loss_disc_real_4: 0.2510662376880646 (0.25137256773618555) | > loss_disc_real_5: 0.25654149055480957 (0.24420871299046737) | > loss_0: 2.738131284713745 (2.684680003386277) | > grad_norm_0: tensor(10.2233, device='cuda:0') (tensor(112.8259, device='cuda:0')) | > loss_gen: 2.1509878635406494 (2.0127755036720862) | > loss_kl: 1.9131524562835693 (1.900045477426969) | > loss_feat: 3.4441077709198 (3.34741473197937) | > loss_mel: 16.77875518798828 (17.67882537841797) | > loss_duration: 1.4659287929534912 (1.5508744991742647) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.752931594848633 (26.489935361422024) | > grad_norm_1: tensor(2628.3218, device='cuda:0') (tensor(5848.1343, device='cuda:0')) | > current_lr_0: 0.00018351785975066927 | > current_lr_1: 0.00018351785975066927 | > step_time: 0.4669 (0.43522682556739223) | > loader_time: 0.0072 (0.0066896585317758415)  --> TIME: 2025-02-25 15:18:00 -- STEP: 63/72 -- GLOBAL_STEP: 1049600 | > loss_disc: 2.553358554840088 (2.7044434850178067) | > loss_disc_real_0: 0.11064555495977402 (0.17598195693322594) | > loss_disc_real_1: 0.27407896518707275 (0.25035971888000996) | > loss_disc_real_2: 0.1823505461215973 (0.23721856635714333) | > loss_disc_real_3: 0.21489843726158142 (0.2340813586636195) | > loss_disc_real_4: 0.23246337473392487 (0.24695549834342237) | > loss_disc_real_5: 0.23228394985198975 (0.2487475053658561) | > loss_0: 2.553358554840088 (2.7044434850178067) | > grad_norm_0: tensor(169.6871, device='cuda:0') (tensor(119.9365, device='cuda:0')) | > loss_gen: 1.949392318725586 (2.0832499057527563) | > loss_kl: 2.2220678329467773 (2.0327246643248054) | > loss_feat: 3.246657133102417 (3.3456832643539185) | > loss_mel: 18.31340217590332 (17.559909411839076) | > loss_duration: 1.613309621810913 (1.5819932600808522) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.344829559326172 (26.603560493105935) | > grad_norm_1: tensor(4376.1309, device='cuda:0') (tensor(5721.4868, device='cuda:0')) | > current_lr_0: 0.00018351785975066927 | > current_lr_1: 0.00018351785975066927 | > step_time: 0.5008 (0.4453246934073312) | > loader_time: 0.0109 (0.008315302076793856) --> EVAL PERFORMANCE | > avg_loader_time: 0.008665121776956912 (+9.249633466694093e-05) | > avg_step_time: 0.45192596274362484 (-0.009247726118060906) | > avg_loss_disc: 2.697117583852419 (+0.048235288808043286) | > avg_loss_disc_real_0: 0.16806647721940365 (+0.02343251797514906) | > avg_loss_disc_real_1: 0.2504596835081006 (+0.02996747919791176) | > avg_loss_disc_real_2: 0.23698425125068343 (-0.0016607933900725513) | > avg_loss_disc_real_3: 0.2352637787100295 (-0.0016083140188539735) | > avg_loss_disc_real_4: 0.24511095713561695 (+0.00278695767194459) | > avg_loss_disc_real_5: 0.24730146686795731 (+0.0030857324600219727) | > avg_loss_0: 2.697117583852419 (+0.048235288808043286) | > avg_grad_norm_0: tensor(122.8786, device='cuda:0') (tensor(-5.5837, device='cuda:0')) | > avg_loss_gen: 2.0722908167771883 (-0.009697242521904403) | > avg_loss_kl: 2.046752143913591 (-0.20497764164293386) | > avg_loss_feat: 3.3291765636121724 (-0.1927185327234402) | > avg_loss_mel: 17.60466265342605 (-0.9036092758178711) | > avg_loss_duration: 1.5893327115287244 (-0.031529053835801735) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.642214869109676 (-1.3425314460002227) | > avg_grad_norm_1: tensor(5644.1582, device='cuda:0') (+tensor(107.3760, device='cuda:0'))  > EPOCH: 689/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:18:05)   --> TIME: 2025-02-25 15:18:26 -- STEP: 41/72 -- GLOBAL_STEP: 1049650 | > loss_disc: 2.746906280517578 (2.6724051731388747) | > loss_disc_real_0: 0.16786284744739532 (0.13144089726776612) | > loss_disc_real_1: 0.1862698644399643 (0.21447102689161535) | > loss_disc_real_2: 0.23135201632976532 (0.24328503975781007) | > loss_disc_real_3: 0.27496281266212463 (0.23448427730217214) | > loss_disc_real_4: 0.2427842766046524 (0.24433255922503588) | > loss_disc_real_5: 0.22362633049488068 (0.25012463803698376) | > loss_0: 2.746906280517578 (2.6724051731388747) | > grad_norm_0: tensor(14.9416, device='cuda:0') (tensor(134.8702, device='cuda:0')) | > loss_gen: 1.9203191995620728 (2.026316096143024) | > loss_kl: 2.0928657054901123 (2.020663124759023) | > loss_feat: 3.238013505935669 (3.380586251979921) | > loss_mel: 17.715839385986328 (17.673790071068737) | > loss_duration: 1.5858609676361084 (1.5595514338190963) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.552898406982422 (26.660906954509457) | > grad_norm_1: tensor(2616.9836, device='cuda:0') (tensor(5204.8262, device='cuda:0')) | > current_lr_0: 0.00018349492001820043 | > current_lr_1: 0.00018349492001820043 | > step_time: 0.452 (0.44005300359028143) | > loader_time: 0.009 (0.007294288495691811) --> EVAL PERFORMANCE | > avg_loader_time: 0.008566826162203936 (-9.829561475297593e-05) | > avg_step_time: 0.45547162646978673 (+0.003545663726161885) | > avg_loss_disc: 2.6540551387088396 (-0.043062445143579264) | > avg_loss_disc_real_0: 0.14474643929533557 (-0.023320037924068088) | > avg_loss_disc_real_1: 0.21985552176623277 (-0.030604161741867836) | > avg_loss_disc_real_2: 0.2397435058800268 (+0.0027592546293433717) | > avg_loss_disc_real_3: 0.23057755250746095 (-0.004686226202568533) | > avg_loss_disc_real_4: 0.24229878837793645 (-0.0028121687576805054) | > avg_loss_disc_real_5: 0.24859699858746054 (+0.0012955317195032234) | > avg_loss_0: 2.6540551387088396 (-0.043062445143579264) | > avg_grad_norm_0: tensor(133.0481, device='cuda:0') (+tensor(10.1695, device='cuda:0')) | > avg_loss_gen: 2.08646202759004 (+0.014171210812851776) | > avg_loss_kl: 2.040749878950522 (-0.006002264963068882) | > avg_loss_feat: 3.4065907941737645 (+0.07741423056159213) | > avg_loss_mel: 17.386915394957636 (-0.21774725846841392) | > avg_loss_duration: 1.5833882647500912 (-0.00594444677863315) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.50410649474238 (-0.13810837436729528) | > avg_grad_norm_1: tensor(5253.2944, device='cuda:0') (tensor(-390.8638, device='cuda:0'))  > EPOCH: 690/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:18:42)   --> TIME: 2025-02-25 15:18:53 -- STEP: 19/72 -- GLOBAL_STEP: 1049700 | > loss_disc: 2.935734987258911 (2.6980277362622713) | > loss_disc_real_0: 0.06469801813364029 (0.1325650354357142) | > loss_disc_real_1: 0.16236832737922668 (0.22392557169261731) | > loss_disc_real_2: 0.27886492013931274 (0.24624527049692055) | > loss_disc_real_3: 0.2641024589538574 (0.24698279484322197) | > loss_disc_real_4: 0.2633216381072998 (0.2523877691281469) | > loss_disc_real_5: 0.22623293101787567 (0.24406535374490837) | > loss_0: 2.935734987258911 (2.6980277362622713) | > grad_norm_0: tensor(292.4523, device='cuda:0') (tensor(120.5126, device='cuda:0')) | > loss_gen: 1.6043181419372559 (1.9751721746043156) | > loss_kl: 1.9241855144500732 (1.962918626634698) | > loss_feat: 2.919455051422119 (3.34617354995326) | > loss_mel: 16.562355041503906 (17.468708339490387) | > loss_duration: 1.513680338859558 (1.5419719972108539) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.52399253845215 (26.294944763183594) | > grad_norm_1: tensor(5713.8916, device='cuda:0') (tensor(5335.5747, device='cuda:0')) | > current_lr_0: 0.00018347198315319814 | > current_lr_1: 0.00018347198315319814 | > step_time: 0.4263 (0.432884881370946) | > loader_time: 0.0065 (0.0068035125732421875)  --> TIME: 2025-02-25 15:19:17 -- STEP: 69/72 -- GLOBAL_STEP: 1049750 | > loss_disc: 2.719801425933838 (2.6752009495444913) | > loss_disc_real_0: 0.24367697536945343 (0.14228814840316767) | > loss_disc_real_1: 0.2063601315021515 (0.221925125821777) | > loss_disc_real_2: 0.26026374101638794 (0.23880714674790707) | > loss_disc_real_3: 0.2353830635547638 (0.23979518638140912) | > loss_disc_real_4: 0.25788775086402893 (0.24668756213741025) | > loss_disc_real_5: 0.2532157301902771 (0.24483845609685648) | > loss_0: 2.719801425933838 (2.6752009495444913) | > grad_norm_0: tensor(216.5426, device='cuda:0') (tensor(123.8257, device='cuda:0')) | > loss_gen: 2.03466796875 (1.992988781652589) | > loss_kl: 1.9542467594146729 (2.0741511358731035) | > loss_feat: 3.2242369651794434 (3.2846108277638755) | > loss_mel: 18.42595672607422 (17.483478933140844) | > loss_duration: 1.6652048826217651 (1.588690161705017) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.30431365966797 (26.423919843590777) | > grad_norm_1: tensor(8189.8232, device='cuda:0') (tensor(5180.7046, device='cuda:0')) | > current_lr_0: 0.00018347198315319814 | > current_lr_1: 0.00018347198315319814 | > step_time: 0.5379 (0.45476789059846295) | > loader_time: 0.0127 (0.008641059847845547) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870163339964101 (+0.00013480723743707415) | > avg_step_time: 0.4569222692032935 (+0.0014506427335067573) | > avg_loss_disc: 2.6756953320033108 (+0.0216401932944712) | > avg_loss_disc_real_0: 0.1429443455917734 (-0.0018020937035621576) | > avg_loss_disc_real_1: 0.2213943898677826 (+0.001538868101549823) | > avg_loss_disc_real_2: 0.23907690358833533 (-0.0006666022916914649) | > avg_loss_disc_real_3: 0.23941228738133336 (+0.008834734873872402) | > avg_loss_disc_real_4: 0.24648990618510985 (+0.004191117807173406) | > avg_loss_disc_real_5: 0.24495675068506054 (-0.003640247902400001) | > avg_loss_0: 2.6756953320033108 (+0.0216401932944712) | > avg_grad_norm_0: tensor(120.8572, device='cuda:0') (tensor(-12.1909, device='cuda:0')) | > avg_loss_gen: 1.9906066236361655 (-0.0958554039538746) | > avg_loss_kl: 2.073130213039022 (+0.03238033408849983) | > avg_loss_feat: 3.278249502182007 (-0.1283412919917577) | > avg_loss_mel: 17.45565617252403 (+0.06874077756639352) | > avg_loss_duration: 1.5890274484392624 (+0.005639183689171157) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.386669964857504 (-0.11743652988487696) | > avg_grad_norm_1: tensor(5108.8906, device='cuda:0') (tensor(-144.4038, device='cuda:0'))  > EPOCH: 691/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:19:20)   --> TIME: 2025-02-25 15:19:43 -- STEP: 47/72 -- GLOBAL_STEP: 1049800 | > loss_disc: 2.8553507328033447 (2.7227231492387483) | > loss_disc_real_0: 0.28278636932373047 (0.18810207808905463) | > loss_disc_real_1: 0.24134576320648193 (0.238077941093039) | > loss_disc_real_2: 0.26240211725234985 (0.23211973493403576) | > loss_disc_real_3: 0.2276318520307541 (0.23973854932379215) | > loss_disc_real_4: 0.23472601175308228 (0.24537263398474835) | > loss_disc_real_5: 0.22562748193740845 (0.24686809074371419) | > loss_0: 2.8553507328033447 (2.7227231492387483) | > grad_norm_0: tensor(44.4949, device='cuda:0') (tensor(149.8593, device='cuda:0')) | > loss_gen: 1.8363838195800781 (2.009618982355646) | > loss_kl: 2.103778839111328 (2.052486493232401) | > loss_feat: 3.007784128189087 (3.247541310939383) | > loss_mel: 16.9207763671875 (17.60996157057742) | > loss_duration: 1.6187617778778076 (1.5581338633882238) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.487483978271484 (26.47774217483845) | > grad_norm_1: tensor(2152.6899, device='cuda:0') (tensor(5701.2241, device='cuda:0')) | > current_lr_0: 0.00018344904915530398 | > current_lr_1: 0.00018344904915530398 | > step_time: 0.458 (0.4435890684736536) | > loader_time: 0.0087 (0.0075253324305757565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515300885052748 (-0.00018633251458826175) | > avg_step_time: 0.4556330828599527 (-0.001289186343340798) | > avg_loss_disc: 2.76012655379067 (+0.08443122178735907) | > avg_loss_disc_real_0: 0.20084915645945245 (+0.05790481086767904) | > avg_loss_disc_real_1: 0.23785883754911558 (+0.016464447681332983) | > avg_loss_disc_real_2: 0.2359667724706757 (-0.0031101311176596202) | > avg_loss_disc_real_3: 0.23859276654015124 (-0.0008195208411821142) | > avg_loss_disc_real_4: 0.2439803460114439 (-0.0025095601736659645) | > avg_loss_disc_real_5: 0.24593170448927812 (+0.000974953804217582) | > avg_loss_0: 2.76012655379067 (+0.08443122178735907) | > avg_grad_norm_0: tensor(141.7985, device='cuda:0') (+tensor(20.9413, device='cuda:0')) | > avg_loss_gen: 1.963677779049941 (-0.026928844586224576) | > avg_loss_kl: 2.1413897443825074 (+0.0682595313434855) | > avg_loss_feat: 3.1528127126290766 (-0.12543678955293025) | > avg_loss_mel: 17.881734740566202 (+0.4260785680421719) | > avg_loss_duration: 1.58347281603746 (-0.005554632401802317) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.72308784807232 (+0.3364178832148177) | > avg_grad_norm_1: tensor(5276.1050, device='cuda:0') (+tensor(167.2144, device='cuda:0'))  > EPOCH: 692/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:19:57)   --> TIME: 2025-02-25 15:20:10 -- STEP: 25/72 -- GLOBAL_STEP: 1049850 | > loss_disc: 2.4336042404174805 (2.5821578693389893) | > loss_disc_real_0: 0.06984131038188934 (0.15103382974863053) | > loss_disc_real_1: 0.2098681628704071 (0.23827659606933593) | > loss_disc_real_2: 0.2521570324897766 (0.23164675176143645) | > loss_disc_real_3: 0.23926058411598206 (0.22383111119270324) | > loss_disc_real_4: 0.25785502791404724 (0.23476580679416656) | > loss_disc_real_5: 0.2769457697868347 (0.23117248892784117) | > loss_0: 2.4336042404174805 (2.5821578693389893) | > grad_norm_0: tensor(106.9713, device='cuda:0') (tensor(94.6071, device='cuda:0')) | > loss_gen: 2.315866231918335 (2.066961026191711) | > loss_kl: 2.43986439704895 (2.593099431991577) | > loss_feat: 3.791558027267456 (3.4065088272094726) | > loss_mel: 19.698339462280273 (20.101514129638666) | > loss_duration: 1.565176248550415 (1.588449010848999) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.81080436706543 (29.75653244018555) | > grad_norm_1: tensor(3451.7815, device='cuda:0') (tensor(4522.2075, device='cuda:0')) | > current_lr_0: 0.00018342611802415955 | > current_lr_1: 0.00018342611802415955 | > step_time: 0.4301 (0.43272584915161133) | > loader_time: 0.0079 (0.006785602569580078) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865421496646505 (+0.00013891408141230265) | > avg_step_time: 0.45375969376362546 (-0.001873389096327227) | > avg_loss_disc: 2.6312081175790705 (-0.12891843621159937) | > avg_loss_disc_real_0: 0.15173683876932506 (-0.04911231769012739) | > avg_loss_disc_real_1: 0.22235503352024188 (-0.015503804028873691) | > avg_loss_disc_real_2: 0.23575581274401974 (-0.00021095972665596818) | > avg_loss_disc_real_3: 0.23296550986632494 (-0.0056272566738262975) | > avg_loss_disc_real_4: 0.24296854058621634 (-0.0010118054252275521) | > avg_loss_disc_real_5: 0.2437854786993752 (-0.0021462257899029225) | > avg_loss_0: 2.6312081175790705 (-0.12891843621159937) | > avg_grad_norm_0: tensor(117.0434, device='cuda:0') (tensor(-24.7551, device='cuda:0')) | > avg_loss_gen: 2.0907236673462557 (+0.1270458882963148) | > avg_loss_kl: 2.3352279914936545 (+0.19383824711114706) | > avg_loss_feat: 3.4549616054749825 (+0.30214889284590596) | > avg_loss_mel: 18.806338981843325 (+0.9246042412771232) | > avg_loss_duration: 1.6061501452620601 (+0.022677329224600085) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 28.293402309149084 (+1.5703144610767623) | > avg_grad_norm_1: tensor(4568.0522, device='cuda:0') (tensor(-708.0527, device='cuda:0'))  > EPOCH: 693/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:20:33)   --> TIME: 2025-02-25 15:20:37 -- STEP: 3/72 -- GLOBAL_STEP: 1049900 | > loss_disc: 2.518507719039917 (2.64201553662618) | > loss_disc_real_0: 0.16699188947677612 (0.17053519686063132) | > loss_disc_real_1: 0.18447263538837433 (0.19959167639414468) | > loss_disc_real_2: 0.22775524854660034 (0.24237306912740073) | > loss_disc_real_3: 0.2601432204246521 (0.23759308954079947) | > loss_disc_real_4: 0.24480588734149933 (0.2525517592827479) | > loss_disc_real_5: 0.26294973492622375 (0.25809260706106824) | > loss_0: 2.518507719039917 (2.64201553662618) | > grad_norm_0: tensor(177.1543, device='cuda:0') (tensor(100.7125, device='cuda:0')) | > loss_gen: 2.3112337589263916 (2.046087384223938) | > loss_kl: 2.037142276763916 (2.042871832847595) | > loss_feat: 3.738645553588867 (3.314201752344767) | > loss_mel: 17.58624839782715 (17.718944549560547) | > loss_duration: 1.6070313453674316 (1.5748907725016277) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.28030014038086 (26.696996053059895) | > grad_norm_1: tensor(8264.1152, device='cuda:0') (tensor(5585.5576, device='cuda:0')) | > current_lr_0: 0.00018340318975940652 | > current_lr_1: 0.00018340318975940652 | > step_time: 0.4365 (0.4369329611460368) | > loader_time: 0.0079 (0.0066754817962646484)  --> TIME: 2025-02-25 15:21:00 -- STEP: 53/72 -- GLOBAL_STEP: 1049950 | > loss_disc: 2.578836679458618 (2.6263793234555215) | > loss_disc_real_0: 0.14838111400604248 (0.13719227642945522) | > loss_disc_real_1: 0.21757997572422028 (0.21822245165986834) | > loss_disc_real_2: 0.21285729110240936 (0.23715249969149535) | > loss_disc_real_3: 0.17791202664375305 (0.23254659097149688) | > loss_disc_real_4: 0.20866110920906067 (0.24231106408362119) | > loss_disc_real_5: 0.24069494009017944 (0.24913651566460449) | > loss_0: 2.578836679458618 (2.6263793234555215) | > grad_norm_0: tensor(82.0051, device='cuda:0') (tensor(115.9816, device='cuda:0')) | > loss_gen: 2.3853440284729004 (2.1088883336984883) | > loss_kl: 2.1901562213897705 (2.061300414913106) | > loss_feat: 4.354743480682373 (3.5608229817084545) | > loss_mel: 16.851882934570312 (17.614676781420446) | > loss_duration: 1.597355604171753 (1.568090870695294) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.379480361938477 (26.913779492648143) | > grad_norm_1: tensor(2645.6150, device='cuda:0') (tensor(4946.2661, device='cuda:0')) | > current_lr_0: 0.00018340318975940652 | > current_lr_1: 0.00018340318975940652 | > step_time: 0.4466 (0.44359113135427797) | > loader_time: 0.0094 (0.007807880077721938) --> EVAL PERFORMANCE | > avg_loader_time: 0.008675652490535255 (+2.1437524070204786e-05) | > avg_step_time: 0.45485569725573904 (+0.0010960034921135842) | > avg_loss_disc: 2.632192027400916 (+0.0009839098218455966) | > avg_loss_disc_real_0: 0.13165171481144253 (-0.020085123957882534) | > avg_loss_disc_real_1: 0.21969357657600458 (-0.002661456944237306) | > avg_loss_disc_real_2: 0.2391144979587743 (+0.0033586852147545476) | > avg_loss_disc_real_3: 0.23196915064899015 (-0.0009963592173347913) | > avg_loss_disc_real_4: 0.24105367715090092 (-0.0019148634353154115) | > avg_loss_disc_real_5: 0.2469450831413269 (+0.0031596044419517078) | > avg_loss_0: 2.632192027400916 (+0.0009839098218455966) | > avg_grad_norm_0: tensor(116.7459, device='cuda:0') (tensor(-0.2975, device='cuda:0')) | > avg_loss_gen: 2.087250613830459 (-0.0034730535157967424) | > avg_loss_kl: 2.070500088409639 (-0.2647279030840153) | > avg_loss_feat: 3.540817680493207 (+0.08585607501822468) | > avg_loss_mel: 17.629731567812644 (-1.1766074140306806) | > avg_loss_duration: 1.5816837465259388 (-0.02446639873612133) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.909983702108892 (-1.3834186070401913) | > avg_grad_norm_1: tensor(4866.7329, device='cuda:0') (+tensor(298.6807, device='cuda:0'))  > EPOCH: 694/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:21:10)   --> TIME: 2025-02-25 15:21:26 -- STEP: 31/72 -- GLOBAL_STEP: 1050000 | > loss_disc: 2.6909873485565186 (2.761970966093002) | > loss_disc_real_0: 0.2704606056213379 (0.17744763544009579) | > loss_disc_real_1: 0.3139168322086334 (0.2238217044261194) | > loss_disc_real_2: 0.24911421537399292 (0.22189147530063505) | > loss_disc_real_3: 0.2847142219543457 (0.22988570313299855) | > loss_disc_real_4: 0.2413249909877777 (0.2525886864431443) | > loss_disc_real_5: 0.25619369745254517 (0.25670200345977656) | > loss_0: 2.6909873485565186 (2.761970966093002) | > grad_norm_0: tensor(146.8636, device='cuda:0') (tensor(121.9031, device='cuda:0')) | > loss_gen: 2.3504865169525146 (1.9514063481361634) | > loss_kl: 1.9363207817077637 (1.9714975049418788) | > loss_feat: 3.1717259883880615 (3.2425185172788558) | > loss_mel: 18.247526168823242 (17.53477730289582) | > loss_duration: 1.5683025121688843 (1.5443868137175036) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.27436065673828 (26.2445863908337) | > grad_norm_1: tensor(6169.9004, device='cuda:0') (tensor(5118.9326, device='cuda:0')) | > current_lr_0: 0.00018338026436068658 | > current_lr_1: 0.00018338026436068658 | > step_time: 0.4366 (0.4355907594003985) | > loader_time: 0.0086 (0.007137767730220672) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1050000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008714105041933729 (+3.845255139847334e-05) | > avg_step_time: 0.4538531538466333 (-0.0010025434091057717) | > avg_loss_disc: 2.7374397331560174 (+0.10524770575510134) | > avg_loss_disc_real_0: 0.166302960864465 (+0.03465124605302247) | > avg_loss_disc_real_1: 0.22340079470419547 (+0.0037072181281908867) | > avg_loss_disc_real_2: 0.2497167595675294 (+0.010602261608755104) | > avg_loss_disc_real_3: 0.23122266144819661 (-0.0007464892007935375) | > avg_loss_disc_real_4: 0.24828387125277188 (+0.0072301941018709515) | > avg_loss_disc_real_5: 0.2506962888677356 (+0.0037512057264086907) | > avg_loss_0: 2.7374397331560174 (+0.10524770575510134) | > avg_grad_norm_0: tensor(118.7254, device='cuda:0') (+tensor(1.9795, device='cuda:0')) | > avg_loss_gen: 1.9700415638131155 (-0.1172090500173435) | > avg_loss_kl: 2.0287716472652586 (-0.04172844114438057) | > avg_loss_feat: 3.1963945173881423 (-0.34442316310506493) | > avg_loss_mel: 17.32676096365486 (-0.3029706041577853) | > avg_loss_duration: 1.5812971910960238 (-0.0003865554299149476) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.103265681736904 (-0.8067180203719886) | > avg_grad_norm_1: tensor(5200.7056, device='cuda:0') (+tensor(333.9727, device='cuda:0'))  > EPOCH: 695/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:21:51)   --> TIME: 2025-02-25 15:21:56 -- STEP: 9/72 -- GLOBAL_STEP: 1050050 | > loss_disc: 2.8214335441589355 (2.6365252335866294) | > loss_disc_real_0: 0.09642946720123291 (0.1304228338930342) | > loss_disc_real_1: 0.2129870504140854 (0.21736294031143188) | > loss_disc_real_2: 0.21440443396568298 (0.21749424934387207) | > loss_disc_real_3: 0.22915945947170258 (0.22941590514447954) | > loss_disc_real_4: 0.24061863124370575 (0.2495320720805062) | > loss_disc_real_5: 0.22108589112758636 (0.23620243701669905) | > loss_0: 2.8214335441589355 (2.6365252335866294) | > grad_norm_0: tensor(285.2587, device='cuda:0') (tensor(162.7099, device='cuda:0')) | > loss_gen: 1.7293692827224731 (1.980384972360399) | > loss_kl: 1.921237587928772 (2.1104975938796997) | > loss_feat: 3.5657854080200195 (3.4446493519677057) | > loss_mel: 18.876131057739258 (18.296527014838325) | > loss_duration: 1.556921124458313 (1.5710871087180243) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.649444580078125 (27.403146531846787) | > grad_norm_1: tensor(5948.0459, device='cuda:0') (tensor(6812.3569, device='cuda:0')) | > current_lr_0: 0.00018335734182764148 | > current_lr_1: 0.00018335734182764148 | > step_time: 0.4262 (0.42713798416985405) | > loader_time: 0.0056 (0.0067132314046223955)  --> TIME: 2025-02-25 15:22:19 -- STEP: 59/72 -- GLOBAL_STEP: 1050100 | > loss_disc: 2.7019190788269043 (2.6243887594190696) | > loss_disc_real_0: 0.11907508969306946 (0.1262651829017421) | > loss_disc_real_1: 0.20986618101596832 (0.22507927680419662) | > loss_disc_real_2: 0.2249101847410202 (0.2336949488874209) | > loss_disc_real_3: 0.21679551899433136 (0.2323128250190767) | > loss_disc_real_4: 0.23839740455150604 (0.24318427579887844) | > loss_disc_real_5: 0.2501213252544403 (0.24409872040910235) | > loss_0: 2.7019190788269043 (2.6243887594190696) | > grad_norm_0: tensor(266.2346, device='cuda:0') (tensor(188.6078, device='cuda:0')) | > loss_gen: 1.8721036911010742 (2.007792406162973) | > loss_kl: 2.123994827270508 (2.0366687027074524) | > loss_feat: 3.4851016998291016 (3.439328011819872) | > loss_mel: 15.620856285095215 (17.292492801860224) | > loss_duration: 1.624468207359314 (1.5676247305789237) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.726526260375977 (26.343906693539378) | > grad_norm_1: tensor(7896.1313, device='cuda:0') (tensor(7665.2847, device='cuda:0')) | > current_lr_0: 0.00018335734182764148 | > current_lr_1: 0.00018335734182764148 | > step_time: 0.466 (0.44254444009166655) | > loader_time: 0.0096 (0.008016016523716816) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857070801963269 (-0.00014339702230103946) | > avg_step_time: 0.4525373317825962 (-0.0013158220640370755) | > avg_loss_disc: 2.6241765458818893 (-0.1132631872741281) | > avg_loss_disc_real_0: 0.12458464770879542 (-0.04171831315566958) | > avg_loss_disc_real_1: 0.22686844602437087 (+0.0034676513201754022) | > avg_loss_disc_real_2: 0.23332293016809813 (-0.016393829399431264) | > avg_loss_disc_real_3: 0.23193843809651657 (+0.0007157766483199546) | > avg_loss_disc_real_4: 0.24206304760046407 (-0.006220823652307805) | > avg_loss_disc_real_5: 0.24447421666602015 (-0.006222072201715445) | > avg_loss_0: 2.6241765458818893 (-0.1132631872741281) | > avg_grad_norm_0: tensor(188.0386, device='cuda:0') (+tensor(69.3132, device='cuda:0')) | > avg_loss_gen: 2.0046216719587076 (+0.03458010814559209) | > avg_loss_kl: 2.0535229192653173 (+0.024751272000058666) | > avg_loss_feat: 3.418152587514528 (+0.22175807012638593) | > avg_loss_mel: 17.255888374758435 (-0.07087258889642456) | > avg_loss_duration: 1.5768263877277644 (-0.004470803368259446) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.30901199663189 (+0.20574631489498785) | > avg_grad_norm_1: tensor(7803.7993, device='cuda:0') (+tensor(2603.0938, device='cuda:0'))  > EPOCH: 696/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:22:27)   --> TIME: 2025-02-25 15:22:46 -- STEP: 37/72 -- GLOBAL_STEP: 1050150 | > loss_disc: 2.596968412399292 (2.564052008293771) | > loss_disc_real_0: 0.1368485987186432 (0.11801712607612481) | > loss_disc_real_1: 0.22090929746627808 (0.21653358396646138) | > loss_disc_real_2: 0.2583707869052887 (0.22842986962279757) | > loss_disc_real_3: 0.23918570578098297 (0.23163842752173142) | > loss_disc_real_4: 0.25464680790901184 (0.2395776820343894) | > loss_disc_real_5: 0.2825133800506592 (0.2441298997885472) | > loss_0: 2.596968412399292 (2.564052008293771) | > grad_norm_0: tensor(128.5254, device='cuda:0') (tensor(201.9560, device='cuda:0')) | > loss_gen: 2.167140245437622 (2.0901290661579845) | > loss_kl: 1.8683308362960815 (1.936481069874119) | > loss_feat: 2.8492982387542725 (3.573652261012309) | > loss_mel: 16.721487045288086 (17.15866743551718) | > loss_duration: 1.5584033727645874 (1.5375267106133539) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.16465950012207 (26.29645656894993) | > grad_norm_1: tensor(7626.4741, device='cuda:0') (tensor(8933.8018, device='cuda:0')) | > current_lr_0: 0.00018333442215991301 | > current_lr_1: 0.00018333442215991301 | > step_time: 0.4491 (0.4380860715299039) | > loader_time: 0.0081 (0.00747973210102803) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874224515028403 (+0.00017153713065133983) | > avg_step_time: 0.4574982448362968 (+0.004960913053700611) | > avg_loss_disc: 2.577547392375033 (-0.04662915350685637) | > avg_loss_disc_real_0: 0.1181638773373315 (-0.006420770371463921) | > avg_loss_disc_real_1: 0.22000880665342573 (-0.006859639370945136) | > avg_loss_disc_real_2: 0.23108184043790253 (-0.0022410897301956) | > avg_loss_disc_real_3: 0.23099562273898594 (-0.0009428153575306297) | > avg_loss_disc_real_4: 0.24128608132751894 (-0.0007769662729451288) | > avg_loss_disc_real_5: 0.24523437715752025 (+0.0007601604915001003) | > avg_loss_0: 2.577547392375033 (-0.04662915350685637) | > avg_grad_norm_0: tensor(203.7900, device='cuda:0') (+tensor(15.7514, device='cuda:0')) | > avg_loss_gen: 2.084053768238552 (+0.07943209627984427) | > avg_loss_kl: 1.9877294335566775 (-0.06579348570863974) | > avg_loss_feat: 3.5442713079318193 (+0.1261187204172911) | > avg_loss_mel: 17.049436394597443 (-0.20645198016099187) | > avg_loss_duration: 1.5686101779131822 (-0.008216209814582154) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.23410114771883 (-0.07491084891305988) | > avg_grad_norm_1: tensor(9081.3516, device='cuda:0') (+tensor(1277.5522, device='cuda:0'))  > EPOCH: 697/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:23:05)   --> TIME: 2025-02-25 15:23:13 -- STEP: 15/72 -- GLOBAL_STEP: 1050200 | > loss_disc: 2.559436082839966 (2.562757619222005) | > loss_disc_real_0: 0.1120762974023819 (0.11438676764567693) | > loss_disc_real_1: 0.20038945972919464 (0.22007982631524403) | > loss_disc_real_2: 0.23284660279750824 (0.2316589593887329) | > loss_disc_real_3: 0.23129519820213318 (0.23616548577944438) | > loss_disc_real_4: 0.240644171833992 (0.24095888038476307) | > loss_disc_real_5: 0.24501244723796844 (0.23899919986724855) | > loss_0: 2.559436082839966 (2.562757619222005) | > grad_norm_0: tensor(298.0378, device='cuda:0') (tensor(227.1169, device='cuda:0')) | > loss_gen: 1.954789400100708 (2.1268706639607746) | > loss_kl: 1.9036133289337158 (1.878865694999695) | > loss_feat: 3.7683167457580566 (3.6817123254140216) | > loss_mel: 17.788660049438477 (17.146935145060223) | > loss_duration: 1.5213704109191895 (1.523271663983663) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.936750411987305 (26.357655334472657) | > grad_norm_1: tensor(10712.6953, device='cuda:0') (tensor(10167.4336, device='cuda:0')) | > current_lr_0: 0.000183311505357143 | > current_lr_1: 0.000183311505357143 | > step_time: 0.4387 (0.43788112004597984) | > loader_time: 0.0076 (0.0067145029703776045)  --> TIME: 2025-02-25 15:23:37 -- STEP: 65/72 -- GLOBAL_STEP: 1050250 | > loss_disc: 2.625528573989868 (2.57567519408006) | > loss_disc_real_0: 0.11102110147476196 (0.11811385383972754) | > loss_disc_real_1: 0.22434380650520325 (0.2201070737380248) | > loss_disc_real_2: 0.21945369243621826 (0.23064619577847995) | > loss_disc_real_3: 0.22377657890319824 (0.23297908718769367) | > loss_disc_real_4: 0.22858692705631256 (0.24104350346785325) | > loss_disc_real_5: 0.22246238589286804 (0.2446110175206111) | > loss_0: 2.625528573989868 (2.57567519408006) | > grad_norm_0: tensor(234.4040, device='cuda:0') (tensor(220.7628, device='cuda:0')) | > loss_gen: 1.9340591430664062 (2.125046135829045) | > loss_kl: 2.200601577758789 (1.9724646880076482) | > loss_feat: 3.311922550201416 (3.6704548799074614) | > loss_mel: 17.430213928222656 (17.024684891333955) | > loss_duration: 1.6273014545440674 (1.5647038734876197) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.504098892211914 (26.3573544722337) | > grad_norm_1: tensor(10065.1475, device='cuda:0') (tensor(10249.4736, device='cuda:0')) | > current_lr_0: 0.000183311505357143 | > current_lr_1: 0.000183311505357143 | > step_time: 0.5 (0.4524233818054199) | > loader_time: 0.0118 (0.008259846613957332) --> EVAL PERFORMANCE | > avg_loader_time: 0.008554263853690994 (-0.0001879812965930351) | > avg_step_time: 0.45884953418248137 (+0.0013512893461845588) | > avg_loss_disc: 2.581130635570473 (+0.0035832431954401223) | > avg_loss_disc_real_0: 0.11744311179073763 (-0.0007207655465938656) | > avg_loss_disc_real_1: 0.22062559551756147 (+0.0006167888641357422) | > avg_loss_disc_real_2: 0.23009946669491244 (-0.0009823737429900947) | > avg_loss_disc_real_3: 0.2332783536592 (+0.0022827309202140544) | > avg_loss_disc_real_4: 0.24131700228637373 (+3.092095885479096e-05) | > avg_loss_disc_real_5: 0.24438265973413495 (-0.0008517174233853009) | > avg_loss_0: 2.581130635570473 (+0.0035832431954401223) | > avg_grad_norm_0: tensor(219.9969, device='cuda:0') (+tensor(16.2069, device='cuda:0')) | > avg_loss_gen: 2.113399688626679 (+0.029345920388127222) | > avg_loss_kl: 1.9842184026476364 (-0.0035110309090411462) | > avg_loss_feat: 3.636229410977431 (+0.09195810304561158) | > avg_loss_mel: 17.01026308032829 (-0.039173314269152826) | > avg_loss_duration: 1.570200706871463 (+0.0015905289582807125) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.31431130959954 (+0.08021016188070718) | > avg_grad_norm_1: tensor(10213.3535, device='cuda:0') (+tensor(1132.0020, device='cuda:0'))  > EPOCH: 698/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:23:42)   --> TIME: 2025-02-25 15:24:04 -- STEP: 43/72 -- GLOBAL_STEP: 1050300 | > loss_disc: 2.70104718208313 (2.611520340276319) | > loss_disc_real_0: 0.10197271406650543 (0.11604736624068993) | > loss_disc_real_1: 0.27595746517181396 (0.22256572405959285) | > loss_disc_real_2: 0.2289489209651947 (0.23554205790508626) | > loss_disc_real_3: 0.2706010341644287 (0.23134199304636135) | > loss_disc_real_4: 0.24705299735069275 (0.24168344639068426) | > loss_disc_real_5: 0.24165473878383636 (0.24234137216279672) | > loss_0: 2.70104718208313 (2.611520340276319) | > grad_norm_0: tensor(345.1735, device='cuda:0') (tensor(214.9287, device='cuda:0')) | > loss_gen: 1.8764269351959229 (2.0871379347734673) | > loss_kl: 1.9373562335968018 (2.01424300116162) | > loss_feat: 3.136486291885376 (3.68984933786614) | > loss_mel: 16.356990814208984 (16.951041044190873) | > loss_duration: 1.5954251289367676 (1.5407652549965436) | > amp_scaler: 8.0 (11.348837209302326) | > loss_1: 24.902685165405273 (26.283036875170332) | > grad_norm_1: tensor(8471.5742, device='cuda:0') (tensor(9950.9668, device='cuda:0')) | > current_lr_0: 0.00018328859141897335 | > current_lr_1: 0.00018328859141897335 | > step_time: 0.4619 (0.441909252211105) | > loader_time: 0.0095 (0.007556388544481854) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864207912498797 (+8.781527129697597e-05) | > avg_step_time: 0.4568214651564477 (-0.002028069026033641) | > avg_loss_disc: 2.604109142867612 (+0.022978507297139128) | > avg_loss_disc_real_0: 0.11593871941448937 (-0.001504392376248262) | > avg_loss_disc_real_1: 0.2243172394137987 (+0.0036916438962372178) | > avg_loss_disc_real_2: 0.23444779616006664 (+0.004348329465154205) | > avg_loss_disc_real_3: 0.2357375034144227 (+0.0024591497552227115) | > avg_loss_disc_real_4: 0.24140380700709116 (+8.680472071742606e-05) | > avg_loss_disc_real_5: 0.24327293647007203 (-0.0011097232640629173) | > avg_loss_0: 2.604109142867612 (+0.022978507297139128) | > avg_grad_norm_0: tensor(211.9660, device='cuda:0') (tensor(-8.0310, device='cuda:0')) | > avg_loss_gen: 2.097300460640813 (-0.016099227985865827) | > avg_loss_kl: 2.036374555507176 (+0.052156152859539784) | > avg_loss_feat: 3.629083754311145 (-0.007145656666285749) | > avg_loss_mel: 17.01097389006279 (+0.0007108097344996622) | > avg_loss_duration: 1.5660011365380087 (-0.0041995703334543055) | > avg_amp_scaler: 10.028169014084506 (-5.971830985915494) | > avg_loss_1: 26.339733875973124 (+0.025422566373585198) | > avg_grad_norm_1: tensor(9905.6895, device='cuda:0') (tensor(-307.6641, device='cuda:0'))  > EPOCH: 699/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:24:19)   --> TIME: 2025-02-25 15:24:30 -- STEP: 21/72 -- GLOBAL_STEP: 1050350 | > loss_disc: 2.934398651123047 (2.620968466713315) | > loss_disc_real_0: 0.05142799764871597 (0.10941351417984281) | > loss_disc_real_1: 0.22938621044158936 (0.22569437892664046) | > loss_disc_real_2: 0.2098822146654129 (0.22732028223219372) | > loss_disc_real_3: 0.2118842452764511 (0.2309209569579079) | > loss_disc_real_4: 0.23606757819652557 (0.24096966641289846) | > loss_disc_real_5: 0.25558406114578247 (0.2480684234982445) | > loss_0: 2.934398651123047 (2.620968466713315) | > grad_norm_0: tensor(340.1528, device='cuda:0') (tensor(230.1612, device='cuda:0')) | > loss_gen: 1.753274917602539 (2.050793085779462) | > loss_kl: 1.94084894657135 (1.9735160384859358) | > loss_feat: 3.431758165359497 (3.5816584201086137) | > loss_mel: 17.01984405517578 (16.85632387797038) | > loss_duration: 1.5429067611694336 (1.5266985098520915) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.68863296508789 (25.988989966256277) | > grad_norm_1: tensor(11784.8047, device='cuda:0') (tensor(10460.0791, device='cuda:0')) | > current_lr_0: 0.00018326568034504598 | > current_lr_1: 0.00018326568034504598 | > step_time: 0.44 (0.4363302162715367) | > loader_time: 0.008 (0.006854057312011719)  --> TIME: 2025-02-25 15:24:55 -- STEP: 71/72 -- GLOBAL_STEP: 1050400 | > loss_disc: 2.558976650238037 (2.607457332208123) | > loss_disc_real_0: 0.17263205349445343 (0.11398079199060587) | > loss_disc_real_1: 0.20493033528327942 (0.22387601204321417) | > loss_disc_real_2: 0.23783060908317566 (0.23182812291131893) | > loss_disc_real_3: 0.2524131238460541 (0.23391328122414334) | > loss_disc_real_4: 0.23054851591587067 (0.24255554881733907) | > loss_disc_real_5: 0.2494039088487625 (0.24746002886496798) | > loss_0: 2.558976650238037 (2.607457332208123) | > grad_norm_0: tensor(270.9716, device='cuda:0') (tensor(232.0202, device='cuda:0')) | > loss_gen: 2.256471872329712 (2.092445788249163) | > loss_kl: 1.9657769203186035 (2.048125424855192) | > loss_feat: 3.4622347354888916 (3.603109376531252) | > loss_mel: 17.049848556518555 (16.95719545659885) | > loss_duration: 1.5653098821640015 (1.5671971650190757) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.299640655517578 (26.26807320285851) | > grad_norm_1: tensor(8002.3828, device='cuda:0') (tensor(10445.6572, device='cuda:0')) | > current_lr_0: 0.00018326568034504598 | > current_lr_1: 0.00018326568034504598 | > step_time: 0.4815 (0.4581694502226064) | > loader_time: 0.0081 (0.008680249603701309) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680249603701309 (+3.817047871333863e-05) | > avg_step_time: 0.4581694502226064 (+0.0013479850661586856) | > avg_loss_disc: 2.607457332208123 (+0.003348189340510732) | > avg_loss_disc_real_0: 0.11398079199060587 (-0.001957927423883496) | > avg_loss_disc_real_1: 0.22387601204321417 (-0.00044122737058452377) | > avg_loss_disc_real_2: 0.23182812291131893 (-0.002619673248747706) | > avg_loss_disc_real_3: 0.23391328122414334 (-0.001824222190279362) | > avg_loss_disc_real_4: 0.24255554881733907 (+0.0011517418102479082) | > avg_loss_disc_real_5: 0.24746002886496798 (+0.004187092394895953) | > avg_loss_0: 2.607457332208123 (+0.003348189340510732) | > avg_grad_norm_0: tensor(232.0202, device='cuda:0') (+tensor(20.0542, device='cuda:0')) | > avg_loss_gen: 2.092445788249163 (-0.004854672391650272) | > avg_loss_kl: 2.048125424855192 (+0.011750869348015947) | > avg_loss_feat: 3.603109376531252 (-0.02597437777989331) | > avg_loss_mel: 16.95719545659885 (-0.053778433463939734) | > avg_loss_duration: 1.5671971650190757 (+0.0011960284810670352) | > avg_amp_scaler: 8.0 (-2.028169014084506) | > avg_loss_1: 26.26807320285851 (-0.07166067311461433) | > avg_grad_norm_1: tensor(10445.6572, device='cuda:0') (+tensor(539.9678, device='cuda:0'))  > EPOCH: 700/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:24:56)   --> TIME: 2025-02-25 15:25:20 -- STEP: 49/72 -- GLOBAL_STEP: 1050450 | > loss_disc: 2.6237361431121826 (2.5979932279002904) | > loss_disc_real_0: 0.051492270082235336 (0.10359175944206667) | > loss_disc_real_1: 0.2169896364212036 (0.22787697552418223) | > loss_disc_real_2: 0.23732653260231018 (0.23071555519590572) | > loss_disc_real_3: 0.22771720588207245 (0.23585610547844243) | > loss_disc_real_4: 0.27527129650115967 (0.24340028513451012) | > loss_disc_real_5: 0.2477986067533493 (0.24437025949663047) | > loss_0: 2.6237361431121826 (2.5979932279002904) | > grad_norm_0: tensor(190.2063, device='cuda:0') (tensor(233.0822, device='cuda:0')) | > loss_gen: 1.8897380828857422 (2.1049292379496056) | > loss_kl: 1.9806638956069946 (2.023439974200969) | > loss_feat: 3.61771559715271 (3.679434703320873) | > loss_mel: 17.370166778564453 (16.94826637968725) | > loss_duration: 1.6410285234451294 (1.5474785079761428) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.499313354492188 (26.303548890717174) | > grad_norm_1: tensor(9093.6406, device='cuda:0') (tensor(10627.4160, device='cuda:0')) | > current_lr_0: 0.00018324277213500285 | > current_lr_1: 0.00018324277213500285 | > step_time: 0.4433 (0.4432332418402847) | > loader_time: 0.0099 (0.007732289178030831) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679154893042334 (-1.094710658974471e-06) | > avg_step_time: 0.4543650183879154 (-0.0038044318346910355) | > avg_loss_disc: 2.6187217806426566 (+0.01126444843453367) | > avg_loss_disc_real_0: 0.10990502811233763 (-0.004075763878268246) | > avg_loss_disc_real_1: 0.23478977445145727 (+0.0109137624082431) | > avg_loss_disc_real_2: 0.23143908129611485 (-0.00038904161520408875) | > avg_loss_disc_real_3: 0.233115338943374 (-0.0007979422807693481) | > avg_loss_disc_real_4: 0.24558464364266733 (+0.00302909482532826) | > avg_loss_disc_real_5: 0.24779472426629404 (+0.0003346954013260517) | > avg_loss_0: 2.6187217806426566 (+0.01126444843453367) | > avg_grad_norm_0: tensor(239.6295, device='cuda:0') (+tensor(7.6093, device='cuda:0')) | > avg_loss_gen: 2.107226662232843 (+0.01478087398368011) | > avg_loss_kl: 2.0526457168686556 (+0.004520292013463489) | > avg_loss_feat: 3.6455130207706503 (+0.042403644239398464) | > avg_loss_mel: 16.970215475055532 (+0.013020018456682436) | > avg_loss_duration: 1.5667758407727095 (-0.0004213242463662059) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.342376762712505 (+0.07430355985399473) | > avg_grad_norm_1: tensor(10641.2158, device='cuda:0') (+tensor(195.5586, device='cuda:0'))  > EPOCH: 701/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:25:33)   --> TIME: 2025-02-25 15:25:48 -- STEP: 27/72 -- GLOBAL_STEP: 1050500 | > loss_disc: 2.7092721462249756 (2.624497652053833) | > loss_disc_real_0: 0.10706210136413574 (0.10794232468362208) | > loss_disc_real_1: 0.2347397655248642 (0.2244773926558318) | > loss_disc_real_2: 0.20279350876808167 (0.22826432353920406) | > loss_disc_real_3: 0.22957845032215118 (0.2317143557248292) | > loss_disc_real_4: 0.23109999299049377 (0.24042753875255585) | > loss_disc_real_5: 0.23521247506141663 (0.23895242258354468) | > loss_0: 2.7092721462249756 (2.624497652053833) | > grad_norm_0: tensor(259.7355, device='cuda:0') (tensor(227.7111, device='cuda:0')) | > loss_gen: 1.970760703086853 (2.0643687822200634) | > loss_kl: 2.0590217113494873 (1.950482929194415) | > loss_feat: 3.3742644786834717 (3.642259377020377) | > loss_mel: 16.575231552124023 (17.209007086577238) | > loss_duration: 1.5289115905761719 (1.5262081446471039) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.508190155029297 (26.392326072410302) | > grad_norm_1: tensor(9290.7207, device='cuda:0') (tensor(9829.8379, device='cuda:0')) | > current_lr_0: 0.00018321986678848597 | > current_lr_1: 0.00018321986678848597 | > step_time: 0.4531 (0.44165764031586824) | > loader_time: 0.0073 (0.007093464886700666) --> EVAL PERFORMANCE | > avg_loader_time: 0.008674067510685451 (-5.087382356883108e-06) | > avg_step_time: 0.4591095279639875 (+0.004744509576072142) | > avg_loss_disc: 2.6473093032836914 (+0.028587522641034813) | > avg_loss_disc_real_0: 0.1162006830350614 (+0.00629565492272377) | > avg_loss_disc_real_1: 0.22244596565273447 (-0.012343808798722794) | > avg_loss_disc_real_2: 0.23896043590256866 (+0.007521354606453812) | > avg_loss_disc_real_3: 0.2354971321535782 (+0.0023817932102042083) | > avg_loss_disc_real_4: 0.2418832627820297 (-0.003701380860637621) | > avg_loss_disc_real_5: 0.2446275687133762 (-0.0031671555529178275) | > avg_loss_0: 2.6473093032836914 (+0.028587522641034813) | > avg_grad_norm_0: tensor(238.3227, device='cuda:0') (tensor(-1.3067, device='cuda:0')) | > avg_loss_gen: 2.0761371263315973 (-0.031089535901245746) | > avg_loss_kl: 2.0389358258583186 (-0.013709891010337039) | > avg_loss_feat: 3.611296704117681 (-0.034216316652969425) | > avg_loss_mel: 17.068503151477216 (+0.09828767642168401) | > avg_loss_duration: 1.5682442994184898 (+0.0014684586457802684) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.363116895648794 (+0.020740132936289513) | > avg_grad_norm_1: tensor(10482.7041, device='cuda:0') (tensor(-158.5117, device='cuda:0'))  > EPOCH: 702/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:26:11)   --> TIME: 2025-02-25 15:26:15 -- STEP: 5/72 -- GLOBAL_STEP: 1050550 | > loss_disc: 2.5346884727478027 (2.6827865123748778) | > loss_disc_real_0: 0.1347476840019226 (0.0802107498049736) | > loss_disc_real_1: 0.2276017814874649 (0.27005264461040496) | > loss_disc_real_2: 0.2050522118806839 (0.30128353536129) | > loss_disc_real_3: 0.24210794270038605 (0.29257037341594694) | > loss_disc_real_4: 0.27242061495780945 (0.27074178457260134) | > loss_disc_real_5: 0.2382756769657135 (0.22738749980926515) | > loss_0: 2.5346884727478027 (2.6827865123748778) | > grad_norm_0: tensor(236.3914, device='cuda:0') (tensor(203.4515, device='cuda:0')) | > loss_gen: 2.194718599319458 (2.1224601030349732) | > loss_kl: 2.088221788406372 (1.9530408143997193) | > loss_feat: 3.6710500717163086 (3.772445869445801) | > loss_mel: 16.85232162475586 (17.249912643432616) | > loss_duration: 1.4983093738555908 (1.5313785076141357) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.304622650146484 (26.629237747192384) | > grad_norm_1: tensor(11034.8350, device='cuda:0') (tensor(10014.1074, device='cuda:0')) | > current_lr_0: 0.0001831969643051374 | > current_lr_1: 0.0001831969643051374 | > step_time: 0.4348 (0.43722057342529297) | > loader_time: 0.0064 (0.0073814868927001955)  --> TIME: 2025-02-25 15:26:38 -- STEP: 55/72 -- GLOBAL_STEP: 1050600 | > loss_disc: 2.880495548248291 (2.707015952196988) | > loss_disc_real_0: 0.14646300673484802 (0.15153736756606537) | > loss_disc_real_1: 0.1654656082391739 (0.25019321726127103) | > loss_disc_real_2: 0.20008008182048798 (0.22945524833419106) | > loss_disc_real_3: 0.31821420788764954 (0.23888351321220397) | > loss_disc_real_4: 0.30074983835220337 (0.24184830486774445) | > loss_disc_real_5: 0.23857788741588593 (0.23857621604746038) | > loss_0: 2.880495548248291 (2.707015952196988) | > grad_norm_0: tensor(161.0792, device='cuda:0') (tensor(226.3016, device='cuda:0')) | > loss_gen: 1.9120030403137207 (2.0756554256785993) | > loss_kl: 2.2182750701904297 (2.0860521489923656) | > loss_feat: 2.8864245414733887 (3.501805712959983) | > loss_mel: 16.37481689453125 (17.14538286382501) | > loss_duration: 1.622178077697754 (1.557348038933494) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.01369857788086 (26.36624412536621) | > grad_norm_1: tensor(9644.2451, device='cuda:0') (tensor(9467.8643, device='cuda:0')) | > current_lr_0: 0.0001831969643051374 | > current_lr_1: 0.0001831969643051374 | > step_time: 0.4747 (0.4468444520776922) | > loader_time: 0.0094 (0.008094176379117097) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754256745459325 (+8.018923477387356e-05) | > avg_step_time: 0.45743268644306023 (-0.0016768415209272858) | > avg_loss_disc: 2.725064932460516 (+0.07775562917682466) | > avg_loss_disc_real_0: 0.16147989276009547 (+0.045279209725034075) | > avg_loss_disc_real_1: 0.2512956472559714 (+0.02884968160323695) | > avg_loss_disc_real_2: 0.2307341992855072 (-0.008226236617061455) | > avg_loss_disc_real_3: 0.23979811454322977 (+0.00430098238965157) | > avg_loss_disc_real_4: 0.24068932361166243 (-0.0011939391703672808) | > avg_loss_disc_real_5: 0.24004469539078188 (-0.0045828733225943274) | > avg_loss_0: 2.725064932460516 (+0.07775562917682466) | > avg_grad_norm_0: tensor(210.5362, device='cuda:0') (tensor(-27.7865, device='cuda:0')) | > avg_loss_gen: 2.0587735276826664 (-0.01736359864893089) | > avg_loss_kl: 2.1480303079309606 (+0.10909448207264205) | > avg_loss_feat: 3.4011826683098163 (-0.21011403580786459) | > avg_loss_mel: 17.21936354838626 (+0.1508603969090423) | > avg_loss_duration: 1.5744083532145325 (+0.00616405379604279) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.401758435746313 (+0.038641540097518856) | > avg_grad_norm_1: tensor(8669.3066, device='cuda:0') (tensor(-1813.3975, device='cuda:0'))  > EPOCH: 703/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:26:48)   --> TIME: 2025-02-25 15:27:05 -- STEP: 33/72 -- GLOBAL_STEP: 1050650 | > loss_disc: 2.553410768508911 (2.6560203234354653) | > loss_disc_real_0: 0.09979251772165298 (0.12062905807838295) | > loss_disc_real_1: 0.2376706600189209 (0.2197302444414659) | > loss_disc_real_2: 0.23122353851795197 (0.23721465468406677) | > loss_disc_real_3: 0.23872791230678558 (0.24355243733434967) | > loss_disc_real_4: 0.23582656681537628 (0.2499540172743075) | > loss_disc_real_5: 0.2503417134284973 (0.24927465256416437) | > loss_0: 2.553410768508911 (2.6560203234354653) | > grad_norm_0: tensor(29.0852, device='cuda:0') (tensor(201.2222, device='cuda:0')) | > loss_gen: 2.207592248916626 (1.9719505960291082) | > loss_kl: 2.089503765106201 (2.1218187303254097) | > loss_feat: 3.888852596282959 (3.3687887625260786) | > loss_mel: 16.701953887939453 (17.347862590443004) | > loss_duration: 1.5538487434387207 (1.5392738616827764) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.441749572753906 (26.349694165316496) | > grad_norm_1: tensor(7955.4487, device='cuda:0') (tensor(8121.6753, device='cuda:0')) | > current_lr_0: 0.00018317406468459923 | > current_lr_1: 0.00018317406468459923 | > step_time: 0.4612 (0.43626267259771173) | > loader_time: 0.0079 (0.0073176875258937025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710525405239053 (-4.373134022027207e-05) | > avg_step_time: 0.45462943802417166 (-0.002803248418888571) | > avg_loss_disc: 2.6913869918232227 (-0.033677940637293347) | > avg_loss_disc_real_0: 0.13057596425355325 (-0.030903928506542222) | > avg_loss_disc_real_1: 0.22729434324821957 (-0.024001304007751856) | > avg_loss_disc_real_2: 0.2426671115025668 (+0.01193291221705961) | > avg_loss_disc_real_3: 0.2463008123384395 (+0.006502697795209733) | > avg_loss_disc_real_4: 0.24780118087647665 (+0.007111857264814225) | > avg_loss_disc_real_5: 0.2465859208728226 (+0.0065412254820407245) | > avg_loss_0: 2.6913869918232227 (-0.033677940637293347) | > avg_grad_norm_0: tensor(191.4551, device='cuda:0') (tensor(-19.0811, device='cuda:0')) | > avg_loss_gen: 2.0416804921459146 (-0.01709303553675179) | > avg_loss_kl: 2.364241343148997 (+0.21621103521803642) | > avg_loss_feat: 3.434381777132061 (+0.03319910882224475) | > avg_loss_mel: 18.094651746078277 (+0.8752881976920186) | > avg_loss_duration: 1.606845864107911 (+0.03243751089337854) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.5418010765398 (+1.1400426407934887) | > avg_grad_norm_1: tensor(7056.8120, device='cuda:0') (tensor(-1612.4946, device='cuda:0'))  > EPOCH: 704/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:27:25)   --> TIME: 2025-02-25 15:27:32 -- STEP: 11/72 -- GLOBAL_STEP: 1050700 | > loss_disc: 2.8464417457580566 (2.9534379785711113) | > loss_disc_real_0: 0.3191336393356323 (0.11960170858285644) | > loss_disc_real_1: 0.20583118498325348 (0.25325797362761066) | > loss_disc_real_2: 0.15478280186653137 (0.3366965123198249) | > loss_disc_real_3: 0.21787960827350616 (0.26802645352753723) | > loss_disc_real_4: 0.26104336977005005 (0.2875457460230047) | > loss_disc_real_5: 0.2739964425563812 (0.2553691823374141) | > loss_0: 2.8464417457580566 (2.9534379785711113) | > grad_norm_0: tensor(77.6849, device='cuda:0') (tensor(147.7991, device='cuda:0')) | > loss_gen: 1.8415292501449585 (1.9385550022125244) | > loss_kl: 2.1519217491149902 (2.266057014465332) | > loss_feat: 2.821145534515381 (3.1444975679570977) | > loss_mel: 17.711233139038086 (18.428828672929242) | > loss_duration: 1.5600804090499878 (1.5786028666929766) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.085908889770508 (27.356541200117633) | > grad_norm_1: tensor(598.3356, device='cuda:0') (tensor(4437.3345, device='cuda:0')) | > current_lr_0: 0.00018315116792651365 | > current_lr_1: 0.00018315116792651365 | > step_time: 0.4191 (0.4371540979905562) | > loader_time: 0.006 (0.0064990953965620565)  --> TIME: 2025-02-25 15:27:55 -- STEP: 61/72 -- GLOBAL_STEP: 1050750 | > loss_disc: 2.571382761001587 (2.7726430541179226) | > loss_disc_real_0: 0.07206757366657257 (0.18993974092309593) | > loss_disc_real_1: 0.20904645323753357 (0.25279939956352376) | > loss_disc_real_2: 0.22658364474773407 (0.24349406581433086) | > loss_disc_real_3: 0.20597048103809357 (0.23631071531381764) | > loss_disc_real_4: 0.21781447529792786 (0.24629588376303185) | > loss_disc_real_5: 0.2345399558544159 (0.24670497218116386) | > loss_0: 2.571382761001587 (2.7726430541179226) | > grad_norm_0: tensor(119.2813, device='cuda:0') (tensor(127.0707, device='cuda:0')) | > loss_gen: 1.9044445753097534 (2.093710588627176) | > loss_kl: 2.5145492553710938 (2.4265242013774935) | > loss_feat: 3.34342885017395 (3.2998547045910946) | > loss_mel: 17.462961196899414 (18.694350899242963) | > loss_duration: 1.6383476257324219 (1.6208040010733682) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.86372947692871 (28.135244213166786) | > grad_norm_1: tensor(7155.8232, device='cuda:0') (tensor(4712.6182, device='cuda:0')) | > current_lr_0: 0.00018315116792651365 | > current_lr_1: 0.00018315116792651365 | > step_time: 0.5035 (0.4493817501380795) | > loader_time: 0.0107 (0.008351314263265644) --> EVAL PERFORMANCE | > avg_loader_time: 0.008781379377338252 (+7.085397209919894e-05) | > avg_step_time: 0.4647138555284957 (+0.01008441750432404) | > avg_loss_disc: 2.7728447208941827 (+0.08145772907095994) | > avg_loss_disc_real_0: 0.17421978549428388 (+0.04364382124073063) | > avg_loss_disc_real_1: 0.246625151012985 (+0.019330807764765423) | > avg_loss_disc_real_2: 0.24144932082001588 (-0.0012177906825509288) | > avg_loss_disc_real_3: 0.23095729086600558 (-0.015343521472433924) | > avg_loss_disc_real_4: 0.24355687117072897 (-0.004244309705747684) | > avg_loss_disc_real_5: 0.24315078535550078 (-0.00343513551732183) | > avg_loss_0: 2.7728447208941827 (+0.08145772907095994) | > avg_grad_norm_0: tensor(136.2948, device='cuda:0') (tensor(-55.1604, device='cuda:0')) | > avg_loss_gen: 2.0311070798148587 (-0.01057341233105591) | > avg_loss_kl: 2.4164393183211206 (+0.052197975172123545) | > avg_loss_feat: 3.2728241497362163 (-0.16155762739584478) | > avg_loss_mel: 18.609449588077165 (+0.5147978419988881) | > avg_loss_duration: 1.6245049657955977 (+0.017659101687686585) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.95432493720256 (+0.41252386066275903) | > avg_grad_norm_1: tensor(4870.7383, device='cuda:0') (tensor(-2186.0737, device='cuda:0'))  > EPOCH: 705/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:28:03)   --> TIME: 2025-02-25 15:28:22 -- STEP: 39/72 -- GLOBAL_STEP: 1050800 | > loss_disc: 2.536288022994995 (2.6526726942795973) | > loss_disc_real_0: 0.21895158290863037 (0.17712392925451964) | > loss_disc_real_1: 0.20634596049785614 (0.22108493153101358) | > loss_disc_real_2: 0.22368383407592773 (0.23234049058877504) | > loss_disc_real_3: 0.22215045988559723 (0.2360350829668534) | > loss_disc_real_4: 0.23288001120090485 (0.24653351574372023) | > loss_disc_real_5: 0.2395244985818863 (0.24611809849739075) | > loss_0: 2.536288022994995 (2.6526726942795973) | > grad_norm_0: tensor(218.7886, device='cuda:0') (tensor(144.8296, device='cuda:0')) | > loss_gen: 2.214797019958496 (2.0051578344442906) | > loss_kl: 2.060645818710327 (2.073412063794258) | > loss_feat: 3.6754722595214844 (3.262540212044349) | > loss_mel: 17.926740646362305 (17.511232033754002) | > loss_duration: 1.5435333251953125 (1.5523388416339192) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.421188354492188 (26.4046806433262) | > grad_norm_1: tensor(9584.2617, device='cuda:0') (tensor(7734.9609, device='cuda:0')) | > current_lr_0: 0.00018312827403052281 | > current_lr_1: 0.00018312827403052281 | > step_time: 0.4616 (0.4442277993911352) | > loader_time: 0.0083 (0.007369897304437099) --> EVAL PERFORMANCE | > avg_loader_time: 0.008657035693316393 (-0.00012434368402185897) | > avg_step_time: 0.4600070295199542 (-0.004706826008541498) | > avg_loss_disc: 2.6557447574508024 (-0.11709996344338025) | > avg_loss_disc_real_0: 0.1617227902399822 (-0.012496995254301674) | > avg_loss_disc_real_1: 0.23115276923062097 (-0.015472381782364014) | > avg_loss_disc_real_2: 0.23210816064351042 (-0.009341160176505464) | > avg_loss_disc_real_3: 0.23553639545407093 (+0.004579104588065347) | > avg_loss_disc_real_4: 0.2433888769485581 (-0.00016799422217086968) | > avg_loss_disc_real_5: 0.2417763158888884 (-0.001374469466612377) | > avg_loss_0: 2.6557447574508024 (-0.11709996344338025) | > avg_grad_norm_0: tensor(151.3212, device='cuda:0') (+tensor(15.0264, device='cuda:0')) | > avg_loss_gen: 2.06772001864205 (+0.03661293882719141) | > avg_loss_kl: 2.0794038739002927 (-0.3370354444208279) | > avg_loss_feat: 3.3773193661595733 (+0.10449521642335702) | > avg_loss_mel: 17.415828946610564 (-1.1936206414666017) | > avg_loss_duration: 1.5785081420146243 (-0.04599682378097336) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.518779996415258 (-1.4355449407873024) | > avg_grad_norm_1: tensor(7443.1660, device='cuda:0') (+tensor(2572.4277, device='cuda:0'))  > EPOCH: 706/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:28:40)   --> TIME: 2025-02-25 15:28:49 -- STEP: 17/72 -- GLOBAL_STEP: 1050850 | > loss_disc: 2.8832266330718994 (2.6472848443423995) | > loss_disc_real_0: 0.10893981158733368 (0.08973010103492175) | > loss_disc_real_1: 0.21483537554740906 (0.21233045353608973) | > loss_disc_real_2: 0.2358648031949997 (0.25182529232081247) | > loss_disc_real_3: 0.23121334612369537 (0.24045510414768667) | > loss_disc_real_4: 0.24755704402923584 (0.2470426848706077) | > loss_disc_real_5: 0.2775656580924988 (0.25009818550418406) | > loss_0: 2.8832266330718994 (2.6472848443423995) | > grad_norm_0: tensor(258.7976, device='cuda:0') (tensor(156.4014, device='cuda:0')) | > loss_gen: 1.75531804561615 (2.1417415001813103) | > loss_kl: 1.9996962547302246 (1.9376950123730827) | > loss_feat: 3.0195343494415283 (3.8078700093662037) | > loss_mel: 17.229387283325195 (17.313005279092227) | > loss_duration: 1.5554490089416504 (1.5274418101591223) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.559385299682617 (26.727753583122702) | > grad_norm_1: tensor(7912.2451, device='cuda:0') (tensor(7609.0088, device='cuda:0')) | > current_lr_0: 0.00018310538299626898 | > current_lr_1: 0.00018310538299626898 | > step_time: 0.4325 (0.4358049981734332) | > loader_time: 0.0082 (0.006856511620914235)  --> TIME: 2025-02-25 15:29:13 -- STEP: 67/72 -- GLOBAL_STEP: 1050900 | > loss_disc: 2.755305290222168 (2.613340573524362) | > loss_disc_real_0: 0.13145077228546143 (0.08964176937492925) | > loss_disc_real_1: 0.32418981194496155 (0.22368715970373865) | > loss_disc_real_2: 0.39787399768829346 (0.23639280204452687) | > loss_disc_real_3: 0.33749401569366455 (0.2345919882628455) | > loss_disc_real_4: 0.23203125596046448 (0.23894977324934147) | > loss_disc_real_5: 0.22945614159107208 (0.24814011023115756) | > loss_0: 2.755305290222168 (2.613340573524362) | > grad_norm_0: tensor(144.4995, device='cuda:0') (tensor(147.2811, device='cuda:0')) | > loss_gen: 2.705000162124634 (2.135360185779742) | > loss_kl: 2.218782424926758 (2.0727798173676684) | > loss_feat: 4.161281108856201 (3.7901138512056267) | > loss_mel: 18.28553009033203 (17.348376886168523) | > loss_duration: 1.686631441116333 (1.575193988743113) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.057226181030273 (26.921824981917197) | > grad_norm_1: tensor(6283.3350, device='cuda:0') (tensor(7072.2769, device='cuda:0')) | > current_lr_0: 0.00018310538299626898 | > current_lr_1: 0.00018310538299626898 | > step_time: 0.5125 (0.4524376250025052) | > loader_time: 0.012 (0.00853923185547786) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710679873614244 (+5.364418029785156e-05) | > avg_step_time: 0.45601940155029297 (-0.003987627969661234) | > avg_loss_disc: 2.610817492847712 (-0.04492726460309049) | > avg_loss_disc_real_0: 0.09337117171413459 (-0.06835161852584762) | > avg_loss_disc_real_1: 0.2253378866843774 (-0.005814882546243572) | > avg_loss_disc_real_2: 0.2413948029279709 (+0.009286642284460467) | > avg_loss_disc_real_3: 0.23527816420709582 (-0.0002582312469751047) | > avg_loss_disc_real_4: 0.23921276733908856 (-0.004176109609469542) | > avg_loss_disc_real_5: 0.2476164012307852 (+0.005840085341896811) | > avg_loss_0: 2.610817492847712 (-0.04492726460309049) | > avg_grad_norm_0: tensor(146.8748, device='cuda:0') (tensor(-4.4464, device='cuda:0')) | > avg_loss_gen: 2.145590132390949 (+0.07787011374889907) | > avg_loss_kl: 2.081846866809147 (+0.0024429929088545244) | > avg_loss_feat: 3.7774355713750274 (+0.4001162052154541) | > avg_loss_mel: 17.34754387761506 (-0.0682850689955039) | > avg_loss_duration: 1.5782105889118894 (-0.0002975531027349376) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.9306273393228 (+0.4118473429075422) | > avg_grad_norm_1: tensor(7016.8804, device='cuda:0') (tensor(-426.2856, device='cuda:0'))  > EPOCH: 707/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:29:17)   --> TIME: 2025-02-25 15:29:40 -- STEP: 45/72 -- GLOBAL_STEP: 1050950 | > loss_disc: 2.8707902431488037 (2.5469116634792752) | > loss_disc_real_0: 0.08008626103401184 (0.08303471621539854) | > loss_disc_real_1: 0.24422886967658997 (0.2185468037923177) | > loss_disc_real_2: 0.21571224927902222 (0.21530127425988516) | > loss_disc_real_3: 0.26653483510017395 (0.23017055425379013) | > loss_disc_real_4: 0.2656168043613434 (0.2415482607152727) | > loss_disc_real_5: 0.2594850957393646 (0.24520795477761162) | > loss_0: 2.8707902431488037 (2.5469116634792752) | > grad_norm_0: tensor(230.0087, device='cuda:0') (tensor(165.7174, device='cuda:0')) | > loss_gen: 1.7419393062591553 (2.131784971555074) | > loss_kl: 1.9544275999069214 (1.9790064785215589) | > loss_feat: 2.7871758937835693 (3.758229441112942) | > loss_mel: 16.598175048828125 (17.186877738104922) | > loss_duration: 1.590263843536377 (1.5512951109144422) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.671981811523438 (26.607193756103516) | > grad_norm_1: tensor(8247.0684, device='cuda:0') (tensor(7622.7739, device='cuda:0')) | > current_lr_0: 0.00018308249482339444 | > current_lr_1: 0.00018308249482339444 | > step_time: 0.4619 (0.44656798044840496) | > loader_time: 0.0088 (0.0076134045918782554) --> EVAL PERFORMANCE | > avg_loader_time: 0.008662717443116957 (-4.7962430497287475e-05) | > avg_step_time: 0.45883449702195717 (+0.0028150954716641974) | > avg_loss_disc: 2.547633154291502 (-0.06318433855620986) | > avg_loss_disc_real_0: 0.08271145888827214 (-0.01065971282586245) | > avg_loss_disc_real_1: 0.22105544385775713 (-0.00428244282662027) | > avg_loss_disc_real_2: 0.21736683350213817 (-0.024027969425832713) | > avg_loss_disc_real_3: 0.2272310689301558 (-0.00804709527694003) | > avg_loss_disc_real_4: 0.24332610780084638 (+0.0041133404617578195) | > avg_loss_disc_real_5: 0.24615807012772897 (-0.0014583311030562374) | > avg_loss_0: 2.547633154291502 (-0.06318433855620986) | > avg_grad_norm_0: tensor(165.8149, device='cuda:0') (+tensor(18.9401, device='cuda:0')) | > avg_loss_gen: 2.138830048937193 (-0.006760083453756227) | > avg_loss_kl: 2.0103921017176667 (-0.07145476509148052) | > avg_loss_feat: 3.7253204701651987 (-0.05211510120982865) | > avg_loss_mel: 17.08537990946165 (-0.26216396815340914) | > avg_loss_duration: 1.572720641821203 (-0.005489947090686398) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.532643304744234 (-0.3979840345785668) | > avg_grad_norm_1: tensor(7635.2471, device='cuda:0') (+tensor(618.3667, device='cuda:0'))  > EPOCH: 708/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:29:54)   --> TIME: 2025-02-25 15:30:06 -- STEP: 23/72 -- GLOBAL_STEP: 1051000 | > loss_disc: 3.4544613361358643 (2.645769336949224) | > loss_disc_real_0: 0.2358628809452057 (0.09535565243467041) | > loss_disc_real_1: 0.2157866656780243 (0.22604208925495978) | > loss_disc_real_2: 0.5463725924491882 (0.259006402414778) | > loss_disc_real_3: 0.29246363043785095 (0.23979819727980572) | > loss_disc_real_4: 0.24682264029979706 (0.24373286573783212) | > loss_disc_real_5: 0.2662818729877472 (0.2472301281016806) | > loss_0: 3.4544613361358643 (2.645769336949224) | > grad_norm_0: tensor(181.5372, device='cuda:0') (tensor(167.1382, device='cuda:0')) | > loss_gen: 2.1169514656066895 (2.1774961274603144) | > loss_kl: 2.2458648681640625 (1.9603426974752676) | > loss_feat: 3.5868277549743652 (3.7615072830863623) | > loss_mel: 18.755416870117188 (17.38322631172512) | > loss_duration: 1.5488923788070679 (1.5293528930000637) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.25395393371582 (26.811925307564113) | > grad_norm_1: tensor(2357.1509, device='cuda:0') (tensor(7058.1675, device='cuda:0')) | > current_lr_0: 0.0001830596095115415 | > current_lr_1: 0.0001830596095115415 | > step_time: 0.4336 (0.4324836316316024) | > loader_time: 0.0062 (0.006754387979922088) --> EVAL PERFORMANCE | > avg_loader_time: 0.008682764751810425 (+2.0047308693468824e-05) | > avg_step_time: 0.461826912114318 (+0.0029924150923608117) | > avg_loss_disc: 2.6849125237532068 (+0.13727936946170471) | > avg_loss_disc_real_0: 0.1214024952914513 (+0.03869103640317917) | > avg_loss_disc_real_1: 0.22237581780678786 (+0.0013203739490307287) | > avg_loss_disc_real_2: 0.2500488235497138 (+0.03268199004757563) | > avg_loss_disc_real_3: 0.2416874264747324 (+0.014456357544576609) | > avg_loss_disc_real_4: 0.24787943296029535 (+0.004553325159448979) | > avg_loss_disc_real_5: 0.2489444722591991 (+0.0027864021314701337) | > avg_loss_0: 2.6849125237532068 (+0.13727936946170471) | > avg_grad_norm_0: tensor(149.2439, device='cuda:0') (tensor(-16.5709, device='cuda:0')) | > avg_loss_gen: 2.1008292755610514 (-0.038000773376141606) | > avg_loss_kl: 2.305891671650847 (+0.29549956993318016) | > avg_loss_feat: 3.557452954037089 (-0.1678675161281098) | > avg_loss_mel: 18.421397544968297 (+1.3360176355066464) | > avg_loss_duration: 1.605495917964989 (+0.03277527614378606) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.991067241614974 (+1.4584239368707408) | > avg_grad_norm_1: tensor(6027.6382, device='cuda:0') (tensor(-1607.6089, device='cuda:0'))  > EPOCH: 709/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:30:32)   --> TIME: 2025-02-25 15:30:34 -- STEP: 1/72 -- GLOBAL_STEP: 1051050 | > loss_disc: 2.7555460929870605 (2.7555460929870605) | > loss_disc_real_0: 0.057926829904317856 (0.057926829904317856) | > loss_disc_real_1: 0.19660604000091553 (0.19660604000091553) | > loss_disc_real_2: 0.2187359482049942 (0.2187359482049942) | > loss_disc_real_3: 0.24037174880504608 (0.24037174880504608) | > loss_disc_real_4: 0.2441745549440384 (0.2441745549440384) | > loss_disc_real_5: 0.23749688267707825 (0.23749688267707825) | > loss_0: 2.7555460929870605 (2.7555460929870605) | > grad_norm_0: tensor(227.2894, device='cuda:0') (tensor(227.2894, device='cuda:0')) | > loss_gen: 1.8503789901733398 (1.8503789901733398) | > loss_kl: 2.1444764137268066 (2.1444764137268066) | > loss_feat: 3.4037299156188965 (3.4037299156188965) | > loss_mel: 18.427623748779297 (18.427623748779297) | > loss_duration: 1.5814616680145264 (1.5814616680145264) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.407670974731445 (27.407670974731445) | > grad_norm_1: tensor(9524.0088, device='cuda:0') (tensor(9524.0088, device='cuda:0')) | > current_lr_0: 0.00018303672706035257 | > current_lr_1: 0.00018303672706035257 | > step_time: 0.4385 (0.43850255012512207) | > loader_time: 0.0056 (0.005632162094116211)  --> TIME: 2025-02-25 15:30:57 -- STEP: 51/72 -- GLOBAL_STEP: 1051100 | > loss_disc: 2.264434337615967 (2.5492412810232126) | > loss_disc_real_0: 0.08007101714611053 (0.10734338262209706) | > loss_disc_real_1: 0.22399823367595673 (0.22458157527680492) | > loss_disc_real_2: 0.2091352939605713 (0.21970082673372007) | > loss_disc_real_3: 0.22185492515563965 (0.23133555171536466) | > loss_disc_real_4: 0.2447211742401123 (0.24109262958461164) | > loss_disc_real_5: 0.2453422248363495 (0.24284698770326726) | > loss_0: 2.264434337615967 (2.5492412810232126) | > grad_norm_0: tensor(71.9425, device='cuda:0') (tensor(150.0096, device='cuda:0')) | > loss_gen: 2.7585933208465576 (2.192737322227627) | > loss_kl: 2.01235294342041 (2.042725778093525) | > loss_feat: 4.550549030303955 (3.67644188450832) | > loss_mel: 17.448387145996094 (17.280110134797933) | > loss_duration: 1.5944112539291382 (1.5601293596566892) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.364294052124023 (26.752144458247166) | > grad_norm_1: tensor(6143.9341, device='cuda:0') (tensor(7147.2715, device='cuda:0')) | > current_lr_0: 0.00018303672706035257 | > current_lr_1: 0.00018303672706035257 | > step_time: 0.4699 (0.4459350296095306) | > loader_time: 0.0089 (0.007809302386115579) --> EVAL PERFORMANCE | > avg_loader_time: 0.008659510545327629 (-2.325420648279647e-05) | > avg_step_time: 0.4579699610320615 (-0.0038569510822564768) | > avg_loss_disc: 2.551106550324131 (-0.13380597342907574) | > avg_loss_disc_real_0: 0.10421903730488159 (-0.017183457986569714) | > avg_loss_disc_real_1: 0.22296171247119634 (+0.0005858946644084762) | > avg_loss_disc_real_2: 0.2225838603687958 (-0.02746496318091801) | > avg_loss_disc_real_3: 0.22984278244031986 (-0.011844644034412544) | > avg_loss_disc_real_4: 0.24033931254501073 (-0.0075401204152846235) | > avg_loss_disc_real_5: 0.2430087195735582 (-0.005935752685640894) | > avg_loss_0: 2.551106550324131 (-0.13380597342907574) | > avg_grad_norm_0: tensor(151.0013, device='cuda:0') (+tensor(1.7573, device='cuda:0')) | > avg_loss_gen: 2.182148088871593 (+0.08131881331054158) | > avg_loss_kl: 2.0584860872214943 (-0.24740558442935257) | > avg_loss_feat: 3.677330581235214 (+0.11987762719812523) | > avg_loss_mel: 17.299672610323196 (-1.1217249346451013) | > avg_loss_duration: 1.5745880150459182 (-0.030907902919070862) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.79222534072231 (-1.1988419008926634) | > avg_grad_norm_1: tensor(7237.4038, device='cuda:0') (+tensor(1209.7656, device='cuda:0'))  > EPOCH: 710/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:31:09)   --> TIME: 2025-02-25 15:31:24 -- STEP: 29/72 -- GLOBAL_STEP: 1051150 | > loss_disc: 2.645822525024414 (2.51496592883406) | > loss_disc_real_0: 0.13195598125457764 (0.08786681268749566) | > loss_disc_real_1: 0.22409187257289886 (0.2168693244457245) | > loss_disc_real_2: 0.2614343762397766 (0.20591574804536228) | > loss_disc_real_3: 0.2690448760986328 (0.22638086769087562) | > loss_disc_real_4: 0.239356130361557 (0.24048085870413943) | > loss_disc_real_5: 0.25128239393234253 (0.24654529526315885) | > loss_0: 2.645822525024414 (2.51496592883406) | > grad_norm_0: tensor(127.7929, device='cuda:0') (tensor(154.8006, device='cuda:0')) | > loss_gen: 2.0921897888183594 (2.201217071763401) | > loss_kl: 1.978530764579773 (1.9707508457118068) | > loss_feat: 3.188206911087036 (3.837397846682318) | > loss_mel: 17.213573455810547 (17.249036065463358) | > loss_duration: 1.546709418296814 (1.5244915033208912) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.019208908081055 (26.782893082191205) | > grad_norm_1: tensor(7885.7676, device='cuda:0') (tensor(7848.5010, device='cuda:0')) | > current_lr_0: 0.00018301384746947002 | > current_lr_1: 0.00018301384746947002 | > step_time: 0.4344 (0.4349103878284323) | > loader_time: 0.0076 (0.007021361383898505) --> EVAL PERFORMANCE | > avg_loader_time: 0.008636212684738805 (-2.329786058882441e-05) | > avg_step_time: 0.45268054411444864 (-0.005289416917612866) | > avg_loss_disc: 2.5895869731903076 (+0.03848042286617659) | > avg_loss_disc_real_0: 0.12157718375535079 (+0.017358146450469195) | > avg_loss_disc_real_1: 0.21922933950390613 (-0.003732372967290204) | > avg_loss_disc_real_2: 0.2141787355214777 (-0.00840512484731809) | > avg_loss_disc_real_3: 0.22459388230468183 (-0.005248900135638029) | > avg_loss_disc_real_4: 0.24149468310282263 (+0.0011553705578118967) | > avg_loss_disc_real_5: 0.2440844910665297 (+0.0010757714929715) | > avg_loss_0: 2.5895869731903076 (+0.03848042286617659) | > avg_grad_norm_0: tensor(155.4449, device='cuda:0') (+tensor(4.4436, device='cuda:0')) | > avg_loss_gen: 2.1472729696354405 (-0.034875119236152496) | > avg_loss_kl: 2.0271627164222825 (-0.03132337079921177) | > avg_loss_feat: 3.6888736671125386 (+0.011543085877324444) | > avg_loss_mel: 17.126804432398828 (-0.17286817792436793) | > avg_loss_duration: 1.5652406114927484 (-0.009347403553169764) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.555354239235466 (-0.23687110148684454) | > avg_grad_norm_1: tensor(7755.9067, device='cuda:0') (+tensor(518.5029, device='cuda:0'))  > EPOCH: 711/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:31:45)   --> TIME: 2025-02-25 15:31:51 -- STEP: 7/72 -- GLOBAL_STEP: 1051200 | > loss_disc: 2.616386890411377 (2.630521331514631) | > loss_disc_real_0: 0.20379549264907837 (0.17363326038633073) | > loss_disc_real_1: 0.18939949572086334 (0.20657997259071895) | > loss_disc_real_2: 0.22196827828884125 (0.22536442961011613) | > loss_disc_real_3: 0.23116768896579742 (0.24045918881893158) | > loss_disc_real_4: 0.21332114934921265 (0.22999277923788344) | > loss_disc_real_5: 0.2472550868988037 (0.24367806315422058) | > loss_0: 2.616386890411377 (2.630521331514631) | > grad_norm_0: tensor(66.2617, device='cuda:0') (tensor(178.9138, device='cuda:0')) | > loss_gen: 2.0093603134155273 (1.9573351996285575) | > loss_kl: 1.8409557342529297 (1.8979849815368652) | > loss_feat: 3.4394726753234863 (3.2139924594334195) | > loss_mel: 17.968111038208008 (17.17879922049386) | > loss_duration: 1.5014303922653198 (1.5254421404429845) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.75933074951172 (25.7735538482666) | > grad_norm_1: tensor(8647.8193, device='cuda:0') (tensor(7792.3931, device='cuda:0')) | > current_lr_0: 0.00018299097073853633 | > current_lr_1: 0.00018299097073853633 | > step_time: 0.4535 (0.43513628414699007) | > loader_time: 0.0085 (0.006305490221296038)  --> TIME: 2025-02-25 15:32:14 -- STEP: 57/72 -- GLOBAL_STEP: 1051250 | > loss_disc: 2.4822258949279785 (2.7102633442795065) | > loss_disc_real_0: 0.08311836421489716 (0.1551884173562653) | > loss_disc_real_1: 0.2408500760793686 (0.26388188543026914) | > loss_disc_real_2: 0.23905088007450104 (0.2352542631458818) | > loss_disc_real_3: 0.2695558965206146 (0.2405122542067578) | > loss_disc_real_4: 0.25307151675224304 (0.24892150780610867) | > loss_disc_real_5: 0.25275617837905884 (0.24807270263370715) | > loss_0: 2.4822258949279785 (2.7102633442795065) | > grad_norm_0: tensor(127.8983, device='cuda:0') (tensor(196.4805, device='cuda:0')) | > loss_gen: 2.1925406455993652 (2.016945228242038) | > loss_kl: 2.0936737060546875 (2.022623681185537) | > loss_feat: 3.7619869709014893 (3.2811467773035954) | > loss_mel: 17.024646759033203 (17.005096736707188) | > loss_duration: 1.6026248931884766 (1.5540630921982883) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.675472259521484 (25.879875250029983) | > grad_norm_1: tensor(12553.8584, device='cuda:0') (tensor(8605.0518, device='cuda:0')) | > current_lr_0: 0.00018299097073853633 | > current_lr_1: 0.00018299097073853633 | > step_time: 0.4685 (0.45546334668209676) | > loader_time: 0.0097 (0.008007446924845377) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864544049115248 (+9.22780641367596e-06) | > avg_step_time: 0.46705190564545107 (+0.014371361531002436) | > avg_loss_disc: 2.7121839758376 (+0.12259700264729245) | > avg_loss_disc_real_0: 0.15171918314947214 (+0.03014199939412135) | > avg_loss_disc_real_1: 0.2586470747917471 (+0.039417735287840955) | > avg_loss_disc_real_2: 0.2344931699860264 (+0.020314434464548697) | > avg_loss_disc_real_3: 0.2405847619536897 (+0.01599087964900786) | > avg_loss_disc_real_4: 0.24565199017524714 (+0.00415730707242451) | > avg_loss_disc_real_5: 0.2454115005446152 (+0.001327009478085478) | > avg_loss_0: 2.7121839758376 (+0.12259700264729245) | > avg_grad_norm_0: tensor(190.5688, device='cuda:0') (+tensor(35.1239, device='cuda:0')) | > avg_loss_gen: 1.9883050565988245 (-0.15896791303661595) | > avg_loss_kl: 2.0543778882899746 (+0.027215171867692067) | > avg_loss_feat: 3.235241836225483 (-0.4536318308870557) | > avg_loss_mel: 17.002537431851238 (-0.1242670005475901) | > avg_loss_duration: 1.5674498685648743 (+0.0022092570721259364) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.847911888444926 (-0.7074423507905401) | > avg_grad_norm_1: tensor(8542.7617, device='cuda:0') (+tensor(786.8550, device='cuda:0'))  > EPOCH: 712/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:32:23)   --> TIME: 2025-02-25 15:32:41 -- STEP: 35/72 -- GLOBAL_STEP: 1051300 | > loss_disc: 2.8036916255950928 (2.6667140415736608) | > loss_disc_real_0: 0.14523552358150482 (0.13997535609773223) | > loss_disc_real_1: 0.20365479588508606 (0.2228997609445027) | > loss_disc_real_2: 0.21309636533260345 (0.2334549307823181) | > loss_disc_real_3: 0.222185418009758 (0.22627704526696887) | > loss_disc_real_4: 0.22174367308616638 (0.24356803979192462) | > loss_disc_real_5: 0.23221847414970398 (0.24539052375725337) | > loss_0: 2.8036916255950928 (2.6667140415736608) | > grad_norm_0: tensor(141.3447, device='cuda:0') (tensor(147.8480, device='cuda:0')) | > loss_gen: 1.7144825458526611 (1.934035243306841) | > loss_kl: 2.3370416164398193 (2.3723304135458805) | > loss_feat: 2.9882097244262695 (3.296540778023856) | > loss_mel: 19.175872802734375 (18.43015463692802) | > loss_duration: 1.6038159132003784 (1.5602537734167916) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.81942367553711 (27.59331512451172) | > grad_norm_1: tensor(1211.5648, device='cuda:0') (tensor(7525.2393, device='cuda:0')) | > current_lr_0: 0.000182968096867194 | > current_lr_1: 0.000182968096867194 | > step_time: 0.4483 (0.43864721570696147) | > loader_time: 0.0083 (0.007118606567382812) --> EVAL PERFORMANCE | > avg_loader_time: 0.008586017178817533 (-5.9423312334947256e-05) | > avg_step_time: 0.45538259559953714 (-0.011669310045913928) | > avg_loss_disc: 2.741122588305406 (+0.028938612467805935) | > avg_loss_disc_real_0: 0.1924143496008826 (+0.04069516645141047) | > avg_loss_disc_real_1: 0.22249114471422116 (-0.03615593007752593) | > avg_loss_disc_real_2: 0.2385039661132114 (+0.004010796127184996) | > avg_loss_disc_real_3: 0.23403107616263377 (-0.006553685791055919) | > avg_loss_disc_real_4: 0.24319224059581757 (-0.002459749579429571) | > avg_loss_disc_real_5: 0.24660322447897684 (+0.0011917239343616493) | > avg_loss_0: 2.741122588305406 (+0.028938612467805935) | > avg_grad_norm_0: tensor(157.7690, device='cuda:0') (tensor(-32.7998, device='cuda:0')) | > avg_loss_gen: 1.9800889273764375 (-0.008216129222387014) | > avg_loss_kl: 2.3142948402485377 (+0.2599169519585631) | > avg_loss_feat: 3.2634234965687066 (+0.02818166034322367) | > avg_loss_mel: 18.181488977351663 (+1.178951545500425) | > avg_loss_duration: 1.5942674959209604 (+0.02681762735608606) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.33356397924289 (+1.485652090797963) | > avg_grad_norm_1: tensor(6430.9023, device='cuda:0') (tensor(-2111.8594, device='cuda:0'))  > EPOCH: 713/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:33:00)   --> TIME: 2025-02-25 15:33:08 -- STEP: 13/72 -- GLOBAL_STEP: 1051350 | > loss_disc: 2.581092357635498 (2.7141634867741513) | > loss_disc_real_0: 0.15599963068962097 (0.15974725897495562) | > loss_disc_real_1: 0.20868952572345734 (0.22562492696138528) | > loss_disc_real_2: 0.32018348574638367 (0.24389687409767738) | > loss_disc_real_3: 0.3205406963825226 (0.248035171857247) | > loss_disc_real_4: 0.3034055531024933 (0.2507237814939939) | > loss_disc_real_5: 0.2816035747528076 (0.24561756161543039) | > loss_0: 2.581092357635498 (2.7141634867741513) | > grad_norm_0: tensor(163.7807, device='cuda:0') (tensor(114.5133, device='cuda:0')) | > loss_gen: 3.402482748031616 (2.118661541205186) | > loss_kl: 4.711792469024658 (3.2630453293140116) | > loss_feat: 5.049127101898193 (3.607787957558265) | > loss_mel: 24.390884399414062 (21.34718601520245) | > loss_duration: 1.8009138107299805 (1.6862649000608003) | > amp_scaler: 8.0 (8.0) | > loss_1: 39.35519790649414 (32.02294481717623) | > grad_norm_1: tensor(11108.8076, device='cuda:0') (tensor(6786.0308, device='cuda:0')) | > current_lr_0: 0.0001829452258550856 | > current_lr_1: 0.0001829452258550856 | > step_time: 0.4723 (0.4323753210214468) | > loader_time: 0.006 (0.00677941395686223)  --> TIME: 2025-02-25 15:33:31 -- STEP: 63/72 -- GLOBAL_STEP: 1051400 | > loss_disc: 2.6395559310913086 (2.6874830836341497) | > loss_disc_real_0: 0.13175781071186066 (0.13935903323784707) | > loss_disc_real_1: 0.3587886691093445 (0.2289047957885833) | > loss_disc_real_2: 0.13739365339279175 (0.24553762269871565) | > loss_disc_real_3: 0.1579725742340088 (0.24176077118941716) | > loss_disc_real_4: 0.22562669217586517 (0.24779690612876232) | > loss_disc_real_5: 0.24229677021503448 (0.24935186027534426) | > loss_0: 2.6395559310913086 (2.6874830836341497) | > grad_norm_0: tensor(62.4865, device='cuda:0') (tensor(135.6035, device='cuda:0')) | > loss_gen: 1.8442014455795288 (2.144325074695406) | > loss_kl: 2.2343873977661133 (2.9945609720926436) | > loss_feat: 3.31215238571167 (3.5711674501025485) | > loss_mel: 18.229211807250977 (20.510495503743485) | > loss_duration: 1.6101425886154175 (1.6680586224510556) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.23009490966797 (30.888607539827863) | > grad_norm_1: tensor(7795.9346, device='cuda:0') (tensor(6337.0942, device='cuda:0')) | > current_lr_0: 0.0001829452258550856 | > current_lr_1: 0.0001829452258550856 | > step_time: 0.4966 (0.4476866192287869) | > loader_time: 0.0112 (0.008384375345139273) --> EVAL PERFORMANCE | > avg_loader_time: 0.008746187451859592 (+0.00016017027304205886) | > avg_step_time: 0.45394191607623036 (-0.001440679523306787) | > avg_loss_disc: 2.681378075774287 (-0.05974451253111912) | > avg_loss_disc_real_0: 0.14180338618830896 (-0.05061096341257365) | > avg_loss_disc_real_1: 0.2269773670065571 (+0.004486222292335945) | > avg_loss_disc_real_2: 0.24078918458290494 (+0.0022852184696935396) | > avg_loss_disc_real_3: 0.23790707386715312 (+0.0038759977045193517) | > avg_loss_disc_real_4: 0.24602069338442578 (+0.0028284527886082156) | > avg_loss_disc_real_5: 0.2484066475864867 (+0.0018034231075098606) | > avg_loss_0: 2.681378075774287 (-0.05974451253111912) | > avg_grad_norm_0: tensor(132.1737, device='cuda:0') (tensor(-25.5953, device='cuda:0')) | > avg_loss_gen: 2.1283284543265766 (+0.1482395269501391) | > avg_loss_kl: 2.927950476256895 (+0.6136556360083572) | > avg_loss_feat: 3.539234960582894 (+0.27581146401418755) | > avg_loss_mel: 20.235100208873476 (+2.0536112315218134) | > avg_loss_duration: 1.66673387608058 (+0.0724663801596197) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 30.497347791429977 (+3.163783812187088) | > avg_grad_norm_1: tensor(6273.6904, device='cuda:0') (tensor(-157.2119, device='cuda:0'))  > EPOCH: 714/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:33:37)   --> TIME: 2025-02-25 15:33:58 -- STEP: 41/72 -- GLOBAL_STEP: 1051450 | > loss_disc: 2.6969761848449707 (2.549371643764217) | > loss_disc_real_0: 0.12355205416679382 (0.10652395710349083) | > loss_disc_real_1: 0.254270076751709 (0.22234795515130207) | > loss_disc_real_2: 0.24948091804981232 (0.22637458363684212) | > loss_disc_real_3: 0.22993654012680054 (0.23873112841350277) | > loss_disc_real_4: 0.24939532577991486 (0.2417199498996502) | > loss_disc_real_5: 0.2690468430519104 (0.24478736701535014) | > loss_0: 2.6969761848449707 (2.549371643764217) | > grad_norm_0: tensor(199.1400, device='cuda:0') (tensor(130.0545, device='cuda:0')) | > loss_gen: 1.8540440797805786 (2.216435886010891) | > loss_kl: 2.0335137844085693 (2.17228059652375) | > loss_feat: 2.976865291595459 (3.768128959144034) | > loss_mel: 16.60728645324707 (18.12222276082853) | > loss_duration: 1.5767388343811035 (1.5729397709776716) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.048446655273438 (27.852007796124713) | > grad_norm_1: tensor(6427.0425, device='cuda:0') (tensor(5883.7559, device='cuda:0')) | > current_lr_0: 0.0001829223577018537 | > current_lr_1: 0.0001829223577018537 | > step_time: 0.4488 (0.4365599736934755) | > loader_time: 0.0091 (0.007570877307798804) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669090942597726 (-7.709650926186577e-05) | > avg_step_time: 0.46121701724092723 (+0.007275101164696873) | > avg_loss_disc: 2.539303312838917 (-0.14207476293537002) | > avg_loss_disc_real_0: 0.10509356914061897 (-0.03670981704768998) | > avg_loss_disc_real_1: 0.2210744898503935 (-0.005902877156163611) | > avg_loss_disc_real_2: 0.22728553309407032 (-0.013503651488834617) | > avg_loss_disc_real_3: 0.2361554702822591 (-0.0017516035848940126) | > avg_loss_disc_real_4: 0.24142030490116334 (-0.004600388483262441) | > avg_loss_disc_real_5: 0.2471310987438954 (-0.0012755488425912898) | > avg_loss_0: 2.539303312838917 (-0.14207476293537002) | > avg_grad_norm_0: tensor(129.5352, device='cuda:0') (tensor(-2.6385, device='cuda:0')) | > avg_loss_gen: 2.2170653091350063 (+0.08873685480842974) | > avg_loss_kl: 2.152978520997813 (-0.7749719552590819) | > avg_loss_feat: 3.7801158763992952 (+0.24088091581640114) | > avg_loss_mel: 17.862953172603124 (-2.372147036270352) | > avg_loss_duration: 1.5914700098440682 (-0.07526386623651193) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.604582773128026 (-2.892765018301951) | > avg_grad_norm_1: tensor(6152.1084, device='cuda:0') (tensor(-121.5820, device='cuda:0'))  > EPOCH: 715/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:34:15)   --> TIME: 2025-02-25 15:34:25 -- STEP: 19/72 -- GLOBAL_STEP: 1051500 | > loss_disc: 2.740201473236084 (2.5320872382113806) | > loss_disc_real_0: 0.10889241099357605 (0.12238041860492606) | > loss_disc_real_1: 0.2381797879934311 (0.21140892960523305) | > loss_disc_real_2: 0.20822429656982422 (0.21105398472986722) | > loss_disc_real_3: 0.18296074867248535 (0.21403313781085767) | > loss_disc_real_4: 0.22964511811733246 (0.23341236459581474) | > loss_disc_real_5: 0.23561640083789825 (0.24068516492843628) | > loss_0: 2.740201473236084 (2.5320872382113806) | > grad_norm_0: tensor(355.8596, device='cuda:0') (tensor(158.9735, device='cuda:0')) | > loss_gen: 1.8347288370132446 (2.241330956157885) | > loss_kl: 1.8163564205169678 (1.9901826632650275) | > loss_feat: 4.20060396194458 (3.976481324747989) | > loss_mel: 17.69753646850586 (17.662069521452253) | > loss_duration: 1.4969733953475952 (1.5370847802413137) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.04619789123535 (27.407148963526677) | > grad_norm_1: tensor(9805.3984, device='cuda:0') (tensor(7048.2153, device='cuda:0')) | > current_lr_0: 0.00018289949240714095 | > current_lr_1: 0.00018289949240714095 | > step_time: 0.4513 (0.4367958871941817) | > loader_time: 0.0073 (0.006721847935726768)  --> TIME: 2025-02-25 15:34:49 -- STEP: 69/72 -- GLOBAL_STEP: 1051550 | > loss_disc: 2.784489870071411 (2.647145063980766) | > loss_disc_real_0: 0.26183533668518066 (0.1566255668691103) | > loss_disc_real_1: 0.20071332156658173 (0.2092343778281972) | > loss_disc_real_2: 0.25044843554496765 (0.21649919353101565) | > loss_disc_real_3: 0.30462804436683655 (0.22059974808623825) | > loss_disc_real_4: 0.25386661291122437 (0.23579126186128976) | > loss_disc_real_5: 0.243734210729599 (0.24611905260362488) | > loss_0: 2.784489870071411 (2.647145063980766) | > grad_norm_0: tensor(53.0296, device='cuda:0') (tensor(159.1935, device='cuda:0')) | > loss_gen: 1.8310483694076538 (2.0578787188599073) | > loss_kl: 2.1964218616485596 (2.0353188739306693) | > loss_feat: 2.6368284225463867 (3.5760076391524165) | > loss_mel: 17.266632080078125 (17.217692762181386) | > loss_duration: 1.6539475917816162 (1.5755127409230107) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.58487892150879 (26.462410664212875) | > grad_norm_1: tensor(2254.5454, device='cuda:0') (tensor(6605.5449, device='cuda:0')) | > current_lr_0: 0.00018289949240714095 | > current_lr_1: 0.00018289949240714095 | > step_time: 0.5416 (0.45452145908189856) | > loader_time: 0.0128 (0.008595939995586008) --> EVAL PERFORMANCE | > avg_loader_time: 0.008650245800824232 (-1.884514177349396e-05) | > avg_step_time: 0.4562325813400913 (-0.004984435900835915) | > avg_loss_disc: 2.64536448599587 (+0.10606117315695318) | > avg_loss_disc_real_0: 0.15753817741929646 (+0.05244460827867749) | > avg_loss_disc_real_1: 0.2096135095811226 (-0.011460980269270893) | > avg_loss_disc_real_2: 0.21727792950163424 (-0.010007603592436076) | > avg_loss_disc_real_3: 0.22165790431096521 (-0.014497565971293896) | > avg_loss_disc_real_4: 0.236703152178039 (-0.00471715272312434) | > avg_loss_disc_real_5: 0.24648865448756957 (-0.0006424442563258415) | > avg_loss_0: 2.64536448599587 (+0.10606117315695318) | > avg_grad_norm_0: tensor(157.3669, device='cuda:0') (+tensor(27.8317, device='cuda:0')) | > avg_loss_gen: 2.0639898609107643 (-0.15307544822424202) | > avg_loss_kl: 2.0376230310386343 (-0.11535548995917866) | > avg_loss_feat: 3.5626072682125467 (-0.21750860818674855) | > avg_loss_mel: 17.187543398897414 (-0.6754097737057094) | > avg_loss_duration: 1.5763064454978621 (-0.015163564346206027) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.428069934039048 (-1.1765128390889785) | > avg_grad_norm_1: tensor(6601.4751, device='cuda:0') (+tensor(449.3667, device='cuda:0'))  > EPOCH: 716/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:34:52)   --> TIME: 2025-02-25 15:35:15 -- STEP: 47/72 -- GLOBAL_STEP: 1051600 | > loss_disc: 2.730130672454834 (2.719010606725166) | > loss_disc_real_0: 0.1194429099559784 (0.12232309040871073) | > loss_disc_real_1: 0.40301650762557983 (0.2535634731992762) | > loss_disc_real_2: 0.2510446608066559 (0.25582277013900434) | > loss_disc_real_3: 0.20834289491176605 (0.2372828227408389) | > loss_disc_real_4: 0.25753697752952576 (0.2476171632396414) | > loss_disc_real_5: 0.27937236428260803 (0.24599767904332343) | > loss_0: 2.730130672454834 (2.719010606725166) | > grad_norm_0: tensor(88.0737, device='cuda:0') (tensor(166.5380, device='cuda:0')) | > loss_gen: 2.034679412841797 (2.001637177264436) | > loss_kl: 2.179915189743042 (1.9754172512825499) | > loss_feat: 3.153684139251709 (3.3384656094490213) | > loss_mel: 16.383859634399414 (17.137853500690866) | > loss_duration: 1.6100102663040161 (1.551999010938279) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.36214828491211 (26.00537263586166) | > grad_norm_1: tensor(5052.1162, device='cuda:0') (tensor(6359.4448, device='cuda:0')) | > current_lr_0: 0.00018287662997059005 | > current_lr_1: 0.00018287662997059005 | > step_time: 0.4714 (0.44172449314847906) | > loader_time: 0.0089 (0.007753859175012466) --> EVAL PERFORMANCE | > avg_loader_time: 0.008703312403719191 (+5.3066602894958834e-05) | > avg_step_time: 0.4544146866865561 (-0.0018178946535352125) | > avg_loss_disc: 2.733672518125723 (+0.08830803212985305) | > avg_loss_disc_real_0: 0.15301915819586157 (-0.004519019223434895) | > avg_loss_disc_real_1: 0.24827367172274786 (+0.038660162141625265) | > avg_loss_disc_real_2: 0.25034920220643714 (+0.033071272704802895) | > avg_loss_disc_real_3: 0.23804113079964276 (+0.016383226488677544) | > avg_loss_disc_real_4: 0.24721523615675914 (+0.010512083978720138) | > avg_loss_disc_real_5: 0.24769070513651403 (+0.0012020506489444638) | > avg_loss_0: 2.733672518125723 (+0.08830803212985305) | > avg_grad_norm_0: tensor(168.3374, device='cuda:0') (+tensor(10.9706, device='cuda:0')) | > avg_loss_gen: 1.9817523922718747 (-0.08223746863888959) | > avg_loss_kl: 2.367367128251305 (+0.3297440972126706) | > avg_loss_feat: 3.2589045544745217 (-0.30370271373802504) | > avg_loss_mel: 18.45230056870151 (+1.2647571698040956) | > avg_loss_duration: 1.6018113569474557 (+0.025504911449593548) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.662135997288665 (+1.2340660632496174) | > avg_grad_norm_1: tensor(6102.2354, device='cuda:0') (tensor(-499.2397, device='cuda:0'))  > EPOCH: 717/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:35:28)   --> TIME: 2025-02-25 15:35:42 -- STEP: 25/72 -- GLOBAL_STEP: 1051650 | > loss_disc: 2.602721691131592 (2.596048154830932) | > loss_disc_real_0: 0.07231606543064117 (0.13514545857906343) | > loss_disc_real_1: 0.2291223406791687 (0.20752330362796784) | > loss_disc_real_2: 0.2463759034872055 (0.23835445046424866) | > loss_disc_real_3: 0.30106619000434875 (0.2259762966632843) | > loss_disc_real_4: 0.24838076531887054 (0.24070360600948335) | > loss_disc_real_5: 0.24162696301937103 (0.23940774142742158) | > loss_0: 2.602721691131592 (2.596048154830932) | > grad_norm_0: tensor(237.1863, device='cuda:0') (tensor(147.1899, device='cuda:0')) | > loss_gen: 2.0654544830322266 (2.1021261692047117) | > loss_kl: 2.306187629699707 (2.2418685388565063) | > loss_feat: 3.6259987354278564 (3.6753541564941408) | > loss_mel: 18.109752655029297 (18.30290702819824) | > loss_duration: 1.5590355396270752 (1.559611177444458) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.66642951965332 (27.881867294311522) | > grad_norm_1: tensor(6107.9307, device='cuda:0') (tensor(6829.3906, device='cuda:0')) | > current_lr_0: 0.0001828537703918437 | > current_lr_1: 0.0001828537703918437 | > step_time: 0.4444 (0.4332686710357666) | > loader_time: 0.0063 (0.006810722351074219) --> EVAL PERFORMANCE | > avg_loader_time: 0.008675662564559724 (-2.764983915946699e-05) | > avg_step_time: 0.46137476639008856 (+0.006960079703532462) | > avg_loss_disc: 2.6007463965617434 (-0.13292612156397965) | > avg_loss_disc_real_0: 0.10305593965548868 (-0.04996321854037289) | > avg_loss_disc_real_1: 0.21576281615965803 (-0.03251085556308983) | > avg_loss_disc_real_2: 0.23373557522263325 (-0.016613626983803892) | > avg_loss_disc_real_3: 0.2294361761338274 (-0.008604954665815345) | > avg_loss_disc_real_4: 0.24418022536056141 (-0.0030350107961977235) | > avg_loss_disc_real_5: 0.24438343900190274 (-0.0033072661346112975) | > avg_loss_0: 2.6007463965617434 (-0.13292612156397965) | > avg_grad_norm_0: tensor(169.6448, device='cuda:0') (+tensor(1.3074, device='cuda:0')) | > avg_loss_gen: 2.074462405392822 (+0.09271001312094729) | > avg_loss_kl: 2.172929616041585 (-0.19443751220971972) | > avg_loss_feat: 3.679602065556486 (+0.4206975110819644) | > avg_loss_mel: 17.914457321166996 (-0.5378432475345143) | > avg_loss_duration: 1.5903873762614291 (-0.011423980686026569) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.43183882807342 (-0.23029716921524468) | > avg_grad_norm_1: tensor(7983.0933, device='cuda:0') (+tensor(1880.8579, device='cuda:0'))  > EPOCH: 718/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:36:06)   --> TIME: 2025-02-25 15:36:09 -- STEP: 3/72 -- GLOBAL_STEP: 1051700 | > loss_disc: 2.498624324798584 (2.6280627250671387) | > loss_disc_real_0: 0.0484197735786438 (0.07800055046876271) | > loss_disc_real_1: 0.16665631532669067 (0.1832302063703537) | > loss_disc_real_2: 0.2941078543663025 (0.29584232966105145) | > loss_disc_real_3: 0.2477348893880844 (0.23440618316332498) | > loss_disc_real_4: 0.24971574544906616 (0.2419794797897339) | > loss_disc_real_5: 0.25381729006767273 (0.2503119309743245) | > loss_0: 2.498624324798584 (2.6280627250671387) | > grad_norm_0: tensor(91.8400, device='cuda:0') (tensor(247.3509, device='cuda:0')) | > loss_gen: 2.102785348892212 (2.0148657162984214) | > loss_kl: 2.085566759109497 (2.116346756617228) | > loss_feat: 3.8044095039367676 (3.7581788698832193) | > loss_mel: 18.25439453125 (18.081026713053387) | > loss_duration: 1.5879735946655273 (1.570509711901347) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.835128784179688 (27.54092788696289) | > grad_norm_1: tensor(12136.1777, device='cuda:0') (tensor(9894.6738, device='cuda:0')) | > current_lr_0: 0.00018283091367054472 | > current_lr_1: 0.00018283091367054472 | > step_time: 0.4244 (0.4329833189646403) | > loader_time: 0.0057 (0.006523450215657552)  --> TIME: 2025-02-25 15:36:32 -- STEP: 53/72 -- GLOBAL_STEP: 1051750 | > loss_disc: 2.6020257472991943 (2.573850343812187) | > loss_disc_real_0: 0.08588391542434692 (0.10145302791640444) | > loss_disc_real_1: 0.19878646731376648 (0.21520358099127715) | > loss_disc_real_2: 0.19748984277248383 (0.2330793707438235) | > loss_disc_real_3: 0.20734967291355133 (0.23687442827899502) | > loss_disc_real_4: 0.23468951880931854 (0.2422126497862474) | > loss_disc_real_5: 0.23221978545188904 (0.24530622728590695) | > loss_0: 2.6020257472991943 (2.573850343812187) | > grad_norm_0: tensor(252.2824, device='cuda:0') (tensor(221.6678, device='cuda:0')) | > loss_gen: 1.9788179397583008 (2.1001847555052566) | > loss_kl: 2.077996015548706 (2.0315722249588872) | > loss_feat: 3.6800060272216797 (3.696714261792741) | > loss_mel: 16.974040985107422 (17.382608971505796) | > loss_duration: 1.602489948272705 (1.559117389175127) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.313350677490234 (26.770197670414763) | > grad_norm_1: tensor(11210.6416, device='cuda:0') (tensor(9945.3965, device='cuda:0')) | > current_lr_0: 0.00018283091367054472 | > current_lr_1: 0.00018283091367054472 | > step_time: 0.4502 (0.4440325701011802) | > loader_time: 0.009 (0.007858060440927182) --> EVAL PERFORMANCE | > avg_loader_time: 0.008662556258725449 (-1.3106305834275148e-05) | > avg_step_time: 0.4565481938106913 (-0.004826572579397281) | > avg_loss_disc: 2.5870994346242555 (-0.013646961937487934) | > avg_loss_disc_real_0: 0.1016759729196488 (-0.0013799667358398715) | > avg_loss_disc_real_1: 0.21703450528668686 (+0.0012716891270288244) | > avg_loss_disc_real_2: 0.23459391820598657 (+0.0008583429833533274) | > avg_loss_disc_real_3: 0.23448834284930162 (+0.0050521667154742045) | > avg_loss_disc_real_4: 0.24257141519600237 (-0.0016088101645590491) | > avg_loss_disc_real_5: 0.24555548186033543 (+0.001172042858432698) | > avg_loss_0: 2.5870994346242555 (-0.013646961937487934) | > avg_grad_norm_0: tensor(229.8106, device='cuda:0') (+tensor(60.1658, device='cuda:0')) | > avg_loss_gen: 2.085687823698555 (+0.011225418305733204) | > avg_loss_kl: 2.0433001081708455 (-0.12962950787073968) | > avg_loss_feat: 3.6812976615529664 (+0.0016955959964803924) | > avg_loss_mel: 17.37507524624677 (-0.5393820749202263) | > avg_loss_duration: 1.5737585584882279 (-0.016628817773201243) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.759119436774455 (-0.6727193912989655) | > avg_grad_norm_1: tensor(10045.3633, device='cuda:0') (+tensor(2062.2700, device='cuda:0'))  > EPOCH: 719/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:36:43)   --> TIME: 2025-02-25 15:36:59 -- STEP: 31/72 -- GLOBAL_STEP: 1051800 | > loss_disc: 2.543560266494751 (2.6512295738343266) | > loss_disc_real_0: 0.1317993551492691 (0.1419265653577543) | > loss_disc_real_1: 0.17307332158088684 (0.21642626918131305) | > loss_disc_real_2: 0.32061275839805603 (0.2366110112878584) | > loss_disc_real_3: 0.27681201696395874 (0.23163106172315537) | > loss_disc_real_4: 0.22899149358272552 (0.23955436723847542) | > loss_disc_real_5: 0.23232436180114746 (0.24467750999235338) | > loss_0: 2.543560266494751 (2.6512295738343266) | > grad_norm_0: tensor(147.9536, device='cuda:0') (tensor(203.8125, device='cuda:0')) | > loss_gen: 2.3498551845550537 (2.180358056099185) | > loss_kl: 2.053431272506714 (2.0996082905800106) | > loss_feat: 4.185018539428711 (3.8992477001682406) | > loss_mel: 18.55120086669922 (17.92125751126197) | > loss_duration: 1.5880186557769775 (1.5571745595624369) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.727523803710938 (27.657646240726596) | > grad_norm_1: tensor(7072.7407, device='cuda:0') (tensor(7068.4546, device='cuda:0')) | > current_lr_0: 0.0001828080598063359 | > current_lr_1: 0.0001828080598063359 | > step_time: 0.4429 (0.43734459723195723) | > loader_time: 0.0084 (0.006897141856531943) --> EVAL PERFORMANCE | > avg_loader_time: 0.008588921855872785 (-7.363440285266393e-05) | > avg_step_time: 0.4546412212747923 (-0.0019069725358989986) | > avg_loss_disc: 2.6868492549573872 (+0.09974982033313173) | > avg_loss_disc_real_0: 0.1764419756200112 (+0.0747660027003624) | > avg_loss_disc_real_1: 0.22025953065341627 (+0.003225025366729417) | > avg_loss_disc_real_2: 0.23462328954901493 (+2.937134302835598e-05) | > avg_loss_disc_real_3: 0.23195934400592053 (-0.0025289988433810917) | > avg_loss_disc_real_4: 0.23936334881984012 (-0.0032080663761622497) | > avg_loss_disc_real_5: 0.24274395448221286 (-0.002811527378122569) | > avg_loss_0: 2.6868492549573872 (+0.09974982033313173) | > avg_grad_norm_0: tensor(167.7798, device='cuda:0') (tensor(-62.0308, device='cuda:0')) | > avg_loss_gen: 2.0748617900928985 (-0.010826033605656704) | > avg_loss_kl: 2.1779697025325926 (+0.13466959436174708) | > avg_loss_feat: 3.5367316192304585 (-0.14456604232250791) | > avg_loss_mel: 17.895044514830676 (+0.5199692685839068) | > avg_loss_duration: 1.5902350939495467 (+0.016476535461318775) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.274842759253275 (+0.51572332247882) | > avg_grad_norm_1: tensor(6113.5405, device='cuda:0') (tensor(-3931.8228, device='cuda:0'))  > EPOCH: 720/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:37:20)   --> TIME: 2025-02-25 15:37:26 -- STEP: 9/72 -- GLOBAL_STEP: 1051850 | > loss_disc: 2.623260259628296 (2.601260052786933) | > loss_disc_real_0: 0.07164697349071503 (0.11047226190567017) | > loss_disc_real_1: 0.22638826072216034 (0.23562363452381557) | > loss_disc_real_2: 0.2126556783914566 (0.2510999259021547) | > loss_disc_real_3: 0.19143474102020264 (0.22198570436901516) | > loss_disc_real_4: 0.2353934943675995 (0.25519519713189864) | > loss_disc_real_5: 0.2686319649219513 (0.2605804171827104) | > loss_0: 2.623260259628296 (2.601260052786933) | > grad_norm_0: tensor(133.6782, device='cuda:0') (tensor(158.2017, device='cuda:0')) | > loss_gen: 2.021383285522461 (2.244101232952542) | > loss_kl: 2.1127593517303467 (2.0199695825576782) | > loss_feat: 4.332924842834473 (4.082246436013116) | > loss_mel: 17.40058708190918 (17.719794591267902) | > loss_duration: 1.544549584388733 (1.5482484234703913) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.41220474243164 (27.61436038547092) | > grad_norm_1: tensor(7412.1660, device='cuda:0') (tensor(7038.7334, device='cuda:0')) | > current_lr_0: 0.0001827852087988601 | > current_lr_1: 0.0001827852087988601 | > step_time: 0.4404 (0.4350460370381673) | > loader_time: 0.0056 (0.00674181514316135)  --> TIME: 2025-02-25 15:37:49 -- STEP: 59/72 -- GLOBAL_STEP: 1051900 | > loss_disc: 2.810051202774048 (2.7638828956474693) | > loss_disc_real_0: 0.10373689979314804 (0.1718187466642614) | > loss_disc_real_1: 0.2636215388774872 (0.2474743979588403) | > loss_disc_real_2: 0.2323533296585083 (0.23220954835414886) | > loss_disc_real_3: 0.21884851157665253 (0.23964636558193272) | > loss_disc_real_4: 0.26012223958969116 (0.24708499873088577) | > loss_disc_real_5: 0.2546664774417877 (0.25208135908943113) | > loss_0: 2.810051202774048 (2.7638828956474693) | > grad_norm_0: tensor(17.5624, device='cuda:0') (tensor(122.0269, device='cuda:0')) | > loss_gen: 1.8972809314727783 (2.0344095129077724) | > loss_kl: 2.243159532546997 (2.319113511150166) | > loss_feat: 2.822749614715576 (3.340421470545106) | > loss_mel: 17.004915237426758 (19.329412072391833) | > loss_duration: 1.6240581274032593 (1.5828311160459354) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.5921630859375 (28.606187755778684) | > grad_norm_1: tensor(4588.6294, device='cuda:0') (tensor(5473.8926, device='cuda:0')) | > current_lr_0: 0.0001827852087988601 | > current_lr_1: 0.0001827852087988601 | > step_time: 0.4695 (0.4469309095608986) | > loader_time: 0.0104 (0.008011280480077713) --> EVAL PERFORMANCE | > avg_loader_time: 0.008581285745325223 (-7.636110547562525e-06) | > avg_step_time: 0.4566348069150683 (+0.001993585640276019) | > avg_loss_disc: 2.74994592263665 (+0.06309666767926281) | > avg_loss_disc_real_0: 0.17550747185735632 (-0.0009345037626548869) | > avg_loss_disc_real_1: 0.24395447009256183 (+0.02369493943914555) | > avg_loss_disc_real_2: 0.23301906526928218 (-0.0016042242797327522) | > avg_loss_disc_real_3: 0.2389798231527839 (+0.007020479146863379) | > avg_loss_disc_real_4: 0.24471332804417947 (+0.005349979224339357) | > avg_loss_disc_real_5: 0.2522348018599229 (+0.009490847377710054) | > avg_loss_0: 2.74994592263665 (+0.06309666767926281) | > avg_grad_norm_0: tensor(123.3345, device='cuda:0') (tensor(-44.4453, device='cuda:0')) | > avg_loss_gen: 2.0318741093219166 (-0.04298768077098192) | > avg_loss_kl: 2.3002561095734717 (+0.12228640704087912) | > avg_loss_feat: 3.2893051496693784 (-0.24742646956108016) | > avg_loss_mel: 19.09069164705948 (+1.1956471322288031) | > avg_loss_duration: 1.5917364090261323 (+0.0015013150765856853) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.30386360598282 (+1.0290208467295443) | > avg_grad_norm_1: tensor(5452.3325, device='cuda:0') (tensor(-661.2080, device='cuda:0'))  > EPOCH: 721/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:37:57)   --> TIME: 2025-02-25 15:38:16 -- STEP: 37/72 -- GLOBAL_STEP: 1051950 | > loss_disc: 2.4739155769348145 (2.6429245794141614) | > loss_disc_real_0: 0.16254332661628723 (0.14052914868335467) | > loss_disc_real_1: 0.21971789002418518 (0.22598206110902735) | > loss_disc_real_2: 0.274445116519928 (0.2648215297911618) | > loss_disc_real_3: 0.24325823783874512 (0.2342764143202756) | > loss_disc_real_4: 0.239956334233284 (0.24348789896513964) | > loss_disc_real_5: 0.24984806776046753 (0.24529881292098277) | > loss_0: 2.4739155769348145 (2.6429245794141614) | > grad_norm_0: tensor(78.3298, device='cuda:0') (tensor(124.9170, device='cuda:0')) | > loss_gen: 2.469191789627075 (2.0917739513758065) | > loss_kl: 1.8455702066421509 (1.9733967265567265) | > loss_feat: 4.045182228088379 (3.4432611143266834) | > loss_mel: 17.975067138671875 (17.537607373418034) | > loss_duration: 1.5655348300933838 (1.5456139429195508) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.90054702758789 (26.591653102153057) | > grad_norm_1: tensor(4594.4082, device='cuda:0') (tensor(5962.7559, device='cuda:0')) | > current_lr_0: 0.00018276236064776024 | > current_lr_1: 0.00018276236064776024 | > step_time: 0.4578 (0.4387200200879896) | > loader_time: 0.0079 (0.007244683600760795) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653419118531992 (+7.213337320676957e-05) | > avg_step_time: 0.45596473989352376 (-0.0006700670215445403) | > avg_loss_disc: 2.6729393509072317 (-0.07700657172941838) | > avg_loss_disc_real_0: 0.1516868629505937 (-0.02382060890676263) | > avg_loss_disc_real_1: 0.22615435971340664 (-0.017800110379155187) | > avg_loss_disc_real_2: 0.24897494702272002 (+0.015955881753437845) | > avg_loss_disc_real_3: 0.2361441840168456 (-0.002835639135938317) | > avg_loss_disc_real_4: 0.24215083936570395 (-0.0025624886784755274) | > avg_loss_disc_real_5: 0.24478840806954344 (-0.00744639379037948) | > avg_loss_0: 2.6729393509072317 (-0.07700657172941838) | > avg_grad_norm_0: tensor(130.6321, device='cuda:0') (+tensor(7.2976, device='cuda:0')) | > avg_loss_gen: 2.0398582169707393 (+0.007984107648822736) | > avg_loss_kl: 2.0183473640764262 (-0.2819087454970455) | > avg_loss_feat: 3.3090259055016746 (+0.019720755832296266) | > avg_loss_mel: 17.41409352799536 (-1.6765981190641206) | > avg_loss_duration: 1.578797698020935 (-0.012938711005197279) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.360122680664062 (-1.9437409253187568) | > avg_grad_norm_1: tensor(5486.2358, device='cuda:0') (+tensor(33.9033, device='cuda:0'))  > EPOCH: 722/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:38:34)   --> TIME: 2025-02-25 15:38:43 -- STEP: 15/72 -- GLOBAL_STEP: 1052000 | > loss_disc: 2.529193162918091 (2.6420575618743896) | > loss_disc_real_0: 0.2610955238342285 (0.1519150247176488) | > loss_disc_real_1: 0.18482504785060883 (0.22175069948037465) | > loss_disc_real_2: 0.2224375307559967 (0.23184973895549774) | > loss_disc_real_3: 0.218619704246521 (0.23626181582609812) | > loss_disc_real_4: 0.2425432950258255 (0.2519315709670385) | > loss_disc_real_5: 0.2509263753890991 (0.24432379106680552) | > loss_0: 2.529193162918091 (2.6420575618743896) | > grad_norm_0: tensor(267.1429, device='cuda:0') (tensor(160.0957, device='cuda:0')) | > loss_gen: 2.469129800796509 (2.0814992904663083) | > loss_kl: 2.5853028297424316 (2.345637480417887) | > loss_feat: 3.8458902835845947 (3.4140198389689127) | > loss_mel: 20.822418212890625 (18.471577199300132) | > loss_duration: 1.8370150327682495 (1.6951548020044962) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.559757232666016 (28.007888793945312) | > grad_norm_1: tensor(8576.0010, device='cuda:0') (tensor(5599.1294, device='cuda:0')) | > current_lr_0: 0.00018273951535267927 | > current_lr_1: 0.00018273951535267927 | > step_time: 0.4429 (0.43354279200236) | > loader_time: 0.0059 (0.006504042943318685)  --> TIME: 2025-02-25 15:39:06 -- STEP: 65/72 -- GLOBAL_STEP: 1052050 | > loss_disc: 2.597015619277954 (2.56459521146921) | > loss_disc_real_0: 0.09329639375209808 (0.10985951784711614) | > loss_disc_real_1: 0.24045579135417938 (0.2107373128716762) | > loss_disc_real_2: 0.25950607657432556 (0.2319130363372656) | > loss_disc_real_3: 0.2535233199596405 (0.24625854606811817) | > loss_disc_real_4: 0.253351092338562 (0.2416545943571971) | > loss_disc_real_5: 0.2439211905002594 (0.2470966937450262) | > loss_0: 2.597015619277954 (2.56459521146921) | > grad_norm_0: tensor(175.4563, device='cuda:0') (tensor(177.4654, device='cuda:0')) | > loss_gen: 1.982476830482483 (2.11267849115225) | > loss_kl: 2.1892857551574707 (2.2519675713319054) | > loss_feat: 3.0522985458374023 (3.6555783638587367) | > loss_mel: 17.125411987304688 (18.05814552307128) | > loss_duration: 1.6367135047912598 (1.6498649505468514) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.986186981201172 (27.7282348045936) | > grad_norm_1: tensor(8192.7676, device='cuda:0') (tensor(8196.4863, device='cuda:0')) | > current_lr_0: 0.00018273951535267927 | > current_lr_1: 0.00018273951535267927 | > step_time: 0.5164 (0.44867176642784706) | > loader_time: 0.0118 (0.008463676159198468) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087470873980455 (+9.36682795135077e-05) | > avg_step_time: 0.45510776277998805 (-0.0008569771135357174) | > avg_loss_disc: 2.564939740677954 (-0.10799961022927773) | > avg_loss_disc_real_0: 0.10874218750797524 (-0.04294467544261846) | > avg_loss_disc_real_1: 0.21207692977828038 (-0.014077429935126257) | > avg_loss_disc_real_2: 0.23168074811848116 (-0.01729419890423886) | > avg_loss_disc_real_3: 0.24518078103871413 (+0.009036597021868542) | > avg_loss_disc_real_4: 0.24112878655883627 (-0.0010220528068676793) | > avg_loss_disc_real_5: 0.24598127498593128 (+0.0011928669163878414) | > avg_loss_0: 2.564939740677954 (-0.10799961022927773) | > avg_grad_norm_0: tensor(180.8176, device='cuda:0') (+tensor(50.1856, device='cuda:0')) | > avg_loss_gen: 2.1070099800405364 (+0.06715176306979709) | > avg_loss_kl: 2.2449923619418084 (+0.22664499786538217) | > avg_loss_feat: 3.637947247061931 (+0.3289213415602563) | > avg_loss_mel: 17.972040364440044 (+0.5579468364446853) | > avg_loss_duration: 1.6496165957249387 (+0.07081889770400362) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.611606436716 (+1.251483756051936) | > avg_grad_norm_1: tensor(8255.1084, device='cuda:0') (+tensor(2768.8726, device='cuda:0'))  > EPOCH: 723/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:39:11)   --> TIME: 2025-02-25 15:39:33 -- STEP: 43/72 -- GLOBAL_STEP: 1052100 | > loss_disc: 2.6889030933380127 (2.5508783972540567) | > loss_disc_real_0: 0.0749569982290268 (0.09947436562804289) | > loss_disc_real_1: 0.220792755484581 (0.2206786337286927) | > loss_disc_real_2: 0.20983628928661346 (0.2325714438460594) | > loss_disc_real_3: 0.22159869968891144 (0.23345603533955508) | > loss_disc_real_4: 0.22509922087192535 (0.2444558521342832) | > loss_disc_real_5: 0.23529884219169617 (0.24362459432247074) | > loss_0: 2.6889030933380127 (2.5508783972540567) | > grad_norm_0: tensor(264.7188, device='cuda:0') (tensor(217.2206, device='cuda:0')) | > loss_gen: 1.771222710609436 (2.15005759860194) | > loss_kl: 1.9092251062393188 (1.9600238716879557) | > loss_feat: 3.345906972885132 (3.8001413844352543) | > loss_mel: 16.579105377197266 (17.287962536479142) | > loss_duration: 1.5795464515686035 (1.5588079496871594) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.185007095336914 (26.75699322722679) | > grad_norm_1: tensor(9941.1279, device='cuda:0') (tensor(9954.6865, device='cuda:0')) | > current_lr_0: 0.0001827166729132602 | > current_lr_1: 0.0001827166729132602 | > step_time: 0.4622 (0.45293235224346784) | > loader_time: 0.0096 (0.007646982059922329) --> EVAL PERFORMANCE | > avg_loader_time: 0.008912049548726686 (+0.00016496215068118568) | > avg_step_time: 0.46327210480058695 (+0.008164342020598903) | > avg_loss_disc: 2.5582632850593243 (-0.0066764556186296176) | > avg_loss_disc_real_0: 0.09916384665059372 (-0.009578340857381518) | > avg_loss_disc_real_1: 0.2230388083508317 (+0.01096187857255132) | > avg_loss_disc_real_2: 0.2326333992917773 (+0.0009526511732961263) | > avg_loss_disc_real_3: 0.2336069215351427 (-0.011573859503571426) | > avg_loss_disc_real_4: 0.24358411993778928 (+0.002455333378953012) | > avg_loss_disc_real_5: 0.24456982944213168 (-0.0014114455437995999) | > avg_loss_0: 2.5582632850593243 (-0.0066764556186296176) | > avg_grad_norm_0: tensor(220.5603, device='cuda:0') (+tensor(39.7427, device='cuda:0')) | > avg_loss_gen: 2.140696151155821 (+0.03368617111528449) | > avg_loss_kl: 1.9984236687002048 (-0.24656869324160358) | > avg_loss_feat: 3.749882029815459 (+0.11193478275352797) | > avg_loss_mel: 17.210462462734156 (-0.7615779017058877) | > avg_loss_duration: 1.5801590848976457 (-0.06945751082729301) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67962329488405 (-0.9319831418319495) | > avg_grad_norm_1: tensor(9886.9980, device='cuda:0') (+tensor(1631.8896, device='cuda:0'))  > EPOCH: 724/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:39:49)   --> TIME: 2025-02-25 15:40:00 -- STEP: 21/72 -- GLOBAL_STEP: 1052150 | > loss_disc: 2.442307233810425 (2.534738608769008) | > loss_disc_real_0: 0.0739467516541481 (0.10541817474932898) | > loss_disc_real_1: 0.20054735243320465 (0.22174888068721407) | > loss_disc_real_2: 0.2081916481256485 (0.23065841836588724) | > loss_disc_real_3: 0.22572794556617737 (0.23598905475366683) | > loss_disc_real_4: 0.2345254123210907 (0.24212455394722165) | > loss_disc_real_5: 0.23491999506950378 (0.24760616109484718) | > loss_0: 2.442307233810425 (2.534738608769008) | > grad_norm_0: tensor(98.7604, device='cuda:0') (tensor(227.5980, device='cuda:0')) | > loss_gen: 2.3647899627685547 (2.17330463159652) | > loss_kl: 1.923323392868042 (1.9195505210331507) | > loss_feat: 4.3662285804748535 (3.798499572844732) | > loss_mel: 17.247751235961914 (17.178642908732098) | > loss_duration: 1.5456924438476562 (1.535232044401623) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.447786331176758 (26.605229786464147) | > grad_norm_1: tensor(10242.1553, device='cuda:0') (tensor(10195.7432, device='cuda:0')) | > current_lr_0: 0.00018269383332914603 | > current_lr_1: 0.00018269383332914603 | > step_time: 0.4307 (0.4335035823640369) | > loader_time: 0.0062 (0.006920950753348214)  --> TIME: 2025-02-25 15:40:24 -- STEP: 71/72 -- GLOBAL_STEP: 1052200 | > loss_disc: 2.5967605113983154 (2.566743521623209) | > loss_disc_real_0: 0.13223403692245483 (0.10672282044526557) | > loss_disc_real_1: 0.1835835576057434 (0.22488472957006642) | > loss_disc_real_2: 0.22978611290454865 (0.22952361698721496) | > loss_disc_real_3: 0.19481487572193146 (0.2322329841029476) | > loss_disc_real_4: 0.22168955206871033 (0.2406884381468867) | > loss_disc_real_5: 0.21753834187984467 (0.24420096273993103) | > loss_0: 2.5967605113983154 (2.566743521623209) | > grad_norm_0: tensor(303.2098, device='cuda:0') (tensor(235.5158, device='cuda:0')) | > loss_gen: 1.908252239227295 (2.1276348375938303) | > loss_kl: 1.9136154651641846 (1.9705850440011898) | > loss_feat: 3.8455915451049805 (3.6962621178425534) | > loss_mel: 16.595748901367188 (17.01773956460012) | > loss_duration: 1.5724833011627197 (1.5709009489543002) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.835691452026367 (26.383122618769256) | > grad_norm_1: tensor(9893.8174, device='cuda:0') (tensor(10120.9473, device='cuda:0')) | > current_lr_0: 0.00018269383332914603 | > current_lr_1: 0.00018269383332914603 | > step_time: 0.4873 (0.4544035683215504) | > loader_time: 0.008 (0.008693201441160389) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693201441160389 (-0.0002188481075662968) | > avg_step_time: 0.4544035683215504 (-0.00886853647903657) | > avg_loss_disc: 2.566743521623209 (+0.008480236563884525) | > avg_loss_disc_real_0: 0.10672282044526557 (+0.007558973794671853) | > avg_loss_disc_real_1: 0.22488472957006642 (+0.001845921219234714) | > avg_loss_disc_real_2: 0.22952361698721496 (-0.003109782304562325) | > avg_loss_disc_real_3: 0.2322329841029476 (-0.0013739374321951048) | > avg_loss_disc_real_4: 0.2406884381468867 (-0.0028956817909025767) | > avg_loss_disc_real_5: 0.24420096273993103 (-0.0003688667022006542) | > avg_loss_0: 2.566743521623209 (+0.008480236563884525) | > avg_grad_norm_0: tensor(235.5158, device='cuda:0') (+tensor(14.9555, device='cuda:0')) | > avg_loss_gen: 2.1276348375938303 (-0.013061313561990584) | > avg_loss_kl: 1.9705850440011898 (-0.02783862469901499) | > avg_loss_feat: 3.6962621178425534 (-0.0536199119729055) | > avg_loss_mel: 17.01773956460012 (-0.1927228981340363) | > avg_loss_duration: 1.5709009489543002 (-0.009258135943345458) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.383122618769256 (-0.2965006761147926) | > avg_grad_norm_1: tensor(10120.9473, device='cuda:0') (+tensor(233.9492, device='cuda:0'))  > EPOCH: 725/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:40:26)   --> TIME: 2025-02-25 15:40:50 -- STEP: 49/72 -- GLOBAL_STEP: 1052250 | > loss_disc: 2.6737141609191895 (2.5587808501963702) | > loss_disc_real_0: 0.08135595917701721 (0.10511297960670626) | > loss_disc_real_1: 0.23091058433055878 (0.2220790802824254) | > loss_disc_real_2: 0.21887050569057465 (0.2298537231221491) | > loss_disc_real_3: 0.21844227612018585 (0.2330567091703415) | > loss_disc_real_4: 0.24488618969917297 (0.24121719051380547) | > loss_disc_real_5: 0.23806540668010712 (0.2433438033473735) | > loss_0: 2.6737141609191895 (2.5587808501963702) | > grad_norm_0: tensor(404.4278, device='cuda:0') (tensor(238.2596, device='cuda:0')) | > loss_gen: 2.013775110244751 (2.1557966519375245) | > loss_kl: 1.9000593423843384 (1.9477495490288248) | > loss_feat: 4.087366104125977 (3.799573513926292) | > loss_mel: 17.540027618408203 (16.976615497044158) | > loss_duration: 1.6269755363464355 (1.5461253463005533) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.168203353881836 (26.42586038550552) | > grad_norm_1: tensor(12818.1807, device='cuda:0') (tensor(10919.6553, device='cuda:0')) | > current_lr_0: 0.00018267099659997988 | > current_lr_1: 0.00018267099659997988 | > step_time: 0.4467 (0.4437812153173953) | > loader_time: 0.0098 (0.0077143299336336095) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640433700991348 (-5.276774016904108e-05) | > avg_step_time: 0.45552255066347797 (+0.001118982341927588) | > avg_loss_disc: 2.562120807002967 (-0.004622714620241997) | > avg_loss_disc_real_0: 0.10656543296407646 (-0.00015738748118911128) | > avg_loss_disc_real_1: 0.2227547374409689 (-0.0021299921290975077) | > avg_loss_disc_real_2: 0.230456455282762 (+0.0009328382955470504) | > avg_loss_disc_real_3: 0.234109941502692 (+0.001876957399744389) | > avg_loss_disc_real_4: 0.24176783901704868 (+0.0010794008701619784) | > avg_loss_disc_real_5: 0.24422731365956052 (+2.6350919629491987e-05) | > avg_loss_0: 2.562120807002967 (-0.004622714620241997) | > avg_grad_norm_0: tensor(239.0404, device='cuda:0') (+tensor(3.5245, device='cuda:0')) | > avg_loss_gen: 2.149477931815135 (+0.021843094221304593) | > avg_loss_kl: 1.9835311728464047 (+0.012946128845214844) | > avg_loss_feat: 3.738894022686381 (+0.042631904843827506) | > avg_loss_mel: 16.982266976799764 (-0.03547258780035634) | > avg_loss_duration: 1.5656033519288184 (-0.005297597025481826) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.419773262991033 (+0.03665064422177622) | > avg_grad_norm_1: tensor(10803.6006, device='cuda:0') (+tensor(682.6533, device='cuda:0'))  > EPOCH: 726/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:41:03)   --> TIME: 2025-02-25 15:41:17 -- STEP: 27/72 -- GLOBAL_STEP: 1052300 | > loss_disc: 2.572169303894043 (2.5206688774956585) | > loss_disc_real_0: 0.12315670400857925 (0.09573644054708658) | > loss_disc_real_1: 0.2347014993429184 (0.2149370903218234) | > loss_disc_real_2: 0.23772898316383362 (0.22890212028114884) | > loss_disc_real_3: 0.2575214207172394 (0.22975317692315136) | > loss_disc_real_4: 0.2532086670398712 (0.2381777857188825) | > loss_disc_real_5: 0.26443397998809814 (0.23906772942454727) | > loss_0: 2.572169303894043 (2.5206688774956585) | > grad_norm_0: tensor(234.4972, device='cuda:0') (tensor(248.5359, device='cuda:0')) | > loss_gen: 2.4938414096832275 (2.1993294468632447) | > loss_kl: 1.9796390533447266 (1.964184703650298) | > loss_feat: 3.9722394943237305 (3.929304281870524) | > loss_mel: 15.857433319091797 (17.09481196933322) | > loss_duration: 1.5198278427124023 (1.520649508193687) | > amp_scaler: 16.0 (15.407407407407407) | > loss_1: 25.822978973388672 (26.70827971564399) | > grad_norm_1: tensor(11103.5801, device='cuda:0') (tensor(11301.2080, device='cuda:0')) | > current_lr_0: 0.00018264816272540486 | > current_lr_1: 0.00018264816272540486 | > step_time: 0.4538 (0.4508840861143889) | > loader_time: 0.0102 (0.006929256297923901) --> EVAL PERFORMANCE | > avg_loader_time: 0.008662693937059861 (+2.2260236068513337e-05) | > avg_step_time: 0.4624294294437892 (+0.00690687878031121) | > avg_loss_disc: 2.6443304243222086 (+0.0822096173192417) | > avg_loss_disc_real_0: 0.12309929408447844 (+0.016533861120401977) | > avg_loss_disc_real_1: 0.2292163383058259 (+0.006461600864857003) | > avg_loss_disc_real_2: 0.23272748354455114 (+0.0022710282617891264) | > avg_loss_disc_real_3: 0.23156859799170157 (-0.002541343510990418) | > avg_loss_disc_real_4: 0.24648709322365236 (+0.004719254206603685) | > avg_loss_disc_real_5: 0.24098899876567678 (-0.003238314893883737) | > avg_loss_0: 2.6443304243222086 (+0.0822096173192417) | > avg_grad_norm_0: tensor(257.2267, device='cuda:0') (+tensor(18.1863, device='cuda:0')) | > avg_loss_gen: 2.1016942934251164 (-0.047783638390018535) | > avg_loss_kl: 2.025494531846382 (+0.041963358999977496) | > avg_loss_feat: 3.667014733166762 (-0.07187928951961897) | > avg_loss_mel: 17.0035901808403 (+0.02132320404053445) | > avg_loss_duration: 1.5637111126537053 (-0.0018922392751130612) | > avg_amp_scaler: 11.154929577464788 (+3.154929577464788) | > avg_loss_1: 26.361504809957154 (-0.05826845303387884) | > avg_grad_norm_1: tensor(9742.5137, device='cuda:0') (tensor(-1061.0869, device='cuda:0'))  > EPOCH: 727/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:41:40)   --> TIME: 2025-02-25 15:41:44 -- STEP: 5/72 -- GLOBAL_STEP: 1052350 | > loss_disc: 2.5784952640533447 (2.555615282058716) | > loss_disc_real_0: 0.084668830037117 (0.08310639336705208) | > loss_disc_real_1: 0.24483411014080048 (0.22001955211162566) | > loss_disc_real_2: 0.22847791016101837 (0.2263866275548935) | > loss_disc_real_3: 0.21215595304965973 (0.23087974786758422) | > loss_disc_real_4: 0.2289421111345291 (0.22635380029678345) | > loss_disc_real_5: 0.24118079245090485 (0.24580033123493195) | > loss_0: 2.5784952640533447 (2.555615282058716) | > grad_norm_0: tensor(266.4984, device='cuda:0') (tensor(261.7147, device='cuda:0')) | > loss_gen: 2.1434199810028076 (2.1264254808425904) | > loss_kl: 2.0975842475891113 (1.9618663549423219) | > loss_feat: 3.4960994720458984 (3.7410879611968992) | > loss_mel: 16.320589065551758 (16.901963996887208) | > loss_duration: 1.483563780784607 (1.5326643943786622) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.541255950927734 (26.264007568359375) | > grad_norm_1: tensor(11159.8066, device='cuda:0') (tensor(11575.4082, device='cuda:0')) | > current_lr_0: 0.0001826253317050642 | > current_lr_1: 0.0001826253317050642 | > step_time: 0.4313 (0.4350714206695557) | > loader_time: 0.0063 (0.006875514984130859)  --> TIME: 2025-02-25 15:42:08 -- STEP: 55/72 -- GLOBAL_STEP: 1052400 | > loss_disc: 2.5391907691955566 (2.568392523852262) | > loss_disc_real_0: 0.12576188147068024 (0.11115426231514323) | > loss_disc_real_1: 0.21964861452579498 (0.21960850358009337) | > loss_disc_real_2: 0.21727363765239716 (0.22796415198932996) | > loss_disc_real_3: 0.24361097812652588 (0.2336564074863087) | > loss_disc_real_4: 0.25731849670410156 (0.23940237625078722) | > loss_disc_real_5: 0.254608690738678 (0.24355891807512803) | > loss_0: 2.5391907691955566 (2.568392523852262) | > grad_norm_0: tensor(231.0554, device='cuda:0') (tensor(244.1187, device='cuda:0')) | > loss_gen: 2.1723945140838623 (2.1439594398845325) | > loss_kl: 1.983560562133789 (2.0000584602355955) | > loss_feat: 3.700993537902832 (3.7504739891399037) | > loss_mel: 16.880966186523438 (16.8441578951749) | > loss_duration: 1.6129258871078491 (1.5491507335142656) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.350841522216797 (26.287800494107334) | > grad_norm_1: tensor(8520.1270, device='cuda:0') (tensor(10707.0713, device='cuda:0')) | > current_lr_0: 0.0001826253317050642 | > current_lr_1: 0.0001826253317050642 | > step_time: 0.4771 (0.45282032273032446) | > loader_time: 0.0092 (0.007941263372247868) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866052466379085 (-2.169273269011368e-06) | > avg_step_time: 0.4621312450355207 (-0.0002981844082684537) | > avg_loss_disc: 2.5801016545631517 (-0.06422876975905689) | > avg_loss_disc_real_0: 0.11241084683529089 (-0.01068844724918755) | > avg_loss_disc_real_1: 0.22238668484587065 (-0.006829653459955259) | > avg_loss_disc_real_2: 0.22986398280506404 (-0.0028635007394871015) | > avg_loss_disc_real_3: 0.2338181280753982 (+0.0022495300836966325) | > avg_loss_disc_real_4: 0.24065612865165925 (-0.005830964571993114) | > avg_loss_disc_real_5: 0.24325466722669736 (+0.0022656684610205813) | > avg_loss_0: 2.5801016545631517 (-0.06422876975905689) | > avg_grad_norm_0: tensor(248.7165, device='cuda:0') (tensor(-8.5102, device='cuda:0')) | > avg_loss_gen: 2.134708950217341 (+0.033014656792224706) | > avg_loss_kl: 2.0335496815157614 (+0.008055149669379258) | > avg_loss_feat: 3.6934665088922207 (+0.02645177572545876) | > avg_loss_mel: 16.86611152702654 (-0.13747865381375846) | > avg_loss_duration: 1.5633624657778673 (-0.00034864687583802123) | > avg_amp_scaler: 8.0 (-3.154929577464788) | > avg_loss_1: 26.291199052837534 (-0.07030575711961973) | > avg_grad_norm_1: tensor(10667.8457, device='cuda:0') (+tensor(925.3320, device='cuda:0'))  > EPOCH: 728/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:42:18)   --> TIME: 2025-02-25 15:42:34 -- STEP: 33/72 -- GLOBAL_STEP: 1052450 | > loss_disc: 2.560718297958374 (2.5547583536668257) | > loss_disc_real_0: 0.1005907878279686 (0.10470880376118602) | > loss_disc_real_1: 0.2258106917142868 (0.2212606275623495) | > loss_disc_real_2: 0.2453758716583252 (0.2319201903812813) | > loss_disc_real_3: 0.1844172477722168 (0.23144914571082953) | > loss_disc_real_4: 0.2211911678314209 (0.2403597944613659) | > loss_disc_real_5: 0.22476455569267273 (0.24502877394358316) | > loss_0: 2.560718297958374 (2.5547583536668257) | > grad_norm_0: tensor(248.6124, device='cuda:0') (tensor(260.5168, device='cuda:0')) | > loss_gen: 2.0191407203674316 (2.1963701392665054) | > loss_kl: 1.998900055885315 (1.9806219917355161) | > loss_feat: 4.078969955444336 (3.8796188181096856) | > loss_mel: 17.227006912231445 (17.145722360321972) | > loss_duration: 1.5277099609375 (1.5250864028930664) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.851726531982422 (26.7274196509159) | > grad_norm_1: tensor(12159.6729, device='cuda:0') (tensor(11594.0752, device='cuda:0')) | > current_lr_0: 0.00018260250353860106 | > current_lr_1: 0.00018260250353860106 | > step_time: 0.4598 (0.4356879176515521) | > loader_time: 0.0078 (0.00709733096036044) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568864473154848 (-9.166019063600213e-05) | > avg_step_time: 0.4540831203192053 (-0.008048124716315441) | > avg_loss_disc: 2.589437951504345 (+0.0093362969411932) | > avg_loss_disc_real_0: 0.1101453598641174 (-0.002265486971173486) | > avg_loss_disc_real_1: 0.22724419328528392 (+0.004857508439413266) | > avg_loss_disc_real_2: 0.23561160287386934 (+0.005747620068805304) | > avg_loss_disc_real_3: 0.23585333014038248 (+0.0020352020649842817) | > avg_loss_disc_real_4: 0.2420378367246037 (+0.0013817080729444375) | > avg_loss_disc_real_5: 0.246237125195248 (+0.0029824579685506503) | > avg_loss_0: 2.589437951504345 (+0.0093362969411932) | > avg_grad_norm_0: tensor(264.1322, device='cuda:0') (+tensor(15.4157, device='cuda:0')) | > avg_loss_gen: 2.172818506267709 (+0.03810955605036792) | > avg_loss_kl: 2.045482245969101 (+0.011932564453339456) | > avg_loss_feat: 3.802942376741221 (+0.10947586784900043) | > avg_loss_mel: 17.09002860163299 (+0.22391707460645094) | > avg_loss_duration: 1.5631882002655888 (-0.00017426551227850418) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67446007527096 (+0.38326102243342675) | > avg_grad_norm_1: tensor(11230.3125, device='cuda:0') (+tensor(562.4668, device='cuda:0'))  > EPOCH: 729/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:42:55)   --> TIME: 2025-02-25 15:43:01 -- STEP: 11/72 -- GLOBAL_STEP: 1052500 | > loss_disc: 2.608154296875 (2.5587058500810103) | > loss_disc_real_0: 0.0754917562007904 (0.10941608250141144) | > loss_disc_real_1: 0.24451012909412384 (0.21501167524944653) | > loss_disc_real_2: 0.23916147649288177 (0.21747169982303272) | > loss_disc_real_3: 0.2505045235157013 (0.23358859663659876) | > loss_disc_real_4: 0.24448475241661072 (0.24015473235737195) | > loss_disc_real_5: 0.2097163051366806 (0.2392189990390431) | > loss_0: 2.608154296875 (2.5587058500810103) | > grad_norm_0: tensor(236.3611, device='cuda:0') (tensor(240.5167, device='cuda:0')) | > loss_gen: 2.0373106002807617 (2.169528137553822) | > loss_kl: 1.906205654144287 (1.962763255292719) | > loss_feat: 3.610323905944824 (3.864538626237349) | > loss_mel: 16.886152267456055 (17.026653983376242) | > loss_duration: 1.508800983428955 (1.5220982811667703) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.94879150390625 (26.545581991022285) | > grad_norm_1: tensor(10615.0723, device='cuda:0') (tensor(10653.7344, device='cuda:0')) | > current_lr_0: 0.00018257967822565873 | > current_lr_1: 0.00018257967822565873 | > step_time: 0.4372 (0.43544507026672363) | > loader_time: 0.0059 (0.006559198552911932)  --> TIME: 2025-02-25 15:43:25 -- STEP: 61/72 -- GLOBAL_STEP: 1052550 | > loss_disc: 2.5875861644744873 (2.570048484645906) | > loss_disc_real_0: 0.08652795851230621 (0.10942590542015482) | > loss_disc_real_1: 0.19341391324996948 (0.2229357926083393) | > loss_disc_real_2: 0.2644985020160675 (0.22555628834200686) | > loss_disc_real_3: 0.24527780711650848 (0.23421748373352114) | > loss_disc_real_4: 0.24810220301151276 (0.24219935937005965) | > loss_disc_real_5: 0.25093600153923035 (0.24508352479973777) | > loss_0: 2.5875861644744873 (2.570048484645906) | > grad_norm_0: tensor(68.7246, device='cuda:0') (tensor(246.7776, device='cuda:0')) | > loss_gen: 2.3556525707244873 (2.1738700358594047) | > loss_kl: 2.0576858520507812 (2.0355163148192115) | > loss_feat: 4.174131870269775 (3.793199152242942) | > loss_mel: 16.692672729492188 (16.944188633903135) | > loss_duration: 1.6097204685211182 (1.552234221677311) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.889862060546875 (26.49900824124696) | > grad_norm_1: tensor(10145.9229, device='cuda:0') (tensor(10969.4932, device='cuda:0')) | > current_lr_0: 0.00018257967822565873 | > current_lr_1: 0.00018257967822565873 | > step_time: 0.4609 (0.4474373176449635) | > loader_time: 0.0107 (0.008143573510842246) --> EVAL PERFORMANCE | > avg_loader_time: 0.008623462327769105 (+5.459785461425781e-05) | > avg_step_time: 0.45731602252369197 (+0.0032329022044866873) | > avg_loss_disc: 2.5770247889236666 (-0.01241316258067826) | > avg_loss_disc_real_0: 0.1119699138675777 (+0.0018245540034602936) | > avg_loss_disc_real_1: 0.2262086079154216 (-0.001035585369862313) | > avg_loss_disc_real_2: 0.2287276653756558 (-0.006883937498213533) | > avg_loss_disc_real_3: 0.23454290271644862 (-0.0013104274239338631) | > avg_loss_disc_real_4: 0.24360706394826862 (+0.0015692272236649263) | > avg_loss_disc_real_5: 0.24571613631617856 (-0.0005209888790694561) | > avg_loss_0: 2.5770247889236666 (-0.01241316258067826) | > avg_grad_norm_0: tensor(250.9474, device='cuda:0') (tensor(-13.1848, device='cuda:0')) | > avg_loss_gen: 2.1825842168969163 (+0.00976571062920728) | > avg_loss_kl: 2.056014643588537 (+0.01053239761943603) | > avg_loss_feat: 3.78612241274874 (-0.016819963992481313) | > avg_loss_mel: 17.00887005765674 (-0.081158543976251) | > avg_loss_duration: 1.561792685951985 (-0.0013955143136037407) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.59538389931262 (-0.07907617595834182) | > avg_grad_norm_1: tensor(11011.8369, device='cuda:0') (tensor(-218.4756, device='cuda:0'))  > EPOCH: 730/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:43:32)   --> TIME: 2025-02-25 15:43:52 -- STEP: 39/72 -- GLOBAL_STEP: 1052600 | > loss_disc: 2.428424835205078 (2.62385046787751) | > loss_disc_real_0: 0.09520310163497925 (0.1290429355815435) | > loss_disc_real_1: 0.2185615599155426 (0.23020824598960388) | > loss_disc_real_2: 0.23311693966388702 (0.22710200723929283) | > loss_disc_real_3: 0.2396821677684784 (0.23653301061728063) | > loss_disc_real_4: 0.2296430468559265 (0.2446943624661519) | > loss_disc_real_5: 0.24179579317569733 (0.24524003572953054) | > loss_0: 2.428424835205078 (2.62385046787751) | > grad_norm_0: tensor(247.2269, device='cuda:0') (tensor(263.9511, device='cuda:0')) | > loss_gen: 2.191967010498047 (2.17459369622744) | > loss_kl: 2.139359951019287 (2.0861920943626995) | > loss_feat: 3.8829824924468994 (3.7638131349514694) | > loss_mel: 17.20360565185547 (17.098487364940155) | > loss_duration: 1.544198989868164 (1.5320111085207033) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.962114334106445 (26.65509747236203) | > grad_norm_1: tensor(10386.9854, device='cuda:0') (tensor(10610.0938, device='cuda:0')) | > current_lr_0: 0.0001825568557658805 | > current_lr_1: 0.0001825568557658805 | > step_time: 0.4513 (0.4434348253103403) | > loader_time: 0.0085 (0.007403911688388922) --> EVAL PERFORMANCE | > avg_loader_time: 0.008664960592565399 (+4.1498264796293455e-05) | > avg_step_time: 0.4596981162756262 (+0.002382093751934211) | > avg_loss_disc: 2.61435088641207 (+0.03732609748840332) | > avg_loss_disc_real_0: 0.1256409190385275 (+0.01367100517094981) | > avg_loss_disc_real_1: 0.22611836627335616 (-9.024164206544727e-05) | > avg_loss_disc_real_2: 0.2286521742041682 (-7.549117148761275e-05) | > avg_loss_disc_real_3: 0.23339456928447938 (-0.0011483334319692395) | > avg_loss_disc_real_4: 0.24316570603511703 (-0.00044135791315158546) | > avg_loss_disc_real_5: 0.24431501161044752 (-0.0014011247057310328) | > avg_loss_0: 2.61435088641207 (+0.03732609748840332) | > avg_grad_norm_0: tensor(255.6564, device='cuda:0') (+tensor(4.7090, device='cuda:0')) | > avg_loss_gen: 2.156445459580756 (-0.026138757316160444) | > avg_loss_kl: 2.1089124041543887 (+0.052897760565851826) | > avg_loss_feat: 3.703925582724558 (-0.08219683002418199) | > avg_loss_mel: 17.064089197508043 (+0.05521913985130311) | > avg_loss_duration: 1.5652718426476062 (+0.003479156695621155) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.59864428345586 (+0.0032603841432425895) | > avg_grad_norm_1: tensor(10395.6777, device='cuda:0') (tensor(-616.1592, device='cuda:0'))  > EPOCH: 731/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:44:09)   --> TIME: 2025-02-25 15:44:19 -- STEP: 17/72 -- GLOBAL_STEP: 1052650 | > loss_disc: 2.795666456222534 (2.637148758944343) | > loss_disc_real_0: 0.08861684799194336 (0.09784749851507299) | > loss_disc_real_1: 0.22023291885852814 (0.2382200053509544) | > loss_disc_real_2: 0.18864591419696808 (0.22479148121441112) | > loss_disc_real_3: 0.28103888034820557 (0.249075529329917) | > loss_disc_real_4: 0.30286896228790283 (0.2564186480115442) | > loss_disc_real_5: 0.26844149827957153 (0.25043897155453176) | > loss_0: 2.795666456222534 (2.637148758944343) | > grad_norm_0: tensor(412.6425, device='cuda:0') (tensor(274.1512, device='cuda:0')) | > loss_gen: 2.047928810119629 (2.1811497351702522) | > loss_kl: 2.2842519283294678 (2.0254225590649773) | > loss_feat: 3.671806812286377 (3.9107093670788933) | > loss_mel: 17.429454803466797 (17.178672566133386) | > loss_duration: 1.5611839294433594 (1.5139443173128015) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.994625091552734 (26.8098986008588) | > grad_norm_1: tensor(11265.7783, device='cuda:0') (tensor(11582.2051, device='cuda:0')) | > current_lr_0: 0.00018253403615890975 | > current_lr_1: 0.00018253403615890975 | > step_time: 0.4397 (0.4369395059697768) | > loader_time: 0.0061 (0.006774565752814798)  --> TIME: 2025-02-25 15:44:43 -- STEP: 67/72 -- GLOBAL_STEP: 1052700 | > loss_disc: 2.498952865600586 (2.619417329332721) | > loss_disc_real_0: 0.10985890030860901 (0.10869480791821409) | > loss_disc_real_1: 0.227375790476799 (0.2313882352700874) | > loss_disc_real_2: 0.26462504267692566 (0.23163148471668585) | > loss_disc_real_3: 0.2406521439552307 (0.24343858014291792) | > loss_disc_real_4: 0.24624685943126678 (0.24391726148662282) | > loss_disc_real_5: 0.23941382765769958 (0.24591143785127953) | > loss_0: 2.498952865600586 (2.619417329332721) | > grad_norm_0: tensor(117.4523, device='cuda:0') (tensor(253.4342, device='cuda:0')) | > loss_gen: 2.431417226791382 (2.156453792728595) | > loss_kl: 2.22175931930542 (2.0921297891816106) | > loss_feat: 4.169740200042725 (3.7136525929863775) | > loss_mel: 17.30729103088379 (16.95110054870151) | > loss_duration: 1.6564815044403076 (1.5606581150595822) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.78668975830078 (26.473994810189772) | > grad_norm_1: tensor(12050.4229, device='cuda:0') (tensor(11206.2871, device='cuda:0')) | > current_lr_0: 0.00018253403615890975 | > current_lr_1: 0.00018253403615890975 | > step_time: 0.519 (0.4531644173522494) | > loader_time: 0.0122 (0.008477388922847919) --> EVAL PERFORMANCE | > avg_loader_time: 0.008657233815797617 (-7.726776767781607e-06) | > avg_step_time: 0.457058009966998 (-0.002640106308628154) | > avg_loss_disc: 2.6238935195224378 (+0.009542633110367849) | > avg_loss_disc_real_0: 0.10967726743137332 (-0.015963651607154183) | > avg_loss_disc_real_1: 0.23217333034730295 (+0.006054964073946789) | > avg_loss_disc_real_2: 0.2316981698845474 (+0.003045995680379193) | > avg_loss_disc_real_3: 0.2426910530513441 (+0.009296483766864733) | > avg_loss_disc_real_4: 0.2437784430426611 (+0.0006127370075440786) | > avg_loss_disc_real_5: 0.24566318871269763 (+0.0013481771022501032) | > avg_loss_0: 2.6238935195224378 (+0.009542633110367849) | > avg_grad_norm_0: tensor(253.1736, device='cuda:0') (tensor(-2.4828, device='cuda:0')) | > avg_loss_gen: 2.152179325130624 (-0.004266134450131798) | > avg_loss_kl: 2.0982076819513895 (-0.010704722202999228) | > avg_loss_feat: 3.698397156218408 (-0.0055284265061499305) | > avg_loss_mel: 16.942587798749905 (-0.12150139875813792) | > avg_loss_duration: 1.5637669848724147 (-0.0015048577751914927) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.4551388915156 (-0.14350539194026268) | > avg_grad_norm_1: tensor(11110.4648, device='cuda:0') (+tensor(714.7871, device='cuda:0'))  > EPOCH: 732/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:44:47)   --> TIME: 2025-02-25 15:45:09 -- STEP: 45/72 -- GLOBAL_STEP: 1052750 | > loss_disc: 3.038991928100586 (2.7167144722408723) | > loss_disc_real_0: 0.5596418380737305 (0.16933503490355276) | > loss_disc_real_1: 0.24366310238838196 (0.2365882803996404) | > loss_disc_real_2: 0.23753899335861206 (0.2397290097342597) | > loss_disc_real_3: 0.30314719676971436 (0.24100885258780586) | > loss_disc_real_4: 0.24391144514083862 (0.24448465009530385) | > loss_disc_real_5: 0.290897935628891 (0.25122013489405315) | > loss_0: 3.038991928100586 (2.7167144722408723) | > grad_norm_0: tensor(356.1667, device='cuda:0') (tensor(195.1973, device='cuda:0')) | > loss_gen: 2.807600259780884 (2.1675288253360323) | > loss_kl: 2.3541648387908936 (2.3873403045866217) | > loss_feat: 3.306471586227417 (3.479237206776937) | > loss_mel: 18.929365158081055 (18.55678537156847) | > loss_duration: 1.594112753868103 (1.5873990482754177) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.991714477539062 (28.178290769788955) | > grad_norm_1: tensor(4467.1104, device='cuda:0') (tensor(7485.9497, device='cuda:0')) | > current_lr_0: 0.0001825112194043899 | > current_lr_1: 0.0001825112194043899 | > step_time: 0.4715 (0.441508764690823) | > loader_time: 0.0088 (0.007471328311496311) --> EVAL PERFORMANCE | > avg_loader_time: 0.008601709150932205 (-5.552466486541223e-05) | > avg_step_time: 0.4538003021562603 (-0.0032577078107376978) | > avg_loss_disc: 2.7788305584813506 (+0.15493703895891286) | > avg_loss_disc_real_0: 0.17766575916657146 (+0.06798849173519814) | > avg_loss_disc_real_1: 0.2453416732835098 (+0.013168342936206862) | > avg_loss_disc_real_2: 0.24742442915137386 (+0.01572625926682647) | > avg_loss_disc_real_3: 0.23639490843658717 (-0.006296144614756943) | > avg_loss_disc_real_4: 0.2417753989427862 (-0.0020030440998748955) | > avg_loss_disc_real_5: 0.2469363911470898 (+0.0012732024343921622) | > avg_loss_0: 2.7788305584813506 (+0.15493703895891286) | > avg_grad_norm_0: tensor(187.5289, device='cuda:0') (tensor(-65.6447, device='cuda:0')) | > avg_loss_gen: 2.074392704896524 (-0.07778662023410021) | > avg_loss_kl: 2.388181748524518 (+0.2899740665731283) | > avg_loss_feat: 3.316662301479931 (-0.3817348547384771) | > avg_loss_mel: 18.607228614914586 (+1.664640816164681) | > avg_loss_duration: 1.6028711862967049 (+0.039104201424290164) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.989336577939316 (+1.5341976864237168) | > avg_grad_norm_1: tensor(6226.1357, device='cuda:0') (tensor(-4884.3291, device='cuda:0'))  > EPOCH: 733/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:45:24)   --> TIME: 2025-02-25 15:45:36 -- STEP: 23/72 -- GLOBAL_STEP: 1052800 | > loss_disc: 2.851175546646118 (2.694234910218612) | > loss_disc_real_0: 0.22149658203125 (0.14729424906165706) | > loss_disc_real_1: 0.24586068093776703 (0.23245688819366953) | > loss_disc_real_2: 0.25473320484161377 (0.24197878526604694) | > loss_disc_real_3: 0.23566600680351257 (0.23077483669571255) | > loss_disc_real_4: 0.2783004641532898 (0.25018989197585895) | > loss_disc_real_5: 0.24397408962249756 (0.24686147013436194) | > loss_0: 2.851175546646118 (2.694234910218612) | > grad_norm_0: tensor(193.4527, device='cuda:0') (tensor(187.6893, device='cuda:0')) | > loss_gen: 1.8577585220336914 (2.025845688322316) | > loss_kl: 2.162614107131958 (2.288199207057123) | > loss_feat: 2.9443306922912598 (3.4242004000622295) | > loss_mel: 18.54020118713379 (18.61839128577191) | > loss_duration: 1.5925105810165405 (1.5668840615645698) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.097415924072266 (27.92352054430091) | > grad_norm_1: tensor(602.9666, device='cuda:0') (tensor(6285.7949, device='cuda:0')) | > current_lr_0: 0.00018248840550196433 | > current_lr_1: 0.00018248840550196433 | > step_time: 0.4472 (0.44142962538677716) | > loader_time: 0.0063 (0.007087593493254289) --> EVAL PERFORMANCE | > avg_loader_time: 0.008889477017899633 (+0.00028776786696742794) | > avg_step_time: 0.4613861769018039 (+0.007585874745543564) | > avg_loss_disc: 2.695203190118494 (-0.08362736836285656) | > avg_loss_disc_real_0: 0.16311743073690102 (-0.014548328429670437) | > avg_loss_disc_real_1: 0.240128562064238 (-0.005213111219271815) | > avg_loss_disc_real_2: 0.2333618142235447 (-0.014062614927829148) | > avg_loss_disc_real_3: 0.23505028809460116 (-0.0013446203419860137) | > avg_loss_disc_real_4: 0.24949049676807833 (+0.0077150978252921165) | > avg_loss_disc_real_5: 0.24818951584083934 (+0.0012531246937495522) | > avg_loss_0: 2.695203190118494 (-0.08362736836285656) | > avg_grad_norm_0: tensor(176.8383, device='cuda:0') (tensor(-10.6906, device='cuda:0')) | > avg_loss_gen: 2.0811756647808446 (+0.006782959884320761) | > avg_loss_kl: 2.539089572261756 (+0.1509078237372381) | > avg_loss_feat: 3.444962521673928 (+0.12830022019399712) | > avg_loss_mel: 19.667601894324935 (+1.0603732794103493) | > avg_loss_duration: 1.643756539049283 (+0.0408853527525781) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.37658608127648 (+1.3872495033371628) | > avg_grad_norm_1: tensor(6481.2188, device='cuda:0') (+tensor(255.0830, device='cuda:0'))  > EPOCH: 734/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:46:01)   --> TIME: 2025-02-25 15:46:04 -- STEP: 1/72 -- GLOBAL_STEP: 1052850 | > loss_disc: 2.449232578277588 (2.449232578277588) | > loss_disc_real_0: 0.11291921883821487 (0.11291921883821487) | > loss_disc_real_1: 0.21527522802352905 (0.21527522802352905) | > loss_disc_real_2: 0.20673641562461853 (0.20673641562461853) | > loss_disc_real_3: 0.161865234375 (0.161865234375) | > loss_disc_real_4: 0.19648191332817078 (0.19648191332817078) | > loss_disc_real_5: 0.20651362836360931 (0.20651362836360931) | > loss_0: 2.449232578277588 (2.449232578277588) | > grad_norm_0: tensor(66.5163, device='cuda:0') (tensor(66.5163, device='cuda:0')) | > loss_gen: 2.3506219387054443 (2.3506219387054443) | > loss_kl: 2.0859456062316895 (2.0859456062316895) | > loss_feat: 4.200173377990723 (4.200173377990723) | > loss_mel: 19.483386993408203 (19.483386993408203) | > loss_duration: 1.6294748783111572 (1.6294748783111572) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.749603271484375 (29.749603271484375) | > grad_norm_1: tensor(5779.2422, device='cuda:0') (tensor(5779.2422, device='cuda:0')) | > current_lr_0: 0.00018246559445127657 | > current_lr_1: 0.00018246559445127657 | > step_time: 0.436 (0.4359889030456543) | > loader_time: 0.0049 (0.004937887191772461)  --> TIME: 2025-02-25 15:46:27 -- STEP: 51/72 -- GLOBAL_STEP: 1052900 | > loss_disc: 2.566354274749756 (2.543501891341863) | > loss_disc_real_0: 0.12470382452011108 (0.10561440081573) | > loss_disc_real_1: 0.20940475165843964 (0.2183132457966898) | > loss_disc_real_2: 0.2654753625392914 (0.22385788402136633) | > loss_disc_real_3: 0.24844704568386078 (0.22932492664047316) | > loss_disc_real_4: 0.24237580597400665 (0.23914214121360405) | > loss_disc_real_5: 0.25740402936935425 (0.24541977953677083) | > loss_0: 2.566354274749756 (2.543501891341863) | > grad_norm_0: tensor(358.9106, device='cuda:0') (tensor(200.5997, device='cuda:0')) | > loss_gen: 2.078516960144043 (2.1485731554966336) | > loss_kl: 1.7674975395202637 (2.0821470606560797) | > loss_feat: 3.6117000579833984 (3.7794515245101032) | > loss_mel: 18.156414031982422 (17.904633091945275) | > loss_duration: 1.598787546157837 (1.5792960255753759) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.212915420532227 (27.494100645476696) | > grad_norm_1: tensor(12361.9238, device='cuda:0') (tensor(9614.5898, device='cuda:0')) | > current_lr_0: 0.00018246559445127657 | > current_lr_1: 0.00018246559445127657 | > step_time: 0.4535 (0.4474197135252111) | > loader_time: 0.0092 (0.007825355903775083) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696038958052514 (-0.00019343805984711922) | > avg_step_time: 0.45998996748051174 (-0.0013962094212921494) | > avg_loss_disc: 2.539538944271248 (-0.15566424584724592) | > avg_loss_disc_real_0: 0.10577195011813875 (-0.05734548061876227) | > avg_loss_disc_real_1: 0.21822697574823674 (-0.021901586316001254) | > avg_loss_disc_real_2: 0.22288100106615416 (-0.01048081315739055) | > avg_loss_disc_real_3: 0.2301871505841403 (-0.004863137510460858) | > avg_loss_disc_real_4: 0.23885853546605984 (-0.010631961302018489) | > avg_loss_disc_real_5: 0.24544229138065393 (-0.00274722446018541) | > avg_loss_0: 2.539538944271248 (-0.15566424584724592) | > avg_grad_norm_0: tensor(209.6277, device='cuda:0') (+tensor(32.7894, device='cuda:0')) | > avg_loss_gen: 2.1526379769956554 (+0.07146231221481081) | > avg_loss_kl: 2.096980027749504 (-0.44210954451225204) | > avg_loss_feat: 3.757269258230505 (+0.31230673655657704) | > avg_loss_mel: 17.776552361501775 (-1.8910495328231605) | > avg_loss_duration: 1.5902152665903868 (-0.05354127245889617) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.373654755068497 (-2.0029313262079818) | > avg_grad_norm_1: tensor(9737.0557, device='cuda:0') (+tensor(3255.8369, device='cuda:0'))  > EPOCH: 735/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:46:39)   --> TIME: 2025-02-25 15:46:54 -- STEP: 29/72 -- GLOBAL_STEP: 1052950 | > loss_disc: 2.796942710876465 (2.5239676031573066) | > loss_disc_real_0: 0.08219065517187119 (0.09865902467020626) | > loss_disc_real_1: 0.2178308218717575 (0.21720816143627825) | > loss_disc_real_2: 0.23322166502475739 (0.21987588703632355) | > loss_disc_real_3: 0.21802321076393127 (0.23033688551393047) | > loss_disc_real_4: 0.23668469488620758 (0.2390253122510581) | > loss_disc_real_5: 0.24101375043392181 (0.2429554077058003) | > loss_0: 2.796942710876465 (2.5239676031573066) | > grad_norm_0: tensor(350.0984, device='cuda:0') (tensor(240.9264, device='cuda:0')) | > loss_gen: 1.8263492584228516 (2.195295165325034) | > loss_kl: 1.9519964456558228 (1.9282620405328685) | > loss_feat: 3.5345358848571777 (3.9942574994317415) | > loss_mel: 18.07447624206543 (17.488304269724875) | > loss_duration: 1.546950101852417 (1.5372624479491135) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.934307098388672 (27.143381644939556) | > grad_norm_1: tensor(11473.7451, device='cuda:0') (tensor(10813.9170, device='cuda:0')) | > current_lr_0: 0.00018244278625197016 | > current_lr_1: 0.00018244278625197016 | > step_time: 0.4392 (0.4407098786584262) | > loader_time: 0.0077 (0.007084295667451003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008669527483658053 (-2.651147439446111e-05) | > avg_step_time: 0.4606302691177583 (+0.0006403016372465431) | > avg_loss_disc: 2.5327366674450085 (-0.006802276826239684) | > avg_loss_disc_real_0: 0.10027219554488088 (-0.005499754573257867) | > avg_loss_disc_real_1: 0.21915468846408415 (+0.0009277127158474063) | > avg_loss_disc_real_2: 0.22738212263080435 (+0.0045011215646501845) | > avg_loss_disc_real_3: 0.23171835655057935 (+0.0015312059664390476) | > avg_loss_disc_real_4: 0.23986600444350445 (+0.0010074689774446088) | > avg_loss_disc_real_5: 0.2441066755375392 (-0.0013356158431147291) | > avg_loss_0: 2.5327366674450085 (-0.006802276826239684) | > avg_grad_norm_0: tensor(238.1646, device='cuda:0') (+tensor(28.5370, device='cuda:0')) | > avg_loss_gen: 2.181154494554224 (+0.028516517558568655) | > avg_loss_kl: 1.9824749953310254 (-0.11450503241847843) | > avg_loss_feat: 3.8466600297202524 (+0.0893907714897475) | > avg_loss_mel: 17.323161380391724 (-0.4533909811100507) | > avg_loss_duration: 1.5724792211828098 (-0.017736045407577006) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.90593018330319 (-0.4677245717653058) | > avg_grad_norm_1: tensor(10567.1230, device='cuda:0') (+tensor(830.0674, device='cuda:0'))  > EPOCH: 736/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:47:16)   --> TIME: 2025-02-25 15:47:21 -- STEP: 7/72 -- GLOBAL_STEP: 1053000 | > loss_disc: 2.4934728145599365 (2.5546512263161794) | > loss_disc_real_0: 0.09580415487289429 (0.10221926761525017) | > loss_disc_real_1: 0.18693357706069946 (0.21781151635306223) | > loss_disc_real_2: 0.24786747992038727 (0.2261314115353993) | > loss_disc_real_3: 0.27250662446022034 (0.2227817944117955) | > loss_disc_real_4: 0.2657772898674011 (0.23534515500068665) | > loss_disc_real_5: 0.25334757566452026 (0.24290949531963893) | > loss_0: 2.4934728145599365 (2.5546512263161794) | > grad_norm_0: tensor(188.7157, device='cuda:0') (tensor(274.0762, device='cuda:0')) | > loss_gen: 2.3371877670288086 (2.169851303100586) | > loss_kl: 2.0018749237060547 (1.8967768464769637) | > loss_feat: 4.103747844696045 (3.9235375268118724) | > loss_mel: 17.05184555053711 (17.153489930289133) | > loss_duration: 1.509223222732544 (1.5426535946982247) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.00387954711914 (26.686308997017996) | > grad_norm_1: tensor(9524.0039, device='cuda:0') (tensor(11130.9492, device='cuda:0')) | > current_lr_0: 0.00018241998090368867 | > current_lr_1: 0.00018241998090368867 | > step_time: 0.4404 (0.4352590356554304) | > loader_time: 0.007 (0.006527015141078404)  --> TIME: 2025-02-25 15:47:45 -- STEP: 57/72 -- GLOBAL_STEP: 1053050 | > loss_disc: 2.6487438678741455 (2.559489425859953) | > loss_disc_real_0: 0.11481192708015442 (0.1010117180514754) | > loss_disc_real_1: 0.2617819607257843 (0.21953639884789786) | > loss_disc_real_2: 0.2010623663663864 (0.23453078275186973) | > loss_disc_real_3: 0.20434245467185974 (0.2302286156959701) | > loss_disc_real_4: 0.23768584430217743 (0.24261750176287533) | > loss_disc_real_5: 0.24459081888198853 (0.24886193087226466) | > loss_0: 2.6487438678741455 (2.559489425859953) | > grad_norm_0: tensor(301.2818, device='cuda:0') (tensor(261.4918, device='cuda:0')) | > loss_gen: 1.8618104457855225 (2.172316141295851) | > loss_kl: 1.9807941913604736 (1.9457064532397086) | > loss_feat: 3.4121832847595215 (3.803295039294059) | > loss_mel: 15.116927146911621 (17.00804288763749) | > loss_duration: 1.620980978012085 (1.5557244576905902) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.99269676208496 (26.485084935238486) | > grad_norm_1: tensor(9933.5322, device='cuda:0') (tensor(11091.0283, device='cuda:0')) | > current_lr_0: 0.00018241998090368867 | > current_lr_1: 0.00018241998090368867 | > step_time: 0.4648 (0.44730277228773685) | > loader_time: 0.0097 (0.00813761510347065) --> EVAL PERFORMANCE | > avg_loader_time: 0.008757443495199713 (+8.791601154166037e-05) | > avg_step_time: 0.4588245002316757 (-0.0018057688860825816) | > avg_loss_disc: 2.5648495479368827 (+0.03211288049187422) | > avg_loss_disc_real_0: 0.10255882721131956 (+0.0022866316664386793) | > avg_loss_disc_real_1: 0.22065115856452727 (+0.0014964701004431258) | > avg_loss_disc_real_2: 0.23308576181740828 (+0.005703639186603937) | > avg_loss_disc_real_3: 0.23120653146589307 (-0.0005118250846862793) | > avg_loss_disc_real_4: 0.2423306082335996 (+0.002464603790095138) | > avg_loss_disc_real_5: 0.24935537177911946 (+0.005248696241580253) | > avg_loss_0: 2.5648495479368827 (+0.03211288049187422) | > avg_grad_norm_0: tensor(265.0715, device='cuda:0') (+tensor(26.9068, device='cuda:0')) | > avg_loss_gen: 2.1768847300972736 (-0.004269764456950487) | > avg_loss_kl: 1.976308203079331 (-0.006166792251694364) | > avg_loss_feat: 3.793596710957272 (-0.053063318762980316) | > avg_loss_mel: 17.032509038146106 (-0.2906523422456182) | > avg_loss_duration: 1.567622403023948 (-0.004856818158861742) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.54692109873597 (-0.35900908456721936) | > avg_grad_norm_1: tensor(11333.7236, device='cuda:0') (+tensor(766.6006, device='cuda:0'))  > EPOCH: 737/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:47:54)   --> TIME: 2025-02-25 15:48:11 -- STEP: 35/72 -- GLOBAL_STEP: 1053100 | > loss_disc: 2.7475171089172363 (2.551361390522548) | > loss_disc_real_0: 0.09750586748123169 (0.09770930047546114) | > loss_disc_real_1: 0.22962099313735962 (0.2144875475338527) | > loss_disc_real_2: 0.22552229464054108 (0.2277778910739081) | > loss_disc_real_3: 0.24151231348514557 (0.23119287320545742) | > loss_disc_real_4: 0.2337476909160614 (0.2404449407543455) | > loss_disc_real_5: 0.2429976761341095 (0.2421574226447514) | > loss_0: 2.7475171089172363 (2.551361390522548) | > grad_norm_0: tensor(422.7255, device='cuda:0') (tensor(266.7197, device='cuda:0')) | > loss_gen: 1.9541523456573486 (2.1780684198651987) | > loss_kl: 2.0300116539001465 (1.8899188314165387) | > loss_feat: 3.3094124794006348 (3.866087375368391) | > loss_mel: 17.55019187927246 (17.01626796722412) | > loss_duration: 1.5805548429489136 (1.5293538706643242) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.42432403564453 (26.479696382795062) | > grad_norm_1: tensor(12036.3691, device='cuda:0') (tensor(11274.5664, device='cuda:0')) | > current_lr_0: 0.0001823971784060757 | > current_lr_1: 0.0001823971784060757 | > step_time: 0.4467 (0.4402454784938267) | > loader_time: 0.0085 (0.007212897709437779) --> EVAL PERFORMANCE | > avg_loader_time: 0.008616088141857738 (-0.00014135535334197473) | > avg_step_time: 0.45870396452890316 (-0.00012053570277253955) | > avg_loss_disc: 2.5653290849336434 (+0.00047953699676073214) | > avg_loss_disc_real_0: 0.1020860651534208 (-0.0004727620578987707) | > avg_loss_disc_real_1: 0.2175856201161801 (-0.0030655384483471715) | > avg_loss_disc_real_2: 0.22960479389613783 (-0.0034809679212704503) | > avg_loss_disc_real_3: 0.2316101913301038 (+0.0004036598642107314) | > avg_loss_disc_real_4: 0.24242448932688002 (+9.388109328042904e-05) | > avg_loss_disc_real_5: 0.24429390568968276 (-0.005061466089436695) | > avg_loss_0: 2.5653290849336434 (+0.00047953699676073214) | > avg_grad_norm_0: tensor(260.0865, device='cuda:0') (tensor(-4.9850, device='cuda:0')) | > avg_loss_gen: 2.174355906499942 (-0.002528823597331531) | > avg_loss_kl: 1.9527956029059181 (-0.023512600173412945) | > avg_loss_feat: 3.8237699689999434 (+0.030173258042671236) | > avg_loss_mel: 16.89115406090105 (-0.14135497724505441) | > avg_loss_duration: 1.5628831588046652 (-0.0047392442192828454) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.404958617519323 (-0.14196248121664823) | > avg_grad_norm_1: tensor(11302.5830, device='cuda:0') (tensor(-31.1406, device='cuda:0'))  > EPOCH: 738/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:48:31)   --> TIME: 2025-02-25 15:48:39 -- STEP: 13/72 -- GLOBAL_STEP: 1053150 | > loss_disc: 2.4942760467529297 (2.518795031767625) | > loss_disc_real_0: 0.22083449363708496 (0.10614248365163803) | > loss_disc_real_1: 0.1964869499206543 (0.22443247070679298) | > loss_disc_real_2: 0.18529559671878815 (0.23084991367963645) | > loss_disc_real_3: 0.19355502724647522 (0.22909177037385794) | > loss_disc_real_4: 0.22953060269355774 (0.23492992153534523) | > loss_disc_real_5: 0.2514708638191223 (0.23895059869839594) | > loss_0: 2.4942760467529297 (2.518795031767625) | > grad_norm_0: tensor(292.8815, device='cuda:0') (tensor(276.3387, device='cuda:0')) | > loss_gen: 2.5427191257476807 (2.278537997832665) | > loss_kl: 1.8890568017959595 (1.9054184876955473) | > loss_feat: 4.040862083435059 (4.0319740772247314) | > loss_mel: 16.983610153198242 (17.232892696674053) | > loss_duration: 1.443663239479065 (1.5187360690190241) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.89990997314453 (26.96755937429575) | > grad_norm_1: tensor(10775.6396, device='cuda:0') (tensor(11617.2080, device='cuda:0')) | > current_lr_0: 0.00018237437875877493 | > current_lr_1: 0.00018237437875877493 | > step_time: 0.462 (0.43081122178297776) | > loader_time: 0.0072 (0.006948727827805739)  --> TIME: 2025-02-25 15:49:02 -- STEP: 63/72 -- GLOBAL_STEP: 1053200 | > loss_disc: 2.6593055725097656 (2.556455271584647) | > loss_disc_real_0: 0.10748062282800674 (0.0989289665742526) | > loss_disc_real_1: 0.2223864048719406 (0.22531028683223422) | > loss_disc_real_2: 0.34635597467422485 (0.22948148184352452) | > loss_disc_real_3: 0.2917018234729767 (0.23256435257101815) | > loss_disc_real_4: 0.2615523934364319 (0.2406748966092155) | > loss_disc_real_5: 0.249232217669487 (0.24492828429691374) | > loss_0: 2.6593055725097656 (2.556455271584647) | > grad_norm_0: tensor(373.5904, device='cuda:0') (tensor(257.3637, device='cuda:0')) | > loss_gen: 2.0372872352600098 (2.211589126359849) | > loss_kl: 2.1138269901275635 (1.99860756170182) | > loss_feat: 3.025416851043701 (3.920370238167899) | > loss_mel: 15.341182708740234 (16.91221626221189) | > loss_duration: 1.57789146900177 (1.5556503326173812) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.095605850219727 (26.598433237227184) | > grad_norm_1: tensor(8406.2178, device='cuda:0') (tensor(11510.4365, device='cuda:0')) | > current_lr_0: 0.00018237437875877493 | > current_lr_1: 0.00018237437875877493 | > step_time: 0.5075 (0.4490103040422712) | > loader_time: 0.0122 (0.008650227198525083) --> EVAL PERFORMANCE | > avg_loader_time: 0.008994253588394382 (+0.0003781654465366442) | > avg_step_time: 0.457327856144435 (-0.0013761083844681665) | > avg_loss_disc: 2.563855738706992 (-0.0014733462266516284) | > avg_loss_disc_real_0: 0.09813763442593561 (-0.003948430727485186) | > avg_loss_disc_real_1: 0.226131602492131 (+0.008545982375950889) | > avg_loss_disc_real_2: 0.22938793435902663 (-0.00021685953711120254) | > avg_loss_disc_real_3: 0.23331226072680783 (+0.0017020693967040312) | > avg_loss_disc_real_4: 0.24186015590815477 (-0.0005643334187252491) | > avg_loss_disc_real_5: 0.2437145361598109 (-0.0005793695298718649) | > avg_loss_0: 2.563855738706992 (-0.0014733462266516284) | > avg_grad_norm_0: tensor(255.5842, device='cuda:0') (tensor(-4.5024, device='cuda:0')) | > avg_loss_gen: 2.2023721194603074 (+0.028016212960365383) | > avg_loss_kl: 2.014899883471744 (+0.06210428056582584) | > avg_loss_feat: 3.9091630720756423 (+0.08539310307569892) | > avg_loss_mel: 16.91976121445778 (+0.028607153556727383) | > avg_loss_duration: 1.5635224866195463 (+0.0006393278148810744) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.609718591394557 (+0.20475997387523392) | > avg_grad_norm_1: tensor(11570.4102, device='cuda:0') (+tensor(267.8271, device='cuda:0'))  > EPOCH: 739/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:49:08)   --> TIME: 2025-02-25 15:49:29 -- STEP: 41/72 -- GLOBAL_STEP: 1053250 | > loss_disc: 2.7057418823242188 (2.631149425739195) | > loss_disc_real_0: 0.07760071754455566 (0.11274706999339708) | > loss_disc_real_1: 0.21638160943984985 (0.22480865949537696) | > loss_disc_real_2: 0.20559917390346527 (0.225174927856864) | > loss_disc_real_3: 0.2033178061246872 (0.23261864410667885) | > loss_disc_real_4: 0.22159214317798615 (0.24074730749537304) | > loss_disc_real_5: 0.24276724457740784 (0.2506494631127613) | > loss_0: 2.7057418823242188 (2.631149425739195) | > grad_norm_0: tensor(397.6884, device='cuda:0') (tensor(279.5858, device='cuda:0')) | > loss_gen: 1.8621364831924438 (2.1143624375506147) | > loss_kl: 1.932020902633667 (1.9986425812651472) | > loss_feat: 3.7110469341278076 (3.829853052046241) | > loss_mel: 17.26822280883789 (16.977110002099007) | > loss_duration: 1.5621638298034668 (1.5336331855960008) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.33559226989746 (26.45360127890982) | > grad_norm_1: tensor(13357.6631, device='cuda:0') (tensor(11306.5723, device='cuda:0')) | > current_lr_0: 0.00018235158196143006 | > current_lr_1: 0.00018235158196143006 | > step_time: 0.453 (0.4402881889808469) | > loader_time: 0.0094 (0.007368378522919446) --> EVAL PERFORMANCE | > avg_loader_time: 0.008604351903351263 (-0.00038990168504311924) | > avg_step_time: 0.4560967734162237 (-0.001231082728211319) | > avg_loss_disc: 2.6288788184313705 (+0.06502307972437871) | > avg_loss_disc_real_0: 0.10454793531500117 (+0.006410300889065565) | > avg_loss_disc_real_1: 0.23014793807352094 (+0.004016335581389946) | > avg_loss_disc_real_2: 0.23327374300906356 (+0.0038858086500369315) | > avg_loss_disc_real_3: 0.23269241817400488 (-0.0006198425528029516) | > avg_loss_disc_real_4: 0.24071128750351115 (-0.0011488684046436215) | > avg_loss_disc_real_5: 0.24739074958881863 (+0.0036762134290077297) | > avg_loss_0: 2.6288788184313705 (+0.06502307972437871) | > avg_grad_norm_0: tensor(273.9405, device='cuda:0') (+tensor(18.3563, device='cuda:0')) | > avg_loss_gen: 2.1208920394870603 (-0.08148007997324713) | > avg_loss_kl: 2.0416183370939436 (+0.0267184536221996) | > avg_loss_feat: 3.7932383174627597 (-0.11592475461288254) | > avg_loss_mel: 16.927708733249712 (+0.007947518791933561) | > avg_loss_duration: 1.563309479767168 (-0.00021300685237823913) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.446766974220807 (-0.16295161717374995) | > avg_grad_norm_1: tensor(11474.3564, device='cuda:0') (tensor(-96.0537, device='cuda:0'))  > EPOCH: 740/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:49:45)   --> TIME: 2025-02-25 15:49:56 -- STEP: 19/72 -- GLOBAL_STEP: 1053300 | > loss_disc: 2.4303817749023438 (2.6206234881752417) | > loss_disc_real_0: 0.15031775832176208 (0.1137156610034014) | > loss_disc_real_1: 0.24397069215774536 (0.21971739554091504) | > loss_disc_real_2: 0.29419055581092834 (0.2230564567603563) | > loss_disc_real_3: 0.2301197201013565 (0.2377510964870453) | > loss_disc_real_4: 0.22430448234081268 (0.22465447532503227) | > loss_disc_real_5: 0.22884371876716614 (0.2347909086628964) | > loss_0: 2.4303817749023438 (2.6206234881752417) | > grad_norm_0: tensor(154.3708, device='cuda:0') (tensor(265.5555, device='cuda:0')) | > loss_gen: 2.666001796722412 (2.0652547384563245) | > loss_kl: 1.802735686302185 (1.9556696163980585) | > loss_feat: 4.394433498382568 (3.7233916834781042) | > loss_mel: 17.092815399169922 (17.089052702251234) | > loss_duration: 1.5184658765792847 (1.520927015103792) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.47445297241211 (26.35429623252467) | > grad_norm_1: tensor(11872.8965, device='cuda:0') (tensor(10070.9648, device='cuda:0')) | > current_lr_0: 0.00018232878801368487 | > current_lr_1: 0.00018232878801368487 | > step_time: 0.4295 (0.43688800460413885) | > loader_time: 0.0098 (0.006955836948595549)  --> TIME: 2025-02-25 15:50:20 -- STEP: 69/72 -- GLOBAL_STEP: 1053350 | > loss_disc: 2.7265350818634033 (2.5857038947119233) | > loss_disc_real_0: 0.07956188172101974 (0.10133546779769055) | > loss_disc_real_1: 0.32432106137275696 (0.22918238685182904) | > loss_disc_real_2: 0.15673774480819702 (0.22269259781941123) | > loss_disc_real_3: 0.2189386487007141 (0.23564551209194073) | > loss_disc_real_4: 0.2437407523393631 (0.2371434310208196) | > loss_disc_real_5: 0.23005196452140808 (0.24406532597714575) | > loss_0: 2.7265350818634033 (2.5857038947119233) | > grad_norm_0: tensor(312.4209, device='cuda:0') (tensor(254.2638, device='cuda:0')) | > loss_gen: 1.8524519205093384 (2.148093303044637) | > loss_kl: 2.2450714111328125 (2.052194460578587) | > loss_feat: 3.5219860076904297 (3.8418812889983687) | > loss_mel: 17.005359649658203 (17.06100573055986) | > loss_duration: 1.6541165113449097 (1.563927652179331) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.27898406982422 (26.667102260866027) | > grad_norm_1: tensor(12930.8086, device='cuda:0') (tensor(11656.2109, device='cuda:0')) | > current_lr_0: 0.00018232878801368487 | > current_lr_1: 0.00018232878801368487 | > step_time: 0.5185 (0.45717503367990686) | > loader_time: 0.0125 (0.008760113647018654) --> EVAL PERFORMANCE | > avg_loader_time: 0.008821897103752889 (+0.0002175452004016254) | > avg_step_time: 0.45896295426597056 (+0.00286618084974688) | > avg_loss_disc: 2.590618714480333 (-0.038260103951037294) | > avg_loss_disc_real_0: 0.10139933412133809 (-0.0031486011936630864) | > avg_loss_disc_real_1: 0.23087090726049853 (+0.00072296918697759) | > avg_loss_disc_real_2: 0.2215892605378594 (-0.011684482471204155) | > avg_loss_disc_real_3: 0.23503947257995605 (+0.0023470544059511766) | > avg_loss_disc_real_4: 0.23767001465172835 (-0.0030412728517827947) | > avg_loss_disc_real_5: 0.24405664184563597 (-0.003334107743182657) | > avg_loss_0: 2.590618714480333 (-0.038260103951037294) | > avg_grad_norm_0: tensor(260.2379, device='cuda:0') (tensor(-13.7026, device='cuda:0')) | > avg_loss_gen: 2.144676040595686 (+0.023784001108625663) | > avg_loss_kl: 2.0556553585428587 (+0.014037021448915166) | > avg_loss_feat: 3.8411297495936005 (+0.047891432130840794) | > avg_loss_mel: 17.065270665665746 (+0.13756193241603398) | > avg_loss_duration: 1.5643916482656777 (+0.001082168498509617) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67112328972615 (+0.22435631550534296) | > avg_grad_norm_1: tensor(11691.4121, device='cuda:0') (+tensor(217.0557, device='cuda:0'))  > EPOCH: 741/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:50:23)   --> TIME: 2025-02-25 15:50:46 -- STEP: 47/72 -- GLOBAL_STEP: 1053400 | > loss_disc: 2.73089861869812 (2.6169636756815815) | > loss_disc_real_0: 0.1079569011926651 (0.09928060640045937) | > loss_disc_real_1: 0.20412962138652802 (0.22523985454376708) | > loss_disc_real_2: 0.1947939693927765 (0.21933120362302091) | > loss_disc_real_3: 0.20294731855392456 (0.23050032682875368) | > loss_disc_real_4: 0.23608142137527466 (0.238958588622986) | > loss_disc_real_5: 0.22849152982234955 (0.24632758092372975) | > loss_0: 2.73089861869812 (2.6169636756815815) | > grad_norm_0: tensor(226.9518, device='cuda:0') (tensor(265.0125, device='cuda:0')) | > loss_gen: 1.8654839992523193 (2.1085944708357474) | > loss_kl: 2.054691791534424 (2.0616781102850075) | > loss_feat: 3.4127607345581055 (3.809939871443079) | > loss_mel: 15.97830581665039 (16.827194741431693) | > loss_duration: 1.602121114730835 (1.540411079183538) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.91336441040039 (26.347818334051905) | > grad_norm_1: tensor(10014.7266, device='cuda:0') (tensor(11272.1475, device='cuda:0')) | > current_lr_0: 0.00018230599691518315 | > current_lr_1: 0.00018230599691518315 | > step_time: 0.4556 (0.44330596923828125) | > loader_time: 0.0088 (0.00776236615282424) --> EVAL PERFORMANCE | > avg_loader_time: 0.008725465183526697 (-9.643192022619138e-05) | > avg_step_time: 0.45799875259399414 (-0.0009642016719764168) | > avg_loss_disc: 2.61415592381652 (+0.02353720933618675) | > avg_loss_disc_real_0: 0.10232052631991011 (+0.0009211921985720212) | > avg_loss_disc_real_1: 0.2254105760178096 (-0.005460331242688937) | > avg_loss_disc_real_2: 0.22306357219185624 (+0.0014743116539968348) | > avg_loss_disc_real_3: 0.23111554056825773 (-0.003923932011698328) | > avg_loss_disc_real_4: 0.2375522516143154 (-0.00011776303741295457) | > avg_loss_disc_real_5: 0.24567718006355663 (+0.0016205382179206584) | > avg_loss_0: 2.61415592381652 (+0.02353720933618675) | > avg_grad_norm_0: tensor(261.8588, device='cuda:0') (+tensor(1.6209, device='cuda:0')) | > avg_loss_gen: 2.1206510352416776 (-0.02402500535400831) | > avg_loss_kl: 2.087470249390938 (+0.03181489084807909) | > avg_loss_feat: 3.7999764596912224 (-0.0411532899023781) | > avg_loss_mel: 16.837646578399223 (-0.2276240872665234) | > avg_loss_duration: 1.5637529248922644 (-0.000638723373413308) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.409497274479396 (-0.26162601524675466) | > avg_grad_norm_1: tensor(11413.2178, device='cuda:0') (tensor(-278.1943, device='cuda:0'))  > EPOCH: 742/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:51:00)   --> TIME: 2025-02-25 15:51:13 -- STEP: 25/72 -- GLOBAL_STEP: 1053450 | > loss_disc: 2.4754340648651123 (2.678841857910156) | > loss_disc_real_0: 0.2096114158630371 (0.09915905430912972) | > loss_disc_real_1: 0.21293477714061737 (0.22596580445766448) | > loss_disc_real_2: 0.25651976466178894 (0.22370759457349776) | > loss_disc_real_3: 0.20599202811717987 (0.24137288093566894) | > loss_disc_real_4: 0.211441308259964 (0.2376222252845764) | > loss_disc_real_5: 0.22438527643680573 (0.24912023544311523) | > loss_0: 2.4754340648651123 (2.678841857910156) | > grad_norm_0: tensor(521.7579, device='cuda:0') (tensor(301.0270, device='cuda:0')) | > loss_gen: 2.3387084007263184 (2.0244351339340207) | > loss_kl: 2.1909639835357666 (2.0692014837265016) | > loss_feat: 3.938591718673706 (3.7238023567199705) | > loss_mel: 17.445667266845703 (17.191606521606445) | > loss_duration: 1.5512868165969849 (1.5257292318344116) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.46521759033203 (26.534774475097656) | > grad_norm_1: tensor(11588.5723, device='cuda:0') (tensor(10356.2168, device='cuda:0')) | > current_lr_0: 0.00018228320866556875 | > current_lr_1: 0.00018228320866556875 | > step_time: 0.435 (0.4387282085418701) | > loader_time: 0.0082 (0.007021064758300781) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719958050150264 (-5.507133376433243e-06) | > avg_step_time: 0.45763831407251493 (-0.0003604385214792116) | > avg_loss_disc: 2.664411138480818 (+0.05025521466429783) | > avg_loss_disc_real_0: 0.14177805111861566 (+0.039457524798705554) | > avg_loss_disc_real_1: 0.22342774775666252 (-0.0019828282611470682) | > avg_loss_disc_real_2: 0.23542977984942182 (+0.012366207657565575) | > avg_loss_disc_real_3: 0.23696317093473085 (+0.005847630366473122) | > avg_loss_disc_real_4: 0.23482847423620626 (-0.002723777378109138) | > avg_loss_disc_real_5: 0.24332171699530641 (-0.002355463068250213) | > avg_loss_0: 2.664411138480818 (+0.05025521466429783) | > avg_grad_norm_0: tensor(264.9245, device='cuda:0') (+tensor(3.0657, device='cuda:0')) | > avg_loss_gen: 2.089105496943837 (-0.03154553829784046) | > avg_loss_kl: 2.1169424224907236 (+0.02947217309978578) | > avg_loss_feat: 3.5275123925276204 (-0.27246406716360205) | > avg_loss_mel: 17.08086338849135 (+0.24321681009212526) | > avg_loss_duration: 1.5718162647435363 (+0.008063339851271945) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.386239951765035 (-0.023257322714361095) | > avg_grad_norm_1: tensor(9449.6279, device='cuda:0') (tensor(-1963.5898, device='cuda:0'))  > EPOCH: 743/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:51:37)   --> TIME: 2025-02-25 15:51:41 -- STEP: 3/72 -- GLOBAL_STEP: 1053500 | > loss_disc: 2.6077070236206055 (2.598071575164795) | > loss_disc_real_0: 0.06639930605888367 (0.12080609301726024) | > loss_disc_real_1: 0.18344560265541077 (0.26303236683209735) | > loss_disc_real_2: 0.2289581000804901 (0.24357842902342478) | > loss_disc_real_3: 0.22081851959228516 (0.22857427597045898) | > loss_disc_real_4: 0.2429143488407135 (0.2643265128135681) | > loss_disc_real_5: 0.21685630083084106 (0.2248625655968984) | > loss_0: 2.6077070236206055 (2.598071575164795) | > grad_norm_0: tensor(332.7968, device='cuda:0') (tensor(257.9844, device='cuda:0')) | > loss_gen: 1.902171015739441 (2.2386662562688193) | > loss_kl: 2.5596303939819336 (2.7067960103352866) | > loss_feat: 3.879751205444336 (3.9864234924316406) | > loss_mel: 19.913148880004883 (20.47598648071289) | > loss_duration: 1.6227903366088867 (1.6560568412144978) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.877490997314453 (31.063929239908855) | > grad_norm_1: tensor(11153.5938, device='cuda:0') (tensor(9761.0234, device='cuda:0')) | > current_lr_0: 0.00018226042326448556 | > current_lr_1: 0.00018226042326448556 | > step_time: 0.4259 (0.43469007809956867) | > loader_time: 0.0057 (0.006195386250813802)  --> TIME: 2025-02-25 15:52:04 -- STEP: 53/72 -- GLOBAL_STEP: 1053550 | > loss_disc: 2.429492473602295 (2.7172395283321165) | > loss_disc_real_0: 0.10813205689191818 (0.13708874049051747) | > loss_disc_real_1: 0.2463705837726593 (0.236511212053164) | > loss_disc_real_2: 0.23382019996643066 (0.24583741432090975) | > loss_disc_real_3: 0.20738434791564941 (0.23938133244244558) | > loss_disc_real_4: 0.199056938290596 (0.2520239364426092) | > loss_disc_real_5: 0.2512962818145752 (0.24390879217183814) | > loss_0: 2.429492473602295 (2.7172395283321165) | > grad_norm_0: tensor(110.1510, device='cuda:0') (tensor(232.0424, device='cuda:0')) | > loss_gen: 2.404700517654419 (2.0845249216511563) | > loss_kl: 2.168044090270996 (2.313194956419603) | > loss_feat: 4.522510051727295 (3.561463936319891) | > loss_mel: 17.898427963256836 (18.08617647638861) | > loss_duration: 1.5852525234222412 (1.5822892953764718) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.578935623168945 (27.627649595152658) | > grad_norm_1: tensor(12108.1699, device='cuda:0') (tensor(8850.7158, device='cuda:0')) | > current_lr_0: 0.00018226042326448556 | > current_lr_1: 0.00018226042326448556 | > step_time: 0.4494 (0.4451319991417651) | > loader_time: 0.0092 (0.007756480630838646) --> EVAL PERFORMANCE | > avg_loader_time: 0.008553686276288096 (-0.00016627177386216788) | > avg_step_time: 0.4558644160418443 (-0.0017738980306706287) | > avg_loss_disc: 2.70682198900572 (+0.042410850524902344) | > avg_loss_disc_real_0: 0.1384459642667166 (-0.003332086851899063) | > avg_loss_disc_real_1: 0.2337353799334714 (+0.010307632176808884) | > avg_loss_disc_real_2: 0.2443447039580681 (+0.008914924108646277) | > avg_loss_disc_real_3: 0.23971709252243312 (+0.0027539215877022682) | > avg_loss_disc_real_4: 0.2493595086352926 (+0.014531034399086346) | > avg_loss_disc_real_5: 0.24307357383445954 (-0.0002481431608468698) | > avg_loss_0: 2.70682198900572 (+0.042410850524902344) | > avg_grad_norm_0: tensor(229.5156, device='cuda:0') (tensor(-35.4088, device='cuda:0')) | > avg_loss_gen: 2.0721320165714747 (-0.01697348037236246) | > avg_loss_kl: 2.27544848180153 (+0.15850605931080652) | > avg_loss_feat: 3.485530524186685 (-0.041981868340935424) | > avg_loss_mel: 17.799619929891236 (+0.7187565413998875) | > avg_loss_duration: 1.5907367249609718 (+0.018920460217435542) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.223467625362773 (+0.837227673597738) | > avg_grad_norm_1: tensor(8701.7500, device='cuda:0') (tensor(-747.8779, device='cuda:0'))  > EPOCH: 744/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:52:14)   --> TIME: 2025-02-25 15:52:30 -- STEP: 31/72 -- GLOBAL_STEP: 1053600 | > loss_disc: 2.3484346866607666 (2.668229987544398) | > loss_disc_real_0: 0.05144072324037552 (0.13553919979641518) | > loss_disc_real_1: 0.21123333275318146 (0.24738989049388507) | > loss_disc_real_2: 0.2512660324573517 (0.25480909597489143) | > loss_disc_real_3: 0.24313999712467194 (0.24607528553855035) | > loss_disc_real_4: 0.24600954353809357 (0.2564648817623816) | > loss_disc_real_5: 0.2509957253932953 (0.26075483425971) | > loss_0: 2.3484346866607666 (2.668229987544398) | > grad_norm_0: tensor(14.7033, device='cuda:0') (tensor(182.3974, device='cuda:0')) | > loss_gen: 2.514357805252075 (2.243534461144478) | > loss_kl: 2.126927375793457 (2.062381655939164) | > loss_feat: 4.535560607910156 (3.678412006747338) | > loss_mel: 17.86121940612793 (17.327257094844693) | > loss_duration: 1.5677331686019897 (1.5337138099055136) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.60580062866211 (26.845299013199345) | > grad_norm_1: tensor(3418.3064, device='cuda:0') (tensor(6188.2266, device='cuda:0')) | > current_lr_0: 0.00018223764071157749 | > current_lr_1: 0.00018223764071157749 | > step_time: 0.4356 (0.4394610312677199) | > loader_time: 0.0083 (0.007292816715855752) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671857941318566 (+0.00011817166503047018) | > avg_step_time: 0.4557814228702599 (-8.299317158438901e-05) | > avg_loss_disc: 2.699517381023353 (-0.007304607982367095) | > avg_loss_disc_real_0: 0.14170764265975483 (+0.003261678393038231) | > avg_loss_disc_real_1: 0.23828990127838828 (+0.004554521344916879) | > avg_loss_disc_real_2: 0.23855330154929363 (-0.00579140240877446) | > avg_loss_disc_real_3: 0.24225992090265516 (+0.0025428283802220453) | > avg_loss_disc_real_4: 0.24774365798688275 (-0.0016158506484098556) | > avg_loss_disc_real_5: 0.2522659079289772 (+0.00919233409451764) | > avg_loss_0: 2.699517381023353 (-0.007304607982367095) | > avg_grad_norm_0: tensor(214.5585, device='cuda:0') (tensor(-14.9571, device='cuda:0')) | > avg_loss_gen: 2.07827571244307 (+0.006143695871595245) | > avg_loss_kl: 2.1165930418901047 (-0.15885543991142548) | > avg_loss_feat: 3.5173872927544823 (+0.03185676856779729) | > avg_loss_mel: 17.294033722138742 (-0.5055862077524935) | > avg_loss_duration: 1.572382582745082 (-0.018354142215889935) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.578672194145096 (-0.6447954312176769) | > avg_grad_norm_1: tensor(6925.0918, device='cuda:0') (tensor(-1776.6582, device='cuda:0'))  > EPOCH: 745/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:52:51)   --> TIME: 2025-02-25 15:52:57 -- STEP: 9/72 -- GLOBAL_STEP: 1053650 | > loss_disc: 2.8332653045654297 (2.7002978059980602) | > loss_disc_real_0: 0.0874280035495758 (0.09302227240469721) | > loss_disc_real_1: 0.46268606185913086 (0.3089628691474597) | > loss_disc_real_2: 0.38899579644203186 (0.25973881118827397) | > loss_disc_real_3: 0.41337820887565613 (0.26885125206576455) | > loss_disc_real_4: 0.3349595069885254 (0.2510317977931764) | > loss_disc_real_5: 0.3094322085380554 (0.2475631833076477) | > loss_0: 2.8332653045654297 (2.7002978059980602) | > grad_norm_0: tensor(32.8236, device='cuda:0') (tensor(121.4220, device='cuda:0')) | > loss_gen: 2.428717613220215 (2.2292034493552313) | > loss_kl: 3.0516610145568848 (2.315801633728875) | > loss_feat: 3.9061615467071533 (3.8414078288608127) | > loss_mel: 20.101953506469727 (18.48943371242947) | > loss_duration: 1.673613429069519 (1.630451003710429) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.162107467651367 (28.506297641330296) | > grad_norm_1: tensor(11596.5693, device='cuda:0') (tensor(9098.4131, device='cuda:0')) | > current_lr_0: 0.00018221486100648853 | > current_lr_1: 0.00018221486100648853 | > step_time: 0.4462 (0.4397680229610867) | > loader_time: 0.008 (0.006578418943617079)  --> TIME: 2025-02-25 15:53:21 -- STEP: 59/72 -- GLOBAL_STEP: 1053700 | > loss_disc: 2.8413004875183105 (2.6233120247469106) | > loss_disc_real_0: 0.053196948021650314 (0.09540140098434385) | > loss_disc_real_1: 0.23632271587848663 (0.23360965249396987) | > loss_disc_real_2: 0.20629030466079712 (0.23809057903491845) | > loss_disc_real_3: 0.30157577991485596 (0.2367516988414829) | > loss_disc_real_4: 0.28770944476127625 (0.24365777939052904) | > loss_disc_real_5: 0.26161763072013855 (0.24525871367777807) | > loss_0: 2.8413004875183105 (2.6233120247469106) | > grad_norm_0: tensor(340.6055, device='cuda:0') (tensor(186.0776, device='cuda:0')) | > loss_gen: 1.6394915580749512 (2.081438014062784) | > loss_kl: 2.1810178756713867 (2.2654175051188066) | > loss_feat: 2.9100964069366455 (3.616253169916444) | > loss_mel: 14.492311477661133 (17.78374073869091) | > loss_duration: 1.612581491470337 (1.611172047712035) | > amp_scaler: 8.0 (8.0) | > loss_1: 22.835498809814453 (27.358021461357506) | > grad_norm_1: tensor(8936.7090, device='cuda:0') (tensor(8969.0742, device='cuda:0')) | > current_lr_0: 0.00018221486100648853 | > current_lr_1: 0.00018221486100648853 | > step_time: 0.4667 (0.44834401647923355) | > loader_time: 0.0098 (0.008098634622864802) --> EVAL PERFORMANCE | > avg_loader_time: 0.008660957846843018 (-1.0900094475548364e-05) | > avg_step_time: 0.45746549418274784 (+0.0016840713124879336) | > avg_loss_disc: 2.6317025036878983 (-0.06781487733545477) | > avg_loss_disc_real_0: 0.09655775565286759 (-0.04514988700688724) | > avg_loss_disc_real_1: 0.23424508296687838 (-0.004044818311509907) | > avg_loss_disc_real_2: 0.22983705346852962 (-0.008716248080764016) | > avg_loss_disc_real_3: 0.23659697447864103 (-0.005662946424014131) | > avg_loss_disc_real_4: 0.2430719107809201 (-0.00467174720596264) | > avg_loss_disc_real_5: 0.24611438412061878 (-0.0061515238083584) | > avg_loss_0: 2.6317025036878983 (-0.06781487733545477) | > avg_grad_norm_0: tensor(203.0722, device='cuda:0') (tensor(-11.4863, device='cuda:0')) | > avg_loss_gen: 2.058470229027976 (-0.019805483415094027) | > avg_loss_kl: 2.242264494090013 (+0.12567145219990827) | > avg_loss_feat: 3.581574923555616 (+0.06418763080113354) | > avg_loss_mel: 17.6618272754508 (+0.3677935533120582) | > avg_loss_duration: 1.613314257541173 (+0.040931674796091055) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.15745111922143 (+0.5787789250763353) | > avg_grad_norm_1: tensor(9095.5986, device='cuda:0') (+tensor(2170.5068, device='cuda:0'))  > EPOCH: 746/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:53:29)   --> TIME: 2025-02-25 15:53:47 -- STEP: 37/72 -- GLOBAL_STEP: 1053750 | > loss_disc: 2.7440104484558105 (2.557042734043018) | > loss_disc_real_0: 0.08657532930374146 (0.09452440821238466) | > loss_disc_real_1: 0.2199707180261612 (0.21921134196423195) | > loss_disc_real_2: 0.25504738092422485 (0.22426298743969686) | > loss_disc_real_3: 0.22969821095466614 (0.2281527962233569) | > loss_disc_real_4: 0.26392662525177 (0.24081765517995163) | > loss_disc_real_5: 0.25883108377456665 (0.24795501860412392) | > loss_0: 2.7440104484558105 (2.557042734043018) | > grad_norm_0: tensor(340.3197, device='cuda:0') (tensor(271.7184, device='cuda:0')) | > loss_gen: 1.9354283809661865 (2.096307686857275) | > loss_kl: 2.0510385036468506 (1.987934644157822) | > loss_feat: 3.387683153152466 (3.6979063201594995) | > loss_mel: 17.47408103942871 (16.963737101168245) | > loss_duration: 1.5725762844085693 (1.5343265469009812) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.420808792114258 (26.28021240234375) | > grad_norm_1: tensor(11623.1807, device='cuda:0') (tensor(10846.2588, device='cuda:0')) | > current_lr_0: 0.0001821920841488627 | > current_lr_1: 0.0001821920841488627 | > step_time: 0.4758 (0.4384851391251023) | > loader_time: 0.0083 (0.007259459108919711) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613284205047176 (-4.767364179584198e-05) | > avg_step_time: 0.4538307425001977 (-0.003634751682550119) | > avg_loss_disc: 2.546907495444929 (-0.08479500824296915) | > avg_loss_disc_real_0: 0.10190627173009054 (+0.005348516077222948) | > avg_loss_disc_real_1: 0.21868463595148543 (-0.015560447015392942) | > avg_loss_disc_real_2: 0.22181419986234585 (-0.008022853606183766) | > avg_loss_disc_real_3: 0.2295669548108544 (-0.00703001966778663) | > avg_loss_disc_real_4: 0.239851928093064 (-0.0032199826878561155) | > avg_loss_disc_real_5: 0.24730547947782866 (+0.001191095357209876) | > avg_loss_0: 2.546907495444929 (-0.08479500824296915) | > avg_grad_norm_0: tensor(266.9084, device='cuda:0') (+tensor(63.8361, device='cuda:0')) | > avg_loss_gen: 2.125014002894012 (+0.06654377386603594) | > avg_loss_kl: 2.0398197862463934 (-0.20244470784361956) | > avg_loss_feat: 3.6676933228130073 (+0.08611839925739151) | > avg_loss_mel: 16.906309396448272 (-0.7555178790025288) | > avg_loss_duration: 1.5667488524611568 (-0.046565405080016165) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.305585350788814 (-0.8518657684326172) | > avg_grad_norm_1: tensor(10681.8379, device='cuda:0') (+tensor(1586.2393, device='cuda:0'))  > EPOCH: 747/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:54:06)   --> TIME: 2025-02-25 15:54:14 -- STEP: 15/72 -- GLOBAL_STEP: 1053800 | > loss_disc: 2.503056287765503 (2.5416439374287925) | > loss_disc_real_0: 0.1691618412733078 (0.104061525563399) | > loss_disc_real_1: 0.23419268429279327 (0.21455021103223165) | > loss_disc_real_2: 0.24099916219711304 (0.23230492075284323) | > loss_disc_real_3: 0.25456786155700684 (0.2408622682094574) | > loss_disc_real_4: 0.26578769087791443 (0.2458463082710902) | > loss_disc_real_5: 0.27036842703819275 (0.2438215990861257) | > loss_0: 2.503056287765503 (2.5416439374287925) | > grad_norm_0: tensor(307.5154, device='cuda:0') (tensor(273.2760, device='cuda:0')) | > loss_gen: 2.6584057807922363 (2.206918398539225) | > loss_kl: 2.0183753967285156 (1.9421297391255696) | > loss_feat: 3.980891227722168 (3.735440429051717) | > loss_mel: 17.27109718322754 (16.76179135640462) | > loss_duration: 1.511844515800476 (1.5175275007883708) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.44061279296875 (26.163807169596353) | > grad_norm_1: tensor(9530.8154, device='cuda:0') (tensor(10725.2383, device='cuda:0')) | > current_lr_0: 0.00018216931013834408 | > current_lr_1: 0.00018216931013834408 | > step_time: 0.4349 (0.434250275293986) | > loader_time: 0.007 (0.006658728917439779)  --> TIME: 2025-02-25 15:54:38 -- STEP: 65/72 -- GLOBAL_STEP: 1053850 | > loss_disc: 2.5925564765930176 (2.5404796233543983) | > loss_disc_real_0: 0.07494226098060608 (0.10238542671387012) | > loss_disc_real_1: 0.22934190928936005 (0.2198529839515686) | > loss_disc_real_2: 0.2754889130592346 (0.22141937544712653) | > loss_disc_real_3: 0.19676326215267181 (0.23326982488999) | > loss_disc_real_4: 0.241235613822937 (0.24224367370972266) | > loss_disc_real_5: 0.24670656025409698 (0.24616817258871518) | > loss_0: 2.5925564765930176 (2.5404796233543983) | > grad_norm_0: tensor(323.1062, device='cuda:0') (tensor(253.9897, device='cuda:0')) | > loss_gen: 1.921722650527954 (2.2128995620287384) | > loss_kl: 2.2359118461608887 (2.0156681317549485) | > loss_feat: 3.5750274658203125 (3.8216666771815375) | > loss_mel: 17.559537887573242 (16.759687585097087) | > loss_duration: 1.612243413925171 (1.5573321782625638) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.904443740844727 (26.367254051795374) | > grad_norm_1: tensor(12971.7861, device='cuda:0') (tensor(11798.3066, device='cuda:0')) | > current_lr_0: 0.00018216931013834408 | > current_lr_1: 0.00018216931013834408 | > step_time: 0.4942 (0.4498214758359469) | > loader_time: 0.0115 (0.008352591441227837) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619459582046723 (+6.175376999546789e-06) | > avg_step_time: 0.4567832577396447 (+0.002952515239446951) | > avg_loss_disc: 2.5435097822001285 (-0.0033977132448006486) | > avg_loss_disc_real_0: 0.10364629349238436 (+0.0017400217622938197) | > avg_loss_disc_real_1: 0.22076012760820524 (+0.0020754916567198023) | > avg_loss_disc_real_2: 0.22153554709864334 (-0.00027865276370250824) | > avg_loss_disc_real_3: 0.2324931178714188 (+0.0029261630605643962) | > avg_loss_disc_real_4: 0.24167206736517624 (+0.0018201392721122478) | > avg_loss_disc_real_5: 0.24676868298523863 (-0.0005367964925900304) | > avg_loss_0: 2.5435097822001285 (-0.0033977132448006486) | > avg_grad_norm_0: tensor(257.2574, device='cuda:0') (tensor(-9.6510, device='cuda:0')) | > avg_loss_gen: 2.2065077398864315 (+0.08149373699241957) | > avg_loss_kl: 2.0233511840793446 (-0.016468602167048818) | > avg_loss_feat: 3.8106980491691913 (+0.143004726356184) | > avg_loss_mel: 16.806012422266136 (-0.10029697418213601) | > avg_loss_duration: 1.5627458112340578 (-0.004003041227099047) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.409315162981056 (+0.10372981219224187) | > avg_grad_norm_1: tensor(11835.8760, device='cuda:0') (+tensor(1154.0381, device='cuda:0'))  > EPOCH: 748/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:54:43)   --> TIME: 2025-02-25 15:55:04 -- STEP: 43/72 -- GLOBAL_STEP: 1053900 | > loss_disc: 2.8046040534973145 (2.5611829036890073) | > loss_disc_real_0: 0.04696187376976013 (0.09908909948412763) | > loss_disc_real_1: 0.23861747980117798 (0.22377239479575045) | > loss_disc_real_2: 0.23074471950531006 (0.2312286094177601) | > loss_disc_real_3: 0.22205322980880737 (0.23142909969008246) | > loss_disc_real_4: 0.22019940614700317 (0.23965229301951652) | > loss_disc_real_5: 0.22754402458667755 (0.2459671161202497) | > loss_0: 2.8046040534973145 (2.5611829036890073) | > grad_norm_0: tensor(332.4239, device='cuda:0') (tensor(247.8875, device='cuda:0')) | > loss_gen: 1.836242437362671 (2.209265398424725) | > loss_kl: 2.0497336387634277 (2.0130326332047925) | > loss_feat: 3.56315016746521 (3.9285518790400307) | > loss_mel: 16.508373260498047 (16.900336908739664) | > loss_duration: 1.570298433303833 (1.5358154330142708) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.527799606323242 (26.587002199749612) | > grad_norm_1: tensor(12133.8682, device='cuda:0') (tensor(11743.1768, device='cuda:0')) | > current_lr_0: 0.00018214653897457678 | > current_lr_1: 0.00018214653897457678 | > step_time: 0.4557 (0.44219473350879757) | > loader_time: 0.0097 (0.007544295732365098) --> EVAL PERFORMANCE | > avg_loader_time: 0.008720999032678737 (+0.00010153945063201379) | > avg_step_time: 0.4578771456866197 (+0.001093887946975014) | > avg_loss_disc: 2.5737224599005466 (+0.030212677700418133) | > avg_loss_disc_real_0: 0.10858625147334287 (+0.004939957980958518) | > avg_loss_disc_real_1: 0.22709470105842805 (+0.0063345734502228135) | > avg_loss_disc_real_2: 0.22670388914329906 (+0.005168342044655716) | > avg_loss_disc_real_3: 0.23345331805692593 (+0.0009602001855071318) | > avg_loss_disc_real_4: 0.24067189177157175 (-0.001000175593604491) | > avg_loss_disc_real_5: 0.24737592755069196 (+0.0006072445654533298) | > avg_loss_0: 2.5737224599005466 (+0.030212677700418133) | > avg_grad_norm_0: tensor(252.3120, device='cuda:0') (tensor(-4.9453, device='cuda:0')) | > avg_loss_gen: 2.1919706824799654 (-0.014537057406466047) | > avg_loss_kl: 2.063544451350897 (+0.04019326727155237) | > avg_loss_feat: 3.824562623467244 (+0.013864574298052545) | > avg_loss_mel: 16.850330661719948 (+0.04431823945381197) | > avg_loss_duration: 1.5613234865833336 (-0.001422324650724116) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.491731939181477 (+0.08241677620042154) | > avg_grad_norm_1: tensor(11563.2021, device='cuda:0') (tensor(-272.6738, device='cuda:0'))  > EPOCH: 749/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:55:20)   --> TIME: 2025-02-25 15:55:32 -- STEP: 21/72 -- GLOBAL_STEP: 1053950 | > loss_disc: 2.2089552879333496 (2.565690869376773) | > loss_disc_real_0: 0.09138499945402145 (0.10609321633265131) | > loss_disc_real_1: 0.22812709212303162 (0.21702822971911656) | > loss_disc_real_2: 0.23352721333503723 (0.2482854106596538) | > loss_disc_real_3: 0.2173408567905426 (0.24081578779788243) | > loss_disc_real_4: 0.22817999124526978 (0.244569046156747) | > loss_disc_real_5: 0.23935556411743164 (0.24368196016266233) | > loss_0: 2.2089552879333496 (2.565690869376773) | > grad_norm_0: tensor(155.9860, device='cuda:0') (tensor(225.6792, device='cuda:0')) | > loss_gen: 2.858093500137329 (2.266511570839655) | > loss_kl: 1.8969982862472534 (1.9277609643482028) | > loss_feat: 5.074736595153809 (3.8863366558438255) | > loss_mel: 17.12166976928711 (17.078407696315224) | > loss_duration: 1.5330320596694946 (1.5167933588936215) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.484529495239258 (26.675810223533993) | > grad_norm_1: tensor(11802.4326, device='cuda:0') (tensor(10540.1250, device='cuda:0')) | > current_lr_0: 0.00018212377065720495 | > current_lr_1: 0.00018212377065720495 | > step_time: 0.4392 (0.4403689588819231) | > loader_time: 0.0066 (0.007104635238647461)  --> TIME: 2025-02-25 15:55:56 -- STEP: 71/72 -- GLOBAL_STEP: 1054000 | > loss_disc: 2.6246562004089355 (2.568573669648506) | > loss_disc_real_0: 0.034557588398456573 (0.10303637406355898) | > loss_disc_real_1: 0.1862398087978363 (0.22087068247123504) | > loss_disc_real_2: 0.18830475211143494 (0.23014981960746603) | > loss_disc_real_3: 0.21806710958480835 (0.23298541089178812) | > loss_disc_real_4: 0.23120228946208954 (0.2390472154382249) | > loss_disc_real_5: 0.21842214465141296 (0.24428636717124724) | > loss_0: 2.6246562004089355 (2.568573669648506) | > grad_norm_0: tensor(285.9098, device='cuda:0') (tensor(242.3633, device='cuda:0')) | > loss_gen: 1.7306325435638428 (2.193519610754201) | > loss_kl: 2.2050442695617676 (2.0467640614845384) | > loss_feat: 3.838982582092285 (3.8031634814302686) | > loss_mel: 16.98889923095703 (16.87794279716385) | > loss_duration: 1.5632202625274658 (1.561897099857599) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.326780319213867 (26.483286871037013) | > grad_norm_1: tensor(14935.1523, device='cuda:0') (tensor(11550.4287, device='cuda:0')) | > current_lr_0: 0.00018212377065720495 | > current_lr_1: 0.00018212377065720495 | > step_time: 0.4716 (0.4585678711743422) | > loader_time: 0.008 (0.008755304444004113) --> EVAL PERFORMANCE | > avg_loader_time: 0.008755304444004113 (+3.4305411325376636e-05) | > avg_step_time: 0.4585678711743422 (+0.0006907254877225322) | > avg_loss_disc: 2.568573669648506 (-0.005148790252040403) | > avg_loss_disc_real_0: 0.10303637406355898 (-0.005549877409783896) | > avg_loss_disc_real_1: 0.22087068247123504 (-0.006224018587193014) | > avg_loss_disc_real_2: 0.23014981960746603 (+0.0034459304641669686) | > avg_loss_disc_real_3: 0.23298541089178812 (-0.0004679071651378097) | > avg_loss_disc_real_4: 0.2390472154382249 (-0.0016246763333468417) | > avg_loss_disc_real_5: 0.24428636717124724 (-0.003089560379444717) | > avg_loss_0: 2.568573669648506 (-0.005148790252040403) | > avg_grad_norm_0: tensor(242.3633, device='cuda:0') (tensor(-9.9487, device='cuda:0')) | > avg_loss_gen: 2.193519610754201 (+0.0015489282742358057) | > avg_loss_kl: 2.0467640614845384 (-0.016780389866358547) | > avg_loss_feat: 3.8031634814302686 (-0.02139914203697524) | > avg_loss_mel: 16.87794279716385 (+0.02761213544390273) | > avg_loss_duration: 1.561897099857599 (+0.0005736132742653055) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.483286871037013 (-0.00844506814446433) | > avg_grad_norm_1: tensor(11550.4287, device='cuda:0') (tensor(-12.7734, device='cuda:0'))  > EPOCH: 750/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:55:57)   --> TIME: 2025-02-25 15:56:21 -- STEP: 49/72 -- GLOBAL_STEP: 1054050 | > loss_disc: 2.8330721855163574 (2.6004003894572363) | > loss_disc_real_0: 0.08876378834247589 (0.11349817365407944) | > loss_disc_real_1: 0.22071899473667145 (0.22829268781506284) | > loss_disc_real_2: 0.24358607828617096 (0.21786918019761845) | > loss_disc_real_3: 0.26825687289237976 (0.23296244594515586) | > loss_disc_real_4: 0.2420562505722046 (0.24226428020973595) | > loss_disc_real_5: 0.2417333722114563 (0.24738233734150322) | > loss_0: 2.8330721855163574 (2.6004003894572363) | > grad_norm_0: tensor(123.6207, device='cuda:0') (tensor(253.6379, device='cuda:0')) | > loss_gen: 1.8620326519012451 (2.1792924331159007) | > loss_kl: 2.152907609939575 (2.0324073616339238) | > loss_feat: 3.0335168838500977 (3.800128167989303) | > loss_mel: 16.6657657623291 (16.93499031845404) | > loss_duration: 1.6234605312347412 (1.5409843094494877) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.337682723999023 (26.487802661195094) | > grad_norm_1: tensor(8962.8730, device='cuda:0') (tensor(10991.1084, device='cuda:0')) | > current_lr_0: 0.0001821010051858728 | > current_lr_1: 0.0001821010051858728 | > step_time: 0.4415 (0.43987560758785327) | > loader_time: 0.0098 (0.007756306200611348) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688016676566974 (-6.728776743713909e-05) | > avg_step_time: 0.4631072030940526 (+0.004539331919710399) | > avg_loss_disc: 2.57528558247526 (+0.006711912826753874) | > avg_loss_disc_real_0: 0.12070234518655588 (+0.017665971122996907) | > avg_loss_disc_real_1: 0.2260212589737395 (+0.0051505765025044725) | > avg_loss_disc_real_2: 0.2172753258909978 (-0.012874493716468233) | > avg_loss_disc_real_3: 0.23274769602527082 (-0.00023771486651730234) | > avg_loss_disc_real_4: 0.2407105930254493 (+0.0016633775872244017) | > avg_loss_disc_real_5: 0.24578071585003758 (+0.001494348678790336) | > avg_loss_0: 2.57528558247526 (+0.006711912826753874) | > avg_grad_norm_0: tensor(257.6817, device='cuda:0') (+tensor(15.3185, device='cuda:0')) | > avg_loss_gen: 2.205832966616456 (+0.012313355862254571) | > avg_loss_kl: 2.062886706540283 (+0.016122645055744567) | > avg_loss_feat: 3.809763639745578 (+0.006600158315309468) | > avg_loss_mel: 17.0135408723858 (+0.13559807522194944) | > avg_loss_duration: 1.5601841798970397 (-0.0017129199605592227) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.652208408839265 (+0.16892153780225172) | > avg_grad_norm_1: tensor(10980.3594, device='cuda:0') (tensor(-570.0693, device='cuda:0'))  > EPOCH: 751/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:56:35)   --> TIME: 2025-02-25 15:56:49 -- STEP: 27/72 -- GLOBAL_STEP: 1054100 | > loss_disc: 2.6847338676452637 (2.637658110371343) | > loss_disc_real_0: 0.18636417388916016 (0.09871387592068424) | > loss_disc_real_1: 0.19542810320854187 (0.2217521350140925) | > loss_disc_real_2: 0.24918010830879211 (0.22422781752215493) | > loss_disc_real_3: 0.25446775555610657 (0.2198463159578818) | > loss_disc_real_4: 0.2503233551979065 (0.23198396022673007) | > loss_disc_real_5: 0.24759000539779663 (0.24760177124429628) | > loss_0: 2.6847338676452637 (2.637658110371343) | > grad_norm_0: tensor(477.4276, device='cuda:0') (tensor(282.9897, device='cuda:0')) | > loss_gen: 2.038141965866089 (2.077204214202033) | > loss_kl: 2.1578621864318848 (2.02902329409564) | > loss_feat: 3.4378769397735596 (3.7420693415182607) | > loss_mel: 17.178089141845703 (17.14835251702203) | > loss_duration: 1.522153377532959 (1.5214676680388275) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.334123611450195 (26.518117127595126) | > grad_norm_1: tensor(6737.9951, device='cuda:0') (tensor(11102.5166, device='cuda:0')) | > current_lr_0: 0.00018207824256022455 | > current_lr_1: 0.00018207824256022455 | > step_time: 0.474 (0.4443633997881854) | > loader_time: 0.0094 (0.0070488717820909284) --> EVAL PERFORMANCE | > avg_loader_time: 0.008919440524678834 (+0.00023142384811185995) | > avg_step_time: 0.4587593347253934 (-0.004347868368659247) | > avg_loss_disc: 2.69886893285832 (+0.1235833503830599) | > avg_loss_disc_real_0: 0.17429421506297418 (+0.0535918698764183) | > avg_loss_disc_real_1: 0.2250388968788402 (-0.0009823620948992973) | > avg_loss_disc_real_2: 0.22930447085642477 (+0.01202914496542698) | > avg_loss_disc_real_3: 0.2262495258744334 (-0.0064981701508374234) | > avg_loss_disc_real_4: 0.23537002797697631 (-0.005340565048472995) | > avg_loss_disc_real_5: 0.24668527488977138 (+0.0009045590397337988) | > avg_loss_0: 2.69886893285832 (+0.1235833503830599) | > avg_grad_norm_0: tensor(274.5221, device='cuda:0') (+tensor(16.8404, device='cuda:0')) | > avg_loss_gen: 2.1316574912675668 (-0.07417547534888902) | > avg_loss_kl: 2.0978976169102634 (+0.0350109103699805) | > avg_loss_feat: 3.580378613001864 (-0.22938502674371408) | > avg_loss_mel: 17.036749450253772 (+0.02320857786797248) | > avg_loss_duration: 1.5634250976669957 (+0.003240917769955942) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.410108378235723 (-0.24210003060354168) | > avg_grad_norm_1: tensor(9986.8037, device='cuda:0') (tensor(-993.5557, device='cuda:0'))  > EPOCH: 752/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:57:12)   --> TIME: 2025-02-25 15:57:16 -- STEP: 5/72 -- GLOBAL_STEP: 1054150 | > loss_disc: 2.627603530883789 (2.829727220535278) | > loss_disc_real_0: 0.1815010905265808 (0.25317818820476534) | > loss_disc_real_1: 0.2782474756240845 (0.2549132913351059) | > loss_disc_real_2: 0.3561522662639618 (0.27932384312152864) | > loss_disc_real_3: 0.3077034056186676 (0.2536033868789673) | > loss_disc_real_4: 0.3363582193851471 (0.2597604006528854) | > loss_disc_real_5: 0.2767115831375122 (0.2368670165538788) | > loss_0: 2.627603530883789 (2.829727220535278) | > grad_norm_0: tensor(225.3026, device='cuda:0') (tensor(215.7083, device='cuda:0')) | > loss_gen: 2.5841569900512695 (2.300112080574036) | > loss_kl: 3.0291006565093994 (2.4824607372283936) | > loss_feat: 4.020364761352539 (3.733136606216431) | > loss_mel: 24.543188095092773 (21.44432258605957) | > loss_duration: 1.962368130683899 (1.6849678039550782) | > amp_scaler: 8.0 (8.0) | > loss_1: 36.13917922973633 (31.645000457763672) | > grad_norm_1: tensor(3461.6650, device='cuda:0') (tensor(4486.8984, device='cuda:0')) | > current_lr_0: 0.00018205548277990452 | > current_lr_1: 0.00018205548277990452 | > step_time: 0.4277 (0.4305396556854248) | > loader_time: 0.0064 (0.0056193828582763675)  --> TIME: 2025-02-25 15:57:39 -- STEP: 55/72 -- GLOBAL_STEP: 1054200 | > loss_disc: 3.023282766342163 (2.7252281839197336) | > loss_disc_real_0: 0.4071848392486572 (0.18332168094136495) | > loss_disc_real_1: 0.3889869749546051 (0.2417376537214626) | > loss_disc_real_2: 0.15814059972763062 (0.2316562601111152) | > loss_disc_real_3: 0.23696254193782806 (0.24583258574659175) | > loss_disc_real_4: 0.22766441106796265 (0.24736608483574607) | > loss_disc_real_5: 0.24276427924633026 (0.24248885891654276) | > loss_0: 3.023282766342163 (2.7252281839197336) | > grad_norm_0: tensor(389.7007, device='cuda:0') (tensor(177.3922, device='cuda:0')) | > loss_gen: 1.9609439373016357 (2.1639732989397906) | > loss_kl: 2.2122550010681152 (2.5372406916184853) | > loss_feat: 2.958613395690918 (3.5865430571816184) | > loss_mel: 18.895801544189453 (20.249840857765893) | > loss_duration: 1.6561341285705566 (1.638084996830333) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.683746337890625 (30.175682657415216) | > grad_norm_1: tensor(1116.2402, device='cuda:0') (tensor(6619.6685, device='cuda:0')) | > current_lr_0: 0.00018205548277990452 | > current_lr_1: 0.00018205548277990452 | > step_time: 0.4709 (0.44351693933660336) | > loader_time: 0.0092 (0.007820146734064276) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857000619592801 (-0.00034943432875082434) | > avg_step_time: 0.4544485824208864 (-0.0043107523045069485) | > avg_loss_disc: 2.748042573391552 (+0.04917364053323192) | > avg_loss_disc_real_0: 0.1971346270030653 (+0.02284041194009112) | > avg_loss_disc_real_1: 0.24612531905442897 (+0.02108642217558876) | > avg_loss_disc_real_2: 0.2324642473962945 (+0.0031597765398697353) | > avg_loss_disc_real_3: 0.24284311100630693 (+0.016593585131873534) | > avg_loss_disc_real_4: 0.24821539223194122 (+0.012845364254964908) | > avg_loss_disc_real_5: 0.24284968535665055 (-0.0038355895331208256) | > avg_loss_0: 2.748042573391552 (+0.04917364053323192) | > avg_grad_norm_0: tensor(188.0270, device='cuda:0') (tensor(-86.4951, device='cuda:0')) | > avg_loss_gen: 2.142791531455348 (+0.011134040187781302) | > avg_loss_kl: 2.488476192447501 (+0.3905785755372375) | > avg_loss_feat: 3.529908764530236 (-0.050469848471628165) | > avg_loss_mel: 19.79074134289379 (+2.753991892640016) | > avg_loss_duration: 1.6389643860534884 (+0.07553928838649271) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.590882019257883 (+3.18077364102216) | > avg_grad_norm_1: tensor(6132.9429, device='cuda:0') (tensor(-3853.8608, device='cuda:0'))  > EPOCH: 753/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:57:49)   --> TIME: 2025-02-25 15:58:06 -- STEP: 33/72 -- GLOBAL_STEP: 1054250 | > loss_disc: 2.5800182819366455 (2.6889037363456962) | > loss_disc_real_0: 0.1289602369070053 (0.17246928991693444) | > loss_disc_real_1: 0.21331565082073212 (0.24923001861933505) | > loss_disc_real_2: 0.2132217437028885 (0.2292294872529579) | > loss_disc_real_3: 0.24574615061283112 (0.24580778407328058) | > loss_disc_real_4: 0.20746508240699768 (0.2376455172444835) | > loss_disc_real_5: 0.22737371921539307 (0.242643547780586) | > loss_0: 2.5800182819366455 (2.6889037363456962) | > grad_norm_0: tensor(115.3450, device='cuda:0') (tensor(130.1564, device='cuda:0')) | > loss_gen: 2.2434024810791016 (2.101127494465222) | > loss_kl: 2.6995792388916016 (3.187507398200758) | > loss_feat: 4.222081661224365 (3.416657902977683) | > loss_mel: 20.205625534057617 (21.400847695090558) | > loss_duration: 1.709326982498169 (1.8436663512027625) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.08001708984375 (31.949807311549332) | > grad_norm_1: tensor(8191.6733, device='cuda:0') (tensor(6274.4604, device='cuda:0')) | > current_lr_0: 0.00018203272584455703 | > current_lr_1: 0.00018203272584455703 | > step_time: 0.438 (0.43811137748487067) | > loader_time: 0.0079 (0.0072126099557587595) --> EVAL PERFORMANCE | > avg_loader_time: 0.008730985748935752 (+0.00016097955300774236) | > avg_step_time: 0.4562772831446688 (+0.0018287007237823838) | > avg_loss_disc: 2.720154691749895 (-0.027887881641656787) | > avg_loss_disc_real_0: 0.147843920009237 (-0.049290706993828315) | > avg_loss_disc_real_1: 0.2399166679718125 (-0.006208651082616479) | > avg_loss_disc_real_2: 0.23584655808730864 (+0.0033823106910141343) | > avg_loss_disc_real_3: 0.24153962878274246 (-0.0013034822235644672) | > avg_loss_disc_real_4: 0.24359807078267487 (-0.004617321449266354) | > avg_loss_disc_real_5: 0.24459906216238586 (+0.0017493768057353087) | > avg_loss_0: 2.720154691749895 (-0.027887881641656787) | > avg_grad_norm_0: tensor(137.3159, device='cuda:0') (tensor(-50.7111, device='cuda:0')) | > avg_loss_gen: 1.9792060432299778 (-0.16358548822537022) | > avg_loss_kl: 2.724624042779627 (+0.23614785033212593) | > avg_loss_feat: 3.264805656083872 (-0.26510310844636376) | > avg_loss_mel: 19.777971213972066 (-0.012770128921722801) | > avg_loss_duration: 1.7532143173083452 (+0.11424993125485683) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.499821568878605 (-0.09106045037927757) | > avg_grad_norm_1: tensor(6030.5186, device='cuda:0') (tensor(-102.4243, device='cuda:0'))  > EPOCH: 754/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:58:26)   --> TIME: 2025-02-25 15:58:33 -- STEP: 11/72 -- GLOBAL_STEP: 1054300 | > loss_disc: 2.763636350631714 (2.7367211905392734) | > loss_disc_real_0: 0.04273298382759094 (0.21589033508842642) | > loss_disc_real_1: 0.14730004966259003 (0.2869120646606792) | > loss_disc_real_2: 0.19665992259979248 (0.2558950185775757) | > loss_disc_real_3: 0.21357491612434387 (0.22628882933746686) | > loss_disc_real_4: 0.22245363891124725 (0.2437682490457188) | > loss_disc_real_5: 0.22745315730571747 (0.2363457828760147) | > loss_0: 2.763636350631714 (2.7367211905392734) | > grad_norm_0: tensor(189.7443, device='cuda:0') (tensor(195.2325, device='cuda:0')) | > loss_gen: 1.478398323059082 (2.0433862426064233) | > loss_kl: 2.086393117904663 (2.0401699326255103) | > loss_feat: 3.4573917388916016 (3.2590102715925737) | > loss_mel: 18.139404296875 (17.687576987526633) | > loss_duration: 1.565140724182129 (1.5735172033309937) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.726726531982422 (26.603660756891426) | > grad_norm_1: tensor(9402.2529, device='cuda:0') (tensor(6538.7280, device='cuda:0')) | > current_lr_0: 0.00018200997175382645 | > current_lr_1: 0.00018200997175382645 | > step_time: 0.4236 (0.4351524656469172) | > loader_time: 0.006 (0.006562666459517045)  --> TIME: 2025-02-25 15:58:57 -- STEP: 61/72 -- GLOBAL_STEP: 1054350 | > loss_disc: 2.80399489402771 (2.69689983618064) | > loss_disc_real_0: 0.14720779657363892 (0.15435040681088552) | > loss_disc_real_1: 0.1814705729484558 (0.22153235080300784) | > loss_disc_real_2: 0.289963036775589 (0.22655527049400767) | > loss_disc_real_3: 0.19845089316368103 (0.23030229157111684) | > loss_disc_real_4: 0.25373536348342896 (0.24027709956051874) | > loss_disc_real_5: 0.24428550899028778 (0.24430206589034345) | > loss_0: 2.80399489402771 (2.69689983618064) | > grad_norm_0: tensor(114.3705, device='cuda:0') (tensor(167.5138, device='cuda:0')) | > loss_gen: 1.664323329925537 (2.009172121032339) | > loss_kl: 2.0503623485565186 (2.099160565704596) | > loss_feat: 2.7327229976654053 (3.465148042459957) | > loss_mel: 16.829784393310547 (17.64797231017566) | > loss_duration: 1.6117970943450928 (1.5844776122296442) | > amp_scaler: 16.0 (14.163934426229504) | > loss_1: 24.88899040222168 (26.80593077862849) | > grad_norm_1: tensor(1787.5681, device='cuda:0') (tensor(5496.0493, device='cuda:0')) | > current_lr_0: 0.00018200997175382645 | > current_lr_1: 0.00018200997175382645 | > step_time: 0.4869 (0.4603053780852771) | > loader_time: 0.0109 (0.00830770320579654) --> EVAL PERFORMANCE | > avg_loader_time: 0.008764918421355771 (+3.393267242001938e-05) | > avg_step_time: 0.471013132954987 (+0.014735849810318169) | > avg_loss_disc: 2.683467082574334 (-0.03668760917556124) | > avg_loss_disc_real_0: 0.14430687335175524 (-0.0035370466574817516) | > avg_loss_disc_real_1: 0.2223629322899899 (-0.017553735681822585) | > avg_loss_disc_real_2: 0.22525350686529994 (-0.01059305122200871) | > avg_loss_disc_real_3: 0.23071046950111926 (-0.010829159281623202) | > avg_loss_disc_real_4: 0.23840853718804642 (-0.005189533594628454) | > avg_loss_disc_real_5: 0.2448123317789024 (+0.0002132696165165482) | > avg_loss_0: 2.683467082574334 (-0.03668760917556124) | > avg_grad_norm_0: tensor(162.5393, device='cuda:0') (+tensor(25.2234, device='cuda:0')) | > avg_loss_gen: 2.0153422708242714 (+0.03613622759429358) | > avg_loss_kl: 2.1093814154745836 (-0.6152426273050433) | > avg_loss_feat: 3.481273160853856 (+0.21646750476998378) | > avg_loss_mel: 17.63700980871496 (-2.140961405257105) | > avg_loss_duration: 1.5900109905592152 (-0.16320332674912996) | > avg_amp_scaler: 14.422535211267602 (+6.422535211267602) | > avg_loss_1: 26.833017725340078 (-2.6668038435385277) | > avg_grad_norm_1: tensor(5649.3052, device='cuda:0') (tensor(-381.2134, device='cuda:0'))  > EPOCH: 755/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:59:04)   --> TIME: 2025-02-25 15:59:24 -- STEP: 39/72 -- GLOBAL_STEP: 1054400 | > loss_disc: 2.2221696376800537 (2.555784897926526) | > loss_disc_real_0: 0.12465721368789673 (0.10894706291265976) | > loss_disc_real_1: 0.2559383809566498 (0.22117525530167115) | > loss_disc_real_2: 0.1493358612060547 (0.2149584002983876) | > loss_disc_real_3: 0.14745302498340607 (0.23217890774592376) | > loss_disc_real_4: 0.2293395847082138 (0.24307854550007063) | > loss_disc_real_5: 0.2337130457162857 (0.24606883449432174) | > loss_0: 2.2221696376800537 (2.555784897926526) | > grad_norm_0: tensor(75.2974, device='cuda:0') (tensor(171.2096, device='cuda:0')) | > loss_gen: 2.754504680633545 (2.1195176259065294) | > loss_kl: 2.0379154682159424 (1.9642801743287306) | > loss_feat: 4.7717695236206055 (3.6575321478721423) | > loss_mel: 18.11870765686035 (17.245192601130558) | > loss_duration: 1.5753065347671509 (1.5421336094538372) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.258203506469727 (26.52865605476575) | > grad_norm_1: tensor(9129.2432, device='cuda:0') (tensor(7921.1460, device='cuda:0')) | > current_lr_0: 0.0001819872205073572 | > current_lr_1: 0.0001819872205073572 | > step_time: 0.453 (0.44129366141099197) | > loader_time: 0.0084 (0.007376371285854242) --> EVAL PERFORMANCE | > avg_loader_time: 0.008641568707748197 (-0.0001233497136075741) | > avg_step_time: 0.4583335661552322 (-0.012679566799754793) | > avg_loss_disc: 2.5862591568852813 (-0.09720792568905257) | > avg_loss_disc_real_0: 0.1151806364806605 (-0.029126236871094738) | > avg_loss_disc_real_1: 0.22031974036928634 (-0.0020431919207035687) | > avg_loss_disc_real_2: 0.22577068325079663 (+0.0005171763854966982) | > avg_loss_disc_real_3: 0.23745648940684091 (+0.006746019905721656) | > avg_loss_disc_real_4: 0.24147490821254086 (+0.0030663710244944464) | > avg_loss_disc_real_5: 0.24782476962452205 (+0.0030124378456196366) | > avg_loss_0: 2.5862591568852813 (-0.09720792568905257) | > avg_grad_norm_0: tensor(192.7197, device='cuda:0') (+tensor(30.1804, device='cuda:0')) | > avg_loss_gen: 2.103611220776196 (+0.08826894995192447) | > avg_loss_kl: 1.9964993940272802 (-0.11288202144730342) | > avg_loss_feat: 3.551683593803728 (+0.07041043294987226) | > avg_loss_mel: 17.185217709608477 (-0.4517920991064841) | > avg_loss_duration: 1.5696299344720974 (-0.02038105608711782) | > avg_amp_scaler: 16.0 (+1.5774647887323976) | > avg_loss_1: 26.406641758663554 (-0.4263759666765239) | > avg_grad_norm_1: tensor(8486.4316, device='cuda:0') (+tensor(2837.1265, device='cuda:0'))  > EPOCH: 756/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 15:59:41)   --> TIME: 2025-02-25 15:59:51 -- STEP: 17/72 -- GLOBAL_STEP: 1054450 | > loss_disc: 2.561678409576416 (2.53572081117069) | > loss_disc_real_0: 0.1214548796415329 (0.12462946465786766) | > loss_disc_real_1: 0.2002607136964798 (0.21379208038834965) | > loss_disc_real_2: 0.19116443395614624 (0.21905880728188684) | > loss_disc_real_3: 0.2376461625099182 (0.23283503073103287) | > loss_disc_real_4: 0.2165989726781845 (0.23531570417039535) | > loss_disc_real_5: 0.23493187129497528 (0.2411336881272933) | > loss_0: 2.561678409576416 (2.53572081117069) | > grad_norm_0: tensor(140.2972, device='cuda:0') (tensor(202.5946, device='cuda:0')) | > loss_gen: 1.9855350255966187 (2.1745059069465187) | > loss_kl: 2.128398895263672 (1.9274427750531364) | > loss_feat: 3.677182197570801 (3.7008600795970246) | > loss_mel: 17.970869064331055 (17.03824093762566) | > loss_duration: 1.5521247386932373 (1.5224516602123486) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.314109802246094 (26.363501156077664) | > grad_norm_1: tensor(11231.1260, device='cuda:0') (tensor(9367.3643, device='cuda:0')) | > current_lr_0: 0.00018196447210479377 | > current_lr_1: 0.00018196447210479377 | > step_time: 0.4294 (0.43616369191338034) | > loader_time: 0.006 (0.006853117662317613)  --> TIME: 2025-02-25 16:00:15 -- STEP: 67/72 -- GLOBAL_STEP: 1054500 | > loss_disc: 2.340008497238159 (2.594259461360191) | > loss_disc_real_0: 0.1023414134979248 (0.14908920495367764) | > loss_disc_real_1: 0.1871563047170639 (0.21766225059530628) | > loss_disc_real_2: 0.2724516689777374 (0.23091347181974953) | > loss_disc_real_3: 0.23558662831783295 (0.2350286503336323) | > loss_disc_real_4: 0.2536240518093109 (0.2451512181936805) | > loss_disc_real_5: 0.2569079101085663 (0.24613913649053715) | > loss_0: 2.340008497238159 (2.594259461360191) | > grad_norm_0: tensor(108.9342, device='cuda:0') (tensor(201.6845, device='cuda:0')) | > loss_gen: 2.443774938583374 (2.156535308752486) | > loss_kl: 2.153278112411499 (1.978035220459326) | > loss_feat: 4.059581756591797 (3.510025547511542) | > loss_mel: 17.66779327392578 (17.010904155560393) | > loss_duration: 1.6547858715057373 (1.5607821460980087) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.97921371459961 (26.21628248869483) | > grad_norm_1: tensor(9011.6387, device='cuda:0') (tensor(9090.7666, device='cuda:0')) | > current_lr_0: 0.00018196447210479377 | > current_lr_1: 0.00018196447210479377 | > step_time: 0.5133 (0.4525007596656458) | > loader_time: 0.0124 (0.008583986937110103) --> EVAL PERFORMANCE | > avg_loader_time: 0.008748192182728941 (+0.00010662347498074411) | > avg_step_time: 0.4565933254403128 (-0.0017402407149194055) | > avg_loss_disc: 2.5968159487549687 (+0.010556791869687387) | > avg_loss_disc_real_0: 0.14487901655301244 (+0.02969838007235194) | > avg_loss_disc_real_1: 0.21958631801773126 (-0.0007334223515550742) | > avg_loss_disc_real_2: 0.23081874049885173 (+0.005048057248055099) | > avg_loss_disc_real_3: 0.23476944318119908 (-0.002687046225641837) | > avg_loss_disc_real_4: 0.24447611603938357 (+0.003001207826842711) | > avg_loss_disc_real_5: 0.2462150817605811 (-0.001609687863940945) | > avg_loss_0: 2.5968159487549687 (+0.010556791869687387) | > avg_grad_norm_0: tensor(205.1437, device='cuda:0') (+tensor(12.4240, device='cuda:0')) | > avg_loss_gen: 2.1460113491810535 (+0.04240012840485763) | > avg_loss_kl: 1.9859667344832084 (-0.010532659544071787) | > avg_loss_feat: 3.5135299957973856 (-0.03815359800634255) | > avg_loss_mel: 17.034829757582973 (-0.15038795202550403) | > avg_loss_duration: 1.5637582809152737 (-0.0058716535568237305) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.24409621870014 (-0.16254553996341414) | > avg_grad_norm_1: tensor(9125.2285, device='cuda:0') (+tensor(638.7969, device='cuda:0'))  > EPOCH: 757/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:00:18)   --> TIME: 2025-02-25 16:00:41 -- STEP: 45/72 -- GLOBAL_STEP: 1054550 | > loss_disc: 2.6703433990478516 (2.6704286204444045) | > loss_disc_real_0: 0.13096541166305542 (0.1417850913272963) | > loss_disc_real_1: 0.19307838380336761 (0.21697285771369934) | > loss_disc_real_2: 0.27289968729019165 (0.23106289737754399) | > loss_disc_real_3: 0.23963454365730286 (0.23425889412562054) | > loss_disc_real_4: 0.2452787607908249 (0.23974478708373176) | > loss_disc_real_5: 0.2720673382282257 (0.2443684028254615) | > loss_0: 2.6703433990478516 (2.6704286204444045) | > grad_norm_0: tensor(274.0240, device='cuda:0') (tensor(231.4384, device='cuda:0')) | > loss_gen: 2.110380172729492 (2.007172944810656) | > loss_kl: 2.153317451477051 (1.9780426714155408) | > loss_feat: 3.86850905418396 (3.39837171236674) | > loss_mel: 17.462549209594727 (16.890907181633846) | > loss_duration: 1.5896129608154297 (1.5384461906221178) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.184368133544922 (25.81294051276313) | > grad_norm_1: tensor(6131.3809, device='cuda:0') (tensor(8285.4854, device='cuda:0')) | > current_lr_0: 0.00018194172654578068 | > current_lr_1: 0.00018194172654578068 | > step_time: 0.4701 (0.44092132250467936) | > loader_time: 0.009 (0.007569710413614909) --> EVAL PERFORMANCE | > avg_loader_time: 0.008622522085485324 (-0.00012567009724361727) | > avg_step_time: 0.4619531832950216 (+0.005359857854708827) | > avg_loss_disc: 2.6694400511996856 (+0.07262410244471695) | > avg_loss_disc_real_0: 0.14702716553714912 (+0.0021481489841366774) | > avg_loss_disc_real_1: 0.21697642748624504 (-0.0026098905314862197) | > avg_loss_disc_real_2: 0.23580928020913836 (+0.004990539710286623) | > avg_loss_disc_real_3: 0.23568301532470004 (+0.0009135721435009625) | > avg_loss_disc_real_4: 0.24210187184139037 (-0.0023742441979932027) | > avg_loss_disc_real_5: 0.24586150784727553 (-0.000353573913305566) | > avg_loss_0: 2.6694400511996856 (+0.07262410244471695) | > avg_grad_norm_0: tensor(240.2365, device='cuda:0') (+tensor(35.0928, device='cuda:0')) | > avg_loss_gen: 2.035756332773557 (-0.11025501640749669) | > avg_loss_kl: 2.013965359875854 (+0.027998625392645415) | > avg_loss_feat: 3.406211107549533 (-0.1073188882478524) | > avg_loss_mel: 16.88985832644181 (-0.14497143114116184) | > avg_loss_duration: 1.5612801665991125 (-0.002478114316161184) | > avg_amp_scaler: 14.535211267605634 (-1.464788732394366) | > avg_loss_1: 25.907071153882523 (-0.33702506481761674) | > avg_grad_norm_1: tensor(8489.6191, device='cuda:0') (tensor(-635.6094, device='cuda:0'))  > EPOCH: 758/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:00:56)   --> TIME: 2025-02-25 16:01:08 -- STEP: 23/72 -- GLOBAL_STEP: 1054600 | > loss_disc: 2.5995590686798096 (2.717785866364189) | > loss_disc_real_0: 0.0987311452627182 (0.12062039647413336) | > loss_disc_real_1: 0.3059728741645813 (0.22792994004228842) | > loss_disc_real_2: 0.13534288108348846 (0.30974181255568634) | > loss_disc_real_3: 0.168359637260437 (0.24508471527825232) | > loss_disc_real_4: 0.24024273455142975 (0.26751709697039233) | > loss_disc_real_5: 0.26052749156951904 (0.255230747487234) | > loss_0: 2.5995590686798096 (2.717785866364189) | > grad_norm_0: tensor(66.5114, device='cuda:0') (tensor(223.2361, device='cuda:0')) | > loss_gen: 2.3227505683898926 (2.216844071512637) | > loss_kl: 1.9297816753387451 (1.9945280759230903) | > loss_feat: 4.3846516609191895 (3.713941190553748) | > loss_mel: 17.550840377807617 (17.309519394584324) | > loss_duration: 1.5420398712158203 (1.5231875388518623) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.73006248474121 (26.758020235144574) | > grad_norm_1: tensor(8453.7930, device='cuda:0') (tensor(8495.7871, device='cuda:0')) | > current_lr_0: 0.00018191898382996245 | > current_lr_1: 0.00018191898382996245 | > step_time: 0.4354 (0.4367144315139107) | > loader_time: 0.0069 (0.006736465122388757) --> EVAL PERFORMANCE | > avg_loader_time: 0.008733638575379277 (+0.00011111648989395317) | > avg_step_time: 0.4564522085055499 (-0.005500974789471702) | > avg_loss_disc: 2.722940871413324 (+0.053500820213638445) | > avg_loss_disc_real_0: 0.1364799618301257 (-0.010547203707023417) | > avg_loss_disc_real_1: 0.24788325615751922 (+0.030906828671274172) | > avg_loss_disc_real_2: 0.2642160458883768 (+0.02840676567923847) | > avg_loss_disc_real_3: 0.23008773843167532 (-0.00559527689302472) | > avg_loss_disc_real_4: 0.2527371260901572 (+0.010635254248766823) | > avg_loss_disc_real_5: 0.2465654861339381 (+0.0007039782866625766) | > avg_loss_0: 2.722940871413324 (+0.053500820213638445) | > avg_grad_norm_0: tensor(237.0296, device='cuda:0') (tensor(-3.2069, device='cuda:0')) | > avg_loss_gen: 2.106546150126927 (+0.07078981735337031) | > avg_loss_kl: 2.246721836882577 (+0.2327564770067232) | > avg_loss_feat: 3.457738110716914 (+0.05152700316738068) | > avg_loss_mel: 18.017048513385607 (+1.1271901869437961) | > avg_loss_duration: 1.6068073998034837 (+0.04552723320437124) | > avg_amp_scaler: 8.0 (-6.535211267605634) | > avg_loss_1: 27.434862002520493 (+1.52779084863797) | > avg_grad_norm_1: tensor(8601.9551, device='cuda:0') (+tensor(112.3359, device='cuda:0'))  > EPOCH: 759/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:01:33)   --> TIME: 2025-02-25 16:01:35 -- STEP: 1/72 -- GLOBAL_STEP: 1054650 | > loss_disc: 2.6394851207733154 (2.6394851207733154) | > loss_disc_real_0: 0.23431634902954102 (0.23431634902954102) | > loss_disc_real_1: 0.28910601139068604 (0.28910601139068604) | > loss_disc_real_2: 0.23851777613162994 (0.23851777613162994) | > loss_disc_real_3: 0.24058307707309723 (0.24058307707309723) | > loss_disc_real_4: 0.24458380043506622 (0.24458380043506622) | > loss_disc_real_5: 0.22417663037776947 (0.22417663037776947) | > loss_0: 2.6394851207733154 (2.6394851207733154) | > grad_norm_0: tensor(161.1485, device='cuda:0') (tensor(161.1485, device='cuda:0')) | > loss_gen: 2.0746257305145264 (2.0746257305145264) | > loss_kl: 2.3571763038635254 (2.3571763038635254) | > loss_feat: 3.137972116470337 (3.137972116470337) | > loss_mel: 17.805997848510742 (17.805997848510742) | > loss_duration: 1.6030611991882324 (1.6030611991882324) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.97883415222168 (26.97883415222168) | > grad_norm_1: tensor(8248.2461, device='cuda:0') (tensor(8248.2461, device='cuda:0')) | > current_lr_0: 0.0001818962439569837 | > current_lr_1: 0.0001818962439569837 | > step_time: 0.4279 (0.4279050827026367) | > loader_time: 0.0049 (0.004937410354614258)  --> TIME: 2025-02-25 16:01:58 -- STEP: 51/72 -- GLOBAL_STEP: 1054700 | > loss_disc: 2.3471741676330566 (2.630598783493042) | > loss_disc_real_0: 0.14775747060775757 (0.12974259016268397) | > loss_disc_real_1: 0.22119392454624176 (0.22134371394035862) | > loss_disc_real_2: 0.2159523367881775 (0.2307795353379904) | > loss_disc_real_3: 0.24751275777816772 (0.23796695529245862) | > loss_disc_real_4: 0.26348966360092163 (0.2480742352850297) | > loss_disc_real_5: 0.259533166885376 (0.25082898373697315) | > loss_0: 2.3471741676330566 (2.630598783493042) | > grad_norm_0: tensor(270.4460, device='cuda:0') (tensor(204.4912, device='cuda:0')) | > loss_gen: 2.7156896591186523 (2.101586180574753) | > loss_kl: 2.1093146800994873 (2.184592375568315) | > loss_feat: 4.255230903625488 (3.551547985450894) | > loss_mel: 16.7373046875 (17.535958271400602) | > loss_duration: 1.6027449369430542 (1.5691255237541946) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.420284271240234 (26.942810432583677) | > grad_norm_1: tensor(9522.4795, device='cuda:0') (tensor(8651.7568, device='cuda:0')) | > current_lr_0: 0.0001818962439569837 | > current_lr_1: 0.0001818962439569837 | > step_time: 0.4505 (0.44512585097668217) | > loader_time: 0.0091 (0.007869271671070772) --> EVAL PERFORMANCE | > avg_loader_time: 0.008706509227484045 (-2.7129347895232456e-05) | > avg_step_time: 0.45725965164077115 (+0.0008074431352212419) | > avg_loss_disc: 2.6055379048199723 (-0.11740296659335181) | > avg_loss_disc_real_0: 0.1293204552166059 (-0.0071595066135198004) | > avg_loss_disc_real_1: 0.21997620354236014 (-0.027907052615159078) | > avg_loss_disc_real_2: 0.22818667850863766 (-0.03602936737973916) | > avg_loss_disc_real_3: 0.23475633078897504 (+0.004668592357299717) | > avg_loss_disc_real_4: 0.24689662099724086 (-0.005840505092916337) | > avg_loss_disc_real_5: 0.2493174556695239 (+0.002751969535585791) | > avg_loss_0: 2.6055379048199723 (-0.11740296659335181) | > avg_grad_norm_0: tensor(214.9221, device='cuda:0') (tensor(-22.1076, device='cuda:0')) | > avg_loss_gen: 2.119184091057575 (+0.01263794093064785) | > avg_loss_kl: 2.1720611008120247 (-0.07466073607055224) | > avg_loss_feat: 3.542262789229272 (+0.08452467851235834) | > avg_loss_mel: 17.439229427928655 (-0.5778190854569516) | > avg_loss_duration: 1.5812620126025778 (-0.025545387200905934) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.853999473679234 (-0.5808625288412586) | > avg_grad_norm_1: tensor(9196.4941, device='cuda:0') (+tensor(594.5391, device='cuda:0'))  > EPOCH: 760/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:02:10)   --> TIME: 2025-02-25 16:02:25 -- STEP: 29/72 -- GLOBAL_STEP: 1054750 | > loss_disc: 2.7282683849334717 (2.5382286680155786) | > loss_disc_real_0: 0.1068248599767685 (0.10672786844701602) | > loss_disc_real_1: 0.23367741703987122 (0.21965610107471203) | > loss_disc_real_2: 0.13472957909107208 (0.2057873704310121) | > loss_disc_real_3: 0.20287400484085083 (0.22927076241065716) | > loss_disc_real_4: 0.24551868438720703 (0.24301683131990762) | > loss_disc_real_5: 0.24180838465690613 (0.2420198295650811) | > loss_0: 2.7282683849334717 (2.5382286680155786) | > grad_norm_0: tensor(299.1940, device='cuda:0') (tensor(205.8711, device='cuda:0')) | > loss_gen: 1.6502004861831665 (2.1672975934784984) | > loss_kl: 1.9314842224121094 (1.9851360115511665) | > loss_feat: 3.1919262409210205 (3.812616857989081) | > loss_mel: 17.094032287597656 (17.141998554098194) | > loss_duration: 1.5360822677612305 (1.5236116154440518) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.403724670410156 (26.630660616118334) | > grad_norm_1: tensor(5669.7114, device='cuda:0') (tensor(9780.9102, device='cuda:0')) | > current_lr_0: 0.00018187350692648908 | > current_lr_1: 0.00018187350692648908 | > step_time: 0.4466 (0.44077920091563255) | > loader_time: 0.0081 (0.007158904240049165) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868414825117084 (-2.2360976313204675e-05) | > avg_step_time: 0.45494259914881746 (-0.002317052491953686) | > avg_loss_disc: 2.55950303144858 (-0.046034873371392226) | > avg_loss_disc_real_0: 0.11797995860098114 (-0.011340496615624765) | > avg_loss_disc_real_1: 0.22174130904842432 (+0.0017651055060641796) | > avg_loss_disc_real_2: 0.213353446042034 (-0.014833232466603674) | > avg_loss_disc_real_3: 0.22943126483702322 (-0.005325065951951813) | > avg_loss_disc_real_4: 0.2395439586589034 (-0.007352662338337451) | > avg_loss_disc_real_5: 0.24416767291619745 (-0.005149782753326448) | > avg_loss_0: 2.55950303144858 (-0.046034873371392226) | > avg_grad_norm_0: tensor(201.9978, device='cuda:0') (tensor(-12.9243, device='cuda:0')) | > avg_loss_gen: 2.1596306276993023 (+0.04044653664172726) | > avg_loss_kl: 2.0191231898858515 (-0.15293791092617326) | > avg_loss_feat: 3.683678563212005 (+0.14141577398273286) | > avg_loss_mel: 16.965057373046875 (-0.47417205488178027) | > avg_loss_duration: 1.5633723131367858 (-0.017889699465791997) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.390861941055512 (-0.4631375326237226) | > avg_grad_norm_1: tensor(9946.1338, device='cuda:0') (+tensor(749.6396, device='cuda:0'))  > EPOCH: 761/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:02:47)   --> TIME: 2025-02-25 16:02:52 -- STEP: 7/72 -- GLOBAL_STEP: 1054800 | > loss_disc: 2.743853807449341 (2.5647048950195312) | > loss_disc_real_0: 0.10973319411277771 (0.10724110049860817) | > loss_disc_real_1: 0.2475939393043518 (0.2122498814548765) | > loss_disc_real_2: 0.24017608165740967 (0.210893452167511) | > loss_disc_real_3: 0.2173018753528595 (0.2278257587126323) | > loss_disc_real_4: 0.2136293351650238 (0.23577448512826646) | > loss_disc_real_5: 0.23654431104660034 (0.23872689264161245) | > loss_0: 2.743853807449341 (2.5647048950195312) | > grad_norm_0: tensor(77.6083, device='cuda:0') (tensor(270.6687, device='cuda:0')) | > loss_gen: 1.888916254043579 (2.133893983704703) | > loss_kl: 1.8910049200057983 (1.902959670339312) | > loss_feat: 3.229250192642212 (3.8515263625553677) | > loss_mel: 17.058542251586914 (17.063954217093332) | > loss_duration: 1.497200608253479 (1.5280482428414481) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.564912796020508 (26.480382101876394) | > grad_norm_1: tensor(8899.2080, device='cuda:0') (tensor(11654.0527, device='cuda:0')) | > current_lr_0: 0.00018185077273812325 | > current_lr_1: 0.00018185077273812325 | > step_time: 0.4451 (0.43466694014413015) | > loader_time: 0.0062 (0.0067675794873918805)  --> TIME: 2025-02-25 16:03:16 -- STEP: 57/72 -- GLOBAL_STEP: 1054850 | > loss_disc: 2.508700370788574 (2.5867552589951894) | > loss_disc_real_0: 0.1486029475927353 (0.14169449123897052) | > loss_disc_real_1: 0.2516036331653595 (0.22464143053481453) | > loss_disc_real_2: 0.21674752235412598 (0.2167161786765383) | > loss_disc_real_3: 0.23979227244853973 (0.2293661472044493) | > loss_disc_real_4: 0.22861748933792114 (0.2379925467988901) | > loss_disc_real_5: 0.2600651681423187 (0.2453596900429642) | > loss_0: 2.508700370788574 (2.5867552589951894) | > grad_norm_0: tensor(213.0802, device='cuda:0') (tensor(214.1501, device='cuda:0')) | > loss_gen: 2.493929862976074 (2.1829457094794824) | > loss_kl: 2.0942914485931396 (1.983893072395994) | > loss_feat: 4.007401943206787 (3.696531613667806) | > loss_mel: 16.47591781616211 (16.760742187499993) | > loss_duration: 1.606359601020813 (1.5477938108276903) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.677900314331055 (26.17190648798357) | > grad_norm_1: tensor(6246.6489, device='cuda:0') (tensor(9943.6592, device='cuda:0')) | > current_lr_0: 0.00018185077273812325 | > current_lr_1: 0.00018185077273812325 | > step_time: 0.4564 (0.4568141343300803) | > loader_time: 0.0095 (0.008264202820627316) --> EVAL PERFORMANCE | > avg_loader_time: 0.008866975005243869 (+0.0001828267540730285) | > avg_step_time: 0.4660680529097436 (+0.011125453760926163) | > avg_loss_disc: 2.593560692290185 (+0.034057660841604775) | > avg_loss_disc_real_0: 0.14857297380205609 (+0.03059301520107495) | > avg_loss_disc_real_1: 0.2246829220946406 (+0.0029416130462162904) | > avg_loss_disc_real_2: 0.21950831757464878 (+0.006154871532614792) | > avg_loss_disc_real_3: 0.22955623856732543 (+0.00012497373030220804) | > avg_loss_disc_real_4: 0.2378600491604335 (-0.0016839094984699154) | > avg_loss_disc_real_5: 0.24587656242746703 (+0.0017088895112695734) | > avg_loss_0: 2.593560692290185 (+0.034057660841604775) | > avg_grad_norm_0: tensor(222.5998, device='cuda:0') (+tensor(20.6020, device='cuda:0')) | > avg_loss_gen: 2.1760587893741232 (+0.016428161674820974) | > avg_loss_kl: 2.0136949545900587 (-0.005428235295792838) | > avg_loss_feat: 3.6392601207948068 (-0.04441844241719828) | > avg_loss_mel: 16.783261446885653 (-0.18179592616122164) | > avg_loss_duration: 1.5594921313541037 (-0.003880181782682124) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.171767597467127 (-0.21909434358838453) | > avg_grad_norm_1: tensor(9923.4980, device='cuda:0') (tensor(-22.6357, device='cuda:0'))  > EPOCH: 762/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:03:25)   --> TIME: 2025-02-25 16:03:42 -- STEP: 35/72 -- GLOBAL_STEP: 1054900 | > loss_disc: 2.7059319019317627 (2.761943885258266) | > loss_disc_real_0: 0.1367364525794983 (0.13877982103398864) | > loss_disc_real_1: 0.2180355340242386 (0.21701290543590274) | > loss_disc_real_2: 0.23515963554382324 (0.245276358297893) | > loss_disc_real_3: 0.2516809403896332 (0.2514884007828576) | > loss_disc_real_4: 0.251058965921402 (0.25567012897559577) | > loss_disc_real_5: 0.2610522508621216 (0.2518201755625861) | > loss_0: 2.7059319019317627 (2.761943885258266) | > grad_norm_0: tensor(313.0295, device='cuda:0') (tensor(232.0234, device='cuda:0')) | > loss_gen: 1.753769040107727 (2.0186958278928486) | > loss_kl: 2.1193325519561768 (2.011421990394593) | > loss_feat: 2.730167865753174 (3.406534889766148) | > loss_mel: 16.707080841064453 (16.841203471592493) | > loss_duration: 1.5751535892486572 (1.5241888863699777) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.8855037689209 (25.802045168195452) | > grad_norm_1: tensor(8224.4600, device='cuda:0') (tensor(7835.3535, device='cuda:0')) | > current_lr_0: 0.00018182804139153097 | > current_lr_1: 0.00018182804139153097 | > step_time: 0.4372 (0.4386396612439837) | > loader_time: 0.0085 (0.007206501279558454) --> EVAL PERFORMANCE | > avg_loader_time: 0.008663311810560629 (-0.00020366319468323978) | > avg_step_time: 0.4589340888278585 (-0.0071339640818851024) | > avg_loss_disc: 2.6875653569127467 (+0.09400466462256185) | > avg_loss_disc_real_0: 0.13853716351826428 (-0.010035810283791802) | > avg_loss_disc_real_1: 0.22173673722525716 (-0.0029461848693834447) | > avg_loss_disc_real_2: 0.2385109650417113 (+0.01900264746706251) | > avg_loss_disc_real_3: 0.24694695187286592 (+0.017390713305540484) | > avg_loss_disc_real_4: 0.24935339013455618 (+0.01149334097412269) | > avg_loss_disc_real_5: 0.24832929901673761 (+0.0024527365892705877) | > avg_loss_0: 2.6875653569127467 (+0.09400466462256185) | > avg_grad_norm_0: tensor(223.1991, device='cuda:0') (+tensor(0.5993, device='cuda:0')) | > avg_loss_gen: 2.063863205238126 (-0.11219558413599717) | > avg_loss_kl: 2.0459430352063253 (+0.032248080616266606) | > avg_loss_feat: 3.4075770243792465 (-0.23168309641556029) | > avg_loss_mel: 16.86653487783083 (+0.08327343094517659) | > avg_loss_duration: 1.5592193620305665 (-0.00027276932353714756) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.943137639005418 (-0.22862995846170975) | > avg_grad_norm_1: tensor(8156.4663, device='cuda:0') (tensor(-1767.0317, device='cuda:0'))  > EPOCH: 763/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:04:02)   --> TIME: 2025-02-25 16:04:10 -- STEP: 13/72 -- GLOBAL_STEP: 1054950 | > loss_disc: 2.82342267036438 (2.5990108526670017) | > loss_disc_real_0: 0.0852484330534935 (0.12396295655232209) | > loss_disc_real_1: 0.2067490518093109 (0.22867175592825964) | > loss_disc_real_2: 0.20782512426376343 (0.2264593862570249) | > loss_disc_real_3: 0.22409825026988983 (0.23666698428300711) | > loss_disc_real_4: 0.2421569973230362 (0.24454043920223528) | > loss_disc_real_5: 0.227050319314003 (0.24178102268622473) | > loss_0: 2.82342267036438 (2.5990108526670017) | > grad_norm_0: tensor(45.8287, device='cuda:0') (tensor(210.4298, device='cuda:0')) | > loss_gen: 1.8502219915390015 (2.156326559873728) | > loss_kl: 1.9971375465393066 (1.968633459164546) | > loss_feat: 3.601966381072998 (3.68571661068843) | > loss_mel: 16.85853385925293 (16.829229501577522) | > loss_duration: 1.452880859375 (1.5130382776260376) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.760740280151367 (26.152944858257587) | > grad_norm_1: tensor(10178.5742, device='cuda:0') (tensor(9961.7637, device='cuda:0')) | > current_lr_0: 0.000181805312886357 | > current_lr_1: 0.000181805312886357 | > step_time: 0.4687 (0.4403540721306434) | > loader_time: 0.0074 (0.0070986197544978214)  --> TIME: 2025-02-25 16:04:33 -- STEP: 63/72 -- GLOBAL_STEP: 1055000 | > loss_disc: 3.035348415374756 (2.735078183431474) | > loss_disc_real_0: 0.14213189482688904 (0.15710085819637967) | > loss_disc_real_1: 0.2524481415748596 (0.2526191446989302) | > loss_disc_real_2: 0.20485833287239075 (0.2386889826683771) | > loss_disc_real_3: 0.23183229565620422 (0.23587403008862148) | > loss_disc_real_4: 0.2124120593070984 (0.23923351509230478) | > loss_disc_real_5: 0.22846828401088715 (0.24255477010257662) | > loss_0: 3.035348415374756 (2.735078183431474) | > grad_norm_0: tensor(338.2645, device='cuda:0') (tensor(218.1594, device='cuda:0')) | > loss_gen: 1.6071577072143555 (2.0118161212830317) | > loss_kl: 2.163313150405884 (2.0575250387191755) | > loss_feat: 2.779547691345215 (3.2988829574887713) | > loss_mel: 16.676815032958984 (16.80356700836666) | > loss_duration: 1.594762921333313 (1.5522256786861117) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.821596145629883 (25.724016704256574) | > grad_norm_1: tensor(2097.2666, device='cuda:0') (tensor(8254.5977, device='cuda:0')) | > current_lr_0: 0.000181805312886357 | > current_lr_1: 0.000181805312886357 | > step_time: 0.5217 (0.45353792584131636) | > loader_time: 0.012 (0.008564313252766928) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1055000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008909524326593104 (+0.00024621251603247456) | > avg_step_time: 0.46170863970904286 (+0.0027745508811843345) | > avg_loss_disc: 2.755941431287309 (+0.06837607437456228) | > avg_loss_disc_real_0: 0.16294402030991836 (+0.02440685679165408) | > avg_loss_disc_real_1: 0.2555630882860909 (+0.03382635106083376) | > avg_loss_disc_real_2: 0.2401027301667442 (+0.0015917651250328957) | > avg_loss_disc_real_3: 0.23625314340625012 (-0.0106938084666158) | > avg_loss_disc_real_4: 0.23860901153423417 (-0.010744378600322013) | > avg_loss_disc_real_5: 0.24166705553800288 (-0.006662243478734731) | > avg_loss_0: 2.755941431287309 (+0.06837607437456228) | > avg_grad_norm_0: tensor(224.4660, device='cuda:0') (+tensor(1.2669, device='cuda:0')) | > avg_loss_gen: 1.994372503858217 (-0.06949070137990909) | > avg_loss_kl: 2.0812508270774073 (+0.03530779187108202) | > avg_loss_feat: 3.2331787297423458 (-0.1743982946369007) | > avg_loss_mel: 16.85900633100053 (-0.007528546830300797) | > avg_loss_duration: 1.562547913739379 (+0.0033285517088124106) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.730356162702535 (-0.21278147630288302) | > avg_grad_norm_1: tensor(7944.2031, device='cuda:0') (tensor(-212.2632, device='cuda:0'))  > EPOCH: 764/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:04:43)   --> TIME: 2025-02-25 16:05:03 -- STEP: 41/72 -- GLOBAL_STEP: 1055050 | > loss_disc: 3.0653939247131348 (2.6938524536970183) | > loss_disc_real_0: 0.07778584957122803 (0.12343662077697308) | > loss_disc_real_1: 0.3937986195087433 (0.2462955611508067) | > loss_disc_real_2: 0.17487500607967377 (0.24506468998222816) | > loss_disc_real_3: 0.2277306765317917 (0.2490311868307068) | > loss_disc_real_4: 0.2048840969800949 (0.25860186938832447) | > loss_disc_real_5: 0.20057205855846405 (0.25364674155305067) | > loss_0: 3.0653939247131348 (2.6938524536970183) | > grad_norm_0: tensor(42.9394, device='cuda:0') (tensor(174.2151, device='cuda:0')) | > loss_gen: 1.609193205833435 (2.1742705368414157) | > loss_kl: 2.3691232204437256 (2.405062437057495) | > loss_feat: 2.9599862098693848 (3.6015162526107414) | > loss_mel: 19.162900924682617 (18.605537786716372) | > loss_duration: 1.5998175144195557 (1.6077902287971684) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.701021194458008 (28.394177367047565) | > grad_norm_1: tensor(126.1104, device='cuda:0') (tensor(5232.3091, device='cuda:0')) | > current_lr_0: 0.0001817825872222462 | > current_lr_1: 0.0001817825872222462 | > step_time: 0.451 (0.4425116050534132) | > loader_time: 0.0093 (0.007430320832787491) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628455685897611 (-0.0002810686406954921) | > avg_step_time: 0.45830001629574196 (-0.003408623413300893) | > avg_loss_disc: 2.6745358755890756 (-0.08140555569823338) | > avg_loss_disc_real_0: 0.12379548347122223 (-0.03914853683869614) | > avg_loss_disc_real_1: 0.24221619103156344 (-0.013346897254527479) | > avg_loss_disc_real_2: 0.24443111935971487 (+0.004328389192970683) | > avg_loss_disc_real_3: 0.24556364391891056 (+0.009310500512660441) | > avg_loss_disc_real_4: 0.256641471469906 (+0.018032459935671846) | > avg_loss_disc_real_5: 0.2515025512433388 (+0.009835495705335928) | > avg_loss_0: 2.6745358755890756 (-0.08140555569823338) | > avg_grad_norm_0: tensor(175.4019, device='cuda:0') (tensor(-49.0641, device='cuda:0')) | > avg_loss_gen: 2.155473791377645 (+0.16110128751942798) | > avg_loss_kl: 2.6868175687924234 (+0.6055667417150161) | > avg_loss_feat: 3.503643583244002 (+0.27046485350165606) | > avg_loss_mel: 20.414057315235414 (+3.555050984234885) | > avg_loss_duration: 1.6384734217549715 (+0.0759255080155925) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 30.398465814724773 (+4.668109652022238) | > avg_grad_norm_1: tensor(7089.7563, device='cuda:0') (tensor(-854.4468, device='cuda:0'))  > EPOCH: 765/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:05:20)   --> TIME: 2025-02-25 16:05:31 -- STEP: 19/72 -- GLOBAL_STEP: 1055100 | > loss_disc: 2.4367401599884033 (2.545658563312731) | > loss_disc_real_0: 0.1477833390235901 (0.11248339673406199) | > loss_disc_real_1: 0.22327543795108795 (0.2330870393075441) | > loss_disc_real_2: 0.19176581501960754 (0.21666165794196882) | > loss_disc_real_3: 0.20672646164894104 (0.21898772136161201) | > loss_disc_real_4: 0.24317124485969543 (0.23597920646792964) | > loss_disc_real_5: 0.2394905984401703 (0.23544296779130636) | > loss_0: 2.4367401599884033 (2.545658563312731) | > grad_norm_0: tensor(211.2421, device='cuda:0') (tensor(214.1100, device='cuda:0')) | > loss_gen: 2.3720874786376953 (2.118818188968458) | > loss_kl: 2.019294500350952 (2.247752227281269) | > loss_feat: 4.026890277862549 (3.6585349785654167) | > loss_mel: 18.021289825439453 (19.165739862542402) | > loss_duration: 1.5490880012512207 (1.5588093682339317) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.988649368286133 (28.74965476989746) | > grad_norm_1: tensor(7901.9170, device='cuda:0') (tensor(9471.4873, device='cuda:0')) | > current_lr_0: 0.00018175986439884342 | > current_lr_1: 0.00018175986439884342 | > step_time: 0.4314 (0.4408625050594932) | > loader_time: 0.0065 (0.007408468346846731)  --> TIME: 2025-02-25 16:05:54 -- STEP: 69/72 -- GLOBAL_STEP: 1055150 | > loss_disc: 2.635085344314575 (2.5489733115486475) | > loss_disc_real_0: 0.10401041060686111 (0.11006369128607321) | > loss_disc_real_1: 0.23231129348278046 (0.22404621044794717) | > loss_disc_real_2: 0.2830628454685211 (0.2288310361513193) | > loss_disc_real_3: 0.2615456283092499 (0.2274936696757441) | > loss_disc_real_4: 0.2727186977863312 (0.24039634349553482) | > loss_disc_real_5: 0.26663073897361755 (0.24104419134665225) | > loss_0: 2.635085344314575 (2.5489733115486475) | > grad_norm_0: tensor(202.0327, device='cuda:0') (tensor(230.0507, device='cuda:0')) | > loss_gen: 2.0286014080047607 (2.1348465715629468) | > loss_kl: 2.231938362121582 (2.1471886358399312) | > loss_feat: 3.1987791061401367 (3.6137238792751147) | > loss_mel: 18.075706481933594 (18.03895919910375) | > loss_duration: 1.6703383922576904 (1.5857018802476965) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.20536231994629 (27.52042007446289) | > grad_norm_1: tensor(11178.5449, device='cuda:0') (tensor(9831.0234, device='cuda:0')) | > current_lr_0: 0.00018175986439884342 | > current_lr_1: 0.00018175986439884342 | > step_time: 0.5318 (0.4572937454002491) | > loader_time: 0.0125 (0.008911447248597073) --> EVAL PERFORMANCE | > avg_loader_time: 0.008955468594188418 (+0.00032701290829080694) | > avg_step_time: 0.45862574644491705 (+0.00032573014917508525) | > avg_loss_disc: 2.550923357547169 (-0.12361251804190676) | > avg_loss_disc_real_0: 0.10926069172335343 (-0.0145347917478688) | > avg_loss_disc_real_1: 0.22380591991921545 (-0.018410271112347998) | > avg_loss_disc_real_2: 0.22924810228213457 (-0.015183017077580296) | > avg_loss_disc_real_3: 0.2275751291866034 (-0.01798851473230717) | > avg_loss_disc_real_4: 0.24053961794141313 (-0.016101853528492888) | > avg_loss_disc_real_5: 0.2411953588606606 (-0.010307192382678215) | > avg_loss_0: 2.550923357547169 (-0.12361251804190676) | > avg_grad_norm_0: tensor(227.6062, device='cuda:0') (+tensor(52.2043, device='cuda:0')) | > avg_loss_gen: 2.132812798862726 (-0.02266099251491882) | > avg_loss_kl: 2.1461234881844313 (-0.5406940806079921) | > avg_loss_feat: 3.6077137463529345 (+0.10407016310893269) | > avg_loss_mel: 18.02435232887805 (-2.3897049863573656) | > avg_loss_duration: 1.5857944387785146 (-0.0526789829764569) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.496796675131353 (-2.9016691395934195) | > avg_grad_norm_1: tensor(9808.3135, device='cuda:0') (+tensor(2718.5571, device='cuda:0'))  > EPOCH: 766/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:05:57)   --> TIME: 2025-02-25 16:06:20 -- STEP: 47/72 -- GLOBAL_STEP: 1055200 | > loss_disc: 2.6232473850250244 (2.5408887254430894) | > loss_disc_real_0: 0.07207433879375458 (0.10668204328481187) | > loss_disc_real_1: 0.21750520169734955 (0.21698666760262023) | > loss_disc_real_2: 0.1906481385231018 (0.22470775912416743) | > loss_disc_real_3: 0.19719263911247253 (0.2276415561741971) | > loss_disc_real_4: 0.23180249333381653 (0.24175439331125706) | > loss_disc_real_5: 0.25155308842658997 (0.24621357530989546) | > loss_0: 2.6232473850250244 (2.5408887254430894) | > grad_norm_0: tensor(196.6414, device='cuda:0') (tensor(249.5510, device='cuda:0')) | > loss_gen: 2.0538322925567627 (2.1844132124109468) | > loss_kl: 1.9268684387207031 (1.9678133599301602) | > loss_feat: 3.7974398136138916 (3.778433155506215) | > loss_mel: 17.9348087310791 (17.316713251966103) | > loss_duration: 1.5984641313552856 (1.550065299297901) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.31141471862793 (26.797438398320626) | > grad_norm_1: tensor(12409.0537, device='cuda:0') (tensor(10688.6387, device='cuda:0')) | > current_lr_0: 0.00018173714441579354 | > current_lr_1: 0.00018173714441579354 | > step_time: 0.4541 (0.4399454593658447) | > loader_time: 0.0087 (0.00760286919614102) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863584666184976 (-0.0003196219323386585) | > avg_step_time: 0.4538508972651522 (-0.00477484917976484) | > avg_loss_disc: 2.556499350238854 (+0.005575992691685183) | > avg_loss_disc_real_0: 0.1064524002583094 (-0.0028082914650440216) | > avg_loss_disc_real_1: 0.22016049858549946 (-0.0036454213337159813) | > avg_loss_disc_real_2: 0.22716029156261766 (-0.0020878107195169138) | > avg_loss_disc_real_3: 0.22936049859288712 (+0.0017853694062837377) | > avg_loss_disc_real_4: 0.24177799237446046 (+0.0012383744330473345) | > avg_loss_disc_real_5: 0.24681438883425486 (+0.005619029973594258) | > avg_loss_0: 2.556499350238854 (+0.005575992691685183) | > avg_grad_norm_0: tensor(250.1662, device='cuda:0') (+tensor(22.5600, device='cuda:0')) | > avg_loss_gen: 2.163394785263168 (+0.030581986400441963) | > avg_loss_kl: 1.993346847278971 (-0.15277664090546028) | > avg_loss_feat: 3.716865022417525 (+0.10915127606459052) | > avg_loss_mel: 17.20241012035961 (-0.8219422085184398) | > avg_loss_duration: 1.5704761743545532 (-0.015318264423961336) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.646492998364945 (-0.8503036767664085) | > avg_grad_norm_1: tensor(10492.3369, device='cuda:0') (+tensor(684.0234, device='cuda:0'))  > EPOCH: 767/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:06:34)   --> TIME: 2025-02-25 16:06:48 -- STEP: 25/72 -- GLOBAL_STEP: 1055250 | > loss_disc: 2.286336660385132 (2.5212926387786867) | > loss_disc_real_0: 0.11000117659568787 (0.10020553976297379) | > loss_disc_real_1: 0.21253356337547302 (0.21888809263706208) | > loss_disc_real_2: 0.21959644556045532 (0.21709155797958374) | > loss_disc_real_3: 0.22742623090744019 (0.2269303971529007) | > loss_disc_real_4: 0.22033269703388214 (0.244811971783638) | > loss_disc_real_5: 0.23203065991401672 (0.24535031139850616) | > loss_0: 2.286336660385132 (2.5212926387786867) | > grad_norm_0: tensor(241.7380, device='cuda:0') (tensor(249.1904, device='cuda:0')) | > loss_gen: 2.8092405796051025 (2.225000143051147) | > loss_kl: 2.0636403560638428 (1.9594638681411742) | > loss_feat: 5.0002899169921875 (3.913741321563721) | > loss_mel: 17.32073402404785 (17.108736228942874) | > loss_duration: 1.5367056131362915 (1.5233996868133546) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.730609893798828 (26.730341262817383) | > grad_norm_1: tensor(12256.4297, device='cuda:0') (tensor(11336.0693, device='cuda:0')) | > current_lr_0: 0.00018171442727274155 | > current_lr_1: 0.00018171442727274155 | > step_time: 0.4338 (0.4559612655639648) | > loader_time: 0.0074 (0.007076930999755859) --> EVAL PERFORMANCE | > avg_loader_time: 0.008848448874245227 (+0.00021260221239546677) | > avg_step_time: 0.46103453636169434 (+0.007183639096542127) | > avg_loss_disc: 2.543369343583013 (-0.013130006655841076) | > avg_loss_disc_real_0: 0.09826912956548409 (-0.008183270692825317) | > avg_loss_disc_real_1: 0.22174011674565328 (+0.0015796181601538117) | > avg_loss_disc_real_2: 0.2245996401343547 (-0.00256065142826295) | > avg_loss_disc_real_3: 0.23012985100208874 (+0.000769352409201618) | > avg_loss_disc_real_4: 0.24493749712554502 (+0.003159504751084563) | > avg_loss_disc_real_5: 0.24626049357400814 (-0.0005538952602467118) | > avg_loss_0: 2.543369343583013 (-0.013130006655841076) | > avg_grad_norm_0: tensor(251.8360, device='cuda:0') (+tensor(1.6698, device='cuda:0')) | > avg_loss_gen: 2.2072995817157586 (+0.04390479645259049) | > avg_loss_kl: 1.9988118242210067 (+0.0054649769420356265) | > avg_loss_feat: 3.871198738125009 (+0.15433371570748378) | > avg_loss_mel: 16.98957715907567 (-0.21283296128394014) | > avg_loss_duration: 1.5649163924472433 (-0.00555978190730988) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.631803593165436 (-0.014689405199508343) | > avg_grad_norm_1: tensor(11550.2969, device='cuda:0') (+tensor(1057.9600, device='cuda:0'))  > EPOCH: 768/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:07:11)   --> TIME: 2025-02-25 16:07:15 -- STEP: 3/72 -- GLOBAL_STEP: 1055300 | > loss_disc: 2.669712781906128 (2.4726805686950684) | > loss_disc_real_0: 0.104849673807621 (0.09131707499424617) | > loss_disc_real_1: 0.2142844945192337 (0.20429772635300955) | > loss_disc_real_2: 0.24030375480651855 (0.21786150336265564) | > loss_disc_real_3: 0.23412545025348663 (0.21437696119149527) | > loss_disc_real_4: 0.2441958636045456 (0.2257855385541916) | > loss_disc_real_5: 0.24228428304195404 (0.2377299169699351) | > loss_0: 2.669712781906128 (2.4726805686950684) | > grad_norm_0: tensor(274.1397, device='cuda:0') (tensor(306.9573, device='cuda:0')) | > loss_gen: 1.9044734239578247 (2.318558096885681) | > loss_kl: 2.077390193939209 (1.9979602098464966) | > loss_feat: 3.4964935779571533 (4.307935158411662) | > loss_mel: 17.211963653564453 (17.34419059753418) | > loss_duration: 1.536933183670044 (1.5285886923472087) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.22725486755371 (27.497233072916668) | > grad_norm_1: tensor(11697.0576, device='cuda:0') (tensor(13748.5508, device='cuda:0')) | > current_lr_0: 0.00018169171296933244 | > current_lr_1: 0.00018169171296933244 | > step_time: 0.4257 (0.4364606539408366) | > loader_time: 0.007 (0.006520509719848633)  --> TIME: 2025-02-25 16:07:38 -- STEP: 53/72 -- GLOBAL_STEP: 1055350 | > loss_disc: 2.789806365966797 (2.5554455361276305) | > loss_disc_real_0: 0.08607245981693268 (0.10301622804605735) | > loss_disc_real_1: 0.23072291910648346 (0.2209525189872058) | > loss_disc_real_2: 0.2156798541545868 (0.22382839361451706) | > loss_disc_real_3: 0.23306111991405487 (0.22925320633177487) | > loss_disc_real_4: 0.22177451848983765 (0.23967881388259385) | > loss_disc_real_5: 0.23667289316654205 (0.24330305940700028) | > loss_0: 2.789806365966797 (2.5554455361276305) | > grad_norm_0: tensor(380.9688, device='cuda:0') (tensor(253.4085, device='cuda:0')) | > loss_gen: 1.8958951234817505 (2.213144567777526) | > loss_kl: 2.036339044570923 (1.9786721198063977) | > loss_feat: 3.255009412765503 (3.9398901372585655) | > loss_mel: 16.75716209411621 (16.836821196214203) | > loss_duration: 1.600598692893982 (1.5429486148762253) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.545005798339844 (26.511476480735922) | > grad_norm_1: tensor(11520.4844, device='cuda:0') (tensor(12080.1738, device='cuda:0')) | > current_lr_0: 0.00018169171296933244 | > current_lr_1: 0.00018169171296933244 | > step_time: 0.459 (0.447260874622273) | > loader_time: 0.0092 (0.007819643560445533) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591520954185809 (-0.0002569279200594181) | > avg_step_time: 0.45914180178037833 (-0.0018927345813160024) | > avg_loss_disc: 2.557260771872292 (+0.01389142828927925) | > avg_loss_disc_real_0: 0.10309673530954712 (+0.004827605744063032) | > avg_loss_disc_real_1: 0.22137562447870282 (-0.0003644922669504558) | > avg_loss_disc_real_2: 0.22584944019015407 (+0.0012498000557993605) | > avg_loss_disc_real_3: 0.23059611757036666 (+0.0004662665682779177) | > avg_loss_disc_real_4: 0.2401840753958259 (-0.004753421729719126) | > avg_loss_disc_real_5: 0.24304411276965074 (-0.003216380804357405) | > avg_loss_0: 2.557260771872292 (+0.01389142828927925) | > avg_grad_norm_0: tensor(254.4598, device='cuda:0') (+tensor(2.6238, device='cuda:0')) | > avg_loss_gen: 2.20202046716717 (-0.005279114548588648) | > avg_loss_kl: 2.010320984141927 (+0.011509159920920498) | > avg_loss_feat: 3.87309751040499 (+0.0018987722799810314) | > avg_loss_mel: 16.827989537950973 (-0.1615876211246956) | > avg_loss_duration: 1.5585892217260011 (-0.006327170721242226) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.472017529984594 (-0.15978606318084232) | > avg_grad_norm_1: tensor(11923.5947, device='cuda:0') (+tensor(373.2979, device='cuda:0'))  > EPOCH: 769/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:07:49)   --> TIME: 2025-02-25 16:08:05 -- STEP: 31/72 -- GLOBAL_STEP: 1055400 | > loss_disc: 2.3843681812286377 (2.525651039615754) | > loss_disc_real_0: 0.10340359807014465 (0.10109745602934592) | > loss_disc_real_1: 0.19715633988380432 (0.22255677273196559) | > loss_disc_real_2: 0.20435731112957 (0.22529456596220693) | > loss_disc_real_3: 0.2299719750881195 (0.23378088012818368) | > loss_disc_real_4: 0.271534264087677 (0.24584272792262415) | > loss_disc_real_5: 0.2573339641094208 (0.24634906841862586) | > loss_0: 2.3843681812286377 (2.525651039615754) | > grad_norm_0: tensor(142.6411, device='cuda:0') (tensor(275.9663, device='cuda:0')) | > loss_gen: 2.567842960357666 (2.2727567418929078) | > loss_kl: 1.9070450067520142 (1.9934427315188992) | > loss_feat: 4.612672328948975 (4.026543278847971) | > loss_mel: 17.625228881835938 (16.922449142702163) | > loss_duration: 1.5568504333496094 (1.517902293512898) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.26963996887207 (26.733094123102003) | > grad_norm_1: tensor(11100.6348, device='cuda:0') (tensor(12409.7012, device='cuda:0')) | > current_lr_0: 0.00018166900150521126 | > current_lr_1: 0.00018166900150521126 | > step_time: 0.4385 (0.435902641665551) | > loader_time: 0.0087 (0.007080854908112557) --> EVAL PERFORMANCE | > avg_loader_time: 0.008581527521912484 (-9.993432273324582e-06) | > avg_step_time: 0.4547468440633425 (-0.004394957717035852) | > avg_loss_disc: 2.5585473557593117 (+0.0012865838870195212) | > avg_loss_disc_real_0: 0.10114437105580115 (-0.0019523642537459668) | > avg_loss_disc_real_1: 0.22521515526402164 (+0.0038395307853188176) | > avg_loss_disc_real_2: 0.2238698316292024 (-0.001979608560951668) | > avg_loss_disc_real_3: 0.2341828228722156 (+0.0035867053018489303) | > avg_loss_disc_real_4: 0.24168574432252157 (+0.0015016689266956762) | > avg_loss_disc_real_5: 0.2458127153591371 (+0.0027686025894863575) | > avg_loss_0: 2.5585473557593117 (+0.0012865838870195212) | > avg_grad_norm_0: tensor(282.9702, device='cuda:0') (+tensor(28.5104, device='cuda:0')) | > avg_loss_gen: 2.2167418842584317 (+0.014721417091261735) | > avg_loss_kl: 2.0353797926029684 (+0.02505880846104125) | > avg_loss_feat: 3.859481485796646 (-0.013616024608343658) | > avg_loss_mel: 16.82126230589102 (-0.006727232059954957) | > avg_loss_duration: 1.5577631248554713 (-0.0008260968705298133) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.49062852456536 (+0.018610994580765094) | > avg_grad_norm_1: tensor(12184.3428, device='cuda:0') (+tensor(260.7480, device='cuda:0'))  > EPOCH: 770/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:08:26)   --> TIME: 2025-02-25 16:08:32 -- STEP: 9/72 -- GLOBAL_STEP: 1055450 | > loss_disc: 2.8460330963134766 (2.521275440851847) | > loss_disc_real_0: 0.061896733939647675 (0.09402111669381459) | > loss_disc_real_1: 0.2580570876598358 (0.22238557040691376) | > loss_disc_real_2: 0.3033847212791443 (0.23586931162410313) | > loss_disc_real_3: 0.2802712917327881 (0.23651244077417585) | > loss_disc_real_4: 0.28580164909362793 (0.2535526901483536) | > loss_disc_real_5: 0.26286250352859497 (0.24214513268735674) | > loss_0: 2.8460330963134766 (2.521275440851847) | > grad_norm_0: tensor(394.4367, device='cuda:0') (tensor(255.5694, device='cuda:0')) | > loss_gen: 1.8319463729858398 (2.2936307986577353) | > loss_kl: 2.294769763946533 (2.0141774813334146) | > loss_feat: 3.327754497528076 (4.028611951404148) | > loss_mel: 17.18466567993164 (16.756810188293457) | > loss_duration: 1.5243631601333618 (1.5190820164150662) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.16349983215332 (26.612312104966904) | > grad_norm_1: tensor(10527.7324, device='cuda:0') (tensor(11328.4824, device='cuda:0')) | > current_lr_0: 0.0001816462928800231 | > current_lr_1: 0.0001816462928800231 | > step_time: 0.4374 (0.4315599335564507) | > loader_time: 0.0071 (0.0064105987548828125)  --> TIME: 2025-02-25 16:08:55 -- STEP: 59/72 -- GLOBAL_STEP: 1055500 | > loss_disc: 2.8593204021453857 (2.571974006749815) | > loss_disc_real_0: 0.08023490011692047 (0.09640161244798515) | > loss_disc_real_1: 0.2184930443763733 (0.2251521101442434) | > loss_disc_real_2: 0.2876490652561188 (0.22441993729542878) | > loss_disc_real_3: 0.2568093538284302 (0.23361981066606813) | > loss_disc_real_4: 0.2601456940174103 (0.24061255333787304) | > loss_disc_real_5: 0.22276221215724945 (0.24344861835746442) | > loss_0: 2.8593204021453857 (2.571974006749815) | > grad_norm_0: tensor(327.5322, device='cuda:0') (tensor(283.6962, device='cuda:0')) | > loss_gen: 1.929227352142334 (2.2219549397290765) | > loss_kl: 2.1975557804107666 (2.037483502242524) | > loss_feat: 3.2878003120422363 (4.014526973336431) | > loss_mel: 16.2290096282959 (16.816590099011435) | > loss_duration: 1.5890748500823975 (1.5505065796738966) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.232669830322266 (26.641062009132515) | > grad_norm_1: tensor(15212.0244, device='cuda:0') (tensor(12602.3193, device='cuda:0')) | > current_lr_0: 0.0001816462928800231 | > current_lr_1: 0.0001816462928800231 | > step_time: 0.4649 (0.44667376097986256) | > loader_time: 0.01 (0.008081537182048213) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866160258440904 (+8.007506249655544e-05) | > avg_step_time: 0.4570709080763266 (+0.0023240640129841283) | > avg_loss_disc: 2.580212116241454 (+0.021664760482142498) | > avg_loss_disc_real_0: 0.10197670679067222 (+0.0008323357348710703) | > avg_loss_disc_real_1: 0.22455058047469234 (-0.0006645747893292975) | > avg_loss_disc_real_2: 0.2252818371208621 (+0.001412005491659707) | > avg_loss_disc_real_3: 0.23476766763438642 (+0.0005848447621708275) | > avg_loss_disc_real_4: 0.2425510975676523 (+0.0008653532451307344) | > avg_loss_disc_real_5: 0.24548715625850248 (-0.0003255591006346148) | > avg_loss_0: 2.580212116241454 (+0.021664760482142498) | > avg_grad_norm_0: tensor(288.8778, device='cuda:0') (+tensor(5.9076, device='cuda:0')) | > avg_loss_gen: 2.2159550676883106 (-0.0007868165701210295) | > avg_loss_kl: 2.0647490964808934 (+0.029369303877925024) | > avg_loss_feat: 3.953245364444358 (+0.09376387864771196) | > avg_loss_mel: 16.896503958903565 (+0.07524165301254726) | > avg_loss_duration: 1.561128010212536 (+0.0033648853570646953) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.691581511161697 (+0.20095298659633798) | > avg_grad_norm_1: tensor(12556.8594, device='cuda:0') (+tensor(372.5166, device='cuda:0'))  > EPOCH: 771/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:09:03)   --> TIME: 2025-02-25 16:09:22 -- STEP: 37/72 -- GLOBAL_STEP: 1055550 | > loss_disc: 2.5819740295410156 (2.5575466220443315) | > loss_disc_real_0: 0.17822334170341492 (0.10256352547455479) | > loss_disc_real_1: 0.24610492587089539 (0.22380022422687426) | > loss_disc_real_2: 0.24681243300437927 (0.2335833866853972) | > loss_disc_real_3: 0.21081802248954773 (0.23329935887375394) | > loss_disc_real_4: 0.23156128823757172 (0.24258810120659904) | > loss_disc_real_5: 0.2576867341995239 (0.24492105076441895) | > loss_0: 2.5819740295410156 (2.5575466220443315) | > grad_norm_0: tensor(270.4165, device='cuda:0') (tensor(288.1469, device='cuda:0')) | > loss_gen: 2.346501350402832 (2.2456049886909697) | > loss_kl: 2.008026123046875 (2.0165783166885376) | > loss_feat: 3.958076238632202 (3.9231349004281535) | > loss_mel: 16.256126403808594 (16.83031700752877) | > loss_duration: 1.5629510879516602 (1.523516464877773) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.13167953491211 (26.539151681436074) | > grad_norm_1: tensor(7517.4277, device='cuda:0') (tensor(12654.5820, device='cuda:0')) | > current_lr_0: 0.0001816235870934131 | > current_lr_1: 0.0001816235870934131 | > step_time: 0.4532 (0.4446969290037413) | > loader_time: 0.008 (0.0071603994111757024) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854779968799 (-0.00011380289641903935) | > avg_step_time: 0.460028759190734 (+0.002957851114407384) | > avg_loss_disc: 2.5801037230961756 (-0.00010839314527855137) | > avg_loss_disc_real_0: 0.10722356877276595 (+0.005246861982093731) | > avg_loss_disc_real_1: 0.2251981841006749 (+0.0006476036259825679) | > avg_loss_disc_real_2: 0.23305284074494537 (+0.007771003624083256) | > avg_loss_disc_real_3: 0.23411688964131852 (-0.000650777993067897) | > avg_loss_disc_real_4: 0.24416049181575505 (+0.0016093942481027468) | > avg_loss_disc_real_5: 0.24496076140605227 (-0.0005263948524502071) | > avg_loss_0: 2.5801037230961756 (-0.00010839314527855137) | > avg_grad_norm_0: tensor(289.1103, device='cuda:0') (+tensor(0.2325, device='cuda:0')) | > avg_loss_gen: 2.2148210448278514 (-0.0011340228604592717) | > avg_loss_kl: 2.0583476163971604 (-0.006401480083733091) | > avg_loss_feat: 3.8351709607621314 (-0.11807440368222677) | > avg_loss_mel: 16.857939438081125 (-0.03856452082244033) | > avg_loss_duration: 1.5591297468668979 (-0.0019982633456381294) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.52540883883624 (-0.16617267232545885) | > avg_grad_norm_1: tensor(12399.0254, device='cuda:0') (tensor(-157.8340, device='cuda:0'))  > EPOCH: 772/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:09:40)   --> TIME: 2025-02-25 16:09:49 -- STEP: 15/72 -- GLOBAL_STEP: 1055600 | > loss_disc: 3.5769269466400146 (2.768889077504476) | > loss_disc_real_0: 0.9948875904083252 (0.2568993868927161) | > loss_disc_real_1: 0.23418259620666504 (0.2352594385544459) | > loss_disc_real_2: 0.2958533763885498 (0.20494260936975478) | > loss_disc_real_3: 0.2691076695919037 (0.21899257798989613) | > loss_disc_real_4: 0.1900482177734375 (0.218702099720637) | > loss_disc_real_5: 0.25258585810661316 (0.24009549717108408) | > loss_0: 3.5769269466400146 (2.768889077504476) | > grad_norm_0: tensor(504.5052, device='cuda:0') (tensor(306.7011, device='cuda:0')) | > loss_gen: 2.9751784801483154 (2.338854114214579) | > loss_kl: 2.185115098953247 (1.9461699724197388) | > loss_feat: 3.353708267211914 (3.8996936957041424) | > loss_mel: 18.10543441772461 (17.0511775970459) | > loss_duration: 1.5544979572296143 (1.5239553213119508) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.173933029174805 (26.759850692749023) | > grad_norm_1: tensor(4993.0464, device='cuda:0') (tensor(9699.5117, device='cuda:0')) | > current_lr_0: 0.0001816008841450264 | > current_lr_1: 0.0001816008841450264 | > step_time: 0.4335 (0.4366389592488607) | > loader_time: 0.0086 (0.006876262029012045)  --> TIME: 2025-02-25 16:10:13 -- STEP: 65/72 -- GLOBAL_STEP: 1055650 | > loss_disc: 2.7773544788360596 (2.805260159419133) | > loss_disc_real_0: 0.10303743183612823 (0.2102574718113129) | > loss_disc_real_1: 0.25072142481803894 (0.2728179104053056) | > loss_disc_real_2: 0.22787532210350037 (0.2602346407679411) | > loss_disc_real_3: 0.23588800430297852 (0.24983351482794847) | > loss_disc_real_4: 0.24929040670394897 (0.25378794670105004) | > loss_disc_real_5: 0.25487658381462097 (0.2495975143634356) | > loss_0: 2.7773544788360596 (2.805260159419133) | > grad_norm_0: tensor(220.0681, device='cuda:0') (tensor(270.7449, device='cuda:0')) | > loss_gen: 1.906192660331726 (2.1726166101602407) | > loss_kl: 2.2506179809570312 (2.165421476730934) | > loss_feat: 2.771239757537842 (3.397568288216224) | > loss_mel: 17.013290405273438 (17.32524987734281) | > loss_duration: 1.6266510486602783 (1.5774409165749184) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.5679931640625 (26.638297213040865) | > grad_norm_1: tensor(11868.3799, device='cuda:0') (tensor(8353.9180, device='cuda:0')) | > current_lr_0: 0.0001816008841450264 | > current_lr_1: 0.0001816008841450264 | > step_time: 0.516 (0.4529645773080679) | > loader_time: 0.012 (0.008483065091646638) --> EVAL PERFORMANCE | > avg_loader_time: 0.008746687795074896 (+0.000198888107084896) | > avg_step_time: 0.4590104297852852 (-0.0010183294054487702) | > avg_loss_disc: 2.7986421719403336 (+0.21853844884415796) | > avg_loss_disc_real_0: 0.2073082649476931 (+0.10008469617492716) | > avg_loss_disc_real_1: 0.2701547723001157 (+0.04495658819944079) | > avg_loss_disc_real_2: 0.2582658880823095 (+0.025213047337364125) | > avg_loss_disc_real_3: 0.24713750519383132 (+0.013020615552512799) | > avg_loss_disc_real_4: 0.25334179968061604 (+0.009181307864860988) | > avg_loss_disc_real_5: 0.24942596109820084 (+0.004465199692148564) | > avg_loss_0: 2.7986421719403336 (+0.21853844884415796) | > avg_grad_norm_0: tensor(279.3161, device='cuda:0') (tensor(-9.7942, device='cuda:0')) | > avg_loss_gen: 2.158364821487749 (-0.05645622334010225) | > avg_loss_kl: 2.1656455271680595 (+0.10729791077089912) | > avg_loss_feat: 3.3843580232539647 (-0.45081293750816664) | > avg_loss_mel: 17.305996545603577 (+0.44805710752245176) | > avg_loss_duration: 1.581285293673126 (+0.02215554680622822) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.59565028338365 (+0.07024144454741332) | > avg_grad_norm_1: tensor(8505.8740, device='cuda:0') (tensor(-3893.1514, device='cuda:0'))  > EPOCH: 773/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:10:18)   --> TIME: 2025-02-25 16:10:39 -- STEP: 43/72 -- GLOBAL_STEP: 1055700 | > loss_disc: 2.4926514625549316 (2.6207591932873395) | > loss_disc_real_0: 0.16937848925590515 (0.136708551887856) | > loss_disc_real_1: 0.18439973890781403 (0.22731240786785303) | > loss_disc_real_2: 0.21178220212459564 (0.2229403745989467) | > loss_disc_real_3: 0.2185528725385666 (0.23787248342536216) | > loss_disc_real_4: 0.22062034904956818 (0.2419513124366139) | > loss_disc_real_5: 0.22811545431613922 (0.24613524661507719) | > loss_0: 2.4926514625549316 (2.6207591932873395) | > grad_norm_0: tensor(239.9605, device='cuda:0') (tensor(277.4274, device='cuda:0')) | > loss_gen: 2.730513572692871 (2.1079116893369103) | > loss_kl: 2.1661555767059326 (2.0163038497747374) | > loss_feat: 4.522884368896484 (3.4981129280356473) | > loss_mel: 16.745445251464844 (16.754552042761514) | > loss_duration: 1.562319278717041 (1.5359173957691636) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.727319717407227 (25.912797927856445) | > grad_norm_1: tensor(10293.0039, device='cuda:0') (tensor(9841.3682, device='cuda:0')) | > current_lr_0: 0.00018157818403450826 | > current_lr_1: 0.00018157818403450826 | > step_time: 0.458 (0.4444268082463464) | > loader_time: 0.0119 (0.007672564927921738) --> EVAL PERFORMANCE | > avg_loader_time: 0.008760747775225574 (+1.4059980150677928e-05) | > avg_step_time: 0.45676958393043193 (-0.0022408458548532906) | > avg_loss_disc: 2.6169923493559932 (-0.18164982258434037) | > avg_loss_disc_real_0: 0.14860598606542794 (-0.058702278882265174) | > avg_loss_disc_real_1: 0.22526183787366033 (-0.044892934426455366) | > avg_loss_disc_real_2: 0.22500184975879292 (-0.03326403832351657) | > avg_loss_disc_real_3: 0.2350613235587805 (-0.012076181635050809) | > avg_loss_disc_real_4: 0.24285096265900302 (-0.01049083702161302) | > avg_loss_disc_real_5: 0.24581312041887096 (-0.003612840679329876) | > avg_loss_0: 2.6169923493559932 (-0.18164982258434037) | > avg_grad_norm_0: tensor(283.8394, device='cuda:0') (+tensor(4.5234, device='cuda:0')) | > avg_loss_gen: 2.1332241917999704 (-0.02514062968777875) | > avg_loss_kl: 2.052777708416254 (-0.11286781875180552) | > avg_loss_feat: 3.5193484098138943 (+0.13499038655992956) | > avg_loss_mel: 16.83479575036278 (-0.4712007952407973) | > avg_loss_duration: 1.5612950492912614 (-0.019990244381864697) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.10144099383287 (-0.49420928955078125) | > avg_grad_norm_1: tensor(10252.4062, device='cuda:0') (+tensor(1746.5322, device='cuda:0'))  > EPOCH: 774/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:10:55)   --> TIME: 2025-02-25 16:11:07 -- STEP: 21/72 -- GLOBAL_STEP: 1055750 | > loss_disc: 2.2939722537994385 (2.6618464560735795) | > loss_disc_real_0: 0.05274209380149841 (0.12925628661399796) | > loss_disc_real_1: 0.14453592896461487 (0.21987143726575942) | > loss_disc_real_2: 0.19554859399795532 (0.23159653941790262) | > loss_disc_real_3: 0.20715296268463135 (0.22540109994865598) | > loss_disc_real_4: 0.2348105013370514 (0.23579244599455879) | > loss_disc_real_5: 0.22471517324447632 (0.23213096033959163) | > loss_0: 2.2939722537994385 (2.6618464560735795) | > grad_norm_0: tensor(63.9678, device='cuda:0') (tensor(125.8201, device='cuda:0')) | > loss_gen: 2.568126678466797 (2.160433530807495) | > loss_kl: 3.114544153213501 (3.02227250734965) | > loss_feat: 4.653777122497559 (3.674198513939267) | > loss_mel: 23.012502670288086 (21.488867986769904) | > loss_duration: 1.7285387516021729 (1.7425144854046049) | > amp_scaler: 8.0 (8.0) | > loss_1: 35.07748794555664 (32.088286990211124) | > grad_norm_1: tensor(3261.0427, device='cuda:0') (tensor(5052.5439, device='cuda:0')) | > current_lr_0: 0.00018155548676150393 | > current_lr_1: 0.00018155548676150393 | > step_time: 0.4405 (0.44093297776721774) | > loader_time: 0.0064 (0.006849243527367001)  --> TIME: 2025-02-25 16:11:31 -- STEP: 71/72 -- GLOBAL_STEP: 1055800 | > loss_disc: 2.904825448989868 (2.69082385721341) | > loss_disc_real_0: 0.07277613133192062 (0.11492999378119556) | > loss_disc_real_1: 0.31543436646461487 (0.23323664740777353) | > loss_disc_real_2: 0.27244967222213745 (0.2532238217306808) | > loss_disc_real_3: 0.21961462497711182 (0.2354303965686073) | > loss_disc_real_4: 0.24436084926128387 (0.24233621107020847) | > loss_disc_real_5: 0.19928587973117828 (0.24358277862340633) | > loss_0: 2.904825448989868 (2.69082385721341) | > grad_norm_0: tensor(81.9332, device='cuda:0') (tensor(143.7896, device='cuda:0')) | > loss_gen: 1.638818383216858 (2.166594226595382) | > loss_kl: 2.0608370304107666 (2.548583843338657) | > loss_feat: 2.8886005878448486 (3.6006314183624695) | > loss_mel: 17.614383697509766 (19.593121864426305) | > loss_duration: 1.6079825162887573 (1.6811329562899093) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.810623168945312 (29.5900644651601) | > grad_norm_1: tensor(766.9947, device='cuda:0') (tensor(5586.7441, device='cuda:0')) | > current_lr_0: 0.00018155548676150393 | > current_lr_1: 0.00018155548676150393 | > step_time: 0.4854 (0.4604391950956533) | > loader_time: 0.0081 (0.008777658704301002) --> EVAL PERFORMANCE | > avg_loader_time: 0.008777658704301002 (+1.691092907542789e-05) | > avg_step_time: 0.4604391950956533 (+0.0036696111652213537) | > avg_loss_disc: 2.69082385721341 (+0.07383150785741677) | > avg_loss_disc_real_0: 0.11492999378119556 (-0.033675992284232376) | > avg_loss_disc_real_1: 0.23323664740777353 (+0.007974809534113197) | > avg_loss_disc_real_2: 0.2532238217306808 (+0.028221971971887888) | > avg_loss_disc_real_3: 0.2354303965686073 (+0.0003690730098267758) | > avg_loss_disc_real_4: 0.24233621107020847 (-0.0005147515887945486) | > avg_loss_disc_real_5: 0.24358277862340633 (-0.0022303417954646354) | > avg_loss_0: 2.69082385721341 (+0.07383150785741677) | > avg_grad_norm_0: tensor(143.7896, device='cuda:0') (tensor(-140.0499, device='cuda:0')) | > avg_loss_gen: 2.166594226595382 (+0.03337003479541156) | > avg_loss_kl: 2.548583843338657 (+0.495806134922403) | > avg_loss_feat: 3.6006314183624695 (+0.08128300854857518) | > avg_loss_mel: 19.593121864426305 (+2.758326114063525) | > avg_loss_duration: 1.6811329562899093 (+0.11983790699864794) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.5900644651601 (+3.48862347132723) | > avg_grad_norm_1: tensor(5586.7441, device='cuda:0') (tensor(-4665.6621, device='cuda:0'))  > EPOCH: 775/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:11:33)   --> TIME: 2025-02-25 16:11:57 -- STEP: 49/72 -- GLOBAL_STEP: 1055850 | > loss_disc: 2.4573159217834473 (2.6112644623736943) | > loss_disc_real_0: 0.06984128057956696 (0.08647386288764522) | > loss_disc_real_1: 0.21808841824531555 (0.23495623773458052) | > loss_disc_real_2: 0.2791474163532257 (0.24885021484628014) | > loss_disc_real_3: 0.2636273205280304 (0.24986354520126264) | > loss_disc_real_4: 0.25289788842201233 (0.24975235425696088) | > loss_disc_real_5: 0.2608832120895386 (0.2500810960725862) | > loss_0: 2.4573159217834473 (2.6112644623736943) | > grad_norm_0: tensor(51.8837, device='cuda:0') (tensor(163.1049, device='cuda:0')) | > loss_gen: 2.157642364501953 (2.1885508834099276) | > loss_kl: 1.96494460105896 (2.0594162819336885) | > loss_feat: 3.8479647636413574 (3.7202992731211135) | > loss_mel: 16.235042572021484 (17.456550267277922) | > loss_duration: 1.6254359483718872 (1.5631737563074852) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.831029891967773 (26.98799028201979) | > grad_norm_1: tensor(2148.6985, device='cuda:0') (tensor(6687.6606, device='cuda:0')) | > current_lr_0: 0.00018153279232565873 | > current_lr_1: 0.00018153279232565873 | > step_time: 0.442 (0.4461330345698765) | > loader_time: 0.0096 (0.007820883575750857) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754115709116763 (-2.3542995184238497e-05) | > avg_step_time: 0.458039253530368 (-0.0023999415652852796) | > avg_loss_disc: 2.663885281119548 (-0.026938576093861855) | > avg_loss_disc_real_0: 0.1085568011646539 (-0.006373192616541665) | > avg_loss_disc_real_1: 0.22667805766555624 (-0.006558589742217291) | > avg_loss_disc_real_2: 0.2461167122696487 (-0.007107109461032102) | > avg_loss_disc_real_3: 0.2438982257121046 (+0.008467829143497307) | > avg_loss_disc_real_4: 0.24798723656526758 (+0.005651025495059109) | > avg_loss_disc_real_5: 0.24863724011770436 (+0.005054461494298029) | > avg_loss_0: 2.663885281119548 (-0.026938576093861855) | > avg_grad_norm_0: tensor(180.1917, device='cuda:0') (+tensor(36.4022, device='cuda:0')) | > avg_loss_gen: 2.0771128325395183 (-0.08948139405586364) | > avg_loss_kl: 2.0703131128364896 (-0.4782707305021674) | > avg_loss_feat: 3.4874703212523124 (-0.1131610971101571) | > avg_loss_mel: 17.34478444112858 (-2.2483374232977233) | > avg_loss_duration: 1.5776678877817074 (-0.10346506850820192) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.557348439391227 (-3.032716025768874) | > avg_grad_norm_1: tensor(6652.4507, device='cuda:0') (+tensor(1065.7065, device='cuda:0'))  > EPOCH: 776/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:12:10)   --> TIME: 2025-02-25 16:12:24 -- STEP: 27/72 -- GLOBAL_STEP: 1055900 | > loss_disc: 2.921621799468994 (2.7629258544356734) | > loss_disc_real_0: 0.03955036401748657 (0.24467769706690756) | > loss_disc_real_1: 0.20534175634384155 (0.21387179251070376) | > loss_disc_real_2: 0.2337072491645813 (0.23508917347148614) | > loss_disc_real_3: 0.24655741453170776 (0.2353835337691837) | > loss_disc_real_4: 0.23864351212978363 (0.23801822518860852) | > loss_disc_real_5: 0.22504174709320068 (0.23493361859409898) | > loss_0: 2.921621799468994 (2.7629258544356734) | > grad_norm_0: tensor(58.9458, device='cuda:0') (tensor(259.8983, device='cuda:0')) | > loss_gen: 1.6422852277755737 (2.0426280807565758) | > loss_kl: 2.2286953926086426 (2.144992060131497) | > loss_feat: 3.0187978744506836 (3.446084428716589) | > loss_mel: 17.809587478637695 (18.111418406168614) | > loss_duration: 1.543035626411438 (1.5487245497880158) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.242401123046875 (27.293847613864475) | > grad_norm_1: tensor(999.1645, device='cuda:0') (tensor(4757.9531, device='cuda:0')) | > current_lr_0: 0.000181510100726618 | > current_lr_1: 0.000181510100726618 | > step_time: 0.4411 (0.4410788394786693) | > loader_time: 0.0082 (0.00716009846440068) --> EVAL PERFORMANCE | > avg_loader_time: 0.00875739984109368 (+3.284131976916474e-06) | > avg_step_time: 0.459150277392965 (+0.0011110238625969826) | > avg_loss_disc: 2.731573457449256 (+0.0676881763297077) | > avg_loss_disc_real_0: 0.18777729911913338 (+0.07922049795447948) | > avg_loss_disc_real_1: 0.22445420417147624 (-0.002223853494079997) | > avg_loss_disc_real_2: 0.24411572725840017 (-0.0020009850112485406) | > avg_loss_disc_real_3: 0.2399320058839422 (-0.003966219828162393) | > avg_loss_disc_real_4: 0.24562851832786076 (-0.002358718237406826) | > avg_loss_disc_real_5: 0.24609331471819273 (-0.0025439253995116207) | > avg_loss_0: 2.731573457449256 (+0.0676881763297077) | > avg_grad_norm_0: tensor(212.7110, device='cuda:0') (+tensor(32.5193, device='cuda:0')) | > avg_loss_gen: 2.010900650225895 (-0.06621218231362347) | > avg_loss_kl: 2.241200332910242 (+0.17088722007375257) | > avg_loss_feat: 3.2900358085900967 (-0.19743451266221568) | > avg_loss_mel: 18.152378512100427 (+0.8075940709718452) | > avg_loss_duration: 1.5895846141895777 (+0.011916726407870337) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.284099820634008 (+0.7267513812427815) | > avg_grad_norm_1: tensor(6847.5293, device='cuda:0') (+tensor(195.0786, device='cuda:0'))  > EPOCH: 777/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:12:47)   --> TIME: 2025-02-25 16:12:51 -- STEP: 5/72 -- GLOBAL_STEP: 1055950 | > loss_disc: 2.5391101837158203 (2.603626823425293) | > loss_disc_real_0: 0.1577964872121811 (0.15075822472572326) | > loss_disc_real_1: 0.25892582535743713 (0.2285005956888199) | > loss_disc_real_2: 0.22480008006095886 (0.22899928390979768) | > loss_disc_real_3: 0.23151333630084991 (0.23083640336990358) | > loss_disc_real_4: 0.2511020302772522 (0.23987087309360505) | > loss_disc_real_5: 0.24857197701931 (0.23697245121002197) | > loss_0: 2.5391101837158203 (2.603626823425293) | > grad_norm_0: tensor(22.5947, device='cuda:0') (tensor(237.2927, device='cuda:0')) | > loss_gen: 1.9964817762374878 (2.0098870992660522) | > loss_kl: 2.0710060596466064 (2.0772101879119873) | > loss_feat: 3.4804723262786865 (3.336779165267944) | > loss_mel: 17.3729248046875 (17.841609954833984) | > loss_duration: 1.5429308414459229 (1.5536142587661743) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.463815689086914 (26.81910057067871) | > grad_norm_1: tensor(10815.1855, device='cuda:0') (tensor(10733.9531, device='cuda:0')) | > current_lr_0: 0.00018148741196402716 | > current_lr_1: 0.00018148741196402716 | > step_time: 0.4374 (0.4311997890472412) | > loader_time: 0.0063 (0.006452083587646484)  --> TIME: 2025-02-25 16:13:15 -- STEP: 55/72 -- GLOBAL_STEP: 1056000 | > loss_disc: 3.099355459213257 (2.6395784031261096) | > loss_disc_real_0: 0.12227018177509308 (0.13786107559095725) | > loss_disc_real_1: 0.25406673550605774 (0.2268371113321998) | > loss_disc_real_2: 0.11900319159030914 (0.23422003307125785) | > loss_disc_real_3: 0.23950645327568054 (0.23246282501654192) | > loss_disc_real_4: 0.247965469956398 (0.2350314205343073) | > loss_disc_real_5: 0.24103794991970062 (0.23578342605720867) | > loss_0: 3.099355459213257 (2.6395784031261096) | > grad_norm_0: tensor(39.6270, device='cuda:0') (tensor(241.3586, device='cuda:0')) | > loss_gen: 1.5024080276489258 (2.064598798751832) | > loss_kl: 1.8739426136016846 (2.0127108118750825) | > loss_feat: 2.76313853263855 (3.5606201778758657) | > loss_mel: 16.569278717041016 (17.26257943240079) | > loss_duration: 1.6041375398635864 (1.5581478032198819) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.312904357910156 (26.458657004616477) | > grad_norm_1: tensor(3480.2651, device='cuda:0') (tensor(8543.6113, device='cuda:0')) | > current_lr_0: 0.00018148741196402716 | > current_lr_1: 0.00018148741196402716 | > step_time: 0.465 (0.4467882286418568) | > loader_time: 0.0092 (0.007983489470048382) --> EVAL PERFORMANCE | > avg_loader_time: 0.008748000776264026 (-9.39906482965401e-06) | > avg_step_time: 0.46393605017326245 (+0.004785772780297459) | > avg_loss_disc: 2.641845286731989 (-0.08972817071726702) | > avg_loss_disc_real_0: 0.12314572022624413 (-0.06463157889288924) | > avg_loss_disc_real_1: 0.2210746361336238 (-0.003379568037852443) | > avg_loss_disc_real_2: 0.24661820056572767 (+0.002502473307327502) | > avg_loss_disc_real_3: 0.22881520612978598 (-0.01111679975415622) | > avg_loss_disc_real_4: 0.23893193333921298 (-0.00669658498864778) | > avg_loss_disc_real_5: 0.23786347432875296 (-0.008229840389439774) | > avg_loss_0: 2.641845286731989 (-0.08972817071726702) | > avg_grad_norm_0: tensor(216.3827, device='cuda:0') (+tensor(3.6717, device='cuda:0')) | > avg_loss_gen: 2.0847742272094942 (+0.07387357698359942) | > avg_loss_kl: 2.0344974776388893 (-0.2067028552713528) | > avg_loss_feat: 3.560472394379092 (+0.2704365857889952) | > avg_loss_mel: 17.222482802162713 (-0.9298957099377141) | > avg_loss_duration: 1.5703853402339236 (-0.019199273955654128) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.472612273525183 (-0.8114875471088254) | > avg_grad_norm_1: tensor(8016.5264, device='cuda:0') (+tensor(1168.9971, device='cuda:0'))  > EPOCH: 778/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:13:25)   --> TIME: 2025-02-25 16:13:42 -- STEP: 33/72 -- GLOBAL_STEP: 1056050 | > loss_disc: 2.500514030456543 (2.5429042758363667) | > loss_disc_real_0: 0.11499998718500137 (0.07089833254841241) | > loss_disc_real_1: 0.22304782271385193 (0.233231552622535) | > loss_disc_real_2: 0.231111079454422 (0.238897800897107) | > loss_disc_real_3: 0.21662887930870056 (0.23429247466000644) | > loss_disc_real_4: 0.22713659703731537 (0.23781917944098963) | > loss_disc_real_5: 0.24966467916965485 (0.24176457027594248) | > loss_0: 2.500514030456543 (2.5429042758363667) | > grad_norm_0: tensor(186.5138, device='cuda:0') (tensor(152.9145, device='cuda:0')) | > loss_gen: 2.308732271194458 (2.1845407955574268) | > loss_kl: 2.063992977142334 (1.9534020640633323) | > loss_feat: 3.952435255050659 (3.8292380751985493) | > loss_mel: 16.85906982421875 (17.135700341426965) | > loss_duration: 1.5340121984481812 (1.5226545442234387) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.718242645263672 (26.6255356759736) | > grad_norm_1: tensor(4643.5864, device='cuda:0') (tensor(7420.2920, device='cuda:0')) | > current_lr_0: 0.00018146472603753166 | > current_lr_1: 0.00018146472603753166 | > step_time: 0.4551 (0.44281057155493536) | > loader_time: 0.0095 (0.007143670862371271) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867282840567575 (-7.517237058827543e-05) | > avg_step_time: 0.458420175901601 (-0.005515874271661447) | > avg_loss_disc: 2.5450531193907833 (-0.09679216734120555) | > avg_loss_disc_real_0: 0.07608820579316415 (-0.04705751443307998) | > avg_loss_disc_real_1: 0.22329080566554002 (+0.0022161695319162233) | > avg_loss_disc_real_2: 0.2376177470448991 (-0.009000453520828566) | > avg_loss_disc_real_3: 0.2355111107020311 (+0.006695904572245115) | > avg_loss_disc_real_4: 0.24140815352889852 (+0.0024762201896855462) | > avg_loss_disc_real_5: 0.24574969708919525 (+0.00788622276044229) | > avg_loss_0: 2.5450531193907833 (-0.09679216734120555) | > avg_grad_norm_0: tensor(162.3229, device='cuda:0') (tensor(-54.0599, device='cuda:0')) | > avg_loss_gen: 2.182266540930305 (+0.09749231372081058) | > avg_loss_kl: 1.993481998712244 (-0.041015478926645255) | > avg_loss_feat: 3.7406228495315768 (+0.18015045515248485) | > avg_loss_mel: 16.945553443801234 (-0.27692935836147825) | > avg_loss_duration: 1.5594076992760244 (-0.010977640957899215) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42133253393039 (-0.05127973959479348) | > avg_grad_norm_1: tensor(7505.2568, device='cuda:0') (tensor(-511.2695, device='cuda:0'))  > EPOCH: 779/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:14:02)   --> TIME: 2025-02-25 16:14:09 -- STEP: 11/72 -- GLOBAL_STEP: 1056100 | > loss_disc: 2.604989767074585 (2.5307696732607754) | > loss_disc_real_0: 0.07030981779098511 (0.08185434375296939) | > loss_disc_real_1: 0.20499619841575623 (0.22295123880559747) | > loss_disc_real_2: 0.23309050500392914 (0.22624594921415503) | > loss_disc_real_3: 0.24218665063381195 (0.24148089777339588) | > loss_disc_real_4: 0.23587864637374878 (0.24231936579400842) | > loss_disc_real_5: 0.25924742221832275 (0.2460659918459979) | > loss_0: 2.604989767074585 (2.5307696732607754) | > grad_norm_0: tensor(251.3419, device='cuda:0') (tensor(184.5157, device='cuda:0')) | > loss_gen: 2.0598793029785156 (2.266282016580755) | > loss_kl: 2.1274099349975586 (1.9819553331895308) | > loss_feat: 3.6811161041259766 (3.9613999019969595) | > loss_mel: 17.12961196899414 (17.002072767777875) | > loss_duration: 1.5057979822158813 (1.5247293710708618) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.503814697265625 (26.736439098011363) | > grad_norm_1: tensor(10393.7139, device='cuda:0') (tensor(7704.1606, device='cuda:0')) | > current_lr_0: 0.00018144204294677697 | > current_lr_1: 0.00018144204294677697 | > step_time: 0.4389 (0.43605338443409314) | > loader_time: 0.0057 (0.006652008403431286)  --> TIME: 2025-02-25 16:14:32 -- STEP: 61/72 -- GLOBAL_STEP: 1056150 | > loss_disc: 2.8269715309143066 (2.5523021377500936) | > loss_disc_real_0: 0.1043686717748642 (0.0888091535231129) | > loss_disc_real_1: 0.22611026465892792 (0.223232594425561) | > loss_disc_real_2: 0.26003339886665344 (0.23285009333344756) | > loss_disc_real_3: 0.281767338514328 (0.2374377394797372) | > loss_disc_real_4: 0.21879364550113678 (0.24258237735169833) | > loss_disc_real_5: 0.27076417207717896 (0.24631687290355808) | > loss_0: 2.8269715309143066 (2.5523021377500936) | > grad_norm_0: tensor(72.4221, device='cuda:0') (tensor(160.8990, device='cuda:0')) | > loss_gen: 2.2372374534606934 (2.2312987007078577) | > loss_kl: 2.30583119392395 (2.0039969878118544) | > loss_feat: 4.20304536819458 (3.870059419850834) | > loss_mel: 18.189905166625977 (16.869888524540134) | > loss_duration: 1.6160916090011597 (1.5496196336433539) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.55211067199707 (26.52486325873703) | > grad_norm_1: tensor(5656.1719, device='cuda:0') (tensor(7502.4595, device='cuda:0')) | > current_lr_0: 0.00018144204294677697 | > current_lr_1: 0.00018144204294677697 | > step_time: 0.4885 (0.44884695381414697) | > loader_time: 0.0108 (0.008263970984787239) --> EVAL PERFORMANCE | > avg_loader_time: 0.008721207229184434 (+4.837882350868308e-05) | > avg_step_time: 0.45592364123169804 (-0.002496534669902961) | > avg_loss_disc: 2.555036047814597 (+0.009982928423813764) | > avg_loss_disc_real_0: 0.08964191791667064 (+0.013553712123506492) | > avg_loss_disc_real_1: 0.22309491730911632 (-0.00019588835642370128) | > avg_loss_disc_real_2: 0.2314692693696895 (-0.006148477675209596) | > avg_loss_disc_real_3: 0.23581824651066685 (+0.0003071358086357556) | > avg_loss_disc_real_4: 0.23959776297421523 (-0.0018103905546832944) | > avg_loss_disc_real_5: 0.24657702424996336 (+0.0008273271607681099) | > avg_loss_0: 2.555036047814597 (+0.009982928423813764) | > avg_grad_norm_0: tensor(154.9415, device='cuda:0') (tensor(-7.3813, device='cuda:0')) | > avg_loss_gen: 2.2146498905101293 (+0.03238334957982447) | > avg_loss_kl: 2.030818141682047 (+0.03733614296980292) | > avg_loss_feat: 3.850315940212196 (+0.10969309068061905) | > avg_loss_mel: 17.00109775973038 (+0.0555443159291471) | > avg_loss_duration: 1.5597977117753365 (+0.0003900124993121423) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.656679422083037 (+0.23534688815264815) | > avg_grad_norm_1: tensor(7247.6523, device='cuda:0') (tensor(-257.6045, device='cuda:0'))  > EPOCH: 780/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:14:39)   --> TIME: 2025-02-25 16:14:59 -- STEP: 39/72 -- GLOBAL_STEP: 1056200 | > loss_disc: 2.5739071369171143 (2.677585314481686) | > loss_disc_real_0: 0.13324430584907532 (0.17246416784249818) | > loss_disc_real_1: 0.26782017946243286 (0.23109280298917723) | > loss_disc_real_2: 0.20987683534622192 (0.22838284189884478) | > loss_disc_real_3: 0.2387654334306717 (0.23575889720366552) | > loss_disc_real_4: 0.2327602356672287 (0.23741102677125198) | > loss_disc_real_5: 0.22069168090820312 (0.24241228783741975) | > loss_0: 2.5739071369171143 (2.677585314481686) | > grad_norm_0: tensor(218.7588, device='cuda:0') (tensor(195.2747, device='cuda:0')) | > loss_gen: 2.1144585609436035 (2.093607040551993) | > loss_kl: 2.068514823913574 (2.043382085286654) | > loss_feat: 3.934447765350342 (3.6098783138470774) | > loss_mel: 17.82204818725586 (17.240078559288612) | > loss_duration: 1.5621237754821777 (1.5354125224626982) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.5015926361084 (26.522358674269455) | > grad_norm_1: tensor(8877.2432, device='cuda:0') (tensor(7113.2363, device='cuda:0')) | > current_lr_0: 0.0001814193626914086 | > current_lr_1: 0.0001814193626914086 | > step_time: 0.4552 (0.44116445688100964) | > loader_time: 0.0084 (0.007430871327718099) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868274796176964 (-3.84592674147928e-05) | > avg_step_time: 0.45859380507133374 (+0.0026701638396356975) | > avg_loss_disc: 2.650249162190397 (+0.09521311437580016) | > avg_loss_disc_real_0: 0.15504226928025905 (+0.0654003513635884) | > avg_loss_disc_real_1: 0.22683136479955324 (+0.003736447490436917) | > avg_loss_disc_real_2: 0.2339599997224942 (+0.0024907303528046987) | > avg_loss_disc_real_3: 0.23689159541062907 (+0.0010733488999622176) | > avg_loss_disc_real_4: 0.23979397747718112 (+0.00019621450296589127) | > avg_loss_disc_real_5: 0.2426372468471527 (-0.003939777402810651) | > avg_loss_0: 2.650249162190397 (+0.09521311437580016) | > avg_grad_norm_0: tensor(191.4203, device='cuda:0') (+tensor(36.4787, device='cuda:0')) | > avg_loss_gen: 2.1403538475573893 (-0.07429604295273995) | > avg_loss_kl: 2.0727664198673947 (+0.04194827818534774) | > avg_loss_feat: 3.6405587968691973 (-0.20975714334299855) | > avg_loss_mel: 17.102377676628013 (+0.1012799168976315) | > avg_loss_duration: 1.5642896151878465 (+0.004491903412509934) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.520346466924103 (-0.1363329551589345) | > avg_grad_norm_1: tensor(6608.4722, device='cuda:0') (tensor(-639.1802, device='cuda:0'))  > EPOCH: 781/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:15:17)   --> TIME: 2025-02-25 16:15:26 -- STEP: 17/72 -- GLOBAL_STEP: 1056250 | > loss_disc: 2.80787992477417 (2.658602279775283) | > loss_disc_real_0: 0.03041522204875946 (0.06267457159564775) | > loss_disc_real_1: 0.28176793456077576 (0.24622632826075835) | > loss_disc_real_2: 0.20666815340518951 (0.20749929547309875) | > loss_disc_real_3: 0.23516130447387695 (0.22466770866337946) | > loss_disc_real_4: 0.36611223220825195 (0.2542106710812625) | > loss_disc_real_5: 0.25608590245246887 (0.24447469588588266) | > loss_0: 2.80787992477417 (2.658602279775283) | > grad_norm_0: tensor(445.9040, device='cuda:0') (tensor(179.5177, device='cuda:0')) | > loss_gen: 1.9362092018127441 (2.0396665264578426) | > loss_kl: 2.0072622299194336 (2.112349643426783) | > loss_feat: 3.4794938564300537 (3.8335672967574177) | > loss_mel: 17.359848022460938 (17.647425090565402) | > loss_duration: 1.5349153280258179 (1.5260508551317102) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.31772804260254 (27.159059524536133) | > grad_norm_1: tensor(12914.9014, device='cuda:0') (tensor(7503.0850, device='cuda:0')) | > current_lr_0: 0.00018139668527107218 | > current_lr_1: 0.00018139668527107218 | > step_time: 0.4379 (0.4403126520269057) | > loader_time: 0.0061 (0.007074762793148265)  --> TIME: 2025-02-25 16:15:50 -- STEP: 67/72 -- GLOBAL_STEP: 1056300 | > loss_disc: 2.5410263538360596 (2.6033041619542816) | > loss_disc_real_0: 0.06809015572071075 (0.11872461046189514) | > loss_disc_real_1: 0.22071588039398193 (0.23049445850635641) | > loss_disc_real_2: 0.24630112946033478 (0.22414357381970135) | > loss_disc_real_3: 0.21578268706798553 (0.22934875248083428) | > loss_disc_real_4: 0.24668249487876892 (0.24299350091770514) | > loss_disc_real_5: 0.22926180064678192 (0.24025697205493698) | > loss_0: 2.5410263538360596 (2.6033041619542816) | > grad_norm_0: tensor(134.5362, device='cuda:0') (tensor(186.1811, device='cuda:0')) | > loss_gen: 2.143801689147949 (2.1429848457450302) | > loss_kl: 2.4287521839141846 (2.470157950671751) | > loss_feat: 3.64620041847229 (3.716078719096397) | > loss_mel: 17.908222198486328 (18.63000632044095) | > loss_duration: 1.6643126010894775 (1.5915712135941236) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.791288375854492 (28.550799384045956) | > grad_norm_1: tensor(11197.3643, device='cuda:0') (tensor(7929.5698, device='cuda:0')) | > current_lr_0: 0.00018139668527107218 | > current_lr_1: 0.00018139668527107218 | > step_time: 0.5077 (0.45694870379433705) | > loader_time: 0.0121 (0.008502942412646845) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684299361537876 (+1.551399768234818e-06) | > avg_step_time: 0.46026498499050944 (+0.0016711799191757026) | > avg_loss_disc: 2.6013028487353256 (-0.04894631345507161) | > avg_loss_disc_real_0: 0.11736264077185744 (-0.037679628508401605) | > avg_loss_disc_real_1: 0.23177213903883814 (+0.004940774239284906) | > avg_loss_disc_real_2: 0.22557555789678868 (-0.008384441825705524) | > avg_loss_disc_real_3: 0.23050608727293956 (-0.006385508137689511) | > avg_loss_disc_real_4: 0.24317623821782394 (+0.003382260740642823) | > avg_loss_disc_real_5: 0.2398025047191432 (-0.002834742128009521) | > avg_loss_0: 2.6013028487353256 (-0.04894631345507161) | > avg_grad_norm_0: tensor(188.2632, device='cuda:0') (tensor(-3.1570, device='cuda:0')) | > avg_loss_gen: 2.145585029897556 (+0.005231182340166729) | > avg_loss_kl: 2.46662651988822 (+0.39386010002082505) | > avg_loss_feat: 3.7129877721759637 (+0.0724289753067664) | > avg_loss_mel: 18.5829432984473 (+1.480565621819288) | > avg_loss_duration: 1.593988408505077 (+0.029698793317230576) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.502131341208873 (+1.9817848742847701) | > avg_grad_norm_1: tensor(7970.8643, device='cuda:0') (+tensor(1362.3921, device='cuda:0'))  > EPOCH: 782/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:15:54)   --> TIME: 2025-02-25 16:16:17 -- STEP: 45/72 -- GLOBAL_STEP: 1056350 | > loss_disc: 2.6057472229003906 (2.5173882590399836) | > loss_disc_real_0: 0.26724281907081604 (0.10189557067222066) | > loss_disc_real_1: 0.243150532245636 (0.21374328467581008) | > loss_disc_real_2: 0.21843944489955902 (0.22050523062547048) | > loss_disc_real_3: 0.23367933928966522 (0.2316584308942159) | > loss_disc_real_4: 0.24891027808189392 (0.2403330193625556) | > loss_disc_real_5: 0.25860321521759033 (0.24338015649053785) | > loss_0: 2.6057472229003906 (2.5173882590399836) | > grad_norm_0: tensor(332.5468, device='cuda:0') (tensor(198.8518, device='cuda:0')) | > loss_gen: 2.5325775146484375 (2.235990476608277) | > loss_kl: 2.200477361679077 (2.186192546950446) | > loss_feat: 3.98081636428833 (3.927861780590481) | > loss_mel: 17.81927490234375 (17.68661130269369) | > loss_duration: 1.5870702266693115 (1.5530063761605157) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.120216369628906 (27.589662255181207) | > grad_norm_1: tensor(7664.0869, device='cuda:0') (tensor(9366.2188, device='cuda:0')) | > current_lr_0: 0.0001813740106854133 | > current_lr_1: 0.0001813740106854133 | > step_time: 0.4573 (0.4443789694044325) | > loader_time: 0.0091 (0.00760098033481174) --> EVAL PERFORMANCE | > avg_loader_time: 0.008681871521640832 (-2.4278398970435655e-06) | > avg_step_time: 0.4579355615965078 (-0.0023294233940016618) | > avg_loss_disc: 2.54693167981967 (-0.05437116891565541) | > avg_loss_disc_real_0: 0.10465571845711118 (-0.012706922314746266) | > avg_loss_disc_real_1: 0.2201604918694832 (-0.01161164716935495) | > avg_loss_disc_real_2: 0.23465895358945282 (+0.009083395692664142) | > avg_loss_disc_real_3: 0.23312714402104767 (+0.0026210567481081137) | > avg_loss_disc_real_4: 0.24211917271916295 (-0.0010570654986609973) | > avg_loss_disc_real_5: 0.24491331149154985 (+0.005110806772406662) | > avg_loss_0: 2.54693167981967 (-0.05437116891565541) | > avg_grad_norm_0: tensor(201.7525, device='cuda:0') (+tensor(13.4893, device='cuda:0')) | > avg_loss_gen: 2.211442428575436 (+0.06585739867787987) | > avg_loss_kl: 2.1814548247297045 (-0.28517169515851526) | > avg_loss_feat: 3.7894746384150544 (+0.07648686623909073) | > avg_loss_mel: 17.472836131780927 (-1.1101071666663742) | > avg_loss_duration: 1.5736064994838876 (-0.020381909021189415) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.22881429967746 (-1.2733170415314135) | > avg_grad_norm_1: tensor(9047.8154, device='cuda:0') (+tensor(1076.9512, device='cuda:0'))  > EPOCH: 783/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:16:31)   --> TIME: 2025-02-25 16:16:44 -- STEP: 23/72 -- GLOBAL_STEP: 1056400 | > loss_disc: 2.8890116214752197 (2.7123898423236352) | > loss_disc_real_0: 0.4510880708694458 (0.16660483829353168) | > loss_disc_real_1: 0.18038612604141235 (0.24496793034284012) | > loss_disc_real_2: 0.21662570536136627 (0.23279019088848776) | > loss_disc_real_3: 0.245939239859581 (0.23484454271586044) | > loss_disc_real_4: 0.21954840421676636 (0.24982184106888977) | > loss_disc_real_5: 0.22978751361370087 (0.2509978083164796) | > loss_0: 2.8890116214752197 (2.7123898423236352) | > grad_norm_0: tensor(386.0732, device='cuda:0') (tensor(243.4938, device='cuda:0')) | > loss_gen: 2.9620065689086914 (2.154655036718949) | > loss_kl: 1.9544938802719116 (2.023840453313745) | > loss_feat: 3.6648030281066895 (3.4709702471028203) | > loss_mel: 16.924158096313477 (17.422977198725157) | > loss_duration: 1.5421377420425415 (1.5308647103931592) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.04759979248047 (26.60330755814262) | > grad_norm_1: tensor(3761.8232, device='cuda:0') (tensor(8794.6143, device='cuda:0')) | > current_lr_0: 0.00018135133893407761 | > current_lr_1: 0.00018135133893407761 | > step_time: 0.4379 (0.43612396198770276) | > loader_time: 0.0068 (0.006870943567027216) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651243129246672 (-3.0628392394160125e-05) | > avg_step_time: 0.45523582042103083 (-0.002699741175476944) | > avg_loss_disc: 2.656088530177801 (+0.10915685035813105) | > avg_loss_disc_real_0: 0.13757716895828784 (+0.03292145050117666) | > avg_loss_disc_real_1: 0.2278765471468509 (+0.007716055277367712) | > avg_loss_disc_real_2: 0.247510218494375 (+0.012851264904922166) | > avg_loss_disc_real_3: 0.23476438161352992 (+0.0016372375924822435) | > avg_loss_disc_real_4: 0.24303918783093842 (+0.000920015111775474) | > avg_loss_disc_real_5: 0.24591853186278276 (+0.0010052203712329066) | > avg_loss_0: 2.656088530177801 (+0.10915685035813105) | > avg_grad_norm_0: tensor(204.5996, device='cuda:0') (+tensor(2.8471, device='cuda:0')) | > avg_loss_gen: 2.1610736108162034 (-0.05036881775923252) | > avg_loss_kl: 2.1033525937040087 (-0.0781022310256958) | > avg_loss_feat: 3.6186434215223287 (-0.17083121689272573) | > avg_loss_mel: 17.201854249121435 (-0.2709818826594912) | > avg_loss_duration: 1.568837023117173 (-0.004769476366714542) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.653760829441982 (-0.5750534702354777) | > avg_grad_norm_1: tensor(6651.0220, device='cuda:0') (tensor(-2396.7935, device='cuda:0'))  > EPOCH: 784/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:17:09)   --> TIME: 2025-02-25 16:17:11 -- STEP: 1/72 -- GLOBAL_STEP: 1056450 | > loss_disc: 2.8994288444519043 (2.8994288444519043) | > loss_disc_real_0: 0.05434975028038025 (0.05434975028038025) | > loss_disc_real_1: 0.2264256626367569 (0.2264256626367569) | > loss_disc_real_2: 0.19334806501865387 (0.19334806501865387) | > loss_disc_real_3: 0.26881924271583557 (0.26881924271583557) | > loss_disc_real_4: 0.20805850625038147 (0.20805850625038147) | > loss_disc_real_5: 0.2348279058933258 (0.2348279058933258) | > loss_0: 2.8994288444519043 (2.8994288444519043) | > grad_norm_0: tensor(13.2918, device='cuda:0') (tensor(13.2918, device='cuda:0')) | > loss_gen: 1.8912113904953003 (1.8912113904953003) | > loss_kl: 1.8280102014541626 (1.8280102014541626) | > loss_feat: 3.442605495452881 (3.442605495452881) | > loss_mel: 18.125455856323242 (18.125455856323242) | > loss_duration: 1.5216727256774902 (1.5216727256774902) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.808956146240234 (26.808956146240234) | > grad_norm_1: tensor(2411.7556, device='cuda:0') (tensor(2411.7556, device='cuda:0')) | > current_lr_0: 0.00018132867001671084 | > current_lr_1: 0.00018132867001671084 | > step_time: 0.4426 (0.4425656795501709) | > loader_time: 0.007 (0.007013797760009766)  --> TIME: 2025-02-25 16:17:34 -- STEP: 51/72 -- GLOBAL_STEP: 1056500 | > loss_disc: 3.0564661026000977 (2.698984225591024) | > loss_disc_real_0: 0.3729749619960785 (0.1378386303344194) | > loss_disc_real_1: 0.2379222959280014 (0.23369708321258134) | > loss_disc_real_2: 0.2283976823091507 (0.23125510472877353) | > loss_disc_real_3: 0.2501395642757416 (0.23267353866614549) | > loss_disc_real_4: 0.26254183053970337 (0.24265274288607577) | > loss_disc_real_5: 0.2643389403820038 (0.24596723154479383) | > loss_0: 3.0564661026000977 (2.698984225591024) | > grad_norm_0: tensor(579.9211, device='cuda:0') (tensor(181.8599, device='cuda:0')) | > loss_gen: 1.7401586771011353 (2.01192110192542) | > loss_kl: 2.3932583332061768 (2.0502312253503248) | > loss_feat: 2.944611072540283 (3.434104045232137) | > loss_mel: 16.805339813232422 (16.99979241689046) | > loss_duration: 1.6021792888641357 (1.548303480241813) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.485546112060547 (26.044352325738647) | > grad_norm_1: tensor(677.1354, device='cuda:0') (tensor(6319.6069, device='cuda:0')) | > current_lr_0: 0.00018132867001671084 | > current_lr_1: 0.00018132867001671084 | > step_time: 0.4736 (0.4467221334868786) | > loader_time: 0.0089 (0.007919461119408699) --> EVAL PERFORMANCE | > avg_loader_time: 0.008741731375036105 (+9.048824578943362e-05) | > avg_step_time: 0.45709299033796286 (+0.0018571699169320222) | > avg_loss_disc: 2.6878231411248867 (+0.03173461094708552) | > avg_loss_disc_real_0: 0.14841553883653294 (+0.010838369878245102) | > avg_loss_disc_real_1: 0.23121172389094258 (+0.0033351767440916724) | > avg_loss_disc_real_2: 0.24020058924043683 (-0.00730962925393816) | > avg_loss_disc_real_3: 0.2324359696096098 (-0.002328412003920116) | > avg_loss_disc_real_4: 0.24500189154920443 (+0.0019627037182660123) | > avg_loss_disc_real_5: 0.2463867576189444 (+0.00046822575616164985) | > avg_loss_0: 2.6878231411248867 (+0.03173461094708552) | > avg_grad_norm_0: tensor(187.1955, device='cuda:0') (tensor(-17.4041, device='cuda:0')) | > avg_loss_gen: 2.0490336703582557 (-0.11203994045794774) | > avg_loss_kl: 2.1499096759608096 (+0.046557082256800886) | > avg_loss_feat: 3.4675674270576153 (-0.15107599446471331) | > avg_loss_mel: 17.35554054421438 (+0.15368629509294607) | > avg_loss_duration: 1.5703370436816149 (+0.001500020564441762) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.592388314260564 (-0.06137251518141795) | > avg_grad_norm_1: tensor(6162.3716, device='cuda:0') (tensor(-488.6504, device='cuda:0'))  > EPOCH: 785/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:17:46)   --> TIME: 2025-02-25 16:18:01 -- STEP: 29/72 -- GLOBAL_STEP: 1056550 | > loss_disc: 2.661020040512085 (2.6414334280737517) | > loss_disc_real_0: 0.15793271362781525 (0.12431378585511241) | > loss_disc_real_1: 0.23109203577041626 (0.22153099497844433) | > loss_disc_real_2: 0.23067744076251984 (0.2129730430656466) | > loss_disc_real_3: 0.21973562240600586 (0.22480722645233417) | > loss_disc_real_4: 0.22984910011291504 (0.2359598237892677) | > loss_disc_real_5: 0.261048823595047 (0.24562398524119936) | > loss_0: 2.661020040512085 (2.6414334280737517) | > grad_norm_0: tensor(98.6238, device='cuda:0') (tensor(153.5126, device='cuda:0')) | > loss_gen: 1.8555835485458374 (2.0396089183873145) | > loss_kl: 2.0058670043945312 (1.9999437167726715) | > loss_feat: 3.141650915145874 (3.5803568527616303) | > loss_mel: 17.58212661743164 (17.102569448536844) | > loss_duration: 1.5349206924438477 (1.5213423638508237) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.120147705078125 (26.243821440071894) | > grad_norm_1: tensor(3064.8972, device='cuda:0') (tensor(7740.1392, device='cuda:0')) | > current_lr_0: 0.00018130600393295875 | > current_lr_1: 0.00018130600393295875 | > step_time: 0.4448 (0.4411291007337899) | > loader_time: 0.0076 (0.006979523033931337) --> EVAL PERFORMANCE | > avg_loader_time: 0.008543320105109414 (-0.00019841126992669114) | > avg_step_time: 0.4573374264676806 (+0.0002444361297177511) | > avg_loss_disc: 2.7137774783121036 (+0.025954337187216847) | > avg_loss_disc_real_0: 0.14204299654549282 (-0.0063725422910401175) | > avg_loss_disc_real_1: 0.2177428431074384 (-0.013468880783504167) | > avg_loss_disc_real_2: 0.23544434751843063 (-0.004756241722006199) | > avg_loss_disc_real_3: 0.22268525458557506 (-0.00975071502403474) | > avg_loss_disc_real_4: 0.23300626307306155 (-0.011995628476142883) | > avg_loss_disc_real_5: 0.24054649136435818 (-0.005840266254586224) | > avg_loss_0: 2.7137774783121036 (+0.025954337187216847) | > avg_grad_norm_0: tensor(175.9142, device='cuda:0') (tensor(-11.2813, device='cuda:0')) | > avg_loss_gen: 1.9795374500919392 (-0.06949622026631652) | > avg_loss_kl: 2.2504551477835206 (+0.10054547182271101) | > avg_loss_feat: 3.41716403692541 (-0.050403390132205406) | > avg_loss_mel: 17.593568049686056 (+0.23802750547167406) | > avg_loss_duration: 1.645260836037112 (+0.07492379235549707) | > avg_amp_scaler: 11.267605633802814 (+3.2676056338028143) | > avg_loss_1: 26.885985441610845 (+0.29359712735028154) | > avg_grad_norm_1: tensor(6777.0977, device='cuda:0') (+tensor(614.7261, device='cuda:0'))  > EPOCH: 786/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:18:23)   --> TIME: 2025-02-25 16:18:28 -- STEP: 7/72 -- GLOBAL_STEP: 1056600 | > loss_disc: 2.690171480178833 (2.550546850476946) | > loss_disc_real_0: 0.21746563911437988 (0.10995192559702056) | > loss_disc_real_1: 0.20657199621200562 (0.22490874571459635) | > loss_disc_real_2: 0.22879911959171295 (0.24773132801055908) | > loss_disc_real_3: 0.3978021740913391 (0.2515732113804136) | > loss_disc_real_4: 0.24389195442199707 (0.23419724617685592) | > loss_disc_real_5: 0.23567882180213928 (0.23071060648986272) | > loss_0: 2.690171480178833 (2.550546850476946) | > grad_norm_0: tensor(342.3181, device='cuda:0') (tensor(155.5385, device='cuda:0')) | > loss_gen: 2.375180721282959 (2.260185411998204) | > loss_kl: 2.2558302879333496 (2.150867853845869) | > loss_feat: 4.055912971496582 (3.950701747621809) | > loss_mel: 19.166912078857422 (18.047360011509486) | > loss_duration: 1.6179730892181396 (1.6676882335117884) | > amp_scaler: 16.0 (16.0) | > loss_1: 29.47180938720703 (28.076803752354213) | > grad_norm_1: tensor(3650.2905, device='cuda:0') (tensor(4714.7002, device='cuda:0')) | > current_lr_0: 0.00018128334068246712 | > current_lr_1: 0.00018128334068246712 | > step_time: 0.438 (0.432363714490618) | > loader_time: 0.0064 (0.006725890295846122)  --> TIME: 2025-02-25 16:18:51 -- STEP: 57/72 -- GLOBAL_STEP: 1056650 | > loss_disc: 2.562114715576172 (2.6647802403098657) | > loss_disc_real_0: 0.13192331790924072 (0.11644068458362629) | > loss_disc_real_1: 0.22101807594299316 (0.22768529256184897) | > loss_disc_real_2: 0.2191019356250763 (0.2332770298970373) | > loss_disc_real_3: 0.23486213386058807 (0.25562844360083864) | > loss_disc_real_4: 0.21560874581336975 (0.24349914255895114) | > loss_disc_real_5: 0.23380443453788757 (0.24179869027514206) | > loss_0: 2.562114715576172 (2.6647802403098657) | > grad_norm_0: tensor(305.0639, device='cuda:0') (tensor(213.5855, device='cuda:0')) | > loss_gen: 1.9864751100540161 (2.0964083817967194) | > loss_kl: 2.8256149291992188 (2.3840627084698593) | > loss_feat: 3.3136932849884033 (3.6461499992169832) | > loss_mel: 21.494775772094727 (19.122353453385205) | > loss_duration: 1.6848366260528564 (1.6250838166789006) | > amp_scaler: 16.0 (16.0) | > loss_1: 31.305395126342773 (28.87405840555827) | > grad_norm_1: tensor(6191.4727, device='cuda:0') (tensor(5597.0449, device='cuda:0')) | > current_lr_0: 0.00018128334068246712 | > current_lr_1: 0.00018128334068246712 | > step_time: 0.4585 (0.4461889768901624) | > loader_time: 0.0097 (0.008111865896927683) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737486852726467 (+0.00019416674761705242) | > avg_step_time: 0.45456517918009154 (-0.002772247287589069) | > avg_loss_disc: 2.65450201907628 (-0.05927545923582356) | > avg_loss_disc_real_0: 0.12632258577455938 (-0.015720410770933446) | > avg_loss_disc_real_1: 0.22610018349869151 (+0.008357340391253104) | > avg_loss_disc_real_2: 0.23005677192983492 (-0.0053875755885957055) | > avg_loss_disc_real_3: 0.24820247397456371 (+0.025517219388988654) | > avg_loss_disc_real_4: 0.24363641545806133 (+0.010630152384999786) | > avg_loss_disc_real_5: 0.24202639347230884 (+0.0014799021079506536) | > avg_loss_0: 2.65450201907628 (-0.05927545923582356) | > avg_grad_norm_0: tensor(207.1949, device='cuda:0') (+tensor(31.2807, device='cuda:0')) | > avg_loss_gen: 2.1193203607075644 (+0.1397829106156252) | > avg_loss_kl: 2.4443361020423997 (+0.19388095425887908) | > avg_loss_feat: 3.634273394732408 (+0.21710935780699803) | > avg_loss_mel: 19.429403425942006 (+1.8358353762559503) | > avg_loss_duration: 1.630606150962937 (-0.014654685074174889) | > avg_amp_scaler: 16.0 (+4.732394366197186) | > avg_loss_1: 29.257939459572377 (+2.3719540179615315) | > avg_grad_norm_1: tensor(5860.9814, device='cuda:0') (tensor(-916.1162, device='cuda:0'))  > EPOCH: 787/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:19:00)   --> TIME: 2025-02-25 16:19:18 -- STEP: 35/72 -- GLOBAL_STEP: 1056700 | > loss_disc: 2.456301689147949 (2.6543973854609897) | > loss_disc_real_0: 0.12081689387559891 (0.13030945605465344) | > loss_disc_real_1: 0.21137073636054993 (0.23462550129209245) | > loss_disc_real_2: 0.2571380138397217 (0.23783269013677324) | > loss_disc_real_3: 0.2686021327972412 (0.23568401336669922) | > loss_disc_real_4: 0.25956085324287415 (0.2428368104355676) | > loss_disc_real_5: 0.25020280480384827 (0.23882873271192823) | > loss_0: 2.456301689147949 (2.6543973854609897) | > grad_norm_0: tensor(31.4283, device='cuda:0') (tensor(121.9840, device='cuda:0')) | > loss_gen: 2.433558464050293 (2.069776613371713) | > loss_kl: 2.3653564453125 (2.348527009146553) | > loss_feat: 3.7152585983276367 (3.4894131932939803) | > loss_mel: 17.373567581176758 (18.775065885271346) | > loss_duration: 1.6012035608291626 (1.5895405122212) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.48894500732422 (28.27232289995466) | > grad_norm_1: tensor(8695.2930, device='cuda:0') (tensor(5667.7310, device='cuda:0')) | > current_lr_0: 0.00018126068026488182 | > current_lr_1: 0.00018126068026488182 | > step_time: 0.4462 (0.4409135341644287) | > loader_time: 0.009 (0.007357515607561384) --> EVAL PERFORMANCE | > avg_loader_time: 0.008951183775780906 (+0.0002136969230544395) | > avg_step_time: 0.45647218193806394 (+0.001907002757972398) | > avg_loss_disc: 2.6280864997648856 (-0.02641551931139441) | > avg_loss_disc_real_0: 0.11380815810300934 (-0.012514427671550032) | > avg_loss_disc_real_1: 0.22867421029319226 (+0.00257402679450075) | > avg_loss_disc_real_2: 0.2379407996023205 (+0.007884027672485583) | > avg_loss_disc_real_3: 0.23884823057853 (-0.009354243396033718) | > avg_loss_disc_real_4: 0.24287490819541502 (-0.000761507262646316) | > avg_loss_disc_real_5: 0.24200793974836107 (-1.8453723947764455e-05) | > avg_loss_0: 2.6280864997648856 (-0.02641551931139441) | > avg_grad_norm_0: tensor(147.7306, device='cuda:0') (tensor(-59.4643, device='cuda:0')) | > avg_loss_gen: 2.1119898406552595 (-0.007330520052304834) | > avg_loss_kl: 2.254238793547724 (-0.19009730849467577) | > avg_loss_feat: 3.5984762655177587 (-0.03579712921464928) | > avg_loss_mel: 18.036899378601934 (-1.392504047340072) | > avg_loss_duration: 1.602165079452622 (-0.02844107151031494) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.603769141183772 (-1.654170318388605) | > avg_grad_norm_1: tensor(6039.7930, device='cuda:0') (+tensor(178.8115, device='cuda:0'))  > EPOCH: 788/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:19:37)   --> TIME: 2025-02-25 16:19:45 -- STEP: 13/72 -- GLOBAL_STEP: 1056750 | > loss_disc: 2.972008466720581 (2.6719293961158166) | > loss_disc_real_0: 0.059687431901693344 (0.04749435673539455) | > loss_disc_real_1: 0.1825326681137085 (0.20590813343341535) | > loss_disc_real_2: 0.22665245831012726 (0.20026994210023147) | > loss_disc_real_3: 0.27137497067451477 (0.2496701295559223) | > loss_disc_real_4: 0.21885758638381958 (0.22736230836464807) | > loss_disc_real_5: 0.2078479379415512 (0.2296020778325888) | > loss_0: 2.972008466720581 (2.6719293961158166) | > grad_norm_0: tensor(66.8954, device='cuda:0') (tensor(238.4597, device='cuda:0')) | > loss_gen: 1.6338951587677002 (1.8763480003063495) | > loss_kl: 1.9156807661056519 (1.9204617555324848) | > loss_feat: 3.1250197887420654 (3.6662820669320912) | > loss_mel: 17.10162925720215 (17.423386500431942) | > loss_duration: 1.467910647392273 (1.5295512676239014) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.244136810302734 (26.416030003474308) | > grad_norm_1: tensor(4025.3982, device='cuda:0') (tensor(6217.0684, device='cuda:0')) | > current_lr_0: 0.0001812380226798487 | > current_lr_1: 0.0001812380226798487 | > step_time: 0.4758 (0.4388658633598915) | > loader_time: 0.0059 (0.006406380580021785)  --> TIME: 2025-02-25 16:20:09 -- STEP: 63/72 -- GLOBAL_STEP: 1056800 | > loss_disc: 2.5927748680114746 (2.717835963718475) | > loss_disc_real_0: 0.20341122150421143 (0.13511203758655088) | > loss_disc_real_1: 0.23041772842407227 (0.24817524992284315) | > loss_disc_real_2: 0.2546924650669098 (0.24818255787804014) | > loss_disc_real_3: 0.2595450282096863 (0.23627244125282953) | > loss_disc_real_4: 0.2467895895242691 (0.2408448788854811) | > loss_disc_real_5: 0.25768816471099854 (0.23477425272502597) | > loss_0: 2.5927748680114746 (2.717835963718475) | > grad_norm_0: tensor(123.9158, device='cuda:0') (tensor(191.1237, device='cuda:0')) | > loss_gen: 2.2080330848693848 (2.0443103294523937) | > loss_kl: 2.1530401706695557 (2.0338181597845906) | > loss_feat: 3.083836555480957 (3.450464922284323) | > loss_mel: 17.457761764526367 (17.354678683810768) | > loss_duration: 1.5858538150787354 (1.5657551496747941) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.488523483276367 (26.44902716742622) | > grad_norm_1: tensor(8735.5342, device='cuda:0') (tensor(6053.7227, device='cuda:0')) | > current_lr_0: 0.0001812380226798487 | > current_lr_1: 0.0001812380226798487 | > step_time: 0.5222 (0.45144444041781956) | > loader_time: 0.0116 (0.00817176273890904) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570654291502187 (-0.0003805294842787188) | > avg_step_time: 0.4596429240535682 (+0.003170742115504277) | > avg_loss_disc: 2.7133963175222906 (+0.08530981775740498) | > avg_loss_disc_real_0: 0.13763998666594568 (+0.023831828562936336) | > avg_loss_disc_real_1: 0.2469668304416495 (+0.018292620148457228) | > avg_loss_disc_real_2: 0.24573042170262674 (+0.007789622100306232) | > avg_loss_disc_real_3: 0.23655633565405726 (-0.002291894924472737) | > avg_loss_disc_real_4: 0.24089842992769162 (-0.0019764782677234027) | > avg_loss_disc_real_5: 0.236774108569387 (-0.00523383117897408) | > avg_loss_0: 2.7133963175222906 (+0.08530981775740498) | > avg_grad_norm_0: tensor(181.5020, device='cuda:0') (+tensor(33.7713, device='cuda:0')) | > avg_loss_gen: 2.0353647739114895 (-0.07662506674377001) | > avg_loss_kl: 2.0469729060858075 (-0.20726588746191643) | > avg_loss_feat: 3.4134706644944743 (-0.18500560102328434) | > avg_loss_mel: 17.390045300335956 (-0.646854078265978) | > avg_loss_duration: 1.5734574694029042 (-0.02870761004971789) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 26.459311042033452 (-1.1444580991503202) | > avg_grad_norm_1: tensor(5917.6797, device='cuda:0') (tensor(-122.1133, device='cuda:0'))  > EPOCH: 789/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:20:15)   --> TIME: 2025-02-25 16:20:35 -- STEP: 41/72 -- GLOBAL_STEP: 1056850 | > loss_disc: 2.6219241619110107 (2.649420854521961) | > loss_disc_real_0: 0.17911872267723083 (0.12229520400486342) | > loss_disc_real_1: 0.22358429431915283 (0.2282342107557669) | > loss_disc_real_2: 0.23311074078083038 (0.23825659439331148) | > loss_disc_real_3: 0.2476968616247177 (0.23817928753248074) | > loss_disc_real_4: 0.2621847093105316 (0.23999389206490865) | > loss_disc_real_5: 0.2652205228805542 (0.24962852168373945) | > loss_0: 2.6219241619110107 (2.649420854521961) | > grad_norm_0: tensor(232.0144, device='cuda:0') (tensor(184.4471, device='cuda:0')) | > loss_gen: 2.035130500793457 (2.005553015848485) | > loss_kl: 1.9435805082321167 (2.001445200385117) | > loss_feat: 3.318326711654663 (3.3669615780434956) | > loss_mel: 17.254730224609375 (16.908698663478937) | > loss_duration: 1.551613450050354 (1.538285717731569) | > amp_scaler: 8.0 (11.707317073170731) | > loss_1: 26.103382110595703 (25.82094406500095) | > grad_norm_1: tensor(7929.2402, device='cuda:0') (tensor(5968.8545, device='cuda:0')) | > current_lr_0: 0.00018121536792701373 | > current_lr_1: 0.00018121536792701373 | > step_time: 0.4777 (0.44470956848888865) | > loader_time: 0.0092 (0.007474096809945456) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661693250629265 (+9.103895912707799e-05) | > avg_step_time: 0.4599252183672408 (+0.00028229431367260416) | > avg_loss_disc: 2.6728200778155258 (-0.04057623970676483) | > avg_loss_disc_real_0: 0.14122369156127249 (+0.003583704895326806) | > avg_loss_disc_real_1: 0.23573847723678804 (-0.011228353204861458) | > avg_loss_disc_real_2: 0.23351184771934025 (-0.01221857398328649) | > avg_loss_disc_real_3: 0.2336025590627966 (-0.0029537765912606706) | > avg_loss_disc_real_4: 0.2421721679643846 (+0.0012737380366929785) | > avg_loss_disc_real_5: 0.2448252259005963 (+0.008051117331209307) | > avg_loss_0: 2.6728200778155258 (-0.04057623970676483) | > avg_grad_norm_0: tensor(188.7263, device='cuda:0') (+tensor(7.2243, device='cuda:0')) | > avg_loss_gen: 2.012760585462543 (-0.022604188448946516) | > avg_loss_kl: 2.148599282116957 (+0.10162637603114932) | > avg_loss_feat: 3.364159513527239 (-0.049311150967235484) | > avg_loss_mel: 17.391786991710386 (+0.0017416913744305873) | > avg_loss_duration: 1.590410487752565 (+0.016953018349660898) | > avg_amp_scaler: 10.140845070422534 (-5.859154929577466) | > avg_loss_1: 26.50771678333551 (+0.04840574130205866) | > avg_grad_norm_1: tensor(6149.9502, device='cuda:0') (+tensor(232.2705, device='cuda:0'))  > EPOCH: 790/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:20:52)   --> TIME: 2025-02-25 16:21:03 -- STEP: 19/72 -- GLOBAL_STEP: 1056900 | > loss_disc: 2.546217918395996 (2.6483983742563346) | > loss_disc_real_0: 0.10815564543008804 (0.10649744225175757) | > loss_disc_real_1: 0.2373005747795105 (0.2591612127266432) | > loss_disc_real_2: 0.32010897994041443 (0.26714079631002324) | > loss_disc_real_3: 0.29320868849754333 (0.2406796130694841) | > loss_disc_real_4: 0.24113434553146362 (0.23087929816622482) | > loss_disc_real_5: 0.2519840896129608 (0.24171480458033712) | > loss_0: 2.546217918395996 (2.6483983742563346) | > grad_norm_0: tensor(71.1776, device='cuda:0') (tensor(209.3719, device='cuda:0')) | > loss_gen: 2.662142753601074 (2.1612471655795447) | > loss_kl: 2.071889638900757 (2.146218255946511) | > loss_feat: 4.063473224639893 (3.644814453626934) | > loss_mel: 17.238937377929688 (17.851201007240697) | > loss_duration: 1.5411163568496704 (1.569884268861068) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.577558517456055 (27.37336530183491) | > grad_norm_1: tensor(3912.3430, device='cuda:0') (tensor(5687.6055, device='cuda:0')) | > current_lr_0: 0.00018119271600602286 | > current_lr_1: 0.00018119271600602286 | > step_time: 0.4301 (0.43763602407355057) | > loader_time: 0.0067 (0.006766469855057566)  --> TIME: 2025-02-25 16:21:26 -- STEP: 69/72 -- GLOBAL_STEP: 1056950 | > loss_disc: 2.6128883361816406 (2.6427294931549956) | > loss_disc_real_0: 0.049547065049409866 (0.10323453926737758) | > loss_disc_real_1: 0.1587992161512375 (0.2271733618732812) | > loss_disc_real_2: 0.2764057219028473 (0.24335597001987955) | > loss_disc_real_3: 0.27777546644210815 (0.2360665813304376) | > loss_disc_real_4: 0.2747640907764435 (0.24108589257019153) | > loss_disc_real_5: 0.2821961045265198 (0.24631122618481732) | > loss_0: 2.6128883361816406 (2.6427294931549956) | > grad_norm_0: tensor(165.7518, device='cuda:0') (tensor(188.5998, device='cuda:0')) | > loss_gen: 2.042363166809082 (2.0611963773119277) | > loss_kl: 2.112767219543457 (2.086044644963913) | > loss_feat: 3.182652235031128 (3.4915810737057007) | > loss_mel: 16.88117218017578 (17.265648123146836) | > loss_duration: 1.6648354530334473 (1.5836157746936963) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.883790969848633 (26.48808609229931) | > grad_norm_1: tensor(5031.3330, device='cuda:0') (tensor(6484.8564, device='cuda:0')) | > current_lr_0: 0.00018119271600602286 | > current_lr_1: 0.00018119271600602286 | > step_time: 0.5376 (0.4555416314498238) | > loader_time: 0.0128 (0.008524048155632574) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579653753361234 (-8.203949726803134e-05) | > avg_step_time: 0.45686076056789343 (-0.0030644577993473865) | > avg_loss_disc: 2.6467555072945608 (-0.026064570520965002) | > avg_loss_disc_real_0: 0.10140518943818522 (-0.03981850212308727) | > avg_loss_disc_real_1: 0.22599243215272125 (-0.00974604508406679) | > avg_loss_disc_real_2: 0.2445792618771674 (+0.011067414157827138) | > avg_loss_disc_real_3: 0.2368421690984511 (+0.003239610035654511) | > avg_loss_disc_real_4: 0.2419927439219515 (-0.00017942404243309618) | > avg_loss_disc_real_5: 0.24703005795747462 (+0.0022048320568783253) | > avg_loss_0: 2.6467555072945608 (-0.026064570520965002) | > avg_grad_norm_0: tensor(187.1333, device='cuda:0') (tensor(-1.5929, device='cuda:0')) | > avg_loss_gen: 2.056032884288842 (+0.043272298826298794) | > avg_loss_kl: 2.085523304804949 (-0.06307597731200776) | > avg_loss_feat: 3.4815110052135627 (+0.11735149168632386) | > avg_loss_mel: 17.267054289159642 (-0.12473270255074453) | > avg_loss_duration: 1.583673593024133 (-0.006736894728432086) | > avg_amp_scaler: 8.0 (-2.140845070422534) | > avg_loss_1: 26.47379520577444 (-0.03392157756107039) | > avg_grad_norm_1: tensor(6391.0044, device='cuda:0') (+tensor(241.0542, device='cuda:0'))  > EPOCH: 791/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:21:29)   --> TIME: 2025-02-25 16:21:52 -- STEP: 47/72 -- GLOBAL_STEP: 1057000 | > loss_disc: 2.4301204681396484 (2.6665647486422923) | > loss_disc_real_0: 0.04268147796392441 (0.1686186020837185) | > loss_disc_real_1: 0.19791008532047272 (0.2357464502466486) | > loss_disc_real_2: 0.23246987164020538 (0.23632389275317497) | > loss_disc_real_3: 0.22981426119804382 (0.2350636860157581) | > loss_disc_real_4: 0.2342318445444107 (0.24386834908039012) | > loss_disc_real_5: 0.22750701010227203 (0.2483453503314485) | > loss_0: 2.4301204681396484 (2.6665647486422923) | > grad_norm_0: tensor(94.1004, device='cuda:0') (tensor(155.5715, device='cuda:0')) | > loss_gen: 2.1076431274414062 (2.0515362709126572) | > loss_kl: 2.0565731525421143 (2.1986853340838817) | > loss_feat: 4.324748992919922 (3.3032385034764067) | > loss_mel: 17.624887466430664 (18.40585903411216) | > loss_duration: 1.5883140563964844 (1.5583932374386078) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.702167510986328 (27.51771208580504) | > grad_norm_1: tensor(2480.1755, device='cuda:0') (tensor(6779.1460, device='cuda:0')) | > current_lr_0: 0.0001811700669165221 | > current_lr_1: 0.0001811700669165221 | > step_time: 0.448 (0.44007773094988883) | > loader_time: 0.0088 (0.00759824793389503) --> EVAL PERFORMANCE | > avg_loader_time: 0.008643610376707262 (+6.3956623346028e-05) | > avg_step_time: 0.4543626543501733 (-0.0024981062177201196) | > avg_loss_disc: 2.6457798178766816 (-0.0009756894178791775) | > avg_loss_disc_real_0: 0.15350106479206554 (+0.05209587535388033) | > avg_loss_disc_real_1: 0.2323644459247589 (+0.006372013772037666) | > avg_loss_disc_real_2: 0.23820486266008564 (-0.006374399217081744) | > avg_loss_disc_real_3: 0.23454199773325046 (-0.0023001713652006373) | > avg_loss_disc_real_4: 0.2455217458412681 (+0.003529001919316599) | > avg_loss_disc_real_5: 0.2490335549686996 (+0.002003497011224986) | > avg_loss_0: 2.6457798178766816 (-0.0009756894178791775) | > avg_grad_norm_0: tensor(146.4277, device='cuda:0') (tensor(-40.7056, device='cuda:0')) | > avg_loss_gen: 2.112497606747587 (+0.05646472245874534) | > avg_loss_kl: 2.1612869081362875 (+0.07576360333133847) | > avg_loss_feat: 3.3999523176273834 (-0.08155868758617935) | > avg_loss_mel: 17.982330349129683 (+0.7152760599700407) | > avg_loss_duration: 1.5760006384110785 (-0.007672954613054506) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.232067511115275 (+0.7582723053408351) | > avg_grad_norm_1: tensor(6083.2588, device='cuda:0') (tensor(-307.7456, device='cuda:0'))  > EPOCH: 792/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:22:06)   --> TIME: 2025-02-25 16:22:20 -- STEP: 25/72 -- GLOBAL_STEP: 1057050 | > loss_disc: 2.4839212894439697 (2.659869604110718) | > loss_disc_real_0: 0.0691627636551857 (0.16360061138868331) | > loss_disc_real_1: 0.19769643247127533 (0.20221169412136078) | > loss_disc_real_2: 0.2374521791934967 (0.21961891293525695) | > loss_disc_real_3: 0.20937442779541016 (0.24208412408828736) | > loss_disc_real_4: 0.23982210457324982 (0.24226803123950957) | > loss_disc_real_5: 0.24689142405986786 (0.23969611942768096) | > loss_0: 2.4839212894439697 (2.659869604110718) | > grad_norm_0: tensor(205.0917, device='cuda:0') (tensor(189.9013, device='cuda:0')) | > loss_gen: 1.94871187210083 (1.9689065217971802) | > loss_kl: 2.8279693126678467 (2.2788054752349853) | > loss_feat: 3.008085012435913 (3.3586737537384033) | > loss_mel: 21.76151466369629 (19.637068099975586) | > loss_duration: 1.6189537048339844 (1.5728277492523193) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.165233612060547 (28.816281967163086) | > grad_norm_1: tensor(10169.2080, device='cuda:0') (tensor(5611.7334, device='cuda:0')) | > current_lr_0: 0.00018114742065815752 | > current_lr_1: 0.00018114742065815752 | > step_time: 0.4396 (0.43954575538635254) | > loader_time: 0.0065 (0.007371730804443359) --> EVAL PERFORMANCE | > avg_loader_time: 0.008797138509616045 (+0.00015352813290878294) | > avg_step_time: 0.457717411954638 (+0.0033547576044646865) | > avg_loss_disc: 2.5810234110120316 (-0.06475640686465) | > avg_loss_disc_real_0: 0.12156100927943915 (-0.031940055512626395) | > avg_loss_disc_real_1: 0.21133211989637832 (-0.021032326028380594) | > avg_loss_disc_real_2: 0.22796314849819935 (-0.010241714161886295) | > avg_loss_disc_real_3: 0.23298179788488738 (-0.0015601998483630863) | > avg_loss_disc_real_4: 0.2425042407613405 (-0.003017505079927596) | > avg_loss_disc_real_5: 0.2413754943810718 (-0.007658060587627802) | > avg_loss_0: 2.5810234110120316 (-0.06475640686465) | > avg_grad_norm_0: tensor(181.2183, device='cuda:0') (+tensor(34.7906, device='cuda:0')) | > avg_loss_gen: 2.078378138407856 (-0.03411946833973101) | > avg_loss_kl: 2.373752506685929 (+0.2124655985496413) | > avg_loss_feat: 3.560836436043323 (+0.16088411841593953) | > avg_loss_mel: 19.193894641500123 (+1.21156429237044) | > avg_loss_duration: 1.6114803838058256 (+0.035479745394747075) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.81834218199824 (+1.5862746708829647) | > avg_grad_norm_1: tensor(7056.4907, device='cuda:0') (+tensor(973.2319, device='cuda:0'))  > EPOCH: 793/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:22:44)   --> TIME: 2025-02-25 16:22:47 -- STEP: 3/72 -- GLOBAL_STEP: 1057100 | > loss_disc: 2.6655585765838623 (2.602456569671631) | > loss_disc_real_0: 0.1045089140534401 (0.11771298944950104) | > loss_disc_real_1: 0.23844239115715027 (0.21535993119080862) | > loss_disc_real_2: 0.2406724989414215 (0.24673745036125183) | > loss_disc_real_3: 0.23089814186096191 (0.22726001838843027) | > loss_disc_real_4: 0.23969680070877075 (0.23273973166942596) | > loss_disc_real_5: 0.24717295169830322 (0.24555950363477072) | > loss_0: 2.6655585765838623 (2.602456569671631) | > grad_norm_0: tensor(255.3195, device='cuda:0') (tensor(278.5001, device='cuda:0')) | > loss_gen: 2.063906669616699 (2.1434064706166587) | > loss_kl: 2.052882671356201 (2.130628744761149) | > loss_feat: 3.7785000801086426 (3.8447910944620767) | > loss_mel: 17.21992301940918 (17.59889284769694) | > loss_duration: 1.5964319705963135 (1.5754610300064087) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.71164321899414 (27.293179829915363) | > grad_norm_1: tensor(7298.8643, device='cuda:0') (tensor(6690.3672, device='cuda:0')) | > current_lr_0: 0.00018112477723057524 | > current_lr_1: 0.00018112477723057524 | > step_time: 0.4469 (0.44513551394144696) | > loader_time: 0.0058 (0.006880044937133789)  --> TIME: 2025-02-25 16:23:10 -- STEP: 53/72 -- GLOBAL_STEP: 1057150 | > loss_disc: 2.4017319679260254 (2.5998553959828503) | > loss_disc_real_0: 0.07259048521518707 (0.11179254963150564) | > loss_disc_real_1: 0.18995577096939087 (0.21405794654252394) | > loss_disc_real_2: 0.17187312245368958 (0.22623367973093716) | > loss_disc_real_3: 0.1730593740940094 (0.22411315789762534) | > loss_disc_real_4: 0.21093350648880005 (0.23367402098088894) | > loss_disc_real_5: 0.23458509147167206 (0.24219495957752443) | > loss_0: 2.4017319679260254 (2.5998553959828503) | > grad_norm_0: tensor(241.8584, device='cuda:0') (tensor(269.4487, device='cuda:0')) | > loss_gen: 2.0202150344848633 (2.0761070116510933) | > loss_kl: 2.04050350189209 (2.0903810735018733) | > loss_feat: 3.798572063446045 (3.7279731372617326) | > loss_mel: 17.0246524810791 (17.221851204926125) | > loss_duration: 1.6131138801574707 (1.5618897474037026) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.497055053710938 (26.67820217924297) | > grad_norm_1: tensor(10718.4854, device='cuda:0') (tensor(9816.9590, device='cuda:0')) | > current_lr_0: 0.00018112477723057524 | > current_lr_1: 0.00018112477723057524 | > step_time: 0.4583 (0.4441810868821054) | > loader_time: 0.0095 (0.007795729727115271) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570341996743643 (-0.00022679651287240213) | > avg_step_time: 0.45683542103834557 (-0.00088199091629243) | > avg_loss_disc: 2.594462770811269 (+0.013439359799237494) | > avg_loss_disc_real_0: 0.11260877240081908 (-0.008952236878620068) | > avg_loss_disc_real_1: 0.21657667025713853 (+0.005244550360760214) | > avg_loss_disc_real_2: 0.2262814527666065 (-0.0016816957315928527) | > avg_loss_disc_real_3: 0.2260466755275995 (-0.006935122357287882) | > avg_loss_disc_real_4: 0.234188662658275 (-0.00831557810306549) | > avg_loss_disc_real_5: 0.24355043219008915 (+0.002174937809017341) | > avg_loss_0: 2.594462770811269 (+0.013439359799237494) | > avg_grad_norm_0: tensor(261.6179, device='cuda:0') (+tensor(80.3996, device='cuda:0')) | > avg_loss_gen: 2.093403181559603 (+0.015025043151746686) | > avg_loss_kl: 2.105613429781416 (-0.26813907690451266) | > avg_loss_feat: 3.6810262270376715 (+0.12018979099434857) | > avg_loss_mel: 17.132198911317655 (-2.0616957301824677) | > avg_loss_duration: 1.5750002827442868 (-0.03648010106153876) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.587242099600772 (-2.231100082397468) | > avg_grad_norm_1: tensor(9421.9580, device='cuda:0') (+tensor(2365.4673, device='cuda:0'))  > EPOCH: 794/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:23:21)   --> TIME: 2025-02-25 16:23:37 -- STEP: 31/72 -- GLOBAL_STEP: 1057200 | > loss_disc: 2.551021099090576 (2.592055789885982) | > loss_disc_real_0: 0.06183382123708725 (0.1457409904368462) | > loss_disc_real_1: 0.2561219334602356 (0.2260164531969255) | > loss_disc_real_2: 0.21600869297981262 (0.22785934325187437) | > loss_disc_real_3: 0.227784663438797 (0.22859700120264484) | > loss_disc_real_4: 0.23806467652320862 (0.2369787702637334) | > loss_disc_real_5: 0.25080084800720215 (0.24625735465557344) | > loss_0: 2.551021099090576 (2.592055789885982) | > grad_norm_0: tensor(145.4190, device='cuda:0') (tensor(201.5762, device='cuda:0')) | > loss_gen: 2.145293951034546 (2.225504090709071) | > loss_kl: 2.084019184112549 (1.9544216894334363) | > loss_feat: 3.834261178970337 (3.7603472971147105) | > loss_mel: 16.44606590270996 (16.955030718157374) | > loss_duration: 1.559255599975586 (1.5261213625631025) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.06889533996582 (26.421424927250033) | > grad_norm_1: tensor(5675.9336, device='cuda:0') (tensor(7852.4824, device='cuda:0')) | > current_lr_0: 0.00018110213663342142 | > current_lr_1: 0.00018110213663342142 | > step_time: 0.4447 (0.43983537920059695) | > loader_time: 0.0088 (0.007016704928490424) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613452105454996 (+4.311010871135314e-05) | > avg_step_time: 0.4553375009080054 (-0.001497920130340169) | > avg_loss_disc: 2.592235628987702 (-0.0022271418235670914) | > avg_loss_disc_real_0: 0.1272314724997735 (+0.01462270009895443) | > avg_loss_disc_real_1: 0.22473603130226405 (+0.008159361045125518) | > avg_loss_disc_real_2: 0.229620453967175 (+0.0033390012005685143) | > avg_loss_disc_real_3: 0.2318511017611329 (+0.005804426233533416) | > avg_loss_disc_real_4: 0.2384337583478068 (+0.004245095689531797) | > avg_loss_disc_real_5: 0.24370716680103624 (+0.00015673461094709018) | > avg_loss_0: 2.592235628987702 (-0.0022271418235670914) | > avg_grad_norm_0: tensor(233.7669, device='cuda:0') (tensor(-27.8510, device='cuda:0')) | > avg_loss_gen: 2.1477568989068696 (+0.054353717347266794) | > avg_loss_kl: 2.0177434615685907 (-0.08786996821282544) | > avg_loss_feat: 3.65497325171887 (-0.02605297531880124) | > avg_loss_mel: 16.98561989421575 (-0.14657901710190657) | > avg_loss_duration: 1.565203518934653 (-0.009796763809633902) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.371296869197362 (-0.21594523040340974) | > avg_grad_norm_1: tensor(8520.4492, device='cuda:0') (tensor(-901.5088, device='cuda:0'))  > EPOCH: 795/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:23:58)   --> TIME: 2025-02-25 16:24:04 -- STEP: 9/72 -- GLOBAL_STEP: 1057250 | > loss_disc: 2.6351819038391113 (2.548282888200548) | > loss_disc_real_0: 0.09338537603616714 (0.10287930402490827) | > loss_disc_real_1: 0.2425008863210678 (0.22482897341251373) | > loss_disc_real_2: 0.24495148658752441 (0.23359414438406625) | > loss_disc_real_3: 0.2751425802707672 (0.2464840594265196) | > loss_disc_real_4: 0.2502840757369995 (0.24570657312870026) | > loss_disc_real_5: 0.25234997272491455 (0.24322494864463806) | > loss_0: 2.6351819038391113 (2.548282888200548) | > grad_norm_0: tensor(240.7529, device='cuda:0') (tensor(211.3474, device='cuda:0')) | > loss_gen: 2.1358377933502197 (2.200574901368883) | > loss_kl: 1.9325828552246094 (1.9308806657791138) | > loss_feat: 3.56506085395813 (3.793853203455607) | > loss_mel: 17.060871124267578 (17.178092108832466) | > loss_duration: 1.5606727600097656 (1.530426237318251) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.255023956298828 (26.633826573689777) | > grad_norm_1: tensor(10244.3496, device='cuda:0') (tensor(9404.3799, device='cuda:0')) | > current_lr_0: 0.00018107949886634223 | > current_lr_1: 0.00018107949886634223 | > step_time: 0.4354 (0.433011531829834) | > loader_time: 0.007 (0.006638394461737739)  --> TIME: 2025-02-25 16:24:27 -- STEP: 59/72 -- GLOBAL_STEP: 1057300 | > loss_disc: 2.6205668449401855 (2.5661725108906372) | > loss_disc_real_0: 0.14071162045001984 (0.10345149677941355) | > loss_disc_real_1: 0.2196512073278427 (0.2245297535496243) | > loss_disc_real_2: 0.2292443960905075 (0.22977488424818396) | > loss_disc_real_3: 0.20966817438602448 (0.2373503481937667) | > loss_disc_real_4: 0.2350107878446579 (0.24366590355412435) | > loss_disc_real_5: 0.2559517025947571 (0.24582614206661613) | > loss_0: 2.6205668449401855 (2.5661725108906372) | > grad_norm_0: tensor(430.9386, device='cuda:0') (tensor(266.5891, device='cuda:0')) | > loss_gen: 2.0491788387298584 (2.136878126758641) | > loss_kl: 2.013145923614502 (1.9724812386399608) | > loss_feat: 3.492396831512451 (3.6801110485852777) | > loss_mel: 16.49220085144043 (16.708589828620518) | > loss_duration: 1.5947202444076538 (1.5506904165623552) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.641643524169922 (26.048750473281086) | > grad_norm_1: tensor(10826.1689, device='cuda:0') (tensor(9870.3369, device='cuda:0')) | > current_lr_0: 0.00018107949886634223 | > current_lr_1: 0.00018107949886634223 | > step_time: 0.4681 (0.4465697134955455) | > loader_time: 0.0096 (0.008076340465222378) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638637166627695 (+2.5185061172699336e-05) | > avg_step_time: 0.4556761090184601 (+0.00033860811045471895) | > avg_loss_disc: 2.5704288113285116 (-0.021806817659190347) | > avg_loss_disc_real_0: 0.10288983934991797 (-0.024341633149855546) | > avg_loss_disc_real_1: 0.22409768444551548 (-0.0006383468567485728) | > avg_loss_disc_real_2: 0.23014382787153753 (+0.000523373904362523) | > avg_loss_disc_real_3: 0.23651356176591257 (+0.004662460004779656) | > avg_loss_disc_real_4: 0.24258978811787887 (+0.004156029770072067) | > avg_loss_disc_real_5: 0.24534763689612 (+0.0016404700950837514) | > avg_loss_0: 2.5704288113285116 (-0.021806817659190347) | > avg_grad_norm_0: tensor(272.7749, device='cuda:0') (+tensor(39.0080, device='cuda:0')) | > avg_loss_gen: 2.132877714197402 (-0.014879184709467808) | > avg_loss_kl: 1.9951785802841187 (-0.022564881284472094) | > avg_loss_feat: 3.67735841912283 (+0.022385167403959905) | > avg_loss_mel: 16.76053804746816 (-0.2250818467475888) | > avg_loss_duration: 1.560745209035739 (-0.004458309898913937) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.126697889516056 (-0.24459897968130662) | > avg_grad_norm_1: tensor(10093.1523, device='cuda:0') (+tensor(1572.7031, device='cuda:0'))  > EPOCH: 796/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:24:35)   --> TIME: 2025-02-25 16:24:54 -- STEP: 37/72 -- GLOBAL_STEP: 1057350 | > loss_disc: 2.648914337158203 (2.6487927307953707) | > loss_disc_real_0: 0.0638231709599495 (0.09523159809209206) | > loss_disc_real_1: 0.1533626765012741 (0.22954449822773804) | > loss_disc_real_2: 0.18419836461544037 (0.2872670268690264) | > loss_disc_real_3: 0.3365403413772583 (0.24932333343737834) | > loss_disc_real_4: 0.284390926361084 (0.26194381552773555) | > loss_disc_real_5: 0.2542099058628082 (0.25314687152166626) | > loss_0: 2.648914337158203 (2.6487927307953707) | > grad_norm_0: tensor(410.3214, device='cuda:0') (tensor(297.9174, device='cuda:0')) | > loss_gen: 1.9315695762634277 (2.2065087137995536) | > loss_kl: 2.036057472229004 (1.9362374801893492) | > loss_feat: 3.298332452774048 (3.7865916844960807) | > loss_mel: 16.87851905822754 (16.87281474551639) | > loss_duration: 1.5686920881271362 (1.5230451371218707) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.713171005249023 (26.32519783844819) | > grad_norm_1: tensor(12137.1514, device='cuda:0') (tensor(10444.2451, device='cuda:0')) | > current_lr_0: 0.00018105686392898393 | > current_lr_1: 0.00018105686392898393 | > step_time: 0.4553 (0.44434245857032567) | > loader_time: 0.0082 (0.0073609223236908786) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086708471808635 (+3.2210014235804774e-05) | > avg_step_time: 0.4605984150523871 (+0.00492230603392696) | > avg_loss_disc: 2.6195193881719883 (+0.04909057684347662) | > avg_loss_disc_real_0: 0.10098140733972402 (-0.0019084320101939445) | > avg_loss_disc_real_1: 0.2255344141117284 (+0.0014367296662129159) | > avg_loss_disc_real_2: 0.25505732062836767 (+0.024913492756830136) | > avg_loss_disc_real_3: 0.24225169315304554 (+0.005738131387132972) | > avg_loss_disc_real_4: 0.25541980795457325 (+0.012830019836694379) | > avg_loss_disc_real_5: 0.24883653580302922 (+0.0034888989069092324) | > avg_loss_0: 2.6195193881719883 (+0.04909057684347662) | > avg_grad_norm_0: tensor(288.8206, device='cuda:0') (+tensor(16.0457, device='cuda:0')) | > avg_loss_gen: 2.1501691526090587 (+0.017291438411656923) | > avg_loss_kl: 2.0002360293562984 (+0.005057449072179754) | > avg_loss_feat: 3.6355388231680426 (-0.04181959595478757) | > avg_loss_mel: 16.702677699881537 (-0.057860347586622396) | > avg_loss_duration: 1.5573980153446467 (-0.003347193691092265) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.046019701890543 (-0.08067818762551227) | > avg_grad_norm_1: tensor(10080.8906, device='cuda:0') (tensor(-12.2617, device='cuda:0'))  > EPOCH: 797/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:25:13)   --> TIME: 2025-02-25 16:25:21 -- STEP: 15/72 -- GLOBAL_STEP: 1057400 | > loss_disc: 2.6311850547790527 (2.5726083596547444) | > loss_disc_real_0: 0.07089368999004364 (0.11269398232301077) | > loss_disc_real_1: 0.2313443124294281 (0.22836843927701314) | > loss_disc_real_2: 0.22723662853240967 (0.2249009003241857) | > loss_disc_real_3: 0.23197302222251892 (0.23669826090335847) | > loss_disc_real_4: 0.23697075247764587 (0.24378289977709452) | > loss_disc_real_5: 0.26584750413894653 (0.2489696701367696) | > loss_0: 2.6311850547790527 (2.5726083596547444) | > grad_norm_0: tensor(310.5579, device='cuda:0') (tensor(281.0268, device='cuda:0')) | > loss_gen: 2.0888471603393555 (2.1495311180750525) | > loss_kl: 1.7763733863830566 (1.906113608678182) | > loss_feat: 3.603858709335327 (3.6519511858622233) | > loss_mel: 16.71359634399414 (16.37274710337321) | > loss_duration: 1.504362940788269 (1.512659978866577) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.687036514282227 (25.59300308227539) | > grad_norm_1: tensor(12555.4385, device='cuda:0') (tensor(10162.9648, device='cuda:0')) | > current_lr_0: 0.0001810342318209928 | > current_lr_1: 0.0001810342318209928 | > step_time: 0.4366 (0.43729799588521323) | > loader_time: 0.006 (0.0066211859385172525)  --> TIME: 2025-02-25 16:25:45 -- STEP: 65/72 -- GLOBAL_STEP: 1057450 | > loss_disc: 2.523991584777832 (2.571981063255897) | > loss_disc_real_0: 0.16763798892498016 (0.10775706756573457) | > loss_disc_real_1: 0.20686323940753937 (0.22241568679992968) | > loss_disc_real_2: 0.21843326091766357 (0.2288560667863259) | > loss_disc_real_3: 0.2246973067522049 (0.23411778738865485) | > loss_disc_real_4: 0.24438068270683289 (0.24205677990729993) | > loss_disc_real_5: 0.24485473334789276 (0.24571840579693133) | > loss_0: 2.523991584777832 (2.571981063255897) | > grad_norm_0: tensor(332.3471, device='cuda:0') (tensor(296.5585, device='cuda:0')) | > loss_gen: 2.280729055404663 (2.138560384970444) | > loss_kl: 2.273144483566284 (2.0227336039909942) | > loss_feat: 3.9133269786834717 (3.6634293923011194) | > loss_mel: 17.454376220703125 (16.488862345768855) | > loss_duration: 1.6092292070388794 (1.5515565377015335) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.530807495117188 (25.8651423234206) | > grad_norm_1: tensor(8063.2319, device='cuda:0') (tensor(10518.5020, device='cuda:0')) | > current_lr_0: 0.0001810342318209928 | > current_lr_1: 0.0001810342318209928 | > step_time: 0.5436 (0.4528004903059739) | > loader_time: 0.0116 (0.008339460079486553) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614775160668602 (-5.60720201948986e-05) | > avg_step_time: 0.4574416456088214 (-0.003156769443565688) | > avg_loss_disc: 2.5717336728539264 (-0.04778571531806186) | > avg_loss_disc_real_0: 0.10760102704377242 (+0.006619619704048396) | > avg_loss_disc_real_1: 0.22145535114785317 (-0.004079062963875224) | > avg_loss_disc_real_2: 0.22878835360768815 (-0.026268967020679518) | > avg_loss_disc_real_3: 0.23379392313285613 (-0.008457770020189409) | > avg_loss_disc_real_4: 0.24171806997816328 (-0.013701737976409972) | > avg_loss_disc_real_5: 0.24543052974721075 (-0.003406006055818478) | > avg_loss_0: 2.5717336728539264 (-0.04778571531806186) | > avg_grad_norm_0: tensor(300.5750, device='cuda:0') (+tensor(11.7545, device='cuda:0')) | > avg_loss_gen: 2.1390193364989583 (-0.011149816110100463) | > avg_loss_kl: 2.034293849703291 (+0.034057820346992784) | > avg_loss_feat: 3.664473449680167 (+0.028934626512124417) | > avg_loss_mel: 16.539863855066432 (-0.16281384481510486) | > avg_loss_duration: 1.556984134123359 (-0.0004138812212877152) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.93463465193627 (-0.11138504995427212) | > avg_grad_norm_1: tensor(10550.0713, device='cuda:0') (+tensor(469.1807, device='cuda:0'))  > EPOCH: 798/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:25:50)   --> TIME: 2025-02-25 16:26:11 -- STEP: 43/72 -- GLOBAL_STEP: 1057500 | > loss_disc: 2.5250113010406494 (2.5579257233198303) | > loss_disc_real_0: 0.08135189861059189 (0.10301177203655243) | > loss_disc_real_1: 0.24652963876724243 (0.22321787268616433) | > loss_disc_real_2: 0.26920124888420105 (0.23438765593739444) | > loss_disc_real_3: 0.25640958547592163 (0.23481581204159316) | > loss_disc_real_4: 0.25766250491142273 (0.24364655204983646) | > loss_disc_real_5: 0.25192931294441223 (0.24576633302278297) | > loss_0: 2.5250113010406494 (2.5579257233198303) | > grad_norm_0: tensor(317.1182, device='cuda:0') (tensor(292.4166, device='cuda:0')) | > loss_gen: 2.135657787322998 (2.2094690522482225) | > loss_kl: 2.075601816177368 (2.0288114520006406) | > loss_feat: 3.755411148071289 (3.7995463581972344) | > loss_mel: 16.91501808166504 (16.78725222654121) | > loss_duration: 1.5624933242797852 (1.527576773665672) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.444183349609375 (26.352655721265215) | > grad_norm_1: tensor(16439.1172, device='cuda:0') (tensor(10856.0244, device='cuda:0')) | > current_lr_0: 0.00018101160254201516 | > current_lr_1: 0.00018101160254201516 | > step_time: 0.4499 (0.4409759266431941) | > loader_time: 0.0098 (0.007445967474649119) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655383553303465 (+4.060839263486313e-05) | > avg_step_time: 0.4538156045994288 (-0.0036260410093925666) | > avg_loss_disc: 2.5613495631956718 (-0.010384109658254648) | > avg_loss_disc_real_0: 0.10239868668813101 (-0.005202340355641405) | > avg_loss_disc_real_1: 0.22169126486274557 (+0.00023591371489239954) | > avg_loss_disc_real_2: 0.23105497763190472 (+0.002266624024216568) | > avg_loss_disc_real_3: 0.23631852361517894 (+0.0025246004823228085) | > avg_loss_disc_real_4: 0.24303826060093625 (+0.001320190622772971) | > avg_loss_disc_real_5: 0.24546269883572217 (+3.2169088511418975e-05) | > avg_loss_0: 2.5613495631956718 (-0.010384109658254648) | > avg_grad_norm_0: tensor(290.3427, device='cuda:0') (tensor(-10.2323, device='cuda:0')) | > avg_loss_gen: 2.1984399053412425 (+0.059420568842284194) | > avg_loss_kl: 2.076936124076306 (+0.04264227437301482) | > avg_loss_feat: 3.79382475329117 (+0.12935130361100322) | > avg_loss_mel: 16.797986138034876 (+0.25812228296844353) | > avg_loss_duration: 1.5536126687493124 (-0.003371465374046645) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.420799497147684 (+0.48616484521141246) | > avg_grad_norm_1: tensor(10834.7529, device='cuda:0') (+tensor(284.6816, device='cuda:0'))  > EPOCH: 799/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:26:27)   --> TIME: 2025-02-25 16:26:39 -- STEP: 21/72 -- GLOBAL_STEP: 1057550 | > loss_disc: 2.70141863822937 (2.5768063749585837) | > loss_disc_real_0: 0.09734652936458588 (0.0909898593312218) | > loss_disc_real_1: 0.21820741891860962 (0.2246649676845187) | > loss_disc_real_2: 0.21737369894981384 (0.22474367490836553) | > loss_disc_real_3: 0.2315521389245987 (0.23371283355213346) | > loss_disc_real_4: 0.2398594319820404 (0.24367026558944158) | > loss_disc_real_5: 0.23322416841983795 (0.24534699675582705) | > loss_0: 2.70141863822937 (2.5768063749585837) | > grad_norm_0: tensor(423.7137, device='cuda:0') (tensor(304.1013, device='cuda:0')) | > loss_gen: 2.090545892715454 (2.1761204231353033) | > loss_kl: 1.9937013387680054 (1.973321801140195) | > loss_feat: 3.6858646869659424 (3.8811460676647367) | > loss_mel: 15.95817756652832 (16.83725624992734) | > loss_duration: 1.5190463066101074 (1.5077303818293981) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.24733543395996 (26.375575111025857) | > grad_norm_1: tensor(13653.2197, device='cuda:0') (tensor(11477.3779, device='cuda:0')) | > current_lr_0: 0.00018098897609169741 | > current_lr_1: 0.00018098897609169741 | > step_time: 0.4686 (0.46214847337631954) | > loader_time: 0.0063 (0.006913730076381138)  --> TIME: 2025-02-25 16:27:03 -- STEP: 71/72 -- GLOBAL_STEP: 1057600 | > loss_disc: 2.7520220279693604 (2.5815794199285373) | > loss_disc_real_0: 0.1957601010799408 (0.10447549819946289) | > loss_disc_real_1: 0.14900259673595428 (0.22940485683125508) | > loss_disc_real_2: 0.28872445225715637 (0.22839648119160827) | > loss_disc_real_3: 0.24403570592403412 (0.23457611614549664) | > loss_disc_real_4: 0.2663107216358185 (0.24307140686982115) | > loss_disc_real_5: 0.24549120664596558 (0.24344337574193176) | > loss_0: 2.7520220279693604 (2.5815794199285373) | > grad_norm_0: tensor(454.7484, device='cuda:0') (tensor(287.8569, device='cuda:0')) | > loss_gen: 2.3148622512817383 (2.1856057862160894) | > loss_kl: 1.9717820882797241 (2.0784201487688967) | > loss_feat: 3.615008592605591 (3.820447048670809) | > loss_mel: 16.389421463012695 (16.693297466761628) | > loss_duration: 1.560448408126831 (1.5550684475562941) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.851520538330078 (26.33283883752957) | > grad_norm_1: tensor(9144.1387, device='cuda:0') (tensor(11023.2842, device='cuda:0')) | > current_lr_0: 0.00018098897609169741 | > current_lr_1: 0.00018098897609169741 | > step_time: 0.4871 (0.46643758491731024) | > loader_time: 0.008 (0.008688547241855676) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688547241855676 (+3.3163688552211024e-05) | > avg_step_time: 0.46643758491731024 (+0.01262198031788142) | > avg_loss_disc: 2.5815794199285373 (+0.020229856732865592) | > avg_loss_disc_real_0: 0.10447549819946289 (+0.0020768115113318775) | > avg_loss_disc_real_1: 0.22940485683125508 (+0.0077135919685095145) | > avg_loss_disc_real_2: 0.22839648119160827 (-0.0026584964402964484) | > avg_loss_disc_real_3: 0.23457611614549664 (-0.0017424074696822944) | > avg_loss_disc_real_4: 0.24307140686982115 (+3.3146268884898245e-05) | > avg_loss_disc_real_5: 0.24344337574193176 (-0.0020193230937904094) | > avg_loss_0: 2.5815794199285373 (+0.020229856732865592) | > avg_grad_norm_0: tensor(287.8569, device='cuda:0') (tensor(-2.4858, device='cuda:0')) | > avg_loss_gen: 2.1856057862160894 (-0.012834119125153087) | > avg_loss_kl: 2.0784201487688967 (+0.0014840246925906797) | > avg_loss_feat: 3.820447048670809 (+0.026622295379638672) | > avg_loss_mel: 16.693297466761628 (-0.10468867127324799) | > avg_loss_duration: 1.5550684475562941 (+0.0014557788069817779) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.33283883752957 (-0.08796065961811195) | > avg_grad_norm_1: tensor(11023.2842, device='cuda:0') (+tensor(188.5312, device='cuda:0'))  > EPOCH: 800/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:27:05)   --> TIME: 2025-02-25 16:27:29 -- STEP: 49/72 -- GLOBAL_STEP: 1057650 | > loss_disc: 2.562659978866577 (2.5699257315421584) | > loss_disc_real_0: 0.15345925092697144 (0.0996951808278658) | > loss_disc_real_1: 0.18311187624931335 (0.2224615316609947) | > loss_disc_real_2: 0.22876706719398499 (0.22774710399763926) | > loss_disc_real_3: 0.22058585286140442 (0.23234692033456297) | > loss_disc_real_4: 0.25489720702171326 (0.24107990337877858) | > loss_disc_real_5: 0.2615452706813812 (0.24573154777896647) | > loss_0: 2.562659978866577 (2.5699257315421584) | > grad_norm_0: tensor(312.2997, device='cuda:0') (tensor(261.9031, device='cuda:0')) | > loss_gen: 2.1540818214416504 (2.1814523959646426) | > loss_kl: 2.0417566299438477 (2.0743793760027205) | > loss_feat: 3.2192230224609375 (3.815315241716346) | > loss_mel: 16.86471176147461 (16.8336806394616) | > loss_duration: 1.615827202796936 (1.5383563455270262) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.895599365234375 (26.443183976776744) | > grad_norm_1: tensor(8247.5508, device='cuda:0') (tensor(10447.1787, device='cuda:0')) | > current_lr_0: 0.00018096635246968594 | > current_lr_1: 0.00018096635246968594 | > step_time: 0.4479 (0.4463462975560402) | > loader_time: 0.0093 (0.007820406738592654) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087154079490984 (+2.6860707242724532e-05) | > avg_step_time: 0.4575273856310777 (-0.008910199286232545) | > avg_loss_disc: 2.5765234349479136 (-0.005055984980623762) | > avg_loss_disc_real_0: 0.10562338171080804 (+0.001147883511345149) | > avg_loss_disc_real_1: 0.2242158013330379 (-0.005189055498217188) | > avg_loss_disc_real_2: 0.22887384534721644 (+0.00047736415560817314) | > avg_loss_disc_real_3: 0.23509010210843154 (+0.0005139859629348931) | > avg_loss_disc_real_4: 0.24339855116018108 (+0.000327144290359932) | > avg_loss_disc_real_5: 0.2464393566192036 (+0.0029959808772718477) | > avg_loss_0: 2.5765234349479136 (-0.005055984980623762) | > avg_grad_norm_0: tensor(265.3904, device='cuda:0') (tensor(-22.4665, device='cuda:0')) | > avg_loss_gen: 2.1824722105348617 (-0.0031335756812276294) | > avg_loss_kl: 2.106023073196412 (+0.02760292442751533) | > avg_loss_feat: 3.786363742720913 (-0.03408330594989595) | > avg_loss_mel: 16.88894000523527 (+0.195642538473642) | > avg_loss_duration: 1.557265837427596 (+0.0021973898713019135) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.521064919485173 (+0.18822608195560164) | > avg_grad_norm_1: tensor(10471.0557, device='cuda:0') (tensor(-552.2285, device='cuda:0'))  > EPOCH: 801/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:27:42)   --> TIME: 2025-02-25 16:27:56 -- STEP: 27/72 -- GLOBAL_STEP: 1057700 | > loss_disc: 2.702528953552246 (2.6374066052613436) | > loss_disc_real_0: 0.17659205198287964 (0.10489540671308835) | > loss_disc_real_1: 0.2599179744720459 (0.23048999353691382) | > loss_disc_real_2: 0.14742828905582428 (0.24026026328404745) | > loss_disc_real_3: 0.23218891024589539 (0.23138043781121573) | > loss_disc_real_4: 0.20636466145515442 (0.23811843549763714) | > loss_disc_real_5: 0.238404780626297 (0.24006636882269825) | > loss_0: 2.702528953552246 (2.6374066052613436) | > grad_norm_0: tensor(209.9772, device='cuda:0') (tensor(273.2627, device='cuda:0')) | > loss_gen: 2.1457924842834473 (2.1331388817893138) | > loss_kl: 2.22802472114563 (2.046562636340106) | > loss_feat: 3.5286128520965576 (3.7494009335835776) | > loss_mel: 16.446924209594727 (16.89367824130588) | > loss_duration: 1.5161521434783936 (1.5137736532423232) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.86550521850586 (26.33655428003382) | > grad_norm_1: tensor(7961.8374, device='cuda:0') (tensor(10465.2041, device='cuda:0')) | > current_lr_0: 0.0001809437316756272 | > current_lr_1: 0.0001809437316756272 | > step_time: 0.4565 (0.4370349954675745) | > loader_time: 0.0085 (0.006982344168203848) --> EVAL PERFORMANCE | > avg_loader_time: 0.008672244112256549 (-4.31638368418516e-05) | > avg_step_time: 0.4574458766990984 (-8.150893197927944e-05) | > avg_loss_disc: 2.606089900916731 (+0.02956646596881729) | > avg_loss_disc_real_0: 0.11361305956059778 (+0.00798967784978974) | > avg_loss_disc_real_1: 0.22605675374957876 (+0.0018409524165408642) | > avg_loss_disc_real_2: 0.23073867760913472 (+0.0018648322619182756) | > avg_loss_disc_real_3: 0.23277219395402451 (-0.0023179081544070224) | > avg_loss_disc_real_4: 0.24150749117555753 (-0.0018910599846235499) | > avg_loss_disc_real_5: 0.24438612624792985 (-0.002053230371273751) | > avg_loss_0: 2.606089900916731 (+0.02956646596881729) | > avg_grad_norm_0: tensor(286.3982, device='cuda:0') (+tensor(21.0078, device='cuda:0')) | > avg_loss_gen: 2.1552423927145945 (-0.027229817820267233) | > avg_loss_kl: 2.0930336243669756 (-0.012989448829436423) | > avg_loss_feat: 3.7139778473007845 (-0.07238589542012841) | > avg_loss_mel: 16.979614701069572 (+0.0906746958343021) | > avg_loss_duration: 1.5586193161950985 (+0.0013534787675024695) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.50048787828902 (-0.02057704119615167) | > avg_grad_norm_1: tensor(10352.7656, device='cuda:0') (tensor(-118.2900, device='cuda:0'))  > EPOCH: 802/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:28:19)   --> TIME: 2025-02-25 16:28:23 -- STEP: 5/72 -- GLOBAL_STEP: 1057750 | > loss_disc: 2.6692581176757812 (2.52032527923584) | > loss_disc_real_0: 0.14556467533111572 (0.11429715007543564) | > loss_disc_real_1: 0.21788327395915985 (0.2306297689676285) | > loss_disc_real_2: 0.19785137474536896 (0.21326735913753508) | > loss_disc_real_3: 0.2471301555633545 (0.2463850647211075) | > loss_disc_real_4: 0.2753749191761017 (0.25343752801418307) | > loss_disc_real_5: 0.2751050591468811 (0.2525443732738495) | > loss_0: 2.6692581176757812 (2.52032527923584) | > grad_norm_0: tensor(347.1087, device='cuda:0') (tensor(251.8379, device='cuda:0')) | > loss_gen: 2.244274854660034 (2.3449120998382567) | > loss_kl: 2.0568056106567383 (2.0190213680267335) | > loss_feat: 3.869912624359131 (4.047439289093018) | > loss_mel: 16.861602783203125 (17.17993049621582) | > loss_duration: 1.485966682434082 (1.5299071788787841) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.51856231689453 (27.1212100982666) | > grad_norm_1: tensor(11588.0010, device='cuda:0') (tensor(10920.5156, device='cuda:0')) | > current_lr_0: 0.00018092111370916774 | > current_lr_1: 0.00018092111370916774 | > step_time: 0.42 (0.4295515060424805) | > loader_time: 0.0097 (0.006632423400878907)  --> TIME: 2025-02-25 16:28:46 -- STEP: 55/72 -- GLOBAL_STEP: 1057800 | > loss_disc: 2.445038318634033 (2.5600280328230425) | > loss_disc_real_0: 0.15847241878509521 (0.10317947905172001) | > loss_disc_real_1: 0.2235725373029709 (0.22806778089566665) | > loss_disc_real_2: 0.16047732532024384 (0.22529845562848178) | > loss_disc_real_3: 0.22429651021957397 (0.23703585510904138) | > loss_disc_real_4: 0.22486816346645355 (0.2432625087824735) | > loss_disc_real_5: 0.23109018802642822 (0.24524614269083198) | > loss_0: 2.445038318634033 (2.5600280328230425) | > grad_norm_0: tensor(168.1321, device='cuda:0') (tensor(267.1356, device='cuda:0')) | > loss_gen: 2.367527723312378 (2.215783242745833) | > loss_kl: 2.199115753173828 (2.06386236927726) | > loss_feat: 3.869410514831543 (3.8723029656843706) | > loss_mel: 16.602115631103516 (16.97060893665661) | > loss_duration: 1.6069376468658447 (1.5461442383852873) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.64510726928711 (26.668701726740057) | > grad_norm_1: tensor(8836.8037, device='cuda:0') (tensor(10899.7607, device='cuda:0')) | > current_lr_0: 0.00018092111370916774 | > current_lr_1: 0.00018092111370916774 | > step_time: 0.4702 (0.44292491566051134) | > loader_time: 0.0093 (0.007846255735917526) --> EVAL PERFORMANCE | > avg_loader_time: 0.008567803342577437 (-0.00010444076967911116) | > avg_step_time: 0.4545247722679461 (-0.0029211044311523438) | > avg_loss_disc: 2.5657591450382284 (-0.04033075587850243) | > avg_loss_disc_real_0: 0.10348378347469048 (-0.010129276085907302) | > avg_loss_disc_real_1: 0.2280285205219833 (+0.001971766772404543) | > avg_loss_disc_real_2: 0.22434821640941457 (-0.006390461199720143) | > avg_loss_disc_real_3: 0.23413976271387557 (+0.0013675687598510566) | > avg_loss_disc_real_4: 0.24174451261339053 (+0.000237021437832996) | > avg_loss_disc_real_5: 0.2443951406109501 (+9.014363020254379e-06) | > avg_loss_0: 2.5657591450382284 (-0.04033075587850243) | > avg_grad_norm_0: tensor(271.5232, device='cuda:0') (tensor(-14.8750, device='cuda:0')) | > avg_loss_gen: 2.197461245765148 (+0.042218853050553395) | > avg_loss_kl: 2.08419019403592 (-0.00884343033105539) | > avg_loss_feat: 3.836218944737609 (+0.12224109743682465) | > avg_loss_mel: 16.98632786979139 (+0.006713168721816487) | > avg_loss_duration: 1.5599796604102767 (+0.0013603442151781664) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.664177854296188 (+0.1636899760071664) | > avg_grad_norm_1: tensor(10968.2705, device='cuda:0') (+tensor(615.5049, device='cuda:0'))  > EPOCH: 803/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:28:56)   --> TIME: 2025-02-25 16:29:13 -- STEP: 33/72 -- GLOBAL_STEP: 1057850 | > loss_disc: 2.663426160812378 (2.55228971712517) | > loss_disc_real_0: 0.0752045065164566 (0.09986846713405667) | > loss_disc_real_1: 0.3091307282447815 (0.21946063231338153) | > loss_disc_real_2: 0.22916226089000702 (0.23156296574708188) | > loss_disc_real_3: 0.22429139912128448 (0.23361388390714471) | > loss_disc_real_4: 0.23118261992931366 (0.24146427033525525) | > loss_disc_real_5: 0.24464677274227142 (0.2443397758585034) | > loss_0: 2.663426160812378 (2.55228971712517) | > grad_norm_0: tensor(195.7911, device='cuda:0') (tensor(245.5987, device='cuda:0')) | > loss_gen: 2.339860200881958 (2.242381709994692) | > loss_kl: 1.9352177381515503 (2.02671860926079) | > loss_feat: 3.816173791885376 (3.9868856993588535) | > loss_mel: 16.433673858642578 (17.20394799203584) | > loss_duration: 1.5323795080184937 (1.5203088666453506) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.05730628967285 (26.98024287368312) | > grad_norm_1: tensor(11423.7822, device='cuda:0') (tensor(10571.8818, device='cuda:0')) | > current_lr_0: 0.0001808984985699541 | > current_lr_1: 0.0001808984985699541 | > step_time: 0.4445 (0.4434407985571659) | > loader_time: 0.0082 (0.00730666969761704) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868955128629443 (+0.00012174794371699188) | > avg_step_time: 0.45831170888014244 (+0.0037869366121963677) | > avg_loss_disc: 2.5942762703962727 (+0.028517125358044293) | > avg_loss_disc_real_0: 0.10019835441465111 (-0.0032854290600393687) | > avg_loss_disc_real_1: 0.22905092235182373 (+0.0010224018298404247) | > avg_loss_disc_real_2: 0.24863385419610531 (+0.02428563778669074) | > avg_loss_disc_real_3: 0.2359949830971973 (+0.0018552203833217262) | > avg_loss_disc_real_4: 0.2412607598472649 (-0.0004837527661256391) | > avg_loss_disc_real_5: 0.2456294102987773 (+0.0012342696878271942) | > avg_loss_0: 2.5942762703962727 (+0.028517125358044293) | > avg_grad_norm_0: tensor(269.2736, device='cuda:0') (tensor(-2.2497, device='cuda:0')) | > avg_loss_gen: 2.201800069338838 (+0.004338823573690309) | > avg_loss_kl: 2.089467236693476 (+0.005277042657555686) | > avg_loss_feat: 3.842257281424294 (+0.006038336686684964) | > avg_loss_mel: 17.141524059671756 (+0.15519618988036754) | > avg_loss_duration: 1.5589045168648303 (-0.0010751435454463554) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.833953185820242 (+0.169775331524054) | > avg_grad_norm_1: tensor(10437.6182, device='cuda:0') (tensor(-530.6523, device='cuda:0'))  > EPOCH: 804/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:29:33)   --> TIME: 2025-02-25 16:29:40 -- STEP: 11/72 -- GLOBAL_STEP: 1057900 | > loss_disc: 2.6462900638580322 (2.5416379191658716) | > loss_disc_real_0: 0.20728355646133423 (0.099093036218123) | > loss_disc_real_1: 0.29956579208374023 (0.23155598071488467) | > loss_disc_real_2: 0.2426171451807022 (0.23194379969076676) | > loss_disc_real_3: 0.2177591323852539 (0.23111025989055634) | > loss_disc_real_4: 0.23643283545970917 (0.24637946621938187) | > loss_disc_real_5: 0.2458813190460205 (0.24379551681605252) | > loss_0: 2.6462900638580322 (2.5416379191658716) | > grad_norm_0: tensor(324.7562, device='cuda:0') (tensor(252.6395, device='cuda:0')) | > loss_gen: 2.4019129276275635 (2.248868324539878) | > loss_kl: 2.074479103088379 (1.9785670042037964) | > loss_feat: 3.7203187942504883 (3.8929365331476387) | > loss_mel: 16.635591506958008 (16.795334209095348) | > loss_duration: 1.5143662691116333 (1.51465782252225) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.346668243408203 (26.430363221602008) | > grad_norm_1: tensor(7060.0078, device='cuda:0') (tensor(9289.0596, device='cuda:0')) | > current_lr_0: 0.00018087588625763284 | > current_lr_1: 0.00018087588625763284 | > step_time: 0.4335 (0.4321165951815518) | > loader_time: 0.0071 (0.006660201332785867)  --> TIME: 2025-02-25 16:30:03 -- STEP: 61/72 -- GLOBAL_STEP: 1057950 | > loss_disc: 2.7926623821258545 (2.6183259721662178) | > loss_disc_real_0: 0.20839181542396545 (0.1014739706135187) | > loss_disc_real_1: 0.28913336992263794 (0.23077257661545864) | > loss_disc_real_2: 0.2789604663848877 (0.2423650553969086) | > loss_disc_real_3: 0.23434634506702423 (0.2365618001242153) | > loss_disc_real_4: 0.26698410511016846 (0.23978339868490814) | > loss_disc_real_5: 0.24037499725818634 (0.24133884247209206) | > loss_0: 2.7926623821258545 (2.6183259721662178) | > grad_norm_0: tensor(418.3999, device='cuda:0') (tensor(263.2258, device='cuda:0')) | > loss_gen: 2.0099403858184814 (2.160928202457116) | > loss_kl: 2.1971561908721924 (2.101575077557173) | > loss_feat: 3.0075559616088867 (3.7599942957768673) | > loss_mel: 16.80852699279785 (16.88344875710909) | > loss_duration: 1.6081171035766602 (1.549649965567667) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.631298065185547 (26.455596361003938) | > grad_norm_1: tensor(7976.5713, device='cuda:0') (tensor(9770.8281, device='cuda:0')) | > current_lr_0: 0.00018087588625763284 | > current_lr_1: 0.00018087588625763284 | > step_time: 0.4829 (0.4479094646016105) | > loader_time: 0.0111 (0.008267496452956904) --> EVAL PERFORMANCE | > avg_loader_time: 0.00873302741789482 (+4.347613160039093e-05) | > avg_step_time: 0.457109229665407 (-0.0012024792147354302) | > avg_loss_disc: 2.639905207593676 (+0.04562893719740346) | > avg_loss_disc_real_0: 0.1027690357410572 (+0.002570681326406088) | > avg_loss_disc_real_1: 0.23937531151402164 (+0.010324389162197911) | > avg_loss_disc_real_2: 0.24359523998179905 (-0.005038614214306264) | > avg_loss_disc_real_3: 0.2366592227153375 (+0.0006642396181402166) | > avg_loss_disc_real_4: 0.2411107062034204 (-0.00015005364384448883) | > avg_loss_disc_real_5: 0.24386269756605927 (-0.0017667127327180288) | > avg_loss_0: 2.639905207593676 (+0.04562893719740346) | > avg_grad_norm_0: tensor(267.1238, device='cuda:0') (tensor(-2.1497, device='cuda:0')) | > avg_loss_gen: 2.1436209141368607 (-0.05817915520197747) | > avg_loss_kl: 2.118889032954901 (+0.029421796261424937) | > avg_loss_feat: 3.6997650677049663 (-0.14249221371932785) | > avg_loss_mel: 16.908243850922922 (-0.23328020874883393) | > avg_loss_duration: 1.5589834183034763 (+7.890143864597299e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42950229913416 (-0.40445088668608165) | > avg_grad_norm_1: tensor(9617.6611, device='cuda:0') (tensor(-819.9570, device='cuda:0'))  > EPOCH: 805/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:30:10)   --> TIME: 2025-02-25 16:30:30 -- STEP: 39/72 -- GLOBAL_STEP: 1058000 | > loss_disc: 2.690779685974121 (2.6303564707438154) | > loss_disc_real_0: 0.10797896981239319 (0.11057360241046318) | > loss_disc_real_1: 0.23207272589206696 (0.23302382077926245) | > loss_disc_real_2: 0.24686171114444733 (0.24540295814856505) | > loss_disc_real_3: 0.2207595556974411 (0.233752401975485) | > loss_disc_real_4: 0.23682916164398193 (0.2408194763538165) | > loss_disc_real_5: 0.22876863181591034 (0.2416545339883902) | > loss_0: 2.690779685974121 (2.6303564707438154) | > grad_norm_0: tensor(212.3187, device='cuda:0') (tensor(299.7179, device='cuda:0')) | > loss_gen: 1.9793115854263306 (2.11320921090933) | > loss_kl: 2.141356945037842 (2.0425505546423106) | > loss_feat: 3.2359042167663574 (3.645493916976146) | > loss_mel: 15.756683349609375 (16.889141645186992) | > loss_duration: 1.5627853870391846 (1.5312019280898266) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.676042556762695 (26.221597182445038) | > grad_norm_1: tensor(8436.4775, device='cuda:0') (tensor(9762.9551, device='cuda:0')) | > current_lr_0: 0.00018085327677185063 | > current_lr_1: 0.00018085327677185063 | > step_time: 0.4624 (0.4387652140397292) | > loader_time: 0.0081 (0.007323167262933193) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638341661909936 (-9.468575598488467e-05) | > avg_step_time: 0.45522858727146204 (-0.0018806423939449757) | > avg_loss_disc: 2.611103440674258 (-0.02880176691941827) | > avg_loss_disc_real_0: 0.11002973388408273 (+0.007260698143025532) | > avg_loss_disc_real_1: 0.22639322364833994 (-0.012982087865681696) | > avg_loss_disc_real_2: 0.24086100009965225 (-0.0027342398821467995) | > avg_loss_disc_real_3: 0.2333377396136942 (-0.003321483101643319) | > avg_loss_disc_real_4: 0.2412768199410237 (+0.00016611373760330728) | > avg_loss_disc_real_5: 0.24330646618151328 (-0.0005562313845459965) | > avg_loss_0: 2.611103440674258 (-0.02880176691941827) | > avg_grad_norm_0: tensor(299.9380, device='cuda:0') (+tensor(32.8142, device='cuda:0')) | > avg_loss_gen: 2.1216006715532774 (-0.02202024258358337) | > avg_loss_kl: 2.078568589519447 (-0.040320443435454045) | > avg_loss_feat: 3.633666330659893 (-0.06609873704507319) | > avg_loss_mel: 16.880261649548167 (-0.027982201374754823) | > avg_loss_duration: 1.5622560675715056 (+0.003272649268029282) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.276353191322006 (-0.15314910781215474) | > avg_grad_norm_1: tensor(9847.3359, device='cuda:0') (+tensor(229.6748, device='cuda:0'))  > EPOCH: 806/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:30:48)   --> TIME: 2025-02-25 16:30:57 -- STEP: 17/72 -- GLOBAL_STEP: 1058050 | > loss_disc: 2.585238456726074 (2.559545839534086) | > loss_disc_real_0: 0.22766786813735962 (0.11002512254259166) | > loss_disc_real_1: 0.2620735764503479 (0.22380412063177893) | > loss_disc_real_2: 0.23433347046375275 (0.22503967583179474) | > loss_disc_real_3: 0.2648880183696747 (0.2362121475093505) | > loss_disc_real_4: 0.24811899662017822 (0.2516195528647479) | > loss_disc_real_5: 0.26046547293663025 (0.2474092630779042) | > loss_0: 2.585238456726074 (2.559545839534086) | > grad_norm_0: tensor(514.6381, device='cuda:0') (tensor(260.6920, device='cuda:0')) | > loss_gen: 2.41020131111145 (2.2232449405333576) | > loss_kl: 2.1619043350219727 (2.051772250848658) | > loss_feat: 3.610513687133789 (3.8818077199599323) | > loss_mel: 16.705913543701172 (17.272021686329563) | > loss_duration: 1.5352578163146973 (1.5131937265396118) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.423791885375977 (26.942040050730984) | > grad_norm_1: tensor(11376.4912, device='cuda:0') (tensor(9214.3525, device='cuda:0')) | > current_lr_0: 0.00018083067011225414 | > current_lr_1: 0.00018083067011225414 | > step_time: 0.4299 (0.4352787522708668) | > loader_time: 0.006 (0.006706826827105354)  --> TIME: 2025-02-25 16:31:21 -- STEP: 67/72 -- GLOBAL_STEP: 1058100 | > loss_disc: 2.658018112182617 (2.571934283669315) | > loss_disc_real_0: 0.10527261346578598 (0.10182760566917819) | > loss_disc_real_1: 0.17989975214004517 (0.22463233279648112) | > loss_disc_real_2: 0.25111815333366394 (0.23361183169172772) | > loss_disc_real_3: 0.23082715272903442 (0.2361391295692814) | > loss_disc_real_4: 0.25114038586616516 (0.24593061275446593) | > loss_disc_real_5: 0.25873228907585144 (0.2468706473930558) | > loss_0: 2.658018112182617 (2.571934283669315) | > grad_norm_0: tensor(230.1053, device='cuda:0') (tensor(272.0713, device='cuda:0')) | > loss_gen: 1.848393201828003 (2.1798506494778316) | > loss_kl: 2.176943063735962 (2.100176553228008) | > loss_feat: 3.0247302055358887 (3.809931153681741) | > loss_mel: 16.266653060913086 (16.917574170810067) | > loss_duration: 1.6536730527877808 (1.55778599141249) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.97039222717285 (26.565318463453607) | > grad_norm_1: tensor(8790.2607, device='cuda:0') (tensor(9922.9658, device='cuda:0')) | > current_lr_0: 0.00018083067011225414 | > current_lr_1: 0.00018083067011225414 | > step_time: 0.4953 (0.45029253390297963) | > loader_time: 0.0124 (0.008400611023404706) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575379008978186 (-6.296265293175007e-05) | > avg_step_time: 0.4537016875307325 (-0.0015268997407295593) | > avg_loss_disc: 2.5703746567309738 (-0.040728783943284164) | > avg_loss_disc_real_0: 0.10203520324028714 (-0.007994530643795586) | > avg_loss_disc_real_1: 0.22428666747791667 (-0.0021065561704232683) | > avg_loss_disc_real_2: 0.23385268869534345 (-0.007008311404308798) | > avg_loss_disc_real_3: 0.23574587891639118 (+0.002408139302696982) | > avg_loss_disc_real_4: 0.24531535783284147 (+0.004038537891817767) | > avg_loss_disc_real_5: 0.24681245693018738 (+0.0035059907486741093) | > avg_loss_0: 2.5703746567309738 (-0.040728783943284164) | > avg_grad_norm_0: tensor(270.5354, device='cuda:0') (tensor(-29.4026, device='cuda:0')) | > avg_loss_gen: 2.1792231176940495 (+0.057622446140772166) | > avg_loss_kl: 2.1063344025276076 (+0.027765813008160833) | > avg_loss_feat: 3.7901244667214407 (+0.15645813606154757) | > avg_loss_mel: 16.912544975818033 (+0.03228332626986585) | > avg_loss_duration: 1.5605966843349832 (-0.0016593832365223715) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.548823584972972 (+0.27247039365096626) | > avg_grad_norm_1: tensor(9882.9492, device='cuda:0') (+tensor(35.6133, device='cuda:0'))  > EPOCH: 807/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:31:25)   --> TIME: 2025-02-25 16:31:47 -- STEP: 45/72 -- GLOBAL_STEP: 1058150 | > loss_disc: 2.6605756282806396 (2.5726387129889594) | > loss_disc_real_0: 0.09219445288181305 (0.10683621217807134) | > loss_disc_real_1: 0.21637172996997833 (0.22502980695830452) | > loss_disc_real_2: 0.25832048058509827 (0.22794932789272732) | > loss_disc_real_3: 0.2804197669029236 (0.23478471338748932) | > loss_disc_real_4: 0.2821083664894104 (0.24447020722760093) | > loss_disc_real_5: 0.28004905581474304 (0.24558762047025892) | > loss_0: 2.6605756282806396 (2.5726387129889594) | > grad_norm_0: tensor(258.9399, device='cuda:0') (tensor(259.6142, device='cuda:0')) | > loss_gen: 2.1054329872131348 (2.199990741411844) | > loss_kl: 2.163877010345459 (2.080423527293735) | > loss_feat: 3.1246416568756104 (3.816801971859402) | > loss_mel: 16.455507278442383 (16.798504214816624) | > loss_duration: 1.5527125597000122 (1.5349765353732638) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.402172088623047 (26.430696868896483) | > grad_norm_1: tensor(10848.3643, device='cuda:0') (tensor(9562.2568, device='cuda:0')) | > current_lr_0: 0.0001808080662784901 | > current_lr_1: 0.0001808080662784901 | > step_time: 0.4559 (0.4407591078016493) | > loader_time: 0.0089 (0.007414144939846463) --> EVAL PERFORMANCE | > avg_loader_time: 0.008533477783203125 (-4.1901225775060544e-05) | > avg_step_time: 0.4574151811465411 (+0.0037134936158086385) | > avg_loss_disc: 2.5712456770346197 (+0.0008710203036459419) | > avg_loss_disc_real_0: 0.1048931822600499 (+0.0028579790197627575) | > avg_loss_disc_real_1: 0.22909200968037188 (+0.0048053422024552095) | > avg_loss_disc_real_2: 0.23078616446172687 (-0.0030665242336165854) | > avg_loss_disc_real_3: 0.23513842825318726 (-0.0006074506632039178) | > avg_loss_disc_real_4: 0.24293580210544694 (-0.002379555727394539) | > avg_loss_disc_real_5: 0.2454589221259238 (-0.0013535348042635897) | > avg_loss_0: 2.5712456770346197 (+0.0008710203036459419) | > avg_grad_norm_0: tensor(257.1041, device='cuda:0') (tensor(-13.4313, device='cuda:0')) | > avg_loss_gen: 2.2115964234714767 (+0.03237330577742714) | > avg_loss_kl: 2.114202522895705 (+0.007868120368097475) | > avg_loss_feat: 3.8057866667357967 (+0.01566220001435603) | > avg_loss_mel: 16.8275488195285 (-0.0849961562895345) | > avg_loss_duration: 1.5589075659362364 (-0.0016891183987468583) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.518042013678755 (-0.030781571294216548) | > avg_grad_norm_1: tensor(9571.7666, device='cuda:0') (tensor(-311.1826, device='cuda:0'))  > EPOCH: 808/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:32:02)   --> TIME: 2025-02-25 16:32:14 -- STEP: 23/72 -- GLOBAL_STEP: 1058200 | > loss_disc: 3.681523084640503 (2.830623025479524) | > loss_disc_real_0: 0.04375311732292175 (0.1975435710147671) | > loss_disc_real_1: 0.296750009059906 (0.24545152096644693) | > loss_disc_real_2: 0.5010169744491577 (0.23701618482237277) | > loss_disc_real_3: 0.2813093662261963 (0.23716350223707117) | > loss_disc_real_4: 0.2990727126598358 (0.23681004993293597) | > loss_disc_real_5: 0.29129379987716675 (0.24481216332186823) | > loss_0: 3.681523084640503 (2.830623025479524) | > grad_norm_0: tensor(194.5664, device='cuda:0') (tensor(237.0978, device='cuda:0')) | > loss_gen: 1.7978296279907227 (2.1594340490258257) | > loss_kl: 2.5805325508117676 (2.9872543086176333) | > loss_feat: 3.2234604358673096 (3.861879265826681) | > loss_mel: 18.8592472076416 (19.73012053448221) | > loss_duration: 1.6331919431686401 (1.66492323771767) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.094263076782227 (30.403611224630605) | > grad_norm_1: tensor(5065.1401, device='cuda:0') (tensor(5957.3047, device='cuda:0')) | > current_lr_0: 0.00018078546527020527 | > current_lr_1: 0.00018078546527020527 | > step_time: 0.4374 (0.44407065018363623) | > loader_time: 0.0065 (0.0070852922356646995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008779555978909342 (+0.0002460781957062171) | > avg_step_time: 0.46041784487979515 (+0.0030026637332540296) | > avg_loss_disc: 2.7923933351543586 (+0.2211476581197389) | > avg_loss_disc_real_0: 0.15894167649913843 (+0.05404849423908853) | > avg_loss_disc_real_1: 0.2507307793892605 (+0.021638769708888644) | > avg_loss_disc_real_2: 0.2432168636103751 (+0.012430699148648222) | > avg_loss_disc_real_3: 0.23613417232540293 (+0.0009957440722156707) | > avg_loss_disc_real_4: 0.2378674825312386 (-0.005068319574208335) | > avg_loss_disc_real_5: 0.24257611388891515 (-0.0028828082370086494) | > avg_loss_0: 2.7923933351543586 (+0.2211476581197389) | > avg_grad_norm_0: tensor(262.1344, device='cuda:0') (+tensor(5.0304, device='cuda:0')) | > avg_loss_gen: 2.0138723682349844 (-0.19772405523649228) | > avg_loss_kl: 2.5223176983040823 (+0.40811517540837716) | > avg_loss_feat: 3.428146308576557 (-0.3776403581592396) | > avg_loss_mel: 18.19406857289059 (+1.3665197533620912) | > avg_loss_duration: 1.6329824303237486 (+0.07407486438751221) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.791387531119334 (+1.2733455174405783) | > avg_grad_norm_1: tensor(9675.0566, device='cuda:0') (+tensor(103.2900, device='cuda:0'))  > EPOCH: 809/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:32:40)   --> TIME: 2025-02-25 16:32:42 -- STEP: 1/72 -- GLOBAL_STEP: 1058250 | > loss_disc: 2.8453168869018555 (2.8453168869018555) | > loss_disc_real_0: 0.1435849666595459 (0.1435849666595459) | > loss_disc_real_1: 0.23275521397590637 (0.23275521397590637) | > loss_disc_real_2: 0.2143321931362152 (0.2143321931362152) | > loss_disc_real_3: 0.22495673596858978 (0.22495673596858978) | > loss_disc_real_4: 0.2292127013206482 (0.2292127013206482) | > loss_disc_real_5: 0.23052062094211578 (0.23052062094211578) | > loss_0: 2.8453168869018555 (2.8453168869018555) | > grad_norm_0: tensor(339.8779, device='cuda:0') (tensor(339.8779, device='cuda:0')) | > loss_gen: 1.6373286247253418 (1.6373286247253418) | > loss_kl: 1.9722058773040771 (1.9722058773040771) | > loss_feat: 2.9560387134552 (2.9560387134552) | > loss_mel: 17.56458854675293 (17.56458854675293) | > loss_duration: 1.5742546319961548 (1.5742546319961548) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.704416275024414 (25.704416275024414) | > grad_norm_1: tensor(2407.7346, device='cuda:0') (tensor(2407.7346, device='cuda:0')) | > current_lr_0: 0.00018076286708704648 | > current_lr_1: 0.00018076286708704648 | > step_time: 0.4462 (0.44620466232299805) | > loader_time: 0.0068 (0.006821870803833008)  --> TIME: 2025-02-25 16:33:05 -- STEP: 51/72 -- GLOBAL_STEP: 1058300 | > loss_disc: 2.651526927947998 (2.7218451967426374) | > loss_disc_real_0: 0.1053980141878128 (0.19783070157555974) | > loss_disc_real_1: 0.22982044517993927 (0.22855127033065348) | > loss_disc_real_2: 0.21414582431316376 (0.232187923263101) | > loss_disc_real_3: 0.20440764725208282 (0.23487631100065567) | > loss_disc_real_4: 0.23830005526542664 (0.24518238709253423) | > loss_disc_real_5: 0.24786041676998138 (0.24413633521865397) | > loss_0: 2.651526927947998 (2.7218451967426374) | > grad_norm_0: tensor(360.4637, device='cuda:0') (tensor(272.0515, device='cuda:0')) | > loss_gen: 2.0383243560791016 (2.0730144439959055) | > loss_kl: 2.1255056858062744 (2.1668138013166534) | > loss_feat: 3.519958257675171 (3.476917584737142) | > loss_mel: 16.448875427246094 (17.509225209554042) | > loss_duration: 1.5988669395446777 (1.565259991907606) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.731531143188477 (26.791231080597523) | > grad_norm_1: tensor(9898.3984, device='cuda:0') (tensor(6472.8403, device='cuda:0')) | > current_lr_0: 0.00018076286708704648 | > current_lr_1: 0.00018076286708704648 | > step_time: 0.4492 (0.44414272027857166) | > loader_time: 0.0093 (0.0077971430385814) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863991656773527 (-0.00013963941117407265) | > avg_step_time: 0.4558726968899579 (-0.004545147989837239) | > avg_loss_disc: 2.680155025401586 (-0.11223830975277282) | > avg_loss_disc_real_0: 0.16744749284756016 (+0.008505816348421735) | > avg_loss_disc_real_1: 0.22273561022651026 (-0.027995169162750272) | > avg_loss_disc_real_2: 0.23712601974396638 (-0.006090843866408707) | > avg_loss_disc_real_3: 0.23280772063094127 (-0.003326451694461663) | > avg_loss_disc_real_4: 0.2442111077023224 (+0.006343625171083805) | > avg_loss_disc_real_5: 0.24369566604285173 (+0.0011195521539365816) | > avg_loss_0: 2.680155025401586 (-0.11223830975277282) | > avg_grad_norm_0: tensor(244.0668, device='cuda:0') (tensor(-18.0676, device='cuda:0')) | > avg_loss_gen: 2.092634639269869 (+0.07876227103488453) | > avg_loss_kl: 2.1754028881099856 (-0.34691481019409665) | > avg_loss_feat: 3.566562682809964 (+0.1384163742334068) | > avg_loss_mel: 17.41109598186655 (-0.7829725910240413) | > avg_loss_duration: 1.577321364846028 (-0.05566106547772054) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.823017711370763 (-0.9683698197485704) | > avg_grad_norm_1: tensor(6815.8335, device='cuda:0') (tensor(-2859.2231, device='cuda:0'))  > EPOCH: 810/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:33:17)   --> TIME: 2025-02-25 16:33:32 -- STEP: 29/72 -- GLOBAL_STEP: 1058350 | > loss_disc: 2.264899253845215 (2.5797279538779425) | > loss_disc_real_0: 0.14433325827121735 (0.1004183325145779) | > loss_disc_real_1: 0.19238613545894623 (0.2281765572983643) | > loss_disc_real_2: 0.22375023365020752 (0.23896718641807294) | > loss_disc_real_3: 0.2209857851266861 (0.22897004464576984) | > loss_disc_real_4: 0.2678774297237396 (0.23793836760109868) | > loss_disc_real_5: 0.24381141364574432 (0.23968509250673756) | > loss_0: 2.264899253845215 (2.5797279538779425) | > grad_norm_0: tensor(168.8204, device='cuda:0') (tensor(193.5479, device='cuda:0')) | > loss_gen: 2.8190338611602783 (2.245801300838076) | > loss_kl: 1.816531777381897 (1.9803043192830578) | > loss_feat: 4.75272274017334 (3.904062846611286) | > loss_mel: 17.045724868774414 (17.106630193776095) | > loss_duration: 1.5151498317718506 (1.5183899690364968) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.94916343688965 (26.75518877752896) | > grad_norm_1: tensor(7186.4824, device='cuda:0') (tensor(7288.9897, device='cuda:0')) | > current_lr_0: 0.0001807402717286606 | > current_lr_1: 0.0001807402717286606 | > step_time: 0.4357 (0.43801009243932265) | > loader_time: 0.0084 (0.007191879995938005) --> EVAL PERFORMANCE | > avg_loader_time: 0.008778958253457512 (+0.00013904168572224235) | > avg_step_time: 0.45775090136998137 (+0.0018782044800234599) | > avg_loss_disc: 2.5549497134248975 (-0.12520531197668827) | > avg_loss_disc_real_0: 0.0901864051661441 (-0.07726108768141607) | > avg_loss_disc_real_1: 0.22348537957164602 (+0.0007497693451357645) | > avg_loss_disc_real_2: 0.23188093052783482 (-0.005245089216131565) | > avg_loss_disc_real_3: 0.2310883965290768 (-0.0017193241018644556) | > avg_loss_disc_real_4: 0.23960767959205198 (-0.004603428110270424) | > avg_loss_disc_real_5: 0.2445096682075044 (+0.0008140021646526607) | > avg_loss_0: 2.5549497134248975 (-0.12520531197668827) | > avg_grad_norm_0: tensor(188.8809, device='cuda:0') (tensor(-55.1860, device='cuda:0')) | > avg_loss_gen: 2.2375600203661845 (+0.1449253810963156) | > avg_loss_kl: 2.0348997703740292 (-0.14050311773595636) | > avg_loss_feat: 3.9054486919456806 (+0.33888600913571665) | > avg_loss_mel: 16.965110953425025 (-0.4459850284415232) | > avg_loss_duration: 1.5579236628304065 (-0.01939770201562152) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.700943100620325 (-0.12207461075043824) | > avg_grad_norm_1: tensor(7542.6396, device='cuda:0') (+tensor(726.8062, device='cuda:0'))  > EPOCH: 811/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:33:54)   --> TIME: 2025-02-25 16:33:59 -- STEP: 7/72 -- GLOBAL_STEP: 1058400 | > loss_disc: 2.820767402648926 (2.527522393635341) | > loss_disc_real_0: 0.06298665702342987 (0.09091214835643768) | > loss_disc_real_1: 0.20522694289684296 (0.2281542888709477) | > loss_disc_real_2: 0.19188915193080902 (0.19973701664379664) | > loss_disc_real_3: 0.19519293308258057 (0.21213107023920333) | > loss_disc_real_4: 0.24130047857761383 (0.25322034103529795) | > loss_disc_real_5: 0.24587883055210114 (0.26160340862614767) | > loss_0: 2.820767402648926 (2.527522393635341) | > grad_norm_0: tensor(75.2242, device='cuda:0') (tensor(229.9820, device='cuda:0')) | > loss_gen: 1.8823214769363403 (2.260031819343567) | > loss_kl: 1.9113167524337769 (1.9612270423344202) | > loss_feat: 3.3304803371429443 (4.002931526729038) | > loss_mel: 17.437297821044922 (17.053334099905832) | > loss_duration: 1.4909296035766602 (1.5150507177625383) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.052345275878906 (26.792575563703263) | > grad_norm_1: tensor(7017.2397, device='cuda:0') (tensor(7938.0977, device='cuda:0')) | > current_lr_0: 0.00018071767919469452 | > current_lr_1: 0.00018071767919469452 | > step_time: 0.428 (0.43137063298906597) | > loader_time: 0.0054 (0.0064808641161237445)  --> TIME: 2025-02-25 16:34:22 -- STEP: 57/72 -- GLOBAL_STEP: 1058450 | > loss_disc: 2.870249032974243 (2.7180825325480678) | > loss_disc_real_0: 0.15221625566482544 (0.14739241169994338) | > loss_disc_real_1: 0.2839735150337219 (0.21422765913762545) | > loss_disc_real_2: 0.17063181102275848 (0.2357910458456006) | > loss_disc_real_3: 0.16430334746837616 (0.22973925097469697) | > loss_disc_real_4: 0.21961575746536255 (0.2515332489682915) | > loss_disc_real_5: 0.20994053781032562 (0.2559500668655363) | > loss_0: 2.870249032974243 (2.7180825325480678) | > grad_norm_0: tensor(168.6843, device='cuda:0') (tensor(218.8489, device='cuda:0')) | > loss_gen: 1.7663737535476685 (2.167535189996686) | > loss_kl: 2.2980730533599854 (2.29651381258379) | > loss_feat: 3.203428030014038 (3.733905892623098) | > loss_mel: 17.48904800415039 (18.17920868856865) | > loss_duration: 1.6095788478851318 (1.5570520183496308) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.36650276184082 (27.934215679503325) | > grad_norm_1: tensor(8985.5508, device='cuda:0') (tensor(6801.1631, device='cuda:0')) | > current_lr_0: 0.00018071767919469452 | > current_lr_1: 0.00018071767919469452 | > step_time: 0.4656 (0.4472301382767527) | > loader_time: 0.0098 (0.00797086849547269) --> EVAL PERFORMANCE | > avg_loader_time: 0.008631518189336213 (-0.00014744006412129897) | > avg_step_time: 0.45718366327420085 (-0.0005672380957805201) | > avg_loss_disc: 2.7207735088509573 (+0.16582379542605974) | > avg_loss_disc_real_0: 0.15736863358129918 (+0.06718222841515509) | > avg_loss_disc_real_1: 0.2216558790122959 (-0.0018295005593501334) | > avg_loss_disc_real_2: 0.2341701524778151 (+0.002289221949980297) | > avg_loss_disc_real_3: 0.23038823535324823 (-0.0007001611758285786) | > avg_loss_disc_real_4: 0.24921435642410317 (+0.009606676832051186) | > avg_loss_disc_real_5: 0.25258033859058165 (+0.008070670383077266) | > avg_loss_0: 2.7207735088509573 (+0.16582379542605974) | > avg_grad_norm_0: tensor(216.5926, device='cuda:0') (+tensor(27.7118, device='cuda:0')) | > avg_loss_gen: 2.13401349497513 (-0.10354652539105436) | > avg_loss_kl: 2.2891153684804144 (+0.25421559810638517) | > avg_loss_feat: 3.5828028531141682 (-0.3226458388315123) | > avg_loss_mel: 18.071658819494115 (+1.1065478660690893) | > avg_loss_duration: 1.5689489975781508 (+0.011025334747744253) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.646539553790024 (+0.9455964531696992) | > avg_grad_norm_1: tensor(6594.7632, device='cuda:0') (tensor(-947.8765, device='cuda:0'))  > EPOCH: 812/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:34:31)   --> TIME: 2025-02-25 16:34:49 -- STEP: 35/72 -- GLOBAL_STEP: 1058500 | > loss_disc: 2.3473000526428223 (2.693139975411551) | > loss_disc_real_0: 0.08451258391141891 (0.15799617224505969) | > loss_disc_real_1: 0.16808241605758667 (0.21625297112124306) | > loss_disc_real_2: 0.20039430260658264 (0.2397943215710776) | > loss_disc_real_3: 0.23455311357975006 (0.24834281631878444) | > loss_disc_real_4: 0.23611725866794586 (0.24822476548807962) | > loss_disc_real_5: 0.2638877034187317 (0.2497961563723428) | > loss_0: 2.3473000526428223 (2.693139975411551) | > grad_norm_0: tensor(120.2736, device='cuda:0') (tensor(225.7881, device='cuda:0')) | > loss_gen: 2.5084221363067627 (2.0442780426570346) | > loss_kl: 3.1430139541625977 (2.6152268409729005) | > loss_feat: 4.480650424957275 (3.405208274296352) | > loss_mel: 21.220596313476562 (19.163159724644252) | > loss_duration: 1.7107765674591064 (1.6263625247137887) | > amp_scaler: 8.0 (8.0) | > loss_1: 33.06346130371094 (28.854235567365375) | > grad_norm_1: tensor(8559.3750, device='cuda:0') (tensor(7673.3135, device='cuda:0')) | > current_lr_0: 0.00018069508948479517 | > current_lr_1: 0.00018069508948479517 | > step_time: 0.4623 (0.4570009299686977) | > loader_time: 0.0083 (0.007241814477103097) --> EVAL PERFORMANCE | > avg_loader_time: 0.008705663009428643 (+7.414482009242968e-05) | > avg_step_time: 0.46620203407717425 (+0.009018370802973408) | > avg_loss_disc: 2.596141684223229 (-0.12463182462772826) | > avg_loss_disc_real_0: 0.12205139542339546 (-0.03531723815790372) | > avg_loss_disc_real_1: 0.21905520222556424 (-0.0026006767867316483) | > avg_loss_disc_real_2: 0.23121877728213727 (-0.002951375195677841) | > avg_loss_disc_real_3: 0.23470746631353673 (+0.004319230960288495) | > avg_loss_disc_real_4: 0.24157050314923406 (-0.007643853274869106) | > avg_loss_disc_real_5: 0.24476410091762812 (-0.00781623767295353) | > avg_loss_0: 2.596141684223229 (-0.12463182462772826) | > avg_grad_norm_0: tensor(211.7754, device='cuda:0') (tensor(-4.8172, device='cuda:0')) | > avg_loss_gen: 2.1198451905183386 (-0.014168304456791514) | > avg_loss_kl: 2.511880867917773 (+0.22276549943735846) | > avg_loss_feat: 3.6420643396780523 (+0.05926148656388408) | > avg_loss_mel: 18.722407354435454 (+0.6507485349413393) | > avg_loss_duration: 1.6326191845074507 (+0.06367018692929993) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.628817088167434 (+0.9822775343774097) | > avg_grad_norm_1: tensor(8150.6348, device='cuda:0') (+tensor(1555.8716, device='cuda:0'))  > EPOCH: 813/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:35:09)   --> TIME: 2025-02-25 16:35:16 -- STEP: 13/72 -- GLOBAL_STEP: 1058550 | > loss_disc: 2.500883102416992 (2.518703295634343) | > loss_disc_real_0: 0.05549784004688263 (0.08354783975161038) | > loss_disc_real_1: 0.19574794173240662 (0.20440968183370736) | > loss_disc_real_2: 0.19320829212665558 (0.22041644155979156) | > loss_disc_real_3: 0.20754094421863556 (0.23120314226700708) | > loss_disc_real_4: 0.23738382756710052 (0.24452374646296868) | > loss_disc_real_5: 0.2443099170923233 (0.24362026957365182) | > loss_0: 2.500883102416992 (2.518703295634343) | > grad_norm_0: tensor(111.2252, device='cuda:0') (tensor(222.9707, device='cuda:0')) | > loss_gen: 2.0872559547424316 (2.1575158651058493) | > loss_kl: 1.9656342267990112 (2.04415016907912) | > loss_feat: 3.9798903465270996 (4.022131296304556) | > loss_mel: 17.404109954833984 (17.616125987126278) | > loss_duration: 1.4731194972991943 (1.537219423514146) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.910009384155273 (27.377142979548527) | > grad_norm_1: tensor(9308.6611, device='cuda:0') (tensor(9193.3975, device='cuda:0')) | > current_lr_0: 0.00018067250259860956 | > current_lr_1: 0.00018067250259860956 | > step_time: 0.4575 (0.432313478910006) | > loader_time: 0.0076 (0.006735471578744741)  --> TIME: 2025-02-25 16:35:40 -- STEP: 63/72 -- GLOBAL_STEP: 1058600 | > loss_disc: 2.6829469203948975 (2.5060999582684227) | > loss_disc_real_0: 0.08319468796253204 (0.09039633277626265) | > loss_disc_real_1: 0.19603973627090454 (0.21304626218856326) | > loss_disc_real_2: 0.23796167969703674 (0.21957708800595904) | > loss_disc_real_3: 0.2381867617368698 (0.22722364772879888) | > loss_disc_real_4: 0.24838599562644958 (0.2373138308998138) | > loss_disc_real_5: 0.23575937747955322 (0.24421203278359913) | > loss_0: 2.6829469203948975 (2.5060999582684227) | > grad_norm_0: tensor(153.0139, device='cuda:0') (tensor(224.7821, device='cuda:0')) | > loss_gen: 1.9582191705703735 (2.200156090751527) | > loss_kl: 2.067790985107422 (2.048515798553587) | > loss_feat: 3.080702781677246 (3.890591102933127) | > loss_mel: 16.591289520263672 (17.202569976685545) | > loss_duration: 1.5770573616027832 (1.5632291975475492) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.275060653686523 (26.905062115381636) | > grad_norm_1: tensor(8451.5400, device='cuda:0') (tensor(9178.2920, device='cuda:0')) | > current_lr_0: 0.00018067250259860956 | > current_lr_1: 0.00018067250259860956 | > step_time: 0.5153 (0.44672636758713496) | > loader_time: 0.0112 (0.008188141716851128) --> EVAL PERFORMANCE | > avg_loader_time: 0.008562413739486479 (-0.00014324926994216391) | > avg_step_time: 0.4541523590893813 (-0.012049674987792969) | > avg_loss_disc: 2.505294971063103 (-0.09084671316012605) | > avg_loss_disc_real_0: 0.09293071920393219 (-0.02912067621946328) | > avg_loss_disc_real_1: 0.21531531550514865 (-0.003739886720415586) | > avg_loss_disc_real_2: 0.22009461068771252 (-0.01112416659442475) | > avg_loss_disc_real_3: 0.22697675752807672 (-0.007730708785460005) | > avg_loss_disc_real_4: 0.23705881246378724 (-0.004511690685446823) | > avg_loss_disc_real_5: 0.24446854813837668 (-0.00029555277925144563) | > avg_loss_0: 2.505294971063103 (-0.09084671316012605) | > avg_grad_norm_0: tensor(225.5584, device='cuda:0') (+tensor(13.7830, device='cuda:0')) | > avg_loss_gen: 2.2054480582895413 (+0.08560286777120263) | > avg_loss_kl: 2.062330573377474 (-0.4495502945402987) | > avg_loss_feat: 3.865087206934539 (+0.2230228672564869) | > avg_loss_mel: 17.18809196310984 (-1.5343153913256131) | > avg_loss_duration: 1.5700911266703 (-0.06252805783715076) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.89104891494966 (-1.7377681732177734) | > avg_grad_norm_1: tensor(9156.6777, device='cuda:0') (+tensor(1006.0430, device='cuda:0'))  > EPOCH: 814/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:35:45)   --> TIME: 2025-02-25 16:36:06 -- STEP: 41/72 -- GLOBAL_STEP: 1058650 | > loss_disc: 2.5266566276550293 (2.548097453466276) | > loss_disc_real_0: 0.1097276359796524 (0.10295496526651265) | > loss_disc_real_1: 0.21389181911945343 (0.21809814797668922) | > loss_disc_real_2: 0.21832145750522614 (0.21864919727895318) | > loss_disc_real_3: 0.2562573254108429 (0.22957756170412388) | > loss_disc_real_4: 0.25842949748039246 (0.23969849180884478) | > loss_disc_real_5: 0.25716444849967957 (0.24369694837709752) | > loss_0: 2.5266566276550293 (2.548097453466276) | > grad_norm_0: tensor(262.3687, device='cuda:0') (tensor(241.1456, device='cuda:0')) | > loss_gen: 2.2497994899749756 (2.1897640373648666) | > loss_kl: 1.8774405717849731 (1.9406198175942027) | > loss_feat: 3.7216930389404297 (3.8684360166875327) | > loss_mel: 16.057668685913086 (16.93786134952452) | > loss_duration: 1.5448482036590576 (1.532660112148378) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.45145034790039 (26.469341464158965) | > grad_norm_1: tensor(7671.3296, device='cuda:0') (tensor(8991.5176, device='cuda:0')) | > current_lr_0: 0.00018064991853578472 | > current_lr_1: 0.00018064991853578472 | > step_time: 0.4439 (0.43921374111640743) | > loader_time: 0.009 (0.007409188805556879) --> EVAL PERFORMANCE | > avg_loader_time: 0.008645181924524441 (+8.276818503796281e-05) | > avg_step_time: 0.4550839645761839 (+0.0009316054868026002) | > avg_loss_disc: 2.55404429032769 (+0.04874931926458714) | > avg_loss_disc_real_0: 0.10657745646968693 (+0.013646737265754744) | > avg_loss_disc_real_1: 0.22192388353213457 (+0.006608568026985923) | > avg_loss_disc_real_2: 0.2199291595270936 (-0.00016545116061891796) | > avg_loss_disc_real_3: 0.23134134374034238 (+0.004364586212265653) | > avg_loss_disc_real_4: 0.2409313738765851 (+0.003872561412797848) | > avg_loss_disc_real_5: 0.2451163372943099 (+0.0006477891559332205) | > avg_loss_0: 2.55404429032769 (+0.04874931926458714) | > avg_grad_norm_0: tensor(245.1725, device='cuda:0') (+tensor(19.6141, device='cuda:0')) | > avg_loss_gen: 2.1899343040627492 (-0.01551375422679202) | > avg_loss_kl: 1.9897846487206472 (-0.07254592465682697) | > avg_loss_feat: 3.8178792100557137 (-0.04720799687882549) | > avg_loss_mel: 16.888088508391043 (-0.30000345471879797) | > avg_loss_duration: 1.5593518287363186 (-0.010739297933981318) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.445038593990702 (-0.4460103209589583) | > avg_grad_norm_1: tensor(9199.6787, device='cuda:0') (+tensor(43.0010, device='cuda:0'))  > EPOCH: 815/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:36:23)   --> TIME: 2025-02-25 16:36:33 -- STEP: 19/72 -- GLOBAL_STEP: 1058700 | > loss_disc: 2.7762134075164795 (2.5427114210630717) | > loss_disc_real_0: 0.1708155870437622 (0.09806370833202412) | > loss_disc_real_1: 0.21522705256938934 (0.21811462625076897) | > loss_disc_real_2: 0.2447008639574051 (0.2278749621228168) | > loss_disc_real_3: 0.22619423270225525 (0.22968053504040367) | > loss_disc_real_4: 0.2253369688987732 (0.23829906473034307) | > loss_disc_real_5: 0.23022949695587158 (0.24206798798159548) | > loss_0: 2.7762134075164795 (2.5427114210630717) | > grad_norm_0: tensor(467.7140, device='cuda:0') (tensor(279.4494, device='cuda:0')) | > loss_gen: 1.729487657546997 (2.166010875450937) | > loss_kl: 1.9377896785736084 (1.934187694599754) | > loss_feat: 3.2252721786499023 (3.893234980733771) | > loss_mel: 16.59930419921875 (17.11291127455862) | > loss_duration: 1.4948238134384155 (1.5116914510726929) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.986679077148438 (26.61803657130191) | > grad_norm_1: tensor(8026.7710, device='cuda:0') (tensor(10118.7695, device='cuda:0')) | > current_lr_0: 0.00018062733729596773 | > current_lr_1: 0.00018062733729596773 | > step_time: 0.4281 (0.436183352219431) | > loader_time: 0.0063 (0.006817303205791272)  --> TIME: 2025-02-25 16:36:57 -- STEP: 69/72 -- GLOBAL_STEP: 1058750 | > loss_disc: 2.9177358150482178 (2.567957577498063) | > loss_disc_real_0: 0.11220363527536392 (0.09777394136872847) | > loss_disc_real_1: 0.2610427439212799 (0.2227942349686139) | > loss_disc_real_2: 0.2207745909690857 (0.2333072687404743) | > loss_disc_real_3: 0.3025040626525879 (0.23219834700010825) | > loss_disc_real_4: 0.2982974648475647 (0.24301934177460877) | > loss_disc_real_5: 0.25273916125297546 (0.24437095505603845) | > loss_0: 2.9177358150482178 (2.567957577498063) | > grad_norm_0: tensor(144.0879, device='cuda:0') (tensor(279.1994, device='cuda:0')) | > loss_gen: 1.8857827186584473 (2.1787993717884673) | > loss_kl: 2.165738344192505 (1.9853631499884785) | > loss_feat: 2.969161033630371 (3.8487132044805996) | > loss_mel: 17.285011291503906 (16.834425691245258) | > loss_duration: 1.6503868103027344 (1.554382312125054) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.956079483032227 (26.401684000872184) | > grad_norm_1: tensor(6348.7578, device='cuda:0') (tensor(10002.0713, device='cuda:0')) | > current_lr_0: 0.00018062733729596773 | > current_lr_1: 0.00018062733729596773 | > step_time: 0.5376 (0.45641365604124207) | > loader_time: 0.0128 (0.008656754009965538) --> EVAL PERFORMANCE | > avg_loader_time: 0.008708550896443111 (+6.336897191866996e-05) | > avg_step_time: 0.4580833139553876 (+0.0029993493792037085) | > avg_loss_disc: 2.5623183183267084 (+0.00827402799901833) | > avg_loss_disc_real_0: 0.09711379148590735 (-0.009463664983779582) | > avg_loss_disc_real_1: 0.2222014050668394 (+0.00027752153470483876) | > avg_loss_disc_real_2: 0.23165570052576737 (+0.011726540998673762) | > avg_loss_disc_real_3: 0.23089719175452916 (-0.0004441519858132126) | > avg_loss_disc_real_4: 0.24291679116202072 (+0.0019854172854356367) | > avg_loss_disc_real_5: 0.24454979178771166 (-0.0005665455065982372) | > avg_loss_0: 2.5623183183267084 (+0.00827402799901833) | > avg_grad_norm_0: tensor(274.0161, device='cuda:0') (+tensor(28.8436, device='cuda:0')) | > avg_loss_gen: 2.1722784814700273 (-0.017655822592721915) | > avg_loss_kl: 1.987931293500981 (-0.0018533552196662306) | > avg_loss_feat: 3.8696944411371796 (+0.05181523108146591) | > avg_loss_mel: 16.8358483650315 (-0.05224014335954408) | > avg_loss_duration: 1.555101112580635 (-0.004250716155683509) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.420853950607945 (-0.024184643382756832) | > avg_grad_norm_1: tensor(9988.2090, device='cuda:0') (+tensor(788.5303, device='cuda:0'))  > EPOCH: 816/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:37:00)   --> TIME: 2025-02-25 16:37:23 -- STEP: 47/72 -- GLOBAL_STEP: 1058800 | > loss_disc: 2.5815374851226807 (2.5956404361319048) | > loss_disc_real_0: 0.2559034824371338 (0.11213744843893862) | > loss_disc_real_1: 0.22329941391944885 (0.22598728093695133) | > loss_disc_real_2: 0.19787119328975677 (0.2206470218110592) | > loss_disc_real_3: 0.22592422366142273 (0.22842657581922857) | > loss_disc_real_4: 0.23009473085403442 (0.23106398211514695) | > loss_disc_real_5: 0.24073562026023865 (0.24312771412920445) | > loss_0: 2.5815374851226807 (2.5956404361319048) | > grad_norm_0: tensor(429.3199, device='cuda:0') (tensor(318.3108, device='cuda:0')) | > loss_gen: 2.529268503189087 (2.1659950550566327) | > loss_kl: 2.0514323711395264 (1.9655576639987054) | > loss_feat: 4.3345746994018555 (3.8518489979683084) | > loss_mel: 16.943967819213867 (16.77783158484926) | > loss_duration: 1.5663585662841797 (1.5288271523536516) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.425601959228516 (26.290060570899477) | > grad_norm_1: tensor(7733.6533, device='cuda:0') (tensor(10808.7236, device='cuda:0')) | > current_lr_0: 0.00018060475887880572 | > current_lr_1: 0.00018060475887880572 | > step_time: 0.455 (0.4446715091137176) | > loader_time: 0.009 (0.007753884538691094) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087143770405944 (+5.82614415128857e-06) | > avg_step_time: 0.45812090014068174 (+3.7586185294147256e-05) | > avg_loss_disc: 2.5892074746145335 (+0.026889156287825067) | > avg_loss_disc_real_0: 0.11316229455487829 (+0.016048503068970946) | > avg_loss_disc_real_1: 0.22404258016129613 (+0.0018411750944567207) | > avg_loss_disc_real_2: 0.22303936368143054 (-0.008616336844336825) | > avg_loss_disc_real_3: 0.2299858454457471 (-0.0009113463087820628) | > avg_loss_disc_real_4: 0.23457833988146043 (-0.008338451280560294) | > avg_loss_disc_real_5: 0.24403833608392259 (-0.0005114557037890766) | > avg_loss_0: 2.5892074746145335 (+0.026889156287825067) | > avg_grad_norm_0: tensor(311.4132, device='cuda:0') (+tensor(37.3972, device='cuda:0')) | > avg_loss_gen: 2.171227250300662 (-0.0010512311693653409) | > avg_loss_kl: 2.003130531646836 (+0.015199238145854999) | > avg_loss_feat: 3.798562342012432 (-0.0711320991247475) | > avg_loss_mel: 16.690239180981273 (-0.14560918405022605) | > avg_loss_duration: 1.5521950419519983 (-0.002906070628636792) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.215354409016356 (-0.205499541591589) | > avg_grad_norm_1: tensor(10895.2461, device='cuda:0') (+tensor(907.0371, device='cuda:0'))  > EPOCH: 817/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:37:37)   --> TIME: 2025-02-25 16:37:50 -- STEP: 25/72 -- GLOBAL_STEP: 1058850 | > loss_disc: 3.030543327331543 (2.735077991485596) | > loss_disc_real_0: 0.1522780954837799 (0.11622722566127777) | > loss_disc_real_1: 0.2295757383108139 (0.22214993596076965) | > loss_disc_real_2: 0.400962769985199 (0.2897277706861496) | > loss_disc_real_3: 0.25389790534973145 (0.2305864667892456) | > loss_disc_real_4: 0.23797471821308136 (0.24458501100540161) | > loss_disc_real_5: 0.25739458203315735 (0.24092196583747863) | > loss_0: 3.030543327331543 (2.735077991485596) | > grad_norm_0: tensor(669.0770, device='cuda:0') (tensor(318.1719, device='cuda:0')) | > loss_gen: 1.9205191135406494 (2.0454102659225466) | > loss_kl: 1.9135836362838745 (2.001481499671936) | > loss_feat: 3.330596685409546 (3.6677088356018066) | > loss_mel: 17.042930603027344 (16.78992534637451) | > loss_duration: 1.5154078006744385 (1.5041113710403442) | > amp_scaler: 8.0 (8.960000000000003) | > loss_1: 25.723037719726562 (26.008637313842772) | > grad_norm_1: tensor(11701.4424, device='cuda:0') (tensor(9089.7021, device='cuda:0')) | > current_lr_0: 0.00018058218328394587 | > current_lr_1: 0.00018058218328394587 | > step_time: 0.4502 (0.4410190200805664) | > loader_time: 0.0077 (0.00707219123840332) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719390546771842 (+5.013506177441868e-06) | > avg_step_time: 0.4560543483411762 (-0.002066551799505545) | > avg_loss_disc: 2.710446162962578 (+0.1212386883480443) | > avg_loss_disc_real_0: 0.12584110727192652 (+0.012678812717048224) | > avg_loss_disc_real_1: 0.22733227163553238 (+0.003289691474236245) | > avg_loss_disc_real_2: 0.2628335090170444 (+0.039794145335613845) | > avg_loss_disc_real_3: 0.23559785455885068 (+0.005612009113103583) | > avg_loss_disc_real_4: 0.2432729572477475 (+0.008694617366287072) | > avg_loss_disc_real_5: 0.24130702081700445 (-0.0027313152669181384) | > avg_loss_0: 2.710446162962578 (+0.1212386883480443) | > avg_grad_norm_0: tensor(258.9481, device='cuda:0') (tensor(-52.4652, device='cuda:0')) | > avg_loss_gen: 2.075541403931631 (-0.09568584636903088) | > avg_loss_kl: 2.2044810107056514 (+0.20135047905881542) | > avg_loss_feat: 3.5841821012362627 (-0.21438024077616946) | > avg_loss_mel: 17.47491557161573 (+0.7846763906344556) | > avg_loss_duration: 1.5837922129832522 (+0.03159717103125392) | > avg_amp_scaler: 8.338028169014084 (+0.33802816901408406) | > avg_loss_1: 26.922912409607793 (+0.7075580005914368) | > avg_grad_norm_1: tensor(8329.1543, device='cuda:0') (tensor(-2566.0918, device='cuda:0'))  > EPOCH: 818/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:38:14)   --> TIME: 2025-02-25 16:38:18 -- STEP: 3/72 -- GLOBAL_STEP: 1058900 | > loss_disc: 2.8661928176879883 (2.74348521232605) | > loss_disc_real_0: 0.04244920611381531 (0.06640114138523738) | > loss_disc_real_1: 0.17614305019378662 (0.21421470244725546) | > loss_disc_real_2: 0.19451087713241577 (0.20855823655923209) | > loss_disc_real_3: 0.16156229376792908 (0.18756084144115448) | > loss_disc_real_4: 0.17495812475681305 (0.21192977329095206) | > loss_disc_real_5: 0.2020294964313507 (0.23826185862223306) | > loss_0: 2.8661928176879883 (2.74348521232605) | > grad_norm_0: tensor(302.2848, device='cuda:0') (tensor(263.9498, device='cuda:0')) | > loss_gen: 1.3891901969909668 (1.760485092798869) | > loss_kl: 2.296266555786133 (2.304691791534424) | > loss_feat: 3.4736826419830322 (3.6959304014841714) | > loss_mel: 18.32412338256836 (18.387011845906574) | > loss_duration: 1.6297292709350586 (1.5839433670043945) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.112991333007812 (27.732062021891277) | > grad_norm_1: tensor(5480.5186, device='cuda:0') (tensor(4728.9893, device='cuda:0')) | > current_lr_0: 0.00018055961051103537 | > current_lr_1: 0.00018055961051103537 | > step_time: 0.4491 (0.6016144752502441) | > loader_time: 0.0094 (0.006846427917480469)  --> TIME: 2025-02-25 16:38:41 -- STEP: 53/72 -- GLOBAL_STEP: 1058950 | > loss_disc: 2.7562918663024902 (2.66756608801068) | > loss_disc_real_0: 0.13821594417095184 (0.1108858383629682) | > loss_disc_real_1: 0.16754288971424103 (0.22335287788004246) | > loss_disc_real_2: 0.21233417093753815 (0.24328077935947565) | > loss_disc_real_3: 0.23251692950725555 (0.23043690481275883) | > loss_disc_real_4: 0.2172391712665558 (0.2372542827759149) | > loss_disc_real_5: 0.22861696779727936 (0.24212887658263152) | > loss_0: 2.7562918663024902 (2.66756608801068) | > grad_norm_0: tensor(639.0336, device='cuda:0') (tensor(252.9406, device='cuda:0')) | > loss_gen: 1.638901948928833 (1.9926661005559958) | > loss_kl: 2.0280826091766357 (2.0948441141056566) | > loss_feat: 3.524542808532715 (3.493645141709526) | > loss_mel: 17.0114803314209 (17.17780467699158) | > loss_duration: 1.5924450159072876 (1.552632322851217) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.795454025268555 (26.311592174026202) | > grad_norm_1: tensor(6194.0317, device='cuda:0') (tensor(7324.4614, device='cuda:0')) | > current_lr_0: 0.00018055961051103537 | > current_lr_1: 0.00018055961051103537 | > step_time: 0.4523 (0.4566760827910225) | > loader_time: 0.0091 (0.007843363959834261) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640292664648781 (-7.909788212306056e-05) | > avg_step_time: 0.46447568208398954 (+0.008421333742813342) | > avg_loss_disc: 2.6792469964900487 (-0.031199166472529072) | > avg_loss_disc_real_0: 0.11509152228983355 (-0.010749584982092969) | > avg_loss_disc_real_1: 0.22509266939801229 (-0.002239602237520094) | > avg_loss_disc_real_2: 0.24384708807501995 (-0.018986420942024435) | > avg_loss_disc_real_3: 0.23146837970740358 (-0.004129474851447101) | > avg_loss_disc_real_4: 0.24241241644805586 (-0.0008605407996916392) | > avg_loss_disc_real_5: 0.24642283416969676 (+0.005115813352692317) | > avg_loss_0: 2.6792469964900487 (-0.031199166472529072) | > avg_grad_norm_0: tensor(271.7182, device='cuda:0') (+tensor(12.7701, device='cuda:0')) | > avg_loss_gen: 1.9908971668968738 (-0.08464423703475732) | > avg_loss_kl: 2.1084217958047358 (-0.09605921490091562) | > avg_loss_feat: 3.4712246572467644 (-0.11295744398949825) | > avg_loss_mel: 17.07809069458867 (-0.3968248770270577) | > avg_loss_duration: 1.5664100596602535 (-0.01738215332299875) | > avg_amp_scaler: 8.0 (-0.33802816901408406) | > avg_loss_1: 26.215044182790837 (-0.7078682268169558) | > avg_grad_norm_1: tensor(7615.3403, device='cuda:0') (tensor(-713.8140, device='cuda:0'))  > EPOCH: 819/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:38:52)   --> TIME: 2025-02-25 16:39:08 -- STEP: 31/72 -- GLOBAL_STEP: 1059000 | > loss_disc: 2.8808326721191406 (2.6660817361647084) | > loss_disc_real_0: 0.0451660081744194 (0.1459043273522008) | > loss_disc_real_1: 0.13250406086444855 (0.2401221533936839) | > loss_disc_real_2: 0.26326534152030945 (0.23501382960427192) | > loss_disc_real_3: 0.24718107283115387 (0.23379229489834077) | > loss_disc_real_4: 0.25245583057403564 (0.24824966153790873) | > loss_disc_real_5: 0.242215096950531 (0.23921711598673173) | > loss_0: 2.8808326721191406 (2.6660817361647084) | > grad_norm_0: tensor(379.0872, device='cuda:0') (tensor(265.5853, device='cuda:0')) | > loss_gen: 1.5892218351364136 (2.1368247770494033) | > loss_kl: 2.021989583969116 (2.1417171186016453) | > loss_feat: 3.5133957862854004 (3.6459731055844213) | > loss_mel: 16.8591365814209 (17.37883764697659) | > loss_duration: 1.5544564723968506 (1.5649766460541756) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.53820037841797 (26.868329078920425) | > grad_norm_1: tensor(13298.3662, device='cuda:0') (tensor(8804.4727, device='cuda:0')) | > current_lr_0: 0.00018053704055972147 | > current_lr_1: 0.00018053704055972147 | > step_time: 0.4449 (0.4368404265372984) | > loader_time: 0.0083 (0.007018481531450825) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585446317430951 (-5.48463472178299e-05) | > avg_step_time: 0.45731100901751454 (-0.0071646730664750025) | > avg_loss_disc: 2.679101027233499 (-0.00014596925654952742) | > avg_loss_disc_real_0: 0.13284115955023698 (+0.017749637260403434) | > avg_loss_disc_real_1: 0.22301500332607352 (-0.0020776660719387663) | > avg_loss_disc_real_2: 0.23507012321915424 (-0.00877696485586571) | > avg_loss_disc_real_3: 0.24086848216157564 (+0.009400102454172055) | > avg_loss_disc_real_4: 0.24657758671632954 (+0.004165170268273677) | > avg_loss_disc_real_5: 0.24590121503447143 (-0.0005216191352253319) | > avg_loss_0: 2.679101027233499 (-0.00014596925654952742) | > avg_grad_norm_0: tensor(287.2069, device='cuda:0') (+tensor(15.4887, device='cuda:0')) | > avg_loss_gen: 2.093233528271528 (+0.10233636137465418) | > avg_loss_kl: 2.1075299608875335 (-0.0008918349172022388) | > avg_loss_feat: 3.5425104121087307 (+0.07128575486196631) | > avg_loss_mel: 16.953623731371376 (-0.12446696321729434) | > avg_loss_duration: 1.579780721328628 (+0.013370661668374417) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.27667830023967 (+0.061634117448832626) | > avg_grad_norm_1: tensor(9206.4463, device='cuda:0') (+tensor(1591.1060, device='cuda:0'))  > EPOCH: 820/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:39:29)   --> TIME: 2025-02-25 16:39:35 -- STEP: 9/72 -- GLOBAL_STEP: 1059050 | > loss_disc: 2.955160140991211 (2.635658105214437) | > loss_disc_real_0: 0.04414726048707962 (0.11592772023545371) | > loss_disc_real_1: 0.1969507336616516 (0.22901275753974915) | > loss_disc_real_2: 0.22526472806930542 (0.2223581936624315) | > loss_disc_real_3: 0.22051483392715454 (0.23874790635373858) | > loss_disc_real_4: 0.2559177577495575 (0.2425843427578608) | > loss_disc_real_5: 0.24551831185817719 (0.23460129565662807) | > loss_0: 2.955160140991211 (2.635658105214437) | > grad_norm_0: tensor(222.1046, device='cuda:0') (tensor(264.4835, device='cuda:0')) | > loss_gen: 1.4218722581863403 (2.006001591682434) | > loss_kl: 1.9980589151382446 (1.9259050289789836) | > loss_feat: 3.3086678981781006 (3.6029504934946694) | > loss_mel: 16.726383209228516 (16.522874620225693) | > loss_duration: 1.510590672492981 (1.5199572907553778) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.965574264526367 (25.57768948872884) | > grad_norm_1: tensor(5047.4751, device='cuda:0') (tensor(10781.3887, device='cuda:0')) | > current_lr_0: 0.0001805144734296515 | > current_lr_1: 0.0001805144734296515 | > step_time: 0.4397 (0.4284731017218696) | > loader_time: 0.0057 (0.006491210725572374)  --> TIME: 2025-02-25 16:39:58 -- STEP: 59/72 -- GLOBAL_STEP: 1059100 | > loss_disc: 2.6774063110351562 (2.7435024552426097) | > loss_disc_real_0: 0.11896031349897385 (0.14801014176869792) | > loss_disc_real_1: 0.2568562924861908 (0.232933050242521) | > loss_disc_real_2: 0.2661195695400238 (0.2415834344544653) | > loss_disc_real_3: 0.2613113522529602 (0.22912164335533725) | > loss_disc_real_4: 0.3304515480995178 (0.24541191605188079) | > loss_disc_real_5: 0.30226758122444153 (0.2403857745861603) | > loss_0: 2.6774063110351562 (2.7435024552426097) | > grad_norm_0: tensor(372.8292, device='cuda:0') (tensor(208.7176, device='cuda:0')) | > loss_gen: 2.305087089538574 (1.9987261517573212) | > loss_kl: 2.3650522232055664 (2.4855961496547123) | > loss_feat: 3.1514763832092285 (3.497837741496199) | > loss_mel: 15.961191177368164 (18.779559216256875) | > loss_duration: 1.5993566513061523 (1.6103630328582506) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.382164001464844 (28.372082403150657) | > grad_norm_1: tensor(6289.9395, device='cuda:0') (tensor(6428.7456, device='cuda:0')) | > current_lr_0: 0.0001805144734296515 | > current_lr_1: 0.0001805144734296515 | > step_time: 0.4696 (0.4440841513164973) | > loader_time: 0.0099 (0.008095510935379285) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640319528714033 (+5.487321128308173e-05) | > avg_step_time: 0.45322509550712475 (-0.0040859135103897826) | > avg_loss_disc: 2.736431276294547 (+0.05733024906104767) | > avg_loss_disc_real_0: 0.15657915080517107 (+0.023737991254934088) | > avg_loss_disc_real_1: 0.23313813226323732 (+0.0101231289371638) | > avg_loss_disc_real_2: 0.23653487903131565 (+0.0014647558121614057) | > avg_loss_disc_real_3: 0.22648054613193996 (-0.014387936029635673) | > avg_loss_disc_real_4: 0.2478413076048166 (+0.0012637208884870577) | > avg_loss_disc_real_5: 0.2440080764427991 (-0.0018931385916723387) | > avg_loss_0: 2.736431276294547 (+0.05733024906104767) | > avg_grad_norm_0: tensor(221.7923, device='cuda:0') (tensor(-65.4146, device='cuda:0')) | > avg_loss_gen: 2.0226975545077255 (-0.0705359737638025) | > avg_loss_kl: 2.468628807806634 (+0.3610988469191003) | > avg_loss_feat: 3.4621926999427903 (-0.08031771216594041) | > avg_loss_mel: 18.71286746817576 (+1.7592437368043825) | > avg_loss_duration: 1.6140068248963693 (+0.03422610356774136) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.280393358687274 (+2.003715058447604) | > avg_grad_norm_1: tensor(6180.4136, device='cuda:0') (tensor(-3026.0327, device='cuda:0'))  > EPOCH: 821/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:40:06)   --> TIME: 2025-02-25 16:40:24 -- STEP: 37/72 -- GLOBAL_STEP: 1059150 | > loss_disc: 2.6001834869384766 (2.6861057152619234) | > loss_disc_real_0: 0.2695988714694977 (0.18226262353159287) | > loss_disc_real_1: 0.17599362134933472 (0.23275524377822876) | > loss_disc_real_2: 0.23804444074630737 (0.23251442208483414) | > loss_disc_real_3: 0.23120255768299103 (0.23606530033253334) | > loss_disc_real_4: 0.27694669365882874 (0.24385798742642273) | > loss_disc_real_5: 0.27497485280036926 (0.24675996641854983) | > loss_0: 2.6001834869384766 (2.6861057152619234) | > grad_norm_0: tensor(445.2348, device='cuda:0') (tensor(179.8331, device='cuda:0')) | > loss_gen: 2.3756966590881348 (2.0522058235632405) | > loss_kl: 2.180394172668457 (2.639579308999551) | > loss_feat: 3.7445833683013916 (3.2987864404111296) | > loss_mel: 17.33391571044922 (18.2168039888949) | > loss_duration: 1.588017225265503 (1.6425924591116003) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.222604751586914 (27.849968111192858) | > grad_norm_1: tensor(4814.4326, device='cuda:0') (tensor(5834.5645, device='cuda:0')) | > current_lr_0: 0.00018049190912047277 | > current_lr_1: 0.00018049190912047277 | > step_time: 0.4445 (0.4374703716587376) | > loader_time: 0.0082 (0.007327266641565271) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737218212073959 (+9.689868335992571e-05) | > avg_step_time: 0.45783250432618905 (+0.004607408819064296) | > avg_loss_disc: 2.6865509966729393 (-0.04988027962160757) | > avg_loss_disc_real_0: 0.15398917069107712 (-0.0025899801140939482) | > avg_loss_disc_real_1: 0.23817671142833333 (+0.005038579165096008) | > avg_loss_disc_real_2: 0.23325164511170185 (-0.0032832339196137983) | > avg_loss_disc_real_3: 0.23608925560830343 (+0.00960870947636347) | > avg_loss_disc_real_4: 0.24521726055044524 (-0.002624047054371359) | > avg_loss_disc_real_5: 0.24601335256872042 (+0.002005276125921329) | > avg_loss_0: 2.6865509966729393 (-0.04988027962160757) | > avg_grad_norm_0: tensor(177.1522, device='cuda:0') (tensor(-44.6401, device='cuda:0')) | > avg_loss_gen: 2.0536287670404136 (+0.030931212532688157) | > avg_loss_kl: 2.4621751912882632 (-0.0064536165183706196) | > avg_loss_feat: 3.3566162015350773 (-0.105576498407713) | > avg_loss_mel: 17.817413612150805 (-0.8954538560249539) | > avg_loss_duration: 1.6305229025827328 (+0.01651607768636354) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.32035669138734 (-0.960036667299935) | > avg_grad_norm_1: tensor(5635.7676, device='cuda:0') (tensor(-544.6460, device='cuda:0'))  > EPOCH: 822/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:40:43)   --> TIME: 2025-02-25 16:40:52 -- STEP: 15/72 -- GLOBAL_STEP: 1059200 | > loss_disc: 2.5566203594207764 (2.6741555531819663) | > loss_disc_real_0: 0.24223659932613373 (0.1828207070628802) | > loss_disc_real_1: 0.2708156108856201 (0.22310252090295155) | > loss_disc_real_2: 0.20109480619430542 (0.24040553172429402) | > loss_disc_real_3: 0.2953197956085205 (0.2422977328300476) | > loss_disc_real_4: 0.25037363171577454 (0.2569352567195892) | > loss_disc_real_5: 0.2523152232170105 (0.2482374946276347) | > loss_0: 2.5566203594207764 (2.6741555531819663) | > grad_norm_0: tensor(635.8369, device='cuda:0') (tensor(251.8263, device='cuda:0')) | > loss_gen: 2.505070686340332 (2.1164237340291345) | > loss_kl: 1.8335696458816528 (2.0372148593266806) | > loss_feat: 3.982738971710205 (3.4348400910695394) | > loss_mel: 18.026470184326172 (17.3672976175944) | > loss_duration: 1.523475170135498 (1.5300167719523112) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.871326446533203 (26.48579279581706) | > grad_norm_1: tensor(11585.7422, device='cuda:0') (tensor(6976.6343, device='cuda:0')) | > current_lr_0: 0.0001804693476318327 | > current_lr_1: 0.0001804693476318327 | > step_time: 0.4512 (0.44011497497558594) | > loader_time: 0.0072 (0.006708701451619466)  --> TIME: 2025-02-25 16:41:16 -- STEP: 65/72 -- GLOBAL_STEP: 1059250 | > loss_disc: 2.684088706970215 (2.697622101123516) | > loss_disc_real_0: 0.25442224740982056 (0.1654539719223976) | > loss_disc_real_1: 0.19153912365436554 (0.24046740715320294) | > loss_disc_real_2: 0.22881408035755157 (0.24510119763704447) | > loss_disc_real_3: 0.22355373203754425 (0.2320460727581611) | > loss_disc_real_4: 0.23483438789844513 (0.24166433627788836) | > loss_disc_real_5: 0.25085484981536865 (0.2448991378912559) | > loss_0: 2.684088706970215 (2.697622101123516) | > grad_norm_0: tensor(338.9724, device='cuda:0') (tensor(209.7984, device='cuda:0')) | > loss_gen: 2.1409695148468018 (2.0545500553571245) | > loss_kl: 2.4089913368225098 (2.0260680327048664) | > loss_feat: 3.481372594833374 (3.321636331998385) | > loss_mel: 17.96614646911621 (17.016322135925297) | > loss_duration: 1.6302168369293213 (1.5625559916863074) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.627696990966797 (25.98113244863657) | > grad_norm_1: tensor(8611.2412, device='cuda:0') (tensor(6876.7803, device='cuda:0')) | > current_lr_0: 0.0001804693476318327 | > current_lr_1: 0.0001804693476318327 | > step_time: 0.4944 (0.4538787034841684) | > loader_time: 0.0119 (0.008646106719970703) --> EVAL PERFORMANCE | > avg_loader_time: 0.00895249004095373 (+0.00021527182887977163) | > avg_step_time: 0.4605069194041507 (+0.002674415077961678) | > avg_loss_disc: 2.68646025321853 (-9.074345440929221e-05) | > avg_loss_disc_real_0: 0.16057335616837085 (+0.006584185477293725) | > avg_loss_disc_real_1: 0.23848444118466175 (+0.00030772975632842314) | > avg_loss_disc_real_2: 0.24331545535947235 (+0.010063810247770505) | > avg_loss_disc_real_3: 0.23102483489144016 (-0.005064420716863277) | > avg_loss_disc_real_4: 0.24225026767858318 (-0.0029669928718620564) | > avg_loss_disc_real_5: 0.24529652528359858 (-0.000716827285121846) | > avg_loss_0: 2.68646025321853 (-9.074345440929221e-05) | > avg_grad_norm_0: tensor(203.2190, device='cuda:0') (+tensor(26.0668, device='cuda:0')) | > avg_loss_gen: 2.0496324720516994 (-0.003996294988714233) | > avg_loss_kl: 2.041880622716017 (-0.42029456857224634) | > avg_loss_feat: 3.334212366963776 (-0.02240383457130113) | > avg_loss_mel: 17.04622820733299 (-0.771185404817814) | > avg_loss_duration: 1.5680389622567406 (-0.06248394032599225) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.03999253393898 (-1.280364157448357) | > avg_grad_norm_1: tensor(6961.5054, device='cuda:0') (+tensor(1325.7378, device='cuda:0'))  > EPOCH: 823/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:41:21)   --> TIME: 2025-02-25 16:41:42 -- STEP: 43/72 -- GLOBAL_STEP: 1059300 | > loss_disc: 2.646956443786621 (2.6481883193171294) | > loss_disc_real_0: 0.07666679471731186 (0.15626653344478716) | > loss_disc_real_1: 0.4668668508529663 (0.24716570349626762) | > loss_disc_real_2: 0.2569636404514313 (0.22177117369895757) | > loss_disc_real_3: 0.21122382581233978 (0.2278970105703487) | > loss_disc_real_4: 0.2458425760269165 (0.23982736710892166) | > loss_disc_real_5: 0.2587200701236725 (0.2463865106882051) | > loss_0: 2.646956443786621 (2.6481883193171294) | > grad_norm_0: tensor(86.6276, device='cuda:0') (tensor(163.3395, device='cuda:0')) | > loss_gen: 2.468095541000366 (2.2636389399683745) | > loss_kl: 1.9001481533050537 (2.0180818091991344) | > loss_feat: 4.0472846031188965 (3.719190070795459) | > loss_mel: 16.858360290527344 (17.1442298223806) | > loss_duration: 1.563915729522705 (1.5395729430886202) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.837804794311523 (26.68471345236135) | > grad_norm_1: tensor(4827.8574, device='cuda:0') (tensor(4463.3047, device='cuda:0')) | > current_lr_0: 0.00018044678896337872 | > current_lr_1: 0.00018044678896337872 | > step_time: 0.4578 (0.4427152899808662) | > loader_time: 0.0097 (0.007554026537163313) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617605961544414 (-0.0003348840794093165) | > avg_step_time: 0.45696394544252206 (-0.0035429739616286704) | > avg_loss_disc: 2.674522692049053 (-0.011937561169477195) | > avg_loss_disc_real_0: 0.15394769430580274 (-0.006625661862568105) | > avg_loss_disc_real_1: 0.26510090118562674 (+0.026616460000964987) | > avg_loss_disc_real_2: 0.2332487616320731 (-0.010066693727399267) | > avg_loss_disc_real_3: 0.22631882382950314 (-0.004706011061937021) | > avg_loss_disc_real_4: 0.23900977028927334 (-0.003240497389309843) | > avg_loss_disc_real_5: 0.24528675578849415 (-9.769495104422488e-06) | > avg_loss_0: 2.674522692049053 (-0.011937561169477195) | > avg_grad_norm_0: tensor(172.4330, device='cuda:0') (tensor(-30.7860, device='cuda:0')) | > avg_loss_gen: 2.159961650069331 (+0.11032917801763142) | > avg_loss_kl: 2.0676290401270694 (+0.025748417411052493) | > avg_loss_feat: 3.502830226656417 (+0.16861785969264087) | > avg_loss_mel: 17.110499167106518 (+0.06427095977352693) | > avg_loss_duration: 1.5677412362165855 (-0.0002977260401550996) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.40866126476879 (+0.3686687308298069) | > avg_grad_norm_1: tensor(4764.2021, device='cuda:0') (tensor(-2197.3032, device='cuda:0'))  > EPOCH: 824/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:41:58)   --> TIME: 2025-02-25 16:42:09 -- STEP: 21/72 -- GLOBAL_STEP: 1059350 | > loss_disc: 2.547679901123047 (2.555571896689279) | > loss_disc_real_0: 0.09889842569828033 (0.09603391694171089) | > loss_disc_real_1: 0.20238804817199707 (0.2154185828708467) | > loss_disc_real_2: 0.21297679841518402 (0.2221879902340117) | > loss_disc_real_3: 0.22844286262989044 (0.2133536090453466) | > loss_disc_real_4: 0.20667137205600739 (0.23465287330604734) | > loss_disc_real_5: 0.19632816314697266 (0.23437152519112542) | > loss_0: 2.547679901123047 (2.555571896689279) | > grad_norm_0: tensor(220.6758, device='cuda:0') (tensor(206.2447, device='cuda:0')) | > loss_gen: 1.9066054821014404 (2.090333966981797) | > loss_kl: 2.2108559608459473 (3.1627995513734364) | > loss_feat: 3.7118873596191406 (3.7427891322544644) | > loss_mel: 19.155492782592773 (22.504563195364817) | > loss_duration: 1.6539589166641235 (1.909390960420881) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.6387996673584 (33.40987695966448) | > grad_norm_1: tensor(7993.1235, device='cuda:0') (tensor(7840.4590, device='cuda:0')) | > current_lr_0: 0.00018042423311475828 | > current_lr_1: 0.00018042423311475828 | > step_time: 0.4365 (0.4354677540915353) | > loader_time: 0.0061 (0.006740025111607143)  --> TIME: 2025-02-25 16:42:33 -- STEP: 71/72 -- GLOBAL_STEP: 1059400 | > loss_disc: 2.640434741973877 (2.547109352031224) | > loss_disc_real_0: 0.09927880018949509 (0.09607022455040837) | > loss_disc_real_1: 0.21899491548538208 (0.22099147527150706) | > loss_disc_real_2: 0.20164243876934052 (0.2213593496822975) | > loss_disc_real_3: 0.21921154856681824 (0.2298363082005944) | > loss_disc_real_4: 0.23368410766124725 (0.23637126994804597) | > loss_disc_real_5: 0.24065370857715607 (0.23927782496935884) | > loss_0: 2.640434741973877 (2.547109352031224) | > grad_norm_0: tensor(27.3513, device='cuda:0') (tensor(210.2075, device='cuda:0')) | > loss_gen: 2.0135483741760254 (2.142876940713801) | > loss_kl: 2.2340035438537598 (2.4840403308331127) | > loss_feat: 3.347057342529297 (3.7786930413313313) | > loss_mel: 16.495513916015625 (18.96075545566183) | > loss_duration: 1.5795679092407227 (1.7036488207293228) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.669689178466797 (29.070014496924173) | > grad_norm_1: tensor(3895.0466, device='cuda:0') (tensor(9019.1455, device='cuda:0')) | > current_lr_0: 0.00018042423311475828 | > current_lr_1: 0.00018042423311475828 | > step_time: 0.5052 (0.45488852850148376) | > loader_time: 0.0083 (0.00877946867069728) --> EVAL PERFORMANCE | > avg_loader_time: 0.00877946867069728 (+0.00016186270915286537) | > avg_step_time: 0.45488852850148376 (-0.0020754169410382994) | > avg_loss_disc: 2.547109352031224 (-0.12741334001782878) | > avg_loss_disc_real_0: 0.09607022455040837 (-0.05787746975539437) | > avg_loss_disc_real_1: 0.22099147527150706 (-0.044109425914119677) | > avg_loss_disc_real_2: 0.2213593496822975 (-0.01188941194977558) | > avg_loss_disc_real_3: 0.2298363082005944 (+0.0035174843710912562) | > avg_loss_disc_real_4: 0.23637126994804597 (-0.0026385003412273678) | > avg_loss_disc_real_5: 0.23927782496935884 (-0.006008930819135311) | > avg_loss_0: 2.547109352031224 (-0.12741334001782878) | > avg_grad_norm_0: tensor(210.2075, device='cuda:0') (+tensor(37.7744, device='cuda:0')) | > avg_loss_gen: 2.142876940713801 (-0.01708470935552997) | > avg_loss_kl: 2.4840403308331127 (+0.4164112907060433) | > avg_loss_feat: 3.7786930413313313 (+0.27586281467491425) | > avg_loss_mel: 18.96075545566183 (+1.850256288555311) | > avg_loss_duration: 1.7036488207293228 (+0.13590758451273732) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.070014496924173 (+2.6613532321553848) | > avg_grad_norm_1: tensor(9019.1455, device='cuda:0') (+tensor(4254.9434, device='cuda:0'))  > EPOCH: 825/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:42:35)   --> TIME: 2025-02-25 16:42:59 -- STEP: 49/72 -- GLOBAL_STEP: 1059450 | > loss_disc: 2.491361141204834 (2.670302610008084) | > loss_disc_real_0: 0.07047416269779205 (0.20681368954935853) | > loss_disc_real_1: 0.17061783373355865 (0.21990327506649252) | > loss_disc_real_2: 0.2195909023284912 (0.2267679994811817) | > loss_disc_real_3: 0.21515892446041107 (0.2382878642909381) | > loss_disc_real_4: 0.24325326085090637 (0.24559598340063682) | > loss_disc_real_5: 0.2391068935394287 (0.24778044771174998) | > loss_0: 2.491361141204834 (2.670302610008084) | > grad_norm_0: tensor(105.3760, device='cuda:0') (tensor(255.3874, device='cuda:0')) | > loss_gen: 1.9421591758728027 (2.1222077924378064) | > loss_kl: 2.2657225131988525 (2.0774714022266623) | > loss_feat: 3.2412710189819336 (3.4727430976167017) | > loss_mel: 17.72828483581543 (17.44397132250727) | > loss_duration: 1.662699580192566 (1.5639528863284053) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.840137481689453 (26.680346819819235) | > grad_norm_1: tensor(6835.2974, device='cuda:0') (tensor(8018.3687, device='cuda:0')) | > current_lr_0: 0.00018040168008561893 | > current_lr_1: 0.00018040168008561893 | > step_time: 0.4374 (0.45233324109291545) | > loader_time: 0.0094 (0.0076531390754544) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575006270072832 (-0.00020446240062444755) | > avg_step_time: 0.4604302863000144 (+0.0055417577985306665) | > avg_loss_disc: 2.6531223512031663 (+0.10601299917194229) | > avg_loss_disc_real_0: 0.17072905825687126 (+0.0746588337064629) | > avg_loss_disc_real_1: 0.22938816669121595 (+0.00839669141970889) | > avg_loss_disc_real_2: 0.24118842478369323 (+0.019829075101395727) | > avg_loss_disc_real_3: 0.23972842202220165 (+0.009892113821607262) | > avg_loss_disc_real_4: 0.24478483661799363 (+0.008413566669947664) | > avg_loss_disc_real_5: 0.2441944377103322 (+0.004916612740973353) | > avg_loss_0: 2.6531223512031663 (+0.10601299917194229) | > avg_grad_norm_0: tensor(220.9496, device='cuda:0') (+tensor(10.7421, device='cuda:0')) | > avg_loss_gen: 2.165225339607453 (+0.022348398893651922) | > avg_loss_kl: 2.1354038144501164 (-0.3486365163829963) | > avg_loss_feat: 3.562011701959959 (-0.21668133937137224) | > avg_loss_mel: 17.464017491945068 (-1.496737963716761) | > avg_loss_duration: 1.5797837032398707 (-0.12386511748945206) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.906442266115 (-2.163572230809173) | > avg_grad_norm_1: tensor(6677.4780, device='cuda:0') (tensor(-2341.6675, device='cuda:0'))  > EPOCH: 826/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:43:12)   --> TIME: 2025-02-25 16:43:26 -- STEP: 27/72 -- GLOBAL_STEP: 1059500 | > loss_disc: 2.7964062690734863 (2.761412337974266) | > loss_disc_real_0: 0.14779701828956604 (0.10997049800223774) | > loss_disc_real_1: 0.2539573907852173 (0.26985310625146935) | > loss_disc_real_2: 0.31094756722450256 (0.2698606887349377) | > loss_disc_real_3: 0.27518197894096375 (0.2571824347531354) | > loss_disc_real_4: 0.2633686065673828 (0.28119024579171775) | > loss_disc_real_5: 0.24984577298164368 (0.24533974848411705) | > loss_0: 2.7964062690734863 (2.761412337974266) | > grad_norm_0: tensor(138.1301, device='cuda:0') (tensor(266.6045, device='cuda:0')) | > loss_gen: 2.101119041442871 (2.1178928922723834) | > loss_kl: 2.568977117538452 (2.0611341617725505) | > loss_feat: 3.113393545150757 (3.465692060965079) | > loss_mel: 19.319561004638672 (17.684459898206924) | > loss_duration: 1.5712378025054932 (1.530900862481859) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.674287796020508 (26.860079800641095) | > grad_norm_1: tensor(12169.1455, device='cuda:0') (tensor(6785.5127, device='cuda:0')) | > current_lr_0: 0.00018037912987560822 | > current_lr_1: 0.00018037912987560822 | > step_time: 0.4532 (0.438118855158488) | > loader_time: 0.0084 (0.007020349855776186) --> EVAL PERFORMANCE | > avg_loader_time: 0.008633277785610143 (+5.827151553731112e-05) | > avg_step_time: 0.4574489022644473 (-0.002981384035567136) | > avg_loss_disc: 2.6717678224536736 (+0.018645471250507306) | > avg_loss_disc_real_0: 0.13415730839044276 (-0.0365717498664285) | > avg_loss_disc_real_1: 0.23955298099719297 (+0.010164814305977016) | > avg_loss_disc_real_2: 0.2552132440704696 (+0.014024819286776347) | > avg_loss_disc_real_3: 0.24299053197175674 (+0.0032621099495550865) | > avg_loss_disc_real_4: 0.2594226386765358 (+0.014637802058542176) | > avg_loss_disc_real_5: 0.24186476687310446 (-0.0023296708372277375) | > avg_loss_0: 2.6717678224536736 (+0.018645471250507306) | > avg_grad_norm_0: tensor(215.0921, device='cuda:0') (tensor(-5.8575, device='cuda:0')) | > avg_loss_gen: 2.137320152470764 (-0.02790518713668888) | > avg_loss_kl: 2.3472663348829244 (+0.21186252043280795) | > avg_loss_feat: 3.5070507593557867 (-0.05496094260417239) | > avg_loss_mel: 18.40535448638487 (+0.9413369944398013) | > avg_loss_duration: 1.5916210295448838 (+0.011837326305013107) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.988612806293325 (+1.0821705401783248) | > avg_grad_norm_1: tensor(6669.4829, device='cuda:0') (tensor(-7.9951, device='cuda:0'))  > EPOCH: 827/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:43:49)   --> TIME: 2025-02-25 16:43:53 -- STEP: 5/72 -- GLOBAL_STEP: 1059550 | > loss_disc: 2.6790833473205566 (2.661929225921631) | > loss_disc_real_0: 0.1342371255159378 (0.11016534268856049) | > loss_disc_real_1: 0.3216683864593506 (0.26404817700386046) | > loss_disc_real_2: 0.21837283670902252 (0.24097525477409362) | > loss_disc_real_3: 0.22100035846233368 (0.2265285223722458) | > loss_disc_real_4: 0.22777901589870453 (0.23321238458156585) | > loss_disc_real_5: 0.2368023842573166 (0.23905282318592072) | > loss_0: 2.6790833473205566 (2.661929225921631) | > grad_norm_0: tensor(142.9222, device='cuda:0') (tensor(197.0270, device='cuda:0')) | > loss_gen: 1.9844720363616943 (2.092735004425049) | > loss_kl: 1.9664067029953003 (2.0273592472076416) | > loss_feat: 3.5390303134918213 (3.632389211654663) | > loss_mel: 18.331462860107422 (17.857493591308593) | > loss_duration: 1.5676852464675903 (1.5531798601150513) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.389057159423828 (27.16315689086914) | > grad_norm_1: tensor(13059.3857, device='cuda:0') (tensor(9504.9727, device='cuda:0')) | > current_lr_0: 0.00018035658248437376 | > current_lr_1: 0.00018035658248437376 | > step_time: 0.4314 (0.4363893508911133) | > loader_time: 0.0065 (0.006742668151855469)  --> TIME: 2025-02-25 16:44:16 -- STEP: 55/72 -- GLOBAL_STEP: 1059600 | > loss_disc: 2.6437630653381348 (2.656761087070812) | > loss_disc_real_0: 0.12492448091506958 (0.11274663162502375) | > loss_disc_real_1: 0.19591186940670013 (0.23620840500701556) | > loss_disc_real_2: 0.2227608561515808 (0.23767707483334974) | > loss_disc_real_3: 0.2443745732307434 (0.22826688912781803) | > loss_disc_real_4: 0.2234693020582199 (0.24222947873852468) | > loss_disc_real_5: 0.24168050289154053 (0.24288979145613585) | > loss_0: 2.6437630653381348 (2.656761087070812) | > grad_norm_0: tensor(91.1861, device='cuda:0') (tensor(218.5524, device='cuda:0')) | > loss_gen: 2.091050863265991 (2.0621113863858302) | > loss_kl: 2.150852918624878 (2.1379998532208524) | > loss_feat: 3.3380918502807617 (3.594287638230757) | > loss_mel: 17.089012145996094 (17.601255052739923) | > loss_duration: 1.608072280883789 (1.574049034985629) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.277080535888672 (26.969702703302556) | > grad_norm_1: tensor(11402.7334, device='cuda:0') (tensor(7721.3350, device='cuda:0')) | > current_lr_0: 0.00018035658248437376 | > current_lr_1: 0.00018035658248437376 | > step_time: 0.4712 (0.44316050789572975) | > loader_time: 0.0095 (0.007884138280695134) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598817905909578 (-3.44598797005647e-05) | > avg_step_time: 0.4543040436758122 (-0.0031448585886351133) | > avg_loss_disc: 2.666680785971628 (-0.005087036482045448) | > avg_loss_disc_real_0: 0.14504997385009913 (+0.010892665459656364) | > avg_loss_disc_real_1: 0.23213139889945447 (-0.007421582097738494) | > avg_loss_disc_real_2: 0.23501627231147929 (-0.020196971758990295) | > avg_loss_disc_real_3: 0.22465937825995433 (-0.01833115371180241) | > avg_loss_disc_real_4: 0.2400807971265954 (-0.019341841549940397) | > avg_loss_disc_real_5: 0.24214058170016384 (+0.00027581482705937854) | > avg_loss_0: 2.666680785971628 (-0.005087036482045448) | > avg_grad_norm_0: tensor(238.1023, device='cuda:0') (+tensor(23.0102, device='cuda:0')) | > avg_loss_gen: 2.0494860410690303 (-0.08783411140173358) | > avg_loss_kl: 2.1291315253351772 (-0.2181348095477471) | > avg_loss_feat: 3.499401552576414 (-0.0076492067793725305) | > avg_loss_mel: 17.438370476306325 (-0.9669840100785443) | > avg_loss_duration: 1.5828095137233464 (-0.008811515821537386) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.699198978047974 (-1.2894138282453511) | > avg_grad_norm_1: tensor(7596.5679, device='cuda:0') (+tensor(927.0850, device='cuda:0'))  > EPOCH: 828/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:44:26)   --> TIME: 2025-02-25 16:44:43 -- STEP: 33/72 -- GLOBAL_STEP: 1059650 | > loss_disc: 2.4861557483673096 (2.563106565764456) | > loss_disc_real_0: 0.13835126161575317 (0.09990222989158197) | > loss_disc_real_1: 0.21427439153194427 (0.21763385064674146) | > loss_disc_real_2: 0.22922614216804504 (0.2279315423784834) | > loss_disc_real_3: 0.21778486669063568 (0.22611570358276367) | > loss_disc_real_4: 0.22801849246025085 (0.23460499490752365) | > loss_disc_real_5: 0.22441786527633667 (0.23226731157664096) | > loss_0: 2.4861557483673096 (2.563106565764456) | > grad_norm_0: tensor(270.2876, device='cuda:0') (tensor(226.9025, device='cuda:0')) | > loss_gen: 2.32266902923584 (2.1040039929476646) | > loss_kl: 2.0265252590179443 (1.9392158009789207) | > loss_feat: 3.9546849727630615 (3.777097702026367) | > loss_mel: 16.640159606933594 (17.114771814057324) | > loss_duration: 1.5209929943084717 (1.5255573879588733) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.465030670166016 (26.46064665823272) | > grad_norm_1: tensor(4598.4819, device='cuda:0') (tensor(8821.8594, device='cuda:0')) | > current_lr_0: 0.0001803340379115632 | > current_lr_1: 0.0001803340379115632 | > step_time: 0.4512 (0.4416742180332993) | > loader_time: 0.0078 (0.00699548287825151) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855320608112174 (-4.561182478783794e-05) | > avg_step_time: 0.4586285033696134 (+0.004324459693801241) | > avg_loss_disc: 2.5576238867262715 (-0.10905689924535666) | > avg_loss_disc_real_0: 0.10185893583045878 (-0.043191038019640346) | > avg_loss_disc_real_1: 0.2203084881037054 (-0.01182291079574907) | > avg_loss_disc_real_2: 0.23058040167244387 (-0.004435870639035416) | > avg_loss_disc_real_3: 0.2301400175816576 (+0.0054806393217032845) | > avg_loss_disc_real_4: 0.23977890379831823 (-0.00030189332827718074) | > avg_loss_disc_real_5: 0.2410578108589414 (-0.0010827708412224357) | > avg_loss_0: 2.5576238867262715 (-0.10905689924535666) | > avg_grad_norm_0: tensor(240.3795, device='cuda:0') (+tensor(2.2771, device='cuda:0')) | > avg_loss_gen: 2.13244544956046 (+0.0829594084914298) | > avg_loss_kl: 1.9817997955940139 (-0.14733172974116338) | > avg_loss_feat: 3.729606497455651 (+0.2302049448792367) | > avg_loss_mel: 16.959464852239044 (-0.47890562406728066) | > avg_loss_duration: 1.5597702180835562 (-0.023039295639790236) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.363086727303518 (-0.3361122507444563) | > avg_grad_norm_1: tensor(9337.5273, device='cuda:0') (+tensor(1740.9595, device='cuda:0'))  > EPOCH: 829/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:45:04)   --> TIME: 2025-02-25 16:45:10 -- STEP: 11/72 -- GLOBAL_STEP: 1059700 | > loss_disc: 2.6135764122009277 (2.544479348442771) | > loss_disc_real_0: 0.099978968501091 (0.09764289585026828) | > loss_disc_real_1: 0.20244310796260834 (0.22592573003335434) | > loss_disc_real_2: 0.23165729641914368 (0.22915373200720007) | > loss_disc_real_3: 0.22335270047187805 (0.2384502806446769) | > loss_disc_real_4: 0.25017663836479187 (0.23989829285578293) | > loss_disc_real_5: 0.25625714659690857 (0.24310945922678168) | > loss_0: 2.6135764122009277 (2.544479348442771) | > grad_norm_0: tensor(346.7155, device='cuda:0') (tensor(273.5927, device='cuda:0')) | > loss_gen: 1.8045525550842285 (2.176160595633767) | > loss_kl: 1.931231141090393 (1.879817247390747) | > loss_feat: 3.108504295349121 (3.7944943471388384) | > loss_mel: 17.157869338989258 (17.079349691217597) | > loss_duration: 1.5049487352371216 (1.520437023856423) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.50710678100586 (26.450258601795543) | > grad_norm_1: tensor(11514.4043, device='cuda:0') (tensor(9653.4199, device='cuda:0')) | > current_lr_0: 0.00018031149615682423 | > current_lr_1: 0.00018031149615682423 | > step_time: 0.4257 (0.42949904095042835) | > loader_time: 0.0067 (0.006530848416415128)  --> TIME: 2025-02-25 16:45:34 -- STEP: 61/72 -- GLOBAL_STEP: 1059750 | > loss_disc: 2.5553860664367676 (2.6938738549341923) | > loss_disc_real_0: 0.07468339055776596 (0.16270752098472388) | > loss_disc_real_1: 0.2372199147939682 (0.27493416872180887) | > loss_disc_real_2: 0.25008252263069153 (0.24245262097139827) | > loss_disc_real_3: 0.2562856078147888 (0.2306888611590276) | > loss_disc_real_4: 0.24755115807056427 (0.2470616375813718) | > loss_disc_real_5: 0.24942144751548767 (0.24659434473905406) | > loss_0: 2.5553860664367676 (2.6938738549341923) | > grad_norm_0: tensor(276.0436, device='cuda:0') (tensor(242.0013, device='cuda:0')) | > loss_gen: 1.9482382535934448 (2.156518861895702) | > loss_kl: 2.126892566680908 (2.141190956850522) | > loss_feat: 3.179586410522461 (3.530431477749934) | > loss_mel: 17.48660659790039 (17.845070104129988) | > loss_duration: 1.6250418424606323 (1.5723724267521844) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.366365432739258 (27.245583925090852) | > grad_norm_1: tensor(14223.9580, device='cuda:0') (tensor(7845.3608, device='cuda:0')) | > current_lr_0: 0.00018031149615682423 | > current_lr_1: 0.00018031149615682423 | > step_time: 0.4703 (0.4475773162529117) | > loader_time: 0.0108 (0.008186527940093495) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661401103919662 (+0.0001081950227979217) | > avg_step_time: 0.4562302441664145 (-0.002398259203198916) | > avg_loss_disc: 2.684416952267499 (+0.1267930655412277) | > avg_loss_disc_real_0: 0.16020460243166332 (+0.058345666601204535) | > avg_loss_disc_real_1: 0.26909261438208576 (+0.04878412627838036) | > avg_loss_disc_real_2: 0.23991434087216015 (+0.009333939199716285) | > avg_loss_disc_real_3: 0.23262117173470243 (+0.0024811541530448122) | > avg_loss_disc_real_4: 0.24566476592715353 (+0.005885862128835295) | > avg_loss_disc_real_5: 0.24571685639905258 (+0.0046590455401111786) | > avg_loss_0: 2.684416952267499 (+0.1267930655412277) | > avg_grad_norm_0: tensor(246.2797, device='cuda:0') (+tensor(5.9002, device='cuda:0')) | > avg_loss_gen: 2.1462250813631942 (+0.013779631802734116) | > avg_loss_kl: 2.1384356676692704 (+0.15663587207525653) | > avg_loss_feat: 3.4889486675531094 (-0.24065782990254148) | > avg_loss_mel: 17.73366388132875 (+0.7741990290897043) | > avg_loss_duration: 1.579040762404321 (+0.019270544320764804) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.086314134194815 (+0.7232274068912972) | > avg_grad_norm_1: tensor(7903.3213, device='cuda:0') (tensor(-1434.2061, device='cuda:0'))  > EPOCH: 830/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:45:41)   --> TIME: 2025-02-25 16:46:00 -- STEP: 39/72 -- GLOBAL_STEP: 1059800 | > loss_disc: 2.5227184295654297 (2.636153612381373) | > loss_disc_real_0: 0.18875402212142944 (0.1396889017942624) | > loss_disc_real_1: 0.19579270482063293 (0.22176260367417946) | > loss_disc_real_2: 0.2338847815990448 (0.23509730169406304) | > loss_disc_real_3: 0.22130534052848816 (0.23395154911738175) | > loss_disc_real_4: 0.2393970638513565 (0.24413939660940415) | > loss_disc_real_5: 0.23990190029144287 (0.25084590376951754) | > loss_0: 2.5227184295654297 (2.636153612381373) | > grad_norm_0: tensor(266.9048, device='cuda:0') (tensor(253.0797, device='cuda:0')) | > loss_gen: 2.3247768878936768 (2.0469627074706245) | > loss_kl: 2.2422852516174316 (1.9417884380389483) | > loss_feat: 3.983351469039917 (3.4692039673145) | > loss_mel: 17.826841354370117 (17.195218404134113) | > loss_duration: 1.5566493272781372 (1.5328360276344495) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.93390655517578 (26.18600967602852) | > grad_norm_1: tensor(7313.6411, device='cuda:0') (tensor(8864.6514, device='cuda:0')) | > current_lr_0: 0.00018028895721980463 | > current_lr_1: 0.00018028895721980463 | > step_time: 0.4547 (0.4403347969055176) | > loader_time: 0.0084 (0.0074576842479216745) --> EVAL PERFORMANCE | > avg_loader_time: 0.008726868830936057 (+6.546772701639461e-05) | > avg_step_time: 0.45598712101788585 (-0.00024312314852864558) | > avg_loss_disc: 2.61843710885921 (-0.065979843408289) | > avg_loss_disc_real_0: 0.12574893272888493 (-0.034455669702778385) | > avg_loss_disc_real_1: 0.22070401550178795 (-0.04838859888029781) | > avg_loss_disc_real_2: 0.23379590519717042 (-0.006118435674989736) | > avg_loss_disc_real_3: 0.23372644207007448 (+0.001105270335372055) | > avg_loss_disc_real_4: 0.24262356988980738 (-0.00304119603734615) | > avg_loss_disc_real_5: 0.24688673103359385 (+0.001169874634541268) | > avg_loss_0: 2.61843710885921 (-0.065979843408289) | > avg_grad_norm_0: tensor(267.9127, device='cuda:0') (+tensor(21.6330, device='cuda:0')) | > avg_loss_gen: 2.0528957994890886 (-0.09332928187410561) | > avg_loss_kl: 1.982485507575559 (-0.15595016009371143) | > avg_loss_feat: 3.4988507754366163 (+0.009902107883506872) | > avg_loss_mel: 17.014262965027708 (-0.7194009163010406) | > avg_loss_duration: 1.5630921729853455 (-0.01594858941897548) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.111587228909347 (-0.9747269052854683) | > avg_grad_norm_1: tensor(9083.9717, device='cuda:0') (+tensor(1180.6504, device='cuda:0'))  > EPOCH: 831/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:46:18)   --> TIME: 2025-02-25 16:46:28 -- STEP: 17/72 -- GLOBAL_STEP: 1059850 | > loss_disc: 2.5405631065368652 (2.542136613060446) | > loss_disc_real_0: 0.07877875864505768 (0.09703566353110706) | > loss_disc_real_1: 0.20380333065986633 (0.21903015585506663) | > loss_disc_real_2: 0.2354598194360733 (0.22549351173288681) | > loss_disc_real_3: 0.23431916534900665 (0.23370703002985785) | > loss_disc_real_4: 0.2351463884115219 (0.2401369266650256) | > loss_disc_real_5: 0.23041772842407227 (0.23408693162833943) | > loss_0: 2.5405631065368652 (2.542136613060446) | > grad_norm_0: tensor(310.6953, device='cuda:0') (tensor(292.1797, device='cuda:0')) | > loss_gen: 2.1993024349212646 (2.162592193659614) | > loss_kl: 1.892956018447876 (1.922545397982878) | > loss_feat: 3.7959113121032715 (3.910564618952134) | > loss_mel: 17.484888076782227 (16.990115894990808) | > loss_duration: 1.5479228496551514 (1.5153190458522123) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.92098045349121 (26.501137452967026) | > grad_norm_1: tensor(13277.0605, device='cuda:0') (tensor(10439.1025, device='cuda:0')) | > current_lr_0: 0.00018026642110015214 | > current_lr_1: 0.00018026642110015214 | > step_time: 0.4303 (0.43671349918141084) | > loader_time: 0.0083 (0.007170550963457893)  --> TIME: 2025-02-25 16:46:51 -- STEP: 67/72 -- GLOBAL_STEP: 1059900 | > loss_disc: 3.050908327102661 (2.5770231538744115) | > loss_disc_real_0: 0.31323692202568054 (0.10736771172551966) | > loss_disc_real_1: 0.3954330384731293 (0.22086235158034226) | > loss_disc_real_2: 0.4091355800628662 (0.24546135806325656) | > loss_disc_real_3: 0.37548205256462097 (0.24077824936873876) | > loss_disc_real_4: 0.2978202998638153 (0.24259151629547573) | > loss_disc_real_5: 0.26818785071372986 (0.2423797725741543) | > loss_0: 3.050908327102661 (2.5770231538744115) | > grad_norm_0: tensor(852.5977, device='cuda:0') (tensor(300.3383, device='cuda:0')) | > loss_gen: 2.5305488109588623 (2.1964875032652666) | > loss_kl: 2.267934799194336 (1.972235430532427) | > loss_feat: 2.790933132171631 (3.8477887110923654) | > loss_mel: 16.978708267211914 (16.878977263151707) | > loss_duration: 1.6520756483078003 (1.554480077615425) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.220199584960938 (26.449969106645728) | > grad_norm_1: tensor(7356.4185, device='cuda:0') (tensor(10223.6641, device='cuda:0')) | > current_lr_0: 0.00018026642110015214 | > current_lr_1: 0.00018026642110015214 | > step_time: 0.4972 (0.45298609448902644) | > loader_time: 0.0126 (0.008608131266351958) --> EVAL PERFORMANCE | > avg_loader_time: 0.008776731894049846 (+4.9863063113789186e-05) | > avg_step_time: 0.4573285680421641 (+0.0013414470242782417) | > avg_loss_disc: 2.592968457181689 (-0.025468651677521326) | > avg_loss_disc_real_0: 0.10822494621847717 (-0.017523986510407763) | > avg_loss_disc_real_1: 0.22929814510362248 (+0.008594129601834533) | > avg_loss_disc_real_2: 0.2512495278472632 (+0.01745362265009276) | > avg_loss_disc_real_3: 0.2447251746772041 (+0.010998732607129608) | > avg_loss_disc_real_4: 0.24503882460191218 (+0.0024152547121047974) | > avg_loss_disc_real_5: 0.24258598349463772 (-0.004300747538956123) | > avg_loss_0: 2.592968457181689 (-0.025468651677521326) | > avg_grad_norm_0: tensor(294.0283, device='cuda:0') (+tensor(26.1156, device='cuda:0')) | > avg_loss_gen: 2.190522615338715 (+0.13762681584962655) | > avg_loss_kl: 1.9829015328850546 (+0.00041602530949558236) | > avg_loss_feat: 3.8265353894569505 (+0.3276846140203342) | > avg_loss_mel: 16.885849093047664 (-0.12841387198004384) | > avg_loss_duration: 1.557627300141563 (-0.005464872843782498) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.44343601817816 (+0.33184878926881467) | > avg_grad_norm_1: tensor(10209.3438, device='cuda:0') (+tensor(1125.3721, device='cuda:0'))  > EPOCH: 832/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:46:55)   --> TIME: 2025-02-25 16:47:18 -- STEP: 45/72 -- GLOBAL_STEP: 1059950 | > loss_disc: 2.6847033500671387 (2.628954373465644) | > loss_disc_real_0: 0.12664037942886353 (0.11592982212702434) | > loss_disc_real_1: 0.2258763164281845 (0.23262240886688232) | > loss_disc_real_2: 0.2445264756679535 (0.23557004299428727) | > loss_disc_real_3: 0.24823559820652008 (0.2359594835175408) | > loss_disc_real_4: 0.24647675454616547 (0.24465262922975753) | > loss_disc_real_5: 0.24203570187091827 (0.24925557076930993) | > loss_0: 2.6847033500671387 (2.628954373465644) | > grad_norm_0: tensor(270.8724, device='cuda:0') (tensor(277.9758, device='cuda:0')) | > loss_gen: 1.8030831813812256 (2.1024487071567113) | > loss_kl: 2.1393234729766846 (1.959294366836548) | > loss_feat: 2.947481155395508 (3.5838171005249024) | > loss_mel: 15.79076099395752 (16.70752224392361) | > loss_duration: 1.568606972694397 (1.5334209971957737) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.249256134033203 (25.886503389146593) | > grad_norm_1: tensor(11014.7314, device='cuda:0') (tensor(9383.3730, device='cuda:0')) | > current_lr_0: 0.0001802438877975146 | > current_lr_1: 0.0001802438877975146 | > step_time: 0.4666 (0.44413901964823405) | > loader_time: 0.0108 (0.007750654220581054) --> EVAL PERFORMANCE | > avg_loader_time: 0.008756160736083984 (-2.0571157965861356e-05) | > avg_step_time: 0.45716905929672885 (-0.00015950874543524396) | > avg_loss_disc: 2.616325680638703 (+0.023357223457014076) | > avg_loss_disc_real_0: 0.11081835449161664 (+0.0025934082731394748) | > avg_loss_disc_real_1: 0.2306823575160873 (+0.001384212412464808) | > avg_loss_disc_real_2: 0.2354103601734403 (-0.015839167673822874) | > avg_loss_disc_real_3: 0.23742885111083448 (-0.007296323566369611) | > avg_loss_disc_real_4: 0.24483592661333756 (-0.0002028979885746185) | > avg_loss_disc_real_5: 0.24868562439797626 (+0.006099640903338532) | > avg_loss_0: 2.616325680638703 (+0.023357223457014076) | > avg_grad_norm_0: tensor(277.4064, device='cuda:0') (tensor(-16.6219, device='cuda:0')) | > avg_loss_gen: 2.108051914564321 (-0.08247070077439433) | > avg_loss_kl: 1.9979382736582152 (+0.01503674077316064) | > avg_loss_feat: 3.5612722819959615 (-0.265263107460989) | > avg_loss_mel: 16.663995339836866 (-0.22185375321079803) | > avg_loss_duration: 1.558081673904204 (+0.0004543737626410138) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.88933942015742 (-0.5540965980207417) | > avg_grad_norm_1: tensor(9659.8564, device='cuda:0') (tensor(-549.4873, device='cuda:0'))  > EPOCH: 833/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:47:32)   --> TIME: 2025-02-25 16:47:45 -- STEP: 23/72 -- GLOBAL_STEP: 1060000 | > loss_disc: 2.7054965496063232 (2.5433642967887544) | > loss_disc_real_0: 0.07203800976276398 (0.09153877214893051) | > loss_disc_real_1: 0.22754737734794617 (0.22379485096620477) | > loss_disc_real_2: 0.23125910758972168 (0.23181738736836807) | > loss_disc_real_3: 0.24192039668560028 (0.23388524742230124) | > loss_disc_real_4: 0.24074384570121765 (0.23973899755788886) | > loss_disc_real_5: 0.25235962867736816 (0.24346609284048495) | > loss_0: 2.7054965496063232 (2.5433642967887544) | > grad_norm_0: tensor(272.3490, device='cuda:0') (tensor(294.1747, device='cuda:0')) | > loss_gen: 1.9905760288238525 (2.1875831769860308) | > loss_kl: 1.7622886896133423 (1.9202470675758694) | > loss_feat: 3.7807698249816895 (3.9651814647342847) | > loss_mel: 16.69780731201172 (17.081742245218027) | > loss_duration: 1.5283739566802979 (1.509005245955094) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.759815216064453 (26.66375914863918) | > grad_norm_1: tensor(12163.3682, device='cuda:0') (tensor(10497.7920, device='cuda:0')) | > current_lr_0: 0.00018022135731153992 | > current_lr_1: 0.00018022135731153992 | > step_time: 0.4367 (0.43803541556648584) | > loader_time: 0.0069 (0.006957292556762695) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1060000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008779582842974596 (+2.342210689061132e-05) | > avg_step_time: 0.45978565954826245 (+0.002616600251533596) | > avg_loss_disc: 2.555363178253174 (-0.06096250238552914) | > avg_loss_disc_real_0: 0.09808160086542787 (-0.012736753626188771) | > avg_loss_disc_real_1: 0.22297319329120743 (-0.007709164224879855) | > avg_loss_disc_real_2: 0.22899059678467226 (-0.0064197633887680405) | > avg_loss_disc_real_3: 0.23659976792167609 (-0.0008290831891583916) | > avg_loss_disc_real_4: 0.24290208476529995 (-0.001933841848037604) | > avg_loss_disc_real_5: 0.24584986962063213 (-0.0028357547773441294) | > avg_loss_0: 2.555363178253174 (-0.06096250238552914) | > avg_grad_norm_0: tensor(270.8372, device='cuda:0') (tensor(-6.5692, device='cuda:0')) | > avg_loss_gen: 2.1789426753218746 (+0.07089076075755374) | > avg_loss_kl: 2.006261756722356 (+0.00832348306414099) | > avg_loss_feat: 3.8104028735362308 (+0.24913059154026929) | > avg_loss_mel: 16.828783330783033 (+0.16478799094616647) | > avg_loss_duration: 1.5539589616614329 (-0.004122712242771165) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.378349438519546 (+0.48901001836212643) | > avg_grad_norm_1: tensor(9809.1621, device='cuda:0') (+tensor(149.3057, device='cuda:0'))  > EPOCH: 834/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:48:13)   --> TIME: 2025-02-25 16:48:15 -- STEP: 1/72 -- GLOBAL_STEP: 1060050 | > loss_disc: 2.45106840133667 (2.45106840133667) | > loss_disc_real_0: 0.11230631172657013 (0.11230631172657013) | > loss_disc_real_1: 0.2010139524936676 (0.2010139524936676) | > loss_disc_real_2: 0.21928580105304718 (0.21928580105304718) | > loss_disc_real_3: 0.23586352169513702 (0.23586352169513702) | > loss_disc_real_4: 0.2404600828886032 (0.2404600828886032) | > loss_disc_real_5: 0.23089051246643066 (0.23089051246643066) | > loss_0: 2.45106840133667 (2.45106840133667) | > grad_norm_0: tensor(310.2841, device='cuda:0') (tensor(310.2841, device='cuda:0')) | > loss_gen: 2.3942551612854004 (2.3942551612854004) | > loss_kl: 1.926943063735962 (1.926943063735962) | > loss_feat: 4.307457447052002 (4.307457447052002) | > loss_mel: 17.43242073059082 (17.43242073059082) | > loss_duration: 1.5223191976547241 (1.5223191976547241) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.58339500427246 (27.58339500427246) | > grad_norm_1: tensor(7704.2573, device='cuda:0') (tensor(7704.2573, device='cuda:0')) | > current_lr_0: 0.00018019882964187596 | > current_lr_1: 0.00018019882964187596 | > step_time: 0.4307 (0.43073463439941406) | > loader_time: 0.0058 (0.005818843841552734)  --> TIME: 2025-02-25 16:48:38 -- STEP: 51/72 -- GLOBAL_STEP: 1060100 | > loss_disc: 2.7185676097869873 (2.598950381372489) | > loss_disc_real_0: 0.08345487713813782 (0.09990521812555837) | > loss_disc_real_1: 0.30204465985298157 (0.21901139879927917) | > loss_disc_real_2: 0.21689462661743164 (0.2345752669315712) | > loss_disc_real_3: 0.19161024689674377 (0.23820553427817776) | > loss_disc_real_4: 0.18205036222934723 (0.2349732436385809) | > loss_disc_real_5: 0.25878599286079407 (0.24679140629721621) | > loss_0: 2.7185676097869873 (2.598950381372489) | > grad_norm_0: tensor(37.9165, device='cuda:0') (tensor(270.7342, device='cuda:0')) | > loss_gen: 2.2825562953948975 (2.161547410721872) | > loss_kl: 2.1663920879364014 (1.9840489018197154) | > loss_feat: 4.100695610046387 (3.8651181202308806) | > loss_mel: 16.344921112060547 (16.907053835251748) | > loss_duration: 1.5866315364837646 (1.537414466633516) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.481197357177734 (26.45518284217984) | > grad_norm_1: tensor(7806.4395, device='cuda:0') (tensor(9317.8105, device='cuda:0')) | > current_lr_0: 0.00018019882964187596 | > current_lr_1: 0.00018019882964187596 | > step_time: 0.4511 (0.4467947483062744) | > loader_time: 0.0089 (0.007870833079020182) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722134039435584 (-5.7448803539011245e-05) | > avg_step_time: 0.45731533077401176 (-0.002470328774250685) | > avg_loss_disc: 2.614395040861318 (+0.05903186260814408) | > avg_loss_disc_real_0: 0.1127442864465042 (+0.01466268558107632) | > avg_loss_disc_real_1: 0.21962921468304916 (-0.003343978608158271) | > avg_loss_disc_real_2: 0.24091459411970326 (+0.011923997335030995) | > avg_loss_disc_real_3: 0.23821262973295132 (+0.0016128618112752346) | > avg_loss_disc_real_4: 0.24595511904065037 (+0.003053034275350419) | > avg_loss_disc_real_5: 0.2458792469451125 (+2.9377324480384104e-05) | > avg_loss_0: 2.614395040861318 (+0.05903186260814408) | > avg_grad_norm_0: tensor(276.9464, device='cuda:0') (+tensor(6.1092, device='cuda:0')) | > avg_loss_gen: 2.1768320311962714 (-0.002110644125603134) | > avg_loss_kl: 2.042199737589123 (+0.03593798086676703) | > avg_loss_feat: 3.79028092639547 (-0.020121947140760543) | > avg_loss_mel: 16.997619588610146 (+0.1688362578271132) | > avg_loss_duration: 1.559299111366272 (+0.005340149704839092) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.56623147239148 (+0.18788203387193292) | > avg_grad_norm_1: tensor(9177.7559, device='cuda:0') (tensor(-631.4062, device='cuda:0'))  > EPOCH: 835/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:48:50)   --> TIME: 2025-02-25 16:49:05 -- STEP: 29/72 -- GLOBAL_STEP: 1060150 | > loss_disc: 2.6828250885009766 (2.5858954725594354) | > loss_disc_real_0: 0.08361487090587616 (0.09681833509741158) | > loss_disc_real_1: 0.22666771709918976 (0.2205397974828194) | > loss_disc_real_2: 0.25755828619003296 (0.21087163481219062) | > loss_disc_real_3: 0.22977866232395172 (0.23051705288475957) | > loss_disc_real_4: 0.2175581306219101 (0.24507416116780248) | > loss_disc_real_5: 0.26288822293281555 (0.24410550193539982) | > loss_0: 2.6828250885009766 (2.5858954725594354) | > grad_norm_0: tensor(176.3827, device='cuda:0') (tensor(238.4106, device='cuda:0')) | > loss_gen: 1.9897006750106812 (2.132703205634808) | > loss_kl: 2.0518765449523926 (2.00195338808257) | > loss_feat: 3.6178181171417236 (3.872430414989077) | > loss_mel: 17.388269424438477 (16.97451275792615) | > loss_duration: 1.5103973150253296 (1.5162921897296249) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.558061599731445 (26.4978918535956) | > grad_norm_1: tensor(7573.5464, device='cuda:0') (tensor(8683.0869, device='cuda:0')) | > current_lr_0: 0.0001801763047881707 | > current_lr_1: 0.0001801763047881707 | > step_time: 0.4567 (0.43675501593228044) | > loader_time: 0.008 (0.006952721497108196) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598257118547467 (-0.0001238769208881177) | > avg_step_time: 0.4557708048484695 (-0.0015445259255422727) | > avg_loss_disc: 2.6094268577199586 (-0.0049681831413592725) | > avg_loss_disc_real_0: 0.10435632605787735 (-0.008387960388626839) | > avg_loss_disc_real_1: 0.23199985564594539 (+0.012370640962896223) | > avg_loss_disc_real_2: 0.22412726681836895 (-0.01678732730133431) | > avg_loss_disc_real_3: 0.2348342237757965 (-0.0033784059571548286) | > avg_loss_disc_real_4: 0.2462940165694331 (+0.00033889752878271673) | > avg_loss_disc_real_5: 0.24536149056864456 (-0.0005177563764679483) | > avg_loss_0: 2.6094268577199586 (-0.0049681831413592725) | > avg_grad_norm_0: tensor(237.7948, device='cuda:0') (tensor(-39.1515, device='cuda:0')) | > avg_loss_gen: 2.156182240432417 (-0.020649790763854536) | > avg_loss_kl: 2.0591893766967337 (+0.016989639107610532) | > avg_loss_feat: 3.7766661744722176 (-0.013614751923252566) | > avg_loss_mel: 16.878828854628008 (-0.11879073398213791) | > avg_loss_duration: 1.5572651876530177 (-0.002033923713254282) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.428131855709452 (-0.13809961668202675) | > avg_grad_norm_1: tensor(8432.9580, device='cuda:0') (tensor(-744.7979, device='cuda:0'))  > EPOCH: 836/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:49:27)   --> TIME: 2025-02-25 16:49:32 -- STEP: 7/72 -- GLOBAL_STEP: 1060200 | > loss_disc: 2.3124313354492188 (2.5451034137180875) | > loss_disc_real_0: 0.1101771742105484 (0.10581060552171298) | > loss_disc_real_1: 0.24025247991085052 (0.23913728977952683) | > loss_disc_real_2: 0.19049103558063507 (0.21982241528374807) | > loss_disc_real_3: 0.22502665221691132 (0.2460397822516305) | > loss_disc_real_4: 0.21784450113773346 (0.24511071826730454) | > loss_disc_real_5: 0.2139529138803482 (0.23479111705507552) | > loss_0: 2.3124313354492188 (2.5451034137180875) | > grad_norm_0: tensor(200.5061, device='cuda:0') (tensor(231.4003, device='cuda:0')) | > loss_gen: 2.563041925430298 (2.247896807534354) | > loss_kl: 1.9696359634399414 (1.9538723741258894) | > loss_feat: 4.67658805847168 (3.873549120766776) | > loss_mel: 17.63138198852539 (16.943991252354213) | > loss_duration: 1.5184142589569092 (1.51286666733878) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.35906219482422 (26.532175881522043) | > grad_norm_1: tensor(8691.0977, device='cuda:0') (tensor(7456.9663, device='cuda:0')) | > current_lr_0: 0.00018015378275007218 | > current_lr_1: 0.00018015378275007218 | > step_time: 0.4521 (0.4349045753479004) | > loader_time: 0.0056 (0.006510121481759208)  --> TIME: 2025-02-25 16:49:55 -- STEP: 57/72 -- GLOBAL_STEP: 1060250 | > loss_disc: 3.1688554286956787 (2.60867622860691) | > loss_disc_real_0: 0.08953322470188141 (0.12892863687062467) | > loss_disc_real_1: 0.18329322338104248 (0.23204242973996883) | > loss_disc_real_2: 0.4058547019958496 (0.23336085232726314) | > loss_disc_real_3: 0.2998391091823578 (0.2352914540914067) | > loss_disc_real_4: 0.2776058614253998 (0.24222655228355475) | > loss_disc_real_5: 0.25669410824775696 (0.2419652149342654) | > loss_0: 3.1688554286956787 (2.60867622860691) | > grad_norm_0: tensor(190.2806, device='cuda:0') (tensor(228.3737, device='cuda:0')) | > loss_gen: 1.7210142612457275 (2.18884317498458) | > loss_kl: 2.1482956409454346 (2.050293763478597) | > loss_feat: 3.411076784133911 (3.7327289413987543) | > loss_mel: 17.115694046020508 (16.991624681573168) | > loss_duration: 1.6061043739318848 (1.5498708800265664) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.00218391418457 (26.513361445644446) | > grad_norm_1: tensor(3505.4182, device='cuda:0') (tensor(7062.2622, device='cuda:0')) | > current_lr_0: 0.00018015378275007218 | > current_lr_1: 0.00018015378275007218 | > step_time: 0.4586 (0.4435740353768332) | > loader_time: 0.0095 (0.008082301993119089) --> EVAL PERFORMANCE | > avg_loader_time: 0.008705283554506973 (+0.00010702643595950599) | > avg_step_time: 0.45333506691623743 (-0.0024357379322320627) | > avg_loss_disc: 2.6515953171421103 (+0.04216845942215164) | > avg_loss_disc_real_0: 0.14774324868241662 (+0.04338692262453926) | > avg_loss_disc_real_1: 0.2359429767853777 (+0.003943121139432321) | > avg_loss_disc_real_2: 0.23011596081122546 (+0.005988693992856509) | > avg_loss_disc_real_3: 0.244446627061132 (+0.009612403285335497) | > avg_loss_disc_real_4: 0.25539376492231664 (+0.009099748352883547) | > avg_loss_disc_real_5: 0.2467589065642424 (+0.0013974159955978394) | > avg_loss_0: 2.6515953171421103 (+0.04216845942215164) | > avg_grad_norm_0: tensor(231.8679, device='cuda:0') (tensor(-5.9270, device='cuda:0')) | > avg_loss_gen: 2.183523199927638 (+0.027340959495221195) | > avg_loss_kl: 2.1034071109664274 (+0.044217734269693665) | > avg_loss_feat: 3.6320242747454574 (-0.14464189972676023) | > avg_loss_mel: 17.077211688941638 (+0.19838283431363024) | > avg_loss_duration: 1.5654592463667965 (+0.008194058713778762) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.561625520947953 (+0.13349366523850037) | > avg_grad_norm_1: tensor(6921.8896, device='cuda:0') (tensor(-1511.0684, device='cuda:0'))  > EPOCH: 837/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:50:04)   --> TIME: 2025-02-25 16:50:22 -- STEP: 35/72 -- GLOBAL_STEP: 1060300 | > loss_disc: 2.531487464904785 (2.7016226291656493) | > loss_disc_real_0: 0.11416872590780258 (0.12633491701313426) | > loss_disc_real_1: 0.24426184594631195 (0.20622806623578072) | > loss_disc_real_2: 0.2480897307395935 (0.2663688263722828) | > loss_disc_real_3: 0.2673322856426239 (0.24836234322616033) | > loss_disc_real_4: 0.2833296060562134 (0.23831212776047841) | > loss_disc_real_5: 0.3115449845790863 (0.25650840742247444) | > loss_0: 2.531487464904785 (2.7016226291656493) | > grad_norm_0: tensor(133.4822, device='cuda:0') (tensor(224.4641, device='cuda:0')) | > loss_gen: 2.2817070484161377 (2.0929731948035104) | > loss_kl: 2.3601737022399902 (2.776109508105687) | > loss_feat: 3.6525988578796387 (3.6311646461486817) | > loss_mel: 17.623531341552734 (19.858935601370675) | > loss_duration: 1.6025861501693726 (1.5860598121370588) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.520597457885742 (29.94524269104004) | > grad_norm_1: tensor(12796.2080, device='cuda:0') (tensor(8566.5576, device='cuda:0')) | > current_lr_0: 0.0001801312635272284 | > current_lr_1: 0.0001801312635272284 | > step_time: 0.4639 (0.44014171872820174) | > loader_time: 0.0085 (0.007329675129481725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008780106692246988 (+7.482313774001548e-05) | > avg_step_time: 0.45847438086926096 (+0.005139313953023528) | > avg_loss_disc: 2.6928261703168843 (+0.04123085317477404) | > avg_loss_disc_real_0: 0.13040347932509974 (-0.017339769357316875) | > avg_loss_disc_real_1: 0.21706386387977802 (-0.018879112905599682) | > avg_loss_disc_real_2: 0.2529266304533247 (+0.02281066964209924) | > avg_loss_disc_real_3: 0.2412432784765539 (-0.003203348584578075) | > avg_loss_disc_real_4: 0.23799940335078978 (-0.01739436157152685) | > avg_loss_disc_real_5: 0.24841685761028628 (+0.0016579510460438818) | > avg_loss_0: 2.6928261703168843 (+0.04123085317477404) | > avg_grad_norm_0: tensor(213.9999, device='cuda:0') (tensor(-17.8680, device='cuda:0')) | > avg_loss_gen: 2.0477605050718273 (-0.13576269485581083) | > avg_loss_kl: 2.4707918150324217 (+0.3673847040659943) | > avg_loss_feat: 3.4726934130762666 (-0.15933086166919086) | > avg_loss_mel: 18.56702995300293 (+1.4898182640612916) | > avg_loss_duration: 1.5992098862016704 (+0.03375063983487392) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.15748553208902 (+1.5958600111410668) | > avg_grad_norm_1: tensor(7340.4053, device='cuda:0') (+tensor(418.5156, device='cuda:0'))  > EPOCH: 838/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:50:41)   --> TIME: 2025-02-25 16:50:49 -- STEP: 13/72 -- GLOBAL_STEP: 1060350 | > loss_disc: 2.5837953090667725 (2.588019004234901) | > loss_disc_real_0: 0.07147103548049927 (0.09766595810651779) | > loss_disc_real_1: 0.47806429862976074 (0.28151624305890155) | > loss_disc_real_2: 0.18615876138210297 (0.23070310400082514) | > loss_disc_real_3: 0.20966172218322754 (0.22513509484437796) | > loss_disc_real_4: 0.2314576357603073 (0.24310922164183396) | > loss_disc_real_5: 0.21370980143547058 (0.2419484292085354) | > loss_0: 2.5837953090667725 (2.588019004234901) | > grad_norm_0: tensor(215.5557, device='cuda:0') (tensor(172.9876, device='cuda:0')) | > loss_gen: 2.278069257736206 (2.2696306705474854) | > loss_kl: 2.186302900314331 (2.350466911609357) | > loss_feat: 4.165876388549805 (4.035849571228027) | > loss_mel: 18.06297492980957 (19.318088091336765) | > loss_duration: 1.4928979873657227 (1.5694936788999116) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.186119079589844 (29.543528630183292) | > grad_norm_1: tensor(5275.2178, device='cuda:0') (tensor(4666.7075, device='cuda:0')) | > current_lr_0: 0.0001801087471192875 | > current_lr_1: 0.0001801087471192875 | > step_time: 0.4586 (0.4331306494199313) | > loader_time: 0.0077 (0.006342502740713267)  --> TIME: 2025-02-25 16:51:12 -- STEP: 63/72 -- GLOBAL_STEP: 1060400 | > loss_disc: 2.550421953201294 (2.5814111384134444) | > loss_disc_real_0: 0.09029978513717651 (0.07384266651102475) | > loss_disc_real_1: 0.2128640115261078 (0.24354600066703458) | > loss_disc_real_2: 0.27038121223449707 (0.2472570453371321) | > loss_disc_real_3: 0.2397596538066864 (0.22218020993565757) | > loss_disc_real_4: 0.2424032986164093 (0.2381914515816976) | > loss_disc_real_5: 0.2656504809856415 (0.24542022981340922) | > loss_0: 2.550421953201294 (2.5814111384134444) | > grad_norm_0: tensor(248.9789, device='cuda:0') (tensor(186.2057, device='cuda:0')) | > loss_gen: 2.0138962268829346 (2.172913180457221) | > loss_kl: 2.0141050815582275 (2.1385560451991976) | > loss_feat: 3.1107630729675293 (3.849978412900652) | > loss_mel: 17.163555145263672 (17.72338709755549) | > loss_duration: 1.5849833488464355 (1.5693897557637049) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.887304306030273 (27.45422457134913) | > grad_norm_1: tensor(8357.5205, device='cuda:0') (tensor(6379.1841, device='cuda:0')) | > current_lr_0: 0.0001801087471192875 | > current_lr_1: 0.0001801087471192875 | > step_time: 0.5048 (0.44689629191444036) | > loader_time: 0.0115 (0.008182200174483043) --> EVAL PERFORMANCE | > avg_loader_time: 0.00856674221200003 (-0.0002133644802469574) | > avg_step_time: 0.4539674402962268 (-0.004506940573034135) | > avg_loss_disc: 2.574998932825008 (-0.11782723749187651) | > avg_loss_disc_real_0: 0.07409335510201857 (-0.05631012422308117) | > avg_loss_disc_real_1: 0.2412731197308486 (+0.024209255851070588) | > avg_loss_disc_real_2: 0.24712641369289082 (-0.0058002167604338795) | > avg_loss_disc_real_3: 0.22288620870717815 (-0.01835706976937576) | > avg_loss_disc_real_4: 0.23859710949407498 (+0.0005977061432851927) | > avg_loss_disc_real_5: 0.2462586370152487 (-0.002158220595037591) | > avg_loss_0: 2.574998932825008 (-0.11782723749187651) | > avg_grad_norm_0: tensor(185.8084, device='cuda:0') (tensor(-28.1915, device='cuda:0')) | > avg_loss_gen: 2.1753553471095124 (+0.12759484203768512) | > avg_loss_kl: 2.132772423851658 (-0.33801939118076385) | > avg_loss_feat: 3.8295680301290163 (+0.3568746170527497) | > avg_loss_mel: 17.68290645975462 (-0.8841234932483104) | > avg_loss_duration: 1.5755875328896751 (-0.023622353311995248) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.396189864252655 (-0.7612956678363645) | > avg_grad_norm_1: tensor(6382.6401, device='cuda:0') (tensor(-957.7651, device='cuda:0'))  > EPOCH: 839/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:51:18)   --> TIME: 2025-02-25 16:51:39 -- STEP: 41/72 -- GLOBAL_STEP: 1060450 | > loss_disc: 2.532301664352417 (2.504134550327208) | > loss_disc_real_0: 0.1085728108882904 (0.0906269523973872) | > loss_disc_real_1: 0.2825812101364136 (0.22350879958490047) | > loss_disc_real_2: 0.24004128575325012 (0.2136433564308213) | > loss_disc_real_3: 0.24448686838150024 (0.22825770952352664) | > loss_disc_real_4: 0.24732793867588043 (0.23373868043829754) | > loss_disc_real_5: 0.2386709451675415 (0.2444180075715228) | > loss_0: 2.532301664352417 (2.504134550327208) | > grad_norm_0: tensor(285.7584, device='cuda:0') (tensor(210.7830, device='cuda:0')) | > loss_gen: 2.4731853008270264 (2.2539619439985694) | > loss_kl: 1.9120231866836548 (1.9333709245774804) | > loss_feat: 4.790711879730225 (4.003018390841601) | > loss_mel: 16.834136962890625 (17.15357082646067) | > loss_duration: 1.5740050077438354 (1.5297005438223117) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.584062576293945 (26.87362261516292) | > grad_norm_1: tensor(4488.3008, device='cuda:0') (tensor(6606.4893, device='cuda:0')) | > current_lr_0: 0.00018008623352589758 | > current_lr_1: 0.00018008623352589758 | > step_time: 0.4434 (0.43752048655254083) | > loader_time: 0.0089 (0.007408031603185142) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605188047382195 (+3.8445835382164284e-05) | > avg_step_time: 0.45406946330003334 (+0.0001020230038065173) | > avg_loss_disc: 2.572540269771092 (-0.002458663053915622) | > avg_loss_disc_real_0: 0.13034628856349997 (+0.0562529334614814) | > avg_loss_disc_real_1: 0.22021622966292878 (-0.02105689006791983) | > avg_loss_disc_real_2: 0.21303496982010317 (-0.03409144387278765) | > avg_loss_disc_real_3: 0.22562834830351278 (+0.002742139596334625) | > avg_loss_disc_real_4: 0.23275594560193344 (-0.005841163892141538) | > avg_loss_disc_real_5: 0.24397342620601117 (-0.00228521080923752) | > avg_loss_0: 2.572540269771092 (-0.002458663053915622) | > avg_grad_norm_0: tensor(217.3768, device='cuda:0') (+tensor(31.5684, device='cuda:0')) | > avg_loss_gen: 2.222442230708163 (+0.04708688359865043) | > avg_loss_kl: 1.9908762011729495 (-0.14189622267870838) | > avg_loss_feat: 3.8261853305386824 (-0.003382699590333882) | > avg_loss_mel: 16.994479904712087 (-0.6884265550425326) | > avg_loss_duration: 1.557329110696282 (-0.01825842219339302) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.59131278454418 (-0.8048770797084757) | > avg_grad_norm_1: tensor(6230.8115, device='cuda:0') (tensor(-151.8286, device='cuda:0'))  > EPOCH: 840/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:51:55)   --> TIME: 2025-02-25 16:52:05 -- STEP: 19/72 -- GLOBAL_STEP: 1060500 | > loss_disc: 3.1604673862457275 (2.72514934288828) | > loss_disc_real_0: 0.3960256576538086 (0.15551181196382172) | > loss_disc_real_1: 0.18846279382705688 (0.21194992724217868) | > loss_disc_real_2: 0.7973973155021667 (0.38212185549108607) | > loss_disc_real_3: 0.4519660770893097 (0.28630278612438004) | > loss_disc_real_4: 0.18161047995090485 (0.23015567268195905) | > loss_disc_real_5: 0.1929652988910675 (0.23845510263192027) | > loss_0: 3.1604673862457275 (2.72514934288828) | > grad_norm_0: tensor(242.4401, device='cuda:0') (tensor(256.1173, device='cuda:0')) | > loss_gen: 3.221585988998413 (2.4999152296467826) | > loss_kl: 1.98232102394104 (1.9037037021235417) | > loss_feat: 4.236077308654785 (4.007916789305837) | > loss_mel: 17.50476837158203 (17.14339958994012) | > loss_duration: 1.500978708267212 (1.510512628053364) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.44573211669922 (27.065448058278935) | > grad_norm_1: tensor(9222.1406, device='cuda:0') (tensor(6393.2896, device='cuda:0')) | > current_lr_0: 0.00018006372274670683 | > current_lr_1: 0.00018006372274670683 | > step_time: 0.4297 (0.4333861878043727) | > loader_time: 0.006 (0.0068576461390445105)  --> TIME: 2025-02-25 16:52:29 -- STEP: 69/72 -- GLOBAL_STEP: 1060550 | > loss_disc: 2.8071084022521973 (2.7578873876212295) | > loss_disc_real_0: 0.149715393781662 (0.20008778199553492) | > loss_disc_real_1: 0.23410393297672272 (0.2275201275512792) | > loss_disc_real_2: 0.2164221704006195 (0.2758253590352293) | > loss_disc_real_3: 0.23396161198616028 (0.25785625067310075) | > loss_disc_real_4: 0.21473342180252075 (0.25997042720732483) | > loss_disc_real_5: 0.23618896305561066 (0.25015248250270233) | > loss_0: 2.8071084022521973 (2.7578873876212295) | > grad_norm_0: tensor(414.3858, device='cuda:0') (tensor(235.1368, device='cuda:0')) | > loss_gen: 1.969344139099121 (2.240899996481081) | > loss_kl: 2.2221176624298096 (2.0118028702943223) | > loss_feat: 3.91327166557312 (3.657585759093796) | > loss_mel: 17.360116958618164 (17.068175647569745) | > loss_duration: 1.664172887802124 (1.558259571807972) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.129024505615234 (26.536723897077035) | > grad_norm_1: tensor(11392.0703, device='cuda:0') (tensor(6054.1724, device='cuda:0')) | > current_lr_0: 0.00018006372274670683 | > current_lr_1: 0.00018006372274670683 | > step_time: 0.5408 (0.4549724364626235) | > loader_time: 0.0125 (0.008516733197198397) --> EVAL PERFORMANCE | > avg_loader_time: 0.008569049163603445 (-3.613888377874963e-05) | > avg_step_time: 0.4561395678721683 (+0.0020701045721349676) | > avg_loss_disc: 2.7627623786388984 (+0.19022210886780622) | > avg_loss_disc_real_0: 0.19641485999167807 (+0.0660685714281781) | > avg_loss_disc_real_1: 0.2279037177352838 (+0.007687488072355031) | > avg_loss_disc_real_2: 0.2743178862081447 (+0.06128291638804151) | > avg_loss_disc_real_3: 0.25783643063525097 (+0.03220808233173819) | > avg_loss_disc_real_4: 0.2593699811210095 (+0.026614035519076068) | > avg_loss_disc_real_5: 0.2503636970486439 (+0.006390270842632745) | > avg_loss_0: 2.7627623786388984 (+0.19022210886780622) | > avg_grad_norm_0: tensor(232.3061, device='cuda:0') (+tensor(14.9293, device='cuda:0')) | > avg_loss_gen: 2.2276340017856016 (+0.005191771077438734) | > avg_loss_kl: 2.016398520536826 (+0.025522319363876278) | > avg_loss_feat: 3.65489811964438 (-0.17128721089430243) | > avg_loss_mel: 17.077426332822995 (+0.08294642811090824) | > avg_loss_duration: 1.559084741162582 (+0.0017556304663000066) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.535441761285487 (-0.05587102325869253) | > avg_grad_norm_1: tensor(6106.0835, device='cuda:0') (tensor(-124.7280, device='cuda:0'))  > EPOCH: 841/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:52:32)   --> TIME: 2025-02-25 16:52:56 -- STEP: 47/72 -- GLOBAL_STEP: 1060600 | > loss_disc: 2.689176559448242 (2.6849505444790456) | > loss_disc_real_0: 0.18267977237701416 (0.1621913239359856) | > loss_disc_real_1: 0.22819745540618896 (0.2288434882747366) | > loss_disc_real_2: 0.1807006150484085 (0.22988780191604127) | > loss_disc_real_3: 0.22709870338439941 (0.24036176946568996) | > loss_disc_real_4: 0.24684423208236694 (0.24340357552183436) | > loss_disc_real_5: 0.2763969600200653 (0.24938219975917894) | > loss_0: 2.689176559448242 (2.6849505444790456) | > grad_norm_0: tensor(229.2447, device='cuda:0') (tensor(191.6729, device='cuda:0')) | > loss_gen: 1.9129371643066406 (2.039630253264244) | > loss_kl: 2.0998587608337402 (2.1038338605393765) | > loss_feat: 3.315012216567993 (3.4517000533164817) | > loss_mel: 17.0034236907959 (17.045915867419957) | > loss_duration: 1.5969669818878174 (1.5440258599342185) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.928199768066406 (26.185105952810733) | > grad_norm_1: tensor(11532.0645, device='cuda:0') (tensor(6983.2739, device='cuda:0')) | > current_lr_0: 0.0001800412147813635 | > current_lr_1: 0.0001800412147813635 | > step_time: 0.4542 (0.45789247370780783) | > loader_time: 0.0091 (0.007658973653265771) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865958106349891 (+9.053189989546503e-05) | > avg_step_time: 0.46381452050007566 (+0.00767495262790735) | > avg_loss_disc: 2.6924019329984423 (-0.07036044564045607) | > avg_loss_disc_real_0: 0.16258719699903273 (-0.033827662992645335) | > avg_loss_disc_real_1: 0.22913182306457575 (+0.0012281053292919342) | > avg_loss_disc_real_2: 0.23457980638658496 (-0.03973807982155972) | > avg_loss_disc_real_3: 0.23848320417840715 (-0.019353226456843814) | > avg_loss_disc_real_4: 0.2437909417588946 (-0.01557903936211491) | > avg_loss_disc_real_5: 0.24606702550196308 (-0.004296671546680841) | > avg_loss_0: 2.6924019329984423 (-0.07036044564045607) | > avg_grad_norm_0: tensor(183.0052, device='cuda:0') (tensor(-49.3009, device='cuda:0')) | > avg_loss_gen: 2.0184742850317066 (-0.209159716753895) | > avg_loss_kl: 2.1437429827703562 (+0.12734446223353046) | > avg_loss_feat: 3.322139907890642 (-0.33275821175373776) | > avg_loss_mel: 17.109511738092134 (+0.0320854052691395) | > avg_loss_duration: 1.567649678445198 (+0.00856493728261598) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.161518634205134 (-0.3739231270803529) | > avg_grad_norm_1: tensor(7313.6250, device='cuda:0') (+tensor(1207.5415, device='cuda:0'))  > EPOCH: 842/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:53:10)   --> TIME: 2025-02-25 16:53:23 -- STEP: 25/72 -- GLOBAL_STEP: 1060650 | > loss_disc: 2.9171133041381836 (2.7523841094970702) | > loss_disc_real_0: 0.5506440997123718 (0.24094905763864516) | > loss_disc_real_1: 0.20731331408023834 (0.21982325315475465) | > loss_disc_real_2: 0.20546592772006989 (0.22706098794937135) | > loss_disc_real_3: 0.21818289160728455 (0.23912681341171257) | > loss_disc_real_4: 0.22196365892887115 (0.2442393833398819) | > loss_disc_real_5: 0.23094891011714935 (0.24868793666362765) | > loss_0: 2.9171133041381836 (2.7523841094970702) | > grad_norm_0: tensor(80.6964, device='cuda:0') (tensor(176.7391, device='cuda:0')) | > loss_gen: 2.046170711517334 (2.03352505683899) | > loss_kl: 2.305527925491333 (2.052517442703247) | > loss_feat: 3.0765323638916016 (3.3296830081939697) | > loss_mel: 18.03379249572754 (17.15360511779785) | > loss_duration: 1.5334031581878662 (1.5208406257629394) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.995426177978516 (26.090171279907228) | > grad_norm_1: tensor(619.2446, device='cuda:0') (tensor(5255.7710, device='cuda:0')) | > current_lr_0: 0.00018001870962951582 | > current_lr_1: 0.00018001870962951582 | > step_time: 0.4305 (0.4378360366821289) | > loader_time: 0.0071 (0.007163076400756836) --> EVAL PERFORMANCE | > avg_loader_time: 0.00875600962571695 (+9.642856221804032e-05) | > avg_step_time: 0.45614648872697855 (-0.007668031773097106) | > avg_loss_disc: 2.7879556333515008 (+0.09555370035305844) | > avg_loss_disc_real_0: 0.19494498195782514 (+0.03235778495879241) | > avg_loss_disc_real_1: 0.22979786194546123 (+0.0006660388808854834) | > avg_loss_disc_real_2: 0.2495494772430876 (+0.014969670856502637) | > avg_loss_disc_real_3: 0.2481973565800089 (+0.009714152401601756) | > avg_loss_disc_real_4: 0.24792825084337047 (+0.004137309084475876) | > avg_loss_disc_real_5: 0.2482181793787111 (+0.0021511538767480243) | > avg_loss_0: 2.7879556333515008 (+0.09555370035305844) | > avg_grad_norm_0: tensor(156.1126, device='cuda:0') (tensor(-26.8925, device='cuda:0')) | > avg_loss_gen: 2.0026119745952977 (-0.015862310436408844) | > avg_loss_kl: 2.3170589863414497 (+0.17331600357109345) | > avg_loss_feat: 3.3198210487903004 (-0.002318859100341797) | > avg_loss_mel: 17.880226377030493 (+0.7707146389383581) | > avg_loss_duration: 1.5933092631084818 (+0.025659584663283663) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.11302765322403 (+0.9515090190188964) | > avg_grad_norm_1: tensor(4340.6206, device='cuda:0') (tensor(-2973.0044, device='cuda:0'))  > EPOCH: 843/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:53:47)   --> TIME: 2025-02-25 16:53:50 -- STEP: 3/72 -- GLOBAL_STEP: 1060700 | > loss_disc: 2.485480546951294 (2.5203612645467124) | > loss_disc_real_0: 0.06653481721878052 (0.1460036834081014) | > loss_disc_real_1: 0.17563657462596893 (0.20163686573505402) | > loss_disc_real_2: 0.2048058658838272 (0.21865020195643106) | > loss_disc_real_3: 0.219478577375412 (0.23310155669848123) | > loss_disc_real_4: 0.21049797534942627 (0.23395796120166779) | > loss_disc_real_5: 0.21570920944213867 (0.23273521661758423) | > loss_0: 2.485480546951294 (2.5203612645467124) | > grad_norm_0: tensor(53.7271, device='cuda:0') (tensor(258.8507, device='cuda:0')) | > loss_gen: 1.915820598602295 (2.2073941230773926) | > loss_kl: 2.2691807746887207 (2.149492343266805) | > loss_feat: 3.932813882827759 (3.7480575243631997) | > loss_mel: 18.707674026489258 (17.96005630493164) | > loss_duration: 1.577378511428833 (1.5658645629882812) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.402868270874023 (27.6308650970459) | > grad_norm_1: tensor(5733.3457, device='cuda:0') (tensor(5377.7803, device='cuda:0')) | > current_lr_0: 0.00017999620729081212 | > current_lr_1: 0.00017999620729081212 | > step_time: 0.4197 (0.4263617197672526) | > loader_time: 0.0056 (0.006490151087443034)  --> TIME: 2025-02-25 16:54:13 -- STEP: 53/72 -- GLOBAL_STEP: 1060750 | > loss_disc: 2.52726411819458 (2.7247476307850964) | > loss_disc_real_0: 0.13087357580661774 (0.14727671212464008) | > loss_disc_real_1: 0.19603228569030762 (0.2325316380779698) | > loss_disc_real_2: 0.20414942502975464 (0.24978814681745914) | > loss_disc_real_3: 0.2395888715982437 (0.23753852242568754) | > loss_disc_real_4: 0.24999766051769257 (0.2443151648314494) | > loss_disc_real_5: 0.26320743560791016 (0.24283000433220053) | > loss_0: 2.52726411819458 (2.7247476307850964) | > grad_norm_0: tensor(263.2070, device='cuda:0') (tensor(204.3263, device='cuda:0')) | > loss_gen: 2.0424559116363525 (2.0100769861689156) | > loss_kl: 2.0182876586914062 (2.0397664331040297) | > loss_feat: 3.637063503265381 (3.43428214091175) | > loss_mel: 17.176605224609375 (17.29315455454701) | > loss_duration: 1.6054332256317139 (1.5558241718220263) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.47984504699707 (26.33310433153836) | > grad_norm_1: tensor(16570.5137, device='cuda:0') (tensor(6453.9644, device='cuda:0')) | > current_lr_0: 0.00017999620729081212 | > current_lr_1: 0.00017999620729081212 | > step_time: 0.4463 (0.44131395501910514) | > loader_time: 0.0092 (0.0078803503288413) --> EVAL PERFORMANCE | > avg_loader_time: 0.008683053540511868 (-7.295608520508333e-05) | > avg_step_time: 0.45451190773869904 (-0.0016345809882795104) | > avg_loss_disc: 2.7122201885975588 (-0.075735444753942) | > avg_loss_disc_real_0: 0.1432433653045708 (-0.051701616653254345) | > avg_loss_disc_real_1: 0.22714851334900923 (-0.0026493485964519947) | > avg_loss_disc_real_2: 0.24526028398057095 (-0.004289193262516644) | > avg_loss_disc_real_3: 0.23763575982040083 (-0.010561596759608077) | > avg_loss_disc_real_4: 0.24425666013234099 (-0.0036715907110294876) | > avg_loss_disc_real_5: 0.2459388681700532 (-0.0022793112086579026) | > avg_loss_0: 2.7122201885975588 (-0.075735444753942) | > avg_grad_norm_0: tensor(187.7220, device='cuda:0') (+tensor(31.6094, device='cuda:0')) | > avg_loss_gen: 2.004759924512514 (+0.0021479499172163408) | > avg_loss_kl: 2.0749676513000277 (-0.242091335041422) | > avg_loss_feat: 3.377252736561735 (+0.0574316877714347) | > avg_loss_mel: 17.237570064168587 (-0.6426563128619058) | > avg_loss_duration: 1.5690066898372816 (-0.024302573271200112) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.263557192305445 (-0.8494704609185852) | > avg_grad_norm_1: tensor(6211.4902, device='cuda:0') (+tensor(1870.8696, device='cuda:0'))  > EPOCH: 844/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:54:24)   --> TIME: 2025-02-25 16:54:40 -- STEP: 31/72 -- GLOBAL_STEP: 1060800 | > loss_disc: 2.509512186050415 (2.7644124723249868) | > loss_disc_real_0: 0.20573651790618896 (0.15144712754314946) | > loss_disc_real_1: 0.23986734449863434 (0.2095446346267577) | > loss_disc_real_2: 0.21030473709106445 (0.22284701033946006) | > loss_disc_real_3: 0.2224646359682083 (0.23070342261945048) | > loss_disc_real_4: 0.23553718626499176 (0.2541608142275964) | > loss_disc_real_5: 0.2558594048023224 (0.24752068711865333) | > loss_0: 2.509512186050415 (2.7644124723249868) | > grad_norm_0: tensor(80.8364, device='cuda:0') (tensor(253.8365, device='cuda:0')) | > loss_gen: 2.3768815994262695 (1.9081067462121286) | > loss_kl: 2.326693058013916 (2.245977113323827) | > loss_feat: 3.709293842315674 (3.314895106900123) | > loss_mel: 17.564130783081055 (18.160744267125285) | > loss_duration: 1.5552103519439697 (1.5314605428326515) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.532207489013672 (27.161183818694084) | > grad_norm_1: tensor(4664.3433, device='cuda:0') (tensor(6053.9814, device='cuda:0')) | > current_lr_0: 0.00017997370776490077 | > current_lr_1: 0.00017997370776490077 | > step_time: 0.4515 (0.43730145116006175) | > loader_time: 0.0088 (0.00696799062913464) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613462179479461 (-6.959136103240615e-05) | > avg_step_time: 0.46189615424250213 (+0.0073842465038030936) | > avg_loss_disc: 2.690213189998143 (-0.022006998599415795) | > avg_loss_disc_real_0: 0.143694767492338 (+0.0004514021877672081) | > avg_loss_disc_real_1: 0.22060367150205962 (-0.006544841846949617) | > avg_loss_disc_real_2: 0.22791481626705384 (-0.01734546771351711) | > avg_loss_disc_real_3: 0.23299195732868894 (-0.004643802491711896) | > avg_loss_disc_real_4: 0.2469086735181406 (+0.002652013385799623) | > avg_loss_disc_real_5: 0.24754579889942224 (+0.0016069307293690438) | > avg_loss_0: 2.690213189998143 (-0.022006998599415795) | > avg_grad_norm_0: tensor(191.2630, device='cuda:0') (+tensor(3.5410, device='cuda:0')) | > avg_loss_gen: 2.003377875811617 (-0.0013820487008970872) | > avg_loss_kl: 2.4573714447693087 (+0.382403793469281) | > avg_loss_feat: 3.4025646498505493 (+0.025311913288814214) | > avg_loss_mel: 18.24361652051899 (+1.0060464563504041) | > avg_loss_duration: 1.6043052236798783 (+0.035298533842596624) | > avg_amp_scaler: 9.014084507042254 (+1.014084507042254) | > avg_loss_1: 27.711235610532086 (+1.4476784182266407) | > avg_grad_norm_1: tensor(5543.9834, device='cuda:0') (tensor(-667.5068, device='cuda:0'))  > EPOCH: 845/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:55:01)   --> TIME: 2025-02-25 16:55:07 -- STEP: 9/72 -- GLOBAL_STEP: 1060850 | > loss_disc: 2.64420747756958 (2.6547130743662515) | > loss_disc_real_0: 0.1667303740978241 (0.0908374513188998) | > loss_disc_real_1: 0.23974965512752533 (0.21829279429382747) | > loss_disc_real_2: 0.22652794420719147 (0.22193507850170135) | > loss_disc_real_3: 0.23560795187950134 (0.22914811472098032) | > loss_disc_real_4: 0.25698497891426086 (0.24250635670291054) | > loss_disc_real_5: 0.249556303024292 (0.2430845962630378) | > loss_0: 2.64420747756958 (2.6547130743662515) | > grad_norm_0: tensor(248.7210, device='cuda:0') (tensor(145.6872, device='cuda:0')) | > loss_gen: 2.0627145767211914 (1.9198351966010199) | > loss_kl: 1.8985462188720703 (2.0281087160110474) | > loss_feat: 3.1527204513549805 (3.3124334812164307) | > loss_mel: 16.692434310913086 (17.102556652492947) | > loss_duration: 1.5550942420959473 (1.548726929558648) | > amp_scaler: 16.0 (16.0) | > loss_1: 25.361509323120117 (25.91166051228841) | > grad_norm_1: tensor(9066.0791, device='cuda:0') (tensor(5824.7114, device='cuda:0')) | > current_lr_0: 0.00017995121105143016 | > current_lr_1: 0.00017995121105143016 | > step_time: 0.4271 (0.43188635508219403) | > loader_time: 0.0075 (0.005992147657606337)  --> TIME: 2025-02-25 16:55:31 -- STEP: 59/72 -- GLOBAL_STEP: 1060900 | > loss_disc: 2.7105822563171387 (2.627848637306084) | > loss_disc_real_0: 0.18168126046657562 (0.0975053859716755) | > loss_disc_real_1: 0.20956802368164062 (0.23938533815286928) | > loss_disc_real_2: 0.24121105670928955 (0.22511816037408375) | > loss_disc_real_3: 0.24511785805225372 (0.2356809983819218) | > loss_disc_real_4: 0.248642697930336 (0.23950817903219643) | > loss_disc_real_5: 0.24668452143669128 (0.2438916283643852) | > loss_0: 2.7105822563171387 (2.627848637306084) | > grad_norm_0: tensor(501.0384, device='cuda:0') (tensor(201.2153, device='cuda:0')) | > loss_gen: 1.808842420578003 (2.0575101032095438) | > loss_kl: 2.0762624740600586 (2.1096255678241533) | > loss_feat: 2.831397771835327 (3.563186693999727) | > loss_mel: 17.18059539794922 (17.55587922112417) | > loss_duration: 1.6055389642715454 (1.568061238628323) | > amp_scaler: 8.0 (15.59322033898305) | > loss_1: 25.50263786315918 (26.854263014712576) | > grad_norm_1: tensor(5375.2368, device='cuda:0') (tensor(8080.3955, device='cuda:0')) | > current_lr_0: 0.00017995121105143016 | > current_lr_1: 0.00017995121105143016 | > step_time: 0.4695 (0.4482787180755098) | > loader_time: 0.0099 (0.008002846927966102) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596400140037 (-1.706203944246143e-05) | > avg_step_time: 0.4579813480377197 (-0.003914806204782406) | > avg_loss_disc: 2.6254058858038674 (-0.0648073041942756) | > avg_loss_disc_real_0: 0.0991307246223302 (-0.0445640428700078) | > avg_loss_disc_real_1: 0.23603429382955524 (+0.015430622327495619) | > avg_loss_disc_real_2: 0.22488289368404468 (-0.003031922583009161) | > avg_loss_disc_real_3: 0.23573503695743184 (+0.0027430796287429005) | > avg_loss_disc_real_4: 0.24041266886281296 (-0.006496004655327653) | > avg_loss_disc_real_5: 0.2448337004638054 (-0.0027120984356168443) | > avg_loss_0: 2.6254058858038674 (-0.0648073041942756) | > avg_grad_norm_0: tensor(213.5794, device='cuda:0') (+tensor(22.3164, device='cuda:0')) | > avg_loss_gen: 2.0503473164330064 (+0.04696944062138941) | > avg_loss_kl: 2.1192947357473235 (-0.33807670902198517) | > avg_loss_feat: 3.538439733881346 (+0.1358750840307965) | > avg_loss_mel: 17.427726638149206 (-0.8158898823697847) | > avg_loss_duration: 1.5766156895059937 (-0.027689534173884534) | > avg_amp_scaler: 14.309859154929578 (+5.295774647887324) | > avg_loss_1: 26.712424264827245 (-0.9988113457048406) | > avg_grad_norm_1: tensor(8331.4404, device='cuda:0') (+tensor(2787.4570, device='cuda:0'))  > EPOCH: 846/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:55:39)   --> TIME: 2025-02-25 16:55:57 -- STEP: 37/72 -- GLOBAL_STEP: 1060950 | > loss_disc: 2.7352235317230225 (2.6257765486433695) | > loss_disc_real_0: 0.040389202535152435 (0.12976541410426837) | > loss_disc_real_1: 0.1995375156402588 (0.21810952070597056) | > loss_disc_real_2: 0.24257683753967285 (0.23782022216835538) | > loss_disc_real_3: 0.3030588626861572 (0.23274575093308011) | > loss_disc_real_4: 0.23768270015716553 (0.2393100011993099) | > loss_disc_real_5: 0.2195519059896469 (0.24572478194494504) | > loss_0: 2.7352235317230225 (2.6257765486433695) | > grad_norm_0: tensor(320.8979, device='cuda:0') (tensor(271.4484, device='cuda:0')) | > loss_gen: 1.8526124954223633 (2.1081391218546273) | > loss_kl: 2.0551300048828125 (2.0054919365290047) | > loss_feat: 4.022558689117432 (3.7223501398756698) | > loss_mel: 17.152156829833984 (17.14467937881882) | > loss_duration: 1.564033031463623 (1.5335325582607373) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.64649200439453 (26.514193096676387) | > grad_norm_1: tensor(8041.1743, device='cuda:0') (tensor(6567.3872, device='cuda:0')) | > current_lr_0: 0.00017992871715004873 | > current_lr_1: 0.00017992871715004873 | > step_time: 0.4475 (0.44046340117583405) | > loader_time: 0.0083 (0.007432447897421347) --> EVAL PERFORMANCE | > avg_loader_time: 0.00942301078581474 (+0.0008266106457777405) | > avg_step_time: 0.45642547540261713 (-0.0015558726351025953) | > avg_loss_disc: 2.6493834710456956 (+0.023977585241828248) | > avg_loss_disc_real_0: 0.11857530332996812 (+0.01944457870763791) | > avg_loss_disc_real_1: 0.21902923810649927 (-0.017005055723055967) | > avg_loss_disc_real_2: 0.2439771017558138 (+0.01909420807176912) | > avg_loss_disc_real_3: 0.23740462895850062 (+0.0016695920010687815) | > avg_loss_disc_real_4: 0.24276933472760967 (+0.0023566658647967142) | > avg_loss_disc_real_5: 0.24400049448013306 (-0.0008332059836723416) | > avg_loss_0: 2.6493834710456956 (+0.023977585241828248) | > avg_grad_norm_0: tensor(286.4304, device='cuda:0') (+tensor(72.8510, device='cuda:0')) | > avg_loss_gen: 2.07678501370927 (+0.02643769727626344) | > avg_loss_kl: 2.0382305434052372 (-0.08106419234208628) | > avg_loss_feat: 3.637006769717579 (+0.09856703583623316) | > avg_loss_mel: 16.97230330991074 (-0.4554233282384672) | > avg_loss_duration: 1.5635386268857498 (-0.013077062620243884) | > avg_amp_scaler: 8.0 (-6.309859154929578) | > avg_loss_1: 26.287864282097615 (-0.4245599827296296) | > avg_grad_norm_1: tensor(7424.6479, device='cuda:0') (tensor(-906.7925, device='cuda:0'))  > EPOCH: 847/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:56:16)   --> TIME: 2025-02-25 16:56:24 -- STEP: 15/72 -- GLOBAL_STEP: 1061000 | > loss_disc: 2.3403525352478027 (2.565857728322347) | > loss_disc_real_0: 0.06492622196674347 (0.08806614850958189) | > loss_disc_real_1: 0.2341100126504898 (0.2210919757684072) | > loss_disc_real_2: 0.21043246984481812 (0.21876860558986663) | > loss_disc_real_3: 0.20577675104141235 (0.22867863674958547) | > loss_disc_real_4: 0.24277900159358978 (0.2387912541627884) | > loss_disc_real_5: 0.2390967607498169 (0.24289514223734537) | > loss_0: 2.3403525352478027 (2.565857728322347) | > grad_norm_0: tensor(13.0494, device='cuda:0') (tensor(279.0502, device='cuda:0')) | > loss_gen: 2.10310435295105 (2.0748243729273477) | > loss_kl: 1.748605728149414 (1.9386073271433513) | > loss_feat: 4.363824844360352 (3.7906521638234456) | > loss_mel: 18.03022575378418 (16.849107742309574) | > loss_duration: 1.5044426918029785 (1.5121248642603555) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.750202178955078 (26.165316263834637) | > grad_norm_1: tensor(14866.6201, device='cuda:0') (tensor(10207.5332, device='cuda:0')) | > current_lr_0: 0.00017990622606040496 | > current_lr_1: 0.00017990622606040496 | > step_time: 0.4395 (0.43778061866760254) | > loader_time: 0.009 (0.0068094889322916664)  --> TIME: 2025-02-25 16:56:48 -- STEP: 65/72 -- GLOBAL_STEP: 1061050 | > loss_disc: 2.8442580699920654 (2.595602519695576) | > loss_disc_real_0: 0.09961147606372833 (0.09285033333760041) | > loss_disc_real_1: 0.22055873274803162 (0.2222822293639183) | > loss_disc_real_2: 0.2672276198863983 (0.2236176142325768) | > loss_disc_real_3: 0.25388890504837036 (0.23331196216436534) | > loss_disc_real_4: 0.2444964200258255 (0.24118240154706516) | > loss_disc_real_5: 0.2502000629901886 (0.2477524140706429) | > loss_0: 2.8442580699920654 (2.595602519695576) | > grad_norm_0: tensor(369.9638, device='cuda:0') (tensor(282.2560, device='cuda:0')) | > loss_gen: 1.9394583702087402 (2.083951922563407) | > loss_kl: 2.1821250915527344 (2.001988535660964) | > loss_feat: 2.998051643371582 (3.717094164628249) | > loss_mel: 16.721586227416992 (16.77308621039758) | > loss_duration: 1.6156561374664307 (1.5496694821577806) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.456878662109375 (26.125790287898138) | > grad_norm_1: tensor(9985.2168, device='cuda:0') (tensor(10255.3057, device='cuda:0')) | > current_lr_0: 0.00017990622606040496 | > current_lr_1: 0.00017990622606040496 | > step_time: 0.488 (0.4582239994635949) | > loader_time: 0.0118 (0.008361053466796876) --> EVAL PERFORMANCE | > avg_loader_time: 0.008649033559879787 (-0.0007739772259349534) | > avg_step_time: 0.4626092944346683 (+0.006183819032051174) | > avg_loss_disc: 2.595966459999622 (-0.05341701104607344) | > avg_loss_disc_real_0: 0.0945733540074926 (-0.024001949322475513) | > avg_loss_disc_real_1: 0.22195419994458346 (+0.002924961838084189) | > avg_loss_disc_real_2: 0.22450544590681368 (-0.019471655849000125) | > avg_loss_disc_real_3: 0.2329342440820076 (-0.004470384876493011) | > avg_loss_disc_real_4: 0.2404534743285515 (-0.0023158603990581783) | > avg_loss_disc_real_5: 0.24623385470517925 (+0.0022333602250461937) | > avg_loss_0: 2.595966459999622 (-0.05341701104607344) | > avg_grad_norm_0: tensor(287.7372, device='cuda:0') (+tensor(1.3068, device='cuda:0')) | > avg_loss_gen: 2.0776346485379724 (+0.0008496348287025945) | > avg_loss_kl: 2.0183094622383657 (-0.019921081166871524) | > avg_loss_feat: 3.689894461295974 (+0.052887691578395124) | > avg_loss_mel: 16.772671753252062 (-0.1996315566586766) | > avg_loss_duration: 1.5554296097285312 (-0.008109017157218679) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.11393990315182 (-0.17392437894579516) | > avg_grad_norm_1: tensor(10326.8428, device='cuda:0') (+tensor(2902.1948, device='cuda:0'))  > EPOCH: 848/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:56:53)   --> TIME: 2025-02-25 16:57:15 -- STEP: 43/72 -- GLOBAL_STEP: 1061100 | > loss_disc: 2.540626049041748 (2.5494399680647746) | > loss_disc_real_0: 0.09708259999752045 (0.09577075392007828) | > loss_disc_real_1: 0.21220892667770386 (0.2167660528144171) | > loss_disc_real_2: 0.23224352300167084 (0.2307391703821892) | > loss_disc_real_3: 0.23890568315982819 (0.23292679911436037) | > loss_disc_real_4: 0.2382703572511673 (0.24216937464337016) | > loss_disc_real_5: 0.22690075635910034 (0.24341889106950096) | > loss_0: 2.540626049041748 (2.5494399680647746) | > grad_norm_0: tensor(224.8849, device='cuda:0') (tensor(287.5396, device='cuda:0')) | > loss_gen: 2.291287422180176 (2.168483007785886) | > loss_kl: 1.8423705101013184 (1.951214352319407) | > loss_feat: 4.195367813110352 (3.7846131768337514) | > loss_mel: 16.890024185180664 (16.703588751859435) | > loss_duration: 1.5493872165679932 (1.5284303676250368) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.7684383392334 (26.13632960652196) | > grad_norm_1: tensor(10141.9072, device='cuda:0') (tensor(10087.1895, device='cuda:0')) | > current_lr_0: 0.0001798837377821474 | > current_lr_1: 0.0001798837377821474 | > step_time: 0.4683 (0.4467234722403593) | > loader_time: 0.0095 (0.007641138032425282) --> EVAL PERFORMANCE | > avg_loader_time: 0.008777128139012297 (+0.00012809457913250978) | > avg_step_time: 0.46235547938816984 (-0.0002538150464984623) | > avg_loss_disc: 2.5586887211866784 (-0.03727773881294372) | > avg_loss_disc_real_0: 0.10018127159753316 (+0.005607917590040562) | > avg_loss_disc_real_1: 0.2189150549156565 (-0.003039145028926965) | > avg_loss_disc_real_2: 0.2289560189549352 (+0.00445057304812152) | > avg_loss_disc_real_3: 0.2329261603069977 (-8.083775009909422e-06) | > avg_loss_disc_real_4: 0.24243265915084894 (+0.0019791848222974473) | > avg_loss_disc_real_5: 0.24481034887508607 (-0.0014235058300931802) | > avg_loss_0: 2.5586887211866784 (-0.03727773881294372) | > avg_grad_norm_0: tensor(276.6282, device='cuda:0') (tensor(-11.1090, device='cuda:0')) | > avg_loss_gen: 2.158958737279327 (+0.08132408874135466) | > avg_loss_kl: 2.0032016374695467 (-0.015107824768819) | > avg_loss_feat: 3.7094865181076693 (+0.01959205681169518) | > avg_loss_mel: 16.685941360366176 (-0.08673039288588669) | > avg_loss_duration: 1.5545096414189943 (-0.0009199683095368894) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.112097861061635 (-0.0018420420901854584) | > avg_grad_norm_1: tensor(9772.0039, device='cuda:0') (tensor(-554.8389, device='cuda:0'))  > EPOCH: 849/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:57:31)   --> TIME: 2025-02-25 16:57:42 -- STEP: 21/72 -- GLOBAL_STEP: 1061150 | > loss_disc: 2.560044288635254 (2.5396493389492942) | > loss_disc_real_0: 0.06481264531612396 (0.08848156673567635) | > loss_disc_real_1: 0.18964551389217377 (0.21971402778511956) | > loss_disc_real_2: 0.20564401149749756 (0.22823622822761536) | > loss_disc_real_3: 0.22314512729644775 (0.22868619220597403) | > loss_disc_real_4: 0.24687428772449493 (0.24206352730592093) | > loss_disc_real_5: 0.24423903226852417 (0.24365866893813723) | > loss_0: 2.560044288635254 (2.5396493389492942) | > grad_norm_0: tensor(267.0404, device='cuda:0') (tensor(282.6201, device='cuda:0')) | > loss_gen: 2.0840611457824707 (2.1735942590804327) | > loss_kl: 1.8896743059158325 (1.9004823480333601) | > loss_feat: 4.147838592529297 (3.7854181357792447) | > loss_mel: 16.89456558227539 (16.66543565477644) | > loss_duration: 1.5243260860443115 (1.508024806068057) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.54046630859375 (26.032955078851607) | > grad_norm_1: tensor(9633.5801, device='cuda:0') (tensor(9853.1123, device='cuda:0')) | > current_lr_0: 0.00017986125231492461 | > current_lr_1: 0.00017986125231492461 | > step_time: 0.4347 (0.4413456689743769) | > loader_time: 0.0062 (0.006699562072753906)  --> TIME: 2025-02-25 16:58:06 -- STEP: 71/72 -- GLOBAL_STEP: 1061200 | > loss_disc: 2.7518372535705566 (2.6332502600172867) | > loss_disc_real_0: 0.32318931818008423 (0.13660428164080834) | > loss_disc_real_1: 0.27658188343048096 (0.2332828474086775) | > loss_disc_real_2: 0.20790541172027588 (0.23094864819251315) | > loss_disc_real_3: 0.2540380358695984 (0.23321159624717605) | > loss_disc_real_4: 0.22796393930912018 (0.23941728262834147) | > loss_disc_real_5: 0.20175360143184662 (0.24527330024981162) | > loss_0: 2.7518372535705566 (2.6332502600172867) | > grad_norm_0: tensor(530.0822, device='cuda:0') (tensor(282.7346, device='cuda:0')) | > loss_gen: 2.138296604156494 (2.162083494831139) | > loss_kl: 2.1424829959869385 (2.010884127146761) | > loss_feat: 3.6372950077056885 (3.6851947206846427) | > loss_mel: 15.936929702758789 (16.7907559166492) | > loss_duration: 1.587550163269043 (1.5555491296338364) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.442554473876953 (26.204467451068716) | > grad_norm_1: tensor(9324.0225, device='cuda:0') (tensor(8303.3691, device='cuda:0')) | > current_lr_0: 0.00017986125231492461 | > current_lr_1: 0.00017986125231492461 | > step_time: 0.5039 (0.45720967776338817) | > loader_time: 0.0079 (0.008591759372764912) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591759372764912 (-0.00018536876624738498) | > avg_step_time: 0.45720967776338817 (-0.005145801624781676) | > avg_loss_disc: 2.6332502600172867 (+0.07456153883060823) | > avg_loss_disc_real_0: 0.13660428164080834 (+0.03642301004327518) | > avg_loss_disc_real_1: 0.2332828474086775 (+0.014367792493021003) | > avg_loss_disc_real_2: 0.23094864819251315 (+0.001992629237577953) | > avg_loss_disc_real_3: 0.23321159624717605 (+0.0002854359401783524) | > avg_loss_disc_real_4: 0.23941728262834147 (-0.003015376522507468) | > avg_loss_disc_real_5: 0.24527330024981162 (+0.00046295137472554537) | > avg_loss_0: 2.6332502600172867 (+0.07456153883060823) | > avg_grad_norm_0: tensor(282.7346, device='cuda:0') (+tensor(6.1064, device='cuda:0')) | > avg_loss_gen: 2.162083494831139 (+0.003124757551812074) | > avg_loss_kl: 2.010884127146761 (+0.007682489677214299) | > avg_loss_feat: 3.6851947206846427 (-0.024291797423026562) | > avg_loss_mel: 16.7907559166492 (+0.10481455628302427) | > avg_loss_duration: 1.5555491296338364 (+0.0010394882148421214) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.204467451068716 (+0.09236959000708111) | > avg_grad_norm_1: tensor(8303.3691, device='cuda:0') (tensor(-1468.6348, device='cuda:0'))  > EPOCH: 850/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:58:08)   --> TIME: 2025-02-25 16:58:32 -- STEP: 49/72 -- GLOBAL_STEP: 1061250 | > loss_disc: 2.507275104522705 (2.658995803521604) | > loss_disc_real_0: 0.0837310403585434 (0.10468595319104436) | > loss_disc_real_1: 0.22540044784545898 (0.22042922036988394) | > loss_disc_real_2: 0.2410319298505783 (0.22855291241893966) | > loss_disc_real_3: 0.2375418096780777 (0.23174776866727945) | > loss_disc_real_4: 0.23238208889961243 (0.2462046903615096) | > loss_disc_real_5: 0.24196311831474304 (0.24034567998380077) | > loss_0: 2.507275104522705 (2.658995803521604) | > grad_norm_0: tensor(255.9763, device='cuda:0') (tensor(236.9144, device='cuda:0')) | > loss_gen: 2.224367618560791 (2.0371633257184714) | > loss_kl: 2.245544910430908 (2.538913391074356) | > loss_feat: 3.655513048171997 (3.716730667620289) | > loss_mel: 17.15408706665039 (19.902101594574596) | > loss_duration: 1.6359052658081055 (1.6237660257183775) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.915416717529297 (29.818674982810506) | > grad_norm_1: tensor(12865.3105, device='cuda:0') (tensor(8403.0977, device='cuda:0')) | > current_lr_0: 0.00017983876965838524 | > current_lr_1: 0.00017983876965838524 | > step_time: 0.444 (0.4472997723793497) | > loader_time: 0.0096 (0.007706330747020488) --> EVAL PERFORMANCE | > avg_loader_time: 0.008697043002491266 (+0.00010528362972635381) | > avg_step_time: 0.4600696395820295 (+0.0028599618186413434) | > avg_loss_disc: 2.631170588479915 (-0.0020796715373716523) | > avg_loss_disc_real_0: 0.09873450859646563 (-0.03786977304434272) | > avg_loss_disc_real_1: 0.22558221032082196 (-0.007700637087855539) | > avg_loss_disc_real_2: 0.23406511513699949 (+0.0031164669444863347) | > avg_loss_disc_real_3: 0.23224306169530035 (-0.0009685345518757049) | > avg_loss_disc_real_4: 0.24480286849216681 (+0.005385585863825343) | > avg_loss_disc_real_5: 0.24388510891249482 (-0.0013881913373167964) | > avg_loss_0: 2.631170588479915 (-0.0020796715373716523) | > avg_grad_norm_0: tensor(248.9252, device='cuda:0') (tensor(-33.8094, device='cuda:0')) | > avg_loss_gen: 2.070730469596218 (-0.09135302523492106) | > avg_loss_kl: 2.4353838269139683 (+0.42449969976720725) | > avg_loss_feat: 3.7215517440312347 (+0.036357023346591966) | > avg_loss_mel: 19.093717951170156 (+2.302962034520956) | > avg_loss_duration: 1.6196126316634702 (+0.06406350202963385) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.94099659986899 (+2.7365291488002725) | > avg_grad_norm_1: tensor(9359.6660, device='cuda:0') (+tensor(1056.2969, device='cuda:0'))  > EPOCH: 851/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:58:45)   --> TIME: 2025-02-25 16:59:00 -- STEP: 27/72 -- GLOBAL_STEP: 1061300 | > loss_disc: 2.462550163269043 (2.6966409065105297) | > loss_disc_real_0: 0.08732688426971436 (0.15380304334340272) | > loss_disc_real_1: 0.28597328066825867 (0.231765772457476) | > loss_disc_real_2: 0.2555308938026428 (0.24669410785039267) | > loss_disc_real_3: 0.26546722650527954 (0.23375320600138771) | > loss_disc_real_4: 0.2601966857910156 (0.24972784739953496) | > loss_disc_real_5: 0.24113374948501587 (0.24698617502495093) | > loss_0: 2.462550163269043 (2.6966409065105297) | > grad_norm_0: tensor(48.7419, device='cuda:0') (tensor(255.8857, device='cuda:0')) | > loss_gen: 2.464637517929077 (2.119549662978561) | > loss_kl: 1.8479008674621582 (1.9740645046587344) | > loss_feat: 3.8774476051330566 (3.629056109322442) | > loss_mel: 16.724349975585938 (16.99931282467312) | > loss_duration: 1.5183420181274414 (1.5219641658994887) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.43267822265625 (26.243947347005207) | > grad_norm_1: tensor(4629.4814, device='cuda:0') (tensor(7898.2124, device='cuda:0')) | > current_lr_0: 0.00017981628981217793 | > current_lr_1: 0.00017981628981217793 | > step_time: 0.4472 (0.4572473808571144) | > loader_time: 0.0086 (0.006919878500479239) --> EVAL PERFORMANCE | > avg_loader_time: 0.008623254131263414 (-7.378887122785199e-05) | > avg_step_time: 0.4638656125941747 (+0.003795973012145182) | > avg_loss_disc: 2.653130930914006 (+0.02196034243409084) | > avg_loss_disc_real_0: 0.12861821850315786 (+0.02988370990669223) | > avg_loss_disc_real_1: 0.23115141312001455 (+0.005569202799192596) | > avg_loss_disc_real_2: 0.2302496569677138 (-0.0038154581692856904) | > avg_loss_disc_real_3: 0.2277901913498489 (-0.004452870345451443) | > avg_loss_disc_real_4: 0.2432628183717459 (-0.0015400501204209105) | > avg_loss_disc_real_5: 0.24459422830964478 (+0.0007091193971499599) | > avg_loss_0: 2.653130930914006 (+0.02196034243409084) | > avg_grad_norm_0: tensor(260.8012, device='cuda:0') (+tensor(11.8759, device='cuda:0')) | > avg_loss_gen: 2.0824787616729745 (+0.011748292076756428) | > avg_loss_kl: 2.3130675822916174 (-0.12231624462235091) | > avg_loss_feat: 3.6479834805072193 (-0.07356826352401535) | > avg_loss_mel: 17.930915134053837 (-1.1628028171163187) | > avg_loss_duration: 1.586047691358647 (-0.033564940304823265) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.560492663316325 (-1.380503936552664) | > avg_grad_norm_1: tensor(8374.0791, device='cuda:0') (tensor(-985.5869, device='cuda:0'))  > EPOCH: 852/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 16:59:23)   --> TIME: 2025-02-25 16:59:27 -- STEP: 5/72 -- GLOBAL_STEP: 1061350 | > loss_disc: 2.1717915534973145 (2.484163427352905) | > loss_disc_real_0: 0.09116392582654953 (0.06676385402679444) | > loss_disc_real_1: 0.22885805368423462 (0.22529149949550628) | > loss_disc_real_2: 0.2584073841571808 (0.3398165166378021) | > loss_disc_real_3: 0.21944227814674377 (0.23353009223937987) | > loss_disc_real_4: 0.21359221637248993 (0.2196286588907242) | > loss_disc_real_5: 0.2248862236738205 (0.2364893674850464) | > loss_0: 2.1717915534973145 (2.484163427352905) | > grad_norm_0: tensor(306.0027, device='cuda:0') (tensor(287.6290, device='cuda:0')) | > loss_gen: 2.572587490081787 (2.346628284454346) | > loss_kl: 2.480103015899658 (2.5177445888519285) | > loss_feat: 4.631999969482422 (4.186996173858643) | > loss_mel: 19.587413787841797 (19.46487808227539) | > loss_duration: 1.5696117877960205 (1.5943621873855591) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.841716766357422 (30.11061019897461) | > grad_norm_1: tensor(6339.2515, device='cuda:0') (tensor(5338.9312, device='cuda:0')) | > current_lr_0: 0.0001797938127759514 | > current_lr_1: 0.0001797938127759514 | > step_time: 0.4284 (0.4292454242706299) | > loader_time: 0.0072 (0.006664085388183594)  --> TIME: 2025-02-25 16:59:50 -- STEP: 55/72 -- GLOBAL_STEP: 1061400 | > loss_disc: 2.8983986377716064 (2.6886074673045766) | > loss_disc_real_0: 0.05164562165737152 (0.14324982427060604) | > loss_disc_real_1: 0.2584804892539978 (0.24374540583653884) | > loss_disc_real_2: 0.2674635350704193 (0.2661833459680731) | > loss_disc_real_3: 0.22981655597686768 (0.2272287894379009) | > loss_disc_real_4: 0.26293569803237915 (0.24895126575773413) | > loss_disc_real_5: 0.24286827445030212 (0.2437204279682853) | > loss_0: 2.8983986377716064 (2.6886074673045766) | > grad_norm_0: tensor(113.6019, device='cuda:0') (tensor(223.6122, device='cuda:0')) | > loss_gen: 1.6361429691314697 (2.134317755699157) | > loss_kl: 2.134284257888794 (2.2933488932522854) | > loss_feat: 2.711717367172241 (3.5755280754782937) | > loss_mel: 16.316184997558594 (18.007021626559176) | > loss_duration: 1.6078627109527588 (1.5739538669586182) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.406190872192383 (27.584170324152165) | > grad_norm_1: tensor(6098.2393, device='cuda:0') (tensor(6633.9502, device='cuda:0')) | > current_lr_0: 0.0001797938127759514 | > current_lr_1: 0.0001797938127759514 | > step_time: 0.4699 (0.4464428424835205) | > loader_time: 0.0095 (0.008085480603304777) --> EVAL PERFORMANCE | > avg_loader_time: 0.008779122795857172 (+0.00015586866459375848) | > avg_step_time: 0.45909458818569987 (-0.004771024408474822) | > avg_loss_disc: 2.6744619691875617 (+0.02133103827355587) | > avg_loss_disc_real_0: 0.1339415681981285 (+0.005323349694970642) | > avg_loss_disc_real_1: 0.2425596491551735 (+0.011408236035158947) | > avg_loss_disc_real_2: 0.25759911516182865 (+0.027349458194114856) | > avg_loss_disc_real_3: 0.2301987000334431 (+0.002408508683594185) | > avg_loss_disc_real_4: 0.24886375329863858 (+0.005600934926892676) | > avg_loss_disc_real_5: 0.2441361091086562 (-0.00045811920098856596) | > avg_loss_0: 2.6744619691875617 (+0.02133103827355587) | > avg_grad_norm_0: tensor(217.9925, device='cuda:0') (tensor(-42.8087, device='cuda:0')) | > avg_loss_gen: 2.1307212080754017 (+0.04824244640242714) | > avg_loss_kl: 2.2688524958113545 (-0.044215086480262844) | > avg_loss_feat: 3.5665328200434296 (-0.08145066046378968) | > avg_loss_mel: 17.84094383347203 (-0.08997130058180858) | > avg_loss_duration: 1.5824872020264746 (-0.003560489332172345) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.389537596366775 (-0.1709550669495492) | > avg_grad_norm_1: tensor(6487.7559, device='cuda:0') (tensor(-1886.3232, device='cuda:0'))  > EPOCH: 853/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:00:00)   --> TIME: 2025-02-25 17:00:17 -- STEP: 33/72 -- GLOBAL_STEP: 1061450 | > loss_disc: 2.851430892944336 (2.7554996663873847) | > loss_disc_real_0: 0.04559977352619171 (0.14613878320563922) | > loss_disc_real_1: 0.20215389132499695 (0.22638474585431995) | > loss_disc_real_2: 0.23247094452381134 (0.2432114996693351) | > loss_disc_real_3: 0.19433839619159698 (0.2140649069439281) | > loss_disc_real_4: 0.22749993205070496 (0.23723008099830512) | > loss_disc_real_5: 0.21604083478450775 (0.2422489266503941) | > loss_0: 2.851430892944336 (2.7554996663873847) | > grad_norm_0: tensor(95.5810, device='cuda:0') (tensor(219.4549, device='cuda:0')) | > loss_gen: 1.6376837491989136 (1.9334794716401533) | > loss_kl: 2.316014289855957 (2.3612079909353554) | > loss_feat: 3.461369514465332 (3.4360359582034024) | > loss_mel: 17.960674285888672 (18.09768801024466) | > loss_duration: 1.596261739730835 (1.6374583569439976) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.972003936767578 (27.465869903564453) | > grad_norm_1: tensor(5643.0205, device='cuda:0') (tensor(6475.3472, device='cuda:0')) | > current_lr_0: 0.0001797713385493544 | > current_lr_1: 0.0001797713385493544 | > step_time: 0.4699 (0.4361907930085153) | > loader_time: 0.0079 (0.007140629219286369) --> EVAL PERFORMANCE | > avg_loader_time: 0.008673258230719768 (-0.00010586456513740454) | > avg_step_time: 0.4552797129456426 (-0.003814875240057247) | > avg_loss_disc: 2.7144251877153422 (+0.03996321852778051) | > avg_loss_disc_real_0: 0.14987807044051066 (+0.01593650224238216) | > avg_loss_disc_real_1: 0.24535434798035824 (+0.0027946988251847382) | > avg_loss_disc_real_2: 0.23620034586375868 (-0.02139876929806997) | > avg_loss_disc_real_3: 0.23018274005030243 (-1.595998314066205e-05) | > avg_loss_disc_real_4: 0.2399525394741918 (-0.00891121382444679) | > avg_loss_disc_real_5: 0.24335892750343807 (-0.0007771816052181413) | > avg_loss_0: 2.7144251877153422 (+0.03996321852778051) | > avg_grad_norm_0: tensor(203.3604, device='cuda:0') (tensor(-14.6320, device='cuda:0')) | > avg_loss_gen: 2.040419568478222 (-0.09030163959717985) | > avg_loss_kl: 2.2642381594214647 (-0.004614336389889839) | > avg_loss_feat: 3.446623859271197 (-0.11990896077223256) | > avg_loss_mel: 17.521514073224136 (-0.3194297602478926) | > avg_loss_duration: 1.626685896389921 (+0.044198694363446256) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.89948159876004 (-0.4900559976067349) | > avg_grad_norm_1: tensor(6480.9434, device='cuda:0') (tensor(-6.8125, device='cuda:0'))  > EPOCH: 854/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:00:37)   --> TIME: 2025-02-25 17:00:45 -- STEP: 11/72 -- GLOBAL_STEP: 1061500 | > loss_disc: 2.7634317874908447 (2.649022340774536) | > loss_disc_real_0: 0.17291554808616638 (0.14919755404645746) | > loss_disc_real_1: 0.2460070699453354 (0.22235178405588324) | > loss_disc_real_2: 0.25083595514297485 (0.24686649983579462) | > loss_disc_real_3: 0.25399017333984375 (0.25331576710397546) | > loss_disc_real_4: 0.2510010004043579 (0.24345474080605942) | > loss_disc_real_5: 0.25852417945861816 (0.24803731116381558) | > loss_0: 2.7634317874908447 (2.649022340774536) | > grad_norm_0: tensor(240.2324, device='cuda:0') (tensor(188.9116, device='cuda:0')) | > loss_gen: 1.9504954814910889 (2.065799052065069) | > loss_kl: 1.8311797380447388 (1.9515188932418823) | > loss_feat: 3.4217050075531006 (3.37404186075384) | > loss_mel: 17.33211326599121 (17.084617701443758) | > loss_duration: 1.5230381488800049 (1.5339856906370684) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.05853271484375 (26.00996346907182) | > grad_norm_1: tensor(7748.1865, device='cuda:0') (tensor(6754.5322, device='cuda:0')) | > current_lr_0: 0.00017974886713203572 | > current_lr_1: 0.00017974886713203572 | > step_time: 0.4328 (0.4786882834001021) | > loader_time: 0.0075 (0.006804336201060902)  --> TIME: 2025-02-25 17:01:08 -- STEP: 61/72 -- GLOBAL_STEP: 1061550 | > loss_disc: 2.7977051734924316 (2.669692528052408) | > loss_disc_real_0: 0.10941819846630096 (0.13745267957937515) | > loss_disc_real_1: 0.2580212354660034 (0.22481081622545837) | > loss_disc_real_2: 0.24743854999542236 (0.2472393725250588) | > loss_disc_real_3: 0.2123403698205948 (0.24142232854835322) | > loss_disc_real_4: 0.23162315785884857 (0.24196567716168577) | > loss_disc_real_5: 0.2311907559633255 (0.2458913871010796) | > loss_0: 2.7977051734924316 (2.669692528052408) | > grad_norm_0: tensor(12.1309, device='cuda:0') (tensor(203.8714, device='cuda:0')) | > loss_gen: 1.8502278327941895 (2.033934698730218) | > loss_kl: 2.1859591007232666 (2.1138760007795736) | > loss_feat: 2.787976026535034 (3.4566365343625427) | > loss_mel: 16.723148345947266 (17.134704027019563) | > loss_duration: 1.6066310405731201 (1.5616956577926386) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.153940200805664 (26.300846912821786) | > grad_norm_1: tensor(4233.5322, device='cuda:0') (tensor(5642.6060, device='cuda:0')) | > current_lr_0: 0.00017974886713203572 | > current_lr_1: 0.00017974886713203572 | > step_time: 0.4681 (0.4551524021586434) | > loader_time: 0.0107 (0.008147954940795897) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624221237612441 (-4.903699310732611e-05) | > avg_step_time: 0.4626198721603608 (+0.0073401592147182) | > avg_loss_disc: 2.7070292855652287 (-0.0073959021501135425) | > avg_loss_disc_real_0: 0.16637769887145132 (+0.01649962843094066) | > avg_loss_disc_real_1: 0.2281186997260846 (-0.01723564825427365) | > avg_loss_disc_real_2: 0.24412776235963257 (+0.00792741649587389) | > avg_loss_disc_real_3: 0.2448452954560938 (+0.014662555405791367) | > avg_loss_disc_real_4: 0.24418395799650272 (+0.004231418522310931) | > avg_loss_disc_real_5: 0.24585603744211332 (+0.002497109938675246) | > avg_loss_0: 2.7070292855652287 (-0.0073959021501135425) | > avg_grad_norm_0: tensor(219.9641, device='cuda:0') (+tensor(16.6037, device='cuda:0')) | > avg_loss_gen: 2.0481698647351325 (+0.007750296256910705) | > avg_loss_kl: 2.1309732937477 (-0.13326486567376472) | > avg_loss_feat: 3.3931862132650026 (-0.053437646006194495) | > avg_loss_mel: 17.18092902277557 (-0.340585050448567) | > avg_loss_duration: 1.5700791301861616 (-0.05660676620375926) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.323337474339446 (-0.576144124420594) | > avg_grad_norm_1: tensor(5696.3525, device='cuda:0') (tensor(-784.5908, device='cuda:0'))  > EPOCH: 855/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:01:15)   --> TIME: 2025-02-25 17:01:34 -- STEP: 39/72 -- GLOBAL_STEP: 1061600 | > loss_disc: 2.5861563682556152 (2.6650970287812052) | > loss_disc_real_0: 0.149329274892807 (0.14738367173152092) | > loss_disc_real_1: 0.2534415125846863 (0.2246548613676658) | > loss_disc_real_2: 0.22351616621017456 (0.22991872941836333) | > loss_disc_real_3: 0.25956687331199646 (0.23103434726213798) | > loss_disc_real_4: 0.24526026844978333 (0.24133529151097322) | > loss_disc_real_5: 0.24306778609752655 (0.24821445766167763) | > loss_0: 2.5861563682556152 (2.6650970287812052) | > grad_norm_0: tensor(251.0635, device='cuda:0') (tensor(145.8662, device='cuda:0')) | > loss_gen: 2.414064645767212 (2.028027607844426) | > loss_kl: 2.2037041187286377 (2.0166468253502474) | > loss_feat: 3.841115951538086 (3.389900360351954) | > loss_mel: 18.273038864135742 (16.90790039453751) | > loss_duration: 1.5511817932128906 (1.5293564613048847) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.283105850219727 (25.871831845014523) | > grad_norm_1: tensor(6846.6499, device='cuda:0') (tensor(5456.1289, device='cuda:0')) | > current_lr_0: 0.0001797263985236442 | > current_lr_1: 0.0001797263985236442 | > step_time: 0.4517 (0.44280334008045685) | > loader_time: 0.0087 (0.0073427481529040215) --> EVAL PERFORMANCE | > avg_loader_time: 0.008600224911327094 (-2.3996326285347785e-05) | > avg_step_time: 0.4580564733961938 (-0.004563398764167004) | > avg_loss_disc: 2.6993635674597503 (-0.007665718105478447) | > avg_loss_disc_real_0: 0.1680778570892945 (+0.001700158217843195) | > avg_loss_disc_real_1: 0.21807737501574231 (-0.010041324710342275) | > avg_loss_disc_real_2: 0.2314689688279595 (-0.012658793531673068) | > avg_loss_disc_real_3: 0.22680911778564183 (-0.01803617767045196) | > avg_loss_disc_real_4: 0.2369489655108519 (-0.007234992485650821) | > avg_loss_disc_real_5: 0.24306083502064288 (-0.0027952024214704385) | > avg_loss_0: 2.6993635674597503 (-0.007665718105478447) | > avg_grad_norm_0: tensor(154.5839, device='cuda:0') (tensor(-65.3802, device='cuda:0')) | > avg_loss_gen: 1.969172486117189 (-0.07899737861794343) | > avg_loss_kl: 2.081502759960336 (-0.04947053378736399) | > avg_loss_feat: 3.292140554374372 (-0.10104565889063055) | > avg_loss_mel: 17.01756930015457 (-0.1633597226209993) | > avg_loss_duration: 1.563421069736212 (-0.006658060449949543) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.92380625765088 (-0.3995312166885654) | > avg_grad_norm_1: tensor(4670.6265, device='cuda:0') (tensor(-1025.7261, device='cuda:0'))  > EPOCH: 856/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:01:52)   --> TIME: 2025-02-25 17:02:01 -- STEP: 17/72 -- GLOBAL_STEP: 1061650 | > loss_disc: 2.4054970741271973 (2.689092145246618) | > loss_disc_real_0: 0.10640506446361542 (0.16460847065729253) | > loss_disc_real_1: 0.20569723844528198 (0.23108066706096425) | > loss_disc_real_2: 0.23708108067512512 (0.24033696598866405) | > loss_disc_real_3: 0.2245253324508667 (0.24623845868250904) | > loss_disc_real_4: 0.2238353043794632 (0.2312807230388417) | > loss_disc_real_5: 0.2374674677848816 (0.24505648455199072) | > loss_0: 2.4054970741271973 (2.689092145246618) | > grad_norm_0: tensor(43.0548, device='cuda:0') (tensor(169.3299, device='cuda:0')) | > loss_gen: 2.4025778770446777 (2.058294134981492) | > loss_kl: 3.1550652980804443 (2.271018554182614) | > loss_feat: 3.7434651851654053 (3.3654246049768783) | > loss_mel: 20.245752334594727 (17.74777715346393) | > loss_duration: 1.6372902393341064 (1.6222542454214657) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.184152603149414 (27.06476873510024) | > grad_norm_1: tensor(2562.9792, device='cuda:0') (tensor(4543.0991, device='cuda:0')) | > current_lr_0: 0.00017970393272382875 | > current_lr_1: 0.00017970393272382875 | > step_time: 0.4367 (0.4374890748192282) | > loader_time: 0.0065 (0.007096669253181009)  --> TIME: 2025-02-25 17:02:25 -- STEP: 67/72 -- GLOBAL_STEP: 1061700 | > loss_disc: 2.625486135482788 (2.5782299290842086) | > loss_disc_real_0: 0.15812017023563385 (0.11861441390060667) | > loss_disc_real_1: 0.2698874771595001 (0.22857734916815117) | > loss_disc_real_2: 0.2801406681537628 (0.22908445280879292) | > loss_disc_real_3: 0.18366281688213348 (0.22710625293539533) | > loss_disc_real_4: 0.25021493434906006 (0.2324175412085519) | > loss_disc_real_5: 0.22873163223266602 (0.24136670934620189) | > loss_0: 2.625486135482788 (2.5782299290842086) | > grad_norm_0: tensor(147.1485, device='cuda:0') (tensor(171.3538, device='cuda:0')) | > loss_gen: 2.1757030487060547 (2.1844965664308447) | > loss_kl: 2.170100450515747 (2.310901629391002) | > loss_feat: 3.6588499546051025 (3.79163874796967) | > loss_mel: 17.405780792236328 (17.693615130524144) | > loss_duration: 1.6469519138336182 (1.623538947817105) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.05738639831543 (27.604191111094917) | > grad_norm_1: tensor(6777.3809, device='cuda:0') (tensor(5661.2656, device='cuda:0')) | > current_lr_0: 0.00017970393272382875 | > current_lr_1: 0.00017970393272382875 | > step_time: 0.5097 (0.4542354896887025) | > loader_time: 0.0123 (0.008482506026082965) --> EVAL PERFORMANCE | > avg_loader_time: 0.008657015545267455 (+5.679063394036128e-05) | > avg_step_time: 0.4578511580614976 (-0.0002053153346961989) | > avg_loss_disc: 2.576851609727026 (-0.12251195773272405) | > avg_loss_disc_real_0: 0.12130200322455084 (-0.04677585386474367) | > avg_loss_disc_real_1: 0.22612144944953247 (+0.008044074433790155) | > avg_loss_disc_real_2: 0.2303877099299095 (-0.001081258898049997) | > avg_loss_disc_real_3: 0.22548744838002702 (-0.001321669405614817) | > avg_loss_disc_real_4: 0.23273017666709256 (-0.004218788843759341) | > avg_loss_disc_real_5: 0.24038676434839276 (-0.0026740706722501173) | > avg_loss_0: 2.576851609727026 (-0.12251195773272405) | > avg_grad_norm_0: tensor(166.5254, device='cuda:0') (+tensor(11.9415, device='cuda:0')) | > avg_loss_gen: 2.1921712411961067 (+0.22299875507891764) | > avg_loss_kl: 2.307917338022045 (+0.22641457806170884) | > avg_loss_feat: 3.7986955004678644 (+0.5065549460934924) | > avg_loss_mel: 17.684649454036236 (+0.6670801538816669) | > avg_loss_duration: 1.623537343992314 (+0.06011627425610189) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.6069709348007 (+1.6831646771498185) | > avg_grad_norm_1: tensor(5517.0693, device='cuda:0') (+tensor(846.4429, device='cuda:0'))  > EPOCH: 857/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:02:29)   --> TIME: 2025-02-25 17:02:51 -- STEP: 45/72 -- GLOBAL_STEP: 1061750 | > loss_disc: 2.7887754440307617 (2.725642357932197) | > loss_disc_real_0: 0.1069643571972847 (0.1272276622139745) | > loss_disc_real_1: 0.20058342814445496 (0.2059053616391288) | > loss_disc_real_2: 0.2209751009941101 (0.2399125936958525) | > loss_disc_real_3: 0.22305868566036224 (0.228185397055414) | > loss_disc_real_4: 0.2719533443450928 (0.23290440837542217) | > loss_disc_real_5: 0.24084481596946716 (0.23569813403818343) | > loss_0: 2.7887754440307617 (2.725642357932197) | > grad_norm_0: tensor(43.4592, device='cuda:0') (tensor(214.9992, device='cuda:0')) | > loss_gen: 1.9273278713226318 (1.8691956493589612) | > loss_kl: 3.1456592082977295 (2.0740022844738424) | > loss_feat: 3.2242612838745117 (3.3477087656656903) | > loss_mel: 20.88134765625 (17.180881330702036) | > loss_duration: 1.622323989868164 (1.5432879262500336) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.800920486450195 (26.01507585313585) | > grad_norm_1: tensor(7301.9595, device='cuda:0') (tensor(6293.2427, device='cuda:0')) | > current_lr_0: 0.00017968146973223827 | > current_lr_1: 0.00017968146973223827 | > step_time: 0.4532 (0.4419537279340956) | > loader_time: 0.0087 (0.007653517193264432) --> EVAL PERFORMANCE | > avg_loader_time: 0.008730629800071181 (+7.361425480372635e-05) | > avg_step_time: 0.4557573123716972 (-0.002093845689800422) | > avg_loss_disc: 2.733077301105983 (+0.1562256913789568) | > avg_loss_disc_real_0: 0.1425137512228438 (+0.02121174799829295) | > avg_loss_disc_real_1: 0.2292345703487665 (+0.0031131208992340176) | > avg_loss_disc_real_2: 0.22908347698164658 (-0.001304232948262929) | > avg_loss_disc_real_3: 0.22302338543912054 (-0.0024640629409064807) | > avg_loss_disc_real_4: 0.230684048273194 (-0.002046128393898561) | > avg_loss_disc_real_5: 0.2365123990555884 (-0.003874365292804355) | > avg_loss_0: 2.733077301105983 (+0.1562256913789568) | > avg_grad_norm_0: tensor(234.1217, device='cuda:0') (+tensor(67.5963, device='cuda:0')) | > avg_loss_gen: 1.9058004987071937 (-0.28637074248891303) | > avg_loss_kl: 2.3112847821813234 (+0.0033674441592785875) | > avg_loss_feat: 3.4116066308088704 (-0.38708886965899403) | > avg_loss_mel: 18.268821796900788 (+0.5841723428645516) | > avg_loss_duration: 1.5745332560069125 (-0.04900408798540146) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.472046892407914 (-0.134924042392786) | > avg_grad_norm_1: tensor(6660.2446, device='cuda:0') (+tensor(1143.1753, device='cuda:0'))  > EPOCH: 858/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:03:06)   --> TIME: 2025-02-25 17:03:19 -- STEP: 23/72 -- GLOBAL_STEP: 1061800 | > loss_disc: 2.532459259033203 (2.6679123380909795) | > loss_disc_real_0: 0.08775836229324341 (0.1567572065345619) | > loss_disc_real_1: 0.2634860873222351 (0.2503153176411338) | > loss_disc_real_2: 0.23992206156253815 (0.22687629383543265) | > loss_disc_real_3: 0.25535091757774353 (0.23781597549500672) | > loss_disc_real_4: 0.27102288603782654 (0.23962210248345914) | > loss_disc_real_5: 0.2647061347961426 (0.23693074156408725) | > loss_0: 2.532459259033203 (2.6679123380909795) | > grad_norm_0: tensor(291.4617, device='cuda:0') (tensor(256.2650, device='cuda:0')) | > loss_gen: 2.236311674118042 (2.102374242699665) | > loss_kl: 2.1763088703155518 (2.1019345314606377) | > loss_feat: 3.4655568599700928 (3.5425508436949356) | > loss_mel: 16.452579498291016 (17.778961762138035) | > loss_duration: 1.5277845859527588 (1.5289726153663967) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.858539581298828 (27.054793399313223) | > grad_norm_1: tensor(7990.5786, device='cuda:0') (tensor(8380.2842, device='cuda:0')) | > current_lr_0: 0.00017965900954852173 | > current_lr_1: 0.00017965900954852173 | > step_time: 0.4359 (0.4381595383519712) | > loader_time: 0.0066 (0.007001700608626656) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870177107797542 (-2.885872209576132e-05) | > avg_step_time: 0.46062241137867244 (+0.0048650990069752376) | > avg_loss_disc: 2.659090283890845 (-0.073987017215138) | > avg_loss_disc_real_0: 0.14607291495506192 (+0.0035591637322181235) | > avg_loss_disc_real_1: 0.23563863672840762 (+0.006404066379641138) | > avg_loss_disc_real_2: 0.23425419187881577 (+0.005170714897169193) | > avg_loss_disc_real_3: 0.23383503165883077 (+0.01081164621971023) | > avg_loss_disc_real_4: 0.24269820749759674 (+0.012014159224402743) | > avg_loss_disc_real_5: 0.24333655498397183 (+0.006824155928383424) | > avg_loss_0: 2.659090283890845 (-0.073987017215138) | > avg_grad_norm_0: tensor(226.7044, device='cuda:0') (tensor(-7.4173, device='cuda:0')) | > avg_loss_gen: 2.0848972260112495 (+0.1790967273040558) | > avg_loss_kl: 2.246460169134006 (-0.06482461304731757) | > avg_loss_feat: 3.524941044794002 (+0.11333441398513155) | > avg_loss_mel: 17.731493291720536 (-0.5373285051802519) | > avg_loss_duration: 1.5791676245944601 (+0.0046343685875476215) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.16695909097161 (-0.3050878014363043) | > avg_grad_norm_1: tensor(6877.1279, device='cuda:0') (+tensor(216.8833, device='cuda:0'))  > EPOCH: 859/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:03:44)   --> TIME: 2025-02-25 17:03:46 -- STEP: 1/72 -- GLOBAL_STEP: 1061850 | > loss_disc: 2.4744110107421875 (2.4744110107421875) | > loss_disc_real_0: 0.18329259753227234 (0.18329259753227234) | > loss_disc_real_1: 0.22290705144405365 (0.22290705144405365) | > loss_disc_real_2: 0.2688673138618469 (0.2688673138618469) | > loss_disc_real_3: 0.26200833916664124 (0.26200833916664124) | > loss_disc_real_4: 0.27517169713974 (0.27517169713974) | > loss_disc_real_5: 0.2741515040397644 (0.2741515040397644) | > loss_0: 2.4744110107421875 (2.4744110107421875) | > grad_norm_0: tensor(430.4080, device='cuda:0') (tensor(430.4080, device='cuda:0')) | > loss_gen: 2.7017617225646973 (2.7017617225646973) | > loss_kl: 2.0183863639831543 (2.0183863639831543) | > loss_feat: 4.135264873504639 (4.135264873504639) | > loss_mel: 18.553701400756836 (18.553701400756836) | > loss_duration: 1.5471683740615845 (1.5471683740615845) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.956281661987305 (28.956281661987305) | > grad_norm_1: tensor(6466.5972, device='cuda:0') (tensor(6466.5972, device='cuda:0')) | > current_lr_0: 0.00017963655217232815 | > current_lr_1: 0.00017963655217232815 | > step_time: 0.4321 (0.4321303367614746) | > loader_time: 0.0059 (0.0059070587158203125)  --> TIME: 2025-02-25 17:04:09 -- STEP: 51/72 -- GLOBAL_STEP: 1061900 | > loss_disc: 2.802154302597046 (2.714421739765241) | > loss_disc_real_0: 0.3719767928123474 (0.24165816816921332) | > loss_disc_real_1: 0.279398113489151 (0.23550059807066823) | > loss_disc_real_2: 0.21446561813354492 (0.23672011669944315) | > loss_disc_real_3: 0.23526506125926971 (0.2417192117256277) | > loss_disc_real_4: 0.25052303075790405 (0.24814362181167976) | > loss_disc_real_5: 0.23897215723991394 (0.2475983062211205) | > loss_0: 2.802154302597046 (2.714421739765241) | > grad_norm_0: tensor(325.4294, device='cuda:0') (tensor(209.5279, device='cuda:0')) | > loss_gen: 2.159416437149048 (2.1669088438445447) | > loss_kl: 2.361191987991333 (2.1018262961331544) | > loss_feat: 3.0932681560516357 (3.4090550740559897) | > loss_mel: 16.719337463378906 (17.432395187078736) | > loss_duration: 1.6203656196594238 (1.5521175136753158) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.95357894897461 (26.662302802590762) | > grad_norm_1: tensor(1691.0238, device='cuda:0') (tensor(4475.2368, device='cuda:0')) | > current_lr_0: 0.00017963655217232815 | > current_lr_1: 0.00017963655217232815 | > step_time: 0.4483 (0.4435216959785013) | > loader_time: 0.0087 (0.007891093983369716) --> EVAL PERFORMANCE | > avg_loader_time: 0.008737258508171834 (+3.548743019641393e-05) | > avg_step_time: 0.4571586293233952 (-0.0034637820552772536) | > avg_loss_disc: 2.716222041089769 (+0.05713175719892405) | > avg_loss_disc_real_0: 0.2232717463982777 (+0.07719883144321579) | > avg_loss_disc_real_1: 0.2339082486192945 (-0.0017303881091131346) | > avg_loss_disc_real_2: 0.24065554246936047 (+0.006401350590544697) | > avg_loss_disc_real_3: 0.23582575018976776 (+0.0019907185309369912) | > avg_loss_disc_real_4: 0.25131633374052986 (+0.008618126242933122) | > avg_loss_disc_real_5: 0.24781109980294402 (+0.004474544818972193) | > avg_loss_0: 2.716222041089769 (+0.05713175719892405) | > avg_grad_norm_0: tensor(188.9463, device='cuda:0') (tensor(-37.7581, device='cuda:0')) | > avg_loss_gen: 2.1248115519402733 (+0.039914325929023775) | > avg_loss_kl: 2.162749419749623 (-0.08371074938438294) | > avg_loss_feat: 3.370050803036757 (-0.15489024175724486) | > avg_loss_mel: 17.57028523297377 (-0.16120805874676591) | > avg_loss_duration: 1.5711525376413908 (-0.008015086953069295) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.7990495386258 (-0.3679095523458109) | > avg_grad_norm_1: tensor(4511.8848, device='cuda:0') (tensor(-2365.2432, device='cuda:0'))  > EPOCH: 860/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:04:21)   --> TIME: 2025-02-25 17:04:36 -- STEP: 29/72 -- GLOBAL_STEP: 1061950 | > loss_disc: 2.8603546619415283 (2.738576182003679) | > loss_disc_real_0: 0.040942512452602386 (0.13305134929973503) | > loss_disc_real_1: 0.20748281478881836 (0.22238486234484048) | > loss_disc_real_2: 0.2310277223587036 (0.228051810429014) | > loss_disc_real_3: 0.26087620854377747 (0.2442538537855806) | > loss_disc_real_4: 0.27913057804107666 (0.24321733106826915) | > loss_disc_real_5: 0.2762738764286041 (0.2507849941993582) | > loss_0: 2.8603546619415283 (2.738576182003679) | > grad_norm_0: tensor(161.3899, device='cuda:0') (tensor(151.8193, device='cuda:0')) | > loss_gen: 1.7510603666305542 (1.9107317554539647) | > loss_kl: 2.5196328163146973 (2.351293152776258) | > loss_feat: 3.2360775470733643 (3.422497798656595) | > loss_mel: 19.278505325317383 (19.130707905210297) | > loss_duration: 1.5728700160980225 (1.5885243251405914) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.35814666748047 (28.403754990676354) | > grad_norm_1: tensor(7404.3374, device='cuda:0') (tensor(5356.1470, device='cuda:0')) | > current_lr_0: 0.0001796140976033066 | > current_lr_1: 0.0001796140976033066 | > step_time: 0.4399 (0.43876026416647024) | > loader_time: 0.0081 (0.007028645482556573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008724676051609953 (-1.2582456561880881e-05) | > avg_step_time: 0.46295010875648174 (+0.005791479433086555) | > avg_loss_disc: 2.644675661140764 (-0.07154637994900526) | > avg_loss_disc_real_0: 0.11752438272388888 (-0.10574736367438883) | > avg_loss_disc_real_1: 0.22028325728967157 (-0.013624991329622915) | > avg_loss_disc_real_2: 0.23408843618883213 (-0.00656710628052834) | > avg_loss_disc_real_3: 0.24053636885864635 (+0.004710618668878591) | > avg_loss_disc_real_4: 0.24446382635915784 (-0.006852507381372025) | > avg_loss_disc_real_5: 0.25232258502026667 (+0.004511485217322642) | > avg_loss_0: 2.644675661140764 (-0.07154637994900526) | > avg_grad_norm_0: tensor(216.1605, device='cuda:0') (+tensor(27.2142, device='cuda:0')) | > avg_loss_gen: 2.0203021277844067 (-0.10450942415586661) | > avg_loss_kl: 2.245670009666765 (+0.08292058991714235) | > avg_loss_feat: 3.4854414866004193 (+0.11539068356366222) | > avg_loss_mel: 18.068041976069058 (+0.4977567430952874) | > avg_loss_duration: 1.5976021256245359 (+0.026449587983145006) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.41705770895515 (+0.6180081703293503) | > avg_grad_norm_1: tensor(7051.2729, device='cuda:0') (+tensor(2539.3882, device='cuda:0'))  > EPOCH: 861/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:04:59)   --> TIME: 2025-02-25 17:05:04 -- STEP: 7/72 -- GLOBAL_STEP: 1062000 | > loss_disc: 2.4807000160217285 (2.5380072593688965) | > loss_disc_real_0: 0.1027272492647171 (0.08769348689488002) | > loss_disc_real_1: 0.22768205404281616 (0.24073433663163865) | > loss_disc_real_2: 0.1916990876197815 (0.16400768075670516) | > loss_disc_real_3: 0.22097694873809814 (0.2260508473430361) | > loss_disc_real_4: 0.23352554440498352 (0.24941796277250564) | > loss_disc_real_5: 0.23289795219898224 (0.24794240295886993) | > loss_0: 2.4807000160217285 (2.5380072593688965) | > grad_norm_0: tensor(147.2394, device='cuda:0') (tensor(280.2814, device='cuda:0')) | > loss_gen: 2.180046796798706 (2.1289394923618863) | > loss_kl: 2.0820391178131104 (1.9416855233056205) | > loss_feat: 3.9558093547821045 (3.888058832713536) | > loss_mel: 17.80287742614746 (17.175110408238) | > loss_duration: 1.5312327146530151 (1.5435741458620345) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.552003860473633 (26.67736898149763) | > grad_norm_1: tensor(7940.6631, device='cuda:0') (tensor(9687.4062, device='cuda:0')) | > current_lr_0: 0.00017959164584110618 | > current_lr_1: 0.00017959164584110618 | > step_time: 0.4318 (0.43337815148489817) | > loader_time: 0.0055 (0.006450074059622628)  --> TIME: 2025-02-25 17:05:27 -- STEP: 57/72 -- GLOBAL_STEP: 1062050 | > loss_disc: 2.3881471157073975 (2.5364472991541813) | > loss_disc_real_0: 0.09767962247133255 (0.09320050876652985) | > loss_disc_real_1: 0.24488116800785065 (0.21991208430967832) | > loss_disc_real_2: 0.2639520466327667 (0.22153335522141374) | > loss_disc_real_3: 0.26439595222473145 (0.230527617690856) | > loss_disc_real_4: 0.24524623155593872 (0.24237265973760372) | > loss_disc_real_5: 0.25011491775512695 (0.24780728837899996) | > loss_0: 2.3881471157073975 (2.5364472991541813) | > grad_norm_0: tensor(211.5573, device='cuda:0') (tensor(245.7520, device='cuda:0')) | > loss_gen: 2.5031991004943848 (2.156083880809317) | > loss_kl: 2.078794002532959 (1.9776876746562488) | > loss_feat: 3.967026948928833 (3.742775059582894) | > loss_mel: 17.62782096862793 (17.076441045393025) | > loss_duration: 1.613463044166565 (1.5546184506332665) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.790302276611328 (26.507606104800576) | > grad_norm_1: tensor(7857.1846, device='cuda:0') (tensor(8735.1855, device='cuda:0')) | > current_lr_0: 0.00017959164584110618 | > current_lr_1: 0.00017959164584110618 | > step_time: 0.4585 (0.4468305236414859) | > loader_time: 0.0097 (0.008103462687709874) --> EVAL PERFORMANCE | > avg_loader_time: 0.008711784658297686 (-1.2891393312267421e-05) | > avg_step_time: 0.456633705488393 (-0.006316403268088744) | > avg_loss_disc: 2.534281828034092 (-0.11039383310667183) | > avg_loss_disc_real_0: 0.09821518148544808 (-0.019309201238440793) | > avg_loss_disc_real_1: 0.2194282502775461 (-0.000855007012125486) | > avg_loss_disc_real_2: 0.22248743180657776 (-0.011601004382254365) | > avg_loss_disc_real_3: 0.22900150800254984 (-0.011534860856096507) | > avg_loss_disc_real_4: 0.24182436499797122 (-0.002639461361186618) | > avg_loss_disc_real_5: 0.24698211756390584 (-0.005340467456360826) | > avg_loss_0: 2.534281828034092 (-0.11039383310667183) | > avg_grad_norm_0: tensor(252.1844, device='cuda:0') (+tensor(36.0240, device='cuda:0')) | > avg_loss_gen: 2.1576847946140134 (+0.13738266682960676) | > avg_loss_kl: 2.000526579333023 (-0.24514343033374209) | > avg_loss_feat: 3.7041331647147593 (+0.21869167811433998) | > avg_loss_mel: 16.993007028606577 (-1.0750349474624805) | > avg_loss_duration: 1.5651433350334705 (-0.03245879059106538) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.420494992968063 (-0.9965627159870856) | > avg_grad_norm_1: tensor(8727.6777, device='cuda:0') (+tensor(1676.4048, device='cuda:0'))  > EPOCH: 862/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:05:36)   --> TIME: 2025-02-25 17:05:53 -- STEP: 35/72 -- GLOBAL_STEP: 1062100 | > loss_disc: 2.5186588764190674 (2.5298902920314243) | > loss_disc_real_0: 0.07412439584732056 (0.10341818769063268) | > loss_disc_real_1: 0.19248543679714203 (0.2185831585100719) | > loss_disc_real_2: 0.22946208715438843 (0.22670947568757194) | > loss_disc_real_3: 0.2475121170282364 (0.23327902129718237) | > loss_disc_real_4: 0.25604143738746643 (0.24018475370747702) | > loss_disc_real_5: 0.25953343510627747 (0.24669659861496518) | > loss_0: 2.5186588764190674 (2.5298902920314243) | > grad_norm_0: tensor(336.9146, device='cuda:0') (tensor(269.8782, device='cuda:0')) | > loss_gen: 2.241037368774414 (2.1936749253954204) | > loss_kl: 1.931033968925476 (1.937596355165754) | > loss_feat: 3.8838753700256348 (3.8215032168797083) | > loss_mel: 17.460020065307617 (17.03499780382429) | > loss_duration: 1.563659906387329 (1.5268660647528511) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.079626083374023 (26.514638573782786) | > grad_norm_1: tensor(9524.2285, device='cuda:0') (tensor(9614.2314, device='cuda:0')) | > current_lr_0: 0.00017956919688537603 | > current_lr_1: 0.00017956919688537603 | > step_time: 0.4465 (0.44105311121259416) | > loader_time: 0.0085 (0.007352161407470703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008761721597590916 (+4.9936939293230426e-05) | > avg_step_time: 0.4572876272067218 (+0.0006539217183287871) | > avg_loss_disc: 2.6031997539627714 (+0.0689179259286794) | > avg_loss_disc_real_0: 0.1243173539323706 (+0.02610217244692252) | > avg_loss_disc_real_1: 0.22573630092009692 (+0.006308050642550828) | > avg_loss_disc_real_2: 0.23377178672333837 (+0.011284354916760608) | > avg_loss_disc_real_3: 0.23240656793957026 (+0.0034050599370204215) | > avg_loss_disc_real_4: 0.24562620885775122 (+0.0038018438597799964) | > avg_loss_disc_real_5: 0.2523407768195783 (+0.005358659255672443) | > avg_loss_0: 2.6031997539627714 (+0.0689179259286794) | > avg_grad_norm_0: tensor(260.2941, device='cuda:0') (+tensor(8.1096, device='cuda:0')) | > avg_loss_gen: 2.163482802014956 (+0.005798007400942495) | > avg_loss_kl: 2.2672809950062915 (+0.2667544156732684) | > avg_loss_feat: 3.727118522348538 (+0.022985357633778847) | > avg_loss_mel: 18.486015709353158 (+1.4930086807465806) | > avg_loss_duration: 1.6064746883553522 (+0.04133135332188176) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.25037284636162 (+1.8298778533935582) | > avg_grad_norm_1: tensor(9815.4844, device='cuda:0') (+tensor(1087.8066, device='cuda:0'))  > EPOCH: 863/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:06:13)   --> TIME: 2025-02-25 17:06:20 -- STEP: 13/72 -- GLOBAL_STEP: 1062150 | > loss_disc: 2.459352493286133 (2.58122260753925) | > loss_disc_real_0: 0.08373939990997314 (0.11920103316123669) | > loss_disc_real_1: 0.18734146654605865 (0.21679213069952452) | > loss_disc_real_2: 0.21602307260036469 (0.22481114933123955) | > loss_disc_real_3: 0.17808569967746735 (0.22443232398766738) | > loss_disc_real_4: 0.20035982131958008 (0.2380664749787404) | > loss_disc_real_5: 0.2250763177871704 (0.22800897634946382) | > loss_0: 2.459352493286133 (2.58122260753925) | > grad_norm_0: tensor(194.2993, device='cuda:0') (tensor(270.1494, device='cuda:0')) | > loss_gen: 2.0805861949920654 (2.230369760439946) | > loss_kl: 2.007709503173828 (2.0240003695854774) | > loss_feat: 3.88169527053833 (3.8658767480116625) | > loss_mel: 17.289466857910156 (17.645868594829853) | > loss_duration: 1.4879869222640991 (1.5271711532886212) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.74744415283203 (27.293286690345177) | > grad_norm_1: tensor(8438.3984, device='cuda:0') (tensor(5226.0410, device='cuda:0')) | > current_lr_0: 0.00017954675073576535 | > current_lr_1: 0.00017954675073576535 | > step_time: 0.4577 (0.4342347291799692) | > loader_time: 0.0058 (0.006961822509765625)  --> TIME: 2025-02-25 17:06:44 -- STEP: 63/72 -- GLOBAL_STEP: 1062200 | > loss_disc: 2.588766574859619 (2.5478947200472395) | > loss_disc_real_0: 0.12743252515792847 (0.09120478510620102) | > loss_disc_real_1: 0.2301778346300125 (0.2199090932096754) | > loss_disc_real_2: 0.1850680410861969 (0.22420127784448957) | > loss_disc_real_3: 0.20885927975177765 (0.2210521222580047) | > loss_disc_real_4: 0.21154268085956573 (0.2329212867078327) | > loss_disc_real_5: 0.24424515664577484 (0.23907755906619724) | > loss_0: 2.588766574859619 (2.5478947200472395) | > grad_norm_0: tensor(219.0121, device='cuda:0') (tensor(220.5125, device='cuda:0')) | > loss_gen: 2.1789495944976807 (2.2121612949976845) | > loss_kl: 1.975799798965454 (2.0302160883706715) | > loss_feat: 3.8589282035827637 (3.943001077288673) | > loss_mel: 16.281597137451172 (17.187034455556713) | > loss_duration: 1.580118179321289 (1.5572478563066512) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.87539291381836 (26.92966067601764) | > grad_norm_1: tensor(3988.0085, device='cuda:0') (tensor(6754.2007, device='cuda:0')) | > current_lr_0: 0.00017954675073576535 | > current_lr_1: 0.00017954675073576535 | > step_time: 0.4863 (0.44765369854276144) | > loader_time: 0.0114 (0.008234266250852551) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624973431439463 (-0.00013674816615145274) | > avg_step_time: 0.4562543580229853 (-0.0010332691837364738) | > avg_loss_disc: 2.5543471490833123 (-0.04885260487945908) | > avg_loss_disc_real_0: 0.0908980125587591 (-0.03341934137361151) | > avg_loss_disc_real_1: 0.22038290660146256 (-0.0053533943186343524) | > avg_loss_disc_real_2: 0.23053644009878937 (-0.003235346624548996) | > avg_loss_disc_real_3: 0.22371802678410435 (-0.008688541155465912) | > avg_loss_disc_real_4: 0.23453096520732825 (-0.01109524365042297) | > avg_loss_disc_real_5: 0.2398313348981696 (-0.012509441921408682) | > avg_loss_0: 2.5543471490833123 (-0.04885260487945908) | > avg_grad_norm_0: tensor(219.7325, device='cuda:0') (tensor(-40.5616, device='cuda:0')) | > avg_loss_gen: 2.220462379321246 (+0.05697957730629) | > avg_loss_kl: 2.036138929111857 (-0.23114206589443453) | > avg_loss_feat: 3.930541320585869 (+0.20342279823733067) | > avg_loss_mel: 17.16816821568448 (-1.3178474936686762) | > avg_loss_duration: 1.5645025951761597 (-0.04197209317919248) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.91981337104045 (-1.33055947532117) | > avg_grad_norm_1: tensor(6896.5908, device='cuda:0') (tensor(-2918.8936, device='cuda:0'))  > EPOCH: 864/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:06:50)   --> TIME: 2025-02-25 17:07:11 -- STEP: 41/72 -- GLOBAL_STEP: 1062250 | > loss_disc: 2.7440009117126465 (2.5136485797602948) | > loss_disc_real_0: 0.06232287734746933 (0.08707439554173772) | > loss_disc_real_1: 0.21245698630809784 (0.2175716855903951) | > loss_disc_real_2: 0.23213693499565125 (0.22330855741733457) | > loss_disc_real_3: 0.2298131287097931 (0.22572367947276045) | > loss_disc_real_4: 0.24792779982089996 (0.2324549468552194) | > loss_disc_real_5: 0.2591210603713989 (0.23948829893658802) | > loss_0: 2.7440009117126465 (2.5136485797602948) | > grad_norm_0: tensor(224.7028, device='cuda:0') (tensor(191.7578, device='cuda:0')) | > loss_gen: 1.9530348777770996 (2.2293925517942843) | > loss_kl: 1.933303713798523 (1.9537957499666911) | > loss_feat: 3.667238712310791 (4.008081046546378) | > loss_mel: 17.339765548706055 (17.141577255435106) | > loss_duration: 1.556099772453308 (1.5267609357833862) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.44944190979004 (26.859607696533203) | > grad_norm_1: tensor(10001.4346, device='cuda:0') (tensor(6982.1226, device='cuda:0')) | > current_lr_0: 0.00017952430739192338 | > current_lr_1: 0.00017952430739192338 | > step_time: 0.4495 (0.4527331096370046) | > loader_time: 0.009 (0.007264858338891006) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618660376105513 (-6.313055333950074e-06) | > avg_step_time: 0.4673580048789441 (+0.011103646855958771) | > avg_loss_disc: 2.535234367343741 (-0.019112781739571094) | > avg_loss_disc_real_0: 0.08615594897681558 (-0.004742063581943512) | > avg_loss_disc_real_1: 0.22352768543740392 (+0.0031447788359413587) | > avg_loss_disc_real_2: 0.2272501957248634 (-0.0032862443739259672) | > avg_loss_disc_real_3: 0.22952297602740812 (+0.00580494924330377) | > avg_loss_disc_real_4: 0.23688701697638337 (+0.002356051769055123) | > avg_loss_disc_real_5: 0.24221824823131025 (+0.0023869133331406445) | > avg_loss_0: 2.535234367343741 (-0.019112781739571094) | > avg_grad_norm_0: tensor(201.2609, device='cuda:0') (tensor(-18.4716, device='cuda:0')) | > avg_loss_gen: 2.232769865385243 (+0.012307486063996897) | > avg_loss_kl: 2.0126601222535254 (-0.02347880685833159) | > avg_loss_feat: 3.959226732522668 (+0.028685411936799188) | > avg_loss_mel: 17.03260539954816 (-0.13556281613632137) | > avg_loss_duration: 1.556194562307546 (-0.008308032868613813) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.793456843201543 (-0.12635652783890805) | > avg_grad_norm_1: tensor(7226.9453, device='cuda:0') (+tensor(330.3545, device='cuda:0'))  > EPOCH: 865/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:07:28)   --> TIME: 2025-02-25 17:07:38 -- STEP: 19/72 -- GLOBAL_STEP: 1062300 | > loss_disc: 2.390031337738037 (2.4975927628968893) | > loss_disc_real_0: 0.061382971704006195 (0.09722543272532914) | > loss_disc_real_1: 0.21537959575653076 (0.22630291000792854) | > loss_disc_real_2: 0.2205166220664978 (0.227458288010798) | > loss_disc_real_3: 0.22371071577072144 (0.22732749738191305) | > loss_disc_real_4: 0.2252711057662964 (0.2360582594808779) | > loss_disc_real_5: 0.22319120168685913 (0.2380166696874719) | > loss_0: 2.390031337738037 (2.4975927628968893) | > grad_norm_0: tensor(182.5224, device='cuda:0') (tensor(205.3402, device='cuda:0')) | > loss_gen: 2.396921396255493 (2.290899910424885) | > loss_kl: 1.9030776023864746 (1.973839232796117) | > loss_feat: 4.755964279174805 (3.9372398476851616) | > loss_mel: 17.237926483154297 (16.87651915299265) | > loss_duration: 1.4935427904129028 (1.5099156216571206) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.787433624267578 (26.58841374045924) | > grad_norm_1: tensor(5237.6802, device='cuda:0') (tensor(7609.0864, device='cuda:0')) | > current_lr_0: 0.0001795018668534994 | > current_lr_1: 0.0001795018668534994 | > step_time: 0.4397 (0.43660545349121094) | > loader_time: 0.0062 (0.006717255240992496)  --> TIME: 2025-02-25 17:08:02 -- STEP: 69/72 -- GLOBAL_STEP: 1062350 | > loss_disc: 2.813127279281616 (2.5203650343245356) | > loss_disc_real_0: 0.07667434215545654 (0.08950331310431164) | > loss_disc_real_1: 0.25421056151390076 (0.22222724340964053) | > loss_disc_real_2: 0.23103410005569458 (0.22425223541432532) | > loss_disc_real_3: 0.23919954895973206 (0.2330380693294) | > loss_disc_real_4: 0.25641608238220215 (0.24053282949371615) | > loss_disc_real_5: 0.25170886516571045 (0.24514095312443332) | > loss_0: 2.813127279281616 (2.5203650343245356) | > grad_norm_0: tensor(89.4819, device='cuda:0') (tensor(197.7152, device='cuda:0')) | > loss_gen: 1.8538098335266113 (2.2582851527393735) | > loss_kl: 2.165257692337036 (2.0130862204924873) | > loss_feat: 2.845670461654663 (3.923309885937235) | > loss_mel: 17.02374839782715 (16.78198311294334) | > loss_duration: 1.6611686944961548 (1.5534520028293997) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.549654006958008 (26.530116288558297) | > grad_norm_1: tensor(6224.7026, device='cuda:0') (tensor(7268.5654, device='cuda:0')) | > current_lr_0: 0.0001795018668534994 | > current_lr_1: 0.0001795018668534994 | > step_time: 0.5391 (0.4556369401406551) | > loader_time: 0.0128 (0.008575764255247252) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627095692594284 (+8.4353164887703e-06) | > avg_step_time: 0.4572316693588042 (-0.010126335520139862) | > avg_loss_disc: 2.5176115774772536 (-0.017622789866487576) | > avg_loss_disc_real_0: 0.09020054434806529 (+0.00404459537124971) | > avg_loss_disc_real_1: 0.22127183956999175 (-0.002255845867412176) | > avg_loss_disc_real_2: 0.22421538913753672 (-0.0030348065873266883) | > avg_loss_disc_real_3: 0.23290561706247465 (+0.003382641035066525) | > avg_loss_disc_real_4: 0.24038259433188908 (+0.003495577355505708) | > avg_loss_disc_real_5: 0.24486240702615658 (+0.002644158794846335) | > avg_loss_0: 2.5176115774772536 (-0.017622789866487576) | > avg_grad_norm_0: tensor(195.8923, device='cuda:0') (tensor(-5.3685, device='cuda:0')) | > avg_loss_gen: 2.2616134462222255 (+0.028843580836982685) | > avg_loss_kl: 2.0162326483659343 (+0.0035725261124088803) | > avg_loss_feat: 3.921349844462435 (-0.03787688806023315) | > avg_loss_mel: 16.76598243982019 (-0.2666229597279717) | > avg_loss_duration: 1.5543429901902104 (-0.0018515721173355537) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.519521283431793 (-0.27393555976975037) | > avg_grad_norm_1: tensor(7189.7520, device='cuda:0') (tensor(-37.1934, device='cuda:0'))  > EPOCH: 866/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:08:05)   --> TIME: 2025-02-25 17:08:28 -- STEP: 47/72 -- GLOBAL_STEP: 1062400 | > loss_disc: 2.2801804542541504 (2.550828989515913) | > loss_disc_real_0: 0.08066193759441376 (0.11920784841826622) | > loss_disc_real_1: 0.226655513048172 (0.23707432062067885) | > loss_disc_real_2: 0.21856793761253357 (0.23408542731974988) | > loss_disc_real_3: 0.23101988434791565 (0.23517033457756042) | > loss_disc_real_4: 0.2480762004852295 (0.24333968853696863) | > loss_disc_real_5: 0.24217788875102997 (0.2486186959642045) | > loss_0: 2.2801804542541504 (2.550828989515913) | > grad_norm_0: tensor(73.6902, device='cuda:0') (tensor(211.2916, device='cuda:0')) | > loss_gen: 2.6753787994384766 (2.3068544890018217) | > loss_kl: 1.9586127996444702 (1.9919212473199723) | > loss_feat: 4.491805076599121 (3.88168628672336) | > loss_mel: 16.512191772460938 (16.83683750477243) | > loss_duration: 1.5873403549194336 (1.5298474266173991) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.225330352783203 (26.547146939216777) | > grad_norm_1: tensor(4750.3271, device='cuda:0') (tensor(6415.3530, device='cuda:0')) | > current_lr_0: 0.0001794794291201427 | > current_lr_1: 0.0001794794291201427 | > step_time: 0.454 (0.4458329474672358) | > loader_time: 0.0091 (0.007696953225643077) --> EVAL PERFORMANCE | > avg_loader_time: 0.008720401307226906 (+9.33056146326227e-05) | > avg_step_time: 0.458607233745951 (+0.0013755643871468104) | > avg_loss_disc: 2.5752210180524373 (+0.05760944057518369) | > avg_loss_disc_real_0: 0.13021068867873137 (+0.040010144330666075) | > avg_loss_disc_real_1: 0.2399672574140656 (+0.01869541784407386) | > avg_loss_disc_real_2: 0.23435890485703106 (+0.010143515719494345) | > avg_loss_disc_real_3: 0.23660596365660008 (+0.0037003465941254365) | > avg_loss_disc_real_4: 0.24348742021641262 (+0.0031048258845235432) | > avg_loss_disc_real_5: 0.24730901273203568 (+0.0024466057058790958) | > avg_loss_0: 2.5752210180524373 (+0.05760944057518369) | > avg_grad_norm_0: tensor(215.9893, device='cuda:0') (+tensor(20.0970, device='cuda:0')) | > avg_loss_gen: 2.321372102683699 (+0.0597586564614736) | > avg_loss_kl: 2.028716789165013 (+0.01248414079907878) | > avg_loss_feat: 3.8124031584027787 (-0.10894668605965618) | > avg_loss_mel: 16.805785071681917 (+0.039802631861729054) | > avg_loss_duration: 1.5542635430752392 (-7.944711497120949e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.52254069690973 (+0.003019413477936439) | > avg_grad_norm_1: tensor(6227.9751, device='cuda:0') (tensor(-961.7769, device='cuda:0'))  > EPOCH: 867/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:08:42)   --> TIME: 2025-02-25 17:08:55 -- STEP: 25/72 -- GLOBAL_STEP: 1062450 | > loss_disc: 2.5938901901245117 (2.7204840755462647) | > loss_disc_real_0: 0.11063769459724426 (0.16997477009892464) | > loss_disc_real_1: 0.16516435146331787 (0.24215908169746395) | > loss_disc_real_2: 0.17375747859477997 (0.23572594702243804) | > loss_disc_real_3: 0.20702576637268066 (0.2349393570423126) | > loss_disc_real_4: 0.22276103496551514 (0.25055958688259133) | > loss_disc_real_5: 0.23306578397750854 (0.2536181771755219) | > loss_0: 2.5938901901245117 (2.7204840755462647) | > grad_norm_0: tensor(300.6340, device='cuda:0') (tensor(205.1260, device='cuda:0')) | > loss_gen: 1.7333214282989502 (2.0560159254074097) | > loss_kl: 2.063251256942749 (2.045047054290771) | > loss_feat: 3.314058780670166 (3.4020928573608398) | > loss_mel: 17.232080459594727 (17.20605308532715) | > loss_duration: 1.524401307106018 (1.5228191661834714) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.86711311340332 (26.232028274536134) | > grad_norm_1: tensor(11280.6172, device='cuda:0') (tensor(7814.3374, device='cuda:0')) | > current_lr_0: 0.00017945699419150268 | > current_lr_1: 0.00017945699419150268 | > step_time: 0.4443 (0.4405835151672363) | > loader_time: 0.0077 (0.007061119079589844) --> EVAL PERFORMANCE | > avg_loader_time: 0.00876532809835085 (+4.4926791123943086e-05) | > avg_step_time: 0.46634108583692097 (+0.007733852090969939) | > avg_loss_disc: 2.6859186501570145 (+0.11069763210457717) | > avg_loss_disc_real_0: 0.16420731000917063 (+0.033996621330439264) | > avg_loss_disc_real_1: 0.23516436445880945 (-0.004802892955256155) | > avg_loss_disc_real_2: 0.2298169473946934 (-0.0045419574623376535) | > avg_loss_disc_real_3: 0.2320389856754894 (-0.0045669779811106925) | > avg_loss_disc_real_4: 0.24059171575895497 (-0.002895704457457654) | > avg_loss_disc_real_5: 0.24641313057550246 (-0.0008958821565332176) | > avg_loss_0: 2.6859186501570145 (+0.11069763210457717) | > avg_grad_norm_0: tensor(238.5749, device='cuda:0') (+tensor(22.5856, device='cuda:0')) | > avg_loss_gen: 2.0273973488471877 (-0.2939747538365114) | > avg_loss_kl: 2.0943017744682186 (+0.06558498530320556) | > avg_loss_feat: 3.421514682366814 (-0.39088847603596477) | > avg_loss_mel: 17.008583727017253 (+0.2027986553353358) | > avg_loss_duration: 1.5614755338346455 (+0.007211990759406373) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.113273029596034 (-0.4092676673136957) | > avg_grad_norm_1: tensor(7530.5669, device='cuda:0') (+tensor(1302.5918, device='cuda:0'))  > EPOCH: 868/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:09:20)   --> TIME: 2025-02-25 17:09:23 -- STEP: 3/72 -- GLOBAL_STEP: 1062500 | > loss_disc: 2.511324882507324 (2.481550455093384) | > loss_disc_real_0: 0.12805698812007904 (0.08576253553231557) | > loss_disc_real_1: 0.1959165334701538 (0.20999425649642944) | > loss_disc_real_2: 0.26516786217689514 (0.2697781523068746) | > loss_disc_real_3: 0.2633648216724396 (0.25605099896589917) | > loss_disc_real_4: 0.2486831098794937 (0.24743196864922842) | > loss_disc_real_5: 0.24579104781150818 (0.23142911990483603) | > loss_0: 2.511324882507324 (2.481550455093384) | > grad_norm_0: tensor(338.8686, device='cuda:0') (tensor(251.3926, device='cuda:0')) | > loss_gen: 2.273721933364868 (2.1972033182779946) | > loss_kl: 2.134350061416626 (2.0172679821650186) | > loss_feat: 3.3251962661743164 (3.670379638671875) | > loss_mel: 16.121978759765625 (16.597408930460613) | > loss_duration: 1.5545564889907837 (1.521535595258077) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.409801483154297 (26.003794987996418) | > grad_norm_1: tensor(4997.1411, device='cuda:0') (tensor(8099.1694, device='cuda:0')) | > current_lr_0: 0.00017943456206722874 | > current_lr_1: 0.00017943456206722874 | > step_time: 0.4357 (0.43970417976379395) | > loader_time: 0.0061 (0.0064245859781901045)  --> TIME: 2025-02-25 17:09:46 -- STEP: 53/72 -- GLOBAL_STEP: 1062550 | > loss_disc: 2.31857967376709 (2.67420602294634) | > loss_disc_real_0: 0.07816685736179352 (0.17054126186753218) | > loss_disc_real_1: 0.1971925050020218 (0.2249046874777326) | > loss_disc_real_2: 0.20903120934963226 (0.23334577004864532) | > loss_disc_real_3: 0.17772223055362701 (0.23854920071250987) | > loss_disc_real_4: 0.21373280882835388 (0.2418059509880138) | > loss_disc_real_5: 0.2379188984632492 (0.24345605918821298) | > loss_0: 2.31857967376709 (2.67420602294634) | > grad_norm_0: tensor(35.6351, device='cuda:0') (tensor(218.0879, device='cuda:0')) | > loss_gen: 2.4171066284179688 (2.095158972830144) | > loss_kl: 2.2134861946105957 (2.059816718101501) | > loss_feat: 4.661126613616943 (3.5524724249569877) | > loss_mel: 16.75627899169922 (16.951417095256296) | > loss_duration: 1.6119742393493652 (1.5460850360258571) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.65997314453125 (26.204950368629312) | > grad_norm_1: tensor(4324.4111, device='cuda:0') (tensor(7077.4463, device='cuda:0')) | > current_lr_0: 0.00017943456206722874 | > current_lr_1: 0.00017943456206722874 | > step_time: 0.4644 (0.4526911141737452) | > loader_time: 0.0094 (0.007979298537632207) --> EVAL PERFORMANCE | > avg_loader_time: 0.008751990089953789 (-1.3338008397060716e-05) | > avg_step_time: 0.4631376501540063 (-0.003203435682914646) | > avg_loss_disc: 2.694459639804464 (+0.008540989647449315) | > avg_loss_disc_real_0: 0.14705266703812173 (-0.017154642971048906) | > avg_loss_disc_real_1: 0.24076866442468806 (+0.005604299965878606) | > avg_loss_disc_real_2: 0.23017970378130254 (+0.00036275638660912546) | > avg_loss_disc_real_3: 0.239570183233476 (+0.007531197557986619) | > avg_loss_disc_real_4: 0.23637603632581067 (-0.004215679433144298) | > avg_loss_disc_real_5: 0.24506178918019147 (-0.0013513413953109843) | > avg_loss_0: 2.694459639804464 (+0.008540989647449315) | > avg_grad_norm_0: tensor(207.9263, device='cuda:0') (tensor(-30.6487, device='cuda:0')) | > avg_loss_gen: 2.108338617942703 (+0.08094126909551536) | > avg_loss_kl: 2.0863513392461854 (-0.007950435222033203) | > avg_loss_feat: 3.6120926393589503 (+0.19057795699213642) | > avg_loss_mel: 16.9792816000925 (-0.029302126924754646) | > avg_loss_duration: 1.561427374960671 (-4.815887397446872e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.347491734464402 (+0.23421870486836838) | > avg_grad_norm_1: tensor(7043.3872, device='cuda:0') (tensor(-487.1797, device='cuda:0'))  > EPOCH: 869/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:09:57)   --> TIME: 2025-02-25 17:10:13 -- STEP: 31/72 -- GLOBAL_STEP: 1062600 | > loss_disc: 2.3680622577667236 (2.593873746933476) | > loss_disc_real_0: 0.04563119262456894 (0.11598855190940437) | > loss_disc_real_1: 0.2310284525156021 (0.2190005745618574) | > loss_disc_real_2: 0.21285660564899445 (0.24269191776552507) | > loss_disc_real_3: 0.2240983098745346 (0.2421840621579078) | > loss_disc_real_4: 0.2359585016965866 (0.25245237783078217) | > loss_disc_real_5: 0.2392810732126236 (0.24399836121066923) | > loss_0: 2.3680622577667236 (2.593873746933476) | > grad_norm_0: tensor(134.6188, device='cuda:0') (tensor(192.4089, device='cuda:0')) | > loss_gen: 2.2464993000030518 (2.1906832841134842) | > loss_kl: 2.085463762283325 (2.010882173815081) | > loss_feat: 4.40291166305542 (3.7452669374404417) | > loss_mel: 18.157394409179688 (16.997124518117598) | > loss_duration: 1.5520474910736084 (1.5199128812359226) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.444318771362305 (26.463869956231886) | > grad_norm_1: tensor(6301.7354, device='cuda:0') (tensor(6740.6499, device='cuda:0')) | > current_lr_0: 0.00017941213274697034 | > current_lr_1: 0.00017941213274697034 | > step_time: 0.4657 (0.4409962700259301) | > loader_time: 0.0085 (0.0071238548524918096) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655924192616637 (-9.606589733715185e-05) | > avg_step_time: 0.4585816054277017 (-0.004556044726304609) | > avg_loss_disc: 2.6370478146512744 (-0.05741182515318943) | > avg_loss_disc_real_0: 0.15082038611068693 (+0.0037677190725652066) | > avg_loss_disc_real_1: 0.22489864410648883 (-0.01587002031819923) | > avg_loss_disc_real_2: 0.23861912018816236 (+0.008439416406859829) | > avg_loss_disc_real_3: 0.2382858330515069 (-0.0012843501819690961) | > avg_loss_disc_real_4: 0.24617581573170674 (+0.009799779405896075) | > avg_loss_disc_real_5: 0.24338222683315547 (-0.0016795623470360066) | > avg_loss_0: 2.6370478146512744 (-0.05741182515318943) | > avg_grad_norm_0: tensor(202.3761, device='cuda:0') (tensor(-5.5502, device='cuda:0')) | > avg_loss_gen: 2.1531897497848735 (+0.04485113184217049) | > avg_loss_kl: 2.2913056309794038 (+0.20495429173321833) | > avg_loss_feat: 3.5735623836517334 (-0.03853025570721691) | > avg_loss_mel: 17.698247667769312 (+0.7189660676768135) | > avg_loss_duration: 1.577739634983976 (+0.016312260023304992) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.29404521995867 (+0.9465534854942668) | > avg_grad_norm_1: tensor(6437.6362, device='cuda:0') (tensor(-605.7510, device='cuda:0'))  > EPOCH: 870/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:10:34)   --> TIME: 2025-02-25 17:10:40 -- STEP: 9/72 -- GLOBAL_STEP: 1062650 | > loss_disc: 2.9124231338500977 (2.611162132687039) | > loss_disc_real_0: 0.05918622016906738 (0.12517061332861582) | > loss_disc_real_1: 0.229951873421669 (0.23434944285286796) | > loss_disc_real_2: 0.24124903976917267 (0.2365307410558065) | > loss_disc_real_3: 0.2514673173427582 (0.2416568481259876) | > loss_disc_real_4: 0.27985718846321106 (0.2502914137310452) | > loss_disc_real_5: 0.2660220265388489 (0.25150047573778367) | > loss_0: 2.9124231338500977 (2.611162132687039) | > grad_norm_0: tensor(107.1989, device='cuda:0') (tensor(77.6447, device='cuda:0')) | > loss_gen: 1.66841459274292 (2.177478459146288) | > loss_kl: 2.2938034534454346 (2.1397498978508844) | > loss_feat: 2.8230113983154297 (3.8304051558176675) | > loss_mel: 18.16901969909668 (18.31525993347168) | > loss_duration: 1.5299502611160278 (1.5473401414023504) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.484201431274414 (28.010234196980793) | > grad_norm_1: tensor(5376.7056, device='cuda:0') (tensor(5067.6992, device='cuda:0')) | > current_lr_0: 0.00017938970623037695 | > current_lr_1: 0.00017938970623037695 | > step_time: 0.4243 (0.43438559108310276) | > loader_time: 0.0057 (0.007054726282755534)  --> TIME: 2025-02-25 17:11:04 -- STEP: 59/72 -- GLOBAL_STEP: 1062700 | > loss_disc: 2.6452982425689697 (2.675089981596349) | > loss_disc_real_0: 0.21348905563354492 (0.14978961056193055) | > loss_disc_real_1: 0.15546874701976776 (0.2364244416861211) | > loss_disc_real_2: 0.2061498761177063 (0.23576279312877332) | > loss_disc_real_3: 0.20897671580314636 (0.23634039067615897) | > loss_disc_real_4: 0.24795380234718323 (0.2463776499538099) | > loss_disc_real_5: 0.23454083502292633 (0.24522983371201207) | > loss_0: 2.6452982425689697 (2.675089981596349) | > grad_norm_0: tensor(28.9591, device='cuda:0') (tensor(196.3388, device='cuda:0')) | > loss_gen: 1.8955714702606201 (2.0466157521231696) | > loss_kl: 2.177372932434082 (2.2089852458339627) | > loss_feat: 3.23539662361145 (3.440138121782723) | > loss_mel: 16.349830627441406 (17.8976344415697) | > loss_duration: 1.6073731184005737 (1.572247480942031) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.26554298400879 (27.165621094784495) | > grad_norm_1: tensor(15881.1748, device='cuda:0') (tensor(7632.5347, device='cuda:0')) | > current_lr_0: 0.00017938970623037695 | > current_lr_1: 0.00017938970623037695 | > step_time: 0.4735 (0.4579516911910752) | > loader_time: 0.0098 (0.008147530636544956) --> EVAL PERFORMANCE | > avg_loader_time: 0.008715501973326777 (+5.9577780710140527e-05) | > avg_step_time: 0.4664346701662305 (+0.007853064738528814) | > avg_loss_disc: 2.68352759723932 (+0.04647978258804564) | > avg_loss_disc_real_0: 0.1437569348482599 (-0.007063451262427023) | > avg_loss_disc_real_1: 0.24575845424977827 (+0.020859810143289442) | > avg_loss_disc_real_2: 0.24187030947544205 (+0.0032511892872796877) | > avg_loss_disc_real_3: 0.2348270531691296 (-0.0034587798823773053) | > avg_loss_disc_real_4: 0.2444610480271595 (-0.001714767704547232) | > avg_loss_disc_real_5: 0.24535320531314528 (+0.00197097847998981) | > avg_loss_0: 2.68352759723932 (+0.04647978258804564) | > avg_grad_norm_0: tensor(191.1261, device='cuda:0') (tensor(-11.2499, device='cuda:0')) | > avg_loss_gen: 2.0566766245264403 (-0.09651312525843325) | > avg_loss_kl: 2.2938835369029515 (+0.0025779059235477852) | > avg_loss_feat: 3.442265302362576 (-0.13129708128915718) | > avg_loss_mel: 18.090034296814824 (+0.3917866290455123) | > avg_loss_duration: 1.5886579063576711 (+0.010918271373695054) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.471517670322473 (+0.17747245036380477) | > avg_grad_norm_1: tensor(7487.1353, device='cuda:0') (+tensor(1049.4990, device='cuda:0'))  > EPOCH: 871/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:11:12)   --> TIME: 2025-02-25 17:11:31 -- STEP: 37/72 -- GLOBAL_STEP: 1062750 | > loss_disc: 2.6655778884887695 (2.5784128485499203) | > loss_disc_real_0: 0.05488918721675873 (0.11262392519494972) | > loss_disc_real_1: 0.26901814341545105 (0.22434367481115702) | > loss_disc_real_2: 0.20432332158088684 (0.22081104283397263) | > loss_disc_real_3: 0.24483618140220642 (0.2337105346692575) | > loss_disc_real_4: 0.25845402479171753 (0.2425497423958134) | > loss_disc_real_5: 0.2584530711174011 (0.2376135084274653) | > loss_0: 2.6655778884887695 (2.5784128485499203) | > grad_norm_0: tensor(261.0593, device='cuda:0') (tensor(177.5544, device='cuda:0')) | > loss_gen: 1.9931445121765137 (2.1742853022910458) | > loss_kl: 2.119807243347168 (2.109192071734248) | > loss_feat: 3.7004783153533936 (3.7907632299371667) | > loss_mel: 17.392133712768555 (17.6890013926738) | > loss_duration: 1.5606876611709595 (1.5505334364401329) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.766252517700195 (27.31377529453587) | > grad_norm_1: tensor(4866.9424, device='cuda:0') (tensor(5863.7939, device='cuda:0')) | > current_lr_0: 0.00017936728251709814 | > current_lr_1: 0.00017936728251709814 | > step_time: 0.4641 (0.4432966000324971) | > loader_time: 0.01 (0.007383752513576198) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871981029779139 (+4.308324464612906e-06) | > avg_step_time: 0.45769734449789556 (-0.008737325668334961) | > avg_loss_disc: 2.6363756690226814 (-0.047151928216638606) | > avg_loss_disc_real_0: 0.1378622506286057 (-0.005894684219654223) | > avg_loss_disc_real_1: 0.22371671674117236 (-0.022041737508605913) | > avg_loss_disc_real_2: 0.2267391690066163 (-0.015131140468825743) | > avg_loss_disc_real_3: 0.2330183722603489 (-0.0018086809087806965) | > avg_loss_disc_real_4: 0.24197727567713026 (-0.002483772350029251) | > avg_loss_disc_real_5: 0.24237910836515292 (-0.0029740969479923607) | > avg_loss_0: 2.6363756690226814 (-0.047151928216638606) | > avg_grad_norm_0: tensor(198.6508, device='cuda:0') (+tensor(7.5246, device='cuda:0')) | > avg_loss_gen: 2.1016828812343973 (+0.04500625670795699) | > avg_loss_kl: 2.1980680495920315 (-0.09581548731092004) | > avg_loss_feat: 3.5899936649161326 (+0.14772836255355637) | > avg_loss_mel: 17.80620146469331 (-0.2838328321215151) | > avg_loss_duration: 1.5917109049541849 (+0.0030529985965137563) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.28765689151388 (-0.18386077880859375) | > avg_grad_norm_1: tensor(6306.8848, device='cuda:0') (tensor(-1180.2505, device='cuda:0'))  > EPOCH: 872/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:11:49)   --> TIME: 2025-02-25 17:11:58 -- STEP: 15/72 -- GLOBAL_STEP: 1062800 | > loss_disc: 2.6089694499969482 (2.7197419007619223) | > loss_disc_real_0: 0.1464819610118866 (0.16683127383391064) | > loss_disc_real_1: 0.17118114233016968 (0.225054528315862) | > loss_disc_real_2: 0.2769107222557068 (0.2764194826285044) | > loss_disc_real_3: 0.19882436096668243 (0.24275187154610953) | > loss_disc_real_4: 0.23329688608646393 (0.24250317911307018) | > loss_disc_real_5: 0.2493314892053604 (0.2392883062362671) | > loss_0: 2.6089694499969482 (2.7197419007619223) | > grad_norm_0: tensor(427.3894, device='cuda:0') (tensor(193.0351, device='cuda:0')) | > loss_gen: 1.7963439226150513 (2.0629748185475667) | > loss_kl: 2.1432526111602783 (2.124898250897726) | > loss_feat: 3.363462448120117 (3.5716168244679767) | > loss_mel: 17.717159271240234 (18.01808853149414) | > loss_duration: 1.530106782913208 (1.5505149920781454) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.550325393676758 (27.328093338012696) | > grad_norm_1: tensor(10784.9346, device='cuda:0') (tensor(6660.3096, device='cuda:0')) | > current_lr_0: 0.0001793448616067835 | > current_lr_1: 0.0001793448616067835 | > step_time: 0.438 (0.4367336908976237) | > loader_time: 0.0075 (0.006772232055664062)  --> TIME: 2025-02-25 17:12:22 -- STEP: 65/72 -- GLOBAL_STEP: 1062850 | > loss_disc: 2.6378931999206543 (2.6721517966343806) | > loss_disc_real_0: 0.30314886569976807 (0.16257366308799154) | > loss_disc_real_1: 0.224162295460701 (0.22593435931664246) | > loss_disc_real_2: 0.19744376838207245 (0.23797804071353032) | > loss_disc_real_3: 0.22672466933727264 (0.22896821132073036) | > loss_disc_real_4: 0.2129645049571991 (0.240408321756583) | > loss_disc_real_5: 0.2183235138654709 (0.24408331857277796) | > loss_0: 2.6378931999206543 (2.6721517966343806) | > grad_norm_0: tensor(397.3691, device='cuda:0') (tensor(174.5140, device='cuda:0')) | > loss_gen: 2.2082581520080566 (2.0545381069183355) | > loss_kl: 2.0835697650909424 (2.080146178832422) | > loss_feat: 3.729883909225464 (3.48264264693627) | > loss_mel: 17.88298797607422 (17.285913628798262) | > loss_duration: 1.6103649139404297 (1.569021463394165) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.515064239501953 (26.47226201570951) | > grad_norm_1: tensor(3438.0017, device='cuda:0') (tensor(5967.9551, device='cuda:0')) | > current_lr_0: 0.0001793448616067835 | > current_lr_1: 0.0001793448616067835 | > step_time: 0.4929 (0.4511646050673265) | > loader_time: 0.0116 (0.008428393877469575) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696361326835525 (-2.3448970955864887e-05) | > avg_step_time: 0.4569978243868116 (-0.0006995201110839844) | > avg_loss_disc: 2.6781626452862377 (+0.04178697626355632) | > avg_loss_disc_real_0: 0.1654504554372438 (+0.027588204808638106) | > avg_loss_disc_real_1: 0.22700924048541296 (+0.0032925237442406052) | > avg_loss_disc_real_2: 0.2363949653128503 (+0.009655796306234005) | > avg_loss_disc_real_3: 0.2306522481458288 (-0.002366124114520113) | > avg_loss_disc_real_4: 0.240019463317495 (-0.0019578123596352692) | > avg_loss_disc_real_5: 0.24410557348123738 (+0.001726465116084458) | > avg_loss_0: 2.6781626452862377 (+0.04178697626355632) | > avg_grad_norm_0: tensor(178.6329, device='cuda:0') (tensor(-20.0179, device='cuda:0')) | > avg_loss_gen: 2.045027899070525 (-0.056654982163872436) | > avg_loss_kl: 2.086380772187677 (-0.11168727740435447) | > avg_loss_feat: 3.4586718216748307 (-0.13132184324130192) | > avg_loss_mel: 17.24776567539698 (-0.558435789296329) | > avg_loss_duration: 1.5737702292455753 (-0.017940675708609577) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.411616311946386 (-0.8760405795674941) | > avg_grad_norm_1: tensor(5834.1777, device='cuda:0') (tensor(-472.7070, device='cuda:0'))  > EPOCH: 873/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:12:27)   --> TIME: 2025-02-25 17:12:49 -- STEP: 43/72 -- GLOBAL_STEP: 1062900 | > loss_disc: 2.649132013320923 (2.6416061368099477) | > loss_disc_real_0: 0.13795113563537598 (0.1272401234438253) | > loss_disc_real_1: 0.3069351017475128 (0.23148219356703204) | > loss_disc_real_2: 0.21110276877880096 (0.24106954904489739) | > loss_disc_real_3: 0.1911405771970749 (0.2404914030502009) | > loss_disc_real_4: 0.2304699867963791 (0.23836270390554917) | > loss_disc_real_5: 0.23290371894836426 (0.24455229522183883) | > loss_0: 2.649132013320923 (2.6416061368099477) | > grad_norm_0: tensor(21.4139, device='cuda:0') (tensor(163.3321, device='cuda:0')) | > loss_gen: 2.184556484222412 (2.0942841629649314) | > loss_kl: 2.0700857639312744 (2.446448109870733) | > loss_feat: 4.274562835693359 (3.717435908872028) | > loss_mel: 17.624082565307617 (18.973277335943177) | > loss_duration: 1.590879201889038 (1.645673668661783) | > amp_scaler: 16.0 (8.55813953488372) | > loss_1: 27.74416732788086 (28.87711928611578) | > grad_norm_1: tensor(3422.7686, device='cuda:0') (tensor(5318.6782, device='cuda:0')) | > current_lr_0: 0.00017932244349908263 | > current_lr_1: 0.00017932244349908263 | > step_time: 0.4526 (0.45479279340699663) | > loader_time: 0.01 (0.007639962573384129) --> EVAL PERFORMANCE | > avg_loader_time: 0.008731257747596421 (+3.489642076089615e-05) | > avg_step_time: 0.4646643786363199 (+0.007666554249508295) | > avg_loss_disc: 2.6659846440167496 (-0.012178001269488092) | > avg_loss_disc_real_0: 0.13112489149813922 (-0.034325563939104575) | > avg_loss_disc_real_1: 0.22727670531037827 (+0.0002674648249653133) | > avg_loss_disc_real_2: 0.24355475121820477 (+0.00715978590535446) | > avg_loss_disc_real_3: 0.2460216800091972 (+0.015369431863368394) | > avg_loss_disc_real_4: 0.2412131474891179 (+0.0011936841716229107) | > avg_loss_disc_real_5: 0.24630161385301133 (+0.002196040371773955) | > avg_loss_0: 2.6659846440167496 (-0.012178001269488092) | > avg_grad_norm_0: tensor(174.4792, device='cuda:0') (tensor(-4.1537, device='cuda:0')) | > avg_loss_gen: 2.03391079499688 (-0.011117104073644857) | > avg_loss_kl: 2.3297246644194702 (+0.2433438922317932) | > avg_loss_feat: 3.5512652262835434 (+0.09259340460871268) | > avg_loss_mel: 18.2845363079662 (+1.036770632569219) | > avg_loss_duration: 1.633098288321159 (+0.059328059075583806) | > avg_amp_scaler: 11.49295774647887 (+3.4929577464788704) | > avg_loss_1: 27.832535407912562 (+1.4209190959661768) | > avg_grad_norm_1: tensor(6077.6338, device='cuda:0') (+tensor(243.4561, device='cuda:0'))  > EPOCH: 874/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:13:04)   --> TIME: 2025-02-25 17:13:16 -- STEP: 21/72 -- GLOBAL_STEP: 1062950 | > loss_disc: 2.7987236976623535 (2.5874800795600525) | > loss_disc_real_0: 0.056341636925935745 (0.11089018785527774) | > loss_disc_real_1: 0.22422245144844055 (0.2105399909473601) | > loss_disc_real_2: 0.22147563099861145 (0.2312827223823184) | > loss_disc_real_3: 0.2233923077583313 (0.2280476213920684) | > loss_disc_real_4: 0.24445286393165588 (0.2412810857806887) | > loss_disc_real_5: 0.24243493378162384 (0.24337797505514963) | > loss_0: 2.7987236976623535 (2.5874800795600525) | > grad_norm_0: tensor(235.8165, device='cuda:0') (tensor(205.5004, device='cuda:0')) | > loss_gen: 1.6907236576080322 (2.044088113875616) | > loss_kl: 2.03728985786438 (1.984536954334804) | > loss_feat: 2.973637580871582 (3.6391267890021917) | > loss_mel: 16.370290756225586 (16.974616232372465) | > loss_duration: 1.539775013923645 (1.5333204553240822) | > amp_scaler: 16.0 (16.0) | > loss_1: 24.611717224121094 (26.17568860735212) | > grad_norm_1: tensor(7732.7964, device='cuda:0') (tensor(6077.7002, device='cuda:0')) | > current_lr_0: 0.00017930002819364525 | > current_lr_1: 0.00017930002819364525 | > step_time: 0.4353 (0.4381905510312035) | > loader_time: 0.0067 (0.006840217681158157)  --> TIME: 2025-02-25 17:13:40 -- STEP: 71/72 -- GLOBAL_STEP: 1063000 | > loss_disc: 2.7009518146514893 (2.6669483654935595) | > loss_disc_real_0: 0.10325812548398972 (0.15661504481669883) | > loss_disc_real_1: 0.2100578099489212 (0.21888427016600756) | > loss_disc_real_2: 0.22110415995121002 (0.23288192673468253) | > loss_disc_real_3: 0.2308061420917511 (0.22988679669272732) | > loss_disc_real_4: 0.2382468581199646 (0.23899159716888213) | > loss_disc_real_5: 0.22453220188617706 (0.24231773614883423) | > loss_0: 2.7009518146514893 (2.6669483654935595) | > grad_norm_0: tensor(74.6330, device='cuda:0') (tensor(187.6496, device='cuda:0')) | > loss_gen: 1.8275631666183472 (2.0216843611757516) | > loss_kl: 2.0535082817077637 (2.087123808726458) | > loss_feat: 3.460237503051758 (3.409466864357532) | > loss_mel: 17.270641326904297 (16.995767203854843) | > loss_duration: 1.5744826793670654 (1.568609570113706) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.186431884765625 (26.08265189049949) | > grad_norm_1: tensor(6652.5483, device='cuda:0') (tensor(5706.5527, device='cuda:0')) | > current_lr_0: 0.00017930002819364525 | > current_lr_1: 0.00017930002819364525 | > step_time: 0.4829 (0.45806704440587004) | > loader_time: 0.0082 (0.00866794586181641) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866794586181641 (-6.331188578001176e-05) | > avg_step_time: 0.45806704440587004 (-0.006597334230449836) | > avg_loss_disc: 2.6669483654935595 (+0.0009637214768098978) | > avg_loss_disc_real_0: 0.15661504481669883 (+0.02549015331855961) | > avg_loss_disc_real_1: 0.21888427016600756 (-0.008392435144370713) | > avg_loss_disc_real_2: 0.23288192673468253 (-0.010672824483522247) | > avg_loss_disc_real_3: 0.22988679669272732 (-0.016134883316469867) | > avg_loss_disc_real_4: 0.23899159716888213 (-0.002221550320235771) | > avg_loss_disc_real_5: 0.24231773614883423 (-0.003983877704177102) | > avg_loss_0: 2.6669483654935595 (+0.0009637214768098978) | > avg_grad_norm_0: tensor(187.6496, device='cuda:0') (+tensor(13.1704, device='cuda:0')) | > avg_loss_gen: 2.0216843611757516 (-0.012226433821128424) | > avg_loss_kl: 2.087123808726458 (-0.24260085569301237) | > avg_loss_feat: 3.409466864357532 (-0.14179836192601147) | > avg_loss_mel: 16.995767203854843 (-1.2887691041113563) | > avg_loss_duration: 1.568609570113706 (-0.06448871820745317) | > avg_amp_scaler: 16.0 (+4.50704225352113) | > avg_loss_1: 26.08265189049949 (-1.7498835174130711) | > avg_grad_norm_1: tensor(5706.5527, device='cuda:0') (tensor(-371.0811, device='cuda:0'))  > EPOCH: 875/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:13:41)   --> TIME: 2025-02-25 17:14:06 -- STEP: 49/72 -- GLOBAL_STEP: 1063050 | > loss_disc: 2.736345052719116 (2.6547643700424506) | > loss_disc_real_0: 0.0757167860865593 (0.15134668217173644) | > loss_disc_real_1: 0.24476571381092072 (0.22926799207925797) | > loss_disc_real_2: 0.18064266443252563 (0.22748735030086673) | > loss_disc_real_3: 0.23316647112369537 (0.24245929900480775) | > loss_disc_real_4: 0.2162666916847229 (0.24890935329758385) | > loss_disc_real_5: 0.2712019383907318 (0.2464400709283595) | > loss_0: 2.736345052719116 (2.6547643700424506) | > grad_norm_0: tensor(81.4190, device='cuda:0') (tensor(208.3551, device='cuda:0')) | > loss_gen: 1.7492742538452148 (2.196331771052614) | > loss_kl: 2.357369899749756 (2.2049664015672645) | > loss_feat: 3.1810617446899414 (3.790798099673524) | > loss_mel: 18.18045997619629 (17.929601377370407) | > loss_duration: 1.6542320251464844 (1.5565618763164595) | > amp_scaler: 16.0 (16.0) | > loss_1: 27.122398376464844 (27.678259752234634) | > grad_norm_1: tensor(5783.6108, device='cuda:0') (tensor(6115.5029, device='cuda:0')) | > current_lr_0: 0.00017927761569012102 | > current_lr_1: 0.00017927761569012102 | > step_time: 0.4465 (0.4449198829884432) | > loader_time: 0.0097 (0.007739952632359096) --> EVAL PERFORMANCE | > avg_loader_time: 0.008698104133068678 (+3.015827125226779e-05) | > avg_step_time: 0.4586016191563136 (+0.0005345747504435505) | > avg_loss_disc: 2.6009866889093964 (-0.06596167658416308) | > avg_loss_disc_real_0: 0.12374222824271298 (-0.032872816573985844) | > avg_loss_disc_real_1: 0.22416416870456346 (+0.005279898538555899) | > avg_loss_disc_real_2: 0.22909635578242826 (-0.0037855709522542635) | > avg_loss_disc_real_3: 0.23638354535673706 (+0.006496748664009733) | > avg_loss_disc_real_4: 0.24586282523585032 (+0.006871228066968188) | > avg_loss_disc_real_5: 0.24494863374972006 (+0.0026308976008858342) | > avg_loss_0: 2.6009866889093964 (-0.06596167658416308) | > avg_grad_norm_0: tensor(182.0206, device='cuda:0') (tensor(-5.6290, device='cuda:0')) | > avg_loss_gen: 2.209195425812628 (+0.1875110646368765) | > avg_loss_kl: 2.2398702812866427 (+0.1527464725601848) | > avg_loss_feat: 3.87010090451845 (+0.4606340401609179) | > avg_loss_mel: 17.84401804964307 (+0.8482508457882254) | > avg_loss_duration: 1.5770834029560357 (+0.00847383284232972) | > avg_amp_scaler: 16.0 (+0.0) | > avg_loss_1: 27.74026819685815 (+1.6576163063586584) | > avg_grad_norm_1: tensor(5868.9834, device='cuda:0') (+tensor(162.4307, device='cuda:0'))  > EPOCH: 876/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:14:19)   --> TIME: 2025-02-25 17:14:33 -- STEP: 27/72 -- GLOBAL_STEP: 1063100 | > loss_disc: 2.532346725463867 (2.4779588381449376) | > loss_disc_real_0: 0.050569117069244385 (0.06626558359022494) | > loss_disc_real_1: 0.19462613761425018 (0.21847785898932703) | > loss_disc_real_2: 0.1777612715959549 (0.219487758146392) | > loss_disc_real_3: 0.23533035814762115 (0.2396450909199538) | > loss_disc_real_4: 0.2347349375486374 (0.24079614656942863) | > loss_disc_real_5: 0.22111046314239502 (0.2466742385316778) | > loss_0: 2.532346725463867 (2.4779588381449376) | > grad_norm_0: tensor(141.2757, device='cuda:0') (tensor(146.1526, device='cuda:0')) | > loss_gen: 2.117905855178833 (2.291182319323222) | > loss_kl: 1.8837125301361084 (2.0005512105094057) | > loss_feat: 4.169893741607666 (4.137039829183508) | > loss_mel: 16.477012634277344 (17.39082188076443) | > loss_duration: 1.521597146987915 (1.5274610033741705) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.170122146606445 (27.34705642417625) | > grad_norm_1: tensor(3372.7764, device='cuda:0') (tensor(6275.9097, device='cuda:0')) | > current_lr_0: 0.00017925520598815976 | > current_lr_1: 0.00017925520598815976 | > step_time: 0.4344 (0.43747973442077637) | > loader_time: 0.0088 (0.007150155526620371) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087782127756468 (+8.010864257812327e-05) | > avg_step_time: 0.4572334759671923 (-0.0013681431891212625) | > avg_loss_disc: 2.646229623069226 (+0.045242934159829495) | > avg_loss_disc_real_0: 0.16169018479405153 (+0.03794795655133855) | > avg_loss_disc_real_1: 0.22989281319396596 (+0.005728644489402496) | > avg_loss_disc_real_2: 0.23751349285454817 (+0.008417137072119912) | > avg_loss_disc_real_3: 0.23584501508256078 (-0.0005385302741762743) | > avg_loss_disc_real_4: 0.240019033912202 (-0.0058437913236483174) | > avg_loss_disc_real_5: 0.24860108108587667 (+0.0036524473361566057) | > avg_loss_0: 2.646229623069226 (+0.045242934159829495) | > avg_grad_norm_0: tensor(198.2736, device='cuda:0') (+tensor(16.2530, device='cuda:0')) | > avg_loss_gen: 2.1748586322220285 (-0.0343367935905996) | > avg_loss_kl: 2.092620183044756 (-0.14725009824188673) | > avg_loss_feat: 3.6792954491897367 (-0.19080545532871307) | > avg_loss_mel: 17.443508725770766 (-0.40050932387230276) | > avg_loss_duration: 1.5693774122587392 (-0.007705990697296494) | > avg_amp_scaler: 12.394366197183098 (-3.605633802816902) | > avg_loss_1: 26.9596605166583 (-0.78060768019985) | > avg_grad_norm_1: tensor(6719.3311, device='cuda:0') (+tensor(850.3477, device='cuda:0'))  > EPOCH: 877/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:14:56)   --> TIME: 2025-02-25 17:15:00 -- STEP: 5/72 -- GLOBAL_STEP: 1063150 | > loss_disc: 2.376551628112793 (2.5069427490234375) | > loss_disc_real_0: 0.1245410367846489 (0.15240734666585923) | > loss_disc_real_1: 0.23389218747615814 (0.2072205811738968) | > loss_disc_real_2: 0.23125028610229492 (0.2376344084739685) | > loss_disc_real_3: 0.20564036071300507 (0.21002425849437714) | > loss_disc_real_4: 0.23861786723136902 (0.22244400084018706) | > loss_disc_real_5: 0.22658118605613708 (0.22589477002620698) | > loss_0: 2.376551628112793 (2.5069427490234375) | > grad_norm_0: tensor(22.5487, device='cuda:0') (tensor(71.3284, device='cuda:0')) | > loss_gen: 2.3433003425598145 (2.247741627693176) | > loss_kl: 2.0305981636047363 (2.099238872528076) | > loss_feat: 4.140064716339111 (3.8360291957855224) | > loss_mel: 17.468957901000977 (17.33199462890625) | > loss_duration: 1.5057718753814697 (1.5399163246154786) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.488693237304688 (27.054921340942382) | > grad_norm_1: tensor(8220.1143, device='cuda:0') (tensor(5571.8550, device='cuda:0')) | > current_lr_0: 0.00017923279908741124 | > current_lr_1: 0.00017923279908741124 | > step_time: 0.4311 (0.43824291229248047) | > loader_time: 0.0095 (0.006675815582275391)  --> TIME: 2025-02-25 17:15:23 -- STEP: 55/72 -- GLOBAL_STEP: 1063200 | > loss_disc: 2.4868154525756836 (2.658138929713857) | > loss_disc_real_0: 0.05312623083591461 (0.13599114695733244) | > loss_disc_real_1: 0.256660133600235 (0.22809562412175266) | > loss_disc_real_2: 0.18552342057228088 (0.2402016257697886) | > loss_disc_real_3: 0.19001387059688568 (0.23110082312063737) | > loss_disc_real_4: 0.25558435916900635 (0.24785606590184298) | > loss_disc_real_5: 0.22101925313472748 (0.2396940824660388) | > loss_0: 2.4868154525756836 (2.658138929713857) | > grad_norm_0: tensor(190.8120, device='cuda:0') (tensor(197.8172, device='cuda:0')) | > loss_gen: 2.236196994781494 (2.051072079485113) | > loss_kl: 3.338620901107788 (2.4472361196171164) | > loss_feat: 4.293288707733154 (3.61469574841586) | > loss_mel: 28.452138900756836 (19.4077576897361) | > loss_duration: 1.8890464305877686 (1.6665812470696189) | > amp_scaler: 8.0 (8.0) | > loss_1: 40.209293365478516 (29.187343042547052) | > grad_norm_1: tensor(9523.0322, device='cuda:0') (tensor(5956.7866, device='cuda:0')) | > current_lr_0: 0.00017923279908741124 | > current_lr_1: 0.00017923279908741124 | > step_time: 0.4753 (0.4459713719107888) | > loader_time: 0.01 (0.007883713462136009) --> EVAL PERFORMANCE | > avg_loader_time: 0.008567360085500797 (-0.0002108526901460038) | > avg_step_time: 0.4583133509461309 (+0.0010798749789385775) | > avg_loss_disc: 2.6667810057250554 (+0.020551382655829453) | > avg_loss_disc_real_0: 0.13506607265329693 (-0.026624112140754608) | > avg_loss_disc_real_1: 0.22987168376714412 (-2.1129426821836494e-05) | > avg_loss_disc_real_2: 0.23779973303767996 (+0.0002862401831317807) | > avg_loss_disc_real_3: 0.23231533820360478 (-0.0035296768789560007) | > avg_loss_disc_real_4: 0.24784581887889917 (+0.00782678496669717) | > avg_loss_disc_real_5: 0.24064323012257965 (-0.007957850963297014) | > avg_loss_0: 2.6667810057250554 (+0.020551382655829453) | > avg_grad_norm_0: tensor(189.0126, device='cuda:0') (tensor(-9.2610, device='cuda:0')) | > avg_loss_gen: 2.0600357156404305 (-0.11482291658159793) | > avg_loss_kl: 2.5363247075551 (+0.4437045245103439) | > avg_loss_feat: 3.564823677842046 (-0.1144717713476906) | > avg_loss_mel: 19.87974092993937 (+2.436232204168604) | > avg_loss_duration: 1.6890665712490889 (+0.11968915899034971) | > avg_amp_scaler: 8.0 (-4.394366197183098) | > avg_loss_1: 29.729991751657405 (+2.770331234999105) | > avg_grad_norm_1: tensor(6053.1602, device='cuda:0') (tensor(-666.1709, device='cuda:0'))  > EPOCH: 878/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:15:33)   --> TIME: 2025-02-25 17:15:50 -- STEP: 33/72 -- GLOBAL_STEP: 1063250 | > loss_disc: 2.5731027126312256 (2.7902937802401455) | > loss_disc_real_0: 0.10551172494888306 (0.2193053996924198) | > loss_disc_real_1: 0.26989325881004333 (0.2622668081612298) | > loss_disc_real_2: 0.20792366564273834 (0.23590048605745492) | > loss_disc_real_3: 0.2835347056388855 (0.22590979211258166) | > loss_disc_real_4: 0.2752363979816437 (0.23993785634185327) | > loss_disc_real_5: 0.26677700877189636 (0.24156362360174005) | > loss_0: 2.5731027126312256 (2.7902937802401455) | > grad_norm_0: tensor(183.0687, device='cuda:0') (tensor(212.2120, device='cuda:0')) | > loss_gen: 2.299816608428955 (2.1451612963820947) | > loss_kl: 2.150167465209961 (2.092909726229581) | > loss_feat: 3.7053046226501465 (3.4472774231072627) | > loss_mel: 17.637332916259766 (17.855619661735766) | > loss_duration: 1.558974266052246 (1.5774997003150708) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.35159683227539 (27.118467677723277) | > grad_norm_1: tensor(3683.7568, device='cuda:0') (tensor(6025.4941, device='cuda:0')) | > current_lr_0: 0.0001792103949875253 | > current_lr_1: 0.0001792103949875253 | > step_time: 0.4705 (0.441494370951797) | > loader_time: 0.0083 (0.0071344664602568655) --> EVAL PERFORMANCE | > avg_loader_time: 0.008676448338468312 (+0.00010908825296751523) | > avg_step_time: 0.4597985610155992 (+0.001485210069468279) | > avg_loss_disc: 2.7785156276864065 (+0.11173462196135109) | > avg_loss_disc_real_0: 0.2244631933494353 (+0.08939712069613837) | > avg_loss_disc_real_1: 0.24833099302691464 (+0.018459309259770518) | > avg_loss_disc_real_2: 0.24286615764591055 (+0.005066424608230591) | > avg_loss_disc_real_3: 0.23393602438375982 (+0.0016206861801550398) | > avg_loss_disc_real_4: 0.24070312733381566 (-0.007142691545083513) | > avg_loss_disc_real_5: 0.24603392918344955 (+0.005390699060869891) | > avg_loss_0: 2.7785156276864065 (+0.11173462196135109) | > avg_grad_norm_0: tensor(224.0775, device='cuda:0') (+tensor(35.0649, device='cuda:0')) | > avg_loss_gen: 2.070688462593186 (+0.010652746952755532) | > avg_loss_kl: 2.1505585388398507 (-0.38576616871524916) | > avg_loss_feat: 3.366557970852919 (-0.19826570698912693) | > avg_loss_mel: 17.685394313973443 (-2.1943466159659266) | > avg_loss_duration: 1.5949352123367955 (-0.0941313589122934) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.86813448516416 (-2.861857266493246) | > avg_grad_norm_1: tensor(6080.9575, device='cuda:0') (+tensor(27.7974, device='cuda:0'))  > EPOCH: 879/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:16:11)   --> TIME: 2025-02-25 17:16:17 -- STEP: 11/72 -- GLOBAL_STEP: 1063300 | > loss_disc: 2.394904136657715 (2.570083661512895) | > loss_disc_real_0: 0.053492918610572815 (0.16360861638730223) | > loss_disc_real_1: 0.21989569067955017 (0.22458032315427606) | > loss_disc_real_2: 0.21412675082683563 (0.2320193133570931) | > loss_disc_real_3: 0.24002940952777863 (0.2314803031357852) | > loss_disc_real_4: 0.21685229241847992 (0.23214658959345383) | > loss_disc_real_5: 0.23813524842262268 (0.24052119796926325) | > loss_0: 2.394904136657715 (2.570083661512895) | > grad_norm_0: tensor(10.4790, device='cuda:0') (tensor(206.3207, device='cuda:0')) | > loss_gen: 2.2632153034210205 (2.13350067355416) | > loss_kl: 2.0198333263397217 (2.032259626822038) | > loss_feat: 4.066105365753174 (3.6409976482391357) | > loss_mel: 18.22405433654785 (17.632979653098367) | > loss_duration: 1.5095598697662354 (1.5378877249631016) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.0827693939209 (26.97762506658381) | > grad_norm_1: tensor(3534.6494, device='cuda:0') (tensor(4713.9893, device='cuda:0')) | > current_lr_0: 0.00017918799368815187 | > current_lr_1: 0.00017918799368815187 | > step_time: 0.4335 (0.4330978176810525) | > loader_time: 0.0057 (0.006675243377685547)  --> TIME: 2025-02-25 17:16:41 -- STEP: 61/72 -- GLOBAL_STEP: 1063350 | > loss_disc: 2.53450083732605 (2.662545802163296) | > loss_disc_real_0: 0.14489442110061646 (0.1546354532608243) | > loss_disc_real_1: 0.1700410693883896 (0.22246299635191433) | > loss_disc_real_2: 0.2214258313179016 (0.23718589446583732) | > loss_disc_real_3: 0.22249269485473633 (0.23711601021837014) | > loss_disc_real_4: 0.2421269416809082 (0.24202965395372422) | > loss_disc_real_5: 0.23601588606834412 (0.2468588423044955) | > loss_0: 2.53450083732605 (2.662545802163296) | > grad_norm_0: tensor(265.5509, device='cuda:0') (tensor(169.3899, device='cuda:0')) | > loss_gen: 2.018407106399536 (2.0125906721490328) | > loss_kl: 2.0982847213745117 (2.0568405565668333) | > loss_feat: 3.294663906097412 (3.437091495169968) | > loss_mel: 16.549888610839844 (17.348161228367534) | > loss_duration: 1.599434733390808 (1.559353554835085) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.560678482055664 (26.414037548127723) | > grad_norm_1: tensor(5136.9927, device='cuda:0') (tensor(5890.5337, device='cuda:0')) | > current_lr_0: 0.00017918799368815187 | > current_lr_1: 0.00017918799368815187 | > step_time: 0.4821 (0.45115522478447584) | > loader_time: 0.0105 (0.00850693124239562) --> EVAL PERFORMANCE | > avg_loader_time: 0.008930609259806889 (+0.0002541609213385764) | > avg_step_time: 0.46008518044377716 (+0.00028661942817798103) | > avg_loss_disc: 2.6654132419908554 (-0.11310238569555109) | > avg_loss_disc_real_0: 0.150998746058051 (-0.0734644472913843) | > avg_loss_disc_real_1: 0.2233085401461158 (-0.025022452880798823) | > avg_loss_disc_real_2: 0.2339588573700945 (-0.008907300275816044) | > avg_loss_disc_real_3: 0.2343320630385842 (+0.0003960386548243766) | > avg_loss_disc_real_4: 0.24014956934351317 (-0.000553557990302489) | > avg_loss_disc_real_5: 0.2463415779278312 (+0.0003076487443816611) | > avg_loss_0: 2.6654132419908554 (-0.11310238569555109) | > avg_grad_norm_0: tensor(170.9732, device='cuda:0') (tensor(-53.1043, device='cuda:0')) | > avg_loss_gen: 2.0012744681935914 (-0.06941399439959461) | > avg_loss_kl: 2.071128294501507 (-0.07943024433834367) | > avg_loss_feat: 3.4034500760091864 (+0.036892105156267174) | > avg_loss_mel: 17.302309304895527 (-0.38308500907791654) | > avg_loss_duration: 1.567204901869868 (-0.027730310466927532) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.345367082407776 (-0.5227674027563829) | > avg_grad_norm_1: tensor(5966.3169, device='cuda:0') (tensor(-114.6406, device='cuda:0'))  > EPOCH: 880/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:16:48)   --> TIME: 2025-02-25 17:17:08 -- STEP: 39/72 -- GLOBAL_STEP: 1063400 | > loss_disc: 2.523583173751831 (2.6225218772888184) | > loss_disc_real_0: 0.091047503054142 (0.13596064291703394) | > loss_disc_real_1: 0.3260675370693207 (0.21001085963768837) | > loss_disc_real_2: 0.24207168817520142 (0.23289419901676667) | > loss_disc_real_3: 0.23328444361686707 (0.2285328270533146) | > loss_disc_real_4: 0.25455060601234436 (0.25013713729687226) | > loss_disc_real_5: 0.2600165605545044 (0.24904122375524962) | > loss_0: 2.523583173751831 (2.6225218772888184) | > grad_norm_0: tensor(113.7588, device='cuda:0') (tensor(178.3849, device='cuda:0')) | > loss_gen: 2.4783079624176025 (2.157472408734835) | > loss_kl: 2.276930809020996 (2.0882687843762913) | > loss_feat: 3.8420779705047607 (3.743770183661045) | > loss_mel: 18.337196350097656 (17.42730610187237) | > loss_duration: 1.5918090343475342 (1.5403604476879804) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.526321411132812 (26.957178018031975) | > grad_norm_1: tensor(3602.1506, device='cuda:0') (tensor(5447.4458, device='cuda:0')) | > current_lr_0: 0.00017916559518894084 | > current_lr_1: 0.00017916559518894084 | > step_time: 0.4578 (0.44135221456870055) | > loader_time: 0.0084 (0.007389062490218725) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864827465003645 (-0.0002823346097704394) | > avg_step_time: 0.45695943899557623 (-0.003125741448200936) | > avg_loss_disc: 2.616457992876079 (-0.04895524911477622) | > avg_loss_disc_real_0: 0.12671622119739018 (-0.02428252486066082) | > avg_loss_disc_real_1: 0.20830475287118427 (-0.01500378727493154) | > avg_loss_disc_real_2: 0.23429788858957692 (+0.0003390312194824219) | > avg_loss_disc_real_3: 0.23252858914120098 (-0.0018034738973832232) | > avg_loss_disc_real_4: 0.2518969004003095 (+0.011747331056796317) | > avg_loss_disc_real_5: 0.25202840125896564 (+0.005686823331134433) | > avg_loss_0: 2.616457992876079 (-0.04895524911477622) | > avg_grad_norm_0: tensor(187.1045, device='cuda:0') (+tensor(16.1313, device='cuda:0')) | > avg_loss_gen: 2.1155415890922007 (+0.11426712089860924) | > avg_loss_kl: 2.425264633877177 (+0.3541363393756698) | > avg_loss_feat: 3.6562001638009516 (+0.2527500877917652) | > avg_loss_mel: 18.65247964187407 (+1.3501703369785432) | > avg_loss_duration: 1.640994129046588 (+0.07378922717671998) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.49048034238144 (+2.1451132599736624) | > avg_grad_norm_1: tensor(7306.2251, device='cuda:0') (+tensor(1339.9082, device='cuda:0'))  > EPOCH: 881/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:17:25)   --> TIME: 2025-02-25 17:17:35 -- STEP: 17/72 -- GLOBAL_STEP: 1063450 | > loss_disc: 2.3728511333465576 (2.483928357853609) | > loss_disc_real_0: 0.10957706719636917 (0.09581481994074934) | > loss_disc_real_1: 0.2663159668445587 (0.2043297124259612) | > loss_disc_real_2: 0.21897870302200317 (0.22821184642174663) | > loss_disc_real_3: 0.21935923397541046 (0.22469406881753137) | > loss_disc_real_4: 0.22934654355049133 (0.23852255414513981) | > loss_disc_real_5: 0.24083033204078674 (0.237759588395848) | > loss_0: 2.3728511333465576 (2.483928357853609) | > grad_norm_0: tensor(30.1754, device='cuda:0') (tensor(255.8680, device='cuda:0')) | > loss_gen: 2.2790260314941406 (2.122451866374296) | > loss_kl: 2.517115831375122 (2.5103008045869717) | > loss_feat: 4.217319965362549 (4.011567480423871) | > loss_mel: 19.288105010986328 (19.368974910062903) | > loss_duration: 1.61098313331604 (1.5974209518993603) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.91254997253418 (29.610716202679804) | > grad_norm_1: tensor(12166.3330, device='cuda:0') (tensor(11567.1260, device='cuda:0')) | > current_lr_0: 0.00017914319948954222 | > current_lr_1: 0.00017914319948954222 | > step_time: 0.4512 (0.4452081708347096) | > loader_time: 0.0061 (0.00677037239074707)  --> TIME: 2025-02-25 17:17:59 -- STEP: 67/72 -- GLOBAL_STEP: 1063500 | > loss_disc: 2.5548136234283447 (2.575585827898623) | > loss_disc_real_0: 0.07233869284391403 (0.10770579788889456) | > loss_disc_real_1: 0.21861982345581055 (0.2134259088493105) | > loss_disc_real_2: 0.19349731504917145 (0.23811378185428791) | > loss_disc_real_3: 0.2161797434091568 (0.225193870378964) | > loss_disc_real_4: 0.20707722008228302 (0.23954468253833144) | > loss_disc_real_5: 0.220763698220253 (0.24277654712769522) | > loss_0: 2.5548136234283447 (2.575585827898623) | > grad_norm_0: tensor(279.9939, device='cuda:0') (tensor(253.1028, device='cuda:0')) | > loss_gen: 2.020796537399292 (2.0767650177229697) | > loss_kl: 2.258539915084839 (2.303241074974857) | > loss_feat: 4.291008472442627 (3.7661002073714984) | > loss_mel: 18.634920120239258 (18.27788563628695) | > loss_duration: 1.6538214683532715 (1.5984544522726714) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.859085083007812 (28.022446646619198) | > grad_norm_1: tensor(14233.2578, device='cuda:0') (tensor(11695.3066, device='cuda:0')) | > current_lr_0: 0.00017914319948954222 | > current_lr_1: 0.00017914319948954222 | > step_time: 0.5399 (0.4558739626585548) | > loader_time: 0.0123 (0.008499622344970703) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867590434114698 (+2.762969111053115e-05) | > avg_step_time: 0.4601330085539482 (+0.003173569558371947) | > avg_loss_disc: 2.5778369399863226 (-0.03862105288975659) | > avg_loss_disc_real_0: 0.1064081297479045 (-0.02030809144948567) | > avg_loss_disc_real_1: 0.21462541766149898 (+0.006320664790314706) | > avg_loss_disc_real_2: 0.23801171632719711 (+0.00371382773762019) | > avg_loss_disc_real_3: 0.22644611206692708 (-0.006082477074273895) | > avg_loss_disc_real_4: 0.23899840783904974 (-0.012898492561259745) | > avg_loss_disc_real_5: 0.2430949372724748 (-0.008933463986490836) | > avg_loss_0: 2.5778369399863226 (-0.03862105288975659) | > avg_grad_norm_0: tensor(260.2955, device='cuda:0') (+tensor(73.1910, device='cuda:0')) | > avg_loss_gen: 2.073451681875847 (-0.04208990721635386) | > avg_loss_kl: 2.3004427325557657 (-0.12482190132141113) | > avg_loss_feat: 3.7517683606752206 (+0.09556819687426898) | > avg_loss_mel: 18.22168403947857 (-0.43079560239549863) | > avg_loss_duration: 1.60004675724137 (-0.040947371805218014) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.947393793455312 (-0.5430865489261265) | > avg_grad_norm_1: tensor(11928.4512, device='cuda:0') (+tensor(4622.2261, device='cuda:0'))  > EPOCH: 882/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:18:03)   --> TIME: 2025-02-25 17:18:25 -- STEP: 45/72 -- GLOBAL_STEP: 1063550 | > loss_disc: 2.5914711952209473 (2.561720604366726) | > loss_disc_real_0: 0.10828134417533875 (0.10901693420277701) | > loss_disc_real_1: 0.23065021634101868 (0.21139193077882132) | > loss_disc_real_2: 0.2345825731754303 (0.23629883229732512) | > loss_disc_real_3: 0.24976828694343567 (0.23290923602051206) | > loss_disc_real_4: 0.25563687086105347 (0.24165658520327674) | > loss_disc_real_5: 0.2631503939628601 (0.24600172738234202) | > loss_0: 2.5914711952209473 (2.561720604366726) | > grad_norm_0: tensor(254.8045, device='cuda:0') (tensor(310.9377, device='cuda:0')) | > loss_gen: 2.0864768028259277 (2.1091837379667493) | > loss_kl: 1.9600636959075928 (1.996592873997158) | > loss_feat: 3.4240591526031494 (3.7218611929151746) | > loss_mel: 16.25309944152832 (17.16204511854384) | > loss_duration: 1.576688528060913 (1.5477364646063911) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.30038833618164 (26.53741912841797) | > grad_norm_1: tensor(13008.4229, device='cuda:0') (tensor(13830.9209, device='cuda:0')) | > current_lr_0: 0.000179120806589606 | > current_lr_1: 0.000179120806589606 | > step_time: 0.4666 (0.44434008598327634) | > loader_time: 0.0088 (0.00776418579949273) --> EVAL PERFORMANCE | > avg_loader_time: 0.008793202924056791 (+0.00011729858290981075) | > avg_step_time: 0.4575400386058109 (-0.0025929699481372914) | > avg_loss_disc: 2.5879520832652774 (+0.010115143278954797) | > avg_loss_disc_real_0: 0.11230244372092502 (+0.00589431397302051) | > avg_loss_disc_real_1: 0.21014415986940893 (-0.004481257792090049) | > avg_loss_disc_real_2: 0.23535761761833246 (-0.002654098708864655) | > avg_loss_disc_real_3: 0.23107564554248058 (+0.0046295334755535045) | > avg_loss_disc_real_4: 0.24329435573497288 (+0.004295947895923136) | > avg_loss_disc_real_5: 0.24631497620696752 (+0.003220038934492714) | > avg_loss_0: 2.5879520832652774 (+0.010115143278954797) | > avg_grad_norm_0: tensor(324.2161, device='cuda:0') (+tensor(63.9206, device='cuda:0')) | > avg_loss_gen: 2.078457209425913 (+0.005005527550066002) | > avg_loss_kl: 2.0532454017182475 (-0.24719733083751816) | > avg_loss_feat: 3.6790126881129304 (-0.07275567256229021) | > avg_loss_mel: 17.22516478954907 (-0.9965192499295021) | > avg_loss_duration: 1.5717152592162011 (-0.028331498025168766) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.607595228813064 (-1.3397985646422477) | > avg_grad_norm_1: tensor(13177.5059, device='cuda:0') (+tensor(1249.0547, device='cuda:0'))  > EPOCH: 883/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:18:40)   --> TIME: 2025-02-25 17:18:52 -- STEP: 23/72 -- GLOBAL_STEP: 1063600 | > loss_disc: 2.7721972465515137 (2.586942030035931) | > loss_disc_real_0: 0.0586656890809536 (0.15951858201752536) | > loss_disc_real_1: 0.20395025610923767 (0.21129275340100992) | > loss_disc_real_2: 0.24617893993854523 (0.23001179617384207) | > loss_disc_real_3: 0.18633078038692474 (0.24227817810100058) | > loss_disc_real_4: 0.22646740078926086 (0.2389057466517324) | > loss_disc_real_5: 0.22594837844371796 (0.24503641089667444) | > loss_0: 2.7721972465515137 (2.586942030035931) | > grad_norm_0: tensor(291.6686, device='cuda:0') (tensor(231.7051, device='cuda:0')) | > loss_gen: 1.6524291038513184 (2.2956486059271772) | > loss_kl: 1.928093433380127 (2.1104543157245796) | > loss_feat: 3.58617901802063 (4.013218962627908) | > loss_mel: 17.008275985717773 (17.74197089153787) | > loss_duration: 1.5385104417800903 (1.535531090653461) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.713489532470703 (27.696823534758195) | > grad_norm_1: tensor(5707.8320, device='cuda:0') (tensor(6498.8970, device='cuda:0')) | > current_lr_0: 0.0001790984164887823 | > current_lr_1: 0.0001790984164887823 | > step_time: 0.4497 (0.43740617710611096) | > loader_time: 0.0065 (0.0069045191225798235) --> EVAL PERFORMANCE | > avg_loader_time: 0.008782578186250072 (-1.0624737806719242e-05) | > avg_step_time: 0.4571854127964503 (-0.00035462580936057586) | > avg_loss_disc: 2.6876397367934106 (+0.09968765352813325) | > avg_loss_disc_real_0: 0.17091329888978474 (+0.05861085516885972) | > avg_loss_disc_real_1: 0.21969006158096693 (+0.009545901711558002) | > avg_loss_disc_real_2: 0.23520219514907245 (-0.00015542246926000813) | > avg_loss_disc_real_3: 0.23444168626422612 (+0.003366040721745539) | > avg_loss_disc_real_4: 0.24039863574672754 (-0.0028957199882453377) | > avg_loss_disc_real_5: 0.24172181457700864 (-0.004593161629958875) | > avg_loss_0: 2.6876397367934106 (+0.09968765352813325) | > avg_grad_norm_0: tensor(203.3440, device='cuda:0') (tensor(-120.8721, device='cuda:0')) | > avg_loss_gen: 2.1101885866111423 (+0.03173137718522945) | > avg_loss_kl: 2.158435361486086 (+0.10518995976783829) | > avg_loss_feat: 3.5770421162457535 (-0.10197057186717684) | > avg_loss_mel: 17.52094631463708 (+0.29578152508801026) | > avg_loss_duration: 1.5706820051435013 (-0.0010332540726998296) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.937294274988307 (+0.3296990461752429) | > avg_grad_norm_1: tensor(6321.1104, device='cuda:0') (tensor(-6856.3955, device='cuda:0'))  > EPOCH: 884/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:19:17)   --> TIME: 2025-02-25 17:19:20 -- STEP: 1/72 -- GLOBAL_STEP: 1063650 | > loss_disc: 2.765784978866577 (2.765784978866577) | > loss_disc_real_0: 0.10574226826429367 (0.10574226826429367) | > loss_disc_real_1: 0.18765932321548462 (0.18765932321548462) | > loss_disc_real_2: 0.24506835639476776 (0.24506835639476776) | > loss_disc_real_3: 0.2778768837451935 (0.2778768837451935) | > loss_disc_real_4: 0.25346487760543823 (0.25346487760543823) | > loss_disc_real_5: 0.2522910237312317 (0.2522910237312317) | > loss_0: 2.765784978866577 (2.765784978866577) | > grad_norm_0: tensor(151.7643, device='cuda:0') (tensor(151.7643, device='cuda:0')) | > loss_gen: 2.0017528533935547 (2.0017528533935547) | > loss_kl: 2.35363507270813 (2.35363507270813) | > loss_feat: 2.986246109008789 (2.986246109008789) | > loss_mel: 19.26921844482422 (19.26921844482422) | > loss_duration: 1.5546658039093018 (1.5546658039093018) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.16551971435547 (28.16551971435547) | > grad_norm_1: tensor(14841.7861, device='cuda:0') (tensor(14841.7861, device='cuda:0')) | > current_lr_0: 0.0001790760291867212 | > current_lr_1: 0.0001790760291867212 | > step_time: 0.4307 (0.43070340156555176) | > loader_time: 0.0049 (0.004949331283569336)  --> TIME: 2025-02-25 17:19:43 -- STEP: 51/72 -- GLOBAL_STEP: 1063700 | > loss_disc: 2.5072712898254395 (2.7084833126442107) | > loss_disc_real_0: 0.0539763942360878 (0.13789804418589557) | > loss_disc_real_1: 0.2652275264263153 (0.21523735467709748) | > loss_disc_real_2: 0.21140529215335846 (0.22953988261082592) | > loss_disc_real_3: 0.25575023889541626 (0.23836248557941586) | > loss_disc_real_4: 0.22597596049308777 (0.24550093213717142) | > loss_disc_real_5: 0.2576272785663605 (0.24981591543730566) | > loss_0: 2.5072712898254395 (2.7084833126442107) | > grad_norm_0: tensor(56.6563, device='cuda:0') (tensor(152.4511, device='cuda:0')) | > loss_gen: 2.3686623573303223 (2.049985619152292) | > loss_kl: 2.1410510540008545 (2.4943223093070244) | > loss_feat: 3.832231044769287 (3.6002681535833023) | > loss_mel: 16.838212966918945 (19.137389762728812) | > loss_duration: 1.6170339584350586 (1.638800817377427) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.797191619873047 (28.920766680848363) | > grad_norm_1: tensor(4155.7983, device='cuda:0') (tensor(7190.5039, device='cuda:0')) | > current_lr_0: 0.0001790760291867212 | > current_lr_1: 0.0001790760291867212 | > step_time: 0.45 (0.4476020102407418) | > loader_time: 0.0093 (0.007704356137443991) --> EVAL PERFORMANCE | > avg_loader_time: 0.008594711061934353 (-0.00018786712431571873) | > avg_step_time: 0.45778118052952727 (+0.0005957677330769573) | > avg_loss_disc: 2.6809208493837167 (-0.006718887409693952) | > avg_loss_disc_real_0: 0.12249007259666078 (-0.048423226293123955) | > avg_loss_disc_real_1: 0.23455277221723342 (+0.014862710636266485) | > avg_loss_disc_real_2: 0.23696419050995732 (+0.001761995360884866) | > avg_loss_disc_real_3: 0.24023391936026828 (+0.005792233096042154) | > avg_loss_disc_real_4: 0.2446850746030539 (+0.004286438856326347) | > avg_loss_disc_real_5: 0.2505299896001816 (+0.008808175023172937) | > avg_loss_0: 2.6809208493837167 (-0.006718887409693952) | > avg_grad_norm_0: tensor(164.7200, device='cuda:0') (tensor(-38.6239, device='cuda:0')) | > avg_loss_gen: 2.0816121537920456 (-0.02857643281909672) | > avg_loss_kl: 2.411112587216875 (+0.252677225730789) | > avg_loss_feat: 3.570032918956918 (-0.007009197288835534) | > avg_loss_mel: 18.584693828099205 (+1.0637475134621255) | > avg_loss_duration: 1.632050119655233 (+0.06136811451173174) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.279501740361603 (+1.3422074653732956) | > avg_grad_norm_1: tensor(7830.2554, device='cuda:0') (+tensor(1509.1450, device='cuda:0'))  > EPOCH: 885/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:19:54)   --> TIME: 2025-02-25 17:20:10 -- STEP: 29/72 -- GLOBAL_STEP: 1063750 | > loss_disc: 2.9585816860198975 (2.6271464495823302) | > loss_disc_real_0: 0.14513078331947327 (0.14519833593532958) | > loss_disc_real_1: 0.2406340092420578 (0.22326062927986012) | > loss_disc_real_2: 0.22563984990119934 (0.2473105181907785) | > loss_disc_real_3: 0.23866425454616547 (0.23356754769539012) | > loss_disc_real_4: 0.256119966506958 (0.242160826921463) | > loss_disc_real_5: 0.24029290676116943 (0.23874617445057836) | > loss_0: 2.9585816860198975 (2.6271464495823302) | > grad_norm_0: tensor(400.0801, device='cuda:0') (tensor(371.5758, device='cuda:0')) | > loss_gen: 1.6666582822799683 (2.1792302254972786) | > loss_kl: 2.0083632469177246 (2.026673473160842) | > loss_feat: 3.0286996364593506 (3.741975759637767) | > loss_mel: 17.565481185913086 (17.385132493643926) | > loss_duration: 1.5239779949188232 (1.525295298675011) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.79317855834961 (26.858307410930767) | > grad_norm_1: tensor(8242.8008, device='cuda:0') (tensor(8294.3594, device='cuda:0')) | > current_lr_0: 0.00017905364468307286 | > current_lr_1: 0.00017905364468307286 | > step_time: 0.4417 (0.4403474495328706) | > loader_time: 0.0079 (0.007102037298268285) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618613363991319 (+2.3902302056965502e-05) | > avg_step_time: 0.4580908862637802 (+0.0003097057342529297) | > avg_loss_disc: 2.600266721886648 (-0.08065412749706846) | > avg_loss_disc_real_0: 0.10473938176119832 (-0.017750690835462463) | > avg_loss_disc_real_1: 0.22323599100952418 (-0.011316781207709237) | > avg_loss_disc_real_2: 0.2421725226539961 (+0.005208332144038791) | > avg_loss_disc_real_3: 0.23467257660879215 (-0.005561342751476128) | > avg_loss_disc_real_4: 0.24430745441309162 (-0.0003776201899622633) | > avg_loss_disc_real_5: 0.24382458501298662 (-0.0067054045871949575) | > avg_loss_0: 2.600266721886648 (-0.08065412749706846) | > avg_grad_norm_0: tensor(297.3729, device='cuda:0') (+tensor(132.6528, device='cuda:0')) | > avg_loss_gen: 2.1202026017954654 (+0.03859044800341982) | > avg_loss_kl: 2.0629312891355704 (-0.3481812980813044) | > avg_loss_feat: 3.654360139873666 (+0.08432722091674805) | > avg_loss_mel: 17.07628739048058 (-1.5084064376186248) | > avg_loss_duration: 1.5623274134917997 (-0.06972270616343335) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.476108765937912 (-1.8033929744236907) | > avg_grad_norm_1: tensor(9731.2969, device='cuda:0') (+tensor(1901.0415, device='cuda:0'))  > EPOCH: 886/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:20:32)   --> TIME: 2025-02-25 17:20:37 -- STEP: 7/72 -- GLOBAL_STEP: 1063800 | > loss_disc: 2.4825685024261475 (2.5221827711377824) | > loss_disc_real_0: 0.07687864452600479 (0.08254064725978034) | > loss_disc_real_1: 0.22909991443157196 (0.20505721228463308) | > loss_disc_real_2: 0.2541569471359253 (0.2340602342571531) | > loss_disc_real_3: 0.2592513859272003 (0.23326006744589126) | > loss_disc_real_4: 0.24372214078903198 (0.23442606840814864) | > loss_disc_real_5: 0.25579479336738586 (0.24261485465935298) | > loss_0: 2.4825685024261475 (2.5221827711377824) | > grad_norm_0: tensor(165.7170, device='cuda:0') (tensor(250.5998, device='cuda:0')) | > loss_gen: 2.304300546646118 (2.1601463726588657) | > loss_kl: 1.9969758987426758 (1.8251850945608956) | > loss_feat: 3.626227855682373 (3.6902590479169572) | > loss_mel: 16.357343673706055 (16.67847660609654) | > loss_duration: 1.4951835870742798 (1.52131153856005) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.780031204223633 (25.875378472464426) | > grad_norm_1: tensor(6398.9385, device='cuda:0') (tensor(10504.1826, device='cuda:0')) | > current_lr_0: 0.00017903126297748746 | > current_lr_1: 0.00017903126297748746 | > step_time: 0.4337 (0.43411411557878765) | > loader_time: 0.0056 (0.006341184888567243)  --> TIME: 2025-02-25 17:21:00 -- STEP: 57/72 -- GLOBAL_STEP: 1063850 | > loss_disc: 2.6296656131744385 (2.533513750946313) | > loss_disc_real_0: 0.1496136635541916 (0.08682306274248842) | > loss_disc_real_1: 0.25581517815589905 (0.21599772990795604) | > loss_disc_real_2: 0.23847709596157074 (0.23462640664033724) | > loss_disc_real_3: 0.23924663662910461 (0.231524013636405) | > loss_disc_real_4: 0.2576671838760376 (0.24099408978955789) | > loss_disc_real_5: 0.26431888341903687 (0.2465614313096331) | > loss_0: 2.6296656131744385 (2.533513750946313) | > grad_norm_0: tensor(583.6268, device='cuda:0') (tensor(289.0602, device='cuda:0')) | > loss_gen: 2.1334619522094727 (2.1596286631467034) | > loss_kl: 2.117733955383301 (1.9389065064881976) | > loss_feat: 3.223149061203003 (3.739341794398793) | > loss_mel: 17.43090057373047 (16.854120087205317) | > loss_duration: 1.6101155281066895 (1.5441526007234005) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.515361785888672 (26.23614960386042) | > grad_norm_1: tensor(9648.5967, device='cuda:0') (tensor(10341.1133, device='cuda:0')) | > current_lr_0: 0.00017903126297748746 | > current_lr_1: 0.00017903126297748746 | > step_time: 0.4576 (0.4550460556097198) | > loader_time: 0.0094 (0.008091407909727935) --> EVAL PERFORMANCE | > avg_loader_time: 0.008717157471347865 (+9.854410735654628e-05) | > avg_step_time: 0.46301844758047184 (+0.004927561316691642) | > avg_loss_disc: 2.5353500171446464 (-0.06491670474200184) | > avg_loss_disc_real_0: 0.08776003232514354 (-0.01697934943605478) | > avg_loss_disc_real_1: 0.21615285017121005 (-0.007083140838314128) | > avg_loss_disc_real_2: 0.2349101871671811 (-0.007262335486815014) | > avg_loss_disc_real_3: 0.23100253645802887 (-0.0036700401507632763) | > avg_loss_disc_real_4: 0.24260978904408467 (-0.001697665369006951) | > avg_loss_disc_real_5: 0.24680409926763722 (+0.002979514254650595) | > avg_loss_0: 2.5353500171446464 (-0.06491670474200184) | > avg_grad_norm_0: tensor(294.5774, device='cuda:0') (tensor(-2.7955, device='cuda:0')) | > avg_loss_gen: 2.1617698954864273 (+0.041567293690961904) | > avg_loss_kl: 1.9730882997244177 (-0.08984298941115276) | > avg_loss_feat: 3.7395783545265737 (+0.08521821465290769) | > avg_loss_mel: 16.83283762864663 (-0.24344976183395062) | > avg_loss_duration: 1.5562564638299001 (-0.006070949661899583) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.263530543152715 (-0.2125782227851971) | > avg_grad_norm_1: tensor(10445.2979, device='cuda:0') (+tensor(714.0010, device='cuda:0'))  > EPOCH: 887/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:21:09)   --> TIME: 2025-02-25 17:21:27 -- STEP: 35/72 -- GLOBAL_STEP: 1063900 | > loss_disc: 2.586249351501465 (2.580998434339251) | > loss_disc_real_0: 0.10005635768175125 (0.11613077351025172) | > loss_disc_real_1: 0.20655763149261475 (0.21904227009841373) | > loss_disc_real_2: 0.22898538410663605 (0.23463063112327032) | > loss_disc_real_3: 0.23480752110481262 (0.23362144018922534) | > loss_disc_real_4: 0.22689293324947357 (0.24616453647613526) | > loss_disc_real_5: 0.24304641783237457 (0.25085517338344027) | > loss_0: 2.586249351501465 (2.580998434339251) | > grad_norm_0: tensor(283.8232, device='cuda:0') (tensor(316.3466, device='cuda:0')) | > loss_gen: 1.9937673807144165 (2.1462237017495287) | > loss_kl: 2.0566112995147705 (1.9108654601233346) | > loss_feat: 3.577617883682251 (3.725500270298549) | > loss_mel: 16.7583065032959 (16.877196775163924) | > loss_duration: 1.5671826601028442 (1.5195103066308158) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.9534854888916 (26.179296439034598) | > grad_norm_1: tensor(14690.3828, device='cuda:0') (tensor(9685.8779, device='cuda:0')) | > current_lr_0: 0.00017900888406961526 | > current_lr_1: 0.00017900888406961526 | > step_time: 0.4437 (0.44061946868896484) | > loader_time: 0.0081 (0.007340683255876813) --> EVAL PERFORMANCE | > avg_loader_time: 0.008808525515274267 (+9.13680439264021e-05) | > avg_step_time: 0.4599188247197111 (-0.0030996228607607645) | > avg_loss_disc: 2.5568889094070655 (+0.021538892262419118) | > avg_loss_disc_real_0: 0.10404088547531988 (+0.01628085315017634) | > avg_loss_disc_real_1: 0.22013792710405 (+0.003985076932839948) | > avg_loss_disc_real_2: 0.23344539126879732 (-0.001464795898383775) | > avg_loss_disc_real_3: 0.2337322965474196 (+0.002729760089390715) | > avg_loss_disc_real_4: 0.24322533061806584 (+0.0006155415739811654) | > avg_loss_disc_real_5: 0.24854368076358044 (+0.001739581495943221) | > avg_loss_0: 2.5568889094070655 (+0.021538892262419118) | > avg_grad_norm_0: tensor(304.8510, device='cuda:0') (+tensor(10.2737, device='cuda:0')) | > avg_loss_gen: 2.157394995152111 (-0.004374900334316312) | > avg_loss_kl: 1.97157793481585 (-0.0015103649085677073) | > avg_loss_feat: 3.7164245961417612 (-0.023153758384812484) | > avg_loss_mel: 16.763869231855374 (-0.06896839679125577) | > avg_loss_duration: 1.5535217254934177 (-0.002734738336482412) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.162788471705476 (-0.10074207144723957) | > avg_grad_norm_1: tensor(10089.2129, device='cuda:0') (tensor(-356.0850, device='cuda:0'))  > EPOCH: 888/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:21:47)   --> TIME: 2025-02-25 17:21:54 -- STEP: 13/72 -- GLOBAL_STEP: 1063950 | > loss_disc: 2.6902804374694824 (2.515355898783757) | > loss_disc_real_0: 0.08206335455179214 (0.08638080037557162) | > loss_disc_real_1: 0.186067134141922 (0.22048885088700515) | > loss_disc_real_2: 0.23539026081562042 (0.2230953390781696) | > loss_disc_real_3: 0.22878651320934296 (0.23116579881081215) | > loss_disc_real_4: 0.23451177775859833 (0.23512499492902023) | > loss_disc_real_5: 0.2382209748029709 (0.24025216354773596) | > loss_0: 2.6902804374694824 (2.515355898783757) | > grad_norm_0: tensor(393.2236, device='cuda:0') (tensor(325.2514, device='cuda:0')) | > loss_gen: 1.9096535444259644 (2.20451021194458) | > loss_kl: 1.7954832315444946 (1.915044876245352) | > loss_feat: 3.778428554534912 (4.071493533941416) | > loss_mel: 15.997529983520508 (16.83074929164006) | > loss_duration: 1.4368871450424194 (1.504342372600849) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.91798210144043 (26.526140506450947) | > grad_norm_1: tensor(13505.7266, device='cuda:0') (tensor(11368.1035, device='cuda:0')) | > current_lr_0: 0.00017898650795910656 | > current_lr_1: 0.00017898650795910656 | > step_time: 0.4799 (0.4373300075531006) | > loader_time: 0.0073 (0.007048295094416692)  --> TIME: 2025-02-25 17:22:18 -- STEP: 63/72 -- GLOBAL_STEP: 1064000 | > loss_disc: 2.5145490169525146 (2.5392848839835516) | > loss_disc_real_0: 0.16074950993061066 (0.09284947723859832) | > loss_disc_real_1: 0.21243181824684143 (0.21835806871217395) | > loss_disc_real_2: 0.19390769302845 (0.22878833091448222) | > loss_disc_real_3: 0.20914910733699799 (0.23015292722081382) | > loss_disc_real_4: 0.2428954690694809 (0.24244193377948942) | > loss_disc_real_5: 0.25760960578918457 (0.24641114709869263) | > loss_0: 2.5145490169525146 (2.5392848839835516) | > grad_norm_0: tensor(359.3167, device='cuda:0') (tensor(318.6657, device='cuda:0')) | > loss_gen: 2.370434522628784 (2.1861982269892617) | > loss_kl: 2.1303348541259766 (1.9545982943640814) | > loss_feat: 3.7438313961029053 (3.870188766055636) | > loss_mel: 17.22081184387207 (16.825362114679244) | > loss_duration: 1.5872631072998047 (1.544146772414919) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.052675247192383 (26.38049416314988) | > grad_norm_1: tensor(7912.0693, device='cuda:0') (tensor(10905.1602, device='cuda:0')) | > current_lr_0: 0.00017898650795910656 | > current_lr_1: 0.00017898650795910656 | > step_time: 0.5017 (0.45215403465997606) | > loader_time: 0.0114 (0.00860596838451567) --> EVAL PERFORMANCE | > avg_loader_time: 0.008945622914273975 (+0.0001370973989997075) | > avg_step_time: 0.4604704715836216 (+0.0005516468639105154) | > avg_loss_disc: 2.5397323050969085 (-0.017156604310156975) | > avg_loss_disc_real_0: 0.09225051461810797 (-0.011790370857211907) | > avg_loss_disc_real_1: 0.22008799834990164 (-4.992875414835951e-05) | > avg_loss_disc_real_2: 0.22877879650659963 (-0.00466659476219769) | > avg_loss_disc_real_3: 0.23128888556655025 (-0.002443410980869337) | > avg_loss_disc_real_4: 0.24021265309461406 (-0.0030126775234517733) | > avg_loss_disc_real_5: 0.24609998267301372 (-0.002443698090566715) | > avg_loss_0: 2.5397323050969085 (-0.017156604310156975) | > avg_grad_norm_0: tensor(317.3889, device='cuda:0') (+tensor(12.5379, device='cuda:0')) | > avg_loss_gen: 2.185172952396769 (+0.027777957244658147) | > avg_loss_kl: 1.9764974419499788 (+0.004919507134128809) | > avg_loss_feat: 3.86237440982335 (+0.14594981368158866) | > avg_loss_mel: 16.820332943553655 (+0.05646371169828157) | > avg_loss_duration: 1.5521410686869017 (-0.0013806568065160008) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.396518841595718 (+0.23373036989024243) | > avg_grad_norm_1: tensor(10947.2637, device='cuda:0') (+tensor(858.0508, device='cuda:0'))  > EPOCH: 889/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:22:24)   --> TIME: 2025-02-25 17:22:45 -- STEP: 41/72 -- GLOBAL_STEP: 1064050 | > loss_disc: 2.5506434440612793 (2.535336761939816) | > loss_disc_real_0: 0.11886033415794373 (0.09327967583042822) | > loss_disc_real_1: 0.18830356001853943 (0.22149668815659312) | > loss_disc_real_2: 0.2315225452184677 (0.22859919907116308) | > loss_disc_real_3: 0.2082541584968567 (0.22973263154669507) | > loss_disc_real_4: 0.24284447729587555 (0.24450800949480475) | > loss_disc_real_5: 0.2579423189163208 (0.24691217011067926) | > loss_0: 2.5506434440612793 (2.535336761939816) | > grad_norm_0: tensor(268.6955, device='cuda:0') (tensor(290.4972, device='cuda:0')) | > loss_gen: 2.0912952423095703 (2.2321261225677116) | > loss_kl: 1.8650104999542236 (1.947516918182373) | > loss_feat: 3.5422756671905518 (3.958266194273786) | > loss_mel: 15.996208190917969 (16.78124316145734) | > loss_duration: 1.55589759349823 (1.5199665354519358) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.05068588256836 (26.43911896682367) | > grad_norm_1: tensor(12217.8008, device='cuda:0') (tensor(10910.2314, device='cuda:0')) | > current_lr_0: 0.00017896413464561168 | > current_lr_1: 0.00017896413464561168 | > step_time: 0.4622 (0.4446271687019162) | > loader_time: 0.009 (0.007537114910963105) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719833803848479 (-0.00022578911042549568) | > avg_step_time: 0.46161467928281974 (+0.001144207699198152) | > avg_loss_disc: 2.5532693157733317 (+0.013537010676423211) | > avg_loss_disc_real_0: 0.09812459454570018 (+0.005874079927592202) | > avg_loss_disc_real_1: 0.2224480140377098 (+0.0023600156878081646) | > avg_loss_disc_real_2: 0.23051842228627542 (+0.0017396257796757908) | > avg_loss_disc_real_3: 0.23086772187494894 (-0.0004211636916013062) | > avg_loss_disc_real_4: 0.24385653100383115 (+0.0036438779092170803) | > avg_loss_disc_real_5: 0.24574994852005597 (-0.00035003415295775664) | > avg_loss_0: 2.5532693157733317 (+0.013537010676423211) | > avg_grad_norm_0: tensor(293.0845, device='cuda:0') (tensor(-24.3044, device='cuda:0')) | > avg_loss_gen: 2.206161237098801 (+0.020988284702031734) | > avg_loss_kl: 2.0019051760015354 (+0.025407734051556607) | > avg_loss_feat: 3.865132365428226 (+0.0027579556048760168) | > avg_loss_mel: 16.748388599342025 (-0.0719443442116301) | > avg_loss_duration: 1.5493774733073278 (-0.0027635953795739354) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.37096482935086 (-0.025554012244857205) | > avg_grad_norm_1: tensor(10300.3418, device='cuda:0') (tensor(-646.9219, device='cuda:0'))  > EPOCH: 890/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:23:01)   --> TIME: 2025-02-25 17:23:12 -- STEP: 19/72 -- GLOBAL_STEP: 1064100 | > loss_disc: 2.676053762435913 (2.5459397466559155) | > loss_disc_real_0: 0.05858969688415527 (0.09890771147451903) | > loss_disc_real_1: 0.20124110579490662 (0.2235741858419619) | > loss_disc_real_2: 0.23623161017894745 (0.22685660500275462) | > loss_disc_real_3: 0.22243766486644745 (0.2365877142077998) | > loss_disc_real_4: 0.2529981732368469 (0.24363527329344498) | > loss_disc_real_5: 0.2431189864873886 (0.24255083030775973) | > loss_0: 2.676053762435913 (2.5459397466559155) | > grad_norm_0: tensor(300.1106, device='cuda:0') (tensor(273.6191, device='cuda:0')) | > loss_gen: 1.9888499975204468 (2.2305403571379814) | > loss_kl: 1.9689379930496216 (1.8926864109541242) | > loss_feat: 3.8162193298339844 (3.899806499481201) | > loss_mel: 17.001644134521484 (16.987040369134203) | > loss_duration: 1.501750111579895 (1.5047274075056378) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.277400970458984 (26.514800925003854) | > grad_norm_1: tensor(11717.0107, device='cuda:0') (tensor(10474.7686, device='cuda:0')) | > current_lr_0: 0.00017894176412878098 | > current_lr_1: 0.00017894176412878098 | > step_time: 0.4329 (0.43025098348918717) | > loader_time: 0.0065 (0.006598623175370066)  --> TIME: 2025-02-25 17:23:36 -- STEP: 69/72 -- GLOBAL_STEP: 1064150 | > loss_disc: 2.5646495819091797 (2.566461476726809) | > loss_disc_real_0: 0.09742581844329834 (0.10794150073459183) | > loss_disc_real_1: 0.2382938116788864 (0.22651154105214105) | > loss_disc_real_2: 0.25728315114974976 (0.23177497434443323) | > loss_disc_real_3: 0.25664085149765015 (0.23358850029931552) | > loss_disc_real_4: 0.25900259613990784 (0.24212607201458752) | > loss_disc_real_5: 0.2531128525733948 (0.24403315437012824) | > loss_0: 2.5646495819091797 (2.566461476726809) | > grad_norm_0: tensor(195.2415, device='cuda:0') (tensor(270.6813, device='cuda:0')) | > loss_gen: 2.3162543773651123 (2.2197093117064326) | > loss_kl: 2.280261516571045 (2.023872565532076) | > loss_feat: 3.8027801513671875 (3.8508973363516987) | > loss_mel: 17.837308883666992 (16.77654285707336) | > loss_duration: 1.6629396677017212 (1.5515332964883335) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.89954376220703 (26.42255531532177) | > grad_norm_1: tensor(6751.6123, device='cuda:0') (tensor(9317.2910, device='cuda:0')) | > current_lr_0: 0.00017894176412878098 | > current_lr_1: 0.00017894176412878098 | > step_time: 0.5227 (0.45989670269731164) | > loader_time: 0.0128 (0.008506626322649527) --> EVAL PERFORMANCE | > avg_loader_time: 0.008561218288582815 (-0.0001586155152656643) | > avg_step_time: 0.4609760264275779 (-0.0006386528552418391) | > avg_loss_disc: 2.568474477445576 (+0.015205161672244394) | > avg_loss_disc_real_0: 0.10737042202496193 (+0.009245827479261753) | > avg_loss_disc_real_1: 0.2265277083071185 (+0.004079694269408685) | > avg_loss_disc_real_2: 0.23189669763538198 (+0.0013782753491065614) | > avg_loss_disc_real_3: 0.23324215831890913 (+0.0023744364439601817) | > avg_loss_disc_real_4: 0.24153286176668087 (-0.002323669237150272) | > avg_loss_disc_real_5: 0.24333725806692957 (-0.0024126904531263926) | > avg_loss_0: 2.568474477445576 (+0.015205161672244394) | > avg_grad_norm_0: tensor(273.1465, device='cuda:0') (tensor(-19.9380, device='cuda:0')) | > avg_loss_gen: 2.213401413299668 (+0.007240176200867143) | > avg_loss_kl: 2.0260343585215823 (+0.02412918252004692) | > avg_loss_feat: 3.8519218303787874 (-0.013210535049438477) | > avg_loss_mel: 16.773586864202798 (+0.025198264860772923) | > avg_loss_duration: 1.5520502077022071 (+0.0026727343948793614) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.416994605265874 (+0.046029775915013005) | > avg_grad_norm_1: tensor(9393.4189, device='cuda:0') (tensor(-906.9229, device='cuda:0'))  > EPOCH: 891/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:23:39)   --> TIME: 2025-02-25 17:24:02 -- STEP: 47/72 -- GLOBAL_STEP: 1064200 | > loss_disc: 2.6423256397247314 (2.5559874747661833) | > loss_disc_real_0: 0.11531712114810944 (0.10519086300058568) | > loss_disc_real_1: 0.20661675930023193 (0.22141060740389723) | > loss_disc_real_2: 0.2357184886932373 (0.22978014578210545) | > loss_disc_real_3: 0.2150631695985794 (0.23259384803315428) | > loss_disc_real_4: 0.2302752286195755 (0.24070825031463136) | > loss_disc_real_5: 0.23075146973133087 (0.24666745580257254) | > loss_0: 2.6423256397247314 (2.5559874747661833) | > grad_norm_0: tensor(314.1679, device='cuda:0') (tensor(301.3616, device='cuda:0')) | > loss_gen: 2.0089731216430664 (2.2226682216563125) | > loss_kl: 2.1548595428466797 (2.033546802845408) | > loss_feat: 3.4712436199188232 (3.8805651766188602) | > loss_mel: 16.78934669494629 (16.906788115805774) | > loss_duration: 1.5830637216567993 (1.5302760372770596) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.00748634338379 (26.573843976284596) | > grad_norm_1: tensor(9906.8438, device='cuda:0') (tensor(9860.3320, device='cuda:0')) | > current_lr_0: 0.00017891939640826487 | > current_lr_1: 0.00017891939640826487 | > step_time: 0.4515 (0.44587381342624094) | > loader_time: 0.0088 (0.007865692706818278) --> EVAL PERFORMANCE | > avg_loader_time: 0.008768995043257592 (+0.00020777675467477733) | > avg_step_time: 0.45941014021215304 (-0.001565886215424861) | > avg_loss_disc: 2.588753109246912 (+0.020278631801335933) | > avg_loss_disc_real_0: 0.11331234143023759 (+0.005941919405275664) | > avg_loss_disc_real_1: 0.23092094609435176 (+0.004393237787233273) | > avg_loss_disc_real_2: 0.2323533636583409 (+0.000456666022958907) | > avg_loss_disc_real_3: 0.2361535992420895 (+0.0029114409231803806) | > avg_loss_disc_real_4: 0.24182271810484604 (+0.00028985633816516754) | > avg_loss_disc_real_5: 0.2466590853644089 (+0.0033218272974793184) | > avg_loss_0: 2.588753109246912 (+0.020278631801335933) | > avg_grad_norm_0: tensor(303.1691, device='cuda:0') (+tensor(30.0226, device='cuda:0')) | > avg_loss_gen: 2.2162548824095394 (+0.002853469109871387) | > avg_loss_kl: 2.071060267972275 (+0.04502590945069285) | > avg_loss_feat: 3.8068444393050505 (-0.04507739107373698) | > avg_loss_mel: 16.951690821580492 (+0.1781039573776937) | > avg_loss_duration: 1.5536810730544617 (+0.0016308653522545402) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.59953128116232 (+0.18253667589644706) | > avg_grad_norm_1: tensor(9532.8506, device='cuda:0') (+tensor(139.4316, device='cuda:0'))  > EPOCH: 892/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:24:16)   --> TIME: 2025-02-25 17:24:29 -- STEP: 25/72 -- GLOBAL_STEP: 1064250 | > loss_disc: 2.5054683685302734 (2.5526634120941156) | > loss_disc_real_0: 0.06341094523668289 (0.0991280274093151) | > loss_disc_real_1: 0.20468971133232117 (0.22903932273387909) | > loss_disc_real_2: 0.24171732366085052 (0.22799026608467102) | > loss_disc_real_3: 0.2590108811855316 (0.24125318169593812) | > loss_disc_real_4: 0.2774909734725952 (0.24483261227607728) | > loss_disc_real_5: 0.23966416716575623 (0.24599942326545715) | > loss_0: 2.5054683685302734 (2.5526634120941156) | > grad_norm_0: tensor(120.1317, device='cuda:0') (tensor(268.4379, device='cuda:0')) | > loss_gen: 2.396458864212036 (2.241787996292114) | > loss_kl: 2.27807354927063 (2.028771815299988) | > loss_feat: 3.908714771270752 (3.9758458614349363) | > loss_mel: 17.050432205200195 (16.959654464721684) | > loss_duration: 1.5056545734405518 (1.506546220779419) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.13933563232422 (26.712606506347655) | > grad_norm_1: tensor(10751.5186, device='cuda:0') (tensor(9347.8096, device='cuda:0')) | > current_lr_0: 0.00017889703148371382 | > current_lr_1: 0.00017889703148371382 | > step_time: 0.4328 (0.4391877460479736) | > loader_time: 0.0074 (0.006972017288208007) --> EVAL PERFORMANCE | > avg_loader_time: 0.00870822181164379 (-6.077323161380113e-05) | > avg_step_time: 0.45876779354793923 (-0.0006423466642138109) | > avg_loss_disc: 2.5952548174790935 (+0.006501708232181436) | > avg_loss_disc_real_0: 0.10997102955277537 (-0.0033413118774622275) | > avg_loss_disc_real_1: 0.2309102905048451 (-1.065558950666401e-05) | > avg_loss_disc_real_2: 0.23543919517960346 (+0.003085831521262572) | > avg_loss_disc_real_3: 0.2364408252104907 (+0.00028722596840119463) | > avg_loss_disc_real_4: 0.2432017985364081 (+0.0013790804315620686) | > avg_loss_disc_real_5: 0.24486972769381296 (-0.0017893576705959313) | > avg_loss_0: 2.5952548174790935 (+0.006501708232181436) | > avg_grad_norm_0: tensor(276.0544, device='cuda:0') (tensor(-27.1147, device='cuda:0')) | > avg_loss_gen: 2.19324882433448 (-0.02300605807505951) | > avg_loss_kl: 2.0998834875267987 (+0.028823219554523494) | > avg_loss_feat: 3.807847755056032 (+0.0010033157509816704) | > avg_loss_mel: 16.936435538278506 (-0.015255283301986111) | > avg_loss_duration: 1.5528077105401268 (-0.0008733625143348966) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.590223231785735 (-0.009308049376585359) | > avg_grad_norm_1: tensor(8969.7656, device='cuda:0') (tensor(-563.0850, device='cuda:0'))  > EPOCH: 893/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:24:53)   --> TIME: 2025-02-25 17:24:56 -- STEP: 3/72 -- GLOBAL_STEP: 1064300 | > loss_disc: 2.4991321563720703 (2.5718401273091636) | > loss_disc_real_0: 0.12814828753471375 (0.1603790819644928) | > loss_disc_real_1: 0.23189665377140045 (0.2411533147096634) | > loss_disc_real_2: 0.23618867993354797 (0.2433810830116272) | > loss_disc_real_3: 0.2454163283109665 (0.23602859675884247) | > loss_disc_real_4: 0.2526105046272278 (0.24452231327692667) | > loss_disc_real_5: 0.26783210039138794 (0.2563841293255488) | > loss_0: 2.4991321563720703 (2.5718401273091636) | > grad_norm_0: tensor(279.4666, device='cuda:0') (tensor(364.9052, device='cuda:0')) | > loss_gen: 2.3018276691436768 (2.2706308364868164) | > loss_kl: 2.1151485443115234 (1.9089893102645874) | > loss_feat: 3.8159875869750977 (3.8288532892862954) | > loss_mel: 17.99163246154785 (17.40479914347331) | > loss_duration: 1.5417433977127075 (1.5170897245407104) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.766340255737305 (26.930362065633137) | > grad_norm_1: tensor(7362.0098, device='cuda:0') (tensor(8440.7578, device='cuda:0')) | > current_lr_0: 0.00017887466935477835 | > current_lr_1: 0.00017887466935477835 | > step_time: 0.4209 (0.430791695912679) | > loader_time: 0.0057 (0.006158192952473958)  --> TIME: 2025-02-25 17:25:19 -- STEP: 53/72 -- GLOBAL_STEP: 1064350 | > loss_disc: 2.564819812774658 (2.5692389956060446) | > loss_disc_real_0: 0.06861113011837006 (0.10124147613093538) | > loss_disc_real_1: 0.1911640465259552 (0.22484515129395252) | > loss_disc_real_2: 0.23418842256069183 (0.22594474452846455) | > loss_disc_real_3: 0.21953192353248596 (0.23537549151564544) | > loss_disc_real_4: 0.19892673194408417 (0.24335531721699913) | > loss_disc_real_5: 0.24161864817142487 (0.24388165040960852) | > loss_0: 2.564819812774658 (2.5692389956060446) | > grad_norm_0: tensor(237.1205, device='cuda:0') (tensor(289.7755, device='cuda:0')) | > loss_gen: 2.0402708053588867 (2.1916782068756397) | > loss_kl: 2.0608229637145996 (2.0339888986551533) | > loss_feat: 3.9896016120910645 (3.898192342722191) | > loss_mel: 17.605194091796875 (16.923606476693774) | > loss_duration: 1.588871955871582 (1.5368548969052873) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.284759521484375 (26.584320680150444) | > grad_norm_1: tensor(12135.3145, device='cuda:0') (tensor(9972.5342, device='cuda:0')) | > current_lr_0: 0.00017887466935477835 | > current_lr_1: 0.00017887466935477835 | > step_time: 0.4478 (0.4418996415048275) | > loader_time: 0.0093 (0.007836769211967036) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629358990091675 (-7.886282155211526e-05) | > avg_step_time: 0.454947206336008 (-0.0038205872119312367) | > avg_loss_disc: 2.5827673287458817 (-0.012487488733211816) | > avg_loss_disc_real_0: 0.10188853079584283 (-0.008082498756932538) | > avg_loss_disc_real_1: 0.22960094939654982 (-0.001309341108295281) | > avg_loss_disc_real_2: 0.2275901132486236 (-0.007849081930979857) | > avg_loss_disc_real_3: 0.23813453253725886 (+0.0016937073267681568) | > avg_loss_disc_real_4: 0.24286477288729708 (-0.0003370256491110335) | > avg_loss_disc_real_5: 0.2445995331230298 (-0.0002701945707831721) | > avg_loss_0: 2.5827673287458817 (-0.012487488733211816) | > avg_grad_norm_0: tensor(294.4825, device='cuda:0') (+tensor(18.4282, device='cuda:0')) | > avg_loss_gen: 2.1826022729067733 (-0.010646551427706541) | > avg_loss_kl: 2.065704570689671 (-0.034178916837127815) | > avg_loss_feat: 3.8300053972593497 (+0.022157642203317618) | > avg_loss_mel: 16.939982656022188 (+0.0035471177436825485) | > avg_loss_duration: 1.5525804321530838 (-0.00022727838704295067) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.570875248438874 (-0.019347983346861497) | > avg_grad_norm_1: tensor(9798.2061, device='cuda:0') (+tensor(828.4404, device='cuda:0'))  > EPOCH: 894/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:25:30)   --> TIME: 2025-02-25 17:25:46 -- STEP: 31/72 -- GLOBAL_STEP: 1064400 | > loss_disc: 2.631761074066162 (2.556245403905069) | > loss_disc_real_0: 0.08442191779613495 (0.10314696606609129) | > loss_disc_real_1: 0.2737669348716736 (0.2268089214640279) | > loss_disc_real_2: 0.23021119832992554 (0.22973267637914227) | > loss_disc_real_3: 0.23318150639533997 (0.23602742870007792) | > loss_disc_real_4: 0.2581261098384857 (0.24404802485819785) | > loss_disc_real_5: 0.24562880396842957 (0.24510528724039754) | > loss_0: 2.631761074066162 (2.556245403905069) | > grad_norm_0: tensor(147.4666, device='cuda:0') (tensor(271.6430, device='cuda:0')) | > loss_gen: 2.1769962310791016 (2.249196817797999) | > loss_kl: 2.3497161865234375 (2.0543499838921333) | > loss_feat: 3.673017740249634 (3.9438290134552987) | > loss_mel: 16.37154197692871 (16.822550004528416) | > loss_duration: 1.5445725917816162 (1.5110158420378161) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.1158447265625 (26.580941600184286) | > grad_norm_1: tensor(8350.4609, device='cuda:0') (tensor(9108.0527, device='cuda:0')) | > current_lr_0: 0.00017885231002110898 | > current_lr_1: 0.00017885231002110898 | > step_time: 0.4528 (0.4419273407228531) | > loader_time: 0.0085 (0.007124331689650013) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865376163536394 (+2.4402645272264198e-05) | > avg_step_time: 0.4579964254943418 (+0.0030492191583337824) | > avg_loss_disc: 2.6075673237652848 (+0.02479999501940311) | > avg_loss_disc_real_0: 0.10631730213341577 (+0.00442877133757294) | > avg_loss_disc_real_1: 0.23482062541682955 (+0.005219676020279729) | > avg_loss_disc_real_2: 0.23895137977432196 (+0.011361266525698355) | > avg_loss_disc_real_3: 0.24015621526140563 (+0.002021682724146767) | > avg_loss_disc_real_4: 0.24316583300980044 (+0.00030106012250336045) | > avg_loss_disc_real_5: 0.24587980332508894 (+0.0012802702020591472) | > avg_loss_0: 2.6075673237652848 (+0.02479999501940311) | > avg_grad_norm_0: tensor(273.5495, device='cuda:0') (tensor(-20.9330, device='cuda:0')) | > avg_loss_gen: 2.203615552942518 (+0.021013280035744764) | > avg_loss_kl: 2.0968942054560493 (+0.031189634766378482) | > avg_loss_feat: 3.8240423538315462 (-0.005963043427803516) | > avg_loss_mel: 16.76087877783976 (-0.17910387818242768) | > avg_loss_duration: 1.5528190723607238 (+0.00023864020763997296) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.43824996410961 (-0.13262528432926501) | > avg_grad_norm_1: tensor(8931.6367, device='cuda:0') (tensor(-866.5693, device='cuda:0'))  > EPOCH: 895/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:26:08)   --> TIME: 2025-02-25 17:26:13 -- STEP: 9/72 -- GLOBAL_STEP: 1064450 | > loss_disc: 2.7793986797332764 (2.6229303412967258) | > loss_disc_real_0: 0.2532423734664917 (0.14178094350629383) | > loss_disc_real_1: 0.2601083815097809 (0.21867387493451437) | > loss_disc_real_2: 0.2446078658103943 (0.22066075603167215) | > loss_disc_real_3: 0.26393017172813416 (0.22763672139909533) | > loss_disc_real_4: 0.27214351296424866 (0.23595789737171596) | > loss_disc_real_5: 0.27017319202423096 (0.23965143660704294) | > loss_0: 2.7793986797332764 (2.6229303412967258) | > grad_norm_0: tensor(573.0575, device='cuda:0') (tensor(339.5625, device='cuda:0')) | > loss_gen: 2.1123909950256348 (2.172384328312344) | > loss_kl: 2.0864779949188232 (1.968485262658861) | > loss_feat: 3.039405107498169 (3.735709296332465) | > loss_mel: 16.213993072509766 (16.61631679534912) | > loss_duration: 1.5064493417739868 (1.5078799194759793) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.958715438842773 (26.000775443183052) | > grad_norm_1: tensor(7009.8574, device='cuda:0') (tensor(8620.0518, device='cuda:0')) | > current_lr_0: 0.00017882995348235633 | > current_lr_1: 0.00017882995348235633 | > step_time: 0.4194 (0.4284629027048747) | > loader_time: 0.0056 (0.006315814124213325)  --> TIME: 2025-02-25 17:26:37 -- STEP: 59/72 -- GLOBAL_STEP: 1064500 | > loss_disc: 2.58872127532959 (2.629535683130814) | > loss_disc_real_0: 0.13657523691654205 (0.12478193148212917) | > loss_disc_real_1: 0.22494658827781677 (0.23845175628439855) | > loss_disc_real_2: 0.2151300460100174 (0.2337859285584951) | > loss_disc_real_3: 0.20993953943252563 (0.23051430714332452) | > loss_disc_real_4: 0.2485719621181488 (0.23812601500648564) | > loss_disc_real_5: 0.23638100922107697 (0.2384932955442849) | > loss_0: 2.58872127532959 (2.629535683130814) | > grad_norm_0: tensor(204.5032, device='cuda:0') (tensor(263.3821, device='cuda:0')) | > loss_gen: 2.1831774711608887 (2.1649507445804135) | > loss_kl: 2.009817361831665 (2.0616836143752293) | > loss_feat: 3.4013710021972656 (3.7701246213104764) | > loss_mel: 16.26114845275879 (16.756636716551693) | > loss_duration: 1.5875710248947144 (1.5425843465126168) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.443084716796875 (26.29598013021178) | > grad_norm_1: tensor(8147.0420, device='cuda:0') (tensor(8104.7607, device='cuda:0')) | > current_lr_0: 0.00017882995348235633 | > current_lr_1: 0.00017882995348235633 | > step_time: 0.4694 (0.4472163491329904) | > loader_time: 0.0098 (0.00823501813209663) --> EVAL PERFORMANCE | > avg_loader_time: 0.008783340454101562 (+0.00012957881873762282) | > avg_step_time: 0.4561332212367528 (-0.0018632042575889574) | > avg_loss_disc: 2.638738299759341 (+0.03117097599405616) | > avg_loss_disc_real_0: 0.1221856779195893 (+0.015868375786173533) | > avg_loss_disc_real_1: 0.2380096315288208 (+0.003189006111991244) | > avg_loss_disc_real_2: 0.23452440344951522 (-0.004426976324806736) | > avg_loss_disc_real_3: 0.23359307038112426 (-0.0065631448802813686) | > avg_loss_disc_real_4: 0.23927225487332948 (-0.0038935781364709543) | > avg_loss_disc_real_5: 0.2393995078516678 (-0.006480295473421133) | > avg_loss_0: 2.638738299759341 (+0.03117097599405616) | > avg_grad_norm_0: tensor(261.9640, device='cuda:0') (tensor(-11.5855, device='cuda:0')) | > avg_loss_gen: 2.1576700059460916 (-0.0459455469964265) | > avg_loss_kl: 2.097187686973896 (+0.0002934815178465833) | > avg_loss_feat: 3.7566364315194143 (-0.06740592231213194) | > avg_loss_mel: 16.78671800586538 (+0.025839228025620997) | > avg_loss_duration: 1.5546414617081763 (+0.001822389347452491) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.352853775024414 (-0.08539618908519486) | > avg_grad_norm_1: tensor(8055.4595, device='cuda:0') (tensor(-876.1772, device='cuda:0'))  > EPOCH: 896/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:26:45)   --> TIME: 2025-02-25 17:27:03 -- STEP: 37/72 -- GLOBAL_STEP: 1064550 | > loss_disc: 3.0456631183624268 (2.6964286082499735) | > loss_disc_real_0: 0.05456763505935669 (0.15667394256672343) | > loss_disc_real_1: 0.29779481887817383 (0.22346587116653854) | > loss_disc_real_2: 0.286312460899353 (0.2435256013193646) | > loss_disc_real_3: 0.3793871998786926 (0.24554683912444758) | > loss_disc_real_4: 0.2841038703918457 (0.24573452086062045) | > loss_disc_real_5: 0.21653731167316437 (0.2417549575502808) | > loss_0: 3.0456631183624268 (2.6964286082499735) | > grad_norm_0: tensor(356.9416, device='cuda:0') (tensor(245.2280, device='cuda:0')) | > loss_gen: 1.9216870069503784 (2.170060634613038) | > loss_kl: 2.141636371612549 (2.1895790132316377) | > loss_feat: 3.136875629425049 (3.6607011975468815) | > loss_mel: 18.16645050048828 (17.45742117392051) | > loss_duration: 1.6068813800811768 (1.5581298293294132) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.973529815673828 (27.035891764872783) | > grad_norm_1: tensor(7180.8237, device='cuda:0') (tensor(6255.1050, device='cuda:0')) | > current_lr_0: 0.00017880759973817103 | > current_lr_1: 0.00017880759973817103 | > step_time: 0.4532 (0.43849078384605616) | > loader_time: 0.0078 (0.007324573155995962) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868111596980565 (-0.00010222448429591212) | > avg_step_time: 0.4552167402186864 (-0.0009164810180664062) | > avg_loss_disc: 2.730121384204273 (+0.09138308444493193) | > avg_loss_disc_real_0: 0.1637943489870555 (+0.0416086710674662) | > avg_loss_disc_real_1: 0.23105701531322909 (-0.006952616215591706) | > avg_loss_disc_real_2: 0.24914607951338866 (+0.014621676063873434) | > avg_loss_disc_real_3: 0.25035862729582986 (+0.016765556914705604) | > avg_loss_disc_real_4: 0.247972814339987 (+0.008700559466657515) | > avg_loss_disc_real_5: 0.24584647677314114 (+0.0064469689214733394) | > avg_loss_0: 2.730121384204273 (+0.09138308444493193) | > avg_grad_norm_0: tensor(260.6874, device='cuda:0') (tensor(-1.2766, device='cuda:0')) | > avg_loss_gen: 2.099106983399727 (-0.058563022546364696) | > avg_loss_kl: 2.241091887715837 (+0.14390420074194088) | > avg_loss_feat: 3.4612196264132646 (-0.29541680510614965) | > avg_loss_mel: 17.539497187439824 (+0.7527791815744429) | > avg_loss_duration: 1.5825298218659951 (+0.027888360157818814) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.923445553846758 (+0.5705917788223438) | > avg_grad_norm_1: tensor(6372.1504, device='cuda:0') (tensor(-1683.3091, device='cuda:0'))  > EPOCH: 897/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:27:21)   --> TIME: 2025-02-25 17:27:30 -- STEP: 15/72 -- GLOBAL_STEP: 1064600 | > loss_disc: 2.640366554260254 (2.607751798629761) | > loss_disc_real_0: 0.1809324324131012 (0.16343486905097962) | > loss_disc_real_1: 0.16706141829490662 (0.21317278146743773) | > loss_disc_real_2: 0.23700423538684845 (0.2314306358496348) | > loss_disc_real_3: 0.1763627678155899 (0.23296917080879212) | > loss_disc_real_4: 0.23089538514614105 (0.24616446693738295) | > loss_disc_real_5: 0.239802286028862 (0.24189865191777546) | > loss_0: 2.640366554260254 (2.607751798629761) | > grad_norm_0: tensor(129.1092, device='cuda:0') (tensor(176.0053, device='cuda:0')) | > loss_gen: 2.558755874633789 (2.3220355351765947) | > loss_kl: 1.9820224046707153 (2.046650536855062) | > loss_feat: 4.136411190032959 (3.847803513209025) | > loss_mel: 17.392881393432617 (17.173422241210936) | > loss_duration: 1.5105222463607788 (1.510807204246521) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.58059310913086 (26.90071881612142) | > grad_norm_1: tensor(2675.9243, device='cuda:0') (tensor(4509.5195, device='cuda:0')) | > current_lr_0: 0.00017878524878820376 | > current_lr_1: 0.00017878524878820376 | > step_time: 0.4351 (0.43517913818359377) | > loader_time: 0.0061 (0.00642552375793457)  --> TIME: 2025-02-25 17:27:54 -- STEP: 65/72 -- GLOBAL_STEP: 1064650 | > loss_disc: 2.6332006454467773 (2.647014515216534) | > loss_disc_real_0: 0.0379210039973259 (0.1308513244184164) | > loss_disc_real_1: 0.12388892471790314 (0.22412017331673548) | > loss_disc_real_2: 0.29361969232559204 (0.2581631483939978) | > loss_disc_real_3: 0.1980258971452713 (0.2381848197716933) | > loss_disc_real_4: 0.2861430048942566 (0.24898692965507507) | > loss_disc_real_5: 0.2594754695892334 (0.24757329248465024) | > loss_0: 2.6332006454467773 (2.647014515216534) | > grad_norm_0: tensor(43.2969, device='cuda:0') (tensor(196.9371, device='cuda:0')) | > loss_gen: 2.3015098571777344 (2.2017338257569534) | > loss_kl: 2.8064286708831787 (2.1959063456608696) | > loss_feat: 4.690807342529297 (3.6762277419750506) | > loss_mel: 18.836685180664062 (17.19487414726845) | > loss_duration: 1.6850488185882568 (1.559367638367873) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.320480346679688 (26.82810962383564) | > grad_norm_1: tensor(1772.8602, device='cuda:0') (tensor(5555.0347, device='cuda:0')) | > current_lr_0: 0.00017878524878820376 | > current_lr_1: 0.00017878524878820376 | > step_time: 0.5118 (0.4586125557239239) | > loader_time: 0.0119 (0.008374867072472205) --> EVAL PERFORMANCE | > avg_loader_time: 0.008654651507525377 (-2.6464462280273438e-05) | > avg_step_time: 0.4630831698296775 (+0.0078664296109911) | > avg_loss_disc: 2.6373920608574237 (-0.09272932334684914) | > avg_loss_disc_real_0: 0.12415040872046645 (-0.039643940266589045) | > avg_loss_disc_real_1: 0.23009083178681386 (-0.0009661835264152263) | > avg_loss_disc_real_2: 0.25463627635593145 (+0.005490196842542788) | > avg_loss_disc_real_3: 0.23780158183104555 (-0.012557045464784311) | > avg_loss_disc_real_4: 0.2458765189832365 (-0.0020962953567504883) | > avg_loss_disc_real_5: 0.24775250969638288 (+0.001906032923241735) | > avg_loss_0: 2.6373920608574237 (-0.09272932334684914) | > avg_grad_norm_0: tensor(192.0052, device='cuda:0') (tensor(-68.6822, device='cuda:0')) | > avg_loss_gen: 2.1993332883002052 (+0.10022630490047835) | > avg_loss_kl: 2.2240072404834583 (-0.017084647232378458) | > avg_loss_feat: 3.7097752866610674 (+0.24855566024780273) | > avg_loss_mel: 17.280056443012942 (-0.2594407444268825) | > avg_loss_duration: 1.5672611216424217 (-0.015268700223573406) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.98043328943387 (+0.05698773558711068) | > avg_grad_norm_1: tensor(5440.9365, device='cuda:0') (tensor(-931.2139, device='cuda:0'))  > EPOCH: 898/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:27:59)   --> TIME: 2025-02-25 17:28:21 -- STEP: 43/72 -- GLOBAL_STEP: 1064700 | > loss_disc: 2.827336072921753 (2.625857092613398) | > loss_disc_real_0: 0.1310499757528305 (0.1310308300131975) | > loss_disc_real_1: 0.27152663469314575 (0.22304881485395653) | > loss_disc_real_2: 0.23428165912628174 (0.23741111471209414) | > loss_disc_real_3: 0.26516804099082947 (0.24536774394124053) | > loss_disc_real_4: 0.28352585434913635 (0.25319037153277285) | > loss_disc_real_5: 0.2657123804092407 (0.24955307467039242) | > loss_0: 2.827336072921753 (2.625857092613398) | > grad_norm_0: tensor(503.1375, device='cuda:0') (tensor(213.4036, device='cuda:0')) | > loss_gen: 2.0011117458343506 (2.1535676080127097) | > loss_kl: 2.1209795475006104 (2.135299873906513) | > loss_feat: 3.4792511463165283 (3.6395728477211886) | > loss_mel: 17.141990661621094 (17.449047044266102) | > loss_duration: 1.5780799388885498 (1.5379441837931789) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.321413040161133 (26.91543144403502) | > grad_norm_1: tensor(11658.6904, device='cuda:0') (tensor(7395.2710, device='cuda:0')) | > current_lr_0: 0.00017876290063210522 | > current_lr_1: 0.00017876290063210522 | > step_time: 0.4532 (0.44094245932823006) | > loader_time: 0.0098 (0.007595893948577171) --> EVAL PERFORMANCE | > avg_loader_time: 0.008694198769582828 (+3.954726205745128e-05) | > avg_step_time: 0.45737752444307567 (-0.005705645386601843) | > avg_loss_disc: 2.6765108444321326 (+0.03911878357470888) | > avg_loss_disc_real_0: 0.12775624144664957 (+0.003605832726183117) | > avg_loss_disc_real_1: 0.23572553505360241 (+0.005634703266788554) | > avg_loss_disc_real_2: 0.2366000705621612 (-0.018036205793770244) | > avg_loss_disc_real_3: 0.2506186114230625 (+0.01281702959201697) | > avg_loss_disc_real_4: 0.2397943873640517 (-0.006082131619184805) | > avg_loss_disc_real_5: 0.2459890626265969 (-0.0017634470697859839) | > avg_loss_0: 2.6765108444321326 (+0.03911878357470888) | > avg_grad_norm_0: tensor(296.6635, device='cuda:0') (+tensor(104.6582, device='cuda:0')) | > avg_loss_gen: 2.0662269290064423 (-0.1331063592937629) | > avg_loss_kl: 2.1450761724525775 (-0.07893106803088079) | > avg_loss_feat: 3.511605742951514 (-0.19816954370955342) | > avg_loss_mel: 17.162744038541554 (-0.11731240447138802) | > avg_loss_duration: 1.5618845677711595 (-0.005376553871262235) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.44753732815595 (-0.5328959612779194) | > avg_grad_norm_1: tensor(9534.2510, device='cuda:0') (+tensor(4093.3145, device='cuda:0'))  > EPOCH: 899/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:28:36)   --> TIME: 2025-02-25 17:28:48 -- STEP: 21/72 -- GLOBAL_STEP: 1064750 | > loss_disc: 2.6974596977233887 (2.603586037953695) | > loss_disc_real_0: 0.16274257004261017 (0.11045968603520166) | > loss_disc_real_1: 0.24972344934940338 (0.23496538187776292) | > loss_disc_real_2: 0.25601693987846375 (0.23129296870458693) | > loss_disc_real_3: 0.24198094010353088 (0.2369560854775565) | > loss_disc_real_4: 0.2579069435596466 (0.23768197851521627) | > loss_disc_real_5: 0.2616735100746155 (0.2406143212602252) | > loss_0: 2.6974596977233887 (2.603586037953695) | > grad_norm_0: tensor(487.5621, device='cuda:0') (tensor(336.0872, device='cuda:0')) | > loss_gen: 2.049145221710205 (2.050184272584461) | > loss_kl: 1.9059009552001953 (1.966503762063526) | > loss_feat: 3.004509687423706 (3.641338620867048) | > loss_mel: 15.118606567382812 (16.766388302757626) | > loss_duration: 1.5253762006759644 (1.5086951028733027) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.603538513183594 (25.933110100882395) | > grad_norm_1: tensor(12794.8340, device='cuda:0') (tensor(12870.3271, device='cuda:0')) | > current_lr_0: 0.0001787405552695262 | > current_lr_1: 0.0001787405552695262 | > step_time: 0.4423 (0.4370378993806385) | > loader_time: 0.0063 (0.006954090935843331)  --> TIME: 2025-02-25 17:29:12 -- STEP: 71/72 -- GLOBAL_STEP: 1064800 | > loss_disc: 2.53886342048645 (2.6108275735881965) | > loss_disc_real_0: 0.0994436964392662 (0.1252052717855279) | > loss_disc_real_1: 0.19280478358268738 (0.22744735655650286) | > loss_disc_real_2: 0.2337917685508728 (0.23575248558756332) | > loss_disc_real_3: 0.21759481728076935 (0.2341627707363854) | > loss_disc_real_4: 0.22990106046199799 (0.240834783080598) | > loss_disc_real_5: 0.23955056071281433 (0.24557399938643817) | > loss_0: 2.53886342048645 (2.6108275735881965) | > grad_norm_0: tensor(274.3112, device='cuda:0') (tensor(348.8146, device='cuda:0')) | > loss_gen: 2.0842669010162354 (2.0473625492042227) | > loss_kl: 1.8738017082214355 (2.0095712970679913) | > loss_feat: 3.444171905517578 (3.476948190742815) | > loss_mel: 16.111373901367188 (16.66772225876929) | > loss_duration: 1.5519500970840454 (1.5494505788238955) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.06556510925293 (25.75105479065801) | > grad_norm_1: tensor(12700.6934, device='cuda:0') (tensor(13066.3096, device='cuda:0')) | > current_lr_0: 0.0001787405552695262 | > current_lr_1: 0.0001787405552695262 | > step_time: 0.5108 (0.4563556187589404) | > loader_time: 0.0081 (0.008705659651420486) --> EVAL PERFORMANCE | > avg_loader_time: 0.008705659651420486 (+1.1460881837658046e-05) | > avg_step_time: 0.4563556187589404 (-0.0010219056841352936) | > avg_loss_disc: 2.6108275735881965 (-0.0656832708439361) | > avg_loss_disc_real_0: 0.1252052717855279 (-0.0025509696611216715) | > avg_loss_disc_real_1: 0.22744735655650286 (-0.008278178497099553) | > avg_loss_disc_real_2: 0.23575248558756332 (-0.000847584974597887) | > avg_loss_disc_real_3: 0.2341627707363854 (-0.01645584068667713) | > avg_loss_disc_real_4: 0.240834783080598 (+0.001040395716546294) | > avg_loss_disc_real_5: 0.24557399938643817 (-0.00041506324015871954) | > avg_loss_0: 2.6108275735881965 (-0.0656832708439361) | > avg_grad_norm_0: tensor(348.8146, device='cuda:0') (+tensor(52.1512, device='cuda:0')) | > avg_loss_gen: 2.0473625492042227 (-0.018864379802219666) | > avg_loss_kl: 2.0095712970679913 (-0.13550487538458622) | > avg_loss_feat: 3.476948190742815 (-0.03465755220869893) | > avg_loss_mel: 16.66772225876929 (-0.49502177977226225) | > avg_loss_duration: 1.5494505788238955 (-0.012433988947263996) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.75105479065801 (-0.6964825374979391) | > avg_grad_norm_1: tensor(13066.3096, device='cuda:0') (+tensor(3532.0586, device='cuda:0'))  > EPOCH: 900/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:29:13)   --> TIME: 2025-02-25 17:29:38 -- STEP: 49/72 -- GLOBAL_STEP: 1064850 | > loss_disc: 2.675267457962036 (2.604187726974487) | > loss_disc_real_0: 0.17083197832107544 (0.1234158425002682) | > loss_disc_real_1: 0.18025623261928558 (0.2176061214841142) | > loss_disc_real_2: 0.22990313172340393 (0.23346418111908193) | > loss_disc_real_3: 0.21639996767044067 (0.2325423445020403) | > loss_disc_real_4: 0.23731988668441772 (0.24198719342144168) | > loss_disc_real_5: 0.2336212396621704 (0.24468596948652852) | > loss_0: 2.675267457962036 (2.604187726974487) | > grad_norm_0: tensor(542.9719, device='cuda:0') (tensor(369.5752, device='cuda:0')) | > loss_gen: 2.1303677558898926 (2.068428659925655) | > loss_kl: 2.0404880046844482 (1.966541599254219) | > loss_feat: 4.149683952331543 (3.6603747581949038) | > loss_mel: 17.159704208374023 (16.627885292987433) | > loss_duration: 1.5892704725265503 (1.5267297540392197) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.06951332092285 (25.849959976819097) | > grad_norm_1: tensor(13589.1299, device='cuda:0') (tensor(13838.1377, device='cuda:0')) | > current_lr_0: 0.0001787182127001175 | > current_lr_1: 0.0001787182127001175 | > step_time: 0.4453 (0.44272025264039333) | > loader_time: 0.0097 (0.007899298959848833) --> EVAL PERFORMANCE | > avg_loader_time: 0.008785496295337945 (+7.983664391745908e-05) | > avg_step_time: 0.46692500651722224 (+0.010569387758281867) | > avg_loss_disc: 2.629759110195535 (+0.018931536607338373) | > avg_loss_disc_real_0: 0.12814173805461798 (+0.0029364662690900822) | > avg_loss_disc_real_1: 0.22295353614108662 (-0.004493820415416239) | > avg_loss_disc_real_2: 0.23747515342604947 (+0.0017226678384861527) | > avg_loss_disc_real_3: 0.23203877475060208 (-0.0021239959857833057) | > avg_loss_disc_real_4: 0.24367273903228867 (+0.002837955951690674) | > avg_loss_disc_real_5: 0.2451180062243636 (-0.00045599316207456386) | > avg_loss_0: 2.629759110195535 (+0.018931536607338373) | > avg_grad_norm_0: tensor(332.0300, device='cuda:0') (tensor(-16.7846, device='cuda:0')) | > avg_loss_gen: 2.064790806300204 (+0.01742825709598117) | > avg_loss_kl: 2.0400001734075413 (+0.030428876339549937) | > avg_loss_feat: 3.592291684217856 (+0.11534349347504103) | > avg_loss_mel: 16.797251002889286 (+0.1295287441199946) | > avg_loss_duration: 1.5507248169939285 (+0.0012742381700330263) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.045058425043667 (+0.2940036343856569) | > avg_grad_norm_1: tensor(12850.2695, device='cuda:0') (tensor(-216.0400, device='cuda:0'))  > EPOCH: 901/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:29:51)   --> TIME: 2025-02-25 17:30:05 -- STEP: 27/72 -- GLOBAL_STEP: 1064900 | > loss_disc: 2.5372166633605957 (2.6516406536102295) | > loss_disc_real_0: 0.1505534052848816 (0.11632954963931331) | > loss_disc_real_1: 0.24397484958171844 (0.20968388296939708) | > loss_disc_real_2: 0.247602641582489 (0.24883899865327058) | > loss_disc_real_3: 0.24582712352275848 (0.24434777487207338) | > loss_disc_real_4: 0.2406618446111679 (0.2448861262312642) | > loss_disc_real_5: 0.23603112995624542 (0.23973813432234306) | > loss_0: 2.5372166633605957 (2.6516406536102295) | > grad_norm_0: tensor(140.7825, device='cuda:0') (tensor(335.1360, device='cuda:0')) | > loss_gen: 2.255558967590332 (2.012244215718022) | > loss_kl: 2.0512592792510986 (1.9522799871585987) | > loss_feat: 3.4512808322906494 (3.5341854183762162) | > loss_mel: 15.70698356628418 (16.717717312000417) | > loss_duration: 1.5179365873336792 (1.5075916449228923) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.983020782470703 (25.724018238208913) | > grad_norm_1: tensor(9608.0957, device='cuda:0') (tensor(14128.2754, device='cuda:0')) | > current_lr_0: 0.00017869587292352996 | > current_lr_1: 0.00017869587292352996 | > step_time: 0.4483 (0.4383424123128255) | > loader_time: 0.0085 (0.006906880272759331) --> EVAL PERFORMANCE | > avg_loader_time: 0.008649661507405024 (-0.00013583478793292164) | > avg_step_time: 0.45948548719916543 (-0.007439519318056809) | > avg_loss_disc: 2.635505159136275 (+0.005746048940740156) | > avg_loss_disc_real_0: 0.1253005824458431 (-0.0028411556087748757) | > avg_loss_disc_real_1: 0.21601386095436526 (-0.006939675186721367) | > avg_loss_disc_real_2: 0.24354255052519516 (+0.006067397099145694) | > avg_loss_disc_real_3: 0.2398906515937456 (+0.007851876843143507) | > avg_loss_disc_real_4: 0.24572031598695568 (+0.0020475769546670075) | > avg_loss_disc_real_5: 0.24365245414451814 (-0.0014655520798454724) | > avg_loss_0: 2.635505159136275 (+0.005746048940740156) | > avg_grad_norm_0: tensor(345.4166, device='cuda:0') (+tensor(13.3866, device='cuda:0')) | > avg_loss_gen: 2.033580242747992 (-0.031210563552211745) | > avg_loss_kl: 2.0128687804853413 (-0.027131392922199904) | > avg_loss_feat: 3.4747934341430664 (-0.11749825007478965) | > avg_loss_mel: 16.59418819320034 (-0.20306280968894797) | > avg_loss_duration: 1.548878659664745 (-0.0018461573291834732) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.66430903152681 (-0.380749393516858) | > avg_grad_norm_1: tensor(13084.1045, device='cuda:0') (+tensor(233.8350, device='cuda:0'))  > EPOCH: 902/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:30:28)   --> TIME: 2025-02-25 17:30:33 -- STEP: 5/72 -- GLOBAL_STEP: 1064950 | > loss_disc: 2.5760879516601562 (2.5975532054901125) | > loss_disc_real_0: 0.09454792737960815 (0.11278558820486069) | > loss_disc_real_1: 0.15695834159851074 (0.19987209737300873) | > loss_disc_real_2: 0.23704269528388977 (0.2139851361513138) | > loss_disc_real_3: 0.1979684829711914 (0.22045804858207702) | > loss_disc_real_4: 0.23086759448051453 (0.23753415048122406) | > loss_disc_real_5: 0.2524421215057373 (0.24006090462207794) | > loss_0: 2.5760879516601562 (2.5975532054901125) | > grad_norm_0: tensor(363.9623, device='cuda:0') (tensor(373.3122, device='cuda:0')) | > loss_gen: 2.149186611175537 (2.0853474617004393) | > loss_kl: 1.9224368333816528 (1.8824548959732055) | > loss_feat: 3.8479225635528564 (3.950532245635986) | > loss_mel: 16.727020263671875 (17.175609588623047) | > loss_duration: 1.4950547218322754 (1.510762333869934) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.141620635986328 (26.6047061920166) | > grad_norm_1: tensor(10914.8838, device='cuda:0') (tensor(14870.9814, device='cuda:0')) | > current_lr_0: 0.00017867353593941453 | > current_lr_1: 0.00017867353593941453 | > step_time: 0.4329 (0.42879362106323243) | > loader_time: 0.009 (0.006915378570556641)  --> TIME: 2025-02-25 17:30:56 -- STEP: 55/72 -- GLOBAL_STEP: 1065000 | > loss_disc: 2.81268310546875 (2.760070527683605) | > loss_disc_real_0: 0.18760815262794495 (0.22118107033046802) | > loss_disc_real_1: 0.1770879030227661 (0.22334130988879639) | > loss_disc_real_2: 0.22566477954387665 (0.22133246768604625) | > loss_disc_real_3: 0.2512888014316559 (0.22761264714327725) | > loss_disc_real_4: 0.2530120313167572 (0.24039294069463557) | > loss_disc_real_5: 0.304241806268692 (0.241299292716113) | > loss_0: 2.81268310546875 (2.760070527683605) | > grad_norm_0: tensor(386.8418, device='cuda:0') (tensor(266.7117, device='cuda:0')) | > loss_gen: 1.9887926578521729 (2.0975847309285953) | > loss_kl: 2.2709295749664307 (2.3668710751966997) | > loss_feat: 3.0704383850097656 (3.5870423447002064) | > loss_mel: 18.650976181030273 (18.305358331853697) | > loss_duration: 1.6304526329040527 (1.5942062833092432) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.611589431762695 (27.95106270530007) | > grad_norm_1: tensor(9026.8574, device='cuda:0') (tensor(6176.2358, device='cuda:0')) | > current_lr_0: 0.00017867353593941453 | > current_lr_1: 0.00017867353593941453 | > step_time: 0.4634 (0.4437009161168879) | > loader_time: 0.0098 (0.007999671589244495) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1065000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008768736476629551 (+0.00011907496922452729) | > avg_step_time: 0.45645933755686585 (-0.0030261496422995804) | > avg_loss_disc: 2.751081043565777 (+0.11557588442950184) | > avg_loss_disc_real_0: 0.22196302589186476 (+0.09666244344602165) | > avg_loss_disc_real_1: 0.2226263027585728 (+0.006612441804207558) | > avg_loss_disc_real_2: 0.22263101939584168 (-0.020911531129353483) | > avg_loss_disc_real_3: 0.2302379935560092 (-0.009652658037736378) | > avg_loss_disc_real_4: 0.2398010211091646 (-0.005919294877791081) | > avg_loss_disc_real_5: 0.2411910310177736 (-0.0024614231267445497) | > avg_loss_0: 2.751081043565777 (+0.11557588442950184) | > avg_grad_norm_0: tensor(255.6575, device='cuda:0') (tensor(-89.7592, device='cuda:0')) | > avg_loss_gen: 2.0810383726173733 (+0.04745812986938125) | > avg_loss_kl: 2.343086662426801 (+0.3302178819414596) | > avg_loss_feat: 3.515667586259439 (+0.040874152116372464) | > avg_loss_mel: 18.131997068163376 (+1.5378088749630372) | > avg_loss_duration: 1.601677197805593 (+0.052798538140848006) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.673466857050506 (+2.0091578255236975) | > avg_grad_norm_1: tensor(6432.5762, device='cuda:0') (tensor(-6651.5283, device='cuda:0'))  > EPOCH: 903/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:31:09)   --> TIME: 2025-02-25 17:31:26 -- STEP: 33/72 -- GLOBAL_STEP: 1065050 | > loss_disc: 2.488250494003296 (2.7044142520788945) | > loss_disc_real_0: 0.04727286100387573 (0.20598863381328006) | > loss_disc_real_1: 0.24453961849212646 (0.20980352682597708) | > loss_disc_real_2: 0.20270343124866486 (0.22826565169926846) | > loss_disc_real_3: 0.19599899649620056 (0.23527069570440234) | > loss_disc_real_4: 0.23853017389774323 (0.2453079977721879) | > loss_disc_real_5: 0.2521761953830719 (0.24180591422500033) | > loss_0: 2.488250494003296 (2.7044142520788945) | > grad_norm_0: tensor(39.5486, device='cuda:0') (tensor(223.6477, device='cuda:0')) | > loss_gen: 2.218334674835205 (2.037948976863515) | > loss_kl: 2.5405361652374268 (2.0711041038686577) | > loss_feat: 4.231988430023193 (3.615163232340957) | > loss_mel: 19.601757049560547 (17.466100403756798) | > loss_duration: 1.5749120712280273 (1.531839370727539) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.167530059814453 (26.722156120069098) | > grad_norm_1: tensor(2812.7107, device='cuda:0') (tensor(5977.6665, device='cuda:0')) | > current_lr_0: 0.0001786512017474221 | > current_lr_1: 0.0001786512017474221 | > step_time: 0.4535 (0.4399786934708104) | > loader_time: 0.0079 (0.007167845061331084) --> EVAL PERFORMANCE | > avg_loader_time: 0.008607468134920363 (-0.00016126834170918786) | > avg_step_time: 0.4578481392121651 (+0.0013888016552992544) | > avg_loss_disc: 2.7288345283185933 (-0.022246515247183574) | > avg_loss_disc_real_0: 0.1634939209571187 (-0.05846910493474605) | > avg_loss_disc_real_1: 0.2141724174291315 (-0.008453885329441302) | > avg_loss_disc_real_2: 0.23150423190123598 (+0.008873212505394301) | > avg_loss_disc_real_3: 0.24151830606057612 (+0.011280312504566903) | > avg_loss_disc_real_4: 0.25075769151600313 (+0.010956670406838537) | > avg_loss_disc_real_5: 0.24157819592616928 (+0.0003871649083956874) | > avg_loss_0: 2.7288345283185933 (-0.022246515247183574) | > avg_grad_norm_0: tensor(212.3133, device='cuda:0') (tensor(-43.3442, device='cuda:0')) | > avg_loss_gen: 2.0161216175052488 (-0.06491675511212458) | > avg_loss_kl: 2.1534093651973025 (-0.18967729722949844) | > avg_loss_feat: 3.5627903535332477 (+0.04712276727380882) | > avg_loss_mel: 17.544484554881773 (-0.5875125132816024) | > avg_loss_duration: 1.5694774540377336 (-0.03219974376785939) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.846283415673483 (-0.8271834413770236) | > avg_grad_norm_1: tensor(6339.2974, device='cuda:0') (tensor(-93.2788, device='cuda:0'))  > EPOCH: 904/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:31:46)   --> TIME: 2025-02-25 17:31:53 -- STEP: 11/72 -- GLOBAL_STEP: 1065100 | > loss_disc: 2.611675500869751 (2.6922828500921074) | > loss_disc_real_0: 0.19423888623714447 (0.08939681574702263) | > loss_disc_real_1: 0.22512395679950714 (0.22677916559306058) | > loss_disc_real_2: 0.2801232635974884 (0.271078884601593) | > loss_disc_real_3: 0.28218382596969604 (0.2171910825100812) | > loss_disc_real_4: 0.24610039591789246 (0.2470069486986507) | > loss_disc_real_5: 0.24998252093791962 (0.25470245561816474) | > loss_0: 2.611675500869751 (2.6922828500921074) | > grad_norm_0: tensor(286.5918, device='cuda:0') (tensor(227.2945, device='cuda:0')) | > loss_gen: 2.438854455947876 (1.98616530678489) | > loss_kl: 2.0561258792877197 (1.9944339773871682) | > loss_feat: 3.5429983139038086 (3.4380906278436836) | > loss_mel: 17.29562759399414 (17.095096934925426) | > loss_duration: 1.5060157775878906 (1.5231704386797817) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.839622497558594 (26.036958000876687) | > grad_norm_1: tensor(5610.0698, device='cuda:0') (tensor(6409.4639, device='cuda:0')) | > current_lr_0: 0.00017862887034720366 | > current_lr_1: 0.00017862887034720366 | > step_time: 0.4206 (0.43235449357466266) | > loader_time: 0.0057 (0.0063982443376020956)  --> TIME: 2025-02-25 17:32:16 -- STEP: 61/72 -- GLOBAL_STEP: 1065150 | > loss_disc: 2.557910442352295 (2.6479274640317825) | > loss_disc_real_0: 0.21197564899921417 (0.1083346386669112) | > loss_disc_real_1: 0.22488288581371307 (0.22474337845552164) | > loss_disc_real_2: 0.2174285501241684 (0.25499490636294014) | > loss_disc_real_3: 0.2452983409166336 (0.23505038658126456) | > loss_disc_real_4: 0.26366692781448364 (0.24605033778753438) | > loss_disc_real_5: 0.26211509108543396 (0.24767378778731236) | > loss_0: 2.557910442352295 (2.6479274640317825) | > grad_norm_0: tensor(332.3324, device='cuda:0') (tensor(196.5838, device='cuda:0')) | > loss_gen: 2.356175661087036 (2.077186058779232) | > loss_kl: 2.2084813117980957 (2.0512182673469903) | > loss_feat: 3.2692556381225586 (3.540644645690918) | > loss_mel: 16.187110900878906 (16.91002536210858) | > loss_duration: 1.6080107688903809 (1.5489966380791587) | > amp_scaler: 16.0 (12.983606557377048) | > loss_1: 25.62903594970703 (26.128071206514953) | > grad_norm_1: tensor(4995.2598, device='cuda:0') (tensor(6364.4414, device='cuda:0')) | > current_lr_0: 0.00017862887034720366 | > current_lr_1: 0.00017862887034720366 | > step_time: 0.4787 (0.4493548948256696) | > loader_time: 0.0109 (0.00821870663126961) --> EVAL PERFORMANCE | > avg_loader_time: 0.008678577315639441 (+7.110918071907835e-05) | > avg_step_time: 0.45766684370981137 (-0.00018129550235373992) | > avg_loss_disc: 2.644666738913093 (-0.0841677894055004) | > avg_loss_disc_real_0: 0.1116633275654954 (-0.05183059339162331) | > avg_loss_disc_real_1: 0.22373752925597445 (+0.009565111826842942) | > avg_loss_disc_real_2: 0.2521531330028051 (+0.02064890110156914) | > avg_loss_disc_real_3: 0.2355560498758101 (-0.005962256184766007) | > avg_loss_disc_real_4: 0.24546872079372406 (-0.005288970722279074) | > avg_loss_disc_real_5: 0.24859015979397464 (+0.007011963867805365) | > avg_loss_0: 2.644666738913093 (-0.0841677894055004) | > avg_grad_norm_0: tensor(193.8359, device='cuda:0') (tensor(-18.4773, device='cuda:0')) | > avg_loss_gen: 2.0735834820169807 (+0.057461864511731964) | > avg_loss_kl: 2.070802742326762 (-0.08260662287054066) | > avg_loss_feat: 3.493964930655251 (-0.06882542287799653) | > avg_loss_mel: 16.903083317716362 (-0.641401237165411) | > avg_loss_duration: 1.558302728223129 (-0.011174725814604658) | > avg_amp_scaler: 13.408450704225352 (+5.408450704225352) | > avg_loss_1: 26.09973743599905 (-0.7465459796744334) | > avg_grad_norm_1: tensor(6304.9434, device='cuda:0') (tensor(-34.3540, device='cuda:0'))  > EPOCH: 905/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:32:23)   --> TIME: 2025-02-25 17:32:43 -- STEP: 39/72 -- GLOBAL_STEP: 1065200 | > loss_disc: 2.6578238010406494 (2.6298385705703344) | > loss_disc_real_0: 0.06276266276836395 (0.13649430231024062) | > loss_disc_real_1: 0.32982876896858215 (0.23467420423642182) | > loss_disc_real_2: 0.2897806465625763 (0.2580521977864779) | > loss_disc_real_3: 0.2505372166633606 (0.24312652188998002) | > loss_disc_real_4: 0.2665509879589081 (0.25016026428112614) | > loss_disc_real_5: 0.2611604928970337 (0.25454801053572923) | > loss_0: 2.6578238010406494 (2.6298385705703344) | > grad_norm_0: tensor(217.9912, device='cuda:0') (tensor(150.7742, device='cuda:0')) | > loss_gen: 2.2640974521636963 (2.2915708285111647) | > loss_kl: 3.963444471359253 (3.0659709526942325) | > loss_feat: 3.806859254837036 (3.8500915246132092) | > loss_mel: 25.422821044921875 (21.215786102490544) | > loss_duration: 1.7042300701141357 (1.6331416368484497) | > amp_scaler: 16.0 (16.0) | > loss_1: 37.16145324707031 (32.0565609565148) | > grad_norm_1: tensor(8538.9922, device='cuda:0') (tensor(5586.5850, device='cuda:0')) | > current_lr_0: 0.00017860654173841025 | > current_lr_1: 0.00017860654173841025 | > step_time: 0.4487 (0.4428267662341778) | > loader_time: 0.0082 (0.007391654528104342) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872095873658086 (+4.2381420941418196e-05) | > avg_step_time: 0.4601581936151209 (+0.0024913499053095256) | > avg_loss_disc: 2.6354025182589678 (-0.009264220654125133) | > avg_loss_disc_real_0: 0.13096755202597296 (+0.019304224460477568) | > avg_loss_disc_real_1: 0.2198487963055221 (-0.0038887329504523616) | > avg_loss_disc_real_2: 0.2428977067621661 (-0.00925542624063902) | > avg_loss_disc_real_3: 0.23857603064725097 (+0.003019980771440861) | > avg_loss_disc_real_4: 0.24639149891658563 (+0.0009227781228615672) | > avg_loss_disc_real_5: 0.2501879011241482 (+0.001597741330173541) | > avg_loss_0: 2.6354025182589678 (-0.009264220654125133) | > avg_grad_norm_0: tensor(164.8816, device='cuda:0') (tensor(-28.9543, device='cuda:0')) | > avg_loss_gen: 2.1899264899777693 (+0.11634300796078856) | > avg_loss_kl: 2.913285735627295 (+0.8424829933005333) | > avg_loss_feat: 3.752322348070816 (+0.25835741741556495) | > avg_loss_mel: 20.417469360459016 (+3.514386042742654) | > avg_loss_duration: 1.6416158776887708 (+0.08331314946564183) | > avg_amp_scaler: 12.619718309859154 (-0.7887323943661979) | > avg_loss_1: 30.91461979503363 (+4.814882359034581) | > avg_grad_norm_1: tensor(5843.8994, device='cuda:0') (tensor(-461.0439, device='cuda:0'))  > EPOCH: 906/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:33:01)   --> TIME: 2025-02-25 17:33:10 -- STEP: 17/72 -- GLOBAL_STEP: 1065250 | > loss_disc: 2.703178882598877 (2.7731076268588795) | > loss_disc_real_0: 0.13838078081607819 (0.32099450423437004) | > loss_disc_real_1: 0.19410093128681183 (0.22567696781719432) | > loss_disc_real_2: 0.20420172810554504 (0.24087774928878336) | > loss_disc_real_3: 0.24654974043369293 (0.24790961952770457) | > loss_disc_real_4: 0.24018119275569916 (0.25859768162755403) | > loss_disc_real_5: 0.2521267831325531 (0.25758376629913554) | > loss_0: 2.703178882598877 (2.7731076268588795) | > grad_norm_0: tensor(123.1214, device='cuda:0') (tensor(268.9685, device='cuda:0')) | > loss_gen: 1.885627269744873 (2.22790088373072) | > loss_kl: 2.267580032348633 (2.140646724139943) | > loss_feat: 3.2609806060791016 (3.27518666491789) | > loss_mel: 17.928991317749023 (17.70185369603774) | > loss_duration: 1.554155707359314 (1.530757237883175) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.897335052490234 (26.87634546616498) | > grad_norm_1: tensor(8705.8945, device='cuda:0') (tensor(6223.8896, device='cuda:0')) | > current_lr_0: 0.00017858421592069294 | > current_lr_1: 0.00017858421592069294 | > step_time: 0.4451 (0.4381740654216093) | > loader_time: 0.0062 (0.007062855888815487)  --> TIME: 2025-02-25 17:33:34 -- STEP: 67/72 -- GLOBAL_STEP: 1065300 | > loss_disc: 2.7628731727600098 (2.706731219789875) | > loss_disc_real_0: 0.09123551845550537 (0.21469737739482916) | > loss_disc_real_1: 0.2232179492712021 (0.22481501502777212) | > loss_disc_real_2: 0.24113641679286957 (0.2649156933844979) | > loss_disc_real_3: 0.20937548577785492 (0.23218693484121294) | > loss_disc_real_4: 0.23187321424484253 (0.24651316008461055) | > loss_disc_real_5: 0.2360466569662094 (0.24937887667719996) | > loss_0: 2.7628731727600098 (2.706731219789875) | > grad_norm_0: tensor(53.7903, device='cuda:0') (tensor(213.2520, device='cuda:0')) | > loss_gen: 1.7270002365112305 (2.1472805845203693) | > loss_kl: 2.146121025085449 (2.1425957359484773) | > loss_feat: 2.964900493621826 (3.3708379055137065) | > loss_mel: 16.964820861816406 (17.421228052964857) | > loss_duration: 1.6421616077423096 (1.5679550277653025) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.445003509521484 (26.649897475740804) | > grad_norm_1: tensor(8529.1309, device='cuda:0') (tensor(5832.1875, device='cuda:0')) | > current_lr_0: 0.00017858421592069294 | > current_lr_1: 0.00017858421592069294 | > step_time: 0.5268 (0.4540626575697714) | > loader_time: 0.012 (0.008548081810794663) --> EVAL PERFORMANCE | > avg_loader_time: 0.008719558447179664 (-1.4002894011960754e-06) | > avg_step_time: 0.45811639705174406 (-0.00204179656337683) | > avg_loss_disc: 2.7055690960145333 (+0.07016657775556556) | > avg_loss_disc_real_0: 0.21014552526700658 (+0.07917797324103362) | > avg_loss_disc_real_1: 0.226472418283073 (+0.0066236219775508975) | > avg_loss_disc_real_2: 0.2616100588314969 (+0.018712352069330807) | > avg_loss_disc_real_3: 0.23210712658687377 (-0.006468904060377201) | > avg_loss_disc_real_4: 0.24617659814760717 (-0.00021490076897845434) | > avg_loss_disc_real_5: 0.24897132184303983 (-0.0012165792811083542) | > avg_loss_0: 2.7055690960145333 (+0.07016657775556556) | > avg_grad_norm_0: tensor(214.2645, device='cuda:0') (+tensor(49.3829, device='cuda:0')) | > avg_loss_gen: 2.128960844496607 (-0.060965645481162234) | > avg_loss_kl: 2.142774054701899 (-0.770511680925396) | > avg_loss_feat: 3.3397216494654267 (-0.4126006986053894) | > avg_loss_mel: 17.402394684267723 (-3.0150746761912934) | > avg_loss_duration: 1.5709271531709483 (-0.07068872451782249) | > avg_amp_scaler: 8.0 (-4.619718309859154) | > avg_loss_1: 26.584778530496948 (-4.329841264536682) | > avg_grad_norm_1: tensor(6025.2153, device='cuda:0') (+tensor(181.3159, device='cuda:0'))  > EPOCH: 907/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:33:38)   --> TIME: 2025-02-25 17:34:01 -- STEP: 45/72 -- GLOBAL_STEP: 1065350 | > loss_disc: 2.666907548904419 (2.675805727640787) | > loss_disc_real_0: 0.19822806119918823 (0.1392829902056191) | > loss_disc_real_1: 0.21975234150886536 (0.23375927607218425) | > loss_disc_real_2: 0.23523908853530884 (0.23318118883503808) | > loss_disc_real_3: 0.2738606631755829 (0.2399524516529507) | > loss_disc_real_4: 0.24783718585968018 (0.24503596391942767) | > loss_disc_real_5: 0.24895425140857697 (0.24538764821158515) | > loss_0: 2.666907548904419 (2.675805727640787) | > grad_norm_0: tensor(173.6213, device='cuda:0') (tensor(180.2239, device='cuda:0')) | > loss_gen: 2.0483486652374268 (2.061611925231086) | > loss_kl: 2.1485562324523926 (2.1463867134518093) | > loss_feat: 3.1146695613861084 (3.41858270433214) | > loss_mel: 17.67826271057129 (17.762218729654947) | > loss_duration: 1.5914428234100342 (1.603934974140591) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.581279754638672 (26.992735205756293) | > grad_norm_1: tensor(6070.6602, device='cuda:0') (tensor(6260.5396, device='cuda:0')) | > current_lr_0: 0.00017856189289370284 | > current_lr_1: 0.00017856189289370284 | > step_time: 0.4846 (0.44629509184095595) | > loader_time: 0.0111 (0.007675266265869141) --> EVAL PERFORMANCE | > avg_loader_time: 0.008741811967231857 (+2.2253520052193873e-05) | > avg_step_time: 0.4583635464520522 (+0.00024714940030812205) | > avg_loss_disc: 2.683702908771138 (-0.021866187243395174) | > avg_loss_disc_real_0: 0.11713957736714635 (-0.09300594789986023) | > avg_loss_disc_real_1: 0.23517017612155056 (+0.008697757838477566) | > avg_loss_disc_real_2: 0.24304575672451878 (-0.018564302106978126) | > avg_loss_disc_real_3: 0.2365819750957086 (+0.004474848508834839) | > avg_loss_disc_real_4: 0.24339516460895538 (-0.0027814335386517897) | > avg_loss_disc_real_5: 0.24277208115853055 (-0.006199240684509277) | > avg_loss_0: 2.683702908771138 (-0.021866187243395174) | > avg_grad_norm_0: tensor(178.4915, device='cuda:0') (tensor(-35.7730, device='cuda:0')) | > avg_loss_gen: 2.012240191580544 (-0.11672065291606293) | > avg_loss_kl: 2.1202568604912555 (-0.022517194210643687) | > avg_loss_feat: 3.406893078709992 (+0.06717142924456532) | > avg_loss_mel: 17.525960586440394 (+0.12356590217267183) | > avg_loss_duration: 1.6056429718581724 (+0.034715818687224065) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67099375120351 (+0.0862152207065634) | > avg_grad_norm_1: tensor(6079.4717, device='cuda:0') (+tensor(54.2563, device='cuda:0'))  > EPOCH: 908/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:34:15)   --> TIME: 2025-02-25 17:34:28 -- STEP: 23/72 -- GLOBAL_STEP: 1065400 | > loss_disc: 2.4982616901397705 (2.725293480831644) | > loss_disc_real_0: 0.09176705777645111 (0.2082975817763287) | > loss_disc_real_1: 0.2515624761581421 (0.25534574493117956) | > loss_disc_real_2: 0.21840336918830872 (0.24121485132238138) | > loss_disc_real_3: 0.25427621603012085 (0.2349787756152775) | > loss_disc_real_4: 0.23983129858970642 (0.25205650536910346) | > loss_disc_real_5: 0.2431362122297287 (0.25298442076081823) | > loss_0: 2.4982616901397705 (2.725293480831644) | > grad_norm_0: tensor(29.5934, device='cuda:0') (tensor(171.1514, device='cuda:0')) | > loss_gen: 2.3483142852783203 (2.130374390146007) | > loss_kl: 1.8530709743499756 (1.9406143530555393) | > loss_feat: 3.741643190383911 (3.398035194562829) | > loss_mel: 15.92699909210205 (16.944261841152024) | > loss_duration: 1.5272715091705322 (1.5198378200116365) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.39729881286621 (25.933123630026113) | > grad_norm_1: tensor(3943.4963, device='cuda:0') (tensor(6053.6006, device='cuda:0')) | > current_lr_0: 0.00017853957265709112 | > current_lr_1: 0.00017853957265709112 | > step_time: 0.4477 (0.43633967897166376) | > loader_time: 0.0063 (0.006696234578671662) --> EVAL PERFORMANCE | > avg_loader_time: 0.008639674791148012 (-0.00010213717608384583) | > avg_step_time: 0.455394039691334 (-0.0029695067607181658) | > avg_loss_disc: 2.675746192394848 (-0.007956716376290363) | > avg_loss_disc_real_0: 0.1379450410079788 (+0.020805463640832453) | > avg_loss_disc_real_1: 0.23059622271799704 (-0.004573953403553516) | > avg_loss_disc_real_2: 0.24565387065981475 (+0.002608113935295975) | > avg_loss_disc_real_3: 0.23788330441629382 (+0.001301329320585215) | > avg_loss_disc_real_4: 0.2510225697302482 (+0.007627405121292841) | > avg_loss_disc_real_5: 0.24906256312215833 (+0.006290481963627775) | > avg_loss_0: 2.675746192394848 (-0.007956716376290363) | > avg_grad_norm_0: tensor(191.3114, device='cuda:0') (+tensor(12.8198, device='cuda:0')) | > avg_loss_gen: 2.0555690191161466 (+0.04332882753560252) | > avg_loss_kl: 2.018395927590383 (-0.10186093290087239) | > avg_loss_feat: 3.467927361877871 (+0.06103428316787918) | > avg_loss_mel: 17.01753743937317 (-0.5084231470672229) | > avg_loss_duration: 1.5630903932410225 (-0.042552578617149894) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.122520231864822 (-0.5484735193386889) | > avg_grad_norm_1: tensor(6324.3940, device='cuda:0') (+tensor(244.9224, device='cuda:0'))  > EPOCH: 909/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:34:53)   --> TIME: 2025-02-25 17:34:55 -- STEP: 1/72 -- GLOBAL_STEP: 1065450 | > loss_disc: 2.3601326942443848 (2.3601326942443848) | > loss_disc_real_0: 0.0437987819314003 (0.0437987819314003) | > loss_disc_real_1: 0.221558079123497 (0.221558079123497) | > loss_disc_real_2: 0.21040143072605133 (0.21040143072605133) | > loss_disc_real_3: 0.22877100110054016 (0.22877100110054016) | > loss_disc_real_4: 0.2436075508594513 (0.2436075508594513) | > loss_disc_real_5: 0.23741421103477478 (0.23741421103477478) | > loss_0: 2.3601326942443848 (2.3601326942443848) | > grad_norm_0: tensor(120.4105, device='cuda:0') (tensor(120.4105, device='cuda:0')) | > loss_gen: 2.4084012508392334 (2.4084012508392334) | > loss_kl: 1.8755273818969727 (1.8755273818969727) | > loss_feat: 4.433705806732178 (4.433705806732178) | > loss_mel: 16.958805084228516 (16.958805084228516) | > loss_duration: 1.5629533529281616 (1.5629533529281616) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.23939323425293 (27.23939323425293) | > grad_norm_1: tensor(8179.7212, device='cuda:0') (tensor(8179.7212, device='cuda:0')) | > current_lr_0: 0.00017851725521050897 | > current_lr_1: 0.00017851725521050897 | > step_time: 0.4517 (0.451678991317749) | > loader_time: 0.0081 (0.008076906204223633)  --> TIME: 2025-02-25 17:35:18 -- STEP: 51/72 -- GLOBAL_STEP: 1065500 | > loss_disc: 2.5706706047058105 (2.650007963180542) | > loss_disc_real_0: 0.1814437210559845 (0.15230669877400582) | > loss_disc_real_1: 0.22953329980373383 (0.22573189437389374) | > loss_disc_real_2: 0.23596502840518951 (0.23383443787986158) | > loss_disc_real_3: 0.22647836804389954 (0.23452289022651374) | > loss_disc_real_4: 0.23152126371860504 (0.24254471209703707) | > loss_disc_real_5: 0.23162119090557098 (0.24654957330694385) | > loss_0: 2.5706706047058105 (2.650007963180542) | > grad_norm_0: tensor(362.1869, device='cuda:0') (tensor(228.9417, device='cuda:0')) | > loss_gen: 2.1279783248901367 (2.0541884922513773) | > loss_kl: 1.8737674951553345 (1.9512866244596594) | > loss_feat: 3.3401927947998047 (3.4997151084974703) | > loss_mel: 16.619388580322266 (16.901315352495985) | > loss_duration: 1.5792245864868164 (1.538993564306521) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.540550231933594 (25.945499120974073) | > grad_norm_1: tensor(10908.5830, device='cuda:0') (tensor(6726.8564, device='cuda:0')) | > current_lr_0: 0.00017851725521050897 | > current_lr_1: 0.00017851725521050897 | > step_time: 0.4462 (0.44436706748663213) | > loader_time: 0.009 (0.007874156914505305) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871452479295328 (+7.485000180526905e-05) | > avg_step_time: 0.4592999773965755 (+0.0039059377052415023) | > avg_loss_disc: 2.6578885333638795 (-0.017857659030968254) | > avg_loss_disc_real_0: 0.1644458950720203 (+0.026500854064041485) | > avg_loss_disc_real_1: 0.225179125935259 (-0.005417096782738051) | > avg_loss_disc_real_2: 0.23569634724670732 (-0.009957523413107433) | > avg_loss_disc_real_3: 0.2348420997740517 (-0.0030412046422421124) | > avg_loss_disc_real_4: 0.24214599564881392 (-0.008876574081434302) | > avg_loss_disc_real_5: 0.2448152205054189 (-0.004247342616739425) | > avg_loss_0: 2.6578885333638795 (-0.017857659030968254) | > avg_grad_norm_0: tensor(220.4086, device='cuda:0') (+tensor(29.0973, device='cuda:0')) | > avg_loss_gen: 2.0531822627698872 (-0.0023867563462594) | > avg_loss_kl: 1.9959759141357851 (-0.022420013454597987) | > avg_loss_feat: 3.4075248980186355 (-0.060402463859235755) | > avg_loss_mel: 16.80856049228722 (-0.20897694708595083) | > avg_loss_duration: 1.5572533070201606 (-0.005837086220861876) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.822496870873678 (-0.300023360991144) | > avg_grad_norm_1: tensor(6464.7402, device='cuda:0') (+tensor(140.3462, device='cuda:0'))  > EPOCH: 910/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:35:30)   --> TIME: 2025-02-25 17:35:45 -- STEP: 29/72 -- GLOBAL_STEP: 1065550 | > loss_disc: 2.4097673892974854 (2.572133352016581) | > loss_disc_real_0: 0.04150576889514923 (0.10498539525373228) | > loss_disc_real_1: 0.2271854430437088 (0.21326605770094642) | > loss_disc_real_2: 0.25866803526878357 (0.2254045122656329) | > loss_disc_real_3: 0.2429996281862259 (0.2433708582458825) | > loss_disc_real_4: 0.2664147913455963 (0.24538917325693985) | > loss_disc_real_5: 0.24251462519168854 (0.25034147603758455) | > loss_0: 2.4097673892974854 (2.572133352016581) | > grad_norm_0: tensor(103.4405, device='cuda:0') (tensor(238.9866, device='cuda:0')) | > loss_gen: 2.1568403244018555 (2.1423018718587947) | > loss_kl: 1.9312341213226318 (1.9234945650758415) | > loss_feat: 3.970618724822998 (3.7420451723296067) | > loss_mel: 17.79276466369629 (17.0712183590593) | > loss_duration: 1.508937954902649 (1.5141887541474968) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.360397338867188 (26.39324878824168) | > grad_norm_1: tensor(9011.6064, device='cuda:0') (tensor(6522.2866, device='cuda:0')) | > current_lr_0: 0.00017849494055360764 | > current_lr_1: 0.00017849494055360764 | > step_time: 0.4387 (0.43623652129337703) | > loader_time: 0.0075 (0.007153395948738888) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655306319115864 (-5.921847383741699e-05) | > avg_step_time: 0.4574688656229368 (-0.001831111773638705) | > avg_loss_disc: 2.6147311606877293 (-0.043157372676150274) | > avg_loss_disc_real_0: 0.12736088436254317 (-0.03708501070947712) | > avg_loss_disc_real_1: 0.22078894362063475 (-0.004390182314624236) | > avg_loss_disc_real_2: 0.23184226789105106 (-0.0038540793556562647) | > avg_loss_disc_real_3: 0.2378035609571027 (+0.0029614611830509896) | > avg_loss_disc_real_4: 0.24322610002168468 (+0.0010801043728707604) | > avg_loss_disc_real_5: 0.24883104776832418 (+0.0040158272629052805) | > avg_loss_0: 2.6147311606877293 (-0.043157372676150274) | > avg_grad_norm_0: tensor(198.8675, device='cuda:0') (tensor(-21.5412, device='cuda:0')) | > avg_loss_gen: 2.1289042385531145 (+0.07572197578322726) | > avg_loss_kl: 2.0087025031237538 (+0.012726588987968634) | > avg_loss_feat: 3.6156872527700075 (+0.20816235475137201) | > avg_loss_mel: 17.109788169323554 (+0.3012276770363336) | > avg_loss_duration: 1.557784958624504 (+0.0005316516043434305) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.420867140863983 (+0.5983702699903048) | > avg_grad_norm_1: tensor(6595.8359, device='cuda:0') (+tensor(131.0957, device='cuda:0'))  > EPOCH: 911/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:36:07)   --> TIME: 2025-02-25 17:36:12 -- STEP: 7/72 -- GLOBAL_STEP: 1065600 | > loss_disc: 2.5782434940338135 (2.6952876704079762) | > loss_disc_real_0: 0.25493931770324707 (0.2688198174749102) | > loss_disc_real_1: 0.15273180603981018 (0.23772431484290532) | > loss_disc_real_2: 0.21124857664108276 (0.2097819937126977) | > loss_disc_real_3: 0.22287918627262115 (0.22320749929973058) | > loss_disc_real_4: 0.2200230062007904 (0.2382839513676507) | > loss_disc_real_5: 0.24472302198410034 (0.2509174219199589) | > loss_0: 2.5782434940338135 (2.6952876704079762) | > grad_norm_0: tensor(277.9052, device='cuda:0') (tensor(234.1275, device='cuda:0')) | > loss_gen: 2.252880811691284 (2.100850224494934) | > loss_kl: 1.9843477010726929 (1.9417042902537756) | > loss_feat: 3.4869279861450195 (3.3109641075134277) | > loss_mel: 17.833148956298828 (16.97361101422991) | > loss_duration: 1.5243932008743286 (1.5231620073318481) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.081697463989258 (25.85029193333217) | > grad_norm_1: tensor(3008.5266, device='cuda:0') (tensor(6814.5698, device='cuda:0')) | > current_lr_0: 0.00017847262868603844 | > current_lr_1: 0.00017847262868603844 | > step_time: 0.4442 (0.43672983986990793) | > loader_time: 0.0063 (0.0069544655936104914)  --> TIME: 2025-02-25 17:36:35 -- STEP: 57/72 -- GLOBAL_STEP: 1065650 | > loss_disc: 2.6609208583831787 (2.6457644052672804) | > loss_disc_real_0: 0.13864177465438843 (0.13201531682882395) | > loss_disc_real_1: 0.21233612298965454 (0.2181248853081151) | > loss_disc_real_2: 0.26487305760383606 (0.23448377324823746) | > loss_disc_real_3: 0.23993967473506927 (0.23268571626721768) | > loss_disc_real_4: 0.2622206211090088 (0.24085233399742528) | > loss_disc_real_5: 0.23364242911338806 (0.24364052503778225) | > loss_0: 2.6609208583831787 (2.6457644052672804) | > grad_norm_0: tensor(44.7402, device='cuda:0') (tensor(190.4630, device='cuda:0')) | > loss_gen: 2.030945301055908 (2.042412642847028) | > loss_kl: 2.1993629932403564 (2.008322052788316) | > loss_feat: 3.414700984954834 (3.5288451972760653) | > loss_mel: 16.910511016845703 (16.784438049584107) | > loss_duration: 1.607496738433838 (1.5421929317608214) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.163015365600586 (25.90621115031995) | > grad_norm_1: tensor(8513.7305, device='cuda:0') (tensor(6191.9731, device='cuda:0')) | > current_lr_0: 0.00017847262868603844 | > current_lr_1: 0.00017847262868603844 | > step_time: 0.4728 (0.44412698243793686) | > loader_time: 0.0097 (0.007989381489000821) --> EVAL PERFORMANCE | > avg_loader_time: 0.008643465982356541 (-1.1840336759322626e-05) | > avg_step_time: 0.45427864706012566 (-0.0031902185628111535) | > avg_loss_disc: 2.646355833805783 (+0.031624673118053614) | > avg_loss_disc_real_0: 0.13572007832182967 (+0.008359193959286498) | > avg_loss_disc_real_1: 0.2177096771102556 (-0.0030792665103791528) | > avg_loss_disc_real_2: 0.23409313047435923 (+0.002250862583308172) | > avg_loss_disc_real_3: 0.23446213884252898 (-0.003341422114573722) | > avg_loss_disc_real_4: 0.2415241517231498 (-0.0017019482985348722) | > avg_loss_disc_real_5: 0.24559140709084523 (-0.00323964067747895) | > avg_loss_0: 2.646355833805783 (+0.031624673118053614) | > avg_grad_norm_0: tensor(189.4171, device='cuda:0') (tensor(-9.4504, device='cuda:0')) | > avg_loss_gen: 2.0292064723834193 (-0.09969776616969517) | > avg_loss_kl: 2.0263454326441592 (+0.01764292952040547) | > avg_loss_feat: 3.4446892805502447 (-0.17099797221976276) | > avg_loss_mel: 16.747299221200006 (-0.3624889481235485) | > avg_loss_duration: 1.5562023444914481 (-0.0015826141330559373) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.80374287887358 (-0.6171242619904014) | > avg_grad_norm_1: tensor(6471.8135, device='cuda:0') (tensor(-124.0225, device='cuda:0'))  > EPOCH: 912/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:36:44)   --> TIME: 2025-02-25 17:37:02 -- STEP: 35/72 -- GLOBAL_STEP: 1065700 | > loss_disc: 2.806560516357422 (2.6512242385319307) | > loss_disc_real_0: 0.07804109156131744 (0.13115456093634878) | > loss_disc_real_1: 0.39868083596229553 (0.2357714465686253) | > loss_disc_real_2: 0.2844657301902771 (0.260604465007782) | > loss_disc_real_3: 0.22477251291275024 (0.23318923158305033) | > loss_disc_real_4: 0.26844677329063416 (0.2510501572063991) | > loss_disc_real_5: 0.2741600275039673 (0.24759920622621265) | > loss_0: 2.806560516357422 (2.6512242385319307) | > grad_norm_0: tensor(141.2104, device='cuda:0') (tensor(232.5590, device='cuda:0')) | > loss_gen: 2.2169010639190674 (2.147274085453578) | > loss_kl: 2.2430026531219482 (2.0143975564411707) | > loss_feat: 3.072885751724243 (3.6461734567369732) | > loss_mel: 17.34750747680664 (16.958106068202426) | > loss_duration: 1.5650603771209717 (1.5208484377179827) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.445356369018555 (26.286799349103656) | > grad_norm_1: tensor(8473.4111, device='cuda:0') (tensor(6278.6958, device='cuda:0')) | > current_lr_0: 0.00017845031960745267 | > current_lr_1: 0.00017845031960745267 | > step_time: 0.4455 (0.456258807863508) | > loader_time: 0.0084 (0.007419613429478236) --> EVAL PERFORMANCE | > avg_loader_time: 0.00876363566224004 (+0.00012016967988349829) | > avg_step_time: 0.46839295642476686 (+0.014114309364641198) | > avg_loss_disc: 2.640464574518338 (-0.005891259287444939) | > avg_loss_disc_real_0: 0.12768330548325896 (-0.00803677283857071) | > avg_loss_disc_real_1: 0.2253310073849181 (+0.007621330274662497) | > avg_loss_disc_real_2: 0.2490168912729747 (+0.01492376079861546) | > avg_loss_disc_real_3: 0.23819042574351942 (+0.0037282869009904462) | > avg_loss_disc_real_4: 0.24166345386437968 (+0.00013930214122986895) | > avg_loss_disc_real_5: 0.2477644942176174 (+0.0021730871267721663) | > avg_loss_0: 2.640464574518338 (-0.005891259287444939) | > avg_grad_norm_0: tensor(210.1601, device='cuda:0') (+tensor(20.7431, device='cuda:0')) | > avg_loss_gen: 2.1323994045526202 (+0.10319293216920089) | > avg_loss_kl: 2.3689840343636526 (+0.34263860171949334) | > avg_loss_feat: 3.703761896616976 (+0.25907261606673115) | > avg_loss_mel: 18.174480330776163 (+1.427181109576157) | > avg_loss_duration: 1.5853726460900106 (+0.029170301598562487) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.96499819151113 (+2.1612553126375467) | > avg_grad_norm_1: tensor(6349.4565, device='cuda:0') (tensor(-122.3569, device='cuda:0'))  > EPOCH: 913/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:37:22)   --> TIME: 2025-02-25 17:37:30 -- STEP: 13/72 -- GLOBAL_STEP: 1065750 | > loss_disc: 2.579960584640503 (2.6882981337033787) | > loss_disc_real_0: 0.09356901049613953 (0.14447884290264207) | > loss_disc_real_1: 0.2373744398355484 (0.2406840851673713) | > loss_disc_real_2: 0.2199546992778778 (0.24813728492993567) | > loss_disc_real_3: 0.1804737001657486 (0.23430647758337167) | > loss_disc_real_4: 0.21351447701454163 (0.2413900437263342) | > loss_disc_real_5: 0.24979646503925323 (0.2407600971368643) | > loss_0: 2.579960584640503 (2.6882981337033787) | > grad_norm_0: tensor(276.3701, device='cuda:0') (tensor(227.0243, device='cuda:0')) | > loss_gen: 2.085733413696289 (2.1590933983142557) | > loss_kl: 2.1081736087799072 (2.0203775625962477) | > loss_feat: 4.231402397155762 (3.725479639493502) | > loss_mel: 17.02748680114746 (17.511304268470177) | > loss_duration: 1.4789549112319946 (1.5299520859351525) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.93174934387207 (26.946206753070538) | > grad_norm_1: tensor(4101.4766, device='cuda:0') (tensor(4558.3242, device='cuda:0')) | > current_lr_0: 0.00017842801331750173 | > current_lr_1: 0.00017842801331750173 | > step_time: 0.4877 (0.442252727655264) | > loader_time: 0.006 (0.0071065425872802734)  --> TIME: 2025-02-25 17:37:53 -- STEP: 63/72 -- GLOBAL_STEP: 1065800 | > loss_disc: 2.7849674224853516 (2.7160820696089) | > loss_disc_real_0: 0.05747634917497635 (0.1543622180109932) | > loss_disc_real_1: 0.2312040477991104 (0.23011707384434957) | > loss_disc_real_2: 0.19683273136615753 (0.24524129856200447) | > loss_disc_real_3: 0.19755984842777252 (0.2354179448078549) | > loss_disc_real_4: 0.23108290135860443 (0.24727970148835862) | > loss_disc_real_5: 0.23420901596546173 (0.24669941952304233) | > loss_0: 2.7849674224853516 (2.7160820696089) | > grad_norm_0: tensor(284.0227, device='cuda:0') (tensor(241.7839, device='cuda:0')) | > loss_gen: 1.668434500694275 (2.0812043859845115) | > loss_kl: 2.127032995223999 (2.070679781928895) | > loss_feat: 3.110234498977661 (3.485804107454088) | > loss_mel: 17.20398712158203 (17.408523226541185) | > loss_duration: 1.5958497524261475 (1.5636100466289218) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.705537796020508 (26.609821410406205) | > grad_norm_1: tensor(7403.1543, device='cuda:0') (tensor(5057.0815, device='cuda:0')) | > current_lr_0: 0.00017842801331750173 | > current_lr_1: 0.00017842801331750173 | > step_time: 0.4862 (0.4516969067709787) | > loader_time: 0.0113 (0.008395698335435655) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874200337369677 (-2.1632288543269734e-05) | > avg_step_time: 0.4587539619123432 (-0.009638994512423671) | > avg_loss_disc: 2.720306997567835 (+0.07984242304949696) | > avg_loss_disc_real_0: 0.1539788948188365 (+0.026295589335577546) | > avg_loss_disc_real_1: 0.23042723347603436 (+0.005096226091116263) | > avg_loss_disc_real_2: 0.2433603095336699 (-0.005656581739304778) | > avg_loss_disc_real_3: 0.23526577945326416 (-0.0029246462902552672) | > avg_loss_disc_real_4: 0.24700238377275602 (+0.005338929908376339) | > avg_loss_disc_real_5: 0.2470642905419981 (-0.0007002036756192931) | > avg_loss_0: 2.720306997567835 (+0.07984242304949696) | > avg_grad_norm_0: tensor(243.8413, device='cuda:0') (+tensor(33.6811, device='cuda:0')) | > avg_loss_gen: 2.0604658781642646 (-0.0719335263883556) | > avg_loss_kl: 2.081068042298438 (-0.2879159920652148) | > avg_loss_feat: 3.4195717153414873 (-0.28419018127548856) | > avg_loss_mel: 17.397545371257085 (-0.7769349595190782) | > avg_loss_duration: 1.5709403467849947 (-0.014432299305015883) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.529591251427018 (-1.4354069400841105) | > avg_grad_norm_1: tensor(4944.8184, device='cuda:0') (tensor(-1404.6382, device='cuda:0'))  > EPOCH: 914/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:37:59)   --> TIME: 2025-02-25 17:38:20 -- STEP: 41/72 -- GLOBAL_STEP: 1065850 | > loss_disc: 2.6503446102142334 (2.6988042505776004) | > loss_disc_real_0: 0.1599050760269165 (0.19073410859195197) | > loss_disc_real_1: 0.11383038014173508 (0.24733441522935537) | > loss_disc_real_2: 0.22301658987998962 (0.2322457236487691) | > loss_disc_real_3: 0.24776294827461243 (0.23919572190540592) | > loss_disc_real_4: 0.23634372651576996 (0.24470586965723737) | > loss_disc_real_5: 0.2506946921348572 (0.24869010942738226) | > loss_0: 2.6503446102142334 (2.6988042505776004) | > grad_norm_0: tensor(148.9087, device='cuda:0') (tensor(185.9748, device='cuda:0')) | > loss_gen: 2.028923988342285 (2.0891636784483745) | > loss_kl: 2.032341718673706 (2.036760754701568) | > loss_feat: 3.489604949951172 (3.4271474349789504) | > loss_mel: 17.6579532623291 (17.542653107061614) | > loss_duration: 1.5667977333068848 (1.5425174875957213) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.77562141418457 (26.638242256350633) | > grad_norm_1: tensor(8380.7178, device='cuda:0') (tensor(6997.1768, device='cuda:0')) | > current_lr_0: 0.00017840570981583705 | > current_lr_1: 0.00017840570981583705 | > step_time: 0.4558 (0.4406715776862168) | > loader_time: 0.0093 (0.007426965527418183) --> EVAL PERFORMANCE | > avg_loader_time: 0.008634016547404545 (-0.00010798682629222477) | > avg_step_time: 0.45601265866991497 (-0.002741303242428217) | > avg_loss_disc: 2.6886952326331333 (-0.031611764934701636) | > avg_loss_disc_real_0: 0.17388123268602598 (+0.01990233786718948) | > avg_loss_disc_real_1: 0.2495918592936556 (+0.01916462581762124) | > avg_loss_disc_real_2: 0.23329145052063632 (-0.010068859013033588) | > avg_loss_disc_real_3: 0.23987645246613193 (+0.004610673012867772) | > avg_loss_disc_real_4: 0.2443669551694897 (-0.002635428603266321) | > avg_loss_disc_real_5: 0.24705293019052962 (-1.1360351468481245e-05) | > avg_loss_0: 2.6886952326331333 (-0.031611764934701636) | > avg_grad_norm_0: tensor(179.6873, device='cuda:0') (tensor(-64.1540, device='cuda:0')) | > avg_loss_gen: 2.1056993511361144 (+0.045233472971849764) | > avg_loss_kl: 2.0555675499875776 (-0.025500492310860157) | > avg_loss_feat: 3.417385873660235 (-0.002185841681252132) | > avg_loss_mel: 17.326299076348956 (-0.0712462949081285) | > avg_loss_duration: 1.568368626312471 (-0.002571720472523742) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.473320544605524 (-0.05627070682149338) | > avg_grad_norm_1: tensor(6130.2866, device='cuda:0') (+tensor(1185.4683, device='cuda:0'))  > EPOCH: 915/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:38:36)   --> TIME: 2025-02-25 17:38:47 -- STEP: 19/72 -- GLOBAL_STEP: 1065900 | > loss_disc: 2.6369142532348633 (2.614792748501426) | > loss_disc_real_0: 0.04776838794350624 (0.156532859527751) | > loss_disc_real_1: 0.20073707401752472 (0.22505540439957067) | > loss_disc_real_2: 0.3182670772075653 (0.24216020813113764) | > loss_disc_real_3: 0.26648029685020447 (0.24002339181147123) | > loss_disc_real_4: 0.28522902727127075 (0.25060324527715383) | > loss_disc_real_5: 0.25296711921691895 (0.24480285221024564) | > loss_0: 2.6369142532348633 (2.614792748501426) | > grad_norm_0: tensor(35.4205, device='cuda:0') (tensor(158.6649, device='cuda:0')) | > loss_gen: 2.046062469482422 (2.1249595190349377) | > loss_kl: 2.0542314052581787 (2.081088994678698) | > loss_feat: 3.3341856002807617 (3.5735802775935124) | > loss_mel: 16.317890167236328 (17.439511750873766) | > loss_duration: 1.5089032649993896 (1.5590627695384778) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.261272430419922 (26.77820356268632) | > grad_norm_1: tensor(4509.0449, device='cuda:0') (tensor(5106.2471, device='cuda:0')) | > current_lr_0: 0.00017838340910211005 | > current_lr_1: 0.00017838340910211005 | > step_time: 0.4471 (0.435872755552593) | > loader_time: 0.0061 (0.006701055325959858)  --> TIME: 2025-02-25 17:39:11 -- STEP: 69/72 -- GLOBAL_STEP: 1065950 | > loss_disc: 2.430569648742676 (2.635463293047919) | > loss_disc_real_0: 0.07325614988803864 (0.11487090976341911) | > loss_disc_real_1: 0.15533769130706787 (0.22669656013233075) | > loss_disc_real_2: 0.24095603823661804 (0.24508974949518839) | > loss_disc_real_3: 0.24449673295021057 (0.23325098208759143) | > loss_disc_real_4: 0.2344963103532791 (0.23529473033504208) | > loss_disc_real_5: 0.2264668494462967 (0.2478073880724285) | > loss_0: 2.430569648742676 (2.635463293047919) | > grad_norm_0: tensor(14.8708, device='cuda:0') (tensor(186.9781, device='cuda:0')) | > loss_gen: 2.4965617656707764 (2.1059040988700977) | > loss_kl: 2.279043197631836 (2.2849366457565963) | > loss_feat: 4.294973850250244 (3.7307180459948555) | > loss_mel: 17.0087947845459 (17.980814063030742) | > loss_duration: 1.6661875247955322 (1.5881040389987007) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.745559692382812 (27.690477205359418) | > grad_norm_1: tensor(3851.3943, device='cuda:0') (tensor(7839.1313, device='cuda:0')) | > current_lr_0: 0.00017838340910211005 | > current_lr_1: 0.00017838340910211005 | > step_time: 0.5394 (0.4528707179470339) | > loader_time: 0.0129 (0.008533809496008833) --> EVAL PERFORMANCE | > avg_loader_time: 0.0085908292045056 (-4.3187342898945436e-05) | > avg_step_time: 0.45465073786990745 (-0.0013619208000075211) | > avg_loss_disc: 2.637942055581321 (-0.05075317705181215) | > avg_loss_disc_real_0: 0.11433460227620433 (-0.05954663040982165) | > avg_loss_disc_real_1: 0.22858440687119122 (-0.02100745242246438) | > avg_loss_disc_real_2: 0.24441097361940733 (+0.011119523098771011) | > avg_loss_disc_real_3: 0.23363862176176528 (-0.006237830704366648) | > avg_loss_disc_real_4: 0.23586393242150966 (-0.008503022747980038) | > avg_loss_disc_real_5: 0.24787090949609247 (+0.0008179793055628493) | > avg_loss_0: 2.637942055581321 (-0.05075317705181215) | > avg_grad_norm_0: tensor(187.9911, device='cuda:0') (+tensor(8.3038, device='cuda:0')) | > avg_loss_gen: 2.0978044019618505 (-0.007894949174263921) | > avg_loss_kl: 2.284073853156935 (+0.2285063031693575) | > avg_loss_feat: 3.720818039397119 (+0.3034321657368837) | > avg_loss_mel: 17.954320450903666 (+0.6280213745547094) | > avg_loss_duration: 1.587853107653873 (+0.019484481341402127) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.644870167047205 (+1.171549622441681) | > avg_grad_norm_1: tensor(7948.0869, device='cuda:0') (+tensor(1817.8003, device='cuda:0'))  > EPOCH: 916/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:39:13)   --> TIME: 2025-02-25 17:39:37 -- STEP: 47/72 -- GLOBAL_STEP: 1066000 | > loss_disc: 2.38340163230896 (2.6316306286669793) | > loss_disc_real_0: 0.04288167506456375 (0.08892148273422364) | > loss_disc_real_1: 0.22282566130161285 (0.22126635972489703) | > loss_disc_real_2: 0.2682543396949768 (0.22904751116925098) | > loss_disc_real_3: 0.21816398203372955 (0.22485373406968218) | > loss_disc_real_4: 0.2372746467590332 (0.24350052501292938) | > loss_disc_real_5: 0.24760544300079346 (0.2443610150763329) | > loss_0: 2.38340163230896 (2.6316306286669793) | > grad_norm_0: tensor(134.8540, device='cuda:0') (tensor(358.0711, device='cuda:0')) | > loss_gen: 2.1663732528686523 (2.033385428976506) | > loss_kl: 2.0174059867858887 (2.0074778693787594) | > loss_feat: 3.965514659881592 (3.7125256213736026) | > loss_mel: 17.44545555114746 (16.8716262046327) | > loss_duration: 1.5885202884674072 (1.5317988852237134) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.183269500732422 (26.156813966467027) | > grad_norm_1: tensor(15492.9033, device='cuda:0') (tensor(10874.6055, device='cuda:0')) | > current_lr_0: 0.00017836111117597227 | > current_lr_1: 0.00017836111117597227 | > step_time: 0.4646 (0.4487634415322162) | > loader_time: 0.009 (0.007831472031613614) --> EVAL PERFORMANCE | > avg_loader_time: 0.008789630003378426 (+0.00019880079887282624) | > avg_step_time: 0.46032017385455926 (+0.005669435984651805) | > avg_loss_disc: 2.62039642602625 (-0.01754562955507133) | > avg_loss_disc_real_0: 0.09826136065620772 (-0.01607324161999661) | > avg_loss_disc_real_1: 0.2203339008798062 (-0.008250505991385021) | > avg_loss_disc_real_2: 0.22974438461619365 (-0.014666589003213687) | > avg_loss_disc_real_3: 0.22850353155337588 (-0.005135090208389398) | > avg_loss_disc_real_4: 0.24238764169350477 (+0.0065237092719951095) | > avg_loss_disc_real_5: 0.24544000037958924 (-0.0024309091165032326) | > avg_loss_0: 2.62039642602625 (-0.01754562955507133) | > avg_grad_norm_0: tensor(352.7363, device='cuda:0') (+tensor(164.7453, device='cuda:0')) | > avg_loss_gen: 2.0647872468115587 (-0.0330171551502918) | > avg_loss_kl: 2.038085462341845 (-0.24598839081509016) | > avg_loss_feat: 3.690547251365554 (-0.030270788031564688) | > avg_loss_mel: 16.82659416467372 (-1.1277262862299473) | > avg_loss_duration: 1.5535172576635656 (-0.03433584999030748) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.173531411399303 (-1.4713387556479027) | > avg_grad_norm_1: tensor(10840.8906, device='cuda:0') (+tensor(2892.8037, device='cuda:0'))  > EPOCH: 917/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:39:51)   --> TIME: 2025-02-25 17:40:04 -- STEP: 25/72 -- GLOBAL_STEP: 1066050 | > loss_disc: 2.6575706005096436 (2.576479263305664) | > loss_disc_real_0: 0.05038249492645264 (0.08438711971044541) | > loss_disc_real_1: 0.16246265172958374 (0.21637562066316604) | > loss_disc_real_2: 0.21281613409519196 (0.23058660328388214) | > loss_disc_real_3: 0.2307153344154358 (0.23340690076351167) | > loss_disc_real_4: 0.23982422053813934 (0.24517923712730408) | > loss_disc_real_5: 0.2410808801651001 (0.2416045618057251) | > loss_0: 2.6575706005096436 (2.576479263305664) | > grad_norm_0: tensor(448.1537, device='cuda:0') (tensor(272.5592, device='cuda:0')) | > loss_gen: 1.9922312498092651 (2.103882517814636) | > loss_kl: 2.2747905254364014 (2.0077741384506225) | > loss_feat: 4.1670989990234375 (3.911555709838867) | > loss_mel: 17.55985450744629 (17.006940994262695) | > loss_duration: 1.5206620693206787 (1.5111186790466309) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.514636993408203 (26.541272048950194) | > grad_norm_1: tensor(15328.5146, device='cuda:0') (tensor(8690.0840, device='cuda:0')) | > current_lr_0: 0.00017833881603707526 | > current_lr_1: 0.00017833881603707526 | > step_time: 0.4413 (0.43572412490844725) | > loader_time: 0.0073 (0.006830005645751953) --> EVAL PERFORMANCE | > avg_loader_time: 0.008621313202549035 (-0.0001683168008293906) | > avg_step_time: 0.4543995185637138 (-0.005920655290845456) | > avg_loss_disc: 2.5546469889896017 (-0.06574943703664804) | > avg_loss_disc_real_0: 0.08922836557030678 (-0.009032995085900941) | > avg_loss_disc_real_1: 0.21823169868177092 (-0.00210220219803528) | > avg_loss_disc_real_2: 0.23375894360139338 (+0.004014558985199729) | > avg_loss_disc_real_3: 0.23536768093915053 (+0.006864149385774648) | > avg_loss_disc_real_4: 0.24467938256935334 (+0.0022917408758485747) | > avg_loss_disc_real_5: 0.245306746850551 (-0.00013325352903822973) | > avg_loss_0: 2.5546469889896017 (-0.06574943703664804) | > avg_grad_norm_0: tensor(298.5723, device='cuda:0') (tensor(-54.1640, device='cuda:0')) | > avg_loss_gen: 2.1446668917024634 (+0.07987964489090471) | > avg_loss_kl: 2.033059531534221 (-0.005025930807624146) | > avg_loss_feat: 3.815437226228311 (+0.12488997486275677) | > avg_loss_mel: 16.858145378005336 (+0.03155121333161759) | > avg_loss_duration: 1.553252379659196 (-0.0002648780043696952) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.40456143231459 (+0.2310300209152878) | > avg_grad_norm_1: tensor(10314.6367, device='cuda:0') (tensor(-526.2539, device='cuda:0'))  > EPOCH: 918/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:40:28)   --> TIME: 2025-02-25 17:40:31 -- STEP: 3/72 -- GLOBAL_STEP: 1066100 | > loss_disc: 2.4444167613983154 (2.491530179977417) | > loss_disc_real_0: 0.11427733302116394 (0.08591995388269424) | > loss_disc_real_1: 0.22321435809135437 (0.21966193616390228) | > loss_disc_real_2: 0.2308233380317688 (0.22606894373893738) | > loss_disc_real_3: 0.22323817014694214 (0.21319339672724405) | > loss_disc_real_4: 0.24062199890613556 (0.24000792702039084) | > loss_disc_real_5: 0.23653189837932587 (0.23475801944732666) | > loss_0: 2.4444167613983154 (2.491530179977417) | > grad_norm_0: tensor(400.5247, device='cuda:0') (tensor(314.1137, device='cuda:0')) | > loss_gen: 2.3294029235839844 (2.213176647822062) | > loss_kl: 1.8877849578857422 (1.9027151266733806) | > loss_feat: 3.836293935775757 (3.9429845809936523) | > loss_mel: 16.43999481201172 (17.227092742919922) | > loss_duration: 1.5180068016052246 (1.508913516998291) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.011484146118164 (26.794883092244465) | > grad_norm_1: tensor(7970.0645, device='cuda:0') (tensor(9951.7637, device='cuda:0')) | > current_lr_0: 0.00017831652368507063 | > current_lr_1: 0.00017831652368507063 | > step_time: 0.4199 (0.42595648765563965) | > loader_time: 0.0058 (0.005901018778483073)  --> TIME: 2025-02-25 17:40:54 -- STEP: 53/72 -- GLOBAL_STEP: 1066150 | > loss_disc: 2.6707546710968018 (2.579729916914454) | > loss_disc_real_0: 0.10015721619129181 (0.09916421658587905) | > loss_disc_real_1: 0.21133077144622803 (0.21860261992463526) | > loss_disc_real_2: 0.24251022934913635 (0.230451301021396) | > loss_disc_real_3: 0.2427249401807785 (0.23546543211307167) | > loss_disc_real_4: 0.22542649507522583 (0.24488754879753544) | > loss_disc_real_5: 0.22989745438098907 (0.24941787179910913) | > loss_0: 2.6707546710968018 (2.579729916914454) | > grad_norm_0: tensor(434.2085, device='cuda:0') (tensor(316.9391, device='cuda:0')) | > loss_gen: 1.7420196533203125 (2.1369838849553515) | > loss_kl: 1.9501155614852905 (1.9822238683700562) | > loss_feat: 3.2813472747802734 (3.75884761900272) | > loss_mel: 15.641864776611328 (16.739991799840386) | > loss_duration: 1.5967450141906738 (1.5322090882175374) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.21209144592285 (26.150256246890663) | > grad_norm_1: tensor(10574.2061, device='cuda:0') (tensor(10524.0146, device='cuda:0')) | > current_lr_0: 0.00017831652368507063 | > current_lr_1: 0.00017831652368507063 | > step_time: 0.4529 (0.4433820697496522) | > loader_time: 0.0093 (0.007937237901507684) --> EVAL PERFORMANCE | > avg_loader_time: 0.008691995916232254 (+7.068271368321916e-05) | > avg_step_time: 0.4616889416331976 (+0.007289423069483825) | > avg_loss_disc: 2.584002817180795 (+0.0293558281911932) | > avg_loss_disc_real_0: 0.09981588258499831 (+0.01058751701469153) | > avg_loss_disc_real_1: 0.21910442810663036 (+0.0008727294248594419) | > avg_loss_disc_real_2: 0.2319669251290845 (-0.001792018472308865) | > avg_loss_disc_real_3: 0.2353407881209548 (-2.689281819573397e-05) | > avg_loss_disc_real_4: 0.24641685922380904 (+0.0017374766544556997) | > avg_loss_disc_real_5: 0.24864472889564407 (+0.0033379820450930575) | > avg_loss_0: 2.584002817180795 (+0.0293558281911932) | > avg_grad_norm_0: tensor(310.9692, device='cuda:0') (+tensor(12.3969, device='cuda:0')) | > avg_loss_gen: 2.1288129830024602 (-0.01585390870000314) | > avg_loss_kl: 2.0163114557803516 (-0.016748075753869163) | > avg_loss_feat: 3.679874692164676 (-0.13556253406363483) | > avg_loss_mel: 16.62997726655342 (-0.2281681114519145) | > avg_loss_duration: 1.5482555980413732 (-0.004996781617822688) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.00323193509814 (-0.4013294972164516) | > avg_grad_norm_1: tensor(10535.0264, device='cuda:0') (+tensor(220.3896, device='cuda:0'))  > EPOCH: 919/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:41:05)   --> TIME: 2025-02-25 17:41:21 -- STEP: 31/72 -- GLOBAL_STEP: 1066200 | > loss_disc: 3.014735221862793 (2.578427283994613) | > loss_disc_real_0: 0.1746279001235962 (0.09471014033882849) | > loss_disc_real_1: 0.1872992068529129 (0.23005847296407145) | > loss_disc_real_2: 0.3935694396495819 (0.23979152450638433) | > loss_disc_real_3: 0.2314247190952301 (0.23972513358439168) | > loss_disc_real_4: 0.2684744596481323 (0.23894548464205959) | > loss_disc_real_5: 0.23626448214054108 (0.24361839121387852) | > loss_0: 3.014735221862793 (2.578427283994613) | > grad_norm_0: tensor(679.1242, device='cuda:0') (tensor(347.0457, device='cuda:0')) | > loss_gen: 1.949622631072998 (2.1679091722734514) | > loss_kl: 2.270473003387451 (1.9401379631411644) | > loss_feat: 3.6662325859069824 (3.8699597158739643) | > loss_mel: 17.725116729736328 (16.727306058329923) | > loss_duration: 1.5632331371307373 (1.5035682839732016) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.174678802490234 (26.208880762900076) | > grad_norm_1: tensor(16320.3711, device='cuda:0') (tensor(10610.1719, device='cuda:0')) | > current_lr_0: 0.00017829423411961 | > current_lr_1: 0.00017829423411961 | > step_time: 0.4405 (0.4368767584523847) | > loader_time: 0.0084 (0.006997877551663307) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853681228530239 (-0.00015518363092986362) | > avg_step_time: 0.4559364218107412 (-0.005752519822456448) | > avg_loss_disc: 2.5795447792805417 (-0.004458037900253231) | > avg_loss_disc_real_0: 0.09063190756969049 (-0.009183975015307821) | > avg_loss_disc_real_1: 0.23967036311055578 (+0.020565935003925423) | > avg_loss_disc_real_2: 0.23209092902465606 (+0.000124003895571545) | > avg_loss_disc_real_3: 0.233727419250448 (-0.0016133688705068094) | > avg_loss_disc_real_4: 0.23329445614781177 (-0.013122403075997269) | > avg_loss_disc_real_5: 0.24220547080039978 (-0.006439258095244288) | > avg_loss_0: 2.5795447792805417 (-0.004458037900253231) | > avg_grad_norm_0: tensor(325.5209, device='cuda:0') (+tensor(14.5517, device='cuda:0')) | > avg_loss_gen: 2.1535171690121504 (+0.024704186009690154) | > avg_loss_kl: 2.0432246856286493 (+0.026913229848297693) | > avg_loss_feat: 3.7827720004068293 (+0.10289730824215315) | > avg_loss_mel: 16.664756828630473 (+0.034779562077051906) | > avg_loss_duration: 1.5489190800089234 (+0.0006634819675501724) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.193189701563874 (+0.18995776646573503) | > avg_grad_norm_1: tensor(10685.4297, device='cuda:0') (+tensor(150.4033, device='cuda:0'))  > EPOCH: 920/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:41:42)   --> TIME: 2025-02-25 17:41:48 -- STEP: 9/72 -- GLOBAL_STEP: 1066250 | > loss_disc: 2.426891326904297 (2.525911728541056) | > loss_disc_real_0: 0.060087546706199646 (0.08336570610602696) | > loss_disc_real_1: 0.2147354632616043 (0.21592746178309122) | > loss_disc_real_2: 0.20051772892475128 (0.21626655591858757) | > loss_disc_real_3: 0.22524897754192352 (0.22606546348995632) | > loss_disc_real_4: 0.21796755492687225 (0.23401585221290588) | > loss_disc_real_5: 0.22280465066432953 (0.2306701143582662) | > loss_0: 2.426891326904297 (2.525911728541056) | > grad_norm_0: tensor(115.3422, device='cuda:0') (tensor(294.2423, device='cuda:0')) | > loss_gen: 2.361208915710449 (2.137948658731249) | > loss_kl: 2.0314435958862305 (1.8508946763144598) | > loss_feat: 4.7142014503479 (3.8721900251176624) | > loss_mel: 17.413801193237305 (16.419037924872505) | > loss_duration: 1.509476900100708 (1.5093649095959134) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.030134201049805 (25.789436128404404) | > grad_norm_1: tensor(7394.7632, device='cuda:0') (tensor(10188.0273, device='cuda:0')) | > current_lr_0: 0.00017827194734034503 | > current_lr_1: 0.00017827194734034503 | > step_time: 0.4382 (0.43379871050516766) | > loader_time: 0.0057 (0.0065818097856309675)  --> TIME: 2025-02-25 17:42:11 -- STEP: 59/72 -- GLOBAL_STEP: 1066300 | > loss_disc: 2.4709291458129883 (2.5448541560415485) | > loss_disc_real_0: 0.08526373654603958 (0.09067589164537898) | > loss_disc_real_1: 0.1898033171892166 (0.21894720979666304) | > loss_disc_real_2: 0.21777421236038208 (0.22838476628570234) | > loss_disc_real_3: 0.22352714836597443 (0.23009983993182748) | > loss_disc_real_4: 0.23844127357006073 (0.2395951209937112) | > loss_disc_real_5: 0.23411662876605988 (0.24554496094331904) | > loss_0: 2.4709291458129883 (2.5448541560415485) | > grad_norm_0: tensor(240.1911, device='cuda:0') (tensor(304.9753, device='cuda:0')) | > loss_gen: 2.1299116611480713 (2.177633503736076) | > loss_kl: 2.0705032348632812 (2.0118296287827575) | > loss_feat: 4.142312049865723 (3.864048513315492) | > loss_mel: 15.867378234863281 (16.510199320518375) | > loss_duration: 1.593292236328125 (1.5387941659507103) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.803396224975586 (26.10250495652021) | > grad_norm_1: tensor(12981.6670, device='cuda:0') (tensor(10606.7920, device='cuda:0')) | > current_lr_0: 0.00017827194734034503 | > current_lr_1: 0.00017827194734034503 | > step_time: 0.4736 (0.4459542904869985) | > loader_time: 0.0101 (0.008121159117100601) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679329509466467 (+0.00014251722416407618) | > avg_step_time: 0.45646451896345114 (+0.0005280971527099609) | > avg_loss_disc: 2.5461275308904505 (-0.03341724839009119) | > avg_loss_disc_real_0: 0.09025517751423406 (-0.0003767300554564229) | > avg_loss_disc_real_1: 0.2201513925068815 (-0.019518970603674274) | > avg_loss_disc_real_2: 0.22821045467551326 (-0.003880474349142793) | > avg_loss_disc_real_3: 0.23070725022067487 (-0.0030201690297731176) | > avg_loss_disc_real_4: 0.24076229186964707 (+0.007467835721835292) | > avg_loss_disc_real_5: 0.2460709163840388 (+0.003865445583639021) | > avg_loss_0: 2.5461275308904505 (-0.03341724839009119) | > avg_grad_norm_0: tensor(304.4187, device='cuda:0') (tensor(-21.1022, device='cuda:0')) | > avg_loss_gen: 2.1798789803410923 (+0.026361811328941887) | > avg_loss_kl: 2.03123485706222 (-0.011989828566429495) | > avg_loss_feat: 3.843638792843886 (+0.06086679243705673) | > avg_loss_mel: 16.583553851490294 (-0.08120297714017966) | > avg_loss_duration: 1.5498107587787462 (+0.000891678769822768) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.18811712130694 (-0.0050725802569324685) | > avg_grad_norm_1: tensor(10732.1406, device='cuda:0') (+tensor(46.7109, device='cuda:0'))  > EPOCH: 921/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:42:19)   --> TIME: 2025-02-25 17:42:38 -- STEP: 37/72 -- GLOBAL_STEP: 1066350 | > loss_disc: 2.6282007694244385 (2.5439331595962114) | > loss_disc_real_0: 0.09115266799926758 (0.0906442609187719) | > loss_disc_real_1: 0.22939634323120117 (0.22700175884607676) | > loss_disc_real_2: 0.2366483062505722 (0.23050306133321813) | > loss_disc_real_3: 0.24770133197307587 (0.23185056488256198) | > loss_disc_real_4: 0.2583213448524475 (0.2378035409225) | > loss_disc_real_5: 0.24855345487594604 (0.24345235526561737) | > loss_0: 2.6282007694244385 (2.5439331595962114) | > grad_norm_0: tensor(273.6606, device='cuda:0') (tensor(321.4625, device='cuda:0')) | > loss_gen: 2.1948769092559814 (2.216257813814524) | > loss_kl: 1.8848724365234375 (2.000627971984245) | > loss_feat: 3.619917392730713 (3.9809337757729195) | > loss_mel: 16.40093421936035 (16.81927861394109) | > loss_duration: 1.5382975339889526 (1.5106388555990684) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.638898849487305 (26.527737127768027) | > grad_norm_1: tensor(11619.2988, device='cuda:0') (tensor(11293.2969, device='cuda:0')) | > current_lr_0: 0.00017824966334692747 | > current_lr_1: 0.00017824966334692747 | > step_time: 0.4478 (0.43626846493901433) | > loader_time: 0.0084 (0.0073151008502857105) --> EVAL PERFORMANCE | > avg_loader_time: 0.008662294334089255 (-1.703517537721133e-05) | > avg_step_time: 0.4558606416406766 (-0.0006038773227745597) | > avg_loss_disc: 2.544637680053711 (-0.0014898508367395813) | > avg_loss_disc_real_0: 0.09322565012204816 (+0.0029704726078140925) | > avg_loss_disc_real_1: 0.2237436548924782 (+0.0035922623855966784) | > avg_loss_disc_real_2: 0.23135868512408833 (+0.003148230448575068) | > avg_loss_disc_real_3: 0.23249617554772067 (+0.0017889253270458039) | > avg_loss_disc_real_4: 0.24025774421826215 (-0.0005045476513849123) | > avg_loss_disc_real_5: 0.24444306295522503 (-0.0016278534288137747) | > avg_loss_0: 2.544637680053711 (-0.0014898508367395813) | > avg_grad_norm_0: tensor(301.8207, device='cuda:0') (tensor(-2.5980, device='cuda:0')) | > avg_loss_gen: 2.2117511655243343 (+0.031872185183241974) | > avg_loss_kl: 2.062001036926054 (+0.030766179863834342) | > avg_loss_feat: 3.945410708306541 (+0.10177191546265485) | > avg_loss_mel: 16.75891189843837 (+0.17535804694807666) | > avg_loss_duration: 1.5468499005680354 (-0.002960858210710793) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.52492480210855 (+0.33680768080160917) | > avg_grad_norm_1: tensor(10617.3867, device='cuda:0') (tensor(-114.7539, device='cuda:0'))  > EPOCH: 922/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:42:56)   --> TIME: 2025-02-25 17:43:05 -- STEP: 15/72 -- GLOBAL_STEP: 1066400 | > loss_disc: 2.4971935749053955 (2.5388691425323486) | > loss_disc_real_0: 0.0828731507062912 (0.08517936393618583) | > loss_disc_real_1: 0.1987435668706894 (0.23060324390729267) | > loss_disc_real_2: 0.22399701178073883 (0.2279259701569875) | > loss_disc_real_3: 0.21659137308597565 (0.23086223701635997) | > loss_disc_real_4: 0.19979840517044067 (0.2332414964834849) | > loss_disc_real_5: 0.21766221523284912 (0.24013595283031464) | > loss_0: 2.4971935749053955 (2.5388691425323486) | > grad_norm_0: tensor(180.9480, device='cuda:0') (tensor(258.1371, device='cuda:0')) | > loss_gen: 2.2460570335388184 (2.241355721155803) | > loss_kl: 1.9708659648895264 (1.9754343032836914) | > loss_feat: 4.5740580558776855 (4.034285704294841) | > loss_mel: 17.436878204345703 (17.010407638549808) | > loss_duration: 1.4789718389511108 (1.4936901648839316) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.706830978393555 (26.755173619588216) | > grad_norm_1: tensor(7749.3882, device='cuda:0') (tensor(9948.5615, device='cuda:0')) | > current_lr_0: 0.0001782273821390091 | > current_lr_1: 0.0001782273821390091 | > step_time: 0.4372 (0.4403764088948568) | > loader_time: 0.006 (0.007166512807210286)  --> TIME: 2025-02-25 17:43:29 -- STEP: 65/72 -- GLOBAL_STEP: 1066450 | > loss_disc: 2.66312837600708 (2.6562669057112474) | > loss_disc_real_0: 0.31988853216171265 (0.14674838964755718) | > loss_disc_real_1: 0.21873384714126587 (0.24886115812338316) | > loss_disc_real_2: 0.25232958793640137 (0.2328078065927212) | > loss_disc_real_3: 0.21584783494472504 (0.23415496143010947) | > loss_disc_real_4: 0.23271073400974274 (0.2418176382780075) | > loss_disc_real_5: 0.24443253874778748 (0.2457562914261451) | > loss_0: 2.66312837600708 (2.6562669057112474) | > grad_norm_0: tensor(528.2050, device='cuda:0') (tensor(291.3258, device='cuda:0')) | > loss_gen: 2.265505790710449 (2.1533931988936215) | > loss_kl: 2.3560452461242676 (2.152974981528061) | > loss_feat: 3.506716012954712 (3.6673638307131253) | > loss_mel: 17.729984283447266 (17.202837518545298) | > loss_duration: 1.6303123235702515 (1.5518224110970131) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.48856544494629 (26.72839211684007) | > grad_norm_1: tensor(13631.6328, device='cuda:0') (tensor(8756.8076, device='cuda:0')) | > current_lr_0: 0.0001782273821390091 | > current_lr_1: 0.0001782273821390091 | > step_time: 0.4944 (0.4524232240823599) | > loader_time: 0.0121 (0.00856425211979793) --> EVAL PERFORMANCE | > avg_loader_time: 0.008813129344456636 (+0.00015083501036738077) | > avg_step_time: 0.4578646438222536 (+0.002004002181577036) | > avg_loss_disc: 2.668180831721131 (+0.12354315166742014) | > avg_loss_disc_real_0: 0.149345194474912 (+0.05611954435286386) | > avg_loss_disc_real_1: 0.24860580769223226 (+0.024862152799754067) | > avg_loss_disc_real_2: 0.23262389780769885 (+0.001265212683610517) | > avg_loss_disc_real_3: 0.23565141562844666 (+0.003155240080725985) | > avg_loss_disc_real_4: 0.24376643207711232 (+0.003508687858850168) | > avg_loss_disc_real_5: 0.24557269039288374 (+0.001129627437658709) | > avg_loss_0: 2.668180831721131 (+0.12354315166742014) | > avg_grad_norm_0: tensor(296.5331, device='cuda:0') (tensor(-5.2875, device='cuda:0')) | > avg_loss_gen: 2.1341281054725116 (-0.07762306005182262) | > avg_loss_kl: 2.1677765023540436 (+0.10577546542798943) | > avg_loss_feat: 3.621467539961909 (-0.3239431683446319) | > avg_loss_mel: 17.17236522889473 (+0.4134533304563597) | > avg_loss_duration: 1.5576271439941836 (+0.010777243426148164) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.653364745663925 (+0.12843994355537447) | > avg_grad_norm_1: tensor(8815.4395, device='cuda:0') (tensor(-1801.9473, device='cuda:0'))  > EPOCH: 923/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:43:34)   --> TIME: 2025-02-25 17:43:55 -- STEP: 43/72 -- GLOBAL_STEP: 1066500 | > loss_disc: 3.1439836025238037 (2.66981250740761) | > loss_disc_real_0: 0.2982899844646454 (0.14065020990579624) | > loss_disc_real_1: 0.2489054650068283 (0.21430561951426572) | > loss_disc_real_2: 0.22096283733844757 (0.22010421960852866) | > loss_disc_real_3: 0.24484166502952576 (0.23734388344509658) | > loss_disc_real_4: 0.22849325835704803 (0.23963770720847818) | > loss_disc_real_5: 0.24008573591709137 (0.23979940248090167) | > loss_0: 3.1439836025238037 (2.66981250740761) | > grad_norm_0: tensor(806.0734, device='cuda:0') (tensor(325.9619, device='cuda:0')) | > loss_gen: 1.7109973430633545 (2.0473796323288314) | > loss_kl: 2.169023036956787 (2.087721608405889) | > loss_feat: 3.3423006534576416 (3.7236673665601154) | > loss_mel: 16.398311614990234 (17.06803346234699) | > loss_duration: 1.5621099472045898 (1.5315524866414625) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.182743072509766 (26.458354506381724) | > grad_norm_1: tensor(153.9749, device='cuda:0') (tensor(8757.9395, device='cuda:0')) | > current_lr_0: 0.0001782051037162417 | > current_lr_1: 0.0001782051037162417 | > step_time: 0.4489 (0.44524509407753166) | > loader_time: 0.0094 (0.007517260174418605) --> EVAL PERFORMANCE | > avg_loader_time: 0.008636739892019352 (-0.00017638945243728457) | > avg_step_time: 0.4585692882537842 (+0.0007046444315305656) | > avg_loss_disc: 2.6340769781193267 (-0.034103853601804346) | > avg_loss_disc_real_0: 0.1293913226732066 (-0.01995387180170541) | > avg_loss_disc_real_1: 0.21786895719632296 (-0.030736850495909296) | > avg_loss_disc_real_2: 0.2200310123218617 (-0.012592885485837146) | > avg_loss_disc_real_3: 0.233151934516262 (-0.0024994811121846483) | > avg_loss_disc_real_4: 0.23727416509473828 (-0.00649226698237404) | > avg_loss_disc_real_5: 0.23750484430454147 (-0.008067846088342268) | > avg_loss_0: 2.6340769781193267 (-0.034103853601804346) | > avg_grad_norm_0: tensor(297.6573, device='cuda:0') (+tensor(1.1242, device='cuda:0')) | > avg_loss_gen: 2.093797834826188 (-0.04033027064632355) | > avg_loss_kl: 2.4517900104254062 (+0.2840135080713626) | > avg_loss_feat: 3.820418911920467 (+0.19895137195855783) | > avg_loss_mel: 18.138307463954867 (+0.9659422350601368) | > avg_loss_duration: 1.584384881274801 (+0.026757737280617366) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.088699045315593 (+1.435334299651668) | > avg_grad_norm_1: tensor(8890.8027, device='cuda:0') (+tensor(75.3633, device='cuda:0'))  > EPOCH: 924/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:44:11)   --> TIME: 2025-02-25 17:44:22 -- STEP: 21/72 -- GLOBAL_STEP: 1066550 | > loss_disc: 2.7360172271728516 (2.600410086768014) | > loss_disc_real_0: 0.17982295155525208 (0.11398537989173617) | > loss_disc_real_1: 0.222046360373497 (0.24523611295790898) | > loss_disc_real_2: 0.21683144569396973 (0.2302728976522173) | > loss_disc_real_3: 0.21677513420581818 (0.23516052251770383) | > loss_disc_real_4: 0.22704583406448364 (0.24041423769224257) | > loss_disc_real_5: 0.22316484153270721 (0.23133178835823423) | > loss_0: 2.7360172271728516 (2.600410086768014) | > grad_norm_0: tensor(697.4367, device='cuda:0') (tensor(280.2244, device='cuda:0')) | > loss_gen: 1.865539312362671 (2.108018744559515) | > loss_kl: 1.995675802230835 (2.1480107648032045) | > loss_feat: 3.490281343460083 (3.5775529884156727) | > loss_mel: 16.37908172607422 (17.582702909197128) | > loss_duration: 1.5473792552947998 (1.5285606384277344) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.277957916259766 (26.94484583536784) | > grad_norm_1: tensor(15176.5977, device='cuda:0') (tensor(11120.1270, device='cuda:0')) | > current_lr_0: 0.00017818282807827715 | > current_lr_1: 0.00017818282807827715 | > step_time: 0.4359 (0.4379090808686756) | > loader_time: 0.0062 (0.006781646183558873)  --> TIME: 2025-02-25 17:44:46 -- STEP: 71/72 -- GLOBAL_STEP: 1066600 | > loss_disc: 2.64188289642334 (2.618225356222877) | > loss_disc_real_0: 0.039191197603940964 (0.1187644822286888) | > loss_disc_real_1: 0.22733986377716064 (0.2360513682516528) | > loss_disc_real_2: 0.23877005279064178 (0.2365303161278577) | > loss_disc_real_3: 0.22045378386974335 (0.23717636604544143) | > loss_disc_real_4: 0.2389945685863495 (0.24151431695676187) | > loss_disc_real_5: 0.2375621199607849 (0.23932473025691342) | > loss_0: 2.64188289642334 (2.618225356222877) | > grad_norm_0: tensor(516.2306, device='cuda:0') (tensor(279.0572, device='cuda:0')) | > loss_gen: 1.84731125831604 (2.136224901172475) | > loss_kl: 2.1143460273742676 (2.3065106365042674) | > loss_feat: 3.7807323932647705 (3.666623548722603) | > loss_mel: 16.27198600769043 (17.96241417737075) | > loss_duration: 1.566258192062378 (1.584615537818049) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.58063316345215 (27.656388726032954) | > grad_norm_1: tensor(15418.9512, device='cuda:0') (tensor(9031.5303, device='cuda:0')) | > current_lr_0: 0.00017818282807827715 | > current_lr_1: 0.00017818282807827715 | > step_time: 0.4781 (0.4564626149728265) | > loader_time: 0.0081 (0.00861314652671277) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861314652671277 (-2.359336530658243e-05) | > avg_step_time: 0.4564626149728265 (-0.002106673280957705) | > avg_loss_disc: 2.618225356222877 (-0.015851621896449508) | > avg_loss_disc_real_0: 0.1187644822286888 (-0.01062684044451781) | > avg_loss_disc_real_1: 0.2360513682516528 (+0.018182411055329833) | > avg_loss_disc_real_2: 0.2365303161278577 (+0.016499303805995985) | > avg_loss_disc_real_3: 0.23717636604544143 (+0.0040244315291794175) | > avg_loss_disc_real_4: 0.24151431695676187 (+0.004240151862023589) | > avg_loss_disc_real_5: 0.23932473025691342 (+0.0018198859523719524) | > avg_loss_0: 2.618225356222877 (-0.015851621896449508) | > avg_grad_norm_0: tensor(279.0572, device='cuda:0') (tensor(-18.6002, device='cuda:0')) | > avg_loss_gen: 2.136224901172475 (+0.04242706634628712) | > avg_loss_kl: 2.3065106365042674 (-0.14527937392113888) | > avg_loss_feat: 3.666623548722603 (-0.1537953631978639) | > avg_loss_mel: 17.96241417737075 (-0.17589328658411674) | > avg_loss_duration: 1.584615537818049 (+0.00023065654324816443) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.656388726032954 (-0.43231031928263874) | > avg_grad_norm_1: tensor(9031.5303, device='cuda:0') (+tensor(140.7275, device='cuda:0'))  > EPOCH: 925/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:44:48)   --> TIME: 2025-02-25 17:45:12 -- STEP: 49/72 -- GLOBAL_STEP: 1066650 | > loss_disc: 2.726332664489746 (2.6855862724537753) | > loss_disc_real_0: 0.07566795498132706 (0.18329009360500745) | > loss_disc_real_1: 0.23015420138835907 (0.22546526394328292) | > loss_disc_real_2: 0.25771716237068176 (0.2350506101335798) | > loss_disc_real_3: 0.2290184497833252 (0.2285240319918613) | > loss_disc_real_4: 0.25548744201660156 (0.24461086763411152) | > loss_disc_real_5: 0.23001042008399963 (0.24712298962534693) | > loss_0: 2.726332664489746 (2.6855862724537753) | > grad_norm_0: tensor(219.1070, device='cuda:0') (tensor(280.0059, device='cuda:0')) | > loss_gen: 1.9109556674957275 (2.1269824991420823) | > loss_kl: 1.955142617225647 (2.001755874984118) | > loss_feat: 3.9565069675445557 (3.6142919112224967) | > loss_mel: 16.112634658813477 (17.0216456043477) | > loss_duration: 1.602370262145996 (1.5370240479099506) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.537609100341797 (26.301700163860712) | > grad_norm_1: tensor(6145.0210, device='cuda:0') (tensor(8591.8154, device='cuda:0')) | > current_lr_0: 0.00017816055522476736 | > current_lr_1: 0.00017816055522476736 | > step_time: 0.4415 (0.4423370799239801) | > loader_time: 0.0095 (0.0077310581596530216) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865508133256939 (+4.193480585662143e-05) | > avg_step_time: 0.45466713502373496 (-0.0017954799490915163) | > avg_loss_disc: 2.664780432069805 (+0.046555075846927974) | > avg_loss_disc_real_0: 0.1617435945696394 (+0.042979112340950615) | > avg_loss_disc_real_1: 0.22560927019992344 (-0.010442098051729354) | > avg_loss_disc_real_2: 0.23854329951212438 (+0.0020129833842666978) | > avg_loss_disc_real_3: 0.2303537711710997 (-0.006822594874341725) | > avg_loss_disc_real_4: 0.24467152608952053 (+0.0031572091327586593) | > avg_loss_disc_real_5: 0.24568644753644164 (+0.006361717279528223) | > avg_loss_0: 2.664780432069805 (+0.046555075846927974) | > avg_grad_norm_0: tensor(251.5731, device='cuda:0') (tensor(-27.4841, device='cuda:0')) | > avg_loss_gen: 2.114001554502567 (-0.022223346669908217) | > avg_loss_kl: 2.0273188889866134 (-0.27919174751765397) | > avg_loss_feat: 3.59553196732427 (-0.07109158139833305) | > avg_loss_mel: 17.002700563887483 (-0.9597136134832667) | > avg_loss_duration: 1.557940511636331 (-0.02667502618171813) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.29749357196647 (-1.3588951540664844) | > avg_grad_norm_1: tensor(8223.6572, device='cuda:0') (tensor(-807.8730, device='cuda:0'))  > EPOCH: 926/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:45:25)   --> TIME: 2025-02-25 17:45:39 -- STEP: 27/72 -- GLOBAL_STEP: 1066700 | > loss_disc: 2.6588122844696045 (2.6668430964152017) | > loss_disc_real_0: 0.0953591912984848 (0.12105200453488915) | > loss_disc_real_1: 0.18250559270381927 (0.25330998279430256) | > loss_disc_real_2: 0.2751772999763489 (0.24916071251586633) | > loss_disc_real_3: 0.24904434382915497 (0.222637716266844) | > loss_disc_real_4: 0.21682602167129517 (0.22477667971893592) | > loss_disc_real_5: 0.23212085664272308 (0.23888531768763507) | > loss_0: 2.6588122844696045 (2.6668430964152017) | > grad_norm_0: tensor(288.7967, device='cuda:0') (tensor(228.5501, device='cuda:0')) | > loss_gen: 1.9453414678573608 (2.1381467757401635) | > loss_kl: 2.388545036315918 (2.712643751391658) | > loss_feat: 3.584076404571533 (3.750228996630068) | > loss_mel: 18.8969783782959 (20.280208799574112) | > loss_duration: 1.6272199153900146 (1.6933787089807015) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.442161560058594 (30.57460700141059) | > grad_norm_1: tensor(8605.4307, device='cuda:0') (tensor(5556.8774, device='cuda:0')) | > current_lr_0: 0.00017813828515536424 | > current_lr_1: 0.00017813828515536424 | > step_time: 0.4457 (0.440797832277086) | > loader_time: 0.0091 (0.007167074415418837) --> EVAL PERFORMANCE | > avg_loader_time: 0.00873331620659627 (+7.823487402687859e-05) | > avg_step_time: 0.4584396389168753 (+0.003772503893140322) | > avg_loss_disc: 2.620789840187825 (-0.04399059188198029) | > avg_loss_disc_real_0: 0.09635492254205992 (-0.06538867202757949) | > avg_loss_disc_real_1: 0.2327641183641595 (+0.007154848164236061) | > avg_loss_disc_real_2: 0.24401650798152869 (+0.0054732084694043) | > avg_loss_disc_real_3: 0.2339953360423236 (+0.0036415648712238846) | > avg_loss_disc_real_4: 0.24388035502232297 (-0.0007911710671975603) | > avg_loss_disc_real_5: 0.245939229575681 (+0.0002527820392393687) | > avg_loss_0: 2.620789840187825 (-0.04399059188198029) | > avg_grad_norm_0: tensor(220.6213, device='cuda:0') (tensor(-30.9518, device='cuda:0')) | > avg_loss_gen: 2.129951883369768 (+0.015950328867201158) | > avg_loss_kl: 2.4136898685509043 (+0.3863709795642909) | > avg_loss_feat: 3.6936810117372323 (+0.09814904441296246) | > avg_loss_mel: 18.471074184901283 (+1.4683736210138) | > avg_loss_duration: 1.6501697184334339 (+0.09222920679710289) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.358566821460993 (+2.061073249494523) | > avg_grad_norm_1: tensor(6756.3311, device='cuda:0') (tensor(-1467.3262, device='cuda:0'))  > EPOCH: 927/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:46:02)   --> TIME: 2025-02-25 17:46:07 -- STEP: 5/72 -- GLOBAL_STEP: 1066750 | > loss_disc: 2.30129337310791 (2.5443777561187746) | > loss_disc_real_0: 0.05772775411605835 (0.07511948943138122) | > loss_disc_real_1: 0.25005555152893066 (0.23360932469367982) | > loss_disc_real_2: 0.19187654554843903 (0.20503692030906678) | > loss_disc_real_3: 0.19627732038497925 (0.21752708554267883) | > loss_disc_real_4: 0.24678680300712585 (0.2448916256427765) | > loss_disc_real_5: 0.2666420042514801 (0.24046873152256013) | > loss_0: 2.30129337310791 (2.5443777561187746) | > grad_norm_0: tensor(49.2361, device='cuda:0') (tensor(230.2891, device='cuda:0')) | > loss_gen: 2.675168991088867 (2.1200801849365236) | > loss_kl: 1.8585178852081299 (1.9011904001235962) | > loss_feat: 4.4747114181518555 (3.8488918781280517) | > loss_mel: 17.183156967163086 (17.03593215942383) | > loss_duration: 1.5702934265136719 (1.559274935722351) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.76184844970703 (26.46536979675293) | > grad_norm_1: tensor(5073.9761, device='cuda:0') (tensor(8776.8135, device='cuda:0')) | > current_lr_0: 0.0001781160178697198 | > current_lr_1: 0.0001781160178697198 | > step_time: 0.4273 (0.43213534355163574) | > loader_time: 0.0063 (0.007464551925659179)  --> TIME: 2025-02-25 17:46:30 -- STEP: 55/72 -- GLOBAL_STEP: 1066800 | > loss_disc: 2.6672239303588867 (2.5307935758070514) | > loss_disc_real_0: 0.0742051750421524 (0.08742209043015131) | > loss_disc_real_1: 0.2039768248796463 (0.21502289419824427) | > loss_disc_real_2: 0.22415217757225037 (0.22203433757478544) | > loss_disc_real_3: 0.25186994671821594 (0.22311226779764348) | > loss_disc_real_4: 0.2531210780143738 (0.23506719280372967) | > loss_disc_real_5: 0.25360700488090515 (0.24205977212299) | > loss_0: 2.6672239303588867 (2.5307935758070514) | > grad_norm_0: tensor(405.3351, device='cuda:0') (tensor(268.3522, device='cuda:0')) | > loss_gen: 1.8594043254852295 (2.1491619218479507) | > loss_kl: 2.2099108695983887 (1.9917828061363914) | > loss_feat: 3.6421382427215576 (3.8965675050562076) | > loss_mel: 17.517393112182617 (16.97731472362172) | > loss_duration: 1.6219642162322998 (1.557648253440857) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.85080909729004 (26.572475363991476) | > grad_norm_1: tensor(15481.9502, device='cuda:0') (tensor(9534.9697, device='cuda:0')) | > current_lr_0: 0.0001781160178697198 | > current_lr_1: 0.0001781160178697198 | > step_time: 0.462 (0.4445435870777477) | > loader_time: 0.0093 (0.008064699172973634) --> EVAL PERFORMANCE | > avg_loader_time: 0.008753444107485494 (+2.0127900889224323e-05) | > avg_step_time: 0.45369901119823186 (-0.00474062771864342) | > avg_loss_disc: 2.538506951130612 (-0.08228288905721293) | > avg_loss_disc_real_0: 0.089839688436666 (-0.0065152341053939244) | > avg_loss_disc_real_1: 0.21553682840206254 (-0.017227289962096964) | > avg_loss_disc_real_2: 0.22420868067674235 (-0.019807827304786335) | > avg_loss_disc_real_3: 0.2262632882511112 (-0.007732047791212393) | > avg_loss_disc_real_4: 0.23661177888722487 (-0.007268576135098098) | > avg_loss_disc_real_5: 0.24480273144345888 (-0.0011364981322221357) | > avg_loss_0: 2.538506951130612 (-0.08228288905721293) | > avg_grad_norm_0: tensor(274.2707, device='cuda:0') (+tensor(53.6494, device='cuda:0')) | > avg_loss_gen: 2.152512269960323 (+0.022560386590555037) | > avg_loss_kl: 2.0108315390600286 (-0.40285832949087563) | > avg_loss_feat: 3.845615004149961 (+0.15193399241272854) | > avg_loss_mel: 16.878232552971646 (-1.5928416319296375) | > avg_loss_duration: 1.5691805272035195 (-0.08098919122991433) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.45637203270281 (-1.9021947887581838) | > avg_grad_norm_1: tensor(9354.9580, device='cuda:0') (+tensor(2598.6270, device='cuda:0'))  > EPOCH: 928/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:46:39)   --> TIME: 2025-02-25 17:46:56 -- STEP: 33/72 -- GLOBAL_STEP: 1066850 | > loss_disc: 2.882633686065674 (2.672295447551843) | > loss_disc_real_0: 0.44888660311698914 (0.17386064407500354) | > loss_disc_real_1: 0.24714432656764984 (0.21512217232675263) | > loss_disc_real_2: 0.29568004608154297 (0.21601509776982394) | > loss_disc_real_3: 0.2199249416589737 (0.219249760111173) | > loss_disc_real_4: 0.20258201658725739 (0.2348587652950576) | > loss_disc_real_5: 0.25404804944992065 (0.24663936324191815) | > loss_0: 2.882633686065674 (2.672295447551843) | > grad_norm_0: tensor(236.9888, device='cuda:0') (tensor(301.7962, device='cuda:0')) | > loss_gen: 2.7452127933502197 (2.102386539632624) | > loss_kl: 2.306257724761963 (1.903472748669711) | > loss_feat: 3.9142465591430664 (3.716789671868989) | > loss_mel: 19.227407455444336 (16.841204065265075) | > loss_duration: 1.609635829925537 (1.5239580978046765) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.80276107788086 (26.087810863148082) | > grad_norm_1: tensor(4100.4902, device='cuda:0') (tensor(8437.7490, device='cuda:0')) | > current_lr_0: 0.0001780937533674861 | > current_lr_1: 0.0001780937533674861 | > step_time: 0.4414 (0.437719236720692) | > loader_time: 0.0078 (0.007318027091748787) --> EVAL PERFORMANCE | > avg_loader_time: 0.008770811725670184 (+1.7367618184689973e-05) | > avg_step_time: 0.45422102028215433 (+0.00052200908392247) | > avg_loss_disc: 2.7435028586589114 (+0.20499590752829944) | > avg_loss_disc_real_0: 0.17042546504190267 (+0.08058577660523668) | > avg_loss_disc_real_1: 0.24363188676431147 (+0.02809505836224893) | > avg_loss_disc_real_2: 0.25033466979651386 (+0.026125989119771514) | > avg_loss_disc_real_3: 0.23776705814919002 (+0.01150376989807883) | > avg_loss_disc_real_4: 0.23719676825362193 (+0.0005849893663970596) | > avg_loss_disc_real_5: 0.25041502966007717 (+0.005612298216618294) | > avg_loss_0: 2.7435028586589114 (+0.20499590752829944) | > avg_grad_norm_0: tensor(288.6942, device='cuda:0') (+tensor(14.4235, device='cuda:0')) | > avg_loss_gen: 2.1646885401766065 (+0.012176270216283314) | > avg_loss_kl: 2.0820001984985774 (+0.07116865943854878) | > avg_loss_feat: 3.6410010700494473 (-0.20461393410051354) | > avg_loss_mel: 17.28862511272161 (+0.41039255974996536) | > avg_loss_duration: 1.5684622727649316 (-0.000718254438587973) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.744777034705795 (+0.28840500200298536) | > avg_grad_norm_1: tensor(6747.6279, device='cuda:0') (tensor(-2607.3301, device='cuda:0'))  > EPOCH: 929/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:47:16)   --> TIME: 2025-02-25 17:47:23 -- STEP: 11/72 -- GLOBAL_STEP: 1066900 | > loss_disc: 2.5952141284942627 (2.626651417125355) | > loss_disc_real_0: 0.15892969071865082 (0.1873758164319125) | > loss_disc_real_1: 0.217790424823761 (0.22110739214853806) | > loss_disc_real_2: 0.2544897496700287 (0.23506845398382706) | > loss_disc_real_3: 0.23278965055942535 (0.288094237446785) | > loss_disc_real_4: 0.23132401704788208 (0.23781844973564148) | > loss_disc_real_5: 0.227659672498703 (0.2382638847286051) | > loss_0: 2.5952141284942627 (2.626651417125355) | > grad_norm_0: tensor(60.2844, device='cuda:0') (tensor(215.4727, device='cuda:0')) | > loss_gen: 2.09899640083313 (2.1422312693162398) | > loss_kl: 1.9564900398254395 (1.9391500191255049) | > loss_feat: 3.375682830810547 (3.355332136154175) | > loss_mel: 17.250173568725586 (17.369496605613016) | > loss_duration: 1.4934368133544922 (1.5195576602762395) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.174779891967773 (26.325767517089844) | > grad_norm_1: tensor(7244.5190, device='cuda:0') (tensor(5376.2329, device='cuda:0')) | > current_lr_0: 0.00017807149164831515 | > current_lr_1: 0.00017807149164831515 | > step_time: 0.4317 (0.4295892932198264) | > loader_time: 0.0057 (0.006659702821211381)  --> TIME: 2025-02-25 17:47:46 -- STEP: 61/72 -- GLOBAL_STEP: 1066950 | > loss_disc: 2.8595755100250244 (2.7133894123014857) | > loss_disc_real_0: 0.13344739377498627 (0.18199868549088963) | > loss_disc_real_1: 0.20465348660945892 (0.23579967363936002) | > loss_disc_real_2: 0.2296312004327774 (0.24225699217593083) | > loss_disc_real_3: 0.2312604784965515 (0.24413880192842644) | > loss_disc_real_4: 0.2388053685426712 (0.24990211622636832) | > loss_disc_real_5: 0.26204532384872437 (0.24859938489609076) | > loss_0: 2.8595755100250244 (2.7133894123014857) | > grad_norm_0: tensor(11.2423, device='cuda:0') (tensor(228.2152, device='cuda:0')) | > loss_gen: 2.097611904144287 (2.0768250559197106) | > loss_kl: 2.0356392860412598 (2.1718976439022617) | > loss_feat: 2.992107391357422 (3.3834990243442724) | > loss_mel: 16.978490829467773 (17.547440794647716) | > loss_duration: 1.5965080261230469 (1.5557040660107722) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.70035743713379 (26.735366633681) | > grad_norm_1: tensor(3175.3848, device='cuda:0') (tensor(5869.9717, device='cuda:0')) | > current_lr_0: 0.00017807149164831515 | > current_lr_1: 0.00017807149164831515 | > step_time: 0.4636 (0.4455840665785993) | > loader_time: 0.0107 (0.008251768643738795) --> EVAL PERFORMANCE | > avg_loader_time: 0.008721076266866334 (-4.973545880384948e-05) | > avg_step_time: 0.4547080657851528 (+0.00048704550299849547) | > avg_loss_disc: 2.7174925669817855 (-0.026010291677125874) | > avg_loss_disc_real_0: 0.1856704626074979 (+0.01524499756559522) | > avg_loss_disc_real_1: 0.24398008723494033 (+0.00034820047062886217) | > avg_loss_disc_real_2: 0.24496334140569392 (-0.0053713283908199405) | > avg_loss_disc_real_3: 0.2450553445329129 (+0.007288286383722864) | > avg_loss_disc_real_4: 0.25077742254230345 (+0.01358065428868152) | > avg_loss_disc_real_5: 0.2476266794641253 (-0.002788350195951861) | > avg_loss_0: 2.7174925669817855 (-0.026010291677125874) | > avg_grad_norm_0: tensor(213.9474, device='cuda:0') (tensor(-74.7468, device='cuda:0')) | > avg_loss_gen: 2.1083872452588155 (-0.05630129491779101) | > avg_loss_kl: 2.1674931099717036 (+0.08549291147312621) | > avg_loss_feat: 3.381939260052963 (-0.2590618099964841) | > avg_loss_mel: 17.479462019154724 (+0.19083690643311257) | > avg_loss_duration: 1.5641776138628032 (-0.004284658902128369) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.70145929363412 (-0.043317741071675187) | > avg_grad_norm_1: tensor(5385.0288, device='cuda:0') (tensor(-1362.5991, device='cuda:0'))  > EPOCH: 930/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:47:54)   --> TIME: 2025-02-25 17:48:13 -- STEP: 39/72 -- GLOBAL_STEP: 1067000 | > loss_disc: 2.7504687309265137 (2.642370419624524) | > loss_disc_real_0: 0.23161806166172028 (0.1427162230396882) | > loss_disc_real_1: 0.22412633895874023 (0.23676047684290472) | > loss_disc_real_2: 0.22672340273857117 (0.23290640536027077) | > loss_disc_real_3: 0.23098574578762054 (0.23438064906841669) | > loss_disc_real_4: 0.24350488185882568 (0.2371252859250093) | > loss_disc_real_5: 0.24118469655513763 (0.2416402403360758) | > loss_0: 2.7504687309265137 (2.642370419624524) | > grad_norm_0: tensor(79.1257, device='cuda:0') (tensor(176.1280, device='cuda:0')) | > loss_gen: 1.9602388143539429 (2.0534889361797233) | > loss_kl: 2.4053616523742676 (2.0445040861765547) | > loss_feat: 3.2822134494781494 (3.4531084390786977) | > loss_mel: 16.81796646118164 (17.12636431669577) | > loss_duration: 1.5862715244293213 (1.5245638657838871) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.052053451538086 (26.20202993735289) | > grad_norm_1: tensor(6709.5947, device='cuda:0') (tensor(5510.9673, device='cuda:0')) | > current_lr_0: 0.00017804923271185912 | > current_lr_1: 0.00017804923271185912 | > step_time: 0.4781 (0.44229838175651354) | > loader_time: 0.0085 (0.007375014133942433) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696898608140543 (-2.4177658725791154e-05) | > avg_step_time: 0.4568148431643634 (+0.002106777379210556) | > avg_loss_disc: 2.650261667412772 (-0.0672308995690134) | > avg_loss_disc_real_0: 0.16467176514192366 (-0.02099869746557423) | > avg_loss_disc_real_1: 0.22879731571170644 (-0.015182771523233884) | > avg_loss_disc_real_2: 0.23551947508059756 (-0.009443866325096362) | > avg_loss_disc_real_3: 0.23621781401231257 (-0.008837530520600323) | > avg_loss_disc_real_4: 0.2434457842732819 (-0.0073316382690215565) | > avg_loss_disc_real_5: 0.24532583943555053 (-0.00230084002857478) | > avg_loss_0: 2.650261667412772 (-0.0672308995690134) | > avg_grad_norm_0: tensor(203.7755, device='cuda:0') (tensor(-10.1719, device='cuda:0')) | > avg_loss_gen: 2.085763108562416 (-0.02262413669639951) | > avg_loss_kl: 2.0733951162284527 (-0.0940979937432509) | > avg_loss_feat: 3.4183428959107736 (+0.03640363585781037) | > avg_loss_mel: 17.01762329692571 (-0.46183872222901456) | > avg_loss_duration: 1.5587490948153213 (-0.005428519047481872) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.153873631651972 (-0.5475856619821471) | > avg_grad_norm_1: tensor(5415.3638, device='cuda:0') (+tensor(30.3350, device='cuda:0'))  > EPOCH: 931/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:48:31)   --> TIME: 2025-02-25 17:48:40 -- STEP: 17/72 -- GLOBAL_STEP: 1067050 | > loss_disc: 2.646054744720459 (2.581720871083877) | > loss_disc_real_0: 0.2622583210468292 (0.09973319685634445) | > loss_disc_real_1: 0.21166151762008667 (0.2183604179059758) | > loss_disc_real_2: 0.23591028153896332 (0.24225668258526745) | > loss_disc_real_3: 0.22435812652111053 (0.23154297558700337) | > loss_disc_real_4: 0.2231370210647583 (0.2321609740748125) | > loss_disc_real_5: 0.23031535744667053 (0.2341339044711169) | > loss_0: 2.646054744720459 (2.581720871083877) | > grad_norm_0: tensor(339.6742, device='cuda:0') (tensor(219.2886, device='cuda:0')) | > loss_gen: 2.106390953063965 (2.110442077412325) | > loss_kl: 2.7049062252044678 (2.0963289106593415) | > loss_feat: 3.3448305130004883 (3.728111084769754) | > loss_mel: 22.13177490234375 (17.520724240471335) | > loss_duration: 1.7386751174926758 (1.5343078234616447) | > amp_scaler: 8.0 (8.0) | > loss_1: 32.02657699584961 (26.989914164823645) | > grad_norm_1: tensor(10936.2246, device='cuda:0') (tensor(7977.8384, device='cuda:0')) | > current_lr_0: 0.00017802697655777013 | > current_lr_1: 0.00017802697655777013 | > step_time: 0.4311 (0.43338073001188393) | > loader_time: 0.0061 (0.006846904754638672)  --> TIME: 2025-02-25 17:49:04 -- STEP: 67/72 -- GLOBAL_STEP: 1067100 | > loss_disc: 2.616342782974243 (2.601690936444411) | > loss_disc_real_0: 0.07494896650314331 (0.11810775835122635) | > loss_disc_real_1: 0.22338902950286865 (0.2232335987153338) | > loss_disc_real_2: 0.2544380724430084 (0.24178719053517528) | > loss_disc_real_3: 0.25618430972099304 (0.23659459379181932) | > loss_disc_real_4: 0.2552058696746826 (0.2431483560088855) | > loss_disc_real_5: 0.26214396953582764 (0.24646714641087092) | > loss_0: 2.616342782974243 (2.601690936444411) | > grad_norm_0: tensor(231.7677, device='cuda:0') (tensor(194.2064, device='cuda:0')) | > loss_gen: 2.0884921550750732 (2.157897200157393) | > loss_kl: 2.134791135787964 (2.1368138896885203) | > loss_feat: 3.441689968109131 (3.6709915061495195) | > loss_mel: 17.457820892333984 (17.175644817636982) | > loss_duration: 1.640815258026123 (1.5673627906770848) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.763608932495117 (26.708710200751007) | > grad_norm_1: tensor(8495.5527, device='cuda:0') (tensor(6629.6411, device='cuda:0')) | > current_lr_0: 0.00017802697655777013 | > current_lr_1: 0.00017802697655777013 | > step_time: 0.5189 (0.4501909747052549) | > loader_time: 0.0121 (0.008397216227517199) --> EVAL PERFORMANCE | > avg_loader_time: 0.008576188288943869 (-0.00012071031919667387) | > avg_step_time: 0.4546229469944054 (-0.002191896169957963) | > avg_loss_disc: 2.602171461347124 (-0.04809020606564829) | > avg_loss_disc_real_0: 0.11526252990457374 (-0.04940923523734993) | > avg_loss_disc_real_1: 0.22338590330221283 (-0.00541141240949361) | > avg_loss_disc_real_2: 0.2412905858855852 (+0.005771110804987628) | > avg_loss_disc_real_3: 0.23669994045311296 (+0.00048212644080039557) | > avg_loss_disc_real_4: 0.24283675765487509 (-0.0006090266184068105) | > avg_loss_disc_real_5: 0.24669292611135563 (+0.001367086675805096) | > avg_loss_0: 2.602171461347124 (-0.04809020606564829) | > avg_grad_norm_0: tensor(191.2617, device='cuda:0') (tensor(-12.5138, device='cuda:0')) | > avg_loss_gen: 2.152647119172862 (+0.0668840106104458) | > avg_loss_kl: 2.134703500170103 (+0.06130838394165039) | > avg_loss_feat: 3.6693564670186647 (+0.25101357110789113) | > avg_loss_mel: 17.16967887609778 (+0.15205557917207102) | > avg_loss_duration: 1.5699199572415419 (+0.011170862426220562) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.69630590626891 (+0.5424322746169388) | > avg_grad_norm_1: tensor(6670.8359, device='cuda:0') (+tensor(1255.4722, device='cuda:0'))  > EPOCH: 932/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:49:08)   --> TIME: 2025-02-25 17:49:30 -- STEP: 45/72 -- GLOBAL_STEP: 1067150 | > loss_disc: 2.57845401763916 (2.6410444418589267) | > loss_disc_real_0: 0.14987494051456451 (0.14273787836233776) | > loss_disc_real_1: 0.21585404872894287 (0.22440073986848194) | > loss_disc_real_2: 0.2240174263715744 (0.23759282231330872) | > loss_disc_real_3: 0.24074223637580872 (0.2308304296599494) | > loss_disc_real_4: 0.22197213768959045 (0.2344415588511361) | > loss_disc_real_5: 0.21926015615463257 (0.240662443306711) | > loss_0: 2.57845401763916 (2.6410444418589267) | > grad_norm_0: tensor(363.6189, device='cuda:0') (tensor(230.6836, device='cuda:0')) | > loss_gen: 2.0902061462402344 (2.047694675127665) | > loss_kl: 2.318899393081665 (2.0414432101779516) | > loss_feat: 3.6955440044403076 (3.5443002170986597) | > loss_mel: 17.128658294677734 (16.71370118459066) | > loss_duration: 1.5603253841400146 (1.524505239062839) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.79363250732422 (25.871644846598308) | > grad_norm_1: tensor(11107.9434, device='cuda:0') (tensor(7302.0210, device='cuda:0')) | > current_lr_0: 0.0001780047231857004 | > current_lr_1: 0.0001780047231857004 | > step_time: 0.4736 (0.4416042274898953) | > loader_time: 0.0089 (0.007526763280232747) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613747610172758 (+3.7559321228888484e-05) | > avg_step_time: 0.46427588731470243 (+0.00965294032029701) | > avg_loss_disc: 2.6624204306535315 (+0.06024896930640766) | > avg_loss_disc_real_0: 0.14664322704496519 (+0.03138069714039145) | > avg_loss_disc_real_1: 0.22395559410813828 (+0.0005696908059254491) | > avg_loss_disc_real_2: 0.23843160166706837 (-0.0028589842185168246) | > avg_loss_disc_real_3: 0.2346115446006748 (-0.0020883958524381685) | > avg_loss_disc_real_4: 0.23914395419644638 (-0.003692803458428706) | > avg_loss_disc_real_5: 0.24411365368836363 (-0.002579272422991996) | > avg_loss_0: 2.6624204306535315 (+0.06024896930640766) | > avg_grad_norm_0: tensor(213.0353, device='cuda:0') (+tensor(21.7736, device='cuda:0')) | > avg_loss_gen: 2.0134899213280475 (-0.13915719784481428) | > avg_loss_kl: 2.0977861511875213 (-0.03691734898258181) | > avg_loss_feat: 3.430744063686317 (-0.2386124033323478) | > avg_loss_mel: 16.762954805938293 (-0.40672407015948764) | > avg_loss_duration: 1.5531246897200464 (-0.016795267521495516) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.858099870278803 (-0.8382060359901082) | > avg_grad_norm_1: tensor(7425.0898, device='cuda:0') (+tensor(754.2539, device='cuda:0'))  > EPOCH: 933/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:49:45)   --> TIME: 2025-02-25 17:49:57 -- STEP: 23/72 -- GLOBAL_STEP: 1067200 | > loss_disc: 2.4379162788391113 (2.6916519558948018) | > loss_disc_real_0: 0.07805125415325165 (0.16120945548881654) | > loss_disc_real_1: 0.22136348485946655 (0.2202545132326043) | > loss_disc_real_2: 0.23312625288963318 (0.2346505976241568) | > loss_disc_real_3: 0.22952885925769806 (0.22887517835782922) | > loss_disc_real_4: 0.2642032504081726 (0.23965814580088077) | > loss_disc_real_5: 0.2554779350757599 (0.2477800716524539) | > loss_0: 2.4379162788391113 (2.6916519558948018) | > grad_norm_0: tensor(15.9393, device='cuda:0') (tensor(303.3114, device='cuda:0')) | > loss_gen: 2.349679946899414 (1.9459620869678) | > loss_kl: 2.0403621196746826 (2.049463033676148) | > loss_feat: 3.763902425765991 (3.38501443033633) | > loss_mel: 16.910980224609375 (17.020876511283543) | > loss_duration: 1.5436201095581055 (1.5142870053001072) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.608543395996094 (25.915603140126105) | > grad_norm_1: tensor(3555.3513, device='cuda:0') (tensor(9582.2656, device='cuda:0')) | > current_lr_0: 0.00017798247259530217 | > current_lr_1: 0.00017798247259530217 | > step_time: 0.4433 (0.4363233421159827) | > loader_time: 0.0069 (0.0068083224089249324) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687032780177155 (+7.328517000439703e-05) | > avg_step_time: 0.45663953499055243 (-0.0076363523241499975) | > avg_loss_disc: 2.7127396455952817 (+0.05031921494175018) | > avg_loss_disc_real_0: 0.17842451294123293 (+0.03178128589626775) | > avg_loss_disc_real_1: 0.22348100303764074 (-0.00047459107049754867) | > avg_loss_disc_real_2: 0.2345968203645357 (-0.003834781302532675) | > avg_loss_disc_real_3: 0.23195761169346285 (-0.002653932907211942) | > avg_loss_disc_real_4: 0.23555948427865203 (-0.0035844699177943473) | > avg_loss_disc_real_5: 0.24526604968057553 (+0.0011523959922118965) | > avg_loss_0: 2.7127396455952817 (+0.05031921494175018) | > avg_grad_norm_0: tensor(264.0167, device='cuda:0') (+tensor(50.9813, device='cuda:0')) | > avg_loss_gen: 2.0220743065149014 (+0.008584385186853893) | > avg_loss_kl: 2.3833893705421767 (+0.2856032193546554) | > avg_loss_feat: 3.515223570272956 (+0.08447950658663927) | > avg_loss_mel: 18.514367681154063 (+1.7514128752157703) | > avg_loss_duration: 1.5820657535338067 (+0.02894106381376038) | > avg_amp_scaler: 9.802816901408448 (+1.8028169014084483) | > avg_loss_1: 28.017120764289103 (+2.1590208940103004) | > avg_grad_norm_1: tensor(6473.3105, device='cuda:0') (tensor(-951.7793, device='cuda:0'))  > EPOCH: 934/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:50:22)   --> TIME: 2025-02-25 17:50:24 -- STEP: 1/72 -- GLOBAL_STEP: 1067250 | > loss_disc: 2.851630449295044 (2.851630449295044) | > loss_disc_real_0: 0.45127835869789124 (0.45127835869789124) | > loss_disc_real_1: 0.22563806176185608 (0.22563806176185608) | > loss_disc_real_2: 0.23189283907413483 (0.23189283907413483) | > loss_disc_real_3: 0.2318415343761444 (0.2318415343761444) | > loss_disc_real_4: 0.23807358741760254 (0.23807358741760254) | > loss_disc_real_5: 0.24274583160877228 (0.24274583160877228) | > loss_0: 2.851630449295044 (2.851630449295044) | > grad_norm_0: tensor(544.0156, device='cuda:0') (tensor(544.0156, device='cuda:0')) | > loss_gen: 2.133212089538574 (2.133212089538574) | > loss_kl: 2.168759346008301 (2.168759346008301) | > loss_feat: 2.767329692840576 (2.767329692840576) | > loss_mel: 17.640562057495117 (17.640562057495117) | > loss_duration: 1.5665525197982788 (1.5665525197982788) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.27641487121582 (26.27641487121582) | > grad_norm_1: tensor(4000.6465, device='cuda:0') (tensor(4000.6465, device='cuda:0')) | > current_lr_0: 0.00017796022478622776 | > current_lr_1: 0.00017796022478622776 | > step_time: 0.4283 (0.4282674789428711) | > loader_time: 0.005 (0.005023002624511719)  --> TIME: 2025-02-25 17:50:47 -- STEP: 51/72 -- GLOBAL_STEP: 1067300 | > loss_disc: 2.313469171524048 (2.710645890703388) | > loss_disc_real_0: 0.04937072843313217 (0.18459751493498391) | > loss_disc_real_1: 0.1655692756175995 (0.2595529828001471) | > loss_disc_real_2: 0.18902702629566193 (0.24657328485273852) | > loss_disc_real_3: 0.17459706962108612 (0.23472316300167756) | > loss_disc_real_4: 0.21156416833400726 (0.2531946111543505) | > loss_disc_real_5: 0.22072580456733704 (0.24617309488502204) | > loss_0: 2.313469171524048 (2.710645890703388) | > grad_norm_0: tensor(45.6689, device='cuda:0') (tensor(272.9295, device='cuda:0')) | > loss_gen: 2.3941736221313477 (2.129630004658419) | > loss_kl: 2.0790114402770996 (2.350839004797093) | > loss_feat: 4.828605651855469 (3.5283421067630543) | > loss_mel: 17.56427764892578 (18.191976958630136) | > loss_duration: 1.6115951538085938 (1.6161965085010903) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.477664947509766 (27.81698455062567) | > grad_norm_1: tensor(6893.0762, device='cuda:0') (tensor(6824.3530, device='cuda:0')) | > current_lr_0: 0.00017796022478622776 | > current_lr_1: 0.00017796022478622776 | > step_time: 0.4433 (0.4440210707047406) | > loader_time: 0.0096 (0.0080412789887073) --> EVAL PERFORMANCE | > avg_loader_time: 0.008853086283509167 (+0.00016605350333201256) | > avg_step_time: 0.456001469786738 (-0.0006380652038144308) | > avg_loss_disc: 2.6771009069093514 (-0.03563873868593026) | > avg_loss_disc_real_0: 0.15532619988834354 (-0.023098313052889397) | > avg_loss_disc_real_1: 0.25059742071259195 (+0.02711641767495121) | > avg_loss_disc_real_2: 0.24443182076366854 (+0.009835000399132848) | > avg_loss_disc_real_3: 0.235899525628963 (+0.003941913935500141) | > avg_loss_disc_real_4: 0.25238729404731525 (+0.016827809768663216) | > avg_loss_disc_real_5: 0.2474585362303425 (+0.0021924865497669754) | > avg_loss_0: 2.6771009069093514 (-0.03563873868593026) | > avg_grad_norm_0: tensor(277.8258, device='cuda:0') (+tensor(13.8091, device='cuda:0')) | > avg_loss_gen: 2.123027309565477 (+0.10095300305057542) | > avg_loss_kl: 2.296986240736196 (-0.08640312980598086) | > avg_loss_feat: 3.5613361983232097 (+0.04611262805025351) | > avg_loss_mel: 17.906685224721134 (-0.6076824564329293) | > avg_loss_duration: 1.6158798211057421 (+0.03381406757193539) | > avg_amp_scaler: 8.0 (-1.8028169014084483) | > avg_loss_1: 27.50391479277275 (-0.5132059715163528) | > avg_grad_norm_1: tensor(8208.0938, device='cuda:0') (+tensor(1734.7832, device='cuda:0'))  > EPOCH: 935/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:50:59)   --> TIME: 2025-02-25 17:51:14 -- STEP: 29/72 -- GLOBAL_STEP: 1067350 | > loss_disc: 2.548656463623047 (2.5791708025439037) | > loss_disc_real_0: 0.06828197091817856 (0.08518000820587421) | > loss_disc_real_1: 0.22895099222660065 (0.24085897515560015) | > loss_disc_real_2: 0.251822829246521 (0.2348950808418208) | > loss_disc_real_3: 0.2380695790052414 (0.2439115797651225) | > loss_disc_real_4: 0.25404661893844604 (0.25613709513483385) | > loss_disc_real_5: 0.25367680191993713 (0.2540941680299825) | > loss_0: 2.548656463623047 (2.5791708025439037) | > grad_norm_0: tensor(73.5527, device='cuda:0') (tensor(272.0227, device='cuda:0')) | > loss_gen: 2.1914989948272705 (2.165549105611341) | > loss_kl: 1.9080368280410767 (1.9676385912401924) | > loss_feat: 3.861938953399658 (3.846525759532534) | > loss_mel: 16.432621002197266 (17.185289382934574) | > loss_duration: 1.5157079696655273 (1.5256974450473129) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.909805297851562 (26.69070026792329) | > grad_norm_1: tensor(6779.1816, device='cuda:0') (tensor(10775.5107, device='cuda:0')) | > current_lr_0: 0.00017793797975812947 | > current_lr_1: 0.00017793797975812947 | > step_time: 0.4441 (0.44047702591994714) | > loader_time: 0.0077 (0.00703851107893319) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871417220209686 (-0.00013891408141230785) | > avg_step_time: 0.46162706361690037 (+0.005625593830162368) | > avg_loss_disc: 2.574779510498047 (-0.10232139641130455) | > avg_loss_disc_real_0: 0.08423736048731165 (-0.07108883940103189) | > avg_loss_disc_real_1: 0.22896534309420788 (-0.021632077618384066) | > avg_loss_disc_real_2: 0.24441833689179218 (-1.3483871876357467e-05) | > avg_loss_disc_real_3: 0.2379194009052196 (+0.0020198752762566052) | > avg_loss_disc_real_4: 0.25018542501288393 (-0.002201869034431314) | > avg_loss_disc_real_5: 0.25006847818132866 (+0.0026099419509861554) | > avg_loss_0: 2.574779510498047 (-0.10232139641130455) | > avg_grad_norm_0: tensor(288.1648, device='cuda:0') (+tensor(10.3391, device='cuda:0')) | > avg_loss_gen: 2.162212974588636 (+0.03918566502315901) | > avg_loss_kl: 2.0111195621356157 (-0.2858666786005801) | > avg_loss_feat: 3.831229545700718 (+0.26989334737750825) | > avg_loss_mel: 17.013653594003596 (-0.8930316307175374) | > avg_loss_duration: 1.561425408846896 (-0.054454412258846174) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.579641046658367 (-0.924273746114384) | > avg_grad_norm_1: tensor(10392.4961, device='cuda:0') (+tensor(2184.4023, device='cuda:0'))  > EPOCH: 936/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:51:37)   --> TIME: 2025-02-25 17:51:42 -- STEP: 7/72 -- GLOBAL_STEP: 1067400 | > loss_disc: 2.6851766109466553 (2.522274971008301) | > loss_disc_real_0: 0.102750264108181 (0.0875086465052196) | > loss_disc_real_1: 0.22824178636074066 (0.22117527893611363) | > loss_disc_real_2: 0.2375224381685257 (0.23201411749635423) | > loss_disc_real_3: 0.2430698722600937 (0.22847730134214675) | > loss_disc_real_4: 0.26069706678390503 (0.24488993414810725) | > loss_disc_real_5: 0.24954994022846222 (0.23910760240895407) | > loss_0: 2.6851766109466553 (2.522274971008301) | > grad_norm_0: tensor(271.4096, device='cuda:0') (tensor(277.0554, device='cuda:0')) | > loss_gen: 2.114363193511963 (2.2403543336050853) | > loss_kl: 1.8048338890075684 (1.9570985862186976) | > loss_feat: 3.6521124839782715 (3.9343443598066057) | > loss_mel: 17.448198318481445 (16.913099016462052) | > loss_duration: 1.5039377212524414 (1.5202324901308333) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.52344512939453 (26.565128871372767) | > grad_norm_1: tensor(10152.8555, device='cuda:0') (tensor(9226.2500, device='cuda:0')) | > current_lr_0: 0.0001779157375106597 | > current_lr_1: 0.0001779157375106597 | > step_time: 0.4295 (0.4349851608276367) | > loader_time: 0.0055 (0.006616524287632534)  --> TIME: 2025-02-25 17:52:05 -- STEP: 57/72 -- GLOBAL_STEP: 1067450 | > loss_disc: 2.676762104034424 (2.5706156136696805) | > loss_disc_real_0: 0.07190543413162231 (0.10962195165063206) | > loss_disc_real_1: 0.3774951100349426 (0.2259331000478644) | > loss_disc_real_2: 0.25661125779151917 (0.2326612697358717) | > loss_disc_real_3: 0.24376174807548523 (0.2340288933431893) | > loss_disc_real_4: 0.2363968938589096 (0.2441208357350868) | > loss_disc_real_5: 0.25754475593566895 (0.24161512731460102) | > loss_0: 2.676762104034424 (2.5706156136696805) | > grad_norm_0: tensor(320.7124, device='cuda:0') (tensor(287.1585, device='cuda:0')) | > loss_gen: 2.2523791790008545 (2.162738227007682) | > loss_kl: 2.1787991523742676 (2.0002688399532387) | > loss_feat: 3.7384238243103027 (3.79038657640156) | > loss_mel: 16.60936737060547 (16.81289868605764) | > loss_duration: 1.6028634309768677 (1.5428639315722281) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.381832122802734 (26.309156150148628) | > grad_norm_1: tensor(7826.6689, device='cuda:0') (tensor(8543.4404, device='cuda:0')) | > current_lr_0: 0.0001779157375106597 | > current_lr_1: 0.0001779157375106597 | > step_time: 0.455 (0.44740154868678045) | > loader_time: 0.0099 (0.007980827699627793) --> EVAL PERFORMANCE | > avg_loader_time: 0.008635517577050436 (-7.865462504642352e-05) | > avg_step_time: 0.4582201527877593 (-0.0034069108291410743) | > avg_loss_disc: 2.576842418858703 (+0.002062908360656035) | > avg_loss_disc_real_0: 0.10858430277923463 (+0.024346942291922982) | > avg_loss_disc_real_1: 0.23162898337337332 (+0.0026636402791654357) | > avg_loss_disc_real_2: 0.23396478257548642 (-0.010453554316305763) | > avg_loss_disc_real_3: 0.23362697493022597 (-0.00429242597499363) | > avg_loss_disc_real_4: 0.2434079000647639 (-0.006777524948120034) | > avg_loss_disc_real_5: 0.24277370706410475 (-0.007294771117223903) | > avg_loss_0: 2.576842418858703 (+0.002062908360656035) | > avg_grad_norm_0: tensor(285.0498, device='cuda:0') (tensor(-3.1151, device='cuda:0')) | > avg_loss_gen: 2.1566639349494183 (-0.00554903963921749) | > avg_loss_kl: 2.0381161192773094 (+0.02699655714169369) | > avg_loss_feat: 3.7533280647976297 (-0.07790148090308824) | > avg_loss_mel: 16.854226757103284 (-0.15942683690031245) | > avg_loss_duration: 1.556036544517732 (-0.005388864329163967) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.358371331658162 (-0.22126971500020431) | > avg_grad_norm_1: tensor(8742.3154, device='cuda:0') (tensor(-1650.1807, device='cuda:0'))  > EPOCH: 937/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:52:14)   --> TIME: 2025-02-25 17:52:32 -- STEP: 35/72 -- GLOBAL_STEP: 1067500 | > loss_disc: 2.496727466583252 (2.518202420643398) | > loss_disc_real_0: 0.08105024695396423 (0.08077381295817239) | > loss_disc_real_1: 0.2280408889055252 (0.21611356820378985) | > loss_disc_real_2: 0.26331645250320435 (0.23137918923582348) | > loss_disc_real_3: 0.2539549171924591 (0.23105313267026628) | > loss_disc_real_4: 0.2628476619720459 (0.24046801711831775) | > loss_disc_real_5: 0.27536505460739136 (0.24263133406639098) | > loss_0: 2.496727466583252 (2.518202420643398) | > grad_norm_0: tensor(190.6577, device='cuda:0') (tensor(307.6077, device='cuda:0')) | > loss_gen: 2.1820733547210693 (2.1698475326810565) | > loss_kl: 2.1490371227264404 (1.9044383355549404) | > loss_feat: 3.5721192359924316 (3.9751401015690395) | > loss_mel: 16.90265464782715 (16.88276345389229) | > loss_duration: 1.5640382766723633 (1.5159271478652954) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.369922637939453 (26.448116411481585) | > grad_norm_1: tensor(12255.8789, device='cuda:0') (tensor(11440.3369, device='cuda:0')) | > current_lr_0: 0.00017789349804347087 | > current_lr_1: 0.00017789349804347087 | > step_time: 0.4431 (0.44062911442347935) | > loader_time: 0.0084 (0.007078347887311663) --> EVAL PERFORMANCE | > avg_loader_time: 0.008574875307754732 (-6.064226929570343e-05) | > avg_step_time: 0.4564320973947015 (-0.0017880553930578191) | > avg_loss_disc: 2.5244682876157096 (-0.05237413124299328) | > avg_loss_disc_real_0: 0.08764798902499844 (-0.020936313754236185) | > avg_loss_disc_real_1: 0.21761676102456912 (-0.014012222348804199) | > avg_loss_disc_real_2: 0.23114214858538668 (-0.0028226339900997433) | > avg_loss_disc_real_3: 0.2320635010239104 (-0.00156347390631556) | > avg_loss_disc_real_4: 0.24224618497029157 (-0.0011617150944723265) | > avg_loss_disc_real_5: 0.24500646473656237 (+0.0022327576724576192) | > avg_loss_0: 2.5244682876157096 (-0.05237413124299328) | > avg_grad_norm_0: tensor(334.9767, device='cuda:0') (+tensor(49.9269, device='cuda:0')) | > avg_loss_gen: 2.1879548354887635 (+0.031290900539345134) | > avg_loss_kl: 1.9752655633738343 (-0.0628505559034751) | > avg_loss_feat: 3.9107485657006924 (+0.1574205009030627) | > avg_loss_mel: 16.76558307862617 (-0.08864367847711208) | > avg_loss_duration: 1.5502795921245092 (-0.005756952393222825) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.389831623560948 (+0.03146029190278554) | > avg_grad_norm_1: tensor(11180.5303, device='cuda:0') (+tensor(2438.2148, device='cuda:0'))  > EPOCH: 938/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:52:51)   --> TIME: 2025-02-25 17:52:59 -- STEP: 13/72 -- GLOBAL_STEP: 1067550 | > loss_disc: 2.5352485179901123 (2.521347394356361) | > loss_disc_real_0: 0.05112260580062866 (0.09198721670187436) | > loss_disc_real_1: 0.18296515941619873 (0.21269230888440058) | > loss_disc_real_2: 0.19321438670158386 (0.22508047291865715) | > loss_disc_real_3: 0.1891191601753235 (0.23660428592791924) | > loss_disc_real_4: 0.2243109941482544 (0.23292950483468863) | > loss_disc_real_5: 0.22230663895606995 (0.23377868418510145) | > loss_0: 2.5352485179901123 (2.521347394356361) | > grad_norm_0: tensor(228.1920, device='cuda:0') (tensor(316.5940, device='cuda:0')) | > loss_gen: 2.0635313987731934 (2.2395219527758083) | > loss_kl: 1.7581207752227783 (1.8859025515042818) | > loss_feat: 4.246896743774414 (4.081362155767588) | > loss_mel: 16.90824317932129 (16.85158436114972) | > loss_duration: 1.4393508434295654 (1.4959990703142607) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.4161434173584 (26.554370439969578) | > grad_norm_1: tensor(12375.3633, device='cuda:0') (tensor(11088.7227, device='cuda:0')) | > current_lr_0: 0.00017787126135621544 | > current_lr_1: 0.00017787126135621544 | > step_time: 0.471 (0.43817811745863694) | > loader_time: 0.0059 (0.006641387939453125)  --> TIME: 2025-02-25 17:53:22 -- STEP: 63/72 -- GLOBAL_STEP: 1067600 | > loss_disc: 2.6325433254241943 (2.566936614021423) | > loss_disc_real_0: 0.23843738436698914 (0.1090948029406487) | > loss_disc_real_1: 0.21649782359600067 (0.22263575664588384) | > loss_disc_real_2: 0.19817809760570526 (0.23117013393886507) | > loss_disc_real_3: 0.23109713196754456 (0.2359651175756303) | > loss_disc_real_4: 0.2609253525733948 (0.23907688969657534) | > loss_disc_real_5: 0.26296037435531616 (0.2467639734820714) | > loss_0: 2.6325433254241943 (2.566936614021423) | > grad_norm_0: tensor(704.5155, device='cuda:0') (tensor(351.6584, device='cuda:0')) | > loss_gen: 2.370866298675537 (2.218863574285356) | > loss_kl: 2.126502752304077 (1.9846811975751604) | > loss_feat: 4.15913724899292 (3.9024778320675795) | > loss_mel: 17.85676383972168 (16.6236087859623) | > loss_duration: 1.5749645233154297 (1.536743578456697) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.088233947753906 (26.266374799940323) | > grad_norm_1: tensor(4842.7422, device='cuda:0') (tensor(10189.7910, device='cuda:0')) | > current_lr_0: 0.00017787126135621544 | > current_lr_1: 0.00017787126135621544 | > step_time: 0.5081 (0.4552387547871423) | > loader_time: 0.012 (0.008207025982084724) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580063430356305 (+5.188122601572712e-06) | > avg_step_time: 0.4617974086546562 (+0.005365311259954708) | > avg_loss_disc: 2.580901139219043 (+0.05643285160333322) | > avg_loss_disc_real_0: 0.1078260964910749 (+0.020178107466076456) | > avg_loss_disc_real_1: 0.22089777446128953 (+0.003281013436720409) | > avg_loss_disc_real_2: 0.23526177251003158 (+0.004119623924644905) | > avg_loss_disc_real_3: 0.23555970569731485 (+0.003496204673404446) | > avg_loss_disc_real_4: 0.24044436741043146 (-0.0018018175598601138) | > avg_loss_disc_real_5: 0.24712632218716848 (+0.0021198574506061107) | > avg_loss_0: 2.580901139219043 (+0.05643285160333322) | > avg_grad_norm_0: tensor(348.9384, device='cuda:0') (+tensor(13.9617, device='cuda:0')) | > avg_loss_gen: 2.1976843568640696 (+0.009729521375306138) | > avg_loss_kl: 2.005273627563261 (+0.03000806418942692) | > avg_loss_feat: 3.8685727388086444 (-0.04217582689204802) | > avg_loss_mel: 16.647997345722896 (-0.1175857329032759) | > avg_loss_duration: 1.545358076901503 (-0.004921515223006212) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.264885996429012 (-0.12494562713193602) | > avg_grad_norm_1: tensor(10024.3516, device='cuda:0') (tensor(-1156.1787, device='cuda:0'))  > EPOCH: 939/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:53:28)   --> TIME: 2025-02-25 17:53:49 -- STEP: 41/72 -- GLOBAL_STEP: 1067650 | > loss_disc: 2.479348659515381 (2.551848068469908) | > loss_disc_real_0: 0.11032983660697937 (0.11122981059115107) | > loss_disc_real_1: 0.21793411672115326 (0.22462698163055791) | > loss_disc_real_2: 0.24819771945476532 (0.22928835815045892) | > loss_disc_real_3: 0.2309941202402115 (0.2314502476918988) | > loss_disc_real_4: 0.2591846287250519 (0.2400179807005859) | > loss_disc_real_5: 0.24699988961219788 (0.24389428954298903) | > loss_0: 2.479348659515381 (2.551848068469908) | > grad_norm_0: tensor(152.6550, device='cuda:0') (tensor(298.2164, device='cuda:0')) | > loss_gen: 2.298475980758667 (2.2409790172809507) | > loss_kl: 1.9890215396881104 (1.9432845755321224) | > loss_feat: 3.745149612426758 (3.992564939871068) | > loss_mel: 14.934077262878418 (16.855353634531912) | > loss_duration: 1.5379716157913208 (1.5178408419213645) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.50469398498535 (26.55002291609601) | > grad_norm_1: tensor(8428.2246, device='cuda:0') (tensor(8653.9346, device='cuda:0')) | > current_lr_0: 0.0001778490274485459 | > current_lr_1: 0.0001778490274485459 | > step_time: 0.458 (0.4449452423467869) | > loader_time: 0.0089 (0.0073577950640422544) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861041646608165 (+3.0353035725344882e-05) | > avg_step_time: 0.4607927832804935 (-0.001004625374162682) | > avg_loss_disc: 2.6265864204353013 (+0.04568528121625848) | > avg_loss_disc_real_0: 0.1131700190492499 (+0.005343922558175007) | > avg_loss_disc_real_1: 0.24384389303519693 (+0.02294611857390741) | > avg_loss_disc_real_2: 0.24321610847829092 (+0.00795433596825934) | > avg_loss_disc_real_3: 0.23180643941315127 (-0.0037532662841635867) | > avg_loss_disc_real_4: 0.2403489691270909 (-9.539828334056977e-05) | > avg_loss_disc_real_5: 0.24342617266614672 (-0.003700149521021767) | > avg_loss_0: 2.6265864204353013 (+0.04568528121625848) | > avg_grad_norm_0: tensor(311.7759, device='cuda:0') (tensor(-37.1624, device='cuda:0')) | > avg_loss_gen: 2.1751790348912627 (-0.022505321972806858) | > avg_loss_kl: 2.018721664455575 (+0.013448036892313997) | > avg_loss_feat: 3.8199424777232425 (-0.048630261085401916) | > avg_loss_mel: 16.811067366264243 (+0.16307002054134756) | > avg_loss_duration: 1.5486886350201892 (+0.003330558118686211) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.37359920018156 (+0.10871320375254712) | > avg_grad_norm_1: tensor(8668.3975, device='cuda:0') (tensor(-1355.9541, device='cuda:0'))  > EPOCH: 940/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:54:06)   --> TIME: 2025-02-25 17:54:16 -- STEP: 19/72 -- GLOBAL_STEP: 1067700 | > loss_disc: 2.576362133026123 (2.7041870543831275) | > loss_disc_real_0: 0.04216689243912697 (0.1260656487000616) | > loss_disc_real_1: 0.2175447642803192 (0.3088940232992172) | > loss_disc_real_2: 0.18800044059753418 (0.2158312350511551) | > loss_disc_real_3: 0.2093142569065094 (0.21209090163833216) | > loss_disc_real_4: 0.20193852484226227 (0.24233088681572362) | > loss_disc_real_5: 0.20147818326950073 (0.2635271031605571) | > loss_0: 2.576362133026123 (2.7041870543831275) | > grad_norm_0: tensor(285.4042, device='cuda:0') (tensor(328.9832, device='cuda:0')) | > loss_gen: 1.6716421842575073 (2.010274316135206) | > loss_kl: 2.687809467315674 (2.3533610042772795) | > loss_feat: 4.086143970489502 (3.3609572209809957) | > loss_mel: 19.416847229003906 (18.005580701326068) | > loss_duration: 1.575555682182312 (1.5393030580721403) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.437997817993164 (27.269476238049958) | > grad_norm_1: tensor(12764.5078, device='cuda:0') (tensor(8871.9971, device='cuda:0')) | > current_lr_0: 0.00017782679632011482 | > current_lr_1: 0.00017782679632011482 | > step_time: 0.4294 (0.4371428740651984) | > loader_time: 0.0063 (0.006759994908383018)  --> TIME: 2025-02-25 17:54:40 -- STEP: 69/72 -- GLOBAL_STEP: 1067750 | > loss_disc: 2.546314239501953 (2.675236463546753) | > loss_disc_real_0: 0.07557974010705948 (0.13736695423722267) | > loss_disc_real_1: 0.2363671213388443 (0.25579072016736737) | > loss_disc_real_2: 0.2321746051311493 (0.22521392137244128) | > loss_disc_real_3: 0.22371532022953033 (0.22135774251343548) | > loss_disc_real_4: 0.24841904640197754 (0.24185560643672943) | > loss_disc_real_5: 0.2585429251194 (0.24815919606581982) | > loss_0: 2.546314239501953 (2.675236463546753) | > grad_norm_0: tensor(202.4908, device='cuda:0') (tensor(270.5711, device='cuda:0')) | > loss_gen: 2.143899917602539 (2.047028101008871) | > loss_kl: 2.107151508331299 (2.2923278100248696) | > loss_feat: 3.599221706390381 (3.5339945157368975) | > loss_mel: 17.106660842895508 (17.833475292592812) | > loss_duration: 1.6487336158752441 (1.5793296703393909) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.605667114257812 (27.286155479541723) | > grad_norm_1: tensor(8955.2871, device='cuda:0') (tensor(8068.0591, device='cuda:0')) | > current_lr_0: 0.00017782679632011482 | > current_lr_1: 0.00017782679632011482 | > step_time: 0.5351 (0.4557293290677278) | > loader_time: 0.0127 (0.008629622666732123) --> EVAL PERFORMANCE | > avg_loader_time: 0.008683480007547727 (+7.306354146607678e-05) | > avg_step_time: 0.45746178358373507 (-0.00333099969675843) | > avg_loss_disc: 2.6725741742362437 (+0.04598775380094233) | > avg_loss_disc_real_0: 0.13561682077780576 (+0.022446801728555857) | > avg_loss_disc_real_1: 0.2549455621292893 (+0.011101669094092381) | > avg_loss_disc_real_2: 0.22503823923393035 (-0.018177869244360573) | > avg_loss_disc_real_3: 0.2211368526791183 (-0.010669586734032954) | > avg_loss_disc_real_4: 0.24168739331440187 (+0.001338424187310977) | > avg_loss_disc_real_5: 0.24843560998708433 (+0.005009437320937615) | > avg_loss_0: 2.6725741742362437 (+0.04598775380094233) | > avg_grad_norm_0: tensor(268.9465, device='cuda:0') (tensor(-42.8294, device='cuda:0')) | > avg_loss_gen: 2.051953159587484 (-0.12322587530377893) | > avg_loss_kl: 2.290556314965369 (+0.2718346505097937) | > avg_loss_feat: 3.5446163533439097 (-0.2753261243793328) | > avg_loss_mel: 17.79714255265787 (+0.9860751863936272) | > avg_loss_duration: 1.5794805237944698 (+0.03079188877428063) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.263749028595402 (+0.8901498284138434) | > avg_grad_norm_1: tensor(8067.2456, device='cuda:0') (tensor(-601.1519, device='cuda:0'))  > EPOCH: 941/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:54:43)   --> TIME: 2025-02-25 17:55:06 -- STEP: 47/72 -- GLOBAL_STEP: 1067800 | > loss_disc: 2.6162078380584717 (2.5520980510305837) | > loss_disc_real_0: 0.054254814982414246 (0.09417952985522594) | > loss_disc_real_1: 0.19384288787841797 (0.2133302850292084) | > loss_disc_real_2: 0.2145286500453949 (0.2311590237820402) | > loss_disc_real_3: 0.21213741600513458 (0.23886683329622796) | > loss_disc_real_4: 0.21267403662204742 (0.2415020744851295) | > loss_disc_real_5: 0.21756650507450104 (0.24368865185595573) | > loss_0: 2.6162078380584717 (2.5520980510305837) | > grad_norm_0: tensor(389.4918, device='cuda:0') (tensor(343.8156, device='cuda:0')) | > loss_gen: 1.9478789567947388 (2.174243855983653) | > loss_kl: 2.0355775356292725 (1.9473378632931) | > loss_feat: 4.289556503295898 (3.9037282720525215) | > loss_mel: 17.113140106201172 (16.9358959197998) | > loss_duration: 1.581847906112671 (1.5314221661141578) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.968002319335938 (26.492628016370407) | > grad_norm_1: tensor(9833.5527, device='cuda:0') (tensor(10821.5986, device='cuda:0')) | > current_lr_0: 0.0001778045679705748 | > current_lr_1: 0.0001778045679705748 | > step_time: 0.4542 (0.4445015268122896) | > loader_time: 0.0093 (0.007760468949662878) --> EVAL PERFORMANCE | > avg_loader_time: 0.008774670077041842 (+9.1190069494115e-05) | > avg_step_time: 0.4661754352945677 (+0.008713651710832604) | > avg_loss_disc: 2.5478945315723682 (-0.12467964266387543) | > avg_loss_disc_real_0: 0.09258526317040686 (-0.04303155760739891) | > avg_loss_disc_real_1: 0.21704485697645537 (-0.03790070515283395) | > avg_loss_disc_real_2: 0.23148908350669162 (+0.006450844272761269) | > avg_loss_disc_real_3: 0.23570692497239987 (+0.014570072293281555) | > avg_loss_disc_real_4: 0.24027014849051623 (-0.0014172448238856383) | > avg_loss_disc_real_5: 0.24498921863629786 (-0.0034463913507864763) | > avg_loss_0: 2.5478945315723682 (-0.12467964266387543) | > avg_grad_norm_0: tensor(338.3193, device='cuda:0') (+tensor(69.3728, device='cuda:0')) | > avg_loss_gen: 2.1699010889295125 (+0.11794792934202869) | > avg_loss_kl: 1.9945765834459117 (-0.2959797315194572) | > avg_loss_feat: 3.8639843665378195 (+0.31936801319390984) | > avg_loss_mel: 16.812241890061067 (-0.9849006625968038) | > avg_loss_duration: 1.553056757215043 (-0.02642376657942669) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.39376065428828 (-0.8699883743071233) | > avg_grad_norm_1: tensor(10671.1211, device='cuda:0') (+tensor(2603.8755, device='cuda:0'))  > EPOCH: 942/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:55:21)   --> TIME: 2025-02-25 17:55:34 -- STEP: 25/72 -- GLOBAL_STEP: 1067850 | > loss_disc: 2.344022274017334 (2.5247460651397704) | > loss_disc_real_0: 0.04794911667704582 (0.08723099693655968) | > loss_disc_real_1: 0.20417121052742004 (0.216355761885643) | > loss_disc_real_2: 0.2376493662595749 (0.23031687021255492) | > loss_disc_real_3: 0.22780893743038177 (0.23188685595989228) | > loss_disc_real_4: 0.24371062219142914 (0.24293878734111785) | > loss_disc_real_5: 0.25520557165145874 (0.24859243273735046) | > loss_0: 2.344022274017334 (2.5247460651397704) | > grad_norm_0: tensor(64.3110, device='cuda:0') (tensor(309.2698, device='cuda:0')) | > loss_gen: 2.7521004676818848 (2.250895261764527) | > loss_kl: 2.1264326572418213 (1.9468362283706666) | > loss_feat: 4.905193328857422 (4.056402368545532) | > loss_mel: 17.01715660095215 (16.693349266052245) | > loss_duration: 1.5129612684249878 (1.5039963293075562) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.3138427734375 (26.451479339599608) | > grad_norm_1: tensor(9163.7285, device='cuda:0') (tensor(10561.9082, device='cuda:0')) | > current_lr_0: 0.00017778234239957847 | > current_lr_1: 0.00017778234239957847 | > step_time: 0.4512 (0.4407588195800781) | > loader_time: 0.0072 (0.006993808746337891) --> EVAL PERFORMANCE | > avg_loader_time: 0.008715522121375713 (-5.914795566612854e-05) | > avg_step_time: 0.4603575753494048 (-0.005817859945162873) | > avg_loss_disc: 2.5478412668469925 (-5.326472537570126e-05) | > avg_loss_disc_real_0: 0.09261265412812501 (+2.7390957718156916e-05) | > avg_loss_disc_real_1: 0.22535440997338632 (+0.008309552996930952) | > avg_loss_disc_real_2: 0.23270931755992727 (+0.0012202340532356526) | > avg_loss_disc_real_3: 0.23623464749732487 (+0.0005277225249250006) | > avg_loss_disc_real_4: 0.24619145733369907 (+0.005921308843182843) | > avg_loss_disc_real_5: 0.24986864962208444 (+0.004879430985786581) | > avg_loss_0: 2.5478412668469925 (-5.326472537570126e-05) | > avg_grad_norm_0: tensor(310.1993, device='cuda:0') (tensor(-28.1200, device='cuda:0')) | > avg_loss_gen: 2.2219337543971096 (+0.052032665467597106) | > avg_loss_kl: 2.000476595381616 (+0.005900011935704219) | > avg_loss_feat: 3.896286430493207 (+0.032302063955387705) | > avg_loss_mel: 16.60533057468038 (-0.20691131538068674) | > avg_loss_duration: 1.5482147361191223 (-0.004842021095920801) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.272241941640075 (-0.12151871264820358) | > avg_grad_norm_1: tensor(10183.8428, device='cuda:0') (tensor(-487.2783, device='cuda:0'))  > EPOCH: 943/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:55:58)   --> TIME: 2025-02-25 17:56:01 -- STEP: 3/72 -- GLOBAL_STEP: 1067900 | > loss_disc: 2.435084819793701 (2.527245283126831) | > loss_disc_real_0: 0.09417347609996796 (0.08952296276887257) | > loss_disc_real_1: 0.20035775005817413 (0.20228353142738342) | > loss_disc_real_2: 0.19471655786037445 (0.18895952900250754) | > loss_disc_real_3: 0.23168686032295227 (0.2234584242105484) | > loss_disc_real_4: 0.24533364176750183 (0.22964748243490854) | > loss_disc_real_5: 0.23833689093589783 (0.23563794294993082) | > loss_0: 2.435084819793701 (2.527245283126831) | > grad_norm_0: tensor(268.8065, device='cuda:0') (tensor(283.0382, device='cuda:0')) | > loss_gen: 2.2758264541625977 (2.1565513610839844) | > loss_kl: 2.0124361515045166 (1.984930117925008) | > loss_feat: 3.952338695526123 (3.944955507914225) | > loss_mel: 16.882369995117188 (16.691582997639973) | > loss_duration: 1.5380539894104004 (1.522745966911316) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.66102409362793 (26.30076535542806) | > grad_norm_1: tensor(7372.3037, device='cuda:0') (tensor(10397.5537, device='cuda:0')) | > current_lr_0: 0.0001777601196067785 | > current_lr_1: 0.0001777601196067785 | > step_time: 0.4264 (0.43366384506225586) | > loader_time: 0.0059 (0.006652514139811198)  --> TIME: 2025-02-25 17:56:24 -- STEP: 53/72 -- GLOBAL_STEP: 1067950 | > loss_disc: 2.5648460388183594 (2.56804902598543) | > loss_disc_real_0: 0.07198449969291687 (0.10234971829461602) | > loss_disc_real_1: 0.20078690350055695 (0.22169646508288834) | > loss_disc_real_2: 0.26406610012054443 (0.22622601856600563) | > loss_disc_real_3: 0.2200535386800766 (0.23447637147498582) | > loss_disc_real_4: 0.23691686987876892 (0.24564851033237745) | > loss_disc_real_5: 0.24702085554599762 (0.24691380166782523) | > loss_0: 2.5648460388183594 (2.56804902598543) | > grad_norm_0: tensor(119.7338, device='cuda:0') (tensor(295.5142, device='cuda:0')) | > loss_gen: 2.165886163711548 (2.193934283166561) | > loss_kl: 2.0650594234466553 (1.9997672674790867) | > loss_feat: 3.6755921840667725 (3.858099231180155) | > loss_mel: 17.08609962463379 (16.86006562214977) | > loss_duration: 1.5921597480773926 (1.5320227281102594) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.584796905517578 (26.443888970141142) | > grad_norm_1: tensor(5472.1553, device='cuda:0') (tensor(9124.8721, device='cuda:0')) | > current_lr_0: 0.0001777601196067785 | > current_lr_1: 0.0001777601196067785 | > step_time: 0.449 (0.4452192243540062) | > loader_time: 0.0092 (0.007894340551124429) --> EVAL PERFORMANCE | > avg_loader_time: 0.008700474886827066 (-1.5047234548646801e-05) | > avg_step_time: 0.4583737312907904 (-0.0019838440586144035) | > avg_loss_disc: 2.5735788479657242 (+0.0257375811187317) | > avg_loss_disc_real_0: 0.10598540725842329 (+0.013372753130298273) | > avg_loss_disc_real_1: 0.22384642883085867 (-0.001507981142527648) | > avg_loss_disc_real_2: 0.22811543731622294 (-0.004593880243704329) | > avg_loss_disc_real_3: 0.23457935746286956 (-0.0016552900344553034) | > avg_loss_disc_real_4: 0.24412487237386302 (-0.002066584959836054) | > avg_loss_disc_real_5: 0.24505787227355258 (-0.004810777348531858) | > avg_loss_0: 2.5735788479657242 (+0.0257375811187317) | > avg_grad_norm_0: tensor(285.7399, device='cuda:0') (tensor(-24.4594, device='cuda:0')) | > avg_loss_gen: 2.1865317603232146 (-0.03540199407389499) | > avg_loss_kl: 2.033739303199339 (+0.033262707817723225) | > avg_loss_feat: 3.8157507634498704 (-0.08053566704333681) | > avg_loss_mel: 16.864048313087142 (+0.25871773840676227) | > avg_loss_duration: 1.5495790881170353 (+0.0013643519979129604) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.449649112325318 (+0.17740717068524248) | > avg_grad_norm_1: tensor(8576.2500, device='cuda:0') (tensor(-1607.5928, device='cuda:0'))  > EPOCH: 944/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:56:35)   --> TIME: 2025-02-25 17:56:52 -- STEP: 31/72 -- GLOBAL_STEP: 1068000 | > loss_disc: 2.5881543159484863 (2.6044531714531685) | > loss_disc_real_0: 0.18427738547325134 (0.12495384028842371) | > loss_disc_real_1: 0.25227513909339905 (0.2208149827295734) | > loss_disc_real_2: 0.22148403525352478 (0.2359528080109627) | > loss_disc_real_3: 0.26253458857536316 (0.2449632147627492) | > loss_disc_real_4: 0.2471432238817215 (0.2510247177654697) | > loss_disc_real_5: 0.24468722939491272 (0.24785593152046204) | > loss_0: 2.5881543159484863 (2.6044531714531685) | > grad_norm_0: tensor(587.0014, device='cuda:0') (tensor(282.1409, device='cuda:0')) | > loss_gen: 2.289945363998413 (2.2133697194437825) | > loss_kl: 2.2132933139801025 (2.0795124999938475) | > loss_feat: 3.63362979888916 (3.8661099172407583) | > loss_mel: 17.335678100585938 (17.125808039019184) | > loss_duration: 1.544739007949829 (1.5144211438394362) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.01728630065918 (26.799221223400487) | > grad_norm_1: tensor(6712.1113, device='cuda:0') (tensor(7657.3765, device='cuda:0')) | > current_lr_0: 0.00017773789959182765 | > current_lr_1: 0.00017773789959182765 | > step_time: 0.4701 (0.4546851496542654) | > loader_time: 0.0084 (0.00717150011370259) --> EVAL PERFORMANCE | > avg_loader_time: 0.008736603696581344 (+3.6128809754277375e-05) | > avg_step_time: 0.46285116168814644 (+0.004477430397356041) | > avg_loss_disc: 2.578287141423829 (+0.004708293458104862) | > avg_loss_disc_real_0: 0.11059375742161778 (+0.004608350163194497) | > avg_loss_disc_real_1: 0.22628455724514707 (+0.002438128414288393) | > avg_loss_disc_real_2: 0.2326319158916742 (+0.00451647857545126) | > avg_loss_disc_real_3: 0.236759010964716 (+0.002179653501846429) | > avg_loss_disc_real_4: 0.24341026578150998 (-0.0007146065923530343) | > avg_loss_disc_real_5: 0.2460434831364054 (+0.0009856108628528149) | > avg_loss_0: 2.578287141423829 (+0.004708293458104862) | > avg_grad_norm_0: tensor(288.9150, device='cuda:0') (+tensor(3.1750, device='cuda:0')) | > avg_loss_gen: 2.213685819800471 (+0.027154059477256354) | > avg_loss_kl: 2.0979273772575486 (+0.06418807405820948) | > avg_loss_feat: 3.872051437136153 (+0.056300673686282554) | > avg_loss_mel: 16.890772577742457 (+0.0267242646553143) | > avg_loss_duration: 1.5525862750872759 (+0.003007186970240605) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.627023401394705 (+0.17737428906938746) | > avg_grad_norm_1: tensor(8606.5586, device='cuda:0') (+tensor(30.3086, device='cuda:0'))  > EPOCH: 945/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:57:13)   --> TIME: 2025-02-25 17:57:19 -- STEP: 9/72 -- GLOBAL_STEP: 1068050 | > loss_disc: 2.5118680000305176 (2.57918373743693) | > loss_disc_real_0: 0.14575837552547455 (0.10964707988831732) | > loss_disc_real_1: 0.23127897083759308 (0.22234713700082567) | > loss_disc_real_2: 0.20654182136058807 (0.22424678007761636) | > loss_disc_real_3: 0.2328329235315323 (0.2506071693367428) | > loss_disc_real_4: 0.24238009750843048 (0.24816618694199455) | > loss_disc_real_5: 0.2598344385623932 (0.24674617085191938) | > loss_0: 2.5118680000305176 (2.57918373743693) | > grad_norm_0: tensor(245.5624, device='cuda:0') (tensor(293.3988, device='cuda:0')) | > loss_gen: 2.2893762588500977 (2.251153906186422) | > loss_kl: 2.1739327907562256 (1.9445079962412517) | > loss_feat: 3.8723838329315186 (4.04955710305108) | > loss_mel: 17.58235740661621 (17.134711583455402) | > loss_duration: 1.5146472454071045 (1.5072002410888672) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.432695388793945 (26.887130737304688) | > grad_norm_1: tensor(7493.1870, device='cuda:0') (tensor(8686.8955, device='cuda:0')) | > current_lr_0: 0.00017771568235437865 | > current_lr_1: 0.00017771568235437865 | > step_time: 0.4295 (0.43546178605821395) | > loader_time: 0.0057 (0.0063272582160102)  --> TIME: 2025-02-25 17:57:42 -- STEP: 59/72 -- GLOBAL_STEP: 1068100 | > loss_disc: 2.5405936241149902 (2.644309120663142) | > loss_disc_real_0: 0.054726630449295044 (0.13163512774695782) | > loss_disc_real_1: 0.21132737398147583 (0.23172410665932347) | > loss_disc_real_2: 0.2186795175075531 (0.23939438745126887) | > loss_disc_real_3: 0.23313193023204803 (0.23451988601078422) | > loss_disc_real_4: 0.24283945560455322 (0.24327447176989866) | > loss_disc_real_5: 0.24295663833618164 (0.24513431044958403) | > loss_0: 2.5405936241149902 (2.644309120663142) | > grad_norm_0: tensor(300.6468, device='cuda:0') (tensor(315.5930, device='cuda:0')) | > loss_gen: 2.1762197017669678 (2.12932133472572) | > loss_kl: 2.118269443511963 (2.092845680349965) | > loss_feat: 4.3120012283325195 (3.6803622932757363) | > loss_mel: 16.769241333007812 (16.898802660279355) | > loss_duration: 1.5814433097839355 (1.540432473360482) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.957176208496094 (26.341764385417356) | > grad_norm_1: tensor(11065.2900, device='cuda:0') (tensor(9004.0010, device='cuda:0')) | > current_lr_0: 0.00017771568235437865 | > current_lr_1: 0.00017771568235437865 | > step_time: 0.4746 (0.4528803017179845) | > loader_time: 0.01 (0.008078134666054932) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671099031475225 (-6.55046651061187e-05) | > avg_step_time: 0.461619625628834 (-0.0012315360593124214) | > avg_loss_disc: 2.6294402807531223 (+0.05115313932929322) | > avg_loss_disc_real_0: 0.1252328577962979 (+0.014639100374680108) | > avg_loss_disc_real_1: 0.22992475016016356 (+0.0036401929150164936) | > avg_loss_disc_real_2: 0.23757224708375796 (+0.004940331192083758) | > avg_loss_disc_real_3: 0.23293234617777273 (-0.003826664786943268) | > avg_loss_disc_real_4: 0.24233385605711333 (-0.0010764097243966542) | > avg_loss_disc_real_5: 0.2454990801257147 (-0.0005444030106906972) | > avg_loss_0: 2.6294402807531223 (+0.05115313932929322) | > avg_grad_norm_0: tensor(313.1895, device='cuda:0') (+tensor(24.2746, device='cuda:0')) | > avg_loss_gen: 2.1285229400849683 (-0.08516287971550263) | > avg_loss_kl: 2.104130652588858 (+0.006203275331309488) | > avg_loss_feat: 3.6842812383678596 (-0.18777019876829337) | > avg_loss_mel: 16.867165081937546 (-0.023607495804910172) | > avg_loss_duration: 1.551335277691693 (-0.0012509973955827647) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.335435061387614 (-0.2915883400070918) | > avg_grad_norm_1: tensor(9187.4863, device='cuda:0') (+tensor(580.9277, device='cuda:0'))  > EPOCH: 946/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:57:50)   --> TIME: 2025-02-25 17:58:09 -- STEP: 37/72 -- GLOBAL_STEP: 1068150 | > loss_disc: 2.802727699279785 (2.6465810698431893) | > loss_disc_real_0: 0.23819884657859802 (0.12331969180219882) | > loss_disc_real_1: 0.33000415563583374 (0.26418005131386424) | > loss_disc_real_2: 0.23842939734458923 (0.21955848545641513) | > loss_disc_real_3: 0.24924680590629578 (0.23276756462213155) | > loss_disc_real_4: 0.21130946278572083 (0.2410337876629185) | > loss_disc_real_5: 0.250499427318573 (0.2366567286285194) | > loss_0: 2.802727699279785 (2.6465810698431893) | > grad_norm_0: tensor(617.2601, device='cuda:0') (tensor(340.0229, device='cuda:0')) | > loss_gen: 2.144907236099243 (2.196873014037674) | > loss_kl: 1.95002281665802 (2.0327625757939103) | > loss_feat: 3.618063449859619 (3.8367487482122473) | > loss_mel: 16.07638931274414 (16.80974261825149) | > loss_duration: 1.560835838317871 (1.51282368157361) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.3502197265625 (26.38895086340002) | > grad_norm_1: tensor(9307.2207, device='cuda:0') (tensor(9399.7734, device='cuda:0')) | > current_lr_0: 0.00017769346789408435 | > current_lr_1: 0.00017769346789408435 | > step_time: 0.4522 (0.4387238090102737) | > loader_time: 0.0079 (0.007225964520428632) --> EVAL PERFORMANCE | > avg_loader_time: 0.008597367246386031 (-7.37317850891938e-05) | > avg_step_time: 0.45678218653504277 (-0.0048374390937912515) | > avg_loss_disc: 2.649149182816627 (+0.01970890206350484) | > avg_loss_disc_real_0: 0.11209495848333331 (-0.01313789931296458) | > avg_loss_disc_real_1: 0.2535725580134861 (+0.023647807853322544) | > avg_loss_disc_real_2: 0.2330085580617609 (-0.004563689021997053) | > avg_loss_disc_real_3: 0.24519958126712854 (+0.012267235089355816) | > avg_loss_disc_real_4: 0.2543115231772543 (+0.011977667120140978) | > avg_loss_disc_real_5: 0.24226981821194501 (-0.003229261913769682) | > avg_loss_0: 2.649149182816627 (+0.01970890206350484) | > avg_grad_norm_0: tensor(299.0779, device='cuda:0') (tensor(-14.1116, device='cuda:0')) | > avg_loss_gen: 2.189369619732172 (+0.06084667964720358) | > avg_loss_kl: 2.2316233927095444 (+0.12749274012068623) | > avg_loss_feat: 3.8061065908888696 (+0.12182535252101001) | > avg_loss_mel: 17.062821280788373 (+0.19565619885082697) | > avg_loss_duration: 1.5607341326458353 (+0.009398854954142166) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.85065500501176 (+0.515219943624146) | > avg_grad_norm_1: tensor(9053.0264, device='cuda:0') (tensor(-134.4600, device='cuda:0'))  > EPOCH: 947/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:58:27)   --> TIME: 2025-02-25 17:58:36 -- STEP: 15/72 -- GLOBAL_STEP: 1068200 | > loss_disc: 2.5129475593566895 (2.6098721822102866) | > loss_disc_real_0: 0.06677111983299255 (0.09701714863379797) | > loss_disc_real_1: 0.2059241235256195 (0.22931811114152273) | > loss_disc_real_2: 0.21772652864456177 (0.23414047062397003) | > loss_disc_real_3: 0.23235905170440674 (0.24364241858323415) | > loss_disc_real_4: 0.2533591091632843 (0.22693512141704558) | > loss_disc_real_5: 0.2567982077598572 (0.24566902418931325) | > loss_0: 2.5129475593566895 (2.6098721822102866) | > grad_norm_0: tensor(175.2314, device='cuda:0') (tensor(328.6720, device='cuda:0')) | > loss_gen: 2.295851469039917 (2.065608135859171) | > loss_kl: 2.006387710571289 (2.089933109283447) | > loss_feat: 4.013865947723389 (3.7219871997833254) | > loss_mel: 18.095409393310547 (17.066340510050455) | > loss_duration: 1.4884840250015259 (1.5043563763300578) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.89999771118164 (26.448225021362305) | > grad_norm_1: tensor(8432.8789, device='cuda:0') (tensor(10374.7725, device='cuda:0')) | > current_lr_0: 0.0001776712562105976 | > current_lr_1: 0.0001776712562105976 | > step_time: 0.4284 (0.4385980129241943) | > loader_time: 0.0059 (0.006654612223307292)  --> TIME: 2025-02-25 17:59:00 -- STEP: 65/72 -- GLOBAL_STEP: 1068250 | > loss_disc: 2.433149814605713 (2.6036917062906118) | > loss_disc_real_0: 0.08582853525876999 (0.09393769402343494) | > loss_disc_real_1: 0.3390832245349884 (0.2295420297063314) | > loss_disc_real_2: 0.1951473206281662 (0.23093684613704682) | > loss_disc_real_3: 0.23879185318946838 (0.2356719413628945) | > loss_disc_real_4: 0.24510163068771362 (0.24486887225737938) | > loss_disc_real_5: 0.26069679856300354 (0.2481082233098837) | > loss_0: 2.433149814605713 (2.6036917062906118) | > grad_norm_0: tensor(54.0175, device='cuda:0') (tensor(307.1792, device='cuda:0')) | > loss_gen: 2.134202480316162 (2.119017028808594) | > loss_kl: 2.103219509124756 (2.0855461835861204) | > loss_feat: 3.428168535232544 (3.78077475474431) | > loss_mel: 17.179845809936523 (16.702692736112155) | > loss_duration: 1.6148300170898438 (1.5453789454240066) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.460268020629883 (26.233409588153545) | > grad_norm_1: tensor(12392.5332, device='cuda:0') (tensor(10137.0615, device='cuda:0')) | > current_lr_0: 0.0001776712562105976 | > current_lr_1: 0.0001776712562105976 | > step_time: 0.5134 (0.45217208862304686) | > loader_time: 0.0118 (0.008458695044884315) --> EVAL PERFORMANCE | > avg_loader_time: 0.008732641246956838 (+0.00013527400057080675) | > avg_step_time: 0.4580665239146058 (+0.0012843373795630164) | > avg_loss_disc: 2.6212118545048675 (-0.027937328311759657) | > avg_loss_disc_real_0: 0.09761799559731718 (-0.014476962886016131) | > avg_loss_disc_real_1: 0.22791399470937085 (-0.025658563304115256) | > avg_loss_disc_real_2: 0.2328862640219675 (-0.00012229403979341358) | > avg_loss_disc_real_3: 0.23498987962662335 (-0.010209701640505192) | > avg_loss_disc_real_4: 0.24459452591311764 (-0.009716997264136662) | > avg_loss_disc_real_5: 0.24719797830346604 (+0.004928160091521028) | > avg_loss_0: 2.6212118545048675 (-0.027937328311759657) | > avg_grad_norm_0: tensor(307.7907, device='cuda:0') (+tensor(8.7128, device='cuda:0')) | > avg_loss_gen: 2.0919233191181243 (-0.09744630061404758) | > avg_loss_kl: 2.0925143251956344 (-0.13910906751390995) | > avg_loss_feat: 3.7187308929335905 (-0.08737569795527911) | > avg_loss_mel: 16.72081860018448 (-0.34200268060389405) | > avg_loss_duration: 1.5519950490602306 (-0.008739083585604712) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.175982139479945 (-0.6746728655318144) | > avg_grad_norm_1: tensor(9741.8682, device='cuda:0') (+tensor(688.8418, device='cuda:0'))  > EPOCH: 948/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:59:05)   --> TIME: 2025-02-25 17:59:27 -- STEP: 43/72 -- GLOBAL_STEP: 1068300 | > loss_disc: 2.8637843132019043 (2.699622076611186) | > loss_disc_real_0: 0.24394884705543518 (0.17378879381820214) | > loss_disc_real_1: 0.21526457369327545 (0.2334812591935313) | > loss_disc_real_2: 0.184200257062912 (0.23813039757484614) | > loss_disc_real_3: 0.2626791000366211 (0.24153498750786448) | > loss_disc_real_4: 0.2693447470664978 (0.24273513222849646) | > loss_disc_real_5: 0.25748854875564575 (0.24383967283160188) | > loss_0: 2.8637843132019043 (2.699622076611186) | > grad_norm_0: tensor(166.6876, device='cuda:0') (tensor(272.8878, device='cuda:0')) | > loss_gen: 2.2249672412872314 (2.12625223814055) | > loss_kl: 2.1299779415130615 (2.589785753294479) | > loss_feat: 3.763046979904175 (3.719299870868062) | > loss_mel: 19.74354362487793 (19.930629508439885) | > loss_duration: 1.6025643348693848 (1.6045639376307643) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.464099884033203 (29.970530842625816) | > grad_norm_1: tensor(2835.8821, device='cuda:0') (tensor(5956.5166, device='cuda:0')) | > current_lr_0: 0.00017764904730357126 | > current_lr_1: 0.00017764904730357126 | > step_time: 0.4647 (0.4434728844221248) | > loader_time: 0.0098 (0.00746379342190055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598502253142884 (-0.00013413899381395367) | > avg_step_time: 0.4569506611622555 (-0.0011158627523502607) | > avg_loss_disc: 2.684483464335052 (+0.06327160983018443) | > avg_loss_disc_real_0: 0.1573116497044832 (+0.05969365410716601) | > avg_loss_disc_real_1: 0.22602535490418824 (-0.0018886398051826125) | > avg_loss_disc_real_2: 0.22729221385129741 (-0.0055940501706700785) | > avg_loss_disc_real_3: 0.23810385167598724 (+0.0031139720493638945) | > avg_loss_disc_real_4: 0.2439819677195079 (-0.0006125581936097524) | > avg_loss_disc_real_5: 0.2434545289462721 (-0.003743449357193951) | > avg_loss_0: 2.684483464335052 (+0.06327160983018443) | > avg_grad_norm_0: tensor(250.8554, device='cuda:0') (tensor(-56.9354, device='cuda:0')) | > avg_loss_gen: 2.086083303035145 (-0.00584001608297946) | > avg_loss_kl: 2.45028130101486 (+0.3577669758192257) | > avg_loss_feat: 3.6174841195764675 (-0.10124677335712295) | > avg_loss_mel: 18.99309921264648 (+2.2722806124620014) | > avg_loss_duration: 1.6056865067549155 (+0.053691457694684974) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.752634155918177 (+2.5766520164382314) | > avg_grad_norm_1: tensor(6757.6050, device='cuda:0') (tensor(-2984.2632, device='cuda:0'))  > EPOCH: 949/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 17:59:42)   --> TIME: 2025-02-25 17:59:54 -- STEP: 21/72 -- GLOBAL_STEP: 1068350 | > loss_disc: 2.447563886642456 (2.617382185799735) | > loss_disc_real_0: 0.21834304928779602 (0.1325109153985977) | > loss_disc_real_1: 0.16185668110847473 (0.22351727457273574) | > loss_disc_real_2: 0.22535279393196106 (0.23581609484695254) | > loss_disc_real_3: 0.2534756362438202 (0.2367628159977141) | > loss_disc_real_4: 0.2276109755039215 (0.24509399774528684) | > loss_disc_real_5: 0.22572395205497742 (0.23873676856358847) | > loss_0: 2.447563886642456 (2.617382185799735) | > grad_norm_0: tensor(160.1981, device='cuda:0') (tensor(157.6387, device='cuda:0')) | > loss_gen: 2.7677838802337646 (2.214659367288862) | > loss_kl: 3.1690914630889893 (3.649360327493577) | > loss_feat: 4.200174331665039 (3.8697668597811745) | > loss_mel: 20.807964324951172 (22.33933548700242) | > loss_duration: 1.9103457927703857 (2.1585785207294284) | > amp_scaler: 8.0 (8.0) | > loss_1: 32.85536193847656 (34.2317008972168) | > grad_norm_1: tensor(1933.7550, device='cuda:0') (tensor(6927.0806, device='cuda:0')) | > current_lr_0: 0.0001776268411726583 | > current_lr_1: 0.0001776268411726583 | > step_time: 0.4327 (0.43975123905000235) | > loader_time: 0.0063 (0.00699811890011742)  --> TIME: 2025-02-25 18:00:18 -- STEP: 71/72 -- GLOBAL_STEP: 1068400 | > loss_disc: 2.5379199981689453 (2.661686524538927) | > loss_disc_real_0: 0.11746526509523392 (0.12884285211772992) | > loss_disc_real_1: 0.25862976908683777 (0.2225484936170175) | > loss_disc_real_2: 0.28081390261650085 (0.21781420015113453) | > loss_disc_real_3: 0.19636477530002594 (0.22853711520282316) | > loss_disc_real_4: 0.22542202472686768 (0.2377427335776074) | > loss_disc_real_5: 0.23005832731723785 (0.2444454593137956) | > loss_0: 2.5379199981689453 (2.661686524538927) | > grad_norm_0: tensor(181.1413, device='cuda:0') (tensor(257.7460, device='cuda:0')) | > loss_gen: 2.0306599140167236 (2.0508477671045653) | > loss_kl: 2.1424355506896973 (2.7537073921149884) | > loss_feat: 3.93355131149292 (3.6787485068952535) | > loss_mel: 18.259389877319336 (19.238374951859598) | > loss_duration: 1.6858748197555542 (1.875347699917538) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.051912307739258 (29.597026368262064) | > grad_norm_1: tensor(12157.4961, device='cuda:0') (tensor(8013.9302, device='cuda:0')) | > current_lr_0: 0.0001776268411726583 | > current_lr_1: 0.0001776268411726583 | > step_time: 0.494 (0.45872250409193444) | > loader_time: 0.0081 (0.008757040534220955) --> EVAL PERFORMANCE | > avg_loader_time: 0.008757040534220955 (+0.00015853828107807028) | > avg_step_time: 0.45872250409193444 (+0.0017718429296789129) | > avg_loss_disc: 2.661686524538927 (-0.02279693979612496) | > avg_loss_disc_real_0: 0.12884285211772992 (-0.028468797586753275) | > avg_loss_disc_real_1: 0.2225484936170175 (-0.0034768612871707294) | > avg_loss_disc_real_2: 0.21781420015113453 (-0.00947801370016288) | > avg_loss_disc_real_3: 0.22853711520282316 (-0.009566736473164084) | > avg_loss_disc_real_4: 0.2377427335776074 (-0.0062392341419005015) | > avg_loss_disc_real_5: 0.2444454593137956 (+0.0009909303675235126) | > avg_loss_0: 2.661686524538927 (-0.02279693979612496) | > avg_grad_norm_0: tensor(257.7460, device='cuda:0') (+tensor(6.8906, device='cuda:0')) | > avg_loss_gen: 2.0508477671045653 (-0.0352355359305796) | > avg_loss_kl: 2.7537073921149884 (+0.3034260911001283) | > avg_loss_feat: 3.6787485068952535 (+0.06126438731878592) | > avg_loss_mel: 19.238374951859598 (+0.24527573921311685) | > avg_loss_duration: 1.875347699917538 (+0.2696611931626225) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.597026368262064 (+0.8443922123438874) | > avg_grad_norm_1: tensor(8013.9302, device='cuda:0') (+tensor(1256.3252, device='cuda:0'))  > EPOCH: 950/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:00:19)   --> TIME: 2025-02-25 18:00:44 -- STEP: 49/72 -- GLOBAL_STEP: 1068450 | > loss_disc: 2.564232349395752 (2.6237776911988555) | > loss_disc_real_0: 0.1077699214220047 (0.08167368238221626) | > loss_disc_real_1: 0.23010440170764923 (0.28298514427579186) | > loss_disc_real_2: 0.2364872246980667 (0.2581280144501705) | > loss_disc_real_3: 0.22412560880184174 (0.22839641692687054) | > loss_disc_real_4: 0.25333699584007263 (0.2404392233916691) | > loss_disc_real_5: 0.24393895268440247 (0.2481612110016297) | > loss_0: 2.564232349395752 (2.6237776911988555) | > grad_norm_0: tensor(220.7946, device='cuda:0') (tensor(311.6925, device='cuda:0')) | > loss_gen: 2.543734073638916 (2.2023483344486787) | > loss_kl: 1.9975500106811523 (2.0722055021597416) | > loss_feat: 4.327674388885498 (3.9223567028434903) | > loss_mel: 15.891955375671387 (17.178289958408897) | > loss_duration: 1.7052910327911377 (1.6620013616522964) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.466205596923828 (27.037202134424327) | > grad_norm_1: tensor(1923.1934, device='cuda:0') (tensor(10793.2998, device='cuda:0')) | > current_lr_0: 0.0001776046378175117 | > current_lr_1: 0.0001776046378175117 | > step_time: 0.4421 (0.4433726047982975) | > loader_time: 0.0101 (0.007921189677958585) --> EVAL PERFORMANCE | > avg_loader_time: 0.008820238247723645 (+6.319771350269017e-05) | > avg_step_time: 0.45678635046515664 (-0.0019361536267777923) | > avg_loss_disc: 2.631120241863627 (-0.030566282675299927) | > avg_loss_disc_real_0: 0.1058050910614326 (-0.023037761056297318) | > avg_loss_disc_real_1: 0.2629611831105932 (+0.04041268949357568) | > avg_loss_disc_real_2: 0.2462740038901987 (+0.028459803739064177) | > avg_loss_disc_real_3: 0.23638924452620494 (+0.007852129323381779) | > avg_loss_disc_real_4: 0.24325836712205914 (+0.0055156335444517535) | > avg_loss_disc_real_5: 0.24701676339330808 (+0.0025713040795124764) | > avg_loss_0: 2.631120241863627 (-0.030566282675299927) | > avg_grad_norm_0: tensor(282.1838, device='cuda:0') (+tensor(24.4379, device='cuda:0')) | > avg_loss_gen: 2.2159511388187685 (+0.1651033717142032) | > avg_loss_kl: 2.094248994975023 (-0.6594583971399652) | > avg_loss_feat: 3.8666740303308194 (+0.1879255234355659) | > avg_loss_mel: 17.229622397624272 (-2.0087525542353255) | > avg_loss_duration: 1.6630555173041115 (-0.21229218261342653) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.069552166361206 (-2.5274742019008585) | > avg_grad_norm_1: tensor(8637.5283, device='cuda:0') (+tensor(623.5981, device='cuda:0'))  > EPOCH: 951/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:00:56)   --> TIME: 2025-02-25 18:01:11 -- STEP: 27/72 -- GLOBAL_STEP: 1068500 | > loss_disc: 2.551851511001587 (2.6314477302409984) | > loss_disc_real_0: 0.05398590862751007 (0.08305191372831662) | > loss_disc_real_1: 0.21733859181404114 (0.2054088800041764) | > loss_disc_real_2: 0.2938462793827057 (0.24161589366418343) | > loss_disc_real_3: 0.2460021823644638 (0.21958873282980035) | > loss_disc_real_4: 0.2619365155696869 (0.24033817924835063) | > loss_disc_real_5: 0.27822569012641907 (0.2344872322347429) | > loss_0: 2.551851511001587 (2.6314477302409984) | > grad_norm_0: tensor(282.5129, device='cuda:0') (tensor(284.1893, device='cuda:0')) | > loss_gen: 2.245216131210327 (1.9732293287913005) | > loss_kl: 2.025522232055664 (2.029525615550853) | > loss_feat: 3.6828715801239014 (3.680325887821339) | > loss_mel: 16.015783309936523 (17.066389825609004) | > loss_duration: 1.5643965005874634 (1.571388809769242) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.533790588378906 (26.32085934391728) | > grad_norm_1: tensor(13414.2979, device='cuda:0') (tensor(8633.4639, device='cuda:0')) | > current_lr_0: 0.00017758243723778452 | > current_lr_1: 0.00017758243723778452 | > step_time: 0.4354 (0.44160720154091165) | > loader_time: 0.0079 (0.007084590417367441) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647828034951652 (-0.0001724102127719923) | > avg_step_time: 0.4595567071941537 (+0.0027703567289970588) | > avg_loss_disc: 2.597248047170505 (-0.03387219469312219) | > avg_loss_disc_real_0: 0.08289782849835678 (-0.02290726256307582) | > avg_loss_disc_real_1: 0.21914636522111758 (-0.04381481788947561) | > avg_loss_disc_real_2: 0.2382042517124767 (-0.008069752177722017) | > avg_loss_disc_real_3: 0.23285637711974935 (-0.0035328674064555865) | > avg_loss_disc_real_4: 0.24389787438050123 (+0.0006395072584420847) | > avg_loss_disc_real_5: 0.2431735524409254 (-0.0038432109523826863) | > avg_loss_0: 2.597248047170505 (-0.03387219469312219) | > avg_grad_norm_0: tensor(300.3199, device='cuda:0') (+tensor(18.1361, device='cuda:0')) | > avg_loss_gen: 2.062347883909521 (-0.15360325490924742) | > avg_loss_kl: 2.0185899432276324 (-0.07565905174739074) | > avg_loss_feat: 3.7100287726227665 (-0.15664525770805282) | > avg_loss_mel: 16.826656354984767 (-0.4029660426395054) | > avg_loss_duration: 1.5961801099105621 (-0.06687540739354936) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.213803062976247 (-0.8557491033849587) | > avg_grad_norm_1: tensor(9918.4346, device='cuda:0') (+tensor(1280.9062, device='cuda:0'))  > EPOCH: 952/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:01:34)   --> TIME: 2025-02-25 18:01:38 -- STEP: 5/72 -- GLOBAL_STEP: 1068550 | > loss_disc: 2.762360095977783 (2.5275134563446047) | > loss_disc_real_0: 0.07622922211885452 (0.08614014387130738) | > loss_disc_real_1: 0.26511871814727783 (0.21873755753040314) | > loss_disc_real_2: 0.2276126742362976 (0.22907815575599672) | > loss_disc_real_3: 0.2301129400730133 (0.22469043731689453) | > loss_disc_real_4: 0.2398267686367035 (0.23087772727012634) | > loss_disc_real_5: 0.23892711102962494 (0.2406539350748062) | > loss_0: 2.762360095977783 (2.5275134563446047) | > grad_norm_0: tensor(417.4115, device='cuda:0') (tensor(344.3741, device='cuda:0')) | > loss_gen: 1.9694347381591797 (2.1810004711151123) | > loss_kl: 2.0676321983337402 (1.9740233421325684) | > loss_feat: 3.6623356342315674 (3.834019660949707) | > loss_mel: 16.31450843811035 (16.65767631530762) | > loss_duration: 1.5486088991165161 (1.5638872861862183) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.562519073486328 (26.2106071472168) | > grad_norm_1: tensor(14976.5264, device='cuda:0') (tensor(10318.0273, device='cuda:0')) | > current_lr_0: 0.0001775602394331298 | > current_lr_1: 0.0001775602394331298 | > step_time: 0.4337 (0.4341588497161865) | > loader_time: 0.0063 (0.007626914978027343)  --> TIME: 2025-02-25 18:02:01 -- STEP: 55/72 -- GLOBAL_STEP: 1068600 | > loss_disc: 2.5736494064331055 (2.532364524494517) | > loss_disc_real_0: 0.18347877264022827 (0.09283434802835638) | > loss_disc_real_1: 0.20893849432468414 (0.22001599967479707) | > loss_disc_real_2: 0.2271983027458191 (0.22984789420257914) | > loss_disc_real_3: 0.23190529644489288 (0.23216270885684273) | > loss_disc_real_4: 0.23100194334983826 (0.24093201810663398) | > loss_disc_real_5: 0.2465878427028656 (0.24499552033164285) | > loss_0: 2.5736494064331055 (2.532364524494517) | > grad_norm_0: tensor(602.7629, device='cuda:0') (tensor(316.1935, device='cuda:0')) | > loss_gen: 2.2544569969177246 (2.1934316049922593) | > loss_kl: 2.0392677783966064 (1.9222881078720093) | > loss_feat: 4.001692295074463 (3.8521613294428043) | > loss_mel: 16.426401138305664 (16.66398565118964) | > loss_duration: 1.618762731552124 (1.5687892978841609) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.3405818939209 (26.200655781139027) | > grad_norm_1: tensor(7550.4160, device='cuda:0') (tensor(9846.2598, device='cuda:0')) | > current_lr_0: 0.0001775602394331298 | > current_lr_1: 0.0001775602394331298 | > step_time: 0.4622 (0.4458117615092884) | > loader_time: 0.0093 (0.007972197099165482) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684947457112053 (+3.7119422160400775e-05) | > avg_step_time: 0.4567468602892379 (-0.0028098469049158137) | > avg_loss_disc: 2.536790149312623 (-0.060457897857881804) | > avg_loss_disc_real_0: 0.09319657013869621 (+0.010298741640339434) | > avg_loss_disc_real_1: 0.22058243713748288 (+0.0014360719163653002) | > avg_loss_disc_real_2: 0.2294469309944502 (-0.008757320718026484) | > avg_loss_disc_real_3: 0.232586113709799 (-0.00027026340995034426) | > avg_loss_disc_real_4: 0.24242045279120056 (-0.0014774215893006704) | > avg_loss_disc_real_5: 0.24668723491715713 (+0.0035136824762317387) | > avg_loss_0: 2.536790149312623 (-0.060457897857881804) | > avg_grad_norm_0: tensor(310.1557, device='cuda:0') (+tensor(9.8358, device='cuda:0')) | > avg_loss_gen: 2.1875836261561217 (+0.1252357422466006) | > avg_loss_kl: 1.956330615030208 (-0.06225932819742441) | > avg_loss_feat: 3.803760283429858 (+0.0937315108070913) | > avg_loss_mel: 16.679619587643053 (-0.14703676734171367) | > avg_loss_duration: 1.579747198333203 (-0.016432911577359155) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.207041135976013 (-0.006761927000233925) | > avg_grad_norm_1: tensor(9629.7695, device='cuda:0') (tensor(-288.6650, device='cuda:0'))  > EPOCH: 953/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:02:11)   --> TIME: 2025-02-25 18:02:28 -- STEP: 33/72 -- GLOBAL_STEP: 1068650 | > loss_disc: 2.473602056503296 (2.55231441151012) | > loss_disc_real_0: 0.08996416628360748 (0.11585421284491365) | > loss_disc_real_1: 0.2414478063583374 (0.2160925535541592) | > loss_disc_real_2: 0.22171403467655182 (0.22672656494559665) | > loss_disc_real_3: 0.23615221679210663 (0.23049413886937228) | > loss_disc_real_4: 0.23210661113262177 (0.2385621400493564) | > loss_disc_real_5: 0.2445298731327057 (0.24107338081706653) | > loss_0: 2.473602056503296 (2.55231441151012) | > grad_norm_0: tensor(269.2268, device='cuda:0') (tensor(279.0141, device='cuda:0')) | > loss_gen: 2.2705845832824707 (2.2110412698803525) | > loss_kl: 1.862841248512268 (1.9083550482085256) | > loss_feat: 4.074934005737305 (3.927808898867983) | > loss_mel: 16.327285766601562 (16.804761915495902) | > loss_duration: 1.540421962738037 (1.5280743584488374) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.076066970825195 (26.380041469227184) | > grad_norm_1: tensor(7867.7139, device='cuda:0') (tensor(8142.0439, device='cuda:0')) | > current_lr_0: 0.00017753804440320063 | > current_lr_1: 0.00017753804440320063 | > step_time: 0.4707 (0.438391468741677) | > loader_time: 0.008 (0.007030154719497218) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547205320546328 (-0.00013774213656572547) | > avg_step_time: 0.45667865578557404 (-6.8204503663849e-05) | > avg_loss_disc: 2.562169363800908 (+0.02537921448828495) | > avg_loss_disc_real_0: 0.10214888052621356 (+0.008952310387517343) | > avg_loss_disc_real_1: 0.2188975775325802 (-0.0016848596049026665) | > avg_loss_disc_real_2: 0.23706013383999677 (+0.007613202845546563) | > avg_loss_disc_real_3: 0.2332166743950105 (+0.0006305606852115009) | > avg_loss_disc_real_4: 0.24471972588921936 (+0.002299273098018806) | > avg_loss_disc_real_5: 0.2451074180048956 (-0.001579816912261528) | > avg_loss_0: 2.562169363800908 (+0.02537921448828495) | > avg_grad_norm_0: tensor(280.3057, device='cuda:0') (tensor(-29.8500, device='cuda:0')) | > avg_loss_gen: 2.206580130147263 (+0.018996503991141278) | > avg_loss_kl: 1.9675093486275472 (+0.01117873359733923) | > avg_loss_feat: 3.8893184460384744 (+0.0855581626086166) | > avg_loss_mel: 16.718670401774656 (+0.03905081413160261) | > avg_loss_duration: 1.5621651192785988 (-0.017582079054604183) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.344243224238006 (+0.13720208826199354) | > avg_grad_norm_1: tensor(8384.4346, device='cuda:0') (tensor(-1245.3350, device='cuda:0'))  > EPOCH: 954/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:02:48)   --> TIME: 2025-02-25 18:02:56 -- STEP: 11/72 -- GLOBAL_STEP: 1068700 | > loss_disc: 2.6096527576446533 (2.587975653735074) | > loss_disc_real_0: 0.06410305202007294 (0.1265549903566187) | > loss_disc_real_1: 0.2270076423883438 (0.22167649458755145) | > loss_disc_real_2: 0.2430574595928192 (0.2333282232284546) | > loss_disc_real_3: 0.24746644496917725 (0.23975549367341129) | > loss_disc_real_4: 0.2533521056175232 (0.23021932759068228) | > loss_disc_real_5: 0.2573997378349304 (0.2387790558013049) | > loss_0: 2.6096527576446533 (2.587975653735074) | > grad_norm_0: tensor(209.0641, device='cuda:0') (tensor(346.5698, device='cuda:0')) | > loss_gen: 2.203636884689331 (2.1940536282279273) | > loss_kl: 1.948530912399292 (1.8349425250833684) | > loss_feat: 3.927999973297119 (3.9537108594721015) | > loss_mel: 17.764741897583008 (16.916758797385476) | > loss_duration: 1.5139117240905762 (1.5198361331766301) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.358821868896484 (26.419302160089668) | > grad_norm_1: tensor(11345.1602, device='cuda:0') (tensor(8132.5498, device='cuda:0')) | > current_lr_0: 0.00017751585214765024 | > current_lr_1: 0.00017751585214765024 | > step_time: 0.4272 (0.43670814687555487) | > loader_time: 0.0057 (0.006453752517700195)  --> TIME: 2025-02-25 18:03:19 -- STEP: 61/72 -- GLOBAL_STEP: 1068750 | > loss_disc: 2.5153567790985107 (2.572403243330659) | > loss_disc_real_0: 0.2445797324180603 (0.11470038425482687) | > loss_disc_real_1: 0.2480345070362091 (0.2210544579830326) | > loss_disc_real_2: 0.22543752193450928 (0.23286043254078412) | > loss_disc_real_3: 0.22341132164001465 (0.23437527945784273) | > loss_disc_real_4: 0.23069876432418823 (0.24087954373633275) | > loss_disc_real_5: 0.23380233347415924 (0.24623614647349373) | > loss_0: 2.5153567790985107 (2.572403243330659) | > grad_norm_0: tensor(589.5775, device='cuda:0') (tensor(296.9551, device='cuda:0')) | > loss_gen: 2.54443097114563 (2.195463553803865) | > loss_kl: 2.029308795928955 (2.0181081646778543) | > loss_feat: 4.012690544128418 (3.8307785753343926) | > loss_mel: 17.33392906188965 (16.850270271301266) | > loss_duration: 1.60287606716156 (1.5484930159615689) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.523237228393555 (26.44311360843846) | > grad_norm_1: tensor(3985.6248, device='cuda:0') (tensor(8042.9487, device='cuda:0')) | > current_lr_0: 0.00017751585214765024 | > current_lr_1: 0.00017751585214765024 | > step_time: 0.4592 (0.4496823920578253) | > loader_time: 0.0106 (0.008152867926925912) --> EVAL PERFORMANCE | > avg_loader_time: 0.008615806069172607 (+6.860074862627928e-05) | > avg_step_time: 0.4591815538809333 (+0.0025028980953592517) | > avg_loss_disc: 2.595617888678968 (+0.03344852487806005) | > avg_loss_disc_real_0: 0.11642387745455957 (+0.01427499692834601) | > avg_loss_disc_real_1: 0.21880581508010205 (-9.176245247816128e-05) | > avg_loss_disc_real_2: 0.23161526087304235 (-0.005444872966954423) | > avg_loss_disc_real_3: 0.23435282728201906 (+0.0011361528870085513) | > avg_loss_disc_real_4: 0.2421398185928103 (-0.0025799072964090564) | > avg_loss_disc_real_5: 0.2469981033617342 (+0.0018906853568385895) | > avg_loss_0: 2.595617888678968 (+0.03344852487806005) | > avg_grad_norm_0: tensor(287.9429, device='cuda:0') (+tensor(7.6372, device='cuda:0')) | > avg_loss_gen: 2.156644943734289 (-0.04993518641297401) | > avg_loss_kl: 2.0410483471104794 (+0.07353899848293222) | > avg_loss_feat: 3.7799809852116546 (-0.10933746082681983) | > avg_loss_mel: 16.91159349092295 (+0.19292308914829448) | > avg_loss_duration: 1.5574673884351489 (-0.004697730843449932) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.44673519403162 (+0.10249196979361486) | > avg_grad_norm_1: tensor(8289.6543, device='cuda:0') (tensor(-94.7803, device='cuda:0'))  > EPOCH: 955/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:03:26)   --> TIME: 2025-02-25 18:03:45 -- STEP: 39/72 -- GLOBAL_STEP: 1068800 | > loss_disc: 2.408168077468872 (2.7693760822980833) | > loss_disc_real_0: 0.131463423371315 (0.14756893452543476) | > loss_disc_real_1: 0.2130574733018875 (0.22073801817038122) | > loss_disc_real_2: 0.25221186876296997 (0.23081869116196266) | > loss_disc_real_3: 0.20863601565361023 (0.22483180883603218) | > loss_disc_real_4: 0.269725501537323 (0.23984763255486122) | > loss_disc_real_5: 0.26440852880477905 (0.2388956627020469) | > loss_0: 2.408168077468872 (2.7693760822980833) | > grad_norm_0: tensor(79.2682, device='cuda:0') (tensor(231.3559, device='cuda:0')) | > loss_gen: 2.501460075378418 (1.9894425899554522) | > loss_kl: 2.2570953369140625 (2.1480115896616225) | > loss_feat: 4.247615337371826 (3.5237422600770607) | > loss_mel: 17.45542335510254 (17.453869501749676) | > loss_duration: 1.5760588645935059 (1.5352661640216143) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.03765296936035 (26.65033213297526) | > grad_norm_1: tensor(2017.2318, device='cuda:0') (tensor(6823.8960, device='cuda:0')) | > current_lr_0: 0.00017749366266613177 | > current_lr_1: 0.00017749366266613177 | > step_time: 0.4618 (0.43805051461244243) | > loader_time: 0.008 (0.007326052739070012) --> EVAL PERFORMANCE | > avg_loader_time: 0.008712143965170414 (+9.633789599780736e-05) | > avg_step_time: 0.4555018787652674 (-0.0036796751156659147) | > avg_loss_disc: 2.717354918869448 (+0.12173703019047988) | > avg_loss_disc_real_0: 0.14628091685368977 (+0.029857039399130203) | > avg_loss_disc_real_1: 0.2295727488440527 (+0.010766933763950659) | > avg_loss_disc_real_2: 0.2297806139563171 (-0.0018346469167252466) | > avg_loss_disc_real_3: 0.23001735646959762 (-0.004335470812421444) | > avg_loss_disc_real_4: 0.2383721953546497 (-0.0037676232381606) | > avg_loss_disc_real_5: 0.23862660497846738 (-0.008371498383266812) | > avg_loss_0: 2.717354918869448 (+0.12173703019047988) | > avg_grad_norm_0: tensor(219.8369, device='cuda:0') (tensor(-68.1060, device='cuda:0')) | > avg_loss_gen: 2.027420287400903 (-0.1292246563333861) | > avg_loss_kl: 2.414327189955913 (+0.37327884284543345) | > avg_loss_feat: 3.5409947449052837 (-0.23898624030637095) | > avg_loss_mel: 18.380742274539564 (+1.469148783616614) | > avg_loss_duration: 1.57635755774001 (+0.018890169304861093) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.9398420629367 (+1.4931068689050804) | > avg_grad_norm_1: tensor(6374.8926, device='cuda:0') (tensor(-1914.7617, device='cuda:0'))  > EPOCH: 956/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:04:03)   --> TIME: 2025-02-25 18:04:13 -- STEP: 17/72 -- GLOBAL_STEP: 1068850 | > loss_disc: 2.2814202308654785 (2.5335567278020523) | > loss_disc_real_0: 0.07442347705364227 (0.12364846117356244) | > loss_disc_real_1: 0.2077624648809433 (0.23373965042478898) | > loss_disc_real_2: 0.25104406476020813 (0.23626298413557165) | > loss_disc_real_3: 0.18065191805362701 (0.2183817090357051) | > loss_disc_real_4: 0.21216347813606262 (0.2418789241243811) | > loss_disc_real_5: 0.20087921619415283 (0.23957116025335648) | > loss_0: 2.2814202308654785 (2.5335567278020523) | > grad_norm_0: tensor(26.3654, device='cuda:0') (tensor(221.9270, device='cuda:0')) | > loss_gen: 2.623924732208252 (2.278070323607501) | > loss_kl: 2.2513513565063477 (2.147287733414594) | > loss_feat: 5.028959274291992 (3.9687672082115624) | > loss_mel: 20.149036407470703 (18.20926980411305) | > loss_duration: 1.5689078569412231 (1.5369418719235588) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.622180938720703 (28.140336878159466) | > grad_norm_1: tensor(4875.0034, device='cuda:0') (tensor(9555.9404, device='cuda:0')) | > current_lr_0: 0.0001774714759582985 | > current_lr_1: 0.0001774714759582985 | > step_time: 0.4354 (0.4322476246777703) | > loader_time: 0.0061 (0.006845670587876264)  --> TIME: 2025-02-25 18:04:36 -- STEP: 67/72 -- GLOBAL_STEP: 1068900 | > loss_disc: 2.684685707092285 (2.61090263679846) | > loss_disc_real_0: 0.07249602675437927 (0.12386905173979583) | > loss_disc_real_1: 0.22716811299324036 (0.22396051961539395) | > loss_disc_real_2: 0.23893119394779205 (0.25415407388067945) | > loss_disc_real_3: 0.2339036911725998 (0.23661128598362652) | > loss_disc_real_4: 0.2561285197734833 (0.2447650750626379) | > loss_disc_real_5: 0.25300440192222595 (0.24542592755004541) | > loss_0: 2.684685707092285 (2.61090263679846) | > grad_norm_0: tensor(61.8645, device='cuda:0') (tensor(245.8748, device='cuda:0')) | > loss_gen: 1.9237205982208252 (2.182765827250124) | > loss_kl: 2.309091329574585 (2.2131891659836262) | > loss_feat: 3.148367166519165 (3.7136002796799388) | > loss_mel: 17.57852554321289 (18.162629967305193) | > loss_duration: 1.6649694442749023 (1.583981572692074) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.624675750732422 (27.85616683959961) | > grad_norm_1: tensor(8386.2129, device='cuda:0') (tensor(7752.4160, device='cuda:0')) | > current_lr_0: 0.0001774714759582985 | > current_lr_1: 0.0001774714759582985 | > step_time: 0.5243 (0.45057294617837934) | > loader_time: 0.0121 (0.008487737000878176) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671948607538785 (-4.0195357631628986e-05) | > avg_step_time: 0.4550036007249859 (-0.0004982780402814635) | > avg_loss_disc: 2.6056411400647232 (-0.11171377880472466) | > avg_loss_disc_real_0: 0.12500326141295293 (-0.021277655440736842) | > avg_loss_disc_real_1: 0.2231411796430467 (-0.0064315692010060155) | > avg_loss_disc_real_2: 0.2535855314681227 (+0.02380491751180558) | > avg_loss_disc_real_3: 0.2370212520931808 (+0.007003895623583195) | > avg_loss_disc_real_4: 0.24505034403901704 (+0.0066781486843673354) | > avg_loss_disc_real_5: 0.2459931381991212 (+0.007366533220653809) | > avg_loss_0: 2.6056411400647232 (-0.11171377880472466) | > avg_grad_norm_0: tensor(240.7591, device='cuda:0') (+tensor(20.9222, device='cuda:0')) | > avg_loss_gen: 2.193630320925107 (+0.16621003352420427) | > avg_loss_kl: 2.2155524593004032 (-0.19877473065550966) | > avg_loss_feat: 3.714758473382869 (+0.17376372847758548) | > avg_loss_mel: 18.106124246624148 (-0.274618027915416) | > avg_loss_duration: 1.586106412847277 (+0.009748855107267085) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.816171968486948 (-0.12367009444975352) | > avg_grad_norm_1: tensor(7758.4868, device='cuda:0') (+tensor(1383.5942, device='cuda:0'))  > EPOCH: 957/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:04:40)   --> TIME: 2025-02-25 18:05:02 -- STEP: 45/72 -- GLOBAL_STEP: 1068950 | > loss_disc: 2.5284199714660645 (2.512596458858914) | > loss_disc_real_0: 0.10946439206600189 (0.08804867164128356) | > loss_disc_real_1: 0.23337602615356445 (0.2176639394627677) | > loss_disc_real_2: 0.2493457943201065 (0.2302295764287313) | > loss_disc_real_3: 0.25488054752349854 (0.2297744611899058) | > loss_disc_real_4: 0.2530006170272827 (0.23987852036952972) | > loss_disc_real_5: 0.2589321732521057 (0.2450815654463238) | > loss_0: 2.5284199714660645 (2.512596458858914) | > grad_norm_0: tensor(455.4658, device='cuda:0') (tensor(264.1791, device='cuda:0')) | > loss_gen: 2.3605213165283203 (2.22019673453437) | > loss_kl: 1.9697096347808838 (1.9873601012759738) | > loss_feat: 4.388428211212158 (3.9685835679372152) | > loss_mel: 17.26203155517578 (16.983083385891383) | > loss_duration: 1.5772804021835327 (1.541733855671353) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.557971954345703 (26.700957828097874) | > grad_norm_1: tensor(7304.3022, device='cuda:0') (tensor(9951.5264, device='cuda:0')) | > current_lr_0: 0.0001774492920238037 | > current_lr_1: 0.0001774492920238037 | > step_time: 0.461 (0.4408725738525391) | > loader_time: 0.0088 (0.007544623480902778) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860691406357456 (-6.503454396422463e-05) | > avg_step_time: 0.4566625004083338 (+0.0016588996833478697) | > avg_loss_disc: 2.5206765322618083 (-0.08496460780291493) | > avg_loss_disc_real_0: 0.08851420519952201 (-0.03648905621343092) | > avg_loss_disc_real_1: 0.21834194975000032 (-0.004799229893046375) | > avg_loss_disc_real_2: 0.23119386296037217 (-0.022391668507750512) | > avg_loss_disc_real_3: 0.23013659661084834 (-0.006884655482332469) | > avg_loss_disc_real_4: 0.24051699587996578 (-0.004533348159051265) | > avg_loss_disc_real_5: 0.24552560839014995 (-0.0004675298089712454) | > avg_loss_0: 2.5206765322618083 (-0.08496460780291493) | > avg_grad_norm_0: tensor(261.1072, device='cuda:0') (+tensor(20.3481, device='cuda:0')) | > avg_loss_gen: 2.2077058973446704 (+0.014075576419563252) | > avg_loss_kl: 2.0188856846849683 (-0.19666677461543491) | > avg_loss_feat: 3.8839659522956524 (+0.16920747891278332) | > avg_loss_mel: 16.825073618284407 (-1.2810506283397416) | > avg_loss_duration: 1.562967710092034 (-0.02313870275524299) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.498598985269037 (-1.3175729832179108) | > avg_grad_norm_1: tensor(9429.9043, device='cuda:0') (+tensor(1671.4175, device='cuda:0'))  > EPOCH: 958/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:05:17)   --> TIME: 2025-02-25 18:05:30 -- STEP: 23/72 -- GLOBAL_STEP: 1069000 | > loss_disc: 2.923780679702759 (2.593610670255578) | > loss_disc_real_0: 0.07706868648529053 (0.1203561616332635) | > loss_disc_real_1: 0.05075782909989357 (0.21731522663132005) | > loss_disc_real_2: 0.22230198979377747 (0.23338890593984854) | > loss_disc_real_3: 0.13059724867343903 (0.2309579648401426) | > loss_disc_real_4: 0.16165880858898163 (0.24210403794827667) | > loss_disc_real_5: 0.22597745060920715 (0.24578602935956873) | > loss_0: 2.923780679702759 (2.593610670255578) | > grad_norm_0: tensor(58.1489, device='cuda:0') (tensor(300.4229, device='cuda:0')) | > loss_gen: 1.9744529724121094 (2.203770570133043) | > loss_kl: 2.0114874839782715 (1.91881959852965) | > loss_feat: 4.2788262367248535 (3.8665483412535293) | > loss_mel: 16.29391098022461 (16.905548137167226) | > loss_duration: 1.5453357696533203 (1.5113799882971721) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.104013442993164 (26.406066562818445) | > grad_norm_1: tensor(8603.2988, device='cuda:0') (tensor(7958.8457, device='cuda:0')) | > current_lr_0: 0.00017742711086230072 | > current_lr_1: 0.00017742711086230072 | > step_time: 0.4479 (0.43879238418910815) | > loader_time: 0.0081 (0.007026848585709282) --> EVAL PERFORMANCE | > avg_loader_time: 0.008682116656236248 (+7.5202592661687e-05) | > avg_step_time: 0.4581813845835941 (+0.0015188841752603044) | > avg_loss_disc: 2.718462991042875 (+0.1977864587810667) | > avg_loss_disc_real_0: 0.18601537779183458 (+0.09750117259231257) | > avg_loss_disc_real_1: 0.2735195490990725 (+0.05517759934907218) | > avg_loss_disc_real_2: 0.2334624535181153 (+0.0022685905577431165) | > avg_loss_disc_real_3: 0.23937833728924604 (+0.009241740678397697) | > avg_loss_disc_real_4: 0.24554312627919964 (+0.005026130399233858) | > avg_loss_disc_real_5: 0.24762618709617937 (+0.002100578706029421) | > avg_loss_0: 2.718462991042875 (+0.1977864587810667) | > avg_grad_norm_0: tensor(300.9242, device='cuda:0') (+tensor(39.8170, device='cuda:0')) | > avg_loss_gen: 2.1474218989761784 (-0.060283998368491964) | > avg_loss_kl: 1.9861164714249087 (-0.03276921326005966) | > avg_loss_feat: 3.38509442436863 (-0.4988715279270224) | > avg_loss_mel: 16.671553678915533 (-0.15351993936887354) | > avg_loss_duration: 1.554649274114152 (-0.008318435977882066) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.74483559836804 (-0.7537633869009959) | > avg_grad_norm_1: tensor(7350.3999, device='cuda:0') (tensor(-2079.5044, device='cuda:0'))  > EPOCH: 959/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:05:54)   --> TIME: 2025-02-25 18:05:57 -- STEP: 1/72 -- GLOBAL_STEP: 1069050 | > loss_disc: 2.651175022125244 (2.651175022125244) | > loss_disc_real_0: 0.04673737287521362 (0.04673737287521362) | > loss_disc_real_1: 0.20687468349933624 (0.20687468349933624) | > loss_disc_real_2: 0.2271144539117813 (0.2271144539117813) | > loss_disc_real_3: 0.1972094625234604 (0.1972094625234604) | > loss_disc_real_4: 0.22853325307369232 (0.22853325307369232) | > loss_disc_real_5: 0.23171357810497284 (0.23171357810497284) | > loss_0: 2.651175022125244 (2.651175022125244) | > grad_norm_0: tensor(364.0363, device='cuda:0') (tensor(364.0363, device='cuda:0')) | > loss_gen: 1.6420929431915283 (1.6420929431915283) | > loss_kl: 1.8389917612075806 (1.8389917612075806) | > loss_feat: 3.356125831604004 (3.356125831604004) | > loss_mel: 17.470308303833008 (17.470308303833008) | > loss_duration: 1.5027203559875488 (1.5027203559875488) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.810239791870117 (25.810239791870117) | > grad_norm_1: tensor(7810.8687, device='cuda:0') (tensor(7810.8687, device='cuda:0')) | > current_lr_0: 0.00017740493247344293 | > current_lr_1: 0.00017740493247344293 | > step_time: 0.4491 (0.4490981101989746) | > loader_time: 0.0057 (0.00574183464050293)  --> TIME: 2025-02-25 18:06:20 -- STEP: 51/72 -- GLOBAL_STEP: 1069100 | > loss_disc: 2.8379435539245605 (2.6514431121302584) | > loss_disc_real_0: 0.2459295392036438 (0.15487407878333448) | > loss_disc_real_1: 0.23655271530151367 (0.22346199201602562) | > loss_disc_real_2: 0.2465958148241043 (0.2439251087752043) | > loss_disc_real_3: 0.17054876685142517 (0.23522361587075627) | > loss_disc_real_4: 0.18188218772411346 (0.24556677832322962) | > loss_disc_real_5: 0.25116246938705444 (0.24721158807184182) | > loss_0: 2.8379435539245605 (2.6514431121302584) | > grad_norm_0: tensor(161.1924, device='cuda:0') (tensor(202.8535, device='cuda:0')) | > loss_gen: 1.6861504316329956 (2.1167050319559433) | > loss_kl: 2.4674413204193115 (1.9889299565670537) | > loss_feat: 2.9899256229400635 (3.5057309375089756) | > loss_mel: 17.08637046813965 (16.93633212295233) | > loss_duration: 1.8573896884918213 (1.5552351054023295) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.087276458740234 (26.10293317308613) | > grad_norm_1: tensor(2389.9695, device='cuda:0') (tensor(5541.1919, device='cuda:0')) | > current_lr_0: 0.00017740493247344293 | > current_lr_1: 0.00017740493247344293 | > step_time: 0.4504 (0.44535427934983196) | > loader_time: 0.009 (0.007801532745361328) --> EVAL PERFORMANCE | > avg_loader_time: 0.008645588243511361 (-3.6528412724886467e-05) | > avg_step_time: 0.45494242117438516 (-0.003238963409208928) | > avg_loss_disc: 2.6460541570690315 (-0.07240883397384357) | > avg_loss_disc_real_0: 0.13658259331550396 (-0.04943278447633062) | > avg_loss_disc_real_1: 0.2280702509090934 (-0.04544929818997909) | > avg_loss_disc_real_2: 0.2459348984797236 (+0.01247244496160832) | > avg_loss_disc_real_3: 0.2378913443273222 (-0.0014869929619238387) | > avg_loss_disc_real_4: 0.2459753495706639 (+0.000432223291464251) | > avg_loss_disc_real_5: 0.2448208621689971 (-0.0028053249271822733) | > avg_loss_0: 2.6460541570690315 (-0.07240883397384357) | > avg_grad_norm_0: tensor(222.0859, device='cuda:0') (tensor(-78.8383, device='cuda:0')) | > avg_loss_gen: 2.1192796767597466 (-0.02814222221643181) | > avg_loss_kl: 2.077467305559508 (+0.09135083413459921) | > avg_loss_feat: 3.5608580716898746 (+0.1757636473212445) | > avg_loss_mel: 17.083717373055464 (+0.4121636941399309) | > avg_loss_duration: 1.5800375501874466 (+0.025388276073294636) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42135996214101 (+0.6765243637729697) | > avg_grad_norm_1: tensor(5506.5386, device='cuda:0') (tensor(-1843.8613, device='cuda:0'))  > EPOCH: 960/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:06:31)   --> TIME: 2025-02-25 18:06:46 -- STEP: 29/72 -- GLOBAL_STEP: 1069150 | > loss_disc: 2.563720464706421 (2.672679407843228) | > loss_disc_real_0: 0.12533050775527954 (0.16012066808240163) | > loss_disc_real_1: 0.22985272109508514 (0.22343318472648488) | > loss_disc_real_2: 0.26834824681282043 (0.22942745171744247) | > loss_disc_real_3: 0.26725438237190247 (0.22611994270620675) | > loss_disc_real_4: 0.2624408006668091 (0.2441634052786334) | > loss_disc_real_5: 0.2747001647949219 (0.24182142471445017) | > loss_0: 2.563720464706421 (2.672679407843228) | > grad_norm_0: tensor(38.8851, device='cuda:0') (tensor(181.3073, device='cuda:0')) | > loss_gen: 2.3162221908569336 (2.0041162885468586) | > loss_kl: 1.8646026849746704 (1.9681118891156952) | > loss_feat: 4.054769992828369 (3.504599530121376) | > loss_mel: 17.05899429321289 (17.256962611757476) | > loss_duration: 1.4952634572982788 (1.5209200669979226) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.789854049682617 (26.25471023033405) | > grad_norm_1: tensor(7245.0859, device='cuda:0') (tensor(7129.8857, device='cuda:0')) | > current_lr_0: 0.00017738275685688374 | > current_lr_1: 0.00017738275685688374 | > step_time: 0.454 (0.4398248524501406) | > loader_time: 0.0075 (0.007151003541617558) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687412235098826 (+4.1823991587464776e-05) | > avg_step_time: 0.4580818229997662 (+0.0031394018253810274) | > avg_loss_disc: 2.720672489891589 (+0.07461833282255759) | > avg_loss_disc_real_0: 0.13672255577755646 (+0.00013996246205250085) | > avg_loss_disc_real_1: 0.24201631315157446 (+0.013946062242481044) | > avg_loss_disc_real_2: 0.2550385963748878 (+0.009103697895164187) | > avg_loss_disc_real_3: 0.2333097237516457 (-0.004581620575676515) | > avg_loss_disc_real_4: 0.24884268704434515 (+0.0028673374736812585) | > avg_loss_disc_real_5: 0.24710019499483243 (+0.0022793328258353396) | > avg_loss_0: 2.720672489891589 (+0.07461833282255759) | > avg_grad_norm_0: tensor(192.0430, device='cuda:0') (tensor(-30.0429, device='cuda:0')) | > avg_loss_gen: 2.011403404491049 (-0.10787627226869745) | > avg_loss_kl: 2.1660138704407377 (+0.0885465648812298) | > avg_loss_feat: 3.4189486369280746 (-0.14190943476179996) | > avg_loss_mel: 17.525208956758735 (+0.4414915837032716) | > avg_loss_duration: 1.5646338210978978 (-0.01540372908954879) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.68620864438339 (+0.26484868224238056) | > avg_grad_norm_1: tensor(6491.4219, device='cuda:0') (+tensor(984.8833, device='cuda:0'))  > EPOCH: 961/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:07:09)   --> TIME: 2025-02-25 18:07:14 -- STEP: 7/72 -- GLOBAL_STEP: 1069200 | > loss_disc: 2.5345466136932373 (2.5469599791935513) | > loss_disc_real_0: 0.17980077862739563 (0.1184314393571445) | > loss_disc_real_1: 0.23508678376674652 (0.2004158922604152) | > loss_disc_real_2: 0.19435307383537292 (0.21540938317775726) | > loss_disc_real_3: 0.20987240970134735 (0.2289783103125436) | > loss_disc_real_4: 0.24221932888031006 (0.2595319471188954) | > loss_disc_real_5: 0.2619457542896271 (0.2695858691419874) | > loss_0: 2.5345466136932373 (2.5469599791935513) | > grad_norm_0: tensor(62.8900, device='cuda:0') (tensor(119.7695, device='cuda:0')) | > loss_gen: 2.4079337120056152 (2.185121604374477) | > loss_kl: 2.8874242305755615 (3.325571434838431) | > loss_feat: 3.846390724182129 (3.9968981742858887) | > loss_mel: 23.000160217285156 (24.308345794677734) | > loss_duration: 1.5825026035308838 (1.5950220482689994) | > amp_scaler: 8.0 (8.0) | > loss_1: 33.72441101074219 (35.41095897129604) | > grad_norm_1: tensor(8214.6836, device='cuda:0') (tensor(6317.4731, device='cuda:0')) | > current_lr_0: 0.00017736058401227663 | > current_lr_1: 0.00017736058401227663 | > step_time: 0.4375 (0.4361316476549421) | > loader_time: 0.0056 (0.006333555494035993)  --> TIME: 2025-02-25 18:07:37 -- STEP: 57/72 -- GLOBAL_STEP: 1069250 | > loss_disc: 2.289147138595581 (2.588927009649445) | > loss_disc_real_0: 0.06569613516330719 (0.1411112208376851) | > loss_disc_real_1: 0.1841374933719635 (0.2274004200048614) | > loss_disc_real_2: 0.21614815294742584 (0.2233692533091495) | > loss_disc_real_3: 0.21867915987968445 (0.2291545365986071) | > loss_disc_real_4: 0.2055588960647583 (0.24062867253495937) | > loss_disc_real_5: 0.2214789241552353 (0.24791129952982846) | > loss_0: 2.289147138595581 (2.588927009649445) | > grad_norm_0: tensor(84.3360, device='cuda:0') (tensor(235.6606, device='cuda:0')) | > loss_gen: 2.4457204341888428 (2.166444485647636) | > loss_kl: 2.33756685256958 (2.567311119614986) | > loss_feat: 4.78263521194458 (3.8162316732239305) | > loss_mel: 17.74974250793457 (19.55501957943566) | > loss_duration: 1.61749267578125 (1.593724953500848) | > amp_scaler: 8.0 (8.701754385964911) | > loss_1: 28.93315887451172 (29.698731740315754) | > grad_norm_1: tensor(4587.7886, device='cuda:0') (tensor(7341.0859, device='cuda:0')) | > current_lr_0: 0.00017736058401227663 | > current_lr_1: 0.00017736058401227663 | > step_time: 0.4652 (0.44827046310692503) | > loader_time: 0.0095 (0.00792645989802846) --> EVAL PERFORMANCE | > avg_loader_time: 0.008603821338062555 (-8.359089703627136e-05) | > avg_step_time: 0.4597705686596078 (+0.0016887456598416173) | > avg_loss_disc: 2.6027921085626313 (-0.11788038132895773) | > avg_loss_disc_real_0: 0.13329299046120174 (-0.0034295653163547157) | > avg_loss_disc_real_1: 0.22999255149297312 (-0.012023761658601334) | > avg_loss_disc_real_2: 0.22692573343364286 (-0.02811286294124493) | > avg_loss_disc_real_3: 0.23152668102526328 (-0.0017830427263824111) | > avg_loss_disc_real_4: 0.24241117629366862 (-0.006431510750676522) | > avg_loss_disc_real_5: 0.24775597264229407 (+0.0006557776474616361) | > avg_loss_0: 2.6027921085626313 (-0.11788038132895773) | > avg_grad_norm_0: tensor(231.6042, device='cuda:0') (+tensor(39.5613, device='cuda:0')) | > avg_loss_gen: 2.164923256551715 (+0.15351985206066576) | > avg_loss_kl: 2.5479328397294165 (+0.38191896928867886) | > avg_loss_feat: 3.814970375786365 (+0.3960217388582903) | > avg_loss_mel: 19.36250638290191 (+1.8372974261431736) | > avg_loss_duration: 1.6028817035782505 (+0.03824788248035271) | > avg_amp_scaler: 8.56338028169014 (+0.5633802816901401) | > avg_loss_1: 29.49321440575828 (+2.8070057613748887) | > avg_grad_norm_1: tensor(7508.9541, device='cuda:0') (+tensor(1017.5322, device='cuda:0'))  > EPOCH: 962/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:07:46)   --> TIME: 2025-02-25 18:08:04 -- STEP: 35/72 -- GLOBAL_STEP: 1069300 | > loss_disc: 2.601288318634033 (2.5429892335619253) | > loss_disc_real_0: 0.10762318968772888 (0.08293358609080315) | > loss_disc_real_1: 0.2682160437107086 (0.22171159854957034) | > loss_disc_real_2: 0.24862657487392426 (0.23293830028602056) | > loss_disc_real_3: 0.23803037405014038 (0.2308127373456955) | > loss_disc_real_4: 0.25670740008354187 (0.23825382930891856) | > loss_disc_real_5: 0.26624369621276855 (0.2501267509801048) | > loss_0: 2.601288318634033 (2.5429892335619253) | > grad_norm_0: tensor(395.7566, device='cuda:0') (tensor(254.4263, device='cuda:0')) | > loss_gen: 2.25175404548645 (2.183886296408517) | > loss_kl: 2.129166841506958 (2.0825505903788977) | > loss_feat: 3.5361733436584473 (3.9623646804264614) | > loss_mel: 17.627355575561523 (17.96874302455357) | > loss_duration: 1.5551199913024902 (1.5370110137122017) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.099571228027344 (27.734555871146068) | > grad_norm_1: tensor(10920.6074, device='cuda:0') (tensor(9827.3291, device='cuda:0')) | > current_lr_0: 0.00017733841393927508 | > current_lr_1: 0.00017733841393927508 | > step_time: 0.4576 (0.43988518714904784) | > loader_time: 0.0085 (0.00715705326625279) --> EVAL PERFORMANCE | > avg_loader_time: 0.008576077474674705 (-2.7743863387849274e-05) | > avg_step_time: 0.4594239651317328 (-0.00034660352787502235) | > avg_loss_disc: 2.540801108722955 (-0.06199099983967615) | > avg_loss_disc_real_0: 0.09121166605135085 (-0.04208132440985089) | > avg_loss_disc_real_1: 0.2215038481312738 (-0.008488703361699323) | > avg_loss_disc_real_2: 0.23213994209195526 (+0.0052142086583124025) | > avg_loss_disc_real_3: 0.23018584515846952 (-0.0013408358667937603) | > avg_loss_disc_real_4: 0.2415098526947935 (-0.0009013235988751256) | > avg_loss_disc_real_5: 0.24884005037831589 (+0.0010840777360218157) | > avg_loss_0: 2.540801108722955 (-0.06199099983967615) | > avg_grad_norm_0: tensor(267.3442, device='cuda:0') (+tensor(35.7399, device='cuda:0')) | > avg_loss_gen: 2.194888454088022 (+0.02996519753630711) | > avg_loss_kl: 2.075581303784545 (-0.47235153594487134) | > avg_loss_feat: 3.873293393094775 (+0.05832301730840994) | > avg_loss_mel: 17.62928057388521 (-1.733225809016698) | > avg_loss_duration: 1.5660316625111539 (-0.03685004106709666) | > avg_amp_scaler: 8.0 (-0.5633802816901401) | > avg_loss_1: 27.33907549146195 (-2.154138914296329) | > avg_grad_norm_1: tensor(9128.1025, device='cuda:0') (+tensor(1619.1484, device='cuda:0'))  > EPOCH: 963/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:08:24)   --> TIME: 2025-02-25 18:08:31 -- STEP: 13/72 -- GLOBAL_STEP: 1069350 | > loss_disc: 2.6859588623046875 (2.595493188271156) | > loss_disc_real_0: 0.09938228130340576 (0.0909343404838672) | > loss_disc_real_1: 0.12968997657299042 (0.2525400003561607) | > loss_disc_real_2: 0.23620203137397766 (0.23370448213357192) | > loss_disc_real_3: 0.2233453392982483 (0.2421226306603505) | > loss_disc_real_4: 0.24159975349903107 (0.24908625391813424) | > loss_disc_real_5: 0.2506346106529236 (0.2508748586361225) | > loss_0: 2.6859588623046875 (2.595493188271156) | > grad_norm_0: tensor(266.2238, device='cuda:0') (tensor(261.2789, device='cuda:0')) | > loss_gen: 1.935164213180542 (2.2051431949322042) | > loss_kl: 1.843470573425293 (1.9928537698892446) | > loss_feat: 3.757110595703125 (3.971115057285016) | > loss_mel: 15.38072395324707 (17.33860793480506) | > loss_duration: 1.4558593034744263 (1.5115553048940806) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.37232780456543 (27.019274931687576) | > grad_norm_1: tensor(7617.4849, device='cuda:0') (tensor(8150.9204, device='cuda:0')) | > current_lr_0: 0.00017731624663753267 | > current_lr_1: 0.00017731624663753267 | > step_time: 0.4522 (0.43746308180002064) | > loader_time: 0.0059 (0.006811912243182843)  --> TIME: 2025-02-25 18:08:55 -- STEP: 63/72 -- GLOBAL_STEP: 1069400 | > loss_disc: 2.6485817432403564 (2.5506454876491) | > loss_disc_real_0: 0.15725567936897278 (0.09907375122346575) | > loss_disc_real_1: 0.23541070520877838 (0.22494827092639982) | > loss_disc_real_2: 0.24896427989006042 (0.2357566538784239) | > loss_disc_real_3: 0.2508784234523773 (0.23306654606546676) | > loss_disc_real_4: 0.25581592321395874 (0.24150944063587795) | > loss_disc_real_5: 0.26423534750938416 (0.24774939247540065) | > loss_0: 2.6485817432403564 (2.5506454876491) | > grad_norm_0: tensor(461.0107, device='cuda:0') (tensor(276.5433, device='cuda:0')) | > loss_gen: 2.147658109664917 (2.205223198920962) | > loss_kl: 2.13313627243042 (1.9928357525477334) | > loss_feat: 3.4910025596618652 (3.836429747324141) | > loss_mel: 17.982627868652344 (16.94152132670084) | > loss_duration: 1.5756897926330566 (1.545970701035999) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.33011245727539 (26.52198073977516) | > grad_norm_1: tensor(9846.8584, device='cuda:0') (tensor(8103.6987, device='cuda:0')) | > current_lr_0: 0.00017731624663753267 | > current_lr_1: 0.00017731624663753267 | > step_time: 0.5099 (0.45140026107666986) | > loader_time: 0.0115 (0.008401745841616676) --> EVAL PERFORMANCE | > avg_loader_time: 0.008775737923635563 (+0.00019966044896085715) | > avg_step_time: 0.45853260873069224 (-0.0008913564010405439) | > avg_loss_disc: 2.5505978624585652 (+0.009796753735610064) | > avg_loss_disc_real_0: 0.09847703072386728 (+0.007265364672516428) | > avg_loss_disc_real_1: 0.2245027443892519 (+0.002998896257978112) | > avg_loss_disc_real_2: 0.235335702417602 (+0.0031957603256467237) | > avg_loss_disc_real_3: 0.23328027464974094 (+0.003094429491271422) | > avg_loss_disc_real_4: 0.24237860277504988 (+0.0008687500802563863) | > avg_loss_disc_real_5: 0.2485499344241451 (-0.0002901159541707776) | > avg_loss_0: 2.5505978624585652 (+0.009796753735610064) | > avg_grad_norm_0: tensor(272.3160, device='cuda:0') (+tensor(4.9718, device='cuda:0')) | > avg_loss_gen: 2.201546428908765 (+0.006657974820742929) | > avg_loss_kl: 2.0083349019708767 (-0.06724640181366848) | > avg_loss_feat: 3.815711471396433 (-0.057581921698341976) | > avg_loss_mel: 16.944465744663287 (-0.6848148292219243) | > avg_loss_duration: 1.554242835917943 (-0.0117888265932109) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.524301394610337 (-0.8147740968516146) | > avg_grad_norm_1: tensor(8039.7690, device='cuda:0') (tensor(-1088.3335, device='cuda:0'))  > EPOCH: 964/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:09:01)   --> TIME: 2025-02-25 18:09:22 -- STEP: 41/72 -- GLOBAL_STEP: 1069450 | > loss_disc: 3.012981653213501 (2.6183926652117475) | > loss_disc_real_0: 0.2206306904554367 (0.15204312734124134) | > loss_disc_real_1: 0.3046340048313141 (0.2445166067379277) | > loss_disc_real_2: 0.35395777225494385 (0.23464403937502606) | > loss_disc_real_3: 0.23048073053359985 (0.23209251681479012) | > loss_disc_real_4: 0.29163312911987305 (0.24300334584422228) | > loss_disc_real_5: 0.2713001072406769 (0.24545832414452623) | > loss_0: 3.012981653213501 (2.6183926652117475) | > grad_norm_0: tensor(391.6331, device='cuda:0') (tensor(271.7455, device='cuda:0')) | > loss_gen: 2.0660250186920166 (2.225120849725677) | > loss_kl: 2.458879232406616 (1.943161071800604) | > loss_feat: 3.2451703548431396 (3.81587688515826) | > loss_mel: 20.195655822753906 (17.08938091557201) | > loss_duration: 1.6992608308792114 (1.5271645493623687) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.66499137878418 (26.600704193115234) | > grad_norm_1: tensor(5317.9087, device='cuda:0') (tensor(6739.5225, device='cuda:0')) | > current_lr_0: 0.00017729408210670296 | > current_lr_1: 0.00017729408210670296 | > step_time: 0.4599 (0.4418238895695384) | > loader_time: 0.009 (0.007455994443195622) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864797914531869 (-0.00012775877831687314) | > avg_step_time: 0.4570234157669712 (-0.00150919296372104) | > avg_loss_disc: 2.671924527262298 (+0.12132666480373278) | > avg_loss_disc_real_0: 0.15206904063971943 (+0.053592009915852154) | > avg_loss_disc_real_1: 0.22928924197462244 (+0.0047864975853705305) | > avg_loss_disc_real_2: 0.24149632139105193 (+0.006160618973449938) | > avg_loss_disc_real_3: 0.2367858823756097 (+0.0035056077258687757) | > avg_loss_disc_real_4: 0.24798633955733876 (+0.005607736782288875) | > avg_loss_disc_real_5: 0.24528404965367115 (-0.003265884770473959) | > avg_loss_0: 2.671924527262298 (+0.12132666480373278) | > avg_grad_norm_0: tensor(271.9622, device='cuda:0') (tensor(-0.3539, device='cuda:0')) | > avg_loss_gen: 2.157264177228364 (-0.0442822516804009) | > avg_loss_kl: 2.145846623769949 (+0.13751172179907245) | > avg_loss_feat: 3.719749094734729 (-0.09596237666170371) | > avg_loss_mel: 17.62790521433656 (+0.6834394696732744) | > avg_loss_duration: 1.5876698342847153 (+0.03342699836677232) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.238434912453236 (+0.7141335178428996) | > avg_grad_norm_1: tensor(6581.9517, device='cuda:0') (tensor(-1457.8174, device='cuda:0'))  > EPOCH: 965/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:09:38)   --> TIME: 2025-02-25 18:09:49 -- STEP: 19/72 -- GLOBAL_STEP: 1069500 | > loss_disc: 2.7011380195617676 (2.8047663538079513) | > loss_disc_real_0: 0.10907919704914093 (0.1770209399493117) | > loss_disc_real_1: 0.22013597190380096 (0.27386340655778585) | > loss_disc_real_2: 0.23409563302993774 (0.257133465848471) | > loss_disc_real_3: 0.23026153445243835 (0.24243031050029554) | > loss_disc_real_4: 0.24727776646614075 (0.24235090456510844) | > loss_disc_real_5: 0.22948212921619415 (0.2359839662125236) | > loss_0: 2.7011380195617676 (2.8047663538079513) | > grad_norm_0: tensor(281.9723, device='cuda:0') (tensor(309.1226, device='cuda:0')) | > loss_gen: 1.7231225967407227 (2.1176269744571887) | > loss_kl: 2.039342164993286 (1.9945075198223716) | > loss_feat: 3.3299672603607178 (3.5032903646167957) | > loss_mel: 17.133317947387695 (17.200396889134453) | > loss_duration: 1.5140380859375 (1.5294037053459568) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.739788055419922 (26.345225283974095) | > grad_norm_1: tensor(12120.1855, device='cuda:0') (tensor(8027.0381, device='cuda:0')) | > current_lr_0: 0.0001772719203464396 | > current_lr_1: 0.0001772719203464396 | > step_time: 0.4365 (0.4361310130671451) | > loader_time: 0.0061 (0.006754561474448756)  --> TIME: 2025-02-25 18:10:13 -- STEP: 69/72 -- GLOBAL_STEP: 1069550 | > loss_disc: 2.599025011062622 (2.703160227208898) | > loss_disc_real_0: 0.24018146097660065 (0.1747672010915003) | > loss_disc_real_1: 0.19725637137889862 (0.23749738823676456) | > loss_disc_real_2: 0.2430756390094757 (0.24152374785879385) | > loss_disc_real_3: 0.2155405580997467 (0.23339262475138126) | > loss_disc_real_4: 0.2334343045949936 (0.2441809268101402) | > loss_disc_real_5: 0.23920893669128418 (0.24214572124723074) | > loss_0: 2.599025011062622 (2.703160227208898) | > grad_norm_0: tensor(488.4974, device='cuda:0') (tensor(279.5012, device='cuda:0')) | > loss_gen: 2.3167049884796143 (2.086671725563381) | > loss_kl: 2.28554105758667 (2.037931475086489) | > loss_feat: 4.100260257720947 (3.4334079044452612) | > loss_mel: 16.415952682495117 (16.90731695423955) | > loss_duration: 1.6521589756011963 (1.5585118894991667) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.770618438720703 (26.023840033489726) | > grad_norm_1: tensor(3398.9253, device='cuda:0') (tensor(6887.7534, device='cuda:0')) | > current_lr_0: 0.0001772719203464396 | > current_lr_1: 0.0001772719203464396 | > step_time: 0.5335 (0.45650453152863874) | > loader_time: 0.0129 (0.008537817692411118) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599261162986217 (-4.8717982332472515e-05) | > avg_step_time: 0.45822350072189116 (+0.0012000849549199666) | > avg_loss_disc: 2.702210651317113 (+0.030286124054815122) | > avg_loss_disc_real_0: 0.1733259038711098 (+0.021256863231390366) | > avg_loss_disc_real_1: 0.23622423347452998 (+0.0069349914999075335) | > avg_loss_disc_real_2: 0.24085511519035824 (-0.0006412062006936892) | > avg_loss_disc_real_3: 0.2336102740865358 (-0.0031756082890739) | > avg_loss_disc_real_4: 0.24413273116232645 (-0.003853608395012309) | > avg_loss_disc_real_5: 0.2418954221295639 (-0.0033886275241072628) | > avg_loss_0: 2.702210651317113 (+0.030286124054815122) | > avg_grad_norm_0: tensor(279.6061, device='cuda:0') (+tensor(7.6440, device='cuda:0')) | > avg_loss_gen: 2.083533918353873 (-0.07373025887449103) | > avg_loss_kl: 2.0412996335768367 (-0.10454699019311242) | > avg_loss_feat: 3.4356154824646428 (-0.2841336122700864) | > avg_loss_mel: 16.897822769594864 (-0.7300824447416971) | > avg_loss_duration: 1.5590524337661098 (-0.028617400518605507) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.01732431331151 (-1.2211105991417277) | > avg_grad_norm_1: tensor(6806.6060, device='cuda:0') (+tensor(224.6543, device='cuda:0'))  > EPOCH: 966/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:10:16)   --> TIME: 2025-02-25 18:10:39 -- STEP: 47/72 -- GLOBAL_STEP: 1069600 | > loss_disc: 2.626471996307373 (2.6793626724405493) | > loss_disc_real_0: 0.12964259088039398 (0.1633818309674872) | > loss_disc_real_1: 0.1552303284406662 (0.23499832730344) | > loss_disc_real_2: 0.22004547715187073 (0.23543246787913302) | > loss_disc_real_3: 0.22935886681079865 (0.23525850284606853) | > loss_disc_real_4: 0.1715877652168274 (0.2357529417631474) | > loss_disc_real_5: 0.27743980288505554 (0.24527829917187385) | > loss_0: 2.626471996307373 (2.6793626724405493) | > grad_norm_0: tensor(148.2322, device='cuda:0') (tensor(244.1874, device='cuda:0')) | > loss_gen: 2.1690375804901123 (2.0668959211795883) | > loss_kl: 2.0730223655700684 (2.20984532478008) | > loss_feat: 3.5077736377716064 (3.577840231834574) | > loss_mel: 16.759746551513672 (17.758857605305128) | > loss_duration: 1.5767128467559814 (1.5457185050274462) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.086292266845703 (27.159157489208464) | > grad_norm_1: tensor(6578.1450, device='cuda:0') (tensor(6118.6680, device='cuda:0')) | > current_lr_0: 0.0001772497613563963 | > current_lr_1: 0.0001772497613563963 | > step_time: 0.4518 (0.4450694804495953) | > loader_time: 0.009 (0.007848217132243703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008789465460978762 (+0.00019020429799254494) | > avg_step_time: 0.4562058751012238 (-0.0020176256206673737) | > avg_loss_disc: 2.693207300884623 (-0.00900335043249001) | > avg_loss_disc_real_0: 0.16386612373548498 (-0.009459780135624818) | > avg_loss_disc_real_1: 0.23721372787381562 (+0.0009894943992856398) | > avg_loss_disc_real_2: 0.2369405830829916 (-0.003914532107366642) | > avg_loss_disc_real_3: 0.23575271477162 (+0.0021424406850841793) | > avg_loss_disc_real_4: 0.23604709665540238 (-0.008085634506924066) | > avg_loss_disc_real_5: 0.2453305188199164 (+0.0034350966903525237) | > avg_loss_0: 2.693207300884623 (-0.00900335043249001) | > avg_grad_norm_0: tensor(245.2154, device='cuda:0') (tensor(-34.3907, device='cuda:0')) | > avg_loss_gen: 2.0647383307067426 (-0.018795587647130407) | > avg_loss_kl: 2.213048605851725 (+0.17174897227488817) | > avg_loss_feat: 3.5238495007367203 (+0.08823401827207755) | > avg_loss_mel: 17.592513366484305 (+0.6946905968894406) | > avg_loss_duration: 1.567110625790878 (+0.008058192024768207) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.961260406064316 (+0.9439360927528071) | > avg_grad_norm_1: tensor(6536.5488, device='cuda:0') (tensor(-270.0571, device='cuda:0'))  > EPOCH: 967/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:10:53)   --> TIME: 2025-02-25 18:11:06 -- STEP: 25/72 -- GLOBAL_STEP: 1069650 | > loss_disc: 2.7383639812469482 (2.7276732921600346) | > loss_disc_real_0: 0.10229894518852234 (0.13171083055436614) | > loss_disc_real_1: 0.2244509607553482 (0.2241354352235794) | > loss_disc_real_2: 0.2539020776748657 (0.24834248900413514) | > loss_disc_real_3: 0.26934248208999634 (0.23342180430889128) | > loss_disc_real_4: 0.25047656893730164 (0.24818995118141174) | > loss_disc_real_5: 0.2633019685745239 (0.25069235026836395) | > loss_0: 2.7383639812469482 (2.7276732921600346) | > grad_norm_0: tensor(181.9423, device='cuda:0') (tensor(192.3528, device='cuda:0')) | > loss_gen: 1.7677929401397705 (1.9498531103134156) | > loss_kl: 2.14532732963562 (2.089761009216309) | > loss_feat: 3.0791211128234863 (3.3832131004333497) | > loss_mel: 17.847692489624023 (17.368471221923834) | > loss_duration: 1.517484188079834 (1.518579397201538) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.357418060302734 (26.309877471923826) | > grad_norm_1: tensor(5755.4927, device='cuda:0') (tensor(5814.5454, device='cuda:0')) | > current_lr_0: 0.00017722760513622673 | > current_lr_1: 0.00017722760513622673 | > step_time: 0.4309 (0.4603846836090088) | > loader_time: 0.0077 (0.006923971176147461) --> EVAL PERFORMANCE | > avg_loader_time: 0.008665880686800244 (-0.00012358477417851767) | > avg_step_time: 0.4656029788541122 (+0.009397103752888436) | > avg_loss_disc: 2.6608082643696953 (-0.032399036514927815) | > avg_loss_disc_real_0: 0.1391896031219774 (-0.024676520613507585) | > avg_loss_disc_real_1: 0.22289046708127142 (-0.014323260792544201) | > avg_loss_disc_real_2: 0.2390662173570042 (+0.0021256342740126055) | > avg_loss_disc_real_3: 0.2387142235964117 (+0.0029615088247917087) | > avg_loss_disc_real_4: 0.24673382958895723 (+0.010686732933554849) | > avg_loss_disc_real_5: 0.24812522110804705 (+0.0027947022881306405) | > avg_loss_0: 2.6608082643696953 (-0.032399036514927815) | > avg_grad_norm_0: tensor(199.9131, device='cuda:0') (tensor(-45.3024, device='cuda:0')) | > avg_loss_gen: 2.0192515195255547 (-0.045486811181187914) | > avg_loss_kl: 2.078051639274812 (-0.13499696657691285) | > avg_loss_feat: 3.3564507927693112 (-0.16739870796740908) | > avg_loss_mel: 16.915561783481653 (-0.6769515830026513) | > avg_loss_duration: 1.5570906753271399 (-0.010019950463738159) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.926406323070257 (-1.034854082994059) | > avg_grad_norm_1: tensor(6512.9385, device='cuda:0') (tensor(-23.6104, device='cuda:0'))  > EPOCH: 968/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:11:30)   --> TIME: 2025-02-25 18:11:34 -- STEP: 3/72 -- GLOBAL_STEP: 1069700 | > loss_disc: 2.500962257385254 (2.4367289543151855) | > loss_disc_real_0: 0.03391673415899277 (0.04599167654911677) | > loss_disc_real_1: 0.20135952532291412 (0.2216527263323466) | > loss_disc_real_2: 0.22271759808063507 (0.22212726374467215) | > loss_disc_real_3: 0.24065850675106049 (0.2421702096859614) | > loss_disc_real_4: 0.2621895670890808 (0.2543553014596303) | > loss_disc_real_5: 0.2632174789905548 (0.2532070378462474) | > loss_0: 2.500962257385254 (2.4367289543151855) | > grad_norm_0: tensor(127.7313, device='cuda:0') (tensor(78.6290, device='cuda:0')) | > loss_gen: 2.0360870361328125 (2.219045956929525) | > loss_kl: 1.9718329906463623 (2.023807168006897) | > loss_feat: 3.8065848350524902 (4.082881689071655) | > loss_mel: 16.538122177124023 (16.71241823832194) | > loss_duration: 1.5257790088653564 (1.5217854181925456) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.878406524658203 (26.55993906656901) | > grad_norm_1: tensor(3194.4099, device='cuda:0') (tensor(2160.7666, device='cuda:0')) | > current_lr_0: 0.0001772054516855847 | > current_lr_1: 0.0001772054516855847 | > step_time: 0.4193 (0.4268932342529297) | > loader_time: 0.0056 (0.006100177764892578)  --> TIME: 2025-02-25 18:11:57 -- STEP: 53/72 -- GLOBAL_STEP: 1069750 | > loss_disc: 2.6738338470458984 (2.654092752708579) | > loss_disc_real_0: 0.2002585232257843 (0.10504769635790924) | > loss_disc_real_1: 0.22784705460071564 (0.22030820486680516) | > loss_disc_real_2: 0.22292347252368927 (0.2388716978284548) | > loss_disc_real_3: 0.22715157270431519 (0.24345916144128116) | > loss_disc_real_4: 0.23549607396125793 (0.24651783817219283) | > loss_disc_real_5: 0.23296713829040527 (0.24820592071650163) | > loss_0: 2.6738338470458984 (2.654092752708579) | > grad_norm_0: tensor(334.5448, device='cuda:0') (tensor(243.2297, device='cuda:0')) | > loss_gen: 2.051631450653076 (2.0391762301606953) | > loss_kl: 2.1938209533691406 (2.0858424901962276) | > loss_feat: 3.2351129055023193 (3.548320747771353) | > loss_mel: 16.594881057739258 (17.021400721567982) | > loss_duration: 1.6007986068725586 (1.5472561480864042) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.676246643066406 (26.24199658519817) | > grad_norm_1: tensor(7827.4380, device='cuda:0') (tensor(6084.6782, device='cuda:0')) | > current_lr_0: 0.0001772054516855847 | > current_lr_1: 0.0001772054516855847 | > step_time: 0.4502 (0.4436560037001124) | > loader_time: 0.0093 (0.008009105358483655) --> EVAL PERFORMANCE | > avg_loader_time: 0.008793810723533094 (+0.00012793003673284946) | > avg_step_time: 0.4558566859070684 (-0.009746292947043822) | > avg_loss_disc: 2.648844389848306 (-0.01196387452138925) | > avg_loss_disc_real_0: 0.13692636459960905 (-0.0022632385223683438) | > avg_loss_disc_real_1: 0.22035336851234166 (-0.002537098568929752) | > avg_loss_disc_real_2: 0.23481715166232955 (-0.0042490656946746475) | > avg_loss_disc_real_3: 0.2409459180395368 (+0.0022316944431250985) | > avg_loss_disc_real_4: 0.24524369563015413 (-0.001490133958803097) | > avg_loss_disc_real_5: 0.24703830795388826 (-0.0010869131541587917) | > avg_loss_0: 2.648844389848306 (-0.01196387452138925) | > avg_grad_norm_0: tensor(242.0683, device='cuda:0') (+tensor(42.1553, device='cuda:0')) | > avg_loss_gen: 2.071711053310985 (+0.052459533785430335) | > avg_loss_kl: 2.1423419562863635 (+0.06429031701155141) | > avg_loss_feat: 3.517003506002292 (+0.1605527132329807) | > avg_loss_mel: 17.193046596688287 (+0.2774848132066339) | > avg_loss_duration: 1.5644893679820315 (+0.007398692654891681) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.488592738836587 (+0.5621864157663303) | > avg_grad_norm_1: tensor(5874.2646, device='cuda:0') (tensor(-638.6738, device='cuda:0'))  > EPOCH: 969/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:12:07)   --> TIME: 2025-02-25 18:12:23 -- STEP: 31/72 -- GLOBAL_STEP: 1069800 | > loss_disc: 2.8027031421661377 (2.6454404246422554) | > loss_disc_real_0: 0.08573175966739655 (0.0997192929348638) | > loss_disc_real_1: 0.19765207171440125 (0.25287444312726287) | > loss_disc_real_2: 0.2478521466255188 (0.2301929049914883) | > loss_disc_real_3: 0.19990263879299164 (0.23961548074599234) | > loss_disc_real_4: 0.2321065366268158 (0.2462232478203312) | > loss_disc_real_5: 0.2458234578371048 (0.24863152542421896) | > loss_0: 2.8027031421661377 (2.6454404246422554) | > grad_norm_0: tensor(18.2943, device='cuda:0') (tensor(159.8050, device='cuda:0')) | > loss_gen: 1.9024661779403687 (2.2038547800433252) | > loss_kl: 1.9774483442306519 (2.1524409863256637) | > loss_feat: 3.0993943214416504 (3.8716634165856147) | > loss_mel: 16.716093063354492 (17.47377167978595) | > loss_duration: 1.5561929941177368 (1.5317662685148177) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.25159454345703 (27.233497250464655) | > grad_norm_1: tensor(1765.4196, device='cuda:0') (tensor(6457.7305, device='cuda:0')) | > current_lr_0: 0.000177183301004124 | > current_lr_1: 0.000177183301004124 | > step_time: 0.4358 (0.4357333721653108) | > loader_time: 0.0087 (0.007057866742534022) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661914879167579 (-0.00013189584436551494) | > avg_step_time: 0.45577412927654426 (-8.25566305241443e-05) | > avg_loss_disc: 2.64932965896499 (+0.00048526911668389516) | > avg_loss_disc_real_0: 0.1315557317624629 (-0.005370632837146144) | > avg_loss_disc_real_1: 0.23650746194409647 (+0.016154093431754807) | > avg_loss_disc_real_2: 0.23129380546825032 (-0.0035233461940792354) | > avg_loss_disc_real_3: 0.23828549053467496 (-0.0026604275048618398) | > avg_loss_disc_real_4: 0.24472389212796386 (-0.0005198035021902747) | > avg_loss_disc_real_5: 0.24837233399001635 (+0.0013340260361280887) | > avg_loss_0: 2.64932965896499 (+0.00048526911668389516) | > avg_grad_norm_0: tensor(181.7995, device='cuda:0') (tensor(-60.2688, device='cuda:0')) | > avg_loss_gen: 2.115269558530457 (+0.04355850521947202) | > avg_loss_kl: 2.1523500832033826 (+0.010008126917019133) | > avg_loss_feat: 3.6196301151329364 (+0.10262660913064447) | > avg_loss_mel: 17.208081312582525 (+0.015034715894238104) | > avg_loss_duration: 1.5644787365282085 (-1.06314538230734e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.659809757286393 (+0.17121701844980564) | > avg_grad_norm_1: tensor(6326.1177, device='cuda:0') (+tensor(451.8530, device='cuda:0'))  > EPOCH: 970/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:12:45)   --> TIME: 2025-02-25 18:12:51 -- STEP: 9/72 -- GLOBAL_STEP: 1069850 | > loss_disc: 2.76649808883667 (2.7461545202467175) | > loss_disc_real_0: 0.2056533545255661 (0.1189172433482276) | > loss_disc_real_1: 0.2580796182155609 (0.2467658238278495) | > loss_disc_real_2: 0.24391549825668335 (0.2334010766612159) | > loss_disc_real_3: 0.2697097063064575 (0.24043241308795082) | > loss_disc_real_4: 0.2610927224159241 (0.24830038679970634) | > loss_disc_real_5: 0.23409457504749298 (0.23265009621779123) | > loss_0: 2.76649808883667 (2.7461545202467175) | > grad_norm_0: tensor(30.7861, device='cuda:0') (tensor(214.7882, device='cuda:0')) | > loss_gen: 1.9017314910888672 (1.959301749865214) | > loss_kl: 1.8537821769714355 (2.087181899282667) | > loss_feat: 3.3266336917877197 (3.380988677342733) | > loss_mel: 17.04352569580078 (18.01059913635254) | > loss_duration: 1.5273845195770264 (1.536814226044549) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.653057098388672 (26.97488572862413) | > grad_norm_1: tensor(8916.5127, device='cuda:0') (tensor(8727.4453, device='cuda:0')) | > current_lr_0: 0.00017716115309149847 | > current_lr_1: 0.00017716115309149847 | > step_time: 0.4274 (0.48250744077894425) | > loader_time: 0.0056 (0.006208393308851454)  --> TIME: 2025-02-25 18:13:14 -- STEP: 59/72 -- GLOBAL_STEP: 1069900 | > loss_disc: 2.722060203552246 (2.719434491658615) | > loss_disc_real_0: 0.12515005469322205 (0.14026227265091265) | > loss_disc_real_1: 0.22595947980880737 (0.22201649832018353) | > loss_disc_real_2: 0.32802441716194153 (0.2699556082992231) | > loss_disc_real_3: 0.2557467520236969 (0.2378782965874268) | > loss_disc_real_4: 0.2194976657629013 (0.25406869538759785) | > loss_disc_real_5: 0.25085458159446716 (0.2424907335790537) | > loss_0: 2.722060203552246 (2.719434491658615) | > grad_norm_0: tensor(100.0594, device='cuda:0') (tensor(218.8424, device='cuda:0')) | > loss_gen: 2.1374943256378174 (2.0852880195035786) | > loss_kl: 1.9936946630477905 (2.0858576903908927) | > loss_feat: 2.9864165782928467 (3.4993025771642134) | > loss_mel: 16.752145767211914 (17.3427299887447) | > loss_duration: 1.5955877304077148 (1.5503621869168038) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.46533966064453 (26.563540377859344) | > grad_norm_1: tensor(5244.5229, device='cuda:0') (tensor(6322.7314, device='cuda:0')) | > current_lr_0: 0.00017716115309149847 | > current_lr_1: 0.00017716115309149847 | > step_time: 0.4799 (0.45531550908492785) | > loader_time: 0.0106 (0.008066868377944169) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862786467646209 (-3.405020270548856e-05) | > avg_step_time: 0.4625093332478698 (+0.006735203971325543) | > avg_loss_disc: 2.713881546342876 (+0.06455188737788609) | > avg_loss_disc_real_0: 0.14587039096464574 (+0.014314659202182833) | > avg_loss_disc_real_1: 0.22381366715884546 (-0.012693794785251011) | > avg_loss_disc_real_2: 0.26785322121331395 (+0.036559415745063634) | > avg_loss_disc_real_3: 0.23963784625832463 (+0.0013523557236496675) | > avg_loss_disc_real_4: 0.25174302030617085 (+0.00701912817820699) | > avg_loss_disc_real_5: 0.24258767404186893 (-0.005784659948147419) | > avg_loss_0: 2.713881546342876 (+0.06455188737788609) | > avg_grad_norm_0: tensor(225.8896, device='cuda:0') (+tensor(44.0901, device='cuda:0')) | > avg_loss_gen: 2.0967069847483044 (-0.018562573782152647) | > avg_loss_kl: 2.1031540716198123 (-0.04919601158357034) | > avg_loss_feat: 3.4642859344751065 (-0.15534418065782996) | > avg_loss_mel: 17.281489600597972 (+0.07340828801544674) | > avg_loss_duration: 1.5605563143609276 (-0.003922422167280892) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.506192811777893 (-0.15361694550849947) | > avg_grad_norm_1: tensor(6363.8716, device='cuda:0') (+tensor(37.7539, device='cuda:0'))  > EPOCH: 971/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:13:22)   --> TIME: 2025-02-25 18:13:41 -- STEP: 37/72 -- GLOBAL_STEP: 1069950 | > loss_disc: 2.7224538326263428 (2.6167103664295093) | > loss_disc_real_0: 0.07999248802661896 (0.13382095269657468) | > loss_disc_real_1: 0.4576016664505005 (0.24880030590134697) | > loss_disc_real_2: 0.36631548404693604 (0.24245764476221962) | > loss_disc_real_3: 0.3228118419647217 (0.2332469007453403) | > loss_disc_real_4: 0.3068566620349884 (0.2386824472530468) | > loss_disc_real_5: 0.28043127059936523 (0.24152357997121038) | > loss_0: 2.7224538326263428 (2.6167103664295093) | > grad_norm_0: tensor(151.5222, device='cuda:0') (tensor(197.6906, device='cuda:0')) | > loss_gen: 2.7308542728424072 (2.2271929206074894) | > loss_kl: 2.064882516860962 (2.2012835322199638) | > loss_feat: 4.322719097137451 (3.862659538114393) | > loss_mel: 17.742443084716797 (17.745410919189457) | > loss_duration: 1.5861616134643555 (1.5407094343288525) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.447059631347656 (27.57725597072292) | > grad_norm_1: tensor(1001.3280, device='cuda:0') (tensor(4896.4248, device='cuda:0')) | > current_lr_0: 0.00017713900794736202 | > current_lr_1: 0.00017713900794736202 | > step_time: 0.4509 (0.4401317093823407) | > loader_time: 0.0079 (0.007087952381855733) --> EVAL PERFORMANCE | > avg_loader_time: 0.008540334835858414 (-8.752984060367633e-05) | > avg_step_time: 0.4580729041301029 (-0.004436429117766927) | > avg_loss_disc: 2.651984611027677 (-0.06189693531519902) | > avg_loss_disc_real_0: 0.12214956102027015 (-0.02372082994437559) | > avg_loss_disc_real_1: 0.2530013167522324 (+0.02918764959338696) | > avg_loss_disc_real_2: 0.24059709297939086 (-0.027256128233923094) | > avg_loss_disc_real_3: 0.23410141950761768 (-0.005536426750706952) | > avg_loss_disc_real_4: 0.2414149524460376 (-0.010328067860133239) | > avg_loss_disc_real_5: 0.24571514906178057 (+0.0031274750199116463) | > avg_loss_0: 2.651984611027677 (-0.06189693531519902) | > avg_grad_norm_0: tensor(184.8043, device='cuda:0') (tensor(-41.0853, device='cuda:0')) | > avg_loss_gen: 2.1413906846247923 (+0.0446836998764879) | > avg_loss_kl: 2.449092324350921 (+0.3459382527311088) | > avg_loss_feat: 3.687074815723258 (+0.22278888124815133) | > avg_loss_mel: 18.352631958437644 (+1.0711423578396726) | > avg_loss_duration: 1.6542336118053382 (+0.09367729744441067) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.284423290843694 (+1.778230479065801) | > avg_grad_norm_1: tensor(4575.6577, device='cuda:0') (tensor(-1788.2139, device='cuda:0'))  > EPOCH: 972/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:13:59)   --> TIME: 2025-02-25 18:14:08 -- STEP: 15/72 -- GLOBAL_STEP: 1070000 | > loss_disc: 2.571589469909668 (2.646208413441976) | > loss_disc_real_0: 0.12733648717403412 (0.16085995808243753) | > loss_disc_real_1: 0.24929223954677582 (0.22984077235062916) | > loss_disc_real_2: 0.23018929362297058 (0.2473072052001953) | > loss_disc_real_3: 0.2178010642528534 (0.24332113862037658) | > loss_disc_real_4: 0.22940631210803986 (0.24539244771003724) | > loss_disc_real_5: 0.23360317945480347 (0.24795681436856587) | > loss_0: 2.571589469909668 (2.646208413441976) | > grad_norm_0: tensor(42.6273, device='cuda:0') (tensor(175.0054, device='cuda:0')) | > loss_gen: 2.2046263217926025 (2.1462066491444904) | > loss_kl: 2.1674602031707764 (2.1726794719696043) | > loss_feat: 3.5707907676696777 (3.5468241850535076) | > loss_mel: 18.34007453918457 (17.839235432942708) | > loss_duration: 1.571014404296875 (1.587808322906494) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.853965759277344 (27.29275436401367) | > grad_norm_1: tensor(7905.5562, device='cuda:0') (tensor(5773.9297, device='cuda:0')) | > current_lr_0: 0.0001771168655713686 | > current_lr_1: 0.0001771168655713686 | > step_time: 0.457 (0.4383599122365316) | > loader_time: 0.0072 (0.006814479827880859) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1070000.pth  --> TIME: 2025-02-25 18:14:34 -- STEP: 65/72 -- GLOBAL_STEP: 1070050 | > loss_disc: 2.4877524375915527 (2.6540534826425404) | > loss_disc_real_0: 0.18033835291862488 (0.1360872687628636) | > loss_disc_real_1: 0.20938032865524292 (0.2240757291133587) | > loss_disc_real_2: 0.22861453890800476 (0.23969202316724336) | > loss_disc_real_3: 0.24337118864059448 (0.2423760570012606) | > loss_disc_real_4: 0.23402957618236542 (0.24623095805828388) | > loss_disc_real_5: 0.21670691668987274 (0.24356228548746842) | > loss_0: 2.4877524375915527 (2.6540534826425404) | > grad_norm_0: tensor(73.3767, device='cuda:0') (tensor(176.8585, device='cuda:0')) | > loss_gen: 2.5175883769989014 (2.0633159967569203) | > loss_kl: 2.299290895462036 (2.1402304337574893) | > loss_feat: 3.9258944988250732 (3.5333883102123553) | > loss_mel: 17.1767578125 (17.44018579629752) | > loss_duration: 1.6275484561920166 (1.590173106927138) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.54707908630371 (26.767293842022237) | > grad_norm_1: tensor(5176.1704, device='cuda:0') (tensor(5735.5049, device='cuda:0')) | > current_lr_0: 0.0001771168655713686 | > current_lr_1: 0.0001771168655713686 | > step_time: 0.4978 (0.4519061932196984) | > loader_time: 0.0124 (0.008569449644822336) --> EVAL PERFORMANCE | > avg_loader_time: 0.008839741559095782 (+0.0002994067232373679) | > avg_step_time: 0.45799664712288013 (-7.625700722274642e-05) | > avg_loss_disc: 2.6518681720948556 (-0.0001164389328214277) | > avg_loss_disc_real_0: 0.13918292884465674 (+0.017033367824386594) | > avg_loss_disc_real_1: 0.2232815747109937 (-0.029719742041238728) | > avg_loss_disc_real_2: 0.24149371914460627 (+0.0008966261652154084) | > avg_loss_disc_real_3: 0.24371823752430125 (+0.00961681801668357) | > avg_loss_disc_real_4: 0.2466611580949434 (+0.005246205648905794) | > avg_loss_disc_real_5: 0.24511826185273453 (-0.0005968872090460486) | > avg_loss_0: 2.6518681720948556 (-0.0001164389328214277) | > avg_grad_norm_0: tensor(184.0336, device='cuda:0') (tensor(-0.7707, device='cuda:0')) | > avg_loss_gen: 2.0761076463779933 (-0.06528303824679904) | > avg_loss_kl: 2.145821898755894 (-0.30327042559502715) | > avg_loss_feat: 3.5172791514598147 (-0.16979566426344306) | > avg_loss_mel: 17.424106087483157 (-0.9285258709544877) | > avg_loss_duration: 1.5933731122755668 (-0.06086049952977146) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.756688050820795 (-1.5277352400228992) | > avg_grad_norm_1: tensor(5928.7959, device='cuda:0') (+tensor(1353.1382, device='cuda:0'))  > EPOCH: 973/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:14:40)   --> TIME: 2025-02-25 18:15:01 -- STEP: 43/72 -- GLOBAL_STEP: 1070100 | > loss_disc: 2.8148434162139893 (2.7152989853260125) | > loss_disc_real_0: 0.36515170335769653 (0.13910458860702293) | > loss_disc_real_1: 0.21738190948963165 (0.22220941407735958) | > loss_disc_real_2: 0.24459873139858246 (0.27139086882735414) | > loss_disc_real_3: 0.22251483798027039 (0.23254386699476906) | > loss_disc_real_4: 0.24328239262104034 (0.24461493831734324) | > loss_disc_real_5: 0.2550392150878906 (0.24081263050090435) | > loss_0: 2.8148434162139893 (2.7152989853260125) | > grad_norm_0: tensor(453.0485, device='cuda:0') (tensor(238.4001, device='cuda:0')) | > loss_gen: 2.1501004695892334 (2.0487460979195533) | > loss_kl: 1.8545242547988892 (2.1042499930359595) | > loss_feat: 2.690859794616699 (3.623309396034063) | > loss_mel: 15.326666831970215 (17.300859340401583) | > loss_duration: 1.5614680051803589 (1.5590033669804417) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.583620071411133 (26.636168280313182) | > grad_norm_1: tensor(6156.4209, device='cuda:0') (tensor(6235.7280, device='cuda:0')) | > current_lr_0: 0.00017709472596317218 | > current_lr_1: 0.00017709472596317218 | > step_time: 0.471 (0.4485700906709183) | > loader_time: 0.0096 (0.007559970367786496) --> EVAL PERFORMANCE | > avg_loader_time: 0.008659943728379804 (-0.00017979783071597755) | > avg_step_time: 0.45939592912163535 (+0.0013992819987552196) | > avg_loss_disc: 2.6706196388728176 (+0.018751466777962023) | > avg_loss_disc_real_0: 0.12190322276257295 (-0.01727970608208379) | > avg_loss_disc_real_1: 0.22602042199020655 (+0.0027388472792128638) | > avg_loss_disc_real_2: 0.2516268917792282 (+0.010133172634621934) | > avg_loss_disc_real_3: 0.23283572998684895 (-0.01088250753745229) | > avg_loss_disc_real_4: 0.24326378967560514 (-0.0033973684193382703) | > avg_loss_disc_real_5: 0.24117195333393526 (-0.003946308518799263) | > avg_loss_0: 2.6706196388728176 (+0.018751466777962023) | > avg_grad_norm_0: tensor(223.2289, device='cuda:0') (+tensor(39.1953, device='cuda:0')) | > avg_loss_gen: 2.062206730036668 (-0.013900916341325065) | > avg_loss_kl: 2.1364214924019826 (-0.009400406353911261) | > avg_loss_feat: 3.677096165401835 (+0.1598170139420203) | > avg_loss_mel: 17.32838035637225 (-0.09572573111090676) | > avg_loss_duration: 1.577214135250575 (-0.016158977024991694) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.78131898691957 (+0.02463093609877376) | > avg_grad_norm_1: tensor(5461.7505, device='cuda:0') (tensor(-467.0454, device='cuda:0'))  > EPOCH: 974/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:15:17)   --> TIME: 2025-02-25 18:15:28 -- STEP: 21/72 -- GLOBAL_STEP: 1070150 | > loss_disc: 2.589775562286377 (2.5709104651496526) | > loss_disc_real_0: 0.12399856746196747 (0.1573700551830587) | > loss_disc_real_1: 0.2094322293996811 (0.2354113608598709) | > loss_disc_real_2: 0.25519201159477234 (0.23999586843308948) | > loss_disc_real_3: 0.2838917672634125 (0.22430074711640677) | > loss_disc_real_4: 0.2863910496234894 (0.24620883947327024) | > loss_disc_real_5: 0.30078959465026855 (0.2492621491352717) | > loss_0: 2.589775562286377 (2.5709104651496526) | > grad_norm_0: tensor(183.6047, device='cuda:0') (tensor(149.1760, device='cuda:0')) | > loss_gen: 2.3233790397644043 (2.4357554004305886) | > loss_kl: 1.9907488822937012 (2.042820243608384) | > loss_feat: 3.815126657485962 (4.087204479035876) | > loss_mel: 17.121408462524414 (17.444727125621977) | > loss_duration: 1.5243260860443115 (1.51711558727991) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.77499008178711 (27.52762267703102) | > grad_norm_1: tensor(8563.1064, device='cuda:0') (tensor(2810.9912, device='cuda:0')) | > current_lr_0: 0.00017707258912242677 | > current_lr_1: 0.00017707258912242677 | > step_time: 0.432 (0.4350273382096064) | > loader_time: 0.0064 (0.006655261630103702)  --> TIME: 2025-02-25 18:15:52 -- STEP: 71/72 -- GLOBAL_STEP: 1070200 | > loss_disc: 2.564350128173828 (2.6282355180928403) | > loss_disc_real_0: 0.08345002681016922 (0.13158115044131247) | > loss_disc_real_1: 0.2726231515407562 (0.24178182071363422) | > loss_disc_real_2: 0.2563973665237427 (0.23560419510787642) | > loss_disc_real_3: 0.22802624106407166 (0.2370548921991402) | > loss_disc_real_4: 0.23873229324817657 (0.2350405859695354) | > loss_disc_real_5: 0.24949730932712555 (0.23744045932528) | > loss_0: 2.564350128173828 (2.6282355180928403) | > grad_norm_0: tensor(71.8798, device='cuda:0') (tensor(174.5203, device='cuda:0')) | > loss_gen: 2.1796460151672363 (2.1931118528607856) | > loss_kl: 1.9374470710754395 (2.269250567530242) | > loss_feat: 3.4830193519592285 (3.756453658493471) | > loss_mel: 16.020706176757812 (17.60141791760082) | > loss_duration: 1.565517783164978 (1.5825661420822144) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.186336517333984 (27.402799901827958) | > grad_norm_1: tensor(1911.3590, device='cuda:0') (tensor(4564.6621, device='cuda:0')) | > current_lr_0: 0.00017707258912242677 | > current_lr_1: 0.00017707258912242677 | > step_time: 0.479 (0.45433698237781794) | > loader_time: 0.008 (0.008494451012409905) --> EVAL PERFORMANCE | > avg_loader_time: 0.008494451012409905 (-0.0001654927159698994) | > avg_step_time: 0.45433698237781794 (-0.005058946743817405) | > avg_loss_disc: 2.6282355180928403 (-0.04238412077997733) | > avg_loss_disc_real_0: 0.13158115044131247 (+0.009677927678739512) | > avg_loss_disc_real_1: 0.24178182071363422 (+0.01576139872342766) | > avg_loss_disc_real_2: 0.23560419510787642 (-0.016022696671351783) | > avg_loss_disc_real_3: 0.2370548921991402 (+0.004219162212291239) | > avg_loss_disc_real_4: 0.2350405859695354 (-0.008223203706069743) | > avg_loss_disc_real_5: 0.23744045932528 (-0.003731494008655273) | > avg_loss_0: 2.6282355180928403 (-0.04238412077997733) | > avg_grad_norm_0: tensor(174.5203, device='cuda:0') (tensor(-48.7086, device='cuda:0')) | > avg_loss_gen: 2.1931118528607856 (+0.13090512282411737) | > avg_loss_kl: 2.269250567530242 (+0.13282907512825926) | > avg_loss_feat: 3.756453658493471 (+0.07935749309163587) | > avg_loss_mel: 17.60141791760082 (+0.2730375612285698) | > avg_loss_duration: 1.5825661420822144 (+0.005352006831639278) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.402799901827958 (+0.6214809149083891) | > avg_grad_norm_1: tensor(4564.6621, device='cuda:0') (tensor(-897.0884, device='cuda:0'))  > EPOCH: 975/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:15:54)   --> TIME: 2025-02-25 18:16:18 -- STEP: 49/72 -- GLOBAL_STEP: 1070250 | > loss_disc: 2.622203826904297 (2.7196843964712962) | > loss_disc_real_0: 0.10414602607488632 (0.18652977885640398) | > loss_disc_real_1: 0.24043726921081543 (0.22436855368468225) | > loss_disc_real_2: 0.21736375987529755 (0.24117887871606009) | > loss_disc_real_3: 0.21855315566062927 (0.23439698316613022) | > loss_disc_real_4: 0.21646347641944885 (0.24331694293995293) | > loss_disc_real_5: 0.23030388355255127 (0.2521556567172615) | > loss_0: 2.622203826904297 (2.7196843964712962) | > grad_norm_0: tensor(359.9753, device='cuda:0') (tensor(246.4139, device='cuda:0')) | > loss_gen: 1.9339518547058105 (2.0366195561934486) | > loss_kl: 2.1170003414154053 (2.092683115784003) | > loss_feat: 3.7426562309265137 (3.4165047090880725) | > loss_mel: 17.102764129638672 (17.25339317321777) | > loss_duration: 1.6169341802597046 (1.5487501499604206) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.513307571411133 (26.347950487720723) | > grad_norm_1: tensor(14839.7568, device='cuda:0') (tensor(6736.7520, device='cuda:0')) | > current_lr_0: 0.00017705045504878646 | > current_lr_1: 0.00017705045504878646 | > step_time: 0.4535 (0.4467943103945985) | > loader_time: 0.0102 (0.00772387154248296) --> EVAL PERFORMANCE | > avg_loader_time: 0.008602763565493301 (+0.00010831255308339609) | > avg_step_time: 0.4584228992462158 (+0.004085916868397876) | > avg_loss_disc: 2.7302374201761164 (+0.10200190208327609) | > avg_loss_disc_real_0: 0.2096893357139238 (+0.07810818527261135) | > avg_loss_disc_real_1: 0.2222694886822096 (-0.019512332031424606) | > avg_loss_disc_real_2: 0.23982440732734303 (+0.0042202122194666125) | > avg_loss_disc_real_3: 0.23428082801926303 (-0.0027740641798771615) | > avg_loss_disc_real_4: 0.24271112743397832 (+0.0076705414644429315) | > avg_loss_disc_real_5: 0.24983621944843884 (+0.01239576012315885) | > avg_loss_0: 2.7302374201761164 (+0.10200190208327609) | > avg_grad_norm_0: tensor(249.2406, device='cuda:0') (+tensor(74.7203, device='cuda:0')) | > avg_loss_gen: 2.04273389762556 (-0.1503779552352258) | > avg_loss_kl: 2.1696995446379757 (-0.09955102289226625) | > avg_loss_feat: 3.352613096505823 (-0.40384056198764773) | > avg_loss_mel: 17.545399719560645 (-0.05601819804017438) | > avg_loss_duration: 1.5727372085544424 (-0.009828933527771921) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.683183267082967 (-0.719616634744991) | > avg_grad_norm_1: tensor(6110.1528, device='cuda:0') (+tensor(1545.4907, device='cuda:0'))  > EPOCH: 976/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:16:31)   --> TIME: 2025-02-25 18:16:46 -- STEP: 27/72 -- GLOBAL_STEP: 1070300 | > loss_disc: 2.448194742202759 (2.5195119469254106) | > loss_disc_real_0: 0.051748231053352356 (0.05137543256084124) | > loss_disc_real_1: 0.2222926765680313 (0.21007282314477144) | > loss_disc_real_2: 0.25505316257476807 (0.2274644645275893) | > loss_disc_real_3: 0.23263125121593475 (0.24056900044282278) | > loss_disc_real_4: 0.2900439500808716 (0.2548498660326004) | > loss_disc_real_5: 0.28524741530418396 (0.25490217573112917) | > loss_0: 2.448194742202759 (2.5195119469254106) | > grad_norm_0: tensor(111.0529, device='cuda:0') (tensor(129.5085, device='cuda:0')) | > loss_gen: 2.409358501434326 (2.1951091598581383) | > loss_kl: 1.8403798341751099 (2.0955710940890846) | > loss_feat: 4.273099899291992 (4.105175874851367) | > loss_mel: 17.097333908081055 (18.2801443029333) | > loss_duration: 1.5433194637298584 (1.535171252709848) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.16349220275879 (28.211171609384042) | > grad_norm_1: tensor(4592.7085, device='cuda:0') (tensor(6136.6548, device='cuda:0')) | > current_lr_0: 0.00017702832374190535 | > current_lr_1: 0.00017702832374190535 | > step_time: 0.4556 (0.45640125981083624) | > loader_time: 0.0105 (0.0069630675845676) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640467281072912 (+3.770371557961123e-05) | > avg_step_time: 0.462894040094295 (+0.0044711408480792025) | > avg_loss_disc: 2.5105777324085503 (-0.21965968776756606) | > avg_loss_disc_real_0: 0.06118837030420841 (-0.1485009654097154) | > avg_loss_disc_real_1: 0.21680941653083746 (-0.005460072151372147) | > avg_loss_disc_real_2: 0.23057625852000546 (-0.00924814880733757) | > avg_loss_disc_real_3: 0.22840823023252085 (-0.005872597786742179) | > avg_loss_disc_real_4: 0.24539444018417686 (+0.0026833127501985354) | > avg_loss_disc_real_5: 0.2515296766035992 (+0.001693457155160366) | > avg_loss_0: 2.5105777324085503 (-0.21965968776756606) | > avg_grad_norm_0: tensor(140.6622, device='cuda:0') (tensor(-108.5785, device='cuda:0')) | > avg_loss_gen: 2.209158395377683 (+0.1664244977521232) | > avg_loss_kl: 2.0541555377799026 (-0.11554400685807309) | > avg_loss_feat: 4.0084678052176885 (+0.6558547087118654) | > avg_loss_mel: 17.414149324658883 (-0.1312503949017625) | > avg_loss_duration: 1.5649401033428356 (-0.007797105211606814) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.250870959859498 (+0.5676876927765306) | > avg_grad_norm_1: tensor(6445.1084, device='cuda:0') (+tensor(334.9556, device='cuda:0'))  > EPOCH: 977/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:17:09)   --> TIME: 2025-02-25 18:17:13 -- STEP: 5/72 -- GLOBAL_STEP: 1070350 | > loss_disc: 2.3259284496307373 (2.42032675743103) | > loss_disc_real_0: 0.05759565532207489 (0.0786644995212555) | > loss_disc_real_1: 0.23160280287265778 (0.21937862336635588) | > loss_disc_real_2: 0.2177671641111374 (0.23223111927509307) | > loss_disc_real_3: 0.2322951704263687 (0.24218499064445495) | > loss_disc_real_4: 0.2277384102344513 (0.24153884649276733) | > loss_disc_real_5: 0.2360105961561203 (0.2411448746919632) | > loss_0: 2.3259284496307373 (2.42032675743103) | > grad_norm_0: tensor(56.3617, device='cuda:0') (tensor(207.4460, device='cuda:0')) | > loss_gen: 2.299799919128418 (2.345689630508423) | > loss_kl: 1.8268139362335205 (1.856994342803955) | > loss_feat: 4.201827526092529 (4.072469902038574) | > loss_mel: 17.055801391601562 (16.765876007080077) | > loss_duration: 1.4965764284133911 (1.5260342359542847) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.88081932067871 (26.56706428527832) | > grad_norm_1: tensor(7237.7412, device='cuda:0') (tensor(6970.0430, device='cuda:0')) | > current_lr_0: 0.00017700619520143762 | > current_lr_1: 0.00017700619520143762 | > step_time: 0.423 (0.4271510601043701) | > loader_time: 0.0063 (0.006850051879882813)  --> TIME: 2025-02-25 18:17:36 -- STEP: 55/72 -- GLOBAL_STEP: 1070400 | > loss_disc: 2.4624030590057373 (2.48746281970631) | > loss_disc_real_0: 0.07809940725564957 (0.07714192677627907) | > loss_disc_real_1: 0.21979542076587677 (0.21856351792812348) | > loss_disc_real_2: 0.22285343706607819 (0.23046356547962535) | > loss_disc_real_3: 0.2306840568780899 (0.23288211063905195) | > loss_disc_real_4: 0.22762326896190643 (0.24163968400521713) | > loss_disc_real_5: 0.2213946133852005 (0.2432709975676103) | > loss_0: 2.4624030590057373 (2.48746281970631) | > grad_norm_0: tensor(125.4700, device='cuda:0') (tensor(179.5177, device='cuda:0')) | > loss_gen: 2.359468936920166 (2.2862155762585723) | > loss_kl: 1.8374736309051514 (1.9604050462896174) | > loss_feat: 4.376552581787109 (4.076402291384612) | > loss_mel: 17.455076217651367 (16.796956755898222) | > loss_duration: 1.5982780456542969 (1.5379031051288952) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.626848220825195 (26.657882690429688) | > grad_norm_1: tensor(4327.0410, device='cuda:0') (tensor(6582.1396, device='cuda:0')) | > current_lr_0: 0.00017700619520143762 | > current_lr_1: 0.00017700619520143762 | > step_time: 0.4638 (0.44427174221385607) | > loader_time: 0.0095 (0.007883756810968572) --> EVAL PERFORMANCE | > avg_loader_time: 0.008594606963681504 (-4.586031739140829e-05) | > avg_step_time: 0.45625714516975513 (-0.006636894924539893) | > avg_loss_disc: 2.507733496142105 (-0.0028442362664451437) | > avg_loss_disc_real_0: 0.0771614611148834 (+0.015973090810674988) | > avg_loss_disc_real_1: 0.22501135061324481 (+0.008201934082407353) | > avg_loss_disc_real_2: 0.23008948207741053 (-0.00048677644259492725) | > avg_loss_disc_real_3: 0.23587375060773233 (+0.007465520375211476) | > avg_loss_disc_real_4: 0.2442729353484973 (-0.0011215048356795487) | > avg_loss_disc_real_5: 0.24384948233483542 (-0.00768019426876379) | > avg_loss_0: 2.507733496142105 (-0.0028442362664451437) | > avg_grad_norm_0: tensor(172.6992, device='cuda:0') (+tensor(32.0370, device='cuda:0')) | > avg_loss_gen: 2.27103178433969 (+0.06187338896200689) | > avg_loss_kl: 1.9894984440064767 (-0.06465709377342588) | > avg_loss_feat: 4.015440272613312 (+0.006972467395623738) | > avg_loss_mel: 16.78528096642293 (-0.6288683582359518) | > avg_loss_duration: 1.551850867943025 (-0.013089235399810661) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.61310238905356 (-0.6377685708059389) | > avg_grad_norm_1: tensor(6297.7456, device='cuda:0') (tensor(-147.3628, device='cuda:0'))  > EPOCH: 978/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:17:46)   --> TIME: 2025-02-25 18:18:03 -- STEP: 33/72 -- GLOBAL_STEP: 1070450 | > loss_disc: 2.6839122772216797 (2.5957729888684824) | > loss_disc_real_0: 0.03145698457956314 (0.135255837756576) | > loss_disc_real_1: 0.26029449701309204 (0.26599105379798205) | > loss_disc_real_2: 0.257610946893692 (0.22095049601612668) | > loss_disc_real_3: 0.2388162910938263 (0.22175379187771768) | > loss_disc_real_4: 0.26199573278427124 (0.22605550650394324) | > loss_disc_real_5: 0.2550199031829834 (0.23722756495981506) | > loss_0: 2.6839122772216797 (2.5957729888684824) | > grad_norm_0: tensor(286.0247, device='cuda:0') (tensor(219.9172, device='cuda:0')) | > loss_gen: 1.9631747007369995 (2.2410956296053794) | > loss_kl: 2.0128796100616455 (1.9428087653535786) | > loss_feat: 3.9701735973358154 (3.838056535431833) | > loss_mel: 16.59081268310547 (16.83662001291911) | > loss_duration: 1.5106823444366455 (1.5082317446217393) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.0477237701416 (26.366812619295988) | > grad_norm_1: tensor(10652.4756, device='cuda:0') (tensor(5263.5352, device='cuda:0')) | > current_lr_0: 0.00017698406942703743 | > current_lr_1: 0.00017698406942703743 | > step_time: 0.4715 (0.43956661946845776) | > loader_time: 0.0081 (0.007231134356874408) --> EVAL PERFORMANCE | > avg_loader_time: 0.008697066508548363 (+0.00010245954486685915) | > avg_step_time: 0.4565254197993749 (+0.000268274629619758) | > avg_loss_disc: 2.620338090708558 (+0.11260459456645266) | > avg_loss_disc_real_0: 0.1504945959633505 (+0.0733331348484671) | > avg_loss_disc_real_1: 0.24180470134170967 (+0.016793350728464856) | > avg_loss_disc_real_2: 0.23409889837805653 (+0.004009416300645996) | > avg_loss_disc_real_3: 0.2318180302079295 (-0.004055720399802842) | > avg_loss_disc_real_4: 0.23642112167788223 (-0.007851813670615077) | > avg_loss_disc_real_5: 0.23941694011150952 (-0.004432542223325897) | > avg_loss_0: 2.620338090708558 (+0.11260459456645266) | > avg_grad_norm_0: tensor(216.2289, device='cuda:0') (+tensor(43.5297, device='cuda:0')) | > avg_loss_gen: 2.1917811867216934 (-0.07925059761799647) | > avg_loss_kl: 2.030816847169903 (+0.041318403163426165) | > avg_loss_feat: 3.7273774214193853 (-0.28806285119392694) | > avg_loss_mel: 16.82753608596157 (+0.04225511953863759) | > avg_loss_duration: 1.5504007708858436 (-0.0014500970571813543) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.32791226346728 (-0.2851901255862792) | > avg_grad_norm_1: tensor(5378.0547, device='cuda:0') (tensor(-919.6909, device='cuda:0'))  > EPOCH: 979/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:18:23)   --> TIME: 2025-02-25 18:18:30 -- STEP: 11/72 -- GLOBAL_STEP: 1070500 | > loss_disc: 2.7553951740264893 (2.780855503949252) | > loss_disc_real_0: 0.11735020577907562 (0.07891449298370969) | > loss_disc_real_1: 0.23573674261569977 (0.23267302323471417) | > loss_disc_real_2: 0.36232441663742065 (0.3900809504769065) | > loss_disc_real_3: 0.24359847605228424 (0.22645370933142575) | > loss_disc_real_4: 0.29006287455558777 (0.28283251415599475) | > loss_disc_real_5: 0.24176651239395142 (0.22748092900622974) | > loss_0: 2.7553951740264893 (2.780855503949252) | > grad_norm_0: tensor(138.4219, device='cuda:0') (tensor(282.0674, device='cuda:0')) | > loss_gen: 2.0335612297058105 (2.02441190589558) | > loss_kl: 2.102358341217041 (1.9512337446212769) | > loss_feat: 3.210029125213623 (3.354225743900646) | > loss_mel: 17.32278823852539 (16.854144529862836) | > loss_duration: 1.5134750604629517 (1.5123482509092852) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.182212829589844 (25.69636448946866) | > grad_norm_1: tensor(8195.3613, device='cuda:0') (tensor(8275.6279, device='cuda:0')) | > current_lr_0: 0.00017696194641835905 | > current_lr_1: 0.00017696194641835905 | > step_time: 0.4454 (0.4331404512578791) | > loader_time: 0.006 (0.006346745924516158)  --> TIME: 2025-02-25 18:18:53 -- STEP: 61/72 -- GLOBAL_STEP: 1070550 | > loss_disc: 2.6794395446777344 (2.6983869974730434) | > loss_disc_real_0: 0.14978167414665222 (0.14512776839928548) | > loss_disc_real_1: 0.2777021825313568 (0.23301942304509585) | > loss_disc_real_2: 0.23453223705291748 (0.2675592867565938) | > loss_disc_real_3: 0.25662854313850403 (0.2323615809933084) | > loss_disc_real_4: 0.24474139511585236 (0.24789143292630306) | > loss_disc_real_5: 0.23594990372657776 (0.23652005635324072) | > loss_0: 2.6794395446777344 (2.6983869974730434) | > grad_norm_0: tensor(180.5142, device='cuda:0') (tensor(237.9556, device='cuda:0')) | > loss_gen: 2.074045181274414 (2.025529386567288) | > loss_kl: 3.952568769454956 (2.1567784958198426) | > loss_feat: 3.2831521034240723 (3.3861339756699858) | > loss_mel: 23.373098373413086 (17.17860118678358) | > loss_duration: 1.78917396068573 (1.554374749543237) | > amp_scaler: 8.0 (8.0) | > loss_1: 34.4720344543457 (26.301417835423205) | > grad_norm_1: tensor(11400.7119, device='cuda:0') (tensor(7360.8008, device='cuda:0')) | > current_lr_0: 0.00017696194641835905 | > current_lr_1: 0.00017696194641835905 | > step_time: 0.4598 (0.44874709160601506) | > loader_time: 0.0109 (0.00814720450854692) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627199790847131 (-6.98667177012318e-05) | > avg_step_time: 0.4581870461853457 (+0.001661626385970838) | > avg_loss_disc: 2.677665173167913 (+0.05732708245935525) | > avg_loss_disc_real_0: 0.14429470518944967 (-0.006199890773900818) | > avg_loss_disc_real_1: 0.22887611494097912 (-0.012928586400730552) | > avg_loss_disc_real_2: 0.26275239129301536 (+0.028653492914958834) | > avg_loss_disc_real_3: 0.23268990092714067 (+0.0008718707192111874) | > avg_loss_disc_real_4: 0.24775216109316114 (+0.01133103941527891) | > avg_loss_disc_real_5: 0.2386485258458366 (-0.0007684142656729231) | > avg_loss_0: 2.677665173167913 (+0.05732708245935525) | > avg_grad_norm_0: tensor(242.4280, device='cuda:0') (+tensor(26.1991, device='cuda:0')) | > avg_loss_gen: 2.0409091677464235 (-0.15087201897526992) | > avg_loss_kl: 2.294984061953048 (+0.2641672147831451) | > avg_loss_feat: 3.404878535740812 (-0.32249888567857354) | > avg_loss_mel: 17.759661553611213 (+0.9321254676496444) | > avg_loss_duration: 1.5742705237697547 (+0.023869752883911133) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.074703888154367 (+0.7467916246870878) | > avg_grad_norm_1: tensor(7864.8579, device='cuda:0') (+tensor(2486.8032, device='cuda:0'))  > EPOCH: 980/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:19:00)   --> TIME: 2025-02-25 18:19:20 -- STEP: 39/72 -- GLOBAL_STEP: 1070600 | > loss_disc: 2.718858003616333 (2.595969725877811) | > loss_disc_real_0: 0.11993512511253357 (0.12552898625532785) | > loss_disc_real_1: 0.19872498512268066 (0.224069533821864) | > loss_disc_real_2: 0.23656348884105682 (0.2312972408074599) | > loss_disc_real_3: 0.25256702303886414 (0.2369252603787642) | > loss_disc_real_4: 0.24402794241905212 (0.24549594330481994) | > loss_disc_real_5: 0.2527136504650116 (0.2480181891184587) | > loss_0: 2.718858003616333 (2.595969725877811) | > grad_norm_0: tensor(295.7937, device='cuda:0') (tensor(286.3224, device='cuda:0')) | > loss_gen: 1.916379690170288 (2.0720449441518545) | > loss_kl: 1.9215922355651855 (2.11236004340343) | > loss_feat: 3.06046462059021 (3.556261184888008) | > loss_mel: 16.386945724487305 (17.1388838596833) | > loss_duration: 1.5411967039108276 (1.5338306640967343) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.826580047607422 (26.413380794036083) | > grad_norm_1: tensor(12520.3447, device='cuda:0') (tensor(11681.1133, device='cuda:0')) | > current_lr_0: 0.00017693982617505673 | > current_lr_1: 0.00017693982617505673 | > step_time: 0.4454 (0.4387578780834491) | > loader_time: 0.0083 (0.007406259194398537) --> EVAL PERFORMANCE | > avg_loader_time: 0.008646800484455805 (+1.9600693608673794e-05) | > avg_step_time: 0.456272605439307 (-0.001914440746038748) | > avg_loss_disc: 2.589603027827303 (-0.08806214534061008) | > avg_loss_disc_real_0: 0.12488050402050287 (-0.019414201168946807) | > avg_loss_disc_real_1: 0.22380518556480677 (-0.005070929376172345) | > avg_loss_disc_real_2: 0.23344442080444014 (-0.029307970488575225) | > avg_loss_disc_real_3: 0.2345847871521829 (+0.0018948862250422194) | > avg_loss_disc_real_4: 0.24332340943141723 (-0.004428751661743918) | > avg_loss_disc_real_5: 0.24629819078344695 (+0.007649664937610351) | > avg_loss_0: 2.589603027827303 (-0.08806214534061008) | > avg_grad_norm_0: tensor(297.3220, device='cuda:0') (+tensor(54.8940, device='cuda:0')) | > avg_loss_gen: 2.0721390549565695 (+0.031229887210145968) | > avg_loss_kl: 2.1038609306577225 (-0.19112313129532543) | > avg_loss_feat: 3.5370245147758808 (+0.13214597903506897) | > avg_loss_mel: 16.870034352154804 (-0.8896272014564097) | > avg_loss_duration: 1.5573357343673704 (-0.016934789402384354) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.140394667504538 (-0.9343092206498298) | > avg_grad_norm_1: tensor(11383.8037, device='cuda:0') (+tensor(3518.9458, device='cuda:0'))  > EPOCH: 981/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:19:37)   --> TIME: 2025-02-25 18:19:47 -- STEP: 17/72 -- GLOBAL_STEP: 1070650 | > loss_disc: 2.5661745071411133 (2.5373455075656666) | > loss_disc_real_0: 0.11812535673379898 (0.1127875228138531) | > loss_disc_real_1: 0.24822692573070526 (0.2163748381768956) | > loss_disc_real_2: 0.23634396493434906 (0.23169045676203334) | > loss_disc_real_3: 0.2251299023628235 (0.23174090858768015) | > loss_disc_real_4: 0.26102662086486816 (0.23755398042061748) | > loss_disc_real_5: 0.2489936351776123 (0.24001156757859624) | > loss_0: 2.5661745071411133 (2.5373455075656666) | > grad_norm_0: tensor(271.8075, device='cuda:0') (tensor(333.4288, device='cuda:0')) | > loss_gen: 2.2761800289154053 (2.1391698122024536) | > loss_kl: 2.188959836959839 (1.950497031211853) | > loss_feat: 3.7949917316436768 (3.837528523276834) | > loss_mel: 17.174171447753906 (16.665538114659924) | > loss_duration: 1.5309464931488037 (1.502318073721493) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.96525001525879 (26.095051597146426) | > grad_norm_1: tensor(9947.9531, device='cuda:0') (tensor(12354.7256, device='cuda:0')) | > current_lr_0: 0.00017691770869678484 | > current_lr_1: 0.00017691770869678484 | > step_time: 0.4398 (0.43629614044638243) | > loader_time: 0.0062 (0.007003265268662397)  --> TIME: 2025-02-25 18:20:11 -- STEP: 67/72 -- GLOBAL_STEP: 1070700 | > loss_disc: 2.666707992553711 (2.573988647603277) | > loss_disc_real_0: 0.10191629081964493 (0.1229090124591073) | > loss_disc_real_1: 0.22863037884235382 (0.2180290482382276) | > loss_disc_real_2: 0.23290786147117615 (0.23197234121721183) | > loss_disc_real_3: 0.2357238531112671 (0.232489527383847) | > loss_disc_real_4: 0.24173182249069214 (0.2430502543253685) | > loss_disc_real_5: 0.2327931523323059 (0.24338278819375964) | > loss_0: 2.666707992553711 (2.573988647603277) | > grad_norm_0: tensor(344.0244, device='cuda:0') (tensor(344.6150, device='cuda:0')) | > loss_gen: 1.9648016691207886 (2.1194529337669485) | > loss_kl: 2.0828561782836914 (2.0030752278085964) | > loss_feat: 3.3222904205322266 (3.7056285765633654) | > loss_mel: 16.078502655029297 (16.555851352748586) | > loss_duration: 1.6294889450073242 (1.5419219287473764) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.07794189453125 (25.92592988085391) | > grad_norm_1: tensor(10674.3242, device='cuda:0') (tensor(12301.5547, device='cuda:0')) | > current_lr_0: 0.00017691770869678484 | > current_lr_1: 0.00017691770869678484 | > step_time: 0.4856 (0.45337219024772074) | > loader_time: 0.012 (0.008492911039893306) --> EVAL PERFORMANCE | > avg_loader_time: 0.008666451548186826 (+1.9651063731021198e-05) | > avg_step_time: 0.4555984248577709 (-0.0006741805815360657) | > avg_loss_disc: 2.5760849428848487 (-0.013518084942454323) | > avg_loss_disc_real_0: 0.1232087888138395 (-0.0016717152066633711) | > avg_loss_disc_real_1: 0.2185970729505512 (-0.005208112614255578) | > avg_loss_disc_real_2: 0.23180876714243015 (-0.0016356536620099893) | > avg_loss_disc_real_3: 0.232579026633585 (-0.002005760518597882) | > avg_loss_disc_real_4: 0.24215497727125462 (-0.0011684321601626024) | > avg_loss_disc_real_5: 0.24244505943546832 (-0.0038531313479786278) | > avg_loss_0: 2.5760849428848487 (-0.013518084942454323) | > avg_grad_norm_0: tensor(343.8333, device='cuda:0') (+tensor(46.5114, device='cuda:0')) | > avg_loss_gen: 2.117333002493415 (+0.04519394753684569) | > avg_loss_kl: 2.0068938698567136 (-0.09696706080100892) | > avg_loss_feat: 3.6967281254244524 (+0.15970361064857164) | > avg_loss_mel: 16.537134828701824 (-0.3328995234529799) | > avg_loss_duration: 1.5454157490125844 (-0.011919985354786) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.903505432773645 (-0.23688923473089218) | > avg_grad_norm_1: tensor(12203.1758, device='cuda:0') (+tensor(819.3721, device='cuda:0'))  > EPOCH: 982/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:20:14)   --> TIME: 2025-02-25 18:20:37 -- STEP: 45/72 -- GLOBAL_STEP: 1070750 | > loss_disc: 2.648488759994507 (2.5745227707756895) | > loss_disc_real_0: 0.15119631588459015 (0.12263638592428631) | > loss_disc_real_1: 0.27492666244506836 (0.22330224249098035) | > loss_disc_real_2: 0.21151430904865265 (0.22813881768120658) | > loss_disc_real_3: 0.240742027759552 (0.23277086648676132) | > loss_disc_real_4: 0.2455264925956726 (0.24070442153347862) | > loss_disc_real_5: 0.2510226368904114 (0.24315142101711698) | > loss_0: 2.648488759994507 (2.5745227707756895) | > grad_norm_0: tensor(429.6123, device='cuda:0') (tensor(303.9087, device='cuda:0')) | > loss_gen: 2.1366662979125977 (2.14391020403968) | > loss_kl: 2.1058008670806885 (1.9464307069778441) | > loss_feat: 3.61130952835083 (3.7897610982259113) | > loss_mel: 16.480947494506836 (16.54936428070068) | > loss_duration: 1.5644456148147583 (1.5201011048422919) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.899169921875 (25.94956724378798) | > grad_norm_1: tensor(10742.7744, device='cuda:0') (tensor(11205.3486, device='cuda:0')) | > current_lr_0: 0.00017689559398319774 | > current_lr_1: 0.00017689559398319774 | > step_time: 0.4633 (0.4390570322672526) | > loader_time: 0.009 (0.007518725925021701) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579821653769052 (-8.662989441777375e-05) | > avg_step_time: 0.46041329477874327 (+0.004814869920972353) | > avg_loss_disc: 2.5983802264844877 (+0.02229528359963906) | > avg_loss_disc_real_0: 0.12843221803786045 (+0.005223429224020956) | > avg_loss_disc_real_1: 0.2229035175602201 (+0.004306444609668891) | > avg_loss_disc_real_2: 0.23050604047070086 (-0.0013027266717292874) | > avg_loss_disc_real_3: 0.23236050782069353 (-0.00021851881289147723) | > avg_loss_disc_real_4: 0.24127831110652065 (-0.0008766661647339746) | > avg_loss_disc_real_5: 0.24468353600569173 (+0.0022384765702234133) | > avg_loss_0: 2.5983802264844877 (+0.02229528359963906) | > avg_grad_norm_0: tensor(324.4887, device='cuda:0') (tensor(-19.3446, device='cuda:0')) | > avg_loss_gen: 2.1080207337795835 (-0.009312268713831617) | > avg_loss_kl: 2.002800525074274 (-0.004093344782439612) | > avg_loss_feat: 3.68252433185846 (-0.014203793565992218) | > avg_loss_mel: 16.557481967227556 (+0.020347138525732333) | > avg_loss_duration: 1.5463041587614677 (+0.0008884097488832676) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.89713155719596 (-0.006373875577686761) | > avg_grad_norm_1: tensor(11378.8809, device='cuda:0') (tensor(-824.2949, device='cuda:0'))  > EPOCH: 983/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:20:52)   --> TIME: 2025-02-25 18:21:04 -- STEP: 23/72 -- GLOBAL_STEP: 1070800 | > loss_disc: 2.6204586029052734 (2.5668456347092343) | > loss_disc_real_0: 0.08736059069633484 (0.11729686642470567) | > loss_disc_real_1: 0.22293144464492798 (0.21491882593735404) | > loss_disc_real_2: 0.22885282337665558 (0.2337548693884974) | > loss_disc_real_3: 0.24794450402259827 (0.23277371409146683) | > loss_disc_real_4: 0.25611719489097595 (0.24350087279858795) | > loss_disc_real_5: 0.2727352976799011 (0.24257253499134726) | > loss_0: 2.6204586029052734 (2.5668456347092343) | > grad_norm_0: tensor(415.3323, device='cuda:0') (tensor(344.1795, device='cuda:0')) | > loss_gen: 2.178873062133789 (2.1442335844039917) | > loss_kl: 2.130441904067993 (1.9648948285890662) | > loss_feat: 3.719449758529663 (3.8899888784989067) | > loss_mel: 16.496448516845703 (16.853766565737516) | > loss_duration: 1.5186389684677124 (1.5003876737926318) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.043851852416992 (26.353271981944207) | > grad_norm_1: tensor(13645.4414, device='cuda:0') (tensor(13150.4033, device='cuda:0')) | > current_lr_0: 0.00017687348203394984 | > current_lr_1: 0.00017687348203394984 | > step_time: 0.4629 (0.4415454346200694) | > loader_time: 0.0068 (0.0067016871079154635) --> EVAL PERFORMANCE | > avg_loader_time: 0.008564630024869674 (-1.5191628899378223e-05) | > avg_step_time: 0.4584922085345631 (-0.0019210862441801924) | > avg_loss_disc: 2.612807461913203 (+0.014427235428715157) | > avg_loss_disc_real_0: 0.11865876940354494 (-0.009773448634315507) | > avg_loss_disc_real_1: 0.2160004036317409 (-0.006903113928479199) | > avg_loss_disc_real_2: 0.23270220445915007 (+0.0021961639884492123) | > avg_loss_disc_real_3: 0.23400165728280242 (+0.0016411494621088873) | > avg_loss_disc_real_4: 0.24140944699166525 (+0.00013113588514460095) | > avg_loss_disc_real_5: 0.2436298883297074 (-0.0010536476759843427) | > avg_loss_0: 2.612807461913203 (+0.014427235428715157) | > avg_grad_norm_0: tensor(342.5975, device='cuda:0') (+tensor(18.1088, device='cuda:0')) | > avg_loss_gen: 2.104139819951125 (-0.0038809138284583433) | > avg_loss_kl: 2.021099204748449 (+0.018298679674174867) | > avg_loss_feat: 3.7380855452846475 (+0.05556121342618736) | > avg_loss_mel: 16.67616151084363 (+0.11867954361607502) | > avg_loss_duration: 1.541620692736666 (-0.00468346602480163) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.081106750058456 (+0.18397519286249775) | > avg_grad_norm_1: tensor(12658.7314, device='cuda:0') (+tensor(1279.8506, device='cuda:0'))  > EPOCH: 984/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:21:29)   --> TIME: 2025-02-25 18:21:31 -- STEP: 1/72 -- GLOBAL_STEP: 1070850 | > loss_disc: 2.6043288707733154 (2.6043288707733154) | > loss_disc_real_0: 0.13714364171028137 (0.13714364171028137) | > loss_disc_real_1: 0.21002474427223206 (0.21002474427223206) | > loss_disc_real_2: 0.22801485657691956 (0.22801485657691956) | > loss_disc_real_3: 0.24538053572177887 (0.24538053572177887) | > loss_disc_real_4: 0.2474505454301834 (0.2474505454301834) | > loss_disc_real_5: 0.25877201557159424 (0.25877201557159424) | > loss_0: 2.6043288707733154 (2.6043288707733154) | > grad_norm_0: tensor(507.4194, device='cuda:0') (tensor(507.4194, device='cuda:0')) | > loss_gen: 2.2272086143493652 (2.2272086143493652) | > loss_kl: 1.8036847114562988 (1.8036847114562988) | > loss_feat: 3.4366543292999268 (3.4366543292999268) | > loss_mel: 16.340206146240234 (16.340206146240234) | > loss_duration: 1.4981743097305298 (1.4981743097305298) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.30592918395996 (25.30592918395996) | > grad_norm_1: tensor(10060.2490, device='cuda:0') (tensor(10060.2490, device='cuda:0')) | > current_lr_0: 0.00017685137284869558 | > current_lr_1: 0.00017685137284869558 | > step_time: 0.4325 (0.4324991703033447) | > loader_time: 0.0058 (0.005814552307128906)  --> TIME: 2025-02-25 18:21:54 -- STEP: 51/72 -- GLOBAL_STEP: 1070900 | > loss_disc: 2.733799457550049 (2.6304235271379066) | > loss_disc_real_0: 0.1357717514038086 (0.13400711119174954) | > loss_disc_real_1: 0.2451147437095642 (0.228981099876703) | > loss_disc_real_2: 0.23207828402519226 (0.23815327678240983) | > loss_disc_real_3: 0.24088147282600403 (0.2399320038510304) | > loss_disc_real_4: 0.2460276335477829 (0.24371171698850744) | > loss_disc_real_5: 0.2613927125930786 (0.24725951137495975) | > loss_0: 2.733799457550049 (2.6304235271379066) | > grad_norm_0: tensor(285.6810, device='cuda:0') (tensor(340.9327, device='cuda:0')) | > loss_gen: 2.0051705837249756 (2.128674254697913) | > loss_kl: 2.053988218307495 (1.991174162602892) | > loss_feat: 3.3055479526519775 (3.649903241325827) | > loss_mel: 16.188627243041992 (16.70993436551561) | > loss_duration: 1.5731019973754883 (1.5245517632540535) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.12643814086914 (26.00423812866211) | > grad_norm_1: tensor(9618.8896, device='cuda:0') (tensor(11203.2646, device='cuda:0')) | > current_lr_0: 0.00017685137284869558 | > current_lr_1: 0.00017685137284869558 | > step_time: 0.4532 (0.4431437838311289) | > loader_time: 0.0089 (0.007827066907695695) --> EVAL PERFORMANCE | > avg_loader_time: 0.008650014098261443 (+8.53840733917692e-05) | > avg_step_time: 0.4549772605090074 (-0.0035149480255556864) | > avg_loss_disc: 2.630719181517481 (+0.017911719604278176) | > avg_loss_disc_real_0: 0.13745904157698993 (+0.018800272173444987) | > avg_loss_disc_real_1: 0.22525759188222214 (+0.009257188250481246) | > avg_loss_disc_real_2: 0.23589452303631206 (+0.0031923185771619844) | > avg_loss_disc_real_3: 0.23943261823184053 (+0.0054309609490381106) | > avg_loss_disc_real_4: 0.2441305931605084 (+0.0027211461688431415) | > avg_loss_disc_real_5: 0.24551461699982763 (+0.0018847286701202393) | > avg_loss_0: 2.630719181517481 (+0.017911719604278176) | > avg_grad_norm_0: tensor(339.4714, device='cuda:0') (tensor(-3.1261, device='cuda:0')) | > avg_loss_gen: 2.105613951951686 (+0.0014741320005606795) | > avg_loss_kl: 2.037702996965865 (+0.016603792217416036) | > avg_loss_feat: 3.5761556961167025 (-0.16192984916794506) | > avg_loss_mel: 16.61177197308607 (-0.06438953775755962) | > avg_loss_duration: 1.5441009175609535 (+0.002480224824287447) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.87534584797604 (-0.20576090208241737) | > avg_grad_norm_1: tensor(11073.6650, device='cuda:0') (tensor(-1585.0664, device='cuda:0'))  > EPOCH: 985/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:22:06)   --> TIME: 2025-02-25 18:22:21 -- STEP: 29/72 -- GLOBAL_STEP: 1070950 | > loss_disc: 2.547311305999756 (2.6179232268497863) | > loss_disc_real_0: 0.1277676820755005 (0.13145367621347823) | > loss_disc_real_1: 0.19621120393276215 (0.23143863215528684) | > loss_disc_real_2: 0.2505187690258026 (0.23521702916457735) | > loss_disc_real_3: 0.26137107610702515 (0.23533049379957133) | > loss_disc_real_4: 0.2736268639564514 (0.24398240806727572) | > loss_disc_real_5: 0.26668781042099 (0.24797765859242143) | > loss_0: 2.547311305999756 (2.6179232268497863) | > grad_norm_0: tensor(210.8426, device='cuda:0') (tensor(284.5458, device='cuda:0')) | > loss_gen: 2.322070360183716 (2.1486427167366298) | > loss_kl: 2.124481439590454 (1.9924718634835605) | > loss_feat: 3.793799638748169 (3.7917038901098845) | > loss_mel: 18.08806610107422 (16.89681023564832) | > loss_duration: 1.5342596769332886 (1.5038297834067509) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.86267852783203 (26.33345860448377) | > grad_norm_1: tensor(11315.7764, device='cuda:0') (tensor(9862.0762, device='cuda:0')) | > current_lr_0: 0.0001768292664270895 | > current_lr_1: 0.0001768292664270895 | > step_time: 0.4302 (0.4330820790652571) | > loader_time: 0.0075 (0.006877035930238921) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860056742815904 (-4.9446670102403986e-05) | > avg_step_time: 0.4537110060033664 (-0.0012662545056409957) | > avg_loss_disc: 2.6339131644074345 (+0.0031939828899534284) | > avg_loss_disc_real_0: 0.1316319278008501 (-0.005827113776139842) | > avg_loss_disc_real_1: 0.23018300071568557 (+0.004925408833463429) | > avg_loss_disc_real_2: 0.233547996887019 (-0.0023465261492930656) | > avg_loss_disc_real_3: 0.23624649811798418 (-0.0031861201138563555) | > avg_loss_disc_real_4: 0.24229922219061514 (-0.0018313709698932479) | > avg_loss_disc_real_5: 0.24595446926607212 (+0.00043985226624448925) | > avg_loss_0: 2.6339131644074345 (+0.0031939828899534284) | > avg_grad_norm_0: tensor(302.2938, device='cuda:0') (tensor(-37.1776, device='cuda:0')) | > avg_loss_gen: 2.116394445929728 (+0.010780493978042038) | > avg_loss_kl: 2.0909050602308463 (+0.05320206326498145) | > avg_loss_feat: 3.6986255645751953 (+0.12246986845849284) | > avg_loss_mel: 16.764528449152564 (+0.152756476066493) | > avg_loss_duration: 1.5495406103805758 (+0.0054396928196223104) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.219994262910227 (+0.34464841493418774) | > avg_grad_norm_1: tensor(10444.3203, device='cuda:0') (tensor(-629.3447, device='cuda:0'))  > EPOCH: 986/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:22:43)   --> TIME: 2025-02-25 18:22:48 -- STEP: 7/72 -- GLOBAL_STEP: 1071000 | > loss_disc: 2.6373116970062256 (2.585263354437692) | > loss_disc_real_0: 0.11097954958677292 (0.12553381813423975) | > loss_disc_real_1: 0.2027316838502884 (0.22323259711265564) | > loss_disc_real_2: 0.22397015988826752 (0.2330263022865568) | > loss_disc_real_3: 0.25518500804901123 (0.22914800686495645) | > loss_disc_real_4: 0.25239548087120056 (0.23689950151102884) | > loss_disc_real_5: 0.24746668338775635 (0.2416103184223175) | > loss_0: 2.6373116970062256 (2.585263354437692) | > grad_norm_0: tensor(174.7791, device='cuda:0') (tensor(236.2017, device='cuda:0')) | > loss_gen: 2.0160913467407227 (2.1374192237854004) | > loss_kl: 2.127732276916504 (1.96433082648686) | > loss_feat: 3.644787311553955 (3.8780898366655623) | > loss_mel: 16.920276641845703 (17.189746039254324) | > loss_duration: 1.4878926277160645 (1.5069004808153426) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.196781158447266 (26.67648696899414) | > grad_norm_1: tensor(7203.8018, device='cuda:0') (tensor(7859.3794, device='cuda:0')) | > current_lr_0: 0.0001768071627687861 | > current_lr_1: 0.0001768071627687861 | > step_time: 0.4407 (0.4346216065543039) | > loader_time: 0.0069 (0.0067402635301862445)  --> TIME: 2025-02-25 18:23:11 -- STEP: 57/72 -- GLOBAL_STEP: 1071050 | > loss_disc: 2.813265323638916 (2.6456180539047507) | > loss_disc_real_0: 0.07222713530063629 (0.155748234440883) | > loss_disc_real_1: 0.2504081428050995 (0.22444513074138708) | > loss_disc_real_2: 0.22923262417316437 (0.23304458171652073) | > loss_disc_real_3: 0.2095474898815155 (0.23421980492901384) | > loss_disc_real_4: 0.2304360568523407 (0.240611854613873) | > loss_disc_real_5: 0.23360803723335266 (0.24462496659211946) | > loss_0: 2.813265323638916 (2.6456180539047507) | > grad_norm_0: tensor(100.7884, device='cuda:0') (tensor(297.8971, device='cuda:0')) | > loss_gen: 1.8808975219726562 (2.1060708782129116) | > loss_kl: 2.146172523498535 (2.0615969770833065) | > loss_feat: 3.1537561416625977 (3.6455537478129068) | > loss_mel: 16.543981552124023 (16.878611748678633) | > loss_duration: 1.5856986045837402 (1.5363438443133703) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.31050682067871 (26.2281773215846) | > grad_norm_1: tensor(8528.7627, device='cuda:0') (tensor(9346.3945, device='cuda:0')) | > current_lr_0: 0.0001768071627687861 | > current_lr_1: 0.0001768071627687861 | > step_time: 0.4669 (0.45367410308436346) | > loader_time: 0.0099 (0.007990561033550059) --> EVAL PERFORMANCE | > avg_loader_time: 0.008616611991130127 (+1.604456297108793e-05) | > avg_step_time: 0.46075920991494623 (+0.007048203911579842) | > avg_loss_disc: 2.6445498432911627 (+0.010636678883728212) | > avg_loss_disc_real_0: 0.1575024408668699 (+0.025870513066019812) | > avg_loss_disc_real_1: 0.22591242802814698 (-0.004270572687538582) | > avg_loss_disc_real_2: 0.23484931214594504 (+0.0013013152589260446) | > avg_loss_disc_real_3: 0.23543623782379527 (-0.0008102602941889026) | > avg_loss_disc_real_4: 0.24203753996063287 (-0.00026168222998226853) | > avg_loss_disc_real_5: 0.24678140165100634 (+0.0008269323849342258) | > avg_loss_0: 2.6445498432911627 (+0.010636678883728212) | > avg_grad_norm_0: tensor(296.7928, device='cuda:0') (tensor(-5.5010, device='cuda:0')) | > avg_loss_gen: 2.114676366389637 (-0.0017180795400908444) | > avg_loss_kl: 2.085793097254256 (-0.005111962976590334) | > avg_loss_feat: 3.6084522395066814 (-0.09017332506851394) | > avg_loss_mel: 16.868976190056593 (+0.10444774090402831) | > avg_loss_duration: 1.5498415987256544 (+0.00030098834507863437) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.227739549018967 (+0.007745286108740146) | > avg_grad_norm_1: tensor(9288.7529, device='cuda:0') (tensor(-1155.5674, device='cuda:0'))  > EPOCH: 987/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:23:20)   --> TIME: 2025-02-25 18:23:38 -- STEP: 35/72 -- GLOBAL_STEP: 1071100 | > loss_disc: 2.6048707962036133 (2.65220160484314) | > loss_disc_real_0: 0.08204469084739685 (0.14977498650550838) | > loss_disc_real_1: 0.12393953651189804 (0.22595050441367284) | > loss_disc_real_2: 0.286838173866272 (0.23000076711177825) | > loss_disc_real_3: 0.2105521559715271 (0.24519041180610657) | > loss_disc_real_4: 0.31413733959198 (0.2508955180644989) | > loss_disc_real_5: 0.25982069969177246 (0.24725120450769153) | > loss_0: 2.6048707962036133 (2.65220160484314) | > grad_norm_0: tensor(146.9073, device='cuda:0') (tensor(287.3842, device='cuda:0')) | > loss_gen: 2.4318346977233887 (2.165332954270499) | > loss_kl: 2.2148900032043457 (2.006990500858852) | > loss_feat: 4.474785804748535 (3.8379479476383755) | > loss_mel: 17.340682983398438 (16.996252196175714) | > loss_duration: 1.5869755744934082 (1.507325577735901) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.049169540405273 (26.513849203927176) | > grad_norm_1: tensor(2231.3435, device='cuda:0') (tensor(10160.4443, device='cuda:0')) | > current_lr_0: 0.00017678506187344 | > current_lr_1: 0.00017678506187344 | > step_time: 0.4358 (0.43541526113237655) | > loader_time: 0.0082 (0.007345383507864816) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619308471679688 (+2.6964805495601624e-06) | > avg_step_time: 0.45351700044014087 (-0.007242209474805361) | > avg_loss_disc: 2.6946824872997452 (+0.05013264400858253) | > avg_loss_disc_real_0: 0.17449812255275082 (+0.01699568168588092) | > avg_loss_disc_real_1: 0.2318829561413174 (+0.00597052811317042) | > avg_loss_disc_real_2: 0.2375952143904189 (+0.0027459022444738523) | > avg_loss_disc_real_3: 0.24178824668199245 (+0.006352008858197172) | > avg_loss_disc_real_4: 0.24529035284485615 (+0.0032528128842232717) | > avg_loss_disc_real_5: 0.2442942801075922 (-0.002487121543414156) | > avg_loss_0: 2.6946824872997452 (+0.05013264400858253) | > avg_grad_norm_0: tensor(259.3471, device='cuda:0') (tensor(-37.4456, device='cuda:0')) | > avg_loss_gen: 2.153870977146523 (+0.03919461075688613) | > avg_loss_kl: 2.1292484797222504 (+0.04345538246799441) | > avg_loss_feat: 3.72499370910752 (+0.11654146960083844) | > avg_loss_mel: 17.111568101694886 (+0.24259191163829286) | > avg_loss_duration: 1.5644373121395918 (+0.014595713413937395) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.684118646970937 (+0.4563790979519702) | > avg_grad_norm_1: tensor(7771.0396, device='cuda:0') (tensor(-1517.7134, device='cuda:0'))  > EPOCH: 988/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:23:57)   --> TIME: 2025-02-25 18:24:05 -- STEP: 13/72 -- GLOBAL_STEP: 1071150 | > loss_disc: 2.945950984954834 (2.927668443092933) | > loss_disc_real_0: 0.1083851158618927 (0.28403542133478016) | > loss_disc_real_1: 0.2197273224592209 (0.22646141510743362) | > loss_disc_real_2: 0.17403827607631683 (0.2802055237384943) | > loss_disc_real_3: 0.20941351354122162 (0.2709688028463951) | > loss_disc_real_4: 0.24566933512687683 (0.2573702289507939) | > loss_disc_real_5: 0.25082749128341675 (0.23891515112840211) | > loss_0: 2.945950984954834 (2.927668443092933) | > grad_norm_0: tensor(54.4275, device='cuda:0') (tensor(61.7234, device='cuda:0')) | > loss_gen: 1.7862098217010498 (1.96671619782081) | > loss_kl: 3.0485222339630127 (2.500242123237023) | > loss_feat: 3.1766443252563477 (3.2461905479431152) | > loss_mel: 20.155101776123047 (19.047467891986553) | > loss_duration: 1.5554949045181274 (1.5622299084296594) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.72197151184082 (28.32284663273738) | > grad_norm_1: tensor(3410.2812, device='cuda:0') (tensor(1255.3767, device='cuda:0')) | > current_lr_0: 0.0001767629637407058 | > current_lr_1: 0.0001767629637407058 | > step_time: 0.4452 (0.4325067446782039) | > loader_time: 0.0058 (0.006661140001737154)  --> TIME: 2025-02-25 18:24:28 -- STEP: 63/72 -- GLOBAL_STEP: 1071200 | > loss_disc: 2.8104445934295654 (2.783375823308551) | > loss_disc_real_0: 0.3165682554244995 (0.21789398954974284) | > loss_disc_real_1: 0.24413801729679108 (0.2439745572351274) | > loss_disc_real_2: 0.24282395839691162 (0.24422502825184475) | > loss_disc_real_3: 0.2409195452928543 (0.24548676609992978) | > loss_disc_real_4: 0.2548196315765381 (0.2456919501225154) | > loss_disc_real_5: 0.26507946848869324 (0.23655857405965292) | > loss_0: 2.8104445934295654 (2.783375823308551) | > grad_norm_0: tensor(137.1149, device='cuda:0') (tensor(173.7262, device='cuda:0')) | > loss_gen: 2.1655664443969727 (2.0286903248892894) | > loss_kl: 2.2675042152404785 (2.435764354372781) | > loss_feat: 3.0473885536193848 (3.388803315541101) | > loss_mel: 18.218486785888672 (18.405675736684646) | > loss_duration: 1.6044118404388428 (1.5868855941863287) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.303356170654297 (27.8458192310636) | > grad_norm_1: tensor(5021.4458, device='cuda:0') (tensor(4553.0898, device='cuda:0')) | > current_lr_0: 0.0001767629637407058 | > current_lr_1: 0.0001767629637407058 | > step_time: 0.5068 (0.44937043719821507) | > loader_time: 0.0113 (0.00826119998144725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627384481295733 (+8.07600961604503e-06) | > avg_step_time: 0.4544466112700986 (+0.0009296108299577388) | > avg_loss_disc: 2.7770910095161114 (+0.08240852221636619) | > avg_loss_disc_real_0: 0.21095362977242807 (+0.03645550721967725) | > avg_loss_disc_real_1: 0.24307561235528596 (+0.01119265621396856) | > avg_loss_disc_real_2: 0.2442897533027219 (+0.006694538912303011) | > avg_loss_disc_real_3: 0.24490320892401143 (+0.003114962242018987) | > avg_loss_disc_real_4: 0.24623004693380549 (+0.0009396940889493388) | > avg_loss_disc_real_5: 0.2376175692803423 (-0.006676710827249882) | > avg_loss_0: 2.7770910095161114 (+0.08240852221636619) | > avg_grad_norm_0: tensor(179.0344, device='cuda:0') (tensor(-80.3127, device='cuda:0')) | > avg_loss_gen: 2.0241723362828647 (-0.12969864086365845) | > avg_loss_kl: 2.4372355602156945 (+0.30798708049344414) | > avg_loss_feat: 3.3842893150490774 (-0.34070439405844244) | > avg_loss_mel: 18.434640830671285 (+1.323072728976399) | > avg_loss_duration: 1.5932347505864963 (+0.028797438446904522) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.87357271221322 (+1.189454065242284) | > avg_grad_norm_1: tensor(4783.1401, device='cuda:0') (tensor(-2987.8994, device='cuda:0'))  > EPOCH: 989/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:24:34)   --> TIME: 2025-02-25 18:24:55 -- STEP: 41/72 -- GLOBAL_STEP: 1071250 | > loss_disc: 2.610093593597412 (2.6670229318665295) | > loss_disc_real_0: 0.15682755410671234 (0.1470779946482763) | > loss_disc_real_1: 0.2352859228849411 (0.22830422731434427) | > loss_disc_real_2: 0.22860001027584076 (0.24005276837000034) | > loss_disc_real_3: 0.26687926054000854 (0.24515252869303633) | > loss_disc_real_4: 0.2615496516227722 (0.24115852321066508) | > loss_disc_real_5: 0.2600610852241516 (0.2484864995973866) | > loss_0: 2.610093593597412 (2.6670229318665295) | > grad_norm_0: tensor(61.1363, device='cuda:0') (tensor(159.1098, device='cuda:0')) | > loss_gen: 2.330118179321289 (2.1187202348941714) | > loss_kl: 2.201228618621826 (2.09098954026292) | > loss_feat: 4.0805559158325195 (3.584407457491247) | > loss_mel: 17.476207733154297 (17.642800400896764) | > loss_duration: 1.5829031467437744 (1.5434154650060141) | > amp_scaler: 16.0 (13.268292682926823) | > loss_1: 27.67101287841797 (26.980333142164277) | > grad_norm_1: tensor(2212.1367, device='cuda:0') (tensor(5655.9092, device='cuda:0')) | > current_lr_0: 0.0001767408683702382 | > current_lr_1: 0.0001767408683702382 | > step_time: 0.4545 (0.4507435531151004) | > loader_time: 0.0088 (0.007294119858160251) --> EVAL PERFORMANCE | > avg_loader_time: 0.008516711248478416 (-0.00011067323281731614) | > avg_step_time: 0.46069686177750707 (+0.006250250507408461) | > avg_loss_disc: 2.65809273048186 (-0.11899827903425164) | > avg_loss_disc_real_0: 0.1297295436682835 (-0.08122408610414458) | > avg_loss_disc_real_1: 0.22455314200528914 (-0.018522470349996822) | > avg_loss_disc_real_2: 0.23766361344868028 (-0.006626139854041618) | > avg_loss_disc_real_3: 0.23934614490455305 (-0.00555706401945838) | > avg_loss_disc_real_4: 0.24209908889213078 (-0.004130958041674709) | > avg_loss_disc_real_5: 0.24707146093878948 (+0.009453891658447178) | > avg_loss_0: 2.65809273048186 (-0.11899827903425164) | > avg_grad_norm_0: tensor(184.9842, device='cuda:0') (+tensor(5.9499, device='cuda:0')) | > avg_loss_gen: 2.058713147338008 (+0.03454081105514328) | > avg_loss_kl: 2.095758575788687 (-0.3414769844270076) | > avg_loss_feat: 3.5677067595468435 (+0.1834174444977661) | > avg_loss_mel: 17.4152780183604 (-1.0193628123108844) | > avg_loss_duration: 1.5676137964490433 (-0.025620954137453023) | > avg_amp_scaler: 14.422535211267602 (+6.422535211267602) | > avg_loss_1: 26.705070280692947 (-1.1685024315202739) | > avg_grad_norm_1: tensor(5834.1611, device='cuda:0') (+tensor(1051.0210, device='cuda:0'))  > EPOCH: 990/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:25:11)   --> TIME: 2025-02-25 18:25:22 -- STEP: 19/72 -- GLOBAL_STEP: 1071300 | > loss_disc: 2.423595428466797 (2.517873713844701) | > loss_disc_real_0: 0.07923781126737595 (0.08831181416386052) | > loss_disc_real_1: 0.24823281168937683 (0.21476954221725464) | > loss_disc_real_2: 0.20105747878551483 (0.22986258331098056) | > loss_disc_real_3: 0.24102997779846191 (0.23617682958904065) | > loss_disc_real_4: 0.23770801723003387 (0.2457699014952308) | > loss_disc_real_5: 0.26983925700187683 (0.24712668045570976) | > loss_0: 2.423595428466797 (2.517873713844701) | > grad_norm_0: tensor(114.7359, device='cuda:0') (tensor(197.0986, device='cuda:0')) | > loss_gen: 2.36098575592041 (2.1442826860829403) | > loss_kl: 1.8491743803024292 (1.9624830860840647) | > loss_feat: 4.277551174163818 (3.9560875516188774) | > loss_mel: 16.654516220092773 (17.287436435097142) | > loss_duration: 1.4965335130691528 (1.5236298975191618) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.638761520385742 (26.873919838353206) | > grad_norm_1: tensor(5315.8726, device='cuda:0') (tensor(7401.8447, device='cuda:0')) | > current_lr_0: 0.00017671877576169192 | > current_lr_1: 0.00017671877576169192 | > step_time: 0.4384 (0.43486076907107707) | > loader_time: 0.0062 (0.00661108368321469)  --> TIME: 2025-02-25 18:25:46 -- STEP: 69/72 -- GLOBAL_STEP: 1071350 | > loss_disc: 2.957145929336548 (2.5516617125359136) | > loss_disc_real_0: 0.38872143626213074 (0.10052459168693295) | > loss_disc_real_1: 0.2116188108921051 (0.2157966226771258) | > loss_disc_real_2: 0.24328193068504333 (0.22987555658471756) | > loss_disc_real_3: 0.2270534485578537 (0.2308066409567128) | > loss_disc_real_4: 0.23900923132896423 (0.24390376823535864) | > loss_disc_real_5: 0.25592368841171265 (0.243533700056698) | > loss_0: 2.957145929336548 (2.5516617125359136) | > grad_norm_0: tensor(423.2770, device='cuda:0') (tensor(246.1963, device='cuda:0')) | > loss_gen: 1.8564784526824951 (2.1239099899927765) | > loss_kl: 2.1824774742126465 (1.9919609798901323) | > loss_feat: 2.7866616249084473 (3.8589995494787246) | > loss_mel: 16.230615615844727 (16.968171064404473) | > loss_duration: 1.666903018951416 (1.557144235873568) | > amp_scaler: 8.0 (15.420289855072463) | > loss_1: 24.723134994506836 (26.500185814456664) | > grad_norm_1: tensor(4130.5122, device='cuda:0') (tensor(7819.4160, device='cuda:0')) | > current_lr_0: 0.00017671877576169192 | > current_lr_1: 0.00017671877576169192 | > step_time: 0.5394 (0.456848687019901) | > loader_time: 0.0125 (0.008533280828724734) --> EVAL PERFORMANCE | > avg_loader_time: 0.008583982225874778 (+6.727097739636125e-05) | > avg_step_time: 0.4581993835073122 (-0.002497478270194864) | > avg_loss_disc: 2.5492810397080974 (-0.1088116907737624) | > avg_loss_disc_real_0: 0.10019760993374907 (-0.029531933734534416) | > avg_loss_disc_real_1: 0.21568311142249846 (-0.008870030582790678) | > avg_loss_disc_real_2: 0.2299868942986072 (-0.007676719150073091) | > avg_loss_disc_real_3: 0.23067907317423483 (-0.008667071730318221) | > avg_loss_disc_real_4: 0.24354105977944926 (+0.0014419708873184833) | > avg_loss_disc_real_5: 0.2432967485256598 (-0.0037747124131296828) | > avg_loss_0: 2.5492810397080974 (-0.1088116907737624) | > avg_grad_norm_0: tensor(242.9427, device='cuda:0') (+tensor(57.9585, device='cuda:0')) | > avg_loss_gen: 2.126334437182251 (+0.06762128984424276) | > avg_loss_kl: 1.995064270328468 (-0.10069430546021896) | > avg_loss_feat: 3.876834506719885 (+0.30912774717304137) | > avg_loss_mel: 16.964805294090592 (-0.4504727242698081) | > avg_loss_duration: 1.5575701871388397 (-0.01004360931020365) | > avg_amp_scaler: 15.211267605633802 (+0.7887323943661997) | > avg_loss_1: 26.520608687065018 (-0.1844615936279297) | > avg_grad_norm_1: tensor(7783.8281, device='cuda:0') (+tensor(1949.6670, device='cuda:0'))  > EPOCH: 991/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:25:48)   --> TIME: 2025-02-25 18:26:12 -- STEP: 47/72 -- GLOBAL_STEP: 1071400 | > loss_disc: 2.5717201232910156 (2.602555604691202) | > loss_disc_real_0: 0.10689844191074371 (0.08888590212990627) | > loss_disc_real_1: 0.22855530679225922 (0.21997715691302686) | > loss_disc_real_2: 0.22196878492832184 (0.2576496718411749) | > loss_disc_real_3: 0.24268895387649536 (0.265656608850398) | > loss_disc_real_4: 0.24039767682552338 (0.2544608930958079) | > loss_disc_real_5: 0.2538297474384308 (0.24821700060621224) | > loss_0: 2.5717201232910156 (2.602555604691202) | > grad_norm_0: tensor(349.4359, device='cuda:0') (tensor(267.5694, device='cuda:0')) | > loss_gen: 2.145637035369873 (2.1986262519308863) | > loss_kl: 2.10725736618042 (1.9115104345565146) | > loss_feat: 3.799665689468384 (3.9384654836451753) | > loss_mel: 16.659278869628906 (16.77890321041676) | > loss_duration: 1.5877009630203247 (1.5298104286193848) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.299541473388672 (26.357315672204848) | > grad_norm_1: tensor(6618.9414, device='cuda:0') (tensor(8589.4434, device='cuda:0')) | > current_lr_0: 0.0001766966859147217 | > current_lr_1: 0.0001766966859147217 | > step_time: 0.4493 (0.44659252369657476) | > loader_time: 0.0089 (0.007587346624820791) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862622596848179 (+4.224374260701318e-05) | > avg_step_time: 0.4578768098858041 (-0.0003225736215081154) | > avg_loss_disc: 2.5845249135729302 (+0.03524387386483285) | > avg_loss_disc_real_0: 0.08804115844549425 (-0.012156451488254824) | > avg_loss_disc_real_1: 0.22018358552120101 (+0.0045004740987025504) | > avg_loss_disc_real_2: 0.2503571747474266 (+0.02037028044881939) | > avg_loss_disc_real_3: 0.25577024380925684 (+0.025091170635022003) | > avg_loss_disc_real_4: 0.24970393344550076 (+0.006162873666051505) | > avg_loss_disc_real_5: 0.2480504869575232 (+0.004753738431863386) | > avg_loss_0: 2.5845249135729302 (+0.03524387386483285) | > avg_grad_norm_0: tensor(260.0841, device='cuda:0') (+tensor(17.1414, device='cuda:0')) | > avg_loss_gen: 2.174365498650241 (+0.04803106146799019) | > avg_loss_kl: 1.9446794869194568 (-0.0503847834090112) | > avg_loss_feat: 3.864637918875251 (-0.012196587844633733) | > avg_loss_mel: 16.66059125980861 (-0.30421403428198346) | > avg_loss_duration: 1.5502487202765236 (-0.007321466862316051) | > avg_amp_scaler: 8.0 (-7.211267605633802) | > avg_loss_1: 26.19452277707382 (-0.32608590999119613) | > avg_grad_norm_1: tensor(8708.9092, device='cuda:0') (+tensor(925.0811, device='cuda:0'))  > EPOCH: 992/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:26:25)   --> TIME: 2025-02-25 18:26:39 -- STEP: 25/72 -- GLOBAL_STEP: 1071450 | > loss_disc: 2.3312675952911377 (2.515242547988892) | > loss_disc_real_0: 0.05912836641073227 (0.0884538708627224) | > loss_disc_real_1: 0.1875687688589096 (0.2180353194475174) | > loss_disc_real_2: 0.19934554398059845 (0.23350593388080598) | > loss_disc_real_3: 0.2132708877325058 (0.23080117285251617) | > loss_disc_real_4: 0.2131025195121765 (0.23812144815921785) | > loss_disc_real_5: 0.2379254400730133 (0.24398382782936096) | > loss_0: 2.3312675952911377 (2.515242547988892) | > grad_norm_0: tensor(164.6389, device='cuda:0') (tensor(274.1859, device='cuda:0')) | > loss_gen: 2.2287044525146484 (2.1960927009582516) | > loss_kl: 1.882882833480835 (1.8957813024520873) | > loss_feat: 4.3102898597717285 (3.9687837505340577) | > loss_mel: 16.762672424316406 (16.550796813964844) | > loss_duration: 1.5156776905059814 (1.500328631401062) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.700227737426758 (26.11178352355957) | > grad_norm_1: tensor(9436.5635, device='cuda:0') (tensor(8497.8496, device='cuda:0')) | > current_lr_0: 0.00017667459882898234 | > current_lr_1: 0.00017667459882898234 | > step_time: 0.4312 (0.4340526485443115) | > loader_time: 0.0073 (0.006941499710083007) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866436286711357 (+3.813689863177948e-05) | > avg_step_time: 0.4543002021144813 (-0.003576607771322793) | > avg_loss_disc: 2.5419831779641173 (-0.042541735608812914) | > avg_loss_disc_real_0: 0.10094682080015328 (+0.012905662354659037) | > avg_loss_disc_real_1: 0.2173307370132124 (-0.0028528485079886146) | > avg_loss_disc_real_2: 0.23356034722126706 (-0.01679682752615952) | > avg_loss_disc_real_3: 0.23590245591083042 (-0.019867787898426414) | > avg_loss_disc_real_4: 0.241426321822153 (-0.008277611623347753) | > avg_loss_disc_real_5: 0.2457531640227412 (-0.0022973229347819923) | > avg_loss_0: 2.5419831779641173 (-0.042541735608812914) | > avg_grad_norm_0: tensor(257.0459, device='cuda:0') (tensor(-3.0382, device='cuda:0')) | > avg_loss_gen: 2.2023232789106766 (+0.027957780260435605) | > avg_loss_kl: 1.9631268239357103 (+0.01844733701625345) | > avg_loss_feat: 3.901849132188609 (+0.03721121331335775) | > avg_loss_mel: 16.603560286508486 (-0.05703097330012241) | > avg_loss_duration: 1.5439051977345644 (-0.006343522541959246) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.21476471592003 (+0.020241938846208285) | > avg_grad_norm_1: tensor(7949.6733, device='cuda:0') (tensor(-759.2358, device='cuda:0'))  > EPOCH: 993/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:27:03)   --> TIME: 2025-02-25 18:27:06 -- STEP: 3/72 -- GLOBAL_STEP: 1071500 | > loss_disc: 2.635568380355835 (2.5601197878519693) | > loss_disc_real_0: 0.08044092357158661 (0.055956028401851654) | > loss_disc_real_1: 0.24074652791023254 (0.21090270578861237) | > loss_disc_real_2: 0.2458384931087494 (0.2221467693646749) | > loss_disc_real_3: 0.22342875599861145 (0.20040927827358246) | > loss_disc_real_4: 0.25887954235076904 (0.23946787416934967) | > loss_disc_real_5: 0.2818793058395386 (0.2633807609478633) | > loss_0: 2.635568380355835 (2.5601197878519693) | > grad_norm_0: tensor(183.1514, device='cuda:0') (tensor(186.3917, device='cuda:0')) | > loss_gen: 2.0628607273101807 (2.098634441693624) | > loss_kl: 1.8596060276031494 (1.8886822859446208) | > loss_feat: 3.389431953430176 (4.048169294993083) | > loss_mel: 16.412216186523438 (16.94422976175944) | > loss_duration: 1.5202614068984985 (1.5104178190231323) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.24437713623047 (26.49013392130534) | > grad_norm_1: tensor(1987.7529, device='cuda:0') (tensor(5851.3608, device='cuda:0')) | > current_lr_0: 0.00017665251450412872 | > current_lr_1: 0.00017665251450412872 | > step_time: 0.4246 (0.4381038347880046) | > loader_time: 0.0066 (0.006879170735677083)  --> TIME: 2025-02-25 18:27:29 -- STEP: 53/72 -- GLOBAL_STEP: 1071550 | > loss_disc: 3.2187118530273438 (2.70939195830867) | > loss_disc_real_0: 0.7357625961303711 (0.2137575327366028) | > loss_disc_real_1: 0.11790823936462402 (0.22836557747620456) | > loss_disc_real_2: 0.2328542172908783 (0.2362042876347056) | > loss_disc_real_3: 0.2657623291015625 (0.23753739162436072) | > loss_disc_real_4: 0.2524835169315338 (0.2469425842447101) | > loss_disc_real_5: 0.2409505993127823 (0.2501692119634377) | > loss_0: 3.2187118530273438 (2.70939195830867) | > grad_norm_0: tensor(512.8249, device='cuda:0') (tensor(285.4568, device='cuda:0')) | > loss_gen: 2.2497291564941406 (2.208716450997118) | > loss_kl: 2.255129098892212 (2.0132209399961076) | > loss_feat: 3.1443917751312256 (3.634467264391341) | > loss_mel: 15.036599159240723 (16.74775812760839) | > loss_duration: 1.601209044456482 (1.5356036379652203) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.287057876586914 (26.139766261262714) | > grad_norm_1: tensor(187.9943, device='cuda:0') (tensor(6185.6792, device='cuda:0')) | > current_lr_0: 0.00017665251450412872 | > current_lr_1: 0.00017665251450412872 | > step_time: 0.4561 (0.44778419890493715) | > loader_time: 0.0091 (0.00788394460138285) --> EVAL PERFORMANCE | > avg_loader_time: 0.008615399750185685 (-4.896311692788487e-05) | > avg_step_time: 0.4601167154983735 (+0.005816513383892219) | > avg_loss_disc: 2.7744703595067413 (+0.23248718154262393) | > avg_loss_disc_real_0: 0.20155102801574787 (+0.10060420721559458) | > avg_loss_disc_real_1: 0.26984522533668603 (+0.052514488323473635) | > avg_loss_disc_real_2: 0.24161362207271683 (+0.008053274851449771) | > avg_loss_disc_real_3: 0.2342321555799162 (-0.0016703003309142261) | > avg_loss_disc_real_4: 0.25123122082629673 (+0.009804899004143719) | > avg_loss_disc_real_5: 0.25746010096979816 (+0.011706936947056962) | > avg_loss_0: 2.7744703595067413 (+0.23248718154262393) | > avg_grad_norm_0: tensor(274.7284, device='cuda:0') (+tensor(17.6825, device='cuda:0')) | > avg_loss_gen: 2.1542450172800414 (-0.04807826163063522) | > avg_loss_kl: 2.2206581948508672 (+0.257531370915157) | > avg_loss_feat: 3.5065996982681917 (-0.3952494339204171) | > avg_loss_mel: 17.31992561716428 (+0.716365330655794) | > avg_loss_duration: 1.5664439234935061 (+0.02253872575894178) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.767872367106694 (+0.5531076511866644) | > avg_grad_norm_1: tensor(6315.2744, device='cuda:0') (tensor(-1634.3989, device='cuda:0'))  > EPOCH: 994/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:27:40)   --> TIME: 2025-02-25 18:27:56 -- STEP: 31/72 -- GLOBAL_STEP: 1071600 | > loss_disc: 2.63077449798584 (2.6369905702529417) | > loss_disc_real_0: 0.07839512825012207 (0.1361647409296805) | > loss_disc_real_1: 0.22755616903305054 (0.20896552983791597) | > loss_disc_real_2: 0.2669689655303955 (0.22898053305764351) | > loss_disc_real_3: 0.25718677043914795 (0.22443213433988632) | > loss_disc_real_4: 0.252049058675766 (0.2447270832715496) | > loss_disc_real_5: 0.2525610625743866 (0.24494350341058546) | > loss_0: 2.63077449798584 (2.6369905702529417) | > grad_norm_0: tensor(295.1409, device='cuda:0') (tensor(201.6707, device='cuda:0')) | > loss_gen: 2.0848565101623535 (1.9289867954869424) | > loss_kl: 2.1653361320495605 (2.0695191621780396) | > loss_feat: 3.574683904647827 (3.437898066736037) | > loss_mel: 17.40074920654297 (17.507518029982045) | > loss_duration: 1.5432298183441162 (1.5227493547624158) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.768856048583984 (26.466671666791363) | > grad_norm_1: tensor(10281.3135, device='cuda:0') (tensor(7791.6909, device='cuda:0')) | > current_lr_0: 0.0001766304329398157 | > current_lr_1: 0.0001766304329398157 | > step_time: 0.481 (0.4375163355181294) | > loader_time: 0.0084 (0.0070050916364116055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008646639300064303 (+3.123954987861721e-05) | > avg_step_time: 0.4522558937610035 (-0.007860821737370016) | > avg_loss_disc: 2.6056562813235 (-0.16881407818324146) | > avg_loss_disc_real_0: 0.12019578010683328 (-0.08135524790891459) | > avg_loss_disc_real_1: 0.21657758111685094 (-0.0532676442198351) | > avg_loss_disc_real_2: 0.23621710547259156 (-0.005396516600125273) | > avg_loss_disc_real_3: 0.2341026313288111 (-0.0001295242511051009) | > avg_loss_disc_real_4: 0.24494788596327877 (-0.006283334863017964) | > avg_loss_disc_real_5: 0.244036830646891 (-0.01342327032290716) | > avg_loss_0: 2.6056562813235 (-0.16881407818324146) | > avg_grad_norm_0: tensor(226.7044, device='cuda:0') (tensor(-48.0240, device='cuda:0')) | > avg_loss_gen: 2.0428017847974536 (-0.1114432324825878) | > avg_loss_kl: 2.2189823321893187 (-0.0016758626615485106) | > avg_loss_feat: 3.589146436100275 (+0.08254673783208322) | > avg_loss_mel: 17.635074857255116 (+0.315149240090836) | > avg_loss_duration: 1.5701029099209207 (+0.003658986427414579) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.056108367275183 (+0.2882360001684887) | > avg_grad_norm_1: tensor(7061.2378, device='cuda:0') (+tensor(745.9634, device='cuda:0'))  > EPOCH: 995/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:28:17)   --> TIME: 2025-02-25 18:28:23 -- STEP: 9/72 -- GLOBAL_STEP: 1071650 | > loss_disc: 2.499708652496338 (2.4748085339864097) | > loss_disc_real_0: 0.10096412897109985 (0.06823090463876724) | > loss_disc_real_1: 0.23631593585014343 (0.2138543360763126) | > loss_disc_real_2: 0.26228874921798706 (0.2317385027805964) | > loss_disc_real_3: 0.23944397270679474 (0.22862893839677176) | > loss_disc_real_4: 0.2514946758747101 (0.2338937504423989) | > loss_disc_real_5: 0.24302786588668823 (0.2413582056760788) | > loss_0: 2.499708652496338 (2.4748085339864097) | > grad_norm_0: tensor(221.5719, device='cuda:0') (tensor(173.4077, device='cuda:0')) | > loss_gen: 2.3893256187438965 (2.292016214794583) | > loss_kl: 1.9382909536361694 (1.9860827525456746) | > loss_feat: 4.355384826660156 (4.2049775918324785) | > loss_mel: 17.5771427154541 (17.24790467156304) | > loss_duration: 1.509610891342163 (1.5218995147281222) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.769756317138672 (27.252880732218426) | > grad_norm_1: tensor(4291.4736, device='cuda:0') (tensor(5212.4771, device='cuda:0')) | > current_lr_0: 0.0001766083541356982 | > current_lr_1: 0.0001766083541356982 | > step_time: 0.4213 (0.4295167658064101) | > loader_time: 0.0056 (0.006286038292778863)  --> TIME: 2025-02-25 18:28:46 -- STEP: 59/72 -- GLOBAL_STEP: 1071700 | > loss_disc: 2.4809224605560303 (2.5058897753893317) | > loss_disc_real_0: 0.06689159572124481 (0.07446188182901528) | > loss_disc_real_1: 0.21713495254516602 (0.21679710887246212) | > loss_disc_real_2: 0.23792815208435059 (0.23892275155600856) | > loss_disc_real_3: 0.24616144597530365 (0.2335551381111145) | > loss_disc_real_4: 0.23132985830307007 (0.24017695375418258) | > loss_disc_real_5: 0.2523817718029022 (0.24555135202609887) | > loss_0: 2.4809224605560303 (2.5058897753893317) | > grad_norm_0: tensor(97.9297, device='cuda:0') (tensor(190.9309, device='cuda:0')) | > loss_gen: 2.2553646564483643 (2.264365899360786) | > loss_kl: 2.063383102416992 (2.0190874762454296) | > loss_feat: 3.8347713947296143 (4.096589795613692) | > loss_mel: 16.172950744628906 (16.910687301118493) | > loss_duration: 1.5912048816680908 (1.5420815156678023) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.91767692565918 (26.832812131461452) | > grad_norm_1: tensor(4383.5718, device='cuda:0') (tensor(6215.8472, device='cuda:0')) | > current_lr_0: 0.0001766083541356982 | > current_lr_1: 0.0001766083541356982 | > step_time: 0.4846 (0.44376263376009667) | > loader_time: 0.0097 (0.008056543641171214) --> EVAL PERFORMANCE | > avg_loader_time: 0.008618072724678148 (-2.8566575386154355e-05) | > avg_step_time: 0.4597691448641495 (+0.00751325110314599) | > avg_loss_disc: 2.5069948082238858 (-0.09866147309961404) | > avg_loss_disc_real_0: 0.07578655164426482 (-0.044409228462568465) | > avg_loss_disc_real_1: 0.2178161083392694 (+0.0012385272224184618) | > avg_loss_disc_real_2: 0.23779508135688138 (+0.0015779758842898173) | > avg_loss_disc_real_3: 0.2339600725492961 (-0.00014255877951499518) | > avg_loss_disc_real_4: 0.2427575554646237 (-0.0021903304986550798) | > avg_loss_disc_real_5: 0.24563505154260448 (+0.001598220895713487) | > avg_loss_0: 2.5069948082238858 (-0.09866147309961404) | > avg_grad_norm_0: tensor(190.9658, device='cuda:0') (tensor(-35.7385, device='cuda:0')) | > avg_loss_gen: 2.2640452955810115 (+0.22124351078355797) | > avg_loss_kl: 2.034509979503257 (-0.18447235268606166) | > avg_loss_feat: 4.059255737653919 (+0.4701093015536437) | > avg_loss_mel: 16.891252759476785 (-0.743822097778331) | > avg_loss_duration: 1.5521985426755018 (-0.017904367245418884) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.80126251972897 (-0.2548458475462141) | > avg_grad_norm_1: tensor(6195.2993, device='cuda:0') (tensor(-865.9385, device='cuda:0'))  > EPOCH: 996/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:28:54)   --> TIME: 2025-02-25 18:29:13 -- STEP: 37/72 -- GLOBAL_STEP: 1071750 | > loss_disc: 2.330359935760498 (2.4933684645472347) | > loss_disc_real_0: 0.06506115943193436 (0.08882286575799053) | > loss_disc_real_1: 0.18471217155456543 (0.2165001691193194) | > loss_disc_real_2: 0.24635222554206848 (0.2317306516138283) | > loss_disc_real_3: 0.2542203664779663 (0.23112004149604487) | > loss_disc_real_4: 0.247543603181839 (0.2387638760579599) | > loss_disc_real_5: 0.2439509630203247 (0.24515819831474409) | > loss_0: 2.330359935760498 (2.4933684645472347) | > grad_norm_0: tensor(124.7973, device='cuda:0') (tensor(199.6137, device='cuda:0')) | > loss_gen: 2.597052574157715 (2.293661629831469) | > loss_kl: 1.936666488647461 (1.9541376732491158) | > loss_feat: 4.700164318084717 (4.086707488910572) | > loss_mel: 16.984716415405273 (16.698176512847077) | > loss_duration: 1.5414905548095703 (1.512956580600223) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.76009178161621 (26.545639811335384) | > grad_norm_1: tensor(5958.5449, device='cuda:0') (tensor(5789.9507, device='cuda:0')) | > current_lr_0: 0.00017658627809143124 | > current_lr_1: 0.00017658627809143124 | > step_time: 0.4397 (0.4388658935959275) | > loader_time: 0.0082 (0.007100975191270983) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854076130289427 (-7.73114217838787e-05) | > avg_step_time: 0.45609574922373597 (-0.0036733956404135237) | > avg_loss_disc: 2.4998323010726713 (-0.00716250715121447) | > avg_loss_disc_real_0: 0.08274558262610937 (+0.006959030981844555) | > avg_loss_disc_real_1: 0.21842615348352512 (+0.000610045144255722) | > avg_loss_disc_real_2: 0.22797591459583227 (-0.009819166761049103) | > avg_loss_disc_real_3: 0.23187300206070216 (-0.0020870704885939395) | > avg_loss_disc_real_4: 0.23992479099354275 (-0.0028327644710809396) | > avg_loss_disc_real_5: 0.24404291640704787 (-0.001592135135556616) | > avg_loss_0: 2.4998323010726713 (-0.00716250715121447) | > avg_grad_norm_0: tensor(192.1514, device='cuda:0') (+tensor(1.1856, device='cuda:0')) | > avg_loss_gen: 2.261673831603897 (-0.0023714639771146295) | > avg_loss_kl: 2.000534433714101 (-0.03397554578915596) | > avg_loss_feat: 4.03714542321756 (-0.022110314436358358) | > avg_loss_mel: 16.689464031810488 (-0.20178872766629752) | > avg_loss_duration: 1.5485687440549825 (-0.0036297986205193755) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.537386477833063 (-0.2638760418959052) | > avg_grad_norm_1: tensor(5811.3398, device='cuda:0') (tensor(-383.9595, device='cuda:0'))  > EPOCH: 997/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:29:31)   --> TIME: 2025-02-25 18:29:40 -- STEP: 15/72 -- GLOBAL_STEP: 1071800 | > loss_disc: 2.500403881072998 (2.5256918748219808) | > loss_disc_real_0: 0.14178413152694702 (0.08038102115194003) | > loss_disc_real_1: 0.2078350931406021 (0.224698676665624) | > loss_disc_real_2: 0.1825498789548874 (0.24130303363005318) | > loss_disc_real_3: 0.22885103523731232 (0.23805438180764515) | > loss_disc_real_4: 0.23320436477661133 (0.24583540856838226) | > loss_disc_real_5: 0.25453123450279236 (0.24201760987440746) | > loss_0: 2.500403881072998 (2.5256918748219808) | > grad_norm_0: tensor(289.1735, device='cuda:0') (tensor(221.2331, device='cuda:0')) | > loss_gen: 2.247427463531494 (2.294782861073812) | > loss_kl: 1.9397392272949219 (1.9318591276804606) | > loss_feat: 3.62362003326416 (4.1291926383972175) | > loss_mel: 17.00370216369629 (16.657407315572105) | > loss_duration: 1.4814643859863281 (1.4896735827128091) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.29595375061035 (26.50291544596354) | > grad_norm_1: tensor(8809.5186, device='cuda:0') (tensor(6259.1147, device='cuda:0')) | > current_lr_0: 0.0001765642048066698 | > current_lr_1: 0.0001765642048066698 | > step_time: 0.4244 (0.43148026466369627) | > loader_time: 0.0058 (0.0065054734547932945)  --> TIME: 2025-02-25 18:30:03 -- STEP: 65/72 -- GLOBAL_STEP: 1071850 | > loss_disc: 2.8184661865234375 (2.5664155409886287) | > loss_disc_real_0: 0.11459693312644958 (0.11201374490673725) | > loss_disc_real_1: 0.39519554376602173 (0.22012296743117846) | > loss_disc_real_2: 0.23757901787757874 (0.23098508669779852) | > loss_disc_real_3: 0.30838167667388916 (0.23505513461736532) | > loss_disc_real_4: 0.2602711319923401 (0.24020611024819888) | > loss_disc_real_5: 0.25358861684799194 (0.24334177030966833) | > loss_0: 2.8184661865234375 (2.5664155409886287) | > grad_norm_0: tensor(65.3844, device='cuda:0') (tensor(204.5271, device='cuda:0')) | > loss_gen: 2.010984182357788 (2.272624421119689) | > loss_kl: 2.185718297958374 (2.0198601209200353) | > loss_feat: 2.740837574005127 (4.0233828691335844) | > loss_mel: 16.35227394104004 (16.67430509420542) | > loss_duration: 1.6216464042663574 (1.5393573999404908) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.911460876464844 (26.52952995300293) | > grad_norm_1: tensor(6279.7754, device='cuda:0') (tensor(5941.2378, device='cuda:0')) | > current_lr_0: 0.0001765642048066698 | > current_lr_1: 0.0001765642048066698 | > step_time: 0.5273 (0.44985581177931566) | > loader_time: 0.0117 (0.008319040445181041) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859710196374168 (+5.634066084740999e-05) | > avg_step_time: 0.4562276752901749 (+0.00013192606643891436) | > avg_loss_disc: 2.5877657440346726 (+0.08793344296200134) | > avg_loss_disc_real_0: 0.11030179744874927 (+0.027556214822639896) | > avg_loss_disc_real_1: 0.2255789800219133 (+0.007152826538388168) | > avg_loss_disc_real_2: 0.2409792639000315 (+0.013003349304199219) | > avg_loss_disc_real_3: 0.24004826978058882 (+0.008175267719886664) | > avg_loss_disc_real_4: 0.24336615015923138 (+0.0034413591656886344) | > avg_loss_disc_real_5: 0.24291144059577457 (-0.0011314758112732914) | > avg_loss_0: 2.5877657440346726 (+0.08793344296200134) | > avg_grad_norm_0: tensor(216.8333, device='cuda:0') (+tensor(24.6819, device='cuda:0')) | > avg_loss_gen: 2.248230942538086 (-0.013442889065810704) | > avg_loss_kl: 2.0299103092139883 (+0.029375875499887183) | > avg_loss_feat: 3.9364806766241385 (-0.10066474659342184) | > avg_loss_mel: 16.662713091138386 (-0.02675094067210182) | > avg_loss_duration: 1.5455147111919565 (-0.0030540328630259683) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.422849708879497 (-0.11453676895356679) | > avg_grad_norm_1: tensor(6112.1650, device='cuda:0') (+tensor(300.8252, device='cuda:0'))  > EPOCH: 998/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:30:08)   --> TIME: 2025-02-25 18:30:29 -- STEP: 43/72 -- GLOBAL_STEP: 1071900 | > loss_disc: 2.564387321472168 (2.6987333242283316) | > loss_disc_real_0: 0.17348000407218933 (0.12853305847492333) | > loss_disc_real_1: 0.2538052499294281 (0.21548783190028611) | > loss_disc_real_2: 0.2128838747739792 (0.2123499796834103) | > loss_disc_real_3: 0.22863589227199554 (0.25973337438217425) | > loss_disc_real_4: 0.24476182460784912 (0.25226635052714236) | > loss_disc_real_5: 0.22560572624206543 (0.24487963630709536) | > loss_0: 2.564387321472168 (2.6987333242283316) | > grad_norm_0: tensor(223.7851, device='cuda:0') (tensor(245.5834, device='cuda:0')) | > loss_gen: 2.422234535217285 (2.1326598705247397) | > loss_kl: 2.4020917415618896 (2.5604890612668774) | > loss_feat: 4.0010175704956055 (3.7877119530078978) | > loss_mel: 17.824275970458984 (18.41097115361413) | > loss_duration: 1.6803054809570312 (1.678883663443632) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.329925537109375 (28.5707156159157) | > grad_norm_1: tensor(1554.7645, device='cuda:0') (tensor(5920.5552, device='cuda:0')) | > current_lr_0: 0.00017654213428106895 | > current_lr_1: 0.00017654213428106895 | > step_time: 0.4536 (0.4397010692330294) | > loader_time: 0.0097 (0.007416386936986169) --> EVAL PERFORMANCE | > avg_loader_time: 0.008571319177117146 (-2.5782786624533108e-05) | > avg_step_time: 0.46656769765934475 (+0.010340022369169866) | > avg_loss_disc: 2.6880119854295756 (+0.10024624139490301) | > avg_loss_disc_real_0: 0.13217044306892745 (+0.02186864562017818) | > avg_loss_disc_real_1: 0.2295082760318904 (+0.003929296009977101) | > avg_loss_disc_real_2: 0.22175226287103036 (-0.01922700102900113) | > avg_loss_disc_real_3: 0.2540836384598638 (+0.01403536867927499) | > avg_loss_disc_real_4: 0.25154806295750837 (+0.008181912798276986) | > avg_loss_disc_real_5: 0.24353911801123282 (+0.0006276774154582443) | > avg_loss_0: 2.6880119854295756 (+0.10024624139490301) | > avg_grad_norm_0: tensor(248.6686, device='cuda:0') (+tensor(31.8353, device='cuda:0')) | > avg_loss_gen: 2.1288463867885965 (-0.1193845557494897) | > avg_loss_kl: 2.5322054238386555 (+0.5022951146246672) | > avg_loss_feat: 3.6903467413405298 (-0.24613393528360872) | > avg_loss_mel: 18.34151400982494 (+1.6788009186865551) | > avg_loss_duration: 1.6712066724266808 (+0.12569196123472426) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.36411916705924 (+1.9412694581797432) | > avg_grad_norm_1: tensor(6231.9429, device='cuda:0') (+tensor(119.7778, device='cuda:0'))  > EPOCH: 999/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:30:46)   --> TIME: 2025-02-25 18:30:57 -- STEP: 21/72 -- GLOBAL_STEP: 1071950 | > loss_disc: 2.593594789505005 (2.616886831465222) | > loss_disc_real_0: 0.1891607642173767 (0.12142773185457502) | > loss_disc_real_1: 0.18766583502292633 (0.21281396775018602) | > loss_disc_real_2: 0.22090217471122742 (0.23366018987837292) | > loss_disc_real_3: 0.24057020246982574 (0.23637196492581142) | > loss_disc_real_4: 0.24085812270641327 (0.2532344091506231) | > loss_disc_real_5: 0.23010049760341644 (0.2480493663322358) | > loss_0: 2.593594789505005 (2.616886831465222) | > grad_norm_0: tensor(124.4465, device='cuda:0') (tensor(188.9107, device='cuda:0')) | > loss_gen: 1.9700875282287598 (2.0937850134713307) | > loss_kl: 1.8421783447265625 (2.23638079279945) | > loss_feat: 3.2818615436553955 (3.7154388314201716) | > loss_mel: 16.451488494873047 (18.127200898670015) | > loss_duration: 1.559476375579834 (1.5477973847162156) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.105091094970703 (27.720602762131463) | > grad_norm_1: tensor(4156.4404, device='cuda:0') (tensor(7387.4854, device='cuda:0')) | > current_lr_0: 0.0001765200665142838 | > current_lr_1: 0.0001765200665142838 | > step_time: 0.4317 (0.43962929362342473) | > loader_time: 0.0063 (0.0068169888995942615)  --> TIME: 2025-02-25 18:31:22 -- STEP: 71/72 -- GLOBAL_STEP: 1072000 | > loss_disc: 2.4501471519470215 (2.5938703980244378) | > loss_disc_real_0: 0.06401803344488144 (0.1293125871411512) | > loss_disc_real_1: 0.21028569340705872 (0.219843339122517) | > loss_disc_real_2: 0.2349424511194229 (0.23842548043795034) | > loss_disc_real_3: 0.22543956339359283 (0.2333947776908606) | > loss_disc_real_4: 0.2388439178466797 (0.2454592572971129) | > loss_disc_real_5: 0.2646653950214386 (0.2453625030081037) | > loss_0: 2.4501471519470215 (2.5938703980244378) | > grad_norm_0: tensor(144.8594, device='cuda:0') (tensor(210.4274, device='cuda:0')) | > loss_gen: 2.344301700592041 (2.127665382036021) | > loss_kl: 2.0909204483032227 (2.2311984384563615) | > loss_feat: 4.289976119995117 (3.7094173464976565) | > loss_mel: 17.562185287475586 (17.880242656654037) | > loss_duration: 1.5630792379379272 (1.5767576895969015) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.850461959838867 (27.525281422574757) | > grad_norm_1: tensor(9115.9043, device='cuda:0') (tensor(7468.8862, device='cuda:0')) | > current_lr_0: 0.0001765200665142838 | > current_lr_1: 0.0001765200665142838 | > step_time: 0.5043 (0.461735466836204) | > loader_time: 0.0081 (0.008957741965710277) --> EVAL PERFORMANCE | > avg_loader_time: 0.008957741965710277 (+0.0003864227885931309) | > avg_step_time: 0.461735466836204 (-0.004832230823140771) | > avg_loss_disc: 2.5938703980244378 (-0.0941415874051379) | > avg_loss_disc_real_0: 0.1293125871411512 (-0.002857855927776237) | > avg_loss_disc_real_1: 0.219843339122517 (-0.009664936909373395) | > avg_loss_disc_real_2: 0.23842548043795034 (+0.016673217566919973) | > avg_loss_disc_real_3: 0.2333947776908606 (-0.020688860769003226) | > avg_loss_disc_real_4: 0.2454592572971129 (-0.006088805660395463) | > avg_loss_disc_real_5: 0.2453625030081037 (+0.0018233849968708749) | > avg_loss_0: 2.5938703980244378 (-0.0941415874051379) | > avg_grad_norm_0: tensor(210.4274, device='cuda:0') (tensor(-38.2412, device='cuda:0')) | > avg_loss_gen: 2.127665382036021 (-0.001181004752575543) | > avg_loss_kl: 2.2311984384563615 (-0.3010069853822941) | > avg_loss_feat: 3.7094173464976565 (+0.019070605157126774) | > avg_loss_mel: 17.880242656654037 (-0.46127135317090406) | > avg_loss_duration: 1.5767576895969015 (-0.0944489828297792) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.525281422574757 (-0.8388377444844828) | > avg_grad_norm_1: tensor(7468.8862, device='cuda:0') (+tensor(1236.9434, device='cuda:0'))  > EPOCH: 1000/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:31:23)   --> TIME: 2025-02-25 18:31:48 -- STEP: 49/72 -- GLOBAL_STEP: 1072050 | > loss_disc: 2.4503026008605957 (2.52496118934787) | > loss_disc_real_0: 0.15622174739837646 (0.10141049363479322) | > loss_disc_real_1: 0.16150040924549103 (0.21433964219628548) | > loss_disc_real_2: 0.23210874199867249 (0.22806479979534539) | > loss_disc_real_3: 0.23557832837104797 (0.23245044782453653) | > loss_disc_real_4: 0.25380316376686096 (0.24252355068313833) | > loss_disc_real_5: 0.2512514293193817 (0.24364257284573146) | > loss_0: 2.4503026008605957 (2.52496118934787) | > grad_norm_0: tensor(120.6836, device='cuda:0') (tensor(232.8345, device='cuda:0')) | > loss_gen: 2.3518872261047363 (2.196576952934265) | > loss_kl: 1.9516619443893433 (1.967510619942023) | > loss_feat: 4.017633438110352 (3.919026258040448) | > loss_mel: 17.299575805664062 (16.900073168229085) | > loss_duration: 1.6059412956237793 (1.5348626983409026) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.226699829101562 (26.518049901845504) | > grad_norm_1: tensor(1064.4752, device='cuda:0') (tensor(7696.1553, device='cuda:0')) | > current_lr_0: 0.0001764980015059695 | > current_lr_1: 0.0001764980015059695 | > step_time: 0.458 (0.44649779066747547) | > loader_time: 0.0098 (0.007596356528145927) --> EVAL PERFORMANCE | > avg_loader_time: 0.00856849173424949 (-0.000389250231460787) | > avg_step_time: 0.4594405167539355 (-0.0022949500822684876) | > avg_loss_disc: 2.5610526514724943 (-0.03281774655194347) | > avg_loss_disc_real_0: 0.11578996130593226 (-0.013522625835218946) | > avg_loss_disc_real_1: 0.22056350468749727 (+0.0007201655649802796) | > avg_loss_disc_real_2: 0.22988246108444643 (-0.008543019353503906) | > avg_loss_disc_real_3: 0.23301886861592952 (-0.00037590907493106895) | > avg_loss_disc_real_4: 0.24216599510589115 (-0.003293262191221752) | > avg_loss_disc_real_5: 0.24371290983448565 (-0.0016495931736180414) | > avg_loss_0: 2.5610526514724943 (-0.03281774655194347) | > avg_grad_norm_0: tensor(228.9843, device='cuda:0') (+tensor(18.5569, device='cuda:0')) | > avg_loss_gen: 2.157349489104579 (+0.029684107068558063) | > avg_loss_kl: 2.0166336633789705 (-0.2145647750773909) | > avg_loss_feat: 3.7932623104310372 (+0.0838449639333807) | > avg_loss_mel: 16.934118082825567 (-0.9461245738284703) | > avg_loss_duration: 1.554091344416981 (-0.0226663451799205) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.455455054699534 (-1.0698263678752227) | > avg_grad_norm_1: tensor(7321.9116, device='cuda:0') (tensor(-146.9746, device='cuda:0'))  > EPOCH: 1001/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:32:01)   --> TIME: 2025-02-25 18:32:15 -- STEP: 27/72 -- GLOBAL_STEP: 1072100 | > loss_disc: 2.4889659881591797 (2.7069106190292924) | > loss_disc_real_0: 0.029564939439296722 (0.20469412952661514) | > loss_disc_real_1: 0.42820391058921814 (0.2600024486029589) | > loss_disc_real_2: 0.2477341592311859 (0.22988260178654282) | > loss_disc_real_3: 0.2074710726737976 (0.24740823441081577) | > loss_disc_real_4: 0.24939140677452087 (0.2483834464240957) | > loss_disc_real_5: 0.2094435840845108 (0.24241786477742372) | > loss_0: 2.4889659881591797 (2.7069106190292924) | > grad_norm_0: tensor(182.5002, device='cuda:0') (tensor(261.8873, device='cuda:0')) | > loss_gen: 2.2139952182769775 (2.214895380867852) | > loss_kl: 2.311631202697754 (2.292521021984241) | > loss_feat: 4.359020233154297 (3.746308847709938) | > loss_mel: 17.606515884399414 (18.145515441894528) | > loss_duration: 1.5353195667266846 (1.5662977033191257) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.0264835357666 (27.96553830747251) | > grad_norm_1: tensor(6456.6641, device='cuda:0') (tensor(6578.5801, device='cuda:0')) | > current_lr_0: 0.00017647593925578124 | > current_lr_1: 0.00017647593925578124 | > step_time: 0.4457 (0.45239762906674985) | > loader_time: 0.0085 (0.0068663756052653) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693742080473564 (+0.00012525034622407408) | > avg_step_time: 0.46264460389043244 (+0.003204087136496947) | > avg_loss_disc: 2.760816469998427 (+0.19976381852593272) | > avg_loss_disc_real_0: 0.19055390258280325 (+0.07476394127687098) | > avg_loss_disc_real_1: 0.2645402309340491 (+0.04397672624655183) | > avg_loss_disc_real_2: 0.2394561077087698 (+0.009573646624323362) | > avg_loss_disc_real_3: 0.24011096127436193 (+0.007092092658432414) | > avg_loss_disc_real_4: 0.25280807156797874 (+0.01064207646208759) | > avg_loss_disc_real_5: 0.24737589271135732 (+0.0036629828768716677) | > avg_loss_0: 2.760816469998427 (+0.19976381852593272) | > avg_grad_norm_0: tensor(244.4899, device='cuda:0') (+tensor(15.5057, device='cuda:0')) | > avg_loss_gen: 2.085099908667551 (-0.072249580437028) | > avg_loss_kl: 2.2544242832022654 (+0.23779061982329486) | > avg_loss_feat: 3.5158356679996974 (-0.27742664243133985) | > avg_loss_mel: 17.808897340801394 (+0.8747792579758276) | > avg_loss_duration: 1.5831820310001643 (+0.029090686583183256) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.247439048659636 (+0.7919839939601019) | > avg_grad_norm_1: tensor(6077.7803, device='cuda:0') (tensor(-1244.1313, device='cuda:0'))  > EPOCH: 1002/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:32:38)   --> TIME: 2025-02-25 18:32:42 -- STEP: 5/72 -- GLOBAL_STEP: 1072150 | > loss_disc: 2.535581588745117 (2.5628835201263427) | > loss_disc_real_0: 0.11161290109157562 (0.17889896035194397) | > loss_disc_real_1: 0.23909364640712738 (0.24039555490016937) | > loss_disc_real_2: 0.22697891294956207 (0.22535553872585296) | > loss_disc_real_3: 0.23929132521152496 (0.2301749587059021) | > loss_disc_real_4: 0.2438179850578308 (0.23214317262172698) | > loss_disc_real_5: 0.24049001932144165 (0.23484552204608916) | > loss_0: 2.535581588745117 (2.5628835201263427) | > grad_norm_0: tensor(110.4688, device='cuda:0') (tensor(301.3664, device='cuda:0')) | > loss_gen: 1.9721903800964355 (2.063767647743225) | > loss_kl: 2.315359592437744 (2.347457838058472) | > loss_feat: 3.31963849067688 (3.447623538970947) | > loss_mel: 17.366748809814453 (18.019356536865235) | > loss_duration: 1.521214246749878 (1.5370382070541382) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.49515151977539 (27.415243911743165) | > grad_norm_1: tensor(6434.7344, device='cuda:0') (tensor(7572.5713, device='cuda:0')) | > current_lr_0: 0.00017645387976337426 | > current_lr_1: 0.00017645387976337426 | > step_time: 0.4503 (0.44586963653564454) | > loader_time: 0.0077 (0.007159042358398438)  --> TIME: 2025-02-25 18:33:05 -- STEP: 55/72 -- GLOBAL_STEP: 1072200 | > loss_disc: 2.5546042919158936 (2.549587102369828) | > loss_disc_real_0: 0.05580851435661316 (0.0989342047070915) | > loss_disc_real_1: 0.18638092279434204 (0.21843695667656984) | > loss_disc_real_2: 0.3155338168144226 (0.2460613120685924) | > loss_disc_real_3: 0.22493363916873932 (0.22762044912034815) | > loss_disc_real_4: 0.2581861913204193 (0.2361213356256485) | > loss_disc_real_5: 0.24601827561855316 (0.23897405808622188) | > loss_0: 2.5546042919158936 (2.549587102369828) | > grad_norm_0: tensor(204.2013, device='cuda:0') (tensor(194.4761, device='cuda:0')) | > loss_gen: 2.219691514968872 (2.171118684248491) | > loss_kl: 2.253204584121704 (2.102411209453236) | > loss_feat: 4.169233798980713 (3.969511305202137) | > loss_mel: 16.220762252807617 (17.143535249883477) | > loss_duration: 1.6045074462890625 (1.5463669928637418) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.46739959716797 (26.93294344815341) | > grad_norm_1: tensor(3966.1436, device='cuda:0') (tensor(5938.6948, device='cuda:0')) | > current_lr_0: 0.00017645387976337426 | > current_lr_1: 0.00017645387976337426 | > step_time: 0.459 (0.4475818287242543) | > loader_time: 0.0093 (0.007993654771284622) --> EVAL PERFORMANCE | > avg_loader_time: 0.008721546388008225 (+2.780430753466026e-05) | > avg_step_time: 0.45847751053286273 (-0.004167093357569707) | > avg_loss_disc: 2.5555316193003037 (-0.2052848506981233) | > avg_loss_disc_real_0: 0.09613202638189558 (-0.09442187620090767) | > avg_loss_disc_real_1: 0.21794491943339228 (-0.04659531150065682) | > avg_loss_disc_real_2: 0.2425439242326038 (+0.003087816523833997) | > avg_loss_disc_real_3: 0.2259139445885806 (-0.014197016685781327) | > avg_loss_disc_real_4: 0.23637110498589528 (-0.016436966582083462) | > avg_loss_disc_real_5: 0.2387534885759085 (-0.008622404135448819) | > avg_loss_0: 2.5555316193003037 (-0.2052848506981233) | > avg_grad_norm_0: tensor(204.1866, device='cuda:0') (tensor(-40.3033, device='cuda:0')) | > avg_loss_gen: 2.162526911413166 (+0.07742700274561498) | > avg_loss_kl: 2.103529594313931 (-0.15089468888833446) | > avg_loss_feat: 3.948471898763952 (+0.43263623076425484) | > avg_loss_mel: 17.054191750539864 (-0.7547055902615298) | > avg_loss_duration: 1.5586430354857108 (-0.024538995514453488) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.827363188837616 (-0.42007585982202045) | > avg_grad_norm_1: tensor(6206.1445, device='cuda:0') (+tensor(128.3643, device='cuda:0'))  > EPOCH: 1003/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:33:15)   --> TIME: 2025-02-25 18:33:32 -- STEP: 33/72 -- GLOBAL_STEP: 1072250 | > loss_disc: 2.3642196655273438 (2.6187380732912007) | > loss_disc_real_0: 0.05488099157810211 (0.09144379915387342) | > loss_disc_real_1: 0.22347526252269745 (0.2219199470498345) | > loss_disc_real_2: 0.21693094074726105 (0.1980613630377885) | > loss_disc_real_3: 0.16417396068572998 (0.22323337197303772) | > loss_disc_real_4: 0.19616514444351196 (0.23054386675357819) | > loss_disc_real_5: 0.22882625460624695 (0.23430779847231778) | > loss_0: 2.3642196655273438 (2.6187380732912007) | > grad_norm_0: tensor(195.9483, device='cuda:0') (tensor(243.0688, device='cuda:0')) | > loss_gen: 2.0641491413116455 (2.0906196399168535) | > loss_kl: 2.100358009338379 (1.9814553694291548) | > loss_feat: 4.636044502258301 (4.0984453432487715) | > loss_mel: 16.851787567138672 (16.78338183778705) | > loss_duration: 1.5177760124206543 (1.5088032484054565) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.170116424560547 (26.462705438787285) | > grad_norm_1: tensor(15197.3467, device='cuda:0') (tensor(7203.1904, device='cuda:0')) | > current_lr_0: 0.00017643182302840384 | > current_lr_1: 0.00017643182302840384 | > step_time: 0.4606 (0.4388337352059104) | > loader_time: 0.0076 (0.007115197904182203) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865341911853199 (-6.812726947623415e-05) | > avg_step_time: 0.4580377625747466 (-0.0004397479581161323) | > avg_loss_disc: 2.5860090591538127 (+0.030477439853509036) | > avg_loss_disc_real_0: 0.10642502076504097 (+0.01029299438314539) | > avg_loss_disc_real_1: 0.22047909737472807 (+0.0025341779413357857) | > avg_loss_disc_real_2: 0.2145470451091377 (-0.0279968791234661) | > avg_loss_disc_real_3: 0.2271102253400104 (+0.0011962807514298013) | > avg_loss_disc_real_4: 0.2360886916728087 (-0.00028241331308659356) | > avg_loss_disc_real_5: 0.2409201102777266 (+0.002166621701818111) | > avg_loss_0: 2.5860090591538127 (+0.030477439853509036) | > avg_grad_norm_0: tensor(210.5519, device='cuda:0') (+tensor(6.3653, device='cuda:0')) | > avg_loss_gen: 2.14501407616575 (-0.017512835247416092) | > avg_loss_kl: 2.023332916514974 (-0.0801966777989569) | > avg_loss_feat: 3.9566337927966035 (+0.008161894032651329) | > avg_loss_mel: 16.766121944911042 (-0.28806980562882245) | > avg_loss_duration: 1.5477666703748032 (-0.010876365110907615) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.43886942259023 (-0.38849376624738596) | > avg_grad_norm_1: tensor(7234.1680, device='cuda:0') (+tensor(1028.0234, device='cuda:0'))  > EPOCH: 1004/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:33:52)   --> TIME: 2025-02-25 18:33:59 -- STEP: 11/72 -- GLOBAL_STEP: 1072300 | > loss_disc: 2.7866029739379883 (2.6551149758425625) | > loss_disc_real_0: 0.1300908327102661 (0.1396656327626922) | > loss_disc_real_1: 0.26250457763671875 (0.23717906393788077) | > loss_disc_real_2: 0.24980439245700836 (0.24121761999346994) | > loss_disc_real_3: 0.2436540573835373 (0.2372652143239975) | > loss_disc_real_4: 0.24938155710697174 (0.24884838272224774) | > loss_disc_real_5: 0.259842187166214 (0.24980555203827945) | > loss_0: 2.7866029739379883 (2.6551149758425625) | > grad_norm_0: tensor(244.6339, device='cuda:0') (tensor(193.3268, device='cuda:0')) | > loss_gen: 1.9961947202682495 (2.060122858394276) | > loss_kl: 1.917951226234436 (1.9400945685126565) | > loss_feat: 3.4591686725616455 (3.482908010482788) | > loss_mel: 17.292686462402344 (16.591412284157492) | > loss_duration: 1.50555419921875 (1.509985089302063) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.17155647277832 (25.584522420709785) | > grad_norm_1: tensor(9234.1992, device='cuda:0') (tensor(6513.2773, device='cuda:0')) | > current_lr_0: 0.00017640976905052529 | > current_lr_1: 0.00017640976905052529 | > step_time: 0.4507 (0.4366283200003884) | > loader_time: 0.0059 (0.006644639101895419)  --> TIME: 2025-02-25 18:34:23 -- STEP: 61/72 -- GLOBAL_STEP: 1072350 | > loss_disc: 2.6585988998413086 (2.6216695543195376) | > loss_disc_real_0: 0.13611799478530884 (0.14016948847985664) | > loss_disc_real_1: 0.22883574664592743 (0.22297057778131765) | > loss_disc_real_2: 0.24893289804458618 (0.23360293468490975) | > loss_disc_real_3: 0.2430112510919571 (0.23470806660222227) | > loss_disc_real_4: 0.2429579347372055 (0.24235937316886713) | > loss_disc_real_5: 0.2507126033306122 (0.24701031676081361) | > loss_0: 2.6585988998413086 (2.6216695543195376) | > grad_norm_0: tensor(405.2468, device='cuda:0') (tensor(229.5997, device='cuda:0')) | > loss_gen: 2.0941433906555176 (2.0532025782788392) | > loss_kl: 2.013263702392578 (2.023096989412777) | > loss_feat: 3.521059513092041 (3.534422843182673) | > loss_mel: 16.47140884399414 (16.57919184887994) | > loss_duration: 1.5915939807891846 (1.5405376836901805) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.691469192504883 (25.73045189654241) | > grad_norm_1: tensor(13528.3291, device='cuda:0') (tensor(10029.3193, device='cuda:0')) | > current_lr_0: 0.00017640976905052529 | > current_lr_1: 0.00017640976905052529 | > step_time: 0.457 (0.44911306412493596) | > loader_time: 0.0107 (0.008303161527289719) --> EVAL PERFORMANCE | > avg_loader_time: 0.008756335352508115 (+0.0001029162339761247) | > avg_step_time: 0.4570911300014442 (-0.0009466325733024084) | > avg_loss_disc: 2.622116370939872 (+0.03610731178605908) | > avg_loss_disc_real_0: 0.14014774686853657 (+0.033722726103495596) | > avg_loss_disc_real_1: 0.22331277606352953 (+0.0028336786888014664) | > avg_loss_disc_real_2: 0.23369224747301823 (+0.019145202363880537) | > avg_loss_disc_real_3: 0.23491083169487162 (+0.0078006063548612115) | > avg_loss_disc_real_4: 0.24212498878928976 (+0.006036297116481071) | > avg_loss_disc_real_5: 0.24610686994774242 (+0.005186759670015806) | > avg_loss_0: 2.622116370939872 (+0.03610731178605908) | > avg_grad_norm_0: tensor(241.9788, device='cuda:0') (+tensor(31.4268, device='cuda:0')) | > avg_loss_gen: 2.0532197364619087 (-0.09179433970384121) | > avg_loss_kl: 2.0376961936413402 (+0.014363277126366203) | > avg_loss_feat: 3.5221919644046835 (-0.43444182839192) | > avg_loss_mel: 16.586045896503276 (-0.18007604840776636) | > avg_loss_duration: 1.5495258049226144 (+0.0017591345478111808) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.748679550600723 (-0.6901898719895065) | > avg_grad_norm_1: tensor(10344.8779, device='cuda:0') (+tensor(3110.7100, device='cuda:0'))  > EPOCH: 1005/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:34:29)   --> TIME: 2025-02-25 18:34:49 -- STEP: 39/72 -- GLOBAL_STEP: 1072400 | > loss_disc: 2.5494649410247803 (2.5713543525108924) | > loss_disc_real_0: 0.10807834565639496 (0.11808844483815707) | > loss_disc_real_1: 0.1908164620399475 (0.21568041963455004) | > loss_disc_real_2: 0.20637978613376617 (0.23027254946720907) | > loss_disc_real_3: 0.23653161525726318 (0.23624394299128118) | > loss_disc_real_4: 0.2590121030807495 (0.24418925092770502) | > loss_disc_real_5: 0.2285250574350357 (0.2396425643028357) | > loss_0: 2.5494649410247803 (2.5713543525108924) | > grad_norm_0: tensor(243.9645, device='cuda:0') (tensor(363.0302, device='cuda:0')) | > loss_gen: 2.081096887588501 (2.0993201304704714) | > loss_kl: 1.999358892440796 (1.947063623330532) | > loss_feat: 3.529188632965088 (3.793786855844351) | > loss_mel: 16.48878288269043 (16.611719840612167) | > loss_duration: 1.5617032051086426 (1.5098355611165364) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.66012954711914 (25.961726017487358) | > grad_norm_1: tensor(10385.1582, device='cuda:0') (tensor(13366.2480, device='cuda:0')) | > current_lr_0: 0.00017638771782939395 | > current_lr_1: 0.00017638771782939395 | > step_time: 0.447 (0.45392463146111905) | > loader_time: 0.008 (0.007415869297125401) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688557315880144 (-6.777803662797073e-05) | > avg_step_time: 0.46386975301823147 (+0.006778623016787277) | > avg_loss_disc: 2.580764572385332 (-0.041351798554539965) | > avg_loss_disc_real_0: 0.11915659474235185 (-0.020991152126184714) | > avg_loss_disc_real_1: 0.220253736620218 (-0.0030590394433115398) | > avg_loss_disc_real_2: 0.22843225090436534 (-0.0052599965686528916) | > avg_loss_disc_real_3: 0.23503836632614405 (+0.0001275346312724357) | > avg_loss_disc_real_4: 0.24354897208616766 (+0.001423983296877901) | > avg_loss_disc_real_5: 0.24036925422473693 (-0.00573761572300549) | > avg_loss_0: 2.580764572385332 (-0.041351798554539965) | > avg_grad_norm_0: tensor(367.7722, device='cuda:0') (+tensor(125.7935, device='cuda:0')) | > avg_loss_gen: 2.096063059820256 (+0.042843323358347174) | > avg_loss_kl: 2.0160756849906813 (-0.0216205086506589) | > avg_loss_feat: 3.739724253264951 (+0.21753228886026754) | > avg_loss_mel: 16.5796341425936 (-0.006411753909674189) | > avg_loss_duration: 1.5422357102514994 (-0.007290094671114966) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.973732827414928 (+0.2250532768142044) | > avg_grad_norm_1: tensor(13396.9736, device='cuda:0') (+tensor(3052.0957, device='cuda:0'))  > EPOCH: 1006/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:35:07)   --> TIME: 2025-02-25 18:35:17 -- STEP: 17/72 -- GLOBAL_STEP: 1072450 | > loss_disc: 2.5073068141937256 (2.573697398690616) | > loss_disc_real_0: 0.06629429757595062 (0.11318394366432638) | > loss_disc_real_1: 0.23101212084293365 (0.21176314266288981) | > loss_disc_real_2: 0.22636418044567108 (0.23051691055297852) | > loss_disc_real_3: 0.23530367016792297 (0.2340757355970495) | > loss_disc_real_4: 0.24453943967819214 (0.24417335057959838) | > loss_disc_real_5: 0.2270950824022293 (0.24954258256098813) | > loss_0: 2.5073068141937256 (2.573697398690616) | > grad_norm_0: tensor(460.3058, device='cuda:0') (tensor(328.1321, device='cuda:0')) | > loss_gen: 2.1628470420837402 (2.1396600709241977) | > loss_kl: 2.322863817214966 (1.9335567460340612) | > loss_feat: 4.054259300231934 (3.9074089947868798) | > loss_mel: 16.772090911865234 (16.696654880748074) | > loss_duration: 1.525118350982666 (1.4891128049177282) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.83717918395996 (26.166393504423255) | > grad_norm_1: tensor(13672.8682, device='cuda:0') (tensor(11936.1074, device='cuda:0')) | > current_lr_0: 0.00017636566936466528 | > current_lr_1: 0.00017636566936466528 | > step_time: 0.433 (0.43819449929630055) | > loader_time: 0.0064 (0.006955427281996783)  --> TIME: 2025-02-25 18:35:40 -- STEP: 67/72 -- GLOBAL_STEP: 1072500 | > loss_disc: 2.5934653282165527 (2.580535781917287) | > loss_disc_real_0: 0.13604338467121124 (0.12019164671203983) | > loss_disc_real_1: 0.22026725113391876 (0.21696229901776384) | > loss_disc_real_2: 0.2248721420764923 (0.23192044225201677) | > loss_disc_real_3: 0.20538349449634552 (0.23371737127873435) | > loss_disc_real_4: 0.24457088112831116 (0.24444834154043624) | > loss_disc_real_5: 0.2289448231458664 (0.24313382604228917) | > loss_0: 2.5934653282165527 (2.580535781917287) | > grad_norm_0: tensor(353.1995, device='cuda:0') (tensor(374.7055, device='cuda:0')) | > loss_gen: 2.137896776199341 (2.1299421253489026) | > loss_kl: 2.086496114730835 (1.9953223822721795) | > loss_feat: 3.8793752193450928 (3.806730907354782) | > loss_mel: 16.767452239990234 (16.510132633038424) | > loss_duration: 1.6266608238220215 (1.534645185541751) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.497882843017578 (25.97677327625787) | > grad_norm_1: tensor(12816.4346, device='cuda:0') (tensor(13269.8389, device='cuda:0')) | > current_lr_0: 0.00017636566936466528 | > current_lr_1: 0.00017636566936466528 | > step_time: 0.5307 (0.45409562338644) | > loader_time: 0.0122 (0.008453931381453329) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624570470460703 (-6.39868454194413e-05) | > avg_step_time: 0.45837694154658787 (-0.005492811471643599) | > avg_loss_disc: 2.5826567327472527 (+0.0018921603619208582) | > avg_loss_disc_real_0: 0.12018326795856718 (+0.0010266732162153291) | > avg_loss_disc_real_1: 0.21788895256082777 (-0.0023647840593902236) | > avg_loss_disc_real_2: 0.23149601254664676 (+0.0030637616422814207) | > avg_loss_disc_real_3: 0.23446346945326094 (-0.0005748968728831183) | > avg_loss_disc_real_4: 0.24513503090596536 (+0.0015860588197977032) | > avg_loss_disc_real_5: 0.24378316377250242 (+0.0034139095477654924) | > avg_loss_0: 2.5826567327472527 (+0.0018921603619208582) | > avg_grad_norm_0: tensor(376.7553, device='cuda:0') (+tensor(8.9831, device='cuda:0')) | > avg_loss_gen: 2.1304037067252146 (+0.03434064690495875) | > avg_loss_kl: 2.0007487501896604 (-0.015326934801020986) | > avg_loss_feat: 3.800599883979475 (+0.060875630714523954) | > avg_loss_mel: 16.523745872605016 (-0.05588826998858565) | > avg_loss_duration: 1.5383710340714793 (-0.0038646761800200657) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.993869297940964 (+0.02013647052603673) | > avg_grad_norm_1: tensor(13262.8086, device='cuda:0') (tensor(-134.1650, device='cuda:0'))  > EPOCH: 1007/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:35:44)   --> TIME: 2025-02-25 18:36:07 -- STEP: 45/72 -- GLOBAL_STEP: 1072550 | > loss_disc: 2.5007052421569824 (2.602842728296916) | > loss_disc_real_0: 0.11486194282770157 (0.12705743138988812) | > loss_disc_real_1: 0.21722012758255005 (0.226361420750618) | > loss_disc_real_2: 0.23574554920196533 (0.23337055477831098) | > loss_disc_real_3: 0.21535681188106537 (0.23443205753962199) | > loss_disc_real_4: 0.23451590538024902 (0.24307223425971136) | > loss_disc_real_5: 0.22089624404907227 (0.24394012722704145) | > loss_0: 2.5007052421569824 (2.602842728296916) | > grad_norm_0: tensor(334.9651, device='cuda:0') (tensor(410.3991, device='cuda:0')) | > loss_gen: 2.146827220916748 (2.145638171831767) | > loss_kl: 2.0775768756866455 (1.9876253366470338) | > loss_feat: 3.700409412384033 (3.7930271201663546) | > loss_mel: 17.054489135742188 (16.665639283921983) | > loss_duration: 1.5626531839370728 (1.5131990141338771) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.541955947875977 (26.105128902859157) | > grad_norm_1: tensor(14401.4609, device='cuda:0') (tensor(14420.5352, device='cuda:0')) | > current_lr_0: 0.0001763436236559947 | > current_lr_1: 0.0001763436236559947 | > step_time: 0.464 (0.44693479008144804) | > loader_time: 0.0089 (0.007590426339043511) --> EVAL PERFORMANCE | > avg_loader_time: 0.008700639429226728 (+7.606895876602522e-05) | > avg_step_time: 0.46067721407178425 (+0.0023002725251963785) | > avg_loss_disc: 2.59930161355247 (+0.016644880805217088) | > avg_loss_disc_real_0: 0.12782080487256314 (+0.007637536913995963) | > avg_loss_disc_real_1: 0.22273899174072373 (+0.00485003917989596) | > avg_loss_disc_real_2: 0.23187269963009258 (+0.00037668708344582025) | > avg_loss_disc_real_3: 0.23488255222918283 (+0.00041908277592189735) | > avg_loss_disc_real_4: 0.24345289170742035 (-0.001682139198545013) | > avg_loss_disc_real_5: 0.24394111943916535 (+0.00015795566666293448) | > avg_loss_0: 2.59930161355247 (+0.016644880805217088) | > avg_grad_norm_0: tensor(413.7190, device='cuda:0') (+tensor(36.9637, device='cuda:0')) | > avg_loss_gen: 2.139404627638804 (+0.009000920913589283) | > avg_loss_kl: 2.0359335785180757 (+0.035184828328415296) | > avg_loss_feat: 3.7598573154127095 (-0.0407425685667655) | > avg_loss_mel: 16.54599975532209 (+0.022253882717073736) | > avg_loss_duration: 1.5399192051148751 (+0.001548171043395774) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.02111432250117 (+0.027245024560205167) | > avg_grad_norm_1: tensor(14542.3408, device='cuda:0') (+tensor(1279.5322, device='cuda:0'))  > EPOCH: 1008/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:36:21)   --> TIME: 2025-02-25 18:36:34 -- STEP: 23/72 -- GLOBAL_STEP: 1072600 | > loss_disc: 2.5257184505462646 (2.5650935484015425) | > loss_disc_real_0: 0.10744127631187439 (0.11444546638623528) | > loss_disc_real_1: 0.17735451459884644 (0.21733550533004428) | > loss_disc_real_2: 0.22210462391376495 (0.23116903007030487) | > loss_disc_real_3: 0.21363981068134308 (0.22919307325197302) | > loss_disc_real_4: 0.24088720977306366 (0.23962864344534668) | > loss_disc_real_5: 0.24741880595684052 (0.2458875755900922) | > loss_0: 2.5257184505462646 (2.5650935484015425) | > grad_norm_0: tensor(260.7362, device='cuda:0') (tensor(414.7480, device='cuda:0')) | > loss_gen: 2.225034475326538 (2.1755837305732397) | > loss_kl: 1.9873838424682617 (2.0163051978401514) | > loss_feat: 4.351587295532227 (3.966375620468803) | > loss_mel: 16.429885864257812 (16.611047786215078) | > loss_duration: 1.5360777378082275 (1.4976452433544656) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.52996826171875 (26.266957573268723) | > grad_norm_1: tensor(14203.0020, device='cuda:0') (tensor(16022.6484, device='cuda:0')) | > current_lr_0: 0.0001763215807030377 | > current_lr_1: 0.0001763215807030377 | > step_time: 0.4402 (0.4380605013474174) | > loader_time: 0.0068 (0.007038344507632049) --> EVAL PERFORMANCE | > avg_loader_time: 0.008813347614986798 (+0.00011270818576007008) | > avg_step_time: 0.46441160457234987 (+0.003734390500565621) | > avg_loss_disc: 2.5905611750105737 (-0.0087404385418961) | > avg_loss_disc_real_0: 0.12795341896339196 (+0.0001326140908288198) | > avg_loss_disc_real_1: 0.2219108095471288 (-0.0008281821935949285) | > avg_loss_disc_real_2: 0.23198159185933395 (+0.0001088922292413752) | > avg_loss_disc_real_3: 0.23329606841147785 (-0.0015864838177049867) | > avg_loss_disc_real_4: 0.24136742802572922 (-0.00208546368169113) | > avg_loss_disc_real_5: 0.24552678222387608 (+0.0015856627847107285) | > avg_loss_0: 2.5905611750105737 (-0.0087404385418961) | > avg_grad_norm_0: tensor(416.2633, device='cuda:0') (+tensor(2.5443, device='cuda:0')) | > avg_loss_gen: 2.1573681193338325 (+0.017963491695028644) | > avg_loss_kl: 2.0646534150754907 (+0.028719836557415057) | > avg_loss_feat: 3.848394058120083 (+0.08853674270737333) | > avg_loss_mel: 16.581250553399745 (+0.03525079807765508) | > avg_loss_duration: 1.5407476525911143 (+0.0008284474762392247) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.192413679310974 (+0.17129935680980424) | > avg_grad_norm_1: tensor(15502.9482, device='cuda:0') (+tensor(960.6074, device='cuda:0'))  > EPOCH: 1009/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:36:59)   --> TIME: 2025-02-25 18:37:01 -- STEP: 1/72 -- GLOBAL_STEP: 1072650 | > loss_disc: 2.725334644317627 (2.725334644317627) | > loss_disc_real_0: 0.10163717716932297 (0.10163717716932297) | > loss_disc_real_1: 0.19985249638557434 (0.19985249638557434) | > loss_disc_real_2: 0.23959553241729736 (0.23959553241729736) | > loss_disc_real_3: 0.2477961629629135 (0.2477961629629135) | > loss_disc_real_4: 0.2604570984840393 (0.2604570984840393) | > loss_disc_real_5: 0.2514038383960724 (0.2514038383960724) | > loss_0: 2.725334644317627 (2.725334644317627) | > grad_norm_0: tensor(617.0152, device='cuda:0') (tensor(617.0152, device='cuda:0')) | > loss_gen: 2.0025131702423096 (2.0025131702423096) | > loss_kl: 1.9182219505310059 (1.9182219505310059) | > loss_feat: 3.6500182151794434 (3.6500182151794434) | > loss_mel: 17.019559860229492 (17.019559860229492) | > loss_duration: 1.4824299812316895 (1.4824299812316895) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.072742462158203 (26.072742462158203) | > grad_norm_1: tensor(15573.6094, device='cuda:0') (tensor(15573.6094, device='cuda:0')) | > current_lr_0: 0.00017629954050544982 | > current_lr_1: 0.00017629954050544982 | > step_time: 0.4299 (0.4298994541168213) | > loader_time: 0.0095 (0.009541749954223633)  --> TIME: 2025-02-25 18:37:24 -- STEP: 51/72 -- GLOBAL_STEP: 1072700 | > loss_disc: 2.490433692932129 (2.626532951990763) | > loss_disc_real_0: 0.09827437996864319 (0.1314199915995785) | > loss_disc_real_1: 0.26192575693130493 (0.21891018631411532) | > loss_disc_real_2: 0.23201322555541992 (0.23150013737818775) | > loss_disc_real_3: 0.2236519604921341 (0.23505865329620884) | > loss_disc_real_4: 0.24450066685676575 (0.24433263172121608) | > loss_disc_real_5: 0.24157270789146423 (0.24449702659074) | > loss_0: 2.490433692932129 (2.626532951990763) | > grad_norm_0: tensor(281.1037, device='cuda:0') (tensor(381.3420, device='cuda:0')) | > loss_gen: 2.42425274848938 (2.135436174916286) | > loss_kl: 2.027719736099243 (2.016952287917044) | > loss_feat: 3.995882749557495 (3.804016290926466) | > loss_mel: 17.00002098083496 (16.600067998848708) | > loss_duration: 1.5702601671218872 (1.5218985992319443) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.018138885498047 (26.07837142196356) | > grad_norm_1: tensor(16359.0176, device='cuda:0') (tensor(14053.0576, device='cuda:0')) | > current_lr_0: 0.00017629954050544982 | > current_lr_1: 0.00017629954050544982 | > step_time: 0.459 (0.44416534666921575) | > loader_time: 0.0091 (0.007752367094451306) --> EVAL PERFORMANCE | > avg_loader_time: 0.008601803175160582 (-0.0002115444398262164) | > avg_step_time: 0.4554177304388772 (-0.00899387413347269) | > avg_loss_disc: 2.6302219108796456 (+0.03966073586907193) | > avg_loss_disc_real_0: 0.13510842432438488 (+0.007155005360992922) | > avg_loss_disc_real_1: 0.22102305356045843 (-0.0008877559866703744) | > avg_loss_disc_real_2: 0.23162059318011916 (-0.0003609986792147968) | > avg_loss_disc_real_3: 0.23523612341410677 (+0.001940055002628921) | > avg_loss_disc_real_4: 0.24431150354130168 (+0.002944075515572464) | > avg_loss_disc_real_5: 0.24583857391082065 (+0.0003117916869445625) | > avg_loss_0: 2.6302219108796456 (+0.03966073586907193) | > avg_grad_norm_0: tensor(396.9095, device='cuda:0') (tensor(-19.3538, device='cuda:0')) | > avg_loss_gen: 2.1273365490873104 (-0.030031570246522143) | > avg_loss_kl: 2.055042342400886 (-0.009611072674604593) | > avg_loss_feat: 3.763441377962139 (-0.08495268015794366) | > avg_loss_mel: 16.646368322238118 (+0.06511776883837328) | > avg_loss_duration: 1.5404829844622547 (-0.0002646681288596131) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.132671597977758 (-0.05974208133321568) | > avg_grad_norm_1: tensor(14362.5967, device='cuda:0') (tensor(-1140.3516, device='cuda:0'))  > EPOCH: 1010/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:37:36)   --> TIME: 2025-02-25 18:37:51 -- STEP: 29/72 -- GLOBAL_STEP: 1072750 | > loss_disc: 2.7850327491760254 (2.674530267715454) | > loss_disc_real_0: 0.12988895177841187 (0.12582851175604193) | > loss_disc_real_1: 0.2565753757953644 (0.22017047117496358) | > loss_disc_real_2: 0.2381444126367569 (0.23328441689754353) | > loss_disc_real_3: 0.24700285494327545 (0.24412548130956188) | > loss_disc_real_4: 0.2675131857395172 (0.25094592108808716) | > loss_disc_real_5: 0.24367369711399078 (0.24644231744881334) | > loss_0: 2.7850327491760254 (2.674530267715454) | > grad_norm_0: tensor(627.8679, device='cuda:0') (tensor(442.8732, device='cuda:0')) | > loss_gen: 2.0486369132995605 (2.105212663782054) | > loss_kl: 2.0634138584136963 (1.986500805821912) | > loss_feat: 3.482715606689453 (3.7998550513695024) | > loss_mel: 16.9108943939209 (16.69486963337866) | > loss_duration: 1.5111770629882812 (1.4959389834568417) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.01683807373047 (26.082377203579608) | > grad_norm_1: tensor(15705.5781, device='cuda:0') (tensor(15036.7344, device='cuda:0')) | > current_lr_0: 0.00017627750306288664 | > current_lr_1: 0.00017627750306288664 | > step_time: 0.4376 (0.4337233592723978) | > loader_time: 0.008 (0.007016486135022393) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638932671345455 (+3.712949618487303e-05) | > avg_step_time: 0.4542792179215122 (-0.0011385125173649646) | > avg_loss_disc: 2.639533832039631 (+0.00931192115998547) | > avg_loss_disc_real_0: 0.1297125538260164 (-0.005395870498368482) | > avg_loss_disc_real_1: 0.22553556255051788 (+0.0045125089900594495) | > avg_loss_disc_real_2: 0.2304951701785477 (-0.0011254230015714517) | > avg_loss_disc_real_3: 0.2386386216106549 (+0.0034024981965481427) | > avg_loss_disc_real_4: 0.24456914587759634 (+0.00025764233629466116) | > avg_loss_disc_real_5: 0.24320021691456647 (-0.002638356996254171) | > avg_loss_0: 2.639533832039631 (+0.00931192115998547) | > avg_grad_norm_0: tensor(434.0396, device='cuda:0') (+tensor(37.1301, device='cuda:0')) | > avg_loss_gen: 2.115791124357305 (-0.011545424730005305) | > avg_loss_kl: 2.050688666357121 (-0.00435367604376502) | > avg_loss_feat: 3.749393809009606 (-0.01404756895253323) | > avg_loss_mel: 16.70699795199113 (+0.06062962975301289) | > avg_loss_duration: 1.5425833060707845 (+0.002100321608529798) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.165454918230076 (+0.03278332025231734) | > avg_grad_norm_1: tensor(14833.1738, device='cuda:0') (+tensor(470.5771, device='cuda:0'))  > EPOCH: 1011/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:38:13)   --> TIME: 2025-02-25 18:38:18 -- STEP: 7/72 -- GLOBAL_STEP: 1072800 | > loss_disc: 2.774890899658203 (2.5797254017421176) | > loss_disc_real_0: 0.1013358011841774 (0.11171453978334155) | > loss_disc_real_1: 0.14621783792972565 (0.20999746450356074) | > loss_disc_real_2: 0.2640218734741211 (0.2438966099705015) | > loss_disc_real_3: 0.19899152219295502 (0.2375223104442869) | > loss_disc_real_4: 0.25414931774139404 (0.24868681175368174) | > loss_disc_real_5: 0.24082987010478973 (0.24599026143550873) | > loss_0: 2.774890899658203 (2.5797254017421176) | > grad_norm_0: tensor(872.0506, device='cuda:0') (tensor(391.2443, device='cuda:0')) | > loss_gen: 1.6263301372528076 (2.0797140257699147) | > loss_kl: 1.8286762237548828 (1.968451874596732) | > loss_feat: 3.801332950592041 (3.7513282980237688) | > loss_mel: 17.956655502319336 (16.859577996390207) | > loss_duration: 1.4711122512817383 (1.4953167949404036) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.684104919433594 (26.15438951764788) | > grad_norm_1: tensor(5055.3594, device='cuda:0') (tensor(12790.7666, device='cuda:0')) | > current_lr_0: 0.00017625546837500377 | > current_lr_1: 0.00017625546837500377 | > step_time: 0.4471 (0.4383507115500314) | > loader_time: 0.0057 (0.006502321788242885)  --> TIME: 2025-02-25 18:38:42 -- STEP: 57/72 -- GLOBAL_STEP: 1072850 | > loss_disc: 3.0280723571777344 (2.702120358483833) | > loss_disc_real_0: 0.3892222046852112 (0.1431215417228247) | > loss_disc_real_1: 0.30313727259635925 (0.23702924554808102) | > loss_disc_real_2: 0.24097001552581787 (0.2326491469876808) | > loss_disc_real_3: 0.2365865856409073 (0.2242680774968967) | > loss_disc_real_4: 0.23989342153072357 (0.24079732047884086) | > loss_disc_real_5: 0.26475563645362854 (0.24321283138634864) | > loss_0: 3.0280723571777344 (2.702120358483833) | > grad_norm_0: tensor(671.9703, device='cuda:0') (tensor(363.6997, device='cuda:0')) | > loss_gen: 1.9245860576629639 (2.065581160679198) | > loss_kl: 2.316016435623169 (2.158194711333827) | > loss_feat: 3.1231541633605957 (3.6774779662751316) | > loss_mel: 18.153244018554688 (17.05572820964612) | > loss_duration: 1.6095694303512573 (1.5415114122524598) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.126569747924805 (26.498493663051672) | > grad_norm_1: tensor(2701.7310, device='cuda:0') (tensor(9790.1436, device='cuda:0')) | > current_lr_0: 0.00017625546837500377 | > current_lr_1: 0.00017625546837500377 | > step_time: 0.4569 (0.45415089841474565) | > loader_time: 0.0097 (0.008056021573250752) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868941024995186 (+5.047757860640427e-05) | > avg_step_time: 0.4618429264552157 (+0.007563708533703473) | > avg_loss_disc: 2.7439979331594118 (+0.10446410111978066) | > avg_loss_disc_real_0: 0.16928890924638426 (+0.03957635542036786) | > avg_loss_disc_real_1: 0.2438021096545206 (+0.018266547104002717) | > avg_loss_disc_real_2: 0.2358240439018733 (+0.005328873723325606) | > avg_loss_disc_real_3: 0.23245047368633914 (-0.006188147924315768) | > avg_loss_disc_real_4: 0.2457948110892739 (+0.0012256652116775513) | > avg_loss_disc_real_5: 0.2459151889656631 (+0.002714972051096637) | > avg_loss_0: 2.7439979331594118 (+0.10446410111978066) | > avg_grad_norm_0: tensor(348.4039, device='cuda:0') (tensor(-85.6357, device='cuda:0')) | > avg_loss_gen: 2.0555188723013433 (-0.06027225205596176) | > avg_loss_kl: 2.414277887680161 (+0.36358922132304006) | > avg_loss_feat: 3.5887278402355354 (-0.16066596877407058) | > avg_loss_mel: 18.34169465723172 (+1.6346967052405894) | > avg_loss_duration: 1.5904355787895095 (+0.04785227271872494) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.990654958805564 (+1.8252000405754885) | > avg_grad_norm_1: tensor(8711.6006, device='cuda:0') (tensor(-6121.5732, device='cuda:0'))  > EPOCH: 1012/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:38:51)   --> TIME: 2025-02-25 18:39:08 -- STEP: 35/72 -- GLOBAL_STEP: 1072900 | > loss_disc: 2.6974973678588867 (2.540493665422712) | > loss_disc_real_0: 0.06376026570796967 (0.07571461296507291) | > loss_disc_real_1: 0.2288304567337036 (0.2214164674282074) | > loss_disc_real_2: 0.22427502274513245 (0.2462635827916009) | > loss_disc_real_3: 0.22384296357631683 (0.23990399198872703) | > loss_disc_real_4: 0.25166237354278564 (0.24450863003730774) | > loss_disc_real_5: 0.2665199935436249 (0.24180105456284115) | > loss_0: 2.6974973678588867 (2.540493665422712) | > grad_norm_0: tensor(179.5669, device='cuda:0') (tensor(186.9277, device='cuda:0')) | > loss_gen: 1.983466625213623 (2.300700879096985) | > loss_kl: 2.012866497039795 (2.3088836669921875) | > loss_feat: 3.434699058532715 (4.281718179157802) | > loss_mel: 17.348833084106445 (18.31888198852539) | > loss_duration: 1.5723121166229248 (1.5732386691229685) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.352176666259766 (28.783423451014926) | > grad_norm_1: tensor(10469.0205, device='cuda:0') (tensor(7347.5981, device='cuda:0')) | > current_lr_0: 0.00017623343644145687 | > current_lr_1: 0.00017623343644145687 | > step_time: 0.4429 (0.4370221274239676) | > loader_time: 0.0086 (0.007196276528494698) --> EVAL PERFORMANCE | > avg_loader_time: 0.008610342589902207 (-7.906766004965246e-05) | > avg_step_time: 0.4544284847420706 (-0.007414441713145092) | > avg_loss_disc: 2.5650359543276506 (-0.17896197883176113) | > avg_loss_disc_real_0: 0.08718153115519335 (-0.08210737809119091) | > avg_loss_disc_real_1: 0.22433209880976612 (-0.01947001084475447) | > avg_loss_disc_real_2: 0.24873920015885798 (+0.012915156256984667) | > avg_loss_disc_real_3: 0.23936731777560544 (+0.006916844089266294) | > avg_loss_disc_real_4: 0.24795613532334987 (+0.002161324234075973) | > avg_loss_disc_real_5: 0.24288711581431643 (-0.0030280731513466774) | > avg_loss_0: 2.5650359543276506 (-0.17896197883176113) | > avg_grad_norm_0: tensor(199.8739, device='cuda:0') (tensor(-148.5300, device='cuda:0')) | > avg_loss_gen: 2.2449480815672533 (+0.18942920926591) | > avg_loss_kl: 2.2154613095270084 (-0.1988165781531528) | > avg_loss_feat: 4.040732084865302 (+0.4520042446297663) | > avg_loss_mel: 17.628730935110166 (-0.7129637221215539) | > avg_loss_duration: 1.5833690535854286 (-0.007066525204080909) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.713241469692175 (-0.27741348911338903) | > avg_grad_norm_1: tensor(6804.4473, device='cuda:0') (tensor(-1907.1533, device='cuda:0'))  > EPOCH: 1013/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:39:27)   --> TIME: 2025-02-25 18:39:35 -- STEP: 13/72 -- GLOBAL_STEP: 1072950 | > loss_disc: 2.4229540824890137 (2.551412582397461) | > loss_disc_real_0: 0.05030278116464615 (0.11962075072985429) | > loss_disc_real_1: 0.22467273473739624 (0.22147180713140047) | > loss_disc_real_2: 0.18973132967948914 (0.20841743166630083) | > loss_disc_real_3: 0.21426327526569366 (0.24695522510088402) | > loss_disc_real_4: 0.23332718014717102 (0.2456349959740272) | > loss_disc_real_5: 0.24554266035556793 (0.2438825643979586) | > loss_0: 2.4229540824890137 (2.551412582397461) | > grad_norm_0: tensor(190.8737, device='cuda:0') (tensor(255.6560, device='cuda:0')) | > loss_gen: 2.2469873428344727 (2.2987539768218994) | > loss_kl: 1.914920687675476 (1.9400187547390277) | > loss_feat: 4.265411376953125 (4.0345479525052586) | > loss_mel: 16.512351989746094 (16.92152404785156) | > loss_duration: 1.4370770454406738 (1.5021872795545137) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.376747131347656 (26.69703219487117) | > grad_norm_1: tensor(8390.4043, device='cuda:0') (tensor(6594.6455, device='cuda:0')) | > current_lr_0: 0.00017621140726190168 | > current_lr_1: 0.00017621140726190168 | > step_time: 0.4695 (0.4351252775925856) | > loader_time: 0.006 (0.006704935660729041)  --> TIME: 2025-02-25 18:39:58 -- STEP: 63/72 -- GLOBAL_STEP: 1073000 | > loss_disc: 2.53678560256958 (2.548552630439637) | > loss_disc_real_0: 0.13860976696014404 (0.10536220799835902) | > loss_disc_real_1: 0.18237701058387756 (0.22515629184624505) | > loss_disc_real_2: 0.2296735644340515 (0.225991609077605) | > loss_disc_real_3: 0.22833648324012756 (0.23373560489170134) | > loss_disc_real_4: 0.24341709911823273 (0.2444046681362485) | > loss_disc_real_5: 0.2647695243358612 (0.2484741009890087) | > loss_0: 2.53678560256958 (2.548552630439637) | > grad_norm_0: tensor(364.5995, device='cuda:0') (tensor(276.3136, device='cuda:0')) | > loss_gen: 2.3205151557922363 (2.2409465994153703) | > loss_kl: 1.927165150642395 (2.0007699311725675) | > loss_feat: 4.065598011016846 (3.9789720679086353) | > loss_mel: 17.543895721435547 (16.865268646724633) | > loss_duration: 1.5758506059646606 (1.5401172429796248) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.433025360107422 (26.626074382237025) | > grad_norm_1: tensor(5987.2217, device='cuda:0') (tensor(7398.9893, device='cuda:0')) | > current_lr_0: 0.00017621140726190168 | > current_lr_1: 0.00017621140726190168 | > step_time: 0.5073 (0.446551277523949) | > loader_time: 0.0116 (0.008179395917862181) --> EVAL PERFORMANCE | > avg_loader_time: 0.008569458840598523 (-4.0883749303683575e-05) | > avg_step_time: 0.452773671754649 (-0.0016548129874215944) | > avg_loss_disc: 2.5528893940885298 (-0.012146560239120863) | > avg_loss_disc_real_0: 0.10578668248695387 (+0.018605151331760522) | > avg_loss_disc_real_1: 0.22459755397178757 (+0.0002654551620214496) | > avg_loss_disc_real_2: 0.22594023842207142 (-0.022798961736786555) | > avg_loss_disc_real_3: 0.23292017360808145 (-0.006447144167523983) | > avg_loss_disc_real_4: 0.243254058167968 (-0.004702077155381873) | > avg_loss_disc_real_5: 0.24745036619649807 (+0.004563250382181638) | > avg_loss_0: 2.5528893940885298 (-0.012146560239120863) | > avg_grad_norm_0: tensor(263.6371, device='cuda:0') (+tensor(63.7632, device='cuda:0')) | > avg_loss_gen: 2.230589319282854 (-0.014358762284399251) | > avg_loss_kl: 2.0133640111332207 (-0.20209729839378765) | > avg_loss_feat: 3.949423078080298 (-0.09130900678500353) | > avg_loss_mel: 16.851449039620405 (-0.7772818954897609) | > avg_loss_duration: 1.5482572938354922 (-0.03511175974993641) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.593082642891037 (-1.1201588268011378) | > avg_grad_norm_1: tensor(7109.9624, device='cuda:0') (+tensor(305.5151, device='cuda:0'))  > EPOCH: 1014/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:40:04)   --> TIME: 2025-02-25 18:40:25 -- STEP: 41/72 -- GLOBAL_STEP: 1073050 | > loss_disc: 2.8118181228637695 (2.762964080019695) | > loss_disc_real_0: 0.1830746978521347 (0.17173930384763858) | > loss_disc_real_1: 0.20126327872276306 (0.27415772273046213) | > loss_disc_real_2: 0.21636788547039032 (0.22936799068276476) | > loss_disc_real_3: 0.1883893758058548 (0.22264689643208574) | > loss_disc_real_4: 0.21719256043434143 (0.23950763373840145) | > loss_disc_real_5: 0.24131812155246735 (0.23945647732513706) | > loss_0: 2.8118181228637695 (2.762964080019695) | > grad_norm_0: tensor(448.9071, device='cuda:0') (tensor(274.5320, device='cuda:0')) | > loss_gen: 1.5740292072296143 (2.114701893271469) | > loss_kl: 2.8235995769500732 (2.031707557236276) | > loss_feat: 3.1453733444213867 (3.6002219072202357) | > loss_mel: 18.658004760742188 (16.973139251150737) | > loss_duration: 1.6319751739501953 (1.525833283982626) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.832983016967773 (26.24560398008765) | > grad_norm_1: tensor(3771.0027, device='cuda:0') (tensor(7606.7861, device='cuda:0')) | > current_lr_0: 0.00017618938083599394 | > current_lr_1: 0.00017618938083599394 | > step_time: 0.4423 (0.43803085350408777) | > loader_time: 0.0091 (0.007417655572658632) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568270105711172 (-1.1887348873515502e-06) | > avg_step_time: 0.4568908617530071 (+0.004117189998358084) | > avg_loss_disc: 2.7550285399799614 (+0.20213914589143167) | > avg_loss_disc_real_0: 0.17261443413059474 (+0.06682775164364087) | > avg_loss_disc_real_1: 0.2591309393165816 (+0.034533385344794026) | > avg_loss_disc_real_2: 0.22923592194704942 (+0.003295683524977999) | > avg_loss_disc_real_3: 0.23412248870016825 (+0.001202315092086792) | > avg_loss_disc_real_4: 0.23931838970788769 (-0.0039356684600803105) | > avg_loss_disc_real_5: 0.24068053381543764 (-0.006769832381060437) | > avg_loss_0: 2.7550285399799614 (+0.20213914589143167) | > avg_grad_norm_0: tensor(294.2255, device='cuda:0') (+tensor(30.5885, device='cuda:0')) | > avg_loss_gen: 2.094703625625288 (-0.1358856936575661) | > avg_loss_kl: 2.164509949549822 (+0.1511459384166014) | > avg_loss_feat: 3.602780194349692 (-0.34664288373060614) | > avg_loss_mel: 17.534679211361304 (+0.6832301717408988) | > avg_loss_duration: 1.566556065854892 (+0.01829877201939989) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.963229246542483 (+0.37014660365144536) | > avg_grad_norm_1: tensor(7243.3916, device='cuda:0') (+tensor(133.4292, device='cuda:0'))  > EPOCH: 1015/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:40:41)   --> TIME: 2025-02-25 18:40:52 -- STEP: 19/72 -- GLOBAL_STEP: 1073100 | > loss_disc: 2.5531704425811768 (2.659347233019377) | > loss_disc_real_0: 0.1279177963733673 (0.19321739477546593) | > loss_disc_real_1: 0.22747984528541565 (0.22963152747405202) | > loss_disc_real_2: 0.1846020668745041 (0.2185627292645605) | > loss_disc_real_3: 0.16236130893230438 (0.2347157934778615) | > loss_disc_real_4: 0.17696604132652283 (0.22811239173537806) | > loss_disc_real_5: 0.18861433863639832 (0.24072535806580594) | > loss_0: 2.5531704425811768 (2.659347233019377) | > grad_norm_0: tensor(13.0977, device='cuda:0') (tensor(237.5890, device='cuda:0')) | > loss_gen: 2.0430819988250732 (2.060744982016714) | > loss_kl: 2.414923667907715 (3.0705676580730237) | > loss_feat: 3.7891509532928467 (3.4887247462021675) | > loss_mel: 20.21967124938965 (21.07624806855854) | > loss_duration: 1.6509106159210205 (1.8741044621718557) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.117738723754883 (31.570390199360094) | > grad_norm_1: tensor(4226.8315, device='cuda:0') (tensor(6535.8638, device='cuda:0')) | > current_lr_0: 0.00017616735716338944 | > current_lr_1: 0.00017616735716338944 | > step_time: 0.4204 (0.4307676490984465) | > loader_time: 0.006 (0.006841032128585012)  --> TIME: 2025-02-25 18:41:15 -- STEP: 69/72 -- GLOBAL_STEP: 1073150 | > loss_disc: 2.657421112060547 (2.6765711722166645) | > loss_disc_real_0: 0.1350073665380478 (0.16173141876208605) | > loss_disc_real_1: 0.18851007521152496 (0.22331629801487576) | > loss_disc_real_2: 0.23963777720928192 (0.23809687663679538) | > loss_disc_real_3: 0.270338773727417 (0.2493558960116428) | > loss_disc_real_4: 0.274729460477829 (0.24673276167848837) | > loss_disc_real_5: 0.2384423315525055 (0.24876920641332434) | > loss_0: 2.657421112060547 (2.6765711722166645) | > grad_norm_0: tensor(266.2222, device='cuda:0') (tensor(240.9187, device='cuda:0')) | > loss_gen: 2.254044532775879 (2.113522123599398) | > loss_kl: 2.4569497108459473 (2.450245915979579) | > loss_feat: 3.4961352348327637 (3.623731319455133) | > loss_mel: 16.928707122802734 (18.543417225713313) | > loss_duration: 1.6748496294021606 (1.6872220523115518) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.810688018798828 (28.418138697527457) | > grad_norm_1: tensor(11475.2578, device='cuda:0') (tensor(7300.3535, device='cuda:0')) | > current_lr_0: 0.00017616735716338944 | > current_lr_1: 0.00017616735716338944 | > step_time: 0.5338 (0.44847341551296954) | > loader_time: 0.0127 (0.00866176080012667) --> EVAL PERFORMANCE | > avg_loader_time: 0.008714323312463895 (+0.00014605320675272274) | > avg_step_time: 0.45019233394676533 (-0.00669852780624175) | > avg_loss_disc: 2.678672394282382 (-0.07635614569757943) | > avg_loss_disc_real_0: 0.16281689163035065 (-0.009797542500244089) | > avg_loss_disc_real_1: 0.22445409986334788 (-0.034676839453233715) | > avg_loss_disc_real_2: 0.23844127734781992 (+0.009205355400770499) | > avg_loss_disc_real_3: 0.2491263745116516 (+0.015003885811483347) | > avg_loss_disc_real_4: 0.24717582300515242 (+0.007857433297264738) | > avg_loss_disc_real_5: 0.24839622013165918 (+0.007715686316221548) | > avg_loss_0: 2.678672394282382 (-0.07635614569757943) | > avg_grad_norm_0: tensor(238.8097, device='cuda:0') (tensor(-55.4158, device='cuda:0')) | > avg_loss_gen: 2.1102479276522783 (+0.015544302026990309) | > avg_loss_kl: 2.443716025688279 (+0.2792060761384567) | > avg_loss_feat: 3.612478293163676 (+0.009698098813983869) | > avg_loss_mel: 18.504486755586004 (+0.9698075442247003) | > avg_loss_duration: 1.6843471426359367 (+0.11779107678104461) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.35527621524435 (+1.392046968701866) | > avg_grad_norm_1: tensor(7210.8022, device='cuda:0') (tensor(-32.5894, device='cuda:0'))  > EPOCH: 1016/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:41:18)   --> TIME: 2025-02-25 18:41:41 -- STEP: 47/72 -- GLOBAL_STEP: 1073200 | > loss_disc: 2.30349063873291 (2.68288418587218) | > loss_disc_real_0: 0.11128032207489014 (0.18840059213974375) | > loss_disc_real_1: 0.22466376423835754 (0.25333076905696955) | > loss_disc_real_2: 0.22433695197105408 (0.24692657463094023) | > loss_disc_real_3: 0.2374345362186432 (0.23925212945075744) | > loss_disc_real_4: 0.2251436561346054 (0.23945664535177516) | > loss_disc_real_5: 0.23034174740314484 (0.24138243337895007) | > loss_0: 2.30349063873291 (2.68288418587218) | > grad_norm_0: tensor(276.0206, device='cuda:0') (tensor(277.7306, device='cuda:0')) | > loss_gen: 2.538444995880127 (2.1827988244117575) | > loss_kl: 3.1784985065460205 (2.394052579047832) | > loss_feat: 4.4610819816589355 (3.7246540759472135) | > loss_mel: 22.456958770751953 (18.84084291661039) | > loss_duration: 1.6857341527938843 (1.5899383844213282) | > amp_scaler: 8.0 (8.0) | > loss_1: 34.320716857910156 (28.7322869402297) | > grad_norm_1: tensor(3059.4333, device='cuda:0') (tensor(5163.4771, device='cuda:0')) | > current_lr_0: 0.000176145336243744 | > current_lr_1: 0.000176145336243744 | > step_time: 0.4514 (0.43783746374414323) | > loader_time: 0.0086 (0.007811835471619951) --> EVAL PERFORMANCE | > avg_loader_time: 0.008741445944342816 (+2.7122631878921666e-05) | > avg_step_time: 0.4500939812458737 (-9.835270089164627e-05) | > avg_loss_disc: 2.6370698901968948 (-0.04160250408548727) | > avg_loss_disc_real_0: 0.14977182547601173 (-0.013045066154338925) | > avg_loss_disc_real_1: 0.23934768602042136 (+0.01489358615707348) | > avg_loss_disc_real_2: 0.24251303769333263 (+0.004071760345512709) | > avg_loss_disc_real_3: 0.2395682261443474 (-0.009558148367304203) | > avg_loss_disc_real_4: 0.2393023820830063 (-0.007873440922146119) | > avg_loss_disc_real_5: 0.24476164327540867 (-0.0036345768562505154) | > avg_loss_0: 2.6370698901968948 (-0.04160250408548727) | > avg_grad_norm_0: tensor(254.1509, device='cuda:0') (+tensor(15.3412, device='cuda:0')) | > avg_loss_gen: 2.204735321058353 (+0.09448739340607482) | > avg_loss_kl: 2.449497311887607 (+0.005781286199328051) | > avg_loss_feat: 3.872346666497244 (+0.25986837333356805) | > avg_loss_mel: 19.049093058411497 (+0.5446063028254926) | > avg_loss_duration: 1.6067303519853404 (-0.07761679065059623) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.182402704803035 (+0.8271264895586867) | > avg_grad_norm_1: tensor(5322.5996, device='cuda:0') (tensor(-1888.2026, device='cuda:0'))  > EPOCH: 1017/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:41:54)   --> TIME: 2025-02-25 18:42:07 -- STEP: 25/72 -- GLOBAL_STEP: 1073250 | > loss_disc: 2.559068202972412 (2.5006770133972167) | > loss_disc_real_0: 0.06216350570321083 (0.05895176909863949) | > loss_disc_real_1: 0.19728310406208038 (0.20351514101028442) | > loss_disc_real_2: 0.21626394987106323 (0.23921931266784668) | > loss_disc_real_3: 0.17718103528022766 (0.23405755519866944) | > loss_disc_real_4: 0.22518260776996613 (0.24615055978298186) | > loss_disc_real_5: 0.26969197392463684 (0.2410221928358078) | > loss_0: 2.559068202972412 (2.5006770133972167) | > grad_norm_0: tensor(199.5076, device='cuda:0') (tensor(161.4630, device='cuda:0')) | > loss_gen: 2.168415069580078 (2.2718046426773073) | > loss_kl: 2.4584145545959473 (2.1510601282119755) | > loss_feat: 4.255825996398926 (4.388178606033325) | > loss_mel: 17.95481300354004 (18.13609085083008) | > loss_duration: 1.525866985321045 (1.5293098258972169) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.36333656311035 (28.47644432067871) | > grad_norm_1: tensor(4663.4546, device='cuda:0') (tensor(6414.3882, device='cuda:0')) | > current_lr_0: 0.00017612331807671352 | > current_lr_1: 0.00017612331807671352 | > step_time: 0.4232 (0.4308489227294922) | > loader_time: 0.0071 (0.007156848907470703) --> EVAL PERFORMANCE | > avg_loader_time: 0.00868779169002049 (-5.3654254322325554e-05) | > avg_step_time: 0.447457192649304 (-0.0026367885965696836) | > avg_loss_disc: 2.506357025092756 (-0.13071286510413893) | > avg_loss_disc_real_0: 0.07991553845644839 (-0.06985628701956334) | > avg_loss_disc_real_1: 0.21430036446578066 (-0.025047321554640706) | > avg_loss_disc_real_2: 0.23407977588579687 (-0.008433261807535758) | > avg_loss_disc_real_3: 0.23062125939718434 (-0.008946966747163054) | > avg_loss_disc_real_4: 0.24188629905102957 (+0.0025839169680232643) | > avg_loss_disc_real_5: 0.24524830638522832 (+0.00048666310981965166) | > avg_loss_0: 2.506357025092756 (-0.13071286510413893) | > avg_grad_norm_0: tensor(204.9848, device='cuda:0') (tensor(-49.1661, device='cuda:0')) | > avg_loss_gen: 2.269818630017025 (+0.06508330895867198) | > avg_loss_kl: 2.1355892587715473 (-0.3139080531160596) | > avg_loss_feat: 4.127265507066753 (+0.25491884056950953) | > avg_loss_mel: 17.483104087937043 (-1.5659889704744536) | > avg_loss_duration: 1.564044923849509 (-0.042685428135831316) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.57982267460353 (-1.6025800301995048) | > avg_grad_norm_1: tensor(5856.8081, device='cuda:0') (+tensor(534.2085, device='cuda:0'))  > EPOCH: 1018/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:42:31)   --> TIME: 2025-02-25 18:42:34 -- STEP: 3/72 -- GLOBAL_STEP: 1073300 | > loss_disc: 2.4337992668151855 (2.4886900583902993) | > loss_disc_real_0: 0.03857288137078285 (0.05541841064890226) | > loss_disc_real_1: 0.17231492698192596 (0.21520825227101645) | > loss_disc_real_2: 0.19163064658641815 (0.21720277269681296) | > loss_disc_real_3: 0.21568253636360168 (0.22183778882026672) | > loss_disc_real_4: 0.23065903782844543 (0.23754185934861502) | > loss_disc_real_5: 0.23802690207958221 (0.242738276720047) | > loss_0: 2.4337992668151855 (2.4886900583902993) | > grad_norm_0: tensor(149.3568, device='cuda:0') (tensor(170.3150, device='cuda:0')) | > loss_gen: 2.117673397064209 (2.1998093922932944) | > loss_kl: 1.9641278982162476 (1.9515165090560913) | > loss_feat: 4.393929481506348 (4.302260239919026) | > loss_mel: 17.32091522216797 (17.303494771321613) | > loss_duration: 1.5307543277740479 (1.5167001088460286) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.32740020751953 (27.273781458536785) | > grad_norm_1: tensor(4974.1577, device='cuda:0') (tensor(4154.0400, device='cuda:0')) | > current_lr_0: 0.00017610130266195392 | > current_lr_1: 0.00017610130266195392 | > step_time: 0.4212 (0.4220872720082601) | > loader_time: 0.0062 (0.005926450093587239)  --> TIME: 2025-02-25 18:42:57 -- STEP: 53/72 -- GLOBAL_STEP: 1073350 | > loss_disc: 2.494891405105591 (2.5737246522363626) | > loss_disc_real_0: 0.050614308565855026 (0.10564672363535413) | > loss_disc_real_1: 0.1483752578496933 (0.24481317569624703) | > loss_disc_real_2: 0.23377454280853271 (0.2307215709731264) | > loss_disc_real_3: 0.19935713708400726 (0.22052188695601696) | > loss_disc_real_4: 0.21666856110095978 (0.2325626549855718) | > loss_disc_real_5: 0.23841840028762817 (0.2412671005388476) | > loss_0: 2.494891405105591 (2.5737246522363626) | > grad_norm_0: tensor(193.6318, device='cuda:0') (tensor(255.0633, device='cuda:0')) | > loss_gen: 2.148052215576172 (2.1831880582953396) | > loss_kl: 2.094327211380005 (2.0262311089713627) | > loss_feat: 4.295648097991943 (3.9183141555426255) | > loss_mel: 16.761926651000977 (16.866288329070464) | > loss_duration: 1.5856459140777588 (1.533936304866143) | > amp_scaler: 8.0 (8.301886792452832) | > loss_1: 26.88559913635254 (26.527957916259766) | > grad_norm_1: tensor(5885.7534, device='cuda:0') (tensor(6692.9453, device='cuda:0')) | > current_lr_0: 0.00017610130266195392 | > current_lr_1: 0.00017610130266195392 | > step_time: 0.4533 (0.4470622674474176) | > loader_time: 0.009 (0.0077435880337121355) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853591233911649 (-0.0001518793509040009) | > avg_step_time: 0.45626748447686855 (+0.008810291827564543) | > avg_loss_disc: 2.6114851320293586 (+0.1051281069366028) | > avg_loss_disc_real_0: 0.12707908970999046 (+0.047163551253542074) | > avg_loss_disc_real_1: 0.23751966277478445 (+0.023219298309003794) | > avg_loss_disc_real_2: 0.23421570763621533 (+0.0001359317504184554) | > avg_loss_disc_real_3: 0.2265350881596686 (-0.004086171237515729) | > avg_loss_disc_real_4: 0.2327233697746841 (-0.00916292927634546) | > avg_loss_disc_real_5: 0.24244603577633977 (-0.0028022706088885463) | > avg_loss_0: 2.6114851320293586 (+0.1051281069366028) | > avg_grad_norm_0: tensor(276.4076, device='cuda:0') (+tensor(71.4229, device='cuda:0')) | > avg_loss_gen: 2.1639442494217778 (-0.10587438059524734) | > avg_loss_kl: 2.092597749871268 (-0.042991508900279474) | > avg_loss_feat: 3.8710409755438144 (-0.256224531522939) | > avg_loss_mel: 17.049877113019917 (-0.43322697491712603) | > avg_loss_duration: 1.552995031988117 (-0.011049891861392158) | > avg_amp_scaler: 8.225352112676056 (+0.22535211267605604) | > avg_loss_1: 26.730455022462664 (-0.8493676521408666) | > avg_grad_norm_1: tensor(6296.0498, device='cuda:0') (+tensor(439.2417, device='cuda:0'))  > EPOCH: 1019/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:43:08)   --> TIME: 2025-02-25 18:43:24 -- STEP: 31/72 -- GLOBAL_STEP: 1073400 | > loss_disc: 2.5523409843444824 (2.5332645447023454) | > loss_disc_real_0: 0.09126774966716766 (0.10373398529425744) | > loss_disc_real_1: 0.22954235970973969 (0.22062350761505864) | > loss_disc_real_2: 0.18376663327217102 (0.23262131502551417) | > loss_disc_real_3: 0.24362030625343323 (0.2338595587399698) | > loss_disc_real_4: 0.2608116865158081 (0.2404261423695472) | > loss_disc_real_5: 0.2520086169242859 (0.24235343115945016) | > loss_0: 2.5523409843444824 (2.5332645447023454) | > grad_norm_0: tensor(59.0634, device='cuda:0') (tensor(192.3216, device='cuda:0')) | > loss_gen: 2.319775104522705 (2.256503570464349) | > loss_kl: 2.1240110397338867 (2.058444219250833) | > loss_feat: 4.518805027008057 (4.0324636044040805) | > loss_mel: 17.336095809936523 (17.411224365234375) | > loss_duration: 1.5522446632385254 (1.5191132445489206) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.85093116760254 (27.277748784711285) | > grad_norm_1: tensor(2763.6284, device='cuda:0') (tensor(4954.0439, device='cuda:0')) | > current_lr_0: 0.00017607928999912117 | > current_lr_1: 0.00017607928999912117 | > step_time: 0.4373 (0.43469161372030934) | > loader_time: 0.0086 (0.007205770861717963) --> EVAL PERFORMANCE | > avg_loader_time: 0.008634641136921624 (+9.872879780513377e-05) | > avg_step_time: 0.4519083332008039 (-0.004359151276064621) | > avg_loss_disc: 2.650272987258266 (+0.038787855228907375) | > avg_loss_disc_real_0: 0.1598608369978381 (+0.032781747287847635) | > avg_loss_disc_real_1: 0.2564483417171827 (+0.018928678942398275) | > avg_loss_disc_real_2: 0.2319259740097422 (-0.0022897336264731394) | > avg_loss_disc_real_3: 0.23213766746118036 (+0.005602579301511756) | > avg_loss_disc_real_4: 0.23847108680597492 (+0.005747717031290817) | > avg_loss_disc_real_5: 0.24844964646117787 (+0.0060036106848380955) | > avg_loss_0: 2.650272987258266 (+0.038787855228907375) | > avg_grad_norm_0: tensor(243.1361, device='cuda:0') (tensor(-33.2715, device='cuda:0')) | > avg_loss_gen: 2.200520868032751 (+0.03657661861097328) | > avg_loss_kl: 2.065002276863851 (-0.027595473007416604) | > avg_loss_feat: 3.703543417890307 (-0.16749755765350738) | > avg_loss_mel: 17.086580652586164 (+0.036703539566246945) | > avg_loss_duration: 1.5549614395893796 (+0.001966407601262654) | > avg_amp_scaler: 8.0 (-0.22535211267605604) | > avg_loss_1: 26.61060854414819 (-0.11984647831447504) | > avg_grad_norm_1: tensor(5595.0483, device='cuda:0') (tensor(-701.0015, device='cuda:0'))  > EPOCH: 1020/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:43:45)   --> TIME: 2025-02-25 18:43:50 -- STEP: 9/72 -- GLOBAL_STEP: 1073450 | > loss_disc: 2.8876163959503174 (2.542606380250719) | > loss_disc_real_0: 0.03534523397684097 (0.1078610469897588) | > loss_disc_real_1: 0.22344006597995758 (0.22766515612602234) | > loss_disc_real_2: 0.2086169570684433 (0.24875453611214957) | > loss_disc_real_3: 0.19390009343624115 (0.2382259021202723) | > loss_disc_real_4: 0.23171326518058777 (0.23905481232537162) | > loss_disc_real_5: 0.22779196500778198 (0.23357987735006544) | > loss_0: 2.8876163959503174 (2.542606380250719) | > grad_norm_0: tensor(164.8548, device='cuda:0') (tensor(198.9272, device='cuda:0')) | > loss_gen: 1.6215482950210571 (2.2195014688703747) | > loss_kl: 2.15139102935791 (2.240577883190579) | > loss_feat: 3.690840721130371 (3.9964759084913464) | > loss_mel: 18.265350341796875 (18.29538959927029) | > loss_duration: 1.5374308824539185 (1.563550975587633) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.26656150817871 (28.315496020846897) | > grad_norm_1: tensor(10274.9248, device='cuda:0') (tensor(6122.2026, device='cuda:0')) | > current_lr_0: 0.00017605728008787127 | > current_lr_1: 0.00017605728008787127 | > step_time: 0.4194 (0.42423121134440106) | > loader_time: 0.0055 (0.006153027216593425)  --> TIME: 2025-02-25 18:44:13 -- STEP: 59/72 -- GLOBAL_STEP: 1073500 | > loss_disc: 2.5145442485809326 (2.5057115797269134) | > loss_disc_real_0: 0.12272170186042786 (0.06800163234189406) | > loss_disc_real_1: 0.22622400522232056 (0.2198518855591952) | > loss_disc_real_2: 0.25531983375549316 (0.23580227665982004) | > loss_disc_real_3: 0.23092865943908691 (0.23445631109051784) | > loss_disc_real_4: 0.2545854449272156 (0.24125438095149349) | > loss_disc_real_5: 0.24989326298236847 (0.24178618849334071) | > loss_0: 2.5145442485809326 (2.5057115797269134) | > grad_norm_0: tensor(207.8227, device='cuda:0') (tensor(182.2799, device='cuda:0')) | > loss_gen: 2.3304085731506348 (2.221656294192299) | > loss_kl: 2.074307441711426 (2.0783506874310764) | > loss_feat: 3.480278253555298 (4.06692413960473) | > loss_mel: 14.446107864379883 (17.150270801479532) | > loss_duration: 1.5753343105316162 (1.5503592450739974) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.906436920166016 (27.067560874809654) | > grad_norm_1: tensor(3063.0559, device='cuda:0') (tensor(6790.5234, device='cuda:0')) | > current_lr_0: 0.00017605728008787127 | > current_lr_1: 0.00017605728008787127 | > step_time: 0.4648 (0.43921917980000125) | > loader_time: 0.0097 (0.008134167073136668) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869005834552604 (+5.541720860441704e-05) | > avg_step_time: 0.44870972633361816 (-0.0031986068671857604) | > avg_loss_disc: 2.5035804661226937 (-0.1466925211355723) | > avg_loss_disc_real_0: 0.0668555182260527 (-0.0930053187717854) | > avg_loss_disc_real_1: 0.2181723762146184 (-0.03827596550256432) | > avg_loss_disc_real_2: 0.2345532970109456 (+0.0026273230012034132) | > avg_loss_disc_real_3: 0.23225480485969865 (+0.00011713739851829108) | > avg_loss_disc_real_4: 0.2397713602428705 (+0.001300273436895566) | > avg_loss_disc_real_5: 0.2421084873273339 (-0.0063411591338439766) | > avg_loss_0: 2.5035804661226937 (-0.1466925211355723) | > avg_grad_norm_0: tensor(184.8421, device='cuda:0') (tensor(-58.2940, device='cuda:0')) | > avg_loss_gen: 2.2224338726258623 (+0.021913004593111296) | > avg_loss_kl: 2.0899169629728287 (+0.024914686108977513) | > avg_loss_feat: 4.065674694491104 (+0.36213127660079714) | > avg_loss_mel: 17.111354895041018 (+0.02477424245485338) | > avg_loss_duration: 1.559145447234033 (+0.00418400764465332) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.04852555503308 (+0.43791701088489177) | > avg_grad_norm_1: tensor(6951.3999, device='cuda:0') (+tensor(1356.3516, device='cuda:0'))  > EPOCH: 1021/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:44:21)   --> TIME: 2025-02-25 18:44:40 -- STEP: 37/72 -- GLOBAL_STEP: 1073550 | > loss_disc: 2.760272979736328 (2.4958909524453654) | > loss_disc_real_0: 0.053946152329444885 (0.05867464342028708) | > loss_disc_real_1: 0.20590992271900177 (0.2231610534158913) | > loss_disc_real_2: 0.24661439657211304 (0.22988756363456314) | > loss_disc_real_3: 0.2417508214712143 (0.23231139698544065) | > loss_disc_real_4: 0.2475959211587906 (0.24090900171447444) | > loss_disc_real_5: 0.25176048278808594 (0.24395910590081601) | > loss_0: 2.760272979736328 (2.4958909524453654) | > grad_norm_0: tensor(297.7910, device='cuda:0') (tensor(198.2688, device='cuda:0')) | > loss_gen: 1.9242830276489258 (2.2585221496788224) | > loss_kl: 1.8549880981445312 (1.971945601540643) | > loss_feat: 3.885910987854004 (4.220341991733861) | > loss_mel: 16.648731231689453 (16.931539870597224) | > loss_duration: 1.5403982400894165 (1.507268325702564) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.85430908203125 (26.88961797147184) | > grad_norm_1: tensor(12467.7129, device='cuda:0') (tensor(7686.0820, device='cuda:0')) | > current_lr_0: 0.00017603527292786028 | > current_lr_1: 0.00017603527292786028 | > step_time: 0.4413 (0.4463865821425979) | > loader_time: 0.008 (0.007057621672346785) --> EVAL PERFORMANCE | > avg_loader_time: 0.008464944194739973 (-0.00022511415078606788) | > avg_step_time: 0.4558663502545424 (+0.007156623920924254) | > avg_loss_disc: 2.577909835627381 (+0.07432936950468738) | > avg_loss_disc_real_0: 0.098227186180966 (+0.03137166795491331) | > avg_loss_disc_real_1: 0.21944094743107406 (+0.001268571216455655) | > avg_loss_disc_real_2: 0.22866781076914827 (-0.0058854862417973275) | > avg_loss_disc_real_3: 0.23535230399017604 (+0.0030974991304773825) | > avg_loss_disc_real_4: 0.2407627827684644 (+0.0009914225255939213) | > avg_loss_disc_real_5: 0.24721656173047885 (+0.005108074403144952) | > avg_loss_0: 2.577909835627381 (+0.07432936950468738) | > avg_grad_norm_0: tensor(212.4234, device='cuda:0') (+tensor(27.5813, device='cuda:0')) | > avg_loss_gen: 2.2152836608215107 (-0.007150211804351603) | > avg_loss_kl: 2.0081656130266863 (-0.08175134994614242) | > avg_loss_feat: 3.9841909307829093 (-0.08148376370819488) | > avg_loss_mel: 16.731456380494883 (-0.37989851454613444) | > avg_loss_duration: 1.5434787592417758 (-0.015666687992257122) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.48257540313291 (-0.5659501519001715) | > avg_grad_norm_1: tensor(6620.2310, device='cuda:0') (tensor(-331.1689, device='cuda:0'))  > EPOCH: 1022/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:44:58)   --> TIME: 2025-02-25 18:45:06 -- STEP: 15/72 -- GLOBAL_STEP: 1073600 | > loss_disc: 2.6770517826080322 (2.575183550516764) | > loss_disc_real_0: 0.1573083996772766 (0.13130675964057445) | > loss_disc_real_1: 0.22291207313537598 (0.21200262606143952) | > loss_disc_real_2: 0.260590523481369 (0.23061013221740723) | > loss_disc_real_3: 0.2189973145723343 (0.2298486739397049) | > loss_disc_real_4: 0.27297598123550415 (0.2329390654961268) | > loss_disc_real_5: 0.2682145833969116 (0.24363329112529755) | > loss_0: 2.6770517826080322 (2.575183550516764) | > grad_norm_0: tensor(364.6219, device='cuda:0') (tensor(183.8766, device='cuda:0')) | > loss_gen: 2.123534917831421 (2.2709492286046347) | > loss_kl: 2.030489683151245 (1.9399048328399657) | > loss_feat: 3.674952745437622 (3.9978844324747715) | > loss_mel: 16.6119441986084 (16.866128094991048) | > loss_duration: 1.5051528215408325 (1.5016363859176636) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.946075439453125 (26.576503245035806) | > grad_norm_1: tensor(1799.6285, device='cuda:0') (tensor(4933.0415, device='cuda:0')) | > current_lr_0: 0.00017601326851874428 | > current_lr_1: 0.00017601326851874428 | > step_time: 0.4274 (0.4307162920633952) | > loader_time: 0.0058 (0.006774044036865235)  --> TIME: 2025-02-25 18:45:30 -- STEP: 65/72 -- GLOBAL_STEP: 1073650 | > loss_disc: 2.5395395755767822 (2.6406909722548266) | > loss_disc_real_0: 0.22747357189655304 (0.1444288090731089) | > loss_disc_real_1: 0.25172320008277893 (0.2318850171107512) | > loss_disc_real_2: 0.2528904974460602 (0.22626096938665097) | > loss_disc_real_3: 0.23031188547611237 (0.2279017164156987) | > loss_disc_real_4: 0.2627183794975281 (0.23244156058018023) | > loss_disc_real_5: 0.22560198605060577 (0.24211544600816873) | > loss_0: 2.5395395755767822 (2.6406909722548266) | > grad_norm_0: tensor(413.1731, device='cuda:0') (tensor(229.3279, device='cuda:0')) | > loss_gen: 2.4464354515075684 (2.1881217883183397) | > loss_kl: 2.2501461505889893 (2.115849507772005) | > loss_feat: 3.816215753555298 (3.88310988499568) | > loss_mel: 16.475622177124023 (17.064876497708834) | > loss_duration: 1.6152117252349854 (1.5488429674735436) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.6036319732666 (26.80080073429988) | > grad_norm_1: tensor(3264.5713, device='cuda:0') (tensor(5406.6689, device='cuda:0')) | > current_lr_0: 0.00017601326851874428 | > current_lr_1: 0.00017601326851874428 | > step_time: 0.5111 (0.44646504108722396) | > loader_time: 0.0115 (0.00832665149982159) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584348248763822 (+0.00011940405402384967) | > avg_step_time: 0.4515423035957444 (-0.004324046658798042) | > avg_loss_disc: 2.637304920545766 (+0.05939508491838508) | > avg_loss_disc_real_0: 0.13965619148188074 (+0.04142900530091474) | > avg_loss_disc_real_1: 0.23555237789389114 (+0.016111430462817072) | > avg_loss_disc_real_2: 0.2287402410112636 (+7.243024211534e-05) | > avg_loss_disc_real_3: 0.22676022875476892 (-0.008592075235407115) | > avg_loss_disc_real_4: 0.23501098827576974 (-0.005751794492694667) | > avg_loss_disc_real_5: 0.2415846158081377 (-0.005631945922341147) | > avg_loss_0: 2.637304920545766 (+0.05939508491838508) | > avg_grad_norm_0: tensor(233.1672, device='cuda:0') (+tensor(20.7438, device='cuda:0')) | > avg_loss_gen: 2.17616667546017 (-0.039116985361340895) | > avg_loss_kl: 2.119509342690588 (+0.11134372966390194) | > avg_loss_feat: 3.848626516234707 (-0.13556441454820245) | > avg_loss_mel: 17.035056449997594 (+0.30360006950271057) | > avg_loss_duration: 1.5541789195906948 (+0.010700160348918963) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.733538023183044 (+0.25096262005013514) | > avg_grad_norm_1: tensor(5482.3115, device='cuda:0') (tensor(-1137.9194, device='cuda:0'))  > EPOCH: 1023/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:45:35)   --> TIME: 2025-02-25 18:45:56 -- STEP: 43/72 -- GLOBAL_STEP: 1073700 | > loss_disc: 2.767726182937622 (2.6644827875980113) | > loss_disc_real_0: 0.1736448109149933 (0.14301168351152607) | > loss_disc_real_1: 0.21464599668979645 (0.24497420947219045) | > loss_disc_real_2: 0.274116575717926 (0.23345537206461264) | > loss_disc_real_3: 0.2662292718887329 (0.24319405056709467) | > loss_disc_real_4: 0.27265191078186035 (0.24897984467273526) | > loss_disc_real_5: 0.24117118120193481 (0.24858615426130065) | > loss_0: 2.767726182937622 (2.6644827875980113) | > grad_norm_0: tensor(108.8533, device='cuda:0') (tensor(239.1208, device='cuda:0')) | > loss_gen: 1.8281983137130737 (2.113229432771372) | > loss_kl: 2.0731873512268066 (2.215864331223244) | > loss_feat: 2.8840224742889404 (3.7112985267195593) | > loss_mel: 16.45844841003418 (17.60464756987816) | > loss_duration: 1.5804102420806885 (1.6227160026860792) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.824264526367188 (27.267755419708962) | > grad_norm_1: tensor(2226.9353, device='cuda:0') (tensor(5818.1177, device='cuda:0')) | > current_lr_0: 0.00017599126686017942 | > current_lr_1: 0.00017599126686017942 | > step_time: 0.4498 (0.43571449435034465) | > loader_time: 0.0098 (0.007571026336315067) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630084319853447 (+4.573607108962485e-05) | > avg_step_time: 0.45053627121616424 (-0.0010060323795801351) | > avg_loss_disc: 2.6638517312600576 (+0.026546810714291436) | > avg_loss_disc_real_0: 0.14161395766890386 (+0.001957766187023119) | > avg_loss_disc_real_1: 0.23357988558184933 (-0.0019724923120418014) | > avg_loss_disc_real_2: 0.23764486191138415 (+0.008904620900120536) | > avg_loss_disc_real_3: 0.2423637223915315 (+0.015603493636762583) | > avg_loss_disc_real_4: 0.24945464436437043 (+0.014443656088600682) | > avg_loss_disc_real_5: 0.2497515055075497 (+0.008166889699412011) | > avg_loss_0: 2.6638517312600576 (+0.026546810714291436) | > avg_grad_norm_0: tensor(242.4595, device='cuda:0') (+tensor(9.2923, device='cuda:0')) | > avg_loss_gen: 2.09200982476624 (-0.08415685069392964) | > avg_loss_kl: 2.168250371033038 (+0.048741028342449866) | > avg_loss_feat: 3.6353184975368875 (-0.21330801869781935) | > avg_loss_mel: 17.26393930676957 (+0.22888285677197473) | > avg_loss_duration: 1.6142868559125443 (+0.06010793632184952) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.77380462431572 (+0.040266601132675106) | > avg_grad_norm_1: tensor(5950.2070, device='cuda:0') (+tensor(467.8955, device='cuda:0'))  > EPOCH: 1024/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:46:11)   --> TIME: 2025-02-25 18:46:23 -- STEP: 21/72 -- GLOBAL_STEP: 1073750 | > loss_disc: 2.555781602859497 (2.6392534460340227) | > loss_disc_real_0: 0.14547514915466309 (0.146149928016322) | > loss_disc_real_1: 0.18885895609855652 (0.2132591697431746) | > loss_disc_real_2: 0.2144625335931778 (0.23915356184755052) | > loss_disc_real_3: 0.22841663658618927 (0.231852141164598) | > loss_disc_real_4: 0.23260341584682465 (0.24543107407433645) | > loss_disc_real_5: 0.2541024684906006 (0.24864315773759568) | > loss_0: 2.555781602859497 (2.6392534460340227) | > grad_norm_0: tensor(102.2853, device='cuda:0') (tensor(236.0468, device='cuda:0')) | > loss_gen: 1.9735643863677979 (2.0472360452016196) | > loss_kl: 1.911742091178894 (2.0269276301066075) | > loss_feat: 3.521116018295288 (3.587810198465983) | > loss_mel: 17.282487869262695 (17.24124104636056) | > loss_duration: 1.518863558769226 (1.5156159173874628) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.207773208618164 (26.418830599103654) | > grad_norm_1: tensor(11786.1875, device='cuda:0') (tensor(9011.3662, device='cuda:0')) | > current_lr_0: 0.00017596926795182189 | > current_lr_1: 0.00017596926795182189 | > step_time: 0.4255 (0.45130689938863117) | > loader_time: 0.0061 (0.006697336832682292)  --> TIME: 2025-02-25 18:46:47 -- STEP: 71/72 -- GLOBAL_STEP: 1073800 | > loss_disc: 2.627493381500244 (2.6381292108079077) | > loss_disc_real_0: 0.13264718651771545 (0.15680801616588108) | > loss_disc_real_1: 0.22601699829101562 (0.2140113113211914) | > loss_disc_real_2: 0.2218814492225647 (0.23801852951587085) | > loss_disc_real_3: 0.22152942419052124 (0.23348473432198377) | > loss_disc_real_4: 0.2686574459075928 (0.24485989208792297) | > loss_disc_real_5: 0.2699092626571655 (0.24746824486154906) | > loss_0: 2.627493381500244 (2.6381292108079077) | > grad_norm_0: tensor(426.6560, device='cuda:0') (tensor(291.3699, device='cuda:0')) | > loss_gen: 2.095062017440796 (2.0487642741539096) | > loss_kl: 2.0408735275268555 (2.0551337759259716) | > loss_feat: 3.379498243331909 (3.5383162901435106) | > loss_mel: 17.203754425048828 (16.88448825352628) | > loss_duration: 1.5569978952407837 (1.5506462547141062) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.276185989379883 (26.077348789698647) | > grad_norm_1: tensor(12560.2588, device='cuda:0') (tensor(9943.7236, device='cuda:0')) | > current_lr_0: 0.00017596926795182189 | > current_lr_1: 0.00017596926795182189 | > step_time: 0.4997 (0.4582857615511182) | > loader_time: 0.0085 (0.0085566178174086) --> EVAL PERFORMANCE | > avg_loader_time: 0.0085566178174086 (-7.346650244484734e-05) | > avg_step_time: 0.4582857615511182 (+0.007749490334953979) | > avg_loss_disc: 2.6381292108079077 (-0.025722520452149844) | > avg_loss_disc_real_0: 0.15680801616588108 (+0.01519405849697722) | > avg_loss_disc_real_1: 0.2140113113211914 (-0.019568574260657945) | > avg_loss_disc_real_2: 0.23801852951587085 (+0.0003736676044867049) | > avg_loss_disc_real_3: 0.23348473432198377 (-0.008878988069547733) | > avg_loss_disc_real_4: 0.24485989208792297 (-0.004594752276447456) | > avg_loss_disc_real_5: 0.24746824486154906 (-0.002283260646000651) | > avg_loss_0: 2.6381292108079077 (-0.025722520452149844) | > avg_grad_norm_0: tensor(291.3699, device='cuda:0') (+tensor(48.9104, device='cuda:0')) | > avg_loss_gen: 2.0487642741539096 (-0.0432455506123306) | > avg_loss_kl: 2.0551337759259716 (-0.11311659510706651) | > avg_loss_feat: 3.5383162901435106 (-0.09700220739337695) | > avg_loss_mel: 16.88448825352628 (-0.3794510532432902) | > avg_loss_duration: 1.5506462547141062 (-0.06364060119843806) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.077348789698647 (-0.6964558346170726) | > avg_grad_norm_1: tensor(9943.7236, device='cuda:0') (+tensor(3993.5166, device='cuda:0'))  > EPOCH: 1025/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:46:48)   --> TIME: 2025-02-25 18:47:12 -- STEP: 49/72 -- GLOBAL_STEP: 1073850 | > loss_disc: 2.524714469909668 (2.5902665245289698) | > loss_disc_real_0: 0.06231971085071564 (0.11877730534392957) | > loss_disc_real_1: 0.2301328331232071 (0.24037102473025418) | > loss_disc_real_2: 0.21836796402931213 (0.23463334599319768) | > loss_disc_real_3: 0.19270752370357513 (0.22835699088719427) | > loss_disc_real_4: 0.1882341057062149 (0.2459652408653376) | > loss_disc_real_5: 0.20396870374679565 (0.2460266981197863) | > loss_0: 2.524714469909668 (2.5902665245289698) | > grad_norm_0: tensor(242.9180, device='cuda:0') (tensor(266.3436, device='cuda:0')) | > loss_gen: 2.081116199493408 (2.2111167494131583) | > loss_kl: 2.188474178314209 (2.132155350276403) | > loss_feat: 4.194934844970703 (3.947201903985471) | > loss_mel: 16.608348846435547 (17.036177751969312) | > loss_duration: 1.6017839908599854 (1.5378031195426474) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.674657821655273 (26.86445493114238) | > grad_norm_1: tensor(9348.8057, device='cuda:0') (tensor(8000.3584, device='cuda:0')) | > current_lr_0: 0.0001759472717933279 | > current_lr_1: 0.0001759472717933279 | > step_time: 0.4397 (0.43823403241683023) | > loader_time: 0.0097 (0.0076655709013646964) --> EVAL PERFORMANCE | > avg_loader_time: 0.008537487244941818 (-1.913057246678146e-05) | > avg_step_time: 0.4507406731726418 (-0.0075450883784763945) | > avg_loss_disc: 2.5703571144963644 (-0.06777209631154335) | > avg_loss_disc_real_0: 0.11087322040972573 (-0.04593479575615535) | > avg_loss_disc_real_1: 0.23769397152141786 (+0.02368266020022647) | > avg_loss_disc_real_2: 0.23510170612536685 (-0.002916823390504003) | > avg_loss_disc_real_3: 0.22884900616088383 (-0.004635728161099945) | > avg_loss_disc_real_4: 0.2430296070139173 (-0.0018302850740056775) | > avg_loss_disc_real_5: 0.24457134915069795 (-0.0028968957108511106) | > avg_loss_0: 2.5703571144963644 (-0.06777209631154335) | > avg_grad_norm_0: tensor(274.3536, device='cuda:0') (tensor(-17.0163, device='cuda:0')) | > avg_loss_gen: 2.2021085829801956 (+0.15334430882628602) | > avg_loss_kl: 2.1392091059348957 (+0.08407533000892409) | > avg_loss_feat: 3.917707923432471 (+0.3793916332889604) | > avg_loss_mel: 16.92187927138637 (+0.03739101786009158) | > avg_loss_duration: 1.5538403870354236 (+0.003194132321317378) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.734745294275417 (+0.65739650457677) | > avg_grad_norm_1: tensor(8592.2295, device='cuda:0') (tensor(-1351.4941, device='cuda:0'))  > EPOCH: 1026/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:47:25)   --> TIME: 2025-02-25 18:47:39 -- STEP: 27/72 -- GLOBAL_STEP: 1073900 | > loss_disc: 2.413522720336914 (2.510457153673525) | > loss_disc_real_0: 0.0940513089299202 (0.0805809012165776) | > loss_disc_real_1: 0.19901800155639648 (0.21695012737203528) | > loss_disc_real_2: 0.23933269083499908 (0.22339760318950372) | > loss_disc_real_3: 0.2329692542552948 (0.23151016180162076) | > loss_disc_real_4: 0.2233746349811554 (0.23787164743299838) | > loss_disc_real_5: 0.23160645365715027 (0.23811850889965339) | > loss_0: 2.413522720336914 (2.510457153673525) | > grad_norm_0: tensor(171.1358, device='cuda:0') (tensor(272.6023, device='cuda:0')) | > loss_gen: 2.28633189201355 (2.1954727393609508) | > loss_kl: 2.110167980194092 (1.9555683930714924) | > loss_feat: 3.938663959503174 (4.033501183545147) | > loss_mel: 16.465662002563477 (16.71070021170157) | > loss_duration: 1.5100712776184082 (1.497993367689627) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.310895919799805 (26.393236089635778) | > grad_norm_1: tensor(5671.5469, device='cuda:0') (tensor(9132.3730, device='cuda:0')) | > current_lr_0: 0.00017592527838435372 | > current_lr_1: 0.00017592527838435372 | > step_time: 0.4325 (0.4304068088531494) | > loader_time: 0.0077 (0.0068303655695032195) --> EVAL PERFORMANCE | > avg_loader_time: 0.008750559578479174 (+0.00021307233353735563) | > avg_step_time: 0.4499383375678264 (-0.0008023356048154073) | > avg_loss_disc: 2.5177111995052286 (-0.0526459149911358) | > avg_loss_disc_real_0: 0.08632432244403263 (-0.02454889796569311) | > avg_loss_disc_real_1: 0.22085924719420957 (-0.016834724327208284) | > avg_loss_disc_real_2: 0.2287629374735792 (-0.006338768651787646) | > avg_loss_disc_real_3: 0.22982499675011972 (+0.0009759905892358922) | > avg_loss_disc_real_4: 0.2412417123855 (-0.0017878946284173025) | > avg_loss_disc_real_5: 0.24453711152916224 (-3.423762153570431e-05) | > avg_loss_0: 2.5177111995052286 (-0.0526459149911358) | > avg_grad_norm_0: tensor(288.3623, device='cuda:0') (+tensor(14.0087, device='cuda:0')) | > avg_loss_gen: 2.2181430954328727 (+0.016034512452677063) | > avg_loss_kl: 2.0317349534639186 (-0.10747415247097702) | > avg_loss_feat: 3.975238487754068 (+0.057530564321596867) | > avg_loss_mel: 16.552426566540355 (-0.3694527048460152) | > avg_loss_duration: 1.540276030419578 (-0.013564356615845652) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.317819299832195 (-0.416925994443222) | > avg_grad_norm_1: tensor(9493.3418, device='cuda:0') (+tensor(901.1123, device='cuda:0'))  > EPOCH: 1027/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:48:01)   --> TIME: 2025-02-25 18:48:06 -- STEP: 5/72 -- GLOBAL_STEP: 1073950 | > loss_disc: 2.4209375381469727 (2.494585704803467) | > loss_disc_real_0: 0.08619889616966248 (0.08401874378323555) | > loss_disc_real_1: 0.240686297416687 (0.2187587320804596) | > loss_disc_real_2: 0.2080615609884262 (0.21883427500724792) | > loss_disc_real_3: 0.20645011961460114 (0.21578296422958373) | > loss_disc_real_4: 0.20983700454235077 (0.22687699198722838) | > loss_disc_real_5: 0.21473263204097748 (0.2257472336292267) | > loss_0: 2.4209375381469727 (2.494585704803467) | > grad_norm_0: tensor(190.9590, device='cuda:0') (tensor(322.6779, device='cuda:0')) | > loss_gen: 2.0826566219329834 (2.203274297714233) | > loss_kl: 1.8353511095046997 (2.0207247018814085) | > loss_feat: 3.879080057144165 (4.085855627059937) | > loss_mel: 16.5980167388916 (16.653850555419922) | > loss_duration: 1.4878525733947754 (1.5021540641784668) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.882957458496094 (26.46585884094238) | > grad_norm_1: tensor(10797.3887, device='cuda:0') (tensor(10430.9873, device='cuda:0')) | > current_lr_0: 0.00017590328772455567 | > current_lr_1: 0.00017590328772455567 | > step_time: 0.4179 (0.4254759311676025) | > loader_time: 0.0064 (0.006869649887084961)  --> TIME: 2025-02-25 18:48:29 -- STEP: 55/72 -- GLOBAL_STEP: 1074000 | > loss_disc: 2.6351380348205566 (2.5251829494129527) | > loss_disc_real_0: 0.06771905720233917 (0.08870417468927123) | > loss_disc_real_1: 0.19301113486289978 (0.2191594356840307) | > loss_disc_real_2: 0.24956956505775452 (0.23082063523205845) | > loss_disc_real_3: 0.22128182649612427 (0.23209981566125695) | > loss_disc_real_4: 0.2367812544107437 (0.24104885079643942) | > loss_disc_real_5: 0.2553277015686035 (0.2441809044642882) | > loss_0: 2.6351380348205566 (2.5251829494129527) | > grad_norm_0: tensor(364.3795, device='cuda:0') (tensor(310.7753, device='cuda:0')) | > loss_gen: 2.11924147605896 (2.2252538984472094) | > loss_kl: 2.1009740829467773 (2.00054688020186) | > loss_feat: 4.336321830749512 (4.021582798524336) | > loss_mel: 17.418663024902344 (16.4610918045044) | > loss_duration: 1.5850107669830322 (1.5238626371730457) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.560211181640625 (26.23233802101829) | > grad_norm_1: tensor(11873.6006, device='cuda:0') (tensor(9929.4951, device='cuda:0')) | > current_lr_0: 0.00017590328772455567 | > current_lr_1: 0.00017590328772455567 | > step_time: 0.4611 (0.44834549643776633) | > loader_time: 0.0091 (0.007736461812799627) --> EVAL PERFORMANCE | > avg_loader_time: 0.008480485056487607 (-0.00027007452199156665) | > avg_step_time: 0.45956765765875157 (+0.009629320090925153) | > avg_loss_disc: 2.5250247700113646 (+0.0073135705061360134) | > avg_loss_disc_real_0: 0.08952062272689712 (+0.003196300282864495) | > avg_loss_disc_real_1: 0.22154884397144048 (+0.0006895967772309053) | > avg_loss_disc_real_2: 0.23243428032163163 (+0.003671342848052428) | > avg_loss_disc_real_3: 0.23208006691764776 (+0.002255070167528045) | > avg_loss_disc_real_4: 0.24106966819561704 (-0.00017204418988295278) | > avg_loss_disc_real_5: 0.24508640191084902 (+0.0005492903816867734) | > avg_loss_0: 2.5250247700113646 (+0.0073135705061360134) | > avg_grad_norm_0: tensor(311.0042, device='cuda:0') (+tensor(22.6419, device='cuda:0')) | > avg_loss_gen: 2.2336169162266684 (+0.015473820793795667) | > avg_loss_kl: 2.0273866754182626 (-0.004348278045656073) | > avg_loss_feat: 4.00257813762611 (+0.027339649872042404) | > avg_loss_mel: 16.538738237300397 (-0.013688329239958108) | > avg_loss_duration: 1.5386260294578444 (-0.001650000961733511) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.340946090053503 (+0.02312679022130837) | > avg_grad_norm_1: tensor(10009.6094, device='cuda:0') (+tensor(516.2676, device='cuda:0'))  > EPOCH: 1028/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:48:39)   --> TIME: 2025-02-25 18:48:55 -- STEP: 33/72 -- GLOBAL_STEP: 1074050 | > loss_disc: 2.4689078330993652 (2.511095393787731) | > loss_disc_real_0: 0.1295846402645111 (0.0990863966219353) | > loss_disc_real_1: 0.21196313202381134 (0.21661808319164044) | > loss_disc_real_2: 0.21281681954860687 (0.22626333119291248) | > loss_disc_real_3: 0.2198742926120758 (0.22962550367369797) | > loss_disc_real_4: 0.2334052473306656 (0.24030795874017657) | > loss_disc_real_5: 0.2580327093601227 (0.24386279194644003) | > loss_0: 2.4689078330993652 (2.511095393787731) | > grad_norm_0: tensor(428.5667, device='cuda:0') (tensor(307.2849, device='cuda:0')) | > loss_gen: 2.41896915435791 (2.2618898586793383) | > loss_kl: 1.8526146411895752 (1.9400273164113362) | > loss_feat: 4.4446516036987305 (4.109614104935617) | > loss_mel: 16.298009872436523 (16.589860482649375) | > loss_duration: 1.5081820487976074 (1.496605356534322) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.52242660522461 (26.39799690246582) | > grad_norm_1: tensor(7078.0469, device='cuda:0') (tensor(9377.9434, device='cuda:0')) | > current_lr_0: 0.0001758812998135901 | > current_lr_1: 0.0001758812998135901 | > step_time: 0.45 (0.43647740826462256) | > loader_time: 0.0084 (0.007085019891912287) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547040778146662 (+6.655572165905482e-05) | > avg_step_time: 0.45494911032663266 (-0.004618547332118916) | > avg_loss_disc: 2.538526128715193 (+0.013501358703828181) | > avg_loss_disc_real_0: 0.09833731045815307 (+0.008816687731255948) | > avg_loss_disc_real_1: 0.22429306616245862 (+0.0027442221910181364) | > avg_loss_disc_real_2: 0.23090160648587724 (-0.0015326738357543945) | > avg_loss_disc_real_3: 0.23149516611871584 (-0.0005849007989319199) | > avg_loss_disc_real_4: 0.2408230292125487 (-0.0002466389830683424) | > avg_loss_disc_real_5: 0.2449459924664296 (-0.00014040944441942593) | > avg_loss_0: 2.538526128715193 (+0.013501358703828181) | > avg_grad_norm_0: tensor(301.7850, device='cuda:0') (tensor(-9.2192, device='cuda:0')) | > avg_loss_gen: 2.238686959508439 (+0.00507004328177052) | > avg_loss_kl: 2.0123651128419686 (-0.015021562576293945) | > avg_loss_feat: 4.0256051714991195 (+0.023027033873009195) | > avg_loss_mel: 16.60245367506859 (+0.06371543776819166) | > avg_loss_duration: 1.5376066909709447 (-0.0010193384868997235) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.41671744870468 (+0.07577135865117768) | > avg_grad_norm_1: tensor(9401.0156, device='cuda:0') (tensor(-608.5938, device='cuda:0'))  > EPOCH: 1029/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:49:16)   --> TIME: 2025-02-25 18:49:22 -- STEP: 11/72 -- GLOBAL_STEP: 1074100 | > loss_disc: 2.6555492877960205 (2.529274116862904) | > loss_disc_real_0: 0.12295803427696228 (0.09769697148691524) | > loss_disc_real_1: 0.22511935234069824 (0.22645486213944174) | > loss_disc_real_2: 0.25337010622024536 (0.2283975752917203) | > loss_disc_real_3: 0.27148759365081787 (0.23238241943446072) | > loss_disc_real_4: 0.2527526021003723 (0.23790426687760788) | > loss_disc_real_5: 0.2383689135313034 (0.24382142993536862) | > loss_0: 2.6555492877960205 (2.529274116862904) | > grad_norm_0: tensor(403.2712, device='cuda:0') (tensor(307.7917, device='cuda:0')) | > loss_gen: 2.021757125854492 (2.240340232849121) | > loss_kl: 1.9630743265151978 (1.9321500929919155) | > loss_feat: 3.439702033996582 (3.9741857051849365) | > loss_mel: 16.448413848876953 (16.50991153717041) | > loss_duration: 1.4920865297317505 (1.4982645511627197) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.365034103393555 (26.15485243363814) | > grad_norm_1: tensor(9100.9199, device='cuda:0') (tensor(8766.2461, device='cuda:0')) | > current_lr_0: 0.0001758593146511134 | > current_lr_1: 0.0001758593146511134 | > step_time: 0.425 (0.4334719614549117) | > loader_time: 0.0058 (0.006895455447110263)  --> TIME: 2025-02-25 18:49:46 -- STEP: 61/72 -- GLOBAL_STEP: 1074150 | > loss_disc: 2.5862209796905518 (2.5507698645357224) | > loss_disc_real_0: 0.054356906563043594 (0.09402491231677963) | > loss_disc_real_1: 0.2333509624004364 (0.22473460558007974) | > loss_disc_real_2: 0.2199965864419937 (0.23377073666111367) | > loss_disc_real_3: 0.2176230400800705 (0.23154015301680955) | > loss_disc_real_4: 0.2320765256881714 (0.24090205768092735) | > loss_disc_real_5: 0.2491607517004013 (0.2445682416685292) | > loss_0: 2.5862209796905518 (2.5507698645357224) | > grad_norm_0: tensor(116.4087, device='cuda:0') (tensor(315.0322, device='cuda:0')) | > loss_gen: 2.243043899536133 (2.236124140317323) | > loss_kl: 2.2349374294281006 (2.0525363820498104) | > loss_feat: 3.866680145263672 (4.063577648069037) | > loss_mel: 16.891639709472656 (16.58258197346672) | > loss_duration: 1.5703718662261963 (1.5294770705895346) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.806673049926758 (26.464297310250704) | > grad_norm_1: tensor(8485.7012, device='cuda:0') (tensor(9837.5791, device='cuda:0')) | > current_lr_0: 0.0001758593146511134 | > current_lr_1: 0.0001758593146511134 | > step_time: 0.4636 (0.4459364531470127) | > loader_time: 0.0111 (0.008243572516519512) --> EVAL PERFORMANCE | > avg_loader_time: 0.008713168157658104 (+0.00016612737951144166) | > avg_step_time: 0.4546512785092206 (-0.0002978318174120531) | > avg_loss_disc: 2.5489198355607585 (+0.010393706845565731) | > avg_loss_disc_real_0: 0.09712006051985313 (-0.0012172499382999413) | > avg_loss_disc_real_1: 0.22332308829670222 (-0.000969977865756394) | > avg_loss_disc_real_2: 0.23478243069749483 (+0.0038808242116175895) | > avg_loss_disc_real_3: 0.2310941303280038 (-0.0004010357907120454) | > avg_loss_disc_real_4: 0.24079358430815415 (-2.9444904394548832e-05) | > avg_loss_disc_real_5: 0.24538550746272986 (+0.00043951499630026647) | > avg_loss_0: 2.5489198355607585 (+0.010393706845565731) | > avg_grad_norm_0: tensor(316.5056, device='cuda:0') (+tensor(14.7206, device='cuda:0')) | > avg_loss_gen: 2.243965974995788 (+0.0052790154873489925) | > avg_loss_kl: 2.080508621645645 (+0.06814350880367659) | > avg_loss_feat: 4.055542858553602 (+0.029937687054482964) | > avg_loss_mel: 16.629245301367536 (+0.026791626298948046) | > avg_loss_duration: 1.5393364983545224 (+0.0017298073835776417) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.548599296892192 (+0.13188184818751125) | > avg_grad_norm_1: tensor(9651.0498, device='cuda:0') (+tensor(250.0342, device='cuda:0'))  > EPOCH: 1030/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:49:52)   --> TIME: 2025-02-25 18:50:12 -- STEP: 39/72 -- GLOBAL_STEP: 1074200 | > loss_disc: 2.762941360473633 (2.7221154005099564) | > loss_disc_real_0: 0.11366403102874756 (0.1609579743578648) | > loss_disc_real_1: 0.21272297203540802 (0.22998833102293503) | > loss_disc_real_2: 0.24895340204238892 (0.25989147562247056) | > loss_disc_real_3: 0.26610589027404785 (0.25097587666450405) | > loss_disc_real_4: 0.24888354539871216 (0.2494221738515756) | > loss_disc_real_5: 0.2588236927986145 (0.24885476361482572) | > loss_0: 2.762941360473633 (2.7221154005099564) | > grad_norm_0: tensor(37.3910, device='cuda:0') (tensor(218.5405, device='cuda:0')) | > loss_gen: 1.9715628623962402 (2.1998582772719555) | > loss_kl: 2.2144854068756104 (2.2439514368008346) | > loss_feat: 3.1761975288391113 (3.86835635625399) | > loss_mel: 17.16752052307129 (17.2875363276555) | > loss_duration: 1.574108362197876 (1.5315301510003896) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.1038761138916 (27.131232872987404) | > grad_norm_1: tensor(6856.7075, device='cuda:0') (tensor(4011.9675, device='cuda:0')) | > current_lr_0: 0.000175837332236782 | > current_lr_1: 0.000175837332236782 | > step_time: 0.466 (0.43938488226670486) | > loader_time: 0.0081 (0.007489705697084084) --> EVAL PERFORMANCE | > avg_loader_time: 0.008723222034078248 (+1.0053876420144242e-05) | > avg_step_time: 0.45452137732169995 (-0.0001299011875206535) | > avg_loss_disc: 2.731890225074661 (+0.18297038951390254) | > avg_loss_disc_real_0: 0.18402553389085008 (+0.08690547337099695) | > avg_loss_disc_real_1: 0.2417797634089497 (+0.018456675112247467) | > avg_loss_disc_real_2: 0.2501750970390481 (+0.015392666341553285) | > avg_loss_disc_real_3: 0.2450573547205455 (+0.01396322439254169) | > avg_loss_disc_real_4: 0.25005362277299586 (+0.009260038464841719) | > avg_loss_disc_real_5: 0.24917899491921278 (+0.003793487456482919) | > avg_loss_0: 2.731890225074661 (+0.18297038951390254) | > avg_grad_norm_0: tensor(228.7251, device='cuda:0') (tensor(-87.7805, device='cuda:0')) | > avg_loss_gen: 2.1740976941417642 (-0.06986828085402363) | > avg_loss_kl: 2.3428520702979934 (+0.26234344865234815) | > avg_loss_feat: 3.708843083448813 (-0.34669977510478933) | > avg_loss_mel: 17.437213521608168 (+0.8079682202406318) | > avg_loss_duration: 1.582286802815719 (+0.04295030446119674) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.245293415768042 (+0.6966941188758504) | > avg_grad_norm_1: tensor(4864.3794, device='cuda:0') (tensor(-4786.6704, device='cuda:0'))  > EPOCH: 1031/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:50:29)   --> TIME: 2025-02-25 18:50:40 -- STEP: 17/72 -- GLOBAL_STEP: 1074250 | > loss_disc: 2.6895034313201904 (2.6046534145579616) | > loss_disc_real_0: 0.12171272933483124 (0.07512577491648056) | > loss_disc_real_1: 0.24168141186237335 (0.2281186449177125) | > loss_disc_real_2: 0.21438802778720856 (0.22337563598857207) | > loss_disc_real_3: 0.2520233988761902 (0.24328688400633194) | > loss_disc_real_4: 0.26963186264038086 (0.2392012257786358) | > loss_disc_real_5: 0.2663160264492035 (0.2360854271580191) | > loss_0: 2.6895034313201904 (2.6046534145579616) | > grad_norm_0: tensor(209.1558, device='cuda:0') (tensor(261.5723, device='cuda:0')) | > loss_gen: 1.9962148666381836 (2.0194259741727043) | > loss_kl: 2.34143328666687 (2.353045730029835) | > loss_feat: 3.582770586013794 (3.831274481380687) | > loss_mel: 18.16037940979004 (17.651114968692553) | > loss_duration: 1.5583374500274658 (1.5356264535118551) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.639137268066406 (27.390487895292395) | > grad_norm_1: tensor(10471.1406, device='cuda:0') (tensor(8397.7500, device='cuda:0')) | > current_lr_0: 0.0001758153525702524 | > current_lr_1: 0.0001758153525702524 | > step_time: 0.432 (0.4646357227774227) | > loader_time: 0.0067 (0.006850186516256893)  --> TIME: 2025-02-25 18:51:03 -- STEP: 67/72 -- GLOBAL_STEP: 1074300 | > loss_disc: 2.497973680496216 (2.5563734182670936) | > loss_disc_real_0: 0.08833999931812286 (0.07953787617274183) | > loss_disc_real_1: 0.21461902558803558 (0.22431049195688163) | > loss_disc_real_2: 0.22172294557094574 (0.22947787638030834) | > loss_disc_real_3: 0.20094884932041168 (0.23294209989149178) | > loss_disc_real_4: 0.2456292361021042 (0.24078321123301094) | > loss_disc_real_5: 0.2591990530490875 (0.2431235273382557) | > loss_0: 2.497973680496216 (2.5563734182670936) | > grad_norm_0: tensor(426.7894, device='cuda:0') (tensor(276.4586, device='cuda:0')) | > loss_gen: 2.143742561340332 (2.128906657446676) | > loss_kl: 2.0721487998962402 (2.163366351554643) | > loss_feat: 3.9739081859588623 (3.8907239081254645) | > loss_mel: 16.34650230407715 (16.917188188922946) | > loss_duration: 1.6422626972198486 (1.554207639907723) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.178564071655273 (26.654392726385773) | > grad_norm_1: tensor(12699.5586, device='cuda:0') (tensor(9766.9561, device='cuda:0')) | > current_lr_0: 0.0001758153525702524 | > current_lr_1: 0.0001758153525702524 | > step_time: 0.5202 (0.46073358806211556) | > loader_time: 0.0123 (0.008352155116067003) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853477397435148 (-0.00018844805972676772) | > avg_step_time: 0.464064144752395 (+0.009542767430695043) | > avg_loss_disc: 2.5603384165696697 (-0.1715518085049914) | > avg_loss_disc_real_0: 0.08164262677162464 (-0.10238290711922544) | > avg_loss_disc_real_1: 0.22552683214906236 (-0.016252931259887332) | > avg_loss_disc_real_2: 0.22997084218011776 (-0.02020425485893035) | > avg_loss_disc_real_3: 0.23251631918927312 (-0.012541035531272365) | > avg_loss_disc_real_4: 0.24032385227545885 (-0.00972977049753701) | > avg_loss_disc_real_5: 0.2426482185091771 (-0.006530776410035688) | > avg_loss_0: 2.5603384165696697 (-0.1715518085049914) | > avg_grad_norm_0: tensor(278.2815, device='cuda:0') (+tensor(49.5564, device='cuda:0')) | > avg_loss_gen: 2.1331610578886218 (-0.04093663625314248) | > avg_loss_kl: 2.1613099826893336 (-0.18154208760865975) | > avg_loss_feat: 3.8858862527659244 (+0.17704316931711128) | > avg_loss_mel: 16.923445741895225 (-0.5137677797129427) | > avg_loss_duration: 1.5567295870310822 (-0.025557215784636877) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.6605325886901 (-0.5847608270779432) | > avg_grad_norm_1: tensor(9674.4492, device='cuda:0') (+tensor(4810.0698, device='cuda:0'))  > EPOCH: 1032/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:51:07)   --> TIME: 2025-02-25 18:51:30 -- STEP: 45/72 -- GLOBAL_STEP: 1074350 | > loss_disc: 2.641423225402832 (2.5445524533589676) | > loss_disc_real_0: 0.16734665632247925 (0.10791540634300974) | > loss_disc_real_1: 0.23584800958633423 (0.2176170719994439) | > loss_disc_real_2: 0.24446895718574524 (0.23499183191193473) | > loss_disc_real_3: 0.2772406041622162 (0.23477284842067295) | > loss_disc_real_4: 0.2493535429239273 (0.24102662569946712) | > loss_disc_real_5: 0.24395668506622314 (0.2455111712217331) | > loss_0: 2.641423225402832 (2.5445524533589676) | > grad_norm_0: tensor(425.7446, device='cuda:0') (tensor(268.2523, device='cuda:0')) | > loss_gen: 2.1796844005584717 (2.2413668923907815) | > loss_kl: 2.1235923767089844 (2.0129769669638744) | > loss_feat: 3.7670235633850098 (3.9355420377519383) | > loss_mel: 16.685022354125977 (16.60833492279053) | > loss_duration: 1.5799118280410767 (1.5196007543139987) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.335233688354492 (26.317821545071073) | > grad_norm_1: tensor(8736.4971, device='cuda:0') (tensor(7603.1045, device='cuda:0')) | > current_lr_0: 0.00017579337565118113 | > current_lr_1: 0.00017579337565118113 | > step_time: 0.4746 (0.44419924947950573) | > loader_time: 0.0085 (0.007460620668199327) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854592591943875 (+1.1151945087269771e-05) | > avg_step_time: 0.4573643140389886 (-0.006699830713406407) | > avg_loss_disc: 2.5637631684961444 (+0.003424751926474734) | > avg_loss_disc_real_0: 0.1143539359556957 (+0.03271130918407107) | > avg_loss_disc_real_1: 0.2235371061194111 (-0.001989726029651251) | > avg_loss_disc_real_2: 0.2353555910604101 (+0.005384748880292334) | > avg_loss_disc_real_3: 0.23749536674627117 (+0.004979047556998045) | > avg_loss_disc_real_4: 0.2433749684985255 (+0.003051116223066641) | > avg_loss_disc_real_5: 0.2457086166865389 (+0.0030603981773618116) | > avg_loss_0: 2.5637631684961444 (+0.003424751926474734) | > avg_grad_norm_0: tensor(277.7123, device='cuda:0') (tensor(-0.5692, device='cuda:0')) | > avg_loss_gen: 2.2268993015020673 (+0.09373824361344552) | > avg_loss_kl: 2.038534013318344 (-0.12277596937098956) | > avg_loss_feat: 3.8885543413565182 (+0.0026680885905938645) | > avg_loss_mel: 16.58136931943222 (-0.3420764224630055) | > avg_loss_duration: 1.5451510389086227 (-0.011578548122459509) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.28050796078964 (-0.38002462790045755) | > avg_grad_norm_1: tensor(7351.1699, device='cuda:0') (tensor(-2323.2793, device='cuda:0'))  > EPOCH: 1033/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:51:44)   --> TIME: 2025-02-25 18:51:57 -- STEP: 23/72 -- GLOBAL_STEP: 1074400 | > loss_disc: 2.799591541290283 (2.593284492907317) | > loss_disc_real_0: 0.051492560654878616 (0.1234715070737445) | > loss_disc_real_1: 0.24818402528762817 (0.23425535274588544) | > loss_disc_real_2: 0.2868768870830536 (0.24405924846296725) | > loss_disc_real_3: 0.27831605076789856 (0.23952279596225076) | > loss_disc_real_4: 0.31891942024230957 (0.24717436730861664) | > loss_disc_real_5: 0.28526410460472107 (0.24812013105205868) | > loss_0: 2.799591541290283 (2.593284492907317) | > grad_norm_0: tensor(518.6000, device='cuda:0') (tensor(327.0934, device='cuda:0')) | > loss_gen: 1.983572006225586 (2.188916361850241) | > loss_kl: 1.977347731590271 (1.9874180555343628) | > loss_feat: 3.680529832839966 (3.89669422481371) | > loss_mel: 16.346540451049805 (16.671296327010445) | > loss_duration: 1.546342134475708 (1.4999610963075056) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.534332275390625 (26.2442860810653) | > grad_norm_1: tensor(14778.0527, device='cuda:0') (tensor(7568.0776, device='cuda:0')) | > current_lr_0: 0.00017577140147922472 | > current_lr_1: 0.00017577140147922472 | > step_time: 0.4374 (0.4388380879941194) | > loader_time: 0.0067 (0.006981735644133195) --> EVAL PERFORMANCE | > avg_loader_time: 0.008714111757950045 (+0.00016818583851129464) | > avg_step_time: 0.4580663325081409 (+0.000702018469152299) | > avg_loss_disc: 2.576539308252469 (+0.012776139756324412) | > avg_loss_disc_real_0: 0.10638669327321187 (-0.007967242682483833) | > avg_loss_disc_real_1: 0.22542620334826724 (+0.0018890972288561347) | > avg_loss_disc_real_2: 0.24048068615752208 (+0.005125095097111981) | > avg_loss_disc_real_3: 0.234702575794408 (-0.002792790951863161) | > avg_loss_disc_real_4: 0.24516413014539531 (+0.001789161646869819) | > avg_loss_disc_real_5: 0.24828716356989353 (+0.0025785468833546266) | > avg_loss_0: 2.576539308252469 (+0.012776139756324412) | > avg_grad_norm_0: tensor(318.8301, device='cuda:0') (+tensor(41.1178, device='cuda:0')) | > avg_loss_gen: 2.1756806944457576 (-0.05121860705630965) | > avg_loss_kl: 2.0493943590513424 (+0.010860345732998322) | > avg_loss_feat: 3.834017283479932 (-0.054537057876586026) | > avg_loss_mel: 16.638813824720778 (+0.057444505288557934) | > avg_loss_duration: 1.5428043200936117 (-0.0023467188150110285) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.24071056742064 (-0.03979739336900323) | > avg_grad_norm_1: tensor(9064.6133, device='cuda:0') (+tensor(1713.4434, device='cuda:0'))  > EPOCH: 1034/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:52:21)   --> TIME: 2025-02-25 18:52:24 -- STEP: 1/72 -- GLOBAL_STEP: 1074450 | > loss_disc: 2.325727939605713 (2.325727939605713) | > loss_disc_real_0: 0.08360941708087921 (0.08360941708087921) | > loss_disc_real_1: 0.22540006041526794 (0.22540006041526794) | > loss_disc_real_2: 0.24063414335250854 (0.24063414335250854) | > loss_disc_real_3: 0.24088223278522491 (0.24088223278522491) | > loss_disc_real_4: 0.2444203794002533 (0.2444203794002533) | > loss_disc_real_5: 0.24614031612873077 (0.24614031612873077) | > loss_0: 2.325727939605713 (2.325727939605713) | > grad_norm_0: tensor(164.9024, device='cuda:0') (tensor(164.9024, device='cuda:0')) | > loss_gen: 2.668616533279419 (2.668616533279419) | > loss_kl: 1.8351441621780396 (1.8351441621780396) | > loss_feat: 4.756397247314453 (4.756397247314453) | > loss_mel: 17.62012481689453 (17.62012481689453) | > loss_duration: 1.4999761581420898 (1.4999761581420898) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.380260467529297 (28.380260467529297) | > grad_norm_1: tensor(7192.7651, device='cuda:0') (tensor(7192.7651, device='cuda:0')) | > current_lr_0: 0.00017574943005403982 | > current_lr_1: 0.00017574943005403982 | > step_time: 0.4383 (0.43833136558532715) | > loader_time: 0.0051 (0.005124092102050781)  --> TIME: 2025-02-25 18:52:47 -- STEP: 51/72 -- GLOBAL_STEP: 1074500 | > loss_disc: 2.336351156234741 (2.540254648993997) | > loss_disc_real_0: 0.16559183597564697 (0.11926266624062669) | > loss_disc_real_1: 0.2594565153121948 (0.2214939202748093) | > loss_disc_real_2: 0.2368544489145279 (0.2255661779758977) | > loss_disc_real_3: 0.21085293591022491 (0.22937346468953526) | > loss_disc_real_4: 0.2530269920825958 (0.24096535000146604) | > loss_disc_real_5: 0.23475205898284912 (0.246284962869158) | > loss_0: 2.336351156234741 (2.540254648993997) | > grad_norm_0: tensor(164.9031, device='cuda:0') (tensor(284.4072, device='cuda:0')) | > loss_gen: 2.892059326171875 (2.276932517687481) | > loss_kl: 2.0796337127685547 (2.0349782588435152) | > loss_feat: 5.067504405975342 (3.9915259211671117) | > loss_mel: 16.44871711730957 (16.673382198109344) | > loss_duration: 1.5563030242919922 (1.520776299869313) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.044218063354492 (26.49759520736395) | > grad_norm_1: tensor(3397.2517, device='cuda:0') (tensor(7490.9199, device='cuda:0')) | > current_lr_0: 0.00017574943005403982 | > current_lr_1: 0.00017574943005403982 | > step_time: 0.4551 (0.44618945028267654) | > loader_time: 0.0092 (0.007824472352570179) --> EVAL PERFORMANCE | > avg_loader_time: 0.008691388116755959 (-2.2723641194086208e-05) | > avg_step_time: 0.45892893092733034 (+0.0008625984191894531) | > avg_loss_disc: 2.6012125552540093 (+0.0246732470015405) | > avg_loss_disc_real_0: 0.13696190211134898 (+0.030575208838137108) | > avg_loss_disc_real_1: 0.23302003552376385 (+0.007593832175496612) | > avg_loss_disc_real_2: 0.2280213173846124 (-0.01245936877290968) | > avg_loss_disc_real_3: 0.2320300901020077 (-0.002672485692400306) | > avg_loss_disc_real_4: 0.24438509093204014 (-0.000779039213355176) | > avg_loss_disc_real_5: 0.24827171862125397 (-1.5444948639559142e-05) | > avg_loss_0: 2.6012125552540093 (+0.0246732470015405) | > avg_grad_norm_0: tensor(270.5669, device='cuda:0') (tensor(-48.2632, device='cuda:0')) | > avg_loss_gen: 2.260142022455243 (+0.08446132800948547) | > avg_loss_kl: 2.0830857904864026 (+0.03369143143506026) | > avg_loss_feat: 3.908163675120179 (+0.07414639164024672) | > avg_loss_mel: 16.734874725341797 (+0.09606090062101913) | > avg_loss_duration: 1.540578780039935 (-0.002225540053676589) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.526845018628617 (+0.2861344512079782) | > avg_grad_norm_1: tensor(7329.7808, device='cuda:0') (tensor(-1734.8325, device='cuda:0'))  > EPOCH: 1035/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:52:59)   --> TIME: 2025-02-25 18:53:14 -- STEP: 29/72 -- GLOBAL_STEP: 1074550 | > loss_disc: 2.708379030227661 (2.728597246367356) | > loss_disc_real_0: 0.1647755205631256 (0.17152323920665116) | > loss_disc_real_1: 0.2828803062438965 (0.23059707109270425) | > loss_disc_real_2: 0.2592417895793915 (0.2303562642171465) | > loss_disc_real_3: 0.26778772473335266 (0.23799484146052394) | > loss_disc_real_4: 0.2588694393634796 (0.23368961595255752) | > loss_disc_real_5: 0.2537499666213989 (0.24103621932966957) | > loss_0: 2.708379030227661 (2.728597246367356) | > grad_norm_0: tensor(63.3560, device='cuda:0') (tensor(217.3210, device='cuda:0')) | > loss_gen: 2.017228841781616 (2.012097864315427) | > loss_kl: 2.36700701713562 (2.199494407094758) | > loss_feat: 3.463595390319824 (3.6373830499320197) | > loss_mel: 18.4451904296875 (17.2768412294059) | > loss_duration: 1.5700914859771729 (1.5483564302839081) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.863113403320312 (26.674172828937397) | > grad_norm_1: tensor(4495.0938, device='cuda:0') (tensor(5374.3267, device='cuda:0')) | > current_lr_0: 0.00017572746137528306 | > current_lr_1: 0.00017572746137528306 | > step_time: 0.454 (0.4437475697747592) | > loader_time: 0.0077 (0.0072557761751372235) --> EVAL PERFORMANCE | > avg_loader_time: 0.008714219214211045 (+2.2831097455086602e-05) | > avg_step_time: 0.4613109373710525 (+0.002382006443722162) | > avg_loss_disc: 2.677109211263523 (+0.07589665600951356) | > avg_loss_disc_real_0: 0.15774147443368403 (+0.020779572322335044) | > avg_loss_disc_real_1: 0.22783036404092546 (-0.005189671482838387) | > avg_loss_disc_real_2: 0.23380921906988386 (+0.005787901685271463) | > avg_loss_disc_real_3: 0.23747950498486908 (+0.005449414882861381) | > avg_loss_disc_real_4: 0.23698847373606455 (-0.007396617195975591) | > avg_loss_disc_real_5: 0.24311693243577448 (-0.005154786185479487) | > avg_loss_0: 2.677109211263523 (+0.07589665600951356) | > avg_grad_norm_0: tensor(211.5703, device='cuda:0') (tensor(-58.9966, device='cuda:0')) | > avg_loss_gen: 2.0545394857164823 (-0.20560253673876083) | > avg_loss_kl: 2.206653804846213 (+0.12356801435981035) | > avg_loss_feat: 3.6082226424150066 (-0.29994103270517236) | > avg_loss_mel: 17.177718256560848 (+0.44284353121905085) | > avg_loss_duration: 1.5757076924955342 (+0.035128912455599126) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.62284176786181 (+0.09599674923319412) | > avg_grad_norm_1: tensor(5594.0229, device='cuda:0') (tensor(-1735.7578, device='cuda:0'))  > EPOCH: 1036/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:53:36)   --> TIME: 2025-02-25 18:53:42 -- STEP: 7/72 -- GLOBAL_STEP: 1074600 | > loss_disc: 2.5114660263061523 (2.6010680879865373) | > loss_disc_real_0: 0.0601995624601841 (0.12783384269901685) | > loss_disc_real_1: 0.22160717844963074 (0.1790302566119603) | > loss_disc_real_2: 0.24581053853034973 (0.24761949905327388) | > loss_disc_real_3: 0.24523821473121643 (0.25779533811977934) | > loss_disc_real_4: 0.22920681536197662 (0.25049370740141186) | > loss_disc_real_5: 0.2148643434047699 (0.2351698194231306) | > loss_0: 2.5114660263061523 (2.6010680879865373) | > grad_norm_0: tensor(391.2647, device='cuda:0') (tensor(365.2961, device='cuda:0')) | > loss_gen: 2.165487289428711 (2.1173282521111623) | > loss_kl: 1.8266401290893555 (1.8895937204360962) | > loss_feat: 4.367728233337402 (3.859651633671352) | > loss_mel: 17.858016967773438 (16.99334171840123) | > loss_duration: 1.4844443798065186 (1.4981134619031633) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.702316284179688 (26.358028139386857) | > grad_norm_1: tensor(8743.0283, device='cuda:0') (tensor(7627.2900, device='cuda:0')) | > current_lr_0: 0.00017570549544261116 | > current_lr_1: 0.00017570549544261116 | > step_time: 0.4575 (0.4379066399165562) | > loader_time: 0.0079 (0.006764037268502372)  --> TIME: 2025-02-25 18:54:05 -- STEP: 57/72 -- GLOBAL_STEP: 1074650 | > loss_disc: 2.9400124549865723 (2.61652424042685) | > loss_disc_real_0: 0.16080771386623383 (0.13135312393046264) | > loss_disc_real_1: 0.23050899803638458 (0.2234898146830107) | > loss_disc_real_2: 0.2531735301017761 (0.24102678973423808) | > loss_disc_real_3: 0.24665647745132446 (0.23720617749189077) | > loss_disc_real_4: 0.2631191313266754 (0.24666498889002883) | > loss_disc_real_5: 0.24314896762371063 (0.24252427722278394) | > loss_0: 2.9400124549865723 (2.61652424042685) | > grad_norm_0: tensor(692.9490, device='cuda:0') (tensor(290.3452, device='cuda:0')) | > loss_gen: 1.9849441051483154 (2.12096086928719) | > loss_kl: 2.269883871078491 (2.0086702066555358) | > loss_feat: 3.5534372329711914 (3.779692850614849) | > loss_mel: 16.39349365234375 (16.888667826066946) | > loss_duration: 1.6033000946044922 (1.5334168923528573) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.8050594329834 (26.3314086847138) | > grad_norm_1: tensor(12171.0479, device='cuda:0') (tensor(7688.8369, device='cuda:0')) | > current_lr_0: 0.00017570549544261116 | > current_lr_1: 0.00017570549544261116 | > step_time: 0.4649 (0.4487729532676831) | > loader_time: 0.0095 (0.00808213468183551) --> EVAL PERFORMANCE | > avg_loader_time: 0.008703110923229809 (-1.1108290981236627e-05) | > avg_step_time: 0.4590850917386337 (-0.002225845632418788) | > avg_loss_disc: 2.6373974504605147 (-0.0397117608030082) | > avg_loss_disc_real_0: 0.13199852548644575 (-0.025742948947238276) | > avg_loss_disc_real_1: 0.22143775235179444 (-0.0063926116891310225) | > avg_loss_disc_real_2: 0.24285881305244608 (+0.00904959398256222) | > avg_loss_disc_real_3: 0.2370537236123018 (-0.00042578137256729653) | > avg_loss_disc_real_4: 0.24483210520005563 (+0.007843631463991085) | > avg_loss_disc_real_5: 0.24187403392623846 (-0.0012428985095360168) | > avg_loss_0: 2.6373974504605147 (-0.0397117608030082) | > avg_grad_norm_0: tensor(285.7409, device='cuda:0') (+tensor(74.1705, device='cuda:0')) | > avg_loss_gen: 2.0846651238454905 (+0.03012563812900826) | > avg_loss_kl: 2.0569172728229574 (-0.14973653202325554) | > avg_loss_feat: 3.705494984774522 (+0.09727234235951565) | > avg_loss_mel: 16.94909541707644 (-0.22862283948440876) | > avg_loss_duration: 1.5484405554516214 (-0.027267137043912815) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.344613518513423 (-0.27822824934838764) | > avg_grad_norm_1: tensor(7510.1294, device='cuda:0') (+tensor(1916.1064, device='cuda:0'))  > EPOCH: 1037/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:54:14)   --> TIME: 2025-02-25 18:54:32 -- STEP: 35/72 -- GLOBAL_STEP: 1074700 | > loss_disc: 2.524446964263916 (2.6781754289354596) | > loss_disc_real_0: 0.06391682475805283 (0.16758057943412236) | > loss_disc_real_1: 0.2887435853481293 (0.23378152251243592) | > loss_disc_real_2: 0.2797926366329193 (0.23185262680053711) | > loss_disc_real_3: 0.2698686122894287 (0.23554524268422808) | > loss_disc_real_4: 0.3014013469219208 (0.2505059237991061) | > loss_disc_real_5: 0.2443077713251114 (0.23920017225401743) | > loss_0: 2.524446964263916 (2.6781754289354596) | > grad_norm_0: tensor(45.1804, device='cuda:0') (tensor(221.8645, device='cuda:0')) | > loss_gen: 2.142056941986084 (2.148714862551009) | > loss_kl: 2.222106456756592 (2.029896640777588) | > loss_feat: 3.731788158416748 (3.759422527040754) | > loss_mel: 17.3761043548584 (17.086128071376255) | > loss_duration: 1.5701664686203003 (1.5121451684406826) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.042221069335938 (26.536307198660715) | > grad_norm_1: tensor(5338.6401, device='cuda:0') (tensor(7119.3965, device='cuda:0')) | > current_lr_0: 0.00017568353225568082 | > current_lr_1: 0.00017568353225568082 | > step_time: 0.436 (0.44091179030282157) | > loader_time: 0.0082 (0.007186508178710938) --> EVAL PERFORMANCE | > avg_loader_time: 0.008663509933041855 (-3.960099018795321e-05) | > avg_step_time: 0.4579697830576292 (-0.001115308681004512) | > avg_loss_disc: 2.6748690202202594 (+0.03747156975974475) | > avg_loss_disc_real_0: 0.1621173074022985 (+0.030118781915852738) | > avg_loss_disc_real_1: 0.23231405082722784 (+0.010876298475433394) | > avg_loss_disc_real_2: 0.23614197297835013 (-0.006716840074095953) | > avg_loss_disc_real_3: 0.23742628790123363 (+0.00037256428893184257) | > avg_loss_disc_real_4: 0.24609269852369603 (+0.0012605933236403966) | > avg_loss_disc_real_5: 0.23802835588723842 (-0.0038456780390000445) | > avg_loss_0: 2.6748690202202594 (+0.03747156975974475) | > avg_grad_norm_0: tensor(221.3538, device='cuda:0') (tensor(-64.3871, device='cuda:0')) | > avg_loss_gen: 2.12412035633141 (+0.039455232485919645) | > avg_loss_kl: 2.238977611904413 (+0.1820603390814557) | > avg_loss_feat: 3.7164428099780014 (+0.010947825203479145) | > avg_loss_mel: 17.86274751475159 (+0.9136520976751505) | > avg_loss_duration: 1.5665188104334011 (+0.01807825498177973) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.508807034559652 (+1.164193516046229) | > avg_grad_norm_1: tensor(6309.3926, device='cuda:0') (tensor(-1200.7368, device='cuda:0'))  > EPOCH: 1038/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:54:51)   --> TIME: 2025-02-25 18:54:59 -- STEP: 13/72 -- GLOBAL_STEP: 1074750 | > loss_disc: 2.679143190383911 (2.574747470708994) | > loss_disc_real_0: 0.05444067344069481 (0.10163087684374589) | > loss_disc_real_1: 0.2420845925807953 (0.23819788029560676) | > loss_disc_real_2: 0.23102015256881714 (0.21787606408962837) | > loss_disc_real_3: 0.20803168416023254 (0.2358408925624994) | > loss_disc_real_4: 0.21654371917247772 (0.22565529667414153) | > loss_disc_real_5: 0.23211592435836792 (0.2279931799723552) | > loss_0: 2.679143190383911 (2.574747470708994) | > grad_norm_0: tensor(179.0404, device='cuda:0') (tensor(216.6877, device='cuda:0')) | > loss_gen: 1.7616751194000244 (2.303142244999225) | > loss_kl: 1.962588906288147 (1.9970388229076679) | > loss_feat: 3.3777408599853516 (4.133801075128408) | > loss_mel: 15.195016860961914 (17.34325100825383) | > loss_duration: 1.447076678276062 (1.5176956561895518) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.744098663330078 (27.294929064237156) | > grad_norm_1: tensor(7069.4717, device='cuda:0') (tensor(6232.5205, device='cuda:0')) | > current_lr_0: 0.00017566157181414884 | > current_lr_1: 0.00017566157181414884 | > step_time: 0.4832 (0.4379419546860915) | > loader_time: 0.0069 (0.006655766413762019)  --> TIME: 2025-02-25 18:55:22 -- STEP: 63/72 -- GLOBAL_STEP: 1074800 | > loss_disc: 2.4935412406921387 (2.620979823763409) | > loss_disc_real_0: 0.12711183726787567 (0.13594255513615078) | > loss_disc_real_1: 0.19943372905254364 (0.21959384944703844) | > loss_disc_real_2: 0.19643628597259521 (0.23356190962450846) | > loss_disc_real_3: 0.23033039271831512 (0.2363047810300948) | > loss_disc_real_4: 0.27500906586647034 (0.23863662140710012) | > loss_disc_real_5: 0.28992101550102234 (0.24285667377804954) | > loss_0: 2.4935412406921387 (2.620979823763409) | > grad_norm_0: tensor(143.4621, device='cuda:0') (tensor(240.2981, device='cuda:0')) | > loss_gen: 2.4230570793151855 (2.140039247179789) | > loss_kl: 2.0492470264434814 (2.0277762204881697) | > loss_feat: 3.308011531829834 (3.763691958926973) | > loss_mel: 16.63632583618164 (17.00520239935981) | > loss_duration: 1.5781631469726562 (1.5487472689340984) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.99480438232422 (26.485457163008434) | > grad_norm_1: tensor(4181.7295, device='cuda:0') (tensor(7743.8950, device='cuda:0')) | > current_lr_0: 0.00017566157181414884 | > current_lr_1: 0.00017566157181414884 | > step_time: 0.5043 (0.4499387551867773) | > loader_time: 0.0111 (0.00819479851495652) --> EVAL PERFORMANCE | > avg_loader_time: 0.008572971317130076 (-9.05386159117793e-05) | > avg_step_time: 0.45835839862554845 (+0.00038861556791924823) | > avg_loss_disc: 2.624304586732892 (-0.05056443348736739) | > avg_loss_disc_real_0: 0.14438282598702 (-0.017734481415278475) | > avg_loss_disc_real_1: 0.2196314899434506 (-0.012682560883777227) | > avg_loss_disc_real_2: 0.23110998894127321 (-0.005031984037076914) | > avg_loss_disc_real_3: 0.23582441727040518 (-0.0016018706308284503) | > avg_loss_disc_real_4: 0.23922669950505376 (-0.006865999018642266) | > avg_loss_disc_real_5: 0.24436144509785612 (+0.006333089210617704) | > avg_loss_0: 2.624304586732892 (-0.05056443348736739) | > avg_grad_norm_0: tensor(256.4337, device='cuda:0') (+tensor(35.0799, device='cuda:0')) | > avg_loss_gen: 2.13993942569679 (+0.01581906936537969) | > avg_loss_kl: 2.0435328769012235 (-0.1954447350031896) | > avg_loss_feat: 3.721588436986359 (+0.005145627008357856) | > avg_loss_mel: 16.96819361834459 (-0.8945538964070003) | > avg_loss_duration: 1.5563614469178966 (-0.010157363515504558) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.429615880402043 (-1.0791911541576091) | > avg_grad_norm_1: tensor(7221.9434, device='cuda:0') (+tensor(912.5508, device='cuda:0'))  > EPOCH: 1039/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:55:29)   --> TIME: 2025-02-25 18:55:49 -- STEP: 41/72 -- GLOBAL_STEP: 1074850 | > loss_disc: 2.5509605407714844 (2.5938726576363162) | > loss_disc_real_0: 0.05561560019850731 (0.112568107910636) | > loss_disc_real_1: 0.22379668056964874 (0.2209374595706056) | > loss_disc_real_2: 0.22976602613925934 (0.23146163799413821) | > loss_disc_real_3: 0.24408958852291107 (0.2345383527075372) | > loss_disc_real_4: 0.26968368887901306 (0.24518760166517117) | > loss_disc_real_5: 0.2734794616699219 (0.24575415289983515) | > loss_0: 2.5509605407714844 (2.5938726576363162) | > grad_norm_0: tensor(10.6747, device='cuda:0') (tensor(241.8803, device='cuda:0')) | > loss_gen: 2.2901103496551514 (2.1830385836159305) | > loss_kl: 2.0088353157043457 (2.053026117929599) | > loss_feat: 4.007862091064453 (4.06271034333764) | > loss_mel: 16.57293701171875 (17.197392231080595) | > loss_duration: 1.5720829963684082 (1.5238816476449735) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.451826095581055 (27.020048885810667) | > grad_norm_1: tensor(3001.1375, device='cuda:0') (tensor(5054.0669, device='cuda:0')) | > current_lr_0: 0.00017563961411767206 | > current_lr_1: 0.00017563961411767206 | > step_time: 0.45 (0.4417740193808951) | > loader_time: 0.0089 (0.007549117251140315) --> EVAL PERFORMANCE | > avg_loader_time: 0.008723013837572555 (+0.00015004252044247858) | > avg_step_time: 0.45797188181272697 (-0.0003865168128214802) | > avg_loss_disc: 2.6617873856719103 (+0.03748279893901829) | > avg_loss_disc_real_0: 0.13203531619108902 (-0.012347509795930994) | > avg_loss_disc_real_1: 0.23669610183003922 (+0.017064611886588615) | > avg_loss_disc_real_2: 0.23923414294988338 (+0.008124154008610163) | > avg_loss_disc_real_3: 0.2450221560790505 (+0.009197738808645328) | > avg_loss_disc_real_4: 0.25066018272453633 (+0.011433483219482565) | > avg_loss_disc_real_5: 0.25375266016369136 (+0.009391215065835234) | > avg_loss_0: 2.6617873856719103 (+0.03748279893901829) | > avg_grad_norm_0: tensor(243.1098, device='cuda:0') (tensor(-13.3239, device='cuda:0')) | > avg_loss_gen: 2.2014534607739518 (+0.06151403507716191) | > avg_loss_kl: 2.765463174228938 (+0.7219302973277144) | > avg_loss_feat: 4.0113856590969466 (+0.28979722211058734) | > avg_loss_mel: 19.28794915911178 (+2.31975554076719) | > avg_loss_duration: 1.6063406467437744 (+0.04997919982587784) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.872592039511236 (+3.442976159109193) | > avg_grad_norm_1: tensor(5860.0898, device='cuda:0') (tensor(-1361.8535, device='cuda:0'))  > EPOCH: 1040/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:56:06)   --> TIME: 2025-02-25 18:56:16 -- STEP: 19/72 -- GLOBAL_STEP: 1074900 | > loss_disc: 2.570692539215088 (2.5880228720213236) | > loss_disc_real_0: 0.17867153882980347 (0.1038223230525067) | > loss_disc_real_1: 0.24096128344535828 (0.24437437873137624) | > loss_disc_real_2: 0.2269214242696762 (0.23764953017234802) | > loss_disc_real_3: 0.24512559175491333 (0.23071434309608058) | > loss_disc_real_4: 0.22310860455036163 (0.23544259369373322) | > loss_disc_real_5: 0.21448718011379242 (0.2200084226696115) | > loss_0: 2.570692539215088 (2.5880228720213236) | > grad_norm_0: tensor(160.3244, device='cuda:0') (tensor(230.3356, device='cuda:0')) | > loss_gen: 2.394198417663574 (2.158414269748487) | > loss_kl: 2.4012582302093506 (2.5019238622565014) | > loss_feat: 3.913567304611206 (3.797826553645887) | > loss_mel: 17.59126091003418 (19.047829176250257) | > loss_duration: 1.5462827682495117 (1.5632377862930298) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.846569061279297 (29.069231736032588) | > grad_norm_1: tensor(9811.8047, device='cuda:0') (tensor(7530.1709, device='cuda:0')) | > current_lr_0: 0.00017561765916590733 | > current_lr_1: 0.00017561765916590733 | > step_time: 0.4255 (0.43136621776380035) | > loader_time: 0.006 (0.006829537843403064)  --> TIME: 2025-02-25 18:56:40 -- STEP: 69/72 -- GLOBAL_STEP: 1074950 | > loss_disc: 2.5496184825897217 (2.62246344746023) | > loss_disc_real_0: 0.1882334202528 (0.1292982562512591) | > loss_disc_real_1: 0.23539933562278748 (0.23563268424376196) | > loss_disc_real_2: 0.2342696338891983 (0.23034019271532694) | > loss_disc_real_3: 0.22215737402439117 (0.22062039461688718) | > loss_disc_real_4: 0.24906408786773682 (0.23686110411865124) | > loss_disc_real_5: 0.24885274469852448 (0.23577262288418369) | > loss_0: 2.5496184825897217 (2.62246344746023) | > grad_norm_0: tensor(296.5192, device='cuda:0') (tensor(248.2267, device='cuda:0')) | > loss_gen: 2.2798612117767334 (2.1561930974324546) | > loss_kl: 2.3272781372070312 (2.2984175146489894) | > loss_feat: 3.5735723972320557 (3.761508696321128) | > loss_mel: 18.37906837463379 (17.909241731616035) | > loss_duration: 1.6570184230804443 (1.5785405618557031) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.216798782348633 (27.703901677891828) | > grad_norm_1: tensor(10914.4912, device='cuda:0') (tensor(7747.7754, device='cuda:0')) | > current_lr_0: 0.00017561765916590733 | > current_lr_1: 0.00017561765916590733 | > step_time: 0.5236 (0.4526341617971227) | > loader_time: 0.0127 (0.008590760438338566) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647545962266516 (-7.546787530603866e-05) | > avg_step_time: 0.45475016849141725 (-0.003221713321309716) | > avg_loss_disc: 2.6256167149879563 (-0.036170670683953965) | > avg_loss_disc_real_0: 0.13098597096305495 (-0.001049345228034071) | > avg_loss_disc_real_1: 0.23531791032619878 (-0.0013781915038404424) | > avg_loss_disc_real_2: 0.23059458463964327 (-0.008639558310240103) | > avg_loss_disc_real_3: 0.22182147729564722 (-0.02320067878340329) | > avg_loss_disc_real_4: 0.2375544637021884 (-0.013105719022347917) | > avg_loss_disc_real_5: 0.23630877426812347 (-0.01744388589556789) | > avg_loss_0: 2.6256167149879563 (-0.036170670683953965) | > avg_grad_norm_0: tensor(242.5373, device='cuda:0') (tensor(-0.5725, device='cuda:0')) | > avg_loss_gen: 2.153304523145649 (-0.04814893762830286) | > avg_loss_kl: 2.294451221613816 (-0.47101195261512174) | > avg_loss_feat: 3.7463566954706757 (-0.2650289636262708) | > avg_loss_mel: 17.881493165459432 (-1.4064559936523473) | > avg_loss_duration: 1.5785281607802484 (-0.027812485963526035) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.65413383698799 (-2.2184582025232444) | > avg_grad_norm_1: tensor(7836.6230, device='cuda:0') (+tensor(1976.5332, device='cuda:0'))  > EPOCH: 1041/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:56:43)   --> TIME: 2025-02-25 18:57:06 -- STEP: 47/72 -- GLOBAL_STEP: 1075000 | > loss_disc: 2.473665714263916 (2.671421938754143) | > loss_disc_real_0: 0.07042045891284943 (0.1910049986649067) | > loss_disc_real_1: 0.2337898313999176 (0.22752375171539632) | > loss_disc_real_2: 0.26042747497558594 (0.24060580261210177) | > loss_disc_real_3: 0.24131536483764648 (0.2278675817428751) | > loss_disc_real_4: 0.23703482747077942 (0.24776361692459026) | > loss_disc_real_5: 0.23823435604572296 (0.24873935827549468) | > loss_0: 2.473665714263916 (2.671421938754143) | > grad_norm_0: tensor(90.0775, device='cuda:0') (tensor(223.5639, device='cuda:0')) | > loss_gen: 2.479491949081421 (2.0822851252048573) | > loss_kl: 2.103512763977051 (2.127287321902336) | > loss_feat: 4.274814128875732 (3.534989443231136) | > loss_mel: 16.480825424194336 (17.43192802591527) | > loss_duration: 1.5806565284729004 (1.5429371874383155) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.919300079345703 (26.719427068182764) | > grad_norm_1: tensor(3009.9956, device='cuda:0') (tensor(7516.4126, device='cuda:0')) | > current_lr_0: 0.00017559570695851158 | > current_lr_1: 0.00017559570695851158 | > step_time: 0.4528 (0.44232008812275336) | > loader_time: 0.0089 (0.007627740819403466) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1075000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.00861069853876678 (-3.684742349973659e-05) | > avg_step_time: 0.4565088547451395 (+0.0017586862537222747) | > avg_loss_disc: 2.679288561915008 (+0.053671846927051625) | > avg_loss_disc_real_0: 0.19730253958366287 (+0.06631656862060792) | > avg_loss_disc_real_1: 0.222663444532475 (-0.012654465793723785) | > avg_loss_disc_real_2: 0.23829430488633438 (+0.007699720246691105) | > avg_loss_disc_real_3: 0.22953871459188596 (+0.007717237296238744) | > avg_loss_disc_real_4: 0.24606692518146944 (+0.00851246147928103) | > avg_loss_disc_real_5: 0.2463256880431108 (+0.01001691377498734) | > avg_loss_0: 2.679288561915008 (+0.053671846927051625) | > avg_grad_norm_0: tensor(214.4846, device='cuda:0') (tensor(-28.0527, device='cuda:0')) | > avg_loss_gen: 2.0896858967525858 (-0.06361862639306315) | > avg_loss_kl: 2.1236383679886943 (-0.17081285362512189) | > avg_loss_feat: 3.484456183205188 (-0.2619005122654876) | > avg_loss_mel: 17.281737045503004 (-0.5997561199564281) | > avg_loss_duration: 1.5622147086640477 (-0.016313452116200722) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.541732277668697 (-1.112401559319295) | > avg_grad_norm_1: tensor(6976.1099, device='cuda:0') (tensor(-860.5132, device='cuda:0'))  > EPOCH: 1042/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:57:23)   --> TIME: 2025-02-25 18:57:36 -- STEP: 25/72 -- GLOBAL_STEP: 1075050 | > loss_disc: 2.84778094291687 (2.641457462310791) | > loss_disc_real_0: 0.34657514095306396 (0.13973392486572267) | > loss_disc_real_1: 0.11644213646650314 (0.2101183596253395) | > loss_disc_real_2: 0.2478283941745758 (0.23226190984249115) | > loss_disc_real_3: 0.2627113163471222 (0.237819402217865) | > loss_disc_real_4: 0.2786497175693512 (0.24437863051891326) | > loss_disc_real_5: 0.26430439949035645 (0.24908601224422455) | > loss_0: 2.84778094291687 (2.641457462310791) | > grad_norm_0: tensor(586.2896, device='cuda:0') (tensor(153.9588, device='cuda:0')) | > loss_gen: 2.1963164806365967 (2.0534619331359862) | > loss_kl: 2.2187349796295166 (1.9646341180801392) | > loss_feat: 3.881821632385254 (3.701323375701904) | > loss_mel: 17.304677963256836 (17.073539047241205) | > loss_duration: 1.5295606851577759 (1.5059262132644653) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.13111114501953 (26.298884506225587) | > grad_norm_1: tensor(9640.3486, device='cuda:0') (tensor(6073.1172, device='cuda:0')) | > current_lr_0: 0.00017557375749514175 | > current_lr_1: 0.00017557375749514175 | > step_time: 0.4478 (0.43629932403564453) | > loader_time: 0.0077 (0.006959104537963867) --> EVAL PERFORMANCE | > avg_loader_time: 0.008696895250132388 (+8.619671136560897e-05) | > avg_step_time: 0.4542675958552831 (-0.0022412588898564345) | > avg_loss_disc: 2.652476257001849 (-0.02681230491315878) | > avg_loss_disc_real_0: 0.12405814902043678 (-0.07324439056322608) | > avg_loss_disc_real_1: 0.2219750544344875 (-0.0006883900979874902) | > avg_loss_disc_real_2: 0.24380816309384898 (+0.005513858207514599) | > avg_loss_disc_real_3: 0.23651520068376836 (+0.0069764860918823945) | > avg_loss_disc_real_4: 0.24825206356988827 (+0.002185138388418828) | > avg_loss_disc_real_5: 0.2535364619443115 (+0.007210773901200701) | > avg_loss_0: 2.652476257001849 (-0.02681230491315878) | > avg_grad_norm_0: tensor(178.1573, device='cuda:0') (tensor(-36.3273, device='cuda:0')) | > avg_loss_gen: 2.0725647076754496 (-0.017121189077136112) | > avg_loss_kl: 2.3172328807938265 (+0.19359451280513218) | > avg_loss_feat: 3.7220089234096903 (+0.23755274020450212) | > avg_loss_mel: 18.520211904821263 (+1.2384748593182593) | > avg_loss_duration: 1.6599323010780442 (+0.09771759241399658) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.291950628791056 (+1.75021835112236) | > avg_grad_norm_1: tensor(6491.5688, device='cuda:0') (tensor(-484.5410, device='cuda:0'))  > EPOCH: 1043/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:58:00)   --> TIME: 2025-02-25 18:58:03 -- STEP: 3/72 -- GLOBAL_STEP: 1075100 | > loss_disc: 2.446181297302246 (2.4272662003835044) | > loss_disc_real_0: 0.1783359944820404 (0.1912415623664856) | > loss_disc_real_1: 0.24111662805080414 (0.2203737994035085) | > loss_disc_real_2: 0.24542741477489471 (0.23305199046929678) | > loss_disc_real_3: 0.22021371126174927 (0.21555573244889578) | > loss_disc_real_4: 0.21372468769550323 (0.20885586738586426) | > loss_disc_real_5: 0.22501543164253235 (0.2243770013252894) | > loss_0: 2.446181297302246 (2.4272662003835044) | > grad_norm_0: tensor(218.7302, device='cuda:0') (tensor(316.8193, device='cuda:0')) | > loss_gen: 2.7676889896392822 (2.731282393137614) | > loss_kl: 2.446835517883301 (2.5580904483795166) | > loss_feat: 4.250363349914551 (4.180431604385376) | > loss_mel: 20.298065185546875 (20.678266525268555) | > loss_duration: 1.6219831705093384 (1.6112934748331706) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.38493537902832 (31.75936444600423) | > grad_norm_1: tensor(2706.2947, device='cuda:0') (tensor(2909.8359, device='cuda:0')) | > current_lr_0: 0.00017555181077545486 | > current_lr_1: 0.00017555181077545486 | > step_time: 0.4265 (0.42694687843322754) | > loader_time: 0.0057 (0.006348927815755208)  --> TIME: 2025-02-25 18:58:26 -- STEP: 53/72 -- GLOBAL_STEP: 1075150 | > loss_disc: 2.464456081390381 (2.6044701495260556) | > loss_disc_real_0: 0.06549295783042908 (0.10997675081609554) | > loss_disc_real_1: 0.21343553066253662 (0.21492372537558935) | > loss_disc_real_2: 0.23545587062835693 (0.2453842820986262) | > loss_disc_real_3: 0.20577386021614075 (0.21939343143746537) | > loss_disc_real_4: 0.22792813181877136 (0.24084160884596267) | > loss_disc_real_5: 0.23304618895053864 (0.2451609567088901) | > loss_0: 2.464456081390381 (2.6044701495260556) | > grad_norm_0: tensor(148.0198, device='cuda:0') (tensor(186.3069, device='cuda:0')) | > loss_gen: 2.189136505126953 (2.1216909255621568) | > loss_kl: 2.2600412368774414 (2.1435469029084686) | > loss_feat: 3.825474500656128 (3.7455401240654713) | > loss_mel: 15.925260543823242 (17.58682038649073) | > loss_duration: 1.6023180484771729 (1.5611769253352903) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.802230834960938 (27.158775149651294) | > grad_norm_1: tensor(3315.8735, device='cuda:0') (tensor(5844.6699, device='cuda:0')) | > current_lr_0: 0.00017555181077545486 | > current_lr_1: 0.00017555181077545486 | > step_time: 0.4457 (0.4395951235069419) | > loader_time: 0.0094 (0.007827457391990806) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861797534244161 (-7.891990769077865e-05) | > avg_step_time: 0.45388992067793726 (-0.00037767517734582645) | > avg_loss_disc: 2.6242062978341543 (-0.028269959167694925) | > avg_loss_disc_real_0: 0.10328335021163375 (-0.020774798808803036) | > avg_loss_disc_real_1: 0.22127951996427186 (-0.0006955344702156419) | > avg_loss_disc_real_2: 0.25107603047935056 (+0.007267867385501586) | > avg_loss_disc_real_3: 0.22299578880340282 (-0.01351941188036554) | > avg_loss_disc_real_4: 0.23974683335129643 (-0.008505230218591842) | > avg_loss_disc_real_5: 0.24536877870559692 (-0.008167683238714585) | > avg_loss_0: 2.6242062978341543 (-0.028269959167694925) | > avg_grad_norm_0: tensor(188.1395, device='cuda:0') (+tensor(9.9822, device='cuda:0')) | > avg_loss_gen: 2.1070245924130298 (+0.034459884737580104) | > avg_loss_kl: 2.1467974118783437 (-0.17043546891548278) | > avg_loss_feat: 3.6803345512336407 (-0.04167437217604952) | > avg_loss_mel: 17.45572288943008 (-1.0644890153911817) | > avg_loss_duration: 1.5719547036667945 (-0.08797759741124977) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.96183408817775 (-1.3301165406133073) | > avg_grad_norm_1: tensor(5647.2129, device='cuda:0') (tensor(-844.3560, device='cuda:0'))  > EPOCH: 1044/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:58:37)   --> TIME: 2025-02-25 18:58:53 -- STEP: 31/72 -- GLOBAL_STEP: 1075200 | > loss_disc: 2.563232660293579 (2.6489932767806517) | > loss_disc_real_0: 0.08201819658279419 (0.12091329615683327) | > loss_disc_real_1: 0.23803207278251648 (0.2290961160775154) | > loss_disc_real_2: 0.23387311398983002 (0.24378532075112866) | > loss_disc_real_3: 0.25205835700035095 (0.24346034228801727) | > loss_disc_real_4: 0.2600790560245514 (0.24951900157236284) | > loss_disc_real_5: 0.25019434094429016 (0.2554920046560226) | > loss_0: 2.563232660293579 (2.6489932767806517) | > grad_norm_0: tensor(49.6280, device='cuda:0') (tensor(168.2876, device='cuda:0')) | > loss_gen: 2.2414815425872803 (2.080288364041236) | > loss_kl: 2.2449395656585693 (1.981777952563378) | > loss_feat: 3.8873696327209473 (3.6204241091205227) | > loss_mel: 18.817819595336914 (16.936675133243686) | > loss_duration: 1.5601812601089478 (1.5151027748661656) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.75179100036621 (26.134268422280588) | > grad_norm_1: tensor(7855.3867, device='cuda:0') (tensor(5849.8745, device='cuda:0')) | > current_lr_0: 0.00017552986679910792 | > current_lr_1: 0.00017552986679910792 | > step_time: 0.4451 (0.4408550339360391) | > loader_time: 0.0085 (0.00721738415379678) --> EVAL PERFORMANCE | > avg_loader_time: 0.00890335566560987 (+0.0002853803231682595) | > avg_step_time: 0.4576797586091807 (+0.003789837931243434) | > avg_loss_disc: 2.6208293001416703 (-0.0033769976924840073) | > avg_loss_disc_real_0: 0.09527023372725703 (-0.00801311648437672) | > avg_loss_disc_real_1: 0.24935538206301955 (+0.028075862098747684) | > avg_loss_disc_real_2: 0.23971243916263044 (-0.011363591316720129) | > avg_loss_disc_real_3: 0.24112272745286914 (+0.018126938649466323) | > avg_loss_disc_real_4: 0.2462125831926373 (+0.006465749841340879) | > avg_loss_disc_real_5: 0.24999263085110102 (+0.004623852145504093) | > avg_loss_0: 2.6208293001416703 (-0.0033769976924840073) | > avg_grad_norm_0: tensor(219.7706, device='cuda:0') (+tensor(31.6311, device='cuda:0')) | > avg_loss_gen: 2.117532599140221 (+0.010508006727191432) | > avg_loss_kl: 2.190887877638911 (+0.04409046576056719) | > avg_loss_feat: 3.74039480383967 (+0.06006025260602943) | > avg_loss_mel: 17.540364775859132 (+0.08464188642905057) | > avg_loss_duration: 1.5659010712529573 (-0.006053632413837162) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.15508122511313 (+0.19324713693537987) | > avg_grad_norm_1: tensor(8065.9473, device='cuda:0') (+tensor(2418.7344, device='cuda:0'))  > EPOCH: 1045/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:59:14)   --> TIME: 2025-02-25 18:59:20 -- STEP: 9/72 -- GLOBAL_STEP: 1075250 | > loss_disc: 2.574090003967285 (2.505375941594442) | > loss_disc_real_0: 0.13571825623512268 (0.09520633021990459) | > loss_disc_real_1: 0.2042795568704605 (0.22624596291118199) | > loss_disc_real_2: 0.2247074842453003 (0.21997728612687853) | > loss_disc_real_3: 0.21836991608142853 (0.2291162825293011) | > loss_disc_real_4: 0.2289176881313324 (0.23275283806853825) | > loss_disc_real_5: 0.24221280217170715 (0.2399694820245107) | > loss_0: 2.574090003967285 (2.505375941594442) | > grad_norm_0: tensor(502.6145, device='cuda:0') (tensor(336.4980, device='cuda:0')) | > loss_gen: 2.116623878479004 (2.1834584871927896) | > loss_kl: 2.198113441467285 (1.9745427370071411) | > loss_feat: 3.858128070831299 (3.9018792841169567) | > loss_mel: 17.693998336791992 (17.296118206448025) | > loss_duration: 1.5272948741912842 (1.520350906583998) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.39415740966797 (26.876349131266277) | > grad_norm_1: tensor(9757.5283, device='cuda:0') (tensor(9299.6143, device='cuda:0')) | > current_lr_0: 0.00017550792556575803 | > current_lr_1: 0.00017550792556575803 | > step_time: 0.433 (0.42943146493699813) | > loader_time: 0.0056 (0.006279018190171983)  --> TIME: 2025-02-25 18:59:43 -- STEP: 59/72 -- GLOBAL_STEP: 1075300 | > loss_disc: 2.3392300605773926 (2.51424812058271) | > loss_disc_real_0: 0.09139058738946915 (0.09037500929276822) | > loss_disc_real_1: 0.18748633563518524 (0.21701312090380717) | > loss_disc_real_2: 0.21585485339164734 (0.228408283853935) | > loss_disc_real_3: 0.2154877483844757 (0.22908975183963776) | > loss_disc_real_4: 0.2102503478527069 (0.2388136854111138) | > loss_disc_real_5: 0.22106613218784332 (0.2408467066490044) | > loss_0: 2.3392300605773926 (2.51424812058271) | > grad_norm_0: tensor(146.7900, device='cuda:0') (tensor(331.8322, device='cuda:0')) | > loss_gen: 2.4673097133636475 (2.177768436528868) | > loss_kl: 2.1061997413635254 (2.0018952559616605) | > loss_feat: 4.6972150802612305 (3.9313458426524015) | > loss_mel: 17.087526321411133 (16.941782789715266) | > loss_duration: 1.576845645904541 (1.5414161177004797) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.935096740722656 (26.59420824859102) | > grad_norm_1: tensor(8452.8193, device='cuda:0') (tensor(10468.3848, device='cuda:0')) | > current_lr_0: 0.00017550792556575803 | > current_lr_1: 0.00017550792556575803 | > step_time: 0.4673 (0.443524550583403) | > loader_time: 0.01 (0.007952358763096695) --> EVAL PERFORMANCE | > avg_loader_time: 0.008533699411741445 (-0.000369656253868424) | > avg_step_time: 0.4525986355794987 (-0.005081123029682) | > avg_loss_disc: 2.5208172697416495 (-0.10001203040002071) | > avg_loss_disc_real_0: 0.09103888965828318 (-0.004231344068973852) | > avg_loss_disc_real_1: 0.21865890269548122 (-0.030696479367538332) | > avg_loss_disc_real_2: 0.22995376565926512 (-0.009758673503365317) | > avg_loss_disc_real_3: 0.23114980673286276 (-0.009972920720006384) | > avg_loss_disc_real_4: 0.24037155215169342 (-0.005841031040943884) | > avg_loss_disc_real_5: 0.24241072632057567 (-0.007581904530525346) | > avg_loss_0: 2.5208172697416495 (-0.10001203040002071) | > avg_grad_norm_0: tensor(331.2713, device='cuda:0') (+tensor(111.5007, device='cuda:0')) | > avg_loss_gen: 2.180539079115424 (+0.06300647997520281) | > avg_loss_kl: 2.024994752776455 (-0.165893124862456) | > avg_loss_feat: 3.9064179977900544 (+0.16602319395038423) | > avg_loss_mel: 16.928331267665808 (-0.6120335081933241) | > avg_loss_duration: 1.5511655370954056 (-0.014735534157551688) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.591448501801825 (-0.5636327233113043) | > avg_grad_norm_1: tensor(10501.7822, device='cuda:0') (+tensor(2435.8350, device='cuda:0'))  > EPOCH: 1046/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 18:59:51)   --> TIME: 2025-02-25 19:00:10 -- STEP: 37/72 -- GLOBAL_STEP: 1075350 | > loss_disc: 2.6083333492279053 (2.532757933075364) | > loss_disc_real_0: 0.17186032235622406 (0.09277354754708908) | > loss_disc_real_1: 0.20365417003631592 (0.2151612470278869) | > loss_disc_real_2: 0.23517651855945587 (0.2287606931216008) | > loss_disc_real_3: 0.23210158944129944 (0.23091148363577352) | > loss_disc_real_4: 0.2650117576122284 (0.24448759330285563) | > loss_disc_real_5: 0.26038235425949097 (0.24537855547827644) | > loss_0: 2.6083333492279053 (2.532757933075364) | > grad_norm_0: tensor(331.3225, device='cuda:0') (tensor(314.9425, device='cuda:0')) | > loss_gen: 2.2013449668884277 (2.193951207238274) | > loss_kl: 2.035254716873169 (1.9419235023292336) | > loss_feat: 3.4505600929260254 (3.93897017272743) | > loss_mel: 16.41073989868164 (16.782677160727015) | > loss_duration: 1.5544217824935913 (1.5097712439459723) | > amp_scaler: 8.0 (8.216216216216216) | > loss_1: 25.65232276916504 (26.3672933836241) | > grad_norm_1: tensor(13379.2852, device='cuda:0') (tensor(9612.5088, device='cuda:0')) | > current_lr_0: 0.0001754859870750623 | > current_lr_1: 0.0001754859870750623 | > step_time: 0.4414 (0.4321870095021016) | > loader_time: 0.0088 (0.007238922892390071) --> EVAL PERFORMANCE | > avg_loader_time: 0.008677697517502476 (+0.00014399810576103123) | > avg_step_time: 0.4495909247599857 (-0.0030077108195130098) | > avg_loss_disc: 2.5542289975663306 (+0.03341172782468105) | > avg_loss_disc_real_0: 0.09885131205681344 (+0.007812422398530264) | > avg_loss_disc_real_1: 0.21662604871770025 (-0.0020328539777809684) | > avg_loss_disc_real_2: 0.2293856055803702 (-0.0005681600788949304) | > avg_loss_disc_real_3: 0.22989756078787252 (-0.0012522459449902379) | > avg_loss_disc_real_4: 0.24347231526609878 (+0.0031007631144053527) | > avg_loss_disc_real_5: 0.24351525012875946 (+0.0011045238081837938) | > avg_loss_0: 2.5542289975663306 (+0.03341172782468105) | > avg_grad_norm_0: tensor(316.1499, device='cuda:0') (tensor(-15.1214, device='cuda:0')) | > avg_loss_gen: 2.146798088516987 (-0.03374099059843694) | > avg_loss_kl: 1.9934568824902388 (-0.03153787028621613) | > avg_loss_feat: 3.8494549066248074 (-0.056963091165247004) | > avg_loss_mel: 16.683183065602478 (-0.24514820206333) | > avg_loss_duration: 1.543923557644159 (-0.007241979451246605) | > avg_amp_scaler: 8.112676056338028 (+0.11267605633802802) | > avg_loss_1: 26.216816485767634 (-0.37463201603419094) | > avg_grad_norm_1: tensor(9291.6377, device='cuda:0') (tensor(-1210.1445, device='cuda:0'))  > EPOCH: 1047/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:00:28)   --> TIME: 2025-02-25 19:00:36 -- STEP: 15/72 -- GLOBAL_STEP: 1075400 | > loss_disc: 2.450200080871582 (2.5399650732676187) | > loss_disc_real_0: 0.06342557072639465 (0.10410742610692977) | > loss_disc_real_1: 0.2343927025794983 (0.21632974247137707) | > loss_disc_real_2: 0.21682190895080566 (0.23008480966091155) | > loss_disc_real_3: 0.22859744727611542 (0.22822922170162202) | > loss_disc_real_4: 0.23741087317466736 (0.24108570615450542) | > loss_disc_real_5: 0.24955859780311584 (0.24751202166080474) | > loss_0: 2.450200080871582 (2.5399650732676187) | > grad_norm_0: tensor(85.1085, device='cuda:0') (tensor(286.6508, device='cuda:0')) | > loss_gen: 2.3550758361816406 (2.2399765650431314) | > loss_kl: 1.9530725479125977 (1.8761510133743287) | > loss_feat: 4.309690952301025 (4.031463734308878) | > loss_mel: 17.55615997314453 (16.625332514444988) | > loss_duration: 1.501680612564087 (1.491168729464213) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.67568016052246 (26.264093017578126) | > grad_norm_1: tensor(6506.8564, device='cuda:0') (tensor(9738.1143, device='cuda:0')) | > current_lr_0: 0.00017546405132667792 | > current_lr_1: 0.00017546405132667792 | > step_time: 0.4367 (0.43137667973836263) | > loader_time: 0.0109 (0.00694425900777181)  --> TIME: 2025-02-25 19:01:00 -- STEP: 65/72 -- GLOBAL_STEP: 1075450 | > loss_disc: 2.561983823776245 (2.6026908214275655) | > loss_disc_real_0: 0.05499577522277832 (0.1269580768278012) | > loss_disc_real_1: 0.278691828250885 (0.22354876146866726) | > loss_disc_real_2: 0.218964084982872 (0.23993526720083677) | > loss_disc_real_3: 0.2086046189069748 (0.237006191106943) | > loss_disc_real_4: 0.226448655128479 (0.25121409801336436) | > loss_disc_real_5: 0.2253485471010208 (0.24576140275368324) | > loss_0: 2.561983823776245 (2.6026908214275655) | > grad_norm_0: tensor(105.6639, device='cuda:0') (tensor(320.3436, device='cuda:0')) | > loss_gen: 1.9668986797332764 (2.185879100286043) | > loss_kl: 2.2585577964782715 (1.9729243663641123) | > loss_feat: 3.591017484664917 (3.812207871216994) | > loss_mel: 16.36107063293457 (16.53117810762846) | > loss_duration: 1.6236933469772339 (1.5366654671155489) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.801237106323242 (26.0388550391564) | > grad_norm_1: tensor(7653.4497, device='cuda:0') (tensor(8685.3037, device='cuda:0')) | > current_lr_0: 0.00017546405132667792 | > current_lr_1: 0.00017546405132667792 | > step_time: 0.5302 (0.45195562289311336) | > loader_time: 0.0116 (0.008294321940495416) --> EVAL PERFORMANCE | > avg_loader_time: 0.00858449935913086 (-9.31981583716171e-05) | > avg_step_time: 0.45805414629654145 (+0.008463221536555765) | > avg_loss_disc: 2.611942798318997 (+0.05771380075266652) | > avg_loss_disc_real_0: 0.12246446657768438 (+0.02361315452087094) | > avg_loss_disc_real_1: 0.2239766389551297 (+0.007350590237429455) | > avg_loss_disc_real_2: 0.23727886483702862 (+0.00789325925665843) | > avg_loss_disc_real_3: 0.2371657173818266 (+0.007268156593954078) | > avg_loss_disc_real_4: 0.2504615800481447 (+0.006989264782045951) | > avg_loss_disc_real_5: 0.2449946749798009 (+0.0014794248510414387) | > avg_loss_0: 2.611942798318997 (+0.05771380075266652) | > avg_grad_norm_0: tensor(326.0449, device='cuda:0') (+tensor(9.8950, device='cuda:0')) | > avg_loss_gen: 2.16450729336537 (+0.01770920484838312) | > avg_loss_kl: 1.9877433625745102 (-0.005713519915728549) | > avg_loss_feat: 3.821454924596867 (-0.027999982027940185) | > avg_loss_mel: 16.581452436850107 (-0.10173062875237093) | > avg_loss_duration: 1.542907651041595 (-0.0010159066025638985) | > avg_amp_scaler: 8.0 (-0.11267605633802802) | > avg_loss_1: 26.098065819538814 (-0.11875066622882002) | > avg_grad_norm_1: tensor(8561.8564, device='cuda:0') (tensor(-729.7812, device='cuda:0'))  > EPOCH: 1048/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:01:05)   --> TIME: 2025-02-25 19:01:26 -- STEP: 43/72 -- GLOBAL_STEP: 1075500 | > loss_disc: 2.7811989784240723 (2.6530493636463963) | > loss_disc_real_0: 0.31368640065193176 (0.13577783982767616) | > loss_disc_real_1: 0.24238575994968414 (0.2293398638450822) | > loss_disc_real_2: 0.2284708321094513 (0.2284141522507335) | > loss_disc_real_3: 0.27007627487182617 (0.20934500597244085) | > loss_disc_real_4: 0.24245613813400269 (0.23684651699177053) | > loss_disc_real_5: 0.24154631793498993 (0.23537703031717344) | > loss_0: 2.7811989784240723 (2.6530493636463963) | > grad_norm_0: tensor(833.6514, device='cuda:0') (tensor(303.1695, device='cuda:0')) | > loss_gen: 2.261827230453491 (2.0731539005456967) | > loss_kl: 2.117919445037842 (2.334962506626927) | > loss_feat: 3.693540096282959 (3.805303462716036) | > loss_mel: 16.407880783081055 (18.491240922794784) | > loss_duration: 1.5462157726287842 (1.5504237662914186) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.02738380432129 (28.255084459171737) | > grad_norm_1: tensor(12812.9014, device='cuda:0') (tensor(8650.5986, device='cuda:0')) | > current_lr_0: 0.00017544211832026208 | > current_lr_1: 0.00017544211832026208 | > step_time: 0.4744 (0.43666875085165335) | > loader_time: 0.0095 (0.0074090236841246136) --> EVAL PERFORMANCE | > avg_loader_time: 0.008561352608909072 (-2.31467502217874e-05) | > avg_step_time: 0.45018095701513156 (-0.007873189281409898) | > avg_loss_disc: 2.651015415997573 (+0.03907261767857584) | > avg_loss_disc_real_0: 0.12144614855798203 (-0.001018318019702355) | > avg_loss_disc_real_1: 0.22985535660679912 (+0.005878717651669418) | > avg_loss_disc_real_2: 0.2273118447669795 (-0.009967020070049126) | > avg_loss_disc_real_3: 0.21885854740377883 (-0.018307169978047766) | > avg_loss_disc_real_4: 0.24041328316843005 (-0.010048296879714674) | > avg_loss_disc_real_5: 0.23860759823255137 (-0.0063870767472495316) | > avg_loss_0: 2.651015415997573 (+0.03907261767857584) | > avg_grad_norm_0: tensor(300.2130, device='cuda:0') (tensor(-25.8319, device='cuda:0')) | > avg_loss_gen: 2.0415060033260946 (-0.12300129003927562) | > avg_loss_kl: 2.2821937530813083 (+0.2944503905067981) | > avg_loss_feat: 3.721131633704817 (-0.10032329089205039) | > avg_loss_mel: 17.925612610830388 (+1.3441601739802813) | > avg_loss_duration: 1.5678702626429812 (+0.02496261160138613) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.538314308918697 (+1.4402484893798828) | > avg_grad_norm_1: tensor(8681.1904, device='cuda:0') (+tensor(119.3340, device='cuda:0'))  > EPOCH: 1049/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:01:41)   --> TIME: 2025-02-25 19:01:53 -- STEP: 21/72 -- GLOBAL_STEP: 1075550 | > loss_disc: 2.4647345542907715 (2.531447649002075) | > loss_disc_real_0: 0.11922342330217361 (0.11108260584019479) | > loss_disc_real_1: 0.19861102104187012 (0.21016687793391092) | > loss_disc_real_2: 0.2399941384792328 (0.22623327800205775) | > loss_disc_real_3: 0.2154652625322342 (0.22665706134977795) | > loss_disc_real_4: 0.23209832608699799 (0.24415907973334902) | > loss_disc_real_5: 0.21006923913955688 (0.24835650764760517) | > loss_0: 2.4647345542907715 (2.531447649002075) | > grad_norm_0: tensor(348.8979, device='cuda:0') (tensor(198.3057, device='cuda:0')) | > loss_gen: 2.1839797496795654 (2.2269199518930343) | > loss_kl: 2.0832767486572266 (2.561841022400629) | > loss_feat: 4.01006555557251 (4.1356861137208485) | > loss_mel: 17.529830932617188 (19.12240200950986) | > loss_duration: 1.5407294034957886 (1.5432080654870897) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.347881317138672 (29.590056918916247) | > grad_norm_1: tensor(8259.3232, device='cuda:0') (tensor(7810.9883, device='cuda:0')) | > current_lr_0: 0.00017542018805547204 | > current_lr_1: 0.00017542018805547204 | > step_time: 0.4293 (0.43306770778837655) | > loader_time: 0.0066 (0.006518784023466564)  --> TIME: 2025-02-25 19:02:17 -- STEP: 71/72 -- GLOBAL_STEP: 1075600 | > loss_disc: 2.3881328105926514 (2.586095111470827) | > loss_disc_real_0: 0.05673117935657501 (0.10928896914275599) | > loss_disc_real_1: 0.18049104511737823 (0.22417517866886838) | > loss_disc_real_2: 0.22977761924266815 (0.24186928864096252) | > loss_disc_real_3: 0.20906788110733032 (0.2318747729902536) | > loss_disc_real_4: 0.24867919087409973 (0.24450267683452284) | > loss_disc_real_5: 0.23722799122333527 (0.23979995032431375) | > loss_0: 2.3881328105926514 (2.586095111470827) | > grad_norm_0: tensor(218.0312, device='cuda:0') (tensor(245.8820, device='cuda:0')) | > loss_gen: 2.2773349285125732 (2.1825231397655647) | > loss_kl: 2.0030250549316406 (2.3202433233529747) | > loss_feat: 3.8681795597076416 (3.944491127846946) | > loss_mel: 15.957196235656738 (17.761693578370853) | > loss_duration: 1.5619014501571655 (1.56983207816809) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.667634963989258 (27.778782992295817) | > grad_norm_1: tensor(9149.5195, device='cuda:0') (tensor(8455.2754, device='cuda:0')) | > current_lr_0: 0.00017542018805547204 | > current_lr_1: 0.00017542018805547204 | > step_time: 0.5002 (0.45196604392897916) | > loader_time: 0.0082 (0.008603767609932053) --> EVAL PERFORMANCE | > avg_loader_time: 0.008603767609932053 (+4.241500102298082e-05) | > avg_step_time: 0.45196604392897916 (+0.0017850869138476) | > avg_loss_disc: 2.586095111470827 (-0.06492030452674591) | > avg_loss_disc_real_0: 0.10928896914275599 (-0.012157179415226038) | > avg_loss_disc_real_1: 0.22417517866886838 (-0.0056801779379307415) | > avg_loss_disc_real_2: 0.24186928864096252 (+0.014557443873983028) | > avg_loss_disc_real_3: 0.2318747729902536 (+0.013016225586474778) | > avg_loss_disc_real_4: 0.24450267683452284 (+0.004089393666092789) | > avg_loss_disc_real_5: 0.23979995032431375 (+0.001192352091762383) | > avg_loss_0: 2.586095111470827 (-0.06492030452674591) | > avg_grad_norm_0: tensor(245.8820, device='cuda:0') (tensor(-54.3310, device='cuda:0')) | > avg_loss_gen: 2.1825231397655647 (+0.14101713643947011) | > avg_loss_kl: 2.3202433233529747 (+0.03804957027166633) | > avg_loss_feat: 3.944491127846946 (+0.2233594941421293) | > avg_loss_mel: 17.761693578370853 (-0.16391903245953543) | > avg_loss_duration: 1.56983207816809 (+0.0019618155251086566) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.778782992295817 (+0.24046868337712013) | > avg_grad_norm_1: tensor(8455.2754, device='cuda:0') (tensor(-225.9150, device='cuda:0'))  > EPOCH: 1050/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:02:18)   --> TIME: 2025-02-25 19:02:42 -- STEP: 49/72 -- GLOBAL_STEP: 1075650 | > loss_disc: 2.4864063262939453 (2.537674412435416) | > loss_disc_real_0: 0.10835331678390503 (0.08623718576771873) | > loss_disc_real_1: 0.20266462862491608 (0.221451038000535) | > loss_disc_real_2: 0.26022616028785706 (0.23794872420174734) | > loss_disc_real_3: 0.24536773562431335 (0.2291540162903922) | > loss_disc_real_4: 0.2371971607208252 (0.23445751989374355) | > loss_disc_real_5: 0.2473657727241516 (0.24498518754024895) | > loss_0: 2.4864063262939453 (2.537674412435416) | > grad_norm_0: tensor(130.2476, device='cuda:0') (tensor(277.6391, device='cuda:0')) | > loss_gen: 2.4807775020599365 (2.2081720756024725) | > loss_kl: 1.9740042686462402 (2.0041120952489426) | > loss_feat: 4.244494915008545 (4.02152969399277) | > loss_mel: 17.22900390625 (16.852002221710826) | > loss_duration: 1.5931504964828491 (1.528113849309026) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.52143096923828 (26.613930059939015) | > grad_norm_1: tensor(5465.7490, device='cuda:0') (tensor(9460.5215, device='cuda:0')) | > current_lr_0: 0.0001753982605319651 | > current_lr_1: 0.0001753982605319651 | > step_time: 0.4378 (0.4391225844013448) | > loader_time: 0.0097 (0.007626674613174127) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577964675258584 (-2.5802934673468947e-05) | > avg_step_time: 0.4541794246351215 (+0.002213380706142354) | > avg_loss_disc: 2.5320278187872667 (-0.054067292683560364) | > avg_loss_disc_real_0: 0.08953660112661375 (-0.019752368016142233) | > avg_loss_disc_real_1: 0.21897030212509802 (-0.005204876543770359) | > avg_loss_disc_real_2: 0.2376485137452542 (-0.0042207748957083235) | > avg_loss_disc_real_3: 0.22730817320481153 (-0.004566599785442077) | > avg_loss_disc_real_4: 0.23682604562228834 (-0.007676631212234497) | > avg_loss_disc_real_5: 0.24530941331890269 (+0.005509462994588932) | > avg_loss_0: 2.5320278187872667 (-0.054067292683560364) | > avg_grad_norm_0: tensor(290.2352, device='cuda:0') (+tensor(44.3532, device='cuda:0')) | > avg_loss_gen: 2.2121687523076226 (+0.029645612542057886) | > avg_loss_kl: 2.0215489494968466 (-0.29869437385612807) | > avg_loss_feat: 3.9828558505420952 (+0.03836472269514912) | > avg_loss_mel: 16.809794748333132 (-0.9518988300377202) | > avg_loss_duration: 1.5467276908981968 (-0.023104387269893056) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.573095993256906 (-1.2056869990389103) | > avg_grad_norm_1: tensor(8992.2002, device='cuda:0') (+tensor(536.9248, device='cuda:0'))  > EPOCH: 1051/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:02:55)   --> TIME: 2025-02-25 19:03:09 -- STEP: 27/72 -- GLOBAL_STEP: 1075700 | > loss_disc: 2.6397767066955566 (2.5631602340274386) | > loss_disc_real_0: 0.18621958792209625 (0.13682193143500226) | > loss_disc_real_1: 0.2196490466594696 (0.21805448223043372) | > loss_disc_real_2: 0.21175168454647064 (0.22008655965328217) | > loss_disc_real_3: 0.2311975061893463 (0.2344643904103173) | > loss_disc_real_4: 0.22992072999477386 (0.24116474555598366) | > loss_disc_real_5: 0.22456447780132294 (0.24155208358058222) | > loss_0: 2.6397767066955566 (2.5631602340274386) | > grad_norm_0: tensor(65.6545, device='cuda:0') (tensor(294.7007, device='cuda:0')) | > loss_gen: 1.9092066287994385 (2.188694150359542) | > loss_kl: 1.976090908050537 (1.9258787676140114) | > loss_feat: 3.1002557277679443 (3.8260393054397017) | > loss_mel: 15.509720802307129 (16.845611536944354) | > loss_duration: 1.5077557563781738 (1.5012576977411907) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.003028869628906 (26.287481025413232) | > grad_norm_1: tensor(5869.6987, device='cuda:0') (tensor(7127.2466, device='cuda:0')) | > current_lr_0: 0.0001753763357493986 | > current_lr_1: 0.0001753763357493986 | > step_time: 0.4343 (0.43671127601906107) | > loader_time: 0.008 (0.007279245941727249) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871801712143589 (+0.0001400524461773068) | > avg_step_time: 0.4560968842304928 (+0.001917459595371318) | > avg_loss_disc: 2.6083345077407194 (+0.07630668895345272) | > avg_loss_disc_real_0: 0.13074089316520052 (+0.04120429203858676) | > avg_loss_disc_real_1: 0.22273792221512592 (+0.003767620090027901) | > avg_loss_disc_real_2: 0.230281832562366 (-0.007366681182888191) | > avg_loss_disc_real_3: 0.23464069379047608 (+0.007332520585664554) | > avg_loss_disc_real_4: 0.24290671923630675 (+0.0060806736140184) | > avg_loss_disc_real_5: 0.2441703095822267 (-0.0011391037366759726) | > avg_loss_0: 2.6083345077407194 (+0.07630668895345272) | > avg_grad_norm_0: tensor(263.0538, device='cuda:0') (tensor(-27.1814, device='cuda:0')) | > avg_loss_gen: 2.150354853818115 (-0.06181389848950758) | > avg_loss_kl: 2.0885473523341433 (+0.06699840283729674) | > avg_loss_feat: 3.741109404765384 (-0.24174644577671112) | > avg_loss_mel: 17.1760473385663 (+0.3662525902331666) | > avg_loss_duration: 1.553008302836351 (+0.0062806119381542125) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.709066982000646 (+0.1359709887437397) | > avg_grad_norm_1: tensor(7257.3555, device='cuda:0') (tensor(-1734.8447, device='cuda:0'))  > EPOCH: 1052/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:03:32)   --> TIME: 2025-02-25 19:03:37 -- STEP: 5/72 -- GLOBAL_STEP: 1075750 | > loss_disc: 2.876495599746704 (2.7803993225097656) | > loss_disc_real_0: 0.44288891553878784 (0.29748943597078326) | > loss_disc_real_1: 0.24127011001110077 (0.228048700094223) | > loss_disc_real_2: 0.22517164051532745 (0.23369355797767638) | > loss_disc_real_3: 0.2379854917526245 (0.2371962070465088) | > loss_disc_real_4: 0.2505417764186859 (0.23677252233028412) | > loss_disc_real_5: 0.2341468185186386 (0.21987708508968354) | > loss_0: 2.876495599746704 (2.7803993225097656) | > grad_norm_0: tensor(158.0051, device='cuda:0') (tensor(206.1384, device='cuda:0')) | > loss_gen: 2.2067317962646484 (2.251628565788269) | > loss_kl: 2.046095848083496 (2.01807644367218) | > loss_feat: 3.321085214614868 (3.4725298404693605) | > loss_mel: 17.07810401916504 (17.494506072998046) | > loss_duration: 1.4750754833221436 (1.521384835243225) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.127092361450195 (26.75812530517578) | > grad_norm_1: tensor(6999.9824, device='cuda:0') (tensor(6329.6943, device='cuda:0')) | > current_lr_0: 0.0001753544137074299 | > current_lr_1: 0.0001753544137074299 | > step_time: 0.421 (0.42803163528442384) | > loader_time: 0.0065 (0.0056800365447998045)  --> TIME: 2025-02-25 19:03:59 -- STEP: 55/72 -- GLOBAL_STEP: 1075800 | > loss_disc: 2.9329311847686768 (2.714917018196799) | > loss_disc_real_0: 0.05589184910058975 (0.1736927039243958) | > loss_disc_real_1: 0.18160779774188995 (0.21236780109730632) | > loss_disc_real_2: 0.27026164531707764 (0.2322127813642675) | > loss_disc_real_3: 0.23754923045635223 (0.2373172028498216) | > loss_disc_real_4: 0.23368743062019348 (0.2418942765756087) | > loss_disc_real_5: 0.22819022834300995 (0.2434704157439145) | > loss_0: 2.9329311847686768 (2.714917018196799) | > grad_norm_0: tensor(27.1588, device='cuda:0') (tensor(201.0381, device='cuda:0')) | > loss_gen: 1.8075188398361206 (2.1067559307271777) | > loss_kl: 2.0376341342926025 (2.0820054877888063) | > loss_feat: 3.1867048740386963 (3.81814864765514) | > loss_mel: 16.446435928344727 (17.20921327417547) | > loss_duration: 1.5991235971450806 (1.552770354531028) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.077415466308594 (26.768893987482244) | > grad_norm_1: tensor(6626.2197, device='cuda:0') (tensor(6099.0312, device='cuda:0')) | > current_lr_0: 0.0001753544137074299 | > current_lr_1: 0.0001753544137074299 | > step_time: 0.4586 (0.44166828502308236) | > loader_time: 0.0094 (0.007691825519908558) --> EVAL PERFORMANCE | > avg_loader_time: 0.008454950762466643 (-0.0002630663589692477) | > avg_step_time: 0.4529863344111913 (-0.0031105498193015335) | > avg_loss_disc: 2.6980102062225333 (+0.08967569848181389) | > avg_loss_disc_real_0: 0.17991485604098145 (+0.049173962875780935) | > avg_loss_disc_real_1: 0.21900515079918043 (-0.003732771415945496) | > avg_loss_disc_real_2: 0.23179631648768842 (+0.001514483925322413) | > avg_loss_disc_real_3: 0.23788356697055654 (+0.0032428731800804578) | > avg_loss_disc_real_4: 0.24181840852112838 (-0.00108831071517837) | > avg_loss_disc_real_5: 0.2441482923819985 (-2.201720022820819e-05) | > avg_loss_0: 2.6980102062225333 (+0.08967569848181389) | > avg_grad_norm_0: tensor(210.2367, device='cuda:0') (tensor(-52.8171, device='cuda:0')) | > avg_loss_gen: 2.122846304530828 (-0.027508549287286854) | > avg_loss_kl: 2.0885056156507673 (-4.1736683376036865e-05) | > avg_loss_feat: 3.726639750977637 (-0.014469653787747117) | > avg_loss_mel: 17.138040864971323 (-0.03800647359497589) | > avg_loss_duration: 1.5624546752849096 (+0.009446372448558504) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.63848753378425 (-0.07057944821639595) | > avg_grad_norm_1: tensor(6250.5801, device='cuda:0') (tensor(-1006.7754, device='cuda:0'))  > EPOCH: 1053/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:04:09)   --> TIME: 2025-02-25 19:04:26 -- STEP: 33/72 -- GLOBAL_STEP: 1075850 | > loss_disc: 2.775453805923462 (2.645673650683779) | > loss_disc_real_0: 0.08858780562877655 (0.1699052145986846) | > loss_disc_real_1: 0.2657381296157837 (0.2429507952747923) | > loss_disc_real_2: 0.2119189351797104 (0.23432591557502747) | > loss_disc_real_3: 0.2461654245853424 (0.23570956921938693) | > loss_disc_real_4: 0.2173914611339569 (0.2381854278571678) | > loss_disc_real_5: 0.24294842779636383 (0.25009527486382116) | > loss_0: 2.775453805923462 (2.645673650683779) | > grad_norm_0: tensor(55.2989, device='cuda:0') (tensor(258.1298, device='cuda:0')) | > loss_gen: 1.7273674011230469 (2.198932734402743) | > loss_kl: 1.9601778984069824 (1.9230189431797375) | > loss_feat: 3.3962066173553467 (3.687263683839278) | > loss_mel: 16.848100662231445 (16.9086554556182) | > loss_duration: 1.5202264785766602 (1.507504777474837) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.45207977294922 (26.22537578235973) | > grad_norm_1: tensor(151.0945, device='cuda:0') (tensor(6322.1055, device='cuda:0')) | > current_lr_0: 0.00017533249440571648 | > current_lr_1: 0.00017533249440571648 | > step_time: 0.4471 (0.43173842719106964) | > loader_time: 0.0079 (0.006943457054369377) --> EVAL PERFORMANCE | > avg_loader_time: 0.008504024693663691 (+4.907393119704846e-05) | > avg_step_time: 0.45044867421539736 (-0.0025376601957939338) | > avg_loss_disc: 2.646358678038691 (-0.05165152818384211) | > avg_loss_disc_real_0: 0.15914028636376623 (-0.02077456967721522) | > avg_loss_disc_real_1: 0.23176466789044126 (+0.01275951709126083) | > avg_loss_disc_real_2: 0.2362284045404112 (+0.004432088052722771) | > avg_loss_disc_real_3: 0.2321418519590942 (-0.005741715011462339) | > avg_loss_disc_real_4: 0.24031275825601228 (-0.001505650265116093) | > avg_loss_disc_real_5: 0.249069654186007 (+0.004921361804008484) | > avg_loss_0: 2.646358678038691 (-0.05165152818384211) | > avg_grad_norm_0: tensor(220.0049, device='cuda:0') (+tensor(9.7682, device='cuda:0')) | > avg_loss_gen: 2.1438586745463626 (+0.021012370015534465) | > avg_loss_kl: 2.259630742207379 (+0.17112512655661183) | > avg_loss_feat: 3.65435705386417 (-0.07228269711346691) | > avg_loss_mel: 17.690146325339736 (+0.5521054603684128) | > avg_loss_duration: 1.5830591564447107 (+0.02060448115980118) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.33105194736535 (+0.6925644135810991) | > avg_grad_norm_1: tensor(6933.9409, device='cuda:0') (+tensor(683.3608, device='cuda:0'))  > EPOCH: 1054/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:04:46)   --> TIME: 2025-02-25 19:04:53 -- STEP: 11/72 -- GLOBAL_STEP: 1075900 | > loss_disc: 2.6445858478546143 (2.6186189218000933) | > loss_disc_real_0: 0.0530102401971817 (0.10297875614328818) | > loss_disc_real_1: 0.19874782860279083 (0.24108529497276654) | > loss_disc_real_2: 0.2033344954252243 (0.2672788311134685) | > loss_disc_real_3: 0.25974705815315247 (0.24743829396638004) | > loss_disc_real_4: 0.25514668226242065 (0.24480138448151675) | > loss_disc_real_5: 0.2711695730686188 (0.25441478192806244) | > loss_0: 2.6445858478546143 (2.6186189218000933) | > grad_norm_0: tensor(109.8867, device='cuda:0') (tensor(94.6859, device='cuda:0')) | > loss_gen: 1.9234559535980225 (2.2087837024168535) | > loss_kl: 2.696063995361328 (2.264681480147622) | > loss_feat: 3.8592207431793213 (3.9969750100916084) | > loss_mel: 19.23063087463379 (18.472571979869496) | > loss_duration: 1.6208871603012085 (1.564238429069519) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.330259323120117 (28.50725017894398) | > grad_norm_1: tensor(6945.6768, device='cuda:0') (tensor(5758.5469, device='cuda:0')) | > current_lr_0: 0.00017531057784391576 | > current_lr_1: 0.00017531057784391576 | > step_time: 0.4221 (0.42879605293273926) | > loader_time: 0.0065 (0.006288094954057174)  --> TIME: 2025-02-25 19:05:16 -- STEP: 61/72 -- GLOBAL_STEP: 1075950 | > loss_disc: 2.9486749172210693 (2.7635700976262325) | > loss_disc_real_0: 0.21152251958847046 (0.17530882560083125) | > loss_disc_real_1: 0.2776724100112915 (0.22928491845482685) | > loss_disc_real_2: 0.25318020582199097 (0.25867865613249486) | > loss_disc_real_3: 0.2429026961326599 (0.246548023380217) | > loss_disc_real_4: 0.22632664442062378 (0.25094631045568183) | > loss_disc_real_5: 0.22262513637542725 (0.24876058907782447) | > loss_0: 2.9486749172210693 (2.7635700976262325) | > grad_norm_0: tensor(27.0421, device='cuda:0') (tensor(198.1745, device='cuda:0')) | > loss_gen: 1.7822258472442627 (2.1038984236170024) | > loss_kl: 2.5901193618774414 (2.2864457740158333) | > loss_feat: 2.6164162158966064 (3.60945559720524) | > loss_mel: 18.237606048583984 (17.765845564545177) | > loss_duration: 1.7571548223495483 (1.5755689163677027) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.983522415161133 (27.3412142581627) | > grad_norm_1: tensor(620.3531, device='cuda:0') (tensor(4810.9619, device='cuda:0')) | > current_lr_0: 0.00017531057784391576 | > current_lr_1: 0.00017531057784391576 | > step_time: 0.4533 (0.4423227036585573) | > loader_time: 0.0108 (0.00806347659376801) --> EVAL PERFORMANCE | > avg_loader_time: 0.008521916161120777 (+1.7891467457085974e-05) | > avg_step_time: 0.4505420469901932 (+9.337277479581907e-05) | > avg_loss_disc: 2.772989279787305 (+0.12663060174861362) | > avg_loss_disc_real_0: 0.1874066424936476 (+0.028266356129881376) | > avg_loss_disc_real_1: 0.2301340896478841 (-0.0016305782425571624) | > avg_loss_disc_real_2: 0.2608374525123919 (+0.024609047971980685) | > avg_loss_disc_real_3: 0.24998736108692599 (+0.017845509127831782) | > avg_loss_disc_real_4: 0.25081876298071626 (+0.010506004724703977) | > avg_loss_disc_real_5: 0.24744895310469076 (-0.0016207010813162337) | > avg_loss_0: 2.772989279787305 (+0.12663060174861362) | > avg_grad_norm_0: tensor(208.4416, device='cuda:0') (tensor(-11.5633, device='cuda:0')) | > avg_loss_gen: 2.094676128575499 (-0.04918254597086369) | > avg_loss_kl: 2.3490280433439867 (+0.0893973011366076) | > avg_loss_feat: 3.5433465561396638 (-0.11101049772450633) | > avg_loss_mel: 17.979376081009985 (+0.2892297556702488) | > avg_loss_duration: 1.5963688467589903 (+0.013309690314279532) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.56279566590215 (+0.23174371853680142) | > avg_grad_norm_1: tensor(4503.1211, device='cuda:0') (tensor(-2430.8198, device='cuda:0'))  > EPOCH: 1055/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:05:23)   --> TIME: 2025-02-25 19:05:42 -- STEP: 39/72 -- GLOBAL_STEP: 1076000 | > loss_disc: 2.4760918617248535 (2.5824053654303905) | > loss_disc_real_0: 0.051089540123939514 (0.11746890331881168) | > loss_disc_real_1: 0.16249153017997742 (0.23030859919694754) | > loss_disc_real_2: 0.2513769567012787 (0.22215824707960471) | > loss_disc_real_3: 0.2554246485233307 (0.2251301747866166) | > loss_disc_real_4: 0.2274804562330246 (0.23202215822843406) | > loss_disc_real_5: 0.21953216195106506 (0.23152530078704542) | > loss_0: 2.4760918617248535 (2.5824053654303905) | > grad_norm_0: tensor(146.4830, device='cuda:0') (tensor(208.8080, device='cuda:0')) | > loss_gen: 2.0978970527648926 (2.002813984186222) | > loss_kl: 2.164168119430542 (2.2972891269586024) | > loss_feat: 3.997208833694458 (3.7438214192023644) | > loss_mel: 17.695764541625977 (18.306949664384884) | > loss_duration: 1.5525202751159668 (1.578325861539596) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.507558822631836 (27.92920034359663) | > grad_norm_1: tensor(6943.5425, device='cuda:0') (tensor(4959.3521, device='cuda:0')) | > current_lr_0: 0.00017528866402168527 | > current_lr_1: 0.00017528866402168527 | > step_time: 0.4486 (0.43315413670662123) | > loader_time: 0.0085 (0.0073353266104673725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008665266171307633 (+0.00014335001018685525) | > avg_step_time: 0.4513201209860788 (+0.000778073995885642) | > avg_loss_disc: 2.626627482159037 (-0.14636179762826762) | > avg_loss_disc_real_0: 0.09658486760732993 (-0.09082177488631768) | > avg_loss_disc_real_1: 0.22908021677547777 (-0.001053872872406325) | > avg_loss_disc_real_2: 0.230658298227149 (-0.03017915428524287) | > avg_loss_disc_real_3: 0.23604632704190806 (-0.013941034045017925) | > avg_loss_disc_real_4: 0.24149871439161436 (-0.009320048589101904) | > avg_loss_disc_real_5: 0.24185697545467968 (-0.005591977650011071) | > avg_loss_0: 2.626627482159037 (-0.14636179762826762) | > avg_grad_norm_0: tensor(258.0728, device='cuda:0') (+tensor(49.6312, device='cuda:0')) | > avg_loss_gen: 2.00625748029897 (-0.08841864827652879) | > avg_loss_kl: 2.2068891206257772 (-0.1421389227182095) | > avg_loss_feat: 3.7943797883853105 (+0.2510332322456468) | > avg_loss_mel: 17.708218789436447 (-0.2711572915735374) | > avg_loss_duration: 1.5880835610376278 (-0.008285285721362445) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.303828951338648 (-0.2589667145635026) | > avg_grad_norm_1: tensor(7276.0044, device='cuda:0') (+tensor(2772.8833, device='cuda:0'))  > EPOCH: 1056/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:05:59)   --> TIME: 2025-02-25 19:06:09 -- STEP: 17/72 -- GLOBAL_STEP: 1076050 | > loss_disc: 2.348830223083496 (2.503973399891573) | > loss_disc_real_0: 0.051832109689712524 (0.08072456586010315) | > loss_disc_real_1: 0.21767549216747284 (0.2192307552870582) | > loss_disc_real_2: 0.23831237852573395 (0.2401662951006609) | > loss_disc_real_3: 0.20384998619556427 (0.22778193652629852) | > loss_disc_real_4: 0.2319657802581787 (0.25229966202202964) | > loss_disc_real_5: 0.24185357987880707 (0.2548466158263823) | > loss_0: 2.348830223083496 (2.503973399891573) | > grad_norm_0: tensor(120.9641, device='cuda:0') (tensor(249.1812, device='cuda:0')) | > loss_gen: 2.2853214740753174 (2.240264051100787) | > loss_kl: 1.9799184799194336 (1.9704509833279777) | > loss_feat: 4.396073341369629 (4.139025912565343) | > loss_mel: 17.697208404541016 (17.247732050278607) | > loss_duration: 1.5259400606155396 (1.520746616756215) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.884462356567383 (27.118219824398267) | > grad_norm_1: tensor(4520.5366, device='cuda:0') (tensor(8662.3115, device='cuda:0')) | > current_lr_0: 0.00017526675293868255 | > current_lr_1: 0.00017526675293868255 | > step_time: 0.4191 (0.4271359443664551) | > loader_time: 0.0059 (0.006591656628776999)  --> TIME: 2025-02-25 19:06:32 -- STEP: 67/72 -- GLOBAL_STEP: 1076100 | > loss_disc: 2.422389268875122 (2.5165610313415523) | > loss_disc_real_0: 0.05716998875141144 (0.08070541401185204) | > loss_disc_real_1: 0.21614386141300201 (0.21496824697772068) | > loss_disc_real_2: 0.23348908126354218 (0.23866456191041577) | > loss_disc_real_3: 0.22691629827022552 (0.23120228590360328) | > loss_disc_real_4: 0.24441078305244446 (0.247908181441364) | > loss_disc_real_5: 0.253480464220047 (0.2536727379062283) | > loss_0: 2.422389268875122 (2.5165610313415523) | > grad_norm_0: tensor(197.5164, device='cuda:0') (tensor(284.3570, device='cuda:0')) | > loss_gen: 2.2030625343322754 (2.182749024078027) | > loss_kl: 2.185851573944092 (1.994814758870139) | > loss_feat: 3.7932932376861572 (3.940008711459032) | > loss_mel: 17.022422790527344 (16.8335146832822) | > loss_duration: 1.6372251510620117 (1.5555414459598598) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.841854095458984 (26.506628776664165) | > grad_norm_1: tensor(10024.2021, device='cuda:0') (tensor(9219.4453, device='cuda:0')) | > current_lr_0: 0.00017526675293868255 | > current_lr_1: 0.00017526675293868255 | > step_time: 0.52 (0.44465514439255444) | > loader_time: 0.0125 (0.008379765410921464) --> EVAL PERFORMANCE | > avg_loader_time: 0.008558421067788565 (-0.00010684510351906783) | > avg_step_time: 0.4489158811703534 (-0.0024042398157254063) | > avg_loss_disc: 2.515255639250849 (-0.11137184290818825) | > avg_loss_disc_real_0: 0.07998524621968535 (-0.016599621387644578) | > avg_loss_disc_real_1: 0.2154771924858362 (-0.01360302428964158) | > avg_loss_disc_real_2: 0.238559097681247 (+0.007900799454097995) | > avg_loss_disc_real_3: 0.23101566710942228 (-0.005030659932485776) | > avg_loss_disc_real_4: 0.2476500453243793 (+0.006151330932764931) | > avg_loss_disc_real_5: 0.2533028742796939 (+0.011445898825014206) | > avg_loss_0: 2.515255639250849 (-0.11137184290818825) | > avg_grad_norm_0: tensor(280.2914, device='cuda:0') (+tensor(22.2186, device='cuda:0')) | > avg_loss_gen: 2.184936525116503 (+0.17867904481753305) | > avg_loss_kl: 2.000010243603881 (-0.2068788770218961) | > avg_loss_feat: 3.9381641468531647 (+0.1437843584678542) | > avg_loss_mel: 16.82487072743161 (-0.8833480620048384) | > avg_loss_duration: 1.5588682470187334 (-0.029215314018894434) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.506850041134257 (-0.7969789102043912) | > avg_grad_norm_1: tensor(9145.1680, device='cuda:0') (+tensor(1869.1636, device='cuda:0'))  > EPOCH: 1057/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:06:36)   --> TIME: 2025-02-25 19:06:58 -- STEP: 45/72 -- GLOBAL_STEP: 1076150 | > loss_disc: 2.6803643703460693 (2.4934893025292286) | > loss_disc_real_0: 0.07463958859443665 (0.07809559926390647) | > loss_disc_real_1: 0.23455199599266052 (0.21415037976370918) | > loss_disc_real_2: 0.22863057255744934 (0.23314690887928008) | > loss_disc_real_3: 0.23794838786125183 (0.23031402826309205) | > loss_disc_real_4: 0.2388743907213211 (0.23876814676655664) | > loss_disc_real_5: 0.24875620007514954 (0.24546631740199196) | > loss_0: 2.6803643703460693 (2.4934893025292286) | > grad_norm_0: tensor(307.0985, device='cuda:0') (tensor(309.7757, device='cuda:0')) | > loss_gen: 2.0490972995758057 (2.2135861582226224) | > loss_kl: 2.1196541786193848 (1.940951500998603) | > loss_feat: 3.7390313148498535 (4.060363388061523) | > loss_mel: 17.572647094726562 (16.704239484998915) | > loss_duration: 1.5557211637496948 (1.5278869416978624) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.036149978637695 (26.44702801174588) | > grad_norm_1: tensor(13560.5166, device='cuda:0') (tensor(9532.9297, device='cuda:0')) | > current_lr_0: 0.0001752448445945652 | > current_lr_1: 0.0001752448445945652 | > step_time: 0.456 (0.43365460501776804) | > loader_time: 0.0087 (0.0075717290242513025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008672912355879662 (+0.0001144912880910974) | > avg_step_time: 0.44700111133951537 (-0.0019147698308380479) | > avg_loss_disc: 2.492798539954171 (-0.02245709929667772) | > avg_loss_disc_real_0: 0.07849923484552072 (-0.0014860113741646253) | > avg_loss_disc_real_1: 0.21457740643494566 (-0.0008997860508905275) | > avg_loss_disc_real_2: 0.2358795874555346 (-0.0026795102257123893) | > avg_loss_disc_real_3: 0.23023990741078282 (-0.0007757596986394666) | > avg_loss_disc_real_4: 0.23930403674152537 (-0.00834600858285392) | > avg_loss_disc_real_5: 0.24530851652084942 (-0.007994357758844467) | > avg_loss_0: 2.492798539954171 (-0.02245709929667772) | > avg_grad_norm_0: tensor(306.9301, device='cuda:0') (+tensor(26.6387, device='cuda:0')) | > avg_loss_gen: 2.213889482995154 (+0.02895295787865093) | > avg_loss_kl: 1.967863615130035 (-0.032146628473846084) | > avg_loss_feat: 4.04301681317074 (+0.1048526663175755) | > avg_loss_mel: 16.711883531489843 (-0.11298719594176632) | > avg_loss_duration: 1.5518490898777062 (-0.007019157141027188) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.488502905402388 (-0.018347135731868747) | > avg_grad_norm_1: tensor(9583.9990, device='cuda:0') (+tensor(438.8311, device='cuda:0'))  > EPOCH: 1058/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:07:12)   --> TIME: 2025-02-25 19:07:25 -- STEP: 23/72 -- GLOBAL_STEP: 1076200 | > loss_disc: 2.4598982334136963 (2.479443933652795) | > loss_disc_real_0: 0.08861879259347916 (0.08028051532481029) | > loss_disc_real_1: 0.21202217042446136 (0.22121449786683786) | > loss_disc_real_2: 0.20941361784934998 (0.22425526121388312) | > loss_disc_real_3: 0.2640027403831482 (0.23740979640380197) | > loss_disc_real_4: 0.23189596831798553 (0.2434122173682503) | > loss_disc_real_5: 0.23473089933395386 (0.24759236496427786) | > loss_0: 2.4598982334136963 (2.479443933652795) | > grad_norm_0: tensor(252.3799, device='cuda:0') (tensor(263.5024, device='cuda:0')) | > loss_gen: 2.248941421508789 (2.2832335285518477) | > loss_kl: 1.7832539081573486 (1.83179031247678) | > loss_feat: 4.150644779205322 (4.238494935243026) | > loss_mel: 17.438573837280273 (16.82156396948773) | > loss_duration: 1.5377249717712402 (1.508729566698489) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.15913963317871 (26.68381243166716) | > grad_norm_1: tensor(8485.7812, device='cuda:0') (tensor(9269.1904, device='cuda:0')) | > current_lr_0: 0.00017522293898899086 | > current_lr_1: 0.00017522293898899086 | > step_time: 0.4566 (0.4313833817191746) | > loader_time: 0.007 (0.007252848666647206) --> EVAL PERFORMANCE | > avg_loader_time: 0.008800869256677757 (+0.00012795690079809435) | > avg_step_time: 0.45080048937193107 (+0.003799378032415701) | > avg_loss_disc: 2.5229897868465363 (+0.030191246892365076) | > avg_loss_disc_real_0: 0.09964122570736307 (+0.021141990861842347) | > avg_loss_disc_real_1: 0.21664266687043957 (+0.002065260435493904) | > avg_loss_disc_real_2: 0.22833596568712047 (-0.007543621768414138) | > avg_loss_disc_real_3: 0.23312601615005815 (+0.0028861087392753337) | > avg_loss_disc_real_4: 0.24347226825398458 (+0.004168231512459208) | > avg_loss_disc_real_5: 0.24561343814285708 (+0.0003049216220076534) | > avg_loss_0: 2.5229897868465363 (+0.030191246892365076) | > avg_grad_norm_0: tensor(280.3623, device='cuda:0') (tensor(-26.5677, device='cuda:0')) | > avg_loss_gen: 2.233774228834771 (+0.01988474583961697) | > avg_loss_kl: 1.9564572119377028 (-0.011406403192332215) | > avg_loss_feat: 4.032789599727577 (-0.01022721344316313) | > avg_loss_mel: 16.713285607351384 (+0.0014020758615416185) | > avg_loss_duration: 1.5467712660910378 (-0.005077823786668434) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.483077976065626 (-0.005424929336761863) | > avg_grad_norm_1: tensor(9305.5645, device='cuda:0') (tensor(-278.4346, device='cuda:0'))  > EPOCH: 1059/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:07:49)   --> TIME: 2025-02-25 19:07:51 -- STEP: 1/72 -- GLOBAL_STEP: 1076250 | > loss_disc: 2.532719612121582 (2.532719612121582) | > loss_disc_real_0: 0.08895878493785858 (0.08895878493785858) | > loss_disc_real_1: 0.23857177793979645 (0.23857177793979645) | > loss_disc_real_2: 0.29215672612190247 (0.29215672612190247) | > loss_disc_real_3: 0.23042719066143036 (0.23042719066143036) | > loss_disc_real_4: 0.2472575306892395 (0.2472575306892395) | > loss_disc_real_5: 0.22375154495239258 (0.22375154495239258) | > loss_0: 2.532719612121582 (2.532719612121582) | > grad_norm_0: tensor(243.0177, device='cuda:0') (tensor(243.0177, device='cuda:0')) | > loss_gen: 2.1085565090179443 (2.1085565090179443) | > loss_kl: 1.6450294256210327 (1.6450294256210327) | > loss_feat: 3.5619091987609863 (3.5619091987609863) | > loss_mel: 16.7626895904541 (16.7626895904541) | > loss_duration: 1.5170472860336304 (1.5170472860336304) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.595232009887695 (25.595232009887695) | > grad_norm_1: tensor(13968.8467, device='cuda:0') (tensor(13968.8467, device='cuda:0')) | > current_lr_0: 0.00017520103612161722 | > current_lr_1: 0.00017520103612161722 | > step_time: 0.4233 (0.4232966899871826) | > loader_time: 0.0056 (0.005605936050415039)  --> TIME: 2025-02-25 19:08:14 -- STEP: 51/72 -- GLOBAL_STEP: 1076300 | > loss_disc: 2.6350221633911133 (2.6302355457754696) | > loss_disc_real_0: 0.21573860943317413 (0.12409976993997891) | > loss_disc_real_1: 0.2208537459373474 (0.22449180863651574) | > loss_disc_real_2: 0.23897607624530792 (0.22919204480507793) | > loss_disc_real_3: 0.2421889305114746 (0.23153011179437824) | > loss_disc_real_4: 0.2432236522436142 (0.24057956624264812) | > loss_disc_real_5: 0.23372524976730347 (0.24082776495054656) | > loss_0: 2.6350221633911133 (2.6302355457754696) | > grad_norm_0: tensor(501.7834, device='cuda:0') (tensor(332.9953, device='cuda:0')) | > loss_gen: 2.087874412536621 (2.0464176336924234) | > loss_kl: 2.0201175212860107 (1.9121568506839228) | > loss_feat: 3.46553635597229 (3.747061448938706) | > loss_mel: 15.445436477661133 (16.57168066735362) | > loss_duration: 1.545348882675171 (1.5263022955726175) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.564315795898438 (25.803618973376704) | > grad_norm_1: tensor(10647.6152, device='cuda:0') (tensor(11240.8164, device='cuda:0')) | > current_lr_0: 0.00017520103612161722 | > current_lr_1: 0.00017520103612161722 | > step_time: 0.442 (0.4373134584987865) | > loader_time: 0.009 (0.007767111647362802) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862396602899256 (-0.00017690322768519617) | > avg_step_time: 0.44877601677263285 (-0.0020244725992982215) | > avg_loss_disc: 2.6739507628158785 (+0.15096097596934221) | > avg_loss_disc_real_0: 0.12393758362027961 (+0.02429635791291654) | > avg_loss_disc_real_1: 0.22720051987070433 (+0.010557853000264766) | > avg_loss_disc_real_2: 0.2258766228464288 (-0.002459342840691686) | > avg_loss_disc_real_3: 0.2374324874139168 (+0.0043064712638586355) | > avg_loss_disc_real_4: 0.24167787273165206 (-0.0017943955223325148) | > avg_loss_disc_real_5: 0.24892990312106172 (+0.0033164649782046474) | > avg_loss_0: 2.6739507628158785 (+0.15096097596934221) | > avg_grad_norm_0: tensor(379.5040, device='cuda:0') (+tensor(99.1416, device='cuda:0')) | > avg_loss_gen: 2.0237814926765334 (-0.20999273615823766) | > avg_loss_kl: 1.951172200726791 (-0.005285011210911739) | > avg_loss_feat: 3.721403806981906 (-0.3113857927456709) | > avg_loss_mel: 16.632226312664194 (-0.08105929468718998) | > avg_loss_duration: 1.54436087440437 (-0.00241039168666779) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.87294479155205 (-0.6101331845135753) | > avg_grad_norm_1: tensor(11447.7275, device='cuda:0') (+tensor(2142.1631, device='cuda:0'))  > EPOCH: 1060/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:08:26)   --> TIME: 2025-02-25 19:08:40 -- STEP: 29/72 -- GLOBAL_STEP: 1076350 | > loss_disc: 2.5489511489868164 (2.64155728241493) | > loss_disc_real_0: 0.03658590465784073 (0.11244868962415333) | > loss_disc_real_1: 0.20300891995429993 (0.2646698824547489) | > loss_disc_real_2: 0.2559278905391693 (0.23648593394920744) | > loss_disc_real_3: 0.26308247447013855 (0.2410751355105433) | > loss_disc_real_4: 0.25268909335136414 (0.2393171026788909) | > loss_disc_real_5: 0.27158331871032715 (0.24502223935620537) | > loss_0: 2.5489511489868164 (2.64155728241493) | > grad_norm_0: tensor(75.3959, device='cuda:0') (tensor(276.3010, device='cuda:0')) | > loss_gen: 2.275049924850464 (2.2139743155446547) | > loss_kl: 1.9905747175216675 (1.9867643652291134) | > loss_feat: 3.9356205463409424 (3.9529275072032) | > loss_mel: 16.764217376708984 (16.949721106167495) | > loss_duration: 1.5301496982574463 (1.5087858808451686) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.4956111907959 (26.612173080444336) | > grad_norm_1: tensor(2704.0442, device='cuda:0') (tensor(6528.3193, device='cuda:0')) | > current_lr_0: 0.00017517913599210202 | > current_lr_1: 0.00017517913599210202 | > step_time: 0.435 (0.4345330287670267) | > loader_time: 0.0078 (0.006947500952358903) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638798351019205 (+1.4832322026644279e-05) | > avg_step_time: 0.45055527754232916 (+0.0017792607696963114) | > avg_loss_disc: 2.587855513666717 (-0.08609524914916156) | > avg_loss_disc_real_0: 0.0917060756662362 (-0.032231507954043406) | > avg_loss_disc_real_1: 0.26356244743080226 (+0.03636192756009793) | > avg_loss_disc_real_2: 0.23001739424718937 (+0.004140771400760579) | > avg_loss_disc_real_3: 0.23086903170800546 (-0.006563455705911325) | > avg_loss_disc_real_4: 0.2403553178612615 (-0.0013225548703905687) | > avg_loss_disc_real_5: 0.2426699114517427 (-0.006259991669319037) | > avg_loss_0: 2.587855513666717 (-0.08609524914916156) | > avg_grad_norm_0: tensor(267.9672, device='cuda:0') (tensor(-111.5368, device='cuda:0')) | > avg_loss_gen: 2.203203505193684 (+0.17942201251715062) | > avg_loss_kl: 2.0265243876148276 (+0.07535218688803647) | > avg_loss_feat: 3.903400196156032 (+0.18199638917412564) | > avg_loss_mel: 16.81073171320096 (+0.17850540053676411) | > avg_loss_duration: 1.5487287934397305 (+0.00436791903536049) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.492588607358265 (+0.6196438158062136) | > avg_grad_norm_1: tensor(6148.0024, device='cuda:0') (tensor(-5299.7251, device='cuda:0'))  > EPOCH: 1061/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:09:02)   --> TIME: 2025-02-25 19:09:07 -- STEP: 7/72 -- GLOBAL_STEP: 1076400 | > loss_disc: 2.566889762878418 (2.5197505269731795) | > loss_disc_real_0: 0.07141096889972687 (0.08775146837745394) | > loss_disc_real_1: 0.20760230720043182 (0.21790151936667307) | > loss_disc_real_2: 0.22299152612686157 (0.23125251701899938) | > loss_disc_real_3: 0.25285691022872925 (0.23996126864637649) | > loss_disc_real_4: 0.2426932454109192 (0.23814204973833902) | > loss_disc_real_5: 0.25756749510765076 (0.24984794216496603) | > loss_0: 2.566889762878418 (2.5197505269731795) | > grad_norm_0: tensor(280.9159, device='cuda:0') (tensor(274.3733, device='cuda:0')) | > loss_gen: 2.1700000762939453 (2.2947825363704135) | > loss_kl: 2.051421880722046 (1.8797236851283483) | > loss_feat: 3.8214943408966064 (4.0933155332292825) | > loss_mel: 16.391504287719727 (16.73275007520403) | > loss_duration: 1.4720385074615479 (1.4977020536150252) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.906457901000977 (26.498274121965682) | > grad_norm_1: tensor(11299.1143, device='cuda:0') (tensor(5946.0186, device='cuda:0')) | > current_lr_0: 0.000175157238600103 | > current_lr_1: 0.000175157238600103 | > step_time: 0.4213 (0.42344750676836285) | > loader_time: 0.0054 (0.006153787885393415)  --> TIME: 2025-02-25 19:09:30 -- STEP: 57/72 -- GLOBAL_STEP: 1076450 | > loss_disc: 2.5872862339019775 (2.4972927403031724) | > loss_disc_real_0: 0.07774785906076431 (0.07853557476610466) | > loss_disc_real_1: 0.2262829840183258 (0.2168372913933637) | > loss_disc_real_2: 0.2320827692747116 (0.2303972314847143) | > loss_disc_real_3: 0.22467616200447083 (0.23275512404609144) | > loss_disc_real_4: 0.24581845104694366 (0.24040011564890543) | > loss_disc_real_5: 0.24646495282649994 (0.24352605353321946) | > loss_0: 2.5872862339019775 (2.4972927403031724) | > grad_norm_0: tensor(202.5693, device='cuda:0') (tensor(236.6493, device='cuda:0')) | > loss_gen: 2.139251708984375 (2.261868393212034) | > loss_kl: 1.9618816375732422 (1.9770341003150271) | > loss_feat: 3.7603116035461426 (4.0700808993557045) | > loss_mel: 15.482975006103516 (16.643620591414606) | > loss_duration: 1.5735599994659424 (1.5302853709773014) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.917980194091797 (26.482889342726324) | > grad_norm_1: tensor(4354.9854, device='cuda:0') (tensor(6102.8940, device='cuda:0')) | > current_lr_0: 0.000175157238600103 | > current_lr_1: 0.000175157238600103 | > step_time: 0.4598 (0.43583063911973385) | > loader_time: 0.0095 (0.007819723664668569) --> EVAL PERFORMANCE | > avg_loader_time: 0.008506526409740181 (-0.00013227194127902325) | > avg_step_time: 0.4459254573768293 (-0.004629820165499843) | > avg_loss_disc: 2.503989599120448 (-0.0838659145462688) | > avg_loss_disc_real_0: 0.07957317947711741 (-0.012132896189118794) | > avg_loss_disc_real_1: 0.2177492703350497 (-0.045813177095752555) | > avg_loss_disc_real_2: 0.23060385756929155 (+0.0005864633221021875) | > avg_loss_disc_real_3: 0.23133096203837597 (+0.000461930330370508) | > avg_loss_disc_real_4: 0.23964677897977157 (-0.0007085388814899174) | > avg_loss_disc_real_5: 0.2439494361759911 (+0.0012795247242484153) | > avg_loss_0: 2.503989599120448 (-0.0838659145462688) | > avg_grad_norm_0: tensor(236.2116, device='cuda:0') (tensor(-31.7556, device='cuda:0')) | > avg_loss_gen: 2.257289362625337 (+0.05408585743165295) | > avg_loss_kl: 2.004733958714445 (-0.02179042890038252) | > avg_loss_feat: 4.046469711921584 (+0.14306951576555216) | > avg_loss_mel: 16.682638369815454 (-0.1280933433855047) | > avg_loss_duration: 1.543651819229126 (-0.00507697421060449) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.534783161861796 (+0.04219455450353138) | > avg_grad_norm_1: tensor(6232.8809, device='cuda:0') (+tensor(84.8784, device='cuda:0'))  > EPOCH: 1062/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:09:39)   --> TIME: 2025-02-25 19:09:56 -- STEP: 35/72 -- GLOBAL_STEP: 1076500 | > loss_disc: 2.7339706420898438 (2.5049125603267126) | > loss_disc_real_0: 0.07351681590080261 (0.0877100487372705) | > loss_disc_real_1: 0.2617013454437256 (0.21957920151097435) | > loss_disc_real_2: 0.2535723149776459 (0.23081059413296837) | > loss_disc_real_3: 0.2517038583755493 (0.23378868997097016) | > loss_disc_real_4: 0.24356497824192047 (0.24223591004099165) | > loss_disc_real_5: 0.25816619396209717 (0.24687780695302144) | > loss_0: 2.7339706420898438 (2.5049125603267126) | > grad_norm_0: tensor(78.7738, device='cuda:0') (tensor(236.5560, device='cuda:0')) | > loss_gen: 1.9298863410949707 (2.308867597579956) | > loss_kl: 2.1681528091430664 (1.9700386285781861) | > loss_feat: 3.0208449363708496 (4.175968483516148) | > loss_mel: 16.976572036743164 (16.888027790614544) | > loss_duration: 1.5680413246154785 (1.5056060484477451) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.663496017456055 (26.848508453369142) | > grad_norm_1: tensor(9054.3828, device='cuda:0') (tensor(6285.1489, device='cuda:0')) | > current_lr_0: 0.00017513534394527798 | > current_lr_1: 0.00017513534394527798 | > step_time: 0.447 (0.43782823426382883) | > loader_time: 0.0087 (0.007170431954520089) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630436910709865 (+0.00012391050096968378) | > avg_step_time: 0.4563667438399624 (+0.010441286463133082) | > avg_loss_disc: 2.562047384154629 (+0.05805778503418102) | > avg_loss_disc_real_0: 0.1158461027949209 (+0.03627292331780349) | > avg_loss_disc_real_1: 0.21744256718477734 (-0.0003067031502723694) | > avg_loss_disc_real_2: 0.22596453927772145 (-0.004639318291570105) | > avg_loss_disc_real_3: 0.2333707421178549 (+0.0020397800794789422) | > avg_loss_disc_real_4: 0.24135249643258647 (+0.0017057174528148922) | > avg_loss_disc_real_5: 0.2495716046279585 (+0.005622168451967391) | > avg_loss_0: 2.562047384154629 (+0.05805778503418102) | > avg_grad_norm_0: tensor(233.7018, device='cuda:0') (tensor(-2.5098, device='cuda:0')) | > avg_loss_gen: 2.282319952064836 (+0.025030589439499096) | > avg_loss_kl: 2.027286860304819 (+0.02255290159037404) | > avg_loss_feat: 4.111731774370437 (+0.06526206244885291) | > avg_loss_mel: 16.808546831910032 (+0.1259084620945785) | > avg_loss_duration: 1.5427027655319432 (-0.000949053697182789) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.772588165713028 (+0.23780500385123204) | > avg_grad_norm_1: tensor(5443.4355, device='cuda:0') (tensor(-789.4453, device='cuda:0'))  > EPOCH: 1063/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:10:16)   --> TIME: 2025-02-25 19:10:23 -- STEP: 13/72 -- GLOBAL_STEP: 1076550 | > loss_disc: 2.528529644012451 (2.6538237791794996) | > loss_disc_real_0: 0.05239768698811531 (0.11224920044724758) | > loss_disc_real_1: 0.2162933498620987 (0.236421783383076) | > loss_disc_real_2: 0.18401803076267242 (0.22518535875357115) | > loss_disc_real_3: 0.19876761734485626 (0.19761573236722213) | > loss_disc_real_4: 0.24399256706237793 (0.1944504024890753) | > loss_disc_real_5: 0.22912883758544922 (0.23572928057267115) | > loss_0: 2.528529644012451 (2.6538237791794996) | > grad_norm_0: tensor(168.2818, device='cuda:0') (tensor(224.4004, device='cuda:0')) | > loss_gen: 2.041992664337158 (2.002489520953252) | > loss_kl: 2.1455671787261963 (2.066081248796903) | > loss_feat: 4.258843898773193 (4.056122376368596) | > loss_mel: 16.96702003479004 (17.44887058551495) | > loss_duration: 1.4452948570251465 (1.5029409665327806) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.858718872070312 (27.076505220853367) | > grad_norm_1: tensor(6268.5059, device='cuda:0') (tensor(5865.0596, device='cuda:0')) | > current_lr_0: 0.0001751134520272848 | > current_lr_1: 0.0001751134520272848 | > step_time: 0.4728 (0.4289891169621394) | > loader_time: 0.0083 (0.006842283102182241)  --> TIME: 2025-02-25 19:10:46 -- STEP: 63/72 -- GLOBAL_STEP: 1076600 | > loss_disc: 2.5522279739379883 (2.6383867036728628) | > loss_disc_real_0: 0.2180575430393219 (0.12441722262236807) | > loss_disc_real_1: 0.20392398536205292 (0.22799328111466907) | > loss_disc_real_2: 0.22458191215991974 (0.22776639295948875) | > loss_disc_real_3: 0.2605367600917816 (0.2253361145655314) | > loss_disc_real_4: 0.24697351455688477 (0.22859361081842391) | > loss_disc_real_5: 0.25967398285865784 (0.2416394426236077) | > loss_0: 2.5522279739379883 (2.6383867036728628) | > grad_norm_0: tensor(275.6771, device='cuda:0') (tensor(230.9574, device='cuda:0')) | > loss_gen: 2.2402453422546387 (2.0867546248057534) | > loss_kl: 2.226231098175049 (2.1154255904848607) | > loss_feat: 3.374823808670044 (3.758468446277437) | > loss_mel: 17.00133514404297 (16.976137751624687) | > loss_duration: 1.5829896926879883 (1.5496657197437589) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.42562484741211 (26.48645249624101) | > grad_norm_1: tensor(10291.8633, device='cuda:0') (tensor(6673.2578, device='cuda:0')) | > current_lr_0: 0.0001751134520272848 | > current_lr_1: 0.0001751134520272848 | > step_time: 0.4925 (0.44046440200200154) | > loader_time: 0.0112 (0.008226269767397926) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861942600196516 (-1.101090874470502e-05) | > avg_step_time: 0.44705223029767965 (-0.009314513542282743) | > avg_loss_disc: 2.641509714260907 (+0.07946233010627779) | > avg_loss_disc_real_0: 0.1305221141329114 (+0.0146760113379905) | > avg_loss_disc_real_1: 0.22849347935595982 (+0.011050912171182481) | > avg_loss_disc_real_2: 0.22816790157640485 (+0.002203362298683398) | > avg_loss_disc_real_3: 0.22803216920772068 (-0.005338572910134232) | > avg_loss_disc_real_4: 0.2294164648358251 (-0.011936031596761376) | > avg_loss_disc_real_5: 0.24143689765896595 (-0.008134706968992544) | > avg_loss_0: 2.641509714260907 (+0.07946233010627779) | > avg_grad_norm_0: tensor(225.0942, device='cuda:0') (tensor(-8.6075, device='cuda:0')) | > avg_loss_gen: 2.0797926120355097 (-0.20252734002932637) | > avg_loss_kl: 2.126786017082107 (+0.09949915677728782) | > avg_loss_feat: 3.700115005734941 (-0.4116167686354961) | > avg_loss_mel: 16.915189407241165 (+0.10664257533113286) | > avg_loss_duration: 1.5570030682523486 (+0.014300302720405389) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.378886424319845 (-0.39370174139318337) | > avg_grad_norm_1: tensor(6829.5586, device='cuda:0') (+tensor(1386.1230, device='cuda:0'))  > EPOCH: 1064/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:10:52)   --> TIME: 2025-02-25 19:11:12 -- STEP: 41/72 -- GLOBAL_STEP: 1076650 | > loss_disc: 2.5522308349609375 (2.613172635799501) | > loss_disc_real_0: 0.03630930185317993 (0.09608832328784757) | > loss_disc_real_1: 0.15930384397506714 (0.2389988899230957) | > loss_disc_real_2: 0.22529080510139465 (0.24542607221661544) | > loss_disc_real_3: 0.21439960598945618 (0.2297675148016069) | > loss_disc_real_4: 0.26042118668556213 (0.24568177105450048) | > loss_disc_real_5: 0.21806448698043823 (0.23880800023311521) | > loss_0: 2.5522308349609375 (2.613172635799501) | > grad_norm_0: tensor(268.0393, device='cuda:0') (tensor(222.3500, device='cuda:0')) | > loss_gen: 1.826292872428894 (2.1737084970241645) | > loss_kl: 2.0634539127349854 (2.021657469795972) | > loss_feat: 4.093526363372803 (3.930344814207495) | > loss_mel: 17.24626350402832 (16.766548994110853) | > loss_duration: 1.5689287185668945 (1.5236237717837822) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.798465728759766 (26.415883506216655) | > grad_norm_1: tensor(10142.8037, device='cuda:0') (tensor(6705.5400, device='cuda:0')) | > current_lr_0: 0.0001750915628457814 | > current_lr_1: 0.0001750915628457814 | > step_time: 0.4376 (0.43438241539931877) | > loader_time: 0.0091 (0.007226001925584747) --> EVAL PERFORMANCE | > avg_loader_time: 0.008537060777905961 (-8.236522405919919e-05) | > avg_step_time: 0.4480342025488195 (+0.0009819722511398288) | > avg_loss_disc: 2.6176491018751977 (-0.023860612385709246) | > avg_loss_disc_real_0: 0.10019108584859002 (-0.030331028284321382) | > avg_loss_disc_real_1: 0.23182144593185103 (+0.0033279665758912114) | > avg_loss_disc_real_2: 0.2457365611909141 (+0.017568659614509263) | > avg_loss_disc_real_3: 0.23460034732247742 (+0.00656817811475674) | > avg_loss_disc_real_4: 0.24668312009791254 (+0.017266655262087455) | > avg_loss_disc_real_5: 0.24358402339505478 (+0.0021471257360888285) | > avg_loss_0: 2.6176491018751977 (-0.023860612385709246) | > avg_grad_norm_0: tensor(219.6874, device='cuda:0') (tensor(-5.4069, device='cuda:0')) | > avg_loss_gen: 2.1370281300074616 (+0.05723551797195192) | > avg_loss_kl: 2.203317758063196 (+0.07653174098108906) | > avg_loss_feat: 3.796150543320347 (+0.09603553758540606) | > avg_loss_mel: 17.175578815836296 (+0.26038940859513104) | > avg_loss_duration: 1.5621699245882705 (+0.005166856335921954) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.874245039174255 (+0.49535861485441046) | > avg_grad_norm_1: tensor(6506.3750, device='cuda:0') (tensor(-323.1836, device='cuda:0'))  > EPOCH: 1065/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:11:28)   --> TIME: 2025-02-25 19:11:39 -- STEP: 19/72 -- GLOBAL_STEP: 1076700 | > loss_disc: 2.5332212448120117 (2.56302241275185) | > loss_disc_real_0: 0.11357316374778748 (0.10237584714042514) | > loss_disc_real_1: 0.22335198521614075 (0.20397898476374776) | > loss_disc_real_2: 0.24653255939483643 (0.23521897588905535) | > loss_disc_real_3: 0.22924493253231049 (0.2353992815080442) | > loss_disc_real_4: 0.23934781551361084 (0.24373441856158407) | > loss_disc_real_5: 0.242343932390213 (0.2354750272474791) | > loss_0: 2.5332212448120117 (2.56302241275185) | > grad_norm_0: tensor(43.7484, device='cuda:0') (tensor(266.8259, device='cuda:0')) | > loss_gen: 2.0467121601104736 (2.103956015486466) | > loss_kl: 2.3567092418670654 (2.796683562429328) | > loss_feat: 3.716977119445801 (3.8413653499201725) | > loss_mel: 18.46290397644043 (20.29358632940995) | > loss_duration: 1.53769052028656 (1.59458789699956) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.120994567871094 (30.630179254632246) | > grad_norm_1: tensor(13041.9326, device='cuda:0') (tensor(10836.1182, device='cuda:0')) | > current_lr_0: 0.00017506967640042566 | > current_lr_1: 0.00017506967640042566 | > step_time: 0.4258 (0.4320040878496672) | > loader_time: 0.0059 (0.00661376902931615)  --> TIME: 2025-02-25 19:12:02 -- STEP: 69/72 -- GLOBAL_STEP: 1076750 | > loss_disc: 2.5238914489746094 (2.623433586479961) | > loss_disc_real_0: 0.12151556462049484 (0.12191552479845891) | > loss_disc_real_1: 0.2274312525987625 (0.22802716806746912) | > loss_disc_real_2: 0.22451162338256836 (0.2330333074365837) | > loss_disc_real_3: 0.23431284725666046 (0.23342188134573508) | > loss_disc_real_4: 0.2435787171125412 (0.24280154186746347) | > loss_disc_real_5: 0.23270770907402039 (0.2411713736212772) | > loss_0: 2.5238914489746094 (2.623433586479961) | > grad_norm_0: tensor(291.4343, device='cuda:0') (tensor(298.5524, device='cuda:0')) | > loss_gen: 2.258516311645508 (2.100160769794298) | > loss_kl: 2.418161392211914 (2.4737336773803262) | > loss_feat: 3.735978364944458 (3.7136100375134014) | > loss_mel: 18.350910186767578 (18.694146211596504) | > loss_duration: 1.6737297773361206 (1.5928071633629177) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.43729591369629 (28.574458053146582) | > grad_norm_1: tensor(9473.3447, device='cuda:0') (tensor(9916.9160, device='cuda:0')) | > current_lr_0: 0.00017506967640042566 | > current_lr_1: 0.00017506967640042566 | > step_time: 0.5047 (0.4441495874653692) | > loader_time: 0.0129 (0.008428459582121477) --> EVAL PERFORMANCE | > avg_loader_time: 0.00849419244578187 (-4.286833212409011e-05) | > avg_step_time: 0.4456884894572513 (-0.002345713091568169) | > avg_loss_disc: 2.619668272179617 (+0.0020191703044192266) | > avg_loss_disc_real_0: 0.12160186034063218 (+0.021410774492042164) | > avg_loss_disc_real_1: 0.2275559091022317 (-0.004265536829619332) | > avg_loss_disc_real_2: 0.23275306548031283 (-0.012983495710601284) | > avg_loss_disc_real_3: 0.2329906017427713 (-0.0016097455797061122) | > avg_loss_disc_real_4: 0.24223138937647914 (-0.0044517307214334) | > avg_loss_disc_real_5: 0.24060994239760117 (-0.0029740809974536098) | > avg_loss_0: 2.619668272179617 (+0.0020191703044192266) | > avg_grad_norm_0: tensor(299.5923, device='cuda:0') (+tensor(79.9049, device='cuda:0')) | > avg_loss_gen: 2.1053160059619955 (-0.031712124045466084) | > avg_loss_kl: 2.4674125825855087 (+0.2640948245223127) | > avg_loss_feat: 3.727903594433422 (-0.06824694888692484) | > avg_loss_mel: 18.623170866093165 (+1.447592050256869) | > avg_loss_duration: 1.592832138840581 (+0.030662214252310527) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.51663538435815 (+1.6423903451838946) | > avg_grad_norm_1: tensor(9860.4209, device='cuda:0') (+tensor(3354.0459, device='cuda:0'))  > EPOCH: 1066/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:12:05)   --> TIME: 2025-02-25 19:12:28 -- STEP: 47/72 -- GLOBAL_STEP: 1076800 | > loss_disc: 2.5621719360351562 (2.477570680861778) | > loss_disc_real_0: 0.08858631551265717 (0.07587372127207034) | > loss_disc_real_1: 0.2510295510292053 (0.2119966358580488) | > loss_disc_real_2: 0.25778570771217346 (0.23545006551641098) | > loss_disc_real_3: 0.24541343748569489 (0.22923151324404048) | > loss_disc_real_4: 0.25546371936798096 (0.24071388517288453) | > loss_disc_real_5: 0.266620397567749 (0.24440615893678463) | > loss_0: 2.5621719360351562 (2.477570680861778) | > grad_norm_0: tensor(122.2252, device='cuda:0') (tensor(226.6816, device='cuda:0')) | > loss_gen: 2.3091840744018555 (2.2852170365921993) | > loss_kl: 2.1444756984710693 (2.0191131906306494) | > loss_feat: 3.594914436340332 (4.227137103993842) | > loss_mel: 16.2488956451416 (17.188594960151836) | > loss_duration: 1.5814523696899414 (1.5337155778357323) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.878921508789062 (27.2537778894952) | > grad_norm_1: tensor(10220.0127, device='cuda:0') (tensor(7616.5576, device='cuda:0')) | > current_lr_0: 0.0001750477926908756 | > current_lr_1: 0.0001750477926908756 | > step_time: 0.4441 (0.4420520752034289) | > loader_time: 0.0089 (0.007491370464893098) --> EVAL PERFORMANCE | > avg_loader_time: 0.008578726943110076 (+8.453449732820535e-05) | > avg_step_time: 0.4538369279512217 (+0.0081484384939704) | > avg_loss_disc: 2.4833967181998244 (-0.13627155397979251) | > avg_loss_disc_real_0: 0.0801254255093739 (-0.04147643483125828) | > avg_loss_disc_real_1: 0.21340885107785884 (-0.014147058024372855) | > avg_loss_disc_real_2: 0.23186764624756825 (-0.000885419232744572) | > avg_loss_disc_real_3: 0.22960801716421692 (-0.003382584578554393) | > avg_loss_disc_real_4: 0.24014381518666172 (-0.0020875741898174205) | > avg_loss_disc_real_5: 0.24481214394032116 (+0.004202201542719997) | > avg_loss_0: 2.4833967181998244 (-0.13627155397979251) | > avg_grad_norm_0: tensor(230.5674, device='cuda:0') (tensor(-69.0249, device='cuda:0')) | > avg_loss_gen: 2.2814964694036566 (+0.17618046344166105) | > avg_loss_kl: 2.0359469131684644 (-0.43146566941704423) | > avg_loss_feat: 4.1523792475042205 (+0.4244756530707985) | > avg_loss_mel: 17.069636089701042 (-1.553534776392123) | > avg_loss_duration: 1.5538236960558824 (-0.039008442784698616) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.093282350352112 (-1.4233530340060376) | > avg_grad_norm_1: tensor(7535.8447, device='cuda:0') (tensor(-2324.5762, device='cuda:0'))  > EPOCH: 1067/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:12:42)   --> TIME: 2025-02-25 19:12:55 -- STEP: 25/72 -- GLOBAL_STEP: 1076850 | > loss_disc: 2.476094961166382 (2.511966695785522) | > loss_disc_real_0: 0.11249779164791107 (0.09911745339632035) | > loss_disc_real_1: 0.22293739020824432 (0.22349671602249147) | > loss_disc_real_2: 0.2307511270046234 (0.2280401712656021) | > loss_disc_real_3: 0.22728075087070465 (0.23260755836963654) | > loss_disc_real_4: 0.2404957115650177 (0.24079352378845215) | > loss_disc_real_5: 0.24062152206897736 (0.24399773478507997) | > loss_0: 2.476094961166382 (2.511966695785522) | > grad_norm_0: tensor(200.9680, device='cuda:0') (tensor(248.9049, device='cuda:0')) | > loss_gen: 2.418372869491577 (2.2873194885253914) | > loss_kl: 2.1782195568084717 (1.9176663398742675) | > loss_feat: 4.300599098205566 (4.072710151672363) | > loss_mel: 17.098657608032227 (16.946321105957033) | > loss_duration: 1.505854845046997 (1.5002842664718627) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.501705169677734 (26.72430107116699) | > grad_norm_1: tensor(4999.5547, device='cuda:0') (tensor(7537.4434, device='cuda:0')) | > current_lr_0: 0.00017502591171678924 | > current_lr_1: 0.00017502591171678924 | > step_time: 0.4352 (0.43976675033569335) | > loader_time: 0.0076 (0.006828804016113282) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614748296603351 (+3.6021353493275246e-05) | > avg_step_time: 0.4604096983520078 (+0.0065727704007860766) | > avg_loss_disc: 2.5149288446130886 (+0.031532126413264194) | > avg_loss_disc_real_0: 0.09691963434009485 (+0.01679420883072094) | > avg_loss_disc_real_1: 0.2229958818831914 (+0.009587030805332547) | > avg_loss_disc_real_2: 0.2294377632124323 (-0.0024298830351359435) | > avg_loss_disc_real_3: 0.23027540848288738 (+0.0006673913186704683) | > avg_loss_disc_real_4: 0.24076957433996066 (+0.0006257591532989326) | > avg_loss_disc_real_5: 0.24495538586462048 (+0.00014324192429931992) | > avg_loss_0: 2.5149288446130886 (+0.031532126413264194) | > avg_grad_norm_0: tensor(251.5673, device='cuda:0') (+tensor(20.9999, device='cuda:0')) | > avg_loss_gen: 2.2525188469550987 (-0.028977622448557838) | > avg_loss_kl: 1.9804161149011532 (-0.05553079826731122) | > avg_loss_feat: 3.9805723848477217 (-0.1718068626564988) | > avg_loss_mel: 16.768712406427095 (-0.3009236832739468) | > avg_loss_duration: 1.5433589236837038 (-0.010464772372178688) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.525578619728627 (-0.5677037306234851) | > avg_grad_norm_1: tensor(7693.1567, device='cuda:0') (+tensor(157.3120, device='cuda:0'))  > EPOCH: 1068/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:13:19)   --> TIME: 2025-02-25 19:13:23 -- STEP: 3/72 -- GLOBAL_STEP: 1076900 | > loss_disc: 2.1973743438720703 (2.467825015385946) | > loss_disc_real_0: 0.06739456206560135 (0.08493947982788086) | > loss_disc_real_1: 0.17532303929328918 (0.20331831773122153) | > loss_disc_real_2: 0.2088337391614914 (0.2488838185866674) | > loss_disc_real_3: 0.20632250607013702 (0.21390231947104135) | > loss_disc_real_4: 0.23205676674842834 (0.23620804647604624) | > loss_disc_real_5: 0.23577627539634705 (0.24748559296131134) | > loss_0: 2.1973743438720703 (2.467825015385946) | > grad_norm_0: tensor(184.5463, device='cuda:0') (tensor(241.3874, device='cuda:0')) | > loss_gen: 2.548548460006714 (2.2912309169769287) | > loss_kl: 1.7769898176193237 (1.7467254797617595) | > loss_feat: 4.821870803833008 (4.2053665320078535) | > loss_mel: 16.751012802124023 (16.35629876454671) | > loss_duration: 1.5320873260498047 (1.5062244733174641) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.43050765991211 (26.105845769246418) | > grad_norm_1: tensor(9748.8955, device='cuda:0') (tensor(7140.5991, device='cuda:0')) | > current_lr_0: 0.00017500403347782463 | > current_lr_1: 0.00017500403347782463 | > step_time: 0.4313 (0.4359612464904785) | > loader_time: 0.0056 (0.00649110476175944)  --> TIME: 2025-02-25 19:13:45 -- STEP: 53/72 -- GLOBAL_STEP: 1076950 | > loss_disc: 2.6970973014831543 (2.5709643408937275) | > loss_disc_real_0: 0.18255072832107544 (0.12145591337444647) | > loss_disc_real_1: 0.3298593759536743 (0.22763465513598244) | > loss_disc_real_2: 0.22252605855464935 (0.2241621621937122) | > loss_disc_real_3: 0.21587663888931274 (0.23133250761706875) | > loss_disc_real_4: 0.2488492727279663 (0.24041009848972536) | > loss_disc_real_5: 0.251032292842865 (0.24641120096422592) | > loss_0: 2.6970973014831543 (2.5709643408937275) | > grad_norm_0: tensor(290.6109, device='cuda:0') (tensor(303.4412, device='cuda:0')) | > loss_gen: 2.157646656036377 (2.230622568220462) | > loss_kl: 2.1308066844940186 (1.916969362294899) | > loss_feat: 3.4619507789611816 (4.033812698328269) | > loss_mel: 15.996743202209473 (16.69899312505182) | > loss_duration: 1.5863442420959473 (1.5258509168085062) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.3334903717041 (26.406248704442437) | > grad_norm_1: tensor(10916.7061, device='cuda:0') (tensor(7916.6895, device='cuda:0')) | > current_lr_0: 0.00017500403347782463 | > current_lr_1: 0.00017500403347782463 | > step_time: 0.4418 (0.4416307323383835) | > loader_time: 0.0091 (0.007808613327314269) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863964792708276 (+2.4899630479408366e-05) | > avg_step_time: 0.45223313653972785 (-0.008176561812279937) | > avg_loss_disc: 2.5866866649036675 (+0.07175782029057887) | > avg_loss_disc_real_0: 0.11902539851799818 (+0.02210576417790333) | > avg_loss_disc_real_1: 0.22110809028988154 (-0.0018877915933098455) | > avg_loss_disc_real_2: 0.22638775890981647 (-0.003050004302615844) | > avg_loss_disc_real_3: 0.2358817693213342 (+0.005606360838446817) | > avg_loss_disc_real_4: 0.2402074504906023 (-0.0005621238493583591) | > avg_loss_disc_real_5: 0.2452944019730662 (+0.00033901610844572216) | > avg_loss_0: 2.5866866649036675 (+0.07175782029057887) | > avg_grad_norm_0: tensor(314.1165, device='cuda:0') (+tensor(62.5492, device='cuda:0')) | > avg_loss_gen: 2.176080685266306 (-0.07643816168879258) | > avg_loss_kl: 1.9657552242279053 (-0.01466089067324794) | > avg_loss_feat: 3.9243839962381704 (-0.05618838860955133) | > avg_loss_mel: 16.7081311991517 (-0.06058120727539418) | > avg_loss_duration: 1.5436744488460916 (+0.0003155251623878641) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.31802553526113 (-0.20755308446749865) | > avg_grad_norm_1: tensor(8629.5898, device='cuda:0') (+tensor(936.4331, device='cuda:0'))  > EPOCH: 1069/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:13:56)   --> TIME: 2025-02-25 19:14:12 -- STEP: 31/72 -- GLOBAL_STEP: 1077000 | > loss_disc: 2.5318474769592285 (2.5828891031203733) | > loss_disc_real_0: 0.12803173065185547 (0.12141907239152541) | > loss_disc_real_1: 0.21159042418003082 (0.22502328407379887) | > loss_disc_real_2: 0.20386658608913422 (0.237581102598098) | > loss_disc_real_3: 0.22151580452919006 (0.23084858156019641) | > loss_disc_real_4: 0.24209272861480713 (0.24588327302086738) | > loss_disc_real_5: 0.2550007104873657 (0.24474352550122044) | > loss_0: 2.5318474769592285 (2.5828891031203733) | > grad_norm_0: tensor(428.5806, device='cuda:0') (tensor(369.6180, device='cuda:0')) | > loss_gen: 2.2392823696136475 (2.1330367826646377) | > loss_kl: 1.9266729354858398 (1.9458380591484807) | > loss_feat: 4.160130977630615 (3.8259477461538007) | > loss_mel: 16.990999221801758 (16.662799404513457) | > loss_duration: 1.5375657081604004 (1.4990954322199668) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.854650497436523 (26.066717332409276) | > grad_norm_1: tensor(16521.9199, device='cuda:0') (tensor(12709.6865, device='cuda:0')) | > current_lr_0: 0.0001749821579736399 | > current_lr_1: 0.0001749821579736399 | > step_time: 0.4512 (0.4488877788666756) | > loader_time: 0.0081 (0.006877653060420867) --> EVAL PERFORMANCE | > avg_loader_time: 0.008663593883245765 (+2.3945956163005586e-05) | > avg_step_time: 0.4587989457896058 (+0.00656580924987793) | > avg_loss_disc: 2.596202702589439 (+0.009516037685771472) | > avg_loss_disc_real_0: 0.12393438333356885 (+0.004908984815570672) | > avg_loss_disc_real_1: 0.22186124933437562 (+0.0007531590444940783) | > avg_loss_disc_real_2: 0.23369752332358293 (+0.007309764413766462) | > avg_loss_disc_real_3: 0.22953329455684607 (-0.006348474764488132) | > avg_loss_disc_real_4: 0.24376150378039185 (+0.003554053289789555) | > avg_loss_disc_real_5: 0.244561031041011 (-0.000733370932055194) | > avg_loss_0: 2.596202702589439 (+0.009516037685771472) | > avg_grad_norm_0: tensor(363.5425, device='cuda:0') (+tensor(49.4260, device='cuda:0')) | > avg_loss_gen: 2.1191018551168317 (-0.05697883014947447) | > avg_loss_kl: 1.9920857389208297 (+0.026330514692924467) | > avg_loss_feat: 3.746220377129568 (-0.17816361910860223) | > avg_loss_mel: 16.502165364547515 (-0.20596583460418572) | > avg_loss_duration: 1.538632266957995 (-0.005042181888096575) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.898205340748103 (-0.41982019451302577) | > avg_grad_norm_1: tensor(12602.6562, device='cuda:0') (+tensor(3973.0664, device='cuda:0'))  > EPOCH: 1070/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:14:33)   --> TIME: 2025-02-25 19:14:39 -- STEP: 9/72 -- GLOBAL_STEP: 1077050 | > loss_disc: 3.015639066696167 (2.666395902633667) | > loss_disc_real_0: 0.1317393183708191 (0.14972321523560417) | > loss_disc_real_1: 0.1870671510696411 (0.20664511952135298) | > loss_disc_real_2: 0.2238091230392456 (0.2272779726319843) | > loss_disc_real_3: 0.23421348631381989 (0.23264454470740425) | > loss_disc_real_4: 0.2641533613204956 (0.2449010113875071) | > loss_disc_real_5: 0.2637520730495453 (0.24625351859463585) | > loss_0: 3.015639066696167 (2.666395902633667) | > grad_norm_0: tensor(284.7723, device='cuda:0') (tensor(353.7637, device='cuda:0')) | > loss_gen: 1.7849304676055908 (1.954990890291002) | > loss_kl: 1.9940539598464966 (1.8985020187166002) | > loss_feat: 3.3193321228027344 (3.716876745223999) | > loss_mel: 16.086090087890625 (16.61671977572971) | > loss_duration: 1.522924542427063 (1.5033903519312541) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.70733070373535 (25.69047949049208) | > grad_norm_1: tensor(11141.8379, device='cuda:0') (tensor(8870.8252, device='cuda:0')) | > current_lr_0: 0.00017496028520389318 | > current_lr_1: 0.00017496028520389318 | > step_time: 0.4449 (0.43656259112887913) | > loader_time: 0.0056 (0.006849527359008789)  --> TIME: 2025-02-25 19:15:02 -- STEP: 59/72 -- GLOBAL_STEP: 1077100 | > loss_disc: 2.5189990997314453 (2.5916748895483503) | > loss_disc_real_0: 0.13420650362968445 (0.12298638335729052) | > loss_disc_real_1: 0.19824348390102386 (0.21691417769860413) | > loss_disc_real_2: 0.22743622958660126 (0.23368459434832556) | > loss_disc_real_3: 0.23521952331066132 (0.23158234787189355) | > loss_disc_real_4: 0.2160455733537674 (0.2405489776599205) | > loss_disc_real_5: 0.21321289241313934 (0.24553436394465172) | > loss_0: 2.5189990997314453 (2.5916748895483503) | > grad_norm_0: tensor(268.5260, device='cuda:0') (tensor(361.6196, device='cuda:0')) | > loss_gen: 2.3691256046295166 (2.121714523283101) | > loss_kl: 2.0288710594177246 (1.9742672867694144) | > loss_feat: 3.9304521083831787 (3.830534167208914) | > loss_mel: 15.209897994995117 (16.666499800601247) | > loss_duration: 1.5830620527267456 (1.5322561728752266) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.12140655517578 (26.12527184567209) | > grad_norm_1: tensor(14113.1367, device='cuda:0') (tensor(13245.2920, device='cuda:0')) | > current_lr_0: 0.00017496028520389318 | > current_lr_1: 0.00017496028520389318 | > step_time: 0.4647 (0.4476619574983241) | > loader_time: 0.0097 (0.008119061841803082) --> EVAL PERFORMANCE | > avg_loader_time: 0.008676562510745627 (+1.2968627499861454e-05) | > avg_step_time: 0.4580537198295056 (-0.000745225960100182) | > avg_loss_disc: 2.5998212854627156 (+0.0036185828732766545) | > avg_loss_disc_real_0: 0.12148932383303913 (-0.0024450595005297143) | > avg_loss_disc_real_1: 0.22390621921546022 (+0.0020449698810846018) | > avg_loss_disc_real_2: 0.2352477255421625 (+0.0015502022185795716) | > avg_loss_disc_real_3: 0.2310906793449966 (+0.0015573847881505398) | > avg_loss_disc_real_4: 0.2421380109350446 (-0.0016234928453472408) | > avg_loss_disc_real_5: 0.24574792070288054 (+0.001186889661869528) | > avg_loss_0: 2.5998212854627156 (+0.0036185828732766545) | > avg_grad_norm_0: tensor(357.2477, device='cuda:0') (tensor(-6.2947, device='cuda:0')) | > avg_loss_gen: 2.118715504525413 (-0.0003863505914187826) | > avg_loss_kl: 1.9948882133188381 (+0.0028024743980084033) | > avg_loss_feat: 3.807174998269954 (+0.06095462114038597) | > avg_loss_mel: 16.64833563146457 (+0.14617026691705348) | > avg_loss_duration: 1.5433437908199472 (+0.0047115238619521715) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.112458108176646 (+0.21425276742854393) | > avg_grad_norm_1: tensor(12966.0068, device='cuda:0') (+tensor(363.3506, device='cuda:0'))  > EPOCH: 1071/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:15:10)   --> TIME: 2025-02-25 19:15:29 -- STEP: 37/72 -- GLOBAL_STEP: 1077150 | > loss_disc: 2.534672260284424 (2.7170205051834517) | > loss_disc_real_0: 0.11388152092695236 (0.1478514635079616) | > loss_disc_real_1: 0.19680945575237274 (0.22503673526886347) | > loss_disc_real_2: 0.22138726711273193 (0.23828605018757484) | > loss_disc_real_3: 0.2573844790458679 (0.23731910981036522) | > loss_disc_real_4: 0.24122944474220276 (0.243029059187786) | > loss_disc_real_5: 0.23472799360752106 (0.2364197300898062) | > loss_0: 2.534672260284424 (2.7170205051834517) | > grad_norm_0: tensor(257.0582, device='cuda:0') (tensor(406.7566, device='cuda:0')) | > loss_gen: 2.3473503589630127 (2.110394390853675) | > loss_kl: 2.016491413116455 (2.202261763650018) | > loss_feat: 4.09830904006958 (3.948562950701327) | > loss_mel: 17.662065505981445 (17.892687101621885) | > loss_duration: 1.5385072231292725 (1.5512310459807113) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.6627254486084 (27.705137098157728) | > grad_norm_1: tensor(11111.2197, device='cuda:0') (tensor(11142.5596, device='cuda:0')) | > current_lr_0: 0.0001749384151682427 | > current_lr_1: 0.0001749384151682427 | > step_time: 0.4469 (0.4353038426992056) | > loader_time: 0.0081 (0.007286387520867425) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671105747491544 (-5.45676325408237e-06) | > avg_step_time: 0.4519856110425063 (-0.0060681087869993156) | > avg_loss_disc: 2.685971072022345 (+0.08614978655962924) | > avg_loss_disc_real_0: 0.13268140262701142 (+0.011192078793972285) | > avg_loss_disc_real_1: 0.2249621798245 (+0.0010559606090397855) | > avg_loss_disc_real_2: 0.2406351711128799 (+0.0053874455707174085) | > avg_loss_disc_real_3: 0.2348065353195432 (+0.0037158559745466) | > avg_loss_disc_real_4: 0.23974057738210114 (-0.002397433552943473) | > avg_loss_disc_real_5: 0.2379252788886218 (-0.007822641814258735) | > avg_loss_0: 2.685971072022345 (+0.08614978655962924) | > avg_grad_norm_0: tensor(398.1660, device='cuda:0') (+tensor(40.9182, device='cuda:0')) | > avg_loss_gen: 2.060530675968653 (-0.05818482855675988) | > avg_loss_kl: 2.1476533379353264 (+0.15276512461648828) | > avg_loss_feat: 3.7153991108209317 (-0.09177588744902243) | > avg_loss_mel: 17.265219352614718 (+0.616883721150149) | > avg_loss_duration: 1.5676324014932337 (+0.02428861067328647) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.756434937598 (+0.643976829421355) | > avg_grad_norm_1: tensor(12844.6592, device='cuda:0') (tensor(-121.3477, device='cuda:0'))  > EPOCH: 1072/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:15:47)   --> TIME: 2025-02-25 19:15:56 -- STEP: 15/72 -- GLOBAL_STEP: 1077200 | > loss_disc: 2.6469945907592773 (2.6122082074483237) | > loss_disc_real_0: 0.17149534821510315 (0.13494238158067068) | > loss_disc_real_1: 0.2016424983739853 (0.21696927348772685) | > loss_disc_real_2: 0.18409736454486847 (0.2400063415368398) | > loss_disc_real_3: 0.2384813278913498 (0.2331279585758845) | > loss_disc_real_4: 0.22208204865455627 (0.24159490764141084) | > loss_disc_real_5: 0.23937661945819855 (0.24818972249825796) | > loss_0: 2.6469945907592773 (2.6122082074483237) | > grad_norm_0: tensor(752.0750, device='cuda:0') (tensor(419.0914, device='cuda:0')) | > loss_gen: 2.199155807495117 (2.1470130761464437) | > loss_kl: 1.9430747032165527 (1.9251877307891845) | > loss_feat: 4.112181186676025 (3.7637115955352782) | > loss_mel: 17.629196166992188 (16.744009717305502) | > loss_duration: 1.5054786205291748 (1.4971216122309368) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.38908576965332 (26.077043787638345) | > grad_norm_1: tensor(17914.3457, device='cuda:0') (tensor(12824.9404, device='cuda:0')) | > current_lr_0: 0.00017491654786634667 | > current_lr_1: 0.00017491654786634667 | > step_time: 0.4207 (0.4313145319620768) | > loader_time: 0.0059 (0.00671844482421875)  --> TIME: 2025-02-25 19:16:19 -- STEP: 65/72 -- GLOBAL_STEP: 1077250 | > loss_disc: 3.051511287689209 (2.710101384382982) | > loss_disc_real_0: 0.06471053510904312 (0.17057800201269302) | > loss_disc_real_1: 0.12640711665153503 (0.2604080231143878) | > loss_disc_real_2: 0.31660202145576477 (0.23614659171838026) | > loss_disc_real_3: 0.19792550802230835 (0.2398082095843095) | > loss_disc_real_4: 0.23175747692584991 (0.2472999075284371) | > loss_disc_real_5: 0.2632181644439697 (0.24525380042883066) | > loss_0: 3.051511287689209 (2.710101384382982) | > grad_norm_0: tensor(161.9600, device='cuda:0') (tensor(341.0910, device='cuda:0')) | > loss_gen: 1.4351078271865845 (2.1648468182637135) | > loss_kl: 3.1042118072509766 (2.261437173990103) | > loss_feat: 2.712948799133301 (3.7384387052976167) | > loss_mel: 18.268753051757812 (17.351435279846193) | > loss_duration: 1.7054041624069214 (1.5581467335040753) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.22642707824707 (27.074304522000826) | > grad_norm_1: tensor(2806.2334, device='cuda:0') (tensor(7523.7617, device='cuda:0')) | > current_lr_0: 0.00017491654786634667 | > current_lr_1: 0.00017491654786634667 | > step_time: 0.4854 (0.4432604166177603) | > loader_time: 0.0118 (0.008237248200636642) --> EVAL PERFORMANCE | > avg_loader_time: 0.008530552958098937 (-0.00014055278939260722) | > avg_step_time: 0.4556119374825921 (+0.003626326440085814) | > avg_loss_disc: 2.7190257932098825 (+0.033054721187537606) | > avg_loss_disc_real_0: 0.17541156362899596 (+0.04273016100198454) | > avg_loss_disc_real_1: 0.2558180737033696 (+0.03085589387886961) | > avg_loss_disc_real_2: 0.23652776697991598 (-0.004107404132963927) | > avg_loss_disc_real_3: 0.2398307506047504 (+0.00502421528520719) | > avg_loss_disc_real_4: 0.24513271765809663 (+0.005392140275995494) | > avg_loss_disc_real_5: 0.2451050424240005 (+0.0071797635353786915) | > avg_loss_0: 2.7190257932098825 (+0.033054721187537606) | > avg_grad_norm_0: tensor(336.1696, device='cuda:0') (tensor(-61.9964, device='cuda:0')) | > avg_loss_gen: 2.1685424841625585 (+0.10801180819390543) | > avg_loss_kl: 2.318928627900674 (+0.1712752899653478) | > avg_loss_feat: 3.6997435361566677 (-0.015655574664263927) | > avg_loss_mel: 17.560310484657823 (+0.29509113204310466) | > avg_loss_duration: 1.569072597463366 (+0.0014401959701322298) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.316597522144587 (+0.5601625845465854) | > avg_grad_norm_1: tensor(7417.1963, device='cuda:0') (tensor(-5427.4629, device='cuda:0'))  > EPOCH: 1073/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:16:24)   --> TIME: 2025-02-25 19:16:46 -- STEP: 43/72 -- GLOBAL_STEP: 1077300 | > loss_disc: 2.7459449768066406 (2.634978876557461) | > loss_disc_real_0: 0.19709265232086182 (0.16022225079494848) | > loss_disc_real_1: 0.24783475697040558 (0.2632765225892843) | > loss_disc_real_2: 0.2390291690826416 (0.23872645303260448) | > loss_disc_real_3: 0.257197767496109 (0.2390034087868624) | > loss_disc_real_4: 0.2687104344367981 (0.2536024884429089) | > loss_disc_real_5: 0.2655375003814697 (0.2502884715795517) | > loss_0: 2.7459449768066406 (2.634978876557461) | > grad_norm_0: tensor(104.9826, device='cuda:0') (tensor(268.4148, device='cuda:0')) | > loss_gen: 2.164451837539673 (2.2135510583256566) | > loss_kl: 2.0200693607330322 (2.1646900620571405) | > loss_feat: 2.811749219894409 (3.600150147149729) | > loss_mel: 14.667630195617676 (17.437003534893655) | > loss_duration: 1.5515525341033936 (1.5418509716211364) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.215452194213867 (26.95724562711494) | > grad_norm_1: tensor(7011., device='cuda:0') (tensor(5745.5879, device='cuda:0')) | > current_lr_0: 0.00017489468329786337 | > current_lr_1: 0.00017489468329786337 | > step_time: 0.4447 (0.4390587751255479) | > loader_time: 0.0096 (0.007437916689140852) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853772566352092 (+7.172705421982714e-06) | > avg_step_time: 0.4513305274533554 (-0.00428141002923671) | > avg_loss_disc: 2.6697417749485504 (-0.04928401826133211) | > avg_loss_disc_real_0: 0.1664472744918205 (-0.008964289137175452) | > avg_loss_disc_real_1: 0.24836678939386148 (-0.007451284309508144) | > avg_loss_disc_real_2: 0.24093673010947 (+0.004408963129554022) | > avg_loss_disc_real_3: 0.24465836374692515 (+0.004827613142174753) | > avg_loss_disc_real_4: 0.25038998311674093 (+0.0052572654586443) | > avg_loss_disc_real_5: 0.25185265129720663 (+0.0067476088732061346) | > avg_loss_0: 2.6697417749485504 (-0.04928401826133211) | > avg_grad_norm_0: tensor(245.3544, device='cuda:0') (tensor(-90.8151, device='cuda:0')) | > avg_loss_gen: 2.1664889026695575 (-0.0020535814930009444) | > avg_loss_kl: 2.1475285761792904 (-0.17140005172138384) | > avg_loss_feat: 3.5929451861851653 (-0.1067983499715024) | > avg_loss_mel: 17.301839009137222 (-0.2584714755206008) | > avg_loss_duration: 1.5626694988197005 (-0.006403098643665395) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.771470942967376 (-0.5451265791772109) | > avg_grad_norm_1: tensor(5743.4473, device='cuda:0') (tensor(-1673.7490, device='cuda:0'))  > EPOCH: 1074/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:17:01)   --> TIME: 2025-02-25 19:17:12 -- STEP: 21/72 -- GLOBAL_STEP: 1077350 | > loss_disc: 3.2536022663116455 (2.7832745143345425) | > loss_disc_real_0: 0.21339410543441772 (0.2013412474521568) | > loss_disc_real_1: 0.1503727287054062 (0.1986590964453561) | > loss_disc_real_2: 0.28801044821739197 (0.2442532181739807) | > loss_disc_real_3: 0.22166770696640015 (0.21943230714116777) | > loss_disc_real_4: 0.2859615385532379 (0.243923537787937) | > loss_disc_real_5: 0.24714407324790955 (0.24991974589370547) | > loss_0: 3.2536022663116455 (2.7832745143345425) | > grad_norm_0: tensor(26.3973, device='cuda:0') (tensor(162.2964, device='cuda:0')) | > loss_gen: 1.7479289770126343 (2.076326852753049) | > loss_kl: 1.9568217992782593 (2.042751567704337) | > loss_feat: 3.3194074630737305 (3.562333118347895) | > loss_mel: 16.574106216430664 (16.897502626691544) | > loss_duration: 1.517014503479004 (1.5054015942982264) | > amp_scaler: 16.0 (8.761904761904763) | > loss_1: 25.115280151367188 (26.08431607200986) | > grad_norm_1: tensor(1655.1865, device='cuda:0') (tensor(4631.3833, device='cuda:0')) | > current_lr_0: 0.00017487282146245114 | > current_lr_1: 0.00017487282146245114 | > step_time: 0.4305 (0.43281441643124535) | > loader_time: 0.0061 (0.006882406416393462)  --> TIME: 2025-02-25 19:17:36 -- STEP: 71/72 -- GLOBAL_STEP: 1077400 | > loss_disc: 2.4049038887023926 (2.690714903280769) | > loss_disc_real_0: 0.13184933364391327 (0.18859247690145403) | > loss_disc_real_1: 0.22339515388011932 (0.2188044452751187) | > loss_disc_real_2: 0.3296583592891693 (0.23580793574662276) | > loss_disc_real_3: 0.2550520598888397 (0.2346509659374264) | > loss_disc_real_4: 0.27454113960266113 (0.2458201224955035) | > loss_disc_real_5: 0.2361937314271927 (0.2496928568457214) | > loss_0: 2.4049038887023926 (2.690714903280769) | > grad_norm_0: tensor(388.7791, device='cuda:0') (tensor(181.6538, device='cuda:0')) | > loss_gen: 2.414862871170044 (2.105146495389266) | > loss_kl: 2.058551073074341 (2.0604686418049787) | > loss_feat: 4.00506591796875 (3.5288247894233375) | > loss_mel: 17.276294708251953 (16.83355946607992) | > loss_duration: 1.5608432292938232 (1.548155986087423) | > amp_scaler: 16.0 (13.859154929577466) | > loss_1: 27.31561851501465 (26.076155434192067) | > grad_norm_1: tensor(6727.9648, device='cuda:0') (tensor(4891.7188, device='cuda:0')) | > current_lr_0: 0.00017487282146245114 | > current_lr_1: 0.00017487282146245114 | > step_time: 0.4936 (0.45309378059817035) | > loader_time: 0.0079 (0.008629714938956248) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629714938956248 (+9.198927543532798e-05) | > avg_step_time: 0.45309378059817035 (+0.001763253144814958) | > avg_loss_disc: 2.690714903280769 (+0.02097312833221876) | > avg_loss_disc_real_0: 0.18859247690145403 (+0.022145202409633524) | > avg_loss_disc_real_1: 0.2188044452751187 (-0.029562344118742784) | > avg_loss_disc_real_2: 0.23580793574662276 (-0.005128794362847244) | > avg_loss_disc_real_3: 0.2346509659374264 (-0.010007397809498747) | > avg_loss_disc_real_4: 0.2458201224955035 (-0.0045698606212374315) | > avg_loss_disc_real_5: 0.2496928568457214 (-0.0021597944514852307) | > avg_loss_0: 2.690714903280769 (+0.02097312833221876) | > avg_grad_norm_0: tensor(181.6538, device='cuda:0') (tensor(-63.7007, device='cuda:0')) | > avg_loss_gen: 2.105146495389266 (-0.0613424072802915) | > avg_loss_kl: 2.0604686418049787 (-0.0870599343743117) | > avg_loss_feat: 3.5288247894233375 (-0.06412039676182779) | > avg_loss_mel: 16.83355946607992 (-0.468279543057303) | > avg_loss_duration: 1.548155986087423 (-0.014513512732277523) | > avg_amp_scaler: 13.859154929577466 (+5.859154929577466) | > avg_loss_1: 26.076155434192067 (-0.6953155087753089) | > avg_grad_norm_1: tensor(4891.7188, device='cuda:0') (tensor(-851.7285, device='cuda:0'))  > EPOCH: 1075/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:17:38)   --> TIME: 2025-02-25 19:18:02 -- STEP: 49/72 -- GLOBAL_STEP: 1077450 | > loss_disc: 2.467661142349243 (2.623188266948777) | > loss_disc_real_0: 0.05106303095817566 (0.1435929498806291) | > loss_disc_real_1: 0.20963549613952637 (0.24411635830694312) | > loss_disc_real_2: 0.2604941427707672 (0.2578205566625206) | > loss_disc_real_3: 0.229491725564003 (0.23121504090270217) | > loss_disc_real_4: 0.2390851080417633 (0.23645870265912036) | > loss_disc_real_5: 0.24089299142360687 (0.24076078254349378) | > loss_0: 2.467661142349243 (2.623188266948777) | > grad_norm_0: tensor(206.1129, device='cuda:0') (tensor(286.9183, device='cuda:0')) | > loss_gen: 1.9613639116287231 (2.1774003165108824) | > loss_kl: 2.236715793609619 (2.0852380917996776) | > loss_feat: 3.9047882556915283 (3.894167797906058) | > loss_mel: 19.89446258544922 (17.68957612952408) | > loss_duration: 1.6785755157470703 (1.5450790171720543) | > amp_scaler: 8.0 (10.285714285714286) | > loss_1: 29.675907135009766 (27.3914612361363) | > grad_norm_1: tensor(16049.2227, device='cuda:0') (tensor(6038.4458, device='cuda:0')) | > current_lr_0: 0.0001748509623597683 | > current_lr_1: 0.0001748509623597683 | > step_time: 0.4355 (0.4378523777942268) | > loader_time: 0.0098 (0.007783680546040438) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685931353501873 (+5.6216414545624815e-05) | > avg_step_time: 0.45108291128991357 (-0.0020108693082567797) | > avg_loss_disc: 2.6183949389927825 (-0.0723199642879866) | > avg_loss_disc_real_0: 0.14238274706081594 (-0.04620972984063809) | > avg_loss_disc_real_1: 0.23540328065274466 (+0.016598835377625965) | > avg_loss_disc_real_2: 0.24915727049532072 (+0.013349334748697955) | > avg_loss_disc_real_3: 0.2289045079493187 (-0.005746457988107717) | > avg_loss_disc_real_4: 0.2372230736302658 (-0.00859704886523771) | > avg_loss_disc_real_5: 0.23954780966463224 (-0.010145047181089162) | > avg_loss_0: 2.6183949389927825 (-0.0723199642879866) | > avg_grad_norm_0: tensor(281.3001, device='cuda:0') (+tensor(99.6464, device='cuda:0')) | > avg_loss_gen: 2.145733994497381 (+0.040587499108114766) | > avg_loss_kl: 2.122806626306453 (+0.06233798450147443) | > avg_loss_feat: 3.8373008378794498 (+0.3084760484561122) | > avg_loss_mel: 17.640064749919194 (+0.8065052838392752) | > avg_loss_duration: 1.5646660260751213 (+0.01651003998769829) | > avg_amp_scaler: 9.577464788732394 (-4.281690140845072) | > avg_loss_1: 27.310572234677597 (+1.2344168004855298) | > avg_grad_norm_1: tensor(7044.0815, device='cuda:0') (+tensor(2152.3628, device='cuda:0'))  > EPOCH: 1076/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:18:15)   --> TIME: 2025-02-25 19:18:29 -- STEP: 27/72 -- GLOBAL_STEP: 1077500 | > loss_disc: 2.8794236183166504 (2.7691437668270535) | > loss_disc_real_0: 0.1915690153837204 (0.16479598420361677) | > loss_disc_real_1: 0.29848262667655945 (0.2513605461076454) | > loss_disc_real_2: 0.2211018055677414 (0.22923907416838188) | > loss_disc_real_3: 0.291322261095047 (0.2392221098696744) | > loss_disc_real_4: 0.2874677777290344 (0.2471695481627076) | > loss_disc_real_5: 0.2600576877593994 (0.24912369140872248) | > loss_0: 2.8794236183166504 (2.7691437668270535) | > grad_norm_0: tensor(82.3930, device='cuda:0') (tensor(208.8066, device='cuda:0')) | > loss_gen: 2.031428813934326 (2.0166817108790083) | > loss_kl: 3.1130216121673584 (2.528551450482121) | > loss_feat: 3.340019941329956 (3.6865100065867105) | > loss_mel: 23.823766708374023 (19.609867873015226) | > loss_duration: 1.6252936124801636 (1.6425757628900033) | > amp_scaler: 8.0 (8.0) | > loss_1: 33.933528900146484 (29.484186878910773) | > grad_norm_1: tensor(8050.2881, device='cuda:0') (tensor(6888.9365, device='cuda:0')) | > current_lr_0: 0.00017482910598947334 | > current_lr_1: 0.00017482910598947334 | > step_time: 0.434 (0.43474743101331925) | > loader_time: 0.008 (0.007064042267975984) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593290624484209 (-9.264072901766368e-05) | > avg_step_time: 0.45209580072214905 (+0.0010128894322354864) | > avg_loss_disc: 2.717342010686096 (+0.0989470716933134) | > avg_loss_disc_real_0: 0.1678019679925391 (+0.025419220931723163) | > avg_loss_disc_real_1: 0.23442575608340788 (-0.0009775245693367796) | > avg_loss_disc_real_2: 0.24124486492553227 (-0.00791240556978845) | > avg_loss_disc_real_3: 0.2341019716900839 (+0.0051974637407652124) | > avg_loss_disc_real_4: 0.24122117627674425 (+0.0039981026464784575) | > avg_loss_disc_real_5: 0.24530470014457972 (+0.005756890479947485) | > avg_loss_0: 2.717342010686096 (+0.0989470716933134) | > avg_grad_norm_0: tensor(249.8234, device='cuda:0') (tensor(-31.4768, device='cuda:0')) | > avg_loss_gen: 2.0363890335593418 (-0.10934496093803903) | > avg_loss_kl: 2.4025093182711528 (+0.27970269196469966) | > avg_loss_feat: 3.650262913233797 (-0.1870379246456526) | > avg_loss_mel: 18.73213327434701 (+1.092068524427816) | > avg_loss_duration: 1.6268982601837374 (+0.062232234108616113) | > avg_amp_scaler: 8.0 (-1.577464788732394) | > avg_loss_1: 28.448192865076198 (+1.137620630398601) | > avg_grad_norm_1: tensor(6601.4336, device='cuda:0') (tensor(-442.6479, device='cuda:0'))  > EPOCH: 1077/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:18:51)   --> TIME: 2025-02-25 19:18:55 -- STEP: 5/72 -- GLOBAL_STEP: 1077550 | > loss_disc: 2.568971872329712 (2.6562677383422852) | > loss_disc_real_0: 0.11897851526737213 (0.11617112904787064) | > loss_disc_real_1: 0.19902050495147705 (0.20261462926864623) | > loss_disc_real_2: 0.16935600340366364 (0.23872241973876954) | > loss_disc_real_3: 0.21623803675174713 (0.22992075085639954) | > loss_disc_real_4: 0.24107125401496887 (0.24999410212039946) | > loss_disc_real_5: 0.240068718791008 (0.24346213340759276) | > loss_0: 2.568971872329712 (2.6562677383422852) | > grad_norm_0: tensor(52.5345, device='cuda:0') (tensor(92.4495, device='cuda:0')) | > loss_gen: 2.376951217651367 (2.0124920845031737) | > loss_kl: 1.9957904815673828 (1.9215795040130614) | > loss_feat: 4.467237949371338 (3.517339849472046) | > loss_mel: 17.228620529174805 (16.829669189453124) | > loss_duration: 1.5185952186584473 (1.532480525970459) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.587194442749023 (25.813561248779298) | > grad_norm_1: tensor(2510.1912, device='cuda:0') (tensor(5391.9214, device='cuda:0')) | > current_lr_0: 0.00017480725235122466 | > current_lr_1: 0.00017480725235122466 | > step_time: 0.4193 (0.4245706081390381) | > loader_time: 0.0076 (0.0061018943786621095)  --> TIME: 2025-02-25 19:19:18 -- STEP: 55/72 -- GLOBAL_STEP: 1077600 | > loss_disc: 2.577746868133545 (2.7161140138452704) | > loss_disc_real_0: 0.1505872905254364 (0.1510183150456711) | > loss_disc_real_1: 0.23762179911136627 (0.24733450196006085) | > loss_disc_real_2: 0.23266583681106567 (0.247159771214832) | > loss_disc_real_3: 0.25677189230918884 (0.23556989458474245) | > loss_disc_real_4: 0.2347843199968338 (0.24274534664370798) | > loss_disc_real_5: 0.21851961314678192 (0.2443395273251967) | > loss_0: 2.577746868133545 (2.7161140138452704) | > grad_norm_0: tensor(440.7683, device='cuda:0') (tensor(265.2338, device='cuda:0')) | > loss_gen: 2.1191728115081787 (2.0838921416889535) | > loss_kl: 2.071493625640869 (2.167854514988986) | > loss_feat: 3.8261911869049072 (3.732643625952981) | > loss_mel: 16.164295196533203 (17.514759427850894) | > loss_duration: 1.6119756698608398 (1.5762459148060193) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.793128967285156 (27.07539572282271) | > grad_norm_1: tensor(3145.7776, device='cuda:0') (tensor(7090.8286, device='cuda:0')) | > current_lr_0: 0.00017480725235122466 | > current_lr_1: 0.00017480725235122466 | > step_time: 0.4763 (0.4433116696097634) | > loader_time: 0.0095 (0.007790179686112837) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850707040706151 (-8.622021742269934e-05) | > avg_step_time: 0.45397331681050046 (+0.0018775160883514053) | > avg_loss_disc: 2.706228706198679 (-0.011113304487416986) | > avg_loss_disc_real_0: 0.14975120287231158 (-0.01805076512022752) | > avg_loss_disc_real_1: 0.24487246635933996 (+0.01044671027593208) | > avg_loss_disc_real_2: 0.2456236469074035 (+0.0043787819818712215) | > avg_loss_disc_real_3: 0.23809241220145158 (+0.003990440511367682) | > avg_loss_disc_real_4: 0.2416594689580756 (+0.00043829268133135924) | > avg_loss_disc_real_5: 0.24062205704165177 (-0.004682643102927958) | > avg_loss_0: 2.706228706198679 (-0.011113304487416986) | > avg_grad_norm_0: tensor(252.0158, device='cuda:0') (+tensor(2.1924, device='cuda:0')) | > avg_loss_gen: 2.065595693991217 (+0.029206660431875253) | > avg_loss_kl: 2.166318419953467 (-0.2361908983176857) | > avg_loss_feat: 3.6223185431789346 (-0.0279443700548625) | > avg_loss_mel: 17.3535902265092 (-1.378543047837809) | > avg_loss_duration: 1.5829163218887758 (-0.04398193829496155) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.79073919376857 (-1.6574536713076284) | > avg_grad_norm_1: tensor(6893.6011, device='cuda:0') (+tensor(292.1675, device='cuda:0'))  > EPOCH: 1078/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:19:28)   --> TIME: 2025-02-25 19:19:45 -- STEP: 33/72 -- GLOBAL_STEP: 1077650 | > loss_disc: 2.680013656616211 (2.639018550063625) | > loss_disc_real_0: 0.06108392775058746 (0.12563062797893174) | > loss_disc_real_1: 0.37314027547836304 (0.2437256859107451) | > loss_disc_real_2: 0.226075679063797 (0.23732752646460678) | > loss_disc_real_3: 0.34499019384384155 (0.24948688393289392) | > loss_disc_real_4: 0.2870396375656128 (0.24616643199414917) | > loss_disc_real_5: 0.23938041925430298 (0.24715775432008685) | > loss_0: 2.680013656616211 (2.639018550063625) | > grad_norm_0: tensor(146.0748, device='cuda:0') (tensor(241.3367, device='cuda:0')) | > loss_gen: 2.2942192554473877 (2.0945514187668306) | > loss_kl: 1.9394129514694214 (1.9709274913325454) | > loss_feat: 4.524436950683594 (3.7006477948391074) | > loss_mel: 16.622882843017578 (16.975527994560476) | > loss_duration: 1.5072565078735352 (1.5104094779852666) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.88821029663086 (26.252064329205137) | > grad_norm_1: tensor(1511.8177, device='cuda:0') (tensor(6054.6309, device='cuda:0')) | > current_lr_0: 0.00017478540144468075 | > current_lr_1: 0.00017478540144468075 | > step_time: 0.4457 (0.4313601219292843) | > loader_time: 0.0078 (0.0070699633974017516) --> EVAL PERFORMANCE | > avg_loader_time: 0.008598451883020535 (+9.138147595902542e-05) | > avg_step_time: 0.4500824196237913 (-0.0038908971867091524) | > avg_loss_disc: 2.680902988138333 (-0.025325718060345892) | > avg_loss_disc_real_0: 0.1607678736441992 (+0.011016670771887616) | > avg_loss_disc_real_1: 0.2373072408454519 (-0.007565225513888052) | > avg_loss_disc_real_2: 0.23828451566293207 (-0.007339131244471414) | > avg_loss_disc_real_3: 0.23999679403405794 (+0.0019043818326063555) | > avg_loss_disc_real_4: 0.2450596465611122 (+0.003400177603036597) | > avg_loss_disc_real_5: 0.246181748492617 (+0.005559691450965243) | > avg_loss_0: 2.680902988138333 (-0.025325718060345892) | > avg_grad_norm_0: tensor(279.0825, device='cuda:0') (+tensor(27.0667, device='cuda:0')) | > avg_loss_gen: 2.0539441175863775 (-0.011651576404839492) | > avg_loss_kl: 2.1327045467537893 (-0.03361387319967779) | > avg_loss_feat: 3.575872263438265 (-0.04644627974066973) | > avg_loss_mel: 17.587090384792273 (+0.23350015828307136) | > avg_loss_duration: 1.5582118353373569 (-0.02470448655141899) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.907823186525157 (+0.11708399275658721) | > avg_grad_norm_1: tensor(6697.0288, device='cuda:0') (tensor(-196.5723, device='cuda:0'))  > EPOCH: 1079/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:20:05)   --> TIME: 2025-02-25 19:20:12 -- STEP: 11/72 -- GLOBAL_STEP: 1077700 | > loss_disc: 2.575890302658081 (2.586003455248746) | > loss_disc_real_0: 0.12407486140727997 (0.11439103396101431) | > loss_disc_real_1: 0.24083362519741058 (0.2250185635956851) | > loss_disc_real_2: 0.24072027206420898 (0.2363195080648769) | > loss_disc_real_3: 0.26001042127609253 (0.22310124066742984) | > loss_disc_real_4: 0.24550658464431763 (0.23703443326733328) | > loss_disc_real_5: 0.24778220057487488 (0.23887029832059686) | > loss_0: 2.575890302658081 (2.586003455248746) | > grad_norm_0: tensor(193.1723, device='cuda:0') (tensor(289.0645, device='cuda:0')) | > loss_gen: 2.1068854331970215 (2.024125261740251) | > loss_kl: 2.179204225540161 (2.1229070750149814) | > loss_feat: 3.3560216426849365 (3.7007881944829766) | > loss_mel: 17.973270416259766 (18.33873870156028) | > loss_duration: 1.5010910034179688 (1.5342714678157459) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.116472244262695 (27.720830743963067) | > grad_norm_1: tensor(7234.0107, device='cuda:0') (tensor(8774.3291, device='cuda:0')) | > current_lr_0: 0.00017476355326950015 | > current_lr_1: 0.00017476355326950015 | > step_time: 0.4237 (0.43100614981217816) | > loader_time: 0.0058 (0.00626152211969549)  --> TIME: 2025-02-25 19:20:35 -- STEP: 61/72 -- GLOBAL_STEP: 1077750 | > loss_disc: 2.6252315044403076 (2.60671428774224) | > loss_disc_real_0: 0.15783551335334778 (0.14139536914766807) | > loss_disc_real_1: 0.3937070667743683 (0.22241861922819106) | > loss_disc_real_2: 0.2398480772972107 (0.23747695860315543) | > loss_disc_real_3: 0.2834360897541046 (0.23681073525889976) | > loss_disc_real_4: 0.25910577178001404 (0.2430146825118143) | > loss_disc_real_5: 0.2490154504776001 (0.24699885019513426) | > loss_0: 2.6252315044403076 (2.60671428774224) | > grad_norm_0: tensor(379.3163, device='cuda:0') (tensor(269.8939, device='cuda:0')) | > loss_gen: 2.8120625019073486 (2.1060544936383367) | > loss_kl: 2.2606117725372314 (2.1461001243747644) | > loss_feat: 4.090699672698975 (3.6624230400460664) | > loss_mel: 17.36707878112793 (17.816287806776707) | > loss_duration: 1.5905442237854004 (1.5546727512703569) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.12099838256836 (27.28553822001473) | > grad_norm_1: tensor(1682.1471, device='cuda:0') (tensor(9664.7646, device='cuda:0')) | > current_lr_0: 0.00017476355326950015 | > current_lr_1: 0.00017476355326950015 | > step_time: 0.4522 (0.43826383059142066) | > loader_time: 0.0108 (0.008094213047965627) --> EVAL PERFORMANCE | > avg_loader_time: 0.00857067779755928 (-2.7774085461255635e-05) | > avg_step_time: 0.4454454838390082 (-0.0046369357847830805) | > avg_loss_disc: 2.60587485071639 (-0.07502813742194325) | > avg_loss_disc_real_0: 0.1310495038215123 (-0.029718369822686913) | > avg_loss_disc_real_1: 0.2302872316098549 (-0.007020009235597008) | > avg_loss_disc_real_2: 0.237988970439199 (-0.000295545223733068) | > avg_loss_disc_real_3: 0.23538836165213248 (-0.004608432381925459) | > avg_loss_disc_real_4: 0.2450875503496385 (+2.7903788526295603e-05) | > avg_loss_disc_real_5: 0.24681011681825343 (+0.0006283683256364248) | > avg_loss_0: 2.60587485071639 (-0.07502813742194325) | > avg_grad_norm_0: tensor(267.0966, device='cuda:0') (tensor(-11.9859, device='cuda:0')) | > avg_loss_gen: 2.1160543770857267 (+0.06211025949934923) | > avg_loss_kl: 2.150346569611992 (+0.01764202285820282) | > avg_loss_feat: 3.700097208291712 (+0.12422494485344693) | > avg_loss_mel: 17.728890298118056 (+0.14179991332578368) | > avg_loss_duration: 1.5619131746426436 (+0.0037013393052867105) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.257301652935194 (+0.3494784664100372) | > avg_grad_norm_1: tensor(9066.4492, device='cuda:0') (+tensor(2369.4204, device='cuda:0'))  > EPOCH: 1080/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:20:41)   --> TIME: 2025-02-25 19:21:01 -- STEP: 39/72 -- GLOBAL_STEP: 1077800 | > loss_disc: 2.7439985275268555 (2.6912124279217844) | > loss_disc_real_0: 0.1822415590286255 (0.20537165036568275) | > loss_disc_real_1: 0.2593221962451935 (0.23407498173988783) | > loss_disc_real_2: 0.31887558102607727 (0.23638518727742708) | > loss_disc_real_3: 0.24738554656505585 (0.2330339577717659) | > loss_disc_real_4: 0.2783201336860657 (0.24308578708232978) | > loss_disc_real_5: 0.27370649576187134 (0.24662478994100523) | > loss_0: 2.7439985275268555 (2.6912124279217844) | > grad_norm_0: tensor(30.5274, device='cuda:0') (tensor(234.7728, device='cuda:0')) | > loss_gen: 1.8326877355575562 (2.0918849125886574) | > loss_kl: 1.9565025568008423 (1.9948348999023438) | > loss_feat: 2.8093318939208984 (3.5312807132036257) | > loss_mel: 16.804630279541016 (16.96502675765599) | > loss_duration: 1.5592641830444336 (1.514755035058046) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.962417602539062 (26.097782135009766) | > grad_norm_1: tensor(5928.2656, device='cuda:0') (tensor(6336.8486, device='cuda:0')) | > current_lr_0: 0.00017474170782534146 | > current_lr_1: 0.00017474170782534146 | > step_time: 0.4395 (0.431940726744823) | > loader_time: 0.0083 (0.00744167352334047) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722748554928201 (+0.00015207075736892173) | > avg_step_time: 0.44704414085603095 (+0.00159865701702272) | > avg_loss_disc: 2.66728847463366 (+0.06141362391727023) | > avg_loss_disc_real_0: 0.18883383405250562 (+0.05778433023099333) | > avg_loss_disc_real_1: 0.22665002679740878 (-0.0036372048124461154) | > avg_loss_disc_real_2: 0.23420301646413938 (-0.003785953975059625) | > avg_loss_disc_real_3: 0.23476336518643606 (-0.0006249964656964146) | > avg_loss_disc_real_4: 0.2441053457663093 (-0.0009822045833292048) | > avg_loss_disc_real_5: 0.24821463605047953 (+0.0014045192322260924) | > avg_loss_0: 2.66728847463366 (+0.06141362391727023) | > avg_grad_norm_0: tensor(216.9833, device='cuda:0') (tensor(-50.1133, device='cuda:0')) | > avg_loss_gen: 2.100699114127898 (-0.015355262957828852) | > avg_loss_kl: 2.0354065374589303 (-0.11494003215306181) | > avg_loss_feat: 3.528022292634131 (-0.1720749156575807) | > avg_loss_mel: 16.98585844711518 (-0.7430318510028755) | > avg_loss_duration: 1.5464921467740773 (-0.01542102786856625) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.196478373567825 (-1.0608232793673693) | > avg_grad_norm_1: tensor(6485.4160, device='cuda:0') (tensor(-2581.0332, device='cuda:0'))  > EPOCH: 1081/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:21:18)   --> TIME: 2025-02-25 19:21:27 -- STEP: 17/72 -- GLOBAL_STEP: 1077850 | > loss_disc: 2.495304822921753 (2.743711191065171) | > loss_disc_real_0: 0.045937031507492065 (0.11160618461230222) | > loss_disc_real_1: 0.25499722361564636 (0.22617805004119873) | > loss_disc_real_2: 0.22898602485656738 (0.22431686783538146) | > loss_disc_real_3: 0.18140274286270142 (0.21316271056147182) | > loss_disc_real_4: 0.1988803893327713 (0.23855329524068272) | > loss_disc_real_5: 0.2299613505601883 (0.24954741316683152) | > loss_0: 2.495304822921753 (2.743711191065171) | > grad_norm_0: tensor(217.7944, device='cuda:0') (tensor(261.4173, device='cuda:0')) | > loss_gen: 2.0878899097442627 (1.9249604309306425) | > loss_kl: 2.268353223800659 (2.4910781032898845) | > loss_feat: 4.0062408447265625 (3.732623380773208) | > loss_mel: 17.87718391418457 (20.57931698069853) | > loss_duration: 1.5399260520935059 (1.5408763184266931) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.77959632873535 (30.26885503881118) | > grad_norm_1: tensor(8657.7129, device='cuda:0') (tensor(7188.0288, device='cuda:0')) | > current_lr_0: 0.00017471986511186327 | > current_lr_1: 0.00017471986511186327 | > step_time: 0.4499 (0.43195422957925234) | > loader_time: 0.0072 (0.006659577874576344)  --> TIME: 2025-02-25 19:21:51 -- STEP: 67/72 -- GLOBAL_STEP: 1077900 | > loss_disc: 2.5050599575042725 (2.6895567445612665) | > loss_disc_real_0: 0.06971483677625656 (0.1601662179872171) | > loss_disc_real_1: 0.2338196486234665 (0.22608052594448202) | > loss_disc_real_2: 0.2457638382911682 (0.23603277575613849) | > loss_disc_real_3: 0.25538304448127747 (0.229075866181459) | > loss_disc_real_4: 0.2622434198856354 (0.24175447047646367) | > loss_disc_real_5: 0.2626071572303772 (0.24648250594957552) | > loss_0: 2.5050599575042725 (2.6895567445612665) | > grad_norm_0: tensor(155.1517, device='cuda:0') (tensor(206.8846, device='cuda:0')) | > loss_gen: 2.23075532913208 (2.0748079534786843) | > loss_kl: 2.039787769317627 (2.1811157091340028) | > loss_feat: 3.6548895835876465 (3.524423499605549) | > loss_mel: 16.32090950012207 (17.808327247847373) | > loss_duration: 1.6441365480422974 (1.5570463369141763) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.890478134155273 (27.145720581510172) | > grad_norm_1: tensor(3772.5229, device='cuda:0') (tensor(4854.3882, device='cuda:0')) | > current_lr_0: 0.00017471986511186327 | > current_lr_1: 0.00017471986511186327 | > step_time: 0.5023 (0.4479735253462151) | > loader_time: 0.0121 (0.008415236401913771) --> EVAL PERFORMANCE | > avg_loader_time: 0.008589291236769984 (-0.0001334573181582168) | > avg_step_time: 0.4518342286768094 (+0.004790087820778444) | > avg_loss_disc: 2.6873511999425754 (+0.020062725308915397) | > avg_loss_disc_real_0: 0.15512286709018155 (-0.03371096696232406) | > avg_loss_disc_real_1: 0.22503574003635998 (-0.0016142867610487999) | > avg_loss_disc_real_2: 0.23439544397340695 (+0.00019242750926756758) | > avg_loss_disc_real_3: 0.2310038187973936 (-0.0037595463890424674) | > avg_loss_disc_real_4: 0.242821947789528 (-0.0012833979767812864) | > avg_loss_disc_real_5: 0.24768039437246994 (-0.0005342416780095838) | > avg_loss_0: 2.6873511999425754 (+0.020062725308915397) | > avg_grad_norm_0: tensor(210.1097, device='cuda:0') (tensor(-6.8736, device='cuda:0')) | > avg_loss_gen: 2.0648642929506966 (-0.03583482117720127) | > avg_loss_kl: 2.174931495962009 (+0.1395249585030789) | > avg_loss_feat: 3.51713987807153 (-0.010882414562601195) | > avg_loss_mel: 17.73917462792195 (+0.7533161808067703) | > avg_loss_duration: 1.5594685530998338 (+0.012976406325756473) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.055578742228764 (+0.8591003686609398) | > avg_grad_norm_1: tensor(4950.4790, device='cuda:0') (tensor(-1534.9370, device='cuda:0'))  > EPOCH: 1082/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:21:55)   --> TIME: 2025-02-25 19:22:17 -- STEP: 45/72 -- GLOBAL_STEP: 1077950 | > loss_disc: 2.5846800804138184 (2.729008139504327) | > loss_disc_real_0: 0.12768854200839996 (0.20808239628871278) | > loss_disc_real_1: 0.2620224058628082 (0.24427246219582027) | > loss_disc_real_2: 0.24047063291072845 (0.22239169776439666) | > loss_disc_real_3: 0.24013307690620422 (0.23233561217784882) | > loss_disc_real_4: 0.263378381729126 (0.23454234566953447) | > loss_disc_real_5: 0.2499881535768509 (0.23522653745280372) | > loss_0: 2.5846800804138184 (2.729008139504327) | > grad_norm_0: tensor(106.1793, device='cuda:0') (tensor(233.2204, device='cuda:0')) | > loss_gen: 2.005424976348877 (2.0425235191980993) | > loss_kl: 2.0371644496917725 (2.2959412574768066) | > loss_feat: 3.425459384918213 (3.448252142800225) | > loss_mel: 16.891157150268555 (18.53306295606825) | > loss_duration: 1.583021640777588 (1.6389503531985812) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.94222640991211 (27.958730019463435) | > grad_norm_1: tensor(12335.3418, device='cuda:0') (tensor(6326.2085, device='cuda:0')) | > current_lr_0: 0.00017469802512872428 | > current_lr_1: 0.00017469802512872428 | > step_time: 0.4733 (0.4421819951799181) | > loader_time: 0.0086 (0.00752107302347819) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722657888707982 (+0.00013336665193799772) | > avg_step_time: 0.45389519946675905 (+0.0020609707899496565) | > avg_loss_disc: 2.704205479420407 (+0.01685427947783147) | > avg_loss_disc_real_0: 0.19920550160844558 (+0.044082634518264024) | > avg_loss_disc_real_1: 0.24244783042182386 (+0.017412090385463874) | > avg_loss_disc_real_2: 0.22859125426957305 (-0.005804189703833895) | > avg_loss_disc_real_3: 0.23065243388565493 (-0.00035138491173866693) | > avg_loss_disc_real_4: 0.2368475312078503 (-0.005974416581677716) | > avg_loss_disc_real_5: 0.23830127758039554 (-0.009379116792074399) | > avg_loss_0: 2.704205479420407 (+0.01685427947783147) | > avg_grad_norm_0: tensor(218.3154, device='cuda:0') (+tensor(8.2057, device='cuda:0')) | > avg_loss_gen: 2.078062500752193 (+0.013198207801496498) | > avg_loss_kl: 2.2511028689397894 (+0.0761713729777802) | > avg_loss_feat: 3.482702866406508 (-0.03443701166502189) | > avg_loss_mel: 18.07353605350978 (+0.33436142558782933) | > avg_loss_duration: 1.63016578513132 (+0.07069723203148626) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.5155699555303 (+0.45999121330153514) | > avg_grad_norm_1: tensor(6572.7412, device='cuda:0') (+tensor(1622.2622, device='cuda:0'))  > EPOCH: 1083/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:22:32)   --> TIME: 2025-02-25 19:22:44 -- STEP: 23/72 -- GLOBAL_STEP: 1078000 | > loss_disc: 2.9065403938293457 (2.7096109701239546) | > loss_disc_real_0: 0.3481854200363159 (0.2034720655368722) | > loss_disc_real_1: 0.07827696204185486 (0.21015792275252548) | > loss_disc_real_2: 0.21117974817752838 (0.23136365154515143) | > loss_disc_real_3: 0.16666127741336823 (0.23585188129673834) | > loss_disc_real_4: 0.22309371829032898 (0.23875866830348969) | > loss_disc_real_5: 0.22075305879116058 (0.2413944906514624) | > loss_0: 2.9065403938293457 (2.7096109701239546) | > grad_norm_0: tensor(113.8158, device='cuda:0') (tensor(168.5583, device='cuda:0')) | > loss_gen: 1.9028115272521973 (2.0453373401061343) | > loss_kl: 2.0885560512542725 (2.036546235499175) | > loss_feat: 3.601893424987793 (3.52882340679998) | > loss_mel: 18.00368881225586 (17.228089788685676) | > loss_duration: 1.5290193557739258 (1.524208048115606) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.12596893310547 (26.363004850304645) | > grad_norm_1: tensor(4037.4370, device='cuda:0') (tensor(6013.0981, device='cuda:0')) | > current_lr_0: 0.0001746761878755832 | > current_lr_1: 0.0001746761878755832 | > step_time: 0.4326 (0.43737006187438965) | > loader_time: 0.0062 (0.006946232007897418) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624123855375908 (-9.853403333207403e-05) | > avg_step_time: 0.45444663141814756 (+0.0005514319513885146) | > avg_loss_disc: 2.6969287261156967 (-0.007276753304710226) | > avg_loss_disc_real_0: 0.1655552854210558 (-0.03365021618738978) | > avg_loss_disc_real_1: 0.22298035518804066 (-0.0194674752337832) | > avg_loss_disc_real_2: 0.23046561907714522 (+0.0018743648075721653) | > avg_loss_disc_real_3: 0.22488848969969952 (-0.005763944185955411) | > avg_loss_disc_real_4: 0.24175700314447912 (+0.004909471936628829) | > avg_loss_disc_real_5: 0.24074747675741223 (+0.002446199177016689) | > avg_loss_0: 2.6969287261156967 (-0.007276753304710226) | > avg_grad_norm_0: tensor(167.8621, device='cuda:0') (tensor(-50.4533, device='cuda:0')) | > avg_loss_gen: 2.0061266674122358 (-0.07193583333995734) | > avg_loss_kl: 2.2105660656808137 (-0.040536803258975684) | > avg_loss_feat: 3.5114552638900114 (+0.028752397483503334) | > avg_loss_mel: 17.694654599042003 (-0.378881454467777) | > avg_loss_duration: 1.5652747473246615 (-0.0648910378066585) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.98807740547287 (-0.5274925500574312) | > avg_grad_norm_1: tensor(7254.3262, device='cuda:0') (+tensor(681.5850, device='cuda:0'))  > EPOCH: 1084/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:23:08)   --> TIME: 2025-02-25 19:23:11 -- STEP: 1/72 -- GLOBAL_STEP: 1078050 | > loss_disc: 2.4699974060058594 (2.4699974060058594) | > loss_disc_real_0: 0.10065777599811554 (0.10065777599811554) | > loss_disc_real_1: 0.22760766744613647 (0.22760766744613647) | > loss_disc_real_2: 0.24532128870487213 (0.24532128870487213) | > loss_disc_real_3: 0.20691874623298645 (0.20691874623298645) | > loss_disc_real_4: 0.23717978596687317 (0.23717978596687317) | > loss_disc_real_5: 0.23125165700912476 (0.23125165700912476) | > loss_0: 2.4699974060058594 (2.4699974060058594) | > grad_norm_0: tensor(44.3960, device='cuda:0') (tensor(44.3960, device='cuda:0')) | > loss_gen: 2.529702663421631 (2.529702663421631) | > loss_kl: 1.8144299983978271 (1.8144299983978271) | > loss_feat: 4.329410552978516 (4.329410552978516) | > loss_mel: 16.61546516418457 (16.61546516418457) | > loss_duration: 1.5191835165023804 (1.5191835165023804) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.80819320678711 (26.80819320678711) | > grad_norm_1: tensor(4842.9248, device='cuda:0') (tensor(4842.9248, device='cuda:0')) | > current_lr_0: 0.00017465435335209875 | > current_lr_1: 0.00017465435335209875 | > step_time: 0.4248 (0.4247729778289795) | > loader_time: 0.0057 (0.005694866180419922)  --> TIME: 2025-02-25 19:23:34 -- STEP: 51/72 -- GLOBAL_STEP: 1078100 | > loss_disc: 2.729024648666382 (2.619334534102795) | > loss_disc_real_0: 0.32647883892059326 (0.11739841987397157) | > loss_disc_real_1: 0.2172386646270752 (0.22484610974788666) | > loss_disc_real_2: 0.2774314284324646 (0.25156364166269113) | > loss_disc_real_3: 0.2555360198020935 (0.23797273898825927) | > loss_disc_real_4: 0.26424986124038696 (0.2439434835139443) | > loss_disc_real_5: 0.2678394615650177 (0.24742902201764724) | > loss_0: 2.729024648666382 (2.619334534102795) | > grad_norm_0: tensor(337.8483, device='cuda:0') (tensor(191.4399, device='cuda:0')) | > loss_gen: 2.431364059448242 (2.1575162901597857) | > loss_kl: 1.951590895652771 (2.0112604463801658) | > loss_feat: 3.5251238346099854 (3.7725419063194123) | > loss_mel: 15.934852600097656 (16.973648800569425) | > loss_duration: 1.5601465702056885 (1.5323407439624561) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.403078079223633 (26.447308072856828) | > grad_norm_1: tensor(1516.4650, device='cuda:0') (tensor(5341.1875, device='cuda:0')) | > current_lr_0: 0.00017465435335209875 | > current_lr_1: 0.00017465435335209875 | > step_time: 0.4447 (0.4425046724431655) | > loader_time: 0.0096 (0.007957341624241249) --> EVAL PERFORMANCE | > avg_loader_time: 0.00873897780834789 (+0.0001148539529719824) | > avg_step_time: 0.46147859600228325 (+0.007031964584135686) | > avg_loss_disc: 2.624227977134812 (-0.07270074898088463) | > avg_loss_disc_real_0: 0.11664500727619924 (-0.04891027814485656) | > avg_loss_disc_real_1: 0.2237295701470174 (+0.0007492149589767294) | > avg_loss_disc_real_2: 0.2484371930780545 (+0.01797157400090929) | > avg_loss_disc_real_3: 0.2375209006205411 (+0.01263241092084158) | > avg_loss_disc_real_4: 0.24353099184136995 (+0.001773988696890827) | > avg_loss_disc_real_5: 0.2475802589889983 (+0.006832782231586065) | > avg_loss_0: 2.624227977134812 (-0.07270074898088463) | > avg_grad_norm_0: tensor(191.9561, device='cuda:0') (+tensor(24.0940, device='cuda:0')) | > avg_loss_gen: 2.1084264718310934 (+0.10229980441885766) | > avg_loss_kl: 2.055786218441708 (-0.15477984723910554) | > avg_loss_feat: 3.6909304538243255 (+0.1794751899343141) | > avg_loss_mel: 16.980932746135018 (-0.7137218529069855) | > avg_loss_duration: 1.5506662298256242 (-0.014608517499037355) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.386741987416443 (-0.6013354180564257) | > avg_grad_norm_1: tensor(5703.5234, device='cuda:0') (tensor(-1550.8027, device='cuda:0'))  > EPOCH: 1085/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:23:46)   --> TIME: 2025-02-25 19:24:01 -- STEP: 29/72 -- GLOBAL_STEP: 1078150 | > loss_disc: 2.9432952404022217 (2.6770472033270476) | > loss_disc_real_0: 0.06082102656364441 (0.11589891758972201) | > loss_disc_real_1: 0.21263127028942108 (0.2421647048202054) | > loss_disc_real_2: 0.27295568585395813 (0.238679648473345) | > loss_disc_real_3: 0.2196972817182541 (0.23275367864247026) | > loss_disc_real_4: 0.3194481432437897 (0.25816211155776325) | > loss_disc_real_5: 0.2844184637069702 (0.2528139527501732) | > loss_0: 2.9432952404022217 (2.6770472033270476) | > grad_norm_0: tensor(115.7233, device='cuda:0') (tensor(172.9980, device='cuda:0')) | > loss_gen: 1.7447941303253174 (2.055066022379645) | > loss_kl: 2.0221643447875977 (2.0153628752149384) | > loss_feat: 3.2258248329162598 (3.661716304976365) | > loss_mel: 17.724388122558594 (16.95346625097867) | > loss_duration: 1.5061291456222534 (1.5020265702543587) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.22330093383789 (26.18763798680799) | > grad_norm_1: tensor(3373.5464, device='cuda:0') (tensor(6343.7397, device='cuda:0')) | > current_lr_0: 0.00017463252155792972 | > current_lr_1: 0.00017463252155792972 | > step_time: 0.4358 (0.4351089822834936) | > loader_time: 0.0075 (0.006820580054973734) --> EVAL PERFORMANCE | > avg_loader_time: 0.008517319047954717 (-0.00022165876039317335) | > avg_step_time: 0.4506079680483106 (-0.010870627953972645) | > avg_loss_disc: 2.7013942691641795 (+0.07716629202936742) | > avg_loss_disc_real_0: 0.15117238970919394 (+0.034527382432994705) | > avg_loss_disc_real_1: 0.26221388053726136 (+0.03848431039024397) | > avg_loss_disc_real_2: 0.24303394451107777 (-0.005403248566976743) | > avg_loss_disc_real_3: 0.2383394721947925 (+0.0008185715742514144) | > avg_loss_disc_real_4: 0.25410652454470256 (+0.010575532703332613) | > avg_loss_disc_real_5: 0.2486780621636082 (+0.0010978031746098904) | > avg_loss_0: 2.7013942691641795 (+0.07716629202936742) | > avg_grad_norm_0: tensor(191.8512, device='cuda:0') (tensor(-0.1049, device='cuda:0')) | > avg_loss_gen: 2.0805479288101196 (-0.02787854302097381) | > avg_loss_kl: 2.0701724139737405 (+0.014386195532032353) | > avg_loss_feat: 3.5137210530294505 (-0.17720940079487502) | > avg_loss_mel: 16.903753697032652 (-0.07717904910236584) | > avg_loss_duration: 1.5466676530703694 (-0.003998576755254835) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.114862764385386 (-0.27187922303105694) | > avg_grad_norm_1: tensor(5978.6616, device='cuda:0') (+tensor(275.1382, device='cuda:0'))  > EPOCH: 1086/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:24:22)   --> TIME: 2025-02-25 19:24:27 -- STEP: 7/72 -- GLOBAL_STEP: 1078200 | > loss_disc: 2.538978099822998 (2.670130899974278) | > loss_disc_real_0: 0.06697461009025574 (0.10209220009190696) | > loss_disc_real_1: 0.17785371840000153 (0.17678420458521163) | > loss_disc_real_2: 0.21416310966014862 (0.22400876453944615) | > loss_disc_real_3: 0.198141410946846 (0.20949119329452515) | > loss_disc_real_4: 0.2229284793138504 (0.2356823491198676) | > loss_disc_real_5: 0.2198541760444641 (0.24142504589898245) | > loss_0: 2.538978099822998 (2.670130899974278) | > grad_norm_0: tensor(335.0593, device='cuda:0') (tensor(351.4320, device='cuda:0')) | > loss_gen: 1.7801343202590942 (1.7908315658569336) | > loss_kl: 2.1340482234954834 (1.99208698953901) | > loss_feat: 3.7174010276794434 (3.5572225706917897) | > loss_mel: 17.56183433532715 (17.069540977478027) | > loss_duration: 1.4862244129180908 (1.4986074481691634) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.679643630981445 (25.908289228166854) | > grad_norm_1: tensor(15950.5654, device='cuda:0') (tensor(9617.1992, device='cuda:0')) | > current_lr_0: 0.00017461069249273496 | > current_lr_1: 0.00017461069249273496 | > step_time: 0.4554 (0.42597910336085726) | > loader_time: 0.0059 (0.00622483662196568)  --> TIME: 2025-02-25 19:24:50 -- STEP: 57/72 -- GLOBAL_STEP: 1078250 | > loss_disc: 3.0576860904693604 (2.6153349792748166) | > loss_disc_real_0: 0.3526482582092285 (0.1297949389799645) | > loss_disc_real_1: 0.22811433672904968 (0.2202382822308624) | > loss_disc_real_2: 0.2657211422920227 (0.2327469072321005) | > loss_disc_real_3: 0.22654737532138824 (0.23015639578041278) | > loss_disc_real_4: 0.23209981620311737 (0.23971110948345117) | > loss_disc_real_5: 0.20219413936138153 (0.24306310634864003) | > loss_0: 3.0576860904693604 (2.6153349792748166) | > grad_norm_0: tensor(707.6381, device='cuda:0') (tensor(235.8167, device='cuda:0')) | > loss_gen: 1.693441390991211 (2.100656072298685) | > loss_kl: 2.0435967445373535 (2.0077741543451944) | > loss_feat: 3.18795108795166 (3.7061789705042254) | > loss_mel: 17.45621109008789 (16.627200779161953) | > loss_duration: 1.5846995115280151 (1.5306047075673153) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.965900421142578 (25.9724143513462) | > grad_norm_1: tensor(3228.4568, device='cuda:0') (tensor(5833.3071, device='cuda:0')) | > current_lr_0: 0.00017461069249273496 | > current_lr_1: 0.00017461069249273496 | > step_time: 0.4523 (0.44169927898206207) | > loader_time: 0.0096 (0.007905123526589912) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568488376241332 (+5.116932828661512e-05) | > avg_step_time: 0.45165987417731485 (+0.001051906129004243) | > avg_loss_disc: 2.63032475659545 (-0.07106951256872929) | > avg_loss_disc_real_0: 0.1353583203130205 (-0.01581406939617344) | > avg_loss_disc_real_1: 0.22261578075482813 (-0.03959809978243323) | > avg_loss_disc_real_2: 0.2317882087868704 (-0.011245735724207367) | > avg_loss_disc_real_3: 0.23086168459603484 (-0.007477787598757668) | > avg_loss_disc_real_4: 0.24129375038852155 (-0.012812774156181012) | > avg_loss_disc_real_5: 0.24168289421309888 (-0.006995167950509307) | > avg_loss_0: 2.63032475659545 (-0.07106951256872929) | > avg_grad_norm_0: tensor(226.9839, device='cuda:0') (+tensor(35.1328, device='cuda:0')) | > avg_loss_gen: 2.0649013804717793 (-0.015646548338340338) | > avg_loss_kl: 2.030163482880927 (-0.04000893109281334) | > avg_loss_feat: 3.6165538606509355 (+0.10283280762148506) | > avg_loss_mel: 16.639945822702327 (-0.2638078743303254) | > avg_loss_duration: 1.5434647529897556 (-0.0032029000806137464) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.895029067993164 (-0.2198336963922216) | > avg_grad_norm_1: tensor(6121.7969, device='cuda:0') (+tensor(143.1353, device='cuda:0'))  > EPOCH: 1087/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:24:59)   --> TIME: 2025-02-25 19:25:16 -- STEP: 35/72 -- GLOBAL_STEP: 1078300 | > loss_disc: 2.612170457839966 (2.601733248574393) | > loss_disc_real_0: 0.18581390380859375 (0.1651351009096418) | > loss_disc_real_1: 0.22951707243919373 (0.2231512712580817) | > loss_disc_real_2: 0.2762666344642639 (0.23156871242182595) | > loss_disc_real_3: 0.2296917736530304 (0.23001395165920258) | > loss_disc_real_4: 0.22835451364517212 (0.2374739646911621) | > loss_disc_real_5: 0.24105720221996307 (0.23885067616190228) | > loss_0: 2.612170457839966 (2.601733248574393) | > grad_norm_0: tensor(337.0935, device='cuda:0') (tensor(203.1104, device='cuda:0')) | > loss_gen: 2.3452084064483643 (2.1504253489630565) | > loss_kl: 2.1615490913391113 (1.969466757774353) | > loss_feat: 4.061637878417969 (3.812025349480765) | > loss_mel: 17.442554473876953 (16.851402991158622) | > loss_duration: 1.5551552772521973 (1.50265907900674) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.566104888916016 (26.28597946166992) | > grad_norm_1: tensor(8056.4062, device='cuda:0') (tensor(5726.8521, device='cuda:0')) | > current_lr_0: 0.00017458886615617336 | > current_lr_1: 0.00017458886615617336 | > step_time: 0.4319 (0.43490088326590404) | > loader_time: 0.0082 (0.007122421264648437) --> EVAL PERFORMANCE | > avg_loader_time: 0.008632800948451945 (+6.431257221061262e-05) | > avg_step_time: 0.4532956674065388 (+0.0016357932292239696) | > avg_loss_disc: 2.665137878606018 (+0.03481312201056763) | > avg_loss_disc_real_0: 0.17973949770692368 (+0.04438117739390318) | > avg_loss_disc_real_1: 0.22556654353376845 (+0.0029507627789403246) | > avg_loss_disc_real_2: 0.23140934319563314 (-0.0003788655912372596) | > avg_loss_disc_real_3: 0.23444819870129438 (+0.003586514105259536) | > avg_loss_disc_real_4: 0.2396128922700882 (-0.0016808581184333538) | > avg_loss_disc_real_5: 0.23960539908476278 (-0.0020774951283361043) | > avg_loss_0: 2.665137878606018 (+0.03481312201056763) | > avg_grad_norm_0: tensor(184.6644, device='cuda:0') (tensor(-42.3196, device='cuda:0')) | > avg_loss_gen: 2.069203484226279 (+0.0043021037544996865) | > avg_loss_kl: 2.0702300004556156 (+0.040066517574688376) | > avg_loss_feat: 3.618519920698354 (+0.0019660600474185053) | > avg_loss_mel: 16.867666996700663 (+0.22772117399833647) | > avg_loss_duration: 1.5435234862314147 (+5.873324165905913e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.169143945398464 (+0.2741148774052995) | > avg_grad_norm_1: tensor(5770.0767, device='cuda:0') (tensor(-351.7202, device='cuda:0'))  > EPOCH: 1088/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:25:36)   --> TIME: 2025-02-25 19:25:44 -- STEP: 13/72 -- GLOBAL_STEP: 1078350 | > loss_disc: 2.85009765625 (2.7029815086951623) | > loss_disc_real_0: 0.06391018629074097 (0.14080575337776768) | > loss_disc_real_1: 0.22120457887649536 (0.2342633639390652) | > loss_disc_real_2: 0.22504782676696777 (0.23283885075495794) | > loss_disc_real_3: 0.2350281924009323 (0.24347711411806253) | > loss_disc_real_4: 0.2700859308242798 (0.23486381654555982) | > loss_disc_real_5: 0.27227210998535156 (0.25133613783579606) | > loss_0: 2.85009765625 (2.7029815086951623) | > grad_norm_0: tensor(228.4986, device='cuda:0') (tensor(264.5663, device='cuda:0')) | > loss_gen: 1.6425609588623047 (2.0172428259482746) | > loss_kl: 2.2393553256988525 (2.543820381164551) | > loss_feat: 3.746685266494751 (3.774747665111835) | > loss_mel: 18.997875213623047 (21.870563213641827) | > loss_duration: 1.4747672080993652 (1.6426338232480562) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.10124397277832 (31.84900841346154) | > grad_norm_1: tensor(8225.4912, device='cuda:0') (tensor(6565.2627, device='cuda:0')) | > current_lr_0: 0.00017456704254790382 | > current_lr_1: 0.00017456704254790382 | > step_time: 0.4586 (0.42908756549541766) | > loader_time: 0.0058 (0.0067677131065955526)  --> TIME: 2025-02-25 19:26:07 -- STEP: 63/72 -- GLOBAL_STEP: 1078400 | > loss_disc: 2.6567564010620117 (2.661957494796268) | > loss_disc_real_0: 0.21338653564453125 (0.15673880574722138) | > loss_disc_real_1: 0.1969626247882843 (0.23301608623966338) | > loss_disc_real_2: 0.24282516539096832 (0.2344942513912443) | > loss_disc_real_3: 0.24756695330142975 (0.23472247426472012) | > loss_disc_real_4: 0.24266302585601807 (0.2384318094404917) | > loss_disc_real_5: 0.269821435213089 (0.24446593343265474) | > loss_0: 2.6567564010620117 (2.661957494796268) | > grad_norm_0: tensor(330.9842, device='cuda:0') (tensor(197.1505, device='cuda:0')) | > loss_gen: 2.100386142730713 (2.0505596683138885) | > loss_kl: 1.9792424440383911 (2.2232184372251003) | > loss_feat: 2.8603551387786865 (3.6117083458673385) | > loss_mel: 15.960389137268066 (18.20803993467302) | > loss_duration: 1.5525068044662476 (1.5746571904137021) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.452880859375 (27.668183553786506) | > grad_norm_1: tensor(990.4623, device='cuda:0') (tensor(5460.1650, device='cuda:0')) | > current_lr_0: 0.00017456704254790382 | > current_lr_1: 0.00017456704254790382 | > step_time: 0.4959 (0.44802723612104145) | > loader_time: 0.0116 (0.008257298242478143) --> EVAL PERFORMANCE | > avg_loader_time: 0.008626561769297426 (-6.239179154519242e-06) | > avg_step_time: 0.45379804557477926 (+0.0005023781682404471) | > avg_loss_disc: 2.6576109503356498 (-0.00752692827036805) | > avg_loss_disc_real_0: 0.1582230085008581 (-0.02151648920606558) | > avg_loss_disc_real_1: 0.232275805213082 (+0.006709261679313544) | > avg_loss_disc_real_2: 0.23342397851003727 (+0.0020146353144041285) | > avg_loss_disc_real_3: 0.23391927065144122 (-0.0005289280498531612) | > avg_loss_disc_real_4: 0.2377516765829543 (-0.0018612156871339047) | > avg_loss_disc_real_5: 0.24427782628737704 (+0.0046724272026142655) | > avg_loss_0: 2.6576109503356498 (-0.00752692827036805) | > avg_grad_norm_0: tensor(196.5929, device='cuda:0') (+tensor(11.9286, device='cuda:0')) | > avg_loss_gen: 2.0508640500861146 (-0.018339434140164368) | > avg_loss_kl: 2.2216005123836897 (+0.15137051192807416) | > avg_loss_feat: 3.592830785563294 (-0.025689135135059882) | > avg_loss_mel: 18.041330162908004 (+1.1736631662073407) | > avg_loss_duration: 1.5789874221237612 (+0.03546393589234653) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.48561294985489 (+1.316469004456426) | > avg_grad_norm_1: tensor(5377.8872, device='cuda:0') (tensor(-392.1895, device='cuda:0'))  > EPOCH: 1089/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:26:13)   --> TIME: 2025-02-25 19:26:33 -- STEP: 41/72 -- GLOBAL_STEP: 1078450 | > loss_disc: 2.583916425704956 (2.662422610492242) | > loss_disc_real_0: 0.21899151802062988 (0.14059747819130014) | > loss_disc_real_1: 0.21554376184940338 (0.22748627204720567) | > loss_disc_real_2: 0.3129219710826874 (0.22775537629679934) | > loss_disc_real_3: 0.21432289481163025 (0.2374522740521082) | > loss_disc_real_4: 0.23577427864074707 (0.2474650929613811) | > loss_disc_real_5: 0.24395112693309784 (0.2453075589203253) | > loss_0: 2.583916425704956 (2.662422610492242) | > grad_norm_0: tensor(294.3745, device='cuda:0') (tensor(180.5198, device='cuda:0')) | > loss_gen: 2.467104434967041 (2.0785302708788618) | > loss_kl: 2.270146131515503 (2.403073319574682) | > loss_feat: 4.045929908752441 (3.7042651874263113) | > loss_mel: 16.952869415283203 (18.20504428119194) | > loss_duration: 1.5816631317138672 (1.6633340265692733) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.317712783813477 (28.054246948986517) | > grad_norm_1: tensor(1718.3037, device='cuda:0') (tensor(5240.3379, device='cuda:0')) | > current_lr_0: 0.00017454522166758532 | > current_lr_1: 0.00017454522166758532 | > step_time: 0.48 (0.4370453590300025) | > loader_time: 0.0088 (0.007255187848719155) --> EVAL PERFORMANCE | > avg_loader_time: 0.008498782842931613 (-0.00012777892636581245) | > avg_step_time: 0.4495500712327554 (-0.004247974342023841) | > avg_loss_disc: 2.6818444628111076 (+0.024233512475457797) | > avg_loss_disc_real_0: 0.13539123015714363 (-0.02283177834371447) | > avg_loss_disc_real_1: 0.24045195734836686 (+0.008176152135284859) | > avg_loss_disc_real_2: 0.2349321089789901 (+0.0015081304689528452) | > avg_loss_disc_real_3: 0.23283163594527984 (-0.0010876347061613834) | > avg_loss_disc_real_4: 0.23955453659447146 (+0.0018028600115171656) | > avg_loss_disc_real_5: 0.24254097212368333 (-0.0017368541636937074) | > avg_loss_0: 2.6818444628111076 (+0.024233512475457797) | > avg_grad_norm_0: tensor(169.6425, device='cuda:0') (tensor(-26.9504, device='cuda:0')) | > avg_loss_gen: 2.069273124278431 (+0.01840907419231641) | > avg_loss_kl: 2.2721959386073367 (+0.05059542622364699) | > avg_loss_feat: 3.683069336582238 (+0.09023855101894362) | > avg_loss_mel: 17.619375994507696 (-0.4219541684003083) | > avg_loss_duration: 1.6337790992898 (+0.05479167716603883) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.277693412673305 (-0.20791953718158496) | > avg_grad_norm_1: tensor(5099.3311, device='cuda:0') (tensor(-278.5562, device='cuda:0'))  > EPOCH: 1090/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:26:49)   --> TIME: 2025-02-25 19:27:00 -- STEP: 19/72 -- GLOBAL_STEP: 1078500 | > loss_disc: 2.5954418182373047 (2.722360836832147) | > loss_disc_real_0: 0.1266365945339203 (0.17719155471575887) | > loss_disc_real_1: 0.2505560517311096 (0.23087819469602486) | > loss_disc_real_2: 0.22177276015281677 (0.23305063577074753) | > loss_disc_real_3: 0.2529400587081909 (0.2526503566064333) | > loss_disc_real_4: 0.25290992856025696 (0.23670879398521624) | > loss_disc_real_5: 0.2369430512189865 (0.2359259528549094) | > loss_0: 2.5954418182373047 (2.722360836832147) | > grad_norm_0: tensor(251.9099, device='cuda:0') (tensor(222.1034, device='cuda:0')) | > loss_gen: 2.0842092037200928 (2.006108095771388) | > loss_kl: 2.5355424880981445 (2.087279991099709) | > loss_feat: 3.7635159492492676 (3.554473462857698) | > loss_mel: 17.793336868286133 (17.306242290296055) | > loss_duration: 1.5755951404571533 (1.529403103025336) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.752201080322266 (26.48350675482499) | > grad_norm_1: tensor(11875.5576, device='cuda:0') (tensor(7298.8618, device='cuda:0')) | > current_lr_0: 0.00017452340351487686 | > current_lr_1: 0.00017452340351487686 | > step_time: 0.4217 (0.43011495941563654) | > loader_time: 0.0075 (0.0064964921850907175)  --> TIME: 2025-02-25 19:27:23 -- STEP: 69/72 -- GLOBAL_STEP: 1078550 | > loss_disc: 2.649099111557007 (2.6649367671081987) | > loss_disc_real_0: 0.2090577334165573 (0.1520239671935206) | > loss_disc_real_1: 0.22876060009002686 (0.21807753996572632) | > loss_disc_real_2: 0.22739604115486145 (0.23356011185957037) | > loss_disc_real_3: 0.2367323487997055 (0.2382958058429801) | > loss_disc_real_4: 0.2254256159067154 (0.24319998380066693) | > loss_disc_real_5: 0.24218979477882385 (0.2417964229117269) | > loss_0: 2.649099111557007 (2.6649367671081987) | > grad_norm_0: tensor(697.4229, device='cuda:0') (tensor(248.2509, device='cuda:0')) | > loss_gen: 2.01138973236084 (1.9932494059852932) | > loss_kl: 2.5101006031036377 (2.1177893548771953) | > loss_feat: 3.4485888481140137 (3.5798637659653374) | > loss_mel: 18.420398712158203 (17.2308658655139) | > loss_duration: 1.7254419326782227 (1.5641802376595095) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.115921020507812 (26.48594861790754) | > grad_norm_1: tensor(12300.5664, device='cuda:0') (tensor(8097.1450, device='cuda:0')) | > current_lr_0: 0.00017452340351487686 | > current_lr_1: 0.00017452340351487686 | > step_time: 0.5093 (0.44850295522938605) | > loader_time: 0.0127 (0.008408052333887073) --> EVAL PERFORMANCE | > avg_loader_time: 0.008480407822300013 (-1.837502063159989e-05) | > avg_step_time: 0.4500083251738212 (+0.0004582539410658004) | > avg_loss_disc: 2.664624187308298 (-0.017220275502809468) | > avg_loss_disc_real_0: 0.1514258751986732 (+0.01603464504152957) | > avg_loss_disc_real_1: 0.21608497685110065 (-0.02436698049726621) | > avg_loss_disc_real_2: 0.23312528326477802 (-0.0018068257142120903) | > avg_loss_disc_real_3: 0.23724094033241272 (+0.004409304387132884) | > avg_loss_disc_real_4: 0.24266729010662563 (+0.003112753512154176) | > avg_loss_disc_real_5: 0.24132287901052288 (-0.0012180931131604567) | > avg_loss_0: 2.664624187308298 (-0.017220275502809468) | > avg_grad_norm_0: tensor(249.1485, device='cuda:0') (+tensor(79.5060, device='cuda:0')) | > avg_loss_gen: 1.9904306153176536 (-0.07884250896077738) | > avg_loss_kl: 2.1453799966355445 (-0.12681594197179225) | > avg_loss_feat: 3.5871267688106485 (-0.09594256777158927) | > avg_loss_mel: 17.285175511534785 (-0.3342004829729106) | > avg_loss_duration: 1.568733169998921 (-0.06504592929087893) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.576846001853404 (-0.7008474108199003) | > avg_grad_norm_1: tensor(8160.9648, device='cuda:0') (+tensor(3061.6338, device='cuda:0'))  > EPOCH: 1091/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:27:26)   --> TIME: 2025-02-25 19:27:50 -- STEP: 47/72 -- GLOBAL_STEP: 1078600 | > loss_disc: 2.5976412296295166 (2.5746658000540212) | > loss_disc_real_0: 0.06855937838554382 (0.11769118833732098) | > loss_disc_real_1: 0.21974363923072815 (0.24923380765509098) | > loss_disc_real_2: 0.22450335323810577 (0.2255457268750414) | > loss_disc_real_3: 0.19219176471233368 (0.22234728012947327) | > loss_disc_real_4: 0.2289648950099945 (0.23442803450087282) | > loss_disc_real_5: 0.2453027069568634 (0.23608097909612857) | > loss_0: 2.5976412296295166 (2.5746658000540212) | > grad_norm_0: tensor(287.7463, device='cuda:0') (tensor(270.1013, device='cuda:0')) | > loss_gen: 1.7489268779754639 (2.1050091449250568) | > loss_kl: 2.042144775390625 (2.312860866810413) | > loss_feat: 3.50154972076416 (3.8697835739622723) | > loss_mel: 16.637571334838867 (18.051908046641245) | > loss_duration: 1.6031804084777832 (1.576403909541191) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.53337287902832 (27.915965628116687) | > grad_norm_1: tensor(3849.9067, device='cuda:0') (tensor(9486.2881, device='cuda:0')) | > current_lr_0: 0.0001745015880894375 | > current_lr_1: 0.0001745015880894375 | > step_time: 0.4458 (0.4414655807170462) | > loader_time: 0.0089 (0.0076946958582452) --> EVAL PERFORMANCE | > avg_loader_time: 0.008642653344382703 (+0.00016224552208268968) | > avg_step_time: 0.4518859621504663 (+0.0018776369766450585) | > avg_loss_disc: 2.5851326055929684 (-0.07949158171532966) | > avg_loss_disc_real_0: 0.11938511973745386 (-0.032040755461219345) | > avg_loss_disc_real_1: 0.2418585482617499 (+0.02577357141064926) | > avg_loss_disc_real_2: 0.22936096367701678 (-0.0037643195877612445) | > avg_loss_disc_real_3: 0.2262432377942851 (-0.010997702538127624) | > avg_loss_disc_real_4: 0.23736447767472604 (-0.0053028124318995895) | > avg_loss_disc_real_5: 0.23787283288760924 (-0.003450046122913636) | > avg_loss_0: 2.5851326055929684 (-0.07949158171532966) | > avg_grad_norm_0: tensor(265.0609, device='cuda:0') (+tensor(15.9123, device='cuda:0')) | > avg_loss_gen: 2.0894959896383156 (+0.09906537432066198) | > avg_loss_kl: 2.247845814261638 (+0.10246581762609352) | > avg_loss_feat: 3.768363509379642 (+0.18123674056899342) | > avg_loss_mel: 17.7839273533351 (+0.4987518418003134) | > avg_loss_duration: 1.5839203394634622 (+0.015187169464541128) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.473552972498073 (+0.8967069706446686) | > avg_grad_norm_1: tensor(10065.0039, device='cuda:0') (+tensor(1904.0391, device='cuda:0'))  > EPOCH: 1092/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:28:03)   --> TIME: 2025-02-25 19:28:16 -- STEP: 25/72 -- GLOBAL_STEP: 1078650 | > loss_disc: 2.6113975048065186 (2.5814540386199947) | > loss_disc_real_0: 0.12152747809886932 (0.11655172199010849) | > loss_disc_real_1: 0.19779327511787415 (0.22553638517856597) | > loss_disc_real_2: 0.2387506514787674 (0.23894814550876617) | > loss_disc_real_3: 0.2340070903301239 (0.2385653191804886) | > loss_disc_real_4: 0.23248280584812164 (0.24252149283885957) | > loss_disc_real_5: 0.24236415326595306 (0.2433651441335678) | > loss_0: 2.6113975048065186 (2.5814540386199947) | > grad_norm_0: tensor(149.5825, device='cuda:0') (tensor(320.3133, device='cuda:0')) | > loss_gen: 2.015744924545288 (2.11259774684906) | > loss_kl: 2.0405359268188477 (2.0211195755004883) | > loss_feat: 3.8327291011810303 (3.8516554164886476) | > loss_mel: 15.816093444824219 (16.94283000946045) | > loss_duration: 1.5023256540298462 (1.5104551458358764) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.207427978515625 (26.438658142089842) | > grad_norm_1: tensor(8951.0166, device='cuda:0') (tensor(12527.8809, device='cuda:0')) | > current_lr_0: 0.0001744797753909263 | > current_lr_1: 0.0001744797753909263 | > step_time: 0.4406 (0.4301807594299316) | > loader_time: 0.0074 (0.006746835708618164) --> EVAL PERFORMANCE | > avg_loader_time: 0.008587551788544987 (-5.510155583771624e-05) | > avg_step_time: 0.45170841418521507 (-0.00017754796525121241) | > avg_loss_disc: 2.579020298702616 (-0.006112306890352581) | > avg_loss_disc_real_0: 0.12041922606212992 (+0.0010341063246760568) | > avg_loss_disc_real_1: 0.22330116386144933 (-0.01855738440030058) | > avg_loss_disc_real_2: 0.23545730722622132 (+0.006096343549204547) | > avg_loss_disc_real_3: 0.23780083152609813 (+0.011557593731813032) | > avg_loss_disc_real_4: 0.2461396571196301 (+0.008775179444904052) | > avg_loss_disc_real_5: 0.2471066027040213 (+0.009233769816412052) | > avg_loss_0: 2.579020298702616 (-0.006112306890352581) | > avg_grad_norm_0: tensor(323.2744, device='cuda:0') (+tensor(58.2136, device='cuda:0')) | > avg_loss_gen: 2.112111816943531 (+0.022615827305215497) | > avg_loss_kl: 2.0260708332061768 (-0.22177498105546123) | > avg_loss_feat: 3.7385006824009857 (-0.029862826978656276) | > avg_loss_mel: 16.87068088961319 (-0.913246463721908) | > avg_loss_duration: 1.5493535055241112 (-0.034566833939351005) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.296717872082347 (-1.176835100415726) | > avg_grad_norm_1: tensor(11773.7656, device='cuda:0') (+tensor(1708.7617, device='cuda:0'))  > EPOCH: 1093/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:28:40)   --> TIME: 2025-02-25 19:28:43 -- STEP: 3/72 -- GLOBAL_STEP: 1078700 | > loss_disc: 2.5185766220092773 (2.574759562810262) | > loss_disc_real_0: 0.08904331922531128 (0.10528458158175151) | > loss_disc_real_1: 0.2024809718132019 (0.22267263134320578) | > loss_disc_real_2: 0.20740476250648499 (0.2193097323179245) | > loss_disc_real_3: 0.20730753242969513 (0.2269804726044337) | > loss_disc_real_4: 0.23774389922618866 (0.2414255291223526) | > loss_disc_real_5: 0.24728576838970184 (0.24565144379933676) | > loss_0: 2.5185766220092773 (2.574759562810262) | > grad_norm_0: tensor(331.7306, device='cuda:0') (tensor(344.7374, device='cuda:0')) | > loss_gen: 2.055546522140503 (2.0599542458852134) | > loss_kl: 1.8993260860443115 (1.9118995666503906) | > loss_feat: 4.069711685180664 (3.8359994888305664) | > loss_mel: 17.159828186035156 (16.87619463602702) | > loss_duration: 1.5121526718139648 (1.514615575472514) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.696563720703125 (26.198662439982098) | > grad_norm_1: tensor(13642.9668, device='cuda:0') (tensor(12290.4639, device='cuda:0')) | > current_lr_0: 0.00017445796541900243 | > current_lr_1: 0.00017445796541900243 | > step_time: 0.4203 (0.42370009422302246) | > loader_time: 0.0057 (0.007068157196044922)  --> TIME: 2025-02-25 19:29:06 -- STEP: 53/72 -- GLOBAL_STEP: 1078750 | > loss_disc: 2.595781087875366 (2.5859992729043073) | > loss_disc_real_0: 0.09362612664699554 (0.12416318508813966) | > loss_disc_real_1: 0.22059863805770874 (0.22170497842554776) | > loss_disc_real_2: 0.20976568758487701 (0.23257454723682044) | > loss_disc_real_3: 0.22858665883541107 (0.2320171923569913) | > loss_disc_real_4: 0.24699722230434418 (0.24183580391811874) | > loss_disc_real_5: 0.23982743918895721 (0.24444958005311354) | > loss_0: 2.595781087875366 (2.5859992729043073) | > grad_norm_0: tensor(301.2531, device='cuda:0') (tensor(298.3281, device='cuda:0')) | > loss_gen: 1.971928596496582 (2.112714306363519) | > loss_kl: 2.074474573135376 (1.9430247950104047) | > loss_feat: 3.9730567932128906 (3.7410466536036076) | > loss_mel: 16.68263816833496 (16.590725880748824) | > loss_duration: 1.5956830978393555 (1.5261496327958017) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.29778289794922 (25.91366152493459) | > grad_norm_1: tensor(7219.0615, device='cuda:0') (tensor(10767.0332, device='cuda:0')) | > current_lr_0: 0.00017445796541900243 | > current_lr_1: 0.00017445796541900243 | > step_time: 0.4386 (0.4381178145138723) | > loader_time: 0.009 (0.007810628639077241) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605366021814482 (+1.781423326949541e-05) | > avg_step_time: 0.44969873361184565 (-0.002009680573369421) | > avg_loss_disc: 2.5873938278413164 (+0.008373529138700597) | > avg_loss_disc_real_0: 0.12587549768283335 (+0.005456271620703432) | > avg_loss_disc_real_1: 0.22188519571028964 (-0.0014159681511596856) | > avg_loss_disc_real_2: 0.23327255437911396 (-0.0021847528471073685) | > avg_loss_disc_real_3: 0.23276869847740925 (-0.0050321330486888805) | > avg_loss_disc_real_4: 0.24218154981942244 (-0.003958107300207653) | > avg_loss_disc_real_5: 0.2461468518619806 (-0.0009597508420406964) | > avg_loss_0: 2.5873938278413164 (+0.008373529138700597) | > avg_grad_norm_0: tensor(309.3086, device='cuda:0') (tensor(-13.9658, device='cuda:0')) | > avg_loss_gen: 2.1148700411890586 (+0.0027582242455275185) | > avg_loss_kl: 1.9826509163413248 (-0.043419916864851915) | > avg_loss_feat: 3.7242150810402883 (-0.014285601360697342) | > avg_loss_mel: 16.640469725702857 (-0.23021116391033303) | > avg_loss_duration: 1.5423424159976797 (-0.00701108952643148) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.00454832802356 (-0.2921695440587868) | > avg_grad_norm_1: tensor(10974.5293, device='cuda:0') (tensor(-799.2363, device='cuda:0'))  > EPOCH: 1094/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:29:17)   --> TIME: 2025-02-25 19:29:32 -- STEP: 31/72 -- GLOBAL_STEP: 1078800 | > loss_disc: 2.5400686264038086 (2.5715956764836467) | > loss_disc_real_0: 0.06797024607658386 (0.11855976019174821) | > loss_disc_real_1: 0.21045735478401184 (0.2183426825269576) | > loss_disc_real_2: 0.23656533658504486 (0.2291606140713538) | > loss_disc_real_3: 0.20160505175590515 (0.23472615643855063) | > loss_disc_real_4: 0.2544806897640228 (0.2419899461730834) | > loss_disc_real_5: 0.2465670108795166 (0.24348258106939255) | > loss_0: 2.5400686264038086 (2.5715956764836467) | > grad_norm_0: tensor(425.0851, device='cuda:0') (tensor(290.8055, device='cuda:0')) | > loss_gen: 2.0108373165130615 (2.118232180995326) | > loss_kl: 2.0132997035980225 (1.9219378194501322) | > loss_feat: 4.17205810546875 (3.850854689075101) | > loss_mel: 17.070804595947266 (16.72653502802696) | > loss_duration: 1.5370765924453735 (1.4998654857758553) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.8040771484375 (26.11742530330535) | > grad_norm_1: tensor(16590.5547, device='cuda:0') (tensor(10874.7070, device='cuda:0')) | > current_lr_0: 0.00017443615817332506 | > current_lr_1: 0.00017443615817332506 | > step_time: 0.4382 (0.434388276069395) | > loader_time: 0.0091 (0.0070500835295646425) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630292516359142 (+2.49264945446602e-05) | > avg_step_time: 0.45194872332290864 (+0.002249989711062994) | > avg_loss_disc: 2.579536354038077 (-0.007857473803239312) | > avg_loss_disc_real_0: 0.12173023924861155 (-0.004145258434221796) | > avg_loss_disc_real_1: 0.22073130918220735 (-0.001153886528082293) | > avg_loss_disc_real_2: 0.2292415440922052 (-0.004031010286908748) | > avg_loss_disc_real_3: 0.23278622035409363 (+1.752187668438432e-05) | > avg_loss_disc_real_4: 0.24216874363556715 (-1.2806183855296194e-05) | > avg_loss_disc_real_5: 0.2416900589432515 (-0.004456792918729108) | > avg_loss_0: 2.579536354038077 (-0.007857473803239312) | > avg_grad_norm_0: tensor(300.1426, device='cuda:0') (tensor(-9.1660, device='cuda:0')) | > avg_loss_gen: 2.1185688502352003 (+0.0036988090461416157) | > avg_loss_kl: 1.978524901497532 (-0.004126014843792936) | > avg_loss_feat: 3.767821614171418 (+0.0436065331311295) | > avg_loss_mel: 16.63753471911793 (-0.00293500658492718) | > avg_loss_duration: 1.5400935176392676 (-0.0022488983584121325) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.042543706759602 (+0.037995378736042085) | > avg_grad_norm_1: tensor(10934.3242, device='cuda:0') (tensor(-40.2051, device='cuda:0'))  > EPOCH: 1095/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:29:53)   --> TIME: 2025-02-25 19:29:59 -- STEP: 9/72 -- GLOBAL_STEP: 1078850 | > loss_disc: 2.626488447189331 (2.582983308368259) | > loss_disc_real_0: 0.19518430531024933 (0.1273642877737681) | > loss_disc_real_1: 0.22438421845436096 (0.21969817247655657) | > loss_disc_real_2: 0.23245568573474884 (0.23473116755485535) | > loss_disc_real_3: 0.24206463992595673 (0.2315797077284919) | > loss_disc_real_4: 0.23201127350330353 (0.23745496571063995) | > loss_disc_real_5: 0.2406490445137024 (0.2370994488398234) | > loss_0: 2.626488447189331 (2.582983308368259) | > grad_norm_0: tensor(565.0610, device='cuda:0') (tensor(383.1625, device='cuda:0')) | > loss_gen: 2.0641984939575195 (2.118244833416409) | > loss_kl: 1.6980372667312622 (1.8310469918780856) | > loss_feat: 3.4772439002990723 (3.7685848342047796) | > loss_mel: 16.265111923217773 (16.47286605834961) | > loss_duration: 1.510791540145874 (1.5023157464133368) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.015384674072266 (25.693058225843643) | > grad_norm_1: tensor(13607.2412, device='cuda:0') (tensor(12698.7158, device='cuda:0')) | > current_lr_0: 0.0001744143536535534 | > current_lr_1: 0.0001744143536535534 | > step_time: 0.4274 (0.42376044061448837) | > loader_time: 0.0076 (0.006430466969807942)  --> TIME: 2025-02-25 19:30:22 -- STEP: 59/72 -- GLOBAL_STEP: 1078900 | > loss_disc: 2.7972543239593506 (2.5965231798462947) | > loss_disc_real_0: 0.1346377432346344 (0.12754697977738863) | > loss_disc_real_1: 0.2434040606021881 (0.2278477645526498) | > loss_disc_real_2: 0.23224203288555145 (0.23638912151425573) | > loss_disc_real_3: 0.24760772287845612 (0.23720907962928384) | > loss_disc_real_4: 0.26133477687835693 (0.24337045368501695) | > loss_disc_real_5: 0.24790695309638977 (0.24567903502512786) | > loss_0: 2.7972543239593506 (2.5965231798462947) | > grad_norm_0: tensor(498.6352, device='cuda:0') (tensor(313.7736, device='cuda:0')) | > loss_gen: 1.7709648609161377 (2.159391324398882) | > loss_kl: 1.9579250812530518 (1.9635002714092449) | > loss_feat: 3.115453004837036 (3.770085310531875) | > loss_mel: 15.984661102294922 (16.637348546820174) | > loss_duration: 1.5781269073486328 (1.5260893187280429) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.40713119506836 (26.0564145556951) | > grad_norm_1: tensor(5997.1025, device='cuda:0') (tensor(11173.9951, device='cuda:0')) | > current_lr_0: 0.0001744143536535534 | > current_lr_1: 0.0001744143536535534 | > step_time: 0.4646 (0.43997832476082493) | > loader_time: 0.0097 (0.007896718332322977) --> EVAL PERFORMANCE | > avg_loader_time: 0.008455776832473111 (-0.00017451568388603123) | > avg_step_time: 0.44850739962618114 (-0.003441323696727505) | > avg_loss_disc: 2.6286372668306592 (+0.04910091279258211) | > avg_loss_disc_real_0: 0.1301763940759947 (+0.00844615482738316) | > avg_loss_disc_real_1: 0.2344586366498974 (+0.013727327467690065) | > avg_loss_disc_real_2: 0.24358030219732876 (+0.014338758105123556) | > avg_loss_disc_real_3: 0.2359624442080377 (+0.00317622385394406) | > avg_loss_disc_real_4: 0.24154410248910876 (-0.0006246411464583823) | > avg_loss_disc_real_5: 0.24480644036346758 (+0.0031163814202160856) | > avg_loss_0: 2.6286372668306592 (+0.04910091279258211) | > avg_grad_norm_0: tensor(323.2041, device='cuda:0') (+tensor(23.0615, device='cuda:0')) | > avg_loss_gen: 2.146290874817002 (+0.02772202458180173) | > avg_loss_kl: 1.9932446127206507 (+0.01471971122311877) | > avg_loss_feat: 3.71051300411493 (-0.05730861005648791) | > avg_loss_mel: 16.66073897187139 (+0.02320425275345883) | > avg_loss_duration: 1.5379911140656808 (-0.0021024035735868196) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.048778453343353 (+0.00623474658375045) | > avg_grad_norm_1: tensor(11004.3271, device='cuda:0') (+tensor(70.0029, device='cuda:0'))  > EPOCH: 1096/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:30:30)   --> TIME: 2025-02-25 19:30:48 -- STEP: 37/72 -- GLOBAL_STEP: 1078950 | > loss_disc: 2.7421443462371826 (2.791853402111981) | > loss_disc_real_0: 0.08079910278320312 (0.19281807843897794) | > loss_disc_real_1: 0.19746337831020355 (0.25988580004588974) | > loss_disc_real_2: 0.3119491636753082 (0.2362512458820601) | > loss_disc_real_3: 0.2559426724910736 (0.23203136389319962) | > loss_disc_real_4: 0.2962653338909149 (0.23699164994665095) | > loss_disc_real_5: 0.24386714398860931 (0.23203150404466166) | > loss_0: 2.7421443462371826 (2.791853402111981) | > grad_norm_0: tensor(79.3067, device='cuda:0') (tensor(319.1815, device='cuda:0')) | > loss_gen: 1.930898904800415 (2.0751068430977897) | > loss_kl: 2.8887217044830322 (2.5602547155844206) | > loss_feat: 3.6267664432525635 (3.7206266957360343) | > loss_mel: 20.046422958374023 (18.811364070789235) | > loss_duration: 1.676171898841858 (1.6030067914241068) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.168981552124023 (28.77035919395653) | > grad_norm_1: tensor(2640.9714, device='cuda:0') (tensor(6046.8027, device='cuda:0')) | > current_lr_0: 0.0001743925518593467 | > current_lr_1: 0.0001743925518593467 | > step_time: 0.4576 (0.4322714870040481) | > loader_time: 0.0085 (0.00722262666032121) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556402904886597 (+0.00010062607241348616) | > avg_step_time: 0.4474884254831663 (-0.001018974143014828) | > avg_loss_disc: 2.725491157719787 (+0.09685389088912766) | > avg_loss_disc_real_0: 0.18441425395053876 (+0.05423785987454405) | > avg_loss_disc_real_1: 0.25135298807856055 (+0.016894351428663135) | > avg_loss_disc_real_2: 0.23859962296318 (-0.0049806792341487716) | > avg_loss_disc_real_3: 0.24003268967212085 (+0.004070245464083161) | > avg_loss_disc_real_4: 0.23934511693430618 (-0.0021989855548025794) | > avg_loss_disc_real_5: 0.2409133453604201 (-0.0038930950030474643) | > avg_loss_0: 2.725491157719787 (+0.09685389088912766) | > avg_grad_norm_0: tensor(270.9522, device='cuda:0') (tensor(-52.2519, device='cuda:0')) | > avg_loss_gen: 2.161906809873985 (+0.015615935056982888) | > avg_loss_kl: 2.4725344684762023 (+0.4792898557555516) | > avg_loss_feat: 3.7317443330522995 (+0.021231328937369565) | > avg_loss_mel: 18.51992772330701 (+1.859188751435621) | > avg_loss_duration: 1.616129672023612 (+0.07813855795793123) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.50224309572032 (+2.453464642376968) | > avg_grad_norm_1: tensor(5551.9224, device='cuda:0') (tensor(-5452.4048, device='cuda:0'))  > EPOCH: 1097/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:31:06)   --> TIME: 2025-02-25 19:31:15 -- STEP: 15/72 -- GLOBAL_STEP: 1079000 | > loss_disc: 2.3092517852783203 (2.5848130226135253) | > loss_disc_real_0: 0.06927213072776794 (0.12931304747859637) | > loss_disc_real_1: 0.1880207508802414 (0.22691238125165303) | > loss_disc_real_2: 0.22377699613571167 (0.23547837833563487) | > loss_disc_real_3: 0.1981160193681717 (0.23065060277779897) | > loss_disc_real_4: 0.24016158282756805 (0.24268302122751872) | > loss_disc_real_5: 0.2515147924423218 (0.24045323133468627) | > loss_0: 2.3092517852783203 (2.5848130226135253) | > grad_norm_0: tensor(130.8157, device='cuda:0') (tensor(228.8521, device='cuda:0')) | > loss_gen: 2.641489267349243 (2.264421645800273) | > loss_kl: 2.117985963821411 (2.0469098726908364) | > loss_feat: 4.826438903808594 (3.969876019159953) | > loss_mel: 18.783172607421875 (17.512298838297518) | > loss_duration: 1.5066015720367432 (1.5179229577382405) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.875686645507812 (27.31142896016439) | > grad_norm_1: tensor(9158.4824, device='cuda:0') (tensor(5091.1035, device='cuda:0')) | > current_lr_0: 0.00017437075279036428 | > current_lr_1: 0.00017437075279036428 | > step_time: 0.4205 (0.43047294616699217) | > loader_time: 0.0058 (0.00655058224995931)  --> TIME: 2025-02-25 19:31:38 -- STEP: 65/72 -- GLOBAL_STEP: 1079050 | > loss_disc: 2.469102382659912 (2.642541547921988) | > loss_disc_real_0: 0.08060014247894287 (0.140890456449527) | > loss_disc_real_1: 0.17638711631298065 (0.23008337101111045) | > loss_disc_real_2: 0.22608286142349243 (0.23541412766163167) | > loss_disc_real_3: 0.2297208309173584 (0.2446147240125216) | > loss_disc_real_4: 0.2525496780872345 (0.2484822507088001) | > loss_disc_real_5: 0.23567621409893036 (0.24705066268260661) | > loss_0: 2.469102382659912 (2.642541547921988) | > grad_norm_0: tensor(296.4176, device='cuda:0') (tensor(239.0268, device='cuda:0')) | > loss_gen: 2.270059585571289 (2.185190395208506) | > loss_kl: 2.4929966926574707 (2.4406471179081852) | > loss_feat: 4.212003707885742 (3.7552726488847) | > loss_mel: 19.684024810791016 (18.971413979163536) | > loss_duration: 1.6382956504821777 (1.5914771098356981) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.297380447387695 (28.94400124183068) | > grad_norm_1: tensor(15093.8643, device='cuda:0') (tensor(7244.9878, device='cuda:0')) | > current_lr_0: 0.00017437075279036428 | > current_lr_1: 0.00017437075279036428 | > step_time: 0.4897 (0.4451675268319937) | > loader_time: 0.0114 (0.008285265702467699) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555583550896443 (-8.193539901540231e-07) | > avg_step_time: 0.4519544823068968 (+0.004466056823730469) | > avg_loss_disc: 2.6343842190755926 (-0.09110693864419428) | > avg_loss_disc_real_0: 0.13697326403688376 (-0.047440989913655) | > avg_loss_disc_real_1: 0.22848662387737087 (-0.022866364201189676) | > avg_loss_disc_real_2: 0.23559657200961046 (-0.003003050953569536) | > avg_loss_disc_real_3: 0.24239725126347073 (+0.002364561591349873) | > avg_loss_disc_real_4: 0.24791230827989713 (+0.008567191345590947) | > avg_loss_disc_real_5: 0.24848795126021747 (+0.00757460589979736) | > avg_loss_0: 2.6343842190755926 (-0.09110693864419428) | > avg_grad_norm_0: tensor(234.8199, device='cuda:0') (tensor(-36.1323, device='cuda:0')) | > avg_loss_gen: 2.1801735884706748 (+0.018266778596689903) | > avg_loss_kl: 2.434453870209171 (-0.03808059826703136) | > avg_loss_feat: 3.7490928576026166 (+0.0173485245503171) | > avg_loss_mel: 18.917156783627792 (+0.3972290603207824) | > avg_loss_duration: 1.5948650937684825 (-0.021264578255129507) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.875742227258815 (+0.37349913153849457) | > avg_grad_norm_1: tensor(7438.7920, device='cuda:0') (+tensor(1886.8696, device='cuda:0'))  > EPOCH: 1098/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:31:43)   --> TIME: 2025-02-25 19:32:04 -- STEP: 43/72 -- GLOBAL_STEP: 1079100 | > loss_disc: 2.4455206394195557 (2.492394164551136) | > loss_disc_real_0: 0.07282251864671707 (0.08422533214785334) | > loss_disc_real_1: 0.21421436965465546 (0.21364417741465014) | > loss_disc_real_2: 0.22390224039554596 (0.23350070278311885) | > loss_disc_real_3: 0.22906480729579926 (0.23103445664394734) | > loss_disc_real_4: 0.2329568862915039 (0.23991832622261935) | > loss_disc_real_5: 0.2352919727563858 (0.24146013204441513) | > loss_0: 2.4455206394195557 (2.492394164551136) | > grad_norm_0: tensor(95.4007, device='cuda:0') (tensor(261.8298, device='cuda:0')) | > loss_gen: 2.343040704727173 (2.211077326952024) | > loss_kl: 2.1679255962371826 (2.0501684643501457) | > loss_feat: 4.411337375640869 (4.0109488631403725) | > loss_mel: 16.3591251373291 (17.483132473258085) | > loss_duration: 1.5515432357788086 (1.5438347140023874) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.83297348022461 (27.29916199972463) | > grad_norm_1: tensor(4256.8228, device='cuda:0') (tensor(9038.5000, device='cuda:0')) | > current_lr_0: 0.00017434895644626548 | > current_lr_1: 0.00017434895644626548 | > step_time: 0.4863 (0.43895658226900325) | > loader_time: 0.0095 (0.007449815439623456) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612589097358809 (+5.700554646236554e-05) | > avg_step_time: 0.4536490339628408 (+0.00169455165594401) | > avg_loss_disc: 2.4984954679515994 (-0.13588875112399323) | > avg_loss_disc_real_0: 0.0837632398790037 (-0.05321002415788005) | > avg_loss_disc_real_1: 0.2165813876289717 (-0.011905236248399176) | > avg_loss_disc_real_2: 0.23399077124998602 (-0.0016058007596244372) | > avg_loss_disc_real_3: 0.2303906219526076 (-0.012006629310863132) | > avg_loss_disc_real_4: 0.24094761152502517 (-0.006964696754871963) | > avg_loss_disc_real_5: 0.24520521243693125 (-0.0032827388232862242) | > avg_loss_0: 2.4984954679515994 (-0.13588875112399323) | > avg_grad_norm_0: tensor(269.9683, device='cuda:0') (+tensor(35.1484, device='cuda:0')) | > avg_loss_gen: 2.2111782020246475 (+0.03100461355397277) | > avg_loss_kl: 2.07286784514575 (-0.3615860250634211) | > avg_loss_feat: 3.988563295821069 (+0.23947043821845249) | > avg_loss_mel: 17.34315547137194 (-1.5740013122558523) | > avg_loss_duration: 1.5647572782677663 (-0.030107815500716173) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.180522166507345 (-1.6952200607514705) | > avg_grad_norm_1: tensor(9272.3301, device='cuda:0') (+tensor(1833.5381, device='cuda:0'))  > EPOCH: 1099/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:32:20)   --> TIME: 2025-02-25 19:32:31 -- STEP: 21/72 -- GLOBAL_STEP: 1079150 | > loss_disc: 2.5544469356536865 (2.486349117188227) | > loss_disc_real_0: 0.10315360128879547 (0.0850653394701935) | > loss_disc_real_1: 0.22136244177818298 (0.21362888174397604) | > loss_disc_real_2: 0.22666999697685242 (0.23638880820501418) | > loss_disc_real_3: 0.22951087355613708 (0.23318728591714585) | > loss_disc_real_4: 0.229757621884346 (0.24378013965629397) | > loss_disc_real_5: 0.23008404672145844 (0.2514080518767947) | > loss_0: 2.5544469356536865 (2.486349117188227) | > grad_norm_0: tensor(230.9302, device='cuda:0') (tensor(266.1092, device='cuda:0')) | > loss_gen: 2.0625267028808594 (2.279772786867051) | > loss_kl: 1.8166756629943848 (1.9244331121444702) | > loss_feat: 3.768742084503174 (4.144701946349371) | > loss_mel: 16.490264892578125 (16.98542004539853) | > loss_duration: 1.5392239093780518 (1.5096927313577562) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.67743492126465 (26.84402084350586) | > grad_norm_1: tensor(9651.5049, device='cuda:0') (tensor(8862.1133, device='cuda:0')) | > current_lr_0: 0.0001743271628267097 | > current_lr_1: 0.0001743271628267097 | > step_time: 0.4442 (0.43132719539460684) | > loader_time: 0.0061 (0.006876037234351749)  --> TIME: 2025-02-25 19:32:55 -- STEP: 71/72 -- GLOBAL_STEP: 1079200 | > loss_disc: 2.6383302211761475 (2.507223045322257) | > loss_disc_real_0: 0.042965225875377655 (0.08857072590732237) | > loss_disc_real_1: 0.22237437963485718 (0.21902077634569625) | > loss_disc_real_2: 0.22964514791965485 (0.23260597150090714) | > loss_disc_real_3: 0.2364397794008255 (0.23155113907767014) | > loss_disc_real_4: 0.23246686160564423 (0.2411399779605194) | > loss_disc_real_5: 0.23396752774715424 (0.24963951719478822) | > loss_0: 2.6383302211761475 (2.507223045322257) | > grad_norm_0: tensor(309.4873, device='cuda:0') (tensor(271.1534, device='cuda:0')) | > loss_gen: 2.1017723083496094 (2.2511523290419237) | > loss_kl: 2.0344040393829346 (2.001263144990088) | > loss_feat: 3.601252317428589 (4.05330600537045) | > loss_mel: 16.543521881103516 (16.819804755734726) | > loss_duration: 1.562464714050293 (1.5499745372315528) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.843414306640625 (26.675500869750977) | > grad_norm_1: tensor(16357.9971, device='cuda:0') (tensor(8852.2676, device='cuda:0')) | > current_lr_0: 0.0001743271628267097 | > current_lr_1: 0.0001743271628267097 | > step_time: 0.473 (0.4515164872290383) | > loader_time: 0.0082 (0.008636313424983492) --> EVAL PERFORMANCE | > avg_loader_time: 0.008636313424983492 (+2.37243276246836e-05) | > avg_step_time: 0.4515164872290383 (-0.002132546733802476) | > avg_loss_disc: 2.507223045322257 (+0.008727577370657436) | > avg_loss_disc_real_0: 0.08857072590732237 (+0.0048074860283186666) | > avg_loss_disc_real_1: 0.21902077634569625 (+0.0024393887167245554) | > avg_loss_disc_real_2: 0.23260597150090714 (-0.0013847997490788744) | > avg_loss_disc_real_3: 0.23155113907767014 (+0.0011605171250625435) | > avg_loss_disc_real_4: 0.2411399779605194 (+0.0001923664354942234) | > avg_loss_disc_real_5: 0.24963951719478822 (+0.004434304757856972) | > avg_loss_0: 2.507223045322257 (+0.008727577370657436) | > avg_grad_norm_0: tensor(271.1534, device='cuda:0') (+tensor(1.1851, device='cuda:0')) | > avg_loss_gen: 2.2511523290419237 (+0.039974127017276206) | > avg_loss_kl: 2.001263144990088 (-0.07160470015566167) | > avg_loss_feat: 4.05330600537045 (+0.06474270954938088) | > avg_loss_mel: 16.819804755734726 (-0.5233507156372141) | > avg_loss_duration: 1.5499745372315528 (-0.014782741036213576) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.675500869750977 (-0.5050212967563681) | > avg_grad_norm_1: tensor(8852.2676, device='cuda:0') (tensor(-420.0625, device='cuda:0'))  > EPOCH: 1100/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:32:56)   --> TIME: 2025-02-25 19:33:21 -- STEP: 49/72 -- GLOBAL_STEP: 1079250 | > loss_disc: 2.5972163677215576 (2.526523341937941) | > loss_disc_real_0: 0.09260198473930359 (0.096669182333411) | > loss_disc_real_1: 0.1842784583568573 (0.22174276441943888) | > loss_disc_real_2: 0.27265527844429016 (0.23226114681788854) | > loss_disc_real_3: 0.2126026451587677 (0.23198279403910346) | > loss_disc_real_4: 0.24491386115550995 (0.2416218424938163) | > loss_disc_real_5: 0.25422289967536926 (0.24730022312427052) | > loss_0: 2.5972163677215576 (2.526523341937941) | > grad_norm_0: tensor(246.9865, device='cuda:0') (tensor(285.4930, device='cuda:0')) | > loss_gen: 2.0992794036865234 (2.246092791459999) | > loss_kl: 1.9559708833694458 (1.9308640567623838) | > loss_feat: 3.9293057918548584 (4.077934148360272) | > loss_mel: 16.131546020507812 (16.729737729442363) | > loss_duration: 1.6105687618255615 (1.5265873743563279) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.72667121887207 (26.51121606632155) | > grad_norm_1: tensor(6493.8354, device='cuda:0') (tensor(8404.4053, device='cuda:0')) | > current_lr_0: 0.00017430537193135635 | > current_lr_1: 0.00017430537193135635 | > step_time: 0.4349 (0.4468018716695357) | > loader_time: 0.0097 (0.007614568788178113) --> EVAL PERFORMANCE | > avg_loader_time: 0.008583985583882937 (-5.2327841100555106e-05) | > avg_step_time: 0.4550298704227931 (+0.003513383193754771) | > avg_loss_disc: 2.5215328243416804 (+0.01430977901942354) | > avg_loss_disc_real_0: 0.09988344277084714 (+0.011312716863524766) | > avg_loss_disc_real_1: 0.22402308230668727 (+0.005002305960991021) | > avg_loss_disc_real_2: 0.2310410435770599 (-0.0015649279238472424) | > avg_loss_disc_real_3: 0.2302715971436299 (-0.0012795419340402292) | > avg_loss_disc_real_4: 0.24125333214309855 (+0.00011335418257915619) | > avg_loss_disc_real_5: 0.247313347290939 (-0.0023261699038492067) | > avg_loss_0: 2.5215328243416804 (+0.01430977901942354) | > avg_grad_norm_0: tensor(290.5678, device='cuda:0') (+tensor(19.4145, device='cuda:0')) | > avg_loss_gen: 2.2524146446039977 (+0.0012623155620739013) | > avg_loss_kl: 1.9686822387534129 (-0.03258090623667531) | > avg_loss_feat: 4.020186038084431 (-0.033119967286018515) | > avg_loss_mel: 16.708773989072988 (-0.11103076666173806) | > avg_loss_duration: 1.5453283266282418 (-0.004646210603310941) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.495385237143072 (-0.18011563260790453) | > avg_grad_norm_1: tensor(8317.7334, device='cuda:0') (tensor(-534.5342, device='cuda:0'))  > EPOCH: 1101/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:33:33)   --> TIME: 2025-02-25 19:33:47 -- STEP: 27/72 -- GLOBAL_STEP: 1079300 | > loss_disc: 2.4582622051239014 (2.511065792154383) | > loss_disc_real_0: 0.0797547772526741 (0.0903346443341838) | > loss_disc_real_1: 0.23464058339595795 (0.2174085764973252) | > loss_disc_real_2: 0.2207256257534027 (0.2264522694879108) | > loss_disc_real_3: 0.2579600214958191 (0.23128598873262052) | > loss_disc_real_4: 0.24419687688350677 (0.24175599217414856) | > loss_disc_real_5: 0.24061906337738037 (0.24248149825467002) | > loss_0: 2.4582622051239014 (2.511065792154383) | > grad_norm_0: tensor(180.2973, device='cuda:0') (tensor(265.3517, device='cuda:0')) | > loss_gen: 2.2729172706604004 (2.252899421585931) | > loss_kl: 1.9316768646240234 (1.9004584550857544) | > loss_feat: 4.220310211181641 (4.114294802701032) | > loss_mel: 17.194368362426758 (16.715040807370787) | > loss_duration: 1.5151937007904053 (1.5004587835735745) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.13446617126465 (26.483152389526367) | > grad_norm_1: tensor(8291.6602, device='cuda:0') (tensor(8192.0723, device='cuda:0')) | > current_lr_0: 0.00017428358375986492 | > current_lr_1: 0.00017428358375986492 | > step_time: 0.4317 (0.43055563502841526) | > loader_time: 0.008 (0.00678262887177644) --> EVAL PERFORMANCE | > avg_loader_time: 0.008602152408008844 (+1.8166824125906422e-05) | > avg_step_time: 0.4474459836180781 (-0.007583886804715001) | > avg_loss_disc: 2.512601778540813 (-0.008931045800867565) | > avg_loss_disc_real_0: 0.09558335177495446 (-0.00430009099589268) | > avg_loss_disc_real_1: 0.22113756016946176 (-0.00288552213722551) | > avg_loss_disc_real_2: 0.22760930510473923 (-0.0034317384723206723) | > avg_loss_disc_real_3: 0.23142376835917083 (+0.0011521712155409258) | > avg_loss_disc_real_4: 0.24168563140949734 (+0.000432299266398789) | > avg_loss_disc_real_5: 0.24324578571487482 (-0.004067561576064194) | > avg_loss_0: 2.512601778540813 (-0.008931045800867565) | > avg_grad_norm_0: tensor(259.5791, device='cuda:0') (tensor(-30.9887, device='cuda:0')) | > avg_loss_gen: 2.261583296345992 (+0.009168651741994438) | > avg_loss_kl: 1.9678858535390504 (-0.0007963852143624273) | > avg_loss_feat: 4.071163775215686 (+0.05097773713125431) | > avg_loss_mel: 16.72532974834173 (+0.016555759268740644) | > avg_loss_duration: 1.5421774202669172 (-0.0031509063613246457) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.56814013736349 (+0.07275490022041708) | > avg_grad_norm_1: tensor(8143.7817, device='cuda:0') (tensor(-173.9517, device='cuda:0'))  > EPOCH: 1102/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:34:10)   --> TIME: 2025-02-25 19:34:14 -- STEP: 5/72 -- GLOBAL_STEP: 1079350 | > loss_disc: 2.459672451019287 (2.4802515506744385) | > loss_disc_real_0: 0.11068527400493622 (0.11427095085382462) | > loss_disc_real_1: 0.25527599453926086 (0.2320967882871628) | > loss_disc_real_2: 0.2297986000776291 (0.2185857504606247) | > loss_disc_real_3: 0.2444271296262741 (0.2492280572652817) | > loss_disc_real_4: 0.25803104043006897 (0.24883961379528047) | > loss_disc_real_5: 0.25628501176834106 (0.24981179535388948) | > loss_0: 2.459672451019287 (2.4802515506744385) | > grad_norm_0: tensor(197.6821, device='cuda:0') (tensor(290.6404, device='cuda:0')) | > loss_gen: 2.4891984462738037 (2.4793222904205323) | > loss_kl: 1.8900283575057983 (1.8400209665298461) | > loss_feat: 4.3005828857421875 (4.339591312408447) | > loss_mel: 16.896957397460938 (17.107655334472657) | > loss_duration: 1.4677772521972656 (1.5017554998397826) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.044544219970703 (27.26834487915039) | > grad_norm_1: tensor(9843.4863, device='cuda:0') (tensor(8341.0068, device='cuda:0')) | > current_lr_0: 0.00017426179831189493 | > current_lr_1: 0.00017426179831189493 | > step_time: 0.4211 (0.42310380935668945) | > loader_time: 0.0061 (0.006562042236328125)  --> TIME: 2025-02-25 19:34:36 -- STEP: 55/72 -- GLOBAL_STEP: 1079400 | > loss_disc: 2.6984503269195557 (2.5392335588281805) | > loss_disc_real_0: 0.08648152649402618 (0.10719886469570074) | > loss_disc_real_1: 0.2527616024017334 (0.22733890197493814) | > loss_disc_real_2: 0.29213982820510864 (0.23039194508032365) | > loss_disc_real_3: 0.26850560307502747 (0.23503753705458208) | > loss_disc_real_4: 0.28742700815200806 (0.24158028743483803) | > loss_disc_real_5: 0.2522953152656555 (0.24479523544961757) | > loss_0: 2.6984503269195557 (2.5392335588281805) | > grad_norm_0: tensor(449.7718, device='cuda:0') (tensor(284.9299, device='cuda:0')) | > loss_gen: 2.1618363857269287 (2.275172294269909) | > loss_kl: 1.9324562549591064 (1.9781246597116644) | > loss_feat: 3.598344087600708 (4.060513279654762) | > loss_mel: 16.868331909179688 (16.58872829784047) | > loss_duration: 1.5947985649108887 (1.5248394034125587) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.1557674407959 (26.4273775620894) | > grad_norm_1: tensor(7629.8970, device='cuda:0') (tensor(8377.9648, device='cuda:0')) | > current_lr_0: 0.00017426179831189493 | > current_lr_1: 0.00017426179831189493 | > step_time: 0.4562 (0.4368719837882302) | > loader_time: 0.0093 (0.007835795662619851) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570593847355373 (-3.1558560653470805e-05) | > avg_step_time: 0.4485275543911356 (+0.0010815707730575386) | > avg_loss_disc: 2.5322514957105606 (+0.019649717169747838) | > avg_loss_disc_real_0: 0.10483274336012316 (+0.0092493915851687) | > avg_loss_disc_real_1: 0.22536547733864315 (+0.004227917169181389) | > avg_loss_disc_real_2: 0.22952077695181672 (+0.0019114718470774894) | > avg_loss_disc_real_3: 0.23389493495645658 (+0.0024711665972857455) | > avg_loss_disc_real_4: 0.24023783122989492 (-0.0014478001796024154) | > avg_loss_disc_real_5: 0.24328943499377076 (+4.3649278895940835e-05) | > avg_loss_0: 2.5322514957105606 (+0.019649717169747838) | > avg_grad_norm_0: tensor(271.3394, device='cuda:0') (+tensor(11.7603, device='cuda:0')) | > avg_loss_gen: 2.275982893688578 (+0.014399597342586112) | > avg_loss_kl: 2.0063510444802297 (+0.038465190941179284) | > avg_loss_feat: 4.069634890892136 (-0.0015288843235499172) | > avg_loss_mel: 16.61379403127751 (-0.11153571706421772) | > avg_loss_duration: 1.5398655894776465 (-0.0023118307892706635) | > avg_amp_scaler: 8.112676056338028 (+0.11267605633802802) | > avg_loss_1: 26.505628155990387 (-0.06251198137310254) | > avg_grad_norm_1: tensor(8161.8203, device='cuda:0') (+tensor(18.0386, device='cuda:0'))  > EPOCH: 1103/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:34:46)   --> TIME: 2025-02-25 19:35:03 -- STEP: 33/72 -- GLOBAL_STEP: 1079450 | > loss_disc: 2.8904595375061035 (2.6874255483800717) | > loss_disc_real_0: 0.26321786642074585 (0.18166992655306152) | > loss_disc_real_1: 0.5145815014839172 (0.23777437210083008) | > loss_disc_real_2: 0.19790251553058624 (0.2338191925576239) | > loss_disc_real_3: 0.3173670470714569 (0.24425614467172913) | > loss_disc_real_4: 0.18095891177654266 (0.23953540758653122) | > loss_disc_real_5: 0.21649008989334106 (0.24469326481674658) | > loss_0: 2.8904595375061035 (2.6874255483800717) | > grad_norm_0: tensor(182.3211, device='cuda:0') (tensor(300.3300, device='cuda:0')) | > loss_gen: 2.0346243381500244 (2.152927366170017) | > loss_kl: 2.3580808639526367 (2.115209355498805) | > loss_feat: 3.410628318786621 (3.7911605834960938) | > loss_mel: 17.121763229370117 (17.306196877450652) | > loss_duration: 1.5520365238189697 (1.5376718839009602) | > amp_scaler: 8.0 (10.424242424242424) | > loss_1: 26.47713279724121 (26.903165875059187) | > grad_norm_1: tensor(12088.3555, device='cuda:0') (tensor(6811.5459, device='cuda:0')) | > current_lr_0: 0.00017424001558710595 | > current_lr_1: 0.00017424001558710595 | > step_time: 0.4395 (0.44933785091746936) | > loader_time: 0.0082 (0.00710164416920055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008576977420860615 (+6.383573505241991e-06) | > avg_step_time: 0.4609551060367638 (+0.012427551645628199) | > avg_loss_disc: 2.6733866510256914 (+0.14113515531513077) | > avg_loss_disc_real_0: 0.165227147002875 (+0.06039440364275184) | > avg_loss_disc_real_1: 0.23575943099780822 (+0.010393953659165067) | > avg_loss_disc_real_2: 0.23620282365402706 (+0.0066820467022103425) | > avg_loss_disc_real_3: 0.23293684506920023 (-0.000958089887256347) | > avg_loss_disc_real_4: 0.23910426589804637 (-0.0011335653318485517) | > avg_loss_disc_real_5: 0.24231638832831046 (-0.0009730466654603032) | > avg_loss_0: 2.6733866510256914 (+0.14113515531513077) | > avg_grad_norm_0: tensor(290.0828, device='cuda:0') (+tensor(18.7435, device='cuda:0')) | > avg_loss_gen: 2.159696941644372 (-0.11628595204420611) | > avg_loss_kl: 2.2773296866618398 (+0.27097864218161005) | > avg_loss_feat: 3.834866013325436 (-0.23476887756669962) | > avg_loss_mel: 17.706984976647604 (+1.0931909453700932) | > avg_loss_duration: 1.5745140549162744 (+0.03464846543862787) | > avg_amp_scaler: 9.126760563380282 (+1.014084507042254) | > avg_loss_1: 27.553391470036036 (+1.0477633140456497) | > avg_grad_norm_1: tensor(6079.2700, device='cuda:0') (tensor(-2082.5503, device='cuda:0'))  > EPOCH: 1104/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:35:23)   --> TIME: 2025-02-25 19:35:30 -- STEP: 11/72 -- GLOBAL_STEP: 1079500 | > loss_disc: 2.4695987701416016 (2.5620426698164507) | > loss_disc_real_0: 0.040949612855911255 (0.11741127988154237) | > loss_disc_real_1: 0.24824360013008118 (0.19415720349008386) | > loss_disc_real_2: 0.24821358919143677 (0.26125301962549036) | > loss_disc_real_3: 0.17617905139923096 (0.23806780034845526) | > loss_disc_real_4: 0.24727830290794373 (0.23070581934668802) | > loss_disc_real_5: 0.26816326379776 (0.2349561317400499) | > loss_0: 2.4695987701416016 (2.5620426698164507) | > grad_norm_0: tensor(100.2715, device='cuda:0') (tensor(276.5876, device='cuda:0')) | > loss_gen: 2.2278974056243896 (2.2933151721954346) | > loss_kl: 2.118746757507324 (2.0615998181429775) | > loss_feat: 4.317338466644287 (4.198375311764804) | > loss_mel: 17.949142456054688 (18.046005249023438) | > loss_duration: 1.5082361698150635 (1.520079406824979) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.121360778808594 (28.119374882091176) | > grad_norm_1: tensor(5008.4717, device='cuda:0') (tensor(6431.0166, device='cuda:0')) | > current_lr_0: 0.00017421823558515755 | > current_lr_1: 0.00017421823558515755 | > step_time: 0.4165 (0.4283865365115079) | > loader_time: 0.0056 (0.006418856707486239)  --> TIME: 2025-02-25 19:35:53 -- STEP: 61/72 -- GLOBAL_STEP: 1079550 | > loss_disc: 2.484581470489502 (2.5386978016525017) | > loss_disc_real_0: 0.1549031138420105 (0.08561271401580238) | > loss_disc_real_1: 0.2462398260831833 (0.2214827496008795) | > loss_disc_real_2: 0.22791478037834167 (0.23845280169463548) | > loss_disc_real_3: 0.22528181970119476 (0.23715520295940462) | > loss_disc_real_4: 0.22885075211524963 (0.23954030749250632) | > loss_disc_real_5: 0.2292114943265915 (0.24316683509310738) | > loss_0: 2.484581470489502 (2.5386978016525017) | > grad_norm_0: tensor(422.4610, device='cuda:0') (tensor(215.0283, device='cuda:0')) | > loss_gen: 2.290473699569702 (2.2455869365911014) | > loss_kl: 2.11435604095459 (2.021750579114819) | > loss_feat: 3.688729763031006 (4.0573924682179445) | > loss_mel: 17.302391052246094 (17.243471520846008) | > loss_duration: 1.5858957767486572 (1.5431016703120997) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.98184585571289 (27.11130307932369) | > grad_norm_1: tensor(6535.1465, device='cuda:0') (tensor(6139.4087, device='cuda:0')) | > current_lr_0: 0.00017421823558515755 | > current_lr_1: 0.00017421823558515755 | > step_time: 0.4524 (0.4446861548501937) | > loader_time: 0.0111 (0.008071692263493772) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853827973486672 (-3.8697685993894365e-05) | > avg_step_time: 0.4532606769615496 (-0.007694429075214226) | > avg_loss_disc: 2.543224351506838 (-0.1301622995188536) | > avg_loss_disc_real_0: 0.08942630076387398 (-0.07580084623900102) | > avg_loss_disc_real_1: 0.22664961197846373 (-0.00910981901934449) | > avg_loss_disc_real_2: 0.2394590006327965 (+0.003256176978769426) | > avg_loss_disc_real_3: 0.23792401270967134 (+0.004987167640471107) | > avg_loss_disc_real_4: 0.2403659528829682 (+0.0012616869849218226) | > avg_loss_disc_real_5: 0.24364455757846296 (+0.0013281692501525) | > avg_loss_0: 2.543224351506838 (-0.1301622995188536) | > avg_grad_norm_0: tensor(216.3268, device='cuda:0') (tensor(-73.7561, device='cuda:0')) | > avg_loss_gen: 2.2569765003634172 (+0.09727955871904514) | > avg_loss_kl: 2.0312015120412252 (-0.24612817462061454) | > avg_loss_feat: 4.047150994690371 (+0.212284981364935) | > avg_loss_mel: 17.220622425347987 (-0.48636255129961725) | > avg_loss_duration: 1.5516723545504287 (-0.022841700365845652) | > avg_amp_scaler: 8.0 (-1.126760563380282) | > avg_loss_1: 27.107623677858165 (-0.44576779217787177) | > avg_grad_norm_1: tensor(5979.5801, device='cuda:0') (tensor(-99.6899, device='cuda:0'))  > EPOCH: 1105/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:36:00)   --> TIME: 2025-02-25 19:36:19 -- STEP: 39/72 -- GLOBAL_STEP: 1079600 | > loss_disc: 2.8774778842926025 (2.62680861277458) | > loss_disc_real_0: 0.0672527551651001 (0.16120992419429314) | > loss_disc_real_1: 0.19343359768390656 (0.21944681383096254) | > loss_disc_real_2: 0.24237091839313507 (0.2281009826140526) | > loss_disc_real_3: 0.17158280313014984 (0.22870712937452856) | > loss_disc_real_4: 0.2256959229707718 (0.23994120191305113) | > loss_disc_real_5: 0.2384209930896759 (0.24059065144795638) | > loss_0: 2.8774778842926025 (2.62680861277458) | > grad_norm_0: tensor(172.7859, device='cuda:0') (tensor(290.4534, device='cuda:0')) | > loss_gen: 1.6204729080200195 (2.2302760069186873) | > loss_kl: 2.1526858806610107 (1.937489656301645) | > loss_feat: 3.5854196548461914 (3.9432168068029942) | > loss_mel: 18.474863052368164 (17.11724741030962) | > loss_duration: 1.585129737854004 (1.5187010979041078) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.41857147216797 (26.746930733705177) | > grad_norm_1: tensor(8909.0312, device='cuda:0') (tensor(6022.1577, device='cuda:0')) | > current_lr_0: 0.0001741964583057094 | > current_lr_1: 0.0001741964583057094 | > step_time: 0.4684 (0.431385963391035) | > loader_time: 0.0087 (0.0070435389494284606) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847122031198421 (-6.705942288250978e-05) | > avg_step_time: 0.44756053199230783 (-0.0057001449692417605) | > avg_loss_disc: 2.633255182857245 (+0.09003083135040724) | > avg_loss_disc_real_0: 0.1668255664408207 (+0.07739926567694672) | > avg_loss_disc_real_1: 0.22474365461040552 (-0.0019059573680582087) | > avg_loss_disc_real_2: 0.22218942117523138 (-0.017269579457565104) | > avg_loss_disc_real_3: 0.22854498826282124 (-0.009379024446850098) | > avg_loss_disc_real_4: 0.23725351040631953 (-0.003112442476648658) | > avg_loss_disc_real_5: 0.2413481591453015 (-0.002296398433161456) | > avg_loss_0: 2.633255182857245 (+0.09003083135040724) | > avg_grad_norm_0: tensor(281.6935, device='cuda:0') (+tensor(65.3667, device='cuda:0')) | > avg_loss_gen: 2.194300903400905 (-0.06267559696251235) | > avg_loss_kl: 2.027803745068295 (-0.0033977669729301496) | > avg_loss_feat: 3.856933600466016 (-0.19021739422435502) | > avg_loss_mel: 17.156334863582128 (-0.06428756176585892) | > avg_loss_duration: 1.554501765210864 (+0.002829410660435183) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.789874654420665 (-0.3177490234375) | > avg_grad_norm_1: tensor(6936.4873, device='cuda:0') (+tensor(956.9072, device='cuda:0'))  > EPOCH: 1106/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:36:36)   --> TIME: 2025-02-25 19:36:46 -- STEP: 17/72 -- GLOBAL_STEP: 1079650 | > loss_disc: 2.5782713890075684 (2.648155268500833) | > loss_disc_real_0: 0.13411614298820496 (0.08128804516266375) | > loss_disc_real_1: 0.25508955121040344 (0.2218034723225762) | > loss_disc_real_2: 0.22383145987987518 (0.22657110498232) | > loss_disc_real_3: 0.21441487967967987 (0.2553597162751591) | > loss_disc_real_4: 0.26317116618156433 (0.27840882627403035) | > loss_disc_real_5: 0.24977348744869232 (0.2573492044911665) | > loss_0: 2.5782713890075684 (2.648155268500833) | > grad_norm_0: tensor(46.8553, device='cuda:0') (tensor(219.3413, device='cuda:0')) | > loss_gen: 2.2578012943267822 (2.14720625035903) | > loss_kl: 2.012878179550171 (1.9580045938491821) | > loss_feat: 3.7834312915802 (4.015332306132597) | > loss_mel: 17.483388900756836 (16.897304254419662) | > loss_duration: 1.5268316268920898 (1.500930582775789) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.0643310546875 (26.518777510699103) | > grad_norm_1: tensor(10541.9531, device='cuda:0') (tensor(6771.4204, device='cuda:0')) | > current_lr_0: 0.0001741746837484212 | > current_lr_1: 0.0001741746837484212 | > step_time: 0.4219 (0.425472708309398) | > loader_time: 0.0058 (0.006454341551836799)  --> TIME: 2025-02-25 19:37:09 -- STEP: 67/72 -- GLOBAL_STEP: 1079700 | > loss_disc: 2.58573317527771 (2.6502503850566805) | > loss_disc_real_0: 0.10227658599615097 (0.10857619789998922) | > loss_disc_real_1: 0.24782903492450714 (0.2294741238453495) | > loss_disc_real_2: 0.24130508303642273 (0.23483468169596658) | > loss_disc_real_3: 0.21299058198928833 (0.24399527573763435) | > loss_disc_real_4: 0.2352016568183899 (0.2609800024708706) | > loss_disc_real_5: 0.227034330368042 (0.25976935295916326) | > loss_0: 2.58573317527771 (2.6502503850566805) | > grad_norm_0: tensor(436.0966, device='cuda:0') (tensor(263.6710, device='cuda:0')) | > loss_gen: 2.006251573562622 (2.0813931724918424) | > loss_kl: 2.241955518722534 (2.2041927451518046) | > loss_feat: 3.4707534313201904 (3.7347660064697266) | > loss_mel: 16.33818244934082 (17.500922644316255) | > loss_duration: 1.6378796100616455 (1.557868032313105) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.695024490356445 (27.07914252779377) | > grad_norm_1: tensor(14466.9551, device='cuda:0') (tensor(10160.4336, device='cuda:0')) | > current_lr_0: 0.0001741746837484212 | > current_lr_1: 0.0001741746837484212 | > step_time: 0.5197 (0.4424801050727047) | > loader_time: 0.0124 (0.008343646775430707) --> EVAL PERFORMANCE | > avg_loader_time: 0.008535912339116486 (+6.469202713227547e-05) | > avg_step_time: 0.4463793862033898 (-0.0011811457889180366) | > avg_loss_disc: 2.6442458864668725 (+0.010990703609627506) | > avg_loss_disc_real_0: 0.10778482426220262 (-0.05904074217861807) | > avg_loss_disc_real_1: 0.22942387858327007 (+0.00468022397286455) | > avg_loss_disc_real_2: 0.23409650296392576 (+0.011907081788694374) | > avg_loss_disc_real_3: 0.242328577058416 (+0.01378358879559477) | > avg_loss_disc_real_4: 0.2592417446660325 (+0.02198823425971297) | > avg_loss_disc_real_5: 0.25851656727387884 (+0.01716840812857734) | > avg_loss_0: 2.6442458864668725 (+0.010990703609627506) | > avg_grad_norm_0: tensor(268.1604, device='cuda:0') (tensor(-13.5331, device='cuda:0')) | > avg_loss_gen: 2.0809140289333508 (-0.11338687446755413) | > avg_loss_kl: 2.2054965932604293 (+0.17769284819213427) | > avg_loss_feat: 3.7388985660714162 (-0.11803503439459995) | > avg_loss_mel: 17.48886011687802 (+0.33252525329589133) | > avg_loss_duration: 1.560834684842069 (+0.006332919631205192) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.075003906035086 (+0.28512925161442126) | > avg_grad_norm_1: tensor(10276.0195, device='cuda:0') (+tensor(3339.5322, device='cuda:0'))  > EPOCH: 1107/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:37:13)   --> TIME: 2025-02-25 19:37:35 -- STEP: 45/72 -- GLOBAL_STEP: 1079750 | > loss_disc: 2.6424009799957275 (2.5845824241638176) | > loss_disc_real_0: 0.11904947459697723 (0.1168964425722758) | > loss_disc_real_1: 0.23669341206550598 (0.22540493839316897) | > loss_disc_real_2: 0.22560782730579376 (0.23555707236131032) | > loss_disc_real_3: 0.25429567694664 (0.23369032012091742) | > loss_disc_real_4: 0.2354661077260971 (0.24335915479395126) | > loss_disc_real_5: 0.25107213854789734 (0.24335897035068935) | > loss_0: 2.6424009799957275 (2.5845824241638176) | > grad_norm_0: tensor(341.4374, device='cuda:0') (tensor(355.8155, device='cuda:0')) | > loss_gen: 2.033936023712158 (2.1291645103030734) | > loss_kl: 2.094313144683838 (1.9524216466479831) | > loss_feat: 3.6805813312530518 (3.783731206258138) | > loss_mel: 16.15218162536621 (16.842789268493654) | > loss_duration: 1.5507516860961914 (1.5197021298938327) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.511764526367188 (26.227808803982207) | > grad_norm_1: tensor(8091.0889, device='cuda:0') (tensor(11769.1377, device='cuda:0')) | > current_lr_0: 0.00017415291191295263 | > current_lr_1: 0.00017415291191295263 | > step_time: 0.4514 (0.43679921362135143) | > loader_time: 0.0085 (0.007439083523220486) --> EVAL PERFORMANCE | > avg_loader_time: 0.008510697055870379 (-2.521528324610743e-05) | > avg_step_time: 0.45129173574313314 (+0.004912349539743344) | > avg_loss_disc: 2.58733006934045 (-0.056915817126422574) | > avg_loss_disc_real_0: 0.12001961595575575 (+0.012234791693553126) | > avg_loss_disc_real_1: 0.2231326122099245 (-0.0062912663733455665) | > avg_loss_disc_real_2: 0.23551617814621456 (+0.0014196751822888043) | > avg_loss_disc_real_3: 0.2334948291241283 (-0.008833747934287706) | > avg_loss_disc_real_4: 0.242561313467966 (-0.016680431198066503) | > avg_loss_disc_real_5: 0.24335122864011308 (-0.015165338633765763) | > avg_loss_0: 2.58733006934045 (-0.056915817126422574) | > avg_grad_norm_0: tensor(352.9771, device='cuda:0') (+tensor(84.8167, device='cuda:0')) | > avg_loss_gen: 2.115136339630879 (+0.034222310697528346) | > avg_loss_kl: 1.99882160777777 (-0.20667498548265928) | > avg_loss_feat: 3.6946494075613963 (-0.04424915851001998) | > avg_loss_mel: 16.74405991191595 (-0.7448002049620683) | > avg_loss_duration: 1.544741024433727 (-0.016093660408342148) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.097408294677734 (-0.9775956113573514) | > avg_grad_norm_1: tensor(11765.2510, device='cuda:0') (+tensor(1489.2314, device='cuda:0'))  > EPOCH: 1108/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:37:50)   --> TIME: 2025-02-25 19:38:02 -- STEP: 23/72 -- GLOBAL_STEP: 1079800 | > loss_disc: 2.7356350421905518 (2.563339202300362) | > loss_disc_real_0: 0.10017741471529007 (0.11130614837874538) | > loss_disc_real_1: 0.22712364792823792 (0.21434318824954654) | > loss_disc_real_2: 0.23240651190280914 (0.23005608501641647) | > loss_disc_real_3: 0.271854966878891 (0.23116640422655188) | > loss_disc_real_4: 0.2710495889186859 (0.23856536396171735) | > loss_disc_real_5: 0.2597034275531769 (0.24133331944113193) | > loss_0: 2.7356350421905518 (2.563339202300362) | > grad_norm_0: tensor(184.8013, device='cuda:0') (tensor(334.7143, device='cuda:0')) | > loss_gen: 1.9999269247055054 (2.130787025327268) | > loss_kl: 2.0386312007904053 (1.921556856321252) | > loss_feat: 3.2307991981506348 (3.9046842118968135) | > loss_mel: 15.245243072509766 (16.727592053620715) | > loss_duration: 1.5193252563476562 (1.4952969965727434) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.033926010131836 (26.179917211117953) | > grad_norm_1: tensor(11910.1553, device='cuda:0') (tensor(11510.5039, device='cuda:0')) | > current_lr_0: 0.0001741311427989635 | > current_lr_1: 0.0001741311427989635 | > step_time: 0.4335 (0.4310359643853229) | > loader_time: 0.0078 (0.006995304771091627) --> EVAL PERFORMANCE | > avg_loader_time: 0.008816396686392771 (+0.00030569963052239253) | > avg_step_time: 0.4505963224760244 (-0.0006954132671087576) | > avg_loss_disc: 2.5693178445520526 (-0.01801222478839737) | > avg_loss_disc_real_0: 0.11788388215739962 (-0.002135733798356132) | > avg_loss_disc_real_1: 0.2187190832386554 (-0.004413528971269093) | > avg_loss_disc_real_2: 0.23122817676671795 (-0.004288001379496614) | > avg_loss_disc_real_3: 0.23138012138890549 (-0.0021147077352228205) | > avg_loss_disc_real_4: 0.24060121220602115 (-0.0019601012619448466) | > avg_loss_disc_real_5: 0.2439543600653259 (+0.0006031314252128162) | > avg_loss_0: 2.5693178445520526 (-0.01801222478839737) | > avg_grad_norm_0: tensor(347.1750, device='cuda:0') (tensor(-5.8020, device='cuda:0')) | > avg_loss_gen: 2.1422433802779284 (+0.027107040647049274) | > avg_loss_kl: 1.9993345771037356 (+0.0005129693259655621) | > avg_loss_feat: 3.845873866282718 (+0.15122445872132184) | > avg_loss_mel: 16.573711086326917 (-0.1703488255890342) | > avg_loss_duration: 1.5384648719304044 (-0.006276152503322585) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.099627884340958 (+0.0022195896632233314) | > avg_grad_norm_1: tensor(12172.1641, device='cuda:0') (+tensor(406.9131, device='cuda:0'))  > EPOCH: 1109/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:38:26)   --> TIME: 2025-02-25 19:38:29 -- STEP: 1/72 -- GLOBAL_STEP: 1079850 | > loss_disc: 2.523409605026245 (2.523409605026245) | > loss_disc_real_0: 0.09794209897518158 (0.09794209897518158) | > loss_disc_real_1: 0.2501819431781769 (0.2501819431781769) | > loss_disc_real_2: 0.2376159131526947 (0.2376159131526947) | > loss_disc_real_3: 0.24028094112873077 (0.24028094112873077) | > loss_disc_real_4: 0.2533913552761078 (0.2533913552761078) | > loss_disc_real_5: 0.24967993795871735 (0.24967993795871735) | > loss_0: 2.523409605026245 (2.523409605026245) | > grad_norm_0: tensor(182.2387, device='cuda:0') (tensor(182.2387, device='cuda:0')) | > loss_gen: 2.3146610260009766 (2.3146610260009766) | > loss_kl: 1.9471052885055542 (1.9471052885055542) | > loss_feat: 3.9486591815948486 (3.9486591815948486) | > loss_mel: 16.181575775146484 (16.181575775146484) | > loss_duration: 1.498265266418457 (1.498265266418457) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.89026641845703 (25.89026641845703) | > grad_norm_1: tensor(12825.4434, device='cuda:0') (tensor(12825.4434, device='cuda:0')) | > current_lr_0: 0.00017410937640611362 | > current_lr_1: 0.00017410937640611362 | > step_time: 0.4149 (0.4148557186126709) | > loader_time: 0.0087 (0.008678913116455078)  --> TIME: 2025-02-25 19:38:51 -- STEP: 51/72 -- GLOBAL_STEP: 1079900 | > loss_disc: 2.5339250564575195 (2.767878789527744) | > loss_disc_real_0: 0.12894076108932495 (0.13350038822082913) | > loss_disc_real_1: 0.25289425253868103 (0.2079952500176196) | > loss_disc_real_2: 0.23860956728458405 (0.21781363966418252) | > loss_disc_real_3: 0.23611946403980255 (0.2136607248993481) | > loss_disc_real_4: 0.24033543467521667 (0.23547043519861557) | > loss_disc_real_5: 0.2336176186800003 (0.23691887569193745) | > loss_0: 2.5339250564575195 (2.767878789527744) | > grad_norm_0: tensor(287.6064, device='cuda:0') (tensor(406.6255, device='cuda:0')) | > loss_gen: 2.30802583694458 (1.92926424741745) | > loss_kl: 2.2718234062194824 (2.2052060108558798) | > loss_feat: 4.053201675415039 (3.8358765489914837) | > loss_mel: 18.460962295532227 (17.297090231203565) | > loss_duration: 1.6059616804122925 (1.539292992330065) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.699974060058594 (26.806729933794806) | > grad_norm_1: tensor(8254.9766, device='cuda:0') (tensor(11052.3242, device='cuda:0')) | > current_lr_0: 0.00017410937640611362 | > current_lr_1: 0.00017410937640611362 | > step_time: 0.4479 (0.437226557264141) | > loader_time: 0.009 (0.007721989762549307) --> EVAL PERFORMANCE | > avg_loader_time: 0.008609375483553174 (-0.0002070212028395975) | > avg_step_time: 0.4601243516089211 (+0.009528029132896743) | > avg_loss_disc: 2.709288509798722 (+0.13997066524666923) | > avg_loss_disc_real_0: 0.12749847433936426 (+0.009614592181964643) | > avg_loss_disc_real_1: 0.21059729185112766 (-0.00812179138752775) | > avg_loss_disc_real_2: 0.2219114230132439 (-0.009316753753474044) | > avg_loss_disc_real_3: 0.21601937858151718 (-0.015360742807388306) | > avg_loss_disc_real_4: 0.2349761315214802 (-0.005625080684540956) | > avg_loss_disc_real_5: 0.23557704141442204 (-0.008377318650903853) | > avg_loss_0: 2.709288509798722 (+0.13997066524666923) | > avg_grad_norm_0: tensor(392.0293, device='cuda:0') (+tensor(44.8542, device='cuda:0')) | > avg_loss_gen: 1.9733920911668053 (-0.16885128911112313) | > avg_loss_kl: 2.214977516254909 (+0.21564293915117316) | > avg_loss_feat: 3.8204680160737374 (-0.025405850208980674) | > avg_loss_mel: 17.342821685361184 (+0.769110599034267) | > avg_loss_duration: 1.558729930662773 (+0.020265058732368724) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.910389322630117 (+0.810761438289159) | > avg_grad_norm_1: tensor(11724.5645, device='cuda:0') (tensor(-447.5996, device='cuda:0'))  > EPOCH: 1110/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:39:03)   --> TIME: 2025-02-25 19:39:18 -- STEP: 29/72 -- GLOBAL_STEP: 1079950 | > loss_disc: 2.566894054412842 (2.635886940462836) | > loss_disc_real_0: 0.06393840909004211 (0.12372838551628176) | > loss_disc_real_1: 0.29043763875961304 (0.2315300934273621) | > loss_disc_real_2: 0.24404241144657135 (0.23653753718425488) | > loss_disc_real_3: 0.25350189208984375 (0.22937396117325487) | > loss_disc_real_4: 0.24396586418151855 (0.23564735568802933) | > loss_disc_real_5: 0.25236693024635315 (0.24216791216669412) | > loss_0: 2.566894054412842 (2.635886940462836) | > grad_norm_0: tensor(480.2448, device='cuda:0') (tensor(421.0919, device='cuda:0')) | > loss_gen: 2.3060619831085205 (2.083948640987791) | > loss_kl: 2.2250325679779053 (2.0504703809475076) | > loss_feat: 4.052460193634033 (3.970749074015124) | > loss_mel: 18.554515838623047 (17.19538192091317) | > loss_duration: 1.5209035873413086 (1.509408560292474) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.658973693847656 (26.809958425061456) | > grad_norm_1: tensor(12421.8770, device='cuda:0') (tensor(12768.6572, device='cuda:0')) | > current_lr_0: 0.00017408761273406286 | > current_lr_1: 0.00017408761273406286 | > step_time: 0.4619 (0.4346978335544981) | > loader_time: 0.0083 (0.007281706250947097) --> EVAL PERFORMANCE | > avg_loader_time: 0.00888679732739086 (+0.0002774218438376854) | > avg_step_time: 0.45250638102141905 (-0.0076179705875020765) | > avg_loss_disc: 2.6009981363592014 (-0.10829037343952042) | > avg_loss_disc_real_0: 0.11407042803688788 (-0.013428046302476376) | > avg_loss_disc_real_1: 0.22681951627764904 (+0.01622222442652138) | > avg_loss_disc_real_2: 0.23211259699203599 (+0.010201173978792083) | > avg_loss_disc_real_3: 0.23055546342487065 (+0.014536084843353475) | > avg_loss_disc_real_4: 0.23949176198999647 (+0.004515630468516274) | > avg_loss_disc_real_5: 0.24331932483424604 (+0.007742283419823998) | > avg_loss_0: 2.6009981363592014 (-0.10829037343952042) | > avg_grad_norm_0: tensor(374.4531, device='cuda:0') (tensor(-17.5762, device='cuda:0')) | > avg_loss_gen: 2.1159194288119454 (+0.14252733764514014) | > avg_loss_kl: 2.0916069292686363 (-0.12337058698627246) | > avg_loss_feat: 3.896024163340179 (+0.07555614726644144) | > avg_loss_mel: 17.028621001982362 (-0.31420068337882157) | > avg_loss_duration: 1.5525200837095021 (-0.006209846953270981) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.68469146943428 (-0.22569785319583602) | > avg_grad_norm_1: tensor(13025.9033, device='cuda:0') (+tensor(1301.3389, device='cuda:0'))  > EPOCH: 1111/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:39:40)   --> TIME: 2025-02-25 19:39:45 -- STEP: 7/72 -- GLOBAL_STEP: 1080000 | > loss_disc: 2.7441654205322266 (2.5750135353633334) | > loss_disc_real_0: 0.11917869001626968 (0.1005931624344417) | > loss_disc_real_1: 0.2306726723909378 (0.22038314385073526) | > loss_disc_real_2: 0.22261181473731995 (0.2247238733938762) | > loss_disc_real_3: 0.23328107595443726 (0.2520290868622916) | > loss_disc_real_4: 0.23942022025585175 (0.24175588147980825) | > loss_disc_real_5: 0.22585895657539368 (0.2433785625866481) | > loss_0: 2.7441654205322266 (2.5750135353633334) | > grad_norm_0: tensor(446.4306, device='cuda:0') (tensor(341.1194, device='cuda:0')) | > loss_gen: 1.9601552486419678 (2.1798441069466725) | > loss_kl: 1.8527830839157104 (1.936618753841945) | > loss_feat: 3.5859549045562744 (4.028562239238194) | > loss_mel: 17.10752296447754 (16.89752279009138) | > loss_duration: 1.4916174411773682 (1.4975350414003645) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.99803352355957 (26.540082386561803) | > grad_norm_1: tensor(13083.9580, device='cuda:0') (tensor(12745.2520, device='cuda:0')) | > current_lr_0: 0.0001740658517824711 | > current_lr_1: 0.0001740658517824711 | > step_time: 0.4195 (0.42387475286211285) | > loader_time: 0.0056 (0.006424903869628906) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1080000.pth  --> TIME: 2025-02-25 19:40:11 -- STEP: 57/72 -- GLOBAL_STEP: 1080050 | > loss_disc: 2.9522948265075684 (2.643794427838241) | > loss_disc_real_0: 0.37687841057777405 (0.15989810215276584) | > loss_disc_real_1: 0.22439153492450714 (0.2333302761901889) | > loss_disc_real_2: 0.20179393887519836 (0.23402138655645804) | > loss_disc_real_3: 0.23787599802017212 (0.23395326001602307) | > loss_disc_real_4: 0.2652761936187744 (0.241245614854913) | > loss_disc_real_5: 0.24547669291496277 (0.24478528138838315) | > loss_0: 2.9522948265075684 (2.643794427838241) | > grad_norm_0: tensor(32.1330, device='cuda:0') (tensor(338.0299, device='cuda:0')) | > loss_gen: 1.821272611618042 (2.171216111434133) | > loss_kl: 3.7209136486053467 (2.0715426248416566) | > loss_feat: 3.0932726860046387 (3.9127191786180466) | > loss_mel: 20.907554626464844 (16.994184393631784) | > loss_duration: 2.150811195373535 (1.5571043010343586) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.693824768066406 (26.706766597011633) | > grad_norm_1: tensor(1073.6771, device='cuda:0') (tensor(11087.2959, device='cuda:0')) | > current_lr_0: 0.0001740658517824711 | > current_lr_1: 0.0001740658517824711 | > step_time: 0.4465 (0.44151347980164646) | > loader_time: 0.0095 (0.008182680397702936) --> EVAL PERFORMANCE | > avg_loader_time: 0.008756429376736494 (-0.00013036795065436528) | > avg_step_time: 0.4530866952009604 (+0.0005803141795413524) | > avg_loss_disc: 2.669554240267041 (+0.06855610390783973) | > avg_loss_disc_real_0: 0.16414556757245266 (+0.05007513953556478) | > avg_loss_disc_real_1: 0.23303121034528168 (+0.006211694067632639) | > avg_loss_disc_real_2: 0.23393423561479004 (+0.001821638622754057) | > avg_loss_disc_real_3: 0.23046539996711302 (-9.006345775763869e-05) | > avg_loss_disc_real_4: 0.23991833333398255 (+0.0004265713439860763) | > avg_loss_disc_real_5: 0.24367774939033346 (+0.0003584245560874222) | > avg_loss_0: 2.669554240267041 (+0.06855610390783973) | > avg_grad_norm_0: tensor(313.3991, device='cuda:0') (tensor(-61.0540, device='cuda:0')) | > avg_loss_gen: 2.1206999446304753 (+0.004780515818529896) | > avg_loss_kl: 2.2547330805953125 (+0.16312615132667618) | > avg_loss_feat: 3.8076146622778664 (-0.08840950106231249) | > avg_loss_mel: 17.583843795346542 (+0.5552227933641802) | > avg_loss_duration: 1.5973159813545117 (+0.044795897645009575) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.36420746924172 (+0.6795159998074389) | > avg_grad_norm_1: tensor(9580.0098, device='cuda:0') (tensor(-3445.8936, device='cuda:0'))  > EPOCH: 1112/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:40:20)   --> TIME: 2025-02-25 19:40:38 -- STEP: 35/72 -- GLOBAL_STEP: 1080100 | > loss_disc: 3.197434425354004 (2.601588371821812) | > loss_disc_real_0: 0.4467945098876953 (0.10882265227181571) | > loss_disc_real_1: 0.48739519715309143 (0.2681392329079764) | > loss_disc_real_2: 0.220718115568161 (0.25774923861026766) | > loss_disc_real_3: 0.25028011202812195 (0.2375868333237512) | > loss_disc_real_4: 0.333289235830307 (0.25662372282573154) | > loss_disc_real_5: 0.21190078556537628 (0.24029901155403682) | > loss_0: 3.197434425354004 (2.601588371821812) | > grad_norm_0: tensor(511.4708, device='cuda:0') (tensor(206.8352, device='cuda:0')) | > loss_gen: 2.1469922065734863 (2.3009684698922293) | > loss_kl: 2.2860639095306396 (2.3343363285064695) | > loss_feat: 3.5284759998321533 (4.196588352748326) | > loss_mel: 17.69277000427246 (18.043797193254743) | > loss_duration: 1.5781787633895874 (1.5697192907333375) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.232479095458984 (28.445409502301896) | > grad_norm_1: tensor(3128.0701, device='cuda:0') (tensor(3414.8489, device='cuda:0')) | > current_lr_0: 0.0001740440935509983 | > current_lr_1: 0.0001740440935509983 | > step_time: 0.4409 (0.4332252706800188) | > loader_time: 0.0082 (0.007176610401698522) --> EVAL PERFORMANCE | > avg_loader_time: 0.008615070665386358 (-0.0001413587113501362) | > avg_step_time: 0.45516054731019784 (+0.0020738521092374396) | > avg_loss_disc: 2.6748997896489954 (+0.0053455493819543065) | > avg_loss_disc_real_0: 0.1560563697466548 (-0.008089197825797867) | > avg_loss_disc_real_1: 0.2543241935716549 (+0.02129298322637324) | > avg_loss_disc_real_2: 0.237430075734434 (+0.0034958401196439615) | > avg_loss_disc_real_3: 0.23676432732125402 (+0.006298927354141004) | > avg_loss_disc_real_4: 0.2475241880181809 (+0.00760585468419836) | > avg_loss_disc_real_5: 0.23995708327897838 (-0.0037206661113550754) | > avg_loss_0: 2.6748997896489954 (+0.0053455493819543065) | > avg_grad_norm_0: tensor(231.4213, device='cuda:0') (tensor(-81.9778, device='cuda:0')) | > avg_loss_gen: 2.1267770562373425 (+0.0060771116068671915) | > avg_loss_kl: 2.309465127931514 (+0.05473204733620163) | > avg_loss_feat: 3.8091323879403127 (+0.0015177256624463453) | > avg_loss_mel: 17.621791409774566 (+0.03794761442802397) | > avg_loss_duration: 1.585212020806863 (-0.012103960547648729) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.452378017801635 (+0.08817054855991557) | > avg_grad_norm_1: tensor(4564.1860, device='cuda:0') (tensor(-5015.8237, device='cuda:0'))  > EPOCH: 1113/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:40:57)   --> TIME: 2025-02-25 19:41:05 -- STEP: 13/72 -- GLOBAL_STEP: 1080150 | > loss_disc: 2.6851840019226074 (2.63562273979187) | > loss_disc_real_0: 0.1443347930908203 (0.11666101188613819) | > loss_disc_real_1: 0.22038640081882477 (0.22537625982211187) | > loss_disc_real_2: 0.23655061423778534 (0.23115869668813852) | > loss_disc_real_3: 0.25485700368881226 (0.21884856659632462) | > loss_disc_real_4: 0.2365497350692749 (0.2478330800166497) | > loss_disc_real_5: 0.24095222353935242 (0.24286879140597123) | > loss_0: 2.6851840019226074 (2.63562273979187) | > grad_norm_0: tensor(78.8092, device='cuda:0') (tensor(169.8290, device='cuda:0')) | > loss_gen: 1.9504705667495728 (2.135015450991117) | > loss_kl: 2.240644693374634 (2.06576367524954) | > loss_feat: 3.76709246635437 (4.016399695323064) | > loss_mel: 17.695711135864258 (17.221248700068546) | > loss_duration: 1.4413094520568848 (1.513581367639395) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.09522819519043 (26.952009201049805) | > grad_norm_1: tensor(6567.5200, device='cuda:0') (tensor(6994.0552, device='cuda:0')) | > current_lr_0: 0.00017402233803930442 | > current_lr_1: 0.00017402233803930442 | > step_time: 0.5139 (0.43450289506178635) | > loader_time: 0.01 (0.007008075714111328)  --> TIME: 2025-02-25 19:41:28 -- STEP: 63/72 -- GLOBAL_STEP: 1080200 | > loss_disc: 2.7610232830047607 (2.647381706843301) | > loss_disc_real_0: 0.1482800543308258 (0.08600343947136213) | > loss_disc_real_1: 0.24957412481307983 (0.2215877641288061) | > loss_disc_real_2: 0.331417441368103 (0.26388190048081533) | > loss_disc_real_3: 0.30343860387802124 (0.2481676356659995) | > loss_disc_real_4: 0.2699982523918152 (0.24469069876368083) | > loss_disc_real_5: 0.26622629165649414 (0.2393569459044744) | > loss_0: 2.7610232830047607 (2.647381706843301) | > grad_norm_0: tensor(291.1445, device='cuda:0') (tensor(191.2246, device='cuda:0')) | > loss_gen: 2.2904844284057617 (2.1791251443681263) | > loss_kl: 2.129300594329834 (2.138076189964537) | > loss_feat: 3.4741055965423584 (4.09239173313928) | > loss_mel: 16.57032585144043 (17.351129925440237) | > loss_duration: 1.586854338645935 (1.54893398095691) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.05107307434082 (27.309657081725106) | > grad_norm_1: tensor(13764.3223, device='cuda:0') (tensor(6619.5444, device='cuda:0')) | > current_lr_0: 0.00017402233803930442 | > current_lr_1: 0.00017402233803930442 | > step_time: 0.4947 (0.44365059004889595) | > loader_time: 0.0109 (0.008069878532772972) --> EVAL PERFORMANCE | > avg_loader_time: 0.00845157932227766 (-0.0001634913431086977) | > avg_step_time: 0.449128657999173 (-0.0060318893110248295) | > avg_loss_disc: 2.6752068794948958 (+0.0003070898459003324) | > avg_loss_disc_real_0: 0.1162967362349302 (-0.039759633511724596) | > avg_loss_disc_real_1: 0.2226960579274406 (-0.03162813564421432) | > avg_loss_disc_real_2: 0.26005612922386384 (+0.022626053489429837) | > avg_loss_disc_real_3: 0.24664013213674787 (+0.009875804815493855) | > avg_loss_disc_real_4: 0.24225244534687257 (-0.005271742671308338) | > avg_loss_disc_real_5: 0.2405663466369602 (+0.0006092633579818096) | > avg_loss_0: 2.6752068794948958 (+0.0003070898459003324) | > avg_grad_norm_0: tensor(197.3014, device='cuda:0') (tensor(-34.1199, device='cuda:0')) | > avg_loss_gen: 2.18309013272675 (+0.05631307648940753) | > avg_loss_kl: 2.146865362852392 (-0.162599765079122) | > avg_loss_feat: 3.9930027303561357 (+0.183870342415823) | > avg_loss_mel: 17.29330874160982 (-0.3284826681647459) | > avg_loss_duration: 1.5563338273008105 (-0.028878193506052474) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.172600893907145 (-0.2797771238944904) | > avg_grad_norm_1: tensor(6498.9194, device='cuda:0') (+tensor(1934.7334, device='cuda:0'))  > EPOCH: 1114/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:41:34)   --> TIME: 2025-02-25 19:41:54 -- STEP: 41/72 -- GLOBAL_STEP: 1080250 | > loss_disc: 2.615485429763794 (2.685291947388067) | > loss_disc_real_0: 0.15523551404476166 (0.16596882181560121) | > loss_disc_real_1: 0.295244425535202 (0.2318254193154777) | > loss_disc_real_2: 0.221842423081398 (0.23351211373398945) | > loss_disc_real_3: 0.22073477506637573 (0.24176866397625058) | > loss_disc_real_4: 0.2556157410144806 (0.24490134890486553) | > loss_disc_real_5: 0.26924851536750793 (0.24424445738152759) | > loss_0: 2.615485429763794 (2.685291947388067) | > grad_norm_0: tensor(231.5303, device='cuda:0') (tensor(234.9428, device='cuda:0')) | > loss_gen: 2.211191177368164 (2.0283869301400532) | > loss_kl: 5.539342880249023 (2.256244932732931) | > loss_feat: 3.439605951309204 (3.5327760824343053) | > loss_mel: 37.942039489746094 (18.489887446891977) | > loss_duration: 2.485081672668457 (1.5900362410196445) | > amp_scaler: 8.0 (8.0) | > loss_1: 51.61725997924805 (27.897331656479253) | > grad_norm_1: tensor(8765.7139, device='cuda:0') (tensor(7923.4448, device='cuda:0')) | > current_lr_0: 0.0001740005852470495 | > current_lr_1: 0.0001740005852470495 | > step_time: 0.457 (0.43423916653888983) | > loader_time: 0.0088 (0.0072861357433039965) --> EVAL PERFORMANCE | > avg_loader_time: 0.008497772082476549 (+4.619276019888867e-05) | > avg_step_time: 0.449312300749228 (+0.00018364275005500197) | > avg_loss_disc: 2.693167011502763 (+0.01796013200786728) | > avg_loss_disc_real_0: 0.1669058044716506 (+0.0506090682367204) | > avg_loss_disc_real_1: 0.23495319128876002 (+0.012257133361319422) | > avg_loss_disc_real_2: 0.237388379976783 (-0.022667749247080843) | > avg_loss_disc_real_3: 0.2483823450518326 (+0.0017422129150847232) | > avg_loss_disc_real_4: 0.2491651903575575 (+0.006912745010684923) | > avg_loss_disc_real_5: 0.24865072671796234 (+0.008084380081002152) | > avg_loss_0: 2.693167011502763 (+0.01796013200786728) | > avg_grad_norm_0: tensor(222.5829, device='cuda:0') (+tensor(25.2815, device='cuda:0')) | > avg_loss_gen: 2.100945825308141 (-0.08214430741860923) | > avg_loss_kl: 2.6699179226243994 (+0.5230525597720073) | > avg_loss_feat: 3.5507700275367413 (-0.4422327028193944) | > avg_loss_mel: 21.114934746648228 (+3.821626005038407) | > avg_loss_duration: 1.737279018885653 (+0.18094519158484257) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 31.1738477626317 (+4.001246868724554) | > avg_grad_norm_1: tensor(6936.9033, device='cuda:0') (+tensor(437.9839, device='cuda:0'))  > EPOCH: 1115/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:42:10)   --> TIME: 2025-02-25 19:42:21 -- STEP: 19/72 -- GLOBAL_STEP: 1080300 | > loss_disc: 2.6894710063934326 (2.5957047060916296) | > loss_disc_real_0: 0.07456162571907043 (0.17586904881816162) | > loss_disc_real_1: 0.18211036920547485 (0.20419550412579587) | > loss_disc_real_2: 0.2093651443719864 (0.23752786062265696) | > loss_disc_real_3: 0.20774130523204803 (0.23823061114863345) | > loss_disc_real_4: 0.21007564663887024 (0.23691208425321078) | > loss_disc_real_5: 0.22052310407161713 (0.22658001592284754) | > loss_0: 2.6894710063934326 (2.5957047060916296) | > grad_norm_0: tensor(309.3840, device='cuda:0') (tensor(225.0511, device='cuda:0')) | > loss_gen: 1.7572492361068726 (2.108640821356522) | > loss_kl: 2.301739454269409 (2.203004560972515) | > loss_feat: 3.93515944480896 (3.7106487625523616) | > loss_mel: 17.891902923583984 (18.40079779373972) | > loss_duration: 1.5764970779418945 (1.5995236258757741) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.462547302246094 (28.022615733899567) | > grad_norm_1: tensor(11997.7168, device='cuda:0') (tensor(7016.0322, device='cuda:0')) | > current_lr_0: 0.00017397883517389362 | > current_lr_1: 0.00017397883517389362 | > step_time: 0.4141 (0.4311022633000424) | > loader_time: 0.0059 (0.006654764476575349)  --> TIME: 2025-02-25 19:42:44 -- STEP: 69/72 -- GLOBAL_STEP: 1080350 | > loss_disc: 2.5032541751861572 (2.555857547815295) | > loss_disc_real_0: 0.09449713677167892 (0.11882412530805748) | > loss_disc_real_1: 0.23293234407901764 (0.21528788688390152) | > loss_disc_real_2: 0.2382766157388687 (0.23514883760092914) | > loss_disc_real_3: 0.2371617704629898 (0.23451935251553854) | > loss_disc_real_4: 0.24583883583545685 (0.23860192277293274) | > loss_disc_real_5: 0.25788742303848267 (0.23987101407154746) | > loss_0: 2.5032541751861572 (2.555857547815295) | > grad_norm_0: tensor(137.5917, device='cuda:0') (tensor(219.0691, device='cuda:0')) | > loss_gen: 2.240739345550537 (2.1444804357445744) | > loss_kl: 2.239976644515991 (2.115003559900366) | > loss_feat: 3.745178461074829 (3.7958267357038413) | > loss_mel: 15.815495491027832 (17.472264510997828) | > loss_duration: 1.6746318340301514 (1.6016155083974202) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.716022491455078 (27.129190693730894) | > grad_norm_1: tensor(4389.7026, device='cuda:0') (tensor(7057.7256, device='cuda:0')) | > current_lr_0: 0.00017397883517389362 | > current_lr_1: 0.00017397883517389362 | > step_time: 0.5557 (0.4493546244026958) | > loader_time: 0.0125 (0.008414085360540858) --> EVAL PERFORMANCE | > avg_loader_time: 0.008470390883969586 (-2.7381198506962537e-05) | > avg_step_time: 0.4513748800250846 (+0.0020625792758565686) | > avg_loss_disc: 2.559184460572793 (-0.13398255092997013) | > avg_loss_disc_real_0: 0.11775624909451306 (-0.04914955537713754) | > avg_loss_disc_real_1: 0.2160544023967125 (-0.01889878889204752) | > avg_loss_disc_real_2: 0.23523196745926225 (-0.002156412517520745) | > avg_loss_disc_real_3: 0.23441691633681178 (-0.013965428715020822) | > avg_loss_disc_real_4: 0.23904008575728242 (-0.010125104600275076) | > avg_loss_disc_real_5: 0.24090293700426396 (-0.007747789713698383) | > avg_loss_0: 2.559184460572793 (-0.13398255092997013) | > avg_grad_norm_0: tensor(217.8976, device='cuda:0') (tensor(-4.6853, device='cuda:0')) | > avg_loss_gen: 2.1403801340452366 (+0.03943430873709586) | > avg_loss_kl: 2.113666908841737 (-0.5562510137826626) | > avg_loss_feat: 3.7841447977952556 (+0.23337477025851427) | > avg_loss_mel: 17.429944065255178 (-3.6849906813930495) | > avg_loss_duration: 1.6015589405113542 (-0.1357200783742989) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.06969476081955 (-4.104153001812147) | > avg_grad_norm_1: tensor(7053.6201, device='cuda:0') (+tensor(116.7168, device='cuda:0'))  > EPOCH: 1116/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:42:47)   --> TIME: 2025-02-25 19:43:11 -- STEP: 47/72 -- GLOBAL_STEP: 1080400 | > loss_disc: 2.4145543575286865 (2.545723194771625) | > loss_disc_real_0: 0.059769850224256516 (0.09668878394555538) | > loss_disc_real_1: 0.21252509951591492 (0.2176191064905613) | > loss_disc_real_2: 0.23844687640666962 (0.23138564508012) | > loss_disc_real_3: 0.23368555307388306 (0.23090344001638127) | > loss_disc_real_4: 0.2442265897989273 (0.2438809998491977) | > loss_disc_real_5: 0.24684381484985352 (0.2515632665537773) | > loss_0: 2.4145543575286865 (2.545723194771625) | > grad_norm_0: tensor(47.0870, device='cuda:0') (tensor(257.6861, device='cuda:0')) | > loss_gen: 2.2460944652557373 (2.1961098807923345) | > loss_kl: 1.9031589031219482 (1.9809265745447038) | > loss_feat: 4.557706832885742 (4.09715965960888) | > loss_mel: 16.91959571838379 (17.05050462357541) | > loss_duration: 1.6062164306640625 (1.5477242140059775) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.232772827148438 (26.872424714108732) | > grad_norm_1: tensor(8236.1016, device='cuda:0') (tensor(7643.8223, device='cuda:0')) | > current_lr_0: 0.00017395708781949687 | > current_lr_1: 0.00017395708781949687 | > step_time: 0.4481 (0.44259567463651617) | > loader_time: 0.0109 (0.007791539455981965) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722745196920043 (+0.000252354312950457) | > avg_step_time: 0.45313259245644155 (+0.0017577124313569725) | > avg_loss_disc: 2.5575506384943574 (-0.0016338220784355428) | > avg_loss_disc_real_0: 0.09563943489231694 (-0.02211681420219612) | > avg_loss_disc_real_1: 0.22174256662247885 (+0.005688164225766346) | > avg_loss_disc_real_2: 0.23149504565017323 (-0.003736921809089022) | > avg_loss_disc_real_3: 0.23346738412346638 (-0.0009495322133453998) | > avg_loss_disc_real_4: 0.24347063605214508 (+0.004430550294862667) | > avg_loss_disc_real_5: 0.2500028370971411 (+0.009099900092877145) | > avg_loss_0: 2.5575506384943574 (-0.0016338220784355428) | > avg_grad_norm_0: tensor(266.0209, device='cuda:0') (+tensor(48.1233, device='cuda:0')) | > avg_loss_gen: 2.1781561341084235 (+0.03777600006318682) | > avg_loss_kl: 2.019637730759634 (-0.09402917808210276) | > avg_loss_feat: 4.028880119323729 (+0.2447353215284731) | > avg_loss_mel: 16.96185874938964 (-0.4680853158655367) | > avg_loss_duration: 1.5651443877690274 (-0.036414552742326745) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.753677018931217 (-0.3160177418883343) | > avg_grad_norm_1: tensor(8113.5317, device='cuda:0') (+tensor(1059.9116, device='cuda:0'))  > EPOCH: 1117/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:43:24)   --> TIME: 2025-02-25 19:43:37 -- STEP: 25/72 -- GLOBAL_STEP: 1080450 | > loss_disc: 2.5525660514831543 (2.5286018371582033) | > loss_disc_real_0: 0.06590719521045685 (0.07824903920292854) | > loss_disc_real_1: 0.22054830193519592 (0.221752889752388) | > loss_disc_real_2: 0.21345284581184387 (0.2274428564310074) | > loss_disc_real_3: 0.2222873717546463 (0.23223113179206847) | > loss_disc_real_4: 0.2347147911787033 (0.24451643586158753) | > loss_disc_real_5: 0.23767568171024323 (0.25279966652393343) | > loss_0: 2.5525660514831543 (2.5286018371582033) | > grad_norm_0: tensor(207.4906, device='cuda:0') (tensor(313.1017, device='cuda:0')) | > loss_gen: 2.052107810974121 (2.1743131637573243) | > loss_kl: 2.0005552768707275 (1.8719946241378784) | > loss_feat: 3.7064108848571777 (4.01491189956665) | > loss_mel: 16.23911476135254 (16.6758406829834) | > loss_duration: 1.512528657913208 (1.5114265251159669) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.510717391967773 (26.248486785888673) | > grad_norm_1: tensor(11095.2334, device='cuda:0') (tensor(10842.2363, device='cuda:0')) | > current_lr_0: 0.00017393534318351942 | > current_lr_1: 0.00017393534318351942 | > step_time: 0.4443 (0.43514013290405273) | > loader_time: 0.0099 (0.007124414443969726) --> EVAL PERFORMANCE | > avg_loader_time: 0.008794418523009395 (+7.167332608935123e-05) | > avg_step_time: 0.45498020548216056 (+0.0018476130257190082) | > avg_loss_disc: 2.5275101057240654 (-0.03004053277029195) | > avg_loss_disc_real_0: 0.0894355716210016 (-0.006203863271315335) | > avg_loss_disc_real_1: 0.2216357301238557 (-0.00010683649862314182) | > avg_loss_disc_real_2: 0.2289369414809724 (-0.0025581041692008255) | > avg_loss_disc_real_3: 0.23175436153378284 (-0.0017130225896835327) | > avg_loss_disc_real_4: 0.24318413709250974 (-0.00028649895963533956) | > avg_loss_disc_real_5: 0.24883725084889102 (-0.0011655862482500834) | > avg_loss_0: 2.5275101057240654 (-0.03004053277029195) | > avg_grad_norm_0: tensor(269.8508, device='cuda:0') (+tensor(3.8299, device='cuda:0')) | > avg_loss_gen: 2.1916613881017106 (+0.01350525399328717) | > avg_loss_kl: 1.9465797484760554 (-0.07305798228357863) | > avg_loss_feat: 3.9044727741832457 (-0.12440734514048302) | > avg_loss_mel: 16.587414271394966 (-0.3744444779946754) | > avg_loss_duration: 1.5501083303505265 (-0.015036057418500892) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.18023638658121 (-0.5734406323500068) | > avg_grad_norm_1: tensor(8926.2627, device='cuda:0') (+tensor(812.7310, device='cuda:0'))  > EPOCH: 1118/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:44:01)   --> TIME: 2025-02-25 19:44:04 -- STEP: 3/72 -- GLOBAL_STEP: 1080500 | > loss_disc: 2.465116500854492 (2.515780766805013) | > loss_disc_real_0: 0.17285369336605072 (0.13344808667898178) | > loss_disc_real_1: 0.19607077538967133 (0.1957213580608368) | > loss_disc_real_2: 0.2164752185344696 (0.21720650295416513) | > loss_disc_real_3: 0.1837589293718338 (0.20627163847287497) | > loss_disc_real_4: 0.23622113466262817 (0.2368624210357666) | > loss_disc_real_5: 0.2373044192790985 (0.2484700083732605) | > loss_0: 2.465116500854492 (2.515780766805013) | > grad_norm_0: tensor(480.0785, device='cuda:0') (tensor(340.5622, device='cuda:0')) | > loss_gen: 2.452831268310547 (2.3170105616251626) | > loss_kl: 2.047102689743042 (1.8816417455673218) | > loss_feat: 4.383821964263916 (4.130267143249512) | > loss_mel: 16.516515731811523 (16.703694661458332) | > loss_duration: 1.5440315008163452 (1.5134281714757283) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.944303512573242 (26.54604212443034) | > grad_norm_1: tensor(6435.9399, device='cuda:0') (tensor(8708.4746, device='cuda:0')) | > current_lr_0: 0.00017391360126562148 | > current_lr_1: 0.00017391360126562148 | > step_time: 0.4308 (0.43002136548360187) | > loader_time: 0.0057 (0.006974538167317708)  --> TIME: 2025-02-25 19:44:27 -- STEP: 53/72 -- GLOBAL_STEP: 1080550 | > loss_disc: 2.448849678039551 (2.544236192163432) | > loss_disc_real_0: 0.11812897771596909 (0.11430329984089113) | > loss_disc_real_1: 0.22703568637371063 (0.2119379722565975) | > loss_disc_real_2: 0.23515866696834564 (0.21833045387043143) | > loss_disc_real_3: 0.22148144245147705 (0.22326534564764994) | > loss_disc_real_4: 0.20827321708202362 (0.2386947752169843) | > loss_disc_real_5: 0.23350287973880768 (0.24293789633040158) | > loss_0: 2.448849678039551 (2.544236192163432) | > grad_norm_0: tensor(352.4244, device='cuda:0') (tensor(313.9139, device='cuda:0')) | > loss_gen: 2.2257025241851807 (2.192440480556128) | > loss_kl: 1.8973689079284668 (1.9235228502525474) | > loss_feat: 4.061084747314453 (4.032284300282315) | > loss_mel: 15.973523139953613 (16.620092643881737) | > loss_duration: 1.58620285987854 (1.525201163201962) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.743881225585938 (26.293541170516104) | > grad_norm_1: tensor(8039.7783, device='cuda:0') (tensor(8147.0684, device='cuda:0')) | > current_lr_0: 0.00017391360126562148 | > current_lr_1: 0.00017391360126562148 | > step_time: 0.4487 (0.4439443327345938) | > loader_time: 0.0091 (0.007873800565611643) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862995335753535 (-0.00016446516547404498) | > avg_step_time: 0.4575298940631705 (+0.0025496885810099568) | > avg_loss_disc: 2.5612961507179377 (+0.03378604499387228) | > avg_loss_disc_real_0: 0.11533248802305947 (+0.025896916402057865) | > avg_loss_disc_real_1: 0.2116257488517694 (-0.010009981272086299) | > avg_loss_disc_real_2: 0.22227970164426616 (-0.006657239836706241) | > avg_loss_disc_real_3: 0.2235111008647462 (-0.008243260669036634) | > avg_loss_disc_real_4: 0.24104444472722605 (-0.0021396923652836908) | > avg_loss_disc_real_5: 0.2427793522955666 (-0.00605789855332442) | > avg_loss_0: 2.5612961507179377 (+0.03378604499387228) | > avg_grad_norm_0: tensor(326.6878, device='cuda:0') (+tensor(56.8370, device='cuda:0')) | > avg_loss_gen: 2.176212137853595 (-0.015449250248115831) | > avg_loss_kl: 1.9706202594327256 (+0.024040510956670147) | > avg_loss_feat: 4.0042447540122 (+0.09977197982895447) | > avg_loss_mel: 16.625577214738005 (+0.038162943343039046) | > avg_loss_duration: 1.5422787783851086 (-0.007829551965417902) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.31893289592904 (+0.13869650934783095) | > avg_grad_norm_1: tensor(8182.3247, device='cuda:0') (tensor(-743.9380, device='cuda:0'))  > EPOCH: 1119/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:44:38)   --> TIME: 2025-02-25 19:44:54 -- STEP: 31/72 -- GLOBAL_STEP: 1080600 | > loss_disc: 2.4842140674591064 (2.61933066768031) | > loss_disc_real_0: 0.06543998420238495 (0.18465740954683674) | > loss_disc_real_1: 0.2605876922607422 (0.22393645682642538) | > loss_disc_real_2: 0.2279486507177353 (0.2463938319875348) | > loss_disc_real_3: 0.24997787177562714 (0.2279719465201901) | > loss_disc_real_4: 0.260379821062088 (0.22801355056224332) | > loss_disc_real_5: 0.2516762912273407 (0.23681348754513648) | > loss_0: 2.4842140674591064 (2.61933066768031) | > grad_norm_0: tensor(65.3738, device='cuda:0') (tensor(309.9894, device='cuda:0')) | > loss_gen: 2.1983070373535156 (2.273329211819557) | > loss_kl: 1.9048503637313843 (2.0154372261416533) | > loss_feat: 4.097142219543457 (4.024976907237884) | > loss_mel: 16.912126541137695 (16.853789852511497) | > loss_duration: 1.5207237005233765 (1.5021762617172734) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.633150100708008 (26.669709297918505) | > grad_norm_1: tensor(4608.7539, device='cuda:0') (tensor(5272.6060, device='cuda:0')) | > current_lr_0: 0.00017389186206546328 | > current_lr_1: 0.00017389186206546328 | > step_time: 0.4516 (0.4503209052547332) | > loader_time: 0.0082 (0.006912777500767861) --> EVAL PERFORMANCE | > avg_loader_time: 0.0084982220555695 (-0.0001317313019658494) | > avg_step_time: 0.4622513945673553 (+0.004721500504184795) | > avg_loss_disc: 2.6185931152021378 (+0.057296964484200075) | > avg_loss_disc_real_0: 0.1336760240751253 (+0.01834353605206583) | > avg_loss_disc_real_1: 0.22023417003137966 (+0.008608421179610248) | > avg_loss_disc_real_2: 0.24707914626514407 (+0.02479944462087791) | > avg_loss_disc_real_3: 0.2397606826164353 (+0.016249581751689085) | > avg_loss_disc_real_4: 0.23651851881557787 (-0.004525925911648188) | > avg_loss_disc_real_5: 0.24439515467260925 (+0.0016158023770426466) | > avg_loss_0: 2.6185931152021378 (+0.057296964484200075) | > avg_grad_norm_0: tensor(316.0035, device='cuda:0') (tensor(-10.6842, device='cuda:0')) | > avg_loss_gen: 2.1826514274301663 (+0.006439289576571472) | > avg_loss_kl: 2.0345735952887747 (+0.06395333585604912) | > avg_loss_feat: 3.9438107584563777 (-0.060433995555822406) | > avg_loss_mel: 16.72775690320512 (+0.102179688467114) | > avg_loss_duration: 1.54300285560984 (+0.0007240772247314453) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.43179549633617 (+0.11286260040713003) | > avg_grad_norm_1: tensor(7428.0898, device='cuda:0') (tensor(-754.2349, device='cuda:0'))  > EPOCH: 1120/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:45:15)   --> TIME: 2025-02-25 19:45:21 -- STEP: 9/72 -- GLOBAL_STEP: 1080650 | > loss_disc: 2.8640694618225098 (2.6143188211652966) | > loss_disc_real_0: 0.09211136400699615 (0.09915955737233162) | > loss_disc_real_1: 0.26172295212745667 (0.19232307705614302) | > loss_disc_real_2: 0.24686376750469208 (0.24282421668370566) | > loss_disc_real_3: 0.2651025652885437 (0.23311286005708906) | > loss_disc_real_4: 0.26839450001716614 (0.24827583299742806) | > loss_disc_real_5: 0.27848201990127563 (0.24614873031775156) | > loss_0: 2.8640694618225098 (2.6143188211652966) | > grad_norm_0: tensor(541.6682, device='cuda:0') (tensor(348.3017, device='cuda:0')) | > loss_gen: 2.025451183319092 (2.0916792816585965) | > loss_kl: 1.8867034912109375 (1.918867786725362) | > loss_feat: 3.6454837322235107 (3.8540301852756076) | > loss_mel: 17.015222549438477 (16.201080534193252) | > loss_duration: 1.5088491439819336 (1.4973140954971313) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.081710815429688 (25.562971962822807) | > grad_norm_1: tensor(15442.9688, device='cuda:0') (tensor(10022.5312, device='cuda:0')) | > current_lr_0: 0.00017387012558270508 | > current_lr_1: 0.00017387012558270508 | > step_time: 0.4444 (0.43359197510613334) | > loader_time: 0.0069 (0.005796141094631619)  --> TIME: 2025-02-25 19:45:44 -- STEP: 59/72 -- GLOBAL_STEP: 1080700 | > loss_disc: 2.402916669845581 (2.540400844509319) | > loss_disc_real_0: 0.07991056889295578 (0.08949775848600823) | > loss_disc_real_1: 0.18811503052711487 (0.2154166261523457) | > loss_disc_real_2: 0.22872678935527802 (0.23627405676801325) | > loss_disc_real_3: 0.23037797212600708 (0.2323097185058109) | > loss_disc_real_4: 0.23418296873569489 (0.24435292462171132) | > loss_disc_real_5: 0.24583804607391357 (0.24557413059776112) | > loss_0: 2.402916669845581 (2.540400844509319) | > grad_norm_0: tensor(157.1511, device='cuda:0') (tensor(280.1771, device='cuda:0')) | > loss_gen: 2.387349843978882 (2.183931696212898) | > loss_kl: 2.0467352867126465 (1.9612765857728862) | > loss_feat: 4.373989105224609 (3.943464848954799) | > loss_mel: 15.735528945922852 (16.400947586964755) | > loss_duration: 1.5777188539505005 (1.5265427524760617) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.121322631835938 (26.016163551201256) | > grad_norm_1: tensor(7209.7520, device='cuda:0') (tensor(8661.7695, device='cuda:0')) | > current_lr_0: 0.00017387012558270508 | > current_lr_1: 0.00017387012558270508 | > step_time: 0.469 (0.444259025282779) | > loader_time: 0.0096 (0.007766687263876705) --> EVAL PERFORMANCE | > avg_loader_time: 0.008356188384579941 (-0.0001420336709895588) | > avg_step_time: 0.4535420981931015 (-0.008709296374253828) | > avg_loss_disc: 2.544164301643909 (-0.07442881355822895) | > avg_loss_disc_real_0: 0.09110538551295307 (-0.04257063856217223) | > avg_loss_disc_real_1: 0.2176110156824891 (-0.002623154348890544) | > avg_loss_disc_real_2: 0.2355497720795618 (-0.01152937418558228) | > avg_loss_disc_real_3: 0.2327309208856502 (-0.007029761730785095) | > avg_loss_disc_real_4: 0.2443219503046761 (+0.007803431489098234) | > avg_loss_disc_real_5: 0.24515122720893 (+0.0007560725363207621) | > avg_loss_0: 2.544164301643909 (-0.07442881355822895) | > avg_grad_norm_0: tensor(278.0863, device='cuda:0') (tensor(-37.9172, device='cuda:0')) | > avg_loss_gen: 2.184855795242417 (+0.0022043678122507515) | > avg_loss_kl: 1.9864684645558748 (-0.0481051307328999) | > avg_loss_feat: 3.9220558186651955 (-0.02175493979118226) | > avg_loss_mel: 16.411273566769882 (-0.31648333643523685) | > avg_loss_duration: 1.5373161846483256 (-0.005686670961514428) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.041969836597712 (-0.3898256597384595) | > avg_grad_norm_1: tensor(8589.2500, device='cuda:0') (+tensor(1161.1602, device='cuda:0'))  > EPOCH: 1121/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:45:52)   --> TIME: 2025-02-25 19:46:11 -- STEP: 37/72 -- GLOBAL_STEP: 1080750 | > loss_disc: 2.611172676086426 (2.54668437468039) | > loss_disc_real_0: 0.10030192136764526 (0.09586273627103986) | > loss_disc_real_1: 0.25209689140319824 (0.2201436315033887) | > loss_disc_real_2: 0.2387605607509613 (0.22652691723527135) | > loss_disc_real_3: 0.2299155592918396 (0.23706316907663602) | > loss_disc_real_4: 0.2498382180929184 (0.2416378307986904) | > loss_disc_real_5: 0.24416294693946838 (0.24468824952035337) | > loss_0: 2.611172676086426 (2.54668437468039) | > grad_norm_0: tensor(492.3345, device='cuda:0') (tensor(309.4458, device='cuda:0')) | > loss_gen: 2.1026298999786377 (2.220691619692622) | > loss_kl: 2.040365219116211 (2.000868568549285) | > loss_feat: 3.953535556793213 (4.101583042660275) | > loss_mel: 16.105331420898438 (16.715677544877337) | > loss_duration: 1.5530962944030762 (1.505566203916395) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.754959106445312 (26.544387044133366) | > grad_norm_1: tensor(12664.7930, device='cuda:0') (tensor(8344.8047, device='cuda:0')) | > current_lr_0: 0.00017384839181700724 | > current_lr_1: 0.00017384839181700724 | > step_time: 0.4439 (0.4389982545698011) | > loader_time: 0.0078 (0.0073240318813839475) --> EVAL PERFORMANCE | > avg_loader_time: 0.008648422402395328 (+0.0002922340178153869) | > avg_step_time: 0.45405543018394795 (+0.0005133319908464662) | > avg_loss_disc: 2.5484358189811167 (+0.004271517337207875) | > avg_loss_disc_real_0: 0.09512492872669662 (+0.004019543213743551) | > avg_loss_disc_real_1: 0.22132765029517698 (+0.003716634612687869) | > avg_loss_disc_real_2: 0.22751365539053797 (-0.00803611668902382) | > avg_loss_disc_real_3: 0.23571737656291103 (+0.0029864556772608297) | > avg_loss_disc_real_4: 0.2407686859369278 (-0.0035532643677483045) | > avg_loss_disc_real_5: 0.24406502112536363 (-0.0010862060835663823) | > avg_loss_0: 2.5484358189811167 (+0.004271517337207875) | > avg_grad_norm_0: tensor(299.3727, device='cuda:0') (+tensor(21.2863, device='cuda:0')) | > avg_loss_gen: 2.206625374270157 (+0.021769579027739905) | > avg_loss_kl: 2.03671024382954 (+0.05024177927366513) | > avg_loss_feat: 4.017203995879267 (+0.09514817721407187) | > avg_loss_mel: 16.6147182625784 (+0.20344469580851765) | > avg_loss_duration: 1.5397954289342317 (+0.002479244285906068) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.415053354182717 (+0.37308351758500535) | > avg_grad_norm_1: tensor(8316.8115, device='cuda:0') (tensor(-272.4385, device='cuda:0'))  > EPOCH: 1122/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:46:29)   --> TIME: 2025-02-25 19:46:38 -- STEP: 15/72 -- GLOBAL_STEP: 1080800 | > loss_disc: 2.5298871994018555 (2.4984565416971845) | > loss_disc_real_0: 0.1350812315940857 (0.10041505669554075) | > loss_disc_real_1: 0.20635783672332764 (0.2226388196150462) | > loss_disc_real_2: 0.2266997992992401 (0.23055100739002227) | > loss_disc_real_3: 0.20458044111728668 (0.23218510150909424) | > loss_disc_real_4: 0.2350763976573944 (0.24676953653494518) | > loss_disc_real_5: 0.25460466742515564 (0.250441179672877) | > loss_0: 2.5298871994018555 (2.4984565416971845) | > grad_norm_0: tensor(117.1047, device='cuda:0') (tensor(286.0452, device='cuda:0')) | > loss_gen: 2.264119863510132 (2.315651003519694) | > loss_kl: 2.0221896171569824 (1.935665241877238) | > loss_feat: 3.960805892944336 (4.198124488194783) | > loss_mel: 17.275714874267578 (16.7065216700236) | > loss_duration: 1.4809199571609497 (1.4840582211812336) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.003751754760742 (26.640020879109702) | > grad_norm_1: tensor(7186.4170, device='cuda:0') (tensor(7284.8237, device='cuda:0')) | > current_lr_0: 0.0001738266607680301 | > current_lr_1: 0.0001738266607680301 | > step_time: 0.4323 (0.4277054309844971) | > loader_time: 0.0058 (0.0063686370849609375)  --> TIME: 2025-02-25 19:47:01 -- STEP: 65/72 -- GLOBAL_STEP: 1080850 | > loss_disc: 2.5860610008239746 (2.596747845869798) | > loss_disc_real_0: 0.08290904760360718 (0.11850338348975549) | > loss_disc_real_1: 0.2218065857887268 (0.2308090967627672) | > loss_disc_real_2: 0.20754186809062958 (0.24158397683730493) | > loss_disc_real_3: 0.24588699638843536 (0.2351344945339056) | > loss_disc_real_4: 0.24210414290428162 (0.24965231808332308) | > loss_disc_real_5: 0.24460461735725403 (0.24577370056739217) | > loss_0: 2.5860610008239746 (2.596747845869798) | > grad_norm_0: tensor(53.4537, device='cuda:0') (tensor(242.7918, device='cuda:0')) | > loss_gen: 2.056516408920288 (2.232953948241014) | > loss_kl: 2.2434957027435303 (2.0813659539589513) | > loss_feat: 3.232194423675537 (3.9657031829540563) | > loss_mel: 16.79380226135254 (16.917332033010634) | > loss_duration: 1.6182559728622437 (1.541279856975262) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.944265365600586 (26.738634931124174) | > grad_norm_1: tensor(5169.3618, device='cuda:0') (tensor(5977.3457, device='cuda:0')) | > current_lr_0: 0.0001738266607680301 | > current_lr_1: 0.0001738266607680301 | > step_time: 0.5077 (0.444907782627986) | > loader_time: 0.0115 (0.00825781455406776) --> EVAL PERFORMANCE | > avg_loader_time: 0.008521429249938105 (-0.00012699315245722279) | > avg_step_time: 0.45158362052810025 (-0.002471809655847701) | > avg_loss_disc: 2.614289387850694 (+0.06585356886957738) | > avg_loss_disc_real_0: 0.11799522499803086 (+0.022870296271334245) | > avg_loss_disc_real_1: 0.22935626236066012 (+0.008028612065483137) | > avg_loss_disc_real_2: 0.2404804978991898 (+0.012966842508651821) | > avg_loss_disc_real_3: 0.23516861758601498 (-0.0005487589768960466) | > avg_loss_disc_real_4: 0.248361165464764 (+0.0075924795278362045) | > avg_loss_disc_real_5: 0.24500739994183393 (+0.0009423788164703018) | > avg_loss_0: 2.614289387850694 (+0.06585356886957738) | > avg_grad_norm_0: tensor(249.2230, device='cuda:0') (tensor(-50.1496, device='cuda:0')) | > avg_loss_gen: 2.198557615280151 (-0.008067758990006002) | > avg_loss_kl: 2.0932300241900164 (+0.056519780360476446) | > avg_loss_feat: 3.903205159684304 (-0.11399883619496354) | > avg_loss_mel: 16.91731927092647 (+0.3026010083480699) | > avg_loss_duration: 1.5476577483432394 (+0.007862319409007723) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.65996975965903 (+0.24491640547631377) | > avg_grad_norm_1: tensor(6138.3984, device='cuda:0') (tensor(-2178.4131, device='cuda:0'))  > EPOCH: 1123/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:47:06)   --> TIME: 2025-02-25 19:47:28 -- STEP: 43/72 -- GLOBAL_STEP: 1080900 | > loss_disc: 2.531102418899536 (2.7065534369890076) | > loss_disc_real_0: 0.19460950791835785 (0.1802879276923662) | > loss_disc_real_1: 0.1983756572008133 (0.23643217218476673) | > loss_disc_real_2: 0.22190281748771667 (0.24707735139270162) | > loss_disc_real_3: 0.23730416595935822 (0.24128413616224778) | > loss_disc_real_4: 0.21910996735095978 (0.2545726687409157) | > loss_disc_real_5: 0.236241415143013 (0.2503394352835278) | > loss_0: 2.531102418899536 (2.7065534369890076) | > grad_norm_0: tensor(136.9310, device='cuda:0') (tensor(281.8684, device='cuda:0')) | > loss_gen: 2.200968027114868 (2.2236809591914333) | > loss_kl: 2.2663638591766357 (2.166432530380958) | > loss_feat: 4.098750114440918 (3.857462439426156) | > loss_mel: 17.200946807861328 (17.21363934805227) | > loss_duration: 1.578200101852417 (1.538035703259845) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.34522819519043 (26.9992508999137) | > grad_norm_1: tensor(4081.9355, device='cuda:0') (tensor(6137.4766, device='cuda:0')) | > current_lr_0: 0.0001738049324354341 | > current_lr_1: 0.0001738049324354341 | > step_time: 0.4561 (0.4423168093659157) | > loader_time: 0.0099 (0.007636641347131064) --> EVAL PERFORMANCE | > avg_loader_time: 0.008857075597198916 (+0.0003356463472608106) | > avg_step_time: 0.4562495795773788 (+0.0046659590492785785) | > avg_loss_disc: 2.719293668236531 (+0.10500428038583687) | > avg_loss_disc_real_0: 0.1816485109411075 (+0.06365328594307665) | > avg_loss_disc_real_1: 0.22529284404197206 (-0.004063418318688061) | > avg_loss_disc_real_2: 0.23652874793804868 (-0.0039517499611411155) | > avg_loss_disc_real_3: 0.24043535766467242 (+0.005266740078657434) | > avg_loss_disc_real_4: 0.24746555299826076 (-0.0008956124665032428) | > avg_loss_disc_real_5: 0.2508009225130081 (+0.005793522571174187) | > avg_loss_0: 2.719293668236531 (+0.10500428038583687) | > avg_grad_norm_0: tensor(245.6242, device='cuda:0') (tensor(-3.5988, device='cuda:0')) | > avg_loss_gen: 2.117270983440776 (-0.08128663183937501) | > avg_loss_kl: 2.205627998835604 (+0.11239797464558743) | > avg_loss_feat: 3.6024320091999753 (-0.3007731504843285) | > avg_loss_mel: 17.244072363410204 (+0.3267530924837345) | > avg_loss_duration: 1.5617717554871464 (+0.014114007143906981) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.731175221187968 (+0.07120546152893681) | > avg_grad_norm_1: tensor(6351.6929, device='cuda:0') (+tensor(213.2944, device='cuda:0'))  > EPOCH: 1124/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:47:43)   --> TIME: 2025-02-25 19:47:54 -- STEP: 21/72 -- GLOBAL_STEP: 1080950 | > loss_disc: 2.7605020999908447 (2.7347441627865745) | > loss_disc_real_0: 0.16391238570213318 (0.20740032187175184) | > loss_disc_real_1: 0.2571616470813751 (0.22354469696680704) | > loss_disc_real_2: 0.23005901277065277 (0.24441230651878176) | > loss_disc_real_3: 0.2380746454000473 (0.22991246694610232) | > loss_disc_real_4: 0.26277685165405273 (0.24449920086633592) | > loss_disc_real_5: 0.23912248015403748 (0.24835404753684998) | > loss_0: 2.7605020999908447 (2.7347441627865745) | > grad_norm_0: tensor(210.0085, device='cuda:0') (tensor(234.0761, device='cuda:0')) | > loss_gen: 2.0759353637695312 (2.014319487980434) | > loss_kl: 2.177046537399292 (2.0397124858129594) | > loss_feat: 3.731614589691162 (3.4796772457304455) | > loss_mel: 17.648269653320312 (17.060426439557755) | > loss_duration: 1.5494771003723145 (1.5113444441840762) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.182342529296875 (26.105480194091797) | > grad_norm_1: tensor(6710.4004, device='cuda:0') (tensor(6963.3320, device='cuda:0')) | > current_lr_0: 0.00017378320681887968 | > current_lr_1: 0.00017378320681887968 | > step_time: 0.4391 (0.4335771742321196) | > loader_time: 0.0064 (0.006618794940766834)  --> TIME: 2025-02-25 19:48:18 -- STEP: 71/72 -- GLOBAL_STEP: 1081000 | > loss_disc: 2.787755250930786 (2.7100411569568474) | > loss_disc_real_0: 0.08389781415462494 (0.14647199084956042) | > loss_disc_real_1: 0.3226063549518585 (0.237491529592326) | > loss_disc_real_2: 0.19413715600967407 (0.2324093368271707) | > loss_disc_real_3: 0.16790902614593506 (0.2300626732094187) | > loss_disc_real_4: 0.1838550865650177 (0.24281584481957932) | > loss_disc_real_5: 0.16149266064167023 (0.241889303838703) | > loss_0: 2.787755250930786 (2.7100411569568474) | > grad_norm_0: tensor(227.8253, device='cuda:0') (tensor(205.0961, device='cuda:0')) | > loss_gen: 1.9021117687225342 (2.005319612126955) | > loss_kl: 2.922382354736328 (2.476146031433428) | > loss_feat: 4.639411926269531 (3.6167283360387237) | > loss_mel: 24.458572387695312 (19.546647595687663) | > loss_duration: 1.902430534362793 (1.7036517791345085) | > amp_scaler: 8.0 (8.0) | > loss_1: 35.82490921020508 (29.348493441729477) | > grad_norm_1: tensor(2538.3962, device='cuda:0') (tensor(7046.8179, device='cuda:0')) | > current_lr_0: 0.00017378320681887968 | > current_lr_1: 0.00017378320681887968 | > step_time: 0.506 (0.4510136288656315) | > loader_time: 0.0081 (0.008486902210074412) --> EVAL PERFORMANCE | > avg_loader_time: 0.008486902210074412 (-0.000370173387124504) | > avg_step_time: 0.4510136288656315 (-0.005235950711747317) | > avg_loss_disc: 2.7100411569568474 (-0.009252511279683517) | > avg_loss_disc_real_0: 0.14647199084956042 (-0.03517652009154709) | > avg_loss_disc_real_1: 0.237491529592326 (+0.012198685550353944) | > avg_loss_disc_real_2: 0.2324093368271707 (-0.004119411110877991) | > avg_loss_disc_real_3: 0.2300626732094187 (-0.010372684455253717) | > avg_loss_disc_real_4: 0.24281584481957932 (-0.004649708178681433) | > avg_loss_disc_real_5: 0.241889303838703 (-0.008911618674305122) | > avg_loss_0: 2.7100411569568474 (-0.009252511279683517) | > avg_grad_norm_0: tensor(205.0961, device='cuda:0') (tensor(-40.5281, device='cuda:0')) | > avg_loss_gen: 2.005319612126955 (-0.11195137131382094) | > avg_loss_kl: 2.476146031433428 (+0.27051803259782403) | > avg_loss_feat: 3.6167283360387237 (+0.014296326838748374) | > avg_loss_mel: 19.546647595687663 (+2.3025752322774586) | > avg_loss_duration: 1.7036517791345085 (+0.14188002364736207) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.348493441729477 (+2.6173182205415095) | > avg_grad_norm_1: tensor(7046.8179, device='cuda:0') (+tensor(695.1250, device='cuda:0'))  > EPOCH: 1125/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:48:20)   --> TIME: 2025-02-25 19:48:44 -- STEP: 49/72 -- GLOBAL_STEP: 1081050 | > loss_disc: 2.5105721950531006 (2.5757499081747874) | > loss_disc_real_0: 0.07122023403644562 (0.09812302096765868) | > loss_disc_real_1: 0.21423141658306122 (0.22961823429380143) | > loss_disc_real_2: 0.22775138914585114 (0.24003251474730822) | > loss_disc_real_3: 0.23864568769931793 (0.231031884648362) | > loss_disc_real_4: 0.24513134360313416 (0.24140715142902064) | > loss_disc_real_5: 0.2451472282409668 (0.2487969933723917) | > loss_0: 2.5105721950531006 (2.5757499081747874) | > grad_norm_0: tensor(277.4413, device='cuda:0') (tensor(244.9836, device='cuda:0')) | > loss_gen: 2.037172317504883 (2.133327479265175) | > loss_kl: 2.0309267044067383 (2.231583429842579) | > loss_feat: 3.5733678340911865 (3.7699153569279886) | > loss_mel: 16.704925537109375 (18.210923292198963) | > loss_duration: 1.6697252988815308 (1.638805934361049) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.016117095947266 (27.984555653163365) | > grad_norm_1: tensor(12809.6055, device='cuda:0') (tensor(8153.5073, device='cuda:0')) | > current_lr_0: 0.00017376148391802732 | > current_lr_1: 0.00017376148391802732 | > step_time: 0.4357 (0.43725552850840044) | > loader_time: 0.0094 (0.007627336346373266) --> EVAL PERFORMANCE | > avg_loader_time: 0.008539042002718216 (+5.213979264380442e-05) | > avg_step_time: 0.44975410716634395 (-0.001259521699287558) | > avg_loss_disc: 2.572349612141999 (-0.13769154481484858) | > avg_loss_disc_real_0: 0.09929417026504664 (-0.04717782058451378) | > avg_loss_disc_real_1: 0.225436323755224 (-0.012055205837102012) | > avg_loss_disc_real_2: 0.23912536615217236 (+0.006716029325001677) | > avg_loss_disc_real_3: 0.2301708117757045 (+0.00010813856628580365) | > avg_loss_disc_real_4: 0.24197762763836014 (-0.0008382171812191808) | > avg_loss_disc_real_5: 0.24892123631188567 (+0.007031932473182678) | > avg_loss_0: 2.572349612141999 (-0.13769154481484858) | > avg_grad_norm_0: tensor(265.8376, device='cuda:0') (+tensor(60.7415, device='cuda:0')) | > avg_loss_gen: 2.1355150837293815 (+0.1301954716024265) | > avg_loss_kl: 2.2263210189174596 (-0.24982501251596823) | > avg_loss_feat: 3.7573223214753915 (+0.1405939854366678) | > avg_loss_mel: 17.862962857098655 (-1.6836847385890081) | > avg_loss_duration: 1.6364299680145693 (-0.06722181111993919) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.618551227408396 (-1.7299422143210812) | > avg_grad_norm_1: tensor(8431.6279, device='cuda:0') (+tensor(1384.8101, device='cuda:0'))  > EPOCH: 1126/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:48:57)   --> TIME: 2025-02-25 19:49:11 -- STEP: 27/72 -- GLOBAL_STEP: 1081100 | > loss_disc: 2.633563995361328 (2.614631688153302) | > loss_disc_real_0: 0.11086557805538177 (0.10087564594491764) | > loss_disc_real_1: 0.22286875545978546 (0.21715496535654422) | > loss_disc_real_2: 0.19384096562862396 (0.21705440238670068) | > loss_disc_real_3: 0.21115227043628693 (0.22063637718006415) | > loss_disc_real_4: 0.2220497578382492 (0.2478001829650667) | > loss_disc_real_5: 0.22158129513263702 (0.2471701701482137) | > loss_0: 2.633563995361328 (2.614631688153302) | > grad_norm_0: tensor(228.0989, device='cuda:0') (tensor(262.1826, device='cuda:0')) | > loss_gen: 1.9437326192855835 (2.068746050198873) | > loss_kl: 1.9433066844940186 (1.976401483571088) | > loss_feat: 3.799919605255127 (3.9325523376464844) | > loss_mel: 16.928369522094727 (17.126080195109044) | > loss_duration: 1.5340149402618408 (1.5381576308497675) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.14934539794922 (26.64193817421242) | > grad_norm_1: tensor(14405.2158, device='cuda:0') (tensor(8890.9385, device='cuda:0')) | > current_lr_0: 0.00017373976373253756 | > current_lr_1: 0.00017373976373253756 | > step_time: 0.4344 (0.43206628163655597) | > loader_time: 0.0086 (0.006955508832578306) --> EVAL PERFORMANCE | > avg_loader_time: 0.008527020333518445 (-1.2021669199771198e-05) | > avg_step_time: 0.44931752580991935 (-0.0004365813564246035) | > avg_loss_disc: 2.6293550041359914 (+0.05700539199399257) | > avg_loss_disc_real_0: 0.13612504190647265 (+0.03683087164142601) | > avg_loss_disc_real_1: 0.21807049456196773 (-0.0073658291932562625) | > avg_loss_disc_real_2: 0.2282629069727911 (-0.01086245917938125) | > avg_loss_disc_real_3: 0.22798283775927317 (-0.0021879740164313377) | > avg_loss_disc_real_4: 0.2436602424148103 (+0.0016826147764501531) | > avg_loss_disc_real_5: 0.24494886251402573 (-0.003972373797859946) | > avg_loss_0: 2.6293550041359914 (+0.05700539199399257) | > avg_grad_norm_0: tensor(246.5882, device='cuda:0') (tensor(-19.2495, device='cuda:0')) | > avg_loss_gen: 2.0646522162665786 (-0.0708628674628029) | > avg_loss_kl: 2.04526414501835 (-0.1810568738991094) | > avg_loss_feat: 3.6759164199023178 (-0.0814059015730737) | > avg_loss_mel: 16.85142484852965 (-1.0115380085690049) | > avg_loss_duration: 1.5677434692920094 (-0.06868649872255994) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.205001401229644 (-1.4135498261787518) | > avg_grad_norm_1: tensor(7467.0142, device='cuda:0') (tensor(-964.6138, device='cuda:0'))  > EPOCH: 1127/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:49:33)   --> TIME: 2025-02-25 19:49:37 -- STEP: 5/72 -- GLOBAL_STEP: 1081150 | > loss_disc: 2.9185256958007812 (2.6546514987945558) | > loss_disc_real_0: 0.054341234266757965 (0.03660759702324867) | > loss_disc_real_1: 0.2749210298061371 (0.2516871809959412) | > loss_disc_real_2: 0.3499160706996918 (0.29587186872959137) | > loss_disc_real_3: 0.2412044107913971 (0.21352846920490265) | > loss_disc_real_4: 0.25113385915756226 (0.23919176757335664) | > loss_disc_real_5: 0.25988081097602844 (0.24200095236301422) | > loss_0: 2.9185256958007812 (2.6546514987945558) | > grad_norm_0: tensor(571.2968, device='cuda:0') (tensor(259.3326, device='cuda:0')) | > loss_gen: 1.923668384552002 (2.060603475570679) | > loss_kl: 1.8734991550445557 (1.9225091457366943) | > loss_feat: 3.4737021923065186 (4.016137981414795) | > loss_mel: 16.865257263183594 (16.831700134277344) | > loss_duration: 1.4844822883605957 (1.5085273265838623) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.620609283447266 (26.339477920532225) | > grad_norm_1: tensor(4347.1504, device='cuda:0') (tensor(5345.8071, device='cuda:0')) | > current_lr_0: 0.000173718046262071 | > current_lr_1: 0.000173718046262071 | > step_time: 0.4202 (0.4225046634674072) | > loader_time: 0.0066 (0.00601491928100586)  --> TIME: 2025-02-25 19:50:00 -- STEP: 55/72 -- GLOBAL_STEP: 1081200 | > loss_disc: 2.3798389434814453 (2.6843213558197023) | > loss_disc_real_0: 0.07139363884925842 (0.1396946858953346) | > loss_disc_real_1: 0.21304164826869965 (0.2385759483684193) | > loss_disc_real_2: 0.21616986393928528 (0.24050145528533243) | > loss_disc_real_3: 0.20484264194965363 (0.22962936135855588) | > loss_disc_real_4: 0.23831768333911896 (0.2467309293421832) | > loss_disc_real_5: 0.2427462488412857 (0.2473008245229721) | > loss_0: 2.3798389434814453 (2.6843213558197023) | > grad_norm_0: tensor(106.5053, device='cuda:0') (tensor(254.6452, device='cuda:0')) | > loss_gen: 2.4184868335723877 (2.1226539785211735) | > loss_kl: 2.2959277629852295 (2.407298883524808) | > loss_feat: 4.234015941619873 (3.7827102531086316) | > loss_mel: 19.789430618286133 (18.42648282484575) | > loss_duration: 1.6481449604034424 (1.5766065380790018) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.38600730895996 (28.31575251492587) | > grad_norm_1: tensor(6954.5093, device='cuda:0') (tensor(7464.6050, device='cuda:0')) | > current_lr_0: 0.000173718046262071 | > current_lr_1: 0.000173718046262071 | > step_time: 0.4554 (0.43831983479586517) | > loader_time: 0.0092 (0.0076651269739324394) --> EVAL PERFORMANCE | > avg_loader_time: 0.00838389195186991 (-0.000143128381648535) | > avg_step_time: 0.4508813367763036 (+0.0015638109663842492) | > avg_loss_disc: 2.6417878278544253 (+0.012432823718433905) | > avg_loss_disc_real_0: 0.12875031800547115 (-0.007374723901001495) | > avg_loss_disc_real_1: 0.23427037432999678 (+0.016199879768029057) | > avg_loss_disc_real_2: 0.23945966425915838 (+0.011196757286367265) | > avg_loss_disc_real_3: 0.23003569770027216 (+0.002052859940998991) | > avg_loss_disc_real_4: 0.24383214766710576 (+0.00017190525229546227) | > avg_loss_disc_real_5: 0.24635331815397235 (+0.0014044556399466224) | > avg_loss_0: 2.6417878278544253 (+0.012432823718433905) | > avg_grad_norm_0: tensor(244.7237, device='cuda:0') (tensor(-1.8644, device='cuda:0')) | > avg_loss_gen: 2.1537663063532864 (+0.08911409008670779) | > avg_loss_kl: 2.366406007551811 (+0.321141862533461) | > avg_loss_feat: 3.797917772346819 (+0.12200135244450117) | > avg_loss_mel: 18.2619660605847 (+1.4105412120550511) | > avg_loss_duration: 1.5851463918954554 (+0.017402922603446003) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.165202664657375 (+1.9602012634277308) | > avg_grad_norm_1: tensor(7101.1030, device='cuda:0') (tensor(-365.9111, device='cuda:0'))  > EPOCH: 1128/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:50:10)   --> TIME: 2025-02-25 19:50:26 -- STEP: 33/72 -- GLOBAL_STEP: 1081250 | > loss_disc: 2.597219705581665 (2.5090002435626406) | > loss_disc_real_0: 0.11536078155040741 (0.08732003741192096) | > loss_disc_real_1: 0.1986747831106186 (0.2262358660950805) | > loss_disc_real_2: 0.26719656586647034 (0.23265637637990894) | > loss_disc_real_3: 0.24911081790924072 (0.2307766436627417) | > loss_disc_real_4: 0.2669614553451538 (0.24026648474462103) | > loss_disc_real_5: 0.26350533962249756 (0.24512216087543603) | > loss_0: 2.597219705581665 (2.5090002435626406) | > grad_norm_0: tensor(91.9708, device='cuda:0') (tensor(219.2141, device='cuda:0')) | > loss_gen: 2.3471450805664062 (2.248200777805213) | > loss_kl: 2.130485773086548 (1.9784170678167632) | > loss_feat: 3.471670627593994 (3.943417014497699) | > loss_mel: 16.939777374267578 (17.007443283543445) | > loss_duration: 1.5196478366851807 (1.5140944863810684) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.408727645874023 (26.691572882912375) | > grad_norm_1: tensor(5963.7310, device='cuda:0') (tensor(6677.0449, device='cuda:0')) | > current_lr_0: 0.00017369633150628823 | > current_lr_1: 0.00017369633150628823 | > step_time: 0.436 (0.4327754829869126) | > loader_time: 0.0085 (0.0070676008860270185) --> EVAL PERFORMANCE | > avg_loader_time: 0.00856483150535906 (+0.0001809395534891501) | > avg_step_time: 0.450136876442063 (-0.000744460334240582) | > avg_loss_disc: 2.530374647865832 (-0.11141317998859313) | > avg_loss_disc_real_0: 0.09538480844086325 (-0.033365509564607904) | > avg_loss_disc_real_1: 0.22504263318760295 (-0.00922774114239383) | > avg_loss_disc_real_2: 0.2190943715018286 (-0.02036529275732979) | > avg_loss_disc_real_3: 0.2341356329934698 (+0.004099935293197632) | > avg_loss_disc_real_4: 0.24060930626493104 (-0.0032228414021747143) | > avg_loss_disc_real_5: 0.24070633884886622 (-0.005646979305106131) | > avg_loss_0: 2.530374647865832 (-0.11141317998859313) | > avg_grad_norm_0: tensor(234.8511, device='cuda:0') (tensor(-9.8726, device='cuda:0')) | > avg_loss_gen: 2.2242411778006757 (+0.07047487144738929) | > avg_loss_kl: 2.0101188992110774 (-0.35628710834073374) | > avg_loss_feat: 3.985888830372985 (+0.1879710580261662) | > avg_loss_mel: 16.870420509660754 (-1.3915455509239472) | > avg_loss_duration: 1.548910182966313 (-0.036236208929142366) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.63957988040548 (-1.525622784251894) | > avg_grad_norm_1: tensor(6827.0420, device='cuda:0') (tensor(-274.0610, device='cuda:0'))  > EPOCH: 1129/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:50:46)   --> TIME: 2025-02-25 19:50:53 -- STEP: 11/72 -- GLOBAL_STEP: 1081300 | > loss_disc: 2.6734976768493652 (2.616817431016402) | > loss_disc_real_0: 0.04688407480716705 (0.09987518936395645) | > loss_disc_real_1: 0.22917018830776215 (0.2301612835038792) | > loss_disc_real_2: 0.2250102013349533 (0.2422319772568616) | > loss_disc_real_3: 0.23973383009433746 (0.23830077187581497) | > loss_disc_real_4: 0.23364296555519104 (0.24751731211488898) | > loss_disc_real_5: 0.24911005795001984 (0.2310808842832392) | > loss_0: 2.6734976768493652 (2.616817431016402) | > grad_norm_0: tensor(200.3685, device='cuda:0') (tensor(233.0669, device='cuda:0')) | > loss_gen: 2.005920886993408 (2.198969461701133) | > loss_kl: 1.875451683998108 (1.8616053949702869) | > loss_feat: 3.5770390033721924 (3.9717304706573486) | > loss_mel: 17.518165588378906 (16.95231801813299) | > loss_duration: 1.4911912679672241 (1.499823581088673) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.467769622802734 (26.484446785666726) | > grad_norm_1: tensor(5970.8027, device='cuda:0') (tensor(6914.0962, device='cuda:0')) | > current_lr_0: 0.00017367461946484993 | > current_lr_1: 0.00017367461946484993 | > step_time: 0.4494 (0.4314854795282537) | > loader_time: 0.0079 (0.006713368675925515)  --> TIME: 2025-02-25 19:51:16 -- STEP: 61/72 -- GLOBAL_STEP: 1081350 | > loss_disc: 2.6175591945648193 (2.67044222550314) | > loss_disc_real_0: 0.14150488376617432 (0.1559976044370502) | > loss_disc_real_1: 0.26723289489746094 (0.2387767541603964) | > loss_disc_real_2: 0.24703927338123322 (0.2338962965324277) | > loss_disc_real_3: 0.21900752186775208 (0.2345955051848146) | > loss_disc_real_4: 0.24921174347400665 (0.23827583941279865) | > loss_disc_real_5: 0.24624493718147278 (0.23936091290145625) | > loss_0: 2.6175591945648193 (2.67044222550314) | > grad_norm_0: tensor(56.0985, device='cuda:0') (tensor(280.8017, device='cuda:0')) | > loss_gen: 2.237299680709839 (2.1665974562285375) | > loss_kl: 2.034313440322876 (1.9537877821531453) | > loss_feat: 3.127286672592163 (3.7999220793364477) | > loss_mel: 16.749422073364258 (16.754050395527827) | > loss_duration: 1.5946855545043945 (1.5331619293963323) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.74300765991211 (26.20751959378602) | > grad_norm_1: tensor(4658.4683, device='cuda:0') (tensor(5699.4468, device='cuda:0')) | > current_lr_0: 0.00017367461946484993 | > current_lr_1: 0.00017367461946484993 | > step_time: 0.4652 (0.44178268948539356) | > loader_time: 0.0105 (0.008094232590472112) --> EVAL PERFORMANCE | > avg_loader_time: 0.008555811895451076 (-9.019609907984227e-06) | > avg_step_time: 0.4491823894876829 (-0.0009544869543801071) | > avg_loss_disc: 2.686436770667493 (+0.15606212280166076) | > avg_loss_disc_real_0: 0.1678296183301529 (+0.07244480988928964) | > avg_loss_disc_real_1: 0.23999267081979295 (+0.014950037632189994) | > avg_loss_disc_real_2: 0.23455233011447207 (+0.015457958612643485) | > avg_loss_disc_real_3: 0.2342857340691795 (+0.0001501010757097021) | > avg_loss_disc_real_4: 0.238690992476235 (-0.0019183137886960577) | > avg_loss_disc_real_5: 0.23993128748007223 (-0.0007750513687939942) | > avg_loss_0: 2.686436770667493 (+0.15606212280166076) | > avg_grad_norm_0: tensor(278.8428, device='cuda:0') (+tensor(43.9917, device='cuda:0')) | > avg_loss_gen: 2.1279797554016113 (-0.09626142239906432) | > avg_loss_kl: 2.127668664488994 (+0.1175497652779165) | > avg_loss_feat: 3.7255418770749804 (-0.2603469532980047) | > avg_loss_mel: 17.73277554041903 (+0.8623550307582768) | > avg_loss_duration: 1.5628633566305672 (+0.013953173664254193) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.27682922255825 (+0.6372493421527672) | > avg_grad_norm_1: tensor(5399.2515, device='cuda:0') (tensor(-1427.7905, device='cuda:0'))  > EPOCH: 1130/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:51:23)   --> TIME: 2025-02-25 19:51:43 -- STEP: 39/72 -- GLOBAL_STEP: 1081400 | > loss_disc: 2.6274867057800293 (2.5363693176171718) | > loss_disc_real_0: 0.26806479692459106 (0.12712271435138506) | > loss_disc_real_1: 0.26662707328796387 (0.2180274757437217) | > loss_disc_real_2: 0.2364700436592102 (0.21513759707793206) | > loss_disc_real_3: 0.22870691120624542 (0.23022588839133581) | > loss_disc_real_4: 0.24185708165168762 (0.22718365719685188) | > loss_disc_real_5: 0.25886014103889465 (0.236975168188413) | > loss_0: 2.6274867057800293 (2.5363693176171718) | > grad_norm_0: tensor(620.9221, device='cuda:0') (tensor(233.2529, device='cuda:0')) | > loss_gen: 2.351503849029541 (2.2140226119603867) | > loss_kl: 2.3036625385284424 (2.541873919658172) | > loss_feat: 3.7822868824005127 (4.075911161227105) | > loss_mel: 18.339447021484375 (19.839318837874973) | > loss_duration: 1.5676875114440918 (1.577499514971024) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.344587326049805 (30.248626122107872) | > grad_norm_1: tensor(6398.0562, device='cuda:0') (tensor(6052.9058, device='cuda:0')) | > current_lr_0: 0.00017365291013741682 | > current_lr_1: 0.00017365291013741682 | > step_time: 0.4499 (0.4332089974330022) | > loader_time: 0.008 (0.007136485515496669) --> EVAL PERFORMANCE | > avg_loader_time: 0.008406095101799765 (-0.00014971679365131073) | > avg_step_time: 0.4492643786148286 (+8.19891271456874e-05) | > avg_loss_disc: 2.5996393015686894 (-0.08679746909880359) | > avg_loss_disc_real_0: 0.14817186156418963 (-0.019657756765963263) | > avg_loss_disc_real_1: 0.2248821681444074 (-0.015110502675385551) | > avg_loss_disc_real_2: 0.2267705896790598 (-0.007781740435412271) | > avg_loss_disc_real_3: 0.23844421361113938 (+0.004158479541959886) | > avg_loss_disc_real_4: 0.2345671187823927 (-0.004123873693842289) | > avg_loss_disc_real_5: 0.2406191884631842 (+0.000687900983111972) | > avg_loss_0: 2.5996393015686894 (-0.08679746909880359) | > avg_grad_norm_0: tensor(251.8346, device='cuda:0') (tensor(-27.0082, device='cuda:0')) | > avg_loss_gen: 2.1682699146405073 (+0.040290159238896006) | > avg_loss_kl: 2.3761567095635634 (+0.24848804507456945) | > avg_loss_feat: 3.863146835649518 (+0.13760495857453758) | > avg_loss_mel: 18.80864476485991 (+1.0758692244408792) | > avg_loss_duration: 1.5884630428233617 (+0.025599686192794513) | > avg_amp_scaler: 8.56338028169014 (+0.5633802816901401) | > avg_loss_1: 28.804681294400925 (+1.527852071842677) | > avg_grad_norm_1: tensor(6636.0635, device='cuda:0') (+tensor(1236.8120, device='cuda:0'))  > EPOCH: 1131/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:52:00)   --> TIME: 2025-02-25 19:52:09 -- STEP: 17/72 -- GLOBAL_STEP: 1081450 | > loss_disc: 2.6809215545654297 (2.6000382479499367) | > loss_disc_real_0: 0.16207867860794067 (0.1096989596810411) | > loss_disc_real_1: 0.1906924843788147 (0.21869615596883438) | > loss_disc_real_2: 0.29752394556999207 (0.2331766994560466) | > loss_disc_real_3: 0.2339170277118683 (0.23176431918845458) | > loss_disc_real_4: 0.24640750885009766 (0.24193818604244904) | > loss_disc_real_5: 0.24949340522289276 (0.2528083552332485) | > loss_0: 2.6809215545654297 (2.6000382479499367) | > grad_norm_0: tensor(769.7840, device='cuda:0') (tensor(346.5166, device='cuda:0')) | > loss_gen: 2.104877471923828 (2.0959386825561523) | > loss_kl: 2.161940097808838 (1.9707781707539278) | > loss_feat: 3.813626289367676 (3.9108357149011947) | > loss_mel: 17.611896514892578 (17.37525687498205) | > loss_duration: 1.5527734756469727 (1.5245430329266716) | > amp_scaler: 8.0 (9.882352941176471) | > loss_1: 27.245113372802734 (26.877352434046127) | > grad_norm_1: tensor(9843.5098, device='cuda:0') (tensor(8242.4980, device='cuda:0')) | > current_lr_0: 0.00017363120352364964 | > current_lr_1: 0.00017363120352364964 | > step_time: 0.4278 (0.4303429407231948) | > loader_time: 0.0091 (0.006641233668607824)  --> TIME: 2025-02-25 19:52:33 -- STEP: 67/72 -- GLOBAL_STEP: 1081500 | > loss_disc: 2.890692710876465 (2.6142507987235906) | > loss_disc_real_0: 0.04116135463118553 (0.1269244906395229) | > loss_disc_real_1: 0.15018883347511292 (0.21467477950587202) | > loss_disc_real_2: 0.2540385127067566 (0.23659477305056445) | > loss_disc_real_3: 0.18327811360359192 (0.23050988832516456) | > loss_disc_real_4: 0.25810521841049194 (0.24143526291669304) | > loss_disc_real_5: 0.21361762285232544 (0.2459038522706103) | > loss_0: 2.890692710876465 (2.6142507987235906) | > grad_norm_0: tensor(368.7017, device='cuda:0') (tensor(330.8344, device='cuda:0')) | > loss_gen: 1.8101857900619507 (2.1324057828134566) | > loss_kl: 2.0451855659484863 (1.9956636677927047) | > loss_feat: 4.023192405700684 (3.8872163082236675) | > loss_mel: 17.006635665893555 (17.077138815353162) | > loss_duration: 1.6314828395843506 (1.552152283156096) | > amp_scaler: 8.0 (8.477611940298505) | > loss_1: 26.51668357849121 (26.644576827091957) | > grad_norm_1: tensor(17793.7520, device='cuda:0') (tensor(8200.7744, device='cuda:0')) | > current_lr_0: 0.00017363120352364964 | > current_lr_1: 0.00017363120352364964 | > step_time: 0.5141 (0.44924572688430103) | > loader_time: 0.012 (0.00830349993349901) --> EVAL PERFORMANCE | > avg_loader_time: 0.00848128426242882 (+7.518916062905501e-05) | > avg_step_time: 0.45304046886067995 (+0.003776090245851349) | > avg_loss_disc: 2.616244685481971 (+0.016605383913281813) | > avg_loss_disc_real_0: 0.12291223545309526 (-0.025259626111094363) | > avg_loss_disc_real_1: 0.21591171643263857 (-0.008970451711768829) | > avg_loss_disc_real_2: 0.23551708669729635 (+0.008746497018236543) | > avg_loss_disc_real_3: 0.22840070514611796 (-0.010043508465021417) | > avg_loss_disc_real_4: 0.2409975224397552 (+0.006430403657362499) | > avg_loss_disc_real_5: 0.2436054462278393 (+0.002986257764655109) | > avg_loss_0: 2.616244685481971 (+0.016605383913281813) | > avg_grad_norm_0: tensor(332.6116, device='cuda:0') (+tensor(80.7770, device='cuda:0')) | > avg_loss_gen: 2.120456339607776 (-0.047813575032731315) | > avg_loss_kl: 2.0033884014881833 (-0.3727683080753801) | > avg_loss_feat: 3.883436481717607 (+0.020289646068088807) | > avg_loss_mel: 17.093157459312756 (-1.7154873055471533) | > avg_loss_duration: 1.5549388301204627 (-0.03352421270289896) | > avg_amp_scaler: 8.45070422535211 (-0.1126760563380298) | > avg_loss_1: 26.655377482024715 (-2.1493038123762105) | > avg_grad_norm_1: tensor(8527.4697, device='cuda:0') (+tensor(1891.4062, device='cuda:0'))  > EPOCH: 1132/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:52:37)   --> TIME: 2025-02-25 19:52:59 -- STEP: 45/72 -- GLOBAL_STEP: 1081550 | > loss_disc: 2.636798620223999 (2.6743384255303275) | > loss_disc_real_0: 0.17809772491455078 (0.194882539825307) | > loss_disc_real_1: 0.20797786116600037 (0.22444575130939484) | > loss_disc_real_2: 0.22724421322345734 (0.23834423836734558) | > loss_disc_real_3: 0.23397816717624664 (0.24370085034105513) | > loss_disc_real_4: 0.2348429560661316 (0.24914689229594328) | > loss_disc_real_5: 0.2542187571525574 (0.2514058099852668) | > loss_0: 2.636798620223999 (2.6743384255303275) | > grad_norm_0: tensor(485.0012, device='cuda:0') (tensor(228.5962, device='cuda:0')) | > loss_gen: 1.8892340660095215 (2.2059519900216) | > loss_kl: 2.0552473068237305 (1.9463719235526191) | > loss_feat: 3.023861885070801 (3.7383104059431296) | > loss_mel: 16.966638565063477 (16.974053128560385) | > loss_duration: 1.5611547231674194 (1.5273466534084743) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.496137619018555 (26.392034106784397) | > grad_norm_1: tensor(4846.6982, device='cuda:0') (tensor(6152.8599, device='cuda:0')) | > current_lr_0: 0.00017360949962320918 | > current_lr_1: 0.00017360949962320918 | > step_time: 0.449 (0.43556100527445474) | > loader_time: 0.0089 (0.007525142033894857) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863891252329652 (+0.0001576282608676989) | > avg_step_time: 0.4505973533845284 (-0.0024431154761515583) | > avg_loss_disc: 2.6874418292247073 (+0.07119714374273611) | > avg_loss_disc_real_0: 0.17943481833372318 (+0.05652258288062792) | > avg_loss_disc_real_1: 0.22502987265167101 (+0.009118156219032447) | > avg_loss_disc_real_2: 0.23695566694081668 (+0.0014385802435203376) | > avg_loss_disc_real_3: 0.23832032168415232 (+0.00991961653803436) | > avg_loss_disc_real_4: 0.24590617754089994 (+0.004908655101144743) | > avg_loss_disc_real_5: 0.24830191739847957 (+0.0046964711706402595) | > avg_loss_0: 2.6874418292247073 (+0.07119714374273611) | > avg_grad_norm_0: tensor(245.1842, device='cuda:0') (tensor(-87.4273, device='cuda:0')) | > avg_loss_gen: 2.137691640518081 (+0.017235300910305007) | > avg_loss_kl: 1.9871113065262915 (-0.01627709496189178) | > avg_loss_feat: 3.6345025082709084 (-0.24893397344669843) | > avg_loss_mel: 16.836151660328188 (-0.25700579898456866) | > avg_loss_duration: 1.5495838900686987 (-0.005354940051764068) | > avg_amp_scaler: 8.0 (-0.4507042253521103) | > avg_loss_1: 26.1450410225022 (-0.5103364595225166) | > avg_grad_norm_1: tensor(6700.5630, device='cuda:0') (tensor(-1826.9067, device='cuda:0'))  > EPOCH: 1133/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:53:13)   --> TIME: 2025-02-25 19:53:26 -- STEP: 23/72 -- GLOBAL_STEP: 1081600 | > loss_disc: 2.6638264656066895 (2.636598068734874) | > loss_disc_real_0: 0.20275381207466125 (0.16501149535179138) | > loss_disc_real_1: 0.2640811502933502 (0.19779924659625345) | > loss_disc_real_2: 0.24571557343006134 (0.23647244015465613) | > loss_disc_real_3: 0.225962296128273 (0.22789054720298105) | > loss_disc_real_4: 0.2827809751033783 (0.24078532211158588) | > loss_disc_real_5: 0.2684652507305145 (0.2375134799791419) | > loss_0: 2.6638264656066895 (2.636598068734874) | > grad_norm_0: tensor(226.9252, device='cuda:0') (tensor(254.0728, device='cuda:0')) | > loss_gen: 2.257902145385742 (2.1025594161904375) | > loss_kl: 1.7944854497909546 (1.950010491454083) | > loss_feat: 2.904982089996338 (3.5931669629138447) | > loss_mel: 15.336992263793945 (16.67322581747304) | > loss_duration: 1.5251471996307373 (1.5009475739105889) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.819509506225586 (25.819910547007684) | > grad_norm_1: tensor(2709.5273, device='cuda:0') (tensor(5663.3438, device='cuda:0')) | > current_lr_0: 0.00017358779843575626 | > current_lr_1: 0.00017358779843575626 | > step_time: 0.426 (0.43094349944073224) | > loader_time: 0.0063 (0.006760846013608186) --> EVAL PERFORMANCE | > avg_loader_time: 0.008538790152106486 (-0.00010012237119003294) | > avg_step_time: 0.45042615877070896 (-0.00017119461381942536) | > avg_loss_disc: 2.6403168893196214 (-0.04712493990508593) | > avg_loss_disc_real_0: 0.1501989548265095 (-0.02923586350721369) | > avg_loss_disc_real_1: 0.21455913488294037 (-0.010470737768730642) | > avg_loss_disc_real_2: 0.24154265644684644 (+0.004586989506029759) | > avg_loss_disc_real_3: 0.23939347623939244 (+0.0010731545552401245) | > avg_loss_disc_real_4: 0.24613268568482197 (+0.00022650814392202734) | > avg_loss_disc_real_5: 0.24434548406533793 (-0.003956433333141635) | > avg_loss_0: 2.6403168893196214 (-0.04712493990508593) | > avg_grad_norm_0: tensor(228.7096, device='cuda:0') (tensor(-16.4747, device='cuda:0')) | > avg_loss_gen: 2.123890126255197 (-0.013801514262884229) | > avg_loss_kl: 2.090936157065378 (+0.10382485053908663) | > avg_loss_feat: 3.621342292973693 (-0.013160215297215228) | > avg_loss_mel: 16.937801226763654 (+0.10164956643546574) | > avg_loss_duration: 1.5513413875875341 (+0.0017574975188354713) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.325311284669688 (+0.18027026216748965) | > avg_grad_norm_1: tensor(6279.3931, device='cuda:0') (tensor(-421.1699, device='cuda:0'))  > EPOCH: 1134/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:53:50)   --> TIME: 2025-02-25 19:53:52 -- STEP: 1/72 -- GLOBAL_STEP: 1081650 | > loss_disc: 2.4181220531463623 (2.4181220531463623) | > loss_disc_real_0: 0.13645736873149872 (0.13645736873149872) | > loss_disc_real_1: 0.25452202558517456 (0.25452202558517456) | > loss_disc_real_2: 0.23609155416488647 (0.23609155416488647) | > loss_disc_real_3: 0.2349192351102829 (0.2349192351102829) | > loss_disc_real_4: 0.23321381211280823 (0.23321381211280823) | > loss_disc_real_5: 0.21558798849582672 (0.21558798849582672) | > loss_0: 2.4181220531463623 (2.4181220531463623) | > grad_norm_0: tensor(358.3218, device='cuda:0') (tensor(358.3218, device='cuda:0')) | > loss_gen: 2.594590187072754 (2.594590187072754) | > loss_kl: 1.7788527011871338 (1.7788527011871338) | > loss_feat: 4.741487979888916 (4.741487979888916) | > loss_mel: 17.619131088256836 (17.619131088256836) | > loss_duration: 1.5022145509719849 (1.5022145509719849) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.236276626586914 (28.236276626586914) | > grad_norm_1: tensor(2774.8105, device='cuda:0') (tensor(2774.8105, device='cuda:0')) | > current_lr_0: 0.00017356609996095179 | > current_lr_1: 0.00017356609996095179 | > step_time: 0.4224 (0.4224417209625244) | > loader_time: 0.0064 (0.006366729736328125)  --> TIME: 2025-02-25 19:54:15 -- STEP: 51/72 -- GLOBAL_STEP: 1081700 | > loss_disc: 2.844883441925049 (2.6014904835644894) | > loss_disc_real_0: 0.3588308095932007 (0.12916011177003384) | > loss_disc_real_1: 0.18502825498580933 (0.21862644804458992) | > loss_disc_real_2: 0.24414628744125366 (0.22718573054846594) | > loss_disc_real_3: 0.2637299597263336 (0.22855004492928) | > loss_disc_real_4: 0.2827763557434082 (0.24077880119576173) | > loss_disc_real_5: 0.24671822786331177 (0.24291495043857425) | > loss_0: 2.844883441925049 (2.6014904835644894) | > grad_norm_0: tensor(628.5203, device='cuda:0') (tensor(250.1286, device='cuda:0')) | > loss_gen: 2.0209004878997803 (2.147012712908727) | > loss_kl: 2.138939380645752 (2.0126772628110996) | > loss_feat: 3.5242998600006104 (4.0102398535784545) | > loss_mel: 16.31821060180664 (17.092573913873412) | > loss_duration: 1.5609216690063477 (1.5295937926161522) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.563270568847656 (26.79209754046272) | > grad_norm_1: tensor(8565.0723, device='cuda:0') (tensor(6293.2461, device='cuda:0')) | > current_lr_0: 0.00017356609996095179 | > current_lr_1: 0.00017356609996095179 | > step_time: 0.4449 (0.43888293995576744) | > loader_time: 0.0091 (0.007756364111806832) --> EVAL PERFORMANCE | > avg_loader_time: 0.008622075470400526 (+8.328531829403935e-05) | > avg_step_time: 0.44990754799104077 (-0.0005186107796681938) | > avg_loss_disc: 2.60730996937819 (-0.033006919941431345) | > avg_loss_disc_real_0: 0.1278484015869843 (-0.0223505532395252) | > avg_loss_disc_real_1: 0.21785101693280987 (+0.0032918820498694934) | > avg_loss_disc_real_2: 0.23154512215668047 (-0.009997534290165977) | > avg_loss_disc_real_3: 0.23646119215958555 (-0.0029322840798068905) | > avg_loss_disc_real_4: 0.2462716513956097 (+0.00013896571078772513) | > avg_loss_disc_real_5: 0.2453148625266384 (+0.000969378461300463) | > avg_loss_0: 2.60730996937819 (-0.033006919941431345) | > avg_grad_norm_0: tensor(234.5921, device='cuda:0') (+tensor(5.8825, device='cuda:0')) | > avg_loss_gen: 2.1350751695498613 (+0.01118504329466452) | > avg_loss_kl: 2.0567162456646764 (-0.0342199114007018) | > avg_loss_feat: 3.8965831508099185 (+0.2752408578362253) | > avg_loss_mel: 17.121725955479583 (+0.18392472871592958) | > avg_loss_duration: 1.5479252187299057 (-0.00341616885762841) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.758025693221832 (+0.43271440855214394) | > avg_grad_norm_1: tensor(6149.0669, device='cuda:0') (tensor(-130.3262, device='cuda:0'))  > EPOCH: 1135/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:54:27)   --> TIME: 2025-02-25 19:54:41 -- STEP: 29/72 -- GLOBAL_STEP: 1081750 | > loss_disc: 2.357133388519287 (2.6365749835968013) | > loss_disc_real_0: 0.08390697836875916 (0.1378755240604795) | > loss_disc_real_1: 0.1966472715139389 (0.2269883700485887) | > loss_disc_real_2: 0.21347372233867645 (0.22567304711917352) | > loss_disc_real_3: 0.22023506462574005 (0.22845572901183162) | > loss_disc_real_4: 0.24372868239879608 (0.24388158218613987) | > loss_disc_real_5: 0.2459971308708191 (0.24300322758740392) | > loss_0: 2.357133388519287 (2.6365749835968013) | > grad_norm_0: tensor(42.0500, device='cuda:0') (tensor(214.9405, device='cuda:0')) | > loss_gen: 2.4477028846740723 (2.0808372168705382) | > loss_kl: 2.1011288166046143 (1.9734703343490072) | > loss_feat: 4.925218105316162 (3.787186754160914) | > loss_mel: 17.11640739440918 (16.978043424672094) | > loss_duration: 1.5087350606918335 (1.4961869963284196) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.099191665649414 (26.31572486614359) | > grad_norm_1: tensor(2733.4824, device='cuda:0') (tensor(7337.5786, device='cuda:0')) | > current_lr_0: 0.00017354440419845666 | > current_lr_1: 0.00017354440419845666 | > step_time: 0.4304 (0.4304384609748577) | > loader_time: 0.0078 (0.006930614339894262) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585597427797988 (-3.647804260253733e-05) | > avg_step_time: 0.4480814497235795 (-0.0018260982674612736) | > avg_loss_disc: 2.669326943410954 (+0.06201697403276407) | > avg_loss_disc_real_0: 0.14437093716901792 (+0.01652253558203362) | > avg_loss_disc_real_1: 0.22486459873092007 (+0.007013581798110208) | > avg_loss_disc_real_2: 0.24477941721258029 (+0.01323429505589982) | > avg_loss_disc_real_3: 0.2279422549714505 (-0.008518937188135067) | > avg_loss_disc_real_4: 0.23968409925279482 (-0.0065875521428148676) | > avg_loss_disc_real_5: 0.24111473476383047 (-0.004200127762807926) | > avg_loss_0: 2.669326943410954 (+0.06201697403276407) | > avg_grad_norm_0: tensor(231.6202, device='cuda:0') (tensor(-2.9718, device='cuda:0')) | > avg_loss_gen: 2.070524486017898 (-0.06455068353196314) | > avg_loss_kl: 2.031968617103469 (-0.024747628561207158) | > avg_loss_feat: 3.692304010122595 (-0.20427914068732367) | > avg_loss_mel: 16.78632576364866 (-0.3354001918309244) | > avg_loss_duration: 1.5433403488615869 (-0.004584869868318853) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.124463148520025 (-0.6335625447018067) | > avg_grad_norm_1: tensor(7123.6694, device='cuda:0') (+tensor(974.6025, device='cuda:0'))  > EPOCH: 1136/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:55:03)   --> TIME: 2025-02-25 19:55:08 -- STEP: 7/72 -- GLOBAL_STEP: 1081800 | > loss_disc: 2.450896739959717 (2.5498359884534563) | > loss_disc_real_0: 0.1458917260169983 (0.13893471445356095) | > loss_disc_real_1: 0.2160494327545166 (0.2294789914573942) | > loss_disc_real_2: 0.2602596580982208 (0.2369987326008933) | > loss_disc_real_3: 0.24018990993499756 (0.23531584228788102) | > loss_disc_real_4: 0.27326062321662903 (0.23933144978114537) | > loss_disc_real_5: 0.26173555850982666 (0.24514836285795485) | > loss_0: 2.450896739959717 (2.5498359884534563) | > grad_norm_0: tensor(375.1411, device='cuda:0') (tensor(181.6578, device='cuda:0')) | > loss_gen: 2.5392258167266846 (2.220389553478786) | > loss_kl: 1.9633283615112305 (1.891284704208374) | > loss_feat: 3.958098888397217 (3.949230739048549) | > loss_mel: 16.59905433654785 (17.004674911499023) | > loss_duration: 1.4870319366455078 (1.4952864136014665) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.54673957824707 (26.560866492135183) | > grad_norm_1: tensor(2673.5334, device='cuda:0') (tensor(6574.6279, device='cuda:0')) | > current_lr_0: 0.00017352271114793184 | > current_lr_1: 0.00017352271114793184 | > step_time: 0.4215 (0.42329474857875277) | > loader_time: 0.0067 (0.006659541811261859)  --> TIME: 2025-02-25 19:55:31 -- STEP: 57/72 -- GLOBAL_STEP: 1081850 | > loss_disc: 2.547477960586548 (2.6195259637999952) | > loss_disc_real_0: 0.07338210940361023 (0.10291765006095693) | > loss_disc_real_1: 0.2527860403060913 (0.24863180596577494) | > loss_disc_real_2: 0.2996903359889984 (0.24184916103095339) | > loss_disc_real_3: 0.31356173753738403 (0.24792140513135671) | > loss_disc_real_4: 0.256070077419281 (0.2504622416015257) | > loss_disc_real_5: 0.2554880678653717 (0.24572837457322239) | > loss_0: 2.547477960586548 (2.6195259637999952) | > grad_norm_0: tensor(175.8153, device='cuda:0') (tensor(202.9434, device='cuda:0')) | > loss_gen: 2.388823986053467 (2.2029587574172425) | > loss_kl: 2.274413824081421 (2.167123549862912) | > loss_feat: 4.052883148193359 (3.9569438800477146) | > loss_mel: 17.51915168762207 (17.77937500936943) | > loss_duration: 1.606940507888794 (1.5472930460645442) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.842214584350586 (27.653694253218802) | > grad_norm_1: tensor(4392.3711, device='cuda:0') (tensor(5904.5356, device='cuda:0')) | > current_lr_0: 0.00017352271114793184 | > current_lr_1: 0.00017352271114793184 | > step_time: 0.4642 (0.44887647294161614) | > loader_time: 0.0096 (0.007977355990493507) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859164855849575 (+6.051130697761614e-06) | > avg_step_time: 0.45634242514489404 (+0.008260975421314543) | > avg_loss_disc: 2.623825412401011 (-0.04550153100994292) | > avg_loss_disc_real_0: 0.09629028602699999 (-0.04808065114201793) | > avg_loss_disc_real_1: 0.2449169893499831 (+0.020052390619063026) | > avg_loss_disc_real_2: 0.24718694388866425 (+0.0024075266760839598) | > avg_loss_disc_real_3: 0.2471519335055015 (+0.019209678534051006) | > avg_loss_disc_real_4: 0.24918284886319872 (+0.009498749610403895) | > avg_loss_disc_real_5: 0.24656054335580746 (+0.005445808591976992) | > avg_loss_0: 2.623825412401011 (-0.04550153100994292) | > avg_grad_norm_0: tensor(201.5440, device='cuda:0') (tensor(-30.0762, device='cuda:0')) | > avg_loss_gen: 2.190785260267659 (+0.12026077424976078) | > avg_loss_kl: 2.1593228276346776 (+0.12735421053120843) | > avg_loss_feat: 3.9092470054895108 (+0.21694299536691597) | > avg_loss_mel: 17.715376195773274 (+0.9290504321246154) | > avg_loss_duration: 1.5582577678519236 (+0.014917418990336717) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.532988937807755 (+1.4085257892877294) | > avg_grad_norm_1: tensor(6005.4209, device='cuda:0') (tensor(-1118.2485, device='cuda:0'))  > EPOCH: 1137/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:55:40)   --> TIME: 2025-02-25 19:55:58 -- STEP: 35/72 -- GLOBAL_STEP: 1081900 | > loss_disc: 2.3324685096740723 (2.5210200786590575) | > loss_disc_real_0: 0.03750383108854294 (0.05795590728521347) | > loss_disc_real_1: 0.20691514015197754 (0.22552424626691) | > loss_disc_real_2: 0.24293814599514008 (0.25143854320049286) | > loss_disc_real_3: 0.25043386220932007 (0.24466577427727834) | > loss_disc_real_4: 0.24185650050640106 (0.2454867320401328) | > loss_disc_real_5: 0.24782003462314606 (0.24782002099922726) | > loss_0: 2.3324685096740723 (2.5210200786590575) | > grad_norm_0: tensor(161.0027, device='cuda:0') (tensor(179.8795, device='cuda:0')) | > loss_gen: 2.293323278427124 (2.261263942718506) | > loss_kl: 2.191371202468872 (1.9352538755961828) | > loss_feat: 4.168089866638184 (4.11334842954363) | > loss_mel: 18.034385681152344 (17.055492891584123) | > loss_duration: 1.534961223602295 (1.503184529713222) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.222131729125977 (26.86854373386928) | > grad_norm_1: tensor(9871.0801, device='cuda:0') (tensor(6148.1519, device='cuda:0')) | > current_lr_0: 0.00017350102080903833 | > current_lr_1: 0.00017350102080903833 | > step_time: 0.4381 (0.4343117782047817) | > loader_time: 0.0085 (0.007026406696864537) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847162663097113 (-0.00012002192752461928) | > avg_step_time: 0.4521460936103069 (-0.00419633153458715) | > avg_loss_disc: 2.540694545692122 (-0.08313086670888925) | > avg_loss_disc_real_0: 0.07469251841096812 (-0.021597767616031868) | > avg_loss_disc_real_1: 0.22167082235846722 (-0.023246166991515882) | > avg_loss_disc_real_2: 0.2436214802970349 (-0.0035654635916293476) | > avg_loss_disc_real_3: 0.23821997957330354 (-0.00893195393219795) | > avg_loss_disc_real_4: 0.2477524717089156 (-0.0014303771542831245) | > avg_loss_disc_real_5: 0.24816224335784642 (+0.0016017000020389638) | > avg_loss_0: 2.540694545692122 (-0.08313086670888925) | > avg_grad_norm_0: tensor(175.5459, device='cuda:0') (tensor(-25.9981, device='cuda:0')) | > avg_loss_gen: 2.253289845627798 (+0.06250458536013914) | > avg_loss_kl: 1.9802412197623454 (-0.1790816078723323) | > avg_loss_feat: 4.07360151452078 (+0.16435450903126902) | > avg_loss_mel: 16.999983183095154 (-0.7153930126781205) | > avg_loss_duration: 1.5417711163910344 (-0.016486651460889234) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.848886704780686 (-0.684102233027069) | > avg_grad_norm_1: tensor(5799.6162, device='cuda:0') (tensor(-205.8047, device='cuda:0'))  > EPOCH: 1138/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:56:17)   --> TIME: 2025-02-25 19:56:24 -- STEP: 13/72 -- GLOBAL_STEP: 1081950 | > loss_disc: 2.7846953868865967 (2.78146375142611) | > loss_disc_real_0: 0.19759239256381989 (0.3605155222691022) | > loss_disc_real_1: 0.21601495146751404 (0.2155548884318425) | > loss_disc_real_2: 0.24120856821537018 (0.24810860134088075) | > loss_disc_real_3: 0.22290657460689545 (0.24251137444606194) | > loss_disc_real_4: 0.23996610939502716 (0.23218083610901466) | > loss_disc_real_5: 0.2476990669965744 (0.25309013059506047) | > loss_0: 2.7846953868865967 (2.78146375142611) | > grad_norm_0: tensor(154.7317, device='cuda:0') (tensor(255.5172, device='cuda:0')) | > loss_gen: 1.8180952072143555 (2.1447086884425235) | > loss_kl: 1.677316427230835 (1.8338762705142682) | > loss_feat: 3.0993573665618896 (3.2106554324810324) | > loss_mel: 16.605619430541992 (16.846681814927322) | > loss_duration: 1.4316679239273071 (1.484149666932913) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.632057189941406 (25.520072350135216) | > grad_norm_1: tensor(6323.6377, device='cuda:0') (tensor(5879.9600, device='cuda:0')) | > current_lr_0: 0.0001734793331814372 | > current_lr_1: 0.0001734793331814372 | > step_time: 0.4507 (0.42995456548837513) | > loader_time: 0.0057 (0.006725036180936373)  --> TIME: 2025-02-25 19:56:47 -- STEP: 63/72 -- GLOBAL_STEP: 1082000 | > loss_disc: 2.717900037765503 (2.6618550807710677) | > loss_disc_real_0: 0.09610781073570251 (0.18939440120898543) | > loss_disc_real_1: 0.19437934458255768 (0.2299784826380866) | > loss_disc_real_2: 0.20731741189956665 (0.2382220003813032) | > loss_disc_real_3: 0.22805742919445038 (0.2427602954327114) | > loss_disc_real_4: 0.21923796832561493 (0.23967247657359592) | > loss_disc_real_5: 0.2353706657886505 (0.24358352001697298) | > loss_0: 2.717900037765503 (2.6618550807710677) | > grad_norm_0: tensor(153.4304, device='cuda:0') (tensor(219.5308, device='cuda:0')) | > loss_gen: 1.841117262840271 (2.1492938806140223) | > loss_kl: 2.4266598224639893 (1.9941705730226305) | > loss_feat: 3.2757227420806885 (3.5856520751165966) | > loss_mel: 17.40165901184082 (16.70691302466015) | > loss_duration: 1.6285326480865479 (1.5348303696465873) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.573692321777344 (25.97086019364614) | > grad_norm_1: tensor(8000.2349, device='cuda:0') (tensor(5295.1221, device='cuda:0')) | > current_lr_0: 0.0001734793331814372 | > current_lr_1: 0.0001734793331814372 | > step_time: 0.5012 (0.441513977353535) | > loader_time: 0.0111 (0.008113085277496827) --> EVAL PERFORMANCE | > avg_loader_time: 0.008499377210375293 (+2.77505794041618e-05) | > avg_step_time: 0.4485879918219338 (-0.0035581017883730692) | > avg_loss_disc: 2.6617287615655174 (+0.12103421587339547) | > avg_loss_disc_real_0: 0.18761579485111674 (+0.11292327644014861) | > avg_loss_disc_real_1: 0.22960494332750078 (+0.007934120969033565) | > avg_loss_disc_real_2: 0.23852701107381094 (-0.0050944692232239575) | > avg_loss_disc_real_3: 0.24283891664424412 (+0.004618937070940576) | > avg_loss_disc_real_4: 0.2405761977736379 (-0.007176273935277699) | > avg_loss_disc_real_5: 0.24374083125255477 (-0.00442141210529165) | > avg_loss_0: 2.6617287615655174 (+0.12103421587339547) | > avg_grad_norm_0: tensor(227.8732, device='cuda:0') (+tensor(52.3273, device='cuda:0')) | > avg_loss_gen: 2.1449609659087483 (-0.10832887971904981) | > avg_loss_kl: 2.0514652947304954 (+0.07122407496815009) | > avg_loss_feat: 3.563542137683277 (-0.5100593768375026) | > avg_loss_mel: 16.84658045164297 (-0.1534027314521822) | > avg_loss_duration: 1.5480677947192127 (+0.006296678328178373) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.154616826017136 (-0.6942698787635493) | > avg_grad_norm_1: tensor(5544.9258, device='cuda:0') (tensor(-254.6904, device='cuda:0'))  > EPOCH: 1139/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:56:53)   --> TIME: 2025-02-25 19:57:14 -- STEP: 41/72 -- GLOBAL_STEP: 1082050 | > loss_disc: 2.571367025375366 (2.6852709607380194) | > loss_disc_real_0: 0.07412037998437881 (0.11992829683713796) | > loss_disc_real_1: 0.27169734239578247 (0.23167105891355655) | > loss_disc_real_2: 0.30440986156463623 (0.24919658354142818) | > loss_disc_real_3: 0.29657912254333496 (0.24958414275471752) | > loss_disc_real_4: 0.23315691947937012 (0.24323201506603054) | > loss_disc_real_5: 0.2488124817609787 (0.24155251536427474) | > loss_0: 2.571367025375366 (2.6852709607380194) | > grad_norm_0: tensor(167.2746, device='cuda:0') (tensor(209.8642, device='cuda:0')) | > loss_gen: 2.1974472999572754 (2.0456559628975106) | > loss_kl: 2.179826498031616 (2.1272891701721557) | > loss_feat: 3.5310463905334473 (3.7259968024928396) | > loss_mel: 17.717138290405273 (17.389806352010588) | > loss_duration: 1.580383062362671 (1.5312276729723302) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.205841064453125 (26.819975876226657) | > grad_norm_1: tensor(6211.6006, device='cuda:0') (tensor(5700.6895, device='cuda:0')) | > current_lr_0: 0.00017345764826478952 | > current_lr_1: 0.00017345764826478952 | > step_time: 0.4669 (0.44853301164580556) | > loader_time: 0.009 (0.00702174698434225) --> EVAL PERFORMANCE | > avg_loader_time: 0.008382699858974402 (-0.00011667735140089008) | > avg_step_time: 0.4599872206298398 (+0.011399228807906003) | > avg_loss_disc: 2.6218693424278583 (-0.03985941913765911) | > avg_loss_disc_real_0: 0.11448910661881238 (-0.07312668823230435) | > avg_loss_disc_real_1: 0.2312668203048303 (+0.0016618769773295294) | > avg_loss_disc_real_2: 0.24243559279072452 (+0.003908581716913578) | > avg_loss_disc_real_3: 0.243948778636019 (+0.0011098619917748742) | > avg_loss_disc_real_4: 0.23862952455668382 (-0.0019466732169540757) | > avg_loss_disc_real_5: 0.24043860758694124 (-0.0033022236656135295) | > avg_loss_0: 2.6218693424278583 (-0.03985941913765911) | > avg_grad_norm_0: tensor(201.6379, device='cuda:0') (tensor(-26.2353, device='cuda:0')) | > avg_loss_gen: 2.1207150304821183 (-0.024245935426630005) | > avg_loss_kl: 2.1320262240691914 (+0.08056092933869596) | > avg_loss_feat: 3.7766368053328825 (+0.21309466764960527) | > avg_loss_mel: 17.166766327871397 (+0.3201858762284253) | > avg_loss_duration: 1.5565351046306986 (+0.00846730991148581) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.752679475596253 (+0.5980626495791164) | > avg_grad_norm_1: tensor(5409.9521, device='cuda:0') (tensor(-134.9736, device='cuda:0'))  > EPOCH: 1140/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:57:30)   --> TIME: 2025-02-25 19:57:41 -- STEP: 19/72 -- GLOBAL_STEP: 1082100 | > loss_disc: 2.466367244720459 (2.569585737429167) | > loss_disc_real_0: 0.06988345086574554 (0.09122970945348866) | > loss_disc_real_1: 0.22679269313812256 (0.23325753525683754) | > loss_disc_real_2: 0.2018587291240692 (0.23051770894150986) | > loss_disc_real_3: 0.2121364176273346 (0.23339251860191948) | > loss_disc_real_4: 0.23412632942199707 (0.24366319571670733) | > loss_disc_real_5: 0.24011445045471191 (0.2396227662500582) | > loss_0: 2.466367244720459 (2.569585737429167) | > grad_norm_0: tensor(130.3852, device='cuda:0') (tensor(222.6187, device='cuda:0')) | > loss_gen: 2.2062056064605713 (2.1176268740704187) | > loss_kl: 1.7952369451522827 (1.9361679428502132) | > loss_feat: 3.8568098545074463 (3.780387439225849) | > loss_mel: 16.9130859375 (16.84272148734645) | > loss_duration: 1.5170519351959229 (1.501312820534957) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.288389205932617 (26.178216633043792) | > grad_norm_1: tensor(4159.8628, device='cuda:0') (tensor(6054.4292, device='cuda:0')) | > current_lr_0: 0.00017343596605875642 | > current_lr_1: 0.00017343596605875642 | > step_time: 0.4326 (0.4312777644709537) | > loader_time: 0.0059 (0.006866266852930973)  --> TIME: 2025-02-25 19:58:04 -- STEP: 69/72 -- GLOBAL_STEP: 1082150 | > loss_disc: 2.620255947113037 (2.646417621253193) | > loss_disc_real_0: 0.12315145879983902 (0.1344057719722606) | > loss_disc_real_1: 0.2686080038547516 (0.23782142510880594) | > loss_disc_real_2: 0.2182968556880951 (0.22498995605586228) | > loss_disc_real_3: 0.25444766879081726 (0.23070483363193015) | > loss_disc_real_4: 0.25050967931747437 (0.24841609143692514) | > loss_disc_real_5: 0.28171369433403015 (0.24768122965874878) | > loss_0: 2.620255947113037 (2.646417621253193) | > grad_norm_0: tensor(88.5809, device='cuda:0') (tensor(238.9424, device='cuda:0')) | > loss_gen: 2.1662471294403076 (2.136115407598192) | > loss_kl: 2.5045719146728516 (2.608794663263404) | > loss_feat: 3.3365533351898193 (3.798262599585713) | > loss_mel: 18.361011505126953 (18.550611938255418) | > loss_duration: 1.6864839792251587 (1.5976099311441616) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.054866790771484 (28.691394667694535) | > grad_norm_1: tensor(5650.8750, device='cuda:0') (tensor(6957.6470, device='cuda:0')) | > current_lr_0: 0.00017343596605875642 | > current_lr_1: 0.00017343596605875642 | > step_time: 0.535 (0.4503632801166479) | > loader_time: 0.0128 (0.008482152137203493) --> EVAL PERFORMANCE | > avg_loader_time: 0.008543877534463371 (+0.00016117767548896877) | > avg_step_time: 0.45186130093856597 (-0.008125919691273853) | > avg_loss_disc: 2.6449525054071987 (+0.023083162979340432) | > avg_loss_disc_real_0: 0.1333263164622263 (+0.01883720984341393) | > avg_loss_disc_real_1: 0.23773843585185603 (+0.006471615547025716) | > avg_loss_disc_real_2: 0.22507382404636328 (-0.01736176874436124) | > avg_loss_disc_real_3: 0.2302927811380843 (-0.013655997497934697) | > avg_loss_disc_real_4: 0.24750786495040838 (+0.008878340393724565) | > avg_loss_disc_real_5: 0.2476477669158452 (+0.007209159328903952) | > avg_loss_0: 2.6449525054071987 (+0.023083162979340432) | > avg_grad_norm_0: tensor(235.0608, device='cuda:0') (+tensor(33.4229, device='cuda:0')) | > avg_loss_gen: 2.136000322623992 (+0.015285292141873796) | > avg_loss_kl: 2.6042308589102516 (+0.47220463484106023) | > avg_loss_feat: 3.7880762429304524 (+0.011439437597569935) | > avg_loss_mel: 18.531532099549196 (+1.3647657716777992) | > avg_loss_duration: 1.5976260178525685 (+0.041090913221869974) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.657465652680735 (+1.9047861770844818) | > avg_grad_norm_1: tensor(7000.8447, device='cuda:0') (+tensor(1590.8926, device='cuda:0'))  > EPOCH: 1141/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:58:07)   --> TIME: 2025-02-25 19:58:30 -- STEP: 47/72 -- GLOBAL_STEP: 1082200 | > loss_disc: 2.9720025062561035 (2.619251028020332) | > loss_disc_real_0: 0.052747778594493866 (0.11840020389632976) | > loss_disc_real_1: 0.26369038224220276 (0.22738128361549784) | > loss_disc_real_2: 0.2914866805076599 (0.24441614715342827) | > loss_disc_real_3: 0.2482808381319046 (0.2364375502505201) | > loss_disc_real_4: 0.26506954431533813 (0.24586083914371246) | > loss_disc_real_5: 0.26849421858787537 (0.2440415008905086) | > loss_0: 2.9720025062561035 (2.619251028020332) | > grad_norm_0: tensor(359.7328, device='cuda:0') (tensor(200.1957, device='cuda:0')) | > loss_gen: 1.7405309677124023 (2.100925196992591) | > loss_kl: 2.1021792888641357 (2.1259299465950487) | > loss_feat: 3.0405466556549072 (3.783828770860713) | > loss_mel: 17.5883846282959 (17.42077780784445) | > loss_duration: 1.592044711112976 (1.541622204983488) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.06368637084961 (26.973084226567696) | > grad_norm_1: tensor(3654.3171, device='cuda:0') (tensor(7073.3608, device='cuda:0')) | > current_lr_0: 0.00017341428656299907 | > current_lr_1: 0.00017341428656299907 | > step_time: 0.4503 (0.4380441016339241) | > loader_time: 0.0088 (0.0076572590685905296) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864837539028114 (+0.00010449785581776935) | > avg_step_time: 0.45015154422169 (-0.0017097567168759542) | > avg_loss_disc: 2.6196016694458444 (-0.025350835961354345) | > avg_loss_disc_real_0: 0.12915581323101485 (-0.004170503231211459) | > avg_loss_disc_real_1: 0.22536943705988602 (-0.01236899879197001) | > avg_loss_disc_real_2: 0.24249621449222028 (+0.017422390445857) | > avg_loss_disc_real_3: 0.23296005037468923 (+0.00266726923660493) | > avg_loss_disc_real_4: 0.2438640959665809 (-0.0036437689838274834) | > avg_loss_disc_real_5: 0.2430112364006714 (-0.004636530515173792) | > avg_loss_0: 2.6196016694458444 (-0.025350835961354345) | > avg_grad_norm_0: tensor(223.4214, device='cuda:0') (tensor(-11.6393, device='cuda:0')) | > avg_loss_gen: 2.1125518117152478 (-0.023448510908744336) | > avg_loss_kl: 2.12323640769636 (-0.4809944512138915) | > avg_loss_feat: 3.694307142580059 (-0.09376910035039332) | > avg_loss_mel: 17.25215649940599 (-1.2793756001432044) | > avg_loss_duration: 1.560427380279756 (-0.03719863757281261) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.74267938103474 (-1.914786271645994) | > avg_grad_norm_1: tensor(7118.5181, device='cuda:0') (+tensor(117.6733, device='cuda:0'))  > EPOCH: 1142/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:58:44)   --> TIME: 2025-02-25 19:58:57 -- STEP: 25/72 -- GLOBAL_STEP: 1082250 | > loss_disc: 2.853440523147583 (2.7346307182312013) | > loss_disc_real_0: 0.06066156178712845 (0.11680123999714852) | > loss_disc_real_1: 0.13709117472171783 (0.20760697335004807) | > loss_disc_real_2: 0.21867386996746063 (0.2247583943605423) | > loss_disc_real_3: 0.20780380070209503 (0.22746276915073393) | > loss_disc_real_4: 0.2898852229118347 (0.2441796946525574) | > loss_disc_real_5: 0.25099214911460876 (0.2533556205034256) | > loss_0: 2.853440523147583 (2.7346307182312013) | > grad_norm_0: tensor(493.6208, device='cuda:0') (tensor(145.5621, device='cuda:0')) | > loss_gen: 1.8534375429153442 (2.062818655967712) | > loss_kl: 2.509190797805786 (2.3475591373443607) | > loss_feat: 3.835888385772705 (3.9216529750823974) | > loss_mel: 17.405479431152344 (18.311608123779298) | > loss_duration: 1.553203821182251 (1.5549560117721557) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.15719985961914 (28.19859504699707) | > grad_norm_1: tensor(14451.2891, device='cuda:0') (tensor(4821.7910, device='cuda:0')) | > current_lr_0: 0.0001733926097771787 | > current_lr_1: 0.0001733926097771787 | > step_time: 0.4312 (0.45444501876831056) | > loader_time: 0.0072 (0.006738071441650391) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613757684197228 (-3.46177060839125e-05) | > avg_step_time: 0.4597797393798828 (+0.009628195158192798) | > avg_loss_disc: 2.75999594070542 (+0.14039427125957582) | > avg_loss_disc_real_0: 0.18482162383660464 (+0.05566581060558978) | > avg_loss_disc_real_1: 0.2219010821320641 (-0.0034683549278219095) | > avg_loss_disc_real_2: 0.24543879149665296 (+0.0029425770044326782) | > avg_loss_disc_real_3: 0.2365913340743159 (+0.003631283699626675) | > avg_loss_disc_real_4: 0.24920571123210478 (+0.005341615265523875) | > avg_loss_disc_real_5: 0.2483059476378938 (+0.005294711237222388) | > avg_loss_0: 2.75999594070542 (+0.14039427125957582) | > avg_grad_norm_0: tensor(241.7457, device='cuda:0') (+tensor(18.3242, device='cuda:0')) | > avg_loss_gen: 2.0884448323451297 (-0.024106979370118076) | > avg_loss_kl: 2.2585760590056303 (+0.13533965130927017) | > avg_loss_feat: 3.6236635765559235 (-0.07064356602413557) | > avg_loss_mel: 17.714005322523512 (+0.4618488231175206) | > avg_loss_duration: 1.5783888857129593 (+0.017961505433203406) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.263078850759587 (+0.5203994697248469) | > avg_grad_norm_1: tensor(5925.6450, device='cuda:0') (tensor(-1192.8730, device='cuda:0'))  > EPOCH: 1143/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:59:21)   --> TIME: 2025-02-25 19:59:24 -- STEP: 3/72 -- GLOBAL_STEP: 1082300 | > loss_disc: 2.743218183517456 (2.7926549116770425) | > loss_disc_real_0: 0.06540723890066147 (0.21394609659910202) | > loss_disc_real_1: 0.24911516904830933 (0.2546492964029312) | > loss_disc_real_2: 0.20786084234714508 (0.22208906710147858) | > loss_disc_real_3: 0.2380334734916687 (0.2500595698753993) | > loss_disc_real_4: 0.21823857724666595 (0.23621403177579245) | > loss_disc_real_5: 0.2442396879196167 (0.2559637228647868) | > loss_0: 2.743218183517456 (2.7926549116770425) | > grad_norm_0: tensor(245.6310, device='cuda:0') (tensor(253.1089, device='cuda:0')) | > loss_gen: 1.962846040725708 (1.8474253416061401) | > loss_kl: 1.7799561023712158 (1.8994867006937664) | > loss_feat: 3.919276714324951 (3.3189247449239097) | > loss_mel: 17.958145141601562 (17.643258412679035) | > loss_duration: 1.537678837776184 (1.5228819449742634) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.15790557861328 (26.231978098551433) | > grad_norm_1: tensor(9135.3203, device='cuda:0') (tensor(8732.3457, device='cuda:0')) | > current_lr_0: 0.00017337093570095655 | > current_lr_1: 0.00017337093570095655 | > step_time: 0.42 (0.42456523577372235) | > loader_time: 0.0081 (0.007018566131591797)  --> TIME: 2025-02-25 19:59:47 -- STEP: 53/72 -- GLOBAL_STEP: 1082350 | > loss_disc: 2.5923585891723633 (2.77828407737444) | > loss_disc_real_0: 0.11053143441677094 (0.21890372290926163) | > loss_disc_real_1: 0.2282777726650238 (0.22264862173008468) | > loss_disc_real_2: 0.18959228694438934 (0.22650844955219412) | > loss_disc_real_3: 0.2282339632511139 (0.24091226593503412) | > loss_disc_real_4: 0.2403700053691864 (0.23800436196462163) | > loss_disc_real_5: 0.24888131022453308 (0.2452015812104603) | > loss_0: 2.5923585891723633 (2.77828407737444) | > grad_norm_0: tensor(254.1064, device='cuda:0') (tensor(111.5552, device='cuda:0')) | > loss_gen: 1.8968673944473267 (1.922959012805291) | > loss_kl: 2.365997552871704 (2.190328903917996) | > loss_feat: 3.6501402854919434 (3.3472286260352946) | > loss_mel: 17.774795532226562 (17.62090017210763) | > loss_duration: 1.6589163541793823 (1.5605886094975019) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.346717834472656 (26.642005416582215) | > grad_norm_1: tensor(8216.9863, device='cuda:0') (tensor(4111.7930, device='cuda:0')) | > current_lr_0: 0.00017337093570095655 | > current_lr_1: 0.00017337093570095655 | > step_time: 0.4423 (0.4438705354366662) | > loader_time: 0.0089 (0.007924988584698369) --> EVAL PERFORMANCE | > avg_loader_time: 0.008668053318077405 (+5.429563388017686e-05) | > avg_step_time: 0.4545430129682514 (-0.0052367264116314205) | > avg_loss_disc: 2.7426543034298314 (-0.017341637275588795) | > avg_loss_disc_real_0: 0.20039058356008063 (+0.015568959723475995) | > avg_loss_disc_real_1: 0.23136411649240574 (+0.009463034360341627) | > avg_loss_disc_real_2: 0.22643819430344542 (-0.01900059719320754) | > avg_loss_disc_real_3: 0.2367433338937625 (+0.00015199981944660768) | > avg_loss_disc_real_4: 0.2402801557745732 (-0.00892555545753157) | > avg_loss_disc_real_5: 0.24689865028354482 (-0.0014072973543489686) | > avg_loss_0: 2.7426543034298314 (-0.017341637275588795) | > avg_grad_norm_0: tensor(134.2369, device='cuda:0') (tensor(-107.5088, device='cuda:0')) | > avg_loss_gen: 1.953723689200173 (-0.13472114314495665) | > avg_loss_kl: 2.2062930674620076 (-0.0522829915436227) | > avg_loss_feat: 3.3702934762121926 (-0.253370100343731) | > avg_loss_mel: 17.56409631648534 (-0.1499090060381718) | > avg_loss_duration: 1.5784353403977942 (+4.6454684834884574e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67284194516464 (-0.5902369055949457) | > avg_grad_norm_1: tensor(4621.4263, device='cuda:0') (tensor(-1304.2188, device='cuda:0'))  > EPOCH: 1144/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 19:59:58)   --> TIME: 2025-02-25 20:00:14 -- STEP: 31/72 -- GLOBAL_STEP: 1082400 | > loss_disc: 2.5562033653259277 (2.6832341532553396) | > loss_disc_real_0: 0.07539647817611694 (0.12569821277453053) | > loss_disc_real_1: 0.18310970067977905 (0.19510997711650788) | > loss_disc_real_2: 0.2509685456752777 (0.22685603701299237) | > loss_disc_real_3: 0.22723332047462463 (0.23699027492154032) | > loss_disc_real_4: 0.25411152839660645 (0.24115767786579748) | > loss_disc_real_5: 0.26710328459739685 (0.24592992903724795) | > loss_0: 2.5562033653259277 (2.6832341532553396) | > grad_norm_0: tensor(147.2111, device='cuda:0') (tensor(296.3230, device='cuda:0')) | > loss_gen: 1.9451768398284912 (1.9055857158476306) | > loss_kl: 2.1955928802490234 (1.9717025449199062) | > loss_feat: 3.4996817111968994 (3.5409605579991497) | > loss_mel: 16.317975997924805 (17.182125583771732) | > loss_duration: 1.5447205305099487 (1.516985954776887) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.50314712524414 (26.117360453451834) | > grad_norm_1: tensor(14063.9199, device='cuda:0') (tensor(10577.0039, device='cuda:0')) | > current_lr_0: 0.00017334926433399393 | > current_lr_1: 0.00017334926433399393 | > step_time: 0.4417 (0.43659414783600836) | > loader_time: 0.0081 (0.007008037259501796) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852856501727037 (-0.00013948830080703564) | > avg_step_time: 0.4548414492271316 (+0.0002984362588802081) | > avg_loss_disc: 2.6574749845853995 (-0.08517931884443186) | > avg_loss_disc_real_0: 0.12806942408353508 (-0.07232115947654555) | > avg_loss_disc_real_1: 0.2137532858571536 (-0.017610830635252123) | > avg_loss_disc_real_2: 0.23226076868218434 (+0.005822574378738926) | > avg_loss_disc_real_3: 0.23576633863046137 (-0.0009769952633011392) | > avg_loss_disc_real_4: 0.24278659950679457 (+0.002506443732221364) | > avg_loss_disc_real_5: 0.24556824921722142 (-0.0013304010663234) | > avg_loss_0: 2.6574749845853995 (-0.08517931884443186) | > avg_grad_norm_0: tensor(301.6658, device='cuda:0') (+tensor(167.4288, device='cuda:0')) | > avg_loss_gen: 1.9540757847503878 (+0.00035209555021475936) | > avg_loss_kl: 2.02384759003008 (-0.18244547743192774) | > avg_loss_feat: 3.502871251442063 (+0.13257777522987046) | > avg_loss_mel: 17.004671419170542 (-0.5594248973147984) | > avg_loss_duration: 1.552808472807978 (-0.025626867589816138) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.038274576966188 (-0.6345673681984536) | > avg_grad_norm_1: tensor(10261.3125, device='cuda:0') (+tensor(5639.8862, device='cuda:0'))  > EPOCH: 1145/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:00:35)   --> TIME: 2025-02-25 20:00:41 -- STEP: 9/72 -- GLOBAL_STEP: 1082450 | > loss_disc: 2.462329626083374 (2.5548461278279624) | > loss_disc_real_0: 0.10981569439172745 (0.1255109202530649) | > loss_disc_real_1: 0.20962101221084595 (0.2096486141284307) | > loss_disc_real_2: 0.228794664144516 (0.23317234218120575) | > loss_disc_real_3: 0.21964026987552643 (0.23375333183341557) | > loss_disc_real_4: 0.23941458761692047 (0.25166190167268115) | > loss_disc_real_5: 0.2403634786605835 (0.24623790548907387) | > loss_0: 2.462329626083374 (2.5548461278279624) | > grad_norm_0: tensor(11.1812, device='cuda:0') (tensor(330.5587, device='cuda:0')) | > loss_gen: 1.9591602087020874 (2.0704287555482654) | > loss_kl: 1.9235481023788452 (1.8189635939068265) | > loss_feat: 3.960338592529297 (3.6291383372412787) | > loss_mel: 17.846492767333984 (17.01870346069336) | > loss_duration: 1.514587640762329 (1.5148231320910983) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.204126358032227 (26.052057054307724) | > grad_norm_1: tensor(8039.2725, device='cuda:0') (tensor(11782.6826, device='cuda:0')) | > current_lr_0: 0.00017332759567595217 | > current_lr_1: 0.00017332759567595217 | > step_time: 0.4297 (0.42987198299831814) | > loader_time: 0.0055 (0.006950643327501085)  --> TIME: 2025-02-25 20:01:04 -- STEP: 59/72 -- GLOBAL_STEP: 1082500 | > loss_disc: 2.5114684104919434 (2.6412011526398746) | > loss_disc_real_0: 0.15185260772705078 (0.15469341778780443) | > loss_disc_real_1: 0.1098882406949997 (0.22739284776024898) | > loss_disc_real_2: 0.27386197447776794 (0.23805768636323638) | > loss_disc_real_3: 0.25140905380249023 (0.23767628179768388) | > loss_disc_real_4: 0.26692935824394226 (0.24562294023521883) | > loss_disc_real_5: 0.2626045346260071 (0.2416379353252508) | > loss_0: 2.5114684104919434 (2.6412011526398746) | > grad_norm_0: tensor(142.4536, device='cuda:0') (tensor(292.3133, device='cuda:0')) | > loss_gen: 2.042581558227539 (2.1021495996895485) | > loss_kl: 2.0938162803649902 (1.9928123203374573) | > loss_feat: 3.5493674278259277 (3.7105088557227184) | > loss_mel: 16.449092864990234 (17.056820966429626) | > loss_duration: 1.5761629343032837 (1.5513907873024375) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.711021423339844 (26.41368261434264) | > grad_norm_1: tensor(14230.2578, device='cuda:0') (tensor(7754.4829, device='cuda:0')) | > current_lr_0: 0.00017332759567595217 | > current_lr_1: 0.00017332759567595217 | > step_time: 0.4839 (0.4435620752431579) | > loader_time: 0.0098 (0.008165844416214247) --> EVAL PERFORMANCE | > avg_loader_time: 0.008710263480602857 (+0.00018169846333248793) | > avg_step_time: 0.45198739078682915 (-0.002854058440302454) | > avg_loss_disc: 2.6440254332314077 (-0.01344955135399184) | > avg_loss_disc_real_0: 0.16624847791669234 (+0.038179053833157256) | > avg_loss_disc_real_1: 0.2336145884973902 (+0.019861302640236583) | > avg_loss_disc_real_2: 0.2372742425388014 (+0.005013473856617046) | > avg_loss_disc_real_3: 0.2365317659478792 (+0.0007654273174178272) | > avg_loss_disc_real_4: 0.2453396922685731 (+0.002553092761778536) | > avg_loss_disc_real_5: 0.24291941985278062 (-0.0026488293644408023) | > avg_loss_0: 2.6440254332314077 (-0.01344955135399184) | > avg_grad_norm_0: tensor(280.7004, device='cuda:0') (tensor(-20.9653, device='cuda:0')) | > avg_loss_gen: 2.1311065227213044 (+0.17703073797091662) | > avg_loss_kl: 2.0071851518792165 (-0.016662438150863323) | > avg_loss_feat: 3.655813220521094 (+0.15294196907903101) | > avg_loss_mel: 17.033189961608024 (+0.028518542437481642) | > avg_loss_duration: 1.5602428678055884 (+0.0074343949976103385) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.387537808485433 (+0.3492632315192452) | > avg_grad_norm_1: tensor(7360.6499, device='cuda:0') (tensor(-2900.6626, device='cuda:0'))  > EPOCH: 1146/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:01:12)   --> TIME: 2025-02-25 20:01:31 -- STEP: 37/72 -- GLOBAL_STEP: 1082550 | > loss_disc: 2.8623433113098145 (2.6815366744995117) | > loss_disc_real_0: 0.2328350991010666 (0.18951480372531995) | > loss_disc_real_1: 0.23592828214168549 (0.250877013480341) | > loss_disc_real_2: 0.21541686356067657 (0.2308644273796597) | > loss_disc_real_3: 0.2428920716047287 (0.22909078283889875) | > loss_disc_real_4: 0.23695550858974457 (0.2397389512609791) | > loss_disc_real_5: 0.2280217558145523 (0.23516207651512042) | > loss_0: 2.8623433113098145 (2.6815366744995117) | > grad_norm_0: tensor(581.3441, device='cuda:0') (tensor(280.1936, device='cuda:0')) | > loss_gen: 1.8961377143859863 (2.0669814928157906) | > loss_kl: 1.929484486579895 (2.0044366926760286) | > loss_feat: 3.5511913299560547 (3.5716951151151917) | > loss_mel: 16.54587745666504 (17.41441639049633) | > loss_duration: 1.5383198261260986 (1.5202751288542877) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.46101188659668 (26.577804874729466) | > grad_norm_1: tensor(10679.6582, device='cuda:0') (tensor(6025.4888, device='cuda:0')) | > current_lr_0: 0.00017330592972649266 | > current_lr_1: 0.00017330592972649266 | > step_time: 0.445 (0.43741033528302165) | > loader_time: 0.0078 (0.007321905445408177) --> EVAL PERFORMANCE | > avg_loader_time: 0.008620708761080887 (-8.955471952197015e-05) | > avg_step_time: 0.453504676550207 (+0.0015172857633778403) | > avg_loss_disc: 2.6693897314474606 (+0.025364298216052905) | > avg_loss_disc_real_0: 0.1587569034645255 (-0.007491574452166833) | > avg_loss_disc_real_1: 0.23931679318488477 (+0.0057022046874945775) | > avg_loss_disc_real_2: 0.23318508036539587 (-0.0040891621734055195) | > avg_loss_disc_real_3: 0.23067177538300904 (-0.00585999056487016) | > avg_loss_disc_real_4: 0.24165611556718047 (-0.0036835767013926324) | > avg_loss_disc_real_5: 0.24064563382679308 (-0.0022737860259875453) | > avg_loss_0: 2.6693897314474606 (+0.025364298216052905) | > avg_grad_norm_0: tensor(219.6558, device='cuda:0') (tensor(-61.0446, device='cuda:0')) | > avg_loss_gen: 2.0685323812592196 (-0.06257414146208484) | > avg_loss_kl: 2.0696470401656457 (+0.06246188828642918) | > avg_loss_feat: 3.635371248487016 (-0.02044197203407805) | > avg_loss_mel: 17.31135584602893 (+0.2781658844209076) | > avg_loss_duration: 1.5569952911054585 (-0.0032475767001298728) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.641901795293244 (+0.25436398680781025) | > avg_grad_norm_1: tensor(5844.9531, device='cuda:0') (tensor(-1515.6968, device='cuda:0'))  > EPOCH: 1147/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:01:49)   --> TIME: 2025-02-25 20:01:57 -- STEP: 15/72 -- GLOBAL_STEP: 1082600 | > loss_disc: 2.4306840896606445 (2.4596570491790772) | > loss_disc_real_0: 0.09549497812986374 (0.08282568752765655) | > loss_disc_real_1: 0.2107226848602295 (0.2237544486920039) | > loss_disc_real_2: 0.17081604897975922 (0.20246033767859142) | > loss_disc_real_3: 0.18254534900188446 (0.23876717189947763) | > loss_disc_real_4: 0.20884783565998077 (0.23259568512439727) | > loss_disc_real_5: 0.2492743283510208 (0.24384050965309143) | > loss_0: 2.4306840896606445 (2.4596570491790772) | > grad_norm_0: tensor(301.5476, device='cuda:0') (tensor(216.5445, device='cuda:0')) | > loss_gen: 2.0868470668792725 (2.2036364793777468) | > loss_kl: 2.4223291873931885 (2.5676800568898517) | > loss_feat: 3.9690136909484863 (4.063617340723673) | > loss_mel: 18.896142959594727 (19.223323186238606) | > loss_duration: 1.5484447479248047 (1.6045822461446126) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.922779083251953 (29.662839889526367) | > grad_norm_1: tensor(8900.7734, device='cuda:0') (tensor(7897.9199, device='cuda:0')) | > current_lr_0: 0.00017328426648527686 | > current_lr_1: 0.00017328426648527686 | > step_time: 0.4202 (0.4270573298136393) | > loader_time: 0.0064 (0.007004133860270182)  --> TIME: 2025-02-25 20:02:21 -- STEP: 65/72 -- GLOBAL_STEP: 1082650 | > loss_disc: 2.6994385719299316 (2.5324228580181414) | > loss_disc_real_0: 0.09815037995576859 (0.08172843565161411) | > loss_disc_real_1: 0.2586135268211365 (0.22048227202433807) | > loss_disc_real_2: 0.2213217169046402 (0.2390223544377547) | > loss_disc_real_3: 0.21701425313949585 (0.2275715947151184) | > loss_disc_real_4: 0.2242719829082489 (0.23345762514151058) | > loss_disc_real_5: 0.2326442003250122 (0.2423203367453355) | > loss_0: 2.6994385719299316 (2.5324228580181414) | > grad_norm_0: tensor(132.2205, device='cuda:0') (tensor(243.3259, device='cuda:0')) | > loss_gen: 1.9470993280410767 (2.1776269142444327) | > loss_kl: 2.146050453186035 (2.216933155059815) | > loss_feat: 3.3056094646453857 (4.047778349656324) | > loss_mel: 16.620052337646484 (17.889801436204177) | > loss_duration: 1.6081407070159912 (1.5780581822762123) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.626953125 (27.9101981823261) | > grad_norm_1: tensor(6970.3457, device='cuda:0') (tensor(8956.2832, device='cuda:0')) | > current_lr_0: 0.00017328426648527686 | > current_lr_1: 0.00017328426648527686 | > step_time: 0.4942 (0.44476155867943395) | > loader_time: 0.0114 (0.008267707091111404) --> EVAL PERFORMANCE | > avg_loader_time: 0.008526973321404256 (-9.373543967663121e-05) | > avg_step_time: 0.45089012468364875 (-0.0026145518665582346) | > avg_loss_disc: 2.5368602410168717 (-0.13252949043058893) | > avg_loss_disc_real_0: 0.08312820097510244 (-0.07562870248942306) | > avg_loss_disc_real_1: 0.22314676086247806 (-0.016170032322406713) | > avg_loss_disc_real_2: 0.23838029717895345 (+0.005195216813557585) | > avg_loss_disc_real_3: 0.22765959715339498 (-0.0030121782296140542) | > avg_loss_disc_real_4: 0.2335790819685224 (-0.008077033598658079) | > avg_loss_disc_real_5: 0.2423179183207767 (+0.0016722844939836279) | > avg_loss_0: 2.5368602410168717 (-0.13252949043058893) | > avg_grad_norm_0: tensor(248.9637, device='cuda:0') (+tensor(29.3079, device='cuda:0')) | > avg_loss_gen: 2.175957889624046 (+0.10742550836482634) | > avg_loss_kl: 2.212698325304918 (+0.14305128513927245) | > avg_loss_feat: 4.011361269883706 (+0.3759900213966896) | > avg_loss_mel: 17.800829014308015 (+0.4894731682790834) | > avg_loss_duration: 1.5812829951165428 (+0.024287704011084266) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.782129636952575 (+1.1402278416593319) | > avg_grad_norm_1: tensor(8780.0430, device='cuda:0') (+tensor(2935.0898, device='cuda:0'))  > EPOCH: 1148/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:02:26)   --> TIME: 2025-02-25 20:02:47 -- STEP: 43/72 -- GLOBAL_STEP: 1082700 | > loss_disc: 2.5300488471984863 (2.5272792993589888) | > loss_disc_real_0: 0.09550943970680237 (0.08478386734807214) | > loss_disc_real_1: 0.21084924042224884 (0.2175631284020668) | > loss_disc_real_2: 0.22710518538951874 (0.2433704574440801) | > loss_disc_real_3: 0.2289687842130661 (0.2303725543410279) | > loss_disc_real_4: 0.23044352233409882 (0.2397711152254149) | > loss_disc_real_5: 0.24110721051692963 (0.24364870613397555) | > loss_0: 2.5300488471984863 (2.5272792993589888) | > grad_norm_0: tensor(294.4925, device='cuda:0') (tensor(278.4081, device='cuda:0')) | > loss_gen: 2.1853113174438477 (2.178997793862986) | > loss_kl: 2.001685380935669 (1.9294292705003606) | > loss_feat: 3.980581283569336 (4.01064661491749) | > loss_mel: 16.785757064819336 (16.976782310840694) | > loss_duration: 1.536381483078003 (1.5168661433596944) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.489715576171875 (26.612722175065862) | > grad_norm_1: tensor(7735.2827, device='cuda:0') (tensor(8445.8877, device='cuda:0')) | > current_lr_0: 0.00017326260595196618 | > current_lr_1: 0.00017326260595196618 | > step_time: 0.4411 (0.4336078499638757) | > loader_time: 0.0092 (0.007482323535653048) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593972300139951 (+6.699897873569532e-05) | > avg_step_time: 0.4528885760777434 (+0.001998451394094658) | > avg_loss_disc: 2.5340674158553 (-0.0027928251615718658) | > avg_loss_disc_real_0: 0.08770269265686961 (+0.0045744916817671705) | > avg_loss_disc_real_1: 0.21804879836633173 (-0.00509796249614633) | > avg_loss_disc_real_2: 0.2404804355661634 (+0.002100138387209932) | > avg_loss_disc_real_3: 0.22873641970291944 (+0.0010768225495244588) | > avg_loss_disc_real_4: 0.2375805035863124 (+0.004001421617790019) | > avg_loss_disc_real_5: 0.24516462251334123 (+0.0028467041925645253) | > avg_loss_0: 2.5340674158553 (-0.0027928251615718658) | > avg_grad_norm_0: tensor(280.9831, device='cuda:0') (+tensor(32.0194, device='cuda:0')) | > avg_loss_gen: 2.1682727437623797 (-0.007685145861666243) | > avg_loss_kl: 1.9700232626686633 (-0.24267506263625482) | > avg_loss_feat: 3.939438218801794 (-0.07192305108191155) | > avg_loss_mel: 16.879204414260215 (-0.9216246000477994) | > avg_loss_duration: 1.5436587451209485 (-0.03762424999559433) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.500597537403372 (-1.281532099549203) | > avg_grad_norm_1: tensor(8454.7363, device='cuda:0') (tensor(-325.3066, device='cuda:0'))  > EPOCH: 1149/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:03:02)   --> TIME: 2025-02-25 20:03:13 -- STEP: 21/72 -- GLOBAL_STEP: 1082750 | > loss_disc: 2.606111764907837 (2.5167908441452753) | > loss_disc_real_0: 0.09505032002925873 (0.08932829187029884) | > loss_disc_real_1: 0.2488033026456833 (0.21924653578372227) | > loss_disc_real_2: 0.2312796413898468 (0.2312757805699394) | > loss_disc_real_3: 0.2419818490743637 (0.226971692272595) | > loss_disc_real_4: 0.23084336519241333 (0.24103654566265287) | > loss_disc_real_5: 0.24556556344032288 (0.244146357689585) | > loss_0: 2.606111764907837 (2.5167908441452753) | > grad_norm_0: tensor(304.2980, device='cuda:0') (tensor(259.7437, device='cuda:0')) | > loss_gen: 2.050558567047119 (2.2231234425590154) | > loss_kl: 1.729806900024414 (1.8966018926529657) | > loss_feat: 3.4334962368011475 (4.005494923818679) | > loss_mel: 16.100317001342773 (16.886859848385765) | > loss_duration: 1.508237361907959 (1.4940046753202165) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.822416305541992 (26.506084442138672) | > grad_norm_1: tensor(9550.9902, device='cuda:0') (tensor(8151.8931, device='cuda:0')) | > current_lr_0: 0.0001732409481262222 | > current_lr_1: 0.0001732409481262222 | > step_time: 0.4543 (0.4307199092138381) | > loader_time: 0.0069 (0.0067124707358224055)  --> TIME: 2025-02-25 20:03:37 -- STEP: 71/72 -- GLOBAL_STEP: 1082800 | > loss_disc: 2.581794500350952 (2.5779734127958056) | > loss_disc_real_0: 0.2425035685300827 (0.12377629759655874) | > loss_disc_real_1: 0.228087916970253 (0.22615007111723995) | > loss_disc_real_2: 0.2555316388607025 (0.22242373983624955) | > loss_disc_real_3: 0.23162265121936798 (0.2240049314331001) | > loss_disc_real_4: 0.2505207657814026 (0.2375704933639983) | > loss_disc_real_5: 0.21787211298942566 (0.23886100989832004) | > loss_0: 2.581794500350952 (2.5779734127958056) | > grad_norm_0: tensor(578.5735, device='cuda:0') (tensor(269.2123, device='cuda:0')) | > loss_gen: 2.28981876373291 (2.1558097332296238) | > loss_kl: 1.9228386878967285 (1.9632509795712754) | > loss_feat: 3.6984329223632812 (3.8629837976375128) | > loss_mel: 16.995357513427734 (16.76136931567125) | > loss_duration: 1.540665864944458 (1.538690555263573) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.447114944458008 (26.28210414295465) | > grad_norm_1: tensor(10952.4521, device='cuda:0') (tensor(6819.1338, device='cuda:0')) | > current_lr_0: 0.0001732409481262222 | > current_lr_1: 0.0001732409481262222 | > step_time: 0.4822 (0.45285258494632347) | > loader_time: 0.0078 (0.008503440400244483) --> EVAL PERFORMANCE | > avg_loader_time: 0.008503440400244483 (-9.05318998954685e-05) | > avg_step_time: 0.45285258494632347 (-3.599113141994348e-05) | > avg_loss_disc: 2.5779734127958056 (+0.04390599694050579) | > avg_loss_disc_real_0: 0.12377629759655874 (+0.03607360493968913) | > avg_loss_disc_real_1: 0.22615007111723995 (+0.008101272750908217) | > avg_loss_disc_real_2: 0.22242373983624955 (-0.018056695729913835) | > avg_loss_disc_real_3: 0.2240049314331001 (-0.0047314882698193395) | > avg_loss_disc_real_4: 0.2375704933639983 (-1.0010222314116302e-05) | > avg_loss_disc_real_5: 0.23886100989832004 (-0.006303612615021187) | > avg_loss_0: 2.5779734127958056 (+0.04390599694050579) | > avg_grad_norm_0: tensor(269.2123, device='cuda:0') (tensor(-11.7708, device='cuda:0')) | > avg_loss_gen: 2.1558097332296238 (-0.012463010532755892) | > avg_loss_kl: 1.9632509795712754 (-0.006772283097387977) | > avg_loss_feat: 3.8629837976375128 (-0.07645442116428125) | > avg_loss_mel: 16.76136931567125 (-0.11783509858896579) | > avg_loss_duration: 1.538690555263573 (-0.00496818985737546) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.28210414295465 (-0.2184933944487213) | > avg_grad_norm_1: tensor(6819.1338, device='cuda:0') (tensor(-1635.6025, device='cuda:0'))  > EPOCH: 1150/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:03:39)   --> TIME: 2025-02-25 20:04:03 -- STEP: 49/72 -- GLOBAL_STEP: 1082850 | > loss_disc: 2.5058822631835938 (2.7075529195824446) | > loss_disc_real_0: 0.06037759408354759 (0.20935909442451536) | > loss_disc_real_1: 0.21737955510616302 (0.22070476321541532) | > loss_disc_real_2: 0.2255806028842926 (0.22582061086990396) | > loss_disc_real_3: 0.2406775802373886 (0.23396861127444676) | > loss_disc_real_4: 0.24105902016162872 (0.24328429692861986) | > loss_disc_real_5: 0.24079720675945282 (0.24305985229355948) | > loss_0: 2.5058822631835938 (2.7075529195824446) | > grad_norm_0: tensor(165.1231, device='cuda:0') (tensor(280.0562, device='cuda:0')) | > loss_gen: 2.1771247386932373 (2.1262703817717883) | > loss_kl: 2.507031202316284 (2.1015085882070106) | > loss_feat: 4.682240009307861 (3.6502586530179393) | > loss_mel: 20.551025390625 (17.592115791476502) | > loss_duration: 1.678034782409668 (1.5360907851433268) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.595455169677734 (27.006244114467076) | > grad_norm_1: tensor(1975.5630, device='cuda:0') (tensor(6228.6567, device='cuda:0')) | > current_lr_0: 0.0001732192930077064 | > current_lr_1: 0.0001732192930077064 | > step_time: 0.4352 (0.4419892856052944) | > loader_time: 0.0095 (0.007679379716211436) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580177602633623 (+7.67372023891405e-05) | > avg_step_time: 0.45415942433854223 (+0.0013068393922187616) | > avg_loss_disc: 2.677683544830537 (+0.09971013203473156) | > avg_loss_disc_real_0: 0.17058400445106164 (+0.046807706854502903) | > avg_loss_disc_real_1: 0.22478272818343736 (-0.0013673429338025844) | > avg_loss_disc_real_2: 0.21678203270888666 (-0.005641707127362894) | > avg_loss_disc_real_3: 0.24041712021743747 (+0.016412188784337367) | > avg_loss_disc_real_4: 0.25285510823760243 (+0.015284614873604135) | > avg_loss_disc_real_5: 0.24621145909940692 (+0.007350449201086878) | > avg_loss_0: 2.677683544830537 (+0.09971013203473156) | > avg_grad_norm_0: tensor(234.6755, device='cuda:0') (tensor(-34.5368, device='cuda:0')) | > avg_loss_gen: 2.195616107591441 (+0.03980637436181711) | > avg_loss_kl: 2.2114693413318034 (+0.24821836176052803) | > avg_loss_feat: 3.8034967906038526 (-0.059487007033660166) | > avg_loss_mel: 18.074134181922595 (+1.3127648662513458) | > avg_loss_duration: 1.5603957310528822 (+0.021705175789309195) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.845111981244155 (+1.5630078382895043) | > avg_grad_norm_1: tensor(6356.3174, device='cuda:0') (tensor(-462.8164, device='cuda:0'))  > EPOCH: 1151/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:04:16)   --> TIME: 2025-02-25 20:04:30 -- STEP: 27/72 -- GLOBAL_STEP: 1082900 | > loss_disc: 2.857180595397949 (2.6036957016697637) | > loss_disc_real_0: 0.062481313943862915 (0.14968018164789235) | > loss_disc_real_1: 0.2468474805355072 (0.21381816819862084) | > loss_disc_real_2: 0.2316051870584488 (0.22881966222215583) | > loss_disc_real_3: 0.208842933177948 (0.22286512951056162) | > loss_disc_real_4: 0.24525728821754456 (0.23634780115551418) | > loss_disc_real_5: 0.24535205960273743 (0.24076338167543765) | > loss_0: 2.857180595397949 (2.6036957016697637) | > grad_norm_0: tensor(100.4641, device='cuda:0') (tensor(206.3639, device='cuda:0')) | > loss_gen: 1.7906744480133057 (2.260918471548292) | > loss_kl: 1.9504330158233643 (2.0560425784852767) | > loss_feat: 3.6800622940063477 (4.006502098507351) | > loss_mel: 17.644763946533203 (17.448384249651873) | > loss_duration: 1.5054280757904053 (1.5182724308084559) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.571361541748047 (27.290119877568) | > grad_norm_1: tensor(7097.1753, device='cuda:0') (tensor(6923.0835, device='cuda:0')) | > current_lr_0: 0.00017319764059608043 | > current_lr_1: 0.00017319764059608043 | > step_time: 0.4282 (0.43422593893828215) | > loader_time: 0.0083 (0.006806921075891565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679933950934611 (+9.97563483009882e-05) | > avg_step_time: 0.45395191622451997 (-0.000207508114022259) | > avg_loss_disc: 2.620500134750151 (-0.0571834100803863) | > avg_loss_disc_real_0: 0.1532760838072904 (-0.017307920643771235) | > avg_loss_disc_real_1: 0.22068109961462692 (-0.004101628568810439) | > avg_loss_disc_real_2: 0.23540837894862807 (+0.01862634623974141) | > avg_loss_disc_real_3: 0.23036527528729236 (-0.010051844930145104) | > avg_loss_disc_real_4: 0.24204813951337842 (-0.010806968724224014) | > avg_loss_disc_real_5: 0.2450702580767618 (-0.0011412010226451164) | > avg_loss_0: 2.620500134750151 (-0.0571834100803863) | > avg_grad_norm_0: tensor(208.3962, device='cuda:0') (tensor(-26.2794, device='cuda:0')) | > avg_loss_gen: 2.177359849634304 (-0.018256257957137034) | > avg_loss_kl: 2.0773701986796405 (-0.13409914265216294) | > avg_loss_feat: 3.7721320776872234 (-0.031364712916629234) | > avg_loss_mel: 17.064433944057416 (-1.0097002378651787) | > avg_loss_duration: 1.5521343224485156 (-0.008261408604366638) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.643430441198216 (-1.2016815400459393) | > avg_grad_norm_1: tensor(7293.2153, device='cuda:0') (+tensor(936.8979, device='cuda:0'))  > EPOCH: 1152/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:04:53)   --> TIME: 2025-02-25 20:04:57 -- STEP: 5/72 -- GLOBAL_STEP: 1082950 | > loss_disc: 2.4897165298461914 (2.617796468734741) | > loss_disc_real_0: 0.14362779259681702 (0.08375103995203972) | > loss_disc_real_1: 0.21401558816432953 (0.19203705787658693) | > loss_disc_real_2: 0.23661433160305023 (0.23314302861690522) | > loss_disc_real_3: 0.24794542789459229 (0.24062474370002745) | > loss_disc_real_4: 0.2461257129907608 (0.2395373821258545) | > loss_disc_real_5: 0.23939843475818634 (0.24967693388462067) | > loss_0: 2.4897165298461914 (2.617796468734741) | > grad_norm_0: tensor(374.0219, device='cuda:0') (tensor(323.2028, device='cuda:0')) | > loss_gen: 2.4189512729644775 (2.032252883911133) | > loss_kl: 1.7905927896499634 (1.7868432521820068) | > loss_feat: 3.856114149093628 (3.7771223068237303) | > loss_mel: 16.60826873779297 (16.766004180908205) | > loss_duration: 1.4868413209915161 (1.513593888282776) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.160768508911133 (25.875816345214844) | > grad_norm_1: tensor(6538.4922, device='cuda:0') (tensor(8381.8877, device='cuda:0')) | > current_lr_0: 0.0001731759908910059 | > current_lr_1: 0.0001731759908910059 | > step_time: 0.4244 (0.4331823825836182) | > loader_time: 0.0057 (0.006793785095214844)  --> TIME: 2025-02-25 20:05:20 -- STEP: 55/72 -- GLOBAL_STEP: 1083000 | > loss_disc: 2.6228702068328857 (2.6301768346266314) | > loss_disc_real_0: 0.09450455009937286 (0.13451813899657938) | > loss_disc_real_1: 0.21407079696655273 (0.20676659155975688) | > loss_disc_real_2: 0.2249596267938614 (0.23572174988009714) | > loss_disc_real_3: 0.2152751386165619 (0.22751517485488545) | > loss_disc_real_4: 0.2310253232717514 (0.23990524763410742) | > loss_disc_real_5: 0.25010576844215393 (0.24630211374976418) | > loss_0: 2.6228702068328857 (2.6301768346266314) | > grad_norm_0: tensor(352.1469, device='cuda:0') (tensor(204.2066, device='cuda:0')) | > loss_gen: 2.077587127685547 (2.132089376449585) | > loss_kl: 2.232118844985962 (2.0664336833086883) | > loss_feat: 3.97548246383667 (3.8643174691633746) | > loss_mel: 16.325958251953125 (17.215226294777615) | > loss_duration: 1.6954503059387207 (1.552242515303872) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.306596755981445 (26.830309434370562) | > grad_norm_1: tensor(9984.6514, device='cuda:0') (tensor(6738.1543, device='cuda:0')) | > current_lr_0: 0.0001731759908910059 | > current_lr_1: 0.0001731759908910059 | > step_time: 0.4687 (0.44497772997075863) | > loader_time: 0.0096 (0.008014674620194867) --> EVAL PERFORMANCE | > avg_loader_time: 0.008681606238996475 (+1.6722880618637304e-06) | > avg_step_time: 0.4545976309709146 (+0.0006457147463946322) | > avg_loss_disc: 2.6466782496009076 (+0.026178114850756717) | > avg_loss_disc_real_0: 0.15815126848682548 (+0.004875184679535072) | > avg_loss_disc_real_1: 0.21020931152390762 (-0.010471788090719303) | > avg_loss_disc_real_2: 0.233660660998922 (-0.0017477179497060735) | > avg_loss_disc_real_3: 0.2283882198619171 (-0.001977055425375268) | > avg_loss_disc_real_4: 0.23934470180054784 (-0.0027034377128305753) | > avg_loss_disc_real_5: 0.24562199497726603 (+0.0005517369005042239) | > avg_loss_0: 2.6466782496009076 (+0.026178114850756717) | > avg_grad_norm_0: tensor(225.3384, device='cuda:0') (+tensor(16.9422, device='cuda:0')) | > avg_loss_gen: 2.144105152345039 (-0.03325469728926489) | > avg_loss_kl: 2.0944225586635965 (+0.017052359983956045) | > avg_loss_feat: 3.769736797037259 (-0.0023952806499645263) | > avg_loss_mel: 17.219858760565106 (+0.1554248165076899) | > avg_loss_duration: 1.5683342319139293 (+0.01619990946541372) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.796457532426 (+0.15302709122778424) | > avg_grad_norm_1: tensor(6614.7280, device='cuda:0') (tensor(-678.4873, device='cuda:0'))  > EPOCH: 1153/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:05:30)   --> TIME: 2025-02-25 20:05:47 -- STEP: 33/72 -- GLOBAL_STEP: 1083050 | > loss_disc: 2.4233243465423584 (2.6736487330812397) | > loss_disc_real_0: 0.14919239282608032 (0.12947263455752164) | > loss_disc_real_1: 0.2378932237625122 (0.23270589926026078) | > loss_disc_real_2: 0.24214601516723633 (0.239277080152974) | > loss_disc_real_3: 0.2637064754962921 (0.23299647867679596) | > loss_disc_real_4: 0.24035154283046722 (0.25120556219057605) | > loss_disc_real_5: 0.2505491375923157 (0.24888370750528394) | > loss_0: 2.4233243465423584 (2.6736487330812397) | > grad_norm_0: tensor(41.1382, device='cuda:0') (tensor(275.7610, device='cuda:0')) | > loss_gen: 2.680129289627075 (2.0319208267963296) | > loss_kl: 2.15811824798584 (1.9980387181946726) | > loss_feat: 4.277626037597656 (3.695271853244666) | > loss_mel: 17.556140899658203 (17.13643386147239) | > loss_duration: 1.5208890438079834 (1.5099761847293738) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.19290542602539 (26.371641448049836) | > grad_norm_1: tensor(2945.3660, device='cuda:0') (tensor(6174.2197, device='cuda:0')) | > current_lr_0: 0.0001731543438921445 | > current_lr_1: 0.0001731543438921445 | > step_time: 0.4406 (0.43990453084309894) | > loader_time: 0.0083 (0.007211576808582653) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624083559278031 (-5.752267971844381e-05) | > avg_step_time: 0.45670601683603207 (+0.002108385865117468) | > avg_loss_disc: 2.6953811947728537 (+0.048702945171946066) | > avg_loss_disc_real_0: 0.14665436555801975 (-0.01149690292880573) | > avg_loss_disc_real_1: 0.22675990365760426 (+0.01655059213369664) | > avg_loss_disc_real_2: 0.24748360942786848 (+0.013822948428946491) | > avg_loss_disc_real_3: 0.2356420639954822 (+0.007253844133565113) | > avg_loss_disc_real_4: 0.24607774447387373 (+0.00673304267332589) | > avg_loss_disc_real_5: 0.24559635133810445 (-2.5643639161576637e-05) | > avg_loss_0: 2.6953811947728537 (+0.048702945171946066) | > avg_grad_norm_0: tensor(260.1437, device='cuda:0') (+tensor(34.8053, device='cuda:0')) | > avg_loss_gen: 2.0172094173834347 (-0.12689573496160422) | > avg_loss_kl: 2.043796045679441 (-0.050626512984155436) | > avg_loss_feat: 3.576173396177695 (-0.19356340085956392) | > avg_loss_mel: 16.93916653915191 (-0.2806922214131973) | > avg_loss_duration: 1.546103769624737 (-0.022230462289192232) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.122449203276297 (-0.6740083291497037) | > avg_grad_norm_1: tensor(6309.5205, device='cuda:0') (tensor(-305.2075, device='cuda:0'))  > EPOCH: 1154/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:06:07)   --> TIME: 2025-02-25 20:06:14 -- STEP: 11/72 -- GLOBAL_STEP: 1083100 | > loss_disc: 2.5864882469177246 (2.5993723435835405) | > loss_disc_real_0: 0.05633467435836792 (0.11662175811149857) | > loss_disc_real_1: 0.21565864980220795 (0.22093013470823114) | > loss_disc_real_2: 0.21678166091442108 (0.2245252403345975) | > loss_disc_real_3: 0.22742284834384918 (0.2212739735841751) | > loss_disc_real_4: 0.23237229883670807 (0.24747362055561758) | > loss_disc_real_5: 0.2394389659166336 (0.25130903856320813) | > loss_0: 2.5864882469177246 (2.5993723435835405) | > grad_norm_0: tensor(144.5558, device='cuda:0') (tensor(177.5220, device='cuda:0')) | > loss_gen: 1.9506781101226807 (2.112719329920682) | > loss_kl: 2.924997568130493 (2.8794675306840376) | > loss_feat: 3.6725237369537354 (3.920756773515181) | > loss_mel: 20.757736206054688 (20.973132740367543) | > loss_duration: 1.5646531581878662 (1.5921945355155251) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.870588302612305 (31.47827113758434) | > grad_norm_1: tensor(7403.5439, device='cuda:0') (tensor(8483.6426, device='cuda:0')) | > current_lr_0: 0.00017313269959915798 | > current_lr_1: 0.00017313269959915798 | > step_time: 0.4209 (0.4307122230529785) | > loader_time: 0.0061 (0.006235707889903675)  --> TIME: 2025-02-25 20:06:37 -- STEP: 61/72 -- GLOBAL_STEP: 1083150 | > loss_disc: 2.4557759761810303 (2.570297080962384) | > loss_disc_real_0: 0.08969688415527344 (0.08416373820089904) | > loss_disc_real_1: 0.2292046695947647 (0.2418372469847319) | > loss_disc_real_2: 0.2697717249393463 (0.23726694652291594) | > loss_disc_real_3: 0.25602108240127563 (0.23892344070262597) | > loss_disc_real_4: 0.2600868344306946 (0.24097553359680488) | > loss_disc_real_5: 0.2248603105545044 (0.2434494654174711) | > loss_0: 2.4557759761810303 (2.570297080962384) | > grad_norm_0: tensor(153.4737, device='cuda:0') (tensor(226.2123, device='cuda:0')) | > loss_gen: 2.2665114402770996 (2.1570948635945553) | > loss_kl: 2.282050848007202 (2.4193953334308063) | > loss_feat: 3.9253084659576416 (3.9559504008684003) | > loss_mel: 16.169803619384766 (18.70989566552834) | > loss_duration: 1.5955824851989746 (1.574673318472065) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.2392578125 (28.817009691332206) | > grad_norm_1: tensor(8969.2744, device='cuda:0') (tensor(8409.2686, device='cuda:0')) | > current_lr_0: 0.00017313269959915798 | > current_lr_1: 0.00017313269959915798 | > step_time: 0.4525 (0.44471153665761476) | > loader_time: 0.0109 (0.008227438223166544) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687721171849211 (+6.363761257117961e-05) | > avg_step_time: 0.45452094749665595 (-0.0021850693393761222) | > avg_loss_disc: 2.56772037291191 (-0.12766082186094385) | > avg_loss_disc_real_0: 0.08398005932989255 (-0.0626743062281272) | > avg_loss_disc_real_1: 0.23974350703434205 (+0.01298360337673779) | > avg_loss_disc_real_2: 0.23764511795950607 (-0.00983849146836241) | > avg_loss_disc_real_3: 0.2378690889183904 (+0.0022270249229081884) | > avg_loss_disc_real_4: 0.24139083145369947 (-0.004686913020174266) | > avg_loss_disc_real_5: 0.2437214584837497 (-0.0018748928543547427) | > avg_loss_0: 2.56772037291191 (-0.12766082186094385) | > avg_grad_norm_0: tensor(232.6577, device='cuda:0') (tensor(-27.4860, device='cuda:0')) | > avg_loss_gen: 2.1574619168966587 (+0.14025249951322394) | > avg_loss_kl: 2.389527455182143 (+0.34573140950270176) | > avg_loss_feat: 3.9412944786985156 (+0.36512108252082065) | > avg_loss_mel: 18.47601409697197 (+1.5368475578200602) | > avg_loss_duration: 1.5797978706762823 (+0.033694101051545244) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.544095858721665 (+2.421646655445368) | > avg_grad_norm_1: tensor(8570.6162, device='cuda:0') (+tensor(2261.0957, device='cuda:0'))  > EPOCH: 1155/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:06:44)   --> TIME: 2025-02-25 20:07:03 -- STEP: 39/72 -- GLOBAL_STEP: 1083200 | > loss_disc: 2.446363925933838 (2.5089462231367063) | > loss_disc_real_0: 0.05033724009990692 (0.07674555604656537) | > loss_disc_real_1: 0.17120467126369476 (0.21411521006853151) | > loss_disc_real_2: 0.21849796175956726 (0.23668939945025322) | > loss_disc_real_3: 0.21853205561637878 (0.23105200819480112) | > loss_disc_real_4: 0.23710894584655762 (0.2421919550651159) | > loss_disc_real_5: 0.22735582292079926 (0.2452063025572361) | > loss_0: 2.446363925933838 (2.5089462231367063) | > grad_norm_0: tensor(298.3159, device='cuda:0') (tensor(299.4606, device='cuda:0')) | > loss_gen: 2.019747734069824 (2.208111276993385) | > loss_kl: 1.9545904397964478 (1.991079785884955) | > loss_feat: 4.099796295166016 (4.169511581078553) | > loss_mel: 17.82599639892578 (17.14701549823468) | > loss_duration: 1.5367244482040405 (1.5138261776704054) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.43685531616211 (27.029544243445763) | > grad_norm_1: tensor(13087.6973, device='cuda:0') (tensor(9890.3477, device='cuda:0')) | > current_lr_0: 0.00017311105801170808 | > current_lr_1: 0.00017311105801170808 | > step_time: 0.4568 (0.437982229086069) | > loader_time: 0.0087 (0.007740497589111328) --> EVAL PERFORMANCE | > avg_loader_time: 0.008833640058275681 (+0.0001459188864264705) | > avg_step_time: 0.45438408851623535 (-0.00013685898042059552) | > avg_loss_disc: 2.515109941992961 (-0.052610430918949014) | > avg_loss_disc_real_0: 0.07989320694140983 (-0.00408685238848272) | > avg_loss_disc_real_1: 0.21715485545950877 (-0.02258865157483328) | > avg_loss_disc_real_2: 0.2375301267479507 (-0.00011499121155536529) | > avg_loss_disc_real_3: 0.2317697607295614 (-0.006099328188828995) | > avg_loss_disc_real_4: 0.24390638840030615 (+0.002515556946606684) | > avg_loss_disc_real_5: 0.24648259706060652 (+0.0027611385768568086) | > avg_loss_0: 2.515109941992961 (-0.052610430918949014) | > avg_grad_norm_0: tensor(288.0691, device='cuda:0') (+tensor(55.4114, device='cuda:0')) | > avg_loss_gen: 2.2174197603279446 (+0.05995784343128596) | > avg_loss_kl: 2.0176916458237337 (-0.3718358093584091) | > avg_loss_feat: 4.113326861824788 (+0.17203238312627267) | > avg_loss_mel: 17.030486926226555 (-1.4455271707454145) | > avg_loss_duration: 1.5444182510107336 (-0.0353796196655487) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.92334341667068 (-1.6207524420509856) | > avg_grad_norm_1: tensor(9598.1152, device='cuda:0') (+tensor(1027.4990, device='cuda:0'))  > EPOCH: 1156/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:07:21)   --> TIME: 2025-02-25 20:07:31 -- STEP: 17/72 -- GLOBAL_STEP: 1083250 | > loss_disc: 2.563466787338257 (2.496420046862434) | > loss_disc_real_0: 0.0619678720831871 (0.08264134364093051) | > loss_disc_real_1: 0.2393975406885147 (0.21886987633564892) | > loss_disc_real_2: 0.2338477224111557 (0.2315231517833822) | > loss_disc_real_3: 0.23353956639766693 (0.22809757905847886) | > loss_disc_real_4: 0.24611084163188934 (0.24128537405939662) | > loss_disc_real_5: 0.25597962737083435 (0.24188772019217997) | > loss_0: 2.563466787338257 (2.496420046862434) | > grad_norm_0: tensor(267.5719, device='cuda:0') (tensor(278.4559, device='cuda:0')) | > loss_gen: 2.19197940826416 (2.2491252422332764) | > loss_kl: 2.093355178833008 (1.88246250853819) | > loss_feat: 3.9273579120635986 (4.179116824093987) | > loss_mel: 16.660640716552734 (16.727444031659296) | > loss_duration: 1.513380765914917 (1.4926594495773315) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.3867130279541 (26.530808280496036) | > grad_norm_1: tensor(10165.0352, device='cuda:0') (tensor(8727.9756, device='cuda:0')) | > current_lr_0: 0.00017308941912945661 | > current_lr_1: 0.00017308941912945661 | > step_time: 0.4252 (0.44068088251001697) | > loader_time: 0.006 (0.006790918462416705)  --> TIME: 2025-02-25 20:07:54 -- STEP: 67/72 -- GLOBAL_STEP: 1083300 | > loss_disc: 2.588179111480713 (2.5180837289610905) | > loss_disc_real_0: 0.12506164610385895 (0.08977664701307) | > loss_disc_real_1: 0.2349080890417099 (0.21887416381444505) | > loss_disc_real_2: 0.2502347528934479 (0.23212310938692804) | > loss_disc_real_3: 0.2428635209798813 (0.23237679520649696) | > loss_disc_real_4: 0.2587907910346985 (0.2419593654461761) | > loss_disc_real_5: 0.25102952122688293 (0.24691196831304635) | > loss_0: 2.588179111480713 (2.5180837289610905) | > grad_norm_0: tensor(434.6494, device='cuda:0') (tensor(285.9172, device='cuda:0')) | > loss_gen: 2.254065990447998 (2.2333983912396778) | > loss_kl: 1.9654567241668701 (1.9663025194139623) | > loss_feat: 3.8400588035583496 (4.068831810310705) | > loss_mel: 16.91930389404297 (16.707598600814592) | > loss_duration: 1.6386609077453613 (1.5361046026002114) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.61754608154297 (26.512236040029954) | > grad_norm_1: tensor(6573.0420, device='cuda:0') (tensor(8394.1094, device='cuda:0')) | > current_lr_0: 0.00017308941912945661 | > current_lr_1: 0.00017308941912945661 | > step_time: 0.5082 (0.45092129707336426) | > loader_time: 0.0122 (0.008451358595890785) --> EVAL PERFORMANCE | > avg_loader_time: 0.008621756459625674 (-0.00021188359865000755) | > avg_step_time: 0.45442354511207256 (+3.9456595837206176e-05) | > avg_loss_disc: 2.5172074378376275 (+0.0020974958446666747) | > avg_loss_disc_real_0: 0.08846167553688442 (+0.008568468595474582) | > avg_loss_disc_real_1: 0.21910093451889467 (+0.0019460790593859023) | > avg_loss_disc_real_2: 0.23207170086007722 (-0.005458425887873486) | > avg_loss_disc_real_3: 0.2326351588880512 (+0.0008653981584897941) | > avg_loss_disc_real_4: 0.2419722166279672 (-0.001934171772338955) | > avg_loss_disc_real_5: 0.24732482055543173 (+0.0008422234948252116) | > avg_loss_0: 2.5172074378376275 (+0.0020974958446666747) | > avg_grad_norm_0: tensor(282.3300, device='cuda:0') (tensor(-5.7391, device='cuda:0')) | > avg_loss_gen: 2.235708276990433 (+0.01828851666248843) | > avg_loss_kl: 1.971810604484988 (-0.04588104133874582) | > avg_loss_feat: 4.069909408058919 (-0.04341745376586914) | > avg_loss_mel: 16.71620397164788 (-0.31428295457867605) | > avg_loss_duration: 1.5396111750266919 (-0.004807075984041731) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.533243555418203 (-0.39009986125247664) | > avg_grad_norm_1: tensor(8516.0430, device='cuda:0') (tensor(-1082.0723, device='cuda:0'))  > EPOCH: 1157/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:07:58)   --> TIME: 2025-02-25 20:08:20 -- STEP: 45/72 -- GLOBAL_STEP: 1083350 | > loss_disc: 2.5281498432159424 (2.5309950192769373) | > loss_disc_real_0: 0.11269994080066681 (0.10202247608039114) | > loss_disc_real_1: 0.19401638209819794 (0.2193072971370485) | > loss_disc_real_2: 0.2450384646654129 (0.2268042696846856) | > loss_disc_real_3: 0.2036600410938263 (0.23020041684309642) | > loss_disc_real_4: 0.22069837152957916 (0.23762015832795036) | > loss_disc_real_5: 0.2554645240306854 (0.2433012263642417) | > loss_0: 2.5281498432159424 (2.5309950192769373) | > grad_norm_0: tensor(369.0573, device='cuda:0') (tensor(295.7099, device='cuda:0')) | > loss_gen: 2.1350502967834473 (2.22426766819424) | > loss_kl: 2.060572385787964 (1.9014434999889798) | > loss_feat: 3.9856929779052734 (4.057989607916936) | > loss_mel: 17.18613624572754 (16.652195760938856) | > loss_duration: 1.5525000095367432 (1.5087182389365301) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.919952392578125 (26.344614834255644) | > grad_norm_1: tensor(7716.3330, device='cuda:0') (tensor(7932.3022, device='cuda:0')) | > current_lr_0: 0.00017306778295206543 | > current_lr_1: 0.00017306778295206543 | > step_time: 0.4604 (0.4405475510491265) | > loader_time: 0.0088 (0.007518264982435439) --> EVAL PERFORMANCE | > avg_loader_time: 0.008619590544364822 (-2.1659152608516363e-06) | > avg_step_time: 0.4536888599395752 (-0.0007346851724973624) | > avg_loss_disc: 2.5312542747443834 (+0.0140468369067559) | > avg_loss_disc_real_0: 0.09990439438064333 (+0.011442718843758914) | > avg_loss_disc_real_1: 0.21912509916533887 (+2.4164646444196913e-05) | > avg_loss_disc_real_2: 0.22976952917139296 (-0.002302171688684268) | > avg_loss_disc_real_3: 0.23045119013584836 (-0.002183968752202836) | > avg_loss_disc_real_4: 0.2392895479857082 (-0.002682668642258995) | > avg_loss_disc_real_5: 0.24428836686510436 (-0.003036453690327373) | > avg_loss_0: 2.5312542747443834 (+0.0140468369067559) | > avg_grad_norm_0: tensor(276.2836, device='cuda:0') (tensor(-6.0464, device='cuda:0')) | > avg_loss_gen: 2.2296273036741865 (-0.006080973316246574) | > avg_loss_kl: 1.9539396007296066 (-0.01787100375538131) | > avg_loss_feat: 4.03442894572943 (-0.03548046232948909) | > avg_loss_mel: 16.698709058089996 (-0.017494913557882796) | > avg_loss_duration: 1.536188562151412 (-0.0034226128752798957) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.452893566077865 (-0.08034998934033766) | > avg_grad_norm_1: tensor(7518.2964, device='cuda:0') (tensor(-997.7466, device='cuda:0'))  > EPOCH: 1158/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:08:35)   --> TIME: 2025-02-25 20:08:47 -- STEP: 23/72 -- GLOBAL_STEP: 1083400 | > loss_disc: 2.4994451999664307 (2.5166421143904976) | > loss_disc_real_0: 0.05946071445941925 (0.09051044285297394) | > loss_disc_real_1: 0.22233423590660095 (0.22517477299856103) | > loss_disc_real_2: 0.2454284280538559 (0.2275354266166687) | > loss_disc_real_3: 0.2468176782131195 (0.23773379623889923) | > loss_disc_real_4: 0.266396701335907 (0.2430256812468819) | > loss_disc_real_5: 0.2341250628232956 (0.24666908836883047) | > loss_0: 2.4994451999664307 (2.5166421143904976) | > grad_norm_0: tensor(223.7098, device='cuda:0') (tensor(256.9219, device='cuda:0')) | > loss_gen: 2.3206143379211426 (2.2425798229549243) | > loss_kl: 1.9570791721343994 (1.906375677689262) | > loss_feat: 4.204758167266846 (4.1186867485875664) | > loss_mel: 16.848819732666016 (16.89726058296535) | > loss_duration: 1.5313130617141724 (1.4964152470878933) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.862585067749023 (26.661317908245586) | > grad_norm_1: tensor(9564.0596, device='cuda:0') (tensor(7725.9648, device='cuda:0')) | > current_lr_0: 0.0001730461494791964 | > current_lr_1: 0.0001730461494791964 | > step_time: 0.4356 (0.43317337658094324) | > loader_time: 0.0064 (0.00664067268371582) --> EVAL PERFORMANCE | > avg_loader_time: 0.008637401419626151 (+1.781087526132874e-05) | > avg_step_time: 0.4537088837422116 (+2.0023802636381927e-05) | > avg_loss_disc: 2.5251842351026936 (-0.006070039641689728) | > avg_loss_disc_real_0: 0.08913951207109742 (-0.010764882309545906) | > avg_loss_disc_real_1: 0.22126629822690722 (+0.002141199061568344) | > avg_loss_disc_real_2: 0.2339402474987675 (+0.004170718327374534) | > avg_loss_disc_real_3: 0.23405145486475717 (+0.003600264728908814) | > avg_loss_disc_real_4: 0.24219664406608526 (+0.002907096080377064) | > avg_loss_disc_real_5: 0.24551563321704595 (+0.0012272663519415916) | > avg_loss_0: 2.5251842351026936 (-0.006070039641689728) | > avg_grad_norm_0: tensor(274.8739, device='cuda:0') (tensor(-1.4097, device='cuda:0')) | > avg_loss_gen: 2.2205989058588598 (-0.009028397815326716) | > avg_loss_kl: 1.9957690574753453 (+0.04182945674573868) | > avg_loss_feat: 4.0367340840084465 (+0.002305138279016461) | > avg_loss_mel: 16.74961134413598 (+0.05090228604598579) | > avg_loss_duration: 1.537426231612622 (+0.001237669461209956) | > avg_amp_scaler: 9.23943661971831 (+1.23943661971831) | > avg_loss_1: 26.54013942664778 (+0.08724586056991512) | > avg_grad_norm_1: tensor(8515.1045, device='cuda:0') (+tensor(996.8081, device='cuda:0'))  > EPOCH: 1159/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:09:12)   --> TIME: 2025-02-25 20:09:14 -- STEP: 1/72 -- GLOBAL_STEP: 1083450 | > loss_disc: 2.640688896179199 (2.640688896179199) | > loss_disc_real_0: 0.29586896300315857 (0.29586896300315857) | > loss_disc_real_1: 0.24265488982200623 (0.24265488982200623) | > loss_disc_real_2: 0.247284397482872 (0.247284397482872) | > loss_disc_real_3: 0.21658620238304138 (0.21658620238304138) | > loss_disc_real_4: 0.259676069021225 (0.259676069021225) | > loss_disc_real_5: 0.24822643399238586 (0.24822643399238586) | > loss_0: 2.640688896179199 (2.640688896179199) | > grad_norm_0: tensor(582.6446, device='cuda:0') (tensor(582.6446, device='cuda:0')) | > loss_gen: 2.56282377243042 (2.56282377243042) | > loss_kl: 1.9323148727416992 (1.9323148727416992) | > loss_feat: 3.6660733222961426 (3.6660733222961426) | > loss_mel: 18.59052276611328 (18.59052276611328) | > loss_duration: 1.4950082302093506 (1.4950082302093506) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.24674415588379 (28.24674415588379) | > grad_norm_1: tensor(3741.7341, device='cuda:0') (tensor(3741.7341, device='cuda:0')) | > current_lr_0: 0.0001730245187105115 | > current_lr_1: 0.0001730245187105115 | > step_time: 0.4123 (0.41225314140319824) | > loader_time: 0.0051 (0.0050961971282958984)  --> TIME: 2025-02-25 20:09:36 -- STEP: 51/72 -- GLOBAL_STEP: 1083500 | > loss_disc: 2.670412540435791 (2.6916629894106983) | > loss_disc_real_0: 0.0518207848072052 (0.14888162742934977) | > loss_disc_real_1: 0.46825045347213745 (0.22814150256853477) | > loss_disc_real_2: 0.20408204197883606 (0.22850159979334064) | > loss_disc_real_3: 0.3636508285999298 (0.2243127781970828) | > loss_disc_real_4: 0.27035072445869446 (0.2308327462158951) | > loss_disc_real_5: 0.24328701198101044 (0.23921467042436786) | > loss_0: 2.670412540435791 (2.6916629894106983) | > grad_norm_0: tensor(58.2563, device='cuda:0') (tensor(275.5803, device='cuda:0')) | > loss_gen: 2.5506467819213867 (2.1044378420885876) | > loss_kl: 2.1453421115875244 (2.1511440043355914) | > loss_feat: 3.977445602416992 (3.86467028131672) | > loss_mel: 15.875534057617188 (17.339300791422534) | > loss_duration: 1.6524662971496582 (1.5818956880008472) | > amp_scaler: 8.0 (11.92156862745098) | > loss_1: 26.201435089111328 (27.04144874273562) | > grad_norm_1: tensor(1681.4801, device='cuda:0') (tensor(6573.7373, device='cuda:0')) | > current_lr_0: 0.0001730245187105115 | > current_lr_1: 0.0001730245187105115 | > step_time: 0.4355 (0.4373180772743973) | > loader_time: 0.009 (0.007654311610203164) --> EVAL PERFORMANCE | > avg_loader_time: 0.008475736832954517 (-0.00016166458667163375) | > avg_step_time: 0.44809656076028315 (-0.00561232298192843) | > avg_loss_disc: 2.679121091332233 (+0.15393685622953956) | > avg_loss_disc_real_0: 0.13121743206406986 (+0.04207791999297243) | > avg_loss_disc_real_1: 0.2201935964780794 (-0.0010727017488278145) | > avg_loss_disc_real_2: 0.24102874498971752 (+0.00708849749095003) | > avg_loss_disc_real_3: 0.22453378812527994 (-0.009517666739477237) | > avg_loss_disc_real_4: 0.2353926248533625 (-0.006804019212722778) | > avg_loss_disc_real_5: 0.24352849432280366 (-0.0019871388942422907) | > avg_loss_0: 2.679121091332233 (+0.15393685622953956) | > avg_grad_norm_0: tensor(271.4040, device='cuda:0') (tensor(-3.4699, device='cuda:0')) | > avg_loss_gen: 2.09878231438113 (-0.12181659147772983) | > avg_loss_kl: 2.1728350377418635 (+0.1770659802665182) | > avg_loss_feat: 3.84790880579344 (-0.1888252782150066) | > avg_loss_mel: 17.3709717602797 (+0.6213604161437196) | > avg_loss_duration: 1.5952249882926404 (+0.05779875668001844) | > avg_amp_scaler: 10.816901408450704 (+1.577464788732394) | > avg_loss_1: 27.085723017303035 (+0.5455835906552551) | > avg_grad_norm_1: tensor(7716.7139, device='cuda:0') (tensor(-798.3906, device='cuda:0'))  > EPOCH: 1160/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:09:48)   --> TIME: 2025-02-25 20:10:03 -- STEP: 29/72 -- GLOBAL_STEP: 1083550 | > loss_disc: 2.6583292484283447 (2.555856170325443) | > loss_disc_real_0: 0.12528231739997864 (0.08901401968865559) | > loss_disc_real_1: 0.23418210446834564 (0.23672218106943985) | > loss_disc_real_2: 0.2533690631389618 (0.2297011151395995) | > loss_disc_real_3: 0.23355066776275635 (0.229921480191165) | > loss_disc_real_4: 0.2571653425693512 (0.24503925442695618) | > loss_disc_real_5: 0.2590174674987793 (0.243506854978101) | > loss_0: 2.6583292484283447 (2.555856170325443) | > grad_norm_0: tensor(633.9916, device='cuda:0') (tensor(324.4344, device='cuda:0')) | > loss_gen: 2.0068533420562744 (2.1446853876113896) | > loss_kl: 2.115813732147217 (1.9131647553937188) | > loss_feat: 3.2708215713500977 (3.9194158767831735) | > loss_mel: 17.60707664489746 (17.156437413445833) | > loss_duration: 1.5147879123687744 (1.5072693372594899) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.515352249145508 (26.640972729386956) | > grad_norm_1: tensor(11175.6660, device='cuda:0') (tensor(10107.7930, device='cuda:0')) | > current_lr_0: 0.00017300289064567267 | > current_lr_1: 0.00017300289064567267 | > step_time: 0.445 (0.4335499056454363) | > loader_time: 0.0073 (0.00720236219208816) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647572826331773 (+0.00017183599337725584) | > avg_step_time: 0.4521916282009071 (+0.004095067440623934) | > avg_loss_disc: 2.546652249886956 (-0.13246884144527726) | > avg_loss_disc_real_0: 0.08731316014284819 (-0.04390427192122166) | > avg_loss_disc_real_1: 0.22407887414307662 (+0.0038852776649972165) | > avg_loss_disc_real_2: 0.23631004548408616 (-0.004718699505631363) | > avg_loss_disc_real_3: 0.2324917236684074 (+0.007957935543127459) | > avg_loss_disc_real_4: 0.24447959934321928 (+0.009086974489856792) | > avg_loss_disc_real_5: 0.243591662653735 (+6.316833093134022e-05) | > avg_loss_0: 2.546652249886956 (-0.13246884144527726) | > avg_grad_norm_0: tensor(320.9767, device='cuda:0') (+tensor(49.5727, device='cuda:0')) | > avg_loss_gen: 2.156538647665105 (+0.057756333283975) | > avg_loss_kl: 1.9676199244781278 (-0.20521511326373565) | > avg_loss_feat: 3.9355402865879974 (+0.08763148079455751) | > avg_loss_mel: 16.848315238952633 (-0.5226565213270682) | > avg_loss_duration: 1.5450508510562735 (-0.05017413723636688) | > avg_amp_scaler: 8.0 (-2.816901408450704) | > avg_loss_1: 26.4530649319501 (-0.6326580853529364) | > avg_grad_norm_1: tensor(9796.5293, device='cuda:0') (+tensor(2079.8154, device='cuda:0'))  > EPOCH: 1161/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:10:25)   --> TIME: 2025-02-25 20:10:30 -- STEP: 7/72 -- GLOBAL_STEP: 1083600 | > loss_disc: 2.4447946548461914 (2.4844610691070557) | > loss_disc_real_0: 0.09019607305526733 (0.10140731877514295) | > loss_disc_real_1: 0.22669531404972076 (0.20726831895964487) | > loss_disc_real_2: 0.24740523099899292 (0.23120691733700888) | > loss_disc_real_3: 0.24628973007202148 (0.22471352560179575) | > loss_disc_real_4: 0.27356961369514465 (0.2515201951776232) | > loss_disc_real_5: 0.2540868818759918 (0.24843284913471766) | > loss_0: 2.4447946548461914 (2.4844610691070557) | > grad_norm_0: tensor(155.7616, device='cuda:0') (tensor(297.4943, device='cuda:0')) | > loss_gen: 2.4856722354888916 (2.3331617968423024) | > loss_kl: 1.9317519664764404 (1.8418990203312464) | > loss_feat: 4.247568130493164 (4.163110528673444) | > loss_mel: 17.2877254486084 (17.082862036568777) | > loss_duration: 1.467113971710205 (1.4959023339407784) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.419830322265625 (26.916935239519393) | > grad_norm_1: tensor(4827.0996, device='cuda:0') (tensor(7333.0586, device='cuda:0')) | > current_lr_0: 0.00017298126528434197 | > current_lr_1: 0.00017298126528434197 | > step_time: 0.4469 (0.43022118295942036) | > loader_time: 0.0067 (0.006235497338431222)  --> TIME: 2025-02-25 20:10:53 -- STEP: 57/72 -- GLOBAL_STEP: 1083650 | > loss_disc: 2.469247341156006 (2.5400294504667595) | > loss_disc_real_0: 0.052937574684619904 (0.09996227645560314) | > loss_disc_real_1: 0.2195259928703308 (0.2181931856954307) | > loss_disc_real_2: 0.20111772418022156 (0.22950696056349235) | > loss_disc_real_3: 0.190035879611969 (0.23280365372958936) | > loss_disc_real_4: 0.20559489727020264 (0.24342430448322966) | > loss_disc_real_5: 0.24083302915096283 (0.24665966028707076) | > loss_0: 2.469247341156006 (2.5400294504667595) | > grad_norm_0: tensor(231.0834, device='cuda:0') (tensor(291.9657, device='cuda:0')) | > loss_gen: 2.073141098022461 (2.2073296371259192) | > loss_kl: 2.0205883979797363 (1.9739808346095837) | > loss_feat: 4.230712413787842 (3.9920363300725037) | > loss_mel: 16.42634391784668 (16.678585671541985) | > loss_duration: 1.57821524143219 (1.5233205912405985) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.32900047302246 (26.375253175434313) | > grad_norm_1: tensor(8253.4385, device='cuda:0') (tensor(7340.4424, device='cuda:0')) | > current_lr_0: 0.00017298126528434197 | > current_lr_1: 0.00017298126528434197 | > step_time: 0.4596 (0.4427581502680193) | > loader_time: 0.0099 (0.007902919200428743) --> EVAL PERFORMANCE | > avg_loader_time: 0.008541097103709903 (-0.0001064757226218703) | > avg_step_time: 0.4530867321390501 (+0.0008951039381430159) | > avg_loss_disc: 2.569080463597473 (+0.022428213710516953) | > avg_loss_disc_real_0: 0.12648430157085538 (+0.03917114142800719) | > avg_loss_disc_real_1: 0.2251114658486675 (+0.0010325917055908906) | > avg_loss_disc_real_2: 0.23341131965878983 (-0.002898725825296322) | > avg_loss_disc_real_3: 0.23606893944908197 (+0.0035772157806745752) | > avg_loss_disc_real_4: 0.2481774924926355 (+0.0036978931494162093) | > avg_loss_disc_real_5: 0.24925043360448224 (+0.005658770950747238) | > avg_loss_0: 2.569080463597473 (+0.022428213710516953) | > avg_grad_norm_0: tensor(303.3944, device='cuda:0') (tensor(-17.5822, device='cuda:0')) | > avg_loss_gen: 2.2506563932123322 (+0.0941177455472273) | > avg_loss_kl: 2.003689809584282 (+0.036069885106154054) | > avg_loss_feat: 3.9384073573099054 (+0.0028670707219080604) | > avg_loss_mel: 16.73525303854069 (-0.11306220041194237) | > avg_loss_duration: 1.5366084995404097 (-0.00844235151586381) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.464615190532843 (+0.011550258582744277) | > avg_grad_norm_1: tensor(7132.1279, device='cuda:0') (tensor(-2664.4014, device='cuda:0'))  > EPOCH: 1162/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:11:02)   --> TIME: 2025-02-25 20:11:19 -- STEP: 35/72 -- GLOBAL_STEP: 1083700 | > loss_disc: 2.502392053604126 (2.5707559381212506) | > loss_disc_real_0: 0.1349547952413559 (0.09979333845632417) | > loss_disc_real_1: 0.2880307734012604 (0.21811219645398003) | > loss_disc_real_2: 0.24680852890014648 (0.23583626193659646) | > loss_disc_real_3: 0.25001880526542664 (0.22666916847229004) | > loss_disc_real_4: 0.26822343468666077 (0.24755968110901969) | > loss_disc_real_5: 0.25262799859046936 (0.24546150181974682) | > loss_0: 2.502392053604126 (2.5707559381212506) | > grad_norm_0: tensor(329.3052, device='cuda:0') (tensor(336.1961, device='cuda:0')) | > loss_gen: 2.472942590713501 (2.1562313216073172) | > loss_kl: 2.1443614959716797 (1.942115340914045) | > loss_feat: 4.003458023071289 (3.924162857873099) | > loss_mel: 15.950634002685547 (16.703694752284466) | > loss_duration: 1.538347840309143 (1.4980768850871495) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.109743118286133 (26.22428103855678) | > grad_norm_1: tensor(7098.1401, device='cuda:0') (tensor(9384.0039, device='cuda:0')) | > current_lr_0: 0.00017295964262618142 | > current_lr_1: 0.00017295964262618142 | > step_time: 0.441 (0.43338420050484794) | > loader_time: 0.0081 (0.007356848035539899) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671686682902592 (+0.000130589579192689) | > avg_step_time: 0.44933360731098015 (-0.0037531248280699425) | > avg_loss_disc: 2.6076312367345245 (+0.038550773137051575) | > avg_loss_disc_real_0: 0.10720308660201623 (-0.019281214968839153) | > avg_loss_disc_real_1: 0.21939724033147517 (-0.005714225517192334) | > avg_loss_disc_real_2: 0.23398328226217083 (+0.0005719626033809933) | > avg_loss_disc_real_3: 0.23664342958322712 (+0.0005744901341451503) | > avg_loss_disc_real_4: 0.23919949733035664 (-0.008977995162278846) | > avg_loss_disc_real_5: 0.24425630880073762 (-0.004994124803744615) | > avg_loss_0: 2.6076312367345245 (+0.038550773137051575) | > avg_grad_norm_0: tensor(322.6103, device='cuda:0') (+tensor(19.2159, device='cuda:0')) | > avg_loss_gen: 2.125100050174015 (-0.1255563430383173) | > avg_loss_kl: 2.0048775219581496 (+0.001187712373867722) | > avg_loss_feat: 3.8734025283598563 (-0.06500482895004911) | > avg_loss_mel: 16.621335392266936 (-0.11391764627375522) | > avg_loss_duration: 1.5347152545418539 (-0.0018932449985558097) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.159430759053834 (-0.30518443147900953) | > avg_grad_norm_1: tensor(8910.3066, device='cuda:0') (+tensor(1778.1787, device='cuda:0'))  > EPOCH: 1163/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:11:39)   --> TIME: 2025-02-25 20:11:46 -- STEP: 13/72 -- GLOBAL_STEP: 1083750 | > loss_disc: 2.8294568061828613 (2.734805107116699) | > loss_disc_real_0: 0.0505225732922554 (0.2672501713610612) | > loss_disc_real_1: 0.22830094397068024 (0.21098928268139178) | > loss_disc_real_2: 0.21128608286380768 (0.2408293875364157) | > loss_disc_real_3: 0.22053299844264984 (0.21789785417226645) | > loss_disc_real_4: 0.23907093703746796 (0.2630203847701733) | > loss_disc_real_5: 0.22155004739761353 (0.25321394205093384) | > loss_0: 2.8294568061828613 (2.734805107116699) | > grad_norm_0: tensor(293.1503, device='cuda:0') (tensor(292.9264, device='cuda:0')) | > loss_gen: 1.7719987630844116 (2.3666675182489247) | > loss_kl: 1.9729256629943848 (1.8864165177712073) | > loss_feat: 4.027318477630615 (3.766044543339656) | > loss_mel: 17.149187088012695 (16.743752846351036) | > loss_duration: 1.4595831632614136 (1.4855406100933368) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.381013870239258 (26.248422329242413) | > grad_norm_1: tensor(8790.8994, device='cuda:0') (tensor(6120.1064, device='cuda:0')) | > current_lr_0: 0.00017293802267085313 | > current_lr_1: 0.00017293802267085313 | > step_time: 0.4484 (0.43172799623929536) | > loader_time: 0.0057 (0.006914890729464018)  --> TIME: 2025-02-25 20:12:09 -- STEP: 63/72 -- GLOBAL_STEP: 1083800 | > loss_disc: 2.677478313446045 (2.670195647648402) | > loss_disc_real_0: 0.06566481292247772 (0.15116020912925401) | > loss_disc_real_1: 0.22649326920509338 (0.22882404474038925) | > loss_disc_real_2: 0.2413928210735321 (0.2425489238803349) | > loss_disc_real_3: 0.25120529532432556 (0.24583441066363496) | > loss_disc_real_4: 0.24513322114944458 (0.2428089848586491) | > loss_disc_real_5: 0.25244656205177307 (0.25179876032329734) | > loss_0: 2.677478313446045 (2.670195647648402) | > grad_norm_0: tensor(91.6479, device='cuda:0') (tensor(347.8050, device='cuda:0')) | > loss_gen: 1.936004877090454 (2.177406127490695) | > loss_kl: 2.074708938598633 (2.023935806183588) | > loss_feat: 3.2347795963287354 (3.782152698153541) | > loss_mel: 16.165851593017578 (16.723609954591776) | > loss_duration: 1.5591142177581787 (1.5327718125449286) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.970458984375 (26.23987630813841) | > grad_norm_1: tensor(5612.3701, device='cuda:0') (tensor(8213.0684, device='cuda:0')) | > current_lr_0: 0.00017293802267085313 | > current_lr_1: 0.00017293802267085313 | > step_time: 0.4754 (0.4433743575262645) | > loader_time: 0.0112 (0.008212013850136408) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585409379341232 (-8.627730356135926e-05) | > avg_step_time: 0.4515125550014872 (+0.002178947690507027) | > avg_loss_disc: 2.663612117230053 (+0.05598088049552841) | > avg_loss_disc_real_0: 0.15218056334366262 (+0.044977476741646386) | > avg_loss_disc_real_1: 0.22826611890759266 (+0.008868878576117484) | > avg_loss_disc_real_2: 0.24173488499413073 (+0.007751602731959906) | > avg_loss_disc_real_3: 0.24423300674263856 (+0.0075895771594114425) | > avg_loss_disc_real_4: 0.2421722067913539 (+0.002972709460997258) | > avg_loss_disc_real_5: 0.2509365623265924 (+0.0066802535258547735) | > avg_loss_0: 2.663612117230053 (+0.05598088049552841) | > avg_grad_norm_0: tensor(352.3135, device='cuda:0') (+tensor(29.7032, device='cuda:0')) | > avg_loss_gen: 2.183201391931991 (+0.05810134175797588) | > avg_loss_kl: 2.0319253273413214 (+0.027047805383171752) | > avg_loss_feat: 3.763296167615434 (-0.11010636074442237) | > avg_loss_mel: 16.69165628728732 (+0.07032089502038374) | > avg_loss_duration: 1.540634274482727 (+0.0059190199408731825) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.21071342683174 (+0.05128266777790813) | > avg_grad_norm_1: tensor(8106.8608, device='cuda:0') (tensor(-803.4458, device='cuda:0'))  > EPOCH: 1164/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:12:15)   --> TIME: 2025-02-25 20:12:36 -- STEP: 41/72 -- GLOBAL_STEP: 1083850 | > loss_disc: 2.447187900543213 (2.629597413830641) | > loss_disc_real_0: 0.05610796809196472 (0.15133805709277712) | > loss_disc_real_1: 0.23531250655651093 (0.211158952335032) | > loss_disc_real_2: 0.229142427444458 (0.24149231758059525) | > loss_disc_real_3: 0.255076140165329 (0.23190281558327558) | > loss_disc_real_4: 0.2585647702217102 (0.2405237202964178) | > loss_disc_real_5: 0.24177178740501404 (0.2414274012170187) | > loss_0: 2.447187900543213 (2.629597413830641) | > grad_norm_0: tensor(36.7276, device='cuda:0') (tensor(299.9608, device='cuda:0')) | > loss_gen: 2.5088253021240234 (2.176955432426639) | > loss_kl: 2.054044246673584 (1.9728346481555845) | > loss_feat: 4.447299003601074 (3.8030306536976886) | > loss_mel: 17.250818252563477 (16.7636874826943) | > loss_duration: 1.5436140298843384 (1.5085631114680593) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.80459976196289 (26.225071372055424) | > grad_norm_1: tensor(5157.3560, device='cuda:0') (tensor(6816.2446, device='cuda:0')) | > current_lr_0: 0.00017291640541801926 | > current_lr_1: 0.00017291640541801926 | > step_time: 0.4385 (0.4324437990421202) | > loader_time: 0.0091 (0.007421330707829173) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605493626124422 (+2.0084246783189444e-05) | > avg_step_time: 0.45055006591367047 (-0.0009624890878167092) | > avg_loss_disc: 2.6222541567305444 (-0.04135796049950846) | > avg_loss_disc_real_0: 0.1402275671946331 (-0.011952996149029504) | > avg_loss_disc_real_1: 0.22092982308125833 (-0.007336295826334327) | > avg_loss_disc_real_2: 0.2406381714931676 (-0.0010967135009631312) | > avg_loss_disc_real_3: 0.23016614771225083 (-0.014066859030387735) | > avg_loss_disc_real_4: 0.24224774854283937 (+7.554175148546949e-05) | > avg_loss_disc_real_5: 0.2467228656923267 (-0.0042136966342656845) | > avg_loss_0: 2.6222541567305444 (-0.04135796049950846) | > avg_grad_norm_0: tensor(307.8575, device='cuda:0') (tensor(-44.4560, device='cuda:0')) | > avg_loss_gen: 2.180657581544258 (-0.002543810387732659) | > avg_loss_kl: 2.0069128325287724 (-0.02501249481254897) | > avg_loss_feat: 3.7862065207790323 (+0.02291035316359835) | > avg_loss_mel: 16.656268683957382 (-0.03538760332993718) | > avg_loss_duration: 1.5392036152557589 (-0.0014306592269681673) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.169249225670182 (-0.0414642011615598) | > avg_grad_norm_1: tensor(7487.4331, device='cuda:0') (tensor(-619.4277, device='cuda:0'))  > EPOCH: 1165/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:12:52)   --> TIME: 2025-02-25 20:13:02 -- STEP: 19/72 -- GLOBAL_STEP: 1083900 | > loss_disc: 2.865748643875122 (2.6358499777944466) | > loss_disc_real_0: 0.33583101630210876 (0.1317196525633335) | > loss_disc_real_1: 0.5249840617179871 (0.2248139750016363) | > loss_disc_real_2: 0.2785988748073578 (0.21701963794858833) | > loss_disc_real_3: 0.3337947428226471 (0.22745579637979207) | > loss_disc_real_4: 0.23088021576404572 (0.22459373427064797) | > loss_disc_real_5: 0.2396954894065857 (0.23668044256536583) | > loss_0: 2.865748643875122 (2.6358499777944466) | > grad_norm_0: tensor(238.8137, device='cuda:0') (tensor(374.9780, device='cuda:0')) | > loss_gen: 2.951751708984375 (2.1516792899683903) | > loss_kl: 1.9597235918045044 (1.9292371712232892) | > loss_feat: 3.440640449523926 (3.988949185923526) | > loss_mel: 17.359006881713867 (16.898952835484554) | > loss_duration: 1.4810950756072998 (1.486999837975753) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.192216873168945 (26.455818778590153) | > grad_norm_1: tensor(8974.7646, device='cuda:0') (tensor(8588.3457, device='cuda:0')) | > current_lr_0: 0.000172894790867342 | > current_lr_1: 0.000172894790867342 | > step_time: 0.4402 (0.431623659635845) | > loader_time: 0.0072 (0.006607381921065481)  --> TIME: 2025-02-25 20:13:26 -- STEP: 69/72 -- GLOBAL_STEP: 1083950 | > loss_disc: 2.757132053375244 (2.6301201564678256) | > loss_disc_real_0: 0.09470964223146439 (0.12241667590063546) | > loss_disc_real_1: 0.23961660265922546 (0.23810515865899515) | > loss_disc_real_2: 0.19670505821704865 (0.23633406170900317) | > loss_disc_real_3: 0.26685550808906555 (0.2376772523790166) | > loss_disc_real_4: 0.25476470589637756 (0.23782023053238358) | > loss_disc_real_5: 0.2603093981742859 (0.24028396692828854) | > loss_0: 2.757132053375244 (2.6301201564678256) | > grad_norm_0: tensor(173.2320, device='cuda:0') (tensor(297.9100, device='cuda:0')) | > loss_gen: 1.9468891620635986 (2.1486464185991148) | > loss_kl: 2.172109842300415 (2.010938020719998) | > loss_feat: 2.7901453971862793 (3.84584515336631) | > loss_mel: 16.19196128845215 (16.57008426776831) | > loss_duration: 1.6451683044433594 (1.5386251712190933) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.746274948120117 (26.11413908695829) | > grad_norm_1: tensor(11533.9883, device='cuda:0') (tensor(7940.8545, device='cuda:0')) | > current_lr_0: 0.000172894790867342 | > current_lr_1: 0.000172894790867342 | > step_time: 0.5392 (0.44958182003187097) | > loader_time: 0.0125 (0.008409113123796988) --> EVAL PERFORMANCE | > avg_loader_time: 0.008472291516586088 (-0.00013320210953833393) | > avg_step_time: 0.4511304640434158 (+0.0005803981297453076) | > avg_loss_disc: 2.6201216167127592 (-0.0021325400177851783) | > avg_loss_disc_real_0: 0.12201904479256813 (-0.018208522402064983) | > avg_loss_disc_real_1: 0.23703935973241297 (+0.016109536651154638) | > avg_loss_disc_real_2: 0.23495436449285964 (-0.005683807000307961) | > avg_loss_disc_real_3: 0.2370989058219211 (+0.006932758109670284) | > avg_loss_disc_real_4: 0.23706884640203396 (-0.005178902140805408) | > avg_loss_disc_real_5: 0.2406888914779878 (-0.006033974214338905) | > avg_loss_0: 2.6201216167127592 (-0.0021325400177851783) | > avg_grad_norm_0: tensor(299.7591, device='cuda:0') (tensor(-8.0985, device='cuda:0')) | > avg_loss_gen: 2.1591480063720487 (-0.021509575172209416) | > avg_loss_kl: 2.0136613560394503 (+0.006748523510677895) | > avg_loss_feat: 3.8679406609333737 (+0.08173414015434144) | > avg_loss_mel: 16.5941332561869 (-0.062135427770481755) | > avg_loss_duration: 1.539226276773802 (+2.2661518043198825e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.174109579811635 (+0.004860354141452916) | > avg_grad_norm_1: tensor(7949.0508, device='cuda:0') (+tensor(461.6177, device='cuda:0'))  > EPOCH: 1166/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:13:29)   --> TIME: 2025-02-25 20:13:52 -- STEP: 47/72 -- GLOBAL_STEP: 1084000 | > loss_disc: 2.916403293609619 (2.6579167690682923) | > loss_disc_real_0: 0.1007838174700737 (0.18823903648460166) | > loss_disc_real_1: 0.21835550665855408 (0.2168607902019582) | > loss_disc_real_2: 0.2238592505455017 (0.22379693674280288) | > loss_disc_real_3: 0.2291596680879593 (0.2292468398809433) | > loss_disc_real_4: 0.2493961900472641 (0.2346893973807071) | > loss_disc_real_5: 0.23686082661151886 (0.23861715134153974) | > loss_0: 2.916403293609619 (2.6579167690682923) | > grad_norm_0: tensor(238.6950, device='cuda:0') (tensor(372.3647, device='cuda:0')) | > loss_gen: 1.769343376159668 (2.16335556862202) | > loss_kl: 2.032418966293335 (2.018777956353857) | > loss_feat: 3.3622398376464844 (3.790169315135225) | > loss_mel: 17.809968948364258 (16.712316330443045) | > loss_duration: 1.6057008504867554 (1.5197995064106393) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.579673767089844 (26.204418588191906) | > grad_norm_1: tensor(6314.1929, device='cuda:0') (tensor(7438.4307, device='cuda:0')) | > current_lr_0: 0.0001728731790184836 | > current_lr_1: 0.0001728731790184836 | > step_time: 0.4384 (0.4361164062581164) | > loader_time: 0.0089 (0.007582659417010368) --> EVAL PERFORMANCE | > avg_loader_time: 0.008597246358092402 (+0.00012495484150631432) | > avg_step_time: 0.4486731005386567 (-0.002457363504759058) | > avg_loss_disc: 2.687624034747272 (+0.06750241803451296) | > avg_loss_disc_real_0: 0.16756425976333483 (+0.045545214970766704) | > avg_loss_disc_real_1: 0.2168672781595042 (-0.020172081572908757) | > avg_loss_disc_real_2: 0.23696890851141703 (+0.002014544018557385) | > avg_loss_disc_real_3: 0.23514065755085206 (-0.001958248271069052) | > avg_loss_disc_real_4: 0.2395953034431162 (+0.0025264570410822307) | > avg_loss_disc_real_5: 0.23617180153517656 (-0.004517089942811248) | > avg_loss_0: 2.687624034747272 (+0.06750241803451296) | > avg_grad_norm_0: tensor(345.7719, device='cuda:0') (+tensor(46.0128, device='cuda:0')) | > avg_loss_gen: 2.1208064287481174 (-0.03834157762393131) | > avg_loss_kl: 2.070991665544644 (+0.0573303095051938) | > avg_loss_feat: 3.764360854323481 (-0.10357980660989252) | > avg_loss_mel: 16.838966920342248 (+0.24483366415534746) | > avg_loss_duration: 1.5448509488307254 (+0.005624672056923297) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.33997680771519 (+0.16586722790355424) | > avg_grad_norm_1: tensor(7114.1782, device='cuda:0') (tensor(-834.8726, device='cuda:0'))  > EPOCH: 1167/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:14:05)   --> TIME: 2025-02-25 20:14:18 -- STEP: 25/72 -- GLOBAL_STEP: 1084050 | > loss_disc: 2.9209821224212646 (2.6881621742248534) | > loss_disc_real_0: 0.1120825782418251 (0.16410301402211186) | > loss_disc_real_1: 0.2510722279548645 (0.23241741240024566) | > loss_disc_real_2: 0.2573368549346924 (0.23778594493865968) | > loss_disc_real_3: 0.2543933689594269 (0.2346303290128708) | > loss_disc_real_4: 0.2678709626197815 (0.24876961767673492) | > loss_disc_real_5: 0.2628171443939209 (0.24269286334514617) | > loss_0: 2.9209821224212646 (2.6881621742248534) | > grad_norm_0: tensor(462.0522, device='cuda:0') (tensor(301.1071, device='cuda:0')) | > loss_gen: 1.694287896156311 (2.029357047080994) | > loss_kl: 2.0500285625457764 (1.9594945812225342) | > loss_feat: 2.820737361907959 (3.539925937652588) | > loss_mel: 16.770925521850586 (16.77254684448242) | > loss_duration: 1.5109461545944214 (1.5014539098739623) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.846925735473633 (25.802778244018555) | > grad_norm_1: tensor(7003.1665, device='cuda:0') (tensor(7426.5562, device='cuda:0')) | > current_lr_0: 0.00017285156987110627 | > current_lr_1: 0.00017285156987110627 | > step_time: 0.4244 (0.4336403369903564) | > loader_time: 0.0075 (0.006757974624633789) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515371403224031 (-8.187495486837101e-05) | > avg_step_time: 0.45204906396462885 (+0.0033759634259721283) | > avg_loss_disc: 2.6724222881693227 (-0.015201746577949482) | > avg_loss_disc_real_0: 0.12145006079489074 (-0.0461141989684441) | > avg_loss_disc_real_1: 0.23766664436585466 (+0.02079936620635045) | > avg_loss_disc_real_2: 0.23262301255279863 (-0.004345895958618395) | > avg_loss_disc_real_3: 0.23093250383373717 (-0.0042081537171148875) | > avg_loss_disc_real_4: 0.24525595949569218 (+0.005660656052575985) | > avg_loss_disc_real_5: 0.23752039566006458 (+0.001348594124888025) | > avg_loss_0: 2.6724222881693227 (-0.015201746577949482) | > avg_grad_norm_0: tensor(320.5500, device='cuda:0') (tensor(-25.2220, device='cuda:0')) | > avg_loss_gen: 2.0447795374292714 (-0.07602689131884599) | > avg_loss_kl: 2.02015233879358 (-0.050839326751064284) | > avg_loss_feat: 3.695901783419327 (-0.06845907090415437) | > avg_loss_mel: 16.59250694597272 (-0.24645997436952882) | > avg_loss_duration: 1.5429119325019944 (-0.001939016328730947) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.89625256498095 (-0.4437242427342376) | > avg_grad_norm_1: tensor(8231.6855, device='cuda:0') (+tensor(1117.5073, device='cuda:0'))  > EPOCH: 1168/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:14:42)   --> TIME: 2025-02-25 20:14:45 -- STEP: 3/72 -- GLOBAL_STEP: 1084100 | > loss_disc: 2.5002851486206055 (2.557284116744995) | > loss_disc_real_0: 0.12170203775167465 (0.07632132122913997) | > loss_disc_real_1: 0.22285936772823334 (0.2564593901236852) | > loss_disc_real_2: 0.221637561917305 (0.2385986347993215) | > loss_disc_real_3: 0.21148596704006195 (0.22003857791423798) | > loss_disc_real_4: 0.23356425762176514 (0.24867441256841025) | > loss_disc_real_5: 0.26082852482795715 (0.26062538226445514) | > loss_0: 2.5002851486206055 (2.557284116744995) | > grad_norm_0: tensor(413.7164, device='cuda:0') (tensor(333.7435, device='cuda:0')) | > loss_gen: 2.211784839630127 (2.1247244675954184) | > loss_kl: 1.9186960458755493 (1.930373231569926) | > loss_feat: 4.2231245040893555 (3.810938517252604) | > loss_mel: 17.463998794555664 (16.553144137064617) | > loss_duration: 1.524210810661316 (1.4979329506556194) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.341814041137695 (25.917112986246746) | > grad_norm_1: tensor(8011.5894, device='cuda:0') (tensor(9519.4492, device='cuda:0')) | > current_lr_0: 0.00017282996342487237 | > current_lr_1: 0.00017282996342487237 | > step_time: 0.4219 (0.42971984545389813) | > loader_time: 0.0066 (0.005627234776814778)  --> TIME: 2025-02-25 20:15:08 -- STEP: 53/72 -- GLOBAL_STEP: 1084150 | > loss_disc: 2.6104679107666016 (2.6041664492409184) | > loss_disc_real_0: 0.3279511332511902 (0.11083210927416694) | > loss_disc_real_1: 0.19045665860176086 (0.21621052611548947) | > loss_disc_real_2: 0.3026396632194519 (0.24006102894837) | > loss_disc_real_3: 0.20116177201271057 (0.23002027340655057) | > loss_disc_real_4: 0.22052137553691864 (0.24275222111423062) | > loss_disc_real_5: 0.2336374670267105 (0.2443002847568044) | > loss_0: 2.6104679107666016 (2.6041664492409184) | > grad_norm_0: tensor(483.2474, device='cuda:0') (tensor(267.8559, device='cuda:0')) | > loss_gen: 2.49438214302063 (2.1069874111211524) | > loss_kl: 2.208397388458252 (2.016199692240301) | > loss_feat: 4.134539604187012 (3.8171829502537564) | > loss_mel: 17.37386703491211 (16.64802983122052) | > loss_duration: 1.635193109512329 (1.531249631125972) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.84638023376465 (26.119649383257016) | > grad_norm_1: tensor(4106.7837, device='cuda:0') (tensor(7157.6426, device='cuda:0')) | > current_lr_0: 0.00017282996342487237 | > current_lr_1: 0.00017282996342487237 | > step_time: 0.4376 (0.4401882504517177) | > loader_time: 0.0089 (0.007834259069190835) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605721970679055 (+9.03505674550234e-05) | > avg_step_time: 0.45283670156774386 (+0.0007876376031150101) | > avg_loss_disc: 2.6038845458500823 (-0.06853774231924037) | > avg_loss_disc_real_0: 0.1229602435315159 (+0.0015101827366251624) | > avg_loss_disc_real_1: 0.21824776664586137 (-0.01941887771999329) | > avg_loss_disc_real_2: 0.23883502290282452 (+0.006212010350025887) | > avg_loss_disc_real_3: 0.2277336945416222 (-0.003198809292114968) | > avg_loss_disc_real_4: 0.24777404313356105 (+0.002518083637868873) | > avg_loss_disc_real_5: 0.24582812735732174 (+0.008307731697257154) | > avg_loss_0: 2.6038845458500823 (-0.06853774231924037) | > avg_grad_norm_0: tensor(276.7413, device='cuda:0') (tensor(-43.8086, device='cuda:0')) | > avg_loss_gen: 2.1933910007208164 (+0.14861146329154495) | > avg_loss_kl: 2.212092423103224 (+0.19194008430964438) | > avg_loss_feat: 3.9118487062588545 (+0.21594692283952766) | > avg_loss_mel: 17.22556071214273 (+0.6330537661700113) | > avg_loss_duration: 1.5650016203732557 (+0.02208968787126131) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.10789452136402 (+1.211641956383069) | > avg_grad_norm_1: tensor(6602.5000, device='cuda:0') (tensor(-1629.1855, device='cuda:0'))  > EPOCH: 1169/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:15:19)   --> TIME: 2025-02-25 20:15:35 -- STEP: 31/72 -- GLOBAL_STEP: 1084200 | > loss_disc: 2.302297592163086 (2.5337983715918755) | > loss_disc_real_0: 0.03370700776576996 (0.08955297997641948) | > loss_disc_real_1: 0.2621157169342041 (0.23162008533554693) | > loss_disc_real_2: 0.23574481904506683 (0.21373751759529114) | > loss_disc_real_3: 0.2393234670162201 (0.2265396079709453) | > loss_disc_real_4: 0.23418256640434265 (0.22809113562107086) | > loss_disc_real_5: 0.24657398462295532 (0.2323332712534935) | > loss_0: 2.302297592163086 (2.5337983715918755) | > grad_norm_0: tensor(35.6035, device='cuda:0') (tensor(240.2346, device='cuda:0')) | > loss_gen: 2.464064836502075 (2.1448514307698896) | > loss_kl: 1.9815125465393066 (2.2207915782928467) | > loss_feat: 4.685491561889648 (4.087630202693324) | > loss_mel: 17.898109436035156 (18.275443477015337) | > loss_duration: 1.5382250547409058 (1.5343783517037668) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.56740379333496 (28.263094871274888) | > grad_norm_1: tensor(4473.9849, device='cuda:0') (tensor(6368.8242, device='cuda:0')) | > current_lr_0: 0.00017280835967944426 | > current_lr_1: 0.00017280835967944426 | > step_time: 0.4329 (0.43202344832881806) | > loader_time: 0.0093 (0.007165455049084079) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862685391600703 (+2.1131945327974508e-05) | > avg_step_time: 0.45041960393878777 (-0.0024170976289560886) | > avg_loss_disc: 2.53402310022166 (-0.06986144562842211) | > avg_loss_disc_real_0: 0.07915577675465127 (-0.04380446677686463) | > avg_loss_disc_real_1: 0.22062480953377736 (+0.0023770428879159866) | > avg_loss_disc_real_2: 0.2235387318151098 (-0.015296291087714714) | > avg_loss_disc_real_3: 0.22306505727096343 (-0.004668637270658776) | > avg_loss_disc_real_4: 0.23730349016021673 (-0.010470552973344316) | > avg_loss_disc_real_5: 0.24409628418129933 (-0.0017318431760224018) | > avg_loss_0: 2.53402310022166 (-0.06986144562842211) | > avg_grad_norm_0: tensor(257.1266, device='cuda:0') (tensor(-19.6147, device='cuda:0')) | > avg_loss_gen: 2.1520832256532043 (-0.041307775067612074) | > avg_loss_kl: 2.1386287464222447 (-0.07346367668097953) | > avg_loss_feat: 4.013427479166381 (+0.10157877290752682) | > avg_loss_mel: 17.525520311274995 (+0.299959599132265) | > avg_loss_duration: 1.5591413924391841 (-0.0058602279340715935) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.388801118017923 (+0.28090659665390305) | > avg_grad_norm_1: tensor(8021.9146, device='cuda:0') (+tensor(1419.4146, device='cuda:0'))  > EPOCH: 1170/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:15:55)   --> TIME: 2025-02-25 20:16:01 -- STEP: 9/72 -- GLOBAL_STEP: 1084250 | > loss_disc: 2.5407285690307617 (2.476493650012546) | > loss_disc_real_0: 0.09810495376586914 (0.0856599248945713) | > loss_disc_real_1: 0.22781962156295776 (0.2195292827155855) | > loss_disc_real_2: 0.23027151823043823 (0.2272779858774609) | > loss_disc_real_3: 0.24314998090267181 (0.22785884307490456) | > loss_disc_real_4: 0.24786999821662903 (0.24106563131014505) | > loss_disc_real_5: 0.24607712030410767 (0.24299913810359108) | > loss_0: 2.5407285690307617 (2.476493650012546) | > grad_norm_0: tensor(309.4439, device='cuda:0') (tensor(244.1773, device='cuda:0')) | > loss_gen: 2.288804292678833 (2.3103592660692005) | > loss_kl: 1.8439347743988037 (1.872721102502611) | > loss_feat: 4.315815448760986 (4.15970762570699) | > loss_mel: 17.169750213623047 (16.985691494411892) | > loss_duration: 1.5112285614013672 (1.5023368861940172) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.129533767700195 (26.830816480848526) | > grad_norm_1: tensor(8060.6880, device='cuda:0') (tensor(7202.4824, device='cuda:0')) | > current_lr_0: 0.00017278675863448433 | > current_lr_1: 0.00017278675863448433 | > step_time: 0.4117 (0.42477475272284615) | > loader_time: 0.0055 (0.006344795227050781)  --> TIME: 2025-02-25 20:16:24 -- STEP: 59/72 -- GLOBAL_STEP: 1084300 | > loss_disc: 2.569154739379883 (2.5123361328900877) | > loss_disc_real_0: 0.13178667426109314 (0.08282428657857038) | > loss_disc_real_1: 0.22364355623722076 (0.218672924627692) | > loss_disc_real_2: 0.22908002138137817 (0.23223055116200852) | > loss_disc_real_3: 0.20484420657157898 (0.2313216987807872) | > loss_disc_real_4: 0.23062792420387268 (0.24626510552430558) | > loss_disc_real_5: 0.23903900384902954 (0.24946529935982267) | > loss_0: 2.569154739379883 (2.5123361328900877) | > grad_norm_0: tensor(496.5093, device='cuda:0') (tensor(269.4545, device='cuda:0')) | > loss_gen: 2.1942548751831055 (2.2360674223657386) | > loss_kl: 2.0370607376098633 (1.9369143045554726) | > loss_feat: 4.131461143493652 (4.018136731648848) | > loss_mel: 17.10311508178711 (16.78784695318188) | > loss_duration: 1.571497917175293 (1.5302516464459694) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.037391662597656 (26.509217116792325) | > grad_norm_1: tensor(8387.6152, device='cuda:0') (tensor(8643.7178, device='cuda:0')) | > current_lr_0: 0.00017278675863448433 | > current_lr_1: 0.00017278675863448433 | > step_time: 0.4865 (0.4378990965374446) | > loader_time: 0.0102 (0.008016198368395787) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579217212300903 (-4.763670370612656e-05) | > avg_step_time: 0.44811958326420315 (-0.002300020674584624) | > avg_loss_disc: 2.5124476862625342 (-0.02157541395912599) | > avg_loss_disc_real_0: 0.08414101432746565 (+0.004985237572814383) | > avg_loss_disc_real_1: 0.2187879442329138 (-0.0018368653008635494) | > avg_loss_disc_real_2: 0.23154163402570804 (+0.00800290221059824) | > avg_loss_disc_real_3: 0.23053884758076199 (+0.007473790309798556) | > avg_loss_disc_real_4: 0.24449442599860716 (+0.007190935838390422) | > avg_loss_disc_real_5: 0.2477389135411088 (+0.0036426293598094683) | > avg_loss_0: 2.5124476862625342 (-0.02157541395912599) | > avg_grad_norm_0: tensor(269.4756, device='cuda:0') (+tensor(12.3490, device='cuda:0')) | > avg_loss_gen: 2.2296669567135017 (+0.07758373106029737) | > avg_loss_kl: 1.9587948372666264 (-0.17983390915561825) | > avg_loss_feat: 3.9951080503598067 (-0.018319428806574667) | > avg_loss_mel: 16.806549851323503 (-0.7189704599514926) | > avg_loss_duration: 1.54056350781884 (-0.018577884620344154) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.53068327567947 (-0.858117842338455) | > avg_grad_norm_1: tensor(8447.0234, device='cuda:0') (+tensor(425.1089, device='cuda:0'))  > EPOCH: 1171/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:16:32)   --> TIME: 2025-02-25 20:16:51 -- STEP: 37/72 -- GLOBAL_STEP: 1084350 | > loss_disc: 2.732611894607544 (2.509881663966823) | > loss_disc_real_0: 0.2301184982061386 (0.08793726392291687) | > loss_disc_real_1: 0.2101515829563141 (0.22029578846854134) | > loss_disc_real_2: 0.2387157827615738 (0.2286011484023687) | > loss_disc_real_3: 0.24193057417869568 (0.2336675641504494) | > loss_disc_real_4: 0.2663204073905945 (0.24304234337162328) | > loss_disc_real_5: 0.2755320966243744 (0.24545681476593018) | > loss_0: 2.732611894607544 (2.509881663966823) | > grad_norm_0: tensor(493.5053, device='cuda:0') (tensor(260.0004, device='cuda:0')) | > loss_gen: 2.321115493774414 (2.2611471704534583) | > loss_kl: 1.7965693473815918 (1.8972871980151615) | > loss_feat: 3.2457115650177 (4.075569455688064) | > loss_mel: 15.926478385925293 (16.71153653634561) | > loss_duration: 1.5383909940719604 (1.499365429620485) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.828266143798828 (26.444905719241582) | > grad_norm_1: tensor(5681.8252, device='cuda:0') (tensor(7836.9707, device='cuda:0')) | > current_lr_0: 0.00017276516028965502 | > current_lr_1: 0.00017276516028965502 | > step_time: 0.4382 (0.4345822592039366) | > loader_time: 0.0081 (0.007081882373706714) --> EVAL PERFORMANCE | > avg_loader_time: 0.008451199867355992 (-0.00012801734494491054) | > avg_step_time: 0.4492172590443786 (+0.0010976757801754444) | > avg_loss_disc: 2.551146171462367 (+0.03869848519983288) | > avg_loss_disc_real_0: 0.1147287957055468 (+0.030587781378081144) | > avg_loss_disc_real_1: 0.22298784868817934 (+0.004199904455265524) | > avg_loss_disc_real_2: 0.23134414431914477 (-0.00019748970656327525) | > avg_loss_disc_real_3: 0.23067968936873154 (+0.00014084178796955338) | > avg_loss_disc_real_4: 0.2416645691428386 (-0.002829856855768559) | > avg_loss_disc_real_5: 0.24726170721188398 (-0.00047720632922482187) | > avg_loss_0: 2.551146171462367 (+0.03869848519983288) | > avg_grad_norm_0: tensor(295.6115, device='cuda:0') (+tensor(26.1358, device='cuda:0')) | > avg_loss_gen: 2.2305307992747134 (+0.0008638425612117473) | > avg_loss_kl: 1.9517346939570468 (-0.007060143309579603) | > avg_loss_feat: 3.971564044415112 (-0.023544005944694746) | > avg_loss_mel: 16.70947330098756 (-0.09707655033594165) | > avg_loss_duration: 1.5345886442023264 (-0.005974863616513559) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.397891487873775 (-0.13279178780569367) | > avg_grad_norm_1: tensor(7475.0361, device='cuda:0') (tensor(-971.9873, device='cuda:0'))  > EPOCH: 1172/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:17:09)   --> TIME: 2025-02-25 20:17:17 -- STEP: 15/72 -- GLOBAL_STEP: 1084400 | > loss_disc: 3.1747803688049316 (2.6325981775919596) | > loss_disc_real_0: 0.34406426548957825 (0.12285233562191328) | > loss_disc_real_1: 0.1519167274236679 (0.2072143922249476) | > loss_disc_real_2: 0.26645419001579285 (0.2299611449241638) | > loss_disc_real_3: 0.25356435775756836 (0.23054390549659728) | > loss_disc_real_4: 0.2708791494369507 (0.23039296468098958) | > loss_disc_real_5: 0.2485550194978714 (0.2335439811150233) | > loss_0: 3.1747803688049316 (2.6325981775919596) | > grad_norm_0: tensor(571.0753, device='cuda:0') (tensor(301.5135, device='cuda:0')) | > loss_gen: 1.585221767425537 (2.060656118392944) | > loss_kl: 2.0296812057495117 (1.8964709679285685) | > loss_feat: 3.3260059356689453 (4.043638912836711) | > loss_mel: 16.768983840942383 (16.667537562052406) | > loss_duration: 1.4963700771331787 (1.480509885152181) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.20626449584961 (26.148814137776693) | > grad_norm_1: tensor(596.1693, device='cuda:0') (tensor(6273.3672, device='cuda:0')) | > current_lr_0: 0.00017274356464461882 | > current_lr_1: 0.00017274356464461882 | > step_time: 0.4219 (0.4356478532155355) | > loader_time: 0.0059 (0.006541029612223307)  --> TIME: 2025-02-25 20:17:41 -- STEP: 65/72 -- GLOBAL_STEP: 1084450 | > loss_disc: 2.721435785293579 (2.6709819096785323) | > loss_disc_real_0: 0.3429383933544159 (0.12319938483146521) | > loss_disc_real_1: 0.18948616087436676 (0.21338328902538006) | > loss_disc_real_2: 0.22215862572193146 (0.24630708384972352) | > loss_disc_real_3: 0.20938067138195038 (0.25494136672753553) | > loss_disc_real_4: 0.2537654936313629 (0.24065149220136495) | > loss_disc_real_5: 0.25261160731315613 (0.24655676965530102) | > loss_0: 2.721435785293579 (2.6709819096785323) | > grad_norm_0: tensor(85.6784, device='cuda:0') (tensor(293.2484, device='cuda:0')) | > loss_gen: 2.6061573028564453 (2.101132480914775) | > loss_kl: 2.1887576580047607 (2.0554336015994727) | > loss_feat: 3.7463927268981934 (3.7511746406555178) | > loss_mel: 17.257707595825195 (16.725587448707) | > loss_duration: 1.5928614139556885 (1.5375563676540671) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.391876220703125 (26.17088473393367) | > grad_norm_1: tensor(3980.8528, device='cuda:0') (tensor(7355.7051, device='cuda:0')) | > current_lr_0: 0.00017274356464461882 | > current_lr_1: 0.00017274356464461882 | > step_time: 0.5302 (0.4477429940150334) | > loader_time: 0.0115 (0.008200179613553562) --> EVAL PERFORMANCE | > avg_loader_time: 0.008466415002312462 (+1.5215134956470325e-05) | > avg_step_time: 0.4517786569998298 (+0.0025613979554511834) | > avg_loss_disc: 2.666968587418677 (+0.11582241595630993) | > avg_loss_disc_real_0: 0.12828736825728082 (+0.013558572551734024) | > avg_loss_disc_real_1: 0.21318957104649341 (-0.00979827764168592) | > avg_loss_disc_real_2: 0.2447136179871962 (+0.01336947366805144) | > avg_loss_disc_real_3: 0.24992665858335897 (+0.01924696921462743) | > avg_loss_disc_real_4: 0.2393022097752128 (-0.0023623593676257992) | > avg_loss_disc_real_5: 0.24531505792073802 (-0.0019466492911459632) | > avg_loss_0: 2.666968587418677 (+0.11582241595630993) | > avg_grad_norm_0: tensor(290.8798, device='cuda:0') (tensor(-4.7316, device='cuda:0')) | > avg_loss_gen: 2.0978021772814452 (-0.13272862199326818) | > avg_loss_kl: 2.066158966279365 (+0.11442427232231833) | > avg_loss_feat: 3.7371749642869116 (-0.23438908012820026) | > avg_loss_mel: 16.75078038766351 (+0.041307086675949733) | > avg_loss_duration: 1.5430329970910515 (+0.008444352888725026) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.194949647070658 (-0.20294184080311695) | > avg_grad_norm_1: tensor(7480.2437, device='cuda:0') (+tensor(5.2075, device='cuda:0'))  > EPOCH: 1173/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:17:45)   --> TIME: 2025-02-25 20:18:07 -- STEP: 43/72 -- GLOBAL_STEP: 1084500 | > loss_disc: 3.4064016342163086 (2.726468235947365) | > loss_disc_real_0: 0.20415166020393372 (0.2153687959666862) | > loss_disc_real_1: 0.21036377549171448 (0.22321755525677703) | > loss_disc_real_2: 0.3965460956096649 (0.24741762426010397) | > loss_disc_real_3: 0.3246673345565796 (0.25127044874568316) | > loss_disc_real_4: 0.3394972085952759 (0.2592504478471223) | > loss_disc_real_5: 0.25857415795326233 (0.25266092669132145) | > loss_0: 3.4064016342163086 (2.726468235947365) | > grad_norm_0: tensor(433.3085, device='cuda:0') (tensor(253.2379, device='cuda:0')) | > loss_gen: 1.988110065460205 (2.2063353505245473) | > loss_kl: 2.0097873210906982 (2.057366052339243) | > loss_feat: 3.502290964126587 (3.6260722016179283) | > loss_mel: 16.712724685668945 (16.88862521149392) | > loss_duration: 1.551234483718872 (1.5249456278113431) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.764148712158203 (26.303344504777776) | > grad_norm_1: tensor(3201.1008, device='cuda:0') (tensor(5753.2769, device='cuda:0')) | > current_lr_0: 0.00017272197169903824 | > current_lr_1: 0.00017272197169903824 | > step_time: 0.4459 (0.439676096273023) | > loader_time: 0.0095 (0.007509492164434388) --> EVAL PERFORMANCE | > avg_loader_time: 0.008615792637139975 (+0.00014937763482751262) | > avg_step_time: 0.4528698719723124 (+0.0010912149724826015) | > avg_loss_disc: 2.7361466851032956 (+0.06917809768461858) | > avg_loss_disc_real_0: 0.18714495218345817 (+0.05885758392617735) | > avg_loss_disc_real_1: 0.22191949125746607 (+0.008729920210972658) | > avg_loss_disc_real_2: 0.2365299295371687 (-0.00818368845002751) | > avg_loss_disc_real_3: 0.24108795359940596 (-0.008838704983953005) | > avg_loss_disc_real_4: 0.24592316213627932 (+0.006620952361066523) | > avg_loss_disc_real_5: 0.24879505915541045 (+0.0034800012346724307) | > avg_loss_0: 2.7361466851032956 (+0.06917809768461858) | > avg_grad_norm_0: tensor(263.0909, device='cuda:0') (tensor(-27.7889, device='cuda:0')) | > avg_loss_gen: 2.114518086675188 (+0.016715909393742567) | > avg_loss_kl: 2.101235027044592 (+0.035076060765226735) | > avg_loss_feat: 3.6163722998659376 (-0.12080266442097409) | > avg_loss_mel: 16.964751189863186 (+0.21397080219967535) | > avg_loss_duration: 1.5509352230689895 (+0.007902225977937993) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.347811927258128 (+0.1528622801874704) | > avg_grad_norm_1: tensor(6132.9785, device='cuda:0') (tensor(-1347.2651, device='cuda:0'))  > EPOCH: 1174/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:18:22)   --> TIME: 2025-02-25 20:18:33 -- STEP: 21/72 -- GLOBAL_STEP: 1084550 | > loss_disc: 2.774862289428711 (2.627665179116385) | > loss_disc_real_0: 0.06676577776670456 (0.15303273321617214) | > loss_disc_real_1: 0.07164079695940018 (0.1985360256263188) | > loss_disc_real_2: 0.29728904366493225 (0.2280582288901011) | > loss_disc_real_3: 0.24298354983329773 (0.22206778398581914) | > loss_disc_real_4: 0.22882018983364105 (0.22787551085154215) | > loss_disc_real_5: 0.21326844394207 (0.23408542076746622) | > loss_0: 2.774862289428711 (2.627665179116385) | > grad_norm_0: tensor(184.1716, device='cuda:0') (tensor(250.3079, device='cuda:0')) | > loss_gen: 1.8511043787002563 (2.0075387841179255) | > loss_kl: 1.8786221742630005 (1.9276988960447765) | > loss_feat: 4.023960113525391 (3.6827865895770846) | > loss_mel: 16.35588836669922 (16.870501382010325) | > loss_duration: 1.519294023513794 (1.4985514595395044) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.628870010375977 (25.987076804751442) | > grad_norm_1: tensor(2486.6714, device='cuda:0') (tensor(6936.3687, device='cuda:0')) | > current_lr_0: 0.00017270038145257585 | > current_lr_1: 0.00017270038145257585 | > step_time: 0.4249 (0.4277609984079997) | > loader_time: 0.0062 (0.006872551781790597)  --> TIME: 2025-02-25 20:18:57 -- STEP: 71/72 -- GLOBAL_STEP: 1084600 | > loss_disc: 2.670896530151367 (2.675986478026484) | > loss_disc_real_0: 0.10220693796873093 (0.1213458961426792) | > loss_disc_real_1: 0.2246256172657013 (0.20844122148315672) | > loss_disc_real_2: 0.18568438291549683 (0.24007041164686982) | > loss_disc_real_3: 0.2329891324043274 (0.23194870961384034) | > loss_disc_real_4: 0.27187836170196533 (0.2410988234718081) | > loss_disc_real_5: 0.23549209535121918 (0.2428828519834599) | > loss_0: 2.670896530151367 (2.675986478026484) | > grad_norm_0: tensor(74.0275, device='cuda:0') (tensor(251.3615, device='cuda:0')) | > loss_gen: 2.140001058578491 (2.008871273255684) | > loss_kl: 2.0145490169525146 (2.1305220244636005) | > loss_feat: 3.53047251701355 (3.6280022641302834) | > loss_mel: 16.731882095336914 (17.271499136803854) | > loss_duration: 1.5592155456542969 (1.5584767754648772) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.976119995117188 (26.597371356588013) | > grad_norm_1: tensor(14165.0986, device='cuda:0') (tensor(7317.3350, device='cuda:0')) | > current_lr_0: 0.00017270038145257585 | > current_lr_1: 0.00017270038145257585 | > step_time: 0.4774 (0.45218134933794046) | > loader_time: 0.0078 (0.008639177805940873) --> EVAL PERFORMANCE | > avg_loader_time: 0.008639177805940873 (+2.3385168800897638e-05) | > avg_step_time: 0.45218134933794046 (-0.0006885226343719131) | > avg_loss_disc: 2.675986478026484 (-0.06016020707681147) | > avg_loss_disc_real_0: 0.1213458961426792 (-0.06579905604077897) | > avg_loss_disc_real_1: 0.20844122148315672 (-0.013478269774309354) | > avg_loss_disc_real_2: 0.24007041164686982 (+0.0035404821097011208) | > avg_loss_disc_real_3: 0.23194870961384034 (-0.00913924398556562) | > avg_loss_disc_real_4: 0.2410988234718081 (-0.004824338664471212) | > avg_loss_disc_real_5: 0.2428828519834599 (-0.00591220717195054) | > avg_loss_0: 2.675986478026484 (-0.06016020707681147) | > avg_grad_norm_0: tensor(251.3615, device='cuda:0') (tensor(-11.7294, device='cuda:0')) | > avg_loss_gen: 2.008871273255684 (-0.10564681341950388) | > avg_loss_kl: 2.1305220244636005 (+0.029286997419008642) | > avg_loss_feat: 3.6280022641302834 (+0.011629964264345816) | > avg_loss_mel: 17.271499136803854 (+0.30674794694066776) | > avg_loss_duration: 1.5584767754648772 (+0.007541552395887718) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.597371356588013 (+0.24955942932988506) | > avg_grad_norm_1: tensor(7317.3350, device='cuda:0') (+tensor(1184.3564, device='cuda:0'))  > EPOCH: 1175/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:18:59)   --> TIME: 2025-02-25 20:19:24 -- STEP: 49/72 -- GLOBAL_STEP: 1084650 | > loss_disc: 2.837961435317993 (2.7293249101054924) | > loss_disc_real_0: 0.449400395154953 (0.20800640384609603) | > loss_disc_real_1: 0.16330406069755554 (0.24072799535126102) | > loss_disc_real_2: 0.18251022696495056 (0.23463792217021084) | > loss_disc_real_3: 0.2782650887966156 (0.23776318254519482) | > loss_disc_real_4: 0.1972944438457489 (0.24563273027235147) | > loss_disc_real_5: 0.19743502140045166 (0.2457470787422997) | > loss_0: 2.837961435317993 (2.7293249101054924) | > grad_norm_0: tensor(324.0456, device='cuda:0') (tensor(290.6770, device='cuda:0')) | > loss_gen: 1.872836947441101 (2.179539665883902) | > loss_kl: 2.2943708896636963 (1.9995976710806087) | > loss_feat: 2.9554710388183594 (3.594327770933813) | > loss_mel: 17.069303512573242 (16.873281634583766) | > loss_duration: 1.6125714778900146 (1.522694359020311) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.804553985595703 (26.169441300995494) | > grad_norm_1: tensor(9063.2441, device='cuda:0') (tensor(5811.5508, device='cuda:0')) | > current_lr_0: 0.00017267879390489427 | > current_lr_1: 0.00017267879390489427 | > step_time: 0.4504 (0.44410560568984675) | > loader_time: 0.0094 (0.0077847218026920245) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864508118427975 (+5.903378338877399e-06) | > avg_step_time: 0.4550392426235575 (+0.0028578932856170303) | > avg_loss_disc: 2.683336795215876 (+0.007350317189391831) | > avg_loss_disc_real_0: 0.18297109944404852 (+0.06162520330136932) | > avg_loss_disc_real_1: 0.23540373477087895 (+0.02696251328772223) | > avg_loss_disc_real_2: 0.23209550808852827 (-0.007974903558341545) | > avg_loss_disc_real_3: 0.24349236257479223 (+0.011543652960951889) | > avg_loss_disc_real_4: 0.24195144860677317 (+0.0008526251349650626) | > avg_loss_disc_real_5: 0.24424920178635015 (+0.0013663498028902432) | > avg_loss_0: 2.683336795215876 (+0.007350317189391831) | > avg_grad_norm_0: tensor(243.7252, device='cuda:0') (tensor(-7.6363, device='cuda:0')) | > avg_loss_gen: 2.1636858003240236 (+0.15481452706833965) | > avg_loss_kl: 2.1208085510092727 (-0.009713473454327826) | > avg_loss_feat: 3.64730828580722 (+0.019306021676936513) | > avg_loss_mel: 17.157034269520935 (-0.11446486728291916) | > avg_loss_duration: 1.5502960127843937 (-0.008180762680483511) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.639133131000357 (+0.0417617744123433) | > avg_grad_norm_1: tensor(4636.9023, device='cuda:0') (tensor(-2680.4326, device='cuda:0'))  > EPOCH: 1176/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:19:36)   --> TIME: 2025-02-25 20:19:50 -- STEP: 27/72 -- GLOBAL_STEP: 1084700 | > loss_disc: 2.6711249351501465 (2.611871012934932) | > loss_disc_real_0: 0.042900219559669495 (0.08220366830075229) | > loss_disc_real_1: 0.23822429776191711 (0.23855119557292373) | > loss_disc_real_2: 0.26551321148872375 (0.19887945000772123) | > loss_disc_real_3: 0.25303730368614197 (0.24678444144902406) | > loss_disc_real_4: 0.26756659150123596 (0.24560093603752278) | > loss_disc_real_5: 0.26572543382644653 (0.2411948447978055) | > loss_0: 2.6711249351501465 (2.611871012934932) | > grad_norm_0: tensor(100.3983, device='cuda:0') (tensor(198.8699, device='cuda:0')) | > loss_gen: 2.2224225997924805 (2.161533338052255) | > loss_kl: 1.897436499595642 (2.0039177735646563) | > loss_feat: 4.326246738433838 (4.169474319175437) | > loss_mel: 16.511186599731445 (17.161185865049013) | > loss_duration: 1.5013427734375 (1.5070875971405595) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.458637237548828 (27.003198835584854) | > grad_norm_1: tensor(1398.7085, device='cuda:0') (tensor(5149.1973, device='cuda:0')) | > current_lr_0: 0.00017265720905565617 | > current_lr_1: 0.00017265720905565617 | > step_time: 0.4377 (0.43453209488480177) | > loader_time: 0.0084 (0.007011908072012442) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871608626674598 (+7.100508246623075e-05) | > avg_step_time: 0.4522515820785308 (-0.0027876605450267156) | > avg_loss_disc: 2.6392193411437557 (-0.04411745407212031) | > avg_loss_disc_real_0: 0.12525665064827657 (-0.05771444879577195) | > avg_loss_disc_real_1: 0.23207162614439575 (-0.003332108626483199) | > avg_loss_disc_real_2: 0.21970434654766405 (-0.012391161540864226) | > avg_loss_disc_real_3: 0.23211254893054425 (-0.011379813644247977) | > avg_loss_disc_real_4: 0.24498756565678287 (+0.0030361170500096957) | > avg_loss_disc_real_5: 0.2462177123402206 (+0.001968510553870456) | > avg_loss_0: 2.6392193411437557 (-0.04411745407212031) | > avg_grad_norm_0: tensor(193.1572, device='cuda:0') (tensor(-50.5680, device='cuda:0')) | > avg_loss_gen: 2.1221375297492657 (-0.04154827057475785) | > avg_loss_kl: 2.0606301720713236 (-0.060178378937949084) | > avg_loss_feat: 3.7861758353004995 (+0.13886754949327962) | > avg_loss_mel: 16.922249605957884 (-0.23478466356305105) | > avg_loss_duration: 1.5471073909544608 (-0.0031886218299328473) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.438300710328868 (-0.20083242067148888) | > avg_grad_norm_1: tensor(5480.1377, device='cuda:0') (+tensor(843.2354, device='cuda:0'))  > EPOCH: 1177/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:20:13)   --> TIME: 2025-02-25 20:20:17 -- STEP: 5/72 -- GLOBAL_STEP: 1084750 | > loss_disc: 2.8258516788482666 (2.5675726413726805) | > loss_disc_real_0: 0.06631723046302795 (0.11650266349315644) | > loss_disc_real_1: 0.2201072871685028 (0.21152757108211517) | > loss_disc_real_2: 0.21787486970424652 (0.21565955579280854) | > loss_disc_real_3: 0.23800894618034363 (0.2420641452074051) | > loss_disc_real_4: 0.2271174192428589 (0.2367222636938095) | > loss_disc_real_5: 0.2393234521150589 (0.23387575447559356) | > loss_0: 2.8258516788482666 (2.5675726413726805) | > grad_norm_0: tensor(95.8243, device='cuda:0') (tensor(240.8724, device='cuda:0')) | > loss_gen: 1.625631332397461 (2.093410587310791) | > loss_kl: 2.1002485752105713 (2.106834602355957) | > loss_feat: 3.2370975017547607 (3.8457991600036623) | > loss_mel: 18.957048416137695 (18.91768569946289) | > loss_duration: 1.5163605213165283 (1.527350115776062) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.43638801574707 (28.49108009338379) | > grad_norm_1: tensor(1311.3680, device='cuda:0') (tensor(4558.7407, device='cuda:0')) | > current_lr_0: 0.0001726356269045242 | > current_lr_1: 0.0001726356269045242 | > step_time: 0.4221 (0.42607707977294923) | > loader_time: 0.0058 (0.006774616241455078)  --> TIME: 2025-02-25 20:20:40 -- STEP: 55/72 -- GLOBAL_STEP: 1084800 | > loss_disc: 2.7963757514953613 (2.5829896926879883) | > loss_disc_real_0: 0.03512771427631378 (0.10475217611952262) | > loss_disc_real_1: 0.24877656996250153 (0.21639476824890483) | > loss_disc_real_2: 0.33081603050231934 (0.2539317838170312) | > loss_disc_real_3: 0.231026753783226 (0.23873736587437716) | > loss_disc_real_4: 0.22440963983535767 (0.2601325671781194) | > loss_disc_real_5: 0.24720396101474762 (0.25011891858144236) | > loss_0: 2.7963757514953613 (2.5829896926879883) | > grad_norm_0: tensor(134.8566, device='cuda:0') (tensor(208.7270, device='cuda:0')) | > loss_gen: 2.0934715270996094 (2.2701840660788792) | > loss_kl: 2.8942806720733643 (2.3323088645935055) | > loss_feat: 3.7908427715301514 (4.089455101706764) | > loss_mel: 20.301292419433594 (18.515806492892185) | > loss_duration: 1.6623345613479614 (1.5698774619535967) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.74222183227539 (28.777632106434215) | > grad_norm_1: tensor(9096.1602, device='cuda:0') (tensor(5683.3765, device='cuda:0')) | > current_lr_0: 0.0001726356269045242 | > current_lr_1: 0.0001726356269045242 | > step_time: 0.4607 (0.44189650362188165) | > loader_time: 0.0093 (0.007761075279929421) --> EVAL PERFORMANCE | > avg_loader_time: 0.00848369531228509 (-0.00023239095446089125) | > avg_step_time: 0.45294153522437725 (+0.0006899531458464669) | > avg_loss_disc: 2.5713438383290463 (-0.06787550281470933) | > avg_loss_disc_real_0: 0.10304978878145486 (-0.022206861866821706) | > avg_loss_disc_real_1: 0.22239011652033097 (-0.009681509624064777) | > avg_loss_disc_real_2: 0.26502335932053317 (+0.04531901277286912) | > avg_loss_disc_real_3: 0.241004380122037 (+0.008891831191492755) | > avg_loss_disc_real_4: 0.25460914050189554 (+0.009621574845112668) | > avg_loss_disc_real_5: 0.2454468267904201 (-0.0007708855498005096) | > avg_loss_0: 2.5713438383290463 (-0.06787550281470933) | > avg_grad_norm_0: tensor(212.7217, device='cuda:0') (+tensor(19.5645, device='cuda:0')) | > avg_loss_gen: 2.3089489970408694 (+0.1868114672916037) | > avg_loss_kl: 2.3860107777823862 (+0.32538060571106264) | > avg_loss_feat: 4.0666552160827205 (+0.280479380782221) | > avg_loss_mel: 18.62323773075158 (+1.700988124793696) | > avg_loss_duration: 1.5844706028280124 (+0.03736321187355163) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.96932335974465 (+2.5310226494157817) | > avg_grad_norm_1: tensor(5748.1631, device='cuda:0') (+tensor(268.0254, device='cuda:0'))  > EPOCH: 1178/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:20:50)   --> TIME: 2025-02-25 20:21:07 -- STEP: 33/72 -- GLOBAL_STEP: 1084850 | > loss_disc: 2.3410937786102295 (2.4472457495602695) | > loss_disc_real_0: 0.08487145602703094 (0.05213735433238925) | > loss_disc_real_1: 0.21842831373214722 (0.22404948193015475) | > loss_disc_real_2: 0.21317864954471588 (0.22061694345690988) | > loss_disc_real_3: 0.20614255964756012 (0.22570371672962652) | > loss_disc_real_4: 0.217208132147789 (0.22794137308091827) | > loss_disc_real_5: 0.2278222143650055 (0.23539543197010504) | > loss_0: 2.3410937786102295 (2.4472457495602695) | > grad_norm_0: tensor(293.0053, device='cuda:0') (tensor(182.0042, device='cuda:0')) | > loss_gen: 2.446540355682373 (2.256083831642613) | > loss_kl: 2.0494534969329834 (2.0878083850398204) | > loss_feat: 4.82352876663208 (4.253679651202577) | > loss_mel: 17.505393981933594 (17.767263932661578) | > loss_duration: 1.503196120262146 (1.52024097514875) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.328113555908203 (27.885076754020922) | > grad_norm_1: tensor(3322.8232, device='cuda:0') (tensor(6291.6353, device='cuda:0')) | > current_lr_0: 0.00017261404745116113 | > current_lr_1: 0.00017261404745116113 | > step_time: 0.4354 (0.432947267185558) | > loader_time: 0.0081 (0.007170641061031457) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647898553122936 (+0.0001642032408378461) | > avg_step_time: 0.45022642780357686 (-0.002715107420800389) | > avg_loss_disc: 2.4729048500598316 (-0.09843898826921471) | > avg_loss_disc_real_0: 0.06126803616192979 (-0.041781752619525075) | > avg_loss_disc_real_1: 0.22026254509536314 (-0.0021275714249678335) | > avg_loss_disc_real_2: 0.22386825735300359 (-0.041155101967529584) | > avg_loss_disc_real_3: 0.22815851762261188 (-0.012845862499425126) | > avg_loss_disc_real_4: 0.23473675800880917 (-0.01987238249308637) | > avg_loss_disc_real_5: 0.2414424440390627 (-0.004004382751357394) | > avg_loss_0: 2.4729048500598316 (-0.09843898826921471) | > avg_grad_norm_0: tensor(195.9053, device='cuda:0') (tensor(-16.8164, device='cuda:0')) | > avg_loss_gen: 2.24698742006866 (-0.061961576972209276) | > avg_loss_kl: 2.0951937923968678 (-0.2908169853855185) | > avg_loss_feat: 4.107428732052654 (+0.04077351596993317) | > avg_loss_mel: 17.291361634160438 (-1.3318760965911416) | > avg_loss_duration: 1.5509361079041386 (-0.03353449492387384) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.291907914927307 (-1.677415444817342) | > avg_grad_norm_1: tensor(6317.0098, device='cuda:0') (+tensor(568.8467, device='cuda:0'))  > EPOCH: 1179/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:21:27)   --> TIME: 2025-02-25 20:21:34 -- STEP: 11/72 -- GLOBAL_STEP: 1084900 | > loss_disc: 2.636896848678589 (2.4813933589241723) | > loss_disc_real_0: 0.051606371998786926 (0.06598635051738132) | > loss_disc_real_1: 0.18508829176425934 (0.2189151644706726) | > loss_disc_real_2: 0.22771045565605164 (0.22662441026080737) | > loss_disc_real_3: 0.25074392557144165 (0.23976039344614203) | > loss_disc_real_4: 0.2625597417354584 (0.2397394911809401) | > loss_disc_real_5: 0.2671036422252655 (0.24211294407194311) | > loss_0: 2.636896848678589 (2.4813933589241723) | > grad_norm_0: tensor(105.6556, device='cuda:0') (tensor(174.9548, device='cuda:0')) | > loss_gen: 1.9649325609207153 (2.275477647781372) | > loss_kl: 2.071268081665039 (1.8632600090720437) | > loss_feat: 3.4328179359436035 (4.148968848315152) | > loss_mel: 16.503128051757812 (16.89598343589089) | > loss_duration: 1.4847757816314697 (1.499093478376215) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.45692253112793 (26.682783473621715) | > grad_norm_1: tensor(7786.4634, device='cuda:0') (tensor(6600.4434, device='cuda:0')) | > current_lr_0: 0.00017259247069522973 | > current_lr_1: 0.00017259247069522973 | > step_time: 0.4181 (0.42744101177562366) | > loader_time: 0.0057 (0.006308317184448242)  --> TIME: 2025-02-25 20:21:57 -- STEP: 61/72 -- GLOBAL_STEP: 1084950 | > loss_disc: 2.3746914863586426 (2.506569014220941) | > loss_disc_real_0: 0.13164733350276947 (0.07543300037257004) | > loss_disc_real_1: 0.17419715225696564 (0.21470695330959852) | > loss_disc_real_2: 0.2271924614906311 (0.23699105665331982) | > loss_disc_real_3: 0.22382107377052307 (0.2363810502603406) | > loss_disc_real_4: 0.23901331424713135 (0.24439592293051424) | > loss_disc_real_5: 0.2555048167705536 (0.24692572067018415) | > loss_0: 2.3746914863586426 (2.506569014220941) | > grad_norm_0: tensor(155.5743, device='cuda:0') (tensor(228.0392, device='cuda:0')) | > loss_gen: 2.4166622161865234 (2.258521371200437) | > loss_kl: 2.0170044898986816 (1.9798333449441878) | > loss_feat: 4.022181987762451 (4.170082580847817) | > loss_mel: 15.394477844238281 (16.75307492740819) | > loss_duration: 1.5697954893112183 (1.5295652366075358) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.420122146606445 (26.69107762321097) | > grad_norm_1: tensor(2080.7402, device='cuda:0') (tensor(6215.2476, device='cuda:0')) | > current_lr_0: 0.00017259247069522973 | > current_lr_1: 0.00017259247069522973 | > step_time: 0.4755 (0.4450359422652448) | > loader_time: 0.0107 (0.008052974450783651) --> EVAL PERFORMANCE | > avg_loader_time: 0.00851245665214431 (-0.00013544190097862507) | > avg_step_time: 0.452656326159625 (+0.0024298983560481213) | > avg_loss_disc: 2.53278296430346 (+0.05987811424362821) | > avg_loss_disc_real_0: 0.08550677516720663 (+0.024238739005276844) | > avg_loss_disc_real_1: 0.22029293632843125 (+3.039123306811109e-05) | > avg_loss_disc_real_2: 0.23230574479405308 (+0.008437487441049496) | > avg_loss_disc_real_3: 0.23444664835090367 (+0.006288130728291791) | > avg_loss_disc_real_4: 0.24304165995456803 (+0.008304901945758864) | > avg_loss_disc_real_5: 0.24501185329027578 (+0.0035694092512130737) | > avg_loss_0: 2.53278296430346 (+0.05987811424362821) | > avg_grad_norm_0: tensor(242.6498, device='cuda:0') (+tensor(46.7446, device='cuda:0')) | > avg_loss_gen: 2.2276931111241733 (-0.019294308944486804) | > avg_loss_kl: 2.061952459980065 (-0.03324133241680283) | > avg_loss_feat: 4.102702385942698 (-0.004726346109955593) | > avg_loss_mel: 17.0446333549392 (-0.2467282792212373) | > avg_loss_duration: 1.553247243585721 (+0.002311135681582366) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.990228733546296 (-0.3016791813810116) | > avg_grad_norm_1: tensor(6062.3638, device='cuda:0') (tensor(-254.6460, device='cuda:0'))  > EPOCH: 1180/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:22:04)   --> TIME: 2025-02-25 20:22:23 -- STEP: 39/72 -- GLOBAL_STEP: 1085000 | > loss_disc: 2.4355416297912598 (2.5987876134041032) | > loss_disc_real_0: 0.14971241354942322 (0.11748906100789706) | > loss_disc_real_1: 0.24766559898853302 (0.25811610160729825) | > loss_disc_real_2: 0.24681346118450165 (0.23471784534362647) | > loss_disc_real_3: 0.2549726963043213 (0.23580574148740524) | > loss_disc_real_4: 0.23866593837738037 (0.2367261357796498) | > loss_disc_real_5: 0.2602512538433075 (0.2496869437969648) | > loss_0: 2.4355416297912598 (2.5987876134041032) | > grad_norm_0: tensor(45.5245, device='cuda:0') (tensor(198.7419, device='cuda:0')) | > loss_gen: 2.619086742401123 (2.286002403650529) | > loss_kl: 2.215940237045288 (2.2436110025797134) | > loss_feat: 4.2300496101379395 (4.0764761460133085) | > loss_mel: 17.91452407836914 (18.05651488670936) | > loss_duration: 1.5262680053710938 (1.539563279885512) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.505868911743164 (28.202167951143704) | > grad_norm_1: tensor(2885.2141, device='cuda:0') (tensor(6871.3472, device='cuda:0')) | > current_lr_0: 0.00017257089663639283 | > current_lr_1: 0.00017257089663639283 | > step_time: 0.4394 (0.4323572562291072) | > loader_time: 0.0079 (0.007225427872095353) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1085000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008553387413562184 (+4.093076141787298e-05) | > avg_step_time: 0.4473212739111672 (-0.005335052248457761) | > avg_loss_disc: 2.5807619800030346 (+0.047979015699574745) | > avg_loss_disc_real_0: 0.11067211612219542 (+0.02516534095498879) | > avg_loss_disc_real_1: 0.2425943061079778 (+0.022301369779546554) | > avg_loss_disc_real_2: 0.23185653344426357 (-0.00044921134978950783) | > avg_loss_disc_real_3: 0.23153996362652576 (-0.0029066847243779115) | > avg_loss_disc_real_4: 0.23858495308479793 (-0.004456706869770094) | > avg_loss_disc_real_5: 0.2498130976734027 (+0.004801244383126929) | > avg_loss_0: 2.5807619800030346 (+0.047979015699574745) | > avg_grad_norm_0: tensor(227.4511, device='cuda:0') (tensor(-15.1988, device='cuda:0')) | > avg_loss_gen: 2.218022151732109 (-0.009670959392064304) | > avg_loss_kl: 2.1921849989555255 (+0.1302325389754606) | > avg_loss_feat: 3.9329678810818085 (-0.1697345048608896) | > avg_loss_mel: 17.583387186829473 (+0.538753831890272) | > avg_loss_duration: 1.5602860568274917 (+0.007038813241770692) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.486848267031387 (+0.4966195334850916) | > avg_grad_norm_1: tensor(7766.2109, device='cuda:0') (+tensor(1703.8472, device='cuda:0'))  > EPOCH: 1181/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:22:43)   --> TIME: 2025-02-25 20:22:52 -- STEP: 17/72 -- GLOBAL_STEP: 1085050 | > loss_disc: 2.5554351806640625 (2.499778088401346) | > loss_disc_real_0: 0.08390888571739197 (0.10438132724341224) | > loss_disc_real_1: 0.21479205787181854 (0.2148200098206015) | > loss_disc_real_2: 0.2499900907278061 (0.23078489566550536) | > loss_disc_real_3: 0.23979322612285614 (0.22615702362621531) | > loss_disc_real_4: 0.2553713321685791 (0.24183539783253388) | > loss_disc_real_5: 0.25310221314430237 (0.2454596775419572) | > loss_0: 2.5554351806640625 (2.499778088401346) | > grad_norm_0: tensor(236.3900, device='cuda:0') (tensor(268.9048, device='cuda:0')) | > loss_gen: 1.9696763753890991 (2.222766742986791) | > loss_kl: 1.901354432106018 (1.8882478756063126) | > loss_feat: 3.57216215133667 (3.939999776728013) | > loss_mel: 16.58226203918457 (16.894911317264334) | > loss_duration: 1.5188508033752441 (1.4985578831504374) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.5443058013916 (26.44448403751149) | > grad_norm_1: tensor(12504.1406, device='cuda:0') (tensor(9413.3779, device='cuda:0')) | > current_lr_0: 0.00017254932527431327 | > current_lr_1: 0.00017254932527431327 | > step_time: 0.4223 (0.42846958777483773) | > loader_time: 0.0063 (0.006542416179881376)  --> TIME: 2025-02-25 20:23:16 -- STEP: 67/72 -- GLOBAL_STEP: 1085100 | > loss_disc: 2.7839367389678955 (2.52865330140982) | > loss_disc_real_0: 0.052277229726314545 (0.09880514520762572) | > loss_disc_real_1: 0.20477847754955292 (0.21893079512154878) | > loss_disc_real_2: 0.2344076782464981 (0.2220833808628481) | > loss_disc_real_3: 0.25056785345077515 (0.2273122087788226) | > loss_disc_real_4: 0.21878278255462646 (0.23787223892425424) | > loss_disc_real_5: 0.24198739230632782 (0.243985726317363) | > loss_0: 2.7839367389678955 (2.52865330140982) | > grad_norm_0: tensor(311.4214, device='cuda:0') (tensor(267.9888, device='cuda:0')) | > loss_gen: 1.8657054901123047 (2.1678835224749444) | > loss_kl: 2.1307356357574463 (1.971468128375153) | > loss_feat: 3.7277369499206543 (3.924945870442177) | > loss_mel: 16.770246505737305 (16.624141038353756) | > loss_duration: 1.6380985975265503 (1.535341990527822) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.132522583007812 (26.223780503913538) | > grad_norm_1: tensor(16295.5537, device='cuda:0') (tensor(8598.4180, device='cuda:0')) | > current_lr_0: 0.00017254932527431327 | > current_lr_1: 0.00017254932527431327 | > step_time: 0.4847 (0.45263483986925723) | > loader_time: 0.0122 (0.008334408945112089) --> EVAL PERFORMANCE | > avg_loader_time: 0.00851107651079205 (-4.231090277013322e-05) | > avg_step_time: 0.4550524966817507 (+0.007731222770583468) | > avg_loss_disc: 2.5239547306383154 (-0.05680724936471915) | > avg_loss_disc_real_0: 0.09820571289935583 (-0.012466403222839595) | > avg_loss_disc_real_1: 0.21915466768640868 (-0.023439638421569126) | > avg_loss_disc_real_2: 0.22330267517499522 (-0.008553858269268355) | > avg_loss_disc_real_3: 0.22801717884943518 (-0.0035227847770905873) | > avg_loss_disc_real_4: 0.23816324623537735 (-0.00042170684942058334) | > avg_loss_disc_real_5: 0.24395303680023678 (-0.005860060873165929) | > avg_loss_0: 2.5239547306383154 (-0.05680724936471915) | > avg_grad_norm_0: tensor(266.3610, device='cuda:0') (+tensor(38.9099, device='cuda:0')) | > avg_loss_gen: 2.173474306791602 (-0.04454784494050701) | > avg_loss_kl: 1.9798920675062797 (-0.21229293144924588) | > avg_loss_feat: 3.920961235610532 (-0.01200664547127639) | > avg_loss_mel: 16.591208686291324 (-0.9921785005381487) | > avg_loss_duration: 1.538942127160623 (-0.02134392966686871) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.20447841160734 (-1.2823698554240472) | > avg_grad_norm_1: tensor(8616.8916, device='cuda:0') (+tensor(850.6807, device='cuda:0'))  > EPOCH: 1182/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:23:20)   --> TIME: 2025-02-25 20:23:42 -- STEP: 45/72 -- GLOBAL_STEP: 1085150 | > loss_disc: 2.373483180999756 (2.483689488304986) | > loss_disc_real_0: 0.08777336776256561 (0.09072879155476889) | > loss_disc_real_1: 0.2528153657913208 (0.21818406052059597) | > loss_disc_real_2: 0.21252289414405823 (0.2162210914823744) | > loss_disc_real_3: 0.236067995429039 (0.22641556329197354) | > loss_disc_real_4: 0.25656142830848694 (0.23805505302217272) | > loss_disc_real_5: 0.2549641728401184 (0.2435748173130883) | > loss_0: 2.373483180999756 (2.483689488304986) | > grad_norm_0: tensor(243.0412, device='cuda:0') (tensor(311.2167, device='cuda:0')) | > loss_gen: 2.5706934928894043 (2.237115491761102) | > loss_kl: 2.12595796585083 (1.9364744901657105) | > loss_feat: 4.568511486053467 (4.156147125032213) | > loss_mel: 16.935136795043945 (16.681499756707087) | > loss_duration: 1.53141188621521 (1.5050300412707858) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.731712341308594 (26.516267098320856) | > grad_norm_1: tensor(7708.0684, device='cuda:0') (tensor(9445.2168, device='cuda:0')) | > current_lr_0: 0.00017252775660865397 | > current_lr_1: 0.00017252775660865397 | > step_time: 0.4618 (0.4378725104861789) | > loader_time: 0.0089 (0.007440906100802952) --> EVAL PERFORMANCE | > avg_loader_time: 0.008543535017631424 (+3.245850683937339e-05) | > avg_step_time: 0.45202944648097937 (-0.0030230502007713156) | > avg_loss_disc: 2.4970905243510932 (-0.026864206287222192) | > avg_loss_disc_real_0: 0.09461496130261622 (-0.0035907515967396053) | > avg_loss_disc_real_1: 0.21883856494661788 (-0.00031610273979079673) | > avg_loss_disc_real_2: 0.21856144763214488 (-0.004741227542850335) | > avg_loss_disc_real_3: 0.22748912262244964 (-0.0005280562269855404) | > avg_loss_disc_real_4: 0.23689301878633634 (-0.0012702274490410115) | > avg_loss_disc_real_5: 0.24464349448680878 (+0.0006904576865719991) | > avg_loss_0: 2.4970905243510932 (-0.026864206287222192) | > avg_grad_norm_0: tensor(315.8448, device='cuda:0') (+tensor(49.4839, device='cuda:0')) | > avg_loss_gen: 2.2275709904415515 (+0.05409668364994946) | > avg_loss_kl: 1.9757198733343204 (-0.004172194171959243) | > avg_loss_feat: 4.069372704331304 (+0.14841146872077182) | > avg_loss_mel: 16.64035447886292 (+0.04914579257159701) | > avg_loss_duration: 1.5321620450892917 (-0.006780082071331295) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.445180167614577 (+0.24070175600723687) | > avg_grad_norm_1: tensor(9465.9229, device='cuda:0') (+tensor(849.0312, device='cuda:0'))  > EPOCH: 1183/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:23:56)   --> TIME: 2025-02-25 20:24:09 -- STEP: 23/72 -- GLOBAL_STEP: 1085200 | > loss_disc: 2.539961338043213 (2.5082804638406504) | > loss_disc_real_0: 0.10121753811836243 (0.09507997512169507) | > loss_disc_real_1: 0.205801323056221 (0.21962814421757407) | > loss_disc_real_2: 0.20611214637756348 (0.23013436470342719) | > loss_disc_real_3: 0.2488214522600174 (0.23793773806613425) | > loss_disc_real_4: 0.25362643599510193 (0.2480522387701532) | > loss_disc_real_5: 0.26347076892852783 (0.24188815316428308) | > loss_0: 2.539961338043213 (2.5082804638406504) | > grad_norm_0: tensor(282.5415, device='cuda:0') (tensor(324.7030, device='cuda:0')) | > loss_gen: 2.06148624420166 (2.2689894696940547) | > loss_kl: 1.9574363231658936 (1.8810573090677676) | > loss_feat: 3.764462471008301 (4.1733070871104365) | > loss_mel: 16.9050235748291 (16.785218943720285) | > loss_duration: 1.5089383125305176 (1.4906528358874114) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.197345733642578 (26.599225500355598) | > grad_norm_1: tensor(10664.3555, device='cuda:0') (tensor(9158.9863, device='cuda:0')) | > current_lr_0: 0.00017250619063907788 | > current_lr_1: 0.00017250619063907788 | > step_time: 0.4347 (0.43101732627205225) | > loader_time: 0.0081 (0.006684873415076215) --> EVAL PERFORMANCE | > avg_loader_time: 0.008623579858054578 (+8.004484042315428e-05) | > avg_step_time: 0.4526614874181613 (+0.0006320409371819435) | > avg_loss_disc: 2.5582164509195677 (+0.06112592656847449) | > avg_loss_disc_real_0: 0.12243118210577629 (+0.02781622080316007) | > avg_loss_disc_real_1: 0.21587480100947368 (-0.0029637639371442037) | > avg_loss_disc_real_2: 0.2299889559057397 (+0.011427508273594816) | > avg_loss_disc_real_3: 0.23426223661698087 (+0.006773113994531232) | > avg_loss_disc_real_4: 0.24355791656064316 (+0.006664897774306816) | > avg_loss_disc_real_5: 0.24226457678096394 (-0.002378917705844835) | > avg_loss_0: 2.5582164509195677 (+0.06112592656847449) | > avg_grad_norm_0: tensor(326.2399, device='cuda:0') (+tensor(10.3951, device='cuda:0')) | > avg_loss_gen: 2.2447137077089763 (+0.01714271726742478) | > avg_loss_kl: 1.9940718932890555 (+0.01835201995473512) | > avg_loss_feat: 4.091376959438054 (+0.02200425510674986) | > avg_loss_mel: 16.84050623128113 (+0.2001517524182077) | > avg_loss_duration: 1.5372321253091512 (+0.005070080219859552) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.707900947248433 (+0.262720779633856) | > avg_grad_norm_1: tensor(7219.5928, device='cuda:0') (tensor(-2246.3301, device='cuda:0'))  > EPOCH: 1184/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:24:33)   --> TIME: 2025-02-25 20:24:35 -- STEP: 1/72 -- GLOBAL_STEP: 1085250 | > loss_disc: 3.1043813228607178 (3.1043813228607178) | > loss_disc_real_0: 0.18334686756134033 (0.18334686756134033) | > loss_disc_real_1: 0.2730798125267029 (0.2730798125267029) | > loss_disc_real_2: 0.24468889832496643 (0.24468889832496643) | > loss_disc_real_3: 0.28969547152519226 (0.28969547152519226) | > loss_disc_real_4: 0.26757049560546875 (0.26757049560546875) | > loss_disc_real_5: 0.23471994698047638 (0.23471994698047638) | > loss_0: 3.1043813228607178 (3.1043813228607178) | > grad_norm_0: tensor(534.4175, device='cuda:0') (tensor(534.4175, device='cuda:0')) | > loss_gen: 1.8517340421676636 (1.8517340421676636) | > loss_kl: 1.909232497215271 (1.909232497215271) | > loss_feat: 3.2938733100891113 (3.2938733100891113) | > loss_mel: 17.385879516601562 (17.385879516601562) | > loss_duration: 1.511894702911377 (1.511894702911377) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.952613830566406 (25.952613830566406) | > grad_norm_1: tensor(13371.5684, device='cuda:0') (tensor(13371.5684, device='cuda:0')) | > current_lr_0: 0.00017248462736524798 | > current_lr_1: 0.00017248462736524798 | > step_time: 0.4211 (0.4210991859436035) | > loader_time: 0.0048 (0.004845142364501953)  --> TIME: 2025-02-25 20:24:58 -- STEP: 51/72 -- GLOBAL_STEP: 1085300 | > loss_disc: 3.0885987281799316 (2.7348537819058287) | > loss_disc_real_0: 0.4461877644062042 (0.24264174809350686) | > loss_disc_real_1: 0.22833329439163208 (0.22526473420507767) | > loss_disc_real_2: 0.09168892353773117 (0.22822995586138145) | > loss_disc_real_3: 0.11910714954137802 (0.2239922808081496) | > loss_disc_real_4: 0.10296224057674408 (0.2305324451596129) | > loss_disc_real_5: 0.1924593448638916 (0.24272526566888772) | > loss_0: 3.0885987281799316 (2.7348537819058287) | > grad_norm_0: tensor(228.6454, device='cuda:0') (tensor(311.0696, device='cuda:0')) | > loss_gen: 2.063603162765503 (2.1360112054675238) | > loss_kl: 2.1306896209716797 (2.1770663915895954) | > loss_feat: 4.207231521606445 (3.7676433394936955) | > loss_mel: 16.777908325195312 (17.51560940462) | > loss_duration: 1.572472333908081 (1.5584881212197097) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.751903533935547 (27.15481844135359) | > grad_norm_1: tensor(790.9185, device='cuda:0') (tensor(6247.1431, device='cuda:0')) | > current_lr_0: 0.00017248462736524798 | > current_lr_1: 0.00017248462736524798 | > step_time: 0.4562 (0.4379581890854181) | > loader_time: 0.0093 (0.007884465011895872) --> EVAL PERFORMANCE | > avg_loader_time: 0.008731868905080875 (+0.00010828904702629705) | > avg_step_time: 0.44886930895523286 (-0.0037921784629284527) | > avg_loss_disc: 2.7340799754774068 (+0.17586352455783905) | > avg_loss_disc_real_0: 0.22825947398660887 (+0.10582829188083258) | > avg_loss_disc_real_1: 0.2200920132684036 (+0.004217212258929931) | > avg_loss_disc_real_2: 0.21641661940326154 (-0.01357233650247816) | > avg_loss_disc_real_3: 0.227446790522253 (-0.006815446094727867) | > avg_loss_disc_real_4: 0.24440250892034718 (+0.0008445923597040217) | > avg_loss_disc_real_5: 0.24203820400674578 (-0.00022637277421816426) | > avg_loss_0: 2.7340799754774068 (+0.17586352455783905) | > avg_grad_norm_0: tensor(272.0171, device='cuda:0') (tensor(-54.2228, device='cuda:0')) | > avg_loss_gen: 2.2251605567797816 (-0.01955315092919463) | > avg_loss_kl: 2.2006574247924373 (+0.20658553150338177) | > avg_loss_feat: 3.901485201338647 (-0.18989175809940662) | > avg_loss_mel: 17.60822083916462 (+0.7677146078834909) | > avg_loss_duration: 1.572597735364672 (+0.03536561005552086) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.508121785983235 (+0.800220838734802) | > avg_grad_norm_1: tensor(5473.6982, device='cuda:0') (tensor(-1745.8945, device='cuda:0'))  > EPOCH: 1185/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:25:10)   --> TIME: 2025-02-25 20:25:25 -- STEP: 29/72 -- GLOBAL_STEP: 1085350 | > loss_disc: 2.5366320610046387 (2.589348529947215) | > loss_disc_real_0: 0.11981192231178284 (0.10047521146720853) | > loss_disc_real_1: 0.20665696263313293 (0.22300931246116243) | > loss_disc_real_2: 0.24498307704925537 (0.2515029100508526) | > loss_disc_real_3: 0.21972429752349854 (0.22586267261669554) | > loss_disc_real_4: 0.26756349205970764 (0.21627065796276618) | > loss_disc_real_5: 0.2717253267765045 (0.22963499765971612) | > loss_0: 2.5366320610046387 (2.589348529947215) | > grad_norm_0: tensor(60.5059, device='cuda:0') (tensor(217.5595, device='cuda:0')) | > loss_gen: 2.2743730545043945 (2.1561659163442157) | > loss_kl: 1.9961845874786377 (1.9885404027741531) | > loss_feat: 3.9895904064178467 (3.9828438758850098) | > loss_mel: 17.437562942504883 (17.05385290343186) | > loss_duration: 1.508223056793213 (1.5068293686570793) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.205934524536133 (26.688232356104358) | > grad_norm_1: tensor(3848.9644, device='cuda:0') (tensor(6600.2051, device='cuda:0')) | > current_lr_0: 0.00017246306678682732 | > current_lr_1: 0.00017246306678682732 | > step_time: 0.4344 (0.4328533534345956) | > loader_time: 0.0071 (0.006967832302225047) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630984266039352 (-0.00010088463904152356) | > avg_step_time: 0.4521370740003989 (+0.0032677650451660156) | > avg_loss_disc: 2.573329509144098 (-0.16075046633330858) | > avg_loss_disc_real_0: 0.09609260341860879 (-0.13216687056800008) | > avg_loss_disc_real_1: 0.25177514888870894 (+0.031683135620305336) | > avg_loss_disc_real_2: 0.23933894008817813 (+0.02292232068491659) | > avg_loss_disc_real_3: 0.22995939263155762 (+0.0025126021093046236) | > avg_loss_disc_real_4: 0.23333753057768647 (-0.011064978342660708) | > avg_loss_disc_real_5: 0.23685061931610107 (-0.005187584690644703) | > avg_loss_0: 2.573329509144098 (-0.16075046633330858) | > avg_grad_norm_0: tensor(211.6129, device='cuda:0') (tensor(-60.4042, device='cuda:0')) | > avg_loss_gen: 2.211459957378012 (-0.013700599401769598) | > avg_loss_kl: 2.0120703015528933 (-0.18858712323954396) | > avg_loss_feat: 3.9407689571380615 (+0.039283755799414344) | > avg_loss_mel: 16.859517634754457 (-0.7487032044101625) | > avg_loss_duration: 1.5434379275415984 (-0.029159807823073702) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.567254832093145 (-0.9408669538900902) | > avg_grad_norm_1: tensor(6228.8838, device='cuda:0') (+tensor(755.1855, device='cuda:0'))  > EPOCH: 1186/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:25:47)   --> TIME: 2025-02-25 20:25:52 -- STEP: 7/72 -- GLOBAL_STEP: 1085400 | > loss_disc: 2.8951032161712646 (2.6425768647875105) | > loss_disc_real_0: 0.03890343755483627 (0.055697403848171234) | > loss_disc_real_1: 0.18453890085220337 (0.2046459444931575) | > loss_disc_real_2: 0.12401411682367325 (0.18810439429112844) | > loss_disc_real_3: 0.21082884073257446 (0.21851147285529546) | > loss_disc_real_4: 0.20230847597122192 (0.22420442317213332) | > loss_disc_real_5: 0.2338990569114685 (0.23742054402828217) | > loss_0: 2.8951032161712646 (2.6425768647875105) | > grad_norm_0: tensor(282.9157, device='cuda:0') (tensor(227.4256, device='cuda:0')) | > loss_gen: 1.5791223049163818 (1.9152772086007255) | > loss_kl: 1.9705042839050293 (1.8937681061880929) | > loss_feat: 3.69824481010437 (4.051044089453561) | > loss_mel: 17.816673278808594 (16.840367589678085) | > loss_duration: 1.4683574438095093 (1.4967031819479806) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.532901763916016 (26.197160175868444) | > grad_norm_1: tensor(11950.3096, device='cuda:0') (tensor(6526.2495, device='cuda:0')) | > current_lr_0: 0.00017244150890347897 | > current_lr_1: 0.00017244150890347897 | > step_time: 0.4212 (0.4228128365107945) | > loader_time: 0.0055 (0.006555250712803432)  --> TIME: 2025-02-25 20:26:15 -- STEP: 57/72 -- GLOBAL_STEP: 1085450 | > loss_disc: 2.51924729347229 (2.6904839089042256) | > loss_disc_real_0: 0.15721753239631653 (0.14669575169682506) | > loss_disc_real_1: 0.30995920300483704 (0.24406350913800692) | > loss_disc_real_2: 0.22538620233535767 (0.2165704292424938) | > loss_disc_real_3: 0.2558997571468353 (0.2318384955849564) | > loss_disc_real_4: 0.2670082449913025 (0.23334763395158867) | > loss_disc_real_5: 0.26898685097694397 (0.23919362797026048) | > loss_0: 2.51924729347229 (2.6904839089042256) | > grad_norm_0: tensor(126.2334, device='cuda:0') (tensor(198.6339, device='cuda:0')) | > loss_gen: 2.506821632385254 (2.0135899175677388) | > loss_kl: 2.1022837162017822 (2.196916563469066) | > loss_feat: 3.9006595611572266 (3.646829605102539) | > loss_mel: 16.885658264160156 (17.76864133801376) | > loss_duration: 1.6110031604766846 (1.6094540357589722) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.006427764892578 (27.235431503831293) | > grad_norm_1: tensor(5756.0581, device='cuda:0') (tensor(7113.1558, device='cuda:0')) | > current_lr_0: 0.00017244150890347897 | > current_lr_1: 0.00017244150890347897 | > step_time: 0.4609 (0.44214296340942383) | > loader_time: 0.0096 (0.008051809511686625) --> EVAL PERFORMANCE | > avg_loader_time: 0.008663650969384424 (+3.266670334507206e-05) | > avg_step_time: 0.45398823644073916 (+0.0018511624403402815) | > avg_loss_disc: 2.6750410610521342 (+0.10171155190803605) | > avg_loss_disc_real_0: 0.1445341950780909 (+0.048441591659482114) | > avg_loss_disc_real_1: 0.24268987879786694 (-0.009085270090842007) | > avg_loss_disc_real_2: 0.21856652273678442 (-0.020772417351393707) | > avg_loss_disc_real_3: 0.23143762538970356 (+0.001478232758145931) | > avg_loss_disc_real_4: 0.23448029160499573 (+0.001142761027309258) | > avg_loss_disc_real_5: 0.24123007310947903 (+0.004379453793377952) | > avg_loss_0: 2.6750410610521342 (+0.10171155190803605) | > avg_grad_norm_0: tensor(207.3039, device='cuda:0') (tensor(-4.3090, device='cuda:0')) | > avg_loss_gen: 2.017982414070989 (-0.19347754330702305) | > avg_loss_kl: 2.333105413007064 (+0.3210351114541705) | > avg_loss_feat: 3.601591684448887 (-0.33917727268917464) | > avg_loss_mel: 18.27183740911349 (+1.4123197743590339) | > avg_loss_duration: 1.6248491011874777 (+0.08141117364587935) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.84936609402509 (+1.282111261931945) | > avg_grad_norm_1: tensor(6689.6655, device='cuda:0') (+tensor(460.7817, device='cuda:0'))  > EPOCH: 1187/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:26:23)   --> TIME: 2025-02-25 20:26:41 -- STEP: 35/72 -- GLOBAL_STEP: 1085500 | > loss_disc: 2.6604843139648438 (2.4577959537506104) | > loss_disc_real_0: 0.11013321578502655 (0.08610572846872466) | > loss_disc_real_1: 0.29109492897987366 (0.21688774185521262) | > loss_disc_real_2: 0.30893674492836 (0.21867020215306962) | > loss_disc_real_3: 0.2967371642589569 (0.2200990391629083) | > loss_disc_real_4: 0.26123854517936707 (0.23300370829445974) | > loss_disc_real_5: 0.24677938222885132 (0.22744932217257363) | > loss_0: 2.6604843139648438 (2.4577959537506104) | > grad_norm_0: tensor(489.0658, device='cuda:0') (tensor(274.9316, device='cuda:0')) | > loss_gen: 2.3223536014556885 (2.186306173460823) | > loss_kl: 2.392605781555176 (2.364905568531581) | > loss_feat: 3.6883978843688965 (4.268277924401421) | > loss_mel: 18.418149948120117 (19.010699735369002) | > loss_duration: 1.5844868421554565 (1.563762889589582) | > amp_scaler: 16.0 (13.942857142857143) | > loss_1: 28.40599250793457 (29.39395261492048) | > grad_norm_1: tensor(10388.9971, device='cuda:0') (tensor(8221.6533, device='cuda:0')) | > current_lr_0: 0.00017241995371486602 | > current_lr_1: 0.00017241995371486602 | > step_time: 0.4443 (0.43646085602896556) | > loader_time: 0.0104 (0.007302216121128627) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866310025604678 (-5.507133376443651e-07) | > avg_step_time: 0.4514737767232975 (-0.0025144597174416305) | > avg_loss_disc: 2.5489447351912387 (-0.1260963258608956) | > avg_loss_disc_real_0: 0.09244461189693129 (-0.05208958318115961) | > avg_loss_disc_real_1: 0.21929201945452617 (-0.023397859343340766) | > avg_loss_disc_real_2: 0.22228049665269717 (+0.003713973915912744) | > avg_loss_disc_real_3: 0.23167031184888223 (+0.00023268645917867703) | > avg_loss_disc_real_4: 0.24595146909565993 (+0.011471177490664203) | > avg_loss_disc_real_5: 0.24578479331144146 (+0.004554720201962431) | > avg_loss_0: 2.5489447351912387 (-0.1260963258608956) | > avg_grad_norm_0: tensor(285.8443, device='cuda:0') (+tensor(78.5403, device='cuda:0')) | > avg_loss_gen: 2.1526670220872033 (+0.1346846080162143) | > avg_loss_kl: 2.3613174834721518 (+0.02821207046508789) | > avg_loss_feat: 4.09830414073568 (+0.49671245628679284) | > avg_loss_mel: 18.705969367228764 (+0.43413195811527316) | > avg_loss_duration: 1.5867195633095754 (-0.0381295378779023) | > avg_amp_scaler: 11.605633802816902 (+3.605633802816902) | > avg_loss_1: 28.904977798461914 (+1.0556117044368243) | > avg_grad_norm_1: tensor(8200.7529, device='cuda:0') (+tensor(1511.0874, device='cuda:0'))  > EPOCH: 1188/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:27:00)   --> TIME: 2025-02-25 20:27:08 -- STEP: 13/72 -- GLOBAL_STEP: 1085550 | > loss_disc: 2.463243246078491 (2.567983792378352) | > loss_disc_real_0: 0.03608914464712143 (0.0884642179768819) | > loss_disc_real_1: 0.25180017948150635 (0.22091358326948607) | > loss_disc_real_2: 0.19378100335597992 (0.23749073193623468) | > loss_disc_real_3: 0.2208119034767151 (0.22353855807047623) | > loss_disc_real_4: 0.2728140950202942 (0.2342136032306231) | > loss_disc_real_5: 0.24515053629875183 (0.2281861179150068) | > loss_0: 2.463243246078491 (2.567983792378352) | > grad_norm_0: tensor(299.4350, device='cuda:0') (tensor(319.6819, device='cuda:0')) | > loss_gen: 2.1055285930633545 (2.1123442283043494) | > loss_kl: 2.1812124252319336 (2.1120672959547777) | > loss_feat: 4.129340648651123 (4.037165128267729) | > loss_mel: 17.076126098632812 (18.070176197932316) | > loss_duration: 1.4390946626663208 (1.5243119093088002) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.93130111694336 (27.856064576369064) | > grad_norm_1: tensor(11597.1689, device='cuda:0') (tensor(7762.3369, device='cuda:0')) | > current_lr_0: 0.00017239840122065166 | > current_lr_1: 0.00017239840122065166 | > step_time: 0.4438 (0.42897653579711914) | > loader_time: 0.007 (0.006901869407066932)  --> TIME: 2025-02-25 20:27:31 -- STEP: 63/72 -- GLOBAL_STEP: 1085600 | > loss_disc: 2.6865203380584717 (2.5968520603482683) | > loss_disc_real_0: 0.16667123138904572 (0.0913000860739322) | > loss_disc_real_1: 0.24524752795696259 (0.2207053724735502) | > loss_disc_real_2: 0.1855245977640152 (0.24657631961126186) | > loss_disc_real_3: 0.2220168113708496 (0.23153802090220982) | > loss_disc_real_4: 0.23171062767505646 (0.24730966701394036) | > loss_disc_real_5: 0.2367042750120163 (0.23880768366276273) | > loss_0: 2.6865203380584717 (2.5968520603482683) | > grad_norm_0: tensor(1061.1740, device='cuda:0') (tensor(317.5301, device='cuda:0')) | > loss_gen: 2.017333507537842 (2.1097403991790045) | > loss_kl: 1.951980471611023 (2.0791614017789315) | > loss_feat: 3.920039653778076 (3.8997520151592435) | > loss_mel: 16.837339401245117 (17.29529120430114) | > loss_duration: 1.5760629177093506 (1.5479475978821042) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.302757263183594 (26.93189251612103) | > grad_norm_1: tensor(14859.6240, device='cuda:0') (tensor(8352.5078, device='cuda:0')) | > current_lr_0: 0.00017239840122065166 | > current_lr_1: 0.00017239840122065166 | > step_time: 0.5191 (0.4436875789884537) | > loader_time: 0.0113 (0.008283316142975337) --> EVAL PERFORMANCE | > avg_loader_time: 0.0086851287895525 (+2.202853350572083e-05) | > avg_step_time: 0.451587052412436 (+0.00011327568913849229) | > avg_loss_disc: 2.590433594206689 (+0.04148885901545052) | > avg_loss_disc_real_0: 0.08888372899570936 (-0.0035608829012219317) | > avg_loss_disc_real_1: 0.22071752350934795 (+0.0014255040548217801) | > avg_loss_disc_real_2: 0.24423395118243268 (+0.021953454529735517) | > avg_loss_disc_real_3: 0.23111321850561758 (-0.0005570933432646474) | > avg_loss_disc_real_4: 0.24519180999675266 (-0.0007596590989072671) | > avg_loss_disc_real_5: 0.23766943665457443 (-0.008115356656867023) | > avg_loss_0: 2.590433594206689 (+0.04148885901545052) | > avg_grad_norm_0: tensor(304.8637, device='cuda:0') (+tensor(19.0195, device='cuda:0')) | > avg_loss_gen: 2.1066697701601913 (-0.045997251927011984) | > avg_loss_kl: 2.0876228507135948 (-0.273694632758557) | > avg_loss_feat: 3.885028207805795 (-0.21327593292988478) | > avg_loss_mel: 17.26203060150147 (-1.443938765727296) | > avg_loss_duration: 1.5550334218522193 (-0.031686141457356154) | > avg_amp_scaler: 8.0 (-3.605633802816902) | > avg_loss_1: 26.896384789910115 (-2.0085930085517987) | > avg_grad_norm_1: tensor(8303.2207, device='cuda:0') (+tensor(102.4678, device='cuda:0'))  > EPOCH: 1189/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:27:37)   --> TIME: 2025-02-25 20:27:58 -- STEP: 41/72 -- GLOBAL_STEP: 1085650 | > loss_disc: 2.463228225708008 (2.5244432949438327) | > loss_disc_real_0: 0.07080116868019104 (0.08838309188623256) | > loss_disc_real_1: 0.21752040088176727 (0.21706217818143891) | > loss_disc_real_2: 0.21968726813793182 (0.23529798083189057) | > loss_disc_real_3: 0.2256818562746048 (0.23801398604381374) | > loss_disc_real_4: 0.23495222628116608 (0.24415429845088865) | > loss_disc_real_5: 0.2344079315662384 (0.24917132316566099) | > loss_0: 2.463228225708008 (2.5244432949438327) | > grad_norm_0: tensor(191.1918, device='cuda:0') (tensor(338.9285, device='cuda:0')) | > loss_gen: 2.1197266578674316 (2.1975824076954913) | > loss_kl: 1.9455389976501465 (1.9460751196233237) | > loss_feat: 4.0656585693359375 (4.008934451312552) | > loss_mel: 17.244951248168945 (16.966705345525973) | > loss_duration: 1.5461677312850952 (1.5167522692098852) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.922042846679688 (26.636049456712676) | > grad_norm_1: tensor(8129.4775, device='cuda:0') (tensor(9740.6230, device='cuda:0')) | > current_lr_0: 0.00017237685142049906 | > current_lr_1: 0.00017237685142049906 | > step_time: 0.445 (0.43830601761980753) | > loader_time: 0.0091 (0.0072142612643358184) --> EVAL PERFORMANCE | > avg_loader_time: 0.008543595461778238 (-0.00014153332777426184) | > avg_step_time: 0.4513915928316788 (-0.00019545958075722902) | > avg_loss_disc: 2.521882826173808 (-0.06855076803288096) | > avg_loss_disc_real_0: 0.08775939847486959 (-0.001124330520839767) | > avg_loss_disc_real_1: 0.21870500583883742 (-0.0020125176705105274) | > avg_loss_disc_real_2: 0.23646522260887523 (-0.007768728573557454) | > avg_loss_disc_real_3: 0.23318298370905324 (+0.0020697652034356584) | > avg_loss_disc_real_4: 0.24389435057069214 (-0.001297459426060521) | > avg_loss_disc_real_5: 0.24855383705924933 (+0.010884400404674893) | > avg_loss_0: 2.521882826173808 (-0.06855076803288096) | > avg_grad_norm_0: tensor(324.6177, device='cuda:0') (+tensor(19.7539, device='cuda:0')) | > avg_loss_gen: 2.204082767728349 (+0.09741299756815769) | > avg_loss_kl: 1.9966590824261519 (-0.0909637682874429) | > avg_loss_feat: 3.9975291910305812 (+0.1125009832247863) | > avg_loss_mel: 16.866402182780526 (-0.3956284187209427) | > avg_loss_duration: 1.5435227877657178 (-0.011510634086501437) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.608195936176138 (-0.2881888537339776) | > avg_grad_norm_1: tensor(9376.1455, device='cuda:0') (+tensor(1072.9248, device='cuda:0'))  > EPOCH: 1190/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:28:14)   --> TIME: 2025-02-25 20:28:24 -- STEP: 19/72 -- GLOBAL_STEP: 1085700 | > loss_disc: 2.491605758666992 (2.5088210105895996) | > loss_disc_real_0: 0.062116801738739014 (0.08642857110029772) | > loss_disc_real_1: 0.20896907150745392 (0.2154091361321901) | > loss_disc_real_2: 0.23704366385936737 (0.23330282380706385) | > loss_disc_real_3: 0.20784509181976318 (0.22865124909501328) | > loss_disc_real_4: 0.22567833960056305 (0.2414110491150304) | > loss_disc_real_5: 0.2168256640434265 (0.24400412252074793) | > loss_0: 2.491605758666992 (2.5088210105895996) | > grad_norm_0: tensor(106.0301, device='cuda:0') (tensor(261.8004, device='cuda:0')) | > loss_gen: 2.1980624198913574 (2.2343926931682385) | > loss_kl: 1.885584831237793 (1.925083875656128) | > loss_feat: 3.9291627407073975 (4.062594062403629) | > loss_mel: 16.25212860107422 (16.72557087948448) | > loss_duration: 1.5065815448760986 (1.4936146673403288) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.771520614624023 (26.441256372552168) | > grad_norm_1: tensor(9348.9268, device='cuda:0') (tensor(8253.1748, device='cuda:0')) | > current_lr_0: 0.00017235530431407149 | > current_lr_1: 0.00017235530431407149 | > step_time: 0.4184 (0.4256184979488975) | > loader_time: 0.0061 (0.006535856347335012)  --> TIME: 2025-02-25 20:28:48 -- STEP: 69/72 -- GLOBAL_STEP: 1085750 | > loss_disc: 2.4902877807617188 (2.5173787656037705) | > loss_disc_real_0: 0.13536053895950317 (0.09729869966057764) | > loss_disc_real_1: 0.22530458867549896 (0.21998105381709943) | > loss_disc_real_2: 0.2344556748867035 (0.23032658968282782) | > loss_disc_real_3: 0.24110276997089386 (0.2319665140863778) | > loss_disc_real_4: 0.22778534889221191 (0.24267585532388825) | > loss_disc_real_5: 0.23931899666786194 (0.24506806420243304) | > loss_0: 2.4902877807617188 (2.5173787656037705) | > grad_norm_0: tensor(415.1903, device='cuda:0') (tensor(288.3387, device='cuda:0')) | > loss_gen: 2.408439874649048 (2.227402308712835) | > loss_kl: 2.1286654472351074 (1.981120911197386) | > loss_feat: 4.231916427612305 (3.956362112708714) | > loss_mel: 17.47794532775879 (16.70473816083825) | > loss_duration: 1.646380066871643 (1.537462421085524) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.893346786499023 (26.407085943913113) | > grad_norm_1: tensor(6175.4028, device='cuda:0') (tensor(7826.6992, device='cuda:0')) | > current_lr_0: 0.00017235530431407149 | > current_lr_1: 0.00017235530431407149 | > step_time: 0.5216 (0.4473982064620308) | > loader_time: 0.0128 (0.008589205534561823) --> EVAL PERFORMANCE | > avg_loader_time: 0.008642932059059681 (+9.933659728144327e-05) | > avg_step_time: 0.4488070380519813 (-0.00258455477969749) | > avg_loss_disc: 2.517358168749742 (-0.00452465742406627) | > avg_loss_disc_real_0: 0.09663852005147598 (+0.008879121576606391) | > avg_loss_disc_real_1: 0.22043877861029665 (+0.0017337727714592244) | > avg_loss_disc_real_2: 0.2301979940122282 (-0.006267228596647023) | > avg_loss_disc_real_3: 0.23145312910348598 (-0.0017298546055672659) | > avg_loss_disc_real_4: 0.24201112985610962 (-0.001883220714582523) | > avg_loss_disc_real_5: 0.24469477190098293 (-0.003859065158266395) | > avg_loss_0: 2.517358168749742 (-0.00452465742406627) | > avg_grad_norm_0: tensor(286.4280, device='cuda:0') (tensor(-38.1896, device='cuda:0')) | > avg_loss_gen: 2.224513072363088 (+0.02043030463473894) | > avg_loss_kl: 1.9830682244099362 (-0.013590858016215623) | > avg_loss_feat: 3.95637589105418 (-0.04115329997640105) | > avg_loss_mel: 16.70413464559636 (-0.1622675371841673) | > avg_loss_duration: 1.538108133933914 (-0.0054146538318038395) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.406200059702698 (-0.20199587647343975) | > avg_grad_norm_1: tensor(7788.0991, device='cuda:0') (tensor(-1588.0464, device='cuda:0'))  > EPOCH: 1191/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:28:50)   --> TIME: 2025-02-25 20:29:13 -- STEP: 47/72 -- GLOBAL_STEP: 1085800 | > loss_disc: 2.8138747215270996 (2.517402019906551) | > loss_disc_real_0: 0.0665884017944336 (0.0912296777947786) | > loss_disc_real_1: 0.22045177221298218 (0.21896648850846798) | > loss_disc_real_2: 0.1850939393043518 (0.2308000237383741) | > loss_disc_real_3: 0.15094301104545593 (0.2251637134146183) | > loss_disc_real_4: 0.2452622801065445 (0.24105861275754076) | > loss_disc_real_5: 0.23999550938606262 (0.24217966928126963) | > loss_0: 2.8138747215270996 (2.517402019906551) | > grad_norm_0: tensor(485.2039, device='cuda:0') (tensor(273.7374, device='cuda:0')) | > loss_gen: 1.684400200843811 (2.2309713820193684) | > loss_kl: 2.008439302444458 (1.9084464083326624) | > loss_feat: 3.6569998264312744 (4.154927659542002) | > loss_mel: 17.147478103637695 (16.659342806389997) | > loss_duration: 1.5623093843460083 (1.5120567565268659) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.059627532958984 (26.465745236011262) | > grad_norm_1: tensor(6355.3208, device='cuda:0') (tensor(7857.5049, device='cuda:0')) | > current_lr_0: 0.00017233375990103223 | > current_lr_1: 0.00017233375990103223 | > step_time: 0.4507 (0.4350694646226599) | > loader_time: 0.0088 (0.007438654595233025) --> EVAL PERFORMANCE | > avg_loader_time: 0.008714014375713506 (+7.108231665382478e-05) | > avg_step_time: 0.44831045916382695 (-0.0004965788881543531) | > avg_loss_disc: 2.5514813544045034 (+0.03412318565476147) | > avg_loss_disc_real_0: 0.09839403464026014 (+0.00175551458878416) | > avg_loss_disc_real_1: 0.22750454637366282 (+0.007065767763366176) | > avg_loss_disc_real_2: 0.22747743339605733 (-0.002720560616170875) | > avg_loss_disc_real_3: 0.22312866801946935 (-0.008324461084016632) | > avg_loss_disc_real_4: 0.2404218311880676 (-0.0015892986680420274) | > avg_loss_disc_real_5: 0.23895673495782932 (-0.005738036943153613) | > avg_loss_0: 2.5514813544045034 (+0.03412318565476147) | > avg_grad_norm_0: tensor(301.4357, device='cuda:0') (+tensor(15.0077, device='cuda:0')) | > avg_loss_gen: 2.204647636749376 (-0.019865435613712012) | > avg_loss_kl: 1.9629596857957436 (-0.020108538614192595) | > avg_loss_feat: 4.0665960983491285 (+0.1102202072949483) | > avg_loss_mel: 16.662529354364107 (-0.04160529123225132) | > avg_loss_duration: 1.5350141239837862 (-0.003094009950127763) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.43174695297026 (+0.02554689326756332) | > avg_grad_norm_1: tensor(7848.3599, device='cuda:0') (+tensor(60.2607, device='cuda:0'))  > EPOCH: 1192/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:29:27)   --> TIME: 2025-02-25 20:29:40 -- STEP: 25/72 -- GLOBAL_STEP: 1085850 | > loss_disc: 2.633732318878174 (2.5976464462280275) | > loss_disc_real_0: 0.14682751893997192 (0.13053344577550888) | > loss_disc_real_1: 0.19112233817577362 (0.22425664514303206) | > loss_disc_real_2: 0.24769368767738342 (0.21916983962059022) | > loss_disc_real_3: 0.21947602927684784 (0.2276133704185486) | > loss_disc_real_4: 0.23728995025157928 (0.22929735898971557) | > loss_disc_real_5: 0.2425883412361145 (0.24361378729343414) | > loss_0: 2.633732318878174 (2.5976464462280275) | > grad_norm_0: tensor(409.2753, device='cuda:0') (tensor(306.8782, device='cuda:0')) | > loss_gen: 2.098414421081543 (2.2085193300247186) | > loss_kl: 1.9764275550842285 (1.9258484506607056) | > loss_feat: 4.013185501098633 (3.9549650287628175) | > loss_mel: 16.13798713684082 (16.619814643859865) | > loss_duration: 1.4962232112884521 (1.4917984676361085) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.72223663330078 (26.20094596862793) | > grad_norm_1: tensor(5096.3408, device='cuda:0') (tensor(6925.0884, device='cuda:0')) | > current_lr_0: 0.0001723122181810446 | > current_lr_1: 0.0001723122181810446 | > step_time: 0.4408 (0.4329476642608643) | > loader_time: 0.0074 (0.006775341033935547) --> EVAL PERFORMANCE | > avg_loader_time: 0.008645880390220965 (-6.813398549254147e-05) | > avg_step_time: 0.451771531306522 (+0.003461072142695032) | > avg_loss_disc: 2.567641322041901 (+0.0161599676373978) | > avg_loss_disc_real_0: 0.12153628793820527 (+0.02314225329794513) | > avg_loss_disc_real_1: 0.21968153717232422 (-0.007823009201338604) | > avg_loss_disc_real_2: 0.22560969204969808 (-0.001867741346359253) | > avg_loss_disc_real_3: 0.23237885219950072 (+0.009250184180031373) | > avg_loss_disc_real_4: 0.23276045708589152 (-0.007661374102176072) | > avg_loss_disc_real_5: 0.24575345994720996 (+0.006796724989380637) | > avg_loss_0: 2.567641322041901 (+0.0161599676373978) | > avg_grad_norm_0: tensor(305.9419, device='cuda:0') (+tensor(4.5062, device='cuda:0')) | > avg_loss_gen: 2.2107680767354823 (+0.006120439986106341) | > avg_loss_kl: 2.0061509223051472 (+0.0431912365094036) | > avg_loss_feat: 3.9542000797432912 (-0.11239601860583726) | > avg_loss_mel: 16.540858913475358 (-0.12167044088874945) | > avg_loss_duration: 1.5362812331025029 (+0.0012671091187166361) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.248259235435807 (-0.18348771753445448) | > avg_grad_norm_1: tensor(7135.8008, device='cuda:0') (tensor(-712.5591, device='cuda:0'))  > EPOCH: 1193/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:30:04)   --> TIME: 2025-02-25 20:30:07 -- STEP: 3/72 -- GLOBAL_STEP: 1085900 | > loss_disc: 2.656242847442627 (2.7457881768544516) | > loss_disc_real_0: 0.1514204442501068 (0.28022412459055585) | > loss_disc_real_1: 0.21066217124462128 (0.24255276223023733) | > loss_disc_real_2: 0.23218907415866852 (0.25942019124825794) | > loss_disc_real_3: 0.29689937829971313 (0.30594176054000854) | > loss_disc_real_4: 0.2918511927127838 (0.29348958532015484) | > loss_disc_real_5: 0.2681647539138794 (0.26745355129241943) | > loss_0: 2.656242847442627 (2.7457881768544516) | > grad_norm_0: tensor(345.9080, device='cuda:0') (tensor(559.3707, device='cuda:0')) | > loss_gen: 2.2926089763641357 (2.47222638130188) | > loss_kl: 2.2695183753967285 (2.1851232051849365) | > loss_feat: 4.846452236175537 (4.249475876490275) | > loss_mel: 18.348251342773438 (18.075501759847004) | > loss_duration: 1.561937689781189 (1.5271740754445393) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.318767547607422 (28.509501775105793) | > grad_norm_1: tensor(1836.3030, device='cuda:0') (tensor(7032.1611, device='cuda:0')) | > current_lr_0: 0.00017229067915377195 | > current_lr_1: 0.00017229067915377195 | > step_time: 0.4251 (0.42921797434488934) | > loader_time: 0.007 (0.006204922993977864)  --> TIME: 2025-02-25 20:30:30 -- STEP: 53/72 -- GLOBAL_STEP: 1085950 | > loss_disc: 2.4827351570129395 (2.6336035593500693) | > loss_disc_real_0: 0.13292405009269714 (0.13392700142455552) | > loss_disc_real_1: 0.2771657407283783 (0.24294745134857465) | > loss_disc_real_2: 0.235895574092865 (0.23468252369817696) | > loss_disc_real_3: 0.19851745665073395 (0.2349418542295132) | > loss_disc_real_4: 0.23202160000801086 (0.2485640324511618) | > loss_disc_real_5: 0.22794502973556519 (0.23918933098046286) | > loss_0: 2.4827351570129395 (2.6336035593500693) | > grad_norm_0: tensor(37.4457, device='cuda:0') (tensor(272.1304, device='cuda:0')) | > loss_gen: 2.0763566493988037 (2.185266177609283) | > loss_kl: 2.146824836730957 (2.1503773050488175) | > loss_feat: 3.889068365097046 (3.9406096980256837) | > loss_mel: 16.838571548461914 (17.254921139411202) | > loss_duration: 1.5867633819580078 (1.5415253256851773) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.537586212158203 (27.07269970875866) | > grad_norm_1: tensor(13920.9297, device='cuda:0') (tensor(5768.8433, device='cuda:0')) | > current_lr_0: 0.00017229067915377195 | > current_lr_1: 0.00017229067915377195 | > step_time: 0.4524 (0.4424321516504828) | > loader_time: 0.009 (0.007861231857875609) --> EVAL PERFORMANCE | > avg_loader_time: 0.008701002094107617 (+5.512170388665208e-05) | > avg_step_time: 0.45375651037189324 (+0.001984979065371262) | > avg_loss_disc: 2.6191845578207107 (+0.05154323577880948) | > avg_loss_disc_real_0: 0.13592472105798584 (+0.014388433119780566) | > avg_loss_disc_real_1: 0.23279816982611803 (+0.013116632653793814) | > avg_loss_disc_real_2: 0.23256499918413834 (+0.006955307134440258) | > avg_loss_disc_real_3: 0.23295471764786144 (+0.0005758654483607184) | > avg_loss_disc_real_4: 0.2460955328924555 (+0.013335075806563984) | > avg_loss_disc_real_5: 0.2396336814887087 (-0.006119778458501257) | > avg_loss_0: 2.6191845578207107 (+0.05154323577880948) | > avg_grad_norm_0: tensor(274.1484, device='cuda:0') (tensor(-31.7935, device='cuda:0')) | > avg_loss_gen: 2.165331775033979 (-0.04543630170150337) | > avg_loss_kl: 2.1459080081590463 (+0.139757085853899) | > avg_loss_feat: 3.84724626742618 (-0.10695381231711121) | > avg_loss_mel: 17.084143195353757 (+0.5432842818783996) | > avg_loss_duration: 1.55391075409634 (+0.017629520993837078) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.796540031970387 (+0.5482807965345806) | > avg_grad_norm_1: tensor(6912.4927, device='cuda:0') (tensor(-223.3081, device='cuda:0'))  > EPOCH: 1194/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:30:41)   --> TIME: 2025-02-25 20:30:56 -- STEP: 31/72 -- GLOBAL_STEP: 1086000 | > loss_disc: 2.6668033599853516 (2.6216243543932514) | > loss_disc_real_0: 0.2363596260547638 (0.1411197591693171) | > loss_disc_real_1: 0.21926343441009521 (0.22929840174413496) | > loss_disc_real_2: 0.24185185134410858 (0.2407745574751208) | > loss_disc_real_3: 0.25485411286354065 (0.23559172499564388) | > loss_disc_real_4: 0.2577064037322998 (0.2430234407224963) | > loss_disc_real_5: 0.24902547895908356 (0.2452381440708714) | > loss_0: 2.6668033599853516 (2.6216243543932514) | > grad_norm_0: tensor(540.1012, device='cuda:0') (tensor(309.2679, device='cuda:0')) | > loss_gen: 2.1442110538482666 (2.0983613306476228) | > loss_kl: 2.1337192058563232 (1.9237954424273582) | > loss_feat: 3.157952070236206 (3.6155846734200754) | > loss_mel: 15.487728118896484 (16.557810321930923) | > loss_duration: 1.5283446311950684 (1.4959187430720176) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.451955795288086 (25.691470607634514) | > grad_norm_1: tensor(9988.7998, device='cuda:0') (tensor(11404.3135, device='cuda:0')) | > current_lr_0: 0.00017226914281887772 | > current_lr_1: 0.00017226914281887772 | > step_time: 0.442 (0.43457108159219066) | > loader_time: 0.0085 (0.0070232191393452305) --> EVAL PERFORMANCE | > avg_loader_time: 0.008806134613466934 (+0.0001051325193593168) | > avg_step_time: 0.4539225907392905 (+0.00016608036739723664) | > avg_loss_disc: 2.599968685230737 (-0.019215872589973593) | > avg_loss_disc_real_0: 0.138700895903396 (+0.002776174845410151) | > avg_loss_disc_real_1: 0.22110571953612312 (-0.01169245028999491) | > avg_loss_disc_real_2: 0.2356898778341186 (+0.0031248786499802617) | > avg_loss_disc_real_3: 0.23278925326508534 (-0.00016546438277609266) | > avg_loss_disc_real_4: 0.24467033672500665 (-0.00142519616744885) | > avg_loss_disc_real_5: 0.2467814946258572 (+0.00714781313714849) | > avg_loss_0: 2.599968685230737 (-0.019215872589973593) | > avg_grad_norm_0: tensor(326.0776, device='cuda:0') (+tensor(51.9292, device='cuda:0')) | > avg_loss_gen: 2.105753394919382 (-0.05957838011459682) | > avg_loss_kl: 1.998706365974856 (-0.14720164218419018) | > avg_loss_feat: 3.6480814571111972 (-0.1991648103149828) | > avg_loss_mel: 16.50417654279252 (-0.5799666525612359) | > avg_loss_duration: 1.535102221327768 (-0.018808532768571862) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.79182012316207 (-1.004719908808319) | > avg_grad_norm_1: tensor(10962.2939, device='cuda:0') (+tensor(4049.8013, device='cuda:0'))  > EPOCH: 1195/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:31:18)   --> TIME: 2025-02-25 20:31:23 -- STEP: 9/72 -- GLOBAL_STEP: 1086050 | > loss_disc: 2.672102928161621 (2.5946575270758734) | > loss_disc_real_0: 0.10045938193798065 (0.15116743495066962) | > loss_disc_real_1: 0.24078966677188873 (0.21836051179303062) | > loss_disc_real_2: 0.21408329904079437 (0.22694753772682613) | > loss_disc_real_3: 0.25237902998924255 (0.23039831221103668) | > loss_disc_real_4: 0.23767420649528503 (0.23827503787146675) | > loss_disc_real_5: 0.2376294881105423 (0.2372172623872757) | > loss_0: 2.672102928161621 (2.5946575270758734) | > grad_norm_0: tensor(212.2245, device='cuda:0') (tensor(313.4899, device='cuda:0')) | > loss_gen: 1.928673267364502 (2.123958362473382) | > loss_kl: 2.0539326667785645 (1.8567384348975287) | > loss_feat: 4.090239524841309 (3.792941623263889) | > loss_mel: 17.049686431884766 (16.693420939975315) | > loss_duration: 1.5131855010986328 (1.5013868411382039) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.635717391967773 (25.96844736735026) | > grad_norm_1: tensor(17744.4824, device='cuda:0') (tensor(10731.4307, device='cuda:0')) | > current_lr_0: 0.00017224760917602534 | > current_lr_1: 0.00017224760917602534 | > step_time: 0.422 (0.4277665615081787) | > loader_time: 0.0055 (0.006809155146280925)  --> TIME: 2025-02-25 20:31:46 -- STEP: 59/72 -- GLOBAL_STEP: 1086100 | > loss_disc: 2.5025362968444824 (2.5945395251451924) | > loss_disc_real_0: 0.11664008349180222 (0.13469750004804734) | > loss_disc_real_1: 0.19918739795684814 (0.22206209448434538) | > loss_disc_real_2: 0.2298138439655304 (0.2329436373912682) | > loss_disc_real_3: 0.23453210294246674 (0.23157901531558925) | > loss_disc_real_4: 0.24317796528339386 (0.24157338602057957) | > loss_disc_real_5: 0.25190120935440063 (0.2437085299673727) | > loss_0: 2.5025362968444824 (2.5945395251451924) | > grad_norm_0: tensor(187.6977, device='cuda:0') (tensor(357.3925, device='cuda:0')) | > loss_gen: 2.174072504043579 (2.1170216616937676) | > loss_kl: 1.9529753923416138 (1.9637625439692352) | > loss_feat: 3.8779213428497314 (3.778179556636487) | > loss_mel: 14.499065399169922 (16.47639382895776) | > loss_duration: 1.5513970851898193 (1.5212756152880393) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.055431365966797 (25.856633428799903) | > grad_norm_1: tensor(11036.1055, device='cuda:0') (tensor(11045.7373, device='cuda:0')) | > current_lr_0: 0.00017224760917602534 | > current_lr_1: 0.00017224760917602534 | > step_time: 0.4778 (0.4388881941973153) | > loader_time: 0.0098 (0.008156251099150056) --> EVAL PERFORMANCE | > avg_loader_time: 0.0088570688811826 (+5.093426771566635e-05) | > avg_step_time: 0.4481008892327967 (-0.005821701506493804) | > avg_loss_disc: 2.5944385797205114 (-0.005530105510225702) | > avg_loss_disc_real_0: 0.13276922325013385 (-0.005931672653262138) | > avg_loss_disc_real_1: 0.22175544416400747 (+0.000649724627884346) | > avg_loss_disc_real_2: 0.2326383160453447 (-0.0030515617887738877) | > avg_loss_disc_real_3: 0.23025932593244902 (-0.0025299273326363225) | > avg_loss_disc_real_4: 0.24133612320456707 (-0.003334213520439583) | > avg_loss_disc_real_5: 0.2434115684787992 (-0.003369926147058) | > avg_loss_0: 2.5944385797205114 (-0.005530105510225702) | > avg_grad_norm_0: tensor(361.1114, device='cuda:0') (+tensor(35.0337, device='cuda:0')) | > avg_loss_gen: 2.1065769716047913 (+0.0008235766854092219) | > avg_loss_kl: 1.9883818374553197 (-0.010324528519536358) | > avg_loss_feat: 3.772216988281465 (+0.12413553117026765) | > avg_loss_mel: 16.532007042790795 (+0.027830499998273694) | > avg_loss_duration: 1.5329887917343998 (-0.0021134295933682434) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.932171781298145 (+0.1403516581360762) | > avg_grad_norm_1: tensor(11357.3789, device='cuda:0') (+tensor(395.0850, device='cuda:0'))  > EPOCH: 1196/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:31:54)   --> TIME: 2025-02-25 20:32:13 -- STEP: 37/72 -- GLOBAL_STEP: 1086150 | > loss_disc: 2.642061710357666 (2.584218379613515) | > loss_disc_real_0: 0.1278252899646759 (0.11250372432373665) | > loss_disc_real_1: 0.19571031630039215 (0.21773898722352208) | > loss_disc_real_2: 0.22383810579776764 (0.2339784921826543) | > loss_disc_real_3: 0.2240692377090454 (0.23380870392193664) | > loss_disc_real_4: 0.22961266338825226 (0.2453497817387452) | > loss_disc_real_5: 0.2304624319076538 (0.2439691984170192) | > loss_0: 2.642061710357666 (2.584218379613515) | > grad_norm_0: tensor(334.0299, device='cuda:0') (tensor(387.4515, device='cuda:0')) | > loss_gen: 1.94514799118042 (2.14066962938051) | > loss_kl: 1.9338412284851074 (1.9332828102885067) | > loss_feat: 3.5367958545684814 (3.9601272956745044) | > loss_mel: 15.609285354614258 (16.51672260181324) | > loss_duration: 1.5316965579986572 (1.493720447694933) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.556766510009766 (26.04452277518607) | > grad_norm_1: tensor(17043.3906, device='cuda:0') (tensor(12618.1221, device='cuda:0')) | > current_lr_0: 0.00017222607822487833 | > current_lr_1: 0.00017222607822487833 | > step_time: 0.4403 (0.4397966990599761) | > loader_time: 0.0078 (0.007075786590576172) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617767145935915 (-0.0002393017352466855) | > avg_step_time: 0.4553109659275538 (+0.00721007669475715) | > avg_loss_disc: 2.584070205688477 (-0.010368374032034389) | > avg_loss_disc_real_0: 0.11492645813965462 (-0.017842765110479233) | > avg_loss_disc_real_1: 0.22036445980340663 (-0.0013909843606008387) | > avg_loss_disc_real_2: 0.23481926447908644 (+0.002180948433741725) | > avg_loss_disc_real_3: 0.23206046349565748 (+0.0018011375632084603) | > avg_loss_disc_real_4: 0.2446482003154889 (+0.003312077110921824) | > avg_loss_disc_real_5: 0.24414553881531031 (+0.000733970336511125) | > avg_loss_0: 2.584070205688477 (-0.010368374032034389) | > avg_grad_norm_0: tensor(405.3145, device='cuda:0') (+tensor(44.2032, device='cuda:0')) | > avg_loss_gen: 2.1447924660964754 (+0.038215494491684154) | > avg_loss_kl: 1.998226867595189 (+0.00984503013986937) | > avg_loss_feat: 3.919878791755354 (+0.14766180347388902) | > avg_loss_mel: 16.509959543255018 (-0.022047499535776893) | > avg_loss_duration: 1.5310688505710013 (-0.001919941163398553) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.103926564606144 (+0.17175478330799976) | > avg_grad_norm_1: tensor(12688.9912, device='cuda:0') (+tensor(1331.6123, device='cuda:0'))  > EPOCH: 1197/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:32:31)   --> TIME: 2025-02-25 20:32:40 -- STEP: 15/72 -- GLOBAL_STEP: 1086200 | > loss_disc: 2.4857678413391113 (2.570614544550578) | > loss_disc_real_0: 0.12840774655342102 (0.11952017098665238) | > loss_disc_real_1: 0.20058806240558624 (0.21658675074577333) | > loss_disc_real_2: 0.2227405160665512 (0.2277586410442988) | > loss_disc_real_3: 0.23233921825885773 (0.2321962575117747) | > loss_disc_real_4: 0.24076424539089203 (0.24167900880177814) | > loss_disc_real_5: 0.24290831387043 (0.24289736449718474) | > loss_0: 2.4857678413391113 (2.570614544550578) | > grad_norm_0: tensor(73.2116, device='cuda:0') (tensor(336.5519, device='cuda:0')) | > loss_gen: 2.2723164558410645 (2.164558418591817) | > loss_kl: 2.008831262588501 (1.9167134523391725) | > loss_feat: 4.137034893035889 (3.9719910621643066) | > loss_mel: 17.12297248840332 (16.648559888203938) | > loss_duration: 1.4799221754074097 (1.480360213915507) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.021076202392578 (26.182182947794598) | > grad_norm_1: tensor(9997.6309, device='cuda:0') (tensor(11098.6445, device='cuda:0')) | > current_lr_0: 0.0001722045499651002 | > current_lr_1: 0.0001722045499651002 | > step_time: 0.4298 (0.43207085927327477) | > loader_time: 0.0058 (0.006826591491699219)  --> TIME: 2025-02-25 20:33:03 -- STEP: 65/72 -- GLOBAL_STEP: 1086250 | > loss_disc: 2.7633395195007324 (2.6718341240516077) | > loss_disc_real_0: 0.09293825924396515 (0.13481763830551735) | > loss_disc_real_1: 0.21487554907798767 (0.24410696350611163) | > loss_disc_real_2: 0.1872006505727768 (0.23643942268995138) | > loss_disc_real_3: 0.21456380188465118 (0.23500123642958126) | > loss_disc_real_4: 0.21218253672122955 (0.24610935541299672) | > loss_disc_real_5: 0.23689518868923187 (0.24430408546557794) | > loss_0: 2.7633395195007324 (2.6718341240516077) | > grad_norm_0: tensor(438.3017, device='cuda:0') (tensor(340.8545, device='cuda:0')) | > loss_gen: 1.727303147315979 (2.140806691463178) | > loss_kl: 2.471919298171997 (2.3320076722365157) | > loss_feat: 3.6763505935668945 (3.8942364362569957) | > loss_mel: 17.69696617126465 (17.768982227032005) | > loss_duration: 1.609650731086731 (1.5571603298187255) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.18218994140625 (27.69319328894982) | > grad_norm_1: tensor(13454.7256, device='cuda:0') (tensor(8878.6797, device='cuda:0')) | > current_lr_0: 0.0001722045499651002 | > current_lr_1: 0.0001722045499651002 | > step_time: 0.4969 (0.44460678834181566) | > loader_time: 0.0115 (0.00833580310528095) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861897267086405 (+1.2055249281363312e-06) | > avg_step_time: 0.4497560380210339 (-0.005554927906519902) | > avg_loss_disc: 2.6713779610647284 (+0.08730775537625135) | > avg_loss_disc_real_0: 0.1372939320937009 (+0.02236747395404627) | > avg_loss_disc_real_1: 0.2410717459631637 (+0.02070728615975706) | > avg_loss_disc_real_2: 0.23528612814318967 (+0.00046686366410322866) | > avg_loss_disc_real_3: 0.2351065647434181 (+0.003046101247760613) | > avg_loss_disc_real_4: 0.243702855328439 (-0.0009453449870499009) | > avg_loss_disc_real_5: 0.24377135302818997 (-0.00037418578712034023) | > avg_loss_0: 2.6713779610647284 (+0.08730775537625135) | > avg_grad_norm_0: tensor(332.6703, device='cuda:0') (tensor(-72.6442, device='cuda:0')) | > avg_loss_gen: 2.1413115380515526 (-0.0034809280449228375) | > avg_loss_kl: 2.3328555839162477 (+0.3346287163210586) | > avg_loss_feat: 3.8791907908211294 (-0.04068800093422453) | > avg_loss_mel: 17.74866963776065 (+1.238710094505631) | > avg_loss_duration: 1.5619706587052682 (+0.030901808134266906) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.663998160563725 (+1.5600715959575808) | > avg_grad_norm_1: tensor(8874.6523, device='cuda:0') (tensor(-3814.3389, device='cuda:0'))  > EPOCH: 1198/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:33:08)   --> TIME: 2025-02-25 20:33:29 -- STEP: 43/72 -- GLOBAL_STEP: 1086300 | > loss_disc: 2.9618515968322754 (2.6856965353322586) | > loss_disc_real_0: 0.451951265335083 (0.14659576945353386) | > loss_disc_real_1: 0.11598934978246689 (0.21925097357395085) | > loss_disc_real_2: 0.2890644073486328 (0.23551718786705372) | > loss_disc_real_3: 0.19664037227630615 (0.23199774498163267) | > loss_disc_real_4: 0.17954428493976593 (0.24359413115091103) | > loss_disc_real_5: 0.26113179326057434 (0.24383637031843497) | > loss_0: 2.9618515968322754 (2.6856965353322586) | > grad_norm_0: tensor(281.1332, device='cuda:0') (tensor(302.2089, device='cuda:0')) | > loss_gen: 2.326846122741699 (2.11865057778913) | > loss_kl: 2.126830577850342 (2.082149294919746) | > loss_feat: 3.7176764011383057 (3.862195902092512) | > loss_mel: 16.249080657958984 (17.046391797620192) | > loss_duration: 1.5422430038452148 (1.5168307298837707) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.962677001953125 (26.626218263493026) | > grad_norm_1: tensor(2947.9114, device='cuda:0') (tensor(7510.4629, device='cuda:0')) | > current_lr_0: 0.00017218302439635455 | > current_lr_1: 0.00017218302439635455 | > step_time: 0.472 (0.4355504180109778) | > loader_time: 0.0098 (0.007343142531638922) --> EVAL PERFORMANCE | > avg_loader_time: 0.008967335795012998 (+0.00034836312414894717) | > avg_step_time: 0.4502123308853364 (+0.0004562928643024877) | > avg_loss_disc: 2.6856763597945092 (+0.014298398729780892) | > avg_loss_disc_real_0: 0.15206167765591347 (+0.01476774556221258) | > avg_loss_disc_real_1: 0.22149624979831803 (-0.019575496164845663) | > avg_loss_disc_real_2: 0.24575300367785172 (+0.010466875534662051) | > avg_loss_disc_real_3: 0.23463828693813002 (-0.0004682778052880754) | > avg_loss_disc_real_4: 0.23743529260998042 (-0.006267562718458575) | > avg_loss_disc_real_5: 0.2469849059699287 (+0.003213552941738723) | > avg_loss_0: 2.6856763597945092 (+0.014298398729780892) | > avg_grad_norm_0: tensor(296.9682, device='cuda:0') (tensor(-35.7021, device='cuda:0')) | > avg_loss_gen: 2.0924890679372874 (-0.048822470114265215) | > avg_loss_kl: 2.127774263771487 (-0.20508132014476077) | > avg_loss_feat: 3.719100663359736 (-0.16009012746139328) | > avg_loss_mel: 16.992750073822446 (-0.7559195639382033) | > avg_loss_duration: 1.5458116934333046 (-0.016158965271963588) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.477925609534896 (-1.186072551028829) | > avg_grad_norm_1: tensor(7111.5547, device='cuda:0') (tensor(-1763.0977, device='cuda:0'))  > EPOCH: 1199/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:33:44)   --> TIME: 2025-02-25 20:33:56 -- STEP: 21/72 -- GLOBAL_STEP: 1086350 | > loss_disc: 2.8625006675720215 (2.6075773806799027) | > loss_disc_real_0: 0.08028411865234375 (0.13902573979326657) | > loss_disc_real_1: 0.222188338637352 (0.21192668946016402) | > loss_disc_real_2: 0.22437119483947754 (0.21983655506656283) | > loss_disc_real_3: 0.26575157046318054 (0.23096501259576707) | > loss_disc_real_4: 0.23497973382472992 (0.2315123663062141) | > loss_disc_real_5: 0.24496185779571533 (0.2357909537497021) | > loss_0: 2.8625006675720215 (2.6075773806799027) | > grad_norm_0: tensor(305.9746, device='cuda:0') (tensor(326.3845, device='cuda:0')) | > loss_gen: 1.6650669574737549 (2.0766480763753252) | > loss_kl: 1.921718716621399 (2.0208075670968917) | > loss_feat: 3.3937761783599854 (3.7805708930605935) | > loss_mel: 17.79380989074707 (17.127145222255162) | > loss_duration: 1.5151938199996948 (1.4926473526727586) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.289566040039062 (26.497819264729817) | > grad_norm_1: tensor(2996.2725, device='cuda:0') (tensor(8330.7773, device='cuda:0')) | > current_lr_0: 0.00017216150151830501 | > current_lr_1: 0.00017216150151830501 | > step_time: 0.437 (0.43125985917590914) | > loader_time: 0.0075 (0.0069026947021484375)  --> TIME: 2025-02-25 20:34:19 -- STEP: 71/72 -- GLOBAL_STEP: 1086400 | > loss_disc: 2.79021954536438 (2.6508252184155965) | > loss_disc_real_0: 0.06959940493106842 (0.12988281365431534) | > loss_disc_real_1: 0.20834200084209442 (0.22244041906276218) | > loss_disc_real_2: 0.1963554471731186 (0.22362285957370007) | > loss_disc_real_3: 0.2903771698474884 (0.22959381971560733) | > loss_disc_real_4: 0.23840485513210297 (0.2368008618203687) | > loss_disc_real_5: 0.24397331476211548 (0.24141054673933646) | > loss_0: 2.79021954536438 (2.6508252184155965) | > grad_norm_0: tensor(153.9692, device='cuda:0') (tensor(220.2067, device='cuda:0')) | > loss_gen: 1.652754545211792 (2.075416413831039) | > loss_kl: 2.0399067401885986 (2.0781335830688477) | > loss_feat: 3.0973997116088867 (3.7760505541949203) | > loss_mel: 17.785043716430664 (17.014261433776) | > loss_duration: 1.5571773052215576 (1.541637042878379) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.132280349731445 (26.48549909994636) | > grad_norm_1: tensor(5118.0020, device='cuda:0') (tensor(6694.9956, device='cuda:0')) | > current_lr_0: 0.00017216150151830501 | > current_lr_1: 0.00017216150151830501 | > step_time: 0.4615 (0.45055480070517095) | > loader_time: 0.008 (0.008628593364232027) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628593364232027 (-0.0003387424307809712) | > avg_step_time: 0.45055480070517095 (+0.0003424698198345455) | > avg_loss_disc: 2.6508252184155965 (-0.034851141378912764) | > avg_loss_disc_real_0: 0.12988281365431534 (-0.02217886400159813) | > avg_loss_disc_real_1: 0.22244041906276218 (+0.0009441692644441557) | > avg_loss_disc_real_2: 0.22362285957370007 (-0.02213014410415165) | > avg_loss_disc_real_3: 0.22959381971560733 (-0.005044467222522692) | > avg_loss_disc_real_4: 0.2368008618203687 (-0.0006344307896117285) | > avg_loss_disc_real_5: 0.24141054673933646 (-0.005574359230592241) | > avg_loss_0: 2.6508252184155965 (-0.034851141378912764) | > avg_grad_norm_0: tensor(220.2067, device='cuda:0') (tensor(-76.7615, device='cuda:0')) | > avg_loss_gen: 2.075416413831039 (-0.017072654106248475) | > avg_loss_kl: 2.0781335830688477 (-0.04964068070263927) | > avg_loss_feat: 3.7760505541949203 (+0.0569498908351842) | > avg_loss_mel: 17.014261433776 (+0.021511359953553466) | > avg_loss_duration: 1.541637042878379 (-0.0041746505549256074) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.48549909994636 (+0.007573490411463268) | > avg_grad_norm_1: tensor(6694.9956, device='cuda:0') (tensor(-416.5591, device='cuda:0'))  > EPOCH: 1200/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:34:21)   --> TIME: 2025-02-25 20:34:45 -- STEP: 49/72 -- GLOBAL_STEP: 1086450 | > loss_disc: 2.8024489879608154 (2.7198977081143125) | > loss_disc_real_0: 0.12901942431926727 (0.11924172652771278) | > loss_disc_real_1: 0.15987975895404816 (0.297643599917694) | > loss_disc_real_2: 0.3046880066394806 (0.2887090839901748) | > loss_disc_real_3: 0.2297496348619461 (0.25278459458935015) | > loss_disc_real_4: 0.23459255695343018 (0.25847042762503336) | > loss_disc_real_5: 0.2658023536205292 (0.24586562295349276) | > loss_0: 2.8024489879608154 (2.7198977081143125) | > grad_norm_0: tensor(70.9769, device='cuda:0') (tensor(222.1768, device='cuda:0')) | > loss_gen: 1.7154662609100342 (2.2548168484045537) | > loss_kl: 2.246161699295044 (2.078996376115448) | > loss_feat: 3.2326579093933105 (3.7855413349307314) | > loss_mel: 17.477176666259766 (17.165390987785496) | > loss_duration: 1.6224547624588013 (1.5267815103336257) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.29391860961914 (26.811526901867925) | > grad_norm_1: tensor(3603.4033, device='cuda:0') (tensor(5683.2275, device='cuda:0')) | > current_lr_0: 0.0001721399813306152 | > current_lr_1: 0.0001721399813306152 | > step_time: 0.4352 (0.4400658461512352) | > loader_time: 0.0095 (0.007631574358258929) --> EVAL PERFORMANCE | > avg_loader_time: 0.008739246449000397 (+0.0001106530847683699) | > avg_step_time: 0.4529887689671046 (+0.00243396826193365) | > avg_loss_disc: 2.7098175035396093 (+0.05899228512401278) | > avg_loss_disc_real_0: 0.13107642534018404 (+0.0011936116858687063) | > avg_loss_disc_real_1: 0.27952013290683997 (+0.057079713844077784) | > avg_loss_disc_real_2: 0.27359689688178845 (+0.04997403730808839) | > avg_loss_disc_real_3: 0.24754097365157704 (+0.017947153935969712) | > avg_loss_disc_real_4: 0.25450393410635663 (+0.017703072285987942) | > avg_loss_disc_real_5: 0.2475567831539772 (+0.006146236414640738) | > avg_loss_0: 2.7098175035396093 (+0.05899228512401278) | > avg_grad_norm_0: tensor(224.5553, device='cuda:0') (+tensor(4.3486, device='cuda:0')) | > avg_loss_gen: 2.1975516651717717 (+0.12213525134073278) | > avg_loss_kl: 2.1149069423406894 (+0.03677335927184178) | > avg_loss_feat: 3.660358647225608 (-0.11569190696931209) | > avg_loss_mel: 17.066254803832145 (+0.05199337005614524) | > avg_loss_duration: 1.5457106140297903 (+0.0040735711514112705) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.584782667562994 (+0.09928356761663437) | > avg_grad_norm_1: tensor(6109.6494, device='cuda:0') (tensor(-585.3462, device='cuda:0'))  > EPOCH: 1201/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:34:58)   --> TIME: 2025-02-25 20:35:12 -- STEP: 27/72 -- GLOBAL_STEP: 1086500 | > loss_disc: 2.563265562057495 (2.60115032725864) | > loss_disc_real_0: 0.14402775466442108 (0.10121325355161119) | > loss_disc_real_1: 0.18526245653629303 (0.22423238334832368) | > loss_disc_real_2: 0.19724585115909576 (0.2342059005189825) | > loss_disc_real_3: 0.23867595195770264 (0.23299390629485803) | > loss_disc_real_4: 0.21255774796009064 (0.24378798846845273) | > loss_disc_real_5: 0.21407850086688995 (0.24638997073526733) | > loss_0: 2.563265562057495 (2.60115032725864) | > grad_norm_0: tensor(59.8431, device='cuda:0') (tensor(223.7918, device='cuda:0')) | > loss_gen: 1.8748230934143066 (2.0991663712042343) | > loss_kl: 1.967307448387146 (2.0750886290161694) | > loss_feat: 3.673758029937744 (3.770858746987802) | > loss_mel: 16.902610778808594 (16.969691841690626) | > loss_duration: 1.508062720298767 (1.4998813072840373) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.92656135559082 (26.414687191998517) | > grad_norm_1: tensor(1426.2894, device='cuda:0') (tensor(5952.6528, device='cuda:0')) | > current_lr_0: 0.00017211846383294888 | > current_lr_1: 0.00017211846383294888 | > step_time: 0.4487 (0.4329949661537453) | > loader_time: 0.0087 (0.006775273217095269) --> EVAL PERFORMANCE | > avg_loader_time: 0.008656061870951047 (-8.318457804934974e-05) | > avg_step_time: 0.45057898843792116 (-0.0024097805291834473) | > avg_loss_disc: 2.730477890498201 (+0.020660386958591737) | > avg_loss_disc_real_0: 0.1691519881690472 (+0.03807556282886315) | > avg_loss_disc_real_1: 0.22360049943688892 (-0.05591963346995105) | > avg_loss_disc_real_2: 0.23721061290149958 (-0.03638628398028887) | > avg_loss_disc_real_3: 0.23291572991391302 (-0.014625243737664023) | > avg_loss_disc_real_4: 0.24236001389127382 (-0.012143920215082815) | > avg_loss_disc_real_5: 0.24349247464831447 (-0.004064308505662723) | > avg_loss_0: 2.730477890498201 (+0.020660386958591737) | > avg_grad_norm_0: tensor(173.6031, device='cuda:0') (tensor(-50.9522, device='cuda:0')) | > avg_loss_gen: 2.0373072086925244 (-0.16024445647924734) | > avg_loss_kl: 2.258032785335057 (+0.1431258429943676) | > avg_loss_feat: 3.5870756464944757 (-0.07328300073113247) | > avg_loss_mel: 17.622027692660485 (+0.55577288882834) | > avg_loss_duration: 1.5726732704001414 (+0.026962656370351112) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.077116737903005 (+0.49233407034001075) | > avg_grad_norm_1: tensor(4516.6987, device='cuda:0') (tensor(-1592.9507, device='cuda:0'))  > EPOCH: 1202/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:35:35)   --> TIME: 2025-02-25 20:35:39 -- STEP: 5/72 -- GLOBAL_STEP: 1086550 | > loss_disc: 2.4708869457244873 (2.654992961883545) | > loss_disc_real_0: 0.051536887884140015 (0.20850469768047333) | > loss_disc_real_1: 0.284659743309021 (0.2808692991733551) | > loss_disc_real_2: 0.19428344070911407 (0.19605418145656586) | > loss_disc_real_3: 0.21235908567905426 (0.20863080024719238) | > loss_disc_real_4: 0.2612035572528839 (0.24435167014598846) | > loss_disc_real_5: 0.26581114530563354 (0.24767338037490844) | > loss_0: 2.4708869457244873 (2.654992961883545) | > grad_norm_0: tensor(46.2671, device='cuda:0') (tensor(166.0900, device='cuda:0')) | > loss_gen: 2.127078056335449 (2.1679407596588134) | > loss_kl: 2.17177677154541 (2.1808153629302978) | > loss_feat: 3.908275604248047 (3.8363035202026365) | > loss_mel: 17.35318374633789 (18.512292861938477) | > loss_duration: 1.5242648124694824 (1.544581913948059) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.084579467773438 (28.24193420410156) | > grad_norm_1: tensor(2959.5544, device='cuda:0') (tensor(4579.2017, device='cuda:0')) | > current_lr_0: 0.00017209694902496975 | > current_lr_1: 0.00017209694902496975 | > step_time: 0.4165 (0.4207607746124268) | > loader_time: 0.0061 (0.006787157058715821)  --> TIME: 2025-02-25 20:36:01 -- STEP: 55/72 -- GLOBAL_STEP: 1086600 | > loss_disc: 2.5971782207489014 (2.6477070071480484) | > loss_disc_real_0: 0.1188710629940033 (0.14288145601749416) | > loss_disc_real_1: 0.2044823318719864 (0.22381116178902713) | > loss_disc_real_2: 0.23477472364902496 (0.23745510523969476) | > loss_disc_real_3: 0.2520178556442261 (0.2338761256499724) | > loss_disc_real_4: 0.25649145245552063 (0.2458857682618228) | > loss_disc_real_5: 0.26429226994514465 (0.2511747430671346) | > loss_0: 2.5971782207489014 (2.6477070071480484) | > grad_norm_0: tensor(290.4276, device='cuda:0') (tensor(216.9775, device='cuda:0')) | > loss_gen: 2.0184266567230225 (2.0394773851741443) | > loss_kl: 2.1371097564697266 (2.0916773687709465) | > loss_feat: 3.347048282623291 (3.4868793834339487) | > loss_mel: 16.613340377807617 (17.346844118291678) | > loss_duration: 1.608097791671753 (1.5428311412984674) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.724021911621094 (26.507709121704103) | > grad_norm_1: tensor(13012.0273, device='cuda:0') (tensor(8970.5244, device='cuda:0')) | > current_lr_0: 0.00017209694902496975 | > current_lr_1: 0.00017209694902496975 | > step_time: 0.4537 (0.43694957386363636) | > loader_time: 0.0093 (0.007901191711425781) --> EVAL PERFORMANCE | > avg_loader_time: 0.008632340901334518 (-2.3720969616529072e-05) | > avg_step_time: 0.4462036717105919 (-0.00437531672732927) | > avg_loss_disc: 2.6416944987337345 (-0.08878339176446648) | > avg_loss_disc_real_0: 0.14032485839766512 (-0.028827129771382076) | > avg_loss_disc_real_1: 0.2221561225367264 (-0.001444376900162525) | > avg_loss_disc_real_2: 0.23814258483094228 (+0.0009319719294426931) | > avg_loss_disc_real_3: 0.23389475257463857 (+0.0009790226607255537) | > avg_loss_disc_real_4: 0.24573452204046115 (+0.0033745081491873274) | > avg_loss_disc_real_5: 0.250955146383232 (+0.0074626717349175475) | > avg_loss_0: 2.6416944987337345 (-0.08878339176446648) | > avg_grad_norm_0: tensor(237.5833, device='cuda:0') (+tensor(63.9802, device='cuda:0')) | > avg_loss_gen: 2.0345361786828917 (-0.0027710300096326357) | > avg_loss_kl: 2.1045490217880465 (-0.15348376354701054) | > avg_loss_feat: 3.4558570317819086 (-0.13121861471256713) | > avg_loss_mel: 17.27965995627389 (-0.34236773638659557) | > avg_loss_duration: 1.555034812067596 (-0.017638458332545293) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.429636861237004 (-0.6474798766660008) | > avg_grad_norm_1: tensor(9153.7461, device='cuda:0') (+tensor(4637.0474, device='cuda:0'))  > EPOCH: 1203/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:36:11)   --> TIME: 2025-02-25 20:36:28 -- STEP: 33/72 -- GLOBAL_STEP: 1086650 | > loss_disc: 2.5452141761779785 (2.5822783094463926) | > loss_disc_real_0: 0.09274140745401382 (0.11659286284085477) | > loss_disc_real_1: 0.20871657133102417 (0.2181118532563701) | > loss_disc_real_2: 0.21654249727725983 (0.23417434412421603) | > loss_disc_real_3: 0.20704658329486847 (0.23484669941844363) | > loss_disc_real_4: 0.20444642007350922 (0.24370357665148648) | > loss_disc_real_5: 0.2130168080329895 (0.24712260699633395) | > loss_0: 2.5452141761779785 (2.5822783094463926) | > grad_norm_0: tensor(247.1105, device='cuda:0') (tensor(294.3553, device='cuda:0')) | > loss_gen: 1.953359842300415 (2.0903573939294526) | > loss_kl: 2.125408887863159 (1.9819167418913408) | > loss_feat: 4.224799156188965 (3.714167529886419) | > loss_mel: 16.70924949645996 (16.96535165382154) | > loss_duration: 1.5032298564910889 (1.507491530794086) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.51604652404785 (26.259284568555426) | > grad_norm_1: tensor(9819.0010, device='cuda:0') (tensor(9606.9902, device='cuda:0')) | > current_lr_0: 0.0001720754369063416 | > current_lr_1: 0.0001720754369063416 | > step_time: 0.4385 (0.4332578615708785) | > loader_time: 0.0079 (0.006969256834550338) --> EVAL PERFORMANCE | > avg_loader_time: 0.008502547170074894 (-0.0001297937312596236) | > avg_step_time: 0.45164633804643656 (+0.00544266633584467) | > avg_loss_disc: 2.6694151985813193 (+0.02772069984758474) | > avg_loss_disc_real_0: 0.15585394366316394 (+0.015529085265498815) | > avg_loss_disc_real_1: 0.21969362211899018 (-0.002462500417736213) | > avg_loss_disc_real_2: 0.23332653792811112 (-0.0048160469028311614) | > avg_loss_disc_real_3: 0.23225959904596838 (-0.0016351535286701913) | > avg_loss_disc_real_4: 0.2427003954497861 (-0.0030341265906750348) | > avg_loss_disc_real_5: 0.2483142765474991 (-0.0026408698357329174) | > avg_loss_0: 2.6694151985813193 (+0.02772069984758474) | > avg_grad_norm_0: tensor(306.7926, device='cuda:0') (+tensor(69.2093, device='cuda:0')) | > avg_loss_gen: 2.042400825191551 (+0.007864646508659145) | > avg_loss_kl: 2.1371848532851314 (+0.03263583149708493) | > avg_loss_feat: 3.610626257641215 (+0.1547692258593063) | > avg_loss_mel: 17.500375371583765 (+0.2207154153098756) | > avg_loss_duration: 1.55268712782524 (-0.002347684242355985) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.843274237404408 (+0.41363737616740437) | > avg_grad_norm_1: tensor(7551.7764, device='cuda:0') (tensor(-1601.9697, device='cuda:0'))  > EPOCH: 1204/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:36:48)   --> TIME: 2025-02-25 20:36:55 -- STEP: 11/72 -- GLOBAL_STEP: 1086700 | > loss_disc: 2.376809597015381 (2.451527963985096) | > loss_disc_real_0: 0.0685010552406311 (0.12308857928622853) | > loss_disc_real_1: 0.18659484386444092 (0.20402349531650543) | > loss_disc_real_2: 0.21337319910526276 (0.21791639653119174) | > loss_disc_real_3: 0.2221072018146515 (0.21915012327107516) | > loss_disc_real_4: 0.21013325452804565 (0.22721827978437598) | > loss_disc_real_5: 0.232340469956398 (0.23335349695249039) | > loss_0: 2.376809597015381 (2.451527963985096) | > grad_norm_0: tensor(90.9991, device='cuda:0') (tensor(202.0087, device='cuda:0')) | > loss_gen: 2.3718526363372803 (2.221870931712064) | > loss_kl: 2.152580738067627 (2.2277187434109775) | > loss_feat: 4.499027729034424 (4.0802106207067315) | > loss_mel: 17.76212501525879 (18.27244654568759) | > loss_duration: 1.5016837120056152 (1.532813386483626) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.28727149963379 (28.335060293024238) | > grad_norm_1: tensor(1694.6868, device='cuda:0') (tensor(5319.3149, device='cuda:0')) | > current_lr_0: 0.0001720539274767283 | > current_lr_1: 0.0001720539274767283 | > step_time: 0.4292 (0.42441669377413666) | > loader_time: 0.0056 (0.00637706843289462)  --> TIME: 2025-02-25 20:37:18 -- STEP: 61/72 -- GLOBAL_STEP: 1086750 | > loss_disc: 2.547823905944824 (2.4797427849691416) | > loss_disc_real_0: 0.14807674288749695 (0.0968986913928243) | > loss_disc_real_1: 0.29303884506225586 (0.21295969896629208) | > loss_disc_real_2: 0.24164944887161255 (0.2272001526883391) | > loss_disc_real_3: 0.27155444025993347 (0.22925844055707337) | > loss_disc_real_4: 0.27453410625457764 (0.23977255234952832) | > loss_disc_real_5: 0.26229745149612427 (0.24501446231466825) | > loss_0: 2.547823905944824 (2.4797427849691416) | > grad_norm_0: tensor(221.8859, device='cuda:0') (tensor(218.9179, device='cuda:0')) | > loss_gen: 2.5174872875213623 (2.248408215944885) | > loss_kl: 1.9536770582199097 (2.0994189215488115) | > loss_feat: 3.618356466293335 (4.078970529994027) | > loss_mel: 15.954501152038574 (17.354423851263334) | > loss_duration: 1.5907270908355713 (1.5470686975072643) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.634748458862305 (27.328290220166817) | > grad_norm_1: tensor(4954.8994, device='cuda:0') (tensor(6194.6812, device='cuda:0')) | > current_lr_0: 0.0001720539274767283 | > current_lr_1: 0.0001720539274767283 | > step_time: 0.4542 (0.4408663061798596) | > loader_time: 0.0107 (0.008237651137054944) --> EVAL PERFORMANCE | > avg_loader_time: 0.00878767900063958 (+0.0002851318305646857) | > avg_step_time: 0.4501186122357006 (-0.0015277258107359581) | > avg_loss_disc: 2.485127079654747 (-0.18428811892657215) | > avg_loss_disc_real_0: 0.09744027508816248 (-0.05841366857500145) | > avg_loss_disc_real_1: 0.2138741952432713 (-0.005819426875718875) | > avg_loss_disc_real_2: 0.22662124247618123 (-0.006705295451929882) | > avg_loss_disc_real_3: 0.22912797016996733 (-0.003131628876001047) | > avg_loss_disc_real_4: 0.2417830417693501 (-0.0009173536804360227) | > avg_loss_disc_real_5: 0.24676283921154452 (-0.0015514373359545863) | > avg_loss_0: 2.485127079654747 (-0.18428811892657215) | > avg_grad_norm_0: tensor(222.9981, device='cuda:0') (tensor(-83.7946, device='cuda:0')) | > avg_loss_gen: 2.246644510349758 (+0.204243685158207) | > avg_loss_kl: 2.106507758019675 (-0.030677095265456256) | > avg_loss_feat: 4.056865618262493 (+0.446239360621278) | > avg_loss_mel: 17.31114716597007 (-0.1892282056136949) | > avg_loss_duration: 1.5552845790352623 (+0.0025974512100221947) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.27644960645219 (+0.4331753690477811) | > avg_grad_norm_1: tensor(6298.1890, device='cuda:0') (tensor(-1253.5874, device='cuda:0'))  > EPOCH: 1205/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:37:25)   --> TIME: 2025-02-25 20:37:44 -- STEP: 39/72 -- GLOBAL_STEP: 1086800 | > loss_disc: 2.645979404449463 (2.538384180802565) | > loss_disc_real_0: 0.1113496720790863 (0.09883551060771331) | > loss_disc_real_1: 0.26673001050949097 (0.23858711562859705) | > loss_disc_real_2: 0.1992628574371338 (0.21940165834549147) | > loss_disc_real_3: 0.2527810037136078 (0.2395630127344376) | > loss_disc_real_4: 0.22642244398593903 (0.24017201211207953) | > loss_disc_real_5: 0.2289678007364273 (0.24345404712053445) | > loss_0: 2.645979404449463 (2.538384180802565) | > grad_norm_0: tensor(191.4678, device='cuda:0') (tensor(264.7591, device='cuda:0')) | > loss_gen: 2.111427068710327 (2.2389344771703086) | > loss_kl: 2.1392595767974854 (1.9473266632128985) | > loss_feat: 3.7253706455230713 (4.117552433258447) | > loss_mel: 17.593509674072266 (17.055451564299748) | > loss_duration: 1.5282042026519775 (1.508708342527732) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.09777069091797 (26.86797352326222) | > grad_norm_1: tensor(10142.2764, device='cuda:0') (tensor(7679.3936, device='cuda:0')) | > current_lr_0: 0.00017203242073579372 | > current_lr_1: 0.00017203242073579372 | > step_time: 0.4517 (0.4358682632446289) | > loader_time: 0.0081 (0.007264950336554112) --> EVAL PERFORMANCE | > avg_loader_time: 0.00892591140639614 (+0.00013823240575656058) | > avg_step_time: 0.4522117561017963 (+0.0020931438660957147) | > avg_loss_disc: 2.54989712339052 (+0.064770043735773) | > avg_loss_disc_real_0: 0.10545796478613162 (+0.008017689697969135) | > avg_loss_disc_real_1: 0.2356004264782852 (+0.021726231235013882) | > avg_loss_disc_real_2: 0.22569930826274442 (-0.0009219342134368091) | > avg_loss_disc_real_3: 0.23400600191573023 (+0.004878031745762901) | > avg_loss_disc_real_4: 0.2417587161483899 (-2.432562096019164e-05) | > avg_loss_disc_real_5: 0.24580036754339513 (-0.0009624716681493894) | > avg_loss_0: 2.54989712339052 (+0.064770043735773) | > avg_grad_norm_0: tensor(285.3211, device='cuda:0') (+tensor(62.3230, device='cuda:0')) | > avg_loss_gen: 2.199664574273874 (-0.0469799360758838) | > avg_loss_kl: 1.9846648447950122 (-0.12184291322466301) | > avg_loss_feat: 3.994500586684321 (-0.06236503157817186) | > avg_loss_mel: 16.83496420148392 (-0.4761829644861493) | > avg_loss_duration: 1.5400509464908654 (-0.015233632544396913) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.55384515036999 (-0.7226044560821983) | > avg_grad_norm_1: tensor(7920.9248, device='cuda:0') (+tensor(1622.7358, device='cuda:0'))  > EPOCH: 1206/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:38:02)   --> TIME: 2025-02-25 20:38:11 -- STEP: 17/72 -- GLOBAL_STEP: 1086850 | > loss_disc: 2.352689266204834 (2.5039358559776757) | > loss_disc_real_0: 0.06346312165260315 (0.0787883979432723) | > loss_disc_real_1: 0.2389669269323349 (0.2081132925608579) | > loss_disc_real_2: 0.17758220434188843 (0.2312926825355081) | > loss_disc_real_3: 0.24838460981845856 (0.22783167747890248) | > loss_disc_real_4: 0.23546859622001648 (0.24246775227434494) | > loss_disc_real_5: 0.26936599612236023 (0.24672921615488388) | > loss_0: 2.352689266204834 (2.5039358559776757) | > grad_norm_0: tensor(157.3417, device='cuda:0') (tensor(275.8907, device='cuda:0')) | > loss_gen: 2.5154526233673096 (2.2025141014772305) | > loss_kl: 1.9830087423324585 (1.8983171196544872) | > loss_feat: 4.390987873077393 (4.1831682850332825) | > loss_mel: 17.6350040435791 (16.715941373039698) | > loss_duration: 1.519561767578125 (1.4898829390020931) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.044015884399414 (26.489823734059055) | > grad_norm_1: tensor(5479.8232, device='cuda:0') (tensor(9318.8594, device='cuda:0')) | > current_lr_0: 0.00017201091668320173 | > current_lr_1: 0.00017201091668320173 | > step_time: 0.4525 (0.4369237843681784) | > loader_time: 0.008 (0.0069884692921358)  --> TIME: 2025-02-25 20:38:35 -- STEP: 67/72 -- GLOBAL_STEP: 1086900 | > loss_disc: 2.513432741165161 (2.544269857121937) | > loss_disc_real_0: 0.0584205724298954 (0.08790148805771303) | > loss_disc_real_1: 0.19757062196731567 (0.21661372660700953) | > loss_disc_real_2: 0.21729952096939087 (0.22591295015456073) | > loss_disc_real_3: 0.19605135917663574 (0.22708000175988496) | > loss_disc_real_4: 0.2488497495651245 (0.23144800872055452) | > loss_disc_real_5: 0.23872683942317963 (0.24392190308713202) | > loss_0: 2.513432741165161 (2.544269857121937) | > grad_norm_0: tensor(333.9771, device='cuda:0') (tensor(293.9853, device='cuda:0')) | > loss_gen: 1.9621992111206055 (2.1642247687524834) | > loss_kl: 2.0970230102539062 (1.950491286035794) | > loss_feat: 3.7208220958709717 (4.064112883895192) | > loss_mel: 17.002300262451172 (16.56986642239699) | > loss_duration: 1.634541392326355 (1.5318336024213193) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.416885375976562 (26.280528908345236) | > grad_norm_1: tensor(10986.6172, device='cuda:0') (tensor(9478.0117, device='cuda:0')) | > current_lr_0: 0.00017201091668320173 | > current_lr_1: 0.00017201091668320173 | > step_time: 0.4941 (0.4506276223196912) | > loader_time: 0.0122 (0.008561159247782696) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872096545259718 (-0.00020494595379896147) | > avg_step_time: 0.45442186274998625 (+0.0022101066481899356) | > avg_loss_disc: 2.5444455381850117 (-0.005451585205508369) | > avg_loss_disc_real_0: 0.08965956493162776 (-0.01579839985450386) | > avg_loss_disc_real_1: 0.21612757255493753 (-0.01947285392334766) | > avg_loss_disc_real_2: 0.22547446841924962 (-0.00022483984349480624) | > avg_loss_disc_real_3: 0.2269513298508147 (-0.007054672064915529) | > avg_loss_disc_real_4: 0.2322748754645737 (-0.009483840683816191) | > avg_loss_disc_real_5: 0.24398508898808924 (-0.0018152785553058881) | > avg_loss_0: 2.5444455381850117 (-0.005451585205508369) | > avg_grad_norm_0: tensor(299.2353, device='cuda:0') (+tensor(13.9142, device='cuda:0')) | > avg_loss_gen: 2.16944783002558 (-0.030216744248293903) | > avg_loss_kl: 1.9558613955135076 (-0.02880344928150458) | > avg_loss_feat: 4.059571031113747 (+0.06507044442942611) | > avg_loss_mel: 16.575836705489902 (-0.25912749599401863) | > avg_loss_duration: 1.5357183688123461 (-0.004332577678519245) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.296435288979975 (-0.2574098613900162) | > avg_grad_norm_1: tensor(9339.5566, device='cuda:0') (+tensor(1418.6318, device='cuda:0'))  > EPOCH: 1207/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:38:39)   --> TIME: 2025-02-25 20:39:01 -- STEP: 45/72 -- GLOBAL_STEP: 1086950 | > loss_disc: 2.6697194576263428 (2.5037471930185955) | > loss_disc_real_0: 0.11435279250144958 (0.09467770796683102) | > loss_disc_real_1: 0.24127759039402008 (0.22043012215031518) | > loss_disc_real_2: 0.24574658274650574 (0.22986750304698944) | > loss_disc_real_3: 0.2702640891075134 (0.23269281884034473) | > loss_disc_real_4: 0.2433895766735077 (0.23954617381095886) | > loss_disc_real_5: 0.2499576210975647 (0.24456700285275776) | > loss_0: 2.6697194576263428 (2.5037471930185955) | > grad_norm_0: tensor(127.4255, device='cuda:0') (tensor(307.2328, device='cuda:0')) | > loss_gen: 1.9272434711456299 (2.2479554467731053) | > loss_kl: 2.0230672359466553 (1.8997978846232095) | > loss_feat: 3.615880012512207 (4.097751590940687) | > loss_mel: 15.93708610534668 (16.726511976453992) | > loss_duration: 1.551303744316101 (1.5101997825834486) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.054582595825195 (26.48221664428711) | > grad_norm_1: tensor(2133.8359, device='cuda:0') (tensor(8270.3604, device='cuda:0')) | > current_lr_0: 0.00017198941531861633 | > current_lr_1: 0.00017198941531861633 | > step_time: 0.45 (0.4353918340471056) | > loader_time: 0.0084 (0.007604927486843533) --> EVAL PERFORMANCE | > avg_loader_time: 0.008842992110991139 (+0.00012202665839395992) | > avg_step_time: 0.4481557490120472 (-0.006266113737939061) | > avg_loss_disc: 2.5721708821578764 (+0.027725343972864636) | > avg_loss_disc_real_0: 0.10198919002858688 (+0.012329625096959118) | > avg_loss_disc_real_1: 0.2345737596632729 (+0.01844618710833537) | > avg_loss_disc_real_2: 0.2268387776025584 (+0.0013643091833087928) | > avg_loss_disc_real_3: 0.2312734240377453 (+0.004322094186930608) | > avg_loss_disc_real_4: 0.24025073458611126 (+0.00797585912153756) | > avg_loss_disc_real_5: 0.24357362133516391 (-0.0004114676529253236) | > avg_loss_0: 2.5721708821578764 (+0.027725343972864636) | > avg_grad_norm_0: tensor(315.3583, device='cuda:0') (+tensor(16.1230, device='cuda:0')) | > avg_loss_gen: 2.2001214917276943 (+0.030673661702114163) | > avg_loss_kl: 1.9528193289125468 (-0.0030420666009607356) | > avg_loss_feat: 3.9704229563054887 (-0.08914807480825848) | > avg_loss_mel: 16.617883870299433 (+0.04204716480953152) | > avg_loss_duration: 1.5357809570473684 (+6.258823502225397e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.27702860765055 (-0.019406681329424913) | > avg_grad_norm_1: tensor(7787.6309, device='cuda:0') (tensor(-1551.9258, device='cuda:0'))  > EPOCH: 1208/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:39:15)   --> TIME: 2025-02-25 20:39:27 -- STEP: 23/72 -- GLOBAL_STEP: 1087000 | > loss_disc: 2.489187240600586 (2.639422126438307) | > loss_disc_real_0: 0.12276093661785126 (0.14004349384618842) | > loss_disc_real_1: 0.2364373505115509 (0.2218814200681189) | > loss_disc_real_2: 0.27985742688179016 (0.2547730408284975) | > loss_disc_real_3: 0.2333206981420517 (0.2442051826611809) | > loss_disc_real_4: 0.2450128048658371 (0.24636014907256418) | > loss_disc_real_5: 0.2695908844470978 (0.25601880835450214) | > loss_0: 2.489187240600586 (2.639422126438307) | > grad_norm_0: tensor(362.9708, device='cuda:0') (tensor(307.9493, device='cuda:0')) | > loss_gen: 2.431208610534668 (2.1421145874521006) | > loss_kl: 2.2809081077575684 (2.1667102316151494) | > loss_feat: 4.158005714416504 (3.8030957346377163) | > loss_mel: 17.956554412841797 (17.625627600628402) | > loss_duration: 1.57327139377594 (1.535030105839605) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.399946212768555 (27.272577949192215) | > grad_norm_1: tensor(4326.2476, device='cuda:0') (tensor(6958.0269, device='cuda:0')) | > current_lr_0: 0.00017196791664170148 | > current_lr_1: 0.00017196791664170148 | > step_time: 0.4229 (0.430132938467938) | > loader_time: 0.0062 (0.00660271229951278) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652858331169877 (-0.0001901337798212617) | > avg_step_time: 0.44673369971799176 (-0.0014220492940554275) | > avg_loss_disc: 2.6369744085929763 (+0.0648035264350999) | > avg_loss_disc_real_0: 0.1229229961902323 (+0.02093380616164542) | > avg_loss_disc_real_1: 0.21252772589804422 (-0.02204603376522868) | > avg_loss_disc_real_2: 0.2419596757267562 (+0.015120898124197785) | > avg_loss_disc_real_3: 0.23183317944197587 (+0.0005597554042305608) | > avg_loss_disc_real_4: 0.24335826555608023 (+0.0031075309699689635) | > avg_loss_disc_real_5: 0.24922724466928295 (+0.0056536233341190345) | > avg_loss_0: 2.6369744085929763 (+0.0648035264350999) | > avg_grad_norm_0: tensor(344.9173, device='cuda:0') (+tensor(29.5590, device='cuda:0')) | > avg_loss_gen: 2.0622692561485407 (-0.1378522355791536) | > avg_loss_kl: 2.1445028513250217 (+0.19168352241247488) | > avg_loss_feat: 3.746875934197869 (-0.22354702210761968) | > avg_loss_mel: 17.231116321724894 (+0.6132324514254606) | > avg_loss_duration: 1.5577720252560896 (+0.02199106820872121) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.74253641047948 (+0.465507802828931) | > avg_grad_norm_1: tensor(9413.3037, device='cuda:0') (+tensor(1625.6729, device='cuda:0'))  > EPOCH: 1209/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:39:52)   --> TIME: 2025-02-25 20:39:54 -- STEP: 1/72 -- GLOBAL_STEP: 1087050 | > loss_disc: 2.73616886138916 (2.73616886138916) | > loss_disc_real_0: 0.08384688943624496 (0.08384688943624496) | > loss_disc_real_1: 0.22289468348026276 (0.22289468348026276) | > loss_disc_real_2: 0.2539975643157959 (0.2539975643157959) | > loss_disc_real_3: 0.23360975086688995 (0.23360975086688995) | > loss_disc_real_4: 0.2465876340866089 (0.2465876340866089) | > loss_disc_real_5: 0.24301327764987946 (0.24301327764987946) | > loss_0: 2.73616886138916 (2.73616886138916) | > grad_norm_0: tensor(436.3036, device='cuda:0') (tensor(436.3036, device='cuda:0')) | > loss_gen: 1.7723026275634766 (1.7723026275634766) | > loss_kl: 1.6122232675552368 (1.6122232675552368) | > loss_feat: 3.6241135597229004 (3.6241135597229004) | > loss_mel: 17.417163848876953 (17.417163848876953) | > loss_duration: 1.5244410037994385 (1.5244410037994385) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.95024299621582 (25.95024299621582) | > grad_norm_1: tensor(9104.1318, device='cuda:0') (tensor(9104.1318, device='cuda:0')) | > current_lr_0: 0.00017194642065212126 | > current_lr_1: 0.00017194642065212126 | > step_time: 0.4242 (0.4241821765899658) | > loader_time: 0.0062 (0.006200075149536133)  --> TIME: 2025-02-25 20:40:17 -- STEP: 51/72 -- GLOBAL_STEP: 1087100 | > loss_disc: 2.5197885036468506 (2.63932814317591) | > loss_disc_real_0: 0.0776439905166626 (0.13103073280231625) | > loss_disc_real_1: 0.24581053853034973 (0.23289479315280914) | > loss_disc_real_2: 0.21547891199588776 (0.24707061139976275) | > loss_disc_real_3: 0.2521188259124756 (0.23090868545513527) | > loss_disc_real_4: 0.1988646239042282 (0.24021045863628387) | > loss_disc_real_5: 0.24527320265769958 (0.2411569038442537) | > loss_0: 2.5197885036468506 (2.63932814317591) | > grad_norm_0: tensor(299.4571, device='cuda:0') (tensor(349.6081, device='cuda:0')) | > loss_gen: 2.250403881072998 (2.114595378146452) | > loss_kl: 1.9868524074554443 (1.9824401444079829) | > loss_feat: 3.9816501140594482 (3.7983871254266477) | > loss_mel: 16.239015579223633 (16.638078932668638) | > loss_duration: 1.5538935661315918 (1.5248984706168085) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.011816024780273 (26.058400135414274) | > grad_norm_1: tensor(12852.3613, device='cuda:0') (tensor(11501.2334, device='cuda:0')) | > current_lr_0: 0.00017194642065212126 | > current_lr_1: 0.00017194642065212126 | > step_time: 0.4574 (0.4426458209168677) | > loader_time: 0.0092 (0.007740273195154527) --> EVAL PERFORMANCE | > avg_loader_time: 0.008602585591061014 (-5.0272740108863595e-05) | > avg_step_time: 0.4537365671614526 (+0.007002867443460847) | > avg_loss_disc: 2.6295104765556223 (-0.00746393203735396) | > avg_loss_disc_real_0: 0.13019374317266572 (+0.007270746982433421) | > avg_loss_disc_real_1: 0.22880670848027082 (+0.016278982582226598) | > avg_loss_disc_real_2: 0.24396157075821515 (+0.00200189503145895) | > avg_loss_disc_real_3: 0.23182662293105058 (-6.556510925292969e-06) | > avg_loss_disc_real_4: 0.2405668751034938 (-0.0027913904525864175) | > avg_loss_disc_real_5: 0.24146012248287738 (-0.007767122186405573) | > avg_loss_0: 2.6295104765556223 (-0.00746393203735396) | > avg_grad_norm_0: tensor(384.6706, device='cuda:0') (+tensor(39.7533, device='cuda:0')) | > avg_loss_gen: 2.1066815685218483 (+0.04441231237330756) | > avg_loss_kl: 2.0182126307151687 (-0.12629022060985307) | > avg_loss_feat: 3.7560269463230185 (+0.00915101212514946) | > avg_loss_mel: 16.641422312024606 (-0.5896940097002883) | > avg_loss_duration: 1.5409170943246764 (-0.016854930931413215) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.06326060227945 (-0.6792758082000319) | > avg_grad_norm_1: tensor(12519.0791, device='cuda:0') (+tensor(3105.7754, device='cuda:0'))  > EPOCH: 1210/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:40:28)   --> TIME: 2025-02-25 20:40:44 -- STEP: 29/72 -- GLOBAL_STEP: 1087150 | > loss_disc: 2.6240603923797607 (2.6016170731906234) | > loss_disc_real_0: 0.12236620485782623 (0.12097335748117546) | > loss_disc_real_1: 0.2170121818780899 (0.21568906615520345) | > loss_disc_real_2: 0.25484442710876465 (0.23120903146678004) | > loss_disc_real_3: 0.25408899784088135 (0.23190989268237147) | > loss_disc_real_4: 0.26000168919563293 (0.24541977715903315) | > loss_disc_real_5: 0.26718664169311523 (0.24922335558924183) | > loss_0: 2.6240603923797607 (2.6016170731906234) | > grad_norm_0: tensor(692.1725, device='cuda:0') (tensor(478.0383, device='cuda:0')) | > loss_gen: 2.212275266647339 (2.1264024890702347) | > loss_kl: 2.0002596378326416 (1.9256124332033355) | > loss_feat: 3.9629669189453125 (3.9317397495795943) | > loss_mel: 16.814346313476562 (16.81002801040123) | > loss_duration: 1.4934290647506714 (1.4889670651534508) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.4832763671875 (26.282749767961175) | > grad_norm_1: tensor(15191.0439, device='cuda:0') (tensor(14619.8506, device='cuda:0')) | > current_lr_0: 0.00017192492734953974 | > current_lr_1: 0.00017192492734953974 | > step_time: 0.4546 (0.4404560039783346) | > loader_time: 0.0081 (0.007173587535989696) --> EVAL PERFORMANCE | > avg_loader_time: 0.008843902131201517 (+0.0002413165401405036) | > avg_step_time: 0.46082315982227595 (+0.007086592660823343) | > avg_loss_disc: 2.674313209426235 (+0.04480273287061287) | > avg_loss_disc_real_0: 0.12796537079651596 (-0.0022283723761497565) | > avg_loss_disc_real_1: 0.21275365966516482 (-0.016053048815105997) | > avg_loss_disc_real_2: 0.23570987792082235 (-0.008251692837392799) | > avg_loss_disc_real_3: 0.22710118369317392 (-0.004725439237876661) | > avg_loss_disc_real_4: 0.2459553555703499 (+0.005388480466856083) | > avg_loss_disc_real_5: 0.24538393465565964 (+0.0039238121727822595) | > avg_loss_0: 2.674313209426235 (+0.04480273287061287) | > avg_grad_norm_0: tensor(472.5722, device='cuda:0') (+tensor(87.9017, device='cuda:0')) | > avg_loss_gen: 2.056061868936242 (-0.05061969958560608) | > avg_loss_kl: 1.9696817482021494 (-0.04853088251301929) | > avg_loss_feat: 3.7783674656505317 (+0.022340519327513242) | > avg_loss_mel: 16.61082120008871 (-0.03060111193589421) | > avg_loss_duration: 1.534052852173926 (-0.0068642421507503215) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.94898522068078 (-0.11427538159867012) | > avg_grad_norm_1: tensor(14167.8574, device='cuda:0') (+tensor(1648.7783, device='cuda:0'))  > EPOCH: 1211/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:41:06)   --> TIME: 2025-02-25 20:41:11 -- STEP: 7/72 -- GLOBAL_STEP: 1087200 | > loss_disc: 2.4727511405944824 (2.55690142086574) | > loss_disc_real_0: 0.122159942984581 (0.1207866540976933) | > loss_disc_real_1: 0.20165692269802094 (0.23093339587960923) | > loss_disc_real_2: 0.16508986055850983 (0.134088468338762) | > loss_disc_real_3: 0.2337639331817627 (0.24120196912969863) | > loss_disc_real_4: 0.22936378419399261 (0.23663525921957834) | > loss_disc_real_5: 0.23222489655017853 (0.24255442832197463) | > loss_0: 2.4727511405944824 (2.55690142086574) | > grad_norm_0: tensor(305.6604, device='cuda:0') (tensor(360.9122, device='cuda:0')) | > loss_gen: 2.4141247272491455 (2.186635800770351) | > loss_kl: 2.9552175998687744 (2.9579414299556186) | > loss_feat: 4.46028470993042 (4.079767772129604) | > loss_mel: 20.488475799560547 (20.87649154663086) | > loss_duration: 1.6079764366149902 (1.6686761038643974) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.92608070373535 (31.76951244899205) | > grad_norm_1: tensor(7406.0767, device='cuda:0') (tensor(11520.7285, device='cuda:0')) | > current_lr_0: 0.00017190343673362105 | > current_lr_1: 0.00017190343673362105 | > step_time: 0.4492 (0.4393421581813267) | > loader_time: 0.0075 (0.006316150937761579)  --> TIME: 2025-02-25 20:41:34 -- STEP: 57/72 -- GLOBAL_STEP: 1087250 | > loss_disc: 2.874812126159668 (2.5951603420993736) | > loss_disc_real_0: 0.062015414237976074 (0.10743739821931772) | > loss_disc_real_1: 0.12622244656085968 (0.2174618313448471) | > loss_disc_real_2: 0.2771743834018707 (0.22016166895627975) | > loss_disc_real_3: 0.31733813881874084 (0.23317062985478787) | > loss_disc_real_4: 0.2713454067707062 (0.23661557467360245) | > loss_disc_real_5: 0.2342320829629898 (0.23833248855774863) | > loss_0: 2.874812126159668 (2.5951603420993736) | > grad_norm_0: tensor(576.0212, device='cuda:0') (tensor(314.4678, device='cuda:0')) | > loss_gen: 1.87396240234375 (2.1361644435347173) | > loss_kl: 2.174514055252075 (2.288549442040294) | > loss_feat: 3.791335344314575 (4.071487234349836) | > loss_mel: 16.05506706237793 (17.793291309423612) | > loss_duration: 1.5981454849243164 (1.5618096004452622) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.493022918701172 (27.85130206325598) | > grad_norm_1: tensor(8867.3320, device='cuda:0') (tensor(9085.9668, device='cuda:0')) | > current_lr_0: 0.00017190343673362105 | > current_lr_1: 0.00017190343673362105 | > step_time: 0.4992 (0.4488424292781897) | > loader_time: 0.0099 (0.008143136375828794) --> EVAL PERFORMANCE | > avg_loader_time: 0.008769824471272211 (-7.407765992930616e-05) | > avg_step_time: 0.45866849388874753 (-0.0021546659335284213) | > avg_loss_disc: 2.63588088666889 (-0.038432322757345094) | > avg_loss_disc_real_0: 0.12091744344838908 (-0.007047927348126884) | > avg_loss_disc_real_1: 0.21753090504609363 (+0.004777245380928802) | > avg_loss_disc_real_2: 0.2237300298793215 (-0.011979848041500846) | > avg_loss_disc_real_3: 0.23711827943022823 (+0.01001709573705431) | > avg_loss_disc_real_4: 0.2404011989563284 (-0.005554156614021505) | > avg_loss_disc_real_5: 0.2389159103934194 (-0.006468024262240246) | > avg_loss_0: 2.63588088666889 (-0.038432322757345094) | > avg_grad_norm_0: tensor(319.7337, device='cuda:0') (tensor(-152.8386, device='cuda:0')) | > avg_loss_gen: 2.1026734281593646 (+0.046611559223122434) | > avg_loss_kl: 2.2690009970060543 (+0.2993192488039049) | > avg_loss_feat: 3.92490444720631 (+0.1465369815557782) | > avg_loss_mel: 17.594195392769826 (+0.983374192681115) | > avg_loss_duration: 1.5679214806623862 (+0.03386862848846017) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.458695747482945 (+1.5097105268021664) | > avg_grad_norm_1: tensor(8883.8467, device='cuda:0') (tensor(-5284.0107, device='cuda:0'))  > EPOCH: 1212/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:41:44)   --> TIME: 2025-02-25 20:42:01 -- STEP: 35/72 -- GLOBAL_STEP: 1087300 | > loss_disc: 2.71404767036438 (2.7205903121403288) | > loss_disc_real_0: 0.09289157390594482 (0.22272940778306552) | > loss_disc_real_1: 0.17082218825817108 (0.23304428032466343) | > loss_disc_real_2: 0.292293906211853 (0.24567230684416635) | > loss_disc_real_3: 0.24135111272335052 (0.24463764301368168) | > loss_disc_real_4: 0.21265481412410736 (0.2504784047603607) | > loss_disc_real_5: 0.2595362663269043 (0.24817311508314951) | > loss_0: 2.71404767036438 (2.7205903121403288) | > grad_norm_0: tensor(174.4612, device='cuda:0') (tensor(370.1443, device='cuda:0')) | > loss_gen: 2.079172372817993 (2.1685263633728034) | > loss_kl: 2.2576282024383545 (1.978117871284485) | > loss_feat: 3.560208320617676 (3.6612294742039273) | > loss_mel: 17.19052505493164 (16.726824923924042) | > loss_duration: 1.5493614673614502 (1.4967510870524816) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.63689613342285 (26.031449835641045) | > grad_norm_1: tensor(11319.5908, device='cuda:0') (tensor(10063.4775, device='cuda:0')) | > current_lr_0: 0.00017188194880402935 | > current_lr_1: 0.00017188194880402935 | > step_time: 0.4396 (0.4407311371394566) | > loader_time: 0.0083 (0.007268973759242466) --> EVAL PERFORMANCE | > avg_loader_time: 0.008741909349468393 (-2.791512180381865e-05) | > avg_step_time: 0.45576328962621554 (-0.0029052042625319907) | > avg_loss_disc: 2.7280913104473705 (+0.09221042377848043) | > avg_loss_disc_real_0: 0.2025767133479387 (+0.08165926989954962) | > avg_loss_disc_real_1: 0.22855220923961048 (+0.011021304193516851) | > avg_loss_disc_real_2: 0.23154692939469512 (+0.007816899515373621) | > avg_loss_disc_real_3: 0.23916540788093082 (+0.0020471284507025955) | > avg_loss_disc_real_4: 0.23949712221051606 (-0.0009040767458123322) | > avg_loss_disc_real_5: 0.24177921909681507 (+0.002863308703395684) | > avg_loss_0: 2.7280913104473705 (+0.09221042377848043) | > avg_grad_norm_0: tensor(368.9036, device='cuda:0') (+tensor(49.1700, device='cuda:0')) | > avg_loss_gen: 2.0787432227336184 (-0.023930205425746198) | > avg_loss_kl: 2.077455532382911 (-0.19154546462314315) | > avg_loss_feat: 3.6935121005689595 (-0.23139234663735042) | > avg_loss_mel: 16.802930428948205 (-0.7912649638216216) | > avg_loss_duration: 1.542250443512285 (-0.025671037150101217) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.19489159382565 (-1.263804153657297) | > avg_grad_norm_1: tensor(7964.1899, device='cuda:0') (tensor(-919.6567, device='cuda:0'))  > EPOCH: 1213/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:42:21)   --> TIME: 2025-02-25 20:42:28 -- STEP: 13/72 -- GLOBAL_STEP: 1087350 | > loss_disc: 2.793943405151367 (2.641476484445425) | > loss_disc_real_0: 0.18217125535011292 (0.1691626450763299) | > loss_disc_real_1: 0.16653454303741455 (0.24991682974191812) | > loss_disc_real_2: 0.28168463706970215 (0.2402195013486422) | > loss_disc_real_3: 0.22352048754692078 (0.23449256786933312) | > loss_disc_real_4: 0.23997433483600616 (0.2435523237173374) | > loss_disc_real_5: 0.28069868683815 (0.25779093228853667) | > loss_0: 2.793943405151367 (2.641476484445425) | > grad_norm_0: tensor(501.7722, device='cuda:0') (tensor(317.0081, device='cuda:0')) | > loss_gen: 1.9050596952438354 (2.203311874316289) | > loss_kl: 2.1200897693634033 (2.0087339327885556) | > loss_feat: 3.8900814056396484 (3.938773173552293) | > loss_mel: 16.703487396240234 (17.14922098013071) | > loss_duration: 1.444791555404663 (1.4963882244550264) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.063508987426758 (26.796428240262546) | > grad_norm_1: tensor(5522.8252, device='cuda:0') (tensor(3646.3958, device='cuda:0')) | > current_lr_0: 0.00017186046356042884 | > current_lr_1: 0.00017186046356042884 | > step_time: 0.4606 (0.43342991975637585) | > loader_time: 0.0058 (0.006519060868483324)  --> TIME: 2025-02-25 20:42:52 -- STEP: 63/72 -- GLOBAL_STEP: 1087400 | > loss_disc: 2.730342149734497 (2.6752777137453596) | > loss_disc_real_0: 0.32149627804756165 (0.1638484766322469) | > loss_disc_real_1: 0.15172837674617767 (0.23191345518543607) | > loss_disc_real_2: 0.2831023335456848 (0.23913615610864428) | > loss_disc_real_3: 0.21023699641227722 (0.23678094054025317) | > loss_disc_real_4: 0.29720792174339294 (0.2447979776632218) | > loss_disc_real_5: 0.29253897070884705 (0.24987695969286422) | > loss_0: 2.730342149734497 (2.6752777137453596) | > grad_norm_0: tensor(465.8071, device='cuda:0') (tensor(279.2899, device='cuda:0')) | > loss_gen: 2.288306474685669 (2.092449767248972) | > loss_kl: 2.0993189811706543 (2.129537993007235) | > loss_feat: 3.3691635131835938 (3.7087031962379577) | > loss_mel: 16.468460083007812 (17.297064887152775) | > loss_duration: 1.5846993923187256 (1.5402985186803908) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.80994987487793 (26.7680542355492) | > grad_norm_1: tensor(1558.8730, device='cuda:0') (tensor(7404.5854, device='cuda:0')) | > current_lr_0: 0.00017186046356042884 | > current_lr_1: 0.00017186046356042884 | > step_time: 0.5 (0.4478260903131394) | > loader_time: 0.0112 (0.008276284687102788) --> EVAL PERFORMANCE | > avg_loader_time: 0.008703406427947568 (-3.850292152082421e-05) | > avg_step_time: 0.4576782139254288 (+0.0019149242992132498) | > avg_loss_disc: 2.677490922766672 (-0.05060038768069841) | > avg_loss_disc_real_0: 0.15837789190487123 (-0.04419882144306747) | > avg_loss_disc_real_1: 0.23589719747993307 (+0.007344988240322592) | > avg_loss_disc_real_2: 0.23921407486351443 (+0.007667145468819303) | > avg_loss_disc_real_3: 0.2348587991486133 (-0.0043066087323175295) | > avg_loss_disc_real_4: 0.2446570012351157 (+0.005159879024599634) | > avg_loss_disc_real_5: 0.2516580236629701 (+0.009878804566155003) | > avg_loss_0: 2.677490922766672 (-0.05060038768069841) | > avg_grad_norm_0: tensor(294.7592, device='cuda:0') (tensor(-74.1444, device='cuda:0')) | > avg_loss_gen: 2.0896615394404243 (+0.010918316706805875) | > avg_loss_kl: 2.1447602292181736 (+0.06730469683526241) | > avg_loss_feat: 3.691982000646457 (-0.0015300999225025258) | > avg_loss_mel: 17.321309962742763 (+0.5183795337945583) | > avg_loss_duration: 1.5484840131141768 (+0.0062335696018918085) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.7961976225947 (+0.6013060287690521) | > avg_grad_norm_1: tensor(7259.1968, device='cuda:0') (tensor(-704.9932, device='cuda:0'))  > EPOCH: 1214/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:42:58)   --> TIME: 2025-02-25 20:43:18 -- STEP: 41/72 -- GLOBAL_STEP: 1087450 | > loss_disc: 2.85835337638855 (2.7071510175379316) | > loss_disc_real_0: 0.18681776523590088 (0.11376442205978603) | > loss_disc_real_1: 0.20538346469402313 (0.2619631359489953) | > loss_disc_real_2: 0.1903940737247467 (0.2646756506547695) | > loss_disc_real_3: 0.26889151334762573 (0.22532820447189053) | > loss_disc_real_4: 0.21983493864536285 (0.2396437522114777) | > loss_disc_real_5: 0.23983734846115112 (0.2438066569043369) | > loss_0: 2.85835337638855 (2.7071510175379316) | > grad_norm_0: tensor(698.2592, device='cuda:0') (tensor(311.7083, device='cuda:0')) | > loss_gen: 1.893621563911438 (2.033156147817286) | > loss_kl: 2.3093101978302 (2.0765732032496755) | > loss_feat: 3.511796474456787 (3.6942361622321895) | > loss_mel: 16.902257919311523 (17.00290854384259) | > loss_duration: 1.5484720468521118 (1.523174631886366) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.16545867919922 (26.330048816960034) | > grad_norm_1: tensor(16333.9824, device='cuda:0') (tensor(10170.4883, device='cuda:0')) | > current_lr_0: 0.00017183898100248379 | > current_lr_1: 0.00017183898100248379 | > step_time: 0.4511 (0.4361834584212885) | > loader_time: 0.0099 (0.007459710283977229) --> EVAL PERFORMANCE | > avg_loader_time: 0.008961758143465283 (+0.00025835171551771494) | > avg_step_time: 0.453326745772026 (-0.0043514681534028155) | > avg_loss_disc: 2.756291171194802 (+0.07880024842812983) | > avg_loss_disc_real_0: 0.13819217792069408 (-0.02018571398417715) | > avg_loss_disc_real_1: 0.25961652230209026 (+0.02371932482215719) | > avg_loss_disc_real_2: 0.2505324770447233 (+0.011318402181208886) | > avg_loss_disc_real_3: 0.24040338669864225 (+0.0055445875500289565) | > avg_loss_disc_real_4: 0.2332063380261542 (-0.011450663208961487) | > avg_loss_disc_real_5: 0.24734038261460586 (-0.0043176410483642125) | > avg_loss_0: 2.756291171194802 (+0.07880024842812983) | > avg_grad_norm_0: tensor(295.1133, device='cuda:0') (+tensor(0.3542, device='cuda:0')) | > avg_loss_gen: 2.050670397113746 (-0.03899114232667822) | > avg_loss_kl: 2.186668661278738 (+0.04190843206056449) | > avg_loss_feat: 3.6846303637598603 (-0.00735163688659668) | > avg_loss_mel: 17.323614268235755 (+0.002304305492991432) | > avg_loss_duration: 1.5613317674314473 (+0.012847754317270477) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.80691565930004 (+0.010718036705338108) | > avg_grad_norm_1: tensor(8317.5469, device='cuda:0') (+tensor(1058.3501, device='cuda:0'))  > EPOCH: 1215/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:43:35)   --> TIME: 2025-02-25 20:43:45 -- STEP: 19/72 -- GLOBAL_STEP: 1087500 | > loss_disc: 2.5577166080474854 (2.6673850260282816) | > loss_disc_real_0: 0.07402181625366211 (0.12167840921565105) | > loss_disc_real_1: 0.2517194151878357 (0.2512769432444321) | > loss_disc_real_2: 0.22978989779949188 (0.23637319238562332) | > loss_disc_real_3: 0.2563200891017914 (0.2580851951712056) | > loss_disc_real_4: 0.23968805372714996 (0.2459219929419066) | > loss_disc_real_5: 0.23522832989692688 (0.2501593961527473) | > loss_0: 2.5577166080474854 (2.6673850260282816) | > grad_norm_0: tensor(157.7981, device='cuda:0') (tensor(271.6305, device='cuda:0')) | > loss_gen: 2.2495388984680176 (2.185105141840483) | > loss_kl: 2.1816961765289307 (2.0678128443266224) | > loss_feat: 3.508646249771118 (3.796153344606098) | > loss_mel: 16.75146484375 (17.282141484712298) | > loss_duration: 1.4929845333099365 (1.51089304371884) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.184329986572266 (26.842105865478516) | > grad_norm_1: tensor(3823.1338, device='cuda:0') (tensor(5551.1968, device='cuda:0')) | > current_lr_0: 0.00017181750112985847 | > current_lr_1: 0.00017181750112985847 | > step_time: 0.4288 (0.42842324156510203) | > loader_time: 0.0079 (0.006695433666831569)  --> TIME: 2025-02-25 20:44:08 -- STEP: 69/72 -- GLOBAL_STEP: 1087550 | > loss_disc: 2.5406572818756104 (2.639569852663123) | > loss_disc_real_0: 0.10772062093019485 (0.11647588279152261) | > loss_disc_real_1: 0.23286867141723633 (0.22985215614671292) | > loss_disc_real_2: 0.2341492474079132 (0.23349396344544232) | > loss_disc_real_3: 0.23426002264022827 (0.2442924455888029) | > loss_disc_real_4: 0.2330394685268402 (0.25594276578530023) | > loss_disc_real_5: 0.2490634173154831 (0.2515377929245216) | > loss_0: 2.5406572818756104 (2.639569852663123) | > grad_norm_0: tensor(193.8652, device='cuda:0') (tensor(276.8151, device='cuda:0')) | > loss_gen: 2.121150493621826 (2.124075435209965) | > loss_kl: 2.193686008453369 (2.068394467450571) | > loss_feat: 3.609081983566284 (3.71977808855582) | > loss_mel: 16.939754486083984 (16.901686198469527) | > loss_duration: 1.6464998722076416 (1.5461666393971099) | > amp_scaler: 8.0 (10.318840579710145) | > loss_1: 26.510173797607422 (26.360100732333418) | > grad_norm_1: tensor(8443.9482, device='cuda:0') (tensor(7105.9487, device='cuda:0')) | > current_lr_0: 0.00017181750112985847 | > current_lr_1: 0.00017181750112985847 | > step_time: 0.5339 (0.4485100697780001) | > loader_time: 0.0125 (0.008525668710902117) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859359620322644 (-0.0003681619402388439) | > avg_step_time: 0.45028212708486637 (-0.003044618687159606) | > avg_loss_disc: 2.632299601192206 (-0.12399157000259597) | > avg_loss_disc_real_0: 0.11501357830326323 (-0.02317859961743085) | > avg_loss_disc_real_1: 0.2302506989996198 (-0.02936582330247045) | > avg_loss_disc_real_2: 0.2326592782013853 (-0.017873198843338017) | > avg_loss_disc_real_3: 0.24371443143193147 (+0.00331104473328922) | > avg_loss_disc_real_4: 0.25520397848646403 (+0.02199764046030983) | > avg_loss_disc_real_5: 0.2511841766011547 (+0.0038437939865488546) | > avg_loss_0: 2.632299601192206 (-0.12399157000259597) | > avg_grad_norm_0: tensor(272.4726, device='cuda:0') (tensor(-22.6408, device='cuda:0')) | > avg_loss_gen: 2.1298282532624797 (+0.07915785614873361) | > avg_loss_kl: 2.0710066640880753 (-0.11566199719066272) | > avg_loss_feat: 3.7319789201440945 (+0.047348556384234186) | > avg_loss_mel: 16.893493209086678 (-0.4301210591490765) | > avg_loss_duration: 1.546417238007129 (-0.014914529424318257) | > avg_amp_scaler: 10.253521126760564 (+2.253521126760564) | > avg_loss_1: 26.372724183848206 (-0.4341914754518328) | > avg_grad_norm_1: tensor(7001.3110, device='cuda:0') (tensor(-1316.2358, device='cuda:0'))  > EPOCH: 1216/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:44:12)   --> TIME: 2025-02-25 20:44:35 -- STEP: 47/72 -- GLOBAL_STEP: 1087600 | > loss_disc: 2.55950927734375 (2.5328784242589424) | > loss_disc_real_0: 0.14977531135082245 (0.07526102777015657) | > loss_disc_real_1: 0.22539326548576355 (0.2202978435348957) | > loss_disc_real_2: 0.23977144062519073 (0.23159801008853506) | > loss_disc_real_3: 0.21883530914783478 (0.23470358931003732) | > loss_disc_real_4: 0.2359185516834259 (0.23962754740359934) | > loss_disc_real_5: 0.24871304631233215 (0.24623059972803643) | > loss_0: 2.55950927734375 (2.5328784242589424) | > grad_norm_0: tensor(396.2126, device='cuda:0') (tensor(252.7640, device='cuda:0')) | > loss_gen: 2.3588621616363525 (2.2413369467917903) | > loss_kl: 1.9830325841903687 (1.9710738684268707) | > loss_feat: 4.256811618804932 (4.100728851683598) | > loss_mel: 16.819366455078125 (16.88956303292132) | > loss_duration: 1.5547881126403809 (1.5184207292313272) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.972862243652344 (26.721123350427504) | > grad_norm_1: tensor(2662.2327, device='cuda:0') (tensor(6352.3467, device='cuda:0')) | > current_lr_0: 0.00017179602394221723 | > current_lr_1: 0.00017179602394221723 | > step_time: 0.447 (0.44127953306157536) | > loader_time: 0.0091 (0.007516353688341506) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652676998729437 (+5.908079550299809e-05) | > avg_step_time: 0.4545002520923883 (+0.0042181250075219445) | > avg_loss_disc: 2.5226307385404345 (-0.10966886265177145) | > avg_loss_disc_real_0: 0.07499132963868094 (-0.04002224866458229) | > avg_loss_disc_real_1: 0.22180378583955093 (-0.008446913160068875) | > avg_loss_disc_real_2: 0.2307039434221429 (-0.001955334779242396) | > avg_loss_disc_real_3: 0.23499514708216762 (-0.00871928434976385) | > avg_loss_disc_real_4: 0.23984057403786083 (-0.015363404448603207) | > avg_loss_disc_real_5: 0.2461253906219778 (-0.005058785979176916) | > avg_loss_0: 2.5226307385404345 (-0.10966886265177145) | > avg_grad_norm_0: tensor(238.2679, device='cuda:0') (tensor(-34.2046, device='cuda:0')) | > avg_loss_gen: 2.24114324509258 (+0.11131499183010041) | > avg_loss_kl: 2.0160715210605673 (-0.054935143027508015) | > avg_loss_feat: 4.038600055264755 (+0.3066211351206607) | > avg_loss_mel: 16.76597260757232 (-0.12752060151435884) | > avg_loss_duration: 1.5400858059735365 (-0.006331432033592543) | > avg_amp_scaler: 8.0 (-2.253521126760564) | > avg_loss_1: 26.601873156050566 (+0.22914897220235986) | > avg_grad_norm_1: tensor(6412.3848, device='cuda:0') (tensor(-588.9263, device='cuda:0'))  > EPOCH: 1217/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:44:49)   --> TIME: 2025-02-25 20:45:02 -- STEP: 25/72 -- GLOBAL_STEP: 1087650 | > loss_disc: 2.599870204925537 (2.4680335235595705) | > loss_disc_real_0: 0.035181887447834015 (0.06740753591060639) | > loss_disc_real_1: 0.19234178960323334 (0.21669421076774598) | > loss_disc_real_2: 0.21903172135353088 (0.22170980155467987) | > loss_disc_real_3: 0.21244727075099945 (0.23336212158203126) | > loss_disc_real_4: 0.22251297533512115 (0.24145858585834504) | > loss_disc_real_5: 0.2283230721950531 (0.24264336109161377) | > loss_0: 2.599870204925537 (2.4680335235595705) | > grad_norm_0: tensor(204.1734, device='cuda:0') (tensor(224.3336, device='cuda:0')) | > loss_gen: 2.004035711288452 (2.2819482994079596) | > loss_kl: 1.9643712043762207 (1.9724762439727783) | > loss_feat: 4.090509414672852 (4.21797700881958) | > loss_mel: 16.566883087158203 (16.7340645980835) | > loss_duration: 1.5039396286010742 (1.4895207166671753) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.129737854003906 (26.69598731994629) | > grad_norm_1: tensor(9859.6152, device='cuda:0') (tensor(6535.1123, device='cuda:0')) | > current_lr_0: 0.00017177454943922445 | > current_lr_1: 0.00017177454943922445 | > step_time: 0.4302 (0.43349233627319333) | > loader_time: 0.0073 (0.006598138809204101) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655403701352404 (+2.726702622966523e-06) | > avg_step_time: 0.4535556208919471 (-0.0009446312004411928) | > avg_loss_disc: 2.484083897630933 (-0.038546840909501334) | > avg_loss_disc_real_0: 0.07371588439588814 (-0.0012754452427927998) | > avg_loss_disc_real_1: 0.22035280961385914 (-0.0014509762256917913) | > avg_loss_disc_real_2: 0.2261830830657986 (-0.004520860356344303) | > avg_loss_disc_real_3: 0.2320414810953006 (-0.0029536659868670267) | > avg_loss_disc_real_4: 0.2396924075946002 (-0.00014816644326062778) | > avg_loss_disc_real_5: 0.24417185006846845 (-0.001953540553509353) | > avg_loss_0: 2.484083897630933 (-0.038546840909501334) | > avg_grad_norm_0: tensor(214.5013, device='cuda:0') (tensor(-23.7667, device='cuda:0')) | > avg_loss_gen: 2.271583548733886 (+0.0304403036413059) | > avg_loss_kl: 2.027378218274721 (+0.011306697214153782) | > avg_loss_feat: 4.111700437438318 (+0.07310038217356318) | > avg_loss_mel: 16.575852864225144 (-0.19011974334717507) | > avg_loss_duration: 1.536695186520966 (-0.0033906194525705757) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.523210525512695 (-0.07866263053787037) | > avg_grad_norm_1: tensor(6378.3408, device='cuda:0') (tensor(-34.0439, device='cuda:0'))  > EPOCH: 1218/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:45:25)   --> TIME: 2025-02-25 20:45:29 -- STEP: 3/72 -- GLOBAL_STEP: 1087700 | > loss_disc: 2.4160497188568115 (2.475341240564982) | > loss_disc_real_0: 0.0869898721575737 (0.0784165511528651) | > loss_disc_real_1: 0.2304016649723053 (0.2237092653910319) | > loss_disc_real_2: 0.18685203790664673 (0.20236272116502127) | > loss_disc_real_3: 0.23026612401008606 (0.22707859178384146) | > loss_disc_real_4: 0.23593057692050934 (0.2449991007645925) | > loss_disc_real_5: 0.23789595067501068 (0.24489751954873404) | > loss_0: 2.4160497188568115 (2.475341240564982) | > grad_norm_0: tensor(233.3092, device='cuda:0') (tensor(251.0732, device='cuda:0')) | > loss_gen: 2.5024850368499756 (2.323302427927653) | > loss_kl: 2.0232112407684326 (1.8125543196996052) | > loss_feat: 4.779818058013916 (4.312832673390706) | > loss_mel: 17.66839027404785 (17.17415936787923) | > loss_duration: 1.5236631631851196 (1.503311276435852) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.497568130493164 (27.12616030375163) | > grad_norm_1: tensor(3677.1064, device='cuda:0') (tensor(6492.6904, device='cuda:0')) | > current_lr_0: 0.00017175307762054453 | > current_lr_1: 0.00017175307762054453 | > step_time: 0.425 (0.42865745226542157) | > loader_time: 0.0057 (0.005512158075968425)  --> TIME: 2025-02-25 20:45:51 -- STEP: 53/72 -- GLOBAL_STEP: 1087750 | > loss_disc: 2.5817971229553223 (2.6760956566288785) | > loss_disc_real_0: 0.24379834532737732 (0.1604806197420606) | > loss_disc_real_1: 0.09661398082971573 (0.24626960696757966) | > loss_disc_real_2: 0.23629909753799438 (0.2256320023311759) | > loss_disc_real_3: 0.21986164152622223 (0.23262064074570277) | > loss_disc_real_4: 0.21552754938602448 (0.24111301662786952) | > loss_disc_real_5: 0.2353539764881134 (0.2485805094804404) | > loss_0: 2.5817971229553223 (2.6760956566288785) | > grad_norm_0: tensor(117.5080, device='cuda:0') (tensor(245.8461, device='cuda:0')) | > loss_gen: 2.486036539077759 (2.1577538904154077) | > loss_kl: 2.2235920429229736 (2.0360873352806523) | > loss_feat: 4.806999206542969 (3.7239496348039154) | > loss_mel: 17.912391662597656 (16.868047084448477) | > loss_duration: 1.602168083190918 (1.5358601133778411) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.031185150146484 (26.3216978468985) | > grad_norm_1: tensor(2500.0081, device='cuda:0') (tensor(5815.6641, device='cuda:0')) | > current_lr_0: 0.00017175307762054453 | > current_lr_1: 0.00017175307762054453 | > step_time: 0.4478 (0.44064105231806916) | > loader_time: 0.0098 (0.008007337462227301) --> EVAL PERFORMANCE | > avg_loader_time: 0.008918815935161752 (+0.00026341223380934795) | > avg_step_time: 0.4598452709090542 (+0.0062896500171070735) | > avg_loss_disc: 2.696765986966415 (+0.21268208933548172) | > avg_loss_disc_real_0: 0.14461845259221506 (+0.07090256819632693) | > avg_loss_disc_real_1: 0.26687056513529434 (+0.046517755521435195) | > avg_loss_disc_real_2: 0.23349647475800045 (+0.007313391692201854) | > avg_loss_disc_real_3: 0.2321852449799927 (+0.00014376388469211632) | > avg_loss_disc_real_4: 0.24660870726679412 (+0.006916299672193926) | > avg_loss_disc_real_5: 0.24568425517686657 (+0.0015124051083981183) | > avg_loss_0: 2.696765986966415 (+0.21268208933548172) | > avg_grad_norm_0: tensor(245.1457, device='cuda:0') (+tensor(30.6444, device='cuda:0')) | > avg_loss_gen: 2.1376041593685966 (-0.13397938936528941) | > avg_loss_kl: 2.0823416491629363 (+0.0549634308882152) | > avg_loss_feat: 3.7077079658776944 (-0.403992471560624) | > avg_loss_mel: 16.951716194690114 (+0.3758633304649699) | > avg_loss_duration: 1.5494579966639128 (+0.01276281014294689) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42882784991198 (-0.09438267560071623) | > avg_grad_norm_1: tensor(6156.8154, device='cuda:0') (tensor(-221.5254, device='cuda:0'))  > EPOCH: 1219/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:46:03)   --> TIME: 2025-02-25 20:46:19 -- STEP: 31/72 -- GLOBAL_STEP: 1087800 | > loss_disc: 2.583542823791504 (2.7086595258405133) | > loss_disc_real_0: 0.17442068457603455 (0.21458700503553113) | > loss_disc_real_1: 0.18687279522418976 (0.2304115160819023) | > loss_disc_real_2: 0.21601535379886627 (0.21344294374988926) | > loss_disc_real_3: 0.253659188747406 (0.22838250090998988) | > loss_disc_real_4: 0.24199937283992767 (0.24279922295001247) | > loss_disc_real_5: 0.2509835660457611 (0.25315080631163805) | > loss_0: 2.583542823791504 (2.7086595258405133) | > grad_norm_0: tensor(111.8047, device='cuda:0') (tensor(249.9108, device='cuda:0')) | > loss_gen: 2.23937726020813 (2.1439258360093647) | > loss_kl: 2.0400617122650146 (1.9937505375954412) | > loss_feat: 3.9746804237365723 (3.615945977549399) | > loss_mel: 17.730852127075195 (17.14796958431121) | > loss_duration: 1.5132267475128174 (1.4960588908964587) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.498199462890625 (26.39765074945265) | > grad_norm_1: tensor(9302.9922, device='cuda:0') (tensor(6555.7144, device='cuda:0')) | > current_lr_0: 0.00017173160848584195 | > current_lr_1: 0.00017173160848584195 | > step_time: 0.4519 (0.4376238469154604) | > loader_time: 0.0087 (0.007200010361209992) --> EVAL PERFORMANCE | > avg_loader_time: 0.008610456762179522 (-0.00030835917298223034) | > avg_step_time: 0.45328082836849587 (-0.006564442540558324) | > avg_loss_disc: 2.716485191398943 (+0.01971920443252806) | > avg_loss_disc_real_0: 0.1761770745031011 (+0.03155862191088604) | > avg_loss_disc_real_1: 0.22795153692574568 (-0.03891902820954865) | > avg_loss_disc_real_2: 0.21353543539282302 (-0.01996103936517743) | > avg_loss_disc_real_3: 0.23444698939860706 (+0.0022617444186143476) | > avg_loss_disc_real_4: 0.2453883907324831 (-0.0012203165343110112) | > avg_loss_disc_real_5: 0.25015079975128174 (+0.004466544574415171) | > avg_loss_0: 2.716485191398943 (+0.01971920443252806) | > avg_grad_norm_0: tensor(218.3493, device='cuda:0') (tensor(-26.7964, device='cuda:0')) | > avg_loss_gen: 2.058054987813385 (-0.07954917155521146) | > avg_loss_kl: 2.0820736918650895 (-0.0002679572978467881) | > avg_loss_feat: 3.6592818146020596 (-0.048426151275634766) | > avg_loss_mel: 17.059920002037373 (+0.10820380734725887) | > avg_loss_duration: 1.5391023108656978 (-0.010355685798214997) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.39843269133232 (-0.030395158579658244) | > avg_grad_norm_1: tensor(6248.4385, device='cuda:0') (+tensor(91.6230, device='cuda:0'))  > EPOCH: 1220/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:46:39)   --> TIME: 2025-02-25 20:46:45 -- STEP: 9/72 -- GLOBAL_STEP: 1087850 | > loss_disc: 2.7901370525360107 (2.677558289633857) | > loss_disc_real_0: 0.03725416958332062 (0.09746450144383642) | > loss_disc_real_1: 0.24309685826301575 (0.22103355328241983) | > loss_disc_real_2: 0.3515866696834564 (0.31674245993296307) | > loss_disc_real_3: 0.23233817517757416 (0.23927100168334114) | > loss_disc_real_4: 0.26121383905410767 (0.24512873921129438) | > loss_disc_real_5: 0.26348692178726196 (0.24168967538409764) | > loss_0: 2.7901370525360107 (2.677558289633857) | > grad_norm_0: tensor(415.9812, device='cuda:0') (tensor(331.4780, device='cuda:0')) | > loss_gen: 1.81459379196167 (2.13910616768731) | > loss_kl: 2.856549024581909 (2.357046683629354) | > loss_feat: 3.5603487491607666 (3.806299024158054) | > loss_mel: 20.758970260620117 (18.533708996242947) | > loss_duration: 1.7933850288391113 (1.690678013695611) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.78384780883789 (28.526839362250435) | > grad_norm_1: tensor(12050.9775, device='cuda:0') (tensor(6040.5122, device='cuda:0')) | > current_lr_0: 0.00017171014203478123 | > current_lr_1: 0.00017171014203478123 | > step_time: 0.4204 (0.42556677924262154) | > loader_time: 0.0057 (0.006482680638631185)  --> TIME: 2025-02-25 20:47:08 -- STEP: 59/72 -- GLOBAL_STEP: 1087900 | > loss_disc: 2.625046968460083 (2.653608798980713) | > loss_disc_real_0: 0.24953380227088928 (0.12905422819115347) | > loss_disc_real_1: 0.23634840548038483 (0.22848182108442663) | > loss_disc_real_2: 0.24536260962486267 (0.24625498359486206) | > loss_disc_real_3: 0.21743255853652954 (0.23531424191038489) | > loss_disc_real_4: 0.2319055199623108 (0.24156762028144582) | > loss_disc_real_5: 0.23385553061962128 (0.24276520993749975) | > loss_0: 2.625046968460083 (2.653608798980713) | > grad_norm_0: tensor(308.9564, device='cuda:0') (tensor(313.0020, device='cuda:0')) | > loss_gen: 1.966180682182312 (2.0397976737911434) | > loss_kl: 2.099709987640381 (2.2789443630283164) | > loss_feat: 3.405749559402466 (3.6310721858073087) | > loss_mel: 16.34085464477539 (17.866054243960615) | > loss_duration: 1.561959981918335 (1.5943997310379805) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.374454498291016 (27.410268363306077) | > grad_norm_1: tensor(7375.3525, device='cuda:0') (tensor(8607.1367, device='cuda:0')) | > current_lr_0: 0.00017171014203478123 | > current_lr_1: 0.00017171014203478123 | > step_time: 0.4611 (0.44068127971584514) | > loader_time: 0.0098 (0.007921251200013241) --> EVAL PERFORMANCE | > avg_loader_time: 0.008678970202593736 (+6.851344041421473e-05) | > avg_step_time: 0.44895392068674866 (-0.004326907681747205) | > avg_loss_disc: 2.645587041344441 (-0.07089815005450184) | > avg_loss_disc_real_0: 0.1281417221040793 (-0.04803535239902182) | > avg_loss_disc_real_1: 0.23613128097544253 (+0.008179744049696847) | > avg_loss_disc_real_2: 0.24255848266709018 (+0.02902304727426716) | > avg_loss_disc_real_3: 0.2331840158768103 (-0.0012629735217967453) | > avg_loss_disc_real_4: 0.24051375309346426 (-0.004874637639018853) | > avg_loss_disc_real_5: 0.2427114103461655 (-0.007439389405116237) | > avg_loss_0: 2.645587041344441 (-0.07089815005450184) | > avg_grad_norm_0: tensor(303.7227, device='cuda:0') (+tensor(85.3734, device='cuda:0')) | > avg_loss_gen: 2.0574041034134356 (-0.0006508843999495362) | > avg_loss_kl: 2.2643033450757954 (+0.18222965321070594) | > avg_loss_feat: 3.6483848699381656 (-0.010896944663894015) | > avg_loss_mel: 17.699195069326475 (+0.6392750672891019) | > avg_loss_duration: 1.595684630770079 (+0.05658231990438112) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.264972149486272 (+0.8665394581539516) | > avg_grad_norm_1: tensor(8020.4058, device='cuda:0') (+tensor(1771.9673, device='cuda:0'))  > EPOCH: 1221/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:47:16)   --> TIME: 2025-02-25 20:47:34 -- STEP: 37/72 -- GLOBAL_STEP: 1087950 | > loss_disc: 2.729487419128418 (2.6379719424892114) | > loss_disc_real_0: 0.1113336831331253 (0.12112179378399977) | > loss_disc_real_1: 0.23975780606269836 (0.22922734916210175) | > loss_disc_real_2: 0.2416251003742218 (0.23353974521160126) | > loss_disc_real_3: 0.23398180305957794 (0.2352425637277397) | > loss_disc_real_4: 0.287117063999176 (0.23744534801792455) | > loss_disc_real_5: 0.2352086752653122 (0.2436570009669742) | > loss_0: 2.729487419128418 (2.6379719424892114) | > grad_norm_0: tensor(337.3004, device='cuda:0') (tensor(212.3242, device='cuda:0')) | > loss_gen: 1.9501657485961914 (2.1549225948952335) | > loss_kl: 2.4333252906799316 (2.8862857109791515) | > loss_feat: 3.5648295879364014 (4.062444396921106) | > loss_mel: 18.759126663208008 (20.84255935050346) | > loss_duration: 1.6076589822769165 (1.5958557257781159) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.315105438232422 (31.542068017495644) | > grad_norm_1: tensor(11445.8008, device='cuda:0') (tensor(7913.9131, device='cuda:0')) | > current_lr_0: 0.00017168867826702688 | > current_lr_1: 0.00017168867826702688 | > step_time: 0.4436 (0.430574977720106) | > loader_time: 0.008 (0.007122007576194969) --> EVAL PERFORMANCE | > avg_loader_time: 0.008535724290659732 (-0.00014324591193400418) | > avg_step_time: 0.4489357605786391 (-1.8160108109566142e-05) | > avg_loss_disc: 2.6407110724650638 (-0.004875968879377357) | > avg_loss_disc_real_0: 0.11136336821380635 (-0.016778353890272935) | > avg_loss_disc_real_1: 0.22336264353402904 (-0.012768637441413488) | > avg_loss_disc_real_2: 0.24758508423684347 (+0.005026601569753292) | > avg_loss_disc_real_3: 0.23778232617277495 (+0.004598310295964636) | > avg_loss_disc_real_4: 0.23883052285708173 (-0.0016832302363825324) | > avg_loss_disc_real_5: 0.24581947964681705 (+0.0031080693006515503) | > avg_loss_0: 2.6407110724650638 (-0.004875968879377357) | > avg_grad_norm_0: tensor(209.0897, device='cuda:0') (tensor(-94.6330, device='cuda:0')) | > avg_loss_gen: 2.1313486703684634 (+0.07394456695502782) | > avg_loss_kl: 2.6037677408943707 (+0.33946439581857524) | > avg_loss_feat: 3.9142661094665527 (+0.26588123952838716) | > avg_loss_mel: 19.440223290886678 (+1.741028221560203) | > avg_loss_duration: 1.5986998601698539 (+0.0030152293997749435) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.68830584136533 (+2.4233336918790584) | > avg_grad_norm_1: tensor(7172.3291, device='cuda:0') (tensor(-848.0767, device='cuda:0'))  > EPOCH: 1222/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:47:53)   --> TIME: 2025-02-25 20:48:01 -- STEP: 15/72 -- GLOBAL_STEP: 1088000 | > loss_disc: 2.5920886993408203 (2.519783592224121) | > loss_disc_real_0: 0.1681375950574875 (0.06296924861768881) | > loss_disc_real_1: 0.21291546523571014 (0.22992559870084125) | > loss_disc_real_2: 0.2256426066160202 (0.18756893376509348) | > loss_disc_real_3: 0.21321411430835724 (0.2467025727033615) | > loss_disc_real_4: 0.22852535545825958 (0.24608864386876425) | > loss_disc_real_5: 0.25028368830680847 (0.236297940214475) | > loss_0: 2.5920886993408203 (2.519783592224121) | > grad_norm_0: tensor(42.5284, device='cuda:0') (tensor(154.4468, device='cuda:0')) | > loss_gen: 2.4437255859375 (2.189121786753336) | > loss_kl: 2.019867420196533 (1.9784699122111002) | > loss_feat: 3.7848124504089355 (3.9628345648447674) | > loss_mel: 17.587200164794922 (17.100245412190755) | > loss_duration: 1.4935340881347656 (1.4988949616750082) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.329139709472656 (26.729566701253255) | > grad_norm_1: tensor(1372.4038, device='cuda:0') (tensor(5185.2178, device='cuda:0')) | > current_lr_0: 0.00017166721718224348 | > current_lr_1: 0.00017166721718224348 | > step_time: 0.4263 (0.43773632049560546) | > loader_time: 0.0058 (0.006480153401692708)  --> TIME: 2025-02-25 20:48:25 -- STEP: 65/72 -- GLOBAL_STEP: 1088050 | > loss_disc: 2.613988161087036 (2.638352878277119) | > loss_disc_real_0: 0.2196817696094513 (0.1620570830141123) | > loss_disc_real_1: 0.1776091754436493 (0.21823703864446053) | > loss_disc_real_2: 0.24084095656871796 (0.22700087542717273) | > loss_disc_real_3: 0.21974419057369232 (0.23577218307898595) | > loss_disc_real_4: 0.2382570058107376 (0.2453684843503512) | > loss_disc_real_5: 0.23663166165351868 (0.2430079636665491) | > loss_0: 2.613988161087036 (2.638352878277119) | > grad_norm_0: tensor(339.6704, device='cuda:0') (tensor(223.7873, device='cuda:0')) | > loss_gen: 2.248401641845703 (2.142371683854323) | > loss_kl: 2.1980719566345215 (2.048940814458407) | > loss_feat: 4.137730598449707 (3.6351311133458064) | > loss_mel: 17.37479019165039 (17.050686205350434) | > loss_duration: 1.597506046295166 (1.5416141179891734) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.556499481201172 (26.418743896484376) | > grad_norm_1: tensor(4125.4819, device='cuda:0') (tensor(5082.1392, device='cuda:0')) | > current_lr_0: 0.00017166721718224348 | > current_lr_1: 0.00017166721718224348 | > step_time: 0.5149 (0.44780744039095366) | > loader_time: 0.0212 (0.00855384239783654) --> EVAL PERFORMANCE | > avg_loader_time: 0.008879053760582295 (+0.0003433294699225625) | > avg_step_time: 0.4545021627990293 (+0.005566402220390176) | > avg_loss_disc: 2.63295728387967 (-0.007753788585393551) | > avg_loss_disc_real_0: 0.15865393116755389 (+0.047290562953747534) | > avg_loss_disc_real_1: 0.21895238221950933 (-0.0044102613145197145) | > avg_loss_disc_real_2: 0.22634655342135632 (-0.021238530815487155) | > avg_loss_disc_real_3: 0.23452530801296234 (-0.003257018159812608) | > avg_loss_disc_real_4: 0.24509421641557988 (+0.006263693558498151) | > avg_loss_disc_real_5: 0.24310986592736042 (-0.0027096137194566328) | > avg_loss_0: 2.63295728387967 (-0.007753788585393551) | > avg_grad_norm_0: tensor(219.5966, device='cuda:0') (+tensor(10.5069, device='cuda:0')) | > avg_loss_gen: 2.135720818815097 (+0.004372148446633695) | > avg_loss_kl: 2.0556069243122153 (-0.5481608165821554) | > avg_loss_feat: 3.6356466320199026 (-0.27861947744665017) | > avg_loss_mel: 17.032144358460332 (-2.4080789324263456) | > avg_loss_duration: 1.5472587514931047 (-0.051441108676749225) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.406377416261485 (-3.281928425103846) | > avg_grad_norm_1: tensor(5041.3628, device='cuda:0') (tensor(-2130.9663, device='cuda:0'))  > EPOCH: 1223/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:48:30)   --> TIME: 2025-02-25 20:48:51 -- STEP: 43/72 -- GLOBAL_STEP: 1088100 | > loss_disc: 2.56363582611084 (2.5846985495367716) | > loss_disc_real_0: 0.06102187559008598 (0.13118773239643075) | > loss_disc_real_1: 0.15044434368610382 (0.22904558653055235) | > loss_disc_real_2: 0.2104058414697647 (0.23265611466973327) | > loss_disc_real_3: 0.23405548930168152 (0.2344510267640269) | > loss_disc_real_4: 0.221506267786026 (0.2422866803962131) | > loss_disc_real_5: 0.2339889109134674 (0.25056803642317305) | > loss_0: 2.56363582611084 (2.5846985495367716) | > grad_norm_0: tensor(132.5710, device='cuda:0') (tensor(240.1273, device='cuda:0')) | > loss_gen: 1.9909350872039795 (2.2319374333980466) | > loss_kl: 2.1560306549072266 (2.334345265876415) | > loss_feat: 3.7729551792144775 (3.9545469062272893) | > loss_mel: 17.168415069580078 (18.210572397986127) | > loss_duration: 1.5559406280517578 (1.5673343996668971) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.644275665283203 (28.298736394837846) | > grad_norm_1: tensor(7948.0024, device='cuda:0') (tensor(5288.1016, device='cuda:0')) | > current_lr_0: 0.0001716457587800957 | > current_lr_1: 0.0001716457587800957 | > step_time: 0.4699 (0.43457169865452966) | > loader_time: 0.0094 (0.007499312245568564) --> EVAL PERFORMANCE | > avg_loader_time: 0.008740535924132438 (-0.00013851783644985675) | > avg_step_time: 0.44874429366957974 (-0.005757869129449533) | > avg_loss_disc: 2.6312058039114508 (-0.0017514799682194493) | > avg_loss_disc_real_0: 0.1471904924427959 (-0.011463438724757996) | > avg_loss_disc_real_1: 0.238481128719491 (+0.01952874649998168) | > avg_loss_disc_real_2: 0.23376390862632807 (+0.007417355204971748) | > avg_loss_disc_real_3: 0.23621495550786945 (+0.0016896474949071039) | > avg_loss_disc_real_4: 0.24652897598038256 (+0.0014347595648026845) | > avg_loss_disc_real_5: 0.2507017349273386 (+0.007591868999978185) | > avg_loss_0: 2.6312058039114508 (-0.0017514799682194493) | > avg_grad_norm_0: tensor(258.2065, device='cuda:0') (+tensor(38.6099, device='cuda:0')) | > avg_loss_gen: 2.220333841485037 (+0.08461302266993975) | > avg_loss_kl: 2.391177610612252 (+0.33557068630003695) | > avg_loss_feat: 3.8907827525071697 (+0.2551361204872671) | > avg_loss_mel: 18.79314613342286 (+1.7610017749625264) | > avg_loss_duration: 1.6012994621841001 (+0.054040710690995475) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.89673982539647 (+2.4903624091349847) | > avg_grad_norm_1: tensor(5772.0913, device='cuda:0') (+tensor(730.7285, device='cuda:0'))  > EPOCH: 1224/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:49:06)   --> TIME: 2025-02-25 20:49:17 -- STEP: 21/72 -- GLOBAL_STEP: 1088150 | > loss_disc: 2.8167927265167236 (2.59156828834897) | > loss_disc_real_0: 0.04879756271839142 (0.1180362936464094) | > loss_disc_real_1: 0.294368714094162 (0.22791649472145808) | > loss_disc_real_2: 0.21997593343257904 (0.2307687486921038) | > loss_disc_real_3: 0.2537142038345337 (0.23237583821728117) | > loss_disc_real_4: 0.24199877679347992 (0.24419743106478736) | > loss_disc_real_5: 0.25410816073417664 (0.24613137259369805) | > loss_0: 2.8167927265167236 (2.59156828834897) | > grad_norm_0: tensor(452.9415, device='cuda:0') (tensor(222.2212, device='cuda:0')) | > loss_gen: 1.7083238363265991 (2.1535064095542547) | > loss_kl: 1.919328212738037 (2.043817945889064) | > loss_feat: 3.291923999786377 (3.9569269816080728) | > loss_mel: 16.71274185180664 (17.666107631865003) | > loss_duration: 1.5213342905044556 (1.52250041280474) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.15365219116211 (27.342859359014604) | > grad_norm_1: tensor(6216.8013, device='cuda:0') (tensor(6736.0767, device='cuda:0')) | > current_lr_0: 0.00017162430306024817 | > current_lr_1: 0.00017162430306024817 | > step_time: 0.4294 (0.4267348902566092) | > loader_time: 0.0064 (0.006508600144159226)  --> TIME: 2025-02-25 20:49:41 -- STEP: 71/72 -- GLOBAL_STEP: 1088200 | > loss_disc: 2.4376049041748047 (2.6508502993785155) | > loss_disc_real_0: 0.14752395451068878 (0.13985517644651335) | > loss_disc_real_1: 0.22178323566913605 (0.2220999652021368) | > loss_disc_real_2: 0.21082668006420135 (0.22739567009496017) | > loss_disc_real_3: 0.20656560361385345 (0.23264416212766942) | > loss_disc_real_4: 0.24914099276065826 (0.24624009199545416) | > loss_disc_real_5: 0.2618643641471863 (0.24396686234944304) | > loss_0: 2.4376049041748047 (2.6508502993785155) | > grad_norm_0: tensor(184.9693, device='cuda:0') (tensor(248.1306, device='cuda:0')) | > loss_gen: 2.4724836349487305 (2.0824903132210317) | > loss_kl: 2.331815242767334 (2.3859814405441284) | > loss_feat: 4.231659889221191 (3.7791149414760965) | > loss_mel: 18.117677688598633 (18.90716000677834) | > loss_duration: 1.599965214729309 (1.601036854193244) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.753602981567383 (28.75578340342347) | > grad_norm_1: tensor(2955.5886, device='cuda:0') (tensor(7396.2412, device='cuda:0')) | > current_lr_0: 0.00017162430306024817 | > current_lr_1: 0.00017162430306024817 | > step_time: 0.4988 (0.45082542929850833) | > loader_time: 0.0079 (0.008751933003815123) --> EVAL PERFORMANCE | > avg_loader_time: 0.008751933003815123 (+1.1397079682685593e-05) | > avg_step_time: 0.45082542929850833 (+0.0020811356289285876) | > avg_loss_disc: 2.6508502993785155 (+0.019644495467064704) | > avg_loss_disc_real_0: 0.13985517644651335 (-0.007335315996282538) | > avg_loss_disc_real_1: 0.2220999652021368 (-0.01638116351735422) | > avg_loss_disc_real_2: 0.22739567009496017 (-0.006368238531367892) | > avg_loss_disc_real_3: 0.23264416212766942 (-0.003570793380200027) | > avg_loss_disc_real_4: 0.24624009199545416 (-0.0002888839849284064) | > avg_loss_disc_real_5: 0.24396686234944304 (-0.006734872577895568) | > avg_loss_0: 2.6508502993785155 (+0.019644495467064704) | > avg_grad_norm_0: tensor(248.1306, device='cuda:0') (tensor(-10.0759, device='cuda:0')) | > avg_loss_gen: 2.0824903132210317 (-0.13784352826400514) | > avg_loss_kl: 2.3859814405441284 (-0.005196170068123784) | > avg_loss_feat: 3.7791149414760965 (-0.11166781103107315) | > avg_loss_mel: 18.90716000677834 (+0.11401387335548208) | > avg_loss_duration: 1.601036854193244 (-0.00026260799085608966) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.75578340342347 (-0.14095642197299796) | > avg_grad_norm_1: tensor(7396.2412, device='cuda:0') (+tensor(1624.1499, device='cuda:0'))  > EPOCH: 1225/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:49:43)   --> TIME: 2025-02-25 20:50:08 -- STEP: 49/72 -- GLOBAL_STEP: 1088250 | > loss_disc: 2.6007590293884277 (2.625671664062812) | > loss_disc_real_0: 0.1732177585363388 (0.1409322183047022) | > loss_disc_real_1: 0.2533794939517975 (0.2261129347037296) | > loss_disc_real_2: 0.26704955101013184 (0.2599710971117019) | > loss_disc_real_3: 0.25689828395843506 (0.22916109829532857) | > loss_disc_real_4: 0.2632749080657959 (0.24199221572097468) | > loss_disc_real_5: 0.2700347304344177 (0.24500784065042222) | > loss_0: 2.6007590293884277 (2.625671664062812) | > grad_norm_0: tensor(245.9427, device='cuda:0') (tensor(192.9175, device='cuda:0')) | > loss_gen: 2.3105077743530273 (2.2108860964677772) | > loss_kl: 2.1205532550811768 (2.1058140968789854) | > loss_feat: 3.5983550548553467 (3.992298870670553) | > loss_mel: 17.170576095581055 (17.404500610974367) | > loss_duration: 1.6045562028884888 (1.5421268891315072) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.804550170898438 (27.255626600615834) | > grad_norm_1: tensor(10663.6621, device='cuda:0') (tensor(5725.0664, device='cuda:0')) | > current_lr_0: 0.00017160285002236562 | > current_lr_1: 0.00017160285002236562 | > step_time: 0.4512 (0.44427238678445624) | > loader_time: 0.0099 (0.007633763916638433) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613841634401133 (-0.00013809136941399063) | > avg_step_time: 0.45902217609781615 (+0.008196746799307819) | > avg_loss_disc: 2.6232295103476084 (-0.027620789030907034) | > avg_loss_disc_real_0: 0.13253978031202102 (-0.007315396134492336) | > avg_loss_disc_real_1: 0.22599558993964128 (+0.0038956247375044883) | > avg_loss_disc_real_2: 0.24821547261426136 (+0.020819802519301184) | > avg_loss_disc_real_3: 0.23239189407355348 (-0.0002522680541159339) | > avg_loss_disc_real_4: 0.24531137313641294 (-0.0009287188590412221) | > avg_loss_disc_real_5: 0.24817082327856144 (+0.0042039609291184) | > avg_loss_0: 2.6232295103476084 (-0.027620789030907034) | > avg_grad_norm_0: tensor(211.6028, device='cuda:0') (tensor(-36.5278, device='cuda:0')) | > avg_loss_gen: 2.16924866824083 (+0.08675835501979812) | > avg_loss_kl: 2.1138825668415544 (-0.272098873702574) | > avg_loss_feat: 3.8972385399778124 (+0.11812359850171594) | > avg_loss_mel: 17.252293747915342 (-1.6548662588629988) | > avg_loss_duration: 1.5574305074315677 (-0.043606346761676296) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.990094117715326 (-1.7656892857081452) | > avg_grad_norm_1: tensor(5878.2852, device='cuda:0') (tensor(-1517.9561, device='cuda:0'))  > EPOCH: 1226/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:50:21)   --> TIME: 2025-02-25 20:50:35 -- STEP: 27/72 -- GLOBAL_STEP: 1088300 | > loss_disc: 2.7768168449401855 (2.635648586131908) | > loss_disc_real_0: 0.47394421696662903 (0.19809438233022336) | > loss_disc_real_1: 0.25878193974494934 (0.21750995130450637) | > loss_disc_real_2: 0.23532263934612274 (0.22642219011430387) | > loss_disc_real_3: 0.23273752629756927 (0.23352624089629562) | > loss_disc_real_4: 0.23918519914150238 (0.2354039748509725) | > loss_disc_real_5: 0.24011467397212982 (0.24690601781562524) | > loss_0: 2.7768168449401855 (2.635648586131908) | > grad_norm_0: tensor(602.8055, device='cuda:0') (tensor(300.4396, device='cuda:0')) | > loss_gen: 2.695904493331909 (2.1531085923865985) | > loss_kl: 1.9523597955703735 (1.9840656121571858) | > loss_feat: 3.7100720405578613 (3.683600240283542) | > loss_mel: 17.97098731994629 (16.970296259279603) | > loss_duration: 1.4876055717468262 (1.5010750867702343) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.81692886352539 (26.292145552458585) | > grad_norm_1: tensor(1796.8402, device='cuda:0') (tensor(7493.9790, device='cuda:0')) | > current_lr_0: 0.0001715813996661128 | > current_lr_1: 0.0001715813996661128 | > step_time: 0.4401 (0.4351917196203161) | > loader_time: 0.0078 (0.006807371422096535) --> EVAL PERFORMANCE | > avg_loader_time: 0.008637975639020891 (+2.413400461975801e-05) | > avg_step_time: 0.45357459699603875 (-0.0054475791017774) | > avg_loss_disc: 2.66964289168237 (+0.04641338133476136) | > avg_loss_disc_real_0: 0.1962862016869263 (+0.06374642137490527) | > avg_loss_disc_real_1: 0.23030619438685163 (+0.004310604447210348) | > avg_loss_disc_real_2: 0.2303341719466196 (-0.017881300667641764) | > avg_loss_disc_real_3: 0.2359732292907339 (+0.0035813352171804036) | > avg_loss_disc_real_4: 0.24102958659051169 (-0.0042817865459012505) | > avg_loss_disc_real_5: 0.24933015922425497 (+0.0011593359456935348) | > avg_loss_0: 2.66964289168237 (+0.04641338133476136) | > avg_grad_norm_0: tensor(310.3767, device='cuda:0') (+tensor(98.7739, device='cuda:0')) | > avg_loss_gen: 2.127827738372372 (-0.0414209298684578) | > avg_loss_kl: 2.2061720297370155 (+0.09228946289546114) | > avg_loss_feat: 3.623706572492358 (-0.2735319674854546) | > avg_loss_mel: 17.558063198143326 (+0.3057694502279844) | > avg_loss_duration: 1.556708384567583 (-0.0007221228639846533) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.07247779738735 (+0.08238367967202365) | > avg_grad_norm_1: tensor(7237.9463, device='cuda:0') (+tensor(1359.6611, device='cuda:0'))  > EPOCH: 1227/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:50:57)   --> TIME: 2025-02-25 20:51:01 -- STEP: 5/72 -- GLOBAL_STEP: 1088350 | > loss_disc: 2.6778059005737305 (2.6958361625671388) | > loss_disc_real_0: 0.07819805294275284 (0.2564111903309822) | > loss_disc_real_1: 0.3855811655521393 (0.24976031631231307) | > loss_disc_real_2: 0.23809804022312164 (0.22221622467041016) | > loss_disc_real_3: 0.2887830138206482 (0.2338025838136673) | > loss_disc_real_4: 0.23816969990730286 (0.2236390084028244) | > loss_disc_real_5: 0.2550983726978302 (0.24411862790584565) | > loss_0: 2.6778059005737305 (2.6958361625671388) | > grad_norm_0: tensor(176.3426, device='cuda:0') (tensor(371.3759, device='cuda:0')) | > loss_gen: 2.0839591026306152 (2.3125938415527343) | > loss_kl: 2.3188388347625732 (2.075503206253052) | > loss_feat: 3.4291529655456543 (3.7002338409423827) | > loss_mel: 16.483936309814453 (16.72039928436279) | > loss_duration: 1.5175398588180542 (1.5293322324752807) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.83342742919922 (26.33806266784668) | > grad_norm_1: tensor(9241.8340, device='cuda:0') (tensor(6981.0034, device='cuda:0')) | > current_lr_0: 0.00017155995199115453 | > current_lr_1: 0.00017155995199115453 | > step_time: 0.4251 (0.42345728874206545) | > loader_time: 0.0062 (0.006716108322143555)  --> TIME: 2025-02-25 20:51:24 -- STEP: 55/72 -- GLOBAL_STEP: 1088400 | > loss_disc: 2.6500203609466553 (2.6842590071938255) | > loss_disc_real_0: 0.153641939163208 (0.18254651203751565) | > loss_disc_real_1: 0.22871096432209015 (0.2481809248978441) | > loss_disc_real_2: 0.20134484767913818 (0.23208398493853483) | > loss_disc_real_3: 0.23013408482074738 (0.23893683363090862) | > loss_disc_real_4: 0.2300138622522354 (0.23996596607294948) | > loss_disc_real_5: 0.24463291466236115 (0.23899208876219663) | > loss_0: 2.6500203609466553 (2.6842590071938255) | > grad_norm_0: tensor(75.6338, device='cuda:0') (tensor(226.7627, device='cuda:0')) | > loss_gen: 1.9694088697433472 (2.1321141503073955) | > loss_kl: 2.0225863456726074 (2.0535955385728304) | > loss_feat: 3.3443260192871094 (3.6255496241829612) | > loss_mel: 16.725706100463867 (16.859202090176677) | > loss_duration: 1.5923867225646973 (1.5324105111035435) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.6544132232666 (26.202871773459695) | > grad_norm_1: tensor(8695.9883, device='cuda:0') (tensor(5720.5366, device='cuda:0')) | > current_lr_0: 0.00017155995199115453 | > current_lr_1: 0.00017155995199115453 | > step_time: 0.4576 (0.4431601047515869) | > loader_time: 0.0098 (0.008297148617831141) --> EVAL PERFORMANCE | > avg_loader_time: 0.008960995875613791 (+0.00032302023659290004) | > avg_step_time: 0.4534675571280466 (-0.00010703986799215359) | > avg_loss_disc: 2.670993230712246 (+0.0013503390298761353) | > avg_loss_disc_real_0: 0.17252762432039623 (-0.02375857736653006) | > avg_loss_disc_real_1: 0.24184439016479842 (+0.011538195777946791) | > avg_loss_disc_real_2: 0.23691022102261933 (+0.006576049075999735) | > avg_loss_disc_real_3: 0.23845659784028228 (+0.0024833685495483926) | > avg_loss_disc_real_4: 0.23932742568808543 (-0.001702160902426253) | > avg_loss_disc_real_5: 0.24070313635846258 (-0.008627022865792394) | > avg_loss_0: 2.670993230712246 (+0.0013503390298761353) | > avg_grad_norm_0: tensor(226.2264, device='cuda:0') (tensor(-84.1503, device='cuda:0')) | > avg_loss_gen: 2.135320525773814 (+0.0074927874014418805) | > avg_loss_kl: 2.0673341717518543 (-0.13883785798516124) | > avg_loss_feat: 3.652114942040242 (+0.02840836954788406) | > avg_loss_mel: 16.879400723417046 (-0.6786624747262806) | > avg_loss_duration: 1.5450866255961675 (-0.011621758971415597) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.279256901270905 (-0.7932208961164449) | > avg_grad_norm_1: tensor(5610.3398, device='cuda:0') (tensor(-1627.6064, device='cuda:0'))  > EPOCH: 1228/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:51:35)   --> TIME: 2025-02-25 20:51:52 -- STEP: 33/72 -- GLOBAL_STEP: 1088450 | > loss_disc: 2.6759862899780273 (2.656332745696559) | > loss_disc_real_0: 0.17234137654304504 (0.2021022690177867) | > loss_disc_real_1: 0.22107809782028198 (0.22465503938270337) | > loss_disc_real_2: 0.19947202503681183 (0.227621082555164) | > loss_disc_real_3: 0.22375500202178955 (0.23016249411033862) | > loss_disc_real_4: 0.2519369423389435 (0.24962951784784143) | > loss_disc_real_5: 0.2525077164173126 (0.24942341175946323) | > loss_0: 2.6759862899780273 (2.656332745696559) | > grad_norm_0: tensor(6.0128, device='cuda:0') (tensor(173.2514, device='cuda:0')) | > loss_gen: 2.0669150352478027 (2.176103848399538) | > loss_kl: 2.61407208442688 (2.2763524958581636) | > loss_feat: 3.679443836212158 (3.761804046052875) | > loss_mel: 18.251689910888672 (17.596040956901785) | > loss_duration: 1.5985190868377686 (1.5513025630604138) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.21063995361328 (27.361603592381332) | > grad_norm_1: tensor(6628.7031, device='cuda:0') (tensor(5635.9839, device='cuda:0')) | > current_lr_0: 0.00017153850699715563 | > current_lr_1: 0.00017153850699715563 | > step_time: 0.4476 (0.44428524104031647) | > loader_time: 0.0083 (0.007471323013305664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008806880091277646 (-0.00015411578433614445) | > avg_step_time: 0.45740171553383413 (+0.003934158405787536) | > avg_loss_disc: 2.660227114046123 (-0.010766116666122816) | > avg_loss_disc_real_0: 0.186619692049186 (+0.014092067728789764) | > avg_loss_disc_real_1: 0.2231253241779099 (-0.018719065986888522) | > avg_loss_disc_real_2: 0.24191741288547786 (+0.005007191862858529) | > avg_loss_disc_real_3: 0.23171104595694744 (-0.006745551883334838) | > avg_loss_disc_real_4: 0.2441218188950713 (+0.004794393206985881) | > avg_loss_disc_real_5: 0.2435697380086066 (+0.0028666016501440184) | > avg_loss_0: 2.660227114046123 (-0.010766116666122816) | > avg_grad_norm_0: tensor(196.2654, device='cuda:0') (tensor(-29.9610, device='cuda:0')) | > avg_loss_gen: 2.129788380273631 (-0.0055321455001831055) | > avg_loss_kl: 2.272068240273166 (+0.20473406852131149) | > avg_loss_feat: 3.7081216657665412 (+0.056006723726299334) | > avg_loss_mel: 17.60134970973916 (+0.7219489863221149) | > avg_loss_duration: 1.5822321109368767 (+0.03714548534070916) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.29355992061991 (+1.0143030193490041) | > avg_grad_norm_1: tensor(5795.2900, device='cuda:0') (+tensor(184.9502, device='cuda:0'))  > EPOCH: 1229/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:52:12)   --> TIME: 2025-02-25 20:52:19 -- STEP: 11/72 -- GLOBAL_STEP: 1088500 | > loss_disc: 2.685779333114624 (2.6201918558640913) | > loss_disc_real_0: 0.13283458352088928 (0.13142370906743137) | > loss_disc_real_1: 0.22087988257408142 (0.20641040124676444) | > loss_disc_real_2: 0.24879032373428345 (0.21972203796560114) | > loss_disc_real_3: 0.20219941437244415 (0.2050893631848422) | > loss_disc_real_4: 0.22966323792934418 (0.23802347481250763) | > loss_disc_real_5: 0.24267925322055817 (0.2512778788805008) | > loss_0: 2.685779333114624 (2.6201918558640913) | > grad_norm_0: tensor(24.3009, device='cuda:0') (tensor(139.5945, device='cuda:0')) | > loss_gen: 1.8902804851531982 (2.0225515257228506) | > loss_kl: 1.9297019243240356 (1.942267428744923) | > loss_feat: 3.5663535594940186 (3.6880615841258657) | > loss_mel: 16.82884407043457 (17.03384954279119) | > loss_duration: 1.5020778179168701 (1.5123827132311733) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.717256546020508 (26.199112632057883) | > grad_norm_1: tensor(11747.3838, device='cuda:0') (tensor(7546.7139, device='cuda:0')) | > current_lr_0: 0.00017151706468378097 | > current_lr_1: 0.00017151706468378097 | > step_time: 0.4248 (0.43030030077153986) | > loader_time: 0.0058 (0.006012786518443714)  --> TIME: 2025-02-25 20:52:42 -- STEP: 61/72 -- GLOBAL_STEP: 1088550 | > loss_disc: 2.6877593994140625 (2.685717008152947) | > loss_disc_real_0: 0.21468210220336914 (0.13675882667303085) | > loss_disc_real_1: 0.2322770357131958 (0.2568172443597043) | > loss_disc_real_2: 0.26576268672943115 (0.2462955227152246) | > loss_disc_real_3: 0.2372664362192154 (0.23458077611981845) | > loss_disc_real_4: 0.23161959648132324 (0.24552036504276464) | > loss_disc_real_5: 0.24971584975719452 (0.24764403358834688) | > loss_0: 2.6877593994140625 (2.685717008152947) | > grad_norm_0: tensor(603.7493, device='cuda:0') (tensor(228.1754, device='cuda:0')) | > loss_gen: 2.025815486907959 (2.131899495593837) | > loss_kl: 2.3066632747650146 (2.1993799463647323) | > loss_feat: 3.2970855236053467 (3.742119695319504) | > loss_mel: 17.43897819519043 (17.790442576173884) | > loss_duration: 1.5932693481445312 (1.5576641618228348) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.66181182861328 (27.421505881137534) | > grad_norm_1: tensor(5597.5029, device='cuda:0') (tensor(6334.8008, device='cuda:0')) | > current_lr_0: 0.00017151706468378097 | > current_lr_1: 0.00017151706468378097 | > step_time: 0.4672 (0.44691591966347616) | > loader_time: 0.0109 (0.008143073222676264) --> EVAL PERFORMANCE | > avg_loader_time: 0.008750737552911465 (-5.614253836618184e-05) | > avg_step_time: 0.45431287523726344 (-0.003088840296570694) | > avg_loss_disc: 2.6713050620656626 (+0.01107794801953954) | > avg_loss_disc_real_0: 0.13255390314988685 (-0.05406578889929914) | > avg_loss_disc_real_1: 0.25129535479444853 (+0.02817003061653864) | > avg_loss_disc_real_2: 0.24606547955895813 (+0.004148066673480277) | > avg_loss_disc_real_3: 0.23238982564546692 (+0.0006787796885194819) | > avg_loss_disc_real_4: 0.24341314380437556 (-0.0007086750906957484) | > avg_loss_disc_real_5: 0.24674938031485383 (+0.0031796423062472323) | > avg_loss_0: 2.6713050620656626 (+0.01107794801953954) | > avg_grad_norm_0: tensor(245.9599, device='cuda:0') (+tensor(49.6946, device='cuda:0')) | > avg_loss_gen: 2.1097430695950146 (-0.020045310678616257) | > avg_loss_kl: 2.2047166690020505 (-0.06735157127111524) | > avg_loss_feat: 3.701554778596045 (-0.0065668871704960274) | > avg_loss_mel: 17.716703710421715 (+0.11535400068255441) | > avg_loss_duration: 1.5662030485314382 (-0.016029062405438443) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.29892128957829 (+0.005361368958379131) | > avg_grad_norm_1: tensor(6807.9785, device='cuda:0') (+tensor(1012.6885, device='cuda:0'))  > EPOCH: 1230/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:52:49)   --> TIME: 2025-02-25 20:53:08 -- STEP: 39/72 -- GLOBAL_STEP: 1088600 | > loss_disc: 2.5034048557281494 (2.5554256378076015) | > loss_disc_real_0: 0.10992402583360672 (0.12289447089036305) | > loss_disc_real_1: 0.22638553380966187 (0.221845948161223) | > loss_disc_real_2: 0.22274595499038696 (0.22551410702558664) | > loss_disc_real_3: 0.2215421497821808 (0.22978499035040537) | > loss_disc_real_4: 0.2537182867527008 (0.2393400481878183) | > loss_disc_real_5: 0.24817532300949097 (0.24341536905521002) | > loss_0: 2.5034048557281494 (2.5554256378076015) | > grad_norm_0: tensor(224.8552, device='cuda:0') (tensor(237.9056, device='cuda:0')) | > loss_gen: 2.057464838027954 (2.125290910402935) | > loss_kl: 2.1563899517059326 (1.9811031176493719) | > loss_feat: 3.6498045921325684 (3.739801571919368) | > loss_mel: 16.627227783203125 (16.94769027905586) | > loss_duration: 1.5443754196166992 (1.5154851858432477) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.035263061523438 (26.309370970114685) | > grad_norm_1: tensor(11890.4072, device='cuda:0') (tensor(7321.8945, device='cuda:0')) | > current_lr_0: 0.0001714956250506955 | > current_lr_1: 0.0001714956250506955 | > step_time: 0.4536 (0.4332921504974365) | > loader_time: 0.0081 (0.007183943039331681) --> EVAL PERFORMANCE | > avg_loader_time: 0.008652052409212354 (-9.868514369911104e-05) | > avg_step_time: 0.44830828317454163 (-0.006004592062721803) | > avg_loss_disc: 2.55912900306809 (-0.11217605899757244) | > avg_loss_disc_real_0: 0.11730749327951753 (-0.01524640987036932) | > avg_loss_disc_real_1: 0.22035059584698208 (-0.030944758947466455) | > avg_loss_disc_real_2: 0.22882223716923888 (-0.01724324238971925) | > avg_loss_disc_real_3: 0.2323080641283116 (-8.176151715533209e-05) | > avg_loss_disc_real_4: 0.24232215591719453 (-0.0010909878871810386) | > avg_loss_disc_real_5: 0.2444477570308766 (-0.0023016232839772333) | > avg_loss_0: 2.55912900306809 (-0.11217605899757244) | > avg_grad_norm_0: tensor(250.6924, device='cuda:0') (+tensor(4.7324, device='cuda:0')) | > avg_loss_gen: 2.130420147533149 (+0.020677077938134225) | > avg_loss_kl: 2.0047457587551065 (-0.19997091024694402) | > avg_loss_feat: 3.746250424586551 (+0.044695645990505906) | > avg_loss_mel: 16.92269765827017 (-0.7940060521515448) | > avg_loss_duration: 1.5446421730686242 (-0.021560875462814044) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.348756145423568 (-0.9501651441547203) | > avg_grad_norm_1: tensor(7574.2241, device='cuda:0') (+tensor(766.2456, device='cuda:0'))  > EPOCH: 1231/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:53:25)   --> TIME: 2025-02-25 20:53:35 -- STEP: 17/72 -- GLOBAL_STEP: 1088650 | > loss_disc: 2.6254754066467285 (2.4803297098945167) | > loss_disc_real_0: 0.08234694600105286 (0.07691849658594412) | > loss_disc_real_1: 0.22102640569210052 (0.21439134373384364) | > loss_disc_real_2: 0.24593877792358398 (0.23144727419404423) | > loss_disc_real_3: 0.2374860942363739 (0.22631551851244533) | > loss_disc_real_4: 0.25272136926651 (0.23725468007957234) | > loss_disc_real_5: 0.26014575362205505 (0.2449988617616541) | > loss_0: 2.6254754066467285 (2.4803297098945167) | > grad_norm_0: tensor(275.5489, device='cuda:0') (tensor(260.4577, device='cuda:0')) | > loss_gen: 2.1242728233337402 (2.1937379837036133) | > loss_kl: 2.0474400520324707 (1.8829938173294067) | > loss_feat: 3.646353006362915 (4.192312198526719) | > loss_mel: 16.60662078857422 (17.133239297305835) | > loss_duration: 1.5178101062774658 (1.4940357909483069) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.9424991607666 (26.89631888445686) | > grad_norm_1: tensor(15288.9023, device='cuda:0') (tensor(9237.6318, device='cuda:0')) | > current_lr_0: 0.00017147418809756415 | > current_lr_1: 0.00017147418809756415 | > step_time: 0.4345 (0.4301918674917782) | > loader_time: 0.0063 (0.006531014161951402)  --> TIME: 2025-02-25 20:53:59 -- STEP: 67/72 -- GLOBAL_STEP: 1088700 | > loss_disc: 2.3734593391418457 (2.505432787226208) | > loss_disc_real_0: 0.12271671742200851 (0.09814159088392754) | > loss_disc_real_1: 0.2350161373615265 (0.2175084207929782) | > loss_disc_real_2: 0.204598531126976 (0.23320177975875228) | > loss_disc_real_3: 0.23004409670829773 (0.23037599427486533) | > loss_disc_real_4: 0.22344167530536652 (0.2406814325211653) | > loss_disc_real_5: 0.22006656229496002 (0.24579558399186205) | > loss_0: 2.3734593391418457 (2.505432787226208) | > grad_norm_0: tensor(215.5947, device='cuda:0') (tensor(294.1193, device='cuda:0')) | > loss_gen: 2.534111976623535 (2.208990116617573) | > loss_kl: 2.0229241847991943 (1.95825751681826) | > loss_feat: 4.556282997131348 (3.9923243095625667) | > loss_mel: 17.438932418823242 (16.723369940003344) | > loss_duration: 1.6209455728530884 (1.5340970203058044) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.173194885253906 (26.417038846371778) | > grad_norm_1: tensor(4320.4570, device='cuda:0') (tensor(8418.6846, device='cuda:0')) | > current_lr_0: 0.00017147418809756415 | > current_lr_1: 0.00017147418809756415 | > step_time: 0.4909 (0.4570657281733271) | > loader_time: 0.012 (0.008624343729730862) --> EVAL PERFORMANCE | > avg_loader_time: 0.008788827439429055 (+0.0001367750302167011) | > avg_step_time: 0.45988084228945453 (+0.011572559114912895) | > avg_loss_disc: 2.514192839743387 (-0.044936163324703404) | > avg_loss_disc_real_0: 0.09966528295001512 (-0.017642210329502417) | > avg_loss_disc_real_1: 0.2190347779804552 (-0.001315817866526875) | > avg_loss_disc_real_2: 0.23326197875217652 (+0.00443974158293764) | > avg_loss_disc_real_3: 0.23068890647149423 (-0.0016191576568173605) | > avg_loss_disc_real_4: 0.24063301904940268 (-0.0016891368677918461) | > avg_loss_disc_real_5: 0.24629727320771821 (+0.0018495161768416202) | > avg_loss_0: 2.514192839743387 (-0.044936163324703404) | > avg_grad_norm_0: tensor(302.8856, device='cuda:0') (+tensor(52.1932, device='cuda:0')) | > avg_loss_gen: 2.2010362467295685 (+0.07061609919641976) | > avg_loss_kl: 1.9648631173120419 (-0.03988264144306464) | > avg_loss_feat: 3.974315885087132 (+0.22806546050058074) | > avg_loss_mel: 16.7055840290768 (-0.21711362919337063) | > avg_loss_duration: 1.5374990298714437 (-0.007143143197180457) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.383298256027867 (+0.034542110604299125) | > avg_grad_norm_1: tensor(8505.4512, device='cuda:0') (+tensor(931.2271, device='cuda:0'))  > EPOCH: 1232/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:54:02)   --> TIME: 2025-02-25 20:54:25 -- STEP: 45/72 -- GLOBAL_STEP: 1088750 | > loss_disc: 2.5122382640838623 (2.510557773378161) | > loss_disc_real_0: 0.07532096654176712 (0.10037835207250383) | > loss_disc_real_1: 0.20822228491306305 (0.2167244166135788) | > loss_disc_real_2: 0.20955295860767365 (0.224657032556004) | > loss_disc_real_3: 0.24224567413330078 (0.22983351084921094) | > loss_disc_real_4: 0.24646663665771484 (0.24227650629149544) | > loss_disc_real_5: 0.24369941651821136 (0.2473664989074071) | > loss_0: 2.5122382640838623 (2.510557773378161) | > grad_norm_0: tensor(189.1840, device='cuda:0') (tensor(287.0876, device='cuda:0')) | > loss_gen: 2.0211479663848877 (2.222543819745382) | > loss_kl: 2.074327230453491 (1.914821908209059) | > loss_feat: 3.878394603729248 (4.056834141413371) | > loss_mel: 16.86974334716797 (16.649827914767794) | > loss_duration: 1.5590052604675293 (1.506258980433146) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.402618408203125 (26.350286950005426) | > grad_norm_1: tensor(16170.4238, device='cuda:0') (tensor(9105.2061, device='cuda:0')) | > current_lr_0: 0.00017145275382405195 | > current_lr_1: 0.00017145275382405195 | > step_time: 0.4752 (0.4509580612182617) | > loader_time: 0.0091 (0.007663239373101129) --> EVAL PERFORMANCE | > avg_loader_time: 0.008785375407044313 (-3.4520323847417322e-06) | > avg_step_time: 0.46266941285469165 (+0.002788570565237125) | > avg_loss_disc: 2.5177734804825045 (+0.0035806407391176798) | > avg_loss_disc_real_0: 0.09939937136122878 (-0.0002659115887863328) | > avg_loss_disc_real_1: 0.2175446333180011 (-0.0014901446624540904) | > avg_loss_disc_real_2: 0.22475220325966955 (-0.008509775492506977) | > avg_loss_disc_real_3: 0.23041660726909907 (-0.0002722992023951598) | > avg_loss_disc_real_4: 0.2390412690354065 (-0.0015917500139961682) | > avg_loss_disc_real_5: 0.24729433068087403 (+0.0009970574731558157) | > avg_loss_0: 2.5177734804825045 (+0.0035806407391176798) | > avg_grad_norm_0: tensor(287.5701, device='cuda:0') (tensor(-15.3155, device='cuda:0')) | > avg_loss_gen: 2.2154056203197428 (+0.014369373590174206) | > avg_loss_kl: 1.958643377666742 (-0.006219739645299871) | > avg_loss_feat: 4.020872525765863 (+0.046556640678730776) | > avg_loss_mel: 16.57303364176146 (-0.13255038731534086) | > avg_loss_duration: 1.5316290519606899 (-0.005869977910753832) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.299584294708684 (-0.08371396131918374) | > avg_grad_norm_1: tensor(9193.2783, device='cuda:0') (+tensor(687.8271, device='cuda:0'))  > EPOCH: 1233/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:54:40)   --> TIME: 2025-02-25 20:54:52 -- STEP: 23/72 -- GLOBAL_STEP: 1088800 | > loss_disc: 2.6267504692077637 (2.514470100402832) | > loss_disc_real_0: 0.06885156780481339 (0.09970403608420621) | > loss_disc_real_1: 0.22184506058692932 (0.22125989328260007) | > loss_disc_real_2: 0.18353429436683655 (0.22671078858168228) | > loss_disc_real_3: 0.22980569303035736 (0.23005099853743677) | > loss_disc_real_4: 0.23530878126621246 (0.23855059885460395) | > loss_disc_real_5: 0.23378616571426392 (0.2476169667814089) | > loss_0: 2.6267504692077637 (2.514470100402832) | > grad_norm_0: tensor(208.5765, device='cuda:0') (tensor(305.9672, device='cuda:0')) | > loss_gen: 2.1626548767089844 (2.2847960358080655) | > loss_kl: 1.8481649160385132 (1.8535261568815813) | > loss_feat: 4.523429870605469 (4.213275277096292) | > loss_mel: 17.448741912841797 (16.686918922092605) | > loss_duration: 1.5112359523773193 (1.4881268636040066) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.49422836303711 (26.526643670123555) | > grad_norm_1: tensor(10875.6807, device='cuda:0') (tensor(9455.7568, device='cuda:0')) | > current_lr_0: 0.00017143132222982392 | > current_lr_1: 0.00017143132222982392 | > step_time: 0.4681 (0.43681328193001123) | > loader_time: 0.0084 (0.007004489069399627) --> EVAL PERFORMANCE | > avg_loader_time: 0.008877310954349142 (+9.193554730482952e-05) | > avg_step_time: 0.45887978311995387 (-0.003789629734737787) | > avg_loss_disc: 2.520203758293474 (+0.0024302778109697165) | > avg_loss_disc_real_0: 0.10205363807543902 (+0.002654266714210235) | > avg_loss_disc_real_1: 0.22126851514191695 (+0.0037238818239158367) | > avg_loss_disc_real_2: 0.22802847455924666 (+0.0032762712995771104) | > avg_loss_disc_real_3: 0.22976900805050218 (-0.0006475992185968893) | > avg_loss_disc_real_4: 0.23790740358157897 (-0.0011338654538275428) | > avg_loss_disc_real_5: 0.24635622115202352 (-0.0009381095288505115) | > avg_loss_0: 2.520203758293474 (+0.0024302778109697165) | > avg_grad_norm_0: tensor(296.2631, device='cuda:0') (+tensor(8.6930, device='cuda:0')) | > avg_loss_gen: 2.2587283161324505 (+0.04332269581270776) | > avg_loss_kl: 1.9556482190817175 (-0.0029951585850245266) | > avg_loss_feat: 4.043591569846784 (+0.022719044080921513) | > avg_loss_mel: 16.528986809958862 (-0.04404683180259639) | > avg_loss_duration: 1.5323322725967623 (+0.0007032206360724214) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.319287367270025 (+0.019703072561341628) | > avg_grad_norm_1: tensor(9226.9668, device='cuda:0') (+tensor(33.6885, device='cuda:0'))  > EPOCH: 1234/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:55:17)   --> TIME: 2025-02-25 20:55:19 -- STEP: 1/72 -- GLOBAL_STEP: 1088850 | > loss_disc: 2.5202815532684326 (2.5202815532684326) | > loss_disc_real_0: 0.22430315613746643 (0.22430315613746643) | > loss_disc_real_1: 0.19810864329338074 (0.19810864329338074) | > loss_disc_real_2: 0.24387255311012268 (0.24387255311012268) | > loss_disc_real_3: 0.25583595037460327 (0.25583595037460327) | > loss_disc_real_4: 0.268271267414093 (0.268271267414093) | > loss_disc_real_5: 0.2460571527481079 (0.2460571527481079) | > loss_0: 2.5202815532684326 (2.5202815532684326) | > grad_norm_0: tensor(696.8549, device='cuda:0') (tensor(696.8549, device='cuda:0')) | > loss_gen: 2.519258975982666 (2.519258975982666) | > loss_kl: 1.9020785093307495 (1.9020785093307495) | > loss_feat: 4.136697292327881 (4.136697292327881) | > loss_mel: 17.01202392578125 (17.01202392578125) | > loss_duration: 1.4855399131774902 (1.4855399131774902) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.055599212646484 (27.055599212646484) | > grad_norm_1: tensor(5425.2817, device='cuda:0') (tensor(5425.2817, device='cuda:0')) | > current_lr_0: 0.0001714098933145452 | > current_lr_1: 0.0001714098933145452 | > step_time: 0.426 (0.4259626865386963) | > loader_time: 0.0101 (0.010055065155029297)  --> TIME: 2025-02-25 20:55:43 -- STEP: 51/72 -- GLOBAL_STEP: 1088900 | > loss_disc: 2.482367992401123 (2.5389876412410364) | > loss_disc_real_0: 0.06907863169908524 (0.10147752547088791) | > loss_disc_real_1: 0.21670405566692352 (0.21652676661809286) | > loss_disc_real_2: 0.2541648745536804 (0.22860508660475412) | > loss_disc_real_3: 0.21747201681137085 (0.2322596183010176) | > loss_disc_real_4: 0.2188643515110016 (0.24149344394020006) | > loss_disc_real_5: 0.24834489822387695 (0.24786769847075144) | > loss_0: 2.482367992401123 (2.5389876412410364) | > grad_norm_0: tensor(161.8765, device='cuda:0') (tensor(315.3139, device='cuda:0')) | > loss_gen: 2.140099287033081 (2.218924363454182) | > loss_kl: 1.7651174068450928 (1.9363840187297148) | > loss_feat: 3.9740958213806152 (4.061128765928978) | > loss_mel: 15.810626983642578 (16.563960804658777) | > loss_duration: 1.5664395093917847 (1.509140692505182) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.256376266479492 (26.289538888370288) | > grad_norm_1: tensor(8733.6484, device='cuda:0') (tensor(8693.8096, device='cuda:0')) | > current_lr_0: 0.0001714098933145452 | > current_lr_1: 0.0001714098933145452 | > step_time: 0.4543 (0.455977481954238) | > loader_time: 0.0092 (0.007905759063421511) --> EVAL PERFORMANCE | > avg_loader_time: 0.008910420914770848 (+3.310996042170562e-05) | > avg_step_time: 0.4666094712808099 (+0.0077296881608560075) | > avg_loss_disc: 2.5356520061761563 (+0.015448247882682065) | > avg_loss_disc_real_0: 0.10006640111686478 (-0.001987236958574237) | > avg_loss_disc_real_1: 0.217158446849232 (-0.004110068292684954) | > avg_loss_disc_real_2: 0.2292498337550902 (+0.001221359195843541) | > avg_loss_disc_real_3: 0.23195417309311075 (+0.0021851650426085723) | > avg_loss_disc_real_4: 0.2390694108227609 (+0.0011620072411819282) | > avg_loss_disc_real_5: 0.24650489255576066 (+0.00014867140373714394) | > avg_loss_0: 2.5356520061761563 (+0.015448247882682065) | > avg_grad_norm_0: tensor(311.0941, device='cuda:0') (+tensor(14.8311, device='cuda:0')) | > avg_loss_gen: 2.218963283888051 (-0.039765032244399645) | > avg_loss_kl: 1.9810109894040604 (+0.02536277032234291) | > avg_loss_feat: 4.012265296049521 (-0.03132627379726305) | > avg_loss_mel: 16.500348104557524 (-0.02863870540133817) | > avg_loss_duration: 1.5286641691772032 (-0.0036681034195591433) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.241251985791703 (-0.07803538147832256) | > avg_grad_norm_1: tensor(8713.8633, device='cuda:0') (tensor(-513.1035, device='cuda:0'))  > EPOCH: 1235/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:55:55)   --> TIME: 2025-02-25 20:56:10 -- STEP: 29/72 -- GLOBAL_STEP: 1088950 | > loss_disc: 2.5426151752471924 (2.516785909389627) | > loss_disc_real_0: 0.06325238943099976 (0.09702886204267373) | > loss_disc_real_1: 0.16380983591079712 (0.22317086057416324) | > loss_disc_real_2: 0.23429971933364868 (0.23116920991190548) | > loss_disc_real_3: 0.23970402777194977 (0.23506010966054325) | > loss_disc_real_4: 0.2569772005081177 (0.24141051594553323) | > loss_disc_real_5: 0.24735824763774872 (0.24669194529796468) | > loss_0: 2.5426151752471924 (2.516785909389627) | > grad_norm_0: tensor(230.1264, device='cuda:0') (tensor(298.6968, device='cuda:0')) | > loss_gen: 2.0999488830566406 (2.288931904167965) | > loss_kl: 2.0288522243499756 (1.924505907913734) | > loss_feat: 4.080197811126709 (4.222916216685855) | > loss_mel: 17.511734008789062 (16.757121579400426) | > loss_duration: 1.4949026107788086 (1.484267600651445) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.21563720703125 (26.677743122495453) | > grad_norm_1: tensor(11597.0303, device='cuda:0') (tensor(9578.4512, device='cuda:0')) | > current_lr_0: 0.00017138846707788087 | > current_lr_1: 0.00017138846707788087 | > step_time: 0.4481 (0.4434263212927457) | > loader_time: 0.0081 (0.007310735768285291) --> EVAL PERFORMANCE | > avg_loader_time: 0.008927446016123598 (+1.7025101352749483e-05) | > avg_step_time: 0.4602021398678632 (-0.006407331412946693) | > avg_loss_disc: 2.538312273965755 (+0.0026602677895986027) | > avg_loss_disc_real_0: 0.10247679191156174 (+0.0024103907946969594) | > avg_loss_disc_real_1: 0.22687342196283206 (+0.00971497511360006) | > avg_loss_disc_real_2: 0.22684523085473288 (-0.002404602900357322) | > avg_loss_disc_real_3: 0.23597780247809183 (+0.004023629384981076) | > avg_loss_disc_real_4: 0.23992657766375744 (+0.0008571668409965427) | > avg_loss_disc_real_5: 0.24606853009949267 (-0.0004363624562679913) | > avg_loss_0: 2.538312273965755 (+0.0026602677895986027) | > avg_grad_norm_0: tensor(302.9055, device='cuda:0') (tensor(-8.1886, device='cuda:0')) | > avg_loss_gen: 2.240354574901957 (+0.021391291013906244) | > avg_loss_kl: 1.993050966464298 (+0.012039977060237517) | > avg_loss_feat: 4.035511923507906 (+0.023246627458385127) | > avg_loss_mel: 16.588632583618164 (+0.08828447906063985) | > avg_loss_duration: 1.5297687188000746 (+0.0011045496228714313) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.387318517120793 (+0.1460665313290903) | > avg_grad_norm_1: tensor(8906.4346, device='cuda:0') (+tensor(192.5713, device='cuda:0'))  > EPOCH: 1236/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:56:32)   --> TIME: 2025-02-25 20:56:37 -- STEP: 7/72 -- GLOBAL_STEP: 1089000 | > loss_disc: 2.534949541091919 (2.5137787887028287) | > loss_disc_real_0: 0.14174315333366394 (0.09345743379422597) | > loss_disc_real_1: 0.22270061075687408 (0.2188139706850052) | > loss_disc_real_2: 0.24327991902828217 (0.2236711404153279) | > loss_disc_real_3: 0.242343008518219 (0.2212384534733636) | > loss_disc_real_4: 0.26828432083129883 (0.23898453158991678) | > loss_disc_real_5: 0.2555074393749237 (0.25911395464624676) | > loss_0: 2.534949541091919 (2.5137787887028287) | > grad_norm_0: tensor(387.7184, device='cuda:0') (tensor(350.6177, device='cuda:0')) | > loss_gen: 2.4945709705352783 (2.3195397513253346) | > loss_kl: 2.0900557041168213 (1.9365180730819702) | > loss_feat: 4.3617753982543945 (4.278483629226685) | > loss_mel: 16.996810913085938 (16.876755033220565) | > loss_duration: 1.4811265468597412 (1.4816323178155082) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.424341201782227 (26.89292880467006) | > grad_norm_1: tensor(5177.9517, device='cuda:0') (tensor(8134.4053, device='cuda:0')) | > current_lr_0: 0.00017136704351949613 | > current_lr_1: 0.00017136704351949613 | > step_time: 0.4397 (0.43420382908412386) | > loader_time: 0.007 (0.006302458899361747)  --> TIME: 2025-02-25 20:57:01 -- STEP: 57/72 -- GLOBAL_STEP: 1089050 | > loss_disc: 2.8079891204833984 (2.5718539639523144) | > loss_disc_real_0: 0.11413921415805817 (0.11317093611547821) | > loss_disc_real_1: 0.06224996969103813 (0.21171789296102106) | > loss_disc_real_2: 0.2328508049249649 (0.22485055499955228) | > loss_disc_real_3: 0.1388252079486847 (0.23303783515043425) | > loss_disc_real_4: 0.2595747411251068 (0.24411740271668686) | > loss_disc_real_5: 0.27208805084228516 (0.2488604946094647) | > loss_0: 2.8079891204833984 (2.5718539639523144) | > grad_norm_0: tensor(179.7561, device='cuda:0') (tensor(294.2428, device='cuda:0')) | > loss_gen: 2.0385518074035645 (2.2650787453902392) | > loss_kl: 2.497843027114868 (2.0690061188580695) | > loss_feat: 4.320794582366943 (4.149601999082063) | > loss_mel: 17.035261154174805 (16.843125025431323) | > loss_duration: 1.6198146343231201 (1.525548677695425) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.512264251708984 (26.852360675209447) | > grad_norm_1: tensor(7749.2603, device='cuda:0') (tensor(7991.0342, device='cuda:0')) | > current_lr_0: 0.00017136704351949613 | > current_lr_1: 0.00017136704351949613 | > step_time: 0.4643 (0.44996234408596103) | > loader_time: 0.01 (0.00801723045215272) --> EVAL PERFORMANCE | > avg_loader_time: 0.008837458113549461 (-8.998790257413666e-05) | > avg_step_time: 0.4601092842263235 (-9.285564153965753e-05) | > avg_loss_disc: 2.5757953818415245 (+0.037483107875769583) | > avg_loss_disc_real_0: 0.11155869717329321 (+0.009081905261731471) | > avg_loss_disc_real_1: 0.21256020565477896 (-0.0143132163080531) | > avg_loss_disc_real_2: 0.2277694388594426 (+0.0009242080047097145) | > avg_loss_disc_real_3: 0.2308458104519777 (-0.0051319920261141405) | > avg_loss_disc_real_4: 0.24397792069005295 (+0.004051343026295506) | > avg_loss_disc_real_5: 0.24957915762780417 (+0.003510627528311494) | > avg_loss_0: 2.5757953818415245 (+0.037483107875769583) | > avg_grad_norm_0: tensor(280.6406, device='cuda:0') (tensor(-22.2649, device='cuda:0')) | > avg_loss_gen: 2.239935562644206 (-0.00041901225775120565) | > avg_loss_kl: 2.1095633960106004 (+0.11651242954630248) | > avg_loss_feat: 4.062836136616451 (+0.02732421310854516) | > avg_loss_mel: 16.917954659797775 (+0.3293220761796114) | > avg_loss_duration: 1.540553487522501 (+0.010784768722426508) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.87084332318373 (+0.48352480606293824) | > avg_grad_norm_1: tensor(7598.5562, device='cuda:0') (tensor(-1307.8784, device='cuda:0'))  > EPOCH: 1237/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:57:10)   --> TIME: 2025-02-25 20:57:28 -- STEP: 35/72 -- GLOBAL_STEP: 1089100 | > loss_disc: 2.3036670684814453 (2.600948940004621) | > loss_disc_real_0: 0.030679330229759216 (0.0892890633216926) | > loss_disc_real_1: 0.248057022690773 (0.22494511199848993) | > loss_disc_real_2: 0.28472909331321716 (0.2344866978270667) | > loss_disc_real_3: 0.22265303134918213 (0.24498112627438137) | > loss_disc_real_4: 0.2178567498922348 (0.24606292588370188) | > loss_disc_real_5: 0.2341795712709427 (0.24555133623736244) | > loss_0: 2.3036670684814453 (2.600948940004621) | > grad_norm_0: tensor(100.3569, device='cuda:0') (tensor(314.4062, device='cuda:0')) | > loss_gen: 2.5542168617248535 (2.140822614942278) | > loss_kl: 2.1482961177825928 (1.9925526312419346) | > loss_feat: 4.920431137084961 (3.9573176656450544) | > loss_mel: 17.617610931396484 (16.854999814714702) | > loss_duration: 1.553830623626709 (1.5023164272308347) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.79438591003418 (26.448009272984095) | > grad_norm_1: tensor(5360.0679, device='cuda:0') (tensor(8976.4854, device='cuda:0')) | > current_lr_0: 0.0001713456226390562 | > current_lr_1: 0.0001713456226390562 | > step_time: 0.4675 (0.4561432157244001) | > loader_time: 0.0086 (0.007240935734340123) --> EVAL PERFORMANCE | > avg_loader_time: 0.008804606719755788 (-3.2851393793673425e-05) | > avg_step_time: 0.4664642877981696 (+0.006355003571846096) | > avg_loss_disc: 2.6424527369754425 (+0.06665735513391802) | > avg_loss_disc_real_0: 0.10523052660512253 (-0.006328170568170685) | > avg_loss_disc_real_1: 0.2220739712807494 (+0.00951376562597045) | > avg_loss_disc_real_2: 0.23528996046999812 (+0.007520521610555525) | > avg_loss_disc_real_3: 0.2479777434762094 (+0.017131933024231716) | > avg_loss_disc_real_4: 0.25470139673897935 (+0.0107234760489264) | > avg_loss_disc_real_5: 0.2522481067079895 (+0.0026689490801853066) | > avg_loss_0: 2.6424527369754425 (+0.06665735513391802) | > avg_grad_norm_0: tensor(303.9514, device='cuda:0') (+tensor(23.3109, device='cuda:0')) | > avg_loss_gen: 2.188058181547782 (-0.051877381096423925) | > avg_loss_kl: 2.0644003841238963 (-0.04516301188670413) | > avg_loss_feat: 3.917536779188774 (-0.1452993574276773) | > avg_loss_mel: 16.714565438284005 (-0.20338922151377048) | > avg_loss_duration: 1.5397390076811885 (-0.0008144798413125542) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.424299938578002 (-0.44654338460572873) | > avg_grad_norm_1: tensor(7026.3057, device='cuda:0') (tensor(-572.2505, device='cuda:0'))  > EPOCH: 1238/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:57:48)   --> TIME: 2025-02-25 20:57:55 -- STEP: 13/72 -- GLOBAL_STEP: 1089150 | > loss_disc: 2.5080454349517822 (2.6077240797189565) | > loss_disc_real_0: 0.1768292486667633 (0.15872352570295334) | > loss_disc_real_1: 0.23053263127803802 (0.23798674230392164) | > loss_disc_real_2: 0.19967120885849 (0.25165701141724217) | > loss_disc_real_3: 0.2359907031059265 (0.25675514798897964) | > loss_disc_real_4: 0.2439621090888977 (0.24735998992736524) | > loss_disc_real_5: 0.2603561580181122 (0.2502704629531273) | > loss_0: 2.5080454349517822 (2.6077240797189565) | > grad_norm_0: tensor(386.4177, device='cuda:0') (tensor(313.1926, device='cuda:0')) | > loss_gen: 2.4420840740203857 (2.263732800116906) | > loss_kl: 2.0657382011413574 (2.0371902264081516) | > loss_feat: 4.252573490142822 (3.8398572298196645) | > loss_mel: 16.704238891601562 (17.310123150165264) | > loss_duration: 1.4591314792633057 (1.502501707810622) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.923765182495117 (26.9534052335299) | > grad_norm_1: tensor(3011.2483, device='cuda:0') (tensor(4377.1045, device='cuda:0')) | > current_lr_0: 0.0001713242044362263 | > current_lr_1: 0.0001713242044362263 | > step_time: 0.4618 (0.4373186918405386) | > loader_time: 0.0059 (0.006677389144897461)  --> TIME: 2025-02-25 20:58:19 -- STEP: 63/72 -- GLOBAL_STEP: 1089200 | > loss_disc: 2.6143805980682373 (2.652127663294474) | > loss_disc_real_0: 0.13234904408454895 (0.16830351020372103) | > loss_disc_real_1: 0.21046490967273712 (0.2266208143934371) | > loss_disc_real_2: 0.19861041009426117 (0.2586826116792738) | > loss_disc_real_3: 0.2562422752380371 (0.22283860627147886) | > loss_disc_real_4: 0.2395944595336914 (0.24903771304895003) | > loss_disc_real_5: 0.25821226835250854 (0.24779020557327877) | > loss_0: 2.6143805980682373 (2.652127663294474) | > grad_norm_0: tensor(367.2505, device='cuda:0') (tensor(285.0187, device='cuda:0')) | > loss_gen: 2.0587234497070312 (2.296858174460274) | > loss_kl: 2.307892084121704 (2.2013865444395284) | > loss_feat: 3.356107234954834 (3.931048798182654) | > loss_mel: 17.358890533447266 (17.53271054464673) | > loss_duration: 1.5784006118774414 (1.5563075807359483) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.660015106201172 (27.51831160651313) | > grad_norm_1: tensor(9194.3477, device='cuda:0') (tensor(4133.6987, device='cuda:0')) | > current_lr_0: 0.0001713242044362263 | > current_lr_1: 0.0001713242044362263 | > step_time: 0.5052 (0.45137029602414086) | > loader_time: 0.0113 (0.008245475708492215) --> EVAL PERFORMANCE | > avg_loader_time: 0.008745962465313114 (-5.8644254442673585e-05) | > avg_step_time: 0.45842156611697776 (-0.008042721681191856) | > avg_loss_disc: 2.6574567741071675 (+0.015004037131725045) | > avg_loss_disc_real_0: 0.16310198222037772 (+0.057871455615255196) | > avg_loss_disc_real_1: 0.22566058937932404 (+0.0035866180985746343) | > avg_loss_disc_real_2: 0.2546398513753648 (+0.019349890905366707) | > avg_loss_disc_real_3: 0.22437905709088687 (-0.02359868638532253) | > avg_loss_disc_real_4: 0.2473533145138915 (-0.007348082225087843) | > avg_loss_disc_real_5: 0.24717797485875412 (-0.005070131849235354) | > avg_loss_0: 2.6574567741071675 (+0.015004037131725045) | > avg_grad_norm_0: tensor(269.6204, device='cuda:0') (tensor(-34.3311, device='cuda:0')) | > avg_loss_gen: 2.2485129262360037 (+0.06045474468822176) | > avg_loss_kl: 2.2090796870245066 (+0.14467930290061037) | > avg_loss_feat: 3.8405072185355174 (-0.07702956065325672) | > avg_loss_mel: 17.464314474186423 (+0.7497490359024184) | > avg_loss_duration: 1.5657040401243825 (+0.025965032443193925) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.328118283983688 (+0.9038183454056856) | > avg_grad_norm_1: tensor(4326.3394, device='cuda:0') (tensor(-2699.9663, device='cuda:0'))  > EPOCH: 1239/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:58:25)   --> TIME: 2025-02-25 20:58:46 -- STEP: 41/72 -- GLOBAL_STEP: 1089250 | > loss_disc: 2.613736629486084 (2.7129755543499456) | > loss_disc_real_0: 0.06807153671979904 (0.16414602873165432) | > loss_disc_real_1: 0.14149270951747894 (0.24131856277221578) | > loss_disc_real_2: 0.23832039535045624 (0.23097923979526613) | > loss_disc_real_3: 0.21798768639564514 (0.2338378847372241) | > loss_disc_real_4: 0.24691054224967957 (0.24376272064883533) | > loss_disc_real_5: 0.25786346197128296 (0.2452892930042453) | > loss_0: 2.613736629486084 (2.7129755543499456) | > grad_norm_0: tensor(258.2581, device='cuda:0') (tensor(240.7266, device='cuda:0')) | > loss_gen: 1.744079351425171 (2.0125287771224984) | > loss_kl: 2.7065374851226807 (2.787828061638809) | > loss_feat: 4.093922138214111 (3.5999600829147713) | > loss_mel: 19.921581268310547 (20.328191059391674) | > loss_duration: 1.7882699966430664 (1.6650523005462274) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.254390716552734 (30.393560456066595) | > grad_norm_1: tensor(12260.9531, device='cuda:0') (tensor(7445.8564, device='cuda:0')) | > current_lr_0: 0.00017130278891067176 | > current_lr_1: 0.00017130278891067176 | > step_time: 0.4469 (0.4427638286497535) | > loader_time: 0.0093 (0.007443788574963081) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628757906631682 (-0.00011720455868143194) | > avg_step_time: 0.4542729283722354 (-0.0041486377447423894) | > avg_loss_disc: 2.644473680308167 (-0.012983093799000311) | > avg_loss_disc_real_0: 0.1283599195450964 (-0.03474206267528132) | > avg_loss_disc_real_1: 0.2271719591718324 (+0.0015113697925083491) | > avg_loss_disc_real_2: 0.23207857973978552 (-0.022561271635579305) | > avg_loss_disc_real_3: 0.23130465519260351 (+0.00692559810171664) | > avg_loss_disc_real_4: 0.24141741911290396 (-0.005935895400987545) | > avg_loss_disc_real_5: 0.24530209412037488 (-0.001875880738379243) | > avg_loss_0: 2.644473680308167 (-0.012983093799000311) | > avg_grad_norm_0: tensor(243.7864, device='cuda:0') (tensor(-25.8340, device='cuda:0')) | > avg_loss_gen: 2.045739969737095 (-0.20277295649890892) | > avg_loss_kl: 2.583805624867829 (+0.3747259378433223) | > avg_loss_feat: 3.679856132453596 (-0.16065108608192125) | > avg_loss_mel: 19.25901104027117 (+1.7946965660847454) | > avg_loss_duration: 1.6521206506541077 (+0.08641661052972527) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.220533397835744 (+1.8924151138520564) | > avg_grad_norm_1: tensor(8262.9404, device='cuda:0') (+tensor(3936.6011, device='cuda:0'))  > EPOCH: 1240/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:59:02)   --> TIME: 2025-02-25 20:59:13 -- STEP: 19/72 -- GLOBAL_STEP: 1089300 | > loss_disc: 2.4474499225616455 (2.4881778390784013) | > loss_disc_real_0: 0.08240339159965515 (0.0688357029698397) | > loss_disc_real_1: 0.23287953436374664 (0.20983095388663442) | > loss_disc_real_2: 0.24417462944984436 (0.2292357612597315) | > loss_disc_real_3: 0.22821341454982758 (0.21855578846053073) | > loss_disc_real_4: 0.2531999349594116 (0.2422647005633304) | > loss_disc_real_5: 0.25592169165611267 (0.24719332001711192) | > loss_0: 2.4474499225616455 (2.4881778390784013) | > grad_norm_0: tensor(94.8130, device='cuda:0') (tensor(261.8105, device='cuda:0')) | > loss_gen: 2.501169443130493 (2.1999053453144275) | > loss_kl: 1.9941723346710205 (1.9878792511789423) | > loss_feat: 4.3707756996154785 (4.167760133743286) | > loss_mel: 17.290584564208984 (17.53447070874666) | > loss_duration: 1.5207408666610718 (1.53105890750885) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.67744255065918 (27.421074214734528) | > grad_norm_1: tensor(6471.6543, device='cuda:0') (tensor(9804.6641, device='cuda:0')) | > current_lr_0: 0.00017128137606205792 | > current_lr_1: 0.00017128137606205792 | > step_time: 0.4458 (0.44146559112950373) | > loader_time: 0.0063 (0.007025932010851409)  --> TIME: 2025-02-25 20:59:37 -- STEP: 69/72 -- GLOBAL_STEP: 1089350 | > loss_disc: 2.480092763900757 (2.4919017570606177) | > loss_disc_real_0: 0.10598398745059967 (0.07275227284517843) | > loss_disc_real_1: 0.20429885387420654 (0.2099232937114826) | > loss_disc_real_2: 0.2281724065542221 (0.2322746104952218) | > loss_disc_real_3: 0.21568343043327332 (0.22735914069673288) | > loss_disc_real_4: 0.22344066202640533 (0.24457948190578516) | > loss_disc_real_5: 0.23238661885261536 (0.2492804546719012) | > loss_0: 2.480092763900757 (2.4919017570606177) | > grad_norm_0: tensor(247.3520, device='cuda:0') (tensor(290.5042, device='cuda:0')) | > loss_gen: 2.1959969997406006 (2.21187418612881) | > loss_kl: 2.2374637126922607 (2.0082223899122598) | > loss_feat: 3.8502395153045654 (4.129786553590194) | > loss_mel: 16.929996490478516 (17.14219243975653) | > loss_duration: 1.6550078392028809 (1.5575028070505113) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.86870574951172 (27.04957840408104) | > grad_norm_1: tensor(9905.1201, device='cuda:0') (tensor(10001.3252, device='cuda:0')) | > current_lr_0: 0.00017128137606205792 | > current_lr_1: 0.00017128137606205792 | > step_time: 0.5428 (0.46011628275332245) | > loader_time: 0.0126 (0.008843933326610624) --> EVAL PERFORMANCE | > avg_loader_time: 0.0088895105979812 (+0.0002607526913495188) | > avg_step_time: 0.46133424194765765 (+0.007061313575422279) | > avg_loss_disc: 2.4920911218079045 (-0.15238255850026272) | > avg_loss_disc_real_0: 0.07384105257585018 (-0.054518866969246224) | > avg_loss_disc_real_1: 0.20893489789794867 (-0.01823706127388372) | > avg_loss_disc_real_2: 0.2320937606650339 (+1.5180925248381438e-05) | > avg_loss_disc_real_3: 0.2270556885591695 (-0.00424896663343402) | > avg_loss_disc_real_4: 0.2442740413924338 (+0.002856622279529847) | > avg_loss_disc_real_5: 0.24909268991208414 (+0.0037905957917092614) | > avg_loss_0: 2.4920911218079045 (-0.15238255850026272) | > avg_grad_norm_0: tensor(293.7954, device='cuda:0') (+tensor(50.0091, device='cuda:0')) | > avg_loss_gen: 2.2114971103802534 (+0.1657571406431586) | > avg_loss_kl: 2.0081823157592558 (-0.5756233091085732) | > avg_loss_feat: 4.121312732427892 (+0.4414565999742961) | > avg_loss_mel: 17.103830606165065 (-2.1551804341061036) | > avg_loss_duration: 1.557637095451355 (-0.09448355520275276) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.002459888726893 (-2.2180735091088515) | > avg_grad_norm_1: tensor(9934.3252, device='cuda:0') (+tensor(1671.3848, device='cuda:0'))  > EPOCH: 1241/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 20:59:40)   --> TIME: 2025-02-25 21:00:04 -- STEP: 47/72 -- GLOBAL_STEP: 1089400 | > loss_disc: 2.5238707065582275 (2.5045893902474257) | > loss_disc_real_0: 0.09186770021915436 (0.09990017544081871) | > loss_disc_real_1: 0.19261272251605988 (0.2131291558133795) | > loss_disc_real_2: 0.24384312331676483 (0.23217576330012463) | > loss_disc_real_3: 0.22619567811489105 (0.23045549557564107) | > loss_disc_real_4: 0.2589501142501831 (0.2421885703472381) | > loss_disc_real_5: 0.2547593414783478 (0.24638666371081738) | > loss_0: 2.5238707065582275 (2.5045893902474257) | > grad_norm_0: tensor(326.2074, device='cuda:0') (tensor(303.6624, device='cuda:0')) | > loss_gen: 2.2723464965820312 (2.2844173096595934) | > loss_kl: 2.112398386001587 (1.910616641348981) | > loss_feat: 4.224751949310303 (4.091078337202681) | > loss_mel: 17.07807159423828 (16.806803439525854) | > loss_duration: 1.5627925395965576 (1.5228218601105061) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.2503604888916 (26.61573767154775) | > grad_norm_1: tensor(7765.6089, device='cuda:0') (tensor(8282.6963, device='cuda:0')) | > current_lr_0: 0.00017125996589005015 | > current_lr_1: 0.00017125996589005015 | > step_time: 0.4578 (0.45098248948442177) | > loader_time: 0.0091 (0.007664584098978245) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684007214828279 (-0.00020550338315292183) | > avg_step_time: 0.46510570485826946 (+0.0037714629106118114) | > avg_loss_disc: 2.5155416777436157 (+0.023450555935711215) | > avg_loss_disc_real_0: 0.10482389416912911 (+0.030982841593278937) | > avg_loss_disc_real_1: 0.21722960723957546 (+0.008294709341626794) | > avg_loss_disc_real_2: 0.23006162425162088 (-0.002032136413413016) | > avg_loss_disc_real_3: 0.22941470901731034 (+0.002359020458140848) | > avg_loss_disc_real_4: 0.2411441097796803 (-0.003129931612753517) | > avg_loss_disc_real_5: 0.24513896481252054 (-0.003953725099563599) | > avg_loss_0: 2.5155416777436157 (+0.023450555935711215) | > avg_grad_norm_0: tensor(306.3374, device='cuda:0') (+tensor(12.5420, device='cuda:0')) | > avg_loss_gen: 2.2655644047428183 (+0.054067294362564855) | > avg_loss_kl: 1.9558033540215292 (-0.052378961737726604) | > avg_loss_feat: 4.01898424726137 (-0.10232848516652204) | > avg_loss_mel: 16.760638908601145 (-0.34319169756392043) | > avg_loss_duration: 1.5433202259977101 (-0.014316869453644854) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.544311039884324 (-0.4581488488425691) | > avg_grad_norm_1: tensor(7978.3765, device='cuda:0') (tensor(-1955.9487, device='cuda:0'))  > EPOCH: 1242/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:00:18)   --> TIME: 2025-02-25 21:00:31 -- STEP: 25/72 -- GLOBAL_STEP: 1089450 | > loss_disc: 2.5099546909332275 (2.547258529663086) | > loss_disc_real_0: 0.17698806524276733 (0.11421595349907875) | > loss_disc_real_1: 0.22696581482887268 (0.21617402374744416) | > loss_disc_real_2: 0.21420711278915405 (0.2233938133716583) | > loss_disc_real_3: 0.19488631188869476 (0.23887698948383332) | > loss_disc_real_4: 0.19653771817684174 (0.24601265847682952) | > loss_disc_real_5: 0.24391193687915802 (0.24202132999897003) | > loss_0: 2.5099546909332275 (2.547258529663086) | > grad_norm_0: tensor(513.2291, device='cuda:0') (tensor(271.9999, device='cuda:0')) | > loss_gen: 2.3158631324768066 (2.270851192474365) | > loss_kl: 2.125112295150757 (1.9325983953475951) | > loss_feat: 4.544647216796875 (4.232170476913452) | > loss_mel: 17.031551361083984 (16.87157146453858) | > loss_duration: 1.5073728561401367 (1.4941076707839966) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.524547576904297 (26.80129928588867) | > grad_norm_1: tensor(5923.0942, device='cuda:0') (tensor(7725.0171, device='cuda:0')) | > current_lr_0: 0.00017123855839431388 | > current_lr_1: 0.00017123855839431388 | > step_time: 0.4521 (0.44526110649108885) | > loader_time: 0.0067 (0.006861200332641601) --> EVAL PERFORMANCE | > avg_loader_time: 0.008879473511601838 (+0.0001954662967735589) | > avg_step_time: 0.464588655552394 (-0.0005170493058754833) | > avg_loss_disc: 2.6234542920555874 (+0.10791261431197174) | > avg_loss_disc_real_0: 0.10646226477455085 (+0.0016383706054217406) | > avg_loss_disc_real_1: 0.2503452313617921 (+0.033115624122216636) | > avg_loss_disc_real_2: 0.24218929002822284 (+0.012127665776601959) | > avg_loss_disc_real_3: 0.2313504651398726 (+0.0019357561225622488) | > avg_loss_disc_real_4: 0.24828676039903946 (+0.0071426506193591754) | > avg_loss_disc_real_5: 0.2421027768245885 (-0.0030361879879320375) | > avg_loss_0: 2.6234542920555874 (+0.10791261431197174) | > avg_grad_norm_0: tensor(267.0963, device='cuda:0') (tensor(-39.2412, device='cuda:0')) | > avg_loss_gen: 2.214219288087228 (-0.051345116655590495) | > avg_loss_kl: 2.169530331248968 (+0.21372697722743905) | > avg_loss_feat: 3.9821435129138787 (-0.036840734347491466) | > avg_loss_mel: 17.513666757395566 (+0.7530278487944209) | > avg_loss_duration: 1.557836015459517 (+0.014515789461806872) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.437395901747152 (+0.8930848618628282) | > avg_grad_norm_1: tensor(7960.7764, device='cuda:0') (tensor(-17.6001, device='cuda:0'))  > EPOCH: 1243/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:00:55)   --> TIME: 2025-02-25 21:00:59 -- STEP: 3/72 -- GLOBAL_STEP: 1089500 | > loss_disc: 2.583920955657959 (2.686988671620687) | > loss_disc_real_0: 0.2723422050476074 (0.21259781469901404) | > loss_disc_real_1: 0.2474515736103058 (0.24586530029773712) | > loss_disc_real_2: 0.21463905274868011 (0.2113140473763148) | > loss_disc_real_3: 0.2560150921344757 (0.24713548024495444) | > loss_disc_real_4: 0.2679429352283478 (0.2592538446187973) | > loss_disc_real_5: 0.2217593789100647 (0.22506978114446005) | > loss_0: 2.583920955657959 (2.686988671620687) | > grad_norm_0: tensor(535.7897, device='cuda:0') (tensor(407.4631, device='cuda:0')) | > loss_gen: 2.447598695755005 (2.210935592651367) | > loss_kl: 2.266396999359131 (2.1448967456817627) | > loss_feat: 4.312617301940918 (3.5792669455210366) | > loss_mel: 18.246477127075195 (17.281606674194336) | > loss_duration: 1.5627297163009644 (1.5204743544260662) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.835819244384766 (26.73718007405599) | > grad_norm_1: tensor(2797.4846, device='cuda:0') (tensor(5025.8716, device='cuda:0')) | > current_lr_0: 0.00017121715357451458 | > current_lr_1: 0.00017121715357451458 | > step_time: 0.4375 (0.4434653123219808) | > loader_time: 0.0068 (0.006440639495849609)  --> TIME: 2025-02-25 21:01:22 -- STEP: 53/72 -- GLOBAL_STEP: 1089550 | > loss_disc: 2.7074673175811768 (2.7644606446320146) | > loss_disc_real_0: 0.15178857743740082 (0.20139993052437621) | > loss_disc_real_1: 0.255123496055603 (0.23380731692853965) | > loss_disc_real_2: 0.2565811574459076 (0.2361559401143272) | > loss_disc_real_3: 0.22227878868579865 (0.23853159794267617) | > loss_disc_real_4: 0.2207242101430893 (0.24958266233498208) | > loss_disc_real_5: 0.24428190290927887 (0.2464529753855939) | > loss_0: 2.7074673175811768 (2.7644606446320146) | > grad_norm_0: tensor(169.1592, device='cuda:0') (tensor(248.4281, device='cuda:0')) | > loss_gen: 1.8368034362792969 (2.010124582164692) | > loss_kl: 2.4324352741241455 (2.0531489489213466) | > loss_feat: 3.033167839050293 (3.5189659865397327) | > loss_mel: 16.409603118896484 (17.09566141524405) | > loss_duration: 1.6141637563705444 (1.5316690094066117) | > amp_scaler: 8.0 (8.150943396226419) | > loss_1: 25.326173782348633 (26.209570003005688) | > grad_norm_1: tensor(797.9208, device='cuda:0') (tensor(5003.7871, device='cuda:0')) | > current_lr_0: 0.00017121715357451458 | > current_lr_1: 0.00017121715357451458 | > step_time: 0.4519 (0.46117922045149895) | > loader_time: 0.0091 (0.00806795875981169) --> EVAL PERFORMANCE | > avg_loader_time: 0.00888935277159785 (+9.879259996011663e-06) | > avg_step_time: 0.4719744769620224 (+0.007385821409628401) | > avg_loss_disc: 2.7224305884938835 (+0.09897629643829609) | > avg_loss_disc_real_0: 0.18673891359021969 (+0.08027664881566883) | > avg_loss_disc_real_1: 0.23200771527391084 (-0.018337516087881256) | > avg_loss_disc_real_2: 0.23638066964250215 (-0.005808620385720692) | > avg_loss_disc_real_3: 0.24057927832637035 (+0.009228813186497764) | > avg_loss_disc_real_4: 0.2461113172097946 (-0.002175443189244869) | > avg_loss_disc_real_5: 0.2462254072159109 (+0.004122630391322407) | > avg_loss_0: 2.7224305884938835 (+0.09897629643829609) | > avg_grad_norm_0: tensor(244.6215, device='cuda:0') (tensor(-22.4748, device='cuda:0')) | > avg_loss_gen: 2.0284857313397895 (-0.1857335567474383) | > avg_loss_kl: 2.117471276874273 (-0.05205905437469527) | > avg_loss_feat: 3.5499375034386005 (-0.4322060094752782) | > avg_loss_mel: 17.100214649254166 (-0.4134521081413993) | > avg_loss_duration: 1.5502030429705767 (-0.007632972488940348) | > avg_amp_scaler: 8.112676056338028 (+0.11267605633802802) | > avg_loss_1: 26.34631232140769 (-1.0910835803394612) | > avg_grad_norm_1: tensor(4650.5142, device='cuda:0') (tensor(-3310.2622, device='cuda:0'))  > EPOCH: 1244/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:01:34)   --> TIME: 2025-02-25 21:01:50 -- STEP: 31/72 -- GLOBAL_STEP: 1089600 | > loss_disc: 2.454502582550049 (2.505077031350905) | > loss_disc_real_0: 0.052901968359947205 (0.07673971583285637) | > loss_disc_real_1: 0.23380598425865173 (0.22942391855101432) | > loss_disc_real_2: 0.23018930852413177 (0.2288868970447971) | > loss_disc_real_3: 0.24731023609638214 (0.23982845150655316) | > loss_disc_real_4: 0.2575491964817047 (0.2453590597837202) | > loss_disc_real_5: 0.2567121386528015 (0.2526242463819442) | > loss_0: 2.454502582550049 (2.505077031350905) | > grad_norm_0: tensor(251.2388, device='cuda:0') (tensor(276.9795, device='cuda:0')) | > loss_gen: 2.374025344848633 (2.2103802696351083) | > loss_kl: 2.1598188877105713 (2.013292647177174) | > loss_feat: 4.318296909332275 (4.144346106436945) | > loss_mel: 17.200931549072266 (17.021620012098744) | > loss_duration: 1.5274879932403564 (1.5022425382368025) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.5805606842041 (26.891881881221646) | > grad_norm_1: tensor(9142.6572, device='cuda:0') (tensor(7763.7915, device='cuda:0')) | > current_lr_0: 0.00017119575143031775 | > current_lr_1: 0.00017119575143031775 | > step_time: 0.4619 (0.44637286278509325) | > loader_time: 0.0085 (0.007115817839099515) --> EVAL PERFORMANCE | > avg_loader_time: 0.008809925804675465 (-7.942696692238467e-05) | > avg_step_time: 0.46623494927312287 (-0.005739527688899515) | > avg_loss_disc: 2.60413246087625 (-0.11829812761763359) | > avg_loss_disc_real_0: 0.09002815795616362 (-0.09671075563405607) | > avg_loss_disc_real_1: 0.23797974861423735 (+0.005972033340326505) | > avg_loss_disc_real_2: 0.2291770886997102 (-0.007203580942791943) | > avg_loss_disc_real_3: 0.2363241495380939 (-0.004255128788276469) | > avg_loss_disc_real_4: 0.24327667813066026 (-0.002834639079134338) | > avg_loss_disc_real_5: 0.2494866831201902 (+0.0032612759042792983) | > avg_loss_0: 2.60413246087625 (-0.11829812761763359) | > avg_grad_norm_0: tensor(309.8594, device='cuda:0') (+tensor(65.2379, device='cuda:0')) | > avg_loss_gen: 2.1361119747161847 (+0.10762624337639526) | > avg_loss_kl: 2.0304703494192853 (-0.08700092745498766) | > avg_loss_feat: 3.9466498099582297 (+0.3967123065196292) | > avg_loss_mel: 16.78302606394593 (-0.31718858530823724) | > avg_loss_duration: 1.541718088405233 (-0.008484954565343594) | > avg_amp_scaler: 8.0 (-0.11267605633802802) | > avg_loss_1: 26.437976299876897 (+0.0916639784692066) | > avg_grad_norm_1: tensor(8556.5869, device='cuda:0') (+tensor(3906.0728, device='cuda:0'))  > EPOCH: 1245/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:02:11)   --> TIME: 2025-02-25 21:02:17 -- STEP: 9/72 -- GLOBAL_STEP: 1089650 | > loss_disc: 2.6138336658477783 (2.6016773647732205) | > loss_disc_real_0: 0.09335843473672867 (0.08137113394008742) | > loss_disc_real_1: 0.22383813560009003 (0.1948658741182751) | > loss_disc_real_2: 0.39910396933555603 (0.23674419025580087) | > loss_disc_real_3: 0.22661928832530975 (0.21355305943224165) | > loss_disc_real_4: 0.23090672492980957 (0.23526017533408272) | > loss_disc_real_5: 0.24887172877788544 (0.23290292256408268) | > loss_0: 2.6138336658477783 (2.6016773647732205) | > grad_norm_0: tensor(575.3287, device='cuda:0') (tensor(461.8327, device='cuda:0')) | > loss_gen: 2.4611852169036865 (2.0690935850143433) | > loss_kl: 1.6988085508346558 (1.8575464751985338) | > loss_feat: 3.8740124702453613 (3.937630123562283) | > loss_mel: 16.990022659301758 (16.691840489705402) | > loss_duration: 1.4917837381362915 (1.4981547726525202) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.515811920166016 (26.05426576402452) | > grad_norm_1: tensor(13718.2725, device='cuda:0') (tensor(10731.1191, device='cuda:0')) | > current_lr_0: 0.00017117435196138895 | > current_lr_1: 0.00017117435196138895 | > step_time: 0.4333 (0.43725400500827366) | > loader_time: 0.0067 (0.00644683837890625)  --> TIME: 2025-02-25 21:02:41 -- STEP: 59/72 -- GLOBAL_STEP: 1089700 | > loss_disc: 2.503441572189331 (2.5564889301687983) | > loss_disc_real_0: 0.06815755367279053 (0.0832989518925295) | > loss_disc_real_1: 0.19864270091056824 (0.21853026293091854) | > loss_disc_real_2: 0.23799633979797363 (0.2324039082405931) | > loss_disc_real_3: 0.22407659888267517 (0.22885631555217809) | > loss_disc_real_4: 0.2348523586988449 (0.23882514895018886) | > loss_disc_real_5: 0.23764581978321075 (0.24188225779492975) | > loss_0: 2.503441572189331 (2.5564889301687983) | > grad_norm_0: tensor(266.5303, device='cuda:0') (tensor(331.1765, device='cuda:0')) | > loss_gen: 2.111931800842285 (2.116524290230314) | > loss_kl: 1.781235933303833 (1.9409726555064573) | > loss_feat: 4.190419673919678 (3.8974367602396818) | > loss_mel: 16.142793655395508 (16.47530811115847) | > loss_duration: 1.5565476417541504 (1.522431488764488) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.782926559448242 (25.95267302302991) | > grad_norm_1: tensor(7284.8560, device='cuda:0') (tensor(9685.8672, device='cuda:0')) | > current_lr_0: 0.00017117435196138895 | > current_lr_1: 0.00017117435196138895 | > step_time: 0.4822 (0.45312563443588) | > loader_time: 0.0101 (0.008182307421150857) --> EVAL PERFORMANCE | > avg_loader_time: 0.008989041959735712 (+0.00017911615506024763) | > avg_step_time: 0.463046198159876 (-0.003188751113246846) | > avg_loss_disc: 2.5568100163634395 (-0.047322444512810424) | > avg_loss_disc_real_0: 0.08580485678894419 (-0.0042233011672194315) | > avg_loss_disc_real_1: 0.2168491459648374 (-0.02113060264939995) | > avg_loss_disc_real_2: 0.2324339252122691 (+0.003256836512558897) | > avg_loss_disc_real_3: 0.2284406685913113 (-0.007883480946782595) | > avg_loss_disc_real_4: 0.23725709298127134 (-0.006019585149388912) | > avg_loss_disc_real_5: 0.2423222967436616 (-0.007164386376528609) | > avg_loss_0: 2.5568100163634395 (-0.047322444512810424) | > avg_grad_norm_0: tensor(340.5080, device='cuda:0') (+tensor(30.6486, device='cuda:0')) | > avg_loss_gen: 2.1184425605854513 (-0.01766941413073342) | > avg_loss_kl: 1.9620434868503625 (-0.06842686256892283) | > avg_loss_feat: 3.899751478517559 (-0.04689833144067057) | > avg_loss_mel: 16.513728477585484 (-0.26929758636044454) | > avg_loss_duration: 1.5339612356373962 (-0.007756852767836886) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.027927022584727 (-0.41004927729217044) | > avg_grad_norm_1: tensor(9851.6240, device='cuda:0') (+tensor(1295.0371, device='cuda:0'))  > EPOCH: 1246/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:02:49)   --> TIME: 2025-02-25 21:03:08 -- STEP: 37/72 -- GLOBAL_STEP: 1089750 | > loss_disc: 2.4206550121307373 (2.5176536005896493) | > loss_disc_real_0: 0.09229868650436401 (0.09030452451190434) | > loss_disc_real_1: 0.2237035036087036 (0.2179550943342415) | > loss_disc_real_2: 0.24627307057380676 (0.23088038289869153) | > loss_disc_real_3: 0.23877349495887756 (0.23445279493525223) | > loss_disc_real_4: 0.2360811084508896 (0.2429023344774504) | > loss_disc_real_5: 0.24992969632148743 (0.247603919860479) | > loss_0: 2.4206550121307373 (2.5176536005896493) | > grad_norm_0: tensor(122.2288, device='cuda:0') (tensor(314.9804, device='cuda:0')) | > loss_gen: 2.325105905532837 (2.2255977198884294) | > loss_kl: 1.796258568763733 (1.9081632511035815) | > loss_feat: 4.078150749206543 (4.072760614188942) | > loss_mel: 16.6909236907959 (16.50619594470874) | > loss_duration: 1.532416820526123 (1.4950385931375865) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.422855377197266 (26.207756145580394) | > grad_norm_1: tensor(10437.6143, device='cuda:0') (tensor(9108.8711, device='cuda:0')) | > current_lr_0: 0.00017115295516739376 | > current_lr_1: 0.00017115295516739376 | > step_time: 0.4821 (0.4514851054629764) | > loader_time: 0.0081 (0.007538602158829973) --> EVAL PERFORMANCE | > avg_loader_time: 0.009144279318796078 (+0.00015523735906036555) | > avg_step_time: 0.476467639627591 (+0.013421441467714956) | > avg_loss_disc: 2.5151424240058575 (-0.041667592357581995) | > avg_loss_disc_real_0: 0.08916883118135825 (+0.0033639743924140653) | > avg_loss_disc_real_1: 0.21835949849074995 (+0.001510352525912556) | > avg_loss_disc_real_2: 0.22901135053433164 (-0.0034225746779374677) | > avg_loss_disc_real_3: 0.23125977226546113 (+0.0028191036741498365) | > avg_loss_disc_real_4: 0.24030015879953412 (+0.0030430658182627746) | > avg_loss_disc_real_5: 0.2453380052892255 (+0.0030157085455639054) | > avg_loss_0: 2.5151424240058575 (-0.041667592357581995) | > avg_grad_norm_0: tensor(321.1238, device='cuda:0') (tensor(-19.3842, device='cuda:0')) | > avg_loss_gen: 2.22199819457363 (+0.10355563398817846) | > avg_loss_kl: 1.9811378751002566 (+0.01909438824989418) | > avg_loss_feat: 4.048149615945951 (+0.14839813742839159) | > avg_loss_mel: 16.44153511692101 (-0.07219336066447468) | > avg_loss_duration: 1.5298848219320809 (-0.004076413705315307) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.222705733608194 (+0.1947787110234671) | > avg_grad_norm_1: tensor(9186.0703, device='cuda:0') (tensor(-665.5537, device='cuda:0'))  > EPOCH: 1247/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:03:28)   --> TIME: 2025-02-25 21:03:36 -- STEP: 15/72 -- GLOBAL_STEP: 1089800 | > loss_disc: 2.5388636589050293 (2.5471594174702963) | > loss_disc_real_0: 0.08723925054073334 (0.10712155376871427) | > loss_disc_real_1: 0.21298080682754517 (0.21272366146246593) | > loss_disc_real_2: 0.21465915441513062 (0.2374838799238205) | > loss_disc_real_3: 0.20304003357887268 (0.23160275022188823) | > loss_disc_real_4: 0.21580684185028076 (0.24303910930951436) | > loss_disc_real_5: 0.2239595204591751 (0.2490221639474233) | > loss_0: 2.5388636589050293 (2.5471594174702963) | > grad_norm_0: tensor(226.5892, device='cuda:0') (tensor(346.9496, device='cuda:0')) | > loss_gen: 2.087641954421997 (2.1946986357371006) | > loss_kl: 2.1562557220458984 (1.8773703654607137) | > loss_feat: 4.00168514251709 (3.9526028474171957) | > loss_mel: 16.473772048950195 (16.361798032124838) | > loss_duration: 1.470242977142334 (1.4790672302246093) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.189599990844727 (25.865537389119467) | > grad_norm_1: tensor(9166.2959, device='cuda:0') (tensor(7644.6514, device='cuda:0')) | > current_lr_0: 0.00017113156104799784 | > current_lr_1: 0.00017113156104799784 | > step_time: 0.4508 (0.4445569356282552) | > loader_time: 0.0064 (0.006740729014078776)  --> TIME: 2025-02-25 21:04:00 -- STEP: 65/72 -- GLOBAL_STEP: 1089850 | > loss_disc: 3.4156789779663086 (2.593409285178551) | > loss_disc_real_0: 0.1501096934080124 (0.11183332760746663) | > loss_disc_real_1: 0.20149250328540802 (0.22342922733380244) | > loss_disc_real_2: 0.32115286588668823 (0.2512045133572358) | > loss_disc_real_3: 0.270874947309494 (0.23666679446513836) | > loss_disc_real_4: 0.24987122416496277 (0.24792552544520452) | > loss_disc_real_5: 0.2821862995624542 (0.24661006767016191) | > loss_0: 3.4156789779663086 (2.593409285178551) | > grad_norm_0: tensor(886.6105, device='cuda:0') (tensor(315.3357, device='cuda:0')) | > loss_gen: 1.682814598083496 (2.221058460382315) | > loss_kl: 2.1649396419525146 (1.97125435425685) | > loss_feat: 3.280743360519409 (3.9790687047518216) | > loss_mel: 16.961811065673828 (16.40845272357648) | > loss_duration: 1.5949817895889282 (1.5245038711107695) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.685291290283203 (26.104338103074294) | > grad_norm_1: tensor(2192.3411, device='cuda:0') (tensor(7389.7349, device='cuda:0')) | > current_lr_0: 0.00017113156104799784 | > current_lr_1: 0.00017113156104799784 | > step_time: 0.5063 (0.45641356981717623) | > loader_time: 0.0115 (0.008357950357290414) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869514572788292 (-0.0004491335909131578) | > avg_step_time: 0.4624294160117566 (-0.0140382236158344) | > avg_loss_disc: 2.6203548370952334 (+0.1052124130893759) | > avg_loss_disc_real_0: 0.11341854872208246 (+0.024249717540724205) | > avg_loss_disc_real_1: 0.22326077856648135 (+0.004901280075731401) | > avg_loss_disc_real_2: 0.2508924989633157 (+0.021881148428984087) | > avg_loss_disc_real_3: 0.23779971918589632 (+0.006539946920435191) | > avg_loss_disc_real_4: 0.24526210549012037 (+0.00496194669058625) | > avg_loss_disc_real_5: 0.24990302702070963 (+0.004565021731484126) | > avg_loss_0: 2.6203548370952334 (+0.1052124130893759) | > avg_grad_norm_0: tensor(312.4219, device='cuda:0') (tensor(-8.7020, device='cuda:0')) | > avg_loss_gen: 2.1931194721812943 (-0.028878722392335465) | > avg_loss_kl: 1.9909899117241443 (+0.009852036623887672) | > avg_loss_feat: 3.935543872940708 (-0.11260574300524251) | > avg_loss_mel: 16.4169929665579 (-0.024542150363110693) | > avg_loss_duration: 1.531182393221788 (+0.0012975712897072) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.06782867539097 (-0.15487705821722386) | > avg_grad_norm_1: tensor(7295.4683, device='cuda:0') (tensor(-1890.6021, device='cuda:0'))  > EPOCH: 1248/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:04:05)   --> TIME: 2025-02-25 21:04:27 -- STEP: 43/72 -- GLOBAL_STEP: 1089900 | > loss_disc: 2.6341166496276855 (2.6890791294186616) | > loss_disc_real_0: 0.1586916744709015 (0.1945986057263474) | > loss_disc_real_1: 0.2220512330532074 (0.23266125886246217) | > loss_disc_real_2: 0.23332077264785767 (0.2524108727310978) | > loss_disc_real_3: 0.25298187136650085 (0.24403518854185602) | > loss_disc_real_4: 0.2372346818447113 (0.2552649271349574) | > loss_disc_real_5: 0.2215953767299652 (0.24701227906138398) | > loss_0: 2.6341166496276855 (2.6890791294186616) | > grad_norm_0: tensor(435.9871, device='cuda:0') (tensor(307.2828, device='cuda:0')) | > loss_gen: 2.0927748680114746 (2.1566465316816816) | > loss_kl: 2.1549885272979736 (1.9507321590601012) | > loss_feat: 3.8714091777801514 (3.6100604367810627) | > loss_mel: 15.771015167236328 (16.585880412611857) | > loss_duration: 1.5350887775421143 (1.5073996560518133) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.425277709960938 (25.810718935589456) | > grad_norm_1: tensor(4236.4453, device='cuda:0') (tensor(6576.6045, device='cuda:0')) | > current_lr_0: 0.00017111016960286683 | > current_lr_1: 0.00017111016960286683 | > step_time: 0.466 (0.4450323193572288) | > loader_time: 0.0096 (0.007804831793141919) --> EVAL PERFORMANCE | > avg_loader_time: 0.008988501320422535 (+0.00029335559253961495) | > avg_step_time: 0.46098753767953793 (-0.001441878332218649) | > avg_loss_disc: 2.6845188006548812 (+0.06416396355964782) | > avg_loss_disc_real_0: 0.185461265835124 (+0.07204271711304155) | > avg_loss_disc_real_1: 0.2242982751676734 (+0.0010374966011920395) | > avg_loss_disc_real_2: 0.24529699624424245 (-0.005595502719073275) | > avg_loss_disc_real_3: 0.2435905708813332 (+0.005790851695436888) | > avg_loss_disc_real_4: 0.25414408155730084 (+0.00888197606718047) | > avg_loss_disc_real_5: 0.24627428281475122 (-0.0036287442059584063) | > avg_loss_0: 2.6845188006548812 (+0.06416396355964782) | > avg_grad_norm_0: tensor(293.4016, device='cuda:0') (tensor(-19.0203, device='cuda:0')) | > avg_loss_gen: 2.1180807083425384 (-0.07503876383875596) | > avg_loss_kl: 2.0233748630738595 (+0.032384951349715196) | > avg_loss_feat: 3.606136741772504 (-0.3294071311682041) | > avg_loss_mel: 16.749525701495962 (+0.3325327349380629) | > avg_loss_duration: 1.535606364129295 (+0.004423970907506902) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.032724192444707 (-0.035104482946263005) | > avg_grad_norm_1: tensor(6316.7900, device='cuda:0') (tensor(-978.6782, device='cuda:0'))  > EPOCH: 1249/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:04:43)   --> TIME: 2025-02-25 21:04:54 -- STEP: 21/72 -- GLOBAL_STEP: 1089950 | > loss_disc: 2.841465950012207 (2.7762735344114757) | > loss_disc_real_0: 0.40616941452026367 (0.2936840085756211) | > loss_disc_real_1: 0.25753477215766907 (0.23291959507124765) | > loss_disc_real_2: 0.244624525308609 (0.23859881361325583) | > loss_disc_real_3: 0.2576160132884979 (0.23165275369371688) | > loss_disc_real_4: 0.2575576603412628 (0.2365881467149371) | > loss_disc_real_5: 0.26769986748695374 (0.24705479471456437) | > loss_0: 2.841465950012207 (2.7762735344114757) | > grad_norm_0: tensor(576.9744, device='cuda:0') (tensor(342.4665, device='cuda:0')) | > loss_gen: 2.1071648597717285 (2.1133184376217065) | > loss_kl: 3.4383554458618164 (2.489944486390977) | > loss_feat: 3.2289578914642334 (3.4881290708269392) | > loss_mel: 22.91292381286621 (18.791719027927943) | > loss_duration: 1.660817265510559 (1.5484429427555628) | > amp_scaler: 8.0 (8.0) | > loss_1: 33.34822082519531 (28.43155388605027) | > grad_norm_1: tensor(1307.3633, device='cuda:0') (tensor(5474.4561, device='cuda:0')) | > current_lr_0: 0.00017108878083166645 | > current_lr_1: 0.00017108878083166645 | > step_time: 0.4274 (0.4375984441666376) | > loader_time: 0.0064 (0.006716330846150716)  --> TIME: 2025-02-25 21:05:18 -- STEP: 71/72 -- GLOBAL_STEP: 1090000 | > loss_disc: 2.7880125045776367 (2.7164483439754434) | > loss_disc_real_0: 0.04683292284607887 (0.19496380689908083) | > loss_disc_real_1: 0.2643817961215973 (0.2269151126834708) | > loss_disc_real_2: 0.2691442668437958 (0.24047713338489263) | > loss_disc_real_3: 0.22205258905887604 (0.24254169997195124) | > loss_disc_real_4: 0.2166529893875122 (0.2415133578256822) | > loss_disc_real_5: 0.1979319155216217 (0.24179272983275668) | > loss_0: 2.7880125045776367 (2.7164483439754434) | > grad_norm_0: tensor(61.9284, device='cuda:0') (tensor(268.5762, device='cuda:0')) | > loss_gen: 1.8917229175567627 (2.1170679129345307) | > loss_kl: 2.1955840587615967 (2.410116603676701) | > loss_feat: 2.8591322898864746 (3.6145172387781277) | > loss_mel: 16.092628479003906 (18.053640983474086) | > loss_duration: 1.5718135833740234 (1.5767965199242175) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.610881805419922 (27.772139455231144) | > grad_norm_1: tensor(6923.9019, device='cuda:0') (tensor(5443.5449, device='cuda:0')) | > current_lr_0: 0.00017108878083166645 | > current_lr_1: 0.00017108878083166645 | > step_time: 0.4816 (0.45590173358648595) | > loader_time: 0.008 (0.00886637727979203) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1090000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.00886637727979203 (-0.0001221240406305054) | > avg_step_time: 0.45590173358648595 (-0.005085804093051982) | > avg_loss_disc: 2.7164483439754434 (+0.03192954332056219) | > avg_loss_disc_real_0: 0.19496380689908083 (+0.009502541063956815) | > avg_loss_disc_real_1: 0.2269151126834708 (+0.0026168375157974155) | > avg_loss_disc_real_2: 0.24047713338489263 (-0.004819862859349822) | > avg_loss_disc_real_3: 0.24254169997195124 (-0.001048870909381966) | > avg_loss_disc_real_4: 0.2415133578256822 (-0.012630723731618637) | > avg_loss_disc_real_5: 0.24179272983275668 (-0.004481552981994541) | > avg_loss_0: 2.7164483439754434 (+0.03192954332056219) | > avg_grad_norm_0: tensor(268.5762, device='cuda:0') (tensor(-24.8254, device='cuda:0')) | > avg_loss_gen: 2.1170679129345307 (-0.001012795408007694) | > avg_loss_kl: 2.410116603676701 (+0.3867417406028415) | > avg_loss_feat: 3.6145172387781277 (+0.0083804970056236) | > avg_loss_mel: 18.053640983474086 (+1.3041152819781239) | > avg_loss_duration: 1.5767965199242175 (+0.04119015579492258) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.772139455231144 (+1.7394152627864372) | > avg_grad_norm_1: tensor(5443.5449, device='cuda:0') (tensor(-873.2451, device='cuda:0'))  > EPOCH: 1250/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:05:23)   --> TIME: 2025-02-25 21:05:48 -- STEP: 49/72 -- GLOBAL_STEP: 1090050 | > loss_disc: 2.800516128540039 (2.637818954428848) | > loss_disc_real_0: 0.33230912685394287 (0.1503024513320047) | > loss_disc_real_1: 0.1957058310508728 (0.21671006904572856) | > loss_disc_real_2: 0.23408277332782745 (0.27439335231878315) | > loss_disc_real_3: 0.24381250143051147 (0.22531652541793123) | > loss_disc_real_4: 0.25318169593811035 (0.24719275868668847) | > loss_disc_real_5: 0.24704661965370178 (0.2433801484959466) | > loss_0: 2.800516128540039 (2.637818954428848) | > grad_norm_0: tensor(400.3090, device='cuda:0') (tensor(233.6710, device='cuda:0')) | > loss_gen: 2.1146225929260254 (2.221107001207313) | > loss_kl: 2.0988879203796387 (2.0879843770241258) | > loss_feat: 3.222330093383789 (3.949562749084161) | > loss_mel: 16.94319725036621 (17.2075517420866) | > loss_duration: 1.593790054321289 (1.5286243351138369) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.972827911376953 (26.994829839589645) | > grad_norm_1: tensor(2089.1611, device='cuda:0') (tensor(3970.0420, device='cuda:0')) | > current_lr_0: 0.0001710673947340625 | > current_lr_1: 0.0001710673947340625 | > step_time: 0.4431 (0.44593358039855957) | > loader_time: 0.0099 (0.00772785167304837) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625634959046268 (-0.00024074232074576193) | > avg_step_time: 0.45692321280358544 (+0.0010214792170994969) | > avg_loss_disc: 2.6283925486282564 (-0.088055795347187) | > avg_loss_disc_real_0: 0.13727406470078818 (-0.05768974219829265) | > avg_loss_disc_real_1: 0.21971700288040538 (-0.007198109803065428) | > avg_loss_disc_real_2: 0.25588328107981617 (+0.015406147694923544) | > avg_loss_disc_real_3: 0.22630633392804106 (-0.016235366043910182) | > avg_loss_disc_real_4: 0.24694323203933072 (+0.005429874213648517) | > avg_loss_disc_real_5: 0.24120383720162888 (-0.0005888926311277964) | > avg_loss_0: 2.6283925486282564 (-0.088055795347187) | > avg_grad_norm_0: tensor(263.6208, device='cuda:0') (tensor(-4.9554, device='cuda:0')) | > avg_loss_gen: 2.163368337590929 (+0.0463004246563985) | > avg_loss_kl: 2.1036022834374872 (-0.30651432023921377) | > avg_loss_feat: 3.906214025658621 (+0.29169678688049316) | > avg_loss_mel: 17.09452191205091 (-0.9591190714231743) | > avg_loss_duration: 1.5457707307708095 (-0.031025789153408034) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.813477046053176 (-0.9586624091779683) | > avg_grad_norm_1: tensor(4967.7773, device='cuda:0') (tensor(-475.7676, device='cuda:0'))  > EPOCH: 1251/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:06:00)   --> TIME: 2025-02-25 21:06:14 -- STEP: 27/72 -- GLOBAL_STEP: 1090100 | > loss_disc: 2.390432357788086 (2.611364585381967) | > loss_disc_real_0: 0.041208792477846146 (0.16653737480993624) | > loss_disc_real_1: 0.22933411598205566 (0.2247244064454679) | > loss_disc_real_2: 0.2228199541568756 (0.23735562187654) | > loss_disc_real_3: 0.2206229567527771 (0.23812752630975512) | > loss_disc_real_4: 0.21224866807460785 (0.2411394593892274) | > loss_disc_real_5: 0.22898489236831665 (0.24257599258864368) | > loss_0: 2.390432357788086 (2.611364585381967) | > grad_norm_0: tensor(16.0853, device='cuda:0') (tensor(274.1739, device='cuda:0')) | > loss_gen: 2.3852055072784424 (2.121466261369211) | > loss_kl: 1.9559401273727417 (1.9166968884291473) | > loss_feat: 4.978574752807617 (3.8581290509965687) | > loss_mel: 17.063426971435547 (16.795305569966636) | > loss_duration: 1.4799550771713257 (1.4922093506212588) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.86310386657715 (26.183807161119248) | > grad_norm_1: tensor(3348.4558, device='cuda:0') (tensor(8000.4658, device='cuda:0')) | > current_lr_0: 0.00017104601130972074 | > current_lr_1: 0.00017104601130972074 | > step_time: 0.4472 (0.4343082198390254) | > loader_time: 0.0091 (0.0070016295821578416) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679158251050496 (+5.3523292004227854e-05) | > avg_step_time: 0.455004762595808 (-0.0019184502077774224) | > avg_loss_disc: 2.61611961311018 (-0.012272935518076622) | > avg_loss_disc_real_0: 0.14957039551415913 (+0.012296330813370948) | > avg_loss_disc_real_1: 0.22507955133914948 (+0.005362548458744093) | > avg_loss_disc_real_2: 0.23696001797494753 (-0.018923263104868637) | > avg_loss_disc_real_3: 0.23723653630471567 (+0.010930202376674608) | > avg_loss_disc_real_4: 0.24411401257548535 (-0.0028292194638453727) | > avg_loss_disc_real_5: 0.24443614629792496 (+0.003232309096296071) | > avg_loss_0: 2.61611961311018 (-0.012272935518076622) | > avg_grad_norm_0: tensor(283.3678, device='cuda:0') (+tensor(19.7470, device='cuda:0')) | > avg_loss_gen: 2.1159466874431563 (-0.04742165014777289) | > avg_loss_kl: 1.972856019584226 (-0.13074626385326127) | > avg_loss_feat: 3.7350123600221017 (-0.17120166563651917) | > avg_loss_mel: 16.566500166772116 (-0.5280217452787959) | > avg_loss_duration: 1.5344814549029713 (-0.011289275867838233) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.924796547688228 (-0.8886804983649483) | > avg_grad_norm_1: tensor(8259.9922, device='cuda:0') (+tensor(3292.2148, device='cuda:0'))  > EPOCH: 1252/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:06:37)   --> TIME: 2025-02-25 21:06:41 -- STEP: 5/72 -- GLOBAL_STEP: 1090150 | > loss_disc: 2.4523301124572754 (2.51117582321167) | > loss_disc_real_0: 0.0914582759141922 (0.09472984671592713) | > loss_disc_real_1: 0.2667903006076813 (0.21823875606060028) | > loss_disc_real_2: 0.22479359805583954 (0.22138675451278686) | > loss_disc_real_3: 0.23685874044895172 (0.22707494497299194) | > loss_disc_real_4: 0.2451847344636917 (0.23117376863956451) | > loss_disc_real_5: 0.24393725395202637 (0.23353528082370759) | > loss_0: 2.4523301124572754 (2.51117582321167) | > grad_norm_0: tensor(85.0336, device='cuda:0') (tensor(279.3221, device='cuda:0')) | > loss_gen: 2.2897417545318604 (2.14291398525238) | > loss_kl: 1.8436791896820068 (1.8852861166000365) | > loss_feat: 4.202746391296387 (4.091555500030518) | > loss_mel: 16.052310943603516 (16.568878173828125) | > loss_duration: 1.4689195156097412 (1.4930078029632567) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.857397079467773 (26.181640625) | > grad_norm_1: tensor(9799.9404, device='cuda:0') (tensor(10880.9463, device='cuda:0')) | > current_lr_0: 0.00017102463055830702 | > current_lr_1: 0.00017102463055830702 | > step_time: 0.4286 (0.43455986976623534) | > loader_time: 0.0059 (0.006949520111083985)  --> TIME: 2025-02-25 21:07:04 -- STEP: 55/72 -- GLOBAL_STEP: 1090200 | > loss_disc: 2.791290044784546 (2.59496340751648) | > loss_disc_real_0: 0.06177157908678055 (0.09829537292773073) | > loss_disc_real_1: 0.2786697745323181 (0.21585716422308576) | > loss_disc_real_2: 0.19475051760673523 (0.23182852214032953) | > loss_disc_real_3: 0.20888429880142212 (0.23282261301170695) | > loss_disc_real_4: 0.25477075576782227 (0.2424715353683992) | > loss_disc_real_5: 0.2739757001399994 (0.2427506750280207) | > loss_0: 2.791290044784546 (2.59496340751648) | > grad_norm_0: tensor(346.7461, device='cuda:0') (tensor(314.4982, device='cuda:0')) | > loss_gen: 2.037633180618286 (2.0973502180793075) | > loss_kl: 1.9542347192764282 (2.025178933143615) | > loss_feat: 3.1885130405426025 (3.9263504418459805) | > loss_mel: 16.164289474487305 (16.799015738747332) | > loss_duration: 1.597426176071167 (1.5242580673911355) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.942096710205078 (26.37215347290039) | > grad_norm_1: tensor(11620.6680, device='cuda:0') (tensor(9287.6191, device='cuda:0')) | > current_lr_0: 0.00017102463055830702 | > current_lr_1: 0.00017102463055830702 | > step_time: 0.4649 (0.44661432613026014) | > loader_time: 0.0094 (0.00799203785982999) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867669683107188 (-2.461419978614862e-06) | > avg_step_time: 0.45792403691251515 (+0.0029192743167071322) | > avg_loss_disc: 2.6011889988267924 (-0.014930614283387378) | > avg_loss_disc_real_0: 0.10057319732199252 (-0.04899719819216661) | > avg_loss_disc_real_1: 0.22467678496745272 (-0.00040276637169675555) | > avg_loss_disc_real_2: 0.23063667643238123 (-0.0063233415425663075) | > avg_loss_disc_real_3: 0.22855821608657567 (-0.008678320218139995) | > avg_loss_disc_real_4: 0.24283666845778346 (-0.0012773441177018896) | > avg_loss_disc_real_5: 0.24156097668996998 (-0.002875169607954975) | > avg_loss_0: 2.6011889988267924 (-0.014930614283387378) | > avg_grad_norm_0: tensor(331.7360, device='cuda:0') (+tensor(48.3682, device='cuda:0')) | > avg_loss_gen: 2.0895816289203277 (-0.026365058522828555) | > avg_loss_kl: 2.053717955737046 (+0.08086193615282022) | > avg_loss_feat: 3.9088912312413604 (+0.1738788712192587) | > avg_loss_mel: 16.83142793682259 (+0.2649277700504733) | > avg_loss_duration: 1.538107306184903 (+0.0036258512819318245) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.421726038758184 (+0.496929491069956) | > avg_grad_norm_1: tensor(9477.1367, device='cuda:0') (+tensor(1217.1445, device='cuda:0'))  > EPOCH: 1253/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:07:15)   --> TIME: 2025-02-25 21:07:31 -- STEP: 33/72 -- GLOBAL_STEP: 1090250 | > loss_disc: 2.349924087524414 (2.561986171838009) | > loss_disc_real_0: 0.0575762614607811 (0.10111487233503298) | > loss_disc_real_1: 0.23294158279895782 (0.2220869746172067) | > loss_disc_real_2: 0.23414884507656097 (0.24227075414224106) | > loss_disc_real_3: 0.21727244555950165 (0.2321134272849921) | > loss_disc_real_4: 0.2290298342704773 (0.24622333636789617) | > loss_disc_real_5: 0.2291075885295868 (0.24927819317037408) | > loss_0: 2.349924087524414 (2.561986171838009) | > grad_norm_0: tensor(154.9221, device='cuda:0') (tensor(255.3579, device='cuda:0')) | > loss_gen: 2.41640567779541 (2.2129916602914994) | > loss_kl: 2.097395420074463 (1.9451792493011013) | > loss_feat: 4.668519020080566 (4.04246086062807) | > loss_mel: 16.404436111450195 (16.623820825056594) | > loss_duration: 1.5161877870559692 (1.490693941260829) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.10294532775879 (26.31514647512725) | > grad_norm_1: tensor(6180.2217, device='cuda:0') (tensor(9648.9961, device='cuda:0')) | > current_lr_0: 0.00017100325247948722 | > current_lr_1: 0.00017100325247948722 | > step_time: 0.4366 (0.43778548818646057) | > loader_time: 0.0081 (0.007081703706221147) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568253315670389 (-0.00010844351540149205) | > avg_step_time: 0.4546872696406405 (-0.003236767271874663) | > avg_loss_disc: 2.5617616411665796 (-0.03942735766021288) | > avg_loss_disc_real_0: 0.09495795872324789 (-0.005615238598744632) | > avg_loss_disc_real_1: 0.22014053354800586 (-0.004536251419446857) | > avg_loss_disc_real_2: 0.2414021567559578 (+0.010765480323576576) | > avg_loss_disc_real_3: 0.2338923257841191 (+0.0053341096975434155) | > avg_loss_disc_real_4: 0.24665740258257154 (+0.003820734124788089) | > avg_loss_disc_real_5: 0.24608267381997176 (+0.004521697130001778) | > avg_loss_0: 2.5617616411665796 (-0.03942735766021288) | > avg_grad_norm_0: tensor(293.8851, device='cuda:0') (tensor(-37.8509, device='cuda:0')) | > avg_loss_gen: 2.1800810753459663 (+0.09049944642563856) | > avg_loss_kl: 1.9931224383098978 (-0.060595517427148415) | > avg_loss_feat: 3.956845390964562 (+0.047954159723201606) | > avg_loss_mel: 16.485099295495257 (-0.3463286413273323) | > avg_loss_duration: 1.5291744705656884 (-0.008932835619214652) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.144322596805196 (-0.2774034419529876) | > avg_grad_norm_1: tensor(10222.1162, device='cuda:0') (+tensor(744.9795, device='cuda:0'))  > EPOCH: 1254/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:07:52)   --> TIME: 2025-02-25 21:07:58 -- STEP: 11/72 -- GLOBAL_STEP: 1090300 | > loss_disc: 2.584689140319824 (2.504431052641435) | > loss_disc_real_0: 0.09537628293037415 (0.08380700139836832) | > loss_disc_real_1: 0.23001043498516083 (0.22121599993922494) | > loss_disc_real_2: 0.22050577402114868 (0.23174171827056192) | > loss_disc_real_3: 0.2428947240114212 (0.23507424918088046) | > loss_disc_real_4: 0.2591135799884796 (0.2369219105352055) | > loss_disc_real_5: 0.2511436343193054 (0.2473625662651929) | > loss_0: 2.584689140319824 (2.504431052641435) | > grad_norm_0: tensor(629.5854, device='cuda:0') (tensor(337.7411, device='cuda:0')) | > loss_gen: 1.985336422920227 (2.2078287493098867) | > loss_kl: 1.8700202703475952 (1.8495223955674605) | > loss_feat: 3.6694514751434326 (4.114353201606057) | > loss_mel: 16.992502212524414 (16.471327694979582) | > loss_duration: 1.4542639255523682 (1.4825320785695857) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.971574783325195 (26.12556370821866) | > grad_norm_1: tensor(13023.7793, device='cuda:0') (tensor(11128.3701, device='cuda:0')) | > current_lr_0: 0.00017098187707292726 | > current_lr_1: 0.00017098187707292726 | > step_time: 0.4179 (0.42621645060452545) | > loader_time: 0.0057 (0.006124778227372603)  --> TIME: 2025-02-25 21:08:21 -- STEP: 61/72 -- GLOBAL_STEP: 1090350 | > loss_disc: 2.7707128524780273 (2.5195446639764496) | > loss_disc_real_0: 0.11202167719602585 (0.08761213475563487) | > loss_disc_real_1: 0.18752798438072205 (0.2128000022446523) | > loss_disc_real_2: 0.19142891466617584 (0.22761630474543962) | > loss_disc_real_3: 0.22268757224082947 (0.23048997096350934) | > loss_disc_real_4: 0.22147305309772491 (0.23703511865412602) | > loss_disc_real_5: 0.255736768245697 (0.24252816717155645) | > loss_0: 2.7707128524780273 (2.5195446639764496) | > grad_norm_0: tensor(708.8154, device='cuda:0') (tensor(344.2278, device='cuda:0')) | > loss_gen: 1.9208276271820068 (2.2034731243477492) | > loss_kl: 1.9885330200195312 (1.943493667196055) | > loss_feat: 4.061967849731445 (4.1395887703192065) | > loss_mel: 15.459064483642578 (16.35955243032486) | > loss_duration: 1.5597978830337524 (1.51554117828119) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.990190505981445 (26.161649016083263) | > grad_norm_1: tensor(17241.6582, device='cuda:0') (tensor(11159.9600, device='cuda:0')) | > current_lr_0: 0.00017098187707292726 | > current_lr_1: 0.00017098187707292726 | > step_time: 0.4561 (0.44267173673285815) | > loader_time: 0.0111 (0.008096124305099737) --> EVAL PERFORMANCE | > avg_loader_time: 0.008595960240968514 (+2.7706925298125185e-05) | > avg_step_time: 0.4509771340329882 (-0.003710135607652265) | > avg_loss_disc: 2.518832465292702 (-0.04292917587387768) | > avg_loss_disc_real_0: 0.08700431962038428 (-0.007953639102863605) | > avg_loss_disc_real_1: 0.2142049535899095 (-0.005935579958096349) | > avg_loss_disc_real_2: 0.22863512517700732 (-0.012767031578950483) | > avg_loss_disc_real_3: 0.23253661478069468 (-0.001355711003424409) | > avg_loss_disc_real_4: 0.2388989400695747 (-0.007758462512996833) | > avg_loss_disc_real_5: 0.2422180717260065 (-0.003864602093965247) | > avg_loss_0: 2.518832465292702 (-0.04292917587387768) | > avg_grad_norm_0: tensor(333.2091, device='cuda:0') (+tensor(39.3240, device='cuda:0')) | > avg_loss_gen: 2.2128636669105206 (+0.03278259156455432) | > avg_loss_kl: 1.96800103993483 (-0.02512139837506777) | > avg_loss_feat: 4.131052208618381 (+0.17420681765381874) | > avg_loss_mel: 16.346328614463264 (-0.13877068103199264) | > avg_loss_duration: 1.525966098610784 (-0.003208371954904532) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.184211489180445 (+0.03988889237524873) | > avg_grad_norm_1: tensor(11079.5127, device='cuda:0') (+tensor(857.3965, device='cuda:0'))  > EPOCH: 1255/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:08:28)   --> TIME: 2025-02-25 21:08:47 -- STEP: 39/72 -- GLOBAL_STEP: 1090400 | > loss_disc: 2.4695866107940674 (2.5184434866293888) | > loss_disc_real_0: 0.11061610281467438 (0.09030852982631095) | > loss_disc_real_1: 0.21967944502830505 (0.22304125626881918) | > loss_disc_real_2: 0.2554168701171875 (0.22545512326252767) | > loss_disc_real_3: 0.22375299036502838 (0.22877553296394837) | > loss_disc_real_4: 0.23383185267448425 (0.2415037418787296) | > loss_disc_real_5: 0.255664199590683 (0.244864794306266) | > loss_0: 2.4695866107940674 (2.5184434866293888) | > grad_norm_0: tensor(311.9310, device='cuda:0') (tensor(342.8022, device='cuda:0')) | > loss_gen: 2.411268472671509 (2.2462741197683873) | > loss_kl: 1.829359531402588 (1.905047248571347) | > loss_feat: 4.419760704040527 (4.210295970623309) | > loss_mel: 16.121973037719727 (16.545082581348908) | > loss_duration: 1.5371315479278564 (1.4940764751189795) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.319494247436523 (26.40077630067483) | > grad_norm_1: tensor(9275.7725, device='cuda:0') (tensor(11147.2998, device='cuda:0')) | > current_lr_0: 0.00017096050433829314 | > current_lr_1: 0.00017096050433829314 | > step_time: 0.4488 (0.4328898833348201) | > loader_time: 0.008 (0.007271124766423152) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852257768872758 (-7.338255224093385e-05) | > avg_step_time: 0.450743920366529 (-0.00023321366645923103) | > avg_loss_disc: 2.5371637613001012 (+0.018331296007399356) | > avg_loss_disc_real_0: 0.0877241683782826 (+0.0007198487578983226) | > avg_loss_disc_real_1: 0.22279611733597768 (+0.008591163746068164) | > avg_loss_disc_real_2: 0.22978294483372863 (+0.0011478196567213106) | > avg_loss_disc_real_3: 0.2327552513337471 (+0.00021863655305243146) | > avg_loss_disc_real_4: 0.23996104677798044 (+0.00106210670840573) | > avg_loss_disc_real_5: 0.24377278542854416 (+0.0015547137025376523) | > avg_loss_0: 2.5371637613001012 (+0.018331296007399356) | > avg_grad_norm_0: tensor(332.1379, device='cuda:0') (tensor(-1.0712, device='cuda:0')) | > avg_loss_gen: 2.2174948269212758 (+0.004631160010755142) | > avg_loss_kl: 1.987622125048033 (+0.01962108511320304) | > avg_loss_feat: 4.0968081883981196 (-0.034244020220261184) | > avg_loss_mel: 16.504887916672395 (+0.15855930220913095) | > avg_loss_duration: 1.5282257560273291 (+0.0022596574165452132) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.335038708968906 (+0.1508272197884608) | > avg_grad_norm_1: tensor(10447.2549, device='cuda:0') (tensor(-632.2578, device='cuda:0'))  > EPOCH: 1256/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:09:05)   --> TIME: 2025-02-25 21:09:14 -- STEP: 17/72 -- GLOBAL_STEP: 1090450 | > loss_disc: 2.6419689655303955 (2.4982039647943832) | > loss_disc_real_0: 0.1024496778845787 (0.08391480914810125) | > loss_disc_real_1: 0.2906421422958374 (0.22147362109492807) | > loss_disc_real_2: 0.25512975454330444 (0.23460345671457403) | > loss_disc_real_3: 0.2572571337223053 (0.2368646395557067) | > loss_disc_real_4: 0.2721680998802185 (0.22946522866978364) | > loss_disc_real_5: 0.25507208704948425 (0.24830354574848623) | > loss_0: 2.6419689655303955 (2.4982039647943832) | > grad_norm_0: tensor(388.2256, device='cuda:0') (tensor(310.9099, device='cuda:0')) | > loss_gen: 2.185133457183838 (2.241575465482824) | > loss_kl: 2.0745394229888916 (1.8906451183206894) | > loss_feat: 3.7771248817443848 (4.17345264378716) | > loss_mel: 16.10624885559082 (16.711803941165698) | > loss_duration: 1.555625557899475 (1.4846996489693136) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.698671340942383 (26.502176733577954) | > grad_norm_1: tensor(12460.5254, device='cuda:0') (tensor(9599.6016, device='cuda:0')) | > current_lr_0: 0.00017093913427525086 | > current_lr_1: 0.00017093913427525086 | > step_time: 0.4418 (0.4293073345633114) | > loader_time: 0.0062 (0.00664220136754653)  --> TIME: 2025-02-25 21:09:38 -- STEP: 67/72 -- GLOBAL_STEP: 1090500 | > loss_disc: 2.7481374740600586 (2.5144202922707173) | > loss_disc_real_0: 0.09260837733745575 (0.08859708499330193) | > loss_disc_real_1: 0.20949696004390717 (0.2236087927177771) | > loss_disc_real_2: 0.2562611997127533 (0.23334445103780546) | > loss_disc_real_3: 0.26595109701156616 (0.23249733714915033) | > loss_disc_real_4: 0.23286432027816772 (0.2370707561720663) | > loss_disc_real_5: 0.24145200848579407 (0.24441156903309608) | > loss_0: 2.7481374740600586 (2.5144202922707173) | > grad_norm_0: tensor(491.1558, device='cuda:0') (tensor(310.2449, device='cuda:0')) | > loss_gen: 2.0483038425445557 (2.2630617049202986) | > loss_kl: 2.157465696334839 (1.9784496453270983) | > loss_feat: 3.7883424758911133 (4.139310448916992) | > loss_mel: 16.398147583007812 (16.516354176535536) | > loss_duration: 1.6254205703735352 (1.526544704366086) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.017681121826172 (26.42372082952243) | > grad_norm_1: tensor(11209.0459, device='cuda:0') (tensor(9944.4482, device='cuda:0')) | > current_lr_0: 0.00017093913427525086 | > current_lr_1: 0.00017093913427525086 | > step_time: 0.5276 (0.45055881898794603) | > loader_time: 0.012 (0.008447725381424178) --> EVAL PERFORMANCE | > avg_loader_time: 0.008615389676161215 (+9.28119874336348e-05) | > avg_step_time: 0.4615746484675878 (+0.010830728101058784) | > avg_loss_disc: 2.512324178722543 (-0.02483958257755825) | > avg_loss_disc_real_0: 0.08949197980929428 (+0.0017678114310116788) | > avg_loss_disc_real_1: 0.22396429617639999 (+0.001168178840422307) | > avg_loss_disc_real_2: 0.23177717814982776 (+0.001994233316099131) | > avg_loss_disc_real_3: 0.2327469337574193 (-8.317576327798726e-06) | > avg_loss_disc_real_4: 0.236196197254557 (-0.0037648495234234303) | > avg_loss_disc_real_5: 0.24411299509901396 (+0.0003402096704697988) | > avg_loss_0: 2.512324178722543 (-0.02483958257755825) | > avg_grad_norm_0: tensor(312.6306, device='cuda:0') (tensor(-19.5073, device='cuda:0')) | > avg_loss_gen: 2.2603774742341374 (+0.04288264731286162) | > avg_loss_kl: 1.99069104564022 (+0.0030689205921869966) | > avg_loss_feat: 4.134589638508542 (+0.037781450110422554) | > avg_loss_mel: 16.53256322296572 (+0.027675306293325264) | > avg_loss_duration: 1.5302034364619723 (+0.001977680434643192) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.4484249437359 (+0.11338623476699539) | > avg_grad_norm_1: tensor(10076.6738, device='cuda:0') (tensor(-370.5811, device='cuda:0'))  > EPOCH: 1257/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:09:42)   --> TIME: 2025-02-25 21:10:04 -- STEP: 45/72 -- GLOBAL_STEP: 1090550 | > loss_disc: 2.406979560852051 (2.5265438768598774) | > loss_disc_real_0: 0.11248116940259933 (0.08878172544969452) | > loss_disc_real_1: 0.21861867606639862 (0.21770306759410435) | > loss_disc_real_2: 0.19131046533584595 (0.23386098941167197) | > loss_disc_real_3: 0.2511242926120758 (0.22903058959378136) | > loss_disc_real_4: 0.2481815665960312 (0.23459463914235432) | > loss_disc_real_5: 0.2381589412689209 (0.2420008510351181) | > loss_0: 2.406979560852051 (2.5265438768598774) | > grad_norm_0: tensor(359.8428, device='cuda:0') (tensor(310.3603, device='cuda:0')) | > loss_gen: 2.411303997039795 (2.260284826490615) | > loss_kl: 2.000040292739868 (1.9704184055328369) | > loss_feat: 4.197250843048096 (4.252340539296466) | > loss_mel: 16.097179412841797 (16.36668383280436) | > loss_duration: 1.551416277885437 (1.5024712562561036) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.257192611694336 (26.35219891866048) | > grad_norm_1: tensor(7407.9829, device='cuda:0') (tensor(10274.0957, device='cuda:0')) | > current_lr_0: 0.00017091776688346643 | > current_lr_1: 0.00017091776688346643 | > step_time: 0.4601 (0.4391717857784695) | > loader_time: 0.0092 (0.007714298036363389) --> EVAL PERFORMANCE | > avg_loader_time: 0.008895239359895946 (+0.0002798496837347307) | > avg_step_time: 0.45268895592488034 (-0.008885692542707435) | > avg_loss_disc: 2.5216144306558963 (+0.00929025193335331) | > avg_loss_disc_real_0: 0.08943228350139 (-5.9696307904283374e-05) | > avg_loss_disc_real_1: 0.21905791969366475 (-0.004906376482735231) | > avg_loss_disc_real_2: 0.23262305221927 (+0.0008458740694422262) | > avg_loss_disc_real_3: 0.22953580991483072 (-0.0032111238425885924) | > avg_loss_disc_real_4: 0.2356474861292772 (-0.0005487111252798216) | > avg_loss_disc_real_5: 0.2418802859077991 (-0.0022327091912148767) | > avg_loss_0: 2.5216144306558963 (+0.00929025193335331) | > avg_grad_norm_0: tensor(312.2079, device='cuda:0') (tensor(-0.4227, device='cuda:0')) | > avg_loss_gen: 2.252305346475522 (-0.008072127758615544) | > avg_loss_kl: 2.0204298042915236 (+0.0297387586513036) | > avg_loss_feat: 4.187660119902916 (+0.05307048139437409) | > avg_loss_mel: 16.362983273788245 (-0.169579949177475) | > avg_loss_duration: 1.5297248128434304 (-0.0004786236185418957) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.35310342278279 (-0.0953215209531102) | > avg_grad_norm_1: tensor(10214.2764, device='cuda:0') (+tensor(137.6025, device='cuda:0'))  > EPOCH: 1258/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:10:19)   --> TIME: 2025-02-25 21:10:31 -- STEP: 23/72 -- GLOBAL_STEP: 1090600 | > loss_disc: 2.653592824935913 (2.5147608155789585) | > loss_disc_real_0: 0.12476607412099838 (0.08982866306019865) | > loss_disc_real_1: 0.17654916644096375 (0.22374277723872144) | > loss_disc_real_2: 0.25010138750076294 (0.22588319946890292) | > loss_disc_real_3: 0.2387552559375763 (0.23042219812455383) | > loss_disc_real_4: 0.2635008692741394 (0.2336198609808217) | > loss_disc_real_5: 0.22735996544361115 (0.24413334869820139) | > loss_0: 2.653592824935913 (2.5147608155789585) | > grad_norm_0: tensor(390.8902, device='cuda:0') (tensor(334.4103, device='cuda:0')) | > loss_gen: 2.0835423469543457 (2.2824255487193232) | > loss_kl: 1.9046282768249512 (1.957338405692059) | > loss_feat: 4.040448188781738 (4.303673381390779) | > loss_mel: 16.636137008666992 (16.76185301075811) | > loss_duration: 1.517582654953003 (1.4917310320812722) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.182336807250977 (26.79702136827552) | > grad_norm_1: tensor(9259.7051, device='cuda:0') (tensor(10172.2129, device='cuda:0')) | > current_lr_0: 0.00017089640216260598 | > current_lr_1: 0.00017089640216260598 | > step_time: 0.439 (0.42823348874631134) | > loader_time: 0.0072 (0.0069766666578209915) --> EVAL PERFORMANCE | > avg_loader_time: 0.00873938412733481 (-0.00015585523256113516) | > avg_step_time: 0.44916490769722095 (-0.003524048227659393) | > avg_loss_disc: 2.5222512097425858 (+0.0006367790866894651) | > avg_loss_disc_real_0: 0.08734709465167895 (-0.0020851888497110532) | > avg_loss_disc_real_1: 0.22380203029639284 (+0.0047441106027280855) | > avg_loss_disc_real_2: 0.22320784775304123 (-0.00941520446622876) | > avg_loss_disc_real_3: 0.2297120474173989 (+0.00017623750256817727) | > avg_loss_disc_real_4: 0.23745583795325856 (+0.001808351823981369) | > avg_loss_disc_real_5: 0.2448558595398782 (+0.0029755736320791204) | > avg_loss_0: 2.5222512097425858 (+0.0006367790866894651) | > avg_grad_norm_0: tensor(327.1965, device='cuda:0') (+tensor(14.9886, device='cuda:0')) | > avg_loss_gen: 2.2642606899771893 (+0.011955343501667492) | > avg_loss_kl: 2.0293883253151264 (+0.008958521023602728) | > avg_loss_feat: 4.2006417731164225 (+0.012981653213506306) | > avg_loss_mel: 16.68717460900965 (+0.3241913352214034) | > avg_loss_duration: 1.531758192559363 (+0.0020333797159326483) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.713223497632523 (+0.36012007484973196) | > avg_grad_norm_1: tensor(10112.4629, device='cuda:0') (tensor(-101.8135, device='cuda:0'))  > EPOCH: 1259/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:10:55)   --> TIME: 2025-02-25 21:10:58 -- STEP: 1/72 -- GLOBAL_STEP: 1090650 | > loss_disc: 2.822636127471924 (2.822636127471924) | > loss_disc_real_0: 0.06440378725528717 (0.06440378725528717) | > loss_disc_real_1: 0.24431736767292023 (0.24431736767292023) | > loss_disc_real_2: 0.24031732976436615 (0.24031732976436615) | > loss_disc_real_3: 0.26461565494537354 (0.26461565494537354) | > loss_disc_real_4: 0.2409512996673584 (0.2409512996673584) | > loss_disc_real_5: 0.23212458193302155 (0.23212458193302155) | > loss_0: 2.822636127471924 (2.822636127471924) | > grad_norm_0: tensor(515.8926, device='cuda:0') (tensor(515.8926, device='cuda:0')) | > loss_gen: 2.0083580017089844 (2.0083580017089844) | > loss_kl: 1.8572982549667358 (1.8572982549667358) | > loss_feat: 3.6137754917144775 (3.6137754917144775) | > loss_mel: 16.63121223449707 (16.63121223449707) | > loss_duration: 1.490311622619629 (1.490311622619629) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.600955963134766 (25.600955963134766) | > grad_norm_1: tensor(15537.1973, device='cuda:0') (tensor(15537.1973, device='cuda:0')) | > current_lr_0: 0.00017087504011233565 | > current_lr_1: 0.00017087504011233565 | > step_time: 0.4248 (0.42479968070983887) | > loader_time: 0.0056 (0.005568027496337891)  --> TIME: 2025-02-25 21:11:21 -- STEP: 51/72 -- GLOBAL_STEP: 1090700 | > loss_disc: 2.392241954803467 (2.5698631370768816) | > loss_disc_real_0: 0.16377274692058563 (0.09737330171115256) | > loss_disc_real_1: 0.16804853081703186 (0.24201656293635274) | > loss_disc_real_2: 0.20831948518753052 (0.22503581730758443) | > loss_disc_real_3: 0.21747742593288422 (0.23753849197836482) | > loss_disc_real_4: 0.19389499723911285 (0.23935980452042) | > loss_disc_real_5: 0.2738650441169739 (0.2435917836778304) | > loss_0: 2.392241954803467 (2.5698631370768816) | > grad_norm_0: tensor(577.1616, device='cuda:0') (tensor(328.0808, device='cuda:0')) | > loss_gen: 2.4584851264953613 (2.279394846336515) | > loss_kl: 2.13484787940979 (1.9930455637913125) | > loss_feat: 4.7135162353515625 (4.186332309947295) | > loss_mel: 17.02446937561035 (16.75364036186069) | > loss_duration: 1.5398495197296143 (1.5095092362048579) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.87116813659668 (26.72192221996831) | > grad_norm_1: tensor(8604.0107, device='cuda:0') (tensor(9809.5430, device='cuda:0')) | > current_lr_0: 0.00017087504011233565 | > current_lr_1: 0.00017087504011233565 | > step_time: 0.4413 (0.445714791615804) | > loader_time: 0.0088 (0.007662272920795516) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542299270629883 (-0.00019708485670492763) | > avg_step_time: 0.4579968888994674 (+0.00883198120224643) | > avg_loss_disc: 2.557451859326429 (+0.03520064958384328) | > avg_loss_disc_real_0: 0.09446328128097761 (+0.007116186629298665) | > avg_loss_disc_real_1: 0.23509730335692286 (+0.01129527306053002) | > avg_loss_disc_real_2: 0.2262784200654903 (+0.0030705723124490603) | > avg_loss_disc_real_3: 0.23728771255889408 (+0.007575665141495186) | > avg_loss_disc_real_4: 0.2390953002261444 (+0.0016394622728858332) | > avg_loss_disc_real_5: 0.24233591388648665 (-0.0025199456533915587) | > avg_loss_0: 2.557451859326429 (+0.03520064958384328) | > avg_grad_norm_0: tensor(327.2470, device='cuda:0') (+tensor(0.0505, device='cuda:0')) | > avg_loss_gen: 2.2638464695970786 (-0.00041422038011074136) | > avg_loss_kl: 2.032577425661222 (+0.0031891003460957634) | > avg_loss_feat: 4.139608107822042 (-0.06103366529438059) | > avg_loss_mel: 16.732035892110474 (+0.04486128310082549) | > avg_loss_duration: 1.528586187832792 (-0.003172004726571087) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.696654064554565 (-0.016569433077957996) | > avg_grad_norm_1: tensor(9714.1152, device='cuda:0') (tensor(-398.3477, device='cuda:0'))  > EPOCH: 1260/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:11:33)   --> TIME: 2025-02-25 21:11:48 -- STEP: 29/72 -- GLOBAL_STEP: 1090750 | > loss_disc: 2.56746244430542 (2.514004896427023) | > loss_disc_real_0: 0.07879884541034698 (0.09979950893541863) | > loss_disc_real_1: 0.25789839029312134 (0.2164359812078805) | > loss_disc_real_2: 0.1967237889766693 (0.2277925317657405) | > loss_disc_real_3: 0.21808025240898132 (0.23578206064372226) | > loss_disc_real_4: 0.224498450756073 (0.24524887471363463) | > loss_disc_real_5: 0.2234313189983368 (0.24424950367417828) | > loss_0: 2.56746244430542 (2.514004896427023) | > grad_norm_0: tensor(296.3856, device='cuda:0') (tensor(325.1437, device='cuda:0')) | > loss_gen: 2.010552406311035 (2.276756599031646) | > loss_kl: 2.072735071182251 (2.023757511171801) | > loss_feat: 4.257752895355225 (4.168855560236964) | > loss_mel: 17.88559341430664 (16.55719707752096) | > loss_duration: 1.5183095932006836 (1.489790287511102) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.744945526123047 (26.51635689570986) | > grad_norm_1: tensor(8688.5156, device='cuda:0') (tensor(8893.7461, device='cuda:0')) | > current_lr_0: 0.00017085368073232158 | > current_lr_1: 0.00017085368073232158 | > step_time: 0.4327 (0.4361445821564773) | > loader_time: 0.0081 (0.006872382657281284) --> EVAL PERFORMANCE | > avg_loader_time: 0.008496388583116129 (-4.591068751375396e-05) | > avg_step_time: 0.4533221889549578 (-0.004674699944509586) | > avg_loss_disc: 2.5252664425003695 (-0.032185416826059576) | > avg_loss_disc_real_0: 0.09892711610021726 (+0.004463834819239648) | > avg_loss_disc_real_1: 0.2220200713671429 (-0.013077231989779947) | > avg_loss_disc_real_2: 0.2261768986641521 (-0.00010152140133817822) | > avg_loss_disc_real_3: 0.2349825123666038 (-0.0023052001922902743) | > avg_loss_disc_real_4: 0.2409605320910333 (+0.0018652318648889055) | > avg_loss_disc_real_5: 0.24459098846140043 (+0.002255074574913779) | > avg_loss_0: 2.5252664425003695 (-0.032185416826059576) | > avg_grad_norm_0: tensor(302.6816, device='cuda:0') (tensor(-24.5654, device='cuda:0')) | > avg_loss_gen: 2.262017473368577 (-0.0018289962285016337) | > avg_loss_kl: 2.1782173153380278 (+0.14563988967680563) | > avg_loss_feat: 4.133628600080247 (-0.005979507741795231) | > avg_loss_mel: 16.9290030036174 (+0.19696711150692536) | > avg_loss_duration: 1.5463906761626123 (+0.01780448832982029) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.04925706352986 (+0.35260299897529634) | > avg_grad_norm_1: tensor(8650.2012, device='cuda:0') (tensor(-1063.9141, device='cuda:0'))  > EPOCH: 1261/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:12:09)   --> TIME: 2025-02-25 21:12:14 -- STEP: 7/72 -- GLOBAL_STEP: 1090800 | > loss_disc: 2.545396089553833 (2.515127726963588) | > loss_disc_real_0: 0.130536288022995 (0.0957465081342629) | > loss_disc_real_1: 0.2202918529510498 (0.23629205354622432) | > loss_disc_real_2: 0.22499538958072662 (0.23475397058895656) | > loss_disc_real_3: 0.23100630939006805 (0.2280436669077192) | > loss_disc_real_4: 0.22486892342567444 (0.23933930482183183) | > loss_disc_real_5: 0.23813804984092712 (0.2487443174634661) | > loss_0: 2.545396089553833 (2.515127726963588) | > grad_norm_0: tensor(333.6694, device='cuda:0') (tensor(335.4705, device='cuda:0')) | > loss_gen: 2.2278494834899902 (2.278446980885097) | > loss_kl: 2.178851842880249 (1.936326163155692) | > loss_feat: 3.9037673473358154 (4.136079617909023) | > loss_mel: 16.148874282836914 (16.793394769941056) | > loss_duration: 1.4970943927764893 (1.4923778942653112) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.956438064575195 (26.63662474496024) | > grad_norm_1: tensor(6587.8110, device='cuda:0') (tensor(8980.5352, device='cuda:0')) | > current_lr_0: 0.00017083232402223005 | > current_lr_1: 0.00017083232402223005 | > step_time: 0.4223 (0.42542147636413574) | > loader_time: 0.0054 (0.00630572863987514)  --> TIME: 2025-02-25 21:12:37 -- STEP: 57/72 -- GLOBAL_STEP: 1090850 | > loss_disc: 2.485562324523926 (2.5604259173075357) | > loss_disc_real_0: 0.06491045653820038 (0.11554812443883795) | > loss_disc_real_1: 0.2025790512561798 (0.2270362659504539) | > loss_disc_real_2: 0.21154168248176575 (0.22791689712750285) | > loss_disc_real_3: 0.19410379230976105 (0.23231880811222813) | > loss_disc_real_4: 0.2312411367893219 (0.24071786304314932) | > loss_disc_real_5: 0.2381054013967514 (0.24687152734973974) | > loss_0: 2.485562324523926 (2.5604259173075357) | > grad_norm_0: tensor(169.7789, device='cuda:0') (tensor(333.9293, device='cuda:0')) | > loss_gen: 2.3801004886627197 (2.2338460014577497) | > loss_kl: 2.0543599128723145 (1.9947209651010078) | > loss_feat: 5.050239086151123 (4.004657854113661) | > loss_mel: 16.389556884765625 (16.616410138314233) | > loss_duration: 1.576292872428894 (1.520293386358964) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.45054817199707 (26.369928192674067) | > grad_norm_1: tensor(9793.1094, device='cuda:0') (tensor(9021.8857, device='cuda:0')) | > current_lr_0: 0.00017083232402223005 | > current_lr_1: 0.00017083232402223005 | > step_time: 0.4542 (0.44339574429026823) | > loader_time: 0.01 (0.007831728249265436) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850207033291669 (+5.681749800560618e-06) | > avg_step_time: 0.45362733787214254 (+0.0003051489171847499) | > avg_loss_disc: 2.5572809971554173 (+0.03201455465504788) | > avg_loss_disc_real_0: 0.11027083907958489 (+0.01134372297936763) | > avg_loss_disc_real_1: 0.22779868899936406 (+0.00577861763222115) | > avg_loss_disc_real_2: 0.2324320369623077 (+0.006255138298155577) | > avg_loss_disc_real_3: 0.23365618267529448 (-0.0013263296913093303) | > avg_loss_disc_real_4: 0.24137126790805602 (+0.00041073581702272266) | > avg_loss_disc_real_5: 0.24682774887958042 (+0.002236760418179995) | > avg_loss_0: 2.5572809971554173 (+0.03201455465504788) | > avg_grad_norm_0: tensor(331.5233, device='cuda:0') (+tensor(28.8417, device='cuda:0')) | > avg_loss_gen: 2.2386536531045405 (-0.023363820264036494) | > avg_loss_kl: 2.0227736687996023 (-0.15544364653842546) | > avg_loss_feat: 4.003096271568619 (-0.13053232851162733) | > avg_loss_mel: 16.6349193143173 (-0.2940836893000984) | > avg_loss_duration: 1.5339490443887849 (-0.0124416317738274) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.433391732229314 (-0.6158653313005473) | > avg_grad_norm_1: tensor(9098.3379, device='cuda:0') (+tensor(448.1367, device='cuda:0'))  > EPOCH: 1262/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:12:46)   --> TIME: 2025-02-25 21:13:04 -- STEP: 35/72 -- GLOBAL_STEP: 1090900 | > loss_disc: 2.6155314445495605 (2.5270650999886644) | > loss_disc_real_0: 0.15210166573524475 (0.09333580732345581) | > loss_disc_real_1: 0.21075676381587982 (0.22659762970038821) | > loss_disc_real_2: 0.23665323853492737 (0.22270742186478207) | > loss_disc_real_3: 0.26788759231567383 (0.2279777228832245) | > loss_disc_real_4: 0.25883615016937256 (0.24284956880978176) | > loss_disc_real_5: 0.22435925900936127 (0.2406487728868212) | > loss_0: 2.6155314445495605 (2.5270650999886644) | > grad_norm_0: tensor(461.4861, device='cuda:0') (tensor(329.3773, device='cuda:0')) | > loss_gen: 2.0767171382904053 (2.2633447238377165) | > loss_kl: 2.1116647720336914 (1.9336763075419834) | > loss_feat: 3.685558795928955 (4.286162233352661) | > loss_mel: 17.15411376953125 (16.66615685054234) | > loss_duration: 1.5334067344665527 (1.4894548892974853) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.56146240234375 (26.638795035226003) | > grad_norm_1: tensor(10232.1953, device='cuda:0') (tensor(10139.8057, device='cuda:0')) | > current_lr_0: 0.00017081096998172726 | > current_lr_1: 0.00017081096998172726 | > step_time: 0.4374 (0.4354478222983224) | > loader_time: 0.0085 (0.007102067129952567) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568716720795967 (+6.664638787927737e-05) | > avg_step_time: 0.46314800625116054 (+0.009520668379017994) | > avg_loss_disc: 2.5430128406470924 (-0.014268156508324914) | > avg_loss_disc_real_0: 0.09206611138414329 (-0.018204727695441597) | > avg_loss_disc_real_1: 0.22558240991243175 (-0.00221627908693231) | > avg_loss_disc_real_2: 0.23046590912509973 (-0.001966127837207954) | > avg_loss_disc_real_3: 0.23598576641418564 (+0.0023295837388911667) | > avg_loss_disc_real_4: 0.24312979213788477 (+0.0017585242298287507) | > avg_loss_disc_real_5: 0.24272779826547058 (-0.004099950614109837) | > avg_loss_0: 2.5430128406470924 (-0.014268156508324914) | > avg_grad_norm_0: tensor(327.0598, device='cuda:0') (tensor(-4.4635, device='cuda:0')) | > avg_loss_gen: 2.2482134993647174 (+0.009559846260176919) | > avg_loss_kl: 1.9905772125217276 (-0.03219645627787471) | > avg_loss_feat: 4.164535549325004 (+0.16143927775638467) | > avg_loss_mel: 16.578125725329766 (-0.05679358898753506) | > avg_loss_duration: 1.527516198829866 (-0.006432845558918876) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.508968084630833 (+0.07557635240151939) | > avg_grad_norm_1: tensor(9835.2842, device='cuda:0') (+tensor(736.9463, device='cuda:0'))  > EPOCH: 1263/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:13:24)   --> TIME: 2025-02-25 21:13:31 -- STEP: 13/72 -- GLOBAL_STEP: 1090950 | > loss_disc: 2.612391471862793 (2.5316331019768348) | > loss_disc_real_0: 0.08211198449134827 (0.09131062546601662) | > loss_disc_real_1: 0.16788621246814728 (0.2150870401125688) | > loss_disc_real_2: 0.2449844926595688 (0.23167948424816132) | > loss_disc_real_3: 0.2196032702922821 (0.2241774728664985) | > loss_disc_real_4: 0.24792689085006714 (0.23935652466920707) | > loss_disc_real_5: 0.24988262355327606 (0.24215323420671317) | > loss_0: 2.612391471862793 (2.5316331019768348) | > grad_norm_0: tensor(276.5882, device='cuda:0') (tensor(357.7549, device='cuda:0')) | > loss_gen: 2.154198169708252 (2.2188597275660586) | > loss_kl: 1.7422642707824707 (1.8732177110818715) | > loss_feat: 4.3128981590271 (4.161719633982732) | > loss_mel: 16.904298782348633 (16.708361405592694) | > loss_duration: 1.4435310363769531 (1.4813625537432158) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.55718994140625 (26.443520619319035) | > grad_norm_1: tensor(9149.6406, device='cuda:0') (tensor(9085.4873, device='cuda:0')) | > current_lr_0: 0.00017078961861047954 | > current_lr_1: 0.00017078961861047954 | > step_time: 0.4621 (0.42998035137469953) | > loader_time: 0.0058 (0.006395725103525015)  --> TIME: 2025-02-25 21:13:55 -- STEP: 63/72 -- GLOBAL_STEP: 1091000 | > loss_disc: 2.3881545066833496 (2.5246974702865357) | > loss_disc_real_0: 0.06026548519730568 (0.101894932547732) | > loss_disc_real_1: 0.20969600975513458 (0.21810712795408946) | > loss_disc_real_2: 0.21228374540805817 (0.2301142882733118) | > loss_disc_real_3: 0.217147096991539 (0.23127170947809067) | > loss_disc_real_4: 0.22638118267059326 (0.2399599552154541) | > loss_disc_real_5: 0.2251649647951126 (0.2420470378701649) | > loss_0: 2.3881545066833496 (2.5246974702865357) | > grad_norm_0: tensor(272.5087, device='cuda:0') (tensor(305.2254, device='cuda:0')) | > loss_gen: 2.2201087474823 (2.266988179040334) | > loss_kl: 2.109361410140991 (2.0441801812913694) | > loss_feat: 4.509922027587891 (4.181002098416523) | > loss_mel: 17.634923934936523 (16.668838894556433) | > loss_duration: 1.5620527267456055 (1.52402182798537) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.03636932373047 (26.68503089178176) | > grad_norm_1: tensor(7796.7505, device='cuda:0') (tensor(7821.8174, device='cuda:0')) | > current_lr_0: 0.00017078961861047954 | > current_lr_1: 0.00017078961861047954 | > step_time: 0.5041 (0.44686821150401285) | > loader_time: 0.0111 (0.008188788852994405) --> EVAL PERFORMANCE | > avg_loader_time: 0.008553562029986314 (-1.5154690809652399e-05) | > avg_step_time: 0.45535718219380983 (-0.007790824057350709) | > avg_loss_disc: 2.5250806472670866 (-0.017932193380005845) | > avg_loss_disc_real_0: 0.10139462388527225 (+0.009328512501128955) | > avg_loss_disc_real_1: 0.2182909646924113 (-0.007291445220020454) | > avg_loss_disc_real_2: 0.23010841851503078 (-0.00035749061006895566) | > avg_loss_disc_real_3: 0.23083786897256342 (-0.005147897441622223) | > avg_loss_disc_real_4: 0.23980952106731038 (-0.003320271070574393) | > avg_loss_disc_real_5: 0.2408076864732823 (-0.0019201117921882893) | > avg_loss_0: 2.5250806472670866 (-0.017932193380005845) | > avg_grad_norm_0: tensor(296.1446, device='cuda:0') (tensor(-30.9152, device='cuda:0')) | > avg_loss_gen: 2.2660484078904277 (+0.017834908525710347) | > avg_loss_kl: 2.060654707357918 (+0.07007749483619041) | > avg_loss_feat: 4.175835837780588 (+0.011300288455584173) | > avg_loss_mel: 16.6823596685705 (+0.10423394324073243) | > avg_loss_duration: 1.5335290180125707 (+0.006012819182704687) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.718427362576335 (+0.209459277945502) | > avg_grad_norm_1: tensor(7543.8379, device='cuda:0') (tensor(-2291.4463, device='cuda:0'))  > EPOCH: 1264/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:14:01)   --> TIME: 2025-02-25 21:14:21 -- STEP: 41/72 -- GLOBAL_STEP: 1091050 | > loss_disc: 2.5416293144226074 (2.8307320839021264) | > loss_disc_real_0: 0.03684873878955841 (0.11934237936284484) | > loss_disc_real_1: 0.3385492265224457 (0.20698469004979947) | > loss_disc_real_2: 0.21551525592803955 (0.32774982147100495) | > loss_disc_real_3: 0.21748192608356476 (0.23862663747333898) | > loss_disc_real_4: 0.2737656235694885 (0.28879963588423835) | > loss_disc_real_5: 0.2439502626657486 (0.24757388052417012) | > loss_0: 2.5416293144226074 (2.8307320839021264) | > grad_norm_0: tensor(327.3156, device='cuda:0') (tensor(272.1802, device='cuda:0')) | > loss_gen: 2.3132851123809814 (2.156001221842882) | > loss_kl: 2.438849449157715 (2.130255053683025) | > loss_feat: 3.7773590087890625 (3.91375214297597) | > loss_mel: 17.58879852294922 (17.143049844881382) | > loss_duration: 1.5646066665649414 (1.5111582424582504) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.682899475097656 (26.854216598882907) | > grad_norm_1: tensor(5806.0103, device='cuda:0') (tensor(7483.2095, device='cuda:0')) | > current_lr_0: 0.00017076826990815322 | > current_lr_1: 0.00017076826990815322 | > step_time: 0.4598 (0.4411477984451666) | > loader_time: 0.0094 (0.007472451140240925) --> EVAL PERFORMANCE | > avg_loader_time: 0.008623059366790345 (+6.949733680403081e-05) | > avg_step_time: 0.4550210388613419 (-0.000336143332467953) | > avg_loss_disc: 2.811500999289499 (+0.28642035202241223) | > avg_loss_disc_real_0: 0.16547044135735067 (+0.06407581747207843) | > avg_loss_disc_real_1: 0.210757777094841 (-0.0075331875975702955) | > avg_loss_disc_real_2: 0.2834492936520509 (+0.05334087513702013) | > avg_loss_disc_real_3: 0.24233429448705324 (+0.011496425514489816) | > avg_loss_disc_real_4: 0.271641174252604 (+0.0318316531852936) | > avg_loss_disc_real_5: 0.2431313345130061 (+0.0023236480397237913) | > avg_loss_0: 2.811500999289499 (+0.28642035202241223) | > avg_grad_norm_0: tensor(278.2026, device='cuda:0') (tensor(-17.9420, device='cuda:0')) | > avg_loss_gen: 2.090521380934917 (-0.17552702695551092) | > avg_loss_kl: 2.2049003789122676 (+0.14424567155434964) | > avg_loss_feat: 3.7493862803553193 (-0.4264495574252689) | > avg_loss_mel: 17.179103931910554 (+0.49674426334005517) | > avg_loss_duration: 1.5452798923975986 (+0.011750874385027865) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.769191876263687 (+0.050764513687351354) | > avg_grad_norm_1: tensor(6519.8560, device='cuda:0') (tensor(-1023.9819, device='cuda:0'))  > EPOCH: 1265/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:14:38)   --> TIME: 2025-02-25 21:14:48 -- STEP: 19/72 -- GLOBAL_STEP: 1091100 | > loss_disc: 2.583712577819824 (2.5786166693034924) | > loss_disc_real_0: 0.0588671900331974 (0.11574280144352662) | > loss_disc_real_1: 0.22332176566123962 (0.21268280164191597) | > loss_disc_real_2: 0.2444939762353897 (0.24132070102189718) | > loss_disc_real_3: 0.22766248881816864 (0.23965640052368767) | > loss_disc_real_4: 0.2522174119949341 (0.2487084238152755) | > loss_disc_real_5: 0.2503950595855713 (0.2530866053543593) | > loss_0: 2.583712577819824 (2.5786166693034924) | > grad_norm_0: tensor(130.3293, device='cuda:0') (tensor(243.2973, device='cuda:0')) | > loss_gen: 2.1792008876800537 (2.2074922009518274) | > loss_kl: 1.9451332092285156 (2.000761678344325) | > loss_feat: 3.6006665229797363 (3.957969866300884) | > loss_mel: 15.762334823608398 (16.799274996707315) | > loss_duration: 1.4711977243423462 (1.4928973787709285) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.958532333374023 (26.458395907753392) | > grad_norm_1: tensor(5838.5698, device='cuda:0') (tensor(7376.8330, device='cuda:0')) | > current_lr_0: 0.00017074692387441468 | > current_lr_1: 0.00017074692387441468 | > step_time: 0.4331 (0.4338863899833278) | > loader_time: 0.0062 (0.006814755891498767)  --> TIME: 2025-02-25 21:15:12 -- STEP: 69/72 -- GLOBAL_STEP: 1091150 | > loss_disc: 2.477555751800537 (2.5878426268480825) | > loss_disc_real_0: 0.051667220890522 (0.09041655015038408) | > loss_disc_real_1: 0.21558928489685059 (0.23618005971977676) | > loss_disc_real_2: 0.23851972818374634 (0.24010591057763583) | > loss_disc_real_3: 0.22930069267749786 (0.23445447070011194) | > loss_disc_real_4: 0.2315230667591095 (0.2445143722537635) | > loss_disc_real_5: 0.2636457085609436 (0.24514279438965564) | > loss_0: 2.477555751800537 (2.5878426268480825) | > grad_norm_0: tensor(106.5159, device='cuda:0') (tensor(287.9365, device='cuda:0')) | > loss_gen: 2.2399489879608154 (2.161298636077107) | > loss_kl: 2.183241605758667 (2.0240743661272353) | > loss_feat: 4.16359281539917 (3.9377386880957563) | > loss_mel: 16.636926651000977 (16.549887823021916) | > loss_duration: 1.644261360168457 (1.5320255514504253) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.867969512939453 (26.205024885094684) | > grad_norm_1: tensor(4650.1050, device='cuda:0') (tensor(9131.0840, device='cuda:0')) | > current_lr_0: 0.00017074692387441468 | > current_lr_1: 0.00017074692387441468 | > step_time: 0.5272 (0.4610985569331957) | > loader_time: 0.0134 (0.008568010468413868) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624264891718478 (+1.2055249281328617e-06) | > avg_step_time: 0.4624194628755811 (+0.0073984240142392355) | > avg_loss_disc: 2.587067308560224 (-0.224433690729275) | > avg_loss_disc_real_0: 0.0894102641237034 (-0.07606017723364727) | > avg_loss_disc_real_1: 0.2356275504743549 (+0.024869773379513904) | > avg_loss_disc_real_2: 0.2397239025629742 (-0.04372539108907669) | > avg_loss_disc_real_3: 0.23385988133893887 (-0.008474413148114368) | > avg_loss_disc_real_4: 0.24395481486555556 (-0.027686359387048415) | > avg_loss_disc_real_5: 0.24546819484569657 (+0.0023368603326904824) | > avg_loss_0: 2.587067308560224 (-0.224433690729275) | > avg_grad_norm_0: tensor(297.7510, device='cuda:0') (+tensor(19.5484, device='cuda:0')) | > avg_loss_gen: 2.1564040570191936 (+0.06588267608427678) | > avg_loss_kl: 2.0259285946966896 (-0.17897178421557802) | > avg_loss_feat: 3.9363506108942166 (+0.18696433053889727) | > avg_loss_mel: 16.57336800534959 (-0.6057359265609641) | > avg_loss_duration: 1.532781310484443 (-0.01249858191315556) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.224832373605647 (-0.5443595026580397) | > avg_grad_norm_1: tensor(9249.5752, device='cuda:0') (+tensor(2729.7192, device='cuda:0'))  > EPOCH: 1266/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:15:15)   --> TIME: 2025-02-25 21:15:39 -- STEP: 47/72 -- GLOBAL_STEP: 1091200 | > loss_disc: 2.4574456214904785 (2.540793434102484) | > loss_disc_real_0: 0.056692976504564285 (0.08847189036772606) | > loss_disc_real_1: 0.20401298999786377 (0.21329792034118739) | > loss_disc_real_2: 0.28003910183906555 (0.22990211590807488) | > loss_disc_real_3: 0.20489227771759033 (0.23545639692468845) | > loss_disc_real_4: 0.24112214148044586 (0.23694047236696203) | > loss_disc_real_5: 0.24307586252689362 (0.23663806661646417) | > loss_0: 2.4574456214904785 (2.540793434102484) | > grad_norm_0: tensor(96.0984, device='cuda:0') (tensor(340.4669, device='cuda:0')) | > loss_gen: 2.241471529006958 (2.1880255557121115) | > loss_kl: 2.001390218734741 (1.9712227963386697) | > loss_feat: 3.9307632446289062 (4.141780817762334) | > loss_mel: 15.727662086486816 (16.569393340577474) | > loss_duration: 1.5503734350204468 (1.502250039831121) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.451662063598633 (26.372672304194023) | > grad_norm_1: tensor(7713.3447, device='cuda:0') (tensor(9293.2998, device='cuda:0')) | > current_lr_0: 0.00017072558050893038 | > current_lr_1: 0.00017072558050893038 | > step_time: 0.4465 (0.4444461680473165) | > loader_time: 0.0097 (0.007840861665441635) --> EVAL PERFORMANCE | > avg_loader_time: 0.008921468761605275 (+0.00029720386988679737) | > avg_step_time: 0.45431565902602505 (-0.008103803849556057) | > avg_loss_disc: 2.5361614294455084 (-0.05090587911471545) | > avg_loss_disc_real_0: 0.09355388083298441 (+0.004143616709281012) | > avg_loss_disc_real_1: 0.21570305757119623 (-0.019924492903158675) | > avg_loss_disc_real_2: 0.22644887255950713 (-0.013275030003467081) | > avg_loss_disc_real_3: 0.23629978215190725 (+0.002439900812968382) | > avg_loss_disc_real_4: 0.23879080821930523 (-0.00516400664625033) | > avg_loss_disc_real_5: 0.23879952120109343 (-0.006668673644603135) | > avg_loss_0: 2.5361614294455084 (-0.05090587911471545) | > avg_grad_norm_0: tensor(337.4817, device='cuda:0') (+tensor(39.7307, device='cuda:0')) | > avg_loss_gen: 2.204166897585695 (+0.047762840566501374) | > avg_loss_kl: 2.021889580807216 (-0.004039013889473697) | > avg_loss_feat: 4.097238167910508 (+0.1608875570162911) | > avg_loss_mel: 16.507269147416242 (-0.06609885793334769) | > avg_loss_duration: 1.5267606839327745 (-0.006020626551668462) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.35732441217127 (+0.13249203856562275) | > avg_grad_norm_1: tensor(8783.3320, device='cuda:0') (tensor(-466.2432, device='cuda:0'))  > EPOCH: 1267/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:15:52)   --> TIME: 2025-02-25 21:16:05 -- STEP: 25/72 -- GLOBAL_STEP: 1091250 | > loss_disc: 2.6041650772094727 (2.513803062438966) | > loss_disc_real_0: 0.08386002480983734 (0.09317148253321647) | > loss_disc_real_1: 0.2628929018974304 (0.22162017166614534) | > loss_disc_real_2: 0.20123858749866486 (0.22275349438190462) | > loss_disc_real_3: 0.2225896716117859 (0.2306874829530716) | > loss_disc_real_4: 0.24617688357830048 (0.23964775502681732) | > loss_disc_real_5: 0.23813870549201965 (0.24265532910823823) | > loss_0: 2.6041650772094727 (2.513803062438966) | > grad_norm_0: tensor(225.5976, device='cuda:0') (tensor(287.7895, device='cuda:0')) | > loss_gen: 2.080838680267334 (2.2540705156326295) | > loss_kl: 1.982263207435608 (1.9510298824310304) | > loss_feat: 3.9217002391815186 (4.165088205337525) | > loss_mel: 15.720114707946777 (16.50815204620361) | > loss_duration: 1.4981626272201538 (1.4834879159927368) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.203081130981445 (26.361828536987304) | > grad_norm_1: tensor(9046.2520, device='cuda:0') (tensor(8230.6396, device='cuda:0')) | > current_lr_0: 0.00017070423981136677 | > current_lr_1: 0.00017070423981136677 | > step_time: 0.4297 (0.43309845924377444) | > loader_time: 0.0064 (0.006741609573364258) --> EVAL PERFORMANCE | > avg_loader_time: 0.008716052686664418 (-0.00020541607494085727) | > avg_step_time: 0.4539353612443091 (-0.00038029778171594364) | > avg_loss_disc: 2.595722087672059 (+0.059560658226550434) | > avg_loss_disc_real_0: 0.14518031876691634 (+0.05162643793393193) | > avg_loss_disc_real_1: 0.22780396904743894 (+0.012100911476242704) | > avg_loss_disc_real_2: 0.23163002498552832 (+0.005181152426021185) | > avg_loss_disc_real_3: 0.2339870637991059 (-0.002312718352801363) | > avg_loss_disc_real_4: 0.24378430675452864 (+0.004993498535223406) | > avg_loss_disc_real_5: 0.2483560281320357 (+0.00955650693094226) | > avg_loss_0: 2.595722087672059 (+0.059560658226550434) | > avg_grad_norm_0: tensor(349.4110, device='cuda:0') (+tensor(11.9294, device='cuda:0')) | > avg_loss_gen: 2.225763910253284 (+0.021597012667589066) | > avg_loss_kl: 2.035787350694897 (+0.01389776988768121) | > avg_loss_feat: 3.891575816651467 (-0.20566235125904075) | > avg_loss_mel: 16.494720673896897 (-0.012548473519345293) | > avg_loss_duration: 1.5284768376551883 (+0.001716153722413738) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.176324548855632 (-0.18099986331563755) | > avg_grad_norm_1: tensor(7889.3696, device='cuda:0') (tensor(-893.9624, device='cuda:0'))  > EPOCH: 1268/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:16:29)   --> TIME: 2025-02-25 21:16:32 -- STEP: 3/72 -- GLOBAL_STEP: 1091300 | > loss_disc: 2.586963653564453 (2.503384749094645) | > loss_disc_real_0: 0.11861836910247803 (0.08407397071520488) | > loss_disc_real_1: 0.2752307057380676 (0.24029269317785898) | > loss_disc_real_2: 0.2898014783859253 (0.25483112533887226) | > loss_disc_real_3: 0.2622646987438202 (0.2539500792821248) | > loss_disc_real_4: 0.3209380805492401 (0.2721940279006958) | > loss_disc_real_5: 0.25668853521347046 (0.245021253824234) | > loss_0: 2.586963653564453 (2.503384749094645) | > grad_norm_0: tensor(466.3325, device='cuda:0') (tensor(262.2492, device='cuda:0')) | > loss_gen: 2.382340669631958 (2.384946664174398) | > loss_kl: 2.105435371398926 (2.0704227288564048) | > loss_feat: 4.015786170959473 (4.335109392801921) | > loss_mel: 17.56944465637207 (17.10706965128581) | > loss_duration: 1.5182020664215088 (1.4961001475652058) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.591209411621094 (27.393648147583008) | > grad_norm_1: tensor(2289.0300, device='cuda:0') (tensor(7685.8354, device='cuda:0')) | > current_lr_0: 0.00017068290178139033 | > current_lr_1: 0.00017068290178139033 | > step_time: 0.4309 (0.43221108118693036) | > loader_time: 0.0076 (0.00661921501159668)  --> TIME: 2025-02-25 21:16:55 -- STEP: 53/72 -- GLOBAL_STEP: 1091350 | > loss_disc: 2.605595588684082 (2.679084494428815) | > loss_disc_real_0: 0.1640942096710205 (0.1866587962041486) | > loss_disc_real_1: 0.13771194219589233 (0.20759569171745823) | > loss_disc_real_2: 0.2337484508752823 (0.2366441288646662) | > loss_disc_real_3: 0.19663184881210327 (0.22762853300796365) | > loss_disc_real_4: 0.21220816671848297 (0.2444454108206731) | > loss_disc_real_5: 0.22366954386234283 (0.24213429160837857) | > loss_0: 2.605595588684082 (2.679084494428815) | > grad_norm_0: tensor(383.5703, device='cuda:0') (tensor(328.0480, device='cuda:0')) | > loss_gen: 2.1571297645568848 (2.1502518361469494) | > loss_kl: 2.257321834564209 (2.1926404777562847) | > loss_feat: 4.506258010864258 (3.890217565140634) | > loss_mel: 15.883111953735352 (17.336855060649373) | > loss_duration: 1.596436858177185 (1.5364690749150403) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.400259017944336 (27.106433868408203) | > grad_norm_1: tensor(2234.2317, device='cuda:0') (tensor(6142.4712, device='cuda:0')) | > current_lr_0: 0.00017068290178139033 | > current_lr_1: 0.00017068290178139033 | > step_time: 0.4419 (0.43981468452597566) | > loader_time: 0.0093 (0.007916441503560767) --> EVAL PERFORMANCE | > avg_loader_time: 0.008668745067757623 (-4.730761890679551e-05) | > avg_step_time: 0.4506338817972532 (-0.0033014794470558884) | > avg_loss_disc: 2.687406539916992 (+0.09168445224493338) | > avg_loss_disc_real_0: 0.18681611752950808 (+0.04163579876259174) | > avg_loss_disc_real_1: 0.21108559933556637 (-0.01671836971187257) | > avg_loss_disc_real_2: 0.2364958367297347 (+0.004865811744206389) | > avg_loss_disc_real_3: 0.23031099224594279 (-0.0036760715531631016) | > avg_loss_disc_real_4: 0.24340084006249066 (-0.00038346669203798145) | > avg_loss_disc_real_5: 0.24208182747095403 (-0.006274200661081669) | > avg_loss_0: 2.687406539916992 (+0.09168445224493338) | > avg_grad_norm_0: tensor(321.7260, device='cuda:0') (tensor(-27.6851, device='cuda:0')) | > avg_loss_gen: 2.1272115186906198 (-0.09855239156266427) | > avg_loss_kl: 2.257907508124768 (+0.2221201574298708) | > avg_loss_feat: 3.827617007242122 (-0.06395880940934484) | > avg_loss_mel: 17.62589098702015 (+1.1311703131232527) | > avg_loss_duration: 1.5694137005738809 (+0.040936862918692585) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.408040597405233 (+1.2317160485496004) | > avg_grad_norm_1: tensor(5981.1504, device='cuda:0') (tensor(-1908.2192, device='cuda:0'))  > EPOCH: 1269/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:17:05)   --> TIME: 2025-02-25 21:17:22 -- STEP: 31/72 -- GLOBAL_STEP: 1091400 | > loss_disc: 2.7565386295318604 (2.584116082037649) | > loss_disc_real_0: 0.2500532865524292 (0.10180489772990826) | > loss_disc_real_1: 0.2069738358259201 (0.20410366837055452) | > loss_disc_real_2: 0.40005138516426086 (0.2829519760224127) | > loss_disc_real_3: 0.3303621709346771 (0.23440559881348763) | > loss_disc_real_4: 0.3133130967617035 (0.24669698121086245) | > loss_disc_real_5: 0.2227885127067566 (0.23608236639730393) | > loss_0: 2.7565386295318604 (2.584116082037649) | > grad_norm_0: tensor(105.9826, device='cuda:0') (tensor(278.1992, device='cuda:0')) | > loss_gen: 2.15509033203125 (2.2431927265659457) | > loss_kl: 1.9312729835510254 (2.0435929798310797) | > loss_feat: 3.0127084255218506 (4.2018275414743735) | > loss_mel: 16.79051399230957 (17.522101556101152) | > loss_duration: 1.5450003147125244 (1.5192467935623661) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.434585571289062 (27.52996149370747) | > grad_norm_1: tensor(7560.2100, device='cuda:0') (tensor(7681.0498, device='cuda:0')) | > current_lr_0: 0.00017066156641866765 | > current_lr_1: 0.00017066156641866765 | > step_time: 0.4321 (0.44655872160388577) | > loader_time: 0.0085 (0.006791268625567036) --> EVAL PERFORMANCE | > avg_loader_time: 0.008465078515066227 (-0.00020366655269139604) | > avg_step_time: 0.45697017118964395 (+0.006336289392390726) | > avg_loss_disc: 2.632753204291975 (-0.054653335625017174) | > avg_loss_disc_real_0: 0.12490994485855944 (-0.06190617267094864) | > avg_loss_disc_real_1: 0.2203143272601383 (+0.009228727924571939) | > avg_loss_disc_real_2: 0.26050196105325724 (+0.024006124323522532) | > avg_loss_disc_real_3: 0.24125698376709306 (+0.010945991521150278) | > avg_loss_disc_real_4: 0.2481961220922605 (+0.004795282029769837) | > avg_loss_disc_real_5: 0.24585161851325504 (+0.0037697910423010184) | > avg_loss_0: 2.632753204291975 (-0.054653335625017174) | > avg_grad_norm_0: tensor(263.1137, device='cuda:0') (tensor(-58.6122, device='cuda:0')) | > avg_loss_gen: 2.1587757325508226 (+0.031564213860202806) | > avg_loss_kl: 2.2681917707685013 (+0.010284262643733388) | > avg_loss_feat: 3.9023973538842 (+0.07478034664207778) | > avg_loss_mel: 18.169538793429528 (+0.5436478064093784) | > avg_loss_duration: 1.571273236207559 (+0.0018595356336781332) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.07017686333455 (+0.6621362659293162) | > avg_grad_norm_1: tensor(7467.4663, device='cuda:0') (+tensor(1486.3159, device='cuda:0'))  > EPOCH: 1270/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:17:42)   --> TIME: 2025-02-25 21:17:48 -- STEP: 9/72 -- GLOBAL_STEP: 1091450 | > loss_disc: 2.645057201385498 (2.7405375904507108) | > loss_disc_real_0: 0.15636661648750305 (0.2839699213703473) | > loss_disc_real_1: 0.253853440284729 (0.18027093013127646) | > loss_disc_real_2: 0.24606414139270782 (0.2405937992864185) | > loss_disc_real_3: 0.24260428547859192 (0.23628967669275072) | > loss_disc_real_4: 0.23091627657413483 (0.23031308750311533) | > loss_disc_real_5: 0.24257859587669373 (0.22868024475044674) | > loss_0: 2.645057201385498 (2.7405375904507108) | > grad_norm_0: tensor(51.6971, device='cuda:0') (tensor(204.0483, device='cuda:0')) | > loss_gen: 2.2488622665405273 (2.267120467291938) | > loss_kl: 2.3164162635803223 (2.0859120157029896) | > loss_feat: 3.9439446926116943 (3.7697083950042725) | > loss_mel: 18.523775100708008 (17.73648452758789) | > loss_duration: 1.5429878234863281 (1.5417925649219089) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.575984954833984 (27.401018142700195) | > grad_norm_1: tensor(6002.2754, device='cuda:0') (tensor(4200.1338, device='cuda:0')) | > current_lr_0: 0.0001706402337228653 | > current_lr_1: 0.0001706402337228653 | > step_time: 0.4226 (0.4280788103739421) | > loader_time: 0.0056 (0.006535953945583767)  --> TIME: 2025-02-25 21:18:11 -- STEP: 59/72 -- GLOBAL_STEP: 1091500 | > loss_disc: 2.6327879428863525 (2.7192820856126687) | > loss_disc_real_0: 0.060859180986881256 (0.18376091476214138) | > loss_disc_real_1: 0.21591497957706451 (0.2493731407290798) | > loss_disc_real_2: 0.23345357179641724 (0.24258895571959221) | > loss_disc_real_3: 0.23939917981624603 (0.23896621698040074) | > loss_disc_real_4: 0.2468186616897583 (0.25300932056823017) | > loss_disc_real_5: 0.2289246767759323 (0.2508043944835664) | > loss_0: 2.6327879428863525 (2.7192820856126687) | > grad_norm_0: tensor(205.4420, device='cuda:0') (tensor(282.8975, device='cuda:0')) | > loss_gen: 1.8110988140106201 (2.17834024106042) | > loss_kl: 2.3444480895996094 (2.2704721507379557) | > loss_feat: 2.8373870849609375 (3.738478846469168) | > loss_mel: 16.505556106567383 (18.154962087081646) | > loss_duration: 1.5894399881362915 (1.5696585259195102) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.087928771972656 (27.91191178661282) | > grad_norm_1: tensor(10551.1602, device='cuda:0') (tensor(5693.9951, device='cuda:0')) | > current_lr_0: 0.0001706402337228653 | > current_lr_1: 0.0001706402337228653 | > step_time: 0.466 (0.4415802228248725) | > loader_time: 0.0101 (0.007997731030997584) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575157380439865 (+0.00011007886537363863) | > avg_step_time: 0.450902327685289 (-0.006067843504354964) | > avg_loss_disc: 2.6980868594747194 (+0.06533365518274437) | > avg_loss_disc_real_0: 0.16551199912185402 (+0.04060205426329458) | > avg_loss_disc_real_1: 0.24105318226444888 (+0.02073885500431058) | > avg_loss_disc_real_2: 0.23820046136077022 (-0.022301499692487015) | > avg_loss_disc_real_3: 0.23658559880626034 (-0.00467138496083272) | > avg_loss_disc_real_4: 0.24688986510458127 (-0.0013062569876792218) | > avg_loss_disc_real_5: 0.24602156583691998 (+0.00016994732366493182) | > avg_loss_0: 2.6980868594747194 (+0.06533365518274437) | > avg_grad_norm_0: tensor(307.0557, device='cuda:0') (+tensor(43.9420, device='cuda:0')) | > avg_loss_gen: 2.1415936056996747 (-0.01718212685114784) | > avg_loss_kl: 2.2730894458126008 (+0.0048976750440994365) | > avg_loss_feat: 3.74997947249614 (-0.15241788138805967) | > avg_loss_mel: 18.1130731340865 (-0.05646565934302927) | > avg_loss_duration: 1.5758779065709718 (+0.004604670363412833) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.85361349078971 (-0.21656337254484015) | > avg_grad_norm_1: tensor(5999.8457, device='cuda:0') (tensor(-1467.6206, device='cuda:0'))  > EPOCH: 1271/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:18:19)   --> TIME: 2025-02-25 21:18:37 -- STEP: 37/72 -- GLOBAL_STEP: 1091550 | > loss_disc: 2.8904473781585693 (2.6892336059261015) | > loss_disc_real_0: 0.44302573800086975 (0.18565257880333308) | > loss_disc_real_1: 0.17218485474586487 (0.2222837036525881) | > loss_disc_real_2: 0.17815878987312317 (0.2397955325809685) | > loss_disc_real_3: 0.22990073263645172 (0.23351417038891767) | > loss_disc_real_4: 0.27044567465782166 (0.24559047375176404) | > loss_disc_real_5: 0.25117772817611694 (0.24673050480919914) | > loss_0: 2.8904473781585693 (2.6892336059261015) | > grad_norm_0: tensor(353.3881, device='cuda:0') (tensor(309.3210, device='cuda:0')) | > loss_gen: 2.157898187637329 (2.0930305944906697) | > loss_kl: 2.076961040496826 (2.07541671314755) | > loss_feat: 3.838698387145996 (3.7440224402659648) | > loss_mel: 17.8902530670166 (17.483872130110456) | > loss_duration: 1.5435129404067993 (1.526209393063107) | > amp_scaler: 16.0 (12.97297297297297) | > loss_1: 27.507322311401367 (26.92255128396524) | > grad_norm_1: tensor(2161.0378, device='cuda:0') (tensor(6861.8931, device='cuda:0')) | > current_lr_0: 0.00017061890369364995 | > current_lr_1: 0.00017061890369364995 | > step_time: 0.4417 (0.43384173109724716) | > loader_time: 0.0083 (0.007340766288138725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008602609097118108 (+2.745171667824231e-05) | > avg_step_time: 0.45141408141230194 (+0.0005117537270129535) | > avg_loss_disc: 2.6705388922086906 (-0.027547967266028817) | > avg_loss_disc_real_0: 0.16066761396195686 (-0.004844385159897158) | > avg_loss_disc_real_1: 0.217764374235986 (-0.023288808028462893) | > avg_loss_disc_real_2: 0.23059960045445133 (-0.007600860906318896) | > avg_loss_disc_real_3: 0.23704601572432987 (+0.00046041691806952834) | > avg_loss_disc_real_4: 0.24622612792841145 (-0.0006637371761698263) | > avg_loss_disc_real_5: 0.24758100719519063 (+0.0015594413582706579) | > avg_loss_0: 2.6705388922086906 (-0.027547967266028817) | > avg_grad_norm_0: tensor(259.9745, device='cuda:0') (tensor(-47.0812, device='cuda:0')) | > avg_loss_gen: 2.0698575217958908 (-0.07173608390378394) | > avg_loss_kl: 2.098284130365077 (-0.17480531544752376) | > avg_loss_feat: 3.69921944174968 (-0.05076003074645996) | > avg_loss_mel: 17.105964351707776 (-1.0071087823787224) | > avg_loss_duration: 1.549650301396007 (-0.026227605174964808) | > avg_amp_scaler: 14.422535211267602 (+6.422535211267602) | > avg_loss_1: 26.522975787310532 (-1.3306377034791765) | > avg_grad_norm_1: tensor(6371.3364, device='cuda:0') (+tensor(371.4907, device='cuda:0'))  > EPOCH: 1272/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:18:56)   --> TIME: 2025-02-25 21:19:04 -- STEP: 15/72 -- GLOBAL_STEP: 1091600 | > loss_disc: 3.0328478813171387 (2.8022221247355144) | > loss_disc_real_0: 0.3095199465751648 (0.06970726425449053) | > loss_disc_real_1: 0.2094946801662445 (0.23887883126735687) | > loss_disc_real_2: 0.20016270875930786 (0.24616748392581939) | > loss_disc_real_3: 0.22121787071228027 (0.2312955011924108) | > loss_disc_real_4: 0.23874668776988983 (0.2408259004354477) | > loss_disc_real_5: 0.24761709570884705 (0.25740252931912744) | > loss_0: 3.0328478813171387 (2.8022221247355144) | > grad_norm_0: tensor(723.9085, device='cuda:0') (tensor(329.1146, device='cuda:0')) | > loss_gen: 1.7759158611297607 (1.8770280996958415) | > loss_kl: 1.9446465969085693 (1.8915751934051515) | > loss_feat: 3.731386423110962 (3.9333665053049724) | > loss_mel: 17.15660285949707 (16.75871156056722) | > loss_duration: 1.4969573020935059 (1.4979710737864176) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.10550880432129 (25.958652623494466) | > grad_norm_1: tensor(5745.7285, device='cuda:0') (tensor(5801.4263, device='cuda:0')) | > current_lr_0: 0.00017059757633068823 | > current_lr_1: 0.00017059757633068823 | > step_time: 0.4316 (0.4273568948109945) | > loader_time: 0.0061 (0.006587870915730794)  --> TIME: 2025-02-25 21:19:27 -- STEP: 65/72 -- GLOBAL_STEP: 1091650 | > loss_disc: 2.465062141418457 (2.738290159518902) | > loss_disc_real_0: 0.09612920880317688 (0.17641171010640952) | > loss_disc_real_1: 0.23427945375442505 (0.22662808620012723) | > loss_disc_real_2: 0.23450274765491486 (0.24304547974696525) | > loss_disc_real_3: 0.18755227327346802 (0.23082644297526433) | > loss_disc_real_4: 0.24125635623931885 (0.24523321527701158) | > loss_disc_real_5: 0.2454732358455658 (0.24889150170179508) | > loss_0: 2.465062141418457 (2.738290159518902) | > grad_norm_0: tensor(39.9073, device='cuda:0') (tensor(264.0846, device='cuda:0')) | > loss_gen: 2.0714223384857178 (1.9765161679341243) | > loss_kl: 2.1635684967041016 (2.025335911604074) | > loss_feat: 3.704483985900879 (3.597126051095816) | > loss_mel: 16.67262077331543 (16.711968671358555) | > loss_duration: 1.5984947681427002 (1.5390110419346736) | > amp_scaler: 16.0 (16.0) | > loss_1: 26.210590362548828 (25.84995797964243) | > grad_norm_1: tensor(7608.6045, device='cuda:0') (tensor(5592.1455, device='cuda:0')) | > current_lr_0: 0.00017059757633068823 | > current_lr_1: 0.00017059757633068823 | > step_time: 0.5165 (0.44305520424476036) | > loader_time: 0.0117 (0.008230348733755255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522053839455186 (-8.055525766292176e-05) | > avg_step_time: 0.4484719759981397 (-0.0029421054141622327) | > avg_loss_disc: 2.7247046987775345 (+0.054165806568843955) | > avg_loss_disc_real_0: 0.16567374949514027 (+0.005006135533183409) | > avg_loss_disc_real_1: 0.226375636500372 (+0.008611262264386021) | > avg_loss_disc_real_2: 0.23625911321018783 (+0.005659512755736507) | > avg_loss_disc_real_3: 0.23752470461415573 (+0.0004786888898258568) | > avg_loss_disc_real_4: 0.24443800033817828 (-0.0017881275902331684) | > avg_loss_disc_real_5: 0.24791804837508935 (+0.0003370411798987172) | > avg_loss_0: 2.7247046987775345 (+0.054165806568843955) | > avg_grad_norm_0: tensor(247.5721, device='cuda:0') (tensor(-12.4024, device='cuda:0')) | > avg_loss_gen: 1.9861285098841492 (-0.08372901191174154) | > avg_loss_kl: 2.0320524497770927 (-0.06623168058798434) | > avg_loss_feat: 3.626573297339426 (-0.07264614441025419) | > avg_loss_mel: 16.68045524812081 (-0.4255091035869647) | > avg_loss_duration: 1.544174113743742 (-0.0054761876522650965) | > avg_amp_scaler: 16.0 (+1.5774647887323976) | > avg_loss_1: 25.86938374479052 (-0.6535920425200104) | > avg_grad_norm_1: tensor(5391.0347, device='cuda:0') (tensor(-980.3018, device='cuda:0'))  > EPOCH: 1273/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:19:32)   --> TIME: 2025-02-25 21:19:54 -- STEP: 43/72 -- GLOBAL_STEP: 1091700 | > loss_disc: 3.0070834159851074 (2.7874510510023245) | > loss_disc_real_0: 0.3239034414291382 (0.15595559230030973) | > loss_disc_real_1: 0.2637577950954437 (0.2041535399161106) | > loss_disc_real_2: 0.20340995490550995 (0.2751807754122935) | > loss_disc_real_3: 0.17665693163871765 (0.21800041891807734) | > loss_disc_real_4: 0.22131092846393585 (0.2279569787341495) | > loss_disc_real_5: 0.21116343140602112 (0.23846651648366174) | > loss_0: 3.0070834159851074 (2.7874510510023245) | > grad_norm_0: tensor(548.0065, device='cuda:0') (tensor(299.1714, device='cuda:0')) | > loss_gen: 1.7585633993148804 (2.016039634859839) | > loss_kl: 2.288175344467163 (2.0174353150434277) | > loss_feat: 3.387072801589966 (3.8344055220138196) | > loss_mel: 17.271530151367188 (17.067063087640804) | > loss_duration: 1.5893548727035522 (1.518676486126212) | > amp_scaler: 8.0 (11.348837209302326) | > loss_1: 26.294696807861328 (26.453620334004246) | > grad_norm_1: tensor(12203.2852, device='cuda:0') (tensor(6601.8315, device='cuda:0')) | > current_lr_0: 0.0001705762516336469 | > current_lr_1: 0.0001705762516336469 | > step_time: 0.4501 (0.4382408108822135) | > loader_time: 0.0095 (0.007372357124506041) --> EVAL PERFORMANCE | > avg_loader_time: 0.008495992338153678 (-2.6061501301508083e-05) | > avg_step_time: 0.45172419346554177 (+0.003252217467402063) | > avg_loss_disc: 2.6822993352379596 (-0.042405363539574914) | > avg_loss_disc_real_0: 0.12418343824609912 (-0.04149031124904115) | > avg_loss_disc_real_1: 0.2142261894760837 (-0.012149447024288301) | > avg_loss_disc_real_2: 0.2536462885393224 (+0.017387175329134585) | > avg_loss_disc_real_3: 0.2208422260804915 (-0.016682478533664225) | > avg_loss_disc_real_4: 0.23219619334583552 (-0.012241806992342757) | > avg_loss_disc_real_5: 0.23878488154478475 (-0.009133166830304601) | > avg_loss_0: 2.6822993352379596 (-0.042405363539574914) | > avg_grad_norm_0: tensor(270.2198, device='cuda:0') (+tensor(22.6477, device='cuda:0')) | > avg_loss_gen: 2.0575482828516356 (+0.07141977296748636) | > avg_loss_kl: 2.2406785404178464 (+0.20862609064075377) | > avg_loss_feat: 3.8331778888971035 (+0.20660459155767752) | > avg_loss_mel: 17.384117663746153 (+0.7036624156253417) | > avg_loss_duration: 1.5884348624189133 (+0.04426074867517138) | > avg_amp_scaler: 10.028169014084506 (-5.971830985915494) | > avg_loss_1: 27.103957485145248 (+1.2345737403547261) | > avg_grad_norm_1: tensor(6303.7988, device='cuda:0') (+tensor(912.7642, device='cuda:0'))  > EPOCH: 1274/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:20:09)   --> TIME: 2025-02-25 21:20:20 -- STEP: 21/72 -- GLOBAL_STEP: 1091750 | > loss_disc: 2.609734058380127 (2.50685791742234) | > loss_disc_real_0: 0.05674097687005997 (0.057228985020802134) | > loss_disc_real_1: 0.240398108959198 (0.21900919789359682) | > loss_disc_real_2: 0.22849400341510773 (0.2317932170061838) | > loss_disc_real_3: 0.2559213936328888 (0.23330803783166976) | > loss_disc_real_4: 0.26064780354499817 (0.2425765898965654) | > loss_disc_real_5: 0.2656523287296295 (0.24312574664751688) | > loss_0: 2.609734058380127 (2.50685791742234) | > grad_norm_0: tensor(230.9087, device='cuda:0') (tensor(165.3242, device='cuda:0')) | > loss_gen: 2.0933971405029297 (2.2064420325415477) | > loss_kl: 1.8515450954437256 (1.956573872339158) | > loss_feat: 3.459226131439209 (4.131289459410169) | > loss_mel: 16.331296920776367 (16.90538415454683) | > loss_duration: 1.516313910484314 (1.5122700759342738) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.25177764892578 (26.711959475562686) | > grad_norm_1: tensor(9292.1279, device='cuda:0') (tensor(5755.0117, device='cuda:0')) | > current_lr_0: 0.00017055492960219267 | > current_lr_1: 0.00017055492960219267 | > step_time: 0.4389 (0.42945708547319683) | > loader_time: 0.0065 (0.006836107798985073)  --> TIME: 2025-02-25 21:20:44 -- STEP: 71/72 -- GLOBAL_STEP: 1091800 | > loss_disc: 2.5470175743103027 (2.5210881300375494) | > loss_disc_real_0: 0.1885586529970169 (0.09184309050545726) | > loss_disc_real_1: 0.21374063193798065 (0.2194191078904649) | > loss_disc_real_2: 0.14181996881961823 (0.22670559018430575) | > loss_disc_real_3: 0.24963483214378357 (0.23136529158538496) | > loss_disc_real_4: 0.2736223638057709 (0.2452065797758774) | > loss_disc_real_5: 0.2543494701385498 (0.24478108341425237) | > loss_0: 2.5470175743103027 (2.5210881300375494) | > grad_norm_0: tensor(432.7739, device='cuda:0') (tensor(209.4870, device='cuda:0')) | > loss_gen: 2.4424824714660645 (2.2352411461548076) | > loss_kl: 1.8890968561172485 (1.9993899355471974) | > loss_feat: 4.29502010345459 (4.047200904765597) | > loss_mel: 16.831151962280273 (16.74566158778231) | > loss_duration: 1.5452299118041992 (1.5414112319409008) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.002979278564453 (26.568904634932398) | > grad_norm_1: tensor(4061.5544, device='cuda:0') (tensor(5624.7314, device='cuda:0')) | > current_lr_0: 0.00017055492960219267 | > current_lr_1: 0.00017055492960219267 | > step_time: 0.475 (0.44980787223493546) | > loader_time: 0.0079 (0.008579469062912634) --> EVAL PERFORMANCE | > avg_loader_time: 0.008579469062912634 (+8.34767247589567e-05) | > avg_step_time: 0.44980787223493546 (-0.0019163212306063104) | > avg_loss_disc: 2.5210881300375494 (-0.1612112052004102) | > avg_loss_disc_real_0: 0.09184309050545726 (-0.03234034774064186) | > avg_loss_disc_real_1: 0.2194191078904649 (+0.005192918414381187) | > avg_loss_disc_real_2: 0.22670559018430575 (-0.026940698355016668) | > avg_loss_disc_real_3: 0.23136529158538496 (+0.010523065504893458) | > avg_loss_disc_real_4: 0.2452065797758774 (+0.013010386430041876) | > avg_loss_disc_real_5: 0.24478108341425237 (+0.005996201869467616) | > avg_loss_0: 2.5210881300375494 (-0.1612112052004102) | > avg_grad_norm_0: tensor(209.4870, device='cuda:0') (tensor(-60.7328, device='cuda:0')) | > avg_loss_gen: 2.2352411461548076 (+0.17769286330317202) | > avg_loss_kl: 1.9993899355471974 (-0.24128860487064907) | > avg_loss_feat: 4.047200904765597 (+0.21402301586849326) | > avg_loss_mel: 16.74566158778231 (-0.6384560759638447) | > avg_loss_duration: 1.5414112319409008 (-0.0470236304780125) | > avg_amp_scaler: 8.0 (-2.028169014084506) | > avg_loss_1: 26.568904634932398 (-0.5350528502128498) | > avg_grad_norm_1: tensor(5624.7314, device='cuda:0') (tensor(-679.0674, device='cuda:0'))  > EPOCH: 1275/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:20:46)   --> TIME: 2025-02-25 21:21:10 -- STEP: 49/72 -- GLOBAL_STEP: 1091850 | > loss_disc: 2.7519493103027344 (2.5651175343260464) | > loss_disc_real_0: 0.17598959803581238 (0.14012293365536907) | > loss_disc_real_1: 0.23766754567623138 (0.22260707069416436) | > loss_disc_real_2: 0.2841728627681732 (0.24567655458742257) | > loss_disc_real_3: 0.25956887006759644 (0.23574013217371337) | > loss_disc_real_4: 0.29005491733551025 (0.2495963068640962) | > loss_disc_real_5: 0.2864112854003906 (0.2505073589938028) | > loss_0: 2.7519493103027344 (2.5651175343260464) | > grad_norm_0: tensor(351.1640, device='cuda:0') (tensor(234.3540, device='cuda:0')) | > loss_gen: 2.2944436073303223 (2.3134029252188544) | > loss_kl: 2.3633036613464355 (1.9236700388849999) | > loss_feat: 3.940788984298706 (4.00751622355714) | > loss_mel: 18.03886604309082 (16.537502249892878) | > loss_duration: 1.6086100339889526 (1.5143816252144016) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.24601173400879 (26.296472899767817) | > grad_norm_1: tensor(3288.6665, device='cuda:0') (tensor(4702.5181, device='cuda:0')) | > current_lr_0: 0.00017053361023599238 | > current_lr_1: 0.00017053361023599238 | > step_time: 0.4368 (0.4387780063006343) | > loader_time: 0.0097 (0.007672786712646484) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580859278289362 (+1.3902153767272885e-06) | > avg_step_time: 0.45093627043173345 (+0.0011283981967979972) | > avg_loss_disc: 2.613049439980949 (+0.0919613099433998) | > avg_loss_disc_real_0: 0.16095930558275168 (+0.06911621507729442) | > avg_loss_disc_real_1: 0.23261377236372988 (+0.013194664473264978) | > avg_loss_disc_real_2: 0.24167024795438202 (+0.014964657770076273) | > avg_loss_disc_real_3: 0.24230305787543177 (+0.010937766290046808) | > avg_loss_disc_real_4: 0.24814596289480237 (+0.002939383118924971) | > avg_loss_disc_real_5: 0.2499684733404241 (+0.005187389926171737) | > avg_loss_0: 2.613049439980949 (+0.0919613099433998) | > avg_grad_norm_0: tensor(255.4313, device='cuda:0') (+tensor(45.9442, device='cuda:0')) | > avg_loss_gen: 2.2366946230471974 (+0.0014534768923897268) | > avg_loss_kl: 2.0967934467423133 (+0.09740351119511592) | > avg_loss_feat: 3.8197762395294603 (-0.22742466523613647) | > avg_loss_mel: 16.938800731175384 (+0.19313914339307559) | > avg_loss_duration: 1.5459391231268222 (+0.004527891185921451) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.638004034337868 (+0.06909939940547005) | > avg_grad_norm_1: tensor(4944.2959, device='cuda:0') (tensor(-680.4355, device='cuda:0'))  > EPOCH: 1276/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:21:22)   --> TIME: 2025-02-25 21:21:36 -- STEP: 27/72 -- GLOBAL_STEP: 1091900 | > loss_disc: 2.456082820892334 (2.48315511809455) | > loss_disc_real_0: 0.11468446254730225 (0.10888054028705314) | > loss_disc_real_1: 0.18962456285953522 (0.20787317830103416) | > loss_disc_real_2: 0.19746960699558258 (0.2183274538428695) | > loss_disc_real_3: 0.2021888941526413 (0.22996665978873218) | > loss_disc_real_4: 0.2440413385629654 (0.23489419398484407) | > loss_disc_real_5: 0.253895103931427 (0.23936821906654923) | > loss_0: 2.456082820892334 (2.48315511809455) | > grad_norm_0: tensor(63.2390, device='cuda:0') (tensor(298.4389, device='cuda:0')) | > loss_gen: 2.3025600910186768 (2.1757453238522575) | > loss_kl: 1.9978352785110474 (1.9884982285676178) | > loss_feat: 3.890892744064331 (4.0922075024357545) | > loss_mel: 16.439250946044922 (17.226852487634726) | > loss_duration: 1.512366771697998 (1.5054508006131206) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.142908096313477 (26.988754413746022) | > grad_norm_1: tensor(7467.7339, device='cuda:0') (tensor(9572.9287, device='cuda:0')) | > current_lr_0: 0.00017051229353471286 | > current_lr_1: 0.00017051229353471286 | > step_time: 0.4302 (0.43111111499645094) | > loader_time: 0.0082 (0.0068838684647171584) --> EVAL PERFORMANCE | > avg_loader_time: 0.008513903953659701 (-6.695532462966045e-05) | > avg_step_time: 0.44930302593070015 (-0.001633244501033304) | > avg_loss_disc: 2.5688261985778804 (-0.04422324140306877) | > avg_loss_disc_real_0: 0.11653975612470802 (-0.04441954945804366) | > avg_loss_disc_real_1: 0.21801043837003306 (-0.014603333993696815) | > avg_loss_disc_real_2: 0.233399790357536 (-0.008270457596846026) | > avg_loss_disc_real_3: 0.23131709094618408 (-0.010985966929247692) | > avg_loss_disc_real_4: 0.24118329710523848 (-0.006962665789563893) | > avg_loss_disc_real_5: 0.24415095192445835 (-0.005817521415965754) | > avg_loss_0: 2.5688261985778804 (-0.04422324140306877) | > avg_grad_norm_0: tensor(299.2783, device='cuda:0') (+tensor(43.8470, device='cuda:0')) | > avg_loss_gen: 2.1063428479181208 (-0.13035177512907659) | > avg_loss_kl: 2.0367645596114676 (-0.06002888713084564) | > avg_loss_feat: 3.8490989040321026 (+0.029322664502642315) | > avg_loss_mel: 16.783618362856576 (-0.15518236831880827) | > avg_loss_duration: 1.5408994731768755 (-0.005039649949946767) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.316724239940374 (-0.3212797943974941) | > avg_grad_norm_1: tensor(9774.0020, device='cuda:0') (+tensor(4829.7061, device='cuda:0'))  > EPOCH: 1277/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:21:59)   --> TIME: 2025-02-25 21:22:03 -- STEP: 5/72 -- GLOBAL_STEP: 1091950 | > loss_disc: 3.025484323501587 (2.696747589111328) | > loss_disc_real_0: 0.0629938542842865 (0.07043925672769547) | > loss_disc_real_1: 0.24223171174526215 (0.22705702781677245) | > loss_disc_real_2: 0.31332308053970337 (0.2679072290658951) | > loss_disc_real_3: 0.23620560765266418 (0.21825752258300782) | > loss_disc_real_4: 0.2861767113208771 (0.2542280375957489) | > loss_disc_real_5: 0.274793803691864 (0.2550699234008789) | > loss_0: 3.025484323501587 (2.696747589111328) | > grad_norm_0: tensor(106.8357, device='cuda:0') (tensor(334.9293, device='cuda:0')) | > loss_gen: 1.9133520126342773 (2.054522466659546) | > loss_kl: 1.919316053390503 (1.941565203666687) | > loss_feat: 3.4914629459381104 (4.090423583984375) | > loss_mel: 16.15291404724121 (16.79280891418457) | > loss_duration: 1.4715774059295654 (1.4995085716247558) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.948623657226562 (26.37882843017578) | > grad_norm_1: tensor(11046.3965, device='cuda:0') (tensor(12648.7422, device='cuda:0')) | > current_lr_0: 0.00017049097949802102 | > current_lr_1: 0.00017049097949802102 | > step_time: 0.4212 (0.43086771965026854) | > loader_time: 0.0061 (0.006722736358642578)  --> TIME: 2025-02-25 21:22:26 -- STEP: 55/72 -- GLOBAL_STEP: 1092000 | > loss_disc: 2.4665727615356445 (2.6532068599354135) | > loss_disc_real_0: 0.04004377871751785 (0.12674512341618538) | > loss_disc_real_1: 0.2200755476951599 (0.22157662456685848) | > loss_disc_real_2: 0.23058122396469116 (0.24720079356973837) | > loss_disc_real_3: 0.2180473357439041 (0.2297697833993218) | > loss_disc_real_4: 0.22255411744117737 (0.24095558686689897) | > loss_disc_real_5: 0.23601315915584564 (0.24115548269315196) | > loss_0: 2.4665727615356445 (2.6532068599354135) | > grad_norm_0: tensor(213.6503, device='cuda:0') (tensor(330.1537, device='cuda:0')) | > loss_gen: 2.1917059421539307 (2.154974790052935) | > loss_kl: 2.135957717895508 (2.100382807038048) | > loss_feat: 4.250662803649902 (3.9198039835149596) | > loss_mel: 17.730072021484375 (17.345635466142127) | > loss_duration: 1.6510639190673828 (1.5511603355407715) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.959461212158203 (27.07195736278187) | > grad_norm_1: tensor(4998.8193, device='cuda:0') (tensor(9870.4189, device='cuda:0')) | > current_lr_0: 0.00017049097949802102 | > current_lr_1: 0.00017049097949802102 | > step_time: 0.4611 (0.442869892987338) | > loader_time: 0.0093 (0.007952013882723721) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653741487315004 (+0.00013983753365530253) | > avg_step_time: 0.45298533036675254 (+0.003682304436052386) | > avg_loss_disc: 2.6314771242544683 (+0.06265092567658792) | > avg_loss_disc_real_0: 0.12226855240657293 (+0.005728796281864915) | > avg_loss_disc_real_1: 0.22059809154188129 (+0.0025876531718482254) | > avg_loss_disc_real_2: 0.24615406654250463 (+0.012754276184968633) | > avg_loss_disc_real_3: 0.2300129900096168 (-0.0013041009365672707) | > avg_loss_disc_real_4: 0.24045710118723587 (-0.0007261959180026034) | > avg_loss_disc_real_5: 0.2401132881641388 (-0.004037663760319554) | > avg_loss_0: 2.6314771242544683 (+0.06265092567658792) | > avg_grad_norm_0: tensor(299.4251, device='cuda:0') (+tensor(0.1468, device='cuda:0')) | > avg_loss_gen: 2.1650456744180606 (+0.05870282649993985) | > avg_loss_kl: 2.1042423097180656 (+0.06747775010659796) | > avg_loss_feat: 3.938443902512671 (+0.08934499848056854) | > avg_loss_mel: 17.273859749377607 (+0.49024138652103133) | > avg_loss_duration: 1.561734463127566 (+0.020834989950690552) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.0433260689319 (+0.7266018289915266) | > avg_grad_norm_1: tensor(8796.9521, device='cuda:0') (tensor(-977.0498, device='cuda:0'))  > EPOCH: 1278/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:22:36)   --> TIME: 2025-02-25 21:22:53 -- STEP: 33/72 -- GLOBAL_STEP: 1092050 | > loss_disc: 2.5573890209198 (2.6015869920903985) | > loss_disc_real_0: 0.08296068012714386 (0.11452710831707175) | > loss_disc_real_1: 0.28807657957077026 (0.21938483304146564) | > loss_disc_real_2: 0.28728097677230835 (0.2187149359872847) | > loss_disc_real_3: 0.2351980060338974 (0.2353071209156152) | > loss_disc_real_4: 0.24957504868507385 (0.23816587256662775) | > loss_disc_real_5: 0.25641578435897827 (0.24214117048364697) | > loss_0: 2.5573890209198 (2.6015869920903985) | > grad_norm_0: tensor(347.0025, device='cuda:0') (tensor(222.9800, device='cuda:0')) | > loss_gen: 2.2611443996429443 (2.132658011985548) | > loss_kl: 2.034982681274414 (2.0608183976375694) | > loss_feat: 4.0086588859558105 (4.051283077760175) | > loss_mel: 16.542020797729492 (17.21444866874001) | > loss_duration: 1.5230674743652344 (1.5197306148933643) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.369874954223633 (26.978938594008937) | > grad_norm_1: tensor(5896.0737, device='cuda:0') (tensor(6528.9600, device='cuda:0')) | > current_lr_0: 0.00017046966812558377 | > current_lr_1: 0.00017046966812558377 | > step_time: 0.4404 (0.434137503306071) | > loader_time: 0.0082 (0.0069055846243193655) --> EVAL PERFORMANCE | > avg_loader_time: 0.008535092985126334 (-0.00011864850218866983) | > avg_step_time: 0.4525216431684897 (-0.000463687198262841) | > avg_loss_disc: 2.575408176637031 (-0.05606894761743719) | > avg_loss_disc_real_0: 0.10954662737712055 (-0.01272192502945238) | > avg_loss_disc_real_1: 0.21874081217487093 (-0.001857279367010356) | > avg_loss_disc_real_2: 0.21567756892509862 (-0.030476497617406006) | > avg_loss_disc_real_3: 0.23501990987381466 (+0.005006919864197851) | > avg_loss_disc_real_4: 0.24324483938620123 (+0.002787738198965356) | > avg_loss_disc_real_5: 0.24536075327597873 (+0.005247465111839933) | > avg_loss_0: 2.575408176637031 (-0.05606894761743719) | > avg_grad_norm_0: tensor(237.6714, device='cuda:0') (tensor(-61.7537, device='cuda:0')) | > avg_loss_gen: 2.1567012860741412 (-0.008344388343919373) | > avg_loss_kl: 2.0603507169535464 (-0.04389159276451915) | > avg_loss_feat: 3.995386429235967 (+0.056942526723295916) | > avg_loss_mel: 16.91589787980201 (-0.35796186957559684) | > avg_loss_duration: 1.5502150696767891 (-0.011519393450776905) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67855117690395 (-0.364774892027949) | > avg_grad_norm_1: tensor(5692.8472, device='cuda:0') (tensor(-3104.1050, device='cuda:0'))  > EPOCH: 1279/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:23:13)   --> TIME: 2025-02-25 21:23:20 -- STEP: 11/72 -- GLOBAL_STEP: 1092100 | > loss_disc: 2.776670217514038 (2.698912577195601) | > loss_disc_real_0: 0.3873637318611145 (0.20566895231604576) | > loss_disc_real_1: 0.18819335103034973 (0.14777975461699747) | > loss_disc_real_2: 0.263268381357193 (0.25419464300979266) | > loss_disc_real_3: 0.25119712948799133 (0.2596675062721426) | > loss_disc_real_4: 0.26349198818206787 (0.2356950816783038) | > loss_disc_real_5: 0.2523854672908783 (0.2558035430583087) | > loss_0: 2.776670217514038 (2.698912577195601) | > grad_norm_0: tensor(673.8420, device='cuda:0') (tensor(491.3292, device='cuda:0')) | > loss_gen: 2.222956657409668 (2.15992073579268) | > loss_kl: 1.8847463130950928 (1.8778096762570469) | > loss_feat: 3.41841459274292 (3.7789799950339575) | > loss_mel: 16.79654312133789 (16.967951514504172) | > loss_duration: 1.489764928817749 (1.5066126043146306) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.812427520751953 (26.29127450422807) | > grad_norm_1: tensor(3946.5068, device='cuda:0') (tensor(6440.5215, device='cuda:0')) | > current_lr_0: 0.00017044835941706807 | > current_lr_1: 0.00017044835941706807 | > step_time: 0.4256 (0.4375696398995139) | > loader_time: 0.0064 (0.006926081397316672)  --> TIME: 2025-02-25 21:23:43 -- STEP: 61/72 -- GLOBAL_STEP: 1092150 | > loss_disc: 2.4815378189086914 (2.6630878526656354) | > loss_disc_real_0: 0.09197039157152176 (0.10659062437957427) | > loss_disc_real_1: 0.2271939218044281 (0.22404219600997988) | > loss_disc_real_2: 0.25945696234703064 (0.21696689908133174) | > loss_disc_real_3: 0.24788278341293335 (0.24487691287134514) | > loss_disc_real_4: 0.2498159408569336 (0.2601898720518488) | > loss_disc_real_5: 0.2598137855529785 (0.24192435721882055) | > loss_0: 2.4815378189086914 (2.6630878526656354) | > grad_norm_0: tensor(170.2934, device='cuda:0') (tensor(262.1357, device='cuda:0')) | > loss_gen: 2.46079683303833 (2.1452594409223464) | > loss_kl: 2.1543712615966797 (2.0883418634289597) | > loss_feat: 4.17761754989624 (4.013996038280549) | > loss_mel: 17.179445266723633 (17.304478551520663) | > loss_duration: 1.5992978811264038 (1.537989170824895) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.571529388427734 (27.090064939905385) | > grad_norm_1: tensor(4066.2400, device='cuda:0') (tensor(6636.9150, device='cuda:0')) | > current_lr_0: 0.00017044835941706807 | > current_lr_1: 0.00017044835941706807 | > step_time: 0.4813 (0.44836938576620133) | > loader_time: 0.011 (0.008484476902445805) --> EVAL PERFORMANCE | > avg_loader_time: 0.00890309374097367 (+0.0003680007558473364) | > avg_step_time: 0.45670421022764396 (+0.004182567059154263) | > avg_loss_disc: 2.6507687031383247 (+0.07536052650129355) | > avg_loss_disc_real_0: 0.10134397984915217 (-0.008202647527968385) | > avg_loss_disc_real_1: 0.2235055469291311 (+0.004764734754260175) | > avg_loss_disc_real_2: 0.22126795634837212 (+0.005590387423273502) | > avg_loss_disc_real_3: 0.24322143511872896 (+0.008201525244914298) | > avg_loss_disc_real_4: 0.2557761121383856 (+0.012531272752184353) | > avg_loss_disc_real_5: 0.2446677327995569 (-0.0006930204764218351) | > avg_loss_0: 2.6507687031383247 (+0.07536052650129355) | > avg_grad_norm_0: tensor(256.0540, device='cuda:0') (+tensor(18.3826, device='cuda:0')) | > avg_loss_gen: 2.1460773424363473 (-0.010623943637793953) | > avg_loss_kl: 2.099316931106674 (+0.03896621415312751) | > avg_loss_feat: 3.976715695690101 (-0.018670733545866014) | > avg_loss_mel: 17.25559261483205 (+0.3396947350300401) | > avg_loss_duration: 1.5469648686932844 (-0.003250200983504703) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.024667336907186 (+0.3461161600032341) | > avg_grad_norm_1: tensor(6771.1836, device='cuda:0') (+tensor(1078.3364, device='cuda:0'))  > EPOCH: 1280/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:23:50)   --> TIME: 2025-02-25 21:24:10 -- STEP: 39/72 -- GLOBAL_STEP: 1092200 | > loss_disc: 2.4418082237243652 (2.6551828934596133) | > loss_disc_real_0: 0.12744057178497314 (0.13980585795182449) | > loss_disc_real_1: 0.24438223242759705 (0.22381445994743934) | > loss_disc_real_2: 0.21674960851669312 (0.23421067725389436) | > loss_disc_real_3: 0.24197809398174286 (0.2366785731835243) | > loss_disc_real_4: 0.24525706470012665 (0.24026120969882378) | > loss_disc_real_5: 0.2354067713022232 (0.24647227006080824) | > loss_0: 2.4418082237243652 (2.6551828934596133) | > grad_norm_0: tensor(169.5073, device='cuda:0') (tensor(250.9258, device='cuda:0')) | > loss_gen: 2.571146249771118 (2.133214244475731) | > loss_kl: 1.9790960550308228 (1.9689967815692608) | > loss_feat: 4.404001712799072 (3.864026687084101) | > loss_mel: 16.840194702148438 (16.766374098948944) | > loss_duration: 1.5310118198394775 (1.5026128047551863) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.325450897216797 (26.235224846081856) | > grad_norm_1: tensor(5317.8164, device='cuda:0') (tensor(7509.7671, device='cuda:0')) | > current_lr_0: 0.00017042705337214092 | > current_lr_1: 0.00017042705337214092 | > step_time: 0.4491 (0.43669913976620406) | > loader_time: 0.0085 (0.007245210500863882) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542977588277467 (-0.0003601161526962035) | > avg_step_time: 0.45230924579459175 (-0.004394964433052206) | > avg_loss_disc: 2.6377990682360153 (-0.012969634902309402) | > avg_loss_disc_real_0: 0.1528107523393463 (+0.05146677249019413) | > avg_loss_disc_real_1: 0.22790911017169416 (+0.004403563242563052) | > avg_loss_disc_real_2: 0.23255576676046344 (+0.011287810412091315) | > avg_loss_disc_real_3: 0.2341167148150189 (-0.009104720303710068) | > avg_loss_disc_real_4: 0.2413626296419493 (-0.014413482496436286) | > avg_loss_disc_real_5: 0.24493857798442034 (+0.00027084518486344966) | > avg_loss_0: 2.6377990682360153 (-0.012969634902309402) | > avg_grad_norm_0: tensor(230.0302, device='cuda:0') (tensor(-26.0238, device='cuda:0')) | > avg_loss_gen: 2.1626472909685583 (+0.016569948532211054) | > avg_loss_kl: 2.0123992050197756 (-0.0869177260868983) | > avg_loss_feat: 3.8348483132644438 (-0.14186738242565733) | > avg_loss_mel: 16.914452942324356 (-0.34113967250769406) | > avg_loss_duration: 1.5365326370991452 (-0.010432231594139196) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.460880440725408 (-0.563786896181778) | > avg_grad_norm_1: tensor(6353.2285, device='cuda:0') (tensor(-417.9551, device='cuda:0'))  > EPOCH: 1281/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:24:27)   --> TIME: 2025-02-25 21:24:36 -- STEP: 17/72 -- GLOBAL_STEP: 1092250 | > loss_disc: 2.7729334831237793 (2.6795982473036823) | > loss_disc_real_0: 0.07557031512260437 (0.1366138775997302) | > loss_disc_real_1: 0.15769676864147186 (0.206602081656456) | > loss_disc_real_2: 0.2351091504096985 (0.24119290446533875) | > loss_disc_real_3: 0.21922820806503296 (0.22918129142592936) | > loss_disc_real_4: 0.2315898835659027 (0.24726033123100505) | > loss_disc_real_5: 0.2514994144439697 (0.24320431141292348) | > loss_0: 2.7729334831237793 (2.6795982473036823) | > grad_norm_0: tensor(211.9324, device='cuda:0') (tensor(294.1551, device='cuda:0')) | > loss_gen: 1.7171952724456787 (2.0385102454353783) | > loss_kl: 2.135685682296753 (1.9272238787482767) | > loss_feat: 3.3821322917938232 (3.805660430122824) | > loss_mel: 17.30975914001465 (16.86730121163761) | > loss_duration: 1.53786301612854 (1.4889491936739754) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.0826358795166 (26.127644707174863) | > grad_norm_1: tensor(15109.4521, device='cuda:0') (tensor(7118.4810, device='cuda:0')) | > current_lr_0: 0.0001704057499904694 | > current_lr_1: 0.0001704057499904694 | > step_time: 0.4247 (0.4336988505195169) | > loader_time: 0.0059 (0.00711522382848403)  --> TIME: 2025-02-25 21:25:00 -- STEP: 67/72 -- GLOBAL_STEP: 1092300 | > loss_disc: 2.642503261566162 (2.6139636466752236) | > loss_disc_real_0: 0.12130270898342133 (0.1495007477255899) | > loss_disc_real_1: 0.21698103845119476 (0.21201544243897966) | > loss_disc_real_2: 0.22744888067245483 (0.23401127555477086) | > loss_disc_real_3: 0.22546392679214478 (0.23062215500803135) | > loss_disc_real_4: 0.2425990253686905 (0.24437848778802956) | > loss_disc_real_5: 0.23516009747982025 (0.24629908777884582) | > loss_0: 2.642503261566162 (2.6139636466752236) | > grad_norm_0: tensor(315.9601, device='cuda:0') (tensor(276.0477, device='cuda:0')) | > loss_gen: 2.0921449661254883 (2.1551251358060703) | > loss_kl: 2.2689719200134277 (2.0048226153672632) | > loss_feat: 3.807490587234497 (3.7888437384989726) | > loss_mel: 16.980947494506836 (16.704494832166986) | > loss_duration: 1.698477864265442 (1.5301324057934889) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.848033905029297 (26.183418928687253) | > grad_norm_1: tensor(12545.8242, device='cuda:0') (tensor(6919.8359, device='cuda:0')) | > current_lr_0: 0.0001704057499904694 | > current_lr_1: 0.0001704057499904694 | > step_time: 0.4919 (0.44977728644413734) | > loader_time: 0.0123 (0.008489021614416321) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865356015487456 (+0.00011058256659709359) | > avg_step_time: 0.4528468696164413 (+0.0005376238218495755) | > avg_loss_disc: 2.620471245805982 (-0.01732782243003328) | > avg_loss_disc_real_0: 0.1491611394768869 (-0.003649612862459406) | > avg_loss_disc_real_1: 0.21479885930746376 (-0.0131102508642304) | > avg_loss_disc_real_2: 0.23575300859733367 (+0.0031972418368702293) | > avg_loss_disc_real_3: 0.23075651429908375 (-0.0033602005159351356) | > avg_loss_disc_real_4: 0.2444832098316139 (+0.0031205801896646013) | > avg_loss_disc_real_5: 0.24585946303018383 (+0.0009208850457634865) | > avg_loss_0: 2.620471245805982 (-0.01732782243003328) | > avg_grad_norm_0: tensor(266.9568, device='cuda:0') (+tensor(36.9266, device='cuda:0')) | > avg_loss_gen: 2.1543989685219787 (-0.008248322446579692) | > avg_loss_kl: 2.0636996064387576 (+0.051300401418981956) | > avg_loss_feat: 3.780177055949896 (-0.054671257314547805) | > avg_loss_mel: 16.922818183898926 (+0.008365241574569637) | > avg_loss_duration: 1.5460010431182216 (+0.009468406019076392) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.46709509298835 (+0.006214652262944043) | > avg_grad_norm_1: tensor(6907.2705, device='cuda:0') (+tensor(554.0420, device='cuda:0'))  > EPOCH: 1282/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:25:04)   --> TIME: 2025-02-25 21:25:26 -- STEP: 45/72 -- GLOBAL_STEP: 1092350 | > loss_disc: 2.6130573749542236 (2.5603782494862872) | > loss_disc_real_0: 0.07616569101810455 (0.10138672126664056) | > loss_disc_real_1: 0.22425521910190582 (0.23027485774623024) | > loss_disc_real_2: 0.2224249392747879 (0.21766042345099978) | > loss_disc_real_3: 0.23810464143753052 (0.23232723673184713) | > loss_disc_real_4: 0.2372385710477829 (0.23407007853190104) | > loss_disc_real_5: 0.24042661488056183 (0.24476984375052985) | > loss_0: 2.6130573749542236 (2.5603782494862872) | > grad_norm_0: tensor(298.7128, device='cuda:0') (tensor(246.5656, device='cuda:0')) | > loss_gen: 2.1075894832611084 (2.158207501305475) | > loss_kl: 2.060197591781616 (2.281810622745089) | > loss_feat: 4.0579609870910645 (3.989925061331855) | > loss_mel: 16.946535110473633 (18.091584565904412) | > loss_duration: 1.5473052263259888 (1.5650257560941907) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.719589233398438 (28.086553446451823) | > grad_norm_1: tensor(8770.7471, device='cuda:0') (tensor(8449.4053, device='cuda:0')) | > current_lr_0: 0.0001703844492717206 | > current_lr_1: 0.0001703844492717206 | > step_time: 0.4811 (0.4409842120276557) | > loader_time: 0.0088 (0.007518418629964193) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613740894156445 (-3.9819260718115465e-05) | > avg_step_time: 0.4527392353810055 (-0.00010763423543580508) | > avg_loss_disc: 2.5278467326097083 (-0.09262451319627374) | > avg_loss_disc_real_0: 0.09435097941420448 (-0.05481016006268241) | > avg_loss_disc_real_1: 0.22402426096755015 (+0.009225401660086396) | > avg_loss_disc_real_2: 0.22245717573333795 (-0.013295832863995716) | > avg_loss_disc_real_3: 0.2302349780646848 (-0.0005215362343989616) | > avg_loss_disc_real_4: 0.2358157405970802 (-0.008667469234533709) | > avg_loss_disc_real_5: 0.24365655846998724 (-0.002202904560196589) | > avg_loss_0: 2.5278467326097083 (-0.09262451319627374) | > avg_grad_norm_0: tensor(254.2860, device='cuda:0') (tensor(-12.6708, device='cuda:0')) | > avg_loss_gen: 2.175524510128398 (+0.02112554160641933) | > avg_loss_kl: 2.24045061897224 (+0.17675101253348258) | > avg_loss_feat: 3.986259685435765 (+0.20608262948586908) | > avg_loss_mel: 17.627162355772217 (+0.7043441718732915) | > avg_loss_duration: 1.5713323975952578 (+0.02533135447703616) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.600729525928767 (+1.1336344329404149) | > avg_grad_norm_1: tensor(8656.8086, device='cuda:0') (+tensor(1749.5381, device='cuda:0'))  > EPOCH: 1283/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:25:41)   --> TIME: 2025-02-25 21:25:53 -- STEP: 23/72 -- GLOBAL_STEP: 1092400 | > loss_disc: 2.4185476303100586 (2.4511042366857114) | > loss_disc_real_0: 0.06471263617277145 (0.075018758683101) | > loss_disc_real_1: 0.2156650424003601 (0.20968303019585816) | > loss_disc_real_2: 0.22372698783874512 (0.21605754870435465) | > loss_disc_real_3: 0.24897891283035278 (0.22778783025948898) | > loss_disc_real_4: 0.2176186740398407 (0.23760231746279675) | > loss_disc_real_5: 0.21200820803642273 (0.23776487423026044) | > loss_0: 2.4185476303100586 (2.4511042366857114) | > grad_norm_0: tensor(164.4249, device='cuda:0') (tensor(298.6220, device='cuda:0')) | > loss_gen: 2.3603904247283936 (2.2512252745421044) | > loss_kl: 2.0846199989318848 (1.940858887589496) | > loss_feat: 4.471105575561523 (4.262332688207212) | > loss_mel: 15.621684074401855 (16.764183873715606) | > loss_duration: 1.530657410621643 (1.4966526394305022) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.068456649780273 (26.715253332386848) | > grad_norm_1: tensor(5264.3447, device='cuda:0') (tensor(9961.2002, device='cuda:0')) | > current_lr_0: 0.00017036315121556162 | > current_lr_1: 0.00017036315121556162 | > step_time: 0.4293 (0.4327599173006804) | > loader_time: 0.0063 (0.006862495256506879) --> EVAL PERFORMANCE | > avg_loader_time: 0.00877268885222959 (+0.00015894795807314469) | > avg_step_time: 0.45388931959447726 (+0.001150084213471736) | > avg_loss_disc: 2.501742960701525 (-0.026103771908183138) | > avg_loss_disc_real_0: 0.09098376716736337 (-0.0033672122468411114) | > avg_loss_disc_real_1: 0.216702357983925 (-0.007321902983625145) | > avg_loss_disc_real_2: 0.22581763754428272 (+0.003360461810944765) | > avg_loss_disc_real_3: 0.2289124137498963 (-0.0013225643147884991) | > avg_loss_disc_real_4: 0.24120387518909617 (+0.005388134592015981) | > avg_loss_disc_real_5: 0.24557697940880144 (+0.001920420938814199) | > avg_loss_0: 2.501742960701525 (-0.026103771908183138) | > avg_grad_norm_0: tensor(298.7850, device='cuda:0') (+tensor(44.4990, device='cuda:0')) | > avg_loss_gen: 2.2264695923093325 (+0.05094508218093452) | > avg_loss_kl: 1.9991543393739513 (-0.24129627959828892) | > avg_loss_feat: 4.0571892597305945 (+0.07092957429482949) | > avg_loss_mel: 16.602698017174088 (-1.0244643385981291) | > avg_loss_duration: 1.533009475385639 (-0.03832292220961886) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.418520699084645 (-1.1822088268441213) | > avg_grad_norm_1: tensor(9027.1348, device='cuda:0') (+tensor(370.3262, device='cuda:0'))  > EPOCH: 1284/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:26:18)   --> TIME: 2025-02-25 21:26:20 -- STEP: 1/72 -- GLOBAL_STEP: 1092450 | > loss_disc: 2.5232481956481934 (2.5232481956481934) | > loss_disc_real_0: 0.0659887045621872 (0.0659887045621872) | > loss_disc_real_1: 0.22130480408668518 (0.22130480408668518) | > loss_disc_real_2: 0.20612436532974243 (0.20612436532974243) | > loss_disc_real_3: 0.2028423249721527 (0.2028423249721527) | > loss_disc_real_4: 0.25127387046813965 (0.25127387046813965) | > loss_disc_real_5: 0.25803372263908386 (0.25803372263908386) | > loss_0: 2.5232481956481934 (2.5232481956481934) | > grad_norm_0: tensor(351.8026, device='cuda:0') (tensor(351.8026, device='cuda:0')) | > loss_gen: 2.1544833183288574 (2.1544833183288574) | > loss_kl: 1.9069534540176392 (1.9069534540176392) | > loss_feat: 3.9822580814361572 (3.9822580814361572) | > loss_mel: 16.812057495117188 (16.812057495117188) | > loss_duration: 1.5192805528640747 (1.5192805528640747) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.37503433227539 (26.37503433227539) | > grad_norm_1: tensor(10549.2314, device='cuda:0') (tensor(10549.2314, device='cuda:0')) | > current_lr_0: 0.00017034185582165968 | > current_lr_1: 0.00017034185582165968 | > step_time: 0.434 (0.43404698371887207) | > loader_time: 0.0066 (0.006649971008300781)  --> TIME: 2025-02-25 21:26:43 -- STEP: 51/72 -- GLOBAL_STEP: 1092500 | > loss_disc: 2.5875918865203857 (2.577041424956976) | > loss_disc_real_0: 0.07189271599054337 (0.11041644902205934) | > loss_disc_real_1: 0.40419456362724304 (0.21920403297625335) | > loss_disc_real_2: 0.26162993907928467 (0.2587072260531724) | > loss_disc_real_3: 0.18280495703220367 (0.23407519331165388) | > loss_disc_real_4: 0.2526988983154297 (0.24469916551720863) | > loss_disc_real_5: 0.2640998959541321 (0.24802575888586978) | > loss_0: 2.5875918865203857 (2.577041424956976) | > grad_norm_0: tensor(179.8405, device='cuda:0') (tensor(303.0463, device='cuda:0')) | > loss_gen: 2.486867904663086 (2.2825809857424564) | > loss_kl: 1.8886919021606445 (1.9411416544633753) | > loss_feat: 3.766761302947998 (4.121125688739852) | > loss_mel: 16.03725242614746 (16.613228480021153) | > loss_duration: 1.537675142288208 (1.5105729290083343) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.717248916625977 (26.46864995769426) | > grad_norm_1: tensor(6454.3564, device='cuda:0') (tensor(7583.6572, device='cuda:0')) | > current_lr_0: 0.00017034185582165968 | > current_lr_1: 0.00017034185582165968 | > step_time: 0.448 (0.44367175476223814) | > loader_time: 0.009 (0.007843017578125) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653026231577697 (-0.00011966262065189227) | > avg_step_time: 0.4538794235444405 (-9.89605003676175e-06) | > avg_loss_disc: 2.5944620723455722 (+0.09271911164404711) | > avg_loss_disc_real_0: 0.11624461821687053 (+0.025260851049507163) | > avg_loss_disc_real_1: 0.22282079103547084 (+0.00611843305154583) | > avg_loss_disc_real_2: 0.2565945127690342 (+0.030776875224751504) | > avg_loss_disc_real_3: 0.2295271483525424 (+0.0006147346026461054) | > avg_loss_disc_real_4: 0.2430063049558183 (+0.0018024297667221245) | > avg_loss_disc_real_5: 0.2442675614860696 (-0.0013094179227318425) | > avg_loss_0: 2.5944620723455722 (+0.09271911164404711) | > avg_grad_norm_0: tensor(323.1971, device='cuda:0') (+tensor(24.4121, device='cuda:0')) | > avg_loss_gen: 2.227513051368821 (+0.0010434590594883275) | > avg_loss_kl: 1.975715249357089 (-0.023439090016862174) | > avg_loss_feat: 4.0057521336515185 (-0.05143712607907602) | > avg_loss_mel: 16.542408177550413 (-0.060289839623674624) | > avg_loss_duration: 1.5306314548975988 (-0.0023780204880401445) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.28202008529448 (-0.1365006137901652) | > avg_grad_norm_1: tensor(7832.0898, device='cuda:0') (tensor(-1195.0449, device='cuda:0'))  > EPOCH: 1285/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:26:55)   --> TIME: 2025-02-25 21:27:10 -- STEP: 29/72 -- GLOBAL_STEP: 1092550 | > loss_disc: 2.6183838844299316 (2.5481725643421043) | > loss_disc_real_0: 0.10073940455913544 (0.14763680884036529) | > loss_disc_real_1: 0.19864895939826965 (0.21822204363757167) | > loss_disc_real_2: 0.21693585813045502 (0.20878242264533864) | > loss_disc_real_3: 0.23420949280261993 (0.22358075503645272) | > loss_disc_real_4: 0.2592553496360779 (0.2206568661434897) | > loss_disc_real_5: 0.2527291774749756 (0.22893571853637695) | > loss_0: 2.6183838844299316 (2.5481725643421043) | > grad_norm_0: tensor(87.7391, device='cuda:0') (tensor(335.6982, device='cuda:0')) | > loss_gen: 2.0866634845733643 (2.220750800494491) | > loss_kl: 2.1130921840667725 (2.5604653440672775) | > loss_feat: 3.8245177268981934 (4.195337813476036) | > loss_mel: 17.477266311645508 (18.801694738453833) | > loss_duration: 1.53043794631958 (1.5740004366841809) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.0319766998291 (29.35224907973717) | > grad_norm_1: tensor(8617.2500, device='cuda:0') (tensor(8168.6465, device='cuda:0')) | > current_lr_0: 0.00017032056308968196 | > current_lr_1: 0.00017032056308968196 | > step_time: 0.4492 (0.4340506011042102) | > loader_time: 0.0078 (0.006998086797780004) --> EVAL PERFORMANCE | > avg_loader_time: 0.008578616128840916 (-7.441010273678134e-05) | > avg_step_time: 0.4543481712609949 (+0.0004687477165544185) | > avg_loss_disc: 2.585098679636565 (-0.009363392709007101) | > avg_loss_disc_real_0: 0.1224289160274284 (+0.006184297810557876) | > avg_loss_disc_real_1: 0.226638074911816 (+0.0038172838763451677) | > avg_loss_disc_real_2: 0.23124036633632553 (-0.025354146432708696) | > avg_loss_disc_real_3: 0.2314962541133585 (+0.0019691057608161033) | > avg_loss_disc_real_4: 0.2368777877008411 (-0.006128517254977178) | > avg_loss_disc_real_5: 0.24253812264388716 (-0.0017294388421824347) | > avg_loss_0: 2.585098679636565 (-0.009363392709007101) | > avg_grad_norm_0: tensor(248.6325, device='cuda:0') (tensor(-74.5646, device='cuda:0')) | > avg_loss_gen: 2.1950870466903907 (-0.032426004678430154) | > avg_loss_kl: 2.347859691566145 (+0.37214444220905585) | > avg_loss_feat: 3.9836230244435056 (-0.02212910920801292) | > avg_loss_mel: 17.716068684215276 (+1.1736605066648629) | > avg_loss_duration: 1.5723188997994006 (+0.04168744490180187) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.814957497825084 (+1.532937412530604) | > avg_grad_norm_1: tensor(7395.2729, device='cuda:0') (tensor(-436.8169, device='cuda:0'))  > EPOCH: 1286/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:27:32)   --> TIME: 2025-02-25 21:27:37 -- STEP: 7/72 -- GLOBAL_STEP: 1092600 | > loss_disc: 2.9206323623657227 (2.518662997654506) | > loss_disc_real_0: 0.08258488774299622 (0.05069110116788319) | > loss_disc_real_1: 0.19021807610988617 (0.2318114936351776) | > loss_disc_real_2: 0.2844711244106293 (0.24257637134620122) | > loss_disc_real_3: 0.21988165378570557 (0.2322333093200411) | > loss_disc_real_4: 0.23660215735435486 (0.23892944625445775) | > loss_disc_real_5: 0.24758754670619965 (0.24560239059584482) | > loss_0: 2.9206323623657227 (2.518662997654506) | > grad_norm_0: tensor(594.3156, device='cuda:0') (tensor(152.3506, device='cuda:0')) | > loss_gen: 1.6251548528671265 (2.2864409685134888) | > loss_kl: 1.9519191980361938 (1.9949767419270106) | > loss_feat: 3.091813564300537 (4.3867358820778986) | > loss_mel: 17.33057403564453 (16.877192905970983) | > loss_duration: 1.4967734813690186 (1.5050380229949951) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.496232986450195 (27.050383704049246) | > grad_norm_1: tensor(269.0804, device='cuda:0') (tensor(3473.1787, device='cuda:0')) | > current_lr_0: 0.00017029927301929573 | > current_lr_1: 0.00017029927301929573 | > step_time: 0.4415 (0.42653417587280273) | > loader_time: 0.0068 (0.006118195397513253)  --> TIME: 2025-02-25 21:28:00 -- STEP: 57/72 -- GLOBAL_STEP: 1092650 | > loss_disc: 2.654409170150757 (2.649240159151847) | > loss_disc_real_0: 0.2401255965232849 (0.0942146721246995) | > loss_disc_real_1: 0.254848837852478 (0.2337382707679481) | > loss_disc_real_2: 0.26220816373825073 (0.23017881183247818) | > loss_disc_real_3: 0.23924675583839417 (0.23277500794644942) | > loss_disc_real_4: 0.2607055902481079 (0.23928878992273098) | > loss_disc_real_5: 0.2562251389026642 (0.24201089379034543) | > loss_0: 2.654409170150757 (2.649240159151847) | > grad_norm_0: tensor(578.2172, device='cuda:0') (tensor(361.2845, device='cuda:0')) | > loss_gen: 2.358586072921753 (2.0222777860206476) | > loss_kl: 2.208597421646118 (2.2866693003135814) | > loss_feat: 3.767047166824341 (3.856497488523785) | > loss_mel: 16.586318969726562 (17.561595214040654) | > loss_duration: 1.5908989906311035 (1.5704859557904696) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.51144790649414 (27.297525840893126) | > grad_norm_1: tensor(5274.4609, device='cuda:0') (tensor(9873.0654, device='cuda:0')) | > current_lr_0: 0.00017029927301929573 | > current_lr_1: 0.00017029927301929573 | > step_time: 0.4515 (0.4414470781359756) | > loader_time: 0.0095 (0.00798880844785456) --> EVAL PERFORMANCE | > avg_loader_time: 0.008603821338062553 (+2.520520922163691e-05) | > avg_step_time: 0.44969278322139256 (-0.0046553880396023595) | > avg_loss_disc: 2.6428512015812835 (+0.05775252194471836) | > avg_loss_disc_real_0: 0.10107079310946057 (-0.021358122917967834) | > avg_loss_disc_real_1: 0.23130260869650773 (+0.004664533784691727) | > avg_loss_disc_real_2: 0.2322416387393441 (+0.0010012724030185605) | > avg_loss_disc_real_3: 0.23302496424023533 (+0.001528710126876831) | > avg_loss_disc_real_4: 0.2403088241395816 (+0.003431036438740487) | > avg_loss_disc_real_5: 0.2426388830366269 (+0.00010076039273973492) | > avg_loss_0: 2.6428512015812835 (+0.05775252194471836) | > avg_grad_norm_0: tensor(368.6793, device='cuda:0') (+tensor(120.0467, device='cuda:0')) | > avg_loss_gen: 2.044443125456153 (-0.15064392123423787) | > avg_loss_kl: 2.3668623205641626 (+0.019002628998017634) | > avg_loss_feat: 3.801710155648245 (-0.1819128687952607) | > avg_loss_mel: 17.8501179117552 (+0.1340492275399221) | > avg_loss_duration: 1.5811363307523056 (+0.008817430952904992) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.6442700238295 (-0.17068747399558504) | > avg_grad_norm_1: tensor(10035.8389, device='cuda:0') (+tensor(2640.5659, device='cuda:0'))  > EPOCH: 1287/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:28:09)   --> TIME: 2025-02-25 21:28:26 -- STEP: 35/72 -- GLOBAL_STEP: 1092700 | > loss_disc: 2.555518627166748 (2.6102593081338066) | > loss_disc_real_0: 0.10568651556968689 (0.13577123986823217) | > loss_disc_real_1: 0.2102358639240265 (0.2253821930715016) | > loss_disc_real_2: 0.28367432951927185 (0.23096312710217068) | > loss_disc_real_3: 0.2638188302516937 (0.23978215200560435) | > loss_disc_real_4: 0.2633529305458069 (0.23590810128620693) | > loss_disc_real_5: 0.24672280251979828 (0.24618374364716666) | > loss_0: 2.555518627166748 (2.6102593081338066) | > grad_norm_0: tensor(399.8342, device='cuda:0') (tensor(284.1651, device='cuda:0')) | > loss_gen: 2.3871994018554688 (2.280078312328884) | > loss_kl: 2.1531567573547363 (2.178194958823068) | > loss_feat: 3.9080281257629395 (4.162606171199253) | > loss_mel: 15.849723815917969 (17.701867512294232) | > loss_duration: 1.5351417064666748 (1.52046035357884) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.833248138427734 (27.843207223074778) | > grad_norm_1: tensor(4764.1602, device='cuda:0') (tensor(5219.5532, device='cuda:0')) | > current_lr_0: 0.0001702779856101683 | > current_lr_1: 0.0001702779856101683 | > step_time: 0.4452 (0.4367544514792306) | > loader_time: 0.0082 (0.006995037623814174) --> EVAL PERFORMANCE | > avg_loader_time: 0.008473745534117796 (-0.00013007580394475658) | > avg_step_time: 0.450642874543096 (+0.0009500913217034279) | > avg_loss_disc: 2.626737074113228 (-0.016114127468055628) | > avg_loss_disc_real_0: 0.16491521810981594 (+0.06384442500035536) | > avg_loss_disc_real_1: 0.22178170210878614 (-0.00952090658772159) | > avg_loss_disc_real_2: 0.23433313424318608 (+0.0020914955038419947) | > avg_loss_disc_real_3: 0.2365340854920132 (+0.0035091212517778803) | > avg_loss_disc_real_4: 0.23928750400811855 (-0.001021320131463055) | > avg_loss_disc_real_5: 0.24212551221881115 (-0.0005133708178157448) | > avg_loss_0: 2.626737074113228 (-0.016114127468055628) | > avg_grad_norm_0: tensor(319.6212, device='cuda:0') (tensor(-49.0581, device='cuda:0')) | > avg_loss_gen: 2.1983607302249317 (+0.15391760476877892) | > avg_loss_kl: 2.1393624124392656 (-0.22749990812489695) | > avg_loss_feat: 3.9055515645255503 (+0.10384140887730542) | > avg_loss_mel: 17.24509677081041 (-0.605021140944789) | > avg_loss_duration: 1.5463691916264277 (-0.03476713912587792) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.034740447998047 (-0.6095295758314521) | > avg_grad_norm_1: tensor(5863.6650, device='cuda:0') (tensor(-4172.1738, device='cuda:0'))  > EPOCH: 1288/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:28:45)   --> TIME: 2025-02-25 21:28:53 -- STEP: 13/72 -- GLOBAL_STEP: 1092750 | > loss_disc: 2.7211296558380127 (2.5488532323103685) | > loss_disc_real_0: 0.054079920053482056 (0.15188578688181364) | > loss_disc_real_1: 0.39326605200767517 (0.21892588986800268) | > loss_disc_real_2: 0.14538894593715668 (0.20598445259607756) | > loss_disc_real_3: 0.15476959943771362 (0.22454456984996796) | > loss_disc_real_4: 0.2272663712501526 (0.23986962323005384) | > loss_disc_real_5: 0.26559796929359436 (0.24272087101752943) | > loss_0: 2.7211296558380127 (2.5488532323103685) | > grad_norm_0: tensor(256.1455, device='cuda:0') (tensor(293.0085, device='cuda:0')) | > loss_gen: 1.951466679573059 (2.2086404103499193) | > loss_kl: 1.924189805984497 (1.9897279464281523) | > loss_feat: 3.8432154655456543 (3.997170319923988) | > loss_mel: 15.907307624816895 (16.797556290259728) | > loss_duration: 1.4376318454742432 (1.4855455068441539) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.063812255859375 (26.478640482975887) | > grad_norm_1: tensor(5940.0156, device='cuda:0') (tensor(5708.9683, device='cuda:0')) | > current_lr_0: 0.00017025670086196702 | > current_lr_1: 0.00017025670086196702 | > step_time: 0.4652 (0.43409998600299543) | > loader_time: 0.0057 (0.006583323845496545)  --> TIME: 2025-02-25 21:29:16 -- STEP: 63/72 -- GLOBAL_STEP: 1092800 | > loss_disc: 2.9177780151367188 (2.6772053658016146) | > loss_disc_real_0: 0.13749320805072784 (0.13664401162947934) | > loss_disc_real_1: 0.14963574707508087 (0.23277366989188714) | > loss_disc_real_2: 0.27015766501426697 (0.22753549426321) | > loss_disc_real_3: 0.23575454950332642 (0.22716783318254682) | > loss_disc_real_4: 0.26180464029312134 (0.2295575033104609) | > loss_disc_real_5: 0.26695260405540466 (0.24024614407902672) | > loss_0: 2.9177780151367188 (2.6772053658016146) | > grad_norm_0: tensor(223.4460, device='cuda:0') (tensor(339.8407, device='cuda:0')) | > loss_gen: 1.6397087574005127 (2.024135727731009) | > loss_kl: 2.129185676574707 (2.0615280923389254) | > loss_feat: 2.9895122051239014 (3.7854118195791093) | > loss_mel: 16.601802825927734 (16.780957161434113) | > loss_duration: 1.5932581424713135 (1.5315805276234944) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.953468322753906 (26.18361342899383) | > grad_norm_1: tensor(1070.9320, device='cuda:0') (tensor(8018.4858, device='cuda:0')) | > current_lr_0: 0.00017025670086196702 | > current_lr_1: 0.00017025670086196702 | > step_time: 0.5017 (0.4464579082670666) | > loader_time: 0.0111 (0.008268174670991443) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655786514282225 (+0.00018204098016442842) | > avg_step_time: 0.4541061193170682 (+0.003463244773972196) | > avg_loss_disc: 2.6800421224513524 (+0.05330504833812455) | > avg_loss_disc_real_0: 0.15095219178728658 (-0.013963026322529354) | > avg_loss_disc_real_1: 0.23757667545701405 (+0.01579497334822791) | > avg_loss_disc_real_2: 0.22906231397474316 (-0.005270820268442916) | > avg_loss_disc_real_3: 0.22792030199312827 (-0.008613783498884947) | > avg_loss_disc_real_4: 0.23102836982465128 (-0.00825913418346727) | > avg_loss_disc_real_5: 0.2410405200551933 (-0.0010849921636178483) | > avg_loss_0: 2.6800421224513524 (+0.05330504833812455) | > avg_grad_norm_0: tensor(348.7441, device='cuda:0') (+tensor(29.1230, device='cuda:0')) | > avg_loss_gen: 2.0526284651017535 (-0.14573226512317827) | > avg_loss_kl: 2.0783129410004952 (-0.061049471438770375) | > avg_loss_feat: 3.749265852108808 (-0.1562857124167425) | > avg_loss_mel: 16.822945272418814 (-0.42215149839159594) | > avg_loss_duration: 1.5423063140519906 (-0.004062877574437129) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.245458898409993 (-0.7892815495880541) | > avg_grad_norm_1: tensor(8068.2007, device='cuda:0') (+tensor(2204.5356, device='cuda:0'))  > EPOCH: 1289/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:29:22)   --> TIME: 2025-02-25 21:29:42 -- STEP: 41/72 -- GLOBAL_STEP: 1092850 | > loss_disc: 3.149200916290283 (2.608375741214287) | > loss_disc_real_0: 0.5795456767082214 (0.1296727332945277) | > loss_disc_real_1: 0.2084047794342041 (0.23356714008784876) | > loss_disc_real_2: 0.25606268644332886 (0.24579125097612056) | > loss_disc_real_3: 0.23881062865257263 (0.23999678616116688) | > loss_disc_real_4: 0.24640916287899017 (0.25187273178158737) | > loss_disc_real_5: 0.27031075954437256 (0.24327262581848516) | > loss_0: 3.149200916290283 (2.608375741214287) | > grad_norm_0: tensor(347.8309, device='cuda:0') (tensor(207.1712, device='cuda:0')) | > loss_gen: 1.8814975023269653 (2.23977551518417) | > loss_kl: 2.3021795749664307 (2.0866061739805266) | > loss_feat: 2.7277307510375977 (4.08036090106499) | > loss_mel: 17.450708389282227 (17.23955126506526) | > loss_duration: 1.679231882095337 (1.542147572447614) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.04134750366211 (27.188441485893435) | > grad_norm_1: tensor(660.8377, device='cuda:0') (tensor(3074.0857, device='cuda:0')) | > current_lr_0: 0.00017023541877435926 | > current_lr_1: 0.00017023541877435926 | > step_time: 0.4392 (0.4337165588285865) | > loader_time: 0.0094 (0.0073948488002870145) --> EVAL PERFORMANCE | > avg_loader_time: 0.008609553457985461 (-4.6233056296763816e-05) | > avg_step_time: 0.4484979736972863 (-0.005608145619781901) | > avg_loss_disc: 2.632753694561166 (-0.04728842789018639) | > avg_loss_disc_real_0: 0.14401899438193982 (-0.006933197405346758) | > avg_loss_disc_real_1: 0.2293150391377194 (-0.008261636319294663) | > avg_loss_disc_real_2: 0.24555518152848096 (+0.0164928675537378) | > avg_loss_disc_real_3: 0.237330512051851 (+0.009410210058722723) | > avg_loss_disc_real_4: 0.2483101808269259 (+0.017281811002274633) | > avg_loss_disc_real_5: 0.244473344213526 (+0.003432824158332709) | > avg_loss_0: 2.632753694561166 (-0.04728842789018639) | > avg_grad_norm_0: tensor(211.3227, device='cuda:0') (tensor(-137.4214, device='cuda:0')) | > avg_loss_gen: 2.163929007422756 (+0.11130054232100273) | > avg_loss_kl: 2.2772651323130435 (+0.19895219131254827) | > avg_loss_feat: 3.886075526895658 (+0.13680967478685035) | > avg_loss_mel: 17.725939253686178 (+0.9029939812673646) | > avg_loss_duration: 1.6197042146199185 (+0.07739790056792795) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.672913269257883 (+1.4274543708478902) | > avg_grad_norm_1: tensor(3458.4351, device='cuda:0') (tensor(-4609.7656, device='cuda:0'))  > EPOCH: 1290/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:29:59)   --> TIME: 2025-02-25 21:30:09 -- STEP: 19/72 -- GLOBAL_STEP: 1092900 | > loss_disc: 2.5764307975769043 (2.7144773633856523) | > loss_disc_real_0: 0.16158635914325714 (0.18251802399754524) | > loss_disc_real_1: 0.2892725169658661 (0.20792045601104436) | > loss_disc_real_2: 0.22692859172821045 (0.2252446334612997) | > loss_disc_real_3: 0.23202480375766754 (0.22358072039328122) | > loss_disc_real_4: 0.23301762342453003 (0.23850680652417636) | > loss_disc_real_5: 0.25079748034477234 (0.2479295573736492) | > loss_0: 2.5764307975769043 (2.7144773633856523) | > grad_norm_0: tensor(274.6653, device='cuda:0') (tensor(220.8327, device='cuda:0')) | > loss_gen: 2.2404723167419434 (1.9620633439013833) | > loss_kl: 2.2239224910736084 (2.1446493801317716) | > loss_feat: 3.7648797035217285 (3.581926521502043) | > loss_mel: 16.746875762939453 (17.721037513331364) | > loss_duration: 1.5338733196258545 (1.568255524886282) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.51002311706543 (26.97793237786544) | > grad_norm_1: tensor(10885.7471, device='cuda:0') (tensor(5840.6665, device='cuda:0')) | > current_lr_0: 0.00017021413934701247 | > current_lr_1: 0.00017021413934701247 | > step_time: 0.4254 (0.42645095524035004) | > loader_time: 0.006 (0.006660674747667815)  --> TIME: 2025-02-25 21:30:33 -- STEP: 69/72 -- GLOBAL_STEP: 1092950 | > loss_disc: 2.7434768676757812 (2.643931482149207) | > loss_disc_real_0: 0.10941153019666672 (0.157725722839435) | > loss_disc_real_1: 0.23234307765960693 (0.218258002637953) | > loss_disc_real_2: 0.24574996531009674 (0.2321550120478091) | > loss_disc_real_3: 0.2517918348312378 (0.2276256309039351) | > loss_disc_real_4: 0.2297341376543045 (0.24102222379566968) | > loss_disc_real_5: 0.252070814371109 (0.2448043261749157) | > loss_0: 2.7434768676757812 (2.643931482149207) | > grad_norm_0: tensor(511.7297, device='cuda:0') (tensor(192.7138, device='cuda:0')) | > loss_gen: 1.7108697891235352 (2.05947918995567) | > loss_kl: 2.161301851272583 (2.1091237171836523) | > loss_feat: 2.8669562339782715 (3.6638016804404883) | > loss_mel: 16.61799430847168 (17.18658100349315) | > loss_duration: 1.6599712371826172 (1.5730427088944807) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.017091751098633 (26.592028327610183) | > grad_norm_1: tensor(6008.7129, device='cuda:0') (tensor(5378.4976, device='cuda:0')) | > current_lr_0: 0.00017021413934701247 | > current_lr_1: 0.00017021413934701247 | > step_time: 0.5275 (0.4477628659510958) | > loader_time: 0.0128 (0.008406652920487999) --> EVAL PERFORMANCE | > avg_loader_time: 0.008466038905398947 (-0.00014351455258651384) | > avg_step_time: 0.4494845397035841 (+0.0009865660062978177) | > avg_loss_disc: 2.641891361961902 (+0.00913766740073596) | > avg_loss_disc_real_0: 0.15494759729735455 (+0.010928602915414726) | > avg_loss_disc_real_1: 0.2193670403999342 (-0.00994799873778518) | > avg_loss_disc_real_2: 0.2326540384494083 (-0.012901143079072669) | > avg_loss_disc_real_3: 0.22835616599506056 (-0.008974346056790428) | > avg_loss_disc_real_4: 0.24096944298542722 (-0.00734073784149869) | > avg_loss_disc_real_5: 0.24468178207605656 (+0.00020843786253055274) | > avg_loss_0: 2.641891361961902 (+0.00913766740073596) | > avg_grad_norm_0: tensor(192.1637, device='cuda:0') (tensor(-19.1590, device='cuda:0')) | > avg_loss_gen: 2.053911591919375 (-0.11001741550338107) | > avg_loss_kl: 2.1103084154531984 (-0.16695671685984514) | > avg_loss_feat: 3.6606810160086187 (-0.22539451088703943) | > avg_loss_mel: 17.15741293195267 (-0.5685263217335077) | > avg_loss_duration: 1.5729041317818868 (-0.04680008283803172) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.555218092152774 (-1.1176951771051087) | > avg_grad_norm_1: tensor(5616.1377, device='cuda:0') (+tensor(2157.7026, device='cuda:0'))  > EPOCH: 1291/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:30:36)   --> TIME: 2025-02-25 21:30:59 -- STEP: 47/72 -- GLOBAL_STEP: 1093000 | > loss_disc: 2.9984724521636963 (2.6596028551142266) | > loss_disc_real_0: 0.08174589276313782 (0.16045095787403427) | > loss_disc_real_1: 0.25846096873283386 (0.2334451770528834) | > loss_disc_real_2: 0.24756596982479095 (0.23011233895382982) | > loss_disc_real_3: 0.24221090972423553 (0.2477030024883595) | > loss_disc_real_4: 0.23388200998306274 (0.2414281945279304) | > loss_disc_real_5: 0.24883903563022614 (0.2389206255369998) | > loss_0: 2.9984724521636963 (2.6596028551142266) | > grad_norm_0: tensor(623.3245, device='cuda:0') (tensor(283.8665, device='cuda:0')) | > loss_gen: 1.7396557331085205 (2.179600606573389) | > loss_kl: 2.043684244155884 (1.939157303343428) | > loss_feat: 3.420429229736328 (3.8768926326264728) | > loss_mel: 16.39202880859375 (16.612802830148247) | > loss_duration: 1.5479888916015625 (1.521242684506355) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.143787384033203 (26.1296961358253) | > grad_norm_1: tensor(13961.5830, device='cuda:0') (tensor(6070.8584, device='cuda:0')) | > current_lr_0: 0.00017019286257959407 | > current_lr_1: 0.00017019286257959407 | > step_time: 0.4463 (0.4384177136928477) | > loader_time: 0.009 (0.00747303252524518) --> EVAL PERFORMANCE | > avg_loader_time: 0.008500122688186004 (+3.4083782787056385e-05) | > avg_step_time: 0.4515355237772767 (+0.0020509840736925855) | > avg_loss_disc: 2.668983600508999 (+0.02709223854709686) | > avg_loss_disc_real_0: 0.15776447563523974 (+0.002816878337885187) | > avg_loss_disc_real_1: 0.2306895245548705 (+0.01132248415493628) | > avg_loss_disc_real_2: 0.23384015157189167 (+0.0011861131224833732) | > avg_loss_disc_real_3: 0.24227739073021312 (+0.01392122473515256) | > avg_loss_disc_real_4: 0.2401327990310293 (-0.0008366439543979198) | > avg_loss_disc_real_5: 0.24171494430219623 (-0.002966837773860337) | > avg_loss_0: 2.668983600508999 (+0.02709223854709686) | > avg_grad_norm_0: tensor(266.3474, device='cuda:0') (+tensor(74.1837, device='cuda:0')) | > avg_loss_gen: 2.142639977831236 (+0.08872838591186083) | > avg_loss_kl: 2.0121558585637054 (-0.09815255688949298) | > avg_loss_feat: 3.7759720607542655 (+0.11529104474564678) | > avg_loss_mel: 16.78977870941162 (-0.3676342225410494) | > avg_loss_duration: 1.5438822709338762 (-0.029021860848010617) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.264429038679097 (-0.29078905347367723) | > avg_grad_norm_1: tensor(6344.2153, device='cuda:0') (+tensor(728.0776, device='cuda:0'))  > EPOCH: 1292/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:31:12)   --> TIME: 2025-02-25 21:31:25 -- STEP: 25/72 -- GLOBAL_STEP: 1093050 | > loss_disc: 2.610337018966675 (2.577829303741455) | > loss_disc_real_0: 0.0806877464056015 (0.08980220153927804) | > loss_disc_real_1: 0.3725301921367645 (0.23013276994228363) | > loss_disc_real_2: 0.218618243932724 (0.24670329451560974) | > loss_disc_real_3: 0.2861849069595337 (0.2658099442720414) | > loss_disc_real_4: 0.3035116493701935 (0.2608405548334122) | > loss_disc_real_5: 0.2726196050643921 (0.25682470500469207) | > loss_0: 2.610337018966675 (2.577829303741455) | > grad_norm_0: tensor(147.2820, device='cuda:0') (tensor(216.8172, device='cuda:0')) | > loss_gen: 2.223783493041992 (2.28451895236969) | > loss_kl: 2.002718687057495 (2.03871675491333) | > loss_feat: 4.143869876861572 (4.06120228767395) | > loss_mel: 16.301973342895508 (17.12016143798828) | > loss_duration: 1.4984619617462158 (1.5006657981872558) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.170808792114258 (27.00526512145996) | > grad_norm_1: tensor(9897.3066, device='cuda:0') (tensor(6104.1196, device='cuda:0')) | > current_lr_0: 0.0001701715884717716 | > current_lr_1: 0.0001701715884717716 | > step_time: 0.4269 (0.42947007179260255) | > loader_time: 0.0064 (0.006739492416381836) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568471586200553 (+6.83488980145492e-05) | > avg_step_time: 0.4521154047737659 (+0.0005798809964892016) | > avg_loss_disc: 2.622817657363247 (-0.04616594314575195) | > avg_loss_disc_real_0: 0.1256115655349174 (-0.03215291010032234) | > avg_loss_disc_real_1: 0.22350729792050913 (-0.007182226634361355) | > avg_loss_disc_real_2: 0.24069242347294176 (+0.006852271901050089) | > avg_loss_disc_real_3: 0.24851093304828858 (+0.0062335423180754634) | > avg_loss_disc_real_4: 0.25450308851792786 (+0.014370289486898558) | > avg_loss_disc_real_5: 0.2564457795569596 (+0.014730835254763347) | > avg_loss_0: 2.622817657363247 (-0.04616594314575195) | > avg_grad_norm_0: tensor(264.2293, device='cuda:0') (tensor(-2.1180, device='cuda:0')) | > avg_loss_gen: 2.148596204502481 (+0.0059562266712451795) | > avg_loss_kl: 2.0536309379926876 (+0.04147507942898221) | > avg_loss_feat: 3.749268071752199 (-0.026703989002066386) | > avg_loss_mel: 16.928562862772342 (+0.1387841533607208) | > avg_loss_duration: 1.5400557652325697 (-0.003826505701306493) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42011384561028 (+0.15568480693118403) | > avg_grad_norm_1: tensor(8645.7490, device='cuda:0') (+tensor(2301.5337, device='cuda:0'))  > EPOCH: 1293/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:31:49)   --> TIME: 2025-02-25 21:31:52 -- STEP: 3/72 -- GLOBAL_STEP: 1093100 | > loss_disc: 2.4707045555114746 (2.5315209229787192) | > loss_disc_real_0: 0.09408599138259888 (0.08294185002644856) | > loss_disc_real_1: 0.2401534765958786 (0.23183407882849374) | > loss_disc_real_2: 0.23764421045780182 (0.23938457171122232) | > loss_disc_real_3: 0.23756051063537598 (0.23252740999062857) | > loss_disc_real_4: 0.23412933945655823 (0.23323917388916016) | > loss_disc_real_5: 0.2504608631134033 (0.25243882834911346) | > loss_0: 2.4707045555114746 (2.5315209229787192) | > grad_norm_0: tensor(298.5156, device='cuda:0') (tensor(282.2521, device='cuda:0')) | > loss_gen: 2.207087755203247 (2.123663147290548) | > loss_kl: 2.0735244750976562 (2.045046329498291) | > loss_feat: 3.7515366077423096 (3.7459747791290283) | > loss_mel: 16.471834182739258 (17.310681025187176) | > loss_duration: 1.5347141027450562 (1.5246619780858357) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.0386962890625 (26.75002670288086) | > grad_norm_1: tensor(8599.7305, device='cuda:0') (tensor(11836.0293, device='cuda:0')) | > current_lr_0: 0.00017015031702321263 | > current_lr_1: 0.00017015031702321263 | > step_time: 0.432 (0.4263758659362793) | > loader_time: 0.0061 (0.0063355763753255205)  --> TIME: 2025-02-25 21:32:15 -- STEP: 53/72 -- GLOBAL_STEP: 1093150 | > loss_disc: 2.7961249351501465 (2.6249215512905475) | > loss_disc_real_0: 0.37438085675239563 (0.13952376215525394) | > loss_disc_real_1: 0.22813370823860168 (0.22972360618834226) | > loss_disc_real_2: 0.24056397378444672 (0.23796650148787588) | > loss_disc_real_3: 0.22307978570461273 (0.23184501618709205) | > loss_disc_real_4: 0.2337629795074463 (0.2460630235244643) | > loss_disc_real_5: 0.22809500992298126 (0.25084193135207555) | > loss_0: 2.7961249351501465 (2.6249215512905475) | > grad_norm_0: tensor(268.5375, device='cuda:0') (tensor(336.9542, device='cuda:0')) | > loss_gen: 2.2132937908172607 (2.1269495307274577) | > loss_kl: 4.375882625579834 (2.4911292206566293) | > loss_feat: 3.743177652359009 (3.8779382750673115) | > loss_mel: 25.667190551757812 (19.062077360333138) | > loss_duration: 1.8364198207855225 (1.5726014443163603) | > amp_scaler: 8.0 (8.0) | > loss_1: 37.83596420288086 (29.13069566690697) | > grad_norm_1: tensor(6356.7612, device='cuda:0') (tensor(9722.6816, device='cuda:0')) | > current_lr_0: 0.00017015031702321263 | > current_lr_1: 0.00017015031702321263 | > step_time: 0.4465 (0.4393457826578392) | > loader_time: 0.0092 (0.00796168255356123) --> EVAL PERFORMANCE | > avg_loader_time: 0.008756234612263424 (+0.00018776302606287112) | > avg_step_time: 0.4504987521910332 (-0.0016166525827326916) | > avg_loss_disc: 2.6053806895941074 (-0.0174369677691395) | > avg_loss_disc_real_0: 0.14415167429497544 (+0.018540108760058044) | > avg_loss_disc_real_1: 0.2214583481701327 (-0.002048949750376422) | > avg_loss_disc_real_2: 0.23627910521668447 (-0.004413318256257293) | > avg_loss_disc_real_3: 0.2307082366355708 (-0.017802696412717783) | > avg_loss_disc_real_4: 0.2475178814689878 (-0.0069852070489400675) | > avg_loss_disc_real_5: 0.24845416487102778 (-0.007991614685931792) | > avg_loss_0: 2.6053806895941074 (-0.0174369677691395) | > avg_grad_norm_0: tensor(307.9424, device='cuda:0') (+tensor(43.7131, device='cuda:0')) | > avg_loss_gen: 2.165314049787923 (+0.016717845285441957) | > avg_loss_kl: 2.7057075315797836 (+0.652076593587096) | > avg_loss_feat: 3.9623799491936054 (+0.21311187744140625) | > avg_loss_mel: 20.103253001898107 (+3.174690139125765) | > avg_loss_duration: 1.6104227338038701 (+0.07036696857130043) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 30.547077232683208 (+4.1269633870729265) | > avg_grad_norm_1: tensor(9024.9941, device='cuda:0') (+tensor(379.2451, device='cuda:0'))  > EPOCH: 1294/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:32:26)   --> TIME: 2025-02-25 21:32:41 -- STEP: 31/72 -- GLOBAL_STEP: 1093200 | > loss_disc: 2.7657053470611572 (2.5530460111556517) | > loss_disc_real_0: 0.3497670888900757 (0.10505809947367638) | > loss_disc_real_1: 0.40374574065208435 (0.21989926311277574) | > loss_disc_real_2: 0.2381087690591812 (0.2438981398459404) | > loss_disc_real_3: 0.2736998498439789 (0.25485996757784196) | > loss_disc_real_4: 0.23505322635173798 (0.2508840224435252) | > loss_disc_real_5: 0.24464061856269836 (0.24703515729596537) | > loss_0: 2.7657053470611572 (2.5530460111556517) | > grad_norm_0: tensor(471.7790, device='cuda:0') (tensor(242.5028, device='cuda:0')) | > loss_gen: 3.007338523864746 (2.311842829950394) | > loss_kl: 2.1854522228240967 (2.24651635462238) | > loss_feat: 2.8682286739349365 (4.127991268711707) | > loss_mel: 16.472070693969727 (18.587749419673795) | > loss_duration: 1.5552868843078613 (1.538827400053701) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.088375091552734 (28.8129273691485) | > grad_norm_1: tensor(3055.4114, device='cuda:0') (tensor(6357.4058, device='cuda:0')) | > current_lr_0: 0.00017012904823358473 | > current_lr_1: 0.00017012904823358473 | > step_time: 0.4348 (0.43308255749364055) | > loader_time: 0.0091 (0.007097298099148658) --> EVAL PERFORMANCE | > avg_loader_time: 0.008645302812818072 (-0.00011093179944535181) | > avg_step_time: 0.4535252342761402 (+0.0030264820851070295) | > avg_loss_disc: 2.6487475180290114 (+0.043366828434904026) | > avg_loss_disc_real_0: 0.1713773867403957 (+0.02722571244542027) | > avg_loss_disc_real_1: 0.2660866736526221 (+0.044628325482489406) | > avg_loss_disc_real_2: 0.2394629993908842 (+0.0031838941741997306) | > avg_loss_disc_real_3: 0.2574514154817017 (+0.026743178846130894) | > avg_loss_disc_real_4: 0.24609482750086717 (-0.001423053968120619) | > avg_loss_disc_real_5: 0.24488350644078052 (-0.003570658430247259) | > avg_loss_0: 2.6487475180290114 (+0.043366828434904026) | > avg_grad_norm_0: tensor(260.9560, device='cuda:0') (tensor(-46.9864, device='cuda:0')) | > avg_loss_gen: 2.276643593546371 (+0.11132954375844806) | > avg_loss_kl: 2.160198233496975 (-0.5455092980828087) | > avg_loss_feat: 3.855161841486542 (-0.1072181077070633) | > avg_loss_mel: 17.819138808989187 (-2.2841141929089197) | > avg_loss_duration: 1.5641801726650184 (-0.04624256113885172) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.675322572949902 (-2.8717546597333055) | > avg_grad_norm_1: tensor(5433.2515, device='cuda:0') (tensor(-3591.7427, device='cuda:0'))  > EPOCH: 1295/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:33:03)   --> TIME: 2025-02-25 21:33:09 -- STEP: 9/72 -- GLOBAL_STEP: 1093250 | > loss_disc: 2.5705578327178955 (2.56428509288364) | > loss_disc_real_0: 0.05677749961614609 (0.1847134716808796) | > loss_disc_real_1: 0.24970941245555878 (0.2185459252860811) | > loss_disc_real_2: 0.2467966377735138 (0.24864398605293697) | > loss_disc_real_3: 0.24568846821784973 (0.24419322941038343) | > loss_disc_real_4: 0.2553299367427826 (0.2421780824661255) | > loss_disc_real_5: 0.24530097842216492 (0.23948467936780718) | > loss_0: 2.5705578327178955 (2.56428509288364) | > grad_norm_0: tensor(254.2753, device='cuda:0') (tensor(265.3540, device='cuda:0')) | > loss_gen: 1.973116159439087 (2.2567732334136963) | > loss_kl: 2.037076950073242 (1.9234544303682115) | > loss_feat: 3.62454891204834 (3.9588683711157904) | > loss_mel: 17.1031551361084 (17.27364985148112) | > loss_duration: 1.5212053060531616 (1.506109184688992) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.259103775024414 (26.918854819403755) | > grad_norm_1: tensor(10026.5732, device='cuda:0') (tensor(5810.8267, device='cuda:0')) | > current_lr_0: 0.00017010778210255553 | > current_lr_1: 0.00017010778210255553 | > step_time: 0.4179 (0.4242173300849067) | > loader_time: 0.0055 (0.00650485356648763)  --> TIME: 2025-02-25 21:33:32 -- STEP: 59/72 -- GLOBAL_STEP: 1093300 | > loss_disc: 2.7244980335235596 (2.5802563004574535) | > loss_disc_real_0: 0.15620167553424835 (0.12490264288449696) | > loss_disc_real_1: 0.18800045549869537 (0.2229499092041436) | > loss_disc_real_2: 0.23587392270565033 (0.23908069148912267) | > loss_disc_real_3: 0.2502775490283966 (0.23874747551093667) | > loss_disc_real_4: 0.23524920642375946 (0.24527807331691354) | > loss_disc_real_5: 0.26611292362213135 (0.2435774616265701) | > loss_0: 2.7244980335235596 (2.5802563004574535) | > grad_norm_0: tensor(75.6499, device='cuda:0') (tensor(225.5041, device='cuda:0')) | > loss_gen: 1.915856957435608 (2.1838204618227683) | > loss_kl: 2.006204128265381 (1.9418581804986728) | > loss_feat: 3.105883836746216 (3.9160131074614446) | > loss_mel: 15.568802833557129 (16.958746118060624) | > loss_duration: 1.5711101293563843 (1.531059620744091) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.167858123779297 (26.531497405747235) | > grad_norm_1: tensor(11790.1504, device='cuda:0') (tensor(6054.3892, device='cuda:0')) | > current_lr_0: 0.00017010778210255553 | > current_lr_1: 0.00017010778210255553 | > step_time: 0.4723 (0.4444671808663061) | > loader_time: 0.01 (0.008014274855791512) --> EVAL PERFORMANCE | > avg_loader_time: 0.008582639022612236 (-6.266379020583579e-05) | > avg_step_time: 0.4551984860863484 (+0.001673251810208165) | > avg_loss_disc: 2.584655271449559 (-0.06409224657945245) | > avg_loss_disc_real_0: 0.12791892508385888 (-0.04345846165653683) | > avg_loss_disc_real_1: 0.22018756413123977 (-0.04589910952138235) | > avg_loss_disc_real_2: 0.23895106643018588 (-0.0005119329606983192) | > avg_loss_disc_real_3: 0.23921095023692493 (-0.018240465244776766) | > avg_loss_disc_real_4: 0.24365296791976607 (-0.0024418595811011023) | > avg_loss_disc_real_5: 0.24235679482070493 (-0.002526711620075589) | > avg_loss_0: 2.584655271449559 (-0.06409224657945245) | > avg_grad_norm_0: tensor(226.1380, device='cuda:0') (tensor(-34.8180, device='cuda:0')) | > avg_loss_gen: 2.1770511781665642 (-0.0995924153798069) | > avg_loss_kl: 1.9724868166614586 (-0.1877114168355163) | > avg_loss_feat: 3.8807592492708016 (+0.025597407784259563) | > avg_loss_mel: 16.99245380347884 (-0.826685005510349) | > avg_loss_duration: 1.542370705537393 (-0.021809467127625437) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.565121690991898 (-1.1102008819580043) | > avg_grad_norm_1: tensor(6084.5859, device='cuda:0') (+tensor(651.3345, device='cuda:0'))  > EPOCH: 1296/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:33:40)   --> TIME: 2025-02-25 21:33:59 -- STEP: 37/72 -- GLOBAL_STEP: 1093350 | > loss_disc: 2.5469565391540527 (2.640953953201706) | > loss_disc_real_0: 0.16751769185066223 (0.13115453387837153) | > loss_disc_real_1: 0.2141162008047104 (0.20898114023981867) | > loss_disc_real_2: 0.21622468531131744 (0.21993015545445518) | > loss_disc_real_3: 0.2255549132823944 (0.23118510560409442) | > loss_disc_real_4: 0.2252139151096344 (0.23907771102479985) | > loss_disc_real_5: 0.22968728840351105 (0.24781721105446686) | > loss_0: 2.5469565391540527 (2.640953953201706) | > grad_norm_0: tensor(31.2587, device='cuda:0') (tensor(222.5334, device='cuda:0')) | > loss_gen: 2.2375571727752686 (2.0497869059846217) | > loss_kl: 1.9881410598754883 (1.9315500903773952) | > loss_feat: 3.713982343673706 (3.7933204238479203) | > loss_mel: 15.93484115600586 (16.87465190887451) | > loss_duration: 1.5444135665893555 (1.508963265934506) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.418933868408203 (26.15827261435019) | > grad_norm_1: tensor(2730.8574, device='cuda:0') (tensor(5837.2197, device='cuda:0')) | > current_lr_0: 0.00017008651862979271 | > current_lr_1: 0.00017008651862979271 | > step_time: 0.4562 (0.4506764540801177) | > loader_time: 0.0082 (0.007443679345620645) --> EVAL PERFORMANCE | > avg_loader_time: 0.008850520765277703 (+0.00026788174266546665) | > avg_step_time: 0.4682051698926469 (+0.013006683806298491) | > avg_loss_disc: 2.6164950619281173 (+0.03183979047855834) | > avg_loss_disc_real_0: 0.13168173566670488 (+0.003762810582845999) | > avg_loss_disc_real_1: 0.22362278800615124 (+0.003435223874911464) | > avg_loss_disc_real_2: 0.22858506000377762 (-0.010366006426408253) | > avg_loss_disc_real_3: 0.23464776869391052 (-0.004563181543014411) | > avg_loss_disc_real_4: 0.2472388981513574 (+0.0035859302315913444) | > avg_loss_disc_real_5: 0.24531666892515103 (+0.002959874104446092) | > avg_loss_0: 2.6164950619281173 (+0.03183979047855834) | > avg_grad_norm_0: tensor(219.3037, device='cuda:0') (tensor(-6.8343, device='cuda:0')) | > avg_loss_gen: 2.146538539671563 (-0.030512638495001188) | > avg_loss_kl: 1.971834548762147 (-0.0006522678993117204) | > avg_loss_feat: 3.8415754680902188 (-0.039183781180582855) | > avg_loss_mel: 16.751175396878953 (-0.24127840659988564) | > avg_loss_duration: 1.5392713832183622 (-0.0030993223190307617) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.25039530472017 (-0.31472638627172955) | > avg_grad_norm_1: tensor(5406.4883, device='cuda:0') (tensor(-678.0977, device='cuda:0'))  > EPOCH: 1297/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:34:18)   --> TIME: 2025-02-25 21:34:26 -- STEP: 15/72 -- GLOBAL_STEP: 1093400 | > loss_disc: 2.559812545776367 (2.5850610891977945) | > loss_disc_real_0: 0.08509931713342667 (0.1417168967425823) | > loss_disc_real_1: 0.19739244878292084 (0.20997152129809063) | > loss_disc_real_2: 0.24582946300506592 (0.2278346965710322) | > loss_disc_real_3: 0.2420026659965515 (0.23014765779177349) | > loss_disc_real_4: 0.26852741837501526 (0.23662071923414865) | > loss_disc_real_5: 0.24808314442634583 (0.24001094500223796) | > loss_0: 2.559812545776367 (2.5850610891977945) | > grad_norm_0: tensor(182.4625, device='cuda:0') (tensor(299.3764, device='cuda:0')) | > loss_gen: 2.1776108741760254 (2.2214967807133994) | > loss_kl: 2.043671131134033 (1.9508585691452027) | > loss_feat: 4.04158353805542 (4.101710923512777) | > loss_mel: 18.235265731811523 (17.36496353149414) | > loss_duration: 1.4981597661972046 (1.4910449504852294) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.99629020690918 (27.130074564615885) | > grad_norm_1: tensor(9451.0381, device='cuda:0') (tensor(6716.8350, device='cuda:0')) | > current_lr_0: 0.000170065257814964 | > current_lr_1: 0.000170065257814964 | > step_time: 0.437 (0.4407146135965983) | > loader_time: 0.0088 (0.0069618701934814455)  --> TIME: 2025-02-25 21:34:50 -- STEP: 65/72 -- GLOBAL_STEP: 1093450 | > loss_disc: 2.5742907524108887 (2.6832774492410514) | > loss_disc_real_0: 0.10119082033634186 (0.16937791590507215) | > loss_disc_real_1: 0.22212664783000946 (0.21356868136387605) | > loss_disc_real_2: 0.2814286947250366 (0.23352319116775805) | > loss_disc_real_3: 0.2603755593299866 (0.23107329102662894) | > loss_disc_real_4: 0.25712940096855164 (0.2431111496228438) | > loss_disc_real_5: 0.2805470824241638 (0.24330362104452574) | > loss_0: 2.5742907524108887 (2.6832774492410514) | > grad_norm_0: tensor(115.3194, device='cuda:0') (tensor(273.1510, device='cuda:0')) | > loss_gen: 2.2198307514190674 (2.0610042168543896) | > loss_kl: 2.6330437660217285 (2.1276614280847395) | > loss_feat: 3.541247844696045 (3.8202865783984845) | > loss_mel: 17.11604881286621 (17.467098705585187) | > loss_duration: 1.6408735513687134 (1.5449951318594126) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.151044845581055 (27.02104606628418) | > grad_norm_1: tensor(7214.7310, device='cuda:0') (tensor(5819.2974, device='cuda:0')) | > current_lr_0: 0.000170065257814964 | > current_lr_1: 0.000170065257814964 | > step_time: 0.5315 (0.4561415452223558) | > loader_time: 0.0118 (0.008477823550884542) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087443405473736 (-0.0001061802179041036) | > avg_step_time: 0.4626110338828933 (-0.005594136009753559) | > avg_loss_disc: 2.6809996114650243 (+0.06450454953690699) | > avg_loss_disc_real_0: 0.17083817363624842 (+0.039156437969543545) | > avg_loss_disc_real_1: 0.21285170851878718 (-0.010771079487364055) | > avg_loss_disc_real_2: 0.23327363922562397 (+0.004688579221846345) | > avg_loss_disc_real_3: 0.2300211722162408 (-0.00462659647766972) | > avg_loss_disc_real_4: 0.24339124497393488 (-0.0038476531774225275) | > avg_loss_disc_real_5: 0.24392924237419183 (-0.001387426550959192) | > avg_loss_0: 2.6809996114650243 (+0.06450454953690699) | > avg_grad_norm_0: tensor(263.2216, device='cuda:0') (+tensor(43.9179, device='cuda:0')) | > avg_loss_gen: 2.0679389943539266 (-0.07859954531763647) | > avg_loss_kl: 2.1500354803783783 (+0.1782009316162314) | > avg_loss_feat: 3.801077748688174 (-0.04049771940204483) | > avg_loss_mel: 17.531404360918927 (+0.7802289640399742) | > avg_loss_duration: 1.5528775957268728 (+0.013606212508510573) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.10333421196736 (+0.8529389072471929) | > avg_grad_norm_1: tensor(5631.4854, device='cuda:0') (+tensor(224.9971, device='cuda:0'))  > EPOCH: 1298/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:34:55)   --> TIME: 2025-02-25 21:35:17 -- STEP: 43/72 -- GLOBAL_STEP: 1093500 | > loss_disc: 2.6194422245025635 (2.655571787856346) | > loss_disc_real_0: 0.18536430597305298 (0.15544205765391503) | > loss_disc_real_1: 0.20404788851737976 (0.24000070729227954) | > loss_disc_real_2: 0.2617030143737793 (0.23313772470452065) | > loss_disc_real_3: 0.22839152812957764 (0.2300170777842056) | > loss_disc_real_4: 0.23572875559329987 (0.24080367628918137) | > loss_disc_real_5: 0.2410620003938675 (0.2428797750278961) | > loss_0: 2.6194422245025635 (2.655571787856346) | > grad_norm_0: tensor(404.0597, device='cuda:0') (tensor(148.3384, device='cuda:0')) | > loss_gen: 2.1798062324523926 (2.109918949215911) | > loss_kl: 1.8396193981170654 (2.0446176085361216) | > loss_feat: 3.4782841205596924 (3.682202699572541) | > loss_mel: 15.687305450439453 (17.376740366913552) | > loss_duration: 1.5397635698318481 (1.5577654090038566) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.72477912902832 (26.77124489185422) | > grad_norm_1: tensor(3241.7925, device='cuda:0') (tensor(4517.2847, device='cuda:0')) | > current_lr_0: 0.0001700439996577371 | > current_lr_1: 0.0001700439996577371 | > step_time: 0.4725 (0.4468425650929296) | > loader_time: 0.0095 (0.007625557655511901) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874506923514353 (+7.286877699314714e-07) | > avg_step_time: 0.464427864047843 (+0.0018168301649496565) | > avg_loss_disc: 2.6337640486972433 (-0.04723556276778096) | > avg_loss_disc_real_0: 0.14936779581115278 (-0.021470377825095643) | > avg_loss_disc_real_1: 0.22952721016088 (+0.016675501642092833) | > avg_loss_disc_real_2: 0.23837004119241742 (+0.005096401966793451) | > avg_loss_disc_real_3: 0.23056187952907992 (+0.0005407073128391171) | > avg_loss_disc_real_4: 0.2433818750818011 (-9.369892133792579e-06) | > avg_loss_disc_real_5: 0.24476185084228785 (+0.0008326084680960188) | > avg_loss_0: 2.6337640486972433 (-0.04723556276778096) | > avg_grad_norm_0: tensor(173.7594, device='cuda:0') (tensor(-89.4622, device='cuda:0')) | > avg_loss_gen: 2.1060837994159107 (+0.038144805061984144) | > avg_loss_kl: 2.05231774692804 (-0.09771773345033852) | > avg_loss_feat: 3.6351845700975876 (-0.1658931785905864) | > avg_loss_mel: 17.12253805617211 (-0.40886630474681596) | > avg_loss_duration: 1.5681376222153784 (+0.015260026488505662) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.484261767964966 (-0.6190724440023949) | > avg_grad_norm_1: tensor(4992.4028, device='cuda:0') (tensor(-639.0825, device='cuda:0'))  > EPOCH: 1299/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:35:33)   --> TIME: 2025-02-25 21:35:45 -- STEP: 21/72 -- GLOBAL_STEP: 1093550 | > loss_disc: 2.758591890335083 (2.65580347606114) | > loss_disc_real_0: 0.07784093171358109 (0.11893874797083084) | > loss_disc_real_1: 0.19938120245933533 (0.22309653460979462) | > loss_disc_real_2: 0.2169000804424286 (0.2259944783789771) | > loss_disc_real_3: 0.2338310331106186 (0.2215331963130406) | > loss_disc_real_4: 0.2205602526664734 (0.23557560642560324) | > loss_disc_real_5: 0.2535766661167145 (0.22856598028114863) | > loss_0: 2.758591890335083 (2.65580347606114) | > grad_norm_0: tensor(312.7108, device='cuda:0') (tensor(237.5663, device='cuda:0')) | > loss_gen: 1.7535886764526367 (1.9198192187717982) | > loss_kl: 2.059556007385254 (2.256840257417588) | > loss_feat: 3.3058009147644043 (3.6742276464189803) | > loss_mel: 16.94745445251465 (18.306449345179967) | > loss_duration: 1.5422139167785645 (1.6061169703801472) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.608612060546875 (27.76345361982073) | > grad_norm_1: tensor(10238.6533, device='cuda:0') (tensor(9125.8008, device='cuda:0')) | > current_lr_0: 0.0001700227441577799 | > current_lr_1: 0.0001700227441577799 | > step_time: 0.453 (0.4450920990535191) | > loader_time: 0.0064 (0.006827967507498605)  --> TIME: 2025-02-25 21:36:10 -- STEP: 71/72 -- GLOBAL_STEP: 1093600 | > loss_disc: 2.7096927165985107 (2.604892327751912) | > loss_disc_real_0: 0.10917995870113373 (0.12353171240276015) | > loss_disc_real_1: 0.25234508514404297 (0.22606965983417673) | > loss_disc_real_2: 0.28157955408096313 (0.2330305748422381) | > loss_disc_real_3: 0.2713279724121094 (0.22983296152571558) | > loss_disc_real_4: 0.26542094349861145 (0.2415492446489737) | > loss_disc_real_5: 0.2953900098800659 (0.2434086946534439) | > loss_0: 2.7096927165985107 (2.604892327751912) | > grad_norm_0: tensor(788.8146, device='cuda:0') (tensor(269.1548, device='cuda:0')) | > loss_gen: 2.1325931549072266 (2.03045614336578) | > loss_kl: 1.8504376411437988 (2.1508461438434225) | > loss_feat: 3.283301830291748 (3.630051891568681) | > loss_mel: 16.53308868408203 (17.235857923265915) | > loss_duration: 1.5585784912109375 (1.5794138303944762) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.357999801635742 (26.626626135597768) | > grad_norm_1: tensor(17108.4082, device='cuda:0') (tensor(10861.7275, device='cuda:0')) | > current_lr_0: 0.0001700227441577799 | > current_lr_1: 0.0001700227441577799 | > step_time: 0.5035 (0.4662322830146467) | > loader_time: 0.0084 (0.008717201125453895) --> EVAL PERFORMANCE | > avg_loader_time: 0.008717201125453895 (-2.7868109689636184e-05) | > avg_step_time: 0.4662322830146467 (+0.0018044189668037225) | > avg_loss_disc: 2.604892327751912 (-0.028871720945331525) | > avg_loss_disc_real_0: 0.12353171240276015 (-0.025836083408392635) | > avg_loss_disc_real_1: 0.22606965983417673 (-0.0034575503267032792) | > avg_loss_disc_real_2: 0.2330305748422381 (-0.005339466350179317) | > avg_loss_disc_real_3: 0.22983296152571558 (-0.0007289180033643317) | > avg_loss_disc_real_4: 0.2415492446489737 (-0.0018326304328273868) | > avg_loss_disc_real_5: 0.2434086946534439 (-0.0013531561888439625) | > avg_loss_0: 2.604892327751912 (-0.028871720945331525) | > avg_grad_norm_0: tensor(269.1548, device='cuda:0') (+tensor(95.3954, device='cuda:0')) | > avg_loss_gen: 2.03045614336578 (-0.07562765605013055) | > avg_loss_kl: 2.1508461438434225 (+0.09852839691538273) | > avg_loss_feat: 3.630051891568681 (-0.005132678528906531) | > avg_loss_mel: 17.235857923265915 (+0.1133198670938036) | > avg_loss_duration: 1.5794138303944762 (+0.011276208179097802) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.626626135597768 (+0.14236436763280125) | > avg_grad_norm_1: tensor(10861.7275, device='cuda:0') (+tensor(5869.3247, device='cuda:0'))  > EPOCH: 1300/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:36:11)   --> TIME: 2025-02-25 21:36:36 -- STEP: 49/72 -- GLOBAL_STEP: 1093650 | > loss_disc: 2.5018961429595947 (2.5729376588548933) | > loss_disc_real_0: 0.08640633523464203 (0.12335888456021038) | > loss_disc_real_1: 0.289174348115921 (0.22061615543706076) | > loss_disc_real_2: 0.24154286086559296 (0.23447667305566827) | > loss_disc_real_3: 0.30832722783088684 (0.23384127203299074) | > loss_disc_real_4: 0.26108047366142273 (0.2438341595080434) | > loss_disc_real_5: 0.25607091188430786 (0.24305749486903755) | > loss_0: 2.5018961429595947 (2.5729376588548933) | > grad_norm_0: tensor(34.8776, device='cuda:0') (tensor(344.6641, device='cuda:0')) | > loss_gen: 2.4204299449920654 (2.112075623200864) | > loss_kl: 2.0193371772766113 (1.9105866685205577) | > loss_feat: 4.061337471008301 (3.876139241821912) | > loss_mel: 15.880254745483398 (16.672178112730684) | > loss_duration: 1.589869737625122 (1.5207970580276178) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.971229553222656 (26.091776633749202) | > grad_norm_1: tensor(2866.9617, device='cuda:0') (tensor(9632.9492, device='cuda:0')) | > current_lr_0: 0.00017000149131476017 | > current_lr_1: 0.00017000149131476017 | > step_time: 0.4538 (0.4515123172682159) | > loader_time: 0.0096 (0.0077337878090994695) --> EVAL PERFORMANCE | > avg_loader_time: 0.008739857606484857 (+2.2656481030962697e-05) | > avg_step_time: 0.46517734124626914 (-0.0010549417683775597) | > avg_loss_disc: 2.5643827411490427 (-0.040509586602869074) | > avg_loss_disc_real_0: 0.1058090731606517 (-0.017722639242108443) | > avg_loss_disc_real_1: 0.21943422459380726 (-0.006635435240369469) | > avg_loss_disc_real_2: 0.23553364923302555 (+0.002503074390787452) | > avg_loss_disc_real_3: 0.22901511297259533 (-0.000817848553120254) | > avg_loss_disc_real_4: 0.23992358840687175 (-0.0016256562421019527) | > avg_loss_disc_real_5: 0.24261441365094252 (-0.000794281002501368) | > avg_loss_0: 2.5643827411490427 (-0.040509586602869074) | > avg_grad_norm_0: tensor(304.1176, device='cuda:0') (+tensor(34.9627, device='cuda:0')) | > avg_loss_gen: 2.130219148917936 (+0.09976300555215589) | > avg_loss_kl: 1.9552329140649716 (-0.19561322977845097) | > avg_loss_feat: 3.9685730766242657 (+0.3385211850555847) | > avg_loss_mel: 16.69335836759754 (-0.5424995556683747) | > avg_loss_duration: 1.5392318057342314 (-0.0401820246602449) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.286615237383774 (-0.34001089821399333) | > avg_grad_norm_1: tensor(8521.8604, device='cuda:0') (tensor(-2339.8672, device='cuda:0'))  > EPOCH: 1301/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:36:49)   --> TIME: 2025-02-25 21:37:04 -- STEP: 27/72 -- GLOBAL_STEP: 1093700 | > loss_disc: 2.6822474002838135 (2.539444022708469) | > loss_disc_real_0: 0.2250741571187973 (0.08543540420079673) | > loss_disc_real_1: 0.23656336963176727 (0.22069218754768372) | > loss_disc_real_2: 0.22184282541275024 (0.23796484812542243) | > loss_disc_real_3: 0.25927847623825073 (0.2345024264521069) | > loss_disc_real_4: 0.2570424973964691 (0.24607309809437505) | > loss_disc_real_5: 0.25754114985466003 (0.24524152610037062) | > loss_0: 2.6822474002838135 (2.539444022708469) | > grad_norm_0: tensor(667.9753, device='cuda:0') (tensor(250.3082, device='cuda:0')) | > loss_gen: 2.146292209625244 (2.19840700979586) | > loss_kl: 1.9930955171585083 (1.9474516665494) | > loss_feat: 3.438310146331787 (4.0863235173402) | > loss_mel: 16.169496536254883 (16.611749719690394) | > loss_duration: 1.4911699295043945 (1.491837324919524) | > amp_scaler: 8.0 (8.296296296296298) | > loss_1: 25.238365173339844 (26.33576915882252) | > grad_norm_1: tensor(4859.5996, device='cuda:0') (tensor(6163.8853, device='cuda:0')) | > current_lr_0: 0.00016998024112834583 | > current_lr_1: 0.00016998024112834583 | > step_time: 0.4553 (0.4511155817243788) | > loader_time: 0.0087 (0.0071916138684308085) --> EVAL PERFORMANCE | > avg_loader_time: 0.008889124427043217 (+0.00014926682055835944) | > avg_step_time: 0.46943373411474093 (+0.004256392868471792) | > avg_loss_disc: 2.528954304439921 (-0.03542843670912177) | > avg_loss_disc_real_0: 0.08626969822380744 (-0.019539374936844267) | > avg_loss_disc_real_1: 0.2202994334445873 (+0.000865208850780036) | > avg_loss_disc_real_2: 0.22221119429024172 (-0.013322454942783835) | > avg_loss_disc_real_3: 0.23103552909804062 (+0.00202041612544529) | > avg_loss_disc_real_4: 0.24403753834710995 (+0.0041139499402381985) | > avg_loss_disc_real_5: 0.24509542298988557 (+0.0024810093389430465) | > avg_loss_0: 2.528954304439921 (-0.03542843670912177) | > avg_grad_norm_0: tensor(235.8562, device='cuda:0') (tensor(-68.2613, device='cuda:0')) | > avg_loss_gen: 2.208488288060041 (+0.07826913914210509) | > avg_loss_kl: 1.9993689110581303 (+0.04413599699315873) | > avg_loss_feat: 4.117542538844363 (+0.14896946222009744) | > avg_loss_mel: 16.569080124438646 (-0.12427824315889424) | > avg_loss_duration: 1.5325431269659122 (-0.006688678768319134) | > avg_amp_scaler: 8.112676056338028 (+0.11267605633802802) | > avg_loss_1: 26.427023041416223 (+0.14040780403244923) | > avg_grad_norm_1: tensor(6862.0386, device='cuda:0') (tensor(-1659.8218, device='cuda:0'))  > EPOCH: 1302/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:37:27)   --> TIME: 2025-02-25 21:37:31 -- STEP: 5/72 -- GLOBAL_STEP: 1093750 | > loss_disc: 2.7826592922210693 (2.547704887390137) | > loss_disc_real_0: 0.03074529580771923 (0.037248629704117776) | > loss_disc_real_1: 0.2482520341873169 (0.21393166184425355) | > loss_disc_real_2: 0.2273881882429123 (0.20304299294948577) | > loss_disc_real_3: 0.2340983897447586 (0.22645331919193268) | > loss_disc_real_4: 0.217873677611351 (0.2222195327281952) | > loss_disc_real_5: 0.23519764840602875 (0.2285801500082016) | > loss_0: 2.7826592922210693 (2.547704887390137) | > grad_norm_0: tensor(56.9403, device='cuda:0') (tensor(190.2797, device='cuda:0')) | > loss_gen: 1.9390828609466553 (2.103771424293518) | > loss_kl: 1.709370493888855 (1.7930526733398438) | > loss_feat: 3.762969970703125 (4.374978828430176) | > loss_mel: 17.0497989654541 (16.915685653686523) | > loss_duration: 1.4761106967926025 (1.4943904876708984) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.937334060668945 (26.681879425048827) | > grad_norm_1: tensor(12275.8545, device='cuda:0') (tensor(9217.2422, device='cuda:0')) | > current_lr_0: 0.00016995899359820477 | > current_lr_1: 0.00016995899359820477 | > step_time: 0.4506 (0.4361257553100586) | > loader_time: 0.006 (0.006926584243774414)  --> TIME: 2025-02-25 21:37:55 -- STEP: 55/72 -- GLOBAL_STEP: 1093800 | > loss_disc: 2.5302538871765137 (2.503430999409069) | > loss_disc_real_0: 0.25031429529190063 (0.09658530235967853) | > loss_disc_real_1: 0.2421630471944809 (0.21601529771631414) | > loss_disc_real_2: 0.18917302787303925 (0.22822982397946445) | > loss_disc_real_3: 0.21442458033561707 (0.2287399327213114) | > loss_disc_real_4: 0.2303411066532135 (0.23916764015501196) | > loss_disc_real_5: 0.22582803666591644 (0.2435782933777029) | > loss_0: 2.5302538871765137 (2.503430999409069) | > grad_norm_0: tensor(497.2343, device='cuda:0') (tensor(234.6674, device='cuda:0')) | > loss_gen: 2.2814812660217285 (2.269463272528216) | > loss_kl: 2.0323359966278076 (1.9269671548496594) | > loss_feat: 3.946911096572876 (4.152098230882126) | > loss_mel: 16.287822723388672 (16.65339265303178) | > loss_duration: 1.5919692516326904 (1.5130721243945036) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.140520095825195 (26.514993286132817) | > grad_norm_1: tensor(3818.2314, device='cuda:0') (tensor(6567.6201, device='cuda:0')) | > current_lr_0: 0.00016995899359820477 | > current_lr_1: 0.00016995899359820477 | > step_time: 0.4699 (0.44991054968400435) | > loader_time: 0.0093 (0.008039049668745562) --> EVAL PERFORMANCE | > avg_loader_time: 0.008759659780582912 (-0.0001294646464603047) | > avg_step_time: 0.46307078213758873 (-0.006362951977152198) | > avg_loss_disc: 2.5111035393996985 (-0.017850765040222427) | > avg_loss_disc_real_0: 0.09803589735127671 (+0.011766199127469276) | > avg_loss_disc_real_1: 0.2172679271496518 (-0.00303150629493551) | > avg_loss_disc_real_2: 0.2304637182346532 (+0.008252523944411477) | > avg_loss_disc_real_3: 0.22866090817350737 (-0.0023746209245332495) | > avg_loss_disc_real_4: 0.2387286361674188 (-0.005308902179691155) | > avg_loss_disc_real_5: 0.24233277456861146 (-0.0027626484212741054) | > avg_loss_0: 2.5111035393996985 (-0.017850765040222427) | > avg_grad_norm_0: tensor(245.8292, device='cuda:0') (+tensor(9.9730, device='cuda:0')) | > avg_loss_gen: 2.2640222294229866 (+0.05553394136294543) | > avg_loss_kl: 1.9667786376577028 (-0.03259027340042753) | > avg_loss_feat: 4.143338552663023 (+0.025796013818659702) | > avg_loss_mel: 16.70561229007345 (+0.13653216563480441) | > avg_loss_duration: 1.5281388474182345 (-0.004404279547677747) | > avg_amp_scaler: 8.0 (-0.11267605633802802) | > avg_loss_1: 26.607890330569845 (+0.18086728915362116) | > avg_grad_norm_1: tensor(6614.9868, device='cuda:0') (tensor(-247.0518, device='cuda:0'))  > EPOCH: 1303/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:38:05)   --> TIME: 2025-02-25 21:38:22 -- STEP: 33/72 -- GLOBAL_STEP: 1093850 | > loss_disc: 2.4437131881713867 (2.4807281638636733) | > loss_disc_real_0: 0.058009929955005646 (0.08874037252231078) | > loss_disc_real_1: 0.25651997327804565 (0.22775609836433874) | > loss_disc_real_2: 0.21369895339012146 (0.2227715700864792) | > loss_disc_real_3: 0.2632234990596771 (0.23008497510895584) | > loss_disc_real_4: 0.24883964657783508 (0.2414913312955336) | > loss_disc_real_5: 0.2577228248119354 (0.2427535237688007) | > loss_0: 2.4437131881713867 (2.4807281638636733) | > grad_norm_0: tensor(98.0845, device='cuda:0') (tensor(215.4557, device='cuda:0')) | > loss_gen: 2.3744149208068848 (2.3340312350880015) | > loss_kl: 1.9973227977752686 (1.9306714065147168) | > loss_feat: 4.252325057983398 (4.32017816919269) | > loss_mel: 16.57371711730957 (16.74956365065141) | > loss_duration: 1.5015554428100586 (1.4949275616443518) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.69933319091797 (26.829371654626094) | > grad_norm_1: tensor(8956.7090, device='cuda:0') (tensor(6050.2695, device='cuda:0')) | > current_lr_0: 0.000169937748724005 | > current_lr_1: 0.000169937748724005 | > step_time: 0.4524 (0.44525501222321484) | > loader_time: 0.0081 (0.007441730210275361) --> EVAL PERFORMANCE | > avg_loader_time: 0.008849268228235377 (+8.960844765246515e-05) | > avg_step_time: 0.46409041780821036 (+0.0010196356706216325) | > avg_loss_disc: 2.5000230728740425 (-0.011080466525656085) | > avg_loss_disc_real_0: 0.09866883474546419 (+0.0006329373941874744) | > avg_loss_disc_real_1: 0.22553708834547392 (+0.008269161195822133) | > avg_loss_disc_real_2: 0.227948301065136 (-0.002515417169517198) | > avg_loss_disc_real_3: 0.23030589353030836 (+0.0016449853568009931) | > avg_loss_disc_real_4: 0.24079536258334844 (+0.002066726415929643) | > avg_loss_disc_real_5: 0.24504999354691573 (+0.0027172189783042644) | > avg_loss_0: 2.5000230728740425 (-0.011080466525656085) | > avg_grad_norm_0: tensor(248.2482, device='cuda:0') (+tensor(2.4190, device='cuda:0')) | > avg_loss_gen: 2.3066654809763736 (+0.042643251553386996) | > avg_loss_kl: 1.9802166374636367 (+0.013437999805933964) | > avg_loss_feat: 4.175944737985101 (+0.03260618532207804) | > avg_loss_mel: 16.628797557992 (-0.07681473208144851) | > avg_loss_duration: 1.5323906330995156 (+0.0042517856812811505) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.624014760406926 (+0.016124429837081067) | > avg_grad_norm_1: tensor(5546.9453, device='cuda:0') (tensor(-1068.0415, device='cuda:0'))  > EPOCH: 1304/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:38:43)   --> TIME: 2025-02-25 21:38:50 -- STEP: 11/72 -- GLOBAL_STEP: 1093900 | > loss_disc: 2.4390673637390137 (2.6988148906014184) | > loss_disc_real_0: 0.02715158648788929 (0.10776008360765198) | > loss_disc_real_1: 0.16457165777683258 (0.14206639474088495) | > loss_disc_real_2: 0.1814592033624649 (0.2052328803322532) | > loss_disc_real_3: 0.17760878801345825 (0.18301740966059946) | > loss_disc_real_4: 0.24382179975509644 (0.2413022287867286) | > loss_disc_real_5: 0.20014166831970215 (0.22552229193123904) | > loss_0: 2.4390673637390137 (2.6988148906014184) | > grad_norm_0: tensor(125.8769, device='cuda:0') (tensor(309.1638, device='cuda:0')) | > loss_gen: 2.1857199668884277 (1.9315246018496426) | > loss_kl: 1.8541984558105469 (1.9001538861881604) | > loss_feat: 4.560638427734375 (4.370902690020475) | > loss_mel: 16.715925216674805 (17.033574537797406) | > loss_duration: 1.4700860977172852 (1.479109991680492) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.78656768798828 (26.715265447443183) | > grad_norm_1: tensor(4933.1304, device='cuda:0') (tensor(5917.4595, device='cuda:0')) | > current_lr_0: 0.00016991650650541448 | > current_lr_1: 0.00016991650650541448 | > step_time: 0.4309 (0.43420360305092554) | > loader_time: 0.0059 (0.006275740536776456)  --> TIME: 2025-02-25 21:39:13 -- STEP: 61/72 -- GLOBAL_STEP: 1093950 | > loss_disc: 2.8144145011901855 (2.63737413140594) | > loss_disc_real_0: 0.06432192772626877 (0.13014470178206436) | > loss_disc_real_1: 0.25177428126335144 (0.21179798514139456) | > loss_disc_real_2: 0.3225056529045105 (0.2322003644753675) | > loss_disc_real_3: 0.2230331152677536 (0.21830505645665965) | > loss_disc_real_4: 0.29688724875450134 (0.2281652234616827) | > loss_disc_real_5: 0.2641139626502991 (0.24148826740804266) | > loss_0: 2.8144145011901855 (2.63737413140594) | > grad_norm_0: tensor(78.7304, device='cuda:0') (tensor(280.6017, device='cuda:0')) | > loss_gen: 1.9144002199172974 (2.1586797081056197) | > loss_kl: 2.0134286880493164 (2.0014592252793864) | > loss_feat: 3.2904105186462402 (4.099425382301455) | > loss_mel: 16.663471221923828 (16.588396557041868) | > loss_duration: 1.5823575258255005 (1.5228273243200583) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.464067459106445 (26.370788230270637) | > grad_norm_1: tensor(5782.5190, device='cuda:0') (tensor(5701.4189, device='cuda:0')) | > current_lr_0: 0.00016991650650541448 | > current_lr_1: 0.00016991650650541448 | > step_time: 0.4878 (0.45154462095166814) | > loader_time: 0.0105 (0.008242669652719966) --> EVAL PERFORMANCE | > avg_loader_time: 0.008699538002551441 (-0.00014973022568393578) | > avg_step_time: 0.45816546090891663 (-0.005924956899293732) | > avg_loss_disc: 2.611155217802021 (+0.11113214492797852) | > avg_loss_disc_real_0: 0.12379807724394429 (+0.0251292424984801) | > avg_loss_disc_real_1: 0.21166279765082077 (-0.013874290694653152) | > avg_loss_disc_real_2: 0.2258463861866736 (-0.0021019148784623964) | > avg_loss_disc_real_3: 0.21943279597121226 (-0.010873097559096101) | > avg_loss_disc_real_4: 0.22564986683952976 (-0.015145495743818682) | > avg_loss_disc_real_5: 0.24108780794580217 (-0.003962185601113555) | > avg_loss_0: 2.611155217802021 (+0.11113214492797852) | > avg_grad_norm_0: tensor(270.4330, device='cuda:0') (+tensor(22.1847, device='cuda:0')) | > avg_loss_gen: 2.16916581274758 (-0.13749966822879367) | > avg_loss_kl: 2.0148770624483134 (+0.034660424984676697) | > avg_loss_feat: 4.119485133130784 (-0.056459604854317114) | > avg_loss_mel: 16.618655715190197 (-0.010141842801804302) | > avg_loss_duration: 1.5322008854906324 (-0.0001897476088832395) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.454384656019613 (-0.16963010438731274) | > avg_grad_norm_1: tensor(5825.4697, device='cuda:0') (+tensor(278.5244, device='cuda:0'))  > EPOCH: 1305/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:39:20)   --> TIME: 2025-02-25 21:39:40 -- STEP: 39/72 -- GLOBAL_STEP: 1094000 | > loss_disc: 2.3345577716827393 (2.4812693840418105) | > loss_disc_real_0: 0.12427903711795807 (0.08843038756495868) | > loss_disc_real_1: 0.22756390273571014 (0.21682817469804716) | > loss_disc_real_2: 0.2148817479610443 (0.227488860870019) | > loss_disc_real_3: 0.22680388391017914 (0.22941271731486687) | > loss_disc_real_4: 0.224091038107872 (0.24000410659190935) | > loss_disc_real_5: 0.2427556812763214 (0.24215575747000864) | > loss_0: 2.3345577716827393 (2.4812693840418105) | > grad_norm_0: tensor(273.0971, device='cuda:0') (tensor(218.9966, device='cuda:0')) | > loss_gen: 2.660229206085205 (2.325369782936879) | > loss_kl: 1.9848238229751587 (1.9842123282261384) | > loss_feat: 5.020623683929443 (4.373203986730331) | > loss_mel: 17.1999454498291 (16.81451391562437) | > loss_duration: 1.5412545204162598 (1.49647926672911) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.406875610351562 (26.99377901126177) | > grad_norm_1: tensor(5850.3374, device='cuda:0') (tensor(5936.5063, device='cuda:0')) | > current_lr_0: 0.0001698952669421013 | > current_lr_1: 0.0001698952669421013 | > step_time: 0.4528 (0.4424896668165158) | > loader_time: 0.0083 (0.007361479294605744) --> EVAL PERFORMANCE | > avg_loader_time: 0.008655924192616637 (-4.3613809934804626e-05) | > avg_step_time: 0.45886170360404 (+0.0006962426951233613) | > avg_loss_disc: 2.5417389332408638 (-0.06941628456115723) | > avg_loss_disc_real_0: 0.10201574851509551 (-0.021782328728848777) | > avg_loss_disc_real_1: 0.22057574630623134 (+0.008912948655410563) | > avg_loss_disc_real_2: 0.2303271942155462 (+0.0044808080288726015) | > avg_loss_disc_real_3: 0.2340264729630779 (+0.014593676991865645) | > avg_loss_disc_real_4: 0.23993038102774553 (+0.01428051418821577) | > avg_loss_disc_real_5: 0.24362804394372753 (+0.0025402359979253553) | > avg_loss_0: 2.5417389332408638 (-0.06941628456115723) | > avg_grad_norm_0: tensor(225.7940, device='cuda:0') (tensor(-44.6390, device='cuda:0')) | > avg_loss_gen: 2.241286603497788 (+0.07212079075020794) | > avg_loss_kl: 2.03067899086106 (+0.015801928412746413) | > avg_loss_feat: 4.125864374805506 (+0.006379241674721925) | > avg_loss_mel: 16.615236336076755 (-0.0034193791134420337) | > avg_loss_duration: 1.528717801604472 (-0.0034830838861603652) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.541783937266175 (+0.0873992812465616) | > avg_grad_norm_1: tensor(6918.0386, device='cuda:0') (+tensor(1092.5688, device='cuda:0'))  > EPOCH: 1306/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:39:58)   --> TIME: 2025-02-25 21:40:08 -- STEP: 17/72 -- GLOBAL_STEP: 1094050 | > loss_disc: 2.8415462970733643 (2.6889324608971092) | > loss_disc_real_0: 0.29259341955184937 (0.10804597288370132) | > loss_disc_real_1: 0.21427518129348755 (0.20503811976488898) | > loss_disc_real_2: 0.22099076211452484 (0.2355866090339773) | > loss_disc_real_3: 0.2445981800556183 (0.23199163114323335) | > loss_disc_real_4: 0.21531149744987488 (0.24911591848906348) | > loss_disc_real_5: 0.22764430940151215 (0.24495859005871942) | > loss_0: 2.8415462970733643 (2.6889324608971092) | > grad_norm_0: tensor(737.7345, device='cuda:0') (tensor(348.8195, device='cuda:0')) | > loss_gen: 1.9073843955993652 (1.956395892535939) | > loss_kl: 2.329801559448242 (1.9483602607951445) | > loss_feat: 3.2514209747314453 (3.832435593885534) | > loss_mel: 17.4193058013916 (16.695083954755) | > loss_duration: 1.5212419033050537 (1.4781264066696167) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.429155349731445 (25.910401961382696) | > grad_norm_1: tensor(4044.8259, device='cuda:0') (tensor(10373.9668, device='cuda:0')) | > current_lr_0: 0.00016987403003373354 | > current_lr_1: 0.00016987403003373354 | > step_time: 0.4579 (0.44249335457296934) | > loader_time: 0.0088 (0.007100399802712833)  --> TIME: 2025-02-25 21:40:32 -- STEP: 67/72 -- GLOBAL_STEP: 1094100 | > loss_disc: 2.632291793823242 (2.6866690080557296) | > loss_disc_real_0: 0.15146978199481964 (0.13324149725819706) | > loss_disc_real_1: 0.29086896777153015 (0.22321317444986372) | > loss_disc_real_2: 0.22479456663131714 (0.23417097285612307) | > loss_disc_real_3: 0.2564640939235687 (0.23206711326962087) | > loss_disc_real_4: 0.25732314586639404 (0.24771719147909932) | > loss_disc_real_5: 0.2494036704301834 (0.24432755873274448) | > loss_0: 2.632291793823242 (2.6866690080557296) | > grad_norm_0: tensor(323.7780, device='cuda:0') (tensor(363.0680, device='cuda:0')) | > loss_gen: 2.3184893131256104 (2.023533554219488) | > loss_kl: 2.420657157897949 (2.281667394424553) | > loss_feat: 3.731900453567505 (3.7561955096116706) | > loss_mel: 17.675615310668945 (17.59423161976374) | > loss_duration: 1.664736032485962 (1.5444251708130339) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.811397552490234 (27.20005308692135) | > grad_norm_1: tensor(10327.8857, device='cuda:0') (tensor(9773.2725, device='cuda:0')) | > current_lr_0: 0.00016987403003373354 | > current_lr_1: 0.00016987403003373354 | > step_time: 0.5113 (0.4653985607090281) | > loader_time: 0.0128 (0.008785461311909692) --> EVAL PERFORMANCE | > avg_loader_time: 0.008958259098966359 (+0.0003023349063497223) | > avg_step_time: 0.46855446990107147 (+0.009692766297031474) | > avg_loss_disc: 2.6858071139161015 (+0.14406818067523774) | > avg_loss_disc_real_0: 0.13570520832714905 (+0.03368945981205354) | > avg_loss_disc_real_1: 0.2264676121338992 (+0.005891865827667875) | > avg_loss_disc_real_2: 0.233596522413509 (+0.003269328197962801) | > avg_loss_disc_real_3: 0.23243232155349894 (-0.00159415140957897) | > avg_loss_disc_real_4: 0.24774102572823914 (+0.007810644700493613) | > avg_loss_disc_real_5: 0.24466315709369282 (+0.0010351131499652944) | > avg_loss_0: 2.6858071139161015 (+0.14406818067523774) | > avg_grad_norm_0: tensor(351.2179, device='cuda:0') (+tensor(125.4239, device='cuda:0')) | > avg_loss_gen: 2.0336119943941147 (-0.2076746091036732) | > avg_loss_kl: 2.2857507030728845 (+0.2550717122118247) | > avg_loss_feat: 3.753395590983646 (-0.3724687838218599) | > avg_loss_mel: 17.614461039153635 (+0.9992247030768802) | > avg_loss_duration: 1.5482950496001984 (+0.019577247995726355) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.235514170686965 (+0.6937302334207907) | > avg_grad_norm_1: tensor(9519.5293, device='cuda:0') (+tensor(2601.4907, device='cuda:0'))  > EPOCH: 1307/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:40:36)   --> TIME: 2025-02-25 21:40:59 -- STEP: 45/72 -- GLOBAL_STEP: 1094150 | > loss_disc: 2.6414568424224854 (2.652206966612074) | > loss_disc_real_0: 0.2467322200536728 (0.15838033598330284) | > loss_disc_real_1: 0.28040656447410583 (0.22363097618023556) | > loss_disc_real_2: 0.25642329454421997 (0.243810232480367) | > loss_disc_real_3: 0.23179318010807037 (0.23377981550163693) | > loss_disc_real_4: 0.23823988437652588 (0.23891226516829597) | > loss_disc_real_5: 0.2549763321876526 (0.24480724136034648) | > loss_0: 2.6414568424224854 (2.652206966612074) | > grad_norm_0: tensor(318.8828, device='cuda:0') (tensor(245.4915, device='cuda:0')) | > loss_gen: 2.3825807571411133 (2.181227829721239) | > loss_kl: 2.1145801544189453 (2.0606675624847424) | > loss_feat: 3.547360420227051 (3.8076111634572345) | > loss_mel: 17.584972381591797 (16.994613817003042) | > loss_duration: 1.5488405227661133 (1.517714007695516) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.178333282470703 (26.56183433532715) | > grad_norm_1: tensor(1819.9437, device='cuda:0') (tensor(7227.6313, device='cuda:0')) | > current_lr_0: 0.0001698527957799793 | > current_lr_1: 0.0001698527957799793 | > step_time: 0.462 (0.44768760999043783) | > loader_time: 0.0089 (0.007822969224717882) --> EVAL PERFORMANCE | > avg_loader_time: 0.00881812270258514 (-0.00014013639638121855) | > avg_step_time: 0.45801350432382504 (-0.01054096557724643) | > avg_loss_disc: 2.6380410026496564 (-0.047766111266445144) | > avg_loss_disc_real_0: 0.15091447128166616 (+0.015209262954517105) | > avg_loss_disc_real_1: 0.22095741237133318 (-0.0055101997625660315) | > avg_loss_disc_real_2: 0.23689992075235072 (+0.0033033983388417176) | > avg_loss_disc_real_3: 0.237708020168291 (+0.005275698614792074) | > avg_loss_disc_real_4: 0.23948093766057996 (-0.008260088067659183) | > avg_loss_disc_real_5: 0.24473799261408793 (+7.483552039511121e-05) | > avg_loss_0: 2.6380410026496564 (-0.047766111266445144) | > avg_grad_norm_0: tensor(236.2158, device='cuda:0') (tensor(-115.0021, device='cuda:0')) | > avg_loss_gen: 2.1295754959885507 (+0.09596350159443601) | > avg_loss_kl: 2.0784654163978478 (-0.20728528667503676) | > avg_loss_feat: 3.7151402822682553 (-0.03825530871539051) | > avg_loss_mel: 16.8993412743152 (-0.715119764838434) | > avg_loss_duration: 1.5390426323447426 (-0.00925241725545578) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.361565119783645 (-0.8739490509033203) | > avg_grad_norm_1: tensor(7332.2466, device='cuda:0') (tensor(-2187.2827, device='cuda:0'))  > EPOCH: 1308/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:41:13)   --> TIME: 2025-02-25 21:41:25 -- STEP: 23/72 -- GLOBAL_STEP: 1094200 | > loss_disc: 2.84145450592041 (2.6844074726104736) | > loss_disc_real_0: 0.11815308034420013 (0.21533949640781982) | > loss_disc_real_1: 0.20643986761569977 (0.2256608994110771) | > loss_disc_real_2: 0.21621333062648773 (0.24192806041758994) | > loss_disc_real_3: 0.2369873821735382 (0.2333574133074802) | > loss_disc_real_4: 0.23183798789978027 (0.2534381300210953) | > loss_disc_real_5: 0.26435625553131104 (0.24444237351417542) | > loss_0: 2.84145450592041 (2.6844074726104736) | > grad_norm_0: tensor(411.1393, device='cuda:0') (tensor(348.8871, device='cuda:0')) | > loss_gen: 1.5885764360427856 (2.139992190443951) | > loss_kl: 2.0034217834472656 (1.9407806759295256) | > loss_feat: 3.6131436824798584 (3.6829448264578115) | > loss_mel: 17.06417465209961 (16.855960265449855) | > loss_duration: 1.5309114456176758 (1.4872627310130908) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.800228118896484 (26.106940725575324) | > grad_norm_1: tensor(6122.4507, device='cuda:0') (tensor(7083.2476, device='cuda:0')) | > current_lr_0: 0.0001698315641805068 | > current_lr_1: 0.0001698315641805068 | > step_time: 0.4411 (0.43496130860370136) | > loader_time: 0.0063 (0.006758347801540209) --> EVAL PERFORMANCE | > avg_loader_time: 0.008701828164114078 (-0.0001162945384710623) | > avg_step_time: 0.45190459237971775 (-0.006108911944107287) | > avg_loss_disc: 2.6817813658378493 (+0.043740363188192966) | > avg_loss_disc_real_0: 0.1733291456082338 (+0.02241467432656763) | > avg_loss_disc_real_1: 0.22088126607344183 (-7.614629789134963e-05) | > avg_loss_disc_real_2: 0.2312169608096002 (-0.005682959942750532) | > avg_loss_disc_real_3: 0.22536965512054066 (-0.01233836504775035) | > avg_loss_disc_real_4: 0.23668004929179876 (-0.0028008883687812014) | > avg_loss_disc_real_5: 0.2450847625732422 (+0.0003467699591542528) | > avg_loss_0: 2.6817813658378493 (+0.043740363188192966) | > avg_grad_norm_0: tensor(274.3568, device='cuda:0') (+tensor(38.1410, device='cuda:0')) | > avg_loss_gen: 2.042003462012385 (-0.08757203397616564) | > avg_loss_kl: 2.028622880787916 (-0.049842535609931904) | > avg_loss_feat: 3.6602628634009564 (-0.05487741886729891) | > avg_loss_mel: 16.779995461584818 (-0.11934581273038347) | > avg_loss_duration: 1.5299706576575696 (-0.009071974687173023) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.04085530025858 (-0.3207098195250637) | > avg_grad_norm_1: tensor(7306.2588, device='cuda:0') (tensor(-25.9878, device='cuda:0'))  > EPOCH: 1309/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:41:50)   --> TIME: 2025-02-25 21:41:52 -- STEP: 1/72 -- GLOBAL_STEP: 1094250 | > loss_disc: 2.689788818359375 (2.689788818359375) | > loss_disc_real_0: 0.25622808933258057 (0.25622808933258057) | > loss_disc_real_1: 0.17979340255260468 (0.17979340255260468) | > loss_disc_real_2: 0.23381735384464264 (0.23381735384464264) | > loss_disc_real_3: 0.24032092094421387 (0.24032092094421387) | > loss_disc_real_4: 0.24660271406173706 (0.24660271406173706) | > loss_disc_real_5: 0.24279864132404327 (0.24279864132404327) | > loss_0: 2.689788818359375 (2.689788818359375) | > grad_norm_0: tensor(904.6825, device='cuda:0') (tensor(904.6825, device='cuda:0')) | > loss_gen: 2.035590410232544 (2.035590410232544) | > loss_kl: 1.6063263416290283 (1.6063263416290283) | > loss_feat: 3.6828253269195557 (3.6828253269195557) | > loss_mel: 17.134958267211914 (17.134958267211914) | > loss_duration: 1.5114586353302002 (1.5114586353302002) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.971160888671875 (25.971160888671875) | > grad_norm_1: tensor(14662.7812, device='cuda:0') (tensor(14662.7812, device='cuda:0')) | > current_lr_0: 0.00016981033523498422 | > current_lr_1: 0.00016981033523498422 | > step_time: 0.4478 (0.4478182792663574) | > loader_time: 0.005 (0.004950284957885742)  --> TIME: 2025-02-25 21:42:15 -- STEP: 51/72 -- GLOBAL_STEP: 1094300 | > loss_disc: 2.575112819671631 (2.6234037922877897) | > loss_disc_real_0: 0.0772552639245987 (0.1630152066548666) | > loss_disc_real_1: 0.22941537201404572 (0.2422440110468397) | > loss_disc_real_2: 0.24257871508598328 (0.2307550197138506) | > loss_disc_real_3: 0.24242427945137024 (0.23089204553295584) | > loss_disc_real_4: 0.2592111825942993 (0.243705433957717) | > loss_disc_real_5: 0.26349306106567383 (0.24404451280247932) | > loss_0: 2.575112819671631 (2.6234037922877897) | > grad_norm_0: tensor(285.8209, device='cuda:0') (tensor(241.3759, device='cuda:0')) | > loss_gen: 2.0656111240386963 (2.223947391790503) | > loss_kl: 2.276641845703125 (2.2855106872670805) | > loss_feat: 3.714174509048462 (3.9015241931466496) | > loss_mel: 16.856943130493164 (17.72002489426557) | > loss_duration: 1.5851807594299316 (1.5450726770887189) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.498550415039062 (27.676079955755494) | > grad_norm_1: tensor(7787.1377, device='cuda:0') (tensor(5829.6479, device='cuda:0')) | > current_lr_0: 0.00016981033523498422 | > current_lr_1: 0.00016981033523498422 | > step_time: 0.452 (0.44371849882836434) | > loader_time: 0.0089 (0.007620465521718941) --> EVAL PERFORMANCE | > avg_loader_time: 0.008505478711195398 (-0.00019634945291868017) | > avg_step_time: 0.45438184872479503 (+0.0024772563450772833) | > avg_loss_disc: 2.642824965463558 (-0.03895640037429127) | > avg_loss_disc_real_0: 0.14103354172597474 (-0.03229560388225905) | > avg_loss_disc_real_1: 0.23589360630008535 (+0.015012340226643522) | > avg_loss_disc_real_2: 0.24843687133889802 (+0.017219910529297833) | > avg_loss_disc_real_3: 0.24501026998943007 (+0.019640614868889406) | > avg_loss_disc_real_4: 0.26211345216757814 (+0.025433402875779387) | > avg_loss_disc_real_5: 0.24078456448837066 (-0.0043001980848715315) | > avg_loss_0: 2.642824965463558 (-0.03895640037429127) | > avg_grad_norm_0: tensor(240.7807, device='cuda:0') (tensor(-33.5761, device='cuda:0')) | > avg_loss_gen: 2.242146527263481 (+0.20014306525109582) | > avg_loss_kl: 2.2706544416051524 (+0.24203156081723654) | > avg_loss_feat: 3.900817404330616 (+0.24055454092965967) | > avg_loss_mel: 17.693715135816117 (+0.9137196742312987) | > avg_loss_duration: 1.5580945384334512 (+0.028123880775881682) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.66542813475703 (+1.6245728344984478) | > avg_grad_norm_1: tensor(5837.5347, device='cuda:0') (tensor(-1468.7241, device='cuda:0'))  > EPOCH: 1310/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:42:27)   --> TIME: 2025-02-25 21:42:42 -- STEP: 29/72 -- GLOBAL_STEP: 1094350 | > loss_disc: 2.763148784637451 (2.6808897297957848) | > loss_disc_real_0: 0.08769600093364716 (0.18238164818492428) | > loss_disc_real_1: 0.27823537588119507 (0.23321920479166097) | > loss_disc_real_2: 0.22658295929431915 (0.24095995210368057) | > loss_disc_real_3: 0.2205987423658371 (0.23645130262292666) | > loss_disc_real_4: 0.25921884179115295 (0.24867479657304697) | > loss_disc_real_5: 0.27679529786109924 (0.24767518557351212) | > loss_0: 2.763148784637451 (2.6808897297957848) | > grad_norm_0: tensor(395.5299, device='cuda:0') (tensor(247.9211, device='cuda:0')) | > loss_gen: 1.8292890787124634 (2.0942062057297806) | > loss_kl: 2.105938196182251 (2.0271355776951236) | > loss_feat: 3.280529022216797 (3.623256000979193) | > loss_mel: 16.91775894165039 (17.120226366766566) | > loss_duration: 1.5218302011489868 (1.4962930473788032) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.655344009399414 (26.36111719854947) | > grad_norm_1: tensor(9150.8682, device='cuda:0') (tensor(7283.3403, device='cuda:0')) | > current_lr_0: 0.00016978910894307984 | > current_lr_1: 0.00016978910894307984 | > step_time: 0.4363 (0.4329079841745311) | > loader_time: 0.0079 (0.007009711758843784) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585459749463578 (+7.99810382681801e-05) | > avg_step_time: 0.45064871411927987 (-0.003733134605515165) | > avg_loss_disc: 2.6628636742981393 (+0.020038708834581254) | > avg_loss_disc_real_0: 0.1683312144497751 (+0.027297672723800376) | > avg_loss_disc_real_1: 0.2324881696365249 (-0.0034054366635604405) | > avg_loss_disc_real_2: 0.2360930600216691 (-0.01234381131722892) | > avg_loss_disc_real_3: 0.23092724634727962 (-0.014083023642150444) | > avg_loss_disc_real_4: 0.2448424944575404 (-0.017270957710037754) | > avg_loss_disc_real_5: 0.24307304578767697 (+0.002288481299306311) | > avg_loss_0: 2.6628636742981393 (+0.020038708834581254) | > avg_grad_norm_0: tensor(245.9614, device='cuda:0') (+tensor(5.1807, device='cuda:0')) | > avg_loss_gen: 2.114891903501161 (-0.12725462376231977) | > avg_loss_kl: 2.069556258094142 (-0.2010981835110104) | > avg_loss_feat: 3.7390882565941608 (-0.1617291477364553) | > avg_loss_mel: 16.984821682244952 (-0.7088934535711644) | > avg_loss_duration: 1.5380056626360181 (-0.020088875797433126) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.446363932649856 (-1.2190642021071731) | > avg_grad_norm_1: tensor(6370.0850, device='cuda:0') (+tensor(532.5503, device='cuda:0'))  > EPOCH: 1311/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:43:04)   --> TIME: 2025-02-25 21:43:09 -- STEP: 7/72 -- GLOBAL_STEP: 1094400 | > loss_disc: 2.5367350578308105 (2.6519661290305003) | > loss_disc_real_0: 0.17292353510856628 (0.1498385550720351) | > loss_disc_real_1: 0.21279661357402802 (0.23440110683441162) | > loss_disc_real_2: 0.2358098030090332 (0.2568927024091993) | > loss_disc_real_3: 0.23647412657737732 (0.23665596970490047) | > loss_disc_real_4: 0.23483124375343323 (0.23669870623520442) | > loss_disc_real_5: 0.21307434141635895 (0.23936751059123448) | > loss_0: 2.5367350578308105 (2.6519661290305003) | > grad_norm_0: tensor(519.5536, device='cuda:0') (tensor(304.0418, device='cuda:0')) | > loss_gen: 2.2134554386138916 (2.055851527622768) | > loss_kl: 2.0063138008117676 (1.979091763496399) | > loss_feat: 4.0974273681640625 (3.769591978618077) | > loss_mel: 17.055776596069336 (17.147886276245117) | > loss_duration: 1.4848169088363647 (1.4927287101745605) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.857789993286133 (26.445150920322963) | > grad_norm_1: tensor(13746.8477, device='cuda:0') (tensor(14005.8330, device='cuda:0')) | > current_lr_0: 0.00016976788530446196 | > current_lr_1: 0.00016976788530446196 | > step_time: 0.4378 (0.4324019636426653) | > loader_time: 0.0056 (0.006381409508841378)  --> TIME: 2025-02-25 21:43:32 -- STEP: 57/72 -- GLOBAL_STEP: 1094450 | > loss_disc: 2.73852801322937 (2.6808811656215727) | > loss_disc_real_0: 0.1508297324180603 (0.13729775824437015) | > loss_disc_real_1: 0.2076795995235443 (0.22216177953962693) | > loss_disc_real_2: 0.23724162578582764 (0.23753434733340614) | > loss_disc_real_3: 0.22569313645362854 (0.23416422150636973) | > loss_disc_real_4: 0.2594192326068878 (0.25651531052171134) | > loss_disc_real_5: 0.243270605802536 (0.2502059528702184) | > loss_0: 2.73852801322937 (2.6808811656215727) | > grad_norm_0: tensor(255.1820, device='cuda:0') (tensor(176.5300, device='cuda:0')) | > loss_gen: 1.7663462162017822 (2.1865240000841912) | > loss_kl: 2.4260332584381104 (2.435971630247015) | > loss_feat: 3.1776881217956543 (4.077678450366907) | > loss_mel: 18.11366081237793 (18.658153098926213) | > loss_duration: 1.622963547706604 (1.5836049778419627) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.106691360473633 (28.94193210936429) | > grad_norm_1: tensor(4089.3623, device='cuda:0') (tensor(6416.5454, device='cuda:0')) | > current_lr_0: 0.00016976788530446196 | > current_lr_1: 0.00016976788530446196 | > step_time: 0.4618 (0.4417113128461336) | > loader_time: 0.0098 (0.007918822137933029) --> EVAL PERFORMANCE | > avg_loader_time: 0.008558340475592814 (-2.711927387076367e-05) | > avg_step_time: 0.4527830741774868 (+0.002134360058206941) | > avg_loss_disc: 2.6665378221323786 (+0.0036741478342392497) | > avg_loss_disc_real_0: 0.13830088055364684 (-0.030030333896128275) | > avg_loss_disc_real_1: 0.22423266965738484 (-0.008255499979140074) | > avg_loss_disc_real_2: 0.2367400346507489 (+0.0006469746290797829) | > avg_loss_disc_real_3: 0.2364196693393546 (+0.0054924229920749745) | > avg_loss_disc_real_4: 0.25392555225063373 (+0.009083057793093346) | > avg_loss_disc_real_5: 0.250681211620989 (+0.0076081658333120306) | > avg_loss_0: 2.6665378221323786 (+0.0036741478342392497) | > avg_grad_norm_0: tensor(170.7907, device='cuda:0') (tensor(-75.1707, device='cuda:0')) | > avg_loss_gen: 2.181394894358138 (+0.06650299085697675) | > avg_loss_kl: 2.4149746878046376 (+0.34541842971049563) | > avg_loss_feat: 3.9609081275026563 (+0.22181987090849553) | > avg_loss_mel: 18.443837447905207 (+1.4590157656602543) | > avg_loss_duration: 1.5890477627096042 (+0.051042100073586116) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.59016289509518 (+2.1437989624453238) | > avg_grad_norm_1: tensor(6300.9580, device='cuda:0') (tensor(-69.1270, device='cuda:0'))  > EPOCH: 1312/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:43:41)   --> TIME: 2025-02-25 21:43:58 -- STEP: 35/72 -- GLOBAL_STEP: 1094500 | > loss_disc: 2.9637298583984375 (2.6484160355159214) | > loss_disc_real_0: 0.03547979146242142 (0.13918711351496835) | > loss_disc_real_1: 0.2639443874359131 (0.23759865505354746) | > loss_disc_real_2: 0.21560479700565338 (0.2528064727783203) | > loss_disc_real_3: 0.21517080068588257 (0.26288682307515826) | > loss_disc_real_4: 0.24640586972236633 (0.2461142144032887) | > loss_disc_real_5: 0.27544182538986206 (0.24668195545673371) | > loss_0: 2.9637298583984375 (2.6484160355159214) | > grad_norm_0: tensor(145.7422, device='cuda:0') (tensor(239.9301, device='cuda:0')) | > loss_gen: 1.6869475841522217 (2.2545068400246757) | > loss_kl: 2.145746946334839 (2.009164251599993) | > loss_feat: 3.3050074577331543 (3.910845729282924) | > loss_mel: 18.39417839050293 (17.22593928745814) | > loss_duration: 1.5469694137573242 (1.51568101474217) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.07884979248047 (26.91613715035575) | > grad_norm_1: tensor(4077.9329, device='cuda:0') (tensor(7624.9492, device='cuda:0')) | > current_lr_0: 0.0001697466643187989 | > current_lr_1: 0.0001697466643187989 | > step_time: 0.4414 (0.4299449920654297) | > loader_time: 0.0084 (0.007120868137904576) --> EVAL PERFORMANCE | > avg_loader_time: 0.008482788650082871 (-7.555182550994348e-05) | > avg_step_time: 0.447084581348258 (-0.005698492829228796) | > avg_loss_disc: 2.6554226472344205 (-0.011115174897958102) | > avg_loss_disc_real_0: 0.13425170457069302 (-0.004049175982953818) | > avg_loss_disc_real_1: 0.23217398977615464 (+0.007941320118769801) | > avg_loss_disc_real_2: 0.24351322462021466 (+0.0067731899694657705) | > avg_loss_disc_real_3: 0.24528300405388148 (+0.008863334714526883) | > avg_loss_disc_real_4: 0.24359938313423748 (-0.01032616911639625) | > avg_loss_disc_real_5: 0.24620586255906332 (-0.004475349061925676) | > avg_loss_0: 2.6554226472344205 (-0.011115174897958102) | > avg_grad_norm_0: tensor(234.4317, device='cuda:0') (+tensor(63.6409, device='cuda:0')) | > avg_loss_gen: 2.1375311327652193 (-0.04386376159291849) | > avg_loss_kl: 2.035282813327413 (-0.37969187447722463) | > avg_loss_feat: 3.7218717991466255 (-0.23903632835603084) | > avg_loss_mel: 17.04423090437768 (-1.3996065435275256) | > avg_loss_duration: 1.5447373893898977 (-0.04431037331970655) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.48365412967306 (-2.10650876542212) | > avg_grad_norm_1: tensor(6409.2412, device='cuda:0') (+tensor(108.2832, device='cuda:0'))  > EPOCH: 1313/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:44:17)   --> TIME: 2025-02-25 21:44:25 -- STEP: 13/72 -- GLOBAL_STEP: 1094550 | > loss_disc: 2.399442434310913 (2.608187492077167) | > loss_disc_real_0: 0.09268207848072052 (0.07358562831695264) | > loss_disc_real_1: 0.2260240763425827 (0.2207903002317135) | > loss_disc_real_2: 0.2095390409231186 (0.23185214629540077) | > loss_disc_real_3: 0.20639342069625854 (0.24579764444094437) | > loss_disc_real_4: 0.22322911024093628 (0.24280354036734655) | > loss_disc_real_5: 0.225216805934906 (0.23314247452295744) | > loss_0: 2.399442434310913 (2.608187492077167) | > grad_norm_0: tensor(19.4164, device='cuda:0') (tensor(252.5785, device='cuda:0')) | > loss_gen: 2.418330192565918 (2.040890253507174) | > loss_kl: 1.9394575357437134 (2.1525083505190334) | > loss_feat: 4.624778747558594 (3.9127138394575853) | > loss_mel: 16.909664154052734 (17.892082654512844) | > loss_duration: 1.447866678237915 (1.4971770965136015) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.340097427368164 (27.495371891902042) | > grad_norm_1: tensor(9426.0830, device='cuda:0') (tensor(7597.8403, device='cuda:0')) | > current_lr_0: 0.00016972544598575903 | > current_lr_1: 0.00016972544598575903 | > step_time: 0.458 (0.4314852311060979) | > loader_time: 0.0057 (0.006974935531616211)  --> TIME: 2025-02-25 21:44:48 -- STEP: 63/72 -- GLOBAL_STEP: 1094600 | > loss_disc: 2.7664804458618164 (2.667460774618482) | > loss_disc_real_0: 0.03331707790493965 (0.10818623370003135) | > loss_disc_real_1: 0.36466240882873535 (0.24643797320978983) | > loss_disc_real_2: 0.20983430743217468 (0.2365178025904156) | > loss_disc_real_3: 0.19976800680160522 (0.2527203921760832) | > loss_disc_real_4: 0.2043730467557907 (0.2617505963832613) | > loss_disc_real_5: 0.25980135798454285 (0.24399297081288837) | > loss_0: 2.7664804458618164 (2.667460774618482) | > grad_norm_0: tensor(373.3614, device='cuda:0') (tensor(271.6104, device='cuda:0')) | > loss_gen: 1.81674063205719 (2.1221245754332774) | > loss_kl: 2.2875261306762695 (2.069804609767975) | > loss_feat: 3.6201560497283936 (3.8746326461670892) | > loss_mel: 18.363718032836914 (17.23431979285347) | > loss_duration: 1.571520209312439 (1.5358908271032665) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.65966033935547 (26.836772404019793) | > grad_norm_1: tensor(12057.0771, device='cuda:0') (tensor(6576.3794, device='cuda:0')) | > current_lr_0: 0.00016972544598575903 | > current_lr_1: 0.00016972544598575903 | > step_time: 0.493 (0.4460818578326513) | > loader_time: 0.0115 (0.008307385066198924) --> EVAL PERFORMANCE | > avg_loader_time: 0.00866235142022791 (+0.00017956277014503919) | > avg_step_time: 0.4533870757465631 (+0.006302494398305103) | > avg_loss_disc: 2.663317787815148 (+0.007895140580727489) | > avg_loss_disc_real_0: 0.10390163619648404 (-0.030350068374208977) | > avg_loss_disc_real_1: 0.25314694328207366 (+0.020972953505919018) | > avg_loss_disc_real_2: 0.23658459874945628 (-0.006928625870758376) | > avg_loss_disc_real_3: 0.24701634930892735 (+0.0017333452550458706) | > avg_loss_disc_real_4: 0.2569522335075996 (+0.013352850373362118) | > avg_loss_disc_real_5: 0.24784911581328217 (+0.0016432532542188516) | > avg_loss_0: 2.663317787815148 (+0.007895140580727489) | > avg_grad_norm_0: tensor(275.3195, device='cuda:0') (+tensor(40.8879, device='cuda:0')) | > avg_loss_gen: 2.119506933319737 (-0.01802419944548239) | > avg_loss_kl: 2.0961183779676205 (+0.06083556464020745) | > avg_loss_feat: 3.8323590721882566 (+0.11048727304163108) | > avg_loss_mel: 17.290015381826493 (+0.2457844774488116) | > avg_loss_duration: 1.5465086715322143 (+0.001771282142316588) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.88450840157522 (+0.40085427190216194) | > avg_grad_norm_1: tensor(7045.9092, device='cuda:0') (+tensor(636.6680, device='cuda:0'))  > EPOCH: 1314/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:44:54)   --> TIME: 2025-02-25 21:45:14 -- STEP: 41/72 -- GLOBAL_STEP: 1094650 | > loss_disc: 2.7550487518310547 (2.597254747297706) | > loss_disc_real_0: 0.07999753206968307 (0.12469494760763357) | > loss_disc_real_1: 0.23389020562171936 (0.21869114186705613) | > loss_disc_real_2: 0.26379022002220154 (0.2396271981117202) | > loss_disc_real_3: 0.2193552702665329 (0.23359040479834486) | > loss_disc_real_4: 0.2372279316186905 (0.23810921900156068) | > loss_disc_real_5: 0.263695627450943 (0.2477307206973797) | > loss_0: 2.7550487518310547 (2.597254747297706) | > grad_norm_0: tensor(44.3898, device='cuda:0') (tensor(293.4736, device='cuda:0')) | > loss_gen: 1.867703914642334 (2.162259197816616) | > loss_kl: 2.040374755859375 (1.9426021837606662) | > loss_feat: 3.3913841247558594 (3.8722326813674552) | > loss_mel: 17.01272964477539 (16.982075063193715) | > loss_duration: 1.5504777431488037 (1.513581979565504) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.8626708984375 (26.472751245266053) | > grad_norm_1: tensor(8889.9121, device='cuda:0') (tensor(8516.4346, device='cuda:0')) | > current_lr_0: 0.0001697042303050108 | > current_lr_1: 0.0001697042303050108 | > step_time: 0.4504 (0.43421919171403095) | > loader_time: 0.0093 (0.007275808148267792) --> EVAL PERFORMANCE | > avg_loader_time: 0.008497553811946383 (-0.0001647976082815271) | > avg_step_time: 0.44788968730980244 (-0.005497388436760675) | > avg_loss_disc: 2.6318380664771714 (-0.031479721337976496) | > avg_loss_disc_real_0: 0.1382287097963649 (+0.03432707359988085) | > avg_loss_disc_real_1: 0.21921222772396787 (-0.03393471555810579) | > avg_loss_disc_real_2: 0.23693368993174863 (+0.0003490911822923437) | > avg_loss_disc_real_3: 0.23421117264620014 (-0.012805176662727208) | > avg_loss_disc_real_4: 0.24252442071135616 (-0.014427812796243444) | > avg_loss_disc_real_5: 0.24607585643378782 (-0.0017732593794943519) | > avg_loss_0: 2.6318380664771714 (-0.031479721337976496) | > avg_grad_norm_0: tensor(286.8072, device='cuda:0') (+tensor(11.4876, device='cuda:0')) | > avg_loss_gen: 2.1479060129380563 (+0.028399079618319334) | > avg_loss_kl: 2.0601580042234606 (-0.035960373744159835) | > avg_loss_feat: 3.8345113908740838 (+0.0021523186858272148) | > avg_loss_mel: 17.14723699865207 (-0.1427783831744236) | > avg_loss_duration: 1.5449433544991722 (-0.0015653170330420796) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.734755878717127 (-0.14975252285809404) | > avg_grad_norm_1: tensor(7763.9673, device='cuda:0') (+tensor(718.0581, device='cuda:0'))  > EPOCH: 1315/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:45:31)   --> TIME: 2025-02-25 21:45:41 -- STEP: 19/72 -- GLOBAL_STEP: 1094700 | > loss_disc: 2.4686696529388428 (2.7325009672265304) | > loss_disc_real_0: 0.04290413483977318 (0.23065353518253878) | > loss_disc_real_1: 0.24741411209106445 (0.23120639982976413) | > loss_disc_real_2: 0.22178468108177185 (0.23035857708830582) | > loss_disc_real_3: 0.235611692070961 (0.23224623187592155) | > loss_disc_real_4: 0.2377012073993683 (0.23416044680695786) | > loss_disc_real_5: 0.23879224061965942 (0.23570370360424645) | > loss_0: 2.4686696529388428 (2.7325009672265304) | > grad_norm_0: tensor(55.7240, device='cuda:0') (tensor(281.6050, device='cuda:0')) | > loss_gen: 2.285553455352783 (1.9770448772530806) | > loss_kl: 2.812412977218628 (2.3311261628803455) | > loss_feat: 4.528073310852051 (3.8206577677475777) | > loss_mel: 19.311365127563477 (18.107745120399876) | > loss_duration: 1.6032025814056396 (1.5535531169489811) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.540607452392578 (27.79012670015034) | > grad_norm_1: tensor(3063.1006, device='cuda:0') (tensor(2471.6230, device='cuda:0')) | > current_lr_0: 0.00016968301727622267 | > current_lr_1: 0.00016968301727622267 | > step_time: 0.4319 (0.4327224430285002) | > loader_time: 0.0078 (0.006772392674496299)  --> TIME: 2025-02-25 21:46:05 -- STEP: 69/72 -- GLOBAL_STEP: 1094750 | > loss_disc: 2.7386980056762695 (2.6682345348855723) | > loss_disc_real_0: 0.16030670702457428 (0.1892953550686007) | > loss_disc_real_1: 0.36720049381256104 (0.2282394027148468) | > loss_disc_real_2: 0.20286938548088074 (0.22434827814931455) | > loss_disc_real_3: 0.1895681917667389 (0.23660862769769586) | > loss_disc_real_4: 0.21982795000076294 (0.2505036279343177) | > loss_disc_real_5: 0.23831506073474884 (0.24428666285846545) | > loss_0: 2.7386980056762695 (2.6682345348855723) | > grad_norm_0: tensor(102.7586, device='cuda:0') (tensor(282.5523, device='cuda:0')) | > loss_gen: 1.8972259759902954 (2.121696676033131) | > loss_kl: 2.2940175533294678 (2.2831814029942383) | > loss_feat: 3.0849506855010986 (3.806032077125881) | > loss_mel: 16.487947463989258 (17.66037433734839) | > loss_duration: 1.673337459564209 (1.578898082608762) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.437480926513672 (27.450182569199715) | > grad_norm_1: tensor(1084.8875, device='cuda:0') (tensor(5668.8232, device='cuda:0')) | > current_lr_0: 0.00016968301727622267 | > current_lr_1: 0.00016968301727622267 | > step_time: 0.5349 (0.44947425178859546) | > loader_time: 0.0129 (0.008540803107662477) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593653289365098 (+9.609947741871447e-05) | > avg_step_time: 0.4507865737861311 (+0.0028968864763286306) | > avg_loss_disc: 2.664746670655801 (+0.03290860417862973) | > avg_loss_disc_real_0: 0.18731261453997922 (+0.04908390474361432) | > avg_loss_disc_real_1: 0.22877605542750426 (+0.009563827703536393) | > avg_loss_disc_real_2: 0.2246778882305387 (-0.012255801701209934) | > avg_loss_disc_real_3: 0.23611847825453314 (+0.0019073056083329931) | > avg_loss_disc_real_4: 0.24981983635627053 (+0.0072954156449143714) | > avg_loss_disc_real_5: 0.24391145164697942 (-0.002164404786808405) | > avg_loss_0: 2.664746670655801 (+0.03290860417862973) | > avg_grad_norm_0: tensor(275.9954, device='cuda:0') (tensor(-10.8118, device='cuda:0')) | > avg_loss_gen: 2.1219727019189114 (-0.025933311019144867) | > avg_loss_kl: 2.276857127606029 (+0.21669912338256836) | > avg_loss_feat: 3.8100051510501913 (-0.02450623982389244) | > avg_loss_mel: 17.615253152981616 (+0.4680161543295469) | > avg_loss_duration: 1.5785997434401176 (+0.03365638894094536) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.402687878675863 (+0.6679319999587356) | > avg_grad_norm_1: tensor(5587.4028, device='cuda:0') (tensor(-2176.5645, device='cuda:0'))  > EPOCH: 1316/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:46:08)   --> TIME: 2025-02-25 21:46:31 -- STEP: 47/72 -- GLOBAL_STEP: 1094800 | > loss_disc: 2.6081740856170654 (2.647985991011275) | > loss_disc_real_0: 0.09486053138971329 (0.16335486081686423) | > loss_disc_real_1: 0.21602478623390198 (0.2284307708131506) | > loss_disc_real_2: 0.2052285522222519 (0.23382109910883803) | > loss_disc_real_3: 0.2171711027622223 (0.23750433579404304) | > loss_disc_real_4: 0.2352810651063919 (0.24772020857384863) | > loss_disc_real_5: 0.2511589527130127 (0.24893355718318452) | > loss_0: 2.6081740856170654 (2.647985991011275) | > grad_norm_0: tensor(48.6259, device='cuda:0') (tensor(220.3831, device='cuda:0')) | > loss_gen: 1.8611563444137573 (2.125080235460972) | > loss_kl: 1.9341944456100464 (1.9824845866954073) | > loss_feat: 3.5278778076171875 (3.7346720847677677) | > loss_mel: 16.716838836669922 (16.937958676764303) | > loss_duration: 1.565211296081543 (1.5228005952023445) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.60527801513672 (26.30299633107287) | > grad_norm_1: tensor(4230.5879, device='cuda:0') (tensor(7394.4912, device='cuda:0')) | > current_lr_0: 0.00016966180689906312 | > current_lr_1: 0.00016966180689906312 | > step_time: 0.4499 (0.43975103662369097) | > loader_time: 0.0086 (0.007495834472331595) --> EVAL PERFORMANCE | > avg_loader_time: 0.008467194060204733 (-0.00012645922916036494) | > avg_step_time: 0.4507582724933893 (-2.8301292741783612e-05) | > avg_loss_disc: 2.639595367539097 (-0.025151303116704327) | > avg_loss_disc_real_0: 0.15760750250077585 (-0.02970511203920337) | > avg_loss_disc_real_1: 0.22895258791010145 (+0.00017653248259719145) | > avg_loss_disc_real_2: 0.23523394595569289 (+0.010556057725154194) | > avg_loss_disc_real_3: 0.2361222784703886 (+3.8002158554706345e-06) | > avg_loss_disc_real_4: 0.24608310133638517 (-0.003736735019885362) | > avg_loss_disc_real_5: 0.2462317578389611 (+0.0023203061919816748) | > avg_loss_0: 2.639595367539097 (-0.025151303116704327) | > avg_grad_norm_0: tensor(204.7836, device='cuda:0') (tensor(-71.2117, device='cuda:0')) | > avg_loss_gen: 2.1172154856399756 (-0.004757216278935772) | > avg_loss_kl: 2.022652674728716 (-0.2542044528773131) | > avg_loss_feat: 3.712296999676127 (-0.0977081513740643) | > avg_loss_mel: 16.853484677596832 (-0.761768475384784) | > avg_loss_duration: 1.5426843921903155 (-0.03591535124980205) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.248334320498184 (-1.1543535581776787) | > avg_grad_norm_1: tensor(7023.2715, device='cuda:0') (+tensor(1435.8687, device='cuda:0'))  > EPOCH: 1317/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:46:44)   --> TIME: 2025-02-25 21:46:57 -- STEP: 25/72 -- GLOBAL_STEP: 1094850 | > loss_disc: 2.6686501502990723 (2.677727460861206) | > loss_disc_real_0: 0.11668474972248077 (0.1631518244743347) | > loss_disc_real_1: 0.25954553484916687 (0.24637805223464965) | > loss_disc_real_2: 0.212198868393898 (0.23456126749515532) | > loss_disc_real_3: 0.22100073099136353 (0.2289677745103836) | > loss_disc_real_4: 0.26443344354629517 (0.24536411285400392) | > loss_disc_real_5: 0.26410216093063354 (0.24323612689971924) | > loss_0: 2.6686501502990723 (2.677727460861206) | > grad_norm_0: tensor(444.5143, device='cuda:0') (tensor(245.4789, device='cuda:0')) | > loss_gen: 1.8120346069335938 (2.1142058086395266) | > loss_kl: 2.11527419090271 (1.951729974746704) | > loss_feat: 3.4711077213287354 (3.7857265186309816) | > loss_mel: 17.496000289916992 (16.93329627990722) | > loss_duration: 1.5110292434692383 (1.5009557390213013) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.405445098876953 (26.285914154052733) | > grad_norm_1: tensor(9878.3066, device='cuda:0') (tensor(7290.1768, device='cuda:0')) | > current_lr_0: 0.00016964059917320073 | > current_lr_1: 0.00016964059917320073 | > step_time: 0.4244 (0.4351853656768799) | > loader_time: 0.0062 (0.006966276168823242) --> EVAL PERFORMANCE | > avg_loader_time: 0.008628794844721404 (+0.0001616007845166717) | > avg_step_time: 0.4530736191171995 (+0.0023153466238102216) | > avg_loss_disc: 2.641499442113957 (+0.0019040745748601928) | > avg_loss_disc_real_0: 0.1531061571563633 (-0.004501345344412544) | > avg_loss_disc_real_1: 0.2334806778481309 (+0.004528089938029445) | > avg_loss_disc_real_2: 0.23649337803813772 (+0.0012594320824448335) | > avg_loss_disc_real_3: 0.2356189831461705 (-0.0005032953242181115) | > avg_loss_disc_real_4: 0.24302721758123855 (-0.003055883755146621) | > avg_loss_disc_real_5: 0.24694628320949177 (+0.0007145253705306831) | > avg_loss_0: 2.641499442113957 (+0.0019040745748601928) | > avg_grad_norm_0: tensor(230.8269, device='cuda:0') (+tensor(26.0433, device='cuda:0')) | > avg_loss_gen: 2.1378316207670824 (+0.020616135127106805) | > avg_loss_kl: 2.0174666542402457 (-0.005186020488470167) | > avg_loss_feat: 3.7246061882502595 (+0.012309188574132435) | > avg_loss_mel: 16.72678240923814 (-0.12670226835869158) | > avg_loss_duration: 1.53775821269398 (-0.004926179496335559) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.144445123806804 (-0.10388919669138019) | > avg_grad_norm_1: tensor(6355.2017, device='cuda:0') (tensor(-668.0698, device='cuda:0'))  > EPOCH: 1318/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:47:21)   --> TIME: 2025-02-25 21:47:24 -- STEP: 3/72 -- GLOBAL_STEP: 1094900 | > loss_disc: 2.6339738368988037 (2.697885831197103) | > loss_disc_real_0: 0.1418558657169342 (0.15854025880495706) | > loss_disc_real_1: 0.17405278980731964 (0.1471132089694341) | > loss_disc_real_2: 0.200102761387825 (0.20439683397610983) | > loss_disc_real_3: 0.21404403448104858 (0.2135323335727056) | > loss_disc_real_4: 0.24286749958992004 (0.23606344064076742) | > loss_disc_real_5: 0.24900314211845398 (0.2442916383345922) | > loss_0: 2.6339738368988037 (2.697885831197103) | > grad_norm_0: tensor(262.5162, device='cuda:0') (tensor(208.4586, device='cuda:0')) | > loss_gen: 1.8762644529342651 (1.7746639251708984) | > loss_kl: 2.0469322204589844 (2.017012357711792) | > loss_feat: 3.850451707839966 (3.5819461345672607) | > loss_mel: 17.05057716369629 (17.119378407796223) | > loss_duration: 1.504136562347412 (1.4987670580546062) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.32836151123047 (25.991767247517902) | > grad_norm_1: tensor(6007.3394, device='cuda:0') (tensor(5347.4775, device='cuda:0')) | > current_lr_0: 0.00016961939409830407 | > current_lr_1: 0.00016961939409830407 | > step_time: 0.4201 (0.4177900155385335) | > loader_time: 0.0096 (0.0077025095621744795)  --> TIME: 2025-02-25 21:47:47 -- STEP: 53/72 -- GLOBAL_STEP: 1094950 | > loss_disc: 2.806103229522705 (2.651068075647894) | > loss_disc_real_0: 0.20467013120651245 (0.1598854755174439) | > loss_disc_real_1: 0.2589849531650543 (0.2214517506226054) | > loss_disc_real_2: 0.2509961426258087 (0.23214879688226953) | > loss_disc_real_3: 0.23975461721420288 (0.23337621705712014) | > loss_disc_real_4: 0.284869521856308 (0.24444496013083547) | > loss_disc_real_5: 0.2549213767051697 (0.24395217535630712) | > loss_0: 2.806103229522705 (2.651068075647894) | > grad_norm_0: tensor(10.9292, device='cuda:0') (tensor(176.7591, device='cuda:0')) | > loss_gen: 1.7881935834884644 (2.1117034858127806) | > loss_kl: 2.054319143295288 (2.12197743496805) | > loss_feat: 2.8886473178863525 (3.7847520585330026) | > loss_mel: 15.778483390808105 (17.256005017262588) | > loss_duration: 1.563030481338501 (1.529138886703635) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.072673797607422 (26.80357684729234) | > grad_norm_1: tensor(4558.5972, device='cuda:0') (tensor(6061.8110, device='cuda:0')) | > current_lr_0: 0.00016961939409830407 | > current_lr_1: 0.00016961939409830407 | > step_time: 0.4468 (0.4396011964330133) | > loader_time: 0.0092 (0.008014638468904316) --> EVAL PERFORMANCE | > avg_loader_time: 0.008761463030962877 (+0.0001326681862414726) | > avg_step_time: 0.4514134400327441 (-0.0016601790844554265) | > avg_loss_disc: 2.6489968904307193 (+0.007497448316762245) | > avg_loss_disc_real_0: 0.15252787556866526 (-0.0005782815876980429) | > avg_loss_disc_real_1: 0.21962309104036276 (-0.013857586807768135) | > avg_loss_disc_real_2: 0.22958151408484284 (-0.006911863953294878) | > avg_loss_disc_real_3: 0.23298324686540683 (-0.002635736280763662) | > avg_loss_disc_real_4: 0.24214348196983337 (-0.0008837356114051731) | > avg_loss_disc_real_5: 0.24332982889363464 (-0.003616454315857137) | > avg_loss_0: 2.6489968904307193 (+0.007497448316762245) | > avg_grad_norm_0: tensor(194.4358, device='cuda:0') (tensor(-36.3911, device='cuda:0')) | > avg_loss_gen: 2.0882026259328272 (-0.0496289948342552) | > avg_loss_kl: 2.137487712040754 (+0.12002105780050831) | > avg_loss_feat: 3.743065588910815 (+0.018459400660555403) | > avg_loss_mel: 17.181605527098753 (+0.454823117860613) | > avg_loss_duration: 1.5434266681402502 (+0.00566845544627026) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.693788179209534 (+0.5493430554027299) | > avg_grad_norm_1: tensor(5698.8354, device='cuda:0') (tensor(-656.3662, device='cuda:0'))  > EPOCH: 1319/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:47:58)   --> TIME: 2025-02-25 21:48:14 -- STEP: 31/72 -- GLOBAL_STEP: 1095000 | > loss_disc: 2.6278626918792725 (2.590653457949239) | > loss_disc_real_0: 0.07148866355419159 (0.18465663096116436) | > loss_disc_real_1: 0.2339274138212204 (0.21595527135556744) | > loss_disc_real_2: 0.23603518307209015 (0.23348514927971747) | > loss_disc_real_3: 0.2032196819782257 (0.23208459346525132) | > loss_disc_real_4: 0.25673699378967285 (0.24043597761661775) | > loss_disc_real_5: 0.23337122797966003 (0.24102733885088273) | > loss_0: 2.6278626918792725 (2.590653457949239) | > grad_norm_0: tensor(282.9700, device='cuda:0') (tensor(266.4585, device='cuda:0')) | > loss_gen: 1.8331279754638672 (2.201080979839448) | > loss_kl: 2.008862018585205 (1.944389408634555) | > loss_feat: 3.8532094955444336 (3.9044764118809856) | > loss_mel: 17.48906898498535 (16.85767472174859) | > loss_duration: 1.5066579580307007 (1.4918606319735124) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.690927505493164 (26.39948223483178) | > grad_norm_1: tensor(9637.6172, device='cuda:0') (tensor(5912.6118, device='cuda:0')) | > current_lr_0: 0.00016959819167404176 | > current_lr_1: 0.00016959819167404176 | > step_time: 0.4412 (0.43229357657894013) | > loader_time: 0.0088 (0.0070371243261521865) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1095000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.0086462161910366 (-0.00011524683992627724) | > avg_step_time: 0.45194871660689234 (+0.0005352765741482579) | > avg_loss_disc: 2.635803628975237 (-0.013193261455482386) | > avg_loss_disc_real_0: 0.1816486021057821 (+0.029120726537116837) | > avg_loss_disc_real_1: 0.21941243259000107 (-0.00021065845036169484) | > avg_loss_disc_real_2: 0.23644221794437353 (+0.006860703859530692) | > avg_loss_disc_real_3: 0.23084582745189397 (-0.0021374194135128644) | > avg_loss_disc_real_4: 0.24137208306453597 (-0.0007713989052973991) | > avg_loss_disc_real_5: 0.2431835603546089 (-0.00014626853902574566) | > avg_loss_0: 2.635803628975237 (-0.013193261455482386) | > avg_grad_norm_0: tensor(226.0119, device='cuda:0') (+tensor(31.5761, device='cuda:0')) | > avg_loss_gen: 2.1450225484203287 (+0.05681992248750145) | > avg_loss_kl: 2.126526610952028 (-0.010961101088725833) | > avg_loss_feat: 3.731083023716027 (-0.01198256519478802) | > avg_loss_mel: 17.11082907125983 (-0.07077645583892433) | > avg_loss_duration: 1.5364412908822718 (-0.006985377257978431) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.64990269298285 (-0.04388548622668509) | > avg_grad_norm_1: tensor(6319.6196, device='cuda:0') (+tensor(620.7842, device='cuda:0'))  > EPOCH: 1320/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:48:38)   --> TIME: 2025-02-25 21:48:44 -- STEP: 9/72 -- GLOBAL_STEP: 1095050 | > loss_disc: 2.725573778152466 (2.7796446747250028) | > loss_disc_real_0: 0.2554406225681305 (0.25744282868173385) | > loss_disc_real_1: 0.20853768289089203 (0.2263772620095147) | > loss_disc_real_2: 0.1908421367406845 (0.2088913851314121) | > loss_disc_real_3: 0.22623641788959503 (0.24586535659101275) | > loss_disc_real_4: 0.2509912848472595 (0.268600485391087) | > loss_disc_real_5: 0.24104568362236023 (0.24776768518818748) | > loss_0: 2.725573778152466 (2.7796446747250028) | > grad_norm_0: tensor(394.0924, device='cuda:0') (tensor(343.1118, device='cuda:0')) | > loss_gen: 2.2156357765197754 (2.0869477457470365) | > loss_kl: 2.5023951530456543 (2.4549105299843683) | > loss_feat: 4.264003276824951 (3.752305269241333) | > loss_mel: 18.530834197998047 (19.53820737202962) | > loss_duration: 1.6003165245056152 (1.6533951097064548) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.11318588256836 (29.485766304863823) | > grad_norm_1: tensor(851.3614, device='cuda:0') (tensor(3891.5378, device='cuda:0')) | > current_lr_0: 0.0001695769919000825 | > current_lr_1: 0.0001695769919000825 | > step_time: 0.4251 (0.4344371159871419) | > loader_time: 0.0084 (0.006849024030897353)  --> TIME: 2025-02-25 21:49:08 -- STEP: 59/72 -- GLOBAL_STEP: 1095100 | > loss_disc: 2.624948263168335 (2.5971830537763694) | > loss_disc_real_0: 0.04634308069944382 (0.14819380293710754) | > loss_disc_real_1: 0.20669059455394745 (0.2142705052333363) | > loss_disc_real_2: 0.20107364654541016 (0.22433799429465148) | > loss_disc_real_3: 0.24450033903121948 (0.23022296317553115) | > loss_disc_real_4: 0.2673719525337219 (0.23920477118532538) | > loss_disc_real_5: 0.2534661293029785 (0.2404702379036758) | > loss_0: 2.624948263168335 (2.5971830537763694) | > grad_norm_0: tensor(183.2411, device='cuda:0') (tensor(231.7999, device='cuda:0')) | > loss_gen: 2.043186902999878 (2.1637606863248156) | > loss_kl: 2.1186208724975586 (2.3319721019874184) | > loss_feat: 3.7379846572875977 (4.0565489914457675) | > loss_mel: 16.303972244262695 (18.12967043407893) | > loss_duration: 1.5794684886932373 (1.6000980441853152) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.783231735229492 (28.282050375211035) | > grad_norm_1: tensor(6069.1714, device='cuda:0') (tensor(6363.4541, device='cuda:0')) | > current_lr_0: 0.0001695769919000825 | > current_lr_1: 0.0001695769919000825 | > step_time: 0.4715 (0.4517996917336674) | > loader_time: 0.0103 (0.008473412465240995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008997715694803588 (+0.0003514995037669881) | > avg_step_time: 0.46077568430296134 (+0.008826967696068999) | > avg_loss_disc: 2.6549064441465995 (+0.019102815171362586) | > avg_loss_disc_real_0: 0.13974613554670776 (-0.04190246655907434) | > avg_loss_disc_real_1: 0.2047674061966614 (-0.014645026393339672) | > avg_loss_disc_real_2: 0.21460168525366716 (-0.021840532690706377) | > avg_loss_disc_real_3: 0.22684212469718826 (-0.0040037027547057125) | > avg_loss_disc_real_4: 0.25295996812867444 (+0.011587885064138465) | > avg_loss_disc_real_5: 0.24481052601001632 (+0.0016269656554074308) | > avg_loss_0: 2.6549064441465995 (+0.019102815171362586) | > avg_grad_norm_0: tensor(274.7534, device='cuda:0') (+tensor(48.7415, device='cuda:0')) | > avg_loss_gen: 2.070549441055513 (-0.0744731073648155) | > avg_loss_kl: 2.3051772352675317 (+0.1786506243155035) | > avg_loss_feat: 3.929737695505921 (+0.19865467178989427) | > avg_loss_mel: 17.940109454410177 (+0.8292803831503477) | > avg_loss_duration: 1.5998189063139368 (+0.06337761543166498) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.845392791318222 (+1.1954900983353731) | > avg_grad_norm_1: tensor(6433.1733, device='cuda:0') (+tensor(113.5537, device='cuda:0'))  > EPOCH: 1321/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:49:16)   --> TIME: 2025-02-25 21:49:34 -- STEP: 37/72 -- GLOBAL_STEP: 1095150 | > loss_disc: 2.7213072776794434 (2.56984543800354) | > loss_disc_real_0: 0.17153431475162506 (0.12527848659334953) | > loss_disc_real_1: 0.22253918647766113 (0.22211536765098572) | > loss_disc_real_2: 0.258140504360199 (0.22828743546395688) | > loss_disc_real_3: 0.23009414970874786 (0.22814386519225868) | > loss_disc_real_4: 0.221750870347023 (0.23342409005036224) | > loss_disc_real_5: 0.2153785228729248 (0.24187983089202159) | > loss_0: 2.7213072776794434 (2.56984543800354) | > grad_norm_0: tensor(686.1896, device='cuda:0') (tensor(307.9146, device='cuda:0')) | > loss_gen: 1.8871886730194092 (2.0865658070590043) | > loss_kl: 1.93198823928833 (1.9795623792184365) | > loss_feat: 3.5802083015441895 (3.8536439715205013) | > loss_mel: 17.119028091430664 (16.89731402010531) | > loss_duration: 1.535444974899292 (1.5156635368192517) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.053857803344727 (26.332749701835013) | > grad_norm_1: tensor(11126.4971, device='cuda:0') (tensor(12360.1758, device='cuda:0')) | > current_lr_0: 0.00016955579477609498 | > current_lr_1: 0.00016955579477609498 | > step_time: 0.4493 (0.43937769451656855) | > loader_time: 0.008 (0.007337544415448163) --> EVAL PERFORMANCE | > avg_loader_time: 0.008698067194978955 (-0.00029964849982463265) | > avg_step_time: 0.45604701780937085 (-0.004728666493590494) | > avg_loss_disc: 2.576098045832674 (-0.07880839831392539) | > avg_loss_disc_real_0: 0.1226941629404753 (-0.01705197260623245) | > avg_loss_disc_real_1: 0.22052407054833964 (+0.015756664351678246) | > avg_loss_disc_real_2: 0.22826939527417572 (+0.013667710020508567) | > avg_loss_disc_real_3: 0.22863458411794313 (+0.0017924594207548716) | > avg_loss_disc_real_4: 0.2351032814509432 (-0.017856686677731243) | > avg_loss_disc_real_5: 0.2438824944932696 (-0.0009280315167467246) | > avg_loss_0: 2.576098045832674 (-0.07880839831392539) | > avg_grad_norm_0: tensor(334.3185, device='cuda:0') (+tensor(59.5651, device='cuda:0')) | > avg_loss_gen: 2.0746243990642923 (+0.004074958008779106) | > avg_loss_kl: 2.011824822761643 (-0.2933524125058886) | > avg_loss_feat: 3.819459790914831 (-0.11027790459108999) | > avg_loss_mel: 16.85140847488188 (-1.0887009795282978) | > avg_loss_duration: 1.5422725795020518 (-0.057546326811884985) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.299590097346776 (-1.5458026939714458) | > avg_grad_norm_1: tensor(11718.5420, device='cuda:0') (+tensor(5285.3687, device='cuda:0'))  > EPOCH: 1322/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:49:53)   --> TIME: 2025-02-25 21:50:01 -- STEP: 15/72 -- GLOBAL_STEP: 1095200 | > loss_disc: 2.4022789001464844 (2.5464701016743976) | > loss_disc_real_0: 0.10888232290744781 (0.11715235362450281) | > loss_disc_real_1: 0.27513909339904785 (0.22284370462099712) | > loss_disc_real_2: 0.24110892415046692 (0.2328198581933975) | > loss_disc_real_3: 0.21789991855621338 (0.22512538731098175) | > loss_disc_real_4: 0.23099389672279358 (0.23890944023927052) | > loss_disc_real_5: 0.24718120694160461 (0.23813753724098205) | > loss_0: 2.4022789001464844 (2.5464701016743976) | > grad_norm_0: tensor(110.6500, device='cuda:0') (tensor(298.9040, device='cuda:0')) | > loss_gen: 2.1050896644592285 (2.127598937352498) | > loss_kl: 1.8366883993148804 (1.882022738456726) | > loss_feat: 3.8344085216522217 (3.954190413157145) | > loss_mel: 17.40876579284668 (16.798425738016764) | > loss_duration: 1.4775954484939575 (1.496979284286499) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.662548065185547 (26.259217071533204) | > grad_norm_1: tensor(14829.7080, device='cuda:0') (tensor(9942.5273, device='cuda:0')) | > current_lr_0: 0.00016953460030174795 | > current_lr_1: 0.00016953460030174795 | > step_time: 0.4247 (0.44115856488545735) | > loader_time: 0.0058 (0.006920242309570312)  --> TIME: 2025-02-25 21:50:25 -- STEP: 65/72 -- GLOBAL_STEP: 1095250 | > loss_disc: 2.8672378063201904 (2.5589151529165406) | > loss_disc_real_0: 0.2265910804271698 (0.12389002992556646) | > loss_disc_real_1: 0.24147020280361176 (0.21875165173640618) | > loss_disc_real_2: 0.2821004092693329 (0.2294265178533701) | > loss_disc_real_3: 0.2428731918334961 (0.22962855444504665) | > loss_disc_real_4: 0.2751636505126953 (0.24143434327382307) | > loss_disc_real_5: 0.2719650864601135 (0.24600244966837076) | > loss_0: 2.8672378063201904 (2.5589151529165406) | > grad_norm_0: tensor(759.8633, device='cuda:0') (tensor(355.8889, device='cuda:0')) | > loss_gen: 1.9459078311920166 (2.1371743330588706) | > loss_kl: 2.1736319065093994 (2.085775562433098) | > loss_feat: 3.0630617141723633 (3.9012038377615124) | > loss_mel: 17.037952423095703 (16.978491460360015) | > loss_duration: 1.5988869667053223 (1.5396746085240292) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.819440841674805 (26.642319869995116) | > grad_norm_1: tensor(7238.5566, device='cuda:0') (tensor(11564.5850, device='cuda:0')) | > current_lr_0: 0.00016953460030174795 | > current_lr_1: 0.00016953460030174795 | > step_time: 0.496 (0.4615929933694693) | > loader_time: 0.012 (0.008436004932110126) --> EVAL PERFORMANCE | > avg_loader_time: 0.008702153890905246 (+4.08669592629092e-06) | > avg_step_time: 0.4664815446020852 (+0.010434526792714338) | > avg_loss_disc: 2.5797564143865874 (+0.003658368553913327) | > avg_loss_disc_real_0: 0.12698895746553449 (+0.004294794525059181) | > avg_loss_disc_real_1: 0.22124133731277895 (+0.0007172667644393116) | > avg_loss_disc_real_2: 0.2268643597481956 (-0.0014050355259801195) | > avg_loss_disc_real_3: 0.22894127478062268 (+0.0003066906626795485) | > avg_loss_disc_real_4: 0.24070470748652875 (+0.005601426035585555) | > avg_loss_disc_real_5: 0.24587974057231152 (+0.00199724607904192) | > avg_loss_0: 2.5797564143865874 (+0.003658368553913327) | > avg_grad_norm_0: tensor(365.5463, device='cuda:0') (+tensor(31.2278, device='cuda:0')) | > avg_loss_gen: 2.1069047585339615 (+0.03228035946966923) | > avg_loss_kl: 2.1009119631538944 (+0.0890871403922513) | > avg_loss_feat: 3.8918978798557333 (+0.0724380889409022) | > avg_loss_mel: 17.01710006552683 (+0.16569159064495054) | > avg_loss_duration: 1.545981204006034 (+0.0037086245039821275) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.66279591305155 (+0.3632058157047737) | > avg_grad_norm_1: tensor(11228.1445, device='cuda:0') (tensor(-490.3975, device='cuda:0'))  > EPOCH: 1323/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:50:30)   --> TIME: 2025-02-25 21:50:52 -- STEP: 43/72 -- GLOBAL_STEP: 1095300 | > loss_disc: 2.6202077865600586 (2.6695424124251965) | > loss_disc_real_0: 0.13274440169334412 (0.1674157412246216) | > loss_disc_real_1: 0.3534424602985382 (0.23964625700961711) | > loss_disc_real_2: 0.2818966805934906 (0.23323465052039125) | > loss_disc_real_3: 0.3030337393283844 (0.24410722699276236) | > loss_disc_real_4: 0.270235151052475 (0.23736264158126918) | > loss_disc_real_5: 0.2503211498260498 (0.24666741386402485) | > loss_0: 2.6202077865600586 (2.6695424124251965) | > grad_norm_0: tensor(549.1000, device='cuda:0') (tensor(308.5509, device='cuda:0')) | > loss_gen: 2.5251686573028564 (2.163311883460643) | > loss_kl: 2.4233322143554688 (2.2661245257355445) | > loss_feat: 3.948246717453003 (3.914093810458516) | > loss_mel: 17.229990005493164 (18.105323392291407) | > loss_duration: 1.5854146480560303 (1.5285790243814157) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.712154388427734 (27.97743282761685) | > grad_norm_1: tensor(7812.4058, device='cuda:0') (tensor(6041.3076, device='cuda:0')) | > current_lr_0: 0.0001695134084767102 | > current_lr_1: 0.0001695134084767102 | > step_time: 0.4511 (0.4377382023389949) | > loader_time: 0.01 (0.007597657137138899) --> EVAL PERFORMANCE | > avg_loader_time: 0.008793394330521707 (+9.124043961646067e-05) | > avg_step_time: 0.45405602119338345 (-0.012425523408701733) | > avg_loss_disc: 2.599037751345567 (+0.01928133695897971) | > avg_loss_disc_real_0: 0.13299180462326796 (+0.006002847157733471) | > avg_loss_disc_real_1: 0.23128739681462168 (+0.01004605950184273) | > avg_loss_disc_real_2: 0.23725379268888017 (+0.010389432940684562) | > avg_loss_disc_real_3: 0.24242887089789752 (+0.013487596117274847) | > avg_loss_disc_real_4: 0.23619302372697373 (-0.0045116837595550185) | > avg_loss_disc_real_5: 0.24559571100792416 (-0.0002840295643873614) | > avg_loss_0: 2.599037751345567 (+0.01928133695897971) | > avg_grad_norm_0: tensor(250.0325, device='cuda:0') (tensor(-115.5138, device='cuda:0')) | > avg_loss_gen: 2.2452933637189187 (+0.13838860518495721) | > avg_loss_kl: 2.330095720962739 (+0.22918375780884448) | > avg_loss_feat: 4.056512144249931 (+0.16461426439419746) | > avg_loss_mel: 18.14070196554695 (+1.1236019000201196) | > avg_loss_duration: 1.56199694687212 (+0.01601574286608609) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.334600206831812 (+1.6718042937802622) | > avg_grad_norm_1: tensor(5636.5312, device='cuda:0') (tensor(-5591.6133, device='cuda:0'))  > EPOCH: 1324/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:51:07)   --> TIME: 2025-02-25 21:51:18 -- STEP: 21/72 -- GLOBAL_STEP: 1095350 | > loss_disc: 2.5687522888183594 (2.6698148591177806) | > loss_disc_real_0: 0.23991110920906067 (0.12229462216297786) | > loss_disc_real_1: 0.23396041989326477 (0.23106381935732706) | > loss_disc_real_2: 0.24970130622386932 (0.2325326736484255) | > loss_disc_real_3: 0.2379843145608902 (0.24057584291412717) | > loss_disc_real_4: 0.24629804491996765 (0.24942441284656525) | > loss_disc_real_5: 0.23582766950130463 (0.24393960265886216) | > loss_0: 2.5687522888183594 (2.6698148591177806) | > grad_norm_0: tensor(102.3928, device='cuda:0') (tensor(192.1845, device='cuda:0')) | > loss_gen: 2.604495048522949 (2.097095977692377) | > loss_kl: 1.9058729410171509 (1.9628665844599407) | > loss_feat: 3.8957314491271973 (3.9267687116350447) | > loss_mel: 16.154640197753906 (17.302500497727163) | > loss_duration: 1.513378620147705 (1.5025381190436227) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.074119567871094 (26.79176984514509) | > grad_norm_1: tensor(4219.9092, device='cuda:0') (tensor(9232.5645, device='cuda:0')) | > current_lr_0: 0.00016949221930065062 | > current_lr_1: 0.00016949221930065062 | > step_time: 0.4259 (0.4316090061551049) | > loader_time: 0.0061 (0.006748369761875698)  --> TIME: 2025-02-25 21:51:42 -- STEP: 71/72 -- GLOBAL_STEP: 1095400 | > loss_disc: 2.615614414215088 (2.6568703987229036) | > loss_disc_real_0: 0.22021833062171936 (0.12177809946973558) | > loss_disc_real_1: 0.22223371267318726 (0.23528133741986584) | > loss_disc_real_2: 0.23792089521884918 (0.23407045426503034) | > loss_disc_real_3: 0.21799391508102417 (0.23788932658417125) | > loss_disc_real_4: 0.2393621951341629 (0.24480655096786122) | > loss_disc_real_5: 0.20405752956867218 (0.2456456722927765) | > loss_0: 2.615614414215088 (2.6568703987229036) | > grad_norm_0: tensor(326.0195, device='cuda:0') (tensor(243.8862, device='cuda:0')) | > loss_gen: 2.425710439682007 (2.076179625282825) | > loss_kl: 1.9240834712982178 (2.0123057650848173) | > loss_feat: 3.756836414337158 (3.79030356608646) | > loss_mel: 16.165895462036133 (16.90006092232717) | > loss_duration: 1.547360897064209 (1.53802679626035) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.819887161254883 (26.316876747238805) | > grad_norm_1: tensor(4787.4697, device='cuda:0') (tensor(8030.3662, device='cuda:0')) | > current_lr_0: 0.00016949221930065062 | > current_lr_1: 0.00016949221930065062 | > step_time: 0.489 (0.45157331815907653) | > loader_time: 0.0084 (0.008692173890664543) --> EVAL PERFORMANCE | > avg_loader_time: 0.008692173890664543 (-0.00010122043985716367) | > avg_step_time: 0.45157331815907653 (-0.002482703034306921) | > avg_loss_disc: 2.6568703987229036 (+0.05783264737733651) | > avg_loss_disc_real_0: 0.12177809946973558 (-0.011213705153532372) | > avg_loss_disc_real_1: 0.23528133741986584 (+0.003993940605244162) | > avg_loss_disc_real_2: 0.23407045426503034 (-0.003183338423849824) | > avg_loss_disc_real_3: 0.23788932658417125 (-0.004539544313726274) | > avg_loss_disc_real_4: 0.24480655096786122 (+0.008613527240887486) | > avg_loss_disc_real_5: 0.2456456722927765 (+4.9961284852351184e-05) | > avg_loss_0: 2.6568703987229036 (+0.05783264737733651) | > avg_grad_norm_0: tensor(243.8862, device='cuda:0') (tensor(-6.1463, device='cuda:0')) | > avg_loss_gen: 2.076179625282825 (-0.1691137384360939) | > avg_loss_kl: 2.0123057650848173 (-0.3177899558779216) | > avg_loss_feat: 3.79030356608646 (-0.26620857816347065) | > avg_loss_mel: 16.90006092232717 (-1.240641043219778) | > avg_loss_duration: 1.53802679626035 (-0.02397015061176999) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.316876747238805 (-2.0177234595930074) | > avg_grad_norm_1: tensor(8030.3662, device='cuda:0') (+tensor(2393.8350, device='cuda:0'))  > EPOCH: 1325/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:51:44)   --> TIME: 2025-02-25 21:52:08 -- STEP: 49/72 -- GLOBAL_STEP: 1095450 | > loss_disc: 2.5163414478302 (2.5996705317983824) | > loss_disc_real_0: 0.09523026645183563 (0.12373067666681448) | > loss_disc_real_1: 0.2443884015083313 (0.23628308791287092) | > loss_disc_real_2: 0.2260027974843979 (0.23495927422630544) | > loss_disc_real_3: 0.23933066427707672 (0.23637877070173927) | > loss_disc_real_4: 0.2563149034976959 (0.24191242699720422) | > loss_disc_real_5: 0.2484763264656067 (0.24233568049207024) | > loss_0: 2.5163414478302 (2.5996705317983824) | > grad_norm_0: tensor(250.8582, device='cuda:0') (tensor(238.4361, device='cuda:0')) | > loss_gen: 2.1245102882385254 (2.124860668668942) | > loss_kl: 2.0670135021209717 (2.1868355006587747) | > loss_feat: 3.679231643676758 (3.917804309300014) | > loss_mel: 17.486839294433594 (17.587407014807876) | > loss_duration: 1.5765756368637085 (1.5416077788995237) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.934171676635742 (27.358515330723353) | > grad_norm_1: tensor(9710.1094, device='cuda:0') (tensor(6525.0063, device='cuda:0')) | > current_lr_0: 0.00016947103277323804 | > current_lr_1: 0.00016947103277323804 | > step_time: 0.4458 (0.437578449443895) | > loader_time: 0.0098 (0.00767668899224729) --> EVAL PERFORMANCE | > avg_loader_time: 0.008613787906270631 (-7.838598439391213e-05) | > avg_step_time: 0.4490487474790762 (-0.0025245706800003376) | > avg_loss_disc: 2.567788762106022 (-0.08908163661688162) | > avg_loss_disc_real_0: 0.11027181405626552 (-0.011506285413470069) | > avg_loss_disc_real_1: 0.2344528016909747 (-0.0008285357288911332) | > avg_loss_disc_real_2: 0.23392118723459646 (-0.00014926703043388612) | > avg_loss_disc_real_3: 0.23451118087264852 (-0.0033781457115227287) | > avg_loss_disc_real_4: 0.24158926203217304 (-0.0032172889356881784) | > avg_loss_disc_real_5: 0.241603220432577 (-0.004042451860199497) | > avg_loss_0: 2.567788762106022 (-0.08908163661688162) | > avg_grad_norm_0: tensor(248.3474, device='cuda:0') (+tensor(4.4612, device='cuda:0')) | > avg_loss_gen: 2.149654913956011 (+0.07347528867318598) | > avg_loss_kl: 2.1808584525551593 (+0.16855268747034202) | > avg_loss_feat: 3.9546362682127616 (+0.16433270212630147) | > avg_loss_mel: 17.383807168879972 (+0.4837462465528013) | > avg_loss_duration: 1.5551911223102624 (+0.01716432604991236) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.22414798467932 (+0.907271237440515) | > avg_grad_norm_1: tensor(7215.0557, device='cuda:0') (tensor(-815.3105, device='cuda:0'))  > EPOCH: 1326/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:52:21)   --> TIME: 2025-02-25 21:52:35 -- STEP: 27/72 -- GLOBAL_STEP: 1095500 | > loss_disc: 2.5578932762145996 (2.497907620889169) | > loss_disc_real_0: 0.06730125099420547 (0.0847409802178542) | > loss_disc_real_1: 0.20564883947372437 (0.22277558236210435) | > loss_disc_real_2: 0.23229989409446716 (0.22759670626234124) | > loss_disc_real_3: 0.2406861037015915 (0.22639444635974038) | > loss_disc_real_4: 0.2543700337409973 (0.24710909249605956) | > loss_disc_real_5: 0.2402893304824829 (0.24634712934494019) | > loss_0: 2.5578932762145996 (2.497907620889169) | > grad_norm_0: tensor(200.2548, device='cuda:0') (tensor(288.4348, device='cuda:0')) | > loss_gen: 2.298784017562866 (2.26177884031225) | > loss_kl: 2.0593245029449463 (1.9271327213004783) | > loss_feat: 4.540918827056885 (4.235273793891625) | > loss_mel: 16.975265502929688 (16.715839174058708) | > loss_duration: 1.49793541431427 (1.496013879776001) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.372228622436523 (26.636038391678422) | > grad_norm_1: tensor(8203.4736, device='cuda:0') (tensor(8548.3252, device='cuda:0')) | > current_lr_0: 0.00016944984889414138 | > current_lr_1: 0.00016944984889414138 | > step_time: 0.4405 (0.435476197136773) | > loader_time: 0.0078 (0.006585121154785156) --> EVAL PERFORMANCE | > avg_loader_time: 0.008500001799892375 (-0.0001137861063782563) | > avg_step_time: 0.455516110003834 (+0.006467362524757825) | > avg_loss_disc: 2.4958444414004473 (-0.07194432070557477) | > avg_loss_disc_real_0: 0.0917065207072547 (-0.018565293349010817) | > avg_loss_disc_real_1: 0.21970491992755675 (-0.014747881763417958) | > avg_loss_disc_real_2: 0.23163005101009154 (-0.002291136224504914) | > avg_loss_disc_real_3: 0.22981487298515482 (-0.0046963078874937) | > avg_loss_disc_real_4: 0.2438826730973284 (+0.0022934110651553485) | > avg_loss_disc_real_5: 0.24543259597160447 (+0.0038293755390274575) | > avg_loss_0: 2.4958444414004473 (-0.07194432070557477) | > avg_grad_norm_0: tensor(291.0877, device='cuda:0') (+tensor(42.7403, device='cuda:0')) | > avg_loss_gen: 2.264663906164571 (+0.1150089922085602) | > avg_loss_kl: 1.992016740248237 (-0.18884171230692237) | > avg_loss_feat: 4.154944235170394 (+0.2003079669576322) | > avg_loss_mel: 16.721974990737262 (-0.6618321781427099) | > avg_loss_duration: 1.5351507630146726 (-0.020040359295589782) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.66875060175506 (-0.5553973829242587) | > avg_grad_norm_1: tensor(8512.9580, device='cuda:0') (+tensor(1297.9023, device='cuda:0'))  > EPOCH: 1327/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:52:58)   --> TIME: 2025-02-25 21:53:02 -- STEP: 5/72 -- GLOBAL_STEP: 1095550 | > loss_disc: 2.4338979721069336 (2.477280855178833) | > loss_disc_real_0: 0.08018939197063446 (0.10837562531232833) | > loss_disc_real_1: 0.2386787384748459 (0.217863592505455) | > loss_disc_real_2: 0.22753727436065674 (0.2355785697698593) | > loss_disc_real_3: 0.2177366465330124 (0.21480668783187867) | > loss_disc_real_4: 0.23725248873233795 (0.2345509946346283) | > loss_disc_real_5: 0.2326245903968811 (0.2384459912776947) | > loss_0: 2.4338979721069336 (2.477280855178833) | > grad_norm_0: tensor(215.9617, device='cuda:0') (tensor(329.8320, device='cuda:0')) | > loss_gen: 2.3567216396331787 (2.2604836940765383) | > loss_kl: 1.6913948059082031 (1.782979393005371) | > loss_feat: 4.514872074127197 (4.18389925956726) | > loss_mel: 16.10665512084961 (16.755414772033692) | > loss_duration: 1.4860305786132812 (1.4938530683517457) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.15567398071289 (26.476630020141602) | > grad_norm_1: tensor(9931.5420, device='cuda:0') (tensor(7497.8823, device='cuda:0')) | > current_lr_0: 0.0001694286676630296 | > current_lr_1: 0.0001694286676630296 | > step_time: 0.4359 (0.437984561920166) | > loader_time: 0.0053 (0.006479215621948242)  --> TIME: 2025-02-25 21:53:25 -- STEP: 55/72 -- GLOBAL_STEP: 1095600 | > loss_disc: 2.567103624343872 (2.567015249078924) | > loss_disc_real_0: 0.08168172091245651 (0.13306447443636985) | > loss_disc_real_1: 0.34933069348335266 (0.24718166576190428) | > loss_disc_real_2: 0.24926650524139404 (0.23248667120933533) | > loss_disc_real_3: 0.31057223677635193 (0.2349985504692251) | > loss_disc_real_4: 0.23975399136543274 (0.24453302161260085) | > loss_disc_real_5: 0.25080111622810364 (0.2456574101339687) | > loss_0: 2.567103624343872 (2.567015249078924) | > grad_norm_0: tensor(140.3611, device='cuda:0') (tensor(311.0049, device='cuda:0')) | > loss_gen: 2.3197438716888428 (2.2777886087244212) | > loss_kl: 2.2318429946899414 (2.0158645694906063) | > loss_feat: 3.4041736125946045 (4.010052628950639) | > loss_mel: 15.929465293884277 (16.761601881547406) | > loss_duration: 1.602142333984375 (1.5232205564325505) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.487369537353516 (26.588528095592153) | > grad_norm_1: tensor(4864.3110, device='cuda:0') (tensor(8364.6699, device='cuda:0')) | > current_lr_0: 0.0001694286676630296 | > current_lr_1: 0.0001694286676630296 | > step_time: 0.4759 (0.4478469241749157) | > loader_time: 0.01 (0.007941540804776277) --> EVAL PERFORMANCE | > avg_loader_time: 0.008695246468127613 (+0.00019524466823523864) | > avg_step_time: 0.45861729769639564 (+0.0031011876925616244) | > avg_loss_disc: 2.6418622446731783 (+0.14601780327273106) | > avg_loss_disc_real_0: 0.1735473071605387 (+0.081840786453284) | > avg_loss_disc_real_1: 0.23121645228124002 (+0.011511532353683268) | > avg_loss_disc_real_2: 0.23966026767878465 (+0.008030216668693108) | > avg_loss_disc_real_3: 0.23304576647113745 (+0.0032308934859826277) | > avg_loss_disc_real_4: 0.24120421833555464 (-0.002678454761773752) | > avg_loss_disc_real_5: 0.24330104110946119 (-0.002131554862143281) | > avg_loss_0: 2.6418622446731783 (+0.14601780327273106) | > avg_grad_norm_0: tensor(286.6980, device='cuda:0') (tensor(-4.3897, device='cuda:0')) | > avg_loss_gen: 2.1958481308440088 (-0.06881577532056227) | > avg_loss_kl: 2.069121426259967 (+0.07710468601172993) | > avg_loss_feat: 3.8516623604465536 (-0.30328187472384016) | > avg_loss_mel: 17.015896488243424 (+0.2939214975061617) | > avg_loss_duration: 1.5471346982767882 (+0.011983935262115608) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.679663053700622 (+0.01091245194556123) | > avg_grad_norm_1: tensor(7168.0557, device='cuda:0') (tensor(-1344.9023, device='cuda:0'))  > EPOCH: 1328/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:53:35)   --> TIME: 2025-02-25 21:53:52 -- STEP: 33/72 -- GLOBAL_STEP: 1095650 | > loss_disc: 2.9998586177825928 (2.6428294181823735) | > loss_disc_real_0: 0.07836826145648956 (0.15152816700212884) | > loss_disc_real_1: 0.23198820650577545 (0.2194797143791661) | > loss_disc_real_2: 0.20960190892219543 (0.24379474421342215) | > loss_disc_real_3: 0.24105218052864075 (0.23940570652484894) | > loss_disc_real_4: 0.24308516085147858 (0.2502589090303941) | > loss_disc_real_5: 0.24642419815063477 (0.2479990774934942) | > loss_0: 2.9998586177825928 (2.6428294181823735) | > grad_norm_0: tensor(69.5924, device='cuda:0') (tensor(187.2363, device='cuda:0')) | > loss_gen: 1.6219723224639893 (2.126423637072245) | > loss_kl: 2.0520904064178467 (2.0680543184280396) | > loss_feat: 3.918386697769165 (3.7902148564656577) | > loss_mel: 17.24799156188965 (17.349103551922425) | > loss_duration: 1.5034816265106201 (1.5158211823665735) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.343921661376953 (26.84961769797585) | > grad_norm_1: tensor(6950.9043, device='cuda:0') (tensor(5058.8936, device='cuda:0')) | > current_lr_0: 0.0001694074890795717 | > current_lr_1: 0.0001694074890795717 | > step_time: 0.4622 (0.43772653377417364) | > loader_time: 0.008 (0.007126107360377457) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629835827249877 (-6.541064087773642e-05) | > avg_step_time: 0.4553785659897495 (-0.003238731706646125) | > avg_loss_disc: 2.6556757134450995 (+0.013813468771921134) | > avg_loss_disc_real_0: 0.15003950474127917 (-0.023507802419259538) | > avg_loss_disc_real_1: 0.22449816238712256 (-0.006718289894117463) | > avg_loss_disc_real_2: 0.24521237842633692 (+0.005552110747552269) | > avg_loss_disc_real_3: 0.23453539274108243 (+0.001489626269944977) | > avg_loss_disc_real_4: 0.23534701302857466 (-0.0058572053069799734) | > avg_loss_disc_real_5: 0.23920701150323304 (-0.00409402960622815) | > avg_loss_0: 2.6556757134450995 (+0.013813468771921134) | > avg_grad_norm_0: tensor(218.5492, device='cuda:0') (tensor(-68.1487, device='cuda:0')) | > avg_loss_gen: 2.0973812761441093 (-0.09846685469989946) | > avg_loss_kl: 2.0957980810756416 (+0.026676654815674716) | > avg_loss_feat: 3.767950397142222 (-0.0837119633043315) | > avg_loss_mel: 17.188928429509552 (+0.1730319412661281) | > avg_loss_duration: 1.5488528231499905 (+0.0017181248732023313) | > avg_amp_scaler: 8.788732394366198 (+0.7887323943661979) | > avg_loss_1: 26.698911183316948 (+0.019248129616325826) | > avg_grad_norm_1: tensor(4782.9868, device='cuda:0') (tensor(-2385.0688, device='cuda:0'))  > EPOCH: 1329/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:54:12)   --> TIME: 2025-02-25 21:54:19 -- STEP: 11/72 -- GLOBAL_STEP: 1095700 | > loss_disc: 2.5242631435394287 (2.5627395022999155) | > loss_disc_real_0: 0.05096923187375069 (0.1469386819411408) | > loss_disc_real_1: 0.1651436686515808 (0.20983963527462698) | > loss_disc_real_2: 0.22896602749824524 (0.23066068237478082) | > loss_disc_real_3: 0.24046313762664795 (0.2365477748892524) | > loss_disc_real_4: 0.2577091157436371 (0.23511927913535724) | > loss_disc_real_5: 0.2585068643093109 (0.22235791114243594) | > loss_0: 2.5242631435394287 (2.5627395022999155) | > grad_norm_0: tensor(54.2751, device='cuda:0') (tensor(234.2026, device='cuda:0')) | > loss_gen: 2.1267473697662354 (2.2701652591878716) | > loss_kl: 1.953559160232544 (1.9067669348283247) | > loss_feat: 3.989166498184204 (4.079127203334462) | > loss_mel: 16.88953971862793 (17.14286084608598) | > loss_duration: 1.4976820945739746 (1.5043071508407593) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.456695556640625 (26.903227892788973) | > grad_norm_1: tensor(2269.1016, device='cuda:0') (tensor(5104.2578, device='cuda:0')) | > current_lr_0: 0.00016938631314343677 | > current_lr_1: 0.00016938631314343677 | > step_time: 0.4292 (0.43319751999594946) | > loader_time: 0.0058 (0.0064189217307350855)  --> TIME: 2025-02-25 21:54:42 -- STEP: 61/72 -- GLOBAL_STEP: 1095750 | > loss_disc: 2.497077465057373 (2.6127135636376555) | > loss_disc_real_0: 0.11941813677549362 (0.1496150471025803) | > loss_disc_real_1: 0.2828504741191864 (0.21961172755624428) | > loss_disc_real_2: 0.24591925740242004 (0.2396516714428292) | > loss_disc_real_3: 0.2344624251127243 (0.23046686365956165) | > loss_disc_real_4: 0.25479087233543396 (0.24151809342571948) | > loss_disc_real_5: 0.24741965532302856 (0.24295408999333618) | > loss_0: 2.497077465057373 (2.6127135636376555) | > grad_norm_0: tensor(19.0102, device='cuda:0') (tensor(261.5830, device='cuda:0')) | > loss_gen: 2.250818967819214 (2.1722229132886794) | > loss_kl: 1.9345245361328125 (1.9430992271079393) | > loss_feat: 4.192031383514404 (3.8204693872420514) | > loss_mel: 15.694451332092285 (16.768327087652494) | > loss_duration: 1.5770869255065918 (1.526082552847315) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.64891242980957 (26.23020112709921) | > grad_norm_1: tensor(2259.7522, device='cuda:0') (tensor(6917.4883, device='cuda:0')) | > current_lr_0: 0.00016938631314343677 | > current_lr_1: 0.00016938631314343677 | > step_time: 0.4908 (0.4485291223056981) | > loader_time: 0.0107 (0.00820772765112705) --> EVAL PERFORMANCE | > avg_loader_time: 0.008698235095386773 (+6.83992681368966e-05) | > avg_step_time: 0.45586133003234863 (+0.0004827640425991153) | > avg_loss_disc: 2.6394657081281636 (-0.016210005316935838) | > avg_loss_disc_real_0: 0.13816228445986628 (-0.011877220281412887) | > avg_loss_disc_real_1: 0.21071953651770736 (-0.013778625869415195) | > avg_loss_disc_real_2: 0.23642578217345225 (-0.008786596252884665) | > avg_loss_disc_real_3: 0.2316657193529774 (-0.0028696733881050374) | > avg_loss_disc_real_4: 0.24157045760624846 (+0.006223444577673792) | > avg_loss_disc_real_5: 0.24395272261659864 (+0.004745711113365608) | > avg_loss_0: 2.6394657081281636 (-0.016210005316935838) | > avg_grad_norm_0: tensor(277.0701, device='cuda:0') (+tensor(58.5209, device='cuda:0')) | > avg_loss_gen: 2.112604864886109 (+0.015223588741999805) | > avg_loss_kl: 1.9786500544615195 (-0.11714802661412205) | > avg_loss_feat: 3.7757605666845615 (+0.0078101695423393735) | > avg_loss_mel: 16.79329099789472 (-0.39563743161483345) | > avg_loss_duration: 1.5361675729214306 (-0.01268525022855993) | > avg_amp_scaler: 8.0 (-0.7887323943661979) | > avg_loss_1: 26.196474021589253 (-0.5024371617276948) | > avg_grad_norm_1: tensor(7295.0244, device='cuda:0') (+tensor(2512.0376, device='cuda:0'))  > EPOCH: 1330/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:54:49)   --> TIME: 2025-02-25 21:55:09 -- STEP: 39/72 -- GLOBAL_STEP: 1095800 | > loss_disc: 2.8717498779296875 (2.684183787076901) | > loss_disc_real_0: 0.25089699029922485 (0.1422351444951999) | > loss_disc_real_1: 0.2167794555425644 (0.24643908326442426) | > loss_disc_real_2: 0.20619381964206696 (0.22391363519888657) | > loss_disc_real_3: 0.2135842889547348 (0.23925189062570915) | > loss_disc_real_4: 0.23613078892230988 (0.23938650771593437) | > loss_disc_real_5: 0.24688293039798737 (0.24552918321047074) | > loss_0: 2.8717498779296875 (2.684183787076901) | > grad_norm_0: tensor(131.7039, device='cuda:0') (tensor(175.2051, device='cuda:0')) | > loss_gen: 1.640688180923462 (2.130589338449331) | > loss_kl: 3.0437512397766113 (2.1720279302352505) | > loss_feat: 3.1590194702148438 (3.9080033607971973) | > loss_mel: 19.744779586791992 (17.548924054854947) | > loss_duration: 1.6268792152404785 (1.5201164453457563) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.215116500854492 (27.27966112968249) | > grad_norm_1: tensor(4924.1401, device='cuda:0') (tensor(5795.0977, device='cuda:0')) | > current_lr_0: 0.00016936513985429382 | > current_lr_1: 0.00016936513985429382 | > step_time: 0.4493 (0.4378973276187212) | > loader_time: 0.0084 (0.007165077405098157) --> EVAL PERFORMANCE | > avg_loader_time: 0.008496103152422836 (-0.00020213194296393733) | > avg_step_time: 0.45460669759293676 (-0.0012546324394118735) | > avg_loss_disc: 2.6005954641691393 (-0.038870243959024275) | > avg_loss_disc_real_0: 0.12666843254381502 (-0.011493851916051262) | > avg_loss_disc_real_1: 0.2352630078372821 (+0.02454347131957474) | > avg_loss_disc_real_2: 0.2281578193667909 (-0.008267962806661366) | > avg_loss_disc_real_3: 0.2316891588795353 (+2.3439526557922363e-05) | > avg_loss_disc_real_4: 0.23899254601606182 (-0.002577911590186638) | > avg_loss_disc_real_5: 0.24313460269444426 (-0.0008181199221543867) | > avg_loss_0: 2.6005954641691393 (-0.038870243959024275) | > avg_grad_norm_0: tensor(188.5991, device='cuda:0') (tensor(-88.4711, device='cuda:0')) | > avg_loss_gen: 2.149120819400733 (+0.03651595451462386) | > avg_loss_kl: 2.2977103481830006 (+0.31906029372148104) | > avg_loss_feat: 3.938258557252481 (+0.16249799056791936) | > avg_loss_mel: 17.861500981827856 (+1.0682099839331372) | > avg_loss_duration: 1.5685401466530813 (+0.0323725737316507) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.81513090536628 (+1.618656883777028) | > avg_grad_norm_1: tensor(6418.1533, device='cuda:0') (tensor(-876.8711, device='cuda:0'))  > EPOCH: 1331/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:55:26)   --> TIME: 2025-02-25 21:55:36 -- STEP: 17/72 -- GLOBAL_STEP: 1095850 | > loss_disc: 2.436352491378784 (2.5014802427852856) | > loss_disc_real_0: 0.07950032502412796 (0.08436616466325872) | > loss_disc_real_1: 0.2454240620136261 (0.21511122584342957) | > loss_disc_real_2: 0.2543189823627472 (0.2272835654370925) | > loss_disc_real_3: 0.24406662583351135 (0.23380609207293568) | > loss_disc_real_4: 0.27140623331069946 (0.24555759745485642) | > loss_disc_real_5: 0.2661941647529602 (0.24002439309568965) | > loss_0: 2.436352491378784 (2.5014802427852856) | > grad_norm_0: tensor(77.2895, device='cuda:0') (tensor(266.5320, device='cuda:0')) | > loss_gen: 2.480713367462158 (2.2134590850156894) | > loss_kl: 2.11970853805542 (1.9908448878456564) | > loss_feat: 4.056488037109375 (4.247008576112636) | > loss_mel: 17.137065887451172 (16.97988796234131) | > loss_duration: 1.5464584827423096 (1.5051177108989042) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.340435028076172 (26.936318117029526) | > grad_norm_1: tensor(6711.7007, device='cuda:0') (tensor(8411.1328, device='cuda:0')) | > current_lr_0: 0.00016934396921181204 | > current_lr_1: 0.00016934396921181204 | > step_time: 0.445 (0.43046144878163056) | > loader_time: 0.0061 (0.006642888574039235)  --> TIME: 2025-02-25 21:56:00 -- STEP: 67/72 -- GLOBAL_STEP: 1095900 | > loss_disc: 2.569031238555908 (2.561162916582022) | > loss_disc_real_0: 0.07521812617778778 (0.10957557852588483) | > loss_disc_real_1: 0.20951679348945618 (0.2231393830100102) | > loss_disc_real_2: 0.22357496619224548 (0.22961417018477595) | > loss_disc_real_3: 0.23779526352882385 (0.2302188917772094) | > loss_disc_real_4: 0.24311234056949615 (0.24236866253525463) | > loss_disc_real_5: 0.23846463859081268 (0.2438590442066762) | > loss_0: 2.569031238555908 (2.561162916582022) | > grad_norm_0: tensor(22.2945, device='cuda:0') (tensor(286.5702, device='cuda:0')) | > loss_gen: 2.451000690460205 (2.1685607486696368) | > loss_kl: 2.3645830154418945 (2.0206645193384656) | > loss_feat: 3.948111057281494 (4.003809459173859) | > loss_mel: 19.31635093688965 (17.064872827102896) | > loss_duration: 1.6621419191360474 (1.541954182866794) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.7421875 (26.799861822555314) | > grad_norm_1: tensor(4871.9551, device='cuda:0') (tensor(8689.8281, device='cuda:0')) | > current_lr_0: 0.00016934396921181204 | > current_lr_1: 0.00016934396921181204 | > step_time: 0.5245 (0.45047230507010844) | > loader_time: 0.0124 (0.008461849013371254) --> EVAL PERFORMANCE | > avg_loader_time: 0.008633888943094603 (+0.00013778579067176729) | > avg_step_time: 0.45390063272395603 (-0.000706064868980727) | > avg_loss_disc: 2.570256444769846 (-0.0303390193992934) | > avg_loss_disc_real_0: 0.11665432260070048 (-0.010014109943114538) | > avg_loss_disc_real_1: 0.22626131982870504 (-0.009001688008577063) | > avg_loss_disc_real_2: 0.23100406246286037 (+0.002846243096069484) | > avg_loss_disc_real_3: 0.2321104541630812 (+0.0004212952835458972) | > avg_loss_disc_real_4: 0.24297182681694837 (+0.003979280800886553) | > avg_loss_disc_real_5: 0.24487726117523623 (+0.0017426584807919698) | > avg_loss_0: 2.570256444769846 (-0.0303390193992934) | > avg_grad_norm_0: tensor(300.5994, device='cuda:0') (+tensor(112.0004, device='cuda:0')) | > avg_loss_gen: 2.181245869314165 (+0.032125049913432235) | > avg_loss_kl: 2.0405361837064717 (-0.25717416447652885) | > avg_loss_feat: 4.002918857923698 (+0.06466030067121675) | > avg_loss_mel: 17.14977721093407 (-0.7117237708937871) | > avg_loss_duration: 1.5465113579387395 (-0.022028788714341774) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.920989614137458 (-0.8941412912288236) | > avg_grad_norm_1: tensor(8659.1992, device='cuda:0') (+tensor(2241.0459, device='cuda:0'))  > EPOCH: 1332/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:56:03)   --> TIME: 2025-02-25 21:56:26 -- STEP: 45/72 -- GLOBAL_STEP: 1095950 | > loss_disc: 2.530463695526123 (2.5683549880981453) | > loss_disc_real_0: 0.05498318374156952 (0.07298711169925001) | > loss_disc_real_1: 0.2211335152387619 (0.23037886752022638) | > loss_disc_real_2: 0.23855817317962646 (0.2427242116795646) | > loss_disc_real_3: 0.24397370219230652 (0.2310472673839993) | > loss_disc_real_4: 0.2614837884902954 (0.2479500048690372) | > loss_disc_real_5: 0.2904047667980194 (0.24871141082710685) | > loss_0: 2.530463695526123 (2.5683549880981453) | > grad_norm_0: tensor(397.7407, device='cuda:0') (tensor(272.2604, device='cuda:0')) | > loss_gen: 1.9756081104278564 (2.132710647583008) | > loss_kl: 1.939397931098938 (1.9967584874894881) | > loss_feat: 3.5054209232330322 (4.076477071974012) | > loss_mel: 16.766042709350586 (16.988624827067056) | > loss_duration: 1.533290147781372 (1.5166904661390517) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.71976089477539 (26.711261367797853) | > grad_norm_1: tensor(11149.9717, device='cuda:0') (tensor(9421.5791, device='cuda:0')) | > current_lr_0: 0.00016932280121566057 | > current_lr_1: 0.00016932280121566057 | > step_time: 0.455 (0.4393633630540636) | > loader_time: 0.0087 (0.007483365800645616) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612746923742158 (-2.114201935244503e-05) | > avg_step_time: 0.4536613578527746 (-0.00023927487118141633) | > avg_loss_disc: 2.5455963007161317 (-0.024660144053714195) | > avg_loss_disc_real_0: 0.07811388050691342 (-0.038540442093787056) | > avg_loss_disc_real_1: 0.22641846243764313 (+0.0001571426089380934) | > avg_loss_disc_real_2: 0.2358838529653952 (+0.004879790502534842) | > avg_loss_disc_real_3: 0.22902011913312992 (-0.003090335029951291) | > avg_loss_disc_real_4: 0.2436816459390479 (+0.0007098191220995254) | > avg_loss_disc_real_5: 0.25371027958225184 (+0.008833018407015614) | > avg_loss_0: 2.5455963007161317 (-0.024660144053714195) | > avg_grad_norm_0: tensor(300.5299, device='cuda:0') (tensor(-0.0695, device='cuda:0')) | > avg_loss_gen: 2.1726381476496304 (-0.008607721664534829) | > avg_loss_kl: 2.002548100243152 (-0.03798808346331972) | > avg_loss_feat: 4.105076276080707 (+0.10215741815700952) | > avg_loss_mel: 16.885515683133836 (-0.26426152780023315) | > avg_loss_duration: 1.5369000434875488 (-0.009611314451190678) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.702678277458944 (-0.21831133667851432) | > avg_grad_norm_1: tensor(9916.4121, device='cuda:0') (+tensor(1257.2129, device='cuda:0'))  > EPOCH: 1333/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:56:41)   --> TIME: 2025-02-25 21:56:53 -- STEP: 23/72 -- GLOBAL_STEP: 1096000 | > loss_disc: 2.476274013519287 (2.504201267076575) | > loss_disc_real_0: 0.09173926711082458 (0.08008197257700174) | > loss_disc_real_1: 0.21958650648593903 (0.22553377047829007) | > loss_disc_real_2: 0.22829404473304749 (0.22694762245468472) | > loss_disc_real_3: 0.25719454884529114 (0.23381372703158337) | > loss_disc_real_4: 0.26131007075309753 (0.23896537267643472) | > loss_disc_real_5: 0.23860013484954834 (0.24547184874182162) | > loss_0: 2.476274013519287 (2.504201267076575) | > grad_norm_0: tensor(331.9868, device='cuda:0') (tensor(332.2634, device='cuda:0')) | > loss_gen: 2.3087711334228516 (2.223852313083151) | > loss_kl: 1.6703609228134155 (1.8446676523789116) | > loss_feat: 4.17218542098999 (4.180113201555998) | > loss_mel: 16.181983947753906 (16.647886400637418) | > loss_duration: 1.5118324756622314 (1.4855027406112007) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.845134735107422 (26.382022111312203) | > grad_norm_1: tensor(10052.3447, device='cuda:0') (tensor(10438.3857, device='cuda:0')) | > current_lr_0: 0.00016930163586550861 | > current_lr_1: 0.00016930163586550861 | > step_time: 0.4399 (0.43670841921930725) | > loader_time: 0.0062 (0.006754066633141559) --> EVAL PERFORMANCE | > avg_loader_time: 0.008668627537472147 (+5.588061372998818e-05) | > avg_step_time: 0.4566116299427731 (+0.0029502720899984847) | > avg_loss_disc: 2.5206283884988703 (-0.024967912217261468) | > avg_loss_disc_real_0: 0.09918815810495699 (+0.02107427759804356) | > avg_loss_disc_real_1: 0.22200894943425353 (-0.004409513003389598) | > avg_loss_disc_real_2: 0.22712822237484892 (-0.008755630590546293) | > avg_loss_disc_real_3: 0.23343989462919637 (+0.004419775496066453) | > avg_loss_disc_real_4: 0.2391457158914754 (-0.004535930047572495) | > avg_loss_disc_real_5: 0.24654314719455342 (-0.007167132387698422) | > avg_loss_0: 2.5206283884988703 (-0.024967912217261468) | > avg_grad_norm_0: tensor(316.7791, device='cuda:0') (+tensor(16.2492, device='cuda:0')) | > avg_loss_gen: 2.2322837819515815 (+0.059645634301951134) | > avg_loss_kl: 1.9386781756307039 (-0.06386992461244811) | > avg_loss_feat: 4.080277076909239 (-0.024799199171468445) | > avg_loss_mel: 16.71917410299811 (-0.1663415801357253) | > avg_loss_duration: 1.5287609268242204 (-0.00813911666332845) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.49917401058573 (-0.20350426687321388) | > avg_grad_norm_1: tensor(9735.8525, device='cuda:0') (tensor(-180.5596, device='cuda:0'))  > EPOCH: 1334/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:57:18)   --> TIME: 2025-02-25 21:57:20 -- STEP: 1/72 -- GLOBAL_STEP: 1096050 | > loss_disc: 2.482950448989868 (2.482950448989868) | > loss_disc_real_0: 0.06290734559297562 (0.06290734559297562) | > loss_disc_real_1: 0.2320381999015808 (0.2320381999015808) | > loss_disc_real_2: 0.2210904359817505 (0.2210904359817505) | > loss_disc_real_3: 0.23714908957481384 (0.23714908957481384) | > loss_disc_real_4: 0.2262231856584549 (0.2262231856584549) | > loss_disc_real_5: 0.2290707230567932 (0.2290707230567932) | > loss_0: 2.482950448989868 (2.482950448989868) | > grad_norm_0: tensor(267.1671, device='cuda:0') (tensor(267.1671, device='cuda:0')) | > loss_gen: 2.2688448429107666 (2.2688448429107666) | > loss_kl: 1.8355032205581665 (1.8355032205581665) | > loss_feat: 4.512404441833496 (4.512404441833496) | > loss_mel: 16.557952880859375 (16.557952880859375) | > loss_duration: 1.4734970331192017 (1.4734970331192017) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.648202896118164 (26.648202896118164) | > grad_norm_1: tensor(7486.0356, device='cuda:0') (tensor(7486.0356, device='cuda:0')) | > current_lr_0: 0.00016928047316102542 | > current_lr_1: 0.00016928047316102542 | > step_time: 0.4466 (0.4465761184692383) | > loader_time: 0.007 (0.006999492645263672)  --> TIME: 2025-02-25 21:57:43 -- STEP: 51/72 -- GLOBAL_STEP: 1096100 | > loss_disc: 2.40451717376709 (2.509803706524418) | > loss_disc_real_0: 0.11019748449325562 (0.09131055880410999) | > loss_disc_real_1: 0.2195972204208374 (0.22182832570636973) | > loss_disc_real_2: 0.23642967641353607 (0.22185835534451054) | > loss_disc_real_3: 0.2441055029630661 (0.23094595325928108) | > loss_disc_real_4: 0.24466560781002045 (0.23800588180037105) | > loss_disc_real_5: 0.2403031289577484 (0.2434088044306811) | > loss_0: 2.40451717376709 (2.509803706524418) | > grad_norm_0: tensor(263.7302, device='cuda:0') (tensor(268.3197, device='cuda:0')) | > loss_gen: 2.397489070892334 (2.247391637633829) | > loss_kl: 1.7448080778121948 (1.8813046263713462) | > loss_feat: 4.030613422393799 (4.176194097481521) | > loss_mel: 15.581356048583984 (16.561970299365473) | > loss_duration: 1.5517754554748535 (1.5062929064619774) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.306041717529297 (26.373153649124443) | > grad_norm_1: tensor(7939.0737, device='cuda:0') (tensor(9718.4951, device='cuda:0')) | > current_lr_0: 0.00016928047316102542 | > current_lr_1: 0.00016928047316102542 | > step_time: 0.4515 (0.44862061855839747) | > loader_time: 0.0093 (0.007980996487187404) --> EVAL PERFORMANCE | > avg_loader_time: 0.008759404571963027 (+9.077703449088086e-05) | > avg_step_time: 0.4570722009094668 (+0.00046057096669371855) | > avg_loss_disc: 2.5142994732923905 (-0.006328915206479824) | > avg_loss_disc_real_0: 0.09695348773204104 (-0.0022346703729159423) | > avg_loss_disc_real_1: 0.2217052633493719 (-0.00030368608488162696) | > avg_loss_disc_real_2: 0.22341550928606113 (-0.003712713088787789) | > avg_loss_disc_real_3: 0.2320248952633898 (-0.0014149993658065796) | > avg_loss_disc_real_4: 0.23798014139625387 (-0.001165574495221533) | > avg_loss_disc_real_5: 0.24365166018546466 (-0.0028914870090887557) | > avg_loss_0: 2.5142994732923905 (-0.006328915206479824) | > avg_grad_norm_0: tensor(274.5417, device='cuda:0') (tensor(-42.2375, device='cuda:0')) | > avg_loss_gen: 2.247923516891372 (+0.01563973493979054) | > avg_loss_kl: 1.9237697544232222 (-0.014908421207481704) | > avg_loss_feat: 4.118668771125901 (+0.038391694216662664) | > avg_loss_mel: 16.593522515095454 (-0.12565158790265585) | > avg_loss_duration: 1.5251613116600145 (-0.003599615164205927) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.409045877590984 (-0.09012813299474587) | > avg_grad_norm_1: tensor(9453.5820, device='cuda:0') (tensor(-282.2705, device='cuda:0'))  > EPOCH: 1335/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:57:55)   --> TIME: 2025-02-25 21:58:10 -- STEP: 29/72 -- GLOBAL_STEP: 1096150 | > loss_disc: 2.548666000366211 (2.540851543689595) | > loss_disc_real_0: 0.0759851410984993 (0.10650723222000845) | > loss_disc_real_1: 0.17813141644001007 (0.21646905973039823) | > loss_disc_real_2: 0.20892742276191711 (0.2354687955872766) | > loss_disc_real_3: 0.23965352773666382 (0.2332434818662446) | > loss_disc_real_4: 0.2601102590560913 (0.2421677359219255) | > loss_disc_real_5: 0.2660655677318573 (0.24676677481881504) | > loss_0: 2.548666000366211 (2.540851543689595) | > grad_norm_0: tensor(147.2507, device='cuda:0') (tensor(269.7722, device='cuda:0')) | > loss_gen: 2.1568520069122314 (2.237160300386363) | > loss_kl: 2.11057186126709 (1.8963352647320977) | > loss_feat: 4.001274108886719 (4.129243250550894) | > loss_mel: 16.841508865356445 (16.645121969025713) | > loss_duration: 1.4940822124481201 (1.4849008609508645) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.60428810119629 (26.392761822404534) | > grad_norm_1: tensor(8687.3955, device='cuda:0') (tensor(7454.6343, device='cuda:0')) | > current_lr_0: 0.00016925931310188028 | > current_lr_1: 0.00016925931310188028 | > step_time: 0.4461 (0.4357670833324564) | > loader_time: 0.0074 (0.0070469790491564524) --> EVAL PERFORMANCE | > avg_loader_time: 0.008731613696460989 (-2.779087550203868e-05) | > avg_step_time: 0.452663928690091 (-0.004408272219375842) | > avg_loss_disc: 2.5933333786440556 (+0.0790339053516651) | > avg_loss_disc_real_0: 0.13859920539486575 (+0.04164571766282471) | > avg_loss_disc_real_1: 0.21781792460193097 (-0.0038873387474409327) | > avg_loss_disc_real_2: 0.2378220344093484 (+0.014406525123287273) | > avg_loss_disc_real_3: 0.232495043898972 (+0.00047014863558220155) | > avg_loss_disc_real_4: 0.23727778838553898 (-0.0007023530107148901) | > avg_loss_disc_real_5: 0.24391040142992854 (+0.0002587412444638726) | > avg_loss_0: 2.5933333786440556 (+0.0790339053516651) | > avg_grad_norm_0: tensor(269.1820, device='cuda:0') (tensor(-5.3596, device='cuda:0')) | > avg_loss_gen: 2.207425411318389 (-0.04049810557298317) | > avg_loss_kl: 2.015578115490121 (+0.09180836106689894) | > avg_loss_feat: 3.9499714576022726 (-0.1686973135236287) | > avg_loss_mel: 16.629617597015805 (+0.036095081920350935) | > avg_loss_duration: 1.5293139538294833 (+0.0041526421694688676) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.331906600737234 (-0.07713927685374955) | > avg_grad_norm_1: tensor(6993.2188, device='cuda:0') (tensor(-2460.3633, device='cuda:0'))  > EPOCH: 1336/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:58:32)   --> TIME: 2025-02-25 21:58:37 -- STEP: 7/72 -- GLOBAL_STEP: 1096200 | > loss_disc: 2.6566646099090576 (2.5963960715702603) | > loss_disc_real_0: 0.20703616738319397 (0.1292606613465718) | > loss_disc_real_1: 0.20054881274700165 (0.21402645536831447) | > loss_disc_real_2: 0.2358953207731247 (0.23288453476769583) | > loss_disc_real_3: 0.21363146603107452 (0.23875766779695237) | > loss_disc_real_4: 0.22548015415668488 (0.24007551584924972) | > loss_disc_real_5: 0.2279847115278244 (0.24112147518566676) | > loss_0: 2.6566646099090576 (2.5963960715702603) | > grad_norm_0: tensor(118.9804, device='cuda:0') (tensor(195.2524, device='cuda:0')) | > loss_gen: 2.0589218139648438 (2.104614462171282) | > loss_kl: 1.8523379564285278 (1.794942089489528) | > loss_feat: 3.759019136428833 (3.7725649220602855) | > loss_mel: 17.223546981811523 (16.927080154418945) | > loss_duration: 1.466827630996704 (1.4870336055755615) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.360652923583984 (26.08623504638672) | > grad_norm_1: tensor(7228.1704, device='cuda:0') (tensor(8455.8652, device='cuda:0')) | > current_lr_0: 0.00016923815568774253 | > current_lr_1: 0.00016923815568774253 | > step_time: 0.4398 (0.4240086759839739) | > loader_time: 0.0077 (0.0067326681954520086)  --> TIME: 2025-02-25 21:58:59 -- STEP: 57/72 -- GLOBAL_STEP: 1096250 | > loss_disc: 2.903759002685547 (2.649776182676616) | > loss_disc_real_0: 0.48700618743896484 (0.17124095127770775) | > loss_disc_real_1: 0.1786627322435379 (0.22167509349814632) | > loss_disc_real_2: 0.21140284836292267 (0.22785003666292158) | > loss_disc_real_3: 0.22620227932929993 (0.23296411727604113) | > loss_disc_real_4: 0.25186997652053833 (0.24674744276623978) | > loss_disc_real_5: 0.2304317057132721 (0.24047661272057316) | > loss_0: 2.903759002685547 (2.649776182676616) | > grad_norm_0: tensor(584.4301, device='cuda:0') (tensor(354.7452, device='cuda:0')) | > loss_gen: 2.268219232559204 (2.161840850846809) | > loss_kl: 2.2227537631988525 (2.017459336080049) | > loss_feat: 4.070662498474121 (3.9247171000430456) | > loss_mel: 16.78656578063965 (16.779067273725538) | > loss_duration: 1.5863187313079834 (1.5224505495606808) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.934520721435547 (26.405534978498494) | > grad_norm_1: tensor(2390.7710, device='cuda:0') (tensor(7520.8774, device='cuda:0')) | > current_lr_0: 0.00016923815568774253 | > current_lr_1: 0.00016923815568774253 | > step_time: 0.4619 (0.4400280274842915) | > loader_time: 0.0097 (0.008097017020509953) --> EVAL PERFORMANCE | > avg_loader_time: 0.008724571953357105 (-7.04174310388328e-06) | > avg_step_time: 0.44994368687481945 (-0.002720241815271529) | > avg_loss_disc: 2.656405153408856 (+0.06307177476480064) | > avg_loss_disc_real_0: 0.1709018240512257 (+0.032302618656359944) | > avg_loss_disc_real_1: 0.2193899819968452 (+0.001572057394914228) | > avg_loss_disc_real_2: 0.2285671309686043 (-0.009254903440744117) | > avg_loss_disc_real_3: 0.23227854261935596 (-0.0002165012796160326) | > avg_loss_disc_real_4: 0.2461434741255263 (+0.008865685739987333) | > avg_loss_disc_real_5: 0.23864514076373947 (-0.00526526066618907) | > avg_loss_0: 2.656405153408856 (+0.06307177476480064) | > avg_grad_norm_0: tensor(367.6704, device='cuda:0') (+tensor(98.4884, device='cuda:0')) | > avg_loss_gen: 2.138750420489781 (-0.06867499082860773) | > avg_loss_kl: 2.054439408678403 (+0.03886129318828191) | > avg_loss_feat: 3.8852610285852998 (-0.06471042901697288) | > avg_loss_mel: 16.808043291870977 (+0.17842569485517146) | > avg_loss_duration: 1.5362296557762254 (+0.00691570194674207) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.422723716413472 (+0.09081711567623785) | > avg_grad_norm_1: tensor(7730.2524, device='cuda:0') (+tensor(737.0337, device='cuda:0'))  > EPOCH: 1337/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:59:09)   --> TIME: 2025-02-25 21:59:26 -- STEP: 35/72 -- GLOBAL_STEP: 1096300 | > loss_disc: 2.473888635635376 (2.5566452298845563) | > loss_disc_real_0: 0.15578973293304443 (0.07990840407354491) | > loss_disc_real_1: 0.2542054057121277 (0.2427686814750944) | > loss_disc_real_2: 0.2137020379304886 (0.22451236248016357) | > loss_disc_real_3: 0.22691114246845245 (0.23312775109495434) | > loss_disc_real_4: 0.24092364311218262 (0.23607521738324846) | > loss_disc_real_5: 0.2500170171260834 (0.2514031716755459) | > loss_0: 2.473888635635376 (2.5566452298845563) | > grad_norm_0: tensor(372.4111, device='cuda:0') (tensor(346.0491, device='cuda:0')) | > loss_gen: 2.2431845664978027 (2.193355584144592) | > loss_kl: 2.230691432952881 (2.026436138153077) | > loss_feat: 3.2839040756225586 (4.133917917524064) | > loss_mel: 16.56462860107422 (17.30862143380301) | > loss_duration: 1.5399445295333862 (1.4952179193496704) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.86235237121582 (27.157548904418945) | > grad_norm_1: tensor(12260.6006, device='cuda:0') (tensor(11872.1807, device='cuda:0')) | > current_lr_0: 0.00016921700091828155 | > current_lr_1: 0.00016921700091828155 | > step_time: 0.4415 (0.437337453024728) | > loader_time: 0.0085 (0.007231133324759347) --> EVAL PERFORMANCE | > avg_loader_time: 0.00871451136092065 (-1.0060592436455032e-05) | > avg_step_time: 0.4555165566189188 (+0.0055728697440993735) | > avg_loss_disc: 2.5365415727588494 (-0.1198635806500068) | > avg_loss_disc_real_0: 0.08474530095995313 (-0.08615652309127257) | > avg_loss_disc_real_1: 0.2323936165218622 (+0.013003634525016988) | > avg_loss_disc_real_2: 0.224664912257396 (-0.0039022187112082996) | > avg_loss_disc_real_3: 0.23186016187701428 (-0.0004183807423416841) | > avg_loss_disc_real_4: 0.23617453201555869 (-0.009968942109967627) | > avg_loss_disc_real_5: 0.2481241291257698 (+0.009478988362030344) | > avg_loss_0: 2.5365415727588494 (-0.1198635806500068) | > avg_grad_norm_0: tensor(325.9722, device='cuda:0') (tensor(-41.6982, device='cuda:0')) | > avg_loss_gen: 2.190056134277666 (+0.051305713787884955) | > avg_loss_kl: 2.032839274742234 (-0.021600133936169108) | > avg_loss_feat: 4.05184651092744 (+0.16658548234214043) | > avg_loss_mel: 16.835482798831574 (+0.02743950696059727) | > avg_loss_duration: 1.5304469105223533 (-0.005782745253872124) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.640671582289144 (+0.21794786587567216) | > avg_grad_norm_1: tensor(11118.5938, device='cuda:0') (+tensor(3388.3413, device='cuda:0'))  > EPOCH: 1338/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 21:59:46)   --> TIME: 2025-02-25 21:59:53 -- STEP: 13/72 -- GLOBAL_STEP: 1096350 | > loss_disc: 2.514940023422241 (2.5006845547602725) | > loss_disc_real_0: 0.06026289612054825 (0.08401295514060901) | > loss_disc_real_1: 0.20435070991516113 (0.21144984089411223) | > loss_disc_real_2: 0.2115176022052765 (0.22075948348412147) | > loss_disc_real_3: 0.22724805772304535 (0.22435000997323257) | > loss_disc_real_4: 0.23187515139579773 (0.2304981259199289) | > loss_disc_real_5: 0.24291887879371643 (0.23787160217761993) | > loss_0: 2.514940023422241 (2.5006845547602725) | > grad_norm_0: tensor(176.8788, device='cuda:0') (tensor(301.0117, device='cuda:0')) | > loss_gen: 2.2323646545410156 (2.1949272889357343) | > loss_kl: 1.7718863487243652 (1.8598858576554518) | > loss_feat: 4.013333797454834 (4.15791812309852) | > loss_mel: 15.742473602294922 (16.348607283372147) | > loss_duration: 1.43072509765625 (1.4754160734323354) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.190784454345703 (26.036755048311672) | > grad_norm_1: tensor(9458.7803, device='cuda:0') (tensor(10731.4521, device='cuda:0')) | > current_lr_0: 0.00016919584879316677 | > current_lr_1: 0.00016919584879316677 | > step_time: 0.4441 (0.43176896755511945) | > loader_time: 0.0077 (0.006701946258544922)  --> TIME: 2025-02-25 22:00:17 -- STEP: 63/72 -- GLOBAL_STEP: 1096400 | > loss_disc: 2.573988914489746 (2.580955289659046) | > loss_disc_real_0: 0.08585280179977417 (0.10423975390574289) | > loss_disc_real_1: 0.206955224275589 (0.22689422799481285) | > loss_disc_real_2: 0.178084135055542 (0.21894182524983846) | > loss_disc_real_3: 0.21097756922245026 (0.23033658734389714) | > loss_disc_real_4: 0.23730695247650146 (0.23408097123342847) | > loss_disc_real_5: 0.25000816583633423 (0.25370496747985716) | > loss_0: 2.573988914489746 (2.580955289659046) | > grad_norm_0: tensor(158.9947, device='cuda:0') (tensor(303.4581, device='cuda:0')) | > loss_gen: 2.161365509033203 (2.1884619224639175) | > loss_kl: 2.213996171951294 (1.956607457191225) | > loss_feat: 4.331835746765137 (4.067597483831738) | > loss_mel: 17.1258544921875 (16.405908811660037) | > loss_duration: 1.5573805570602417 (1.5179335238441591) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.390432357788086 (26.13650924440414) | > grad_norm_1: tensor(4849.2427, device='cuda:0') (tensor(9328.9414, device='cuda:0')) | > current_lr_0: 0.00016919584879316677 | > current_lr_1: 0.00016919584879316677 | > step_time: 0.5037 (0.448063229757642) | > loader_time: 0.0114 (0.008261453537713914) --> EVAL PERFORMANCE | > avg_loader_time: 0.008633919165168008 (-8.059219575264238e-05) | > avg_step_time: 0.4550366065871548 (-0.0004799500317640426) | > avg_loss_disc: 2.5782229430238965 (+0.041681370265047146) | > avg_loss_disc_real_0: 0.10409828842106 (+0.019352987461106877) | > avg_loss_disc_real_1: 0.22592500964520681 (-0.006468606876655375) | > avg_loss_disc_real_2: 0.22184197709593975 (-0.002822935161456236) | > avg_loss_disc_real_3: 0.23146394797613923 (-0.0003962139008750476) | > avg_loss_disc_real_4: 0.23531503996378939 (-0.0008594920517693005) | > avg_loss_disc_real_5: 0.25266879398218345 (+0.004544664856413638) | > avg_loss_0: 2.5782229430238965 (+0.041681370265047146) | > avg_grad_norm_0: tensor(303.9811, device='cuda:0') (tensor(-21.9911, device='cuda:0')) | > avg_loss_gen: 2.196754561343664 (+0.00669842706599777) | > avg_loss_kl: 1.9718826438339663 (-0.06095663090826764) | > avg_loss_feat: 4.07151828685277 (+0.019671775925329982) | > avg_loss_mel: 16.449735251950543 (-0.3857475468810314) | > avg_loss_duration: 1.5268090476452467 (-0.0036378628771065458) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.216699841996313 (-0.42397174029283136) | > avg_grad_norm_1: tensor(9358.8740, device='cuda:0') (tensor(-1759.7197, device='cuda:0'))  > EPOCH: 1339/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:00:23)   --> TIME: 2025-02-25 22:00:43 -- STEP: 41/72 -- GLOBAL_STEP: 1096450 | > loss_disc: 2.38179349899292 (2.5330820897730377) | > loss_disc_real_0: 0.08820813149213791 (0.09666391298538302) | > loss_disc_real_1: 0.23019728064537048 (0.22664215034101068) | > loss_disc_real_2: 0.23393265902996063 (0.22556488070546127) | > loss_disc_real_3: 0.23587004840373993 (0.23008111191958916) | > loss_disc_real_4: 0.2364044487476349 (0.23668632202032136) | > loss_disc_real_5: 0.24243485927581787 (0.24183310786398446) | > loss_0: 2.38179349899292 (2.5330820897730377) | > grad_norm_0: tensor(382.8094, device='cuda:0') (tensor(302.1475, device='cuda:0')) | > loss_gen: 2.3018174171447754 (2.2141407902647807) | > loss_kl: 1.989409327507019 (2.0033864916824715) | > loss_feat: 3.8567562103271484 (4.059111531187848) | > loss_mel: 17.13836669921875 (16.526218251484195) | > loss_duration: 1.5178868770599365 (1.5011560626146268) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.804235458374023 (26.304013089435855) | > grad_norm_1: tensor(10521.6875, device='cuda:0') (tensor(8171.9082, device='cuda:0')) | > current_lr_0: 0.0001691746993120676 | > current_lr_1: 0.0001691746993120676 | > step_time: 0.4534 (0.441066951286502) | > loader_time: 0.0094 (0.007533137391253215) --> EVAL PERFORMANCE | > avg_loader_time: 0.008726344981663664 (+9.242581649565595e-05) | > avg_step_time: 0.4562907789794492 (+0.001254172392294417) | > avg_loss_disc: 2.5309090950119657 (-0.04731384801193084) | > avg_loss_disc_real_0: 0.10120288209176401 (-0.0028954063292959964) | > avg_loss_disc_real_1: 0.2235271005143582 (-0.0023979091308486133) | > avg_loss_disc_real_2: 0.22816139900348556 (+0.006319421907545808) | > avg_loss_disc_real_3: 0.2321791648864746 (+0.0007152169103353812) | > avg_loss_disc_real_4: 0.2379604202341026 (+0.002645380270313219) | > avg_loss_disc_real_5: 0.2428092901975336 (-0.009859503784649837) | > avg_loss_0: 2.5309090950119657 (-0.04731384801193084) | > avg_grad_norm_0: tensor(295.2088, device='cuda:0') (tensor(-8.7723, device='cuda:0')) | > avg_loss_gen: 2.2208835222351726 (+0.024128960891508733) | > avg_loss_kl: 2.048127768744886 (+0.07624512491091973) | > avg_loss_feat: 4.028366820912967 (-0.043151465939803124) | > avg_loss_mel: 16.425805548547014 (-0.02392970340352818) | > avg_loss_duration: 1.5300203041291571 (+0.0032112564839104074) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.25320396960621 (+0.03650412760989852) | > avg_grad_norm_1: tensor(8025.7920, device='cuda:0') (tensor(-1333.0820, device='cuda:0'))  > EPOCH: 1340/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:01:00)   --> TIME: 2025-02-25 22:01:11 -- STEP: 19/72 -- GLOBAL_STEP: 1096500 | > loss_disc: 2.7493042945861816 (2.6086756806624565) | > loss_disc_real_0: 0.18184003233909607 (0.11190344942243476) | > loss_disc_real_1: 0.2789749503135681 (0.25890353792592097) | > loss_disc_real_2: 0.26930779218673706 (0.23826350271701813) | > loss_disc_real_3: 0.2559780180454254 (0.22942805447076497) | > loss_disc_real_4: 0.24522671103477478 (0.23926715082243868) | > loss_disc_real_5: 0.24559107422828674 (0.2369198163873271) | > loss_0: 2.7493042945861816 (2.6086756806624565) | > grad_norm_0: tensor(490.0680, device='cuda:0') (tensor(269.5887, device='cuda:0')) | > loss_gen: 2.138244152069092 (2.2781084524957755) | > loss_kl: 2.0380802154541016 (1.9858946423781545) | > loss_feat: 3.546070098876953 (4.109384837903474) | > loss_mel: 16.814186096191406 (16.916331742939196) | > loss_duration: 1.4777195453643799 (1.4885131308906956) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.014301300048828 (26.778232775236432) | > grad_norm_1: tensor(2220.8474, device='cuda:0') (tensor(6040.2583, device='cuda:0')) | > current_lr_0: 0.0001691535524746536 | > current_lr_1: 0.0001691535524746536 | > step_time: 0.4305 (0.46779429285149826) | > loader_time: 0.0076 (0.0068419607062088815)  --> TIME: 2025-02-25 22:01:35 -- STEP: 69/72 -- GLOBAL_STEP: 1096550 | > loss_disc: 2.995966672897339 (2.678072970846425) | > loss_disc_real_0: 0.4994080662727356 (0.16755331313048588) | > loss_disc_real_1: 0.26911017298698425 (0.23448357547538867) | > loss_disc_real_2: 0.2554109990596771 (0.23725704805574555) | > loss_disc_real_3: 0.2621537446975708 (0.23708067143308945) | > loss_disc_real_4: 0.23814092576503754 (0.24478038691956064) | > loss_disc_real_5: 0.2558315396308899 (0.24409494905368143) | > loss_0: 2.995966672897339 (2.678072970846425) | > grad_norm_0: tensor(393.9202, device='cuda:0') (tensor(304.0647, device='cuda:0')) | > loss_gen: 2.3328499794006348 (2.1667781238970547) | > loss_kl: 2.253542184829712 (2.191988863806793) | > loss_feat: 3.483597755432129 (3.8488525100376294) | > loss_mel: 18.137889862060547 (17.20078243034474) | > loss_duration: 1.6578367948532104 (1.5444758612176646) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.8657169342041 (26.952877763388813) | > grad_norm_1: tensor(804.1670, device='cuda:0') (tensor(6776.9526, device='cuda:0')) | > current_lr_0: 0.0001691535524746536 | > current_lr_1: 0.0001691535524746536 | > step_time: 0.5095 (0.4641573152680328) | > loader_time: 0.0125 (0.00874238083328026) --> EVAL PERFORMANCE | > avg_loader_time: 0.008791856362786092 (+6.551138112242776e-05) | > avg_step_time: 0.4646275882989588 (+0.008336809319509586) | > avg_loss_disc: 2.6796949850001806 (+0.14878588998821485) | > avg_loss_disc_real_0: 0.17047372644006373 (+0.06927084434829972) | > avg_loss_disc_real_1: 0.2351669813965408 (+0.011639880882182596) | > avg_loss_disc_real_2: 0.23665275456200183 (+0.008491355558516267) | > avg_loss_disc_real_3: 0.23745846202675724 (+0.005279297140282635) | > avg_loss_disc_real_4: 0.24397262427168834 (+0.006012204037585733) | > avg_loss_disc_real_5: 0.244116636229233 (+0.0013073460316993801) | > avg_loss_0: 2.6796949850001806 (+0.14878588998821485) | > avg_grad_norm_0: tensor(306.3301, device='cuda:0') (+tensor(11.1213, device='cuda:0')) | > avg_loss_gen: 2.1666064346340335 (-0.05427708760113914) | > avg_loss_kl: 2.1938764226268708 (+0.14574865388198477) | > avg_loss_feat: 3.84035951654676 (-0.18800730436620716) | > avg_loss_mel: 17.171594163061872 (+0.7457886145148578) | > avg_loss_duration: 1.5451830188992997 (+0.015162714770142527) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.917619544015803 (+0.6644155744095919) | > avg_grad_norm_1: tensor(6606.9302, device='cuda:0') (tensor(-1418.8618, device='cuda:0'))  > EPOCH: 1341/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:01:37)   --> TIME: 2025-02-25 22:02:01 -- STEP: 47/72 -- GLOBAL_STEP: 1096600 | > loss_disc: 2.734452962875366 (2.5525811631628814) | > loss_disc_real_0: 0.2451622486114502 (0.11595114566227223) | > loss_disc_real_1: 0.22483612596988678 (0.22785060202821772) | > loss_disc_real_2: 0.1856803596019745 (0.2402980387844938) | > loss_disc_real_3: 0.23255223035812378 (0.23121530229740955) | > loss_disc_real_4: 0.2600243389606476 (0.24952779868815808) | > loss_disc_real_5: 0.24313759803771973 (0.25217652289157216) | > loss_0: 2.734452962875366 (2.5525811631628814) | > grad_norm_0: tensor(483.3560, device='cuda:0') (tensor(265.6910, device='cuda:0')) | > loss_gen: 2.3261375427246094 (2.258686255901417) | > loss_kl: 2.1598362922668457 (2.1245103313567792) | > loss_feat: 4.422753810882568 (4.137543891338591) | > loss_mel: 16.77393341064453 (17.205086079049615) | > loss_duration: 1.562402367591858 (1.5208005499332509) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.24506378173828 (27.24662728005267) | > grad_norm_1: tensor(2134.0193, device='cuda:0') (tensor(7662.3477, device='cuda:0')) | > current_lr_0: 0.00016913240828059426 | > current_lr_1: 0.00016913240828059426 | > step_time: 0.4579 (0.445155235047036) | > loader_time: 0.0088 (0.007709792319764482) --> EVAL PERFORMANCE | > avg_loader_time: 0.008671525498511086 (-0.00012033086427500582) | > avg_step_time: 0.45867159333027585 (-0.005955994968682932) | > avg_loss_disc: 2.617439636042421 (-0.062255348957759526) | > avg_loss_disc_real_0: 0.13713599159054354 (-0.033337734849520184) | > avg_loss_disc_real_1: 0.22479930436107473 (-0.010367677035466066) | > avg_loss_disc_real_2: 0.2294633751603919 (-0.0071893794016099355) | > avg_loss_disc_real_3: 0.24257917240471907 (+0.005120710377961829) | > avg_loss_disc_real_4: 0.2530669565771667 (+0.009094332305478375) | > avg_loss_disc_real_5: 0.24895490013377758 (+0.0048382639045445874) | > avg_loss_0: 2.617439636042421 (-0.062255348957759526) | > avg_grad_norm_0: tensor(261.5183, device='cuda:0') (tensor(-44.8118, device='cuda:0')) | > avg_loss_gen: 2.2438493412984926 (+0.0772429066644591) | > avg_loss_kl: 2.1429120386150524 (-0.05096438401181835) | > avg_loss_feat: 4.021171103061085 (+0.18081158651432494) | > avg_loss_mel: 17.24002458008242 (+0.0684304170205472) | > avg_loss_duration: 1.5435132812446273 (-0.0016697376546723763) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.19147050884408 (+0.2738509648282772) | > avg_grad_norm_1: tensor(6945.7554, device='cuda:0') (+tensor(338.8252, device='cuda:0'))  > EPOCH: 1342/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:02:15)   --> TIME: 2025-02-25 22:02:28 -- STEP: 25/72 -- GLOBAL_STEP: 1096650 | > loss_disc: 2.2725062370300293 (2.6016636276245118) | > loss_disc_real_0: 0.06223602592945099 (0.11201582819223405) | > loss_disc_real_1: 0.19444698095321655 (0.2399652600288391) | > loss_disc_real_2: 0.21505582332611084 (0.2220623028278351) | > loss_disc_real_3: 0.23766043782234192 (0.23506830453872682) | > loss_disc_real_4: 0.2439948469400406 (0.23738675832748413) | > loss_disc_real_5: 0.2293429970741272 (0.2441269063949585) | > loss_0: 2.2725062370300293 (2.6016636276245118) | > grad_norm_0: tensor(91.7455, device='cuda:0') (tensor(284.9911, device='cuda:0')) | > loss_gen: 2.8075764179229736 (2.179920468330384) | > loss_kl: 1.9264047145843506 (1.935608196258545) | > loss_feat: 4.503800392150879 (3.9761821842193603) | > loss_mel: 16.572187423706055 (16.927021560668955) | > loss_duration: 1.4695050716400146 (1.492862458229065) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.27947425842285 (26.51159507751465) | > grad_norm_1: tensor(8667.6943, device='cuda:0') (tensor(6823.6196, device='cuda:0')) | > current_lr_0: 0.00016911126672955918 | > current_lr_1: 0.00016911126672955918 | > step_time: 0.4467 (0.43689417839050293) | > loader_time: 0.0093 (0.007147645950317383) --> EVAL PERFORMANCE | > avg_loader_time: 0.008752429989022266 (+8.090449051117998e-05) | > avg_step_time: 0.4581765188297755 (-0.0004950745005003476) | > avg_loss_disc: 2.594797940321371 (-0.022641695721050237) | > avg_loss_disc_real_0: 0.12015332318317722 (-0.016982668407366328) | > avg_loss_disc_real_1: 0.22837585924376905 (+0.003576554882694316) | > avg_loss_disc_real_2: 0.23891725611518805 (+0.009453880954796157) | > avg_loss_disc_real_3: 0.23082279550357604 (-0.011756376901143034) | > avg_loss_disc_real_4: 0.2410446520842297 (-0.012022304492937008) | > avg_loss_disc_real_5: 0.24877446344200993 (-0.0001804366917676492) | > avg_loss_0: 2.594797940321371 (-0.022641695721050237) | > avg_grad_norm_0: tensor(244.5175, device='cuda:0') (tensor(-17.0009, device='cuda:0')) | > avg_loss_gen: 2.1719995082264205 (-0.07184983307207204) | > avg_loss_kl: 1.991197020235196 (-0.15171501837985635) | > avg_loss_feat: 3.869572122332076 (-0.15159898072900901) | > avg_loss_mel: 16.77689999593816 (-0.4631245841442606) | > avg_loss_duration: 1.5328117525073843 (-0.010701528737242949) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.342480646052824 (-0.8489898627912567) | > avg_grad_norm_1: tensor(6660.2466, device='cuda:0') (tensor(-285.5088, device='cuda:0'))  > EPOCH: 1343/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:02:52)   --> TIME: 2025-02-25 22:02:55 -- STEP: 3/72 -- GLOBAL_STEP: 1096700 | > loss_disc: 2.758523464202881 (2.7583211263020835) | > loss_disc_real_0: 0.11875907331705093 (0.12726027518510818) | > loss_disc_real_1: 0.2103305459022522 (0.25028567512830097) | > loss_disc_real_2: 0.23505555093288422 (0.2298335681358973) | > loss_disc_real_3: 0.2717629075050354 (0.26081567505995434) | > loss_disc_real_4: 0.21957996487617493 (0.2275442580382029) | > loss_disc_real_5: 0.2252485305070877 (0.24574294686317444) | > loss_0: 2.758523464202881 (2.7583211263020835) | > grad_norm_0: tensor(82.2365, device='cuda:0') (tensor(184.0859, device='cuda:0')) | > loss_gen: 1.996692180633545 (1.9372652769088745) | > loss_kl: 2.0926742553710938 (2.1396713256835938) | > loss_feat: 3.849484920501709 (3.607306718826294) | > loss_mel: 18.287700653076172 (18.060740788777668) | > loss_duration: 1.531890630722046 (1.516558329264323) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.75844383239746 (27.26154327392578) | > grad_norm_1: tensor(11243.3643, device='cuda:0') (tensor(11371.7998, device='cuda:0')) | > current_lr_0: 0.00016909012782121796 | > current_lr_1: 0.00016909012782121796 | > step_time: 0.4348 (0.4369163513183594) | > loader_time: 0.0079 (0.006854454676310222)  --> TIME: 2025-02-25 22:03:18 -- STEP: 53/72 -- GLOBAL_STEP: 1096750 | > loss_disc: 2.8622243404388428 (2.66849635682016) | > loss_disc_real_0: 0.29657959938049316 (0.15061348304152483) | > loss_disc_real_1: 0.42971497774124146 (0.2149892954753255) | > loss_disc_real_2: 0.268215537071228 (0.2254439704823044) | > loss_disc_real_3: 0.26276230812072754 (0.24603895280721053) | > loss_disc_real_4: 0.26180019974708557 (0.24866814995711706) | > loss_disc_real_5: 0.24169905483722687 (0.2510383601458567) | > loss_0: 2.8622243404388428 (2.66849635682016) | > grad_norm_0: tensor(352.7887, device='cuda:0') (tensor(225.1727, device='cuda:0')) | > loss_gen: 2.6861300468444824 (2.1633290052413945) | > loss_kl: 2.3181028366088867 (2.0803057315214626) | > loss_feat: 3.6761837005615234 (3.9381390202720206) | > loss_mel: 17.541173934936523 (17.362723350524906) | > loss_duration: 1.6085186004638672 (1.5299689814729511) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.830110549926758 (27.074466129518903) | > grad_norm_1: tensor(2322.9226, device='cuda:0') (tensor(5929.6982, device='cuda:0')) | > current_lr_0: 0.00016909012782121796 | > current_lr_1: 0.00016909012782121796 | > step_time: 0.4544 (0.45244726145042563) | > loader_time: 0.0094 (0.007986905439844674) --> EVAL PERFORMANCE | > avg_loader_time: 0.008800248025168838 (+4.7818036146571666e-05) | > avg_step_time: 0.4646076115084366 (+0.006431092678661099) | > avg_loss_disc: 2.6784183475333196 (+0.08362040721194886) | > avg_loss_disc_real_0: 0.16300654915017138 (+0.04285322596699416) | > avg_loss_disc_real_1: 0.23198028785032285 (+0.0036044286065538) | > avg_loss_disc_real_2: 0.2313313989991873 (-0.007585857116000738) | > avg_loss_disc_real_3: 0.2428885620664543 (+0.01206576656287825) | > avg_loss_disc_real_4: 0.25346787043020763 (+0.012423218345977927) | > avg_loss_disc_real_5: 0.2520259587277829 (+0.003251495285772954) | > avg_loss_0: 2.6784183475333196 (+0.08362040721194886) | > avg_grad_norm_0: tensor(241.8289, device='cuda:0') (tensor(-2.6885, device='cuda:0')) | > avg_loss_gen: 2.189018637361661 (+0.017019129135240618) | > avg_loss_kl: 2.1472863798410122 (+0.15608935960581616) | > avg_loss_feat: 3.8609503390083852 (-0.008621783323690568) | > avg_loss_mel: 17.389651325387014 (+0.6127513294488551) | > avg_loss_duration: 1.5470547860776875 (+0.01424303357030321) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.13396152979891 (+0.791480883746086) | > avg_grad_norm_1: tensor(5901.7554, device='cuda:0') (tensor(-758.4912, device='cuda:0'))  > EPOCH: 1344/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:03:29)   --> TIME: 2025-02-25 22:03:46 -- STEP: 31/72 -- GLOBAL_STEP: 1096800 | > loss_disc: 2.4300127029418945 (2.7462117979603424) | > loss_disc_real_0: 0.09672823548316956 (0.2169787999122374) | > loss_disc_real_1: 0.25030210614204407 (0.21591275449722044) | > loss_disc_real_2: 0.22610457241535187 (0.2509042740829529) | > loss_disc_real_3: 0.2274836152791977 (0.23738565271900547) | > loss_disc_real_4: 0.25697556138038635 (0.258553217976324) | > loss_disc_real_5: 0.26950734853744507 (0.251178391998814) | > loss_0: 2.4300127029418945 (2.7462117979603424) | > grad_norm_0: tensor(150.6527, device='cuda:0') (tensor(328.4087, device='cuda:0')) | > loss_gen: 2.4923911094665527 (2.10155164810919) | > loss_kl: 2.2459518909454346 (2.325911741102895) | > loss_feat: 4.405221939086914 (3.764266367881529) | > loss_mel: 19.18950653076172 (20.126784416937056) | > loss_duration: 1.6727248430252075 (1.7099956350941812) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.005796432495117 (30.02850963223365) | > grad_norm_1: tensor(970.5770, device='cuda:0') (tensor(6781.9604, device='cuda:0')) | > current_lr_0: 0.00016906899155524031 | > current_lr_1: 0.00016906899155524031 | > step_time: 0.4537 (0.46148681640625) | > loader_time: 0.0086 (0.007071918056857201) --> EVAL PERFORMANCE | > avg_loader_time: 0.008672015767701914 (-0.00012823225746692347) | > avg_step_time: 0.4685186433120513 (+0.003911031803614684) | > avg_loss_disc: 2.7015669312275623 (+0.023148583694242664) | > avg_loss_disc_real_0: 0.18004738114459418 (+0.0170408319944228) | > avg_loss_disc_real_1: 0.23012604155171085 (-0.001854246298612) | > avg_loss_disc_real_2: 0.23881474452119478 (+0.007483345522007467) | > avg_loss_disc_real_3: 0.23567957420584182 (-0.00720898786061247) | > avg_loss_disc_real_4: 0.2602037844103826 (+0.006735913980174979) | > avg_loss_disc_real_5: 0.25425726028395373 (+0.0022313015561708505) | > avg_loss_0: 2.7015669312275623 (+0.023148583694242664) | > avg_grad_norm_0: tensor(269.9330, device='cuda:0') (+tensor(28.1041, device='cuda:0')) | > avg_loss_gen: 2.1242945093504138 (-0.06472412801124738) | > avg_loss_kl: 2.5591205761466234 (+0.41183419630561113) | > avg_loss_feat: 3.7757491830369116 (-0.0852011559714736) | > avg_loss_mel: 19.93377497498418 (+2.5441236495971644) | > avg_loss_duration: 1.7127588550809405 (+0.16570406900325296) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 30.105698088525045 (+2.9717365587261355) | > avg_grad_norm_1: tensor(6379.0952, device='cuda:0') (+tensor(477.3398, device='cuda:0'))  > EPOCH: 1345/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:04:07)   --> TIME: 2025-02-25 22:04:13 -- STEP: 9/72 -- GLOBAL_STEP: 1096850 | > loss_disc: 2.6110637187957764 (2.5288887553744845) | > loss_disc_real_0: 0.05194701999425888 (0.10165539301104015) | > loss_disc_real_1: 0.2583498954772949 (0.1917337824900945) | > loss_disc_real_2: 0.21052168309688568 (0.2362950543562571) | > loss_disc_real_3: 0.22919325530529022 (0.20742914742893642) | > loss_disc_real_4: 0.19557370245456696 (0.18225586579905617) | > loss_disc_real_5: 0.22214964032173157 (0.21956100397639805) | > loss_0: 2.6110637187957764 (2.5288887553744845) | > grad_norm_0: tensor(254.0102, device='cuda:0') (tensor(259.2097, device='cuda:0')) | > loss_gen: 1.97344970703125 (2.0846384498808117) | > loss_kl: 2.2248682975769043 (2.2249965402815075) | > loss_feat: 3.8400933742523193 (4.158577230241564) | > loss_mel: 18.24915885925293 (18.525214936998154) | > loss_duration: 1.5961017608642578 (1.5668356948428683) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.8836727142334 (28.56026289198134) | > grad_norm_1: tensor(12853.8184, device='cuda:0') (tensor(6050.4653, device='cuda:0')) | > current_lr_0: 0.0001690478579312959 | > current_lr_1: 0.0001690478579312959 | > step_time: 0.4282 (0.4341757297515869) | > loader_time: 0.0058 (0.006697336832682292)  --> TIME: 2025-02-25 22:04:37 -- STEP: 59/72 -- GLOBAL_STEP: 1096900 | > loss_disc: 2.631624221801758 (2.639054993451652) | > loss_disc_real_0: 0.1619129329919815 (0.14910672901798108) | > loss_disc_real_1: 0.2249845713376999 (0.22665831863375033) | > loss_disc_real_2: 0.22211195528507233 (0.22706920741978337) | > loss_disc_real_3: 0.20063462853431702 (0.22312208757562152) | > loss_disc_real_4: 0.22236454486846924 (0.22918810303938592) | > loss_disc_real_5: 0.20645520091056824 (0.23673265793565976) | > loss_0: 2.631624221801758 (2.639054993451652) | > grad_norm_0: tensor(106.2564, device='cuda:0') (tensor(222.9409, device='cuda:0')) | > loss_gen: 2.049114465713501 (2.0357212434380743) | > loss_kl: 2.2500250339508057 (2.4465670989731616) | > loss_feat: 3.840158700942993 (3.7875789909039512) | > loss_mel: 18.117401123046875 (18.45399174447787) | > loss_duration: 1.5992103815078735 (1.589697680230868) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.85590934753418 (28.313556865110236) | > grad_norm_1: tensor(5685.9155, device='cuda:0') (tensor(6483.1987, device='cuda:0')) | > current_lr_0: 0.0001690478579312959 | > current_lr_1: 0.0001690478579312959 | > step_time: 0.4743 (0.4511165174387269) | > loader_time: 0.0099 (0.00815324460045766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008699642100804289 (+2.762633310237489e-05) | > avg_step_time: 0.4579562872228488 (-0.010562356089202474) | > avg_loss_disc: 2.6284805989601243 (-0.07308633226743799) | > avg_loss_disc_real_0: 0.1512754688065657 (-0.028771912338028477) | > avg_loss_disc_real_1: 0.2272566545387389 (-0.002869387012971958) | > avg_loss_disc_real_2: 0.2302497127945994 (-0.008565031726595368) | > avg_loss_disc_real_3: 0.22412369305818852 (-0.011555881147653296) | > avg_loss_disc_real_4: 0.23142156760457536 (-0.028782216805807254) | > avg_loss_disc_real_5: 0.2386156293288083 (-0.015641630955145425) | > avg_loss_0: 2.6284805989601243 (-0.07308633226743799) | > avg_grad_norm_0: tensor(223.7882, device='cuda:0') (tensor(-46.1448, device='cuda:0')) | > avg_loss_gen: 2.0750411295555007 (-0.049253379794913066) | > avg_loss_kl: 2.434507665499835 (-0.12461291064678814) | > avg_loss_feat: 3.7751861928214487 (-0.0005629902154629107) | > avg_loss_mel: 18.370716659116074 (-1.5630583158681048) | > avg_loss_duration: 1.594129547266893 (-0.11862930781404746) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.249581216086803 (-1.8561168724382426) | > avg_grad_norm_1: tensor(6531.1011, device='cuda:0') (+tensor(152.0059, device='cuda:0'))  > EPOCH: 1346/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:04:45)   --> TIME: 2025-02-25 22:05:03 -- STEP: 37/72 -- GLOBAL_STEP: 1096950 | > loss_disc: 2.5195326805114746 (2.5744487852663607) | > loss_disc_real_0: 0.142165869474411 (0.11493423883173917) | > loss_disc_real_1: 0.24443233013153076 (0.22056566380165718) | > loss_disc_real_2: 0.2573668360710144 (0.24389463099273476) | > loss_disc_real_3: 0.2698264718055725 (0.2170692899742642) | > loss_disc_real_4: 0.22834578156471252 (0.23205618036759867) | > loss_disc_real_5: 0.22301824390888214 (0.23608280718326569) | > loss_0: 2.5195326805114746 (2.5744487852663607) | > grad_norm_0: tensor(62.9555, device='cuda:0') (tensor(241.7543, device='cuda:0')) | > loss_gen: 2.6527445316314697 (2.1647803976729114) | > loss_kl: 1.9561277627944946 (2.0799292905910587) | > loss_feat: 4.423386573791504 (4.065539965758452) | > loss_mel: 17.51137351989746 (17.209822525849212) | > loss_duration: 1.5496056079864502 (1.5256547831200264) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.093238830566406 (27.04572682767301) | > grad_norm_1: tensor(3021.5896, device='cuda:0') (tensor(7676.5381, device='cuda:0')) | > current_lr_0: 0.0001690267269490545 | > current_lr_1: 0.0001690267269490545 | > step_time: 0.4485 (0.43985105849601125) | > loader_time: 0.0083 (0.007319025091222815) --> EVAL PERFORMANCE | > avg_loader_time: 0.008661427967984909 (-3.821413281938045e-05) | > avg_step_time: 0.4543772986237432 (-0.0035789885991056036) | > avg_loss_disc: 2.6137520763236024 (-0.014728522636521912) | > avg_loss_disc_real_0: 0.1591073990287915 (+0.007831930222225808) | > avg_loss_disc_real_1: 0.21591059863567352 (-0.011346055903065366) | > avg_loss_disc_real_2: 0.24007929651670054 (+0.009829583722101132) | > avg_loss_disc_real_3: 0.22604682537871348 (+0.0019231323205249617) | > avg_loss_disc_real_4: 0.24009791205466632 (+0.008676344450090967) | > avg_loss_disc_real_5: 0.24005874718578768 (+0.0014431178569793701) | > avg_loss_0: 2.6137520763236024 (-0.014728522636521912) | > avg_grad_norm_0: tensor(281.2510, device='cuda:0') (+tensor(57.4628, device='cuda:0')) | > avg_loss_gen: 2.178173318715164 (+0.10313218915966349) | > avg_loss_kl: 2.08671465390165 (-0.3477930115981853) | > avg_loss_feat: 3.9018817485218324 (+0.1266955557003837) | > avg_loss_mel: 17.04873744534774 (-1.3219792137683335) | > avg_loss_duration: 1.553574567109766 (-0.040554980157126996) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.76908170673209 (-1.4804995093547113) | > avg_grad_norm_1: tensor(6443.0034, device='cuda:0') (tensor(-88.0977, device='cuda:0'))  > EPOCH: 1347/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:05:22)   --> TIME: 2025-02-25 22:05:30 -- STEP: 15/72 -- GLOBAL_STEP: 1097000 | > loss_disc: 2.6781821250915527 (2.6316047032674152) | > loss_disc_real_0: 0.22430086135864258 (0.08985932245850563) | > loss_disc_real_1: 0.22272469103336334 (0.20310543477535248) | > loss_disc_real_2: 0.164805069565773 (0.3433367133140564) | > loss_disc_real_3: 0.24927590787410736 (0.2999314308166504) | > loss_disc_real_4: 0.2892117500305176 (0.27477238178253166) | > loss_disc_real_5: 0.2622325122356415 (0.27136222620805106) | > loss_0: 2.6781821250915527 (2.6316047032674152) | > grad_norm_0: tensor(463.1948, device='cuda:0') (tensor(180.4130, device='cuda:0')) | > loss_gen: 2.2758901119232178 (2.446610450744629) | > loss_kl: 2.011963129043579 (1.8741053819656373) | > loss_feat: 3.2423529624938965 (4.043288501103719) | > loss_mel: 17.3995418548584 (17.05798454284668) | > loss_duration: 1.5054121017456055 (1.4978984673817952) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.435161590576172 (26.919887415568034) | > grad_norm_1: tensor(5580.6250, device='cuda:0') (tensor(7284.1616, device='cuda:0')) | > current_lr_0: 0.00016900559860818585 | > current_lr_1: 0.00016900559860818585 | > step_time: 0.4245 (0.43129776318868) | > loader_time: 0.0058 (0.0067462921142578125)  --> TIME: 2025-02-25 22:05:53 -- STEP: 65/72 -- GLOBAL_STEP: 1097050 | > loss_disc: 2.5180797576904297 (2.6141007863558254) | > loss_disc_real_0: 0.10446849465370178 (0.11100669829891278) | > loss_disc_real_1: 0.18862216174602509 (0.2126068454522353) | > loss_disc_real_2: 0.19884632527828217 (0.2552329521912796) | > loss_disc_real_3: 0.21563979983329773 (0.24222733584734113) | > loss_disc_real_4: 0.24285508692264557 (0.25136658732707673) | > loss_disc_real_5: 0.24484889209270477 (0.2508026962096874) | > loss_0: 2.5180797576904297 (2.6141007863558254) | > grad_norm_0: tensor(200.4632, device='cuda:0') (tensor(215.9917, device='cuda:0')) | > loss_gen: 2.160200834274292 (2.2172921309104323) | > loss_kl: 2.2682831287384033 (1.9728371473459096) | > loss_feat: 4.304934978485107 (3.950688684903658) | > loss_mel: 16.89637565612793 (16.7470124904926) | > loss_duration: 1.6130205392837524 (1.533210725050706) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.242813110351562 (26.421040989802435) | > grad_norm_1: tensor(3032.3259, device='cuda:0') (tensor(6986.9658, device='cuda:0')) | > current_lr_0: 0.00016900559860818585 | > current_lr_1: 0.00016900559860818585 | > step_time: 0.5106 (0.44547091997586763) | > loader_time: 0.0116 (0.008421006569495569) --> EVAL PERFORMANCE | > avg_loader_time: 0.008709356818400637 (+4.792885041572832e-05) | > avg_step_time: 0.451243390499706 (-0.003133908124037188) | > avg_loss_disc: 2.6037105540154686 (-0.010041522308133821) | > avg_loss_disc_real_0: 0.10936674470422973 (-0.049740654324561775) | > avg_loss_disc_real_1: 0.2129465440209483 (-0.0029640546147252367) | > avg_loss_disc_real_2: 0.252721452377212 (+0.012642155860511456) | > avg_loss_disc_real_3: 0.24162190480971 (+0.015575079430996508) | > avg_loss_disc_real_4: 0.2506925022937881 (+0.010594590239121765) | > avg_loss_disc_real_5: 0.2513030162999328 (+0.011244269114145111) | > avg_loss_0: 2.6037105540154686 (-0.010041522308133821) | > avg_grad_norm_0: tensor(206.5315, device='cuda:0') (tensor(-74.7195, device='cuda:0')) | > avg_loss_gen: 2.220502472259628 (+0.04232915354446387) | > avg_loss_kl: 1.9860259898951356 (-0.10068866400651433) | > avg_loss_feat: 3.9539592904104315 (+0.05207754188859903) | > avg_loss_mel: 16.785680112704423 (-0.2630573326433172) | > avg_loss_duration: 1.538489021046061 (-0.015085546063705157) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.484656615996023 (-0.28442509073606814) | > avg_grad_norm_1: tensor(6808.8652, device='cuda:0') (+tensor(365.8618, device='cuda:0'))  > EPOCH: 1348/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:05:58)   --> TIME: 2025-02-25 22:06:20 -- STEP: 43/72 -- GLOBAL_STEP: 1097100 | > loss_disc: 2.5208334922790527 (2.6274290528408315) | > loss_disc_real_0: 0.24555720388889313 (0.11272365352961906) | > loss_disc_real_1: 0.22990290820598602 (0.22858594357967377) | > loss_disc_real_2: 0.23372726142406464 (0.22087231243765632) | > loss_disc_real_3: 0.23994509875774384 (0.22863119424775588) | > loss_disc_real_4: 0.22259260714054108 (0.23899069255174593) | > loss_disc_real_5: 0.21832548081874847 (0.24428887734579485) | > loss_0: 2.5208334922790527 (2.6274290528408315) | > grad_norm_0: tensor(535.4247, device='cuda:0') (tensor(241.7739, device='cuda:0')) | > loss_gen: 2.481397867202759 (2.0591318773668865) | > loss_kl: 1.9488435983657837 (1.9138849890509317) | > loss_feat: 4.702731132507324 (3.824561584827512) | > loss_mel: 15.808695793151855 (16.680296786995818) | > loss_duration: 1.538071870803833 (1.5091352656830188) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.479740142822266 (25.987010734025823) | > grad_norm_1: tensor(2917.9419, device='cuda:0') (tensor(6609.6201, device='cuda:0')) | > current_lr_0: 0.0001689844729083598 | > current_lr_1: 0.0001689844729083598 | > step_time: 0.4525 (0.44295955813208293) | > loader_time: 0.0094 (0.007463837778845499) --> EVAL PERFORMANCE | > avg_loader_time: 0.008607149124145508 (-0.00010220769425512907) | > avg_step_time: 0.4546998621712268 (+0.0034564716715208) | > avg_loss_disc: 2.6156220939797414 (+0.011911539964272855) | > avg_loss_disc_real_0: 0.1128379287801578 (+0.0034711840759280715) | > avg_loss_disc_real_1: 0.22195488550293613 (+0.009008341481987842) | > avg_loss_disc_real_2: 0.22677446230196616 (-0.02594699007524584) | > avg_loss_disc_real_3: 0.22909828369886104 (-0.012523621110848954) | > avg_loss_disc_real_4: 0.2398997165367637 (-0.010792785757024387) | > avg_loss_disc_real_5: 0.24508239853550012 (-0.006220617764432668) | > avg_loss_0: 2.6156220939797414 (+0.011911539964272855) | > avg_grad_norm_0: tensor(242.6054, device='cuda:0') (+tensor(36.0739, device='cuda:0')) | > avg_loss_gen: 2.079878158972297 (-0.14062431328733105) | > avg_loss_kl: 1.9652452183441378 (-0.02078077155099778) | > avg_loss_feat: 3.8189691792071705 (-0.134990111203261) | > avg_loss_mel: 16.721033969395585 (-0.06464614330883833) | > avg_loss_duration: 1.5346189535839456 (-0.0038700674621152675) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.119745603749447 (-0.3649110122465764) | > avg_grad_norm_1: tensor(6734.0361, device='cuda:0') (tensor(-74.8291, device='cuda:0'))  > EPOCH: 1349/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:06:36)   --> TIME: 2025-02-25 22:06:47 -- STEP: 21/72 -- GLOBAL_STEP: 1097150 | > loss_disc: 2.764068126678467 (2.617208310535976) | > loss_disc_real_0: 0.18957021832466125 (0.09511169720263708) | > loss_disc_real_1: 0.2467399686574936 (0.2301121581168402) | > loss_disc_real_2: 0.2207852452993393 (0.23343710672287715) | > loss_disc_real_3: 0.22815944254398346 (0.2275029250553676) | > loss_disc_real_4: 0.23648424446582794 (0.2396648604245413) | > loss_disc_real_5: 0.22103232145309448 (0.24443556652182624) | > loss_0: 2.764068126678467 (2.617208310535976) | > grad_norm_0: tensor(255.7838, device='cuda:0') (tensor(219.8614, device='cuda:0')) | > loss_gen: 1.8558868169784546 (2.0824961321694517) | > loss_kl: 1.707248568534851 (1.9015000490915208) | > loss_feat: 3.6995646953582764 (3.986396698724656) | > loss_mel: 16.72034454345703 (16.74458372025263) | > loss_duration: 1.5164340734481812 (1.4892687570481074) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.49947738647461 (26.204245158604213) | > grad_norm_1: tensor(2147.7615, device='cuda:0') (tensor(5271.8794, device='cuda:0')) | > current_lr_0: 0.00016896334984924625 | > current_lr_1: 0.00016896334984924625 | > step_time: 0.4318 (0.4360454763684954) | > loader_time: 0.0064 (0.006724709556216285)  --> TIME: 2025-02-25 22:07:11 -- STEP: 71/72 -- GLOBAL_STEP: 1097200 | > loss_disc: 3.098482370376587 (2.6330993813528143) | > loss_disc_real_0: 0.4662773013114929 (0.13464513552230845) | > loss_disc_real_1: 0.3515041768550873 (0.22796076788029201) | > loss_disc_real_2: 0.33451539278030396 (0.23351007343178065) | > loss_disc_real_3: 0.23164451122283936 (0.23205608402339506) | > loss_disc_real_4: 0.24434298276901245 (0.24219846515588359) | > loss_disc_real_5: 0.26237979531288147 (0.24594705604331593) | > loss_0: 3.098482370376587 (2.6330993813528143) | > grad_norm_0: tensor(233.6451, device='cuda:0') (tensor(185.7931, device='cuda:0')) | > loss_gen: 2.303356885910034 (2.1047910200038427) | > loss_kl: 2.009831190109253 (1.9998254473780241) | > loss_feat: 3.189958333969116 (3.7840609516895993) | > loss_mel: 17.255037307739258 (16.76184405743237) | > loss_duration: 1.5299370288848877 (1.5312451308881734) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.28812026977539 (26.18176637568944) | > grad_norm_1: tensor(316.8383, device='cuda:0') (tensor(5911.8076, device='cuda:0')) | > current_lr_0: 0.00016896334984924625 | > current_lr_1: 0.00016896334984924625 | > step_time: 0.4888 (0.4547146441231311) | > loader_time: 0.0083 (0.008605288787626885) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605288787626885 (-1.860336518623093e-06) | > avg_step_time: 0.4547146441231311 (+1.4781951904296875e-05) | > avg_loss_disc: 2.6330993813528143 (+0.01747728737307286) | > avg_loss_disc_real_0: 0.13464513552230845 (+0.021807206742150642) | > avg_loss_disc_real_1: 0.22796076788029201 (+0.006005882377355887) | > avg_loss_disc_real_2: 0.23351007343178065 (+0.006735611129814495) | > avg_loss_disc_real_3: 0.23205608402339506 (+0.002957800324534021) | > avg_loss_disc_real_4: 0.24219846515588359 (+0.0022987486191198836) | > avg_loss_disc_real_5: 0.24594705604331593 (+0.0008646575078158081) | > avg_loss_0: 2.6330993813528143 (+0.01747728737307286) | > avg_grad_norm_0: tensor(185.7931, device='cuda:0') (tensor(-56.8124, device='cuda:0')) | > avg_loss_gen: 2.1047910200038427 (+0.024912861031545663) | > avg_loss_kl: 1.9998254473780241 (+0.034580229033886356) | > avg_loss_feat: 3.7840609516895993 (-0.034908227517571166) | > avg_loss_mel: 16.76184405743237 (+0.04081008803678543) | > avg_loss_duration: 1.5312451308881734 (-0.003373822695772244) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.18176637568944 (+0.06202077193999145) | > avg_grad_norm_1: tensor(5911.8076, device='cuda:0') (tensor(-822.2285, device='cuda:0'))  > EPOCH: 1350/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:07:13)   --> TIME: 2025-02-25 22:07:37 -- STEP: 49/72 -- GLOBAL_STEP: 1097250 | > loss_disc: 2.532845973968506 (2.6482582530196828) | > loss_disc_real_0: 0.08728887140750885 (0.20737706063961497) | > loss_disc_real_1: 0.2396075427532196 (0.22154744075877325) | > loss_disc_real_2: 0.21970416605472565 (0.25451059943559207) | > loss_disc_real_3: 0.23483404517173767 (0.2381052697191433) | > loss_disc_real_4: 0.24380303919315338 (0.23771345919492293) | > loss_disc_real_5: 0.2249630242586136 (0.24541978811731144) | > loss_0: 2.532845973968506 (2.6482582530196828) | > grad_norm_0: tensor(152.7084, device='cuda:0') (tensor(246.1220, device='cuda:0')) | > loss_gen: 2.2133264541625977 (2.179956290186668) | > loss_kl: 1.973994493484497 (2.0820133369796125) | > loss_feat: 4.378903865814209 (3.722940595782533) | > loss_mel: 16.261991500854492 (17.368211259647293) | > loss_duration: 1.589557409286499 (1.5201338918841616) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.417774200439453 (26.873255437734176) | > grad_norm_1: tensor(3487.1089, device='cuda:0') (tensor(5888.0630, device='cuda:0')) | > current_lr_0: 0.00016894222943051508 | > current_lr_1: 0.00016894222943051508 | > step_time: 0.4446 (0.442498508764773) | > loader_time: 0.0098 (0.007805216069124183) --> EVAL PERFORMANCE | > avg_loader_time: 0.008734471361402056 (+0.00012918257377517173) | > avg_step_time: 0.4535996678849341 (-0.0011149762381970074) | > avg_loss_disc: 2.657218751773028 (+0.024119370420213837) | > avg_loss_disc_real_0: 0.2021647190124216 (+0.06751958349011317) | > avg_loss_disc_real_1: 0.22301941994630115 (-0.0049413479339908695) | > avg_loss_disc_real_2: 0.24643910959572848 (+0.012929036163947832) | > avg_loss_disc_real_3: 0.23402539819059237 (+0.0019693141671973113) | > avg_loss_disc_real_4: 0.2390015465272984 (-0.003196918628585177) | > avg_loss_disc_real_5: 0.24422818884043626 (-0.0017188672028796703) | > avg_loss_0: 2.657218751773028 (+0.024119370420213837) | > avg_grad_norm_0: tensor(266.2032, device='cuda:0') (+tensor(80.4101, device='cuda:0')) | > avg_loss_gen: 2.1311813760811176 (+0.026390356077274912) | > avg_loss_kl: 2.0736102050458873 (+0.07378475766786319) | > avg_loss_feat: 3.673191157864853 (-0.11086979382474649) | > avg_loss_mel: 17.167764542807994 (+0.4059204853756242) | > avg_loss_duration: 1.5369244612438577 (+0.005679330355684353) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.582671850499974 (+0.4009054748105356) | > avg_grad_norm_1: tensor(6616.9004, device='cuda:0') (+tensor(705.0928, device='cuda:0'))  > EPOCH: 1351/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:07:50)   --> TIME: 2025-02-25 22:08:04 -- STEP: 27/72 -- GLOBAL_STEP: 1097300 | > loss_disc: 2.537642240524292 (2.6064238106762923) | > loss_disc_real_0: 0.11982069164514542 (0.16716913409807066) | > loss_disc_real_1: 0.1971479058265686 (0.21174095846988536) | > loss_disc_real_2: 0.2702433466911316 (0.23136643844622154) | > loss_disc_real_3: 0.2284480780363083 (0.22301477911295714) | > loss_disc_real_4: 0.24569056928157806 (0.24182220796744028) | > loss_disc_real_5: 0.2524002194404602 (0.23945568281191368) | > loss_0: 2.537642240524292 (2.6064238106762923) | > grad_norm_0: tensor(115.9431, device='cuda:0') (tensor(213.3229, device='cuda:0')) | > loss_gen: 2.3117520809173584 (2.1325522352147983) | > loss_kl: 1.93744695186615 (1.870025422837999) | > loss_feat: 3.997657060623169 (3.829449459358498) | > loss_mel: 16.741168975830078 (16.704253726535374) | > loss_duration: 1.4913524389266968 (1.4823969337675307) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.47937774658203 (26.018677746808088) | > grad_norm_1: tensor(7480.6655, device='cuda:0') (tensor(7158.5317, device='cuda:0')) | > current_lr_0: 0.00016892111165183626 | > current_lr_1: 0.00016892111165183626 | > step_time: 0.4372 (0.4366754337593361) | > loader_time: 0.0095 (0.006950175320660627) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617824232074573 (-0.00011664712932748372) | > avg_step_time: 0.4546533033881389 (+0.0010536355032048084) | > avg_loss_disc: 2.646323751395857 (-0.010895000377171282) | > avg_loss_disc_real_0: 0.14230287117017826 (-0.059861847842243354) | > avg_loss_disc_real_1: 0.23930352569465907 (+0.016284105748357924) | > avg_loss_disc_real_2: 0.2462469542949972 (-0.00019215530073127263) | > avg_loss_disc_real_3: 0.23772486876434004 (+0.0036994705737476707) | > avg_loss_disc_real_4: 0.24848852422035916 (+0.009486977693060755) | > avg_loss_disc_real_5: 0.24984746730663407 (+0.005619278466197808) | > avg_loss_0: 2.646323751395857 (-0.010895000377171282) | > avg_grad_norm_0: tensor(195.8867, device='cuda:0') (tensor(-70.3165, device='cuda:0')) | > avg_loss_gen: 2.153238464409197 (+0.02205708832807929) | > avg_loss_kl: 2.1569487309791664 (+0.08333852593327906) | > avg_loss_feat: 3.830926327638223 (+0.1577351697733702) | > avg_loss_mel: 17.909674134052974 (+0.7419095912449798) | > avg_loss_duration: 1.5576060070118434 (+0.020681545767985643) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.608393709424515 (+1.0257218589245412) | > avg_grad_norm_1: tensor(5530.4312, device='cuda:0') (tensor(-1086.4692, device='cuda:0'))  > EPOCH: 1352/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:08:27)   --> TIME: 2025-02-25 22:08:31 -- STEP: 5/72 -- GLOBAL_STEP: 1097350 | > loss_disc: 2.4795007705688477 (2.6050350189208986) | > loss_disc_real_0: 0.032210782170295715 (0.05138281360268593) | > loss_disc_real_1: 0.23623543977737427 (0.2163788139820099) | > loss_disc_real_2: 0.18886271119117737 (0.15885285586118697) | > loss_disc_real_3: 0.2719475030899048 (0.3001261591911316) | > loss_disc_real_4: 0.24955669045448303 (0.24511261880397797) | > loss_disc_real_5: 0.22207581996917725 (0.23977384567260743) | > loss_0: 2.4795007705688477 (2.6050350189208986) | > grad_norm_0: tensor(35.0824, device='cuda:0') (tensor(100.1837, device='cuda:0')) | > loss_gen: 2.2750518321990967 (2.0849193572998046) | > loss_kl: 2.126995801925659 (2.079701232910156) | > loss_feat: 4.555490016937256 (4.20288200378418) | > loss_mel: 17.720441818237305 (17.670327377319335) | > loss_duration: 1.4951813220977783 (1.534797978401184) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.173160552978516 (27.572628021240234) | > grad_norm_1: tensor(1977.8636, device='cuda:0') (tensor(4206.4775, device='cuda:0')) | > current_lr_0: 0.00016889999651287978 | > current_lr_1: 0.00016889999651287978 | > step_time: 0.4239 (0.43361854553222656) | > loader_time: 0.0063 (0.006041717529296875)  --> TIME: 2025-02-25 22:08:54 -- STEP: 55/72 -- GLOBAL_STEP: 1097400 | > loss_disc: 2.574877977371216 (2.5044014280492606) | > loss_disc_real_0: 0.045750923454761505 (0.0602300512181087) | > loss_disc_real_1: 0.20882557332515717 (0.21737461388111115) | > loss_disc_real_2: 0.26266685128211975 (0.23497601598501205) | > loss_disc_real_3: 0.22244787216186523 (0.23560327752069993) | > loss_disc_real_4: 0.2552953362464905 (0.2415548335422169) | > loss_disc_real_5: 0.2527604103088379 (0.24045031829313798) | > loss_0: 2.574877977371216 (2.5044014280492606) | > grad_norm_0: tensor(237.7802, device='cuda:0') (tensor(150.4996, device='cuda:0')) | > loss_gen: 1.953239917755127 (2.227986940470609) | > loss_kl: 1.931180477142334 (2.0103164022619073) | > loss_feat: 3.797917127609253 (4.199707724831324) | > loss_mel: 17.40447235107422 (16.946216912703083) | > loss_duration: 1.5676569938659668 (1.5295243436639958) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.65446662902832 (26.913752226396042) | > grad_norm_1: tensor(8123.9834, device='cuda:0') (tensor(6978.7471, device='cuda:0')) | > current_lr_0: 0.00016889999651287978 | > current_lr_1: 0.00016889999651287978 | > step_time: 0.4613 (0.44475038701837716) | > loader_time: 0.0092 (0.007854218916459517) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556255152527711 (-6.156907954686132e-05) | > avg_step_time: 0.4553525380685296 (+0.0006992346803906813) | > avg_loss_disc: 2.502497075309216 (-0.14382667608664068) | > avg_loss_disc_real_0: 0.06183662771863837 (-0.08046624345153988) | > avg_loss_disc_real_1: 0.21830230867359 (-0.02100121702106908) | > avg_loss_disc_real_2: 0.2322200968022078 (-0.014026857492789424) | > avg_loss_disc_real_3: 0.23504668495184938 (-0.0026781838124906587) | > avg_loss_disc_real_4: 0.24244354854167346 (-0.006044975678685699) | > avg_loss_disc_real_5: 0.2418429950173472 (-0.008004472289286857) | > avg_loss_0: 2.502497075309216 (-0.14382667608664068) | > avg_grad_norm_0: tensor(157.6163, device='cuda:0') (tensor(-38.2704, device='cuda:0')) | > avg_loss_gen: 2.2318992178205033 (+0.0786607534113064) | > avg_loss_kl: 2.0253747416214214 (-0.131573989357745) | > avg_loss_feat: 4.1832207726760675 (+0.35229444503784446) | > avg_loss_mel: 16.89930406758483 (-1.0103700664681448) | > avg_loss_duration: 1.5416001484427653 (-0.016005858569078102) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.881398886022435 (-0.7269948234020802) | > avg_grad_norm_1: tensor(7083.7295, device='cuda:0') (+tensor(1553.2983, device='cuda:0'))  > EPOCH: 1353/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:09:04)   --> TIME: 2025-02-25 22:09:20 -- STEP: 33/72 -- GLOBAL_STEP: 1097450 | > loss_disc: 2.5342447757720947 (2.4909283103364888) | > loss_disc_real_0: 0.09898307174444199 (0.06243338001271089) | > loss_disc_real_1: 0.21201997995376587 (0.21563425660133362) | > loss_disc_real_2: 0.24930670857429504 (0.2247119920723366) | > loss_disc_real_3: 0.22322200238704681 (0.23049025643955579) | > loss_disc_real_4: 0.24130064249038696 (0.24192607899506888) | > loss_disc_real_5: 0.24247771501541138 (0.24642079255797647) | > loss_0: 2.5342447757720947 (2.4909283103364888) | > grad_norm_0: tensor(237.9448, device='cuda:0') (tensor(205.2453, device='cuda:0')) | > loss_gen: 2.2186391353607178 (2.2688166300455728) | > loss_kl: 1.860893964767456 (1.8939710133003467) | > loss_feat: 4.2709832191467285 (4.355664896242546) | > loss_mel: 16.318431854248047 (16.657016089468286) | > loss_duration: 1.475532054901123 (1.48541500712886) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.144481658935547 (26.66088375900731) | > grad_norm_1: tensor(5004.5723, device='cuda:0') (tensor(7868.1650, device='cuda:0')) | > current_lr_0: 0.00016887888401331567 | > current_lr_1: 0.00016887888401331567 | > step_time: 0.4371 (0.4349672505349824) | > loader_time: 0.0079 (0.00704074628425367) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855481792503679 (-1.4372274909218996e-06) | > avg_step_time: 0.4553690426786181 (+1.650461008850801e-05) | > avg_loss_disc: 2.5001002432594834 (-0.0023968320497327333) | > avg_loss_disc_real_0: 0.07335354867850392 (+0.011516920959865555) | > avg_loss_disc_real_1: 0.21784679381780223 (-0.0004555148557877642) | > avg_loss_disc_real_2: 0.22421881766386434 (-0.008001279138343448) | > avg_loss_disc_real_3: 0.2307460885232603 (-0.004300596428589071) | > avg_loss_disc_real_4: 0.2429209865734611 (+0.0004774380317876248) | > avg_loss_disc_real_5: 0.24715153830991665 (+0.00530854329256944) | > avg_loss_0: 2.5001002432594834 (-0.0023968320497327333) | > avg_grad_norm_0: tensor(196.5968, device='cuda:0') (+tensor(38.9805, device='cuda:0')) | > avg_loss_gen: 2.268795248488305 (+0.03689603066780167) | > avg_loss_kl: 1.9443736546476122 (-0.08100108697380914) | > avg_loss_feat: 4.250893629772563 (+0.06767285709649506) | > avg_loss_mel: 16.615696168281662 (-0.283607899303167) | > avg_loss_duration: 1.5272085112585148 (-0.014391637184250516) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.60696736188002 (-0.2744315241424147) | > avg_grad_norm_1: tensor(7676.5537, device='cuda:0') (+tensor(592.8242, device='cuda:0'))  > EPOCH: 1354/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:09:41)   --> TIME: 2025-02-25 22:09:47 -- STEP: 11/72 -- GLOBAL_STEP: 1097500 | > loss_disc: 2.39129900932312 (2.5060230168429287) | > loss_disc_real_0: 0.06911655515432358 (0.0843059854073958) | > loss_disc_real_1: 0.19844911992549896 (0.2080341184681112) | > loss_disc_real_2: 0.21015900373458862 (0.254644513130188) | > loss_disc_real_3: 0.25507235527038574 (0.23588212511756204) | > loss_disc_real_4: 0.24570809304714203 (0.2329335402358662) | > loss_disc_real_5: 0.26411575078964233 (0.24206164343790573) | > loss_0: 2.39129900932312 (2.5060230168429287) | > grad_norm_0: tensor(120.3072, device='cuda:0') (tensor(212.0963, device='cuda:0')) | > loss_gen: 2.2853736877441406 (2.264662168242715) | > loss_kl: 1.7788901329040527 (1.8199869719418613) | > loss_feat: 4.239730358123779 (4.286700595508922) | > loss_mel: 16.44673728942871 (16.615642460909758) | > loss_duration: 1.4771063327789307 (1.480543472550132) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.22783851623535 (26.46753605929288) | > grad_norm_1: tensor(11867.0762, device='cuda:0') (tensor(8184.6797, device='cuda:0')) | > current_lr_0: 0.000168857774152814 | > current_lr_1: 0.000168857774152814 | > step_time: 0.4269 (0.4270757761868564) | > loader_time: 0.0059 (0.006168690594759854)  --> TIME: 2025-02-25 22:10:10 -- STEP: 61/72 -- GLOBAL_STEP: 1097550 | > loss_disc: 2.324532985687256 (2.490779575754385) | > loss_disc_real_0: 0.07073155045509338 (0.07673082048775719) | > loss_disc_real_1: 0.20835109055042267 (0.21637116055019567) | > loss_disc_real_2: 0.22545504570007324 (0.2323300462277209) | > loss_disc_real_3: 0.25570711493492126 (0.23145826942608005) | > loss_disc_real_4: 0.2340821474790573 (0.2419008727444977) | > loss_disc_real_5: 0.24993982911109924 (0.24368834251263102) | > loss_0: 2.324532985687256 (2.490779575754385) | > grad_norm_0: tensor(136.0839, device='cuda:0') (tensor(192.8869, device='cuda:0')) | > loss_gen: 2.556570291519165 (2.2800495761339774) | > loss_kl: 2.0707132816314697 (1.9098203905293198) | > loss_feat: 4.4024200439453125 (4.3020845475744025) | > loss_mel: 16.50352668762207 (16.517472017006796) | > loss_duration: 1.5530226230621338 (1.5144033901026994) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.086252212524414 (26.523830038602235) | > grad_norm_1: tensor(5891.0674, device='cuda:0') (tensor(7553.4995, device='cuda:0')) | > current_lr_0: 0.000168857774152814 | > current_lr_1: 0.000168857774152814 | > step_time: 0.455 (0.44205722652497836) | > loader_time: 0.0107 (0.008010836898303425) --> EVAL PERFORMANCE | > avg_loader_time: 0.008496643791736015 (-5.817413330077431e-05) | > avg_step_time: 0.45031021346508615 (-0.005058829213531957) | > avg_loss_disc: 2.4950828014964794 (-0.005017441763004005) | > avg_loss_disc_real_0: 0.07547569773356677 (+0.002122149055062847) | > avg_loss_disc_real_1: 0.2161232136504751 (-0.0017235801673271267) | > avg_loss_disc_real_2: 0.2341380583148607 (+0.009919240650996364) | > avg_loss_disc_real_3: 0.22911833730381978 (-0.001627751219440532) | > avg_loss_disc_real_4: 0.2404538330057977 (-0.0024671535676633882) | > avg_loss_disc_real_5: 0.24362838520130642 (-0.003523153108610233) | > avg_loss_0: 2.4950828014964794 (-0.005017441763004005) | > avg_grad_norm_0: tensor(195.4032, device='cuda:0') (tensor(-1.1936, device='cuda:0')) | > avg_loss_gen: 2.265133275112635 (-0.00366197337566998) | > avg_loss_kl: 1.932070533994218 (-0.01230312065339434) | > avg_loss_feat: 4.2742964684123725 (+0.023402838639809964) | > avg_loss_mel: 16.526856570176676 (-0.0888395981049861) | > avg_loss_duration: 1.5249466828896967 (-0.002261828368818053) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.523303636362854 (-0.08366372551716594) | > avg_grad_norm_1: tensor(7679.0410, device='cuda:0') (+tensor(2.4873, device='cuda:0'))  > EPOCH: 1355/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:10:18)   --> TIME: 2025-02-25 22:10:37 -- STEP: 39/72 -- GLOBAL_STEP: 1097600 | > loss_disc: 2.4872164726257324 (2.477268176201062) | > loss_disc_real_0: 0.07063175737857819 (0.07817465534959084) | > loss_disc_real_1: 0.1952066719532013 (0.2183555162105805) | > loss_disc_real_2: 0.20926888287067413 (0.2252265826249734) | > loss_disc_real_3: 0.21871620416641235 (0.22842358434811616) | > loss_disc_real_4: 0.23134566843509674 (0.23790306464219704) | > loss_disc_real_5: 0.2531382441520691 (0.24665490824442643) | > loss_0: 2.4872164726257324 (2.477268176201062) | > grad_norm_0: tensor(136.8032, device='cuda:0') (tensor(226.9706, device='cuda:0')) | > loss_gen: 2.159223794937134 (2.321525246669084) | > loss_kl: 1.9404746294021606 (1.8749659886726966) | > loss_feat: 4.241025447845459 (4.36377438520774) | > loss_mel: 16.81125259399414 (16.679561346005165) | > loss_duration: 1.5358810424804688 (1.491405908878033) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.687856674194336 (26.731232912112507) | > grad_norm_1: tensor(6080.1367, device='cuda:0') (tensor(7358.0127, device='cuda:0')) | > current_lr_0: 0.00016883666693104488 | > current_lr_1: 0.00016883666693104488 | > step_time: 0.453 (0.43880581855773926) | > loader_time: 0.008 (0.0075684327345628) --> EVAL PERFORMANCE | > avg_loader_time: 0.008765368394448721 (+0.00026872460271270583) | > avg_step_time: 0.4549602757037525 (+0.004650062238666375) | > avg_loss_disc: 2.4964167803106174 (+0.0013339788141379927) | > avg_loss_disc_real_0: 0.08646037996235029 (+0.010984682228783516) | > avg_loss_disc_real_1: 0.2199094448710831 (+0.003786231220608005) | > avg_loss_disc_real_2: 0.22213401320114942 (-0.012004045113711281) | > avg_loss_disc_real_3: 0.23161508625661822 (+0.0024967489527984443) | > avg_loss_disc_real_4: 0.23852350615279774 (-0.0019303268529999584) | > avg_loss_disc_real_5: 0.24587378451521968 (+0.0022453993139132655) | > avg_loss_0: 2.4964167803106174 (+0.0013339788141379927) | > avg_grad_norm_0: tensor(221.7264, device='cuda:0') (+tensor(26.3232, device='cuda:0')) | > avg_loss_gen: 2.301302429655907 (+0.036169154543272075) | > avg_loss_kl: 1.9411582896407222 (+0.009087755646504325) | > avg_loss_feat: 4.228140072083811 (-0.04615639632856183) | > avg_loss_mel: 16.521443743101308 (-0.005412827075367943) | > avg_loss_duration: 1.5242733686742647 (-0.0006733142154320415) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.51631782424282 (-0.006985812120035462) | > avg_grad_norm_1: tensor(6784.2402, device='cuda:0') (tensor(-894.8008, device='cuda:0'))  > EPOCH: 1356/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:10:55)   --> TIME: 2025-02-25 22:11:04 -- STEP: 17/72 -- GLOBAL_STEP: 1097650 | > loss_disc: 2.433673143386841 (2.4697056517881504) | > loss_disc_real_0: 0.06130106374621391 (0.08639278854517375) | > loss_disc_real_1: 0.2791323661804199 (0.22171898098552928) | > loss_disc_real_2: 0.24522939324378967 (0.23367874937898972) | > loss_disc_real_3: 0.23990334570407867 (0.23476960290880763) | > loss_disc_real_4: 0.2340487241744995 (0.2349026588832631) | > loss_disc_real_5: 0.26387539505958557 (0.2480415552854538) | > loss_0: 2.433673143386841 (2.4697056517881504) | > grad_norm_0: tensor(197.0521, device='cuda:0') (tensor(211.4190, device='cuda:0')) | > loss_gen: 2.521242618560791 (2.3739389251260197) | > loss_kl: 2.118537187576294 (1.856546114472782) | > loss_feat: 4.9676313400268555 (4.378439608742209) | > loss_mel: 17.235692977905273 (16.45043401157155) | > loss_duration: 1.5134735107421875 (1.4791246792849373) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.356576919555664 (26.538483227000516) | > grad_norm_1: tensor(4390.9795, device='cuda:0') (tensor(6427.7251, device='cuda:0')) | > current_lr_0: 0.0001688155623476785 | > current_lr_1: 0.0001688155623476785 | > step_time: 0.4414 (0.43446869008681355) | > loader_time: 0.0076 (0.007044722052181468)  --> TIME: 2025-02-25 22:11:28 -- STEP: 67/72 -- GLOBAL_STEP: 1097700 | > loss_disc: 2.4381320476531982 (2.5631368871945064) | > loss_disc_real_0: 0.10129036009311676 (0.1108854325182402) | > loss_disc_real_1: 0.18828606605529785 (0.23725149280099728) | > loss_disc_real_2: 0.2006518840789795 (0.23065129078146238) | > loss_disc_real_3: 0.14934423565864563 (0.2575125262808445) | > loss_disc_real_4: 0.2141006588935852 (0.24809945252404283) | > loss_disc_real_5: 0.22420313954353333 (0.24489842244048618) | > loss_0: 2.4381320476531982 (2.5631368871945064) | > grad_norm_0: tensor(124.5611, device='cuda:0') (tensor(220.1996, device='cuda:0')) | > loss_gen: 2.367987871170044 (2.363390821129528) | > loss_kl: 2.2232298851013184 (1.9985085227596227) | > loss_feat: 4.950683116912842 (4.30662843362609) | > loss_mel: 17.729063034057617 (16.83016005558754) | > loss_duration: 1.6244450807571411 (1.5263420307814186) | > amp_scaler: 8.0 (8.358208955223883) | > loss_1: 28.895410537719727 (27.025029936833167) | > grad_norm_1: tensor(3774.5464, device='cuda:0') (tensor(5322.5571, device='cuda:0')) | > current_lr_0: 0.0001688155623476785 | > current_lr_1: 0.0001688155623476785 | > step_time: 0.513 (0.4474898167510531) | > loader_time: 0.012 (0.00848268750888198) --> EVAL PERFORMANCE | > avg_loader_time: 0.008656730114574164 (-0.000108638279874557) | > avg_step_time: 0.45136415454703316 (-0.0035961211567193674) | > avg_loss_disc: 2.5724604465592082 (+0.07604366624859082) | > avg_loss_disc_real_0: 0.11358437918021642 (+0.027123999217866138) | > avg_loss_disc_real_1: 0.23724601176423085 (+0.017336566893147748) | > avg_loss_disc_real_2: 0.22889948667774737 (+0.00676547347659795) | > avg_loss_disc_real_3: 0.25144809757319986 (+0.019833011316581634) | > avg_loss_disc_real_4: 0.2446787523131975 (+0.006155246160399752) | > avg_loss_disc_real_5: 0.2442391772925014 (-0.0016346072227182706) | > avg_loss_0: 2.5724604465592082 (+0.07604366624859082) | > avg_grad_norm_0: tensor(216.3906, device='cuda:0') (tensor(-5.3358, device='cuda:0')) | > avg_loss_gen: 2.3397198243879926 (+0.03841739473208561) | > avg_loss_kl: 2.007611011115598 (+0.0664527214748758) | > avg_loss_feat: 4.273671455786261 (+0.04553138370245069) | > avg_loss_mel: 16.845846203011526 (+0.32440245991021754) | > avg_loss_duration: 1.5301162709652538 (+0.00584290229098916) | > avg_amp_scaler: 8.33802816901409 (+0.3380281690140894) | > avg_loss_1: 26.996964790451695 (+0.4806469662088766) | > avg_grad_norm_1: tensor(5392.2822, device='cuda:0') (tensor(-1391.9580, device='cuda:0'))  > EPOCH: 1357/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:11:32)   --> TIME: 2025-02-25 22:11:54 -- STEP: 45/72 -- GLOBAL_STEP: 1097750 | > loss_disc: 2.626325845718384 (2.677931165695191) | > loss_disc_real_0: 0.14492416381835938 (0.19684821065101363) | > loss_disc_real_1: 0.21472595632076263 (0.23184434043036567) | > loss_disc_real_2: 0.22768664360046387 (0.23830728431542714) | > loss_disc_real_3: 0.2227349430322647 (0.24137307869063485) | > loss_disc_real_4: 0.2500239610671997 (0.24270692831940122) | > loss_disc_real_5: 0.26573628187179565 (0.24440852999687196) | > loss_0: 2.626325845718384 (2.677931165695191) | > grad_norm_0: tensor(614.5804, device='cuda:0') (tensor(338.0107, device='cuda:0')) | > loss_gen: 1.9064972400665283 (2.154584670066833) | > loss_kl: 2.0414836406707764 (1.9747617721557618) | > loss_feat: 2.9521069526672363 (3.7763455867767335) | > loss_mel: 16.145963668823242 (16.706089189317495) | > loss_duration: 1.5488665103912354 (1.5038850757810804) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.59491729736328 (26.115666325887045) | > grad_norm_1: tensor(6702.5288, device='cuda:0') (tensor(5821.5664, device='cuda:0')) | > current_lr_0: 0.00016879446040238502 | > current_lr_1: 0.00016879446040238502 | > step_time: 0.4859 (0.43735069698757595) | > loader_time: 0.009 (0.007591152191162109) --> EVAL PERFORMANCE | > avg_loader_time: 0.008689682248612527 (+3.295213403836303e-05) | > avg_step_time: 0.45180532294259945 (+0.0004411683955662937) | > avg_loss_disc: 2.6452172944243526 (+0.07275684786514436) | > avg_loss_disc_real_0: 0.17190524182793962 (+0.058320862647723196) | > avg_loss_disc_real_1: 0.22808539909376224 (-0.00916061267046861) | > avg_loss_disc_real_2: 0.23526019970296133 (+0.00636071302521396) | > avg_loss_disc_real_3: 0.23750381696392114 (-0.013944280609278714) | > avg_loss_disc_real_4: 0.24292822308103804 (-0.001750529232159459) | > avg_loss_disc_real_5: 0.24362106957066226 (-0.0006181077218391506) | > avg_loss_0: 2.6452172944243526 (+0.07275684786514436) | > avg_grad_norm_0: tensor(275.3804, device='cuda:0') (+tensor(58.9898, device='cuda:0')) | > avg_loss_gen: 2.1448106614636693 (-0.19490916292432336) | > avg_loss_kl: 2.0479532463449828 (+0.04034223522938474) | > avg_loss_feat: 3.752098734949676 (-0.5215727208365855) | > avg_loss_mel: 16.89298465889944 (+0.04713845588791443) | > avg_loss_duration: 1.5332328232241346 (+0.003116552258880745) | > avg_amp_scaler: 8.0 (-0.3380281690140894) | > avg_loss_1: 26.371080076190786 (-0.6258847142609092) | > avg_grad_norm_1: tensor(5893.6143, device='cuda:0') (+tensor(501.3320, device='cuda:0'))  > EPOCH: 1358/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:12:08)   --> TIME: 2025-02-25 22:12:20 -- STEP: 23/72 -- GLOBAL_STEP: 1097800 | > loss_disc: 2.6602516174316406 (2.572955857152524) | > loss_disc_real_0: 0.38031986355781555 (0.13752466764139093) | > loss_disc_real_1: 0.2559815049171448 (0.25142597374708753) | > loss_disc_real_2: 0.22873422503471375 (0.23760047814120416) | > loss_disc_real_3: 0.2311331331729889 (0.21953611270241116) | > loss_disc_real_4: 0.25022438168525696 (0.22004684546719427) | > loss_disc_real_5: 0.27441203594207764 (0.2209574845821961) | > loss_0: 2.6602516174316406 (2.572955857152524) | > grad_norm_0: tensor(782.9260, device='cuda:0') (tensor(321.7473, device='cuda:0')) | > loss_gen: 2.5895981788635254 (2.2511042460151334) | > loss_kl: 2.394984722137451 (2.3403135278950566) | > loss_feat: 3.8935718536376953 (4.281212340230527) | > loss_mel: 17.72484588623047 (18.71473561162534) | > loss_duration: 1.5497868061065674 (1.544281270193017) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.152788162231445 (29.13164694412895) | > grad_norm_1: tensor(3426.1050, device='cuda:0') (tensor(5484.4517, device='cuda:0')) | > current_lr_0: 0.00016877336109483472 | > current_lr_1: 0.00016877336109483472 | > step_time: 0.4284 (0.43359578174093494) | > loader_time: 0.0064 (0.006917383359826129) --> EVAL PERFORMANCE | > avg_loader_time: 0.008685857477322428 (-3.824771290098991e-06) | > avg_step_time: 0.45198919067920096 (+0.00018386773660150624) | > avg_loss_disc: 2.5618915255640595 (-0.08332576886029308) | > avg_loss_disc_real_0: 0.09467041794158204 (-0.07723482388635758) | > avg_loss_disc_real_1: 0.24856945977244563 (+0.020484060678683386) | > avg_loss_disc_real_2: 0.2137379729202096 (-0.02152222678275173) | > avg_loss_disc_real_3: 0.23405095808942553 (-0.003452858874495618) | > avg_loss_disc_real_4: 0.2296152954370203 (-0.01331292764401773) | > avg_loss_disc_real_5: 0.22922247375400973 (-0.014398595816652537) | > avg_loss_0: 2.5618915255640595 (-0.08332576886029308) | > avg_grad_norm_0: tensor(260.6146, device='cuda:0') (tensor(-14.7658, device='cuda:0')) | > avg_loss_gen: 2.1662439695546323 (+0.021433308090963) | > avg_loss_kl: 2.387762381996908 (+0.33980913565192505) | > avg_loss_feat: 4.113399283986695 (+0.3613005490370189) | > avg_loss_mel: 18.06647588165713 (+1.17349122275769) | > avg_loss_duration: 1.5724057916184546 (+0.03917296839432005) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.306287335677887 (+1.9352072594871004) | > avg_grad_norm_1: tensor(8087.4243, device='cuda:0') (+tensor(2193.8101, device='cuda:0'))  > EPOCH: 1359/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:12:45)   --> TIME: 2025-02-25 22:12:48 -- STEP: 1/72 -- GLOBAL_STEP: 1097850 | > loss_disc: 2.7005603313446045 (2.7005603313446045) | > loss_disc_real_0: 0.18787148594856262 (0.18787148594856262) | > loss_disc_real_1: 0.19616936147212982 (0.19616936147212982) | > loss_disc_real_2: 0.23565450310707092 (0.23565450310707092) | > loss_disc_real_3: 0.23233915865421295 (0.23233915865421295) | > loss_disc_real_4: 0.26719939708709717 (0.26719939708709717) | > loss_disc_real_5: 0.242002472281456 (0.242002472281456) | > loss_0: 2.7005603313446045 (2.7005603313446045) | > grad_norm_0: tensor(31.9157, device='cuda:0') (tensor(31.9157, device='cuda:0')) | > loss_gen: 1.9779767990112305 (1.9779767990112305) | > loss_kl: 2.0615243911743164 (2.0615243911743164) | > loss_feat: 3.9379940032958984 (3.9379940032958984) | > loss_mel: 16.963058471679688 (16.963058471679688) | > loss_duration: 1.4956222772598267 (1.4956222772598267) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.43617820739746 (26.43617820739746) | > grad_norm_1: tensor(5756.6851, device='cuda:0') (tensor(5756.6851, device='cuda:0')) | > current_lr_0: 0.00016875226442469786 | > current_lr_1: 0.00016875226442469786 | > step_time: 0.4338 (0.43384885787963867) | > loader_time: 0.0068 (0.006780862808227539)  --> TIME: 2025-02-25 22:13:10 -- STEP: 51/72 -- GLOBAL_STEP: 1097900 | > loss_disc: 2.7065770626068115 (2.544247229894003) | > loss_disc_real_0: 0.07757715880870819 (0.10227485987193442) | > loss_disc_real_1: 0.2189684510231018 (0.22430180655974968) | > loss_disc_real_2: 0.22785206139087677 (0.23378872871398926) | > loss_disc_real_3: 0.2240339070558548 (0.22911523194874034) | > loss_disc_real_4: 0.22762088477611542 (0.23962189344798818) | > loss_disc_real_5: 0.24301394820213318 (0.2498775615411647) | > loss_0: 2.7065770626068115 (2.544247229894003) | > grad_norm_0: tensor(332.1993, device='cuda:0') (tensor(239.8378, device='cuda:0')) | > loss_gen: 1.873306155204773 (2.2061302708644486) | > loss_kl: 1.8500027656555176 (2.0074913992601284) | > loss_feat: 3.7347495555877686 (4.134177404291489) | > loss_mel: 15.587654113769531 (16.77348417394301) | > loss_duration: 1.5713591575622559 (1.5158681822758096) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.6170711517334 (26.637151231952743) | > grad_norm_1: tensor(14876.3213, device='cuda:0') (tensor(9590.8057, device='cuda:0')) | > current_lr_0: 0.00016875226442469786 | > current_lr_1: 0.00016875226442469786 | > step_time: 0.4568 (0.43860577602012485) | > loader_time: 0.009 (0.007566512799730488) --> EVAL PERFORMANCE | > avg_loader_time: 0.008443295116155922 (-0.0002425623611665064) | > avg_step_time: 0.44928975172445806 (-0.0026994389547428943) | > avg_loss_disc: 2.528190894865654 (-0.033700630698405565) | > avg_loss_disc_real_0: 0.09485673925406496 (+0.00018632131248291772) | > avg_loss_disc_real_1: 0.22331434173483244 (-0.025255118037613183) | > avg_loss_disc_real_2: 0.233698984896633 (+0.019961011976423387) | > avg_loss_disc_real_3: 0.22875042469568654 (-0.005300533393738982) | > avg_loss_disc_real_4: 0.2387941603509473 (+0.009178864913926998) | > avg_loss_disc_real_5: 0.24887682707376887 (+0.01965435331975915) | > avg_loss_0: 2.528190894865654 (-0.033700630698405565) | > avg_grad_norm_0: tensor(247.9807, device='cuda:0') (tensor(-12.6339, device='cuda:0')) | > avg_loss_gen: 2.2097912942859477 (+0.043547324731315395) | > avg_loss_kl: 2.022852351967718 (-0.36491003002918987) | > avg_loss_feat: 4.119623375610567 (+0.006224091623872496) | > avg_loss_mel: 16.737993200060345 (-1.3284826815967854) | > avg_loss_duration: 1.533336061826894 (-0.03906972979156054) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.623596110814056 (-1.6826912248638308) | > avg_grad_norm_1: tensor(9614.5713, device='cuda:0') (+tensor(1527.1470, device='cuda:0'))  > EPOCH: 1360/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:13:22)   --> TIME: 2025-02-25 22:13:37 -- STEP: 29/72 -- GLOBAL_STEP: 1097950 | > loss_disc: 2.2502496242523193 (2.460375021243917) | > loss_disc_real_0: 0.08729073405265808 (0.07864378379850553) | > loss_disc_real_1: 0.20002397894859314 (0.21064559190437712) | > loss_disc_real_2: 0.16105009615421295 (0.22514411498760356) | > loss_disc_real_3: 0.20649634301662445 (0.22518831851153537) | > loss_disc_real_4: 0.21183493733406067 (0.23533552936438856) | > loss_disc_real_5: 0.2504400312900543 (0.2458055928863328) | > loss_0: 2.2502496242523193 (2.460375021243917) | > grad_norm_0: tensor(300.1077, device='cuda:0') (tensor(296.1385, device='cuda:0')) | > loss_gen: 2.55230450630188 (2.267994272297827) | > loss_kl: 1.9105535745620728 (1.905654002880228) | > loss_feat: 5.002377033233643 (4.413843598859064) | > loss_mel: 17.427663803100586 (16.972948600505962) | > loss_duration: 1.5025804042816162 (1.4862970730354046) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.39548110961914 (27.046737605127795) | > grad_norm_1: tensor(7572.3687, device='cuda:0') (tensor(9702.8145, device='cuda:0')) | > current_lr_0: 0.00016873117039164476 | > current_lr_1: 0.00016873117039164476 | > step_time: 0.4342 (0.431682825088501) | > loader_time: 0.0076 (0.0068362827958731815) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577393813872005 (+0.00013409869771608372) | > avg_step_time: 0.44955421501482035 (+0.00026446329036228233) | > avg_loss_disc: 2.4877099957264646 (-0.0404808991391894) | > avg_loss_disc_real_0: 0.08322991971188869 (-0.011626819542176273) | > avg_loss_disc_real_1: 0.21841971933002202 (-0.0048946224048104225) | > avg_loss_disc_real_2: 0.22004985620438217 (-0.013649128692250823) | > avg_loss_disc_real_3: 0.22907401684304357 (+0.0003235921473570269) | > avg_loss_disc_real_4: 0.2372145694746098 (-0.0015795908763374944) | > avg_loss_disc_real_5: 0.24590114514592668 (-0.0029756819278421998) | > avg_loss_0: 2.4877099957264646 (-0.0404808991391894) | > avg_grad_norm_0: tensor(288.4233, device='cuda:0') (+tensor(40.4427, device='cuda:0')) | > avg_loss_gen: 2.2465878708261826 (+0.03679657654023494) | > avg_loss_kl: 1.991466784141433 (-0.03138556782628488) | > avg_loss_feat: 4.272157350056606 (+0.15253397444603856) | > avg_loss_mel: 16.72422946338922 (-0.013763736671123183) | > avg_loss_duration: 1.5270421253123754 (-0.006293936514518705) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.761483420788405 (+0.1378873099743494) | > avg_grad_norm_1: tensor(9669.7773, device='cuda:0') (+tensor(55.2061, device='cuda:0'))  > EPOCH: 1361/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:13:58)   --> TIME: 2025-02-25 22:14:03 -- STEP: 7/72 -- GLOBAL_STEP: 1098000 | > loss_disc: 2.7638542652130127 (2.5082170963287354) | > loss_disc_real_0: 0.10112124681472778 (0.07573513420564788) | > loss_disc_real_1: 0.210405632853508 (0.19764293943132674) | > loss_disc_real_2: 0.18115760385990143 (0.21324919164180756) | > loss_disc_real_3: 0.20493513345718384 (0.21867154112883977) | > loss_disc_real_4: 0.2196737825870514 (0.22262103429862432) | > loss_disc_real_5: 0.2150581330060959 (0.2406582576887948) | > loss_0: 2.7638542652130127 (2.5082170963287354) | > grad_norm_0: tensor(641.7037, device='cuda:0') (tensor(312.4980, device='cuda:0')) | > loss_gen: 1.6079521179199219 (2.1225310393742154) | > loss_kl: 2.067211389541626 (1.895287105015346) | > loss_feat: 3.8030993938446045 (4.1994616985321045) | > loss_mel: 16.89715003967285 (16.306440625871932) | > loss_duration: 1.4415433406829834 (1.4717239482062203) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.81695556640625 (25.99544416155134) | > grad_norm_1: tensor(6736.3765, device='cuda:0') (tensor(9454.3018, device='cuda:0')) | > current_lr_0: 0.0001687100789953458 | > current_lr_1: 0.0001687100789953458 | > step_time: 0.4418 (0.4301838534218924) | > loader_time: 0.0069 (0.006362778799874442)  --> TIME: 2025-02-25 22:14:26 -- STEP: 57/72 -- GLOBAL_STEP: 1098050 | > loss_disc: 3.0548367500305176 (2.550181376306634) | > loss_disc_real_0: 0.13660117983818054 (0.09339520936472376) | > loss_disc_real_1: 0.21441565454006195 (0.2184712008426064) | > loss_disc_real_2: 0.21301674842834473 (0.2147137854705777) | > loss_disc_real_3: 0.264395534992218 (0.2294576055648034) | > loss_disc_real_4: 0.2562834918498993 (0.231951172153155) | > loss_disc_real_5: 0.23897334933280945 (0.24575655946606084) | > loss_0: 3.0548367500305176 (2.550181376306634) | > grad_norm_0: tensor(35.3430, device='cuda:0') (tensor(306.0017, device='cuda:0')) | > loss_gen: 1.7537952661514282 (2.1949752192748213) | > loss_kl: 2.0382540225982666 (1.9801131758773536) | > loss_feat: 3.3589320182800293 (4.245401210952223) | > loss_mel: 15.520133018493652 (16.455382815578528) | > loss_duration: 1.571993112564087 (1.5069161026101363) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.24310874938965 (26.382788607948708) | > grad_norm_1: tensor(2177.6394, device='cuda:0') (tensor(10062.3516, device='cuda:0')) | > current_lr_0: 0.0001687100789953458 | > current_lr_1: 0.0001687100789953458 | > step_time: 0.4521 (0.4398868418576425) | > loader_time: 0.0095 (0.00781526481896116) --> EVAL PERFORMANCE | > avg_loader_time: 0.008475978609541772 (-0.00010141520433023382) | > avg_step_time: 0.4493323044038155 (-0.00022191061100484966) | > avg_loss_disc: 2.5544620500483983 (+0.06675205432193376) | > avg_loss_disc_real_0: 0.10411463751339578 (+0.020884717801507094) | > avg_loss_disc_real_1: 0.22103237874910864 (+0.002612659419086616) | > avg_loss_disc_real_2: 0.21657021763459058 (-0.003479638569791582) | > avg_loss_disc_real_3: 0.22664412940052195 (-0.002429887442521622) | > avg_loss_disc_real_4: 0.23368371372491542 (-0.0035308557496943893) | > avg_loss_disc_real_5: 0.243970168728224 (-0.001930976417702679) | > avg_loss_0: 2.5544620500483983 (+0.06675205432193376) | > avg_grad_norm_0: tensor(295.6774, device='cuda:0') (+tensor(7.2540, device='cuda:0')) | > avg_loss_gen: 2.189477433621043 (-0.05711043720513942) | > avg_loss_kl: 2.012040658735893 (+0.02057387459446014) | > avg_loss_feat: 4.160033874108758 (-0.11212347594784777) | > avg_loss_mel: 16.48536183800496 (-0.23886762538426254) | > avg_loss_duration: 1.5214034637934726 (-0.005638661518902799) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.368317295128197 (-0.39316612566020837) | > avg_grad_norm_1: tensor(9574.0234, device='cuda:0') (tensor(-95.7539, device='cuda:0'))  > EPOCH: 1362/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:14:35)   --> TIME: 2025-02-25 22:14:53 -- STEP: 35/72 -- GLOBAL_STEP: 1098100 | > loss_disc: 2.3723466396331787 (2.5464453901563373) | > loss_disc_real_0: 0.09357884526252747 (0.10065566514219557) | > loss_disc_real_1: 0.20745335519313812 (0.21529999205044337) | > loss_disc_real_2: 0.23815278708934784 (0.22592691268239704) | > loss_disc_real_3: 0.24065737426280975 (0.2290904415505273) | > loss_disc_real_4: 0.2353965789079666 (0.23290673196315764) | > loss_disc_real_5: 0.24186524748802185 (0.24288952052593232) | > loss_0: 2.3723466396331787 (2.5464453901563373) | > grad_norm_0: tensor(243.7160, device='cuda:0') (tensor(322.9220, device='cuda:0')) | > loss_gen: 2.5250132083892822 (2.159435885293143) | > loss_kl: 1.9341217279434204 (1.8981146097183228) | > loss_feat: 4.609529972076416 (4.045801952907017) | > loss_mel: 17.812076568603516 (16.420739146641328) | > loss_duration: 1.5481598377227783 (1.4875231095722743) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.42890167236328 (26.011614717756) | > grad_norm_1: tensor(5475.0479, device='cuda:0') (tensor(10224.8076, device='cuda:0')) | > current_lr_0: 0.00016868899023547137 | > current_lr_1: 0.00016868899023547137 | > step_time: 0.4438 (0.43689918518066406) | > loader_time: 0.0082 (0.007172925131661552) --> EVAL PERFORMANCE | > avg_loader_time: 0.008520965844812526 (+4.498723527075407e-05) | > avg_step_time: 0.4503950669731892 (+0.0010627625693737097) | > avg_loss_disc: 2.5648581545117874 (+0.010396104463389122) | > avg_loss_disc_real_0: 0.09214217829662309 (-0.011972459216772688) | > avg_loss_disc_real_1: 0.22203239336819716 (+0.001000014619088524) | > avg_loss_disc_real_2: 0.23697987996356587 (+0.020409662328975287) | > avg_loss_disc_real_3: 0.23397986087161052 (+0.007335731471088569) | > avg_loss_disc_real_4: 0.23618228082925502 (+0.0024985671043395996) | > avg_loss_disc_real_5: 0.2443414565543054 (+0.00037128782608139566) | > avg_loss_0: 2.5648581545117874 (+0.010396104463389122) | > avg_grad_norm_0: tensor(319.7604, device='cuda:0') (+tensor(24.0830, device='cuda:0')) | > avg_loss_gen: 2.1621011596330453 (-0.027376273987997912) | > avg_loss_kl: 1.9796336738156601 (-0.032406984920233084) | > avg_loss_feat: 4.031939540110843 (-0.12809433399791548) | > avg_loss_mel: 16.43077467528867 (-0.05458716271628816) | > avg_loss_duration: 1.5232756087477777 (+0.0018721449543050994) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.127724553497746 (-0.24059274163045075) | > avg_grad_norm_1: tensor(9598.1543, device='cuda:0') (+tensor(24.1309, device='cuda:0'))  > EPOCH: 1363/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:15:12)   --> TIME: 2025-02-25 22:15:20 -- STEP: 13/72 -- GLOBAL_STEP: 1098150 | > loss_disc: 2.576961040496826 (2.6165794959435096) | > loss_disc_real_0: 0.11640132963657379 (0.11936292596734487) | > loss_disc_real_1: 0.2143484205007553 (0.26326820942071766) | > loss_disc_real_2: 0.21419839560985565 (0.22218212256064782) | > loss_disc_real_3: 0.22671222686767578 (0.2346277695435744) | > loss_disc_real_4: 0.19565162062644958 (0.22228099749638483) | > loss_disc_real_5: 0.20681235194206238 (0.24283951406295484) | > loss_0: 2.576961040496826 (2.6165794959435096) | > grad_norm_0: tensor(421.6110, device='cuda:0') (tensor(411.9554, device='cuda:0')) | > loss_gen: 1.9255568981170654 (2.1666655173668494) | > loss_kl: 2.03338623046875 (1.8672620883354774) | > loss_feat: 4.068734169006348 (4.041675787705642) | > loss_mel: 16.397672653198242 (16.646142886235165) | > loss_duration: 1.433030128479004 (1.4720012683134813) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.858379364013672 (26.19374788724459) | > grad_norm_1: tensor(10464.5459, device='cuda:0') (tensor(9164.1445, device='cuda:0')) | > current_lr_0: 0.00016866790411169193 | > current_lr_1: 0.00016866790411169193 | > step_time: 0.4608 (0.42671438363882214) | > loader_time: 0.0057 (0.006563846881573017)  --> TIME: 2025-02-25 22:15:43 -- STEP: 63/72 -- GLOBAL_STEP: 1098200 | > loss_disc: 2.412950038909912 (2.636472902600728) | > loss_disc_real_0: 0.09693518280982971 (0.16132202703091827) | > loss_disc_real_1: 0.2696387469768524 (0.240739096843061) | > loss_disc_real_2: 0.25494199991226196 (0.23106795360171606) | > loss_disc_real_3: 0.27086812257766724 (0.2326951902064066) | > loss_disc_real_4: 0.25773561000823975 (0.23720029469520326) | > loss_disc_real_5: 0.26283562183380127 (0.24521441544805253) | > loss_0: 2.412950038909912 (2.636472902600728) | > grad_norm_0: tensor(266.1635, device='cuda:0') (tensor(365.7861, device='cuda:0')) | > loss_gen: 2.4251608848571777 (2.1537324076607116) | > loss_kl: 2.1565709114074707 (2.008252715307569) | > loss_feat: 4.2413249015808105 (3.8358579022543764) | > loss_mel: 15.895949363708496 (16.551743931240512) | > loss_duration: 1.561434268951416 (1.5177124511627924) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.280441284179688 (26.067299464392285) | > grad_norm_1: tensor(4944.4707, device='cuda:0') (tensor(8731.8389, device='cuda:0')) | > current_lr_0: 0.00016866790411169193 | > current_lr_1: 0.00016866790411169193 | > step_time: 0.4674 (0.4403486100454179) | > loader_time: 0.0113 (0.008096259737771654) --> EVAL PERFORMANCE | > avg_loader_time: 0.00846493747872366 (-5.602836608886545e-05) | > avg_step_time: 0.4477781074147829 (-0.0026169595584062955) | > avg_loss_disc: 2.632684539741194 (+0.06782638522940676) | > avg_loss_disc_real_0: 0.15107091175208623 (+0.058928733455463136) | > avg_loss_disc_real_1: 0.2511839053370583 (+0.02915151196886115) | > avg_loss_disc_real_2: 0.23212150494817277 (-0.004858375015393102) | > avg_loss_disc_real_3: 0.23500418977838167 (+0.0010243289067711492) | > avg_loss_disc_real_4: 0.237947543532076 (+0.0017652627028209855) | > avg_loss_disc_real_5: 0.2448974501498988 (+0.0005559935955934125) | > avg_loss_0: 2.632684539741194 (+0.06782638522940676) | > avg_grad_norm_0: tensor(352.6574, device='cuda:0') (+tensor(32.8970, device='cuda:0')) | > avg_loss_gen: 2.1764971974869853 (+0.014396037853940058) | > avg_loss_kl: 2.031876846098564 (+0.05224317228290398) | > avg_loss_feat: 3.8506665229797354 (-0.18127301713110722) | > avg_loss_mel: 16.585528655790952 (+0.1547539805022815) | > avg_loss_duration: 1.5270507100602269 (+0.0037751013124491806) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.171620033156703 (+0.043895479658957015) | > avg_grad_norm_1: tensor(8597.4043, device='cuda:0') (tensor(-1000.7500, device='cuda:0'))  > EPOCH: 1364/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:15:48)   --> TIME: 2025-02-25 22:16:09 -- STEP: 41/72 -- GLOBAL_STEP: 1098250 | > loss_disc: 2.6797854900360107 (2.565148400097359) | > loss_disc_real_0: 0.04531188681721687 (0.08869128492547244) | > loss_disc_real_1: 0.21481642127037048 (0.2469124314261646) | > loss_disc_real_2: 0.2514556646347046 (0.23585477725761692) | > loss_disc_real_3: 0.2465582638978958 (0.23408184473107502) | > loss_disc_real_4: 0.2294795960187912 (0.23878314545968685) | > loss_disc_real_5: 0.24020208418369293 (0.2460297291598669) | > loss_0: 2.6797854900360107 (2.565148400097359) | > grad_norm_0: tensor(399.6729, device='cuda:0') (tensor(369.4251, device='cuda:0')) | > loss_gen: 1.8718758821487427 (2.1971116705638605) | > loss_kl: 1.8006867170333862 (1.950424272839616) | > loss_feat: 3.8942291736602783 (4.076374943663434) | > loss_mel: 16.565763473510742 (16.732217695654892) | > loss_duration: 1.5223027467727661 (1.4996526328528799) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.654857635498047 (26.455780959710843) | > grad_norm_1: tensor(15843.2197, device='cuda:0') (tensor(11071.6670, device='cuda:0')) | > current_lr_0: 0.00016864682062367795 | > current_lr_1: 0.00016864682062367795 | > step_time: 0.4398 (0.43532274409038263) | > loader_time: 0.0087 (0.00729443387287419) --> EVAL PERFORMANCE | > avg_loader_time: 0.008559569506578043 (+9.463202785438274e-05) | > avg_step_time: 0.4506875427675919 (+0.0029094353528089645) | > avg_loss_disc: 2.567965608247569 (-0.0647189314936254) | > avg_loss_disc_real_0: 0.09048353613052569 (-0.06058737562156054) | > avg_loss_disc_real_1: 0.23792669030142502 (-0.01325721503563329) | > avg_loss_disc_real_2: 0.2406343164998041 (+0.008512811551631333) | > avg_loss_disc_real_3: 0.23563546949708966 (+0.0006312797187079944) | > avg_loss_disc_real_4: 0.23848147522395766 (+0.0005339316918816506) | > avg_loss_disc_real_5: 0.24679221884465555 (+0.0018947686947567433) | > avg_loss_0: 2.567965608247569 (-0.0647189314936254) | > avg_grad_norm_0: tensor(376.4389, device='cuda:0') (+tensor(23.7815, device='cuda:0')) | > avg_loss_gen: 2.2010690091361456 (+0.024571811649160225) | > avg_loss_kl: 1.992896016214935 (-0.038980829883629164) | > avg_loss_feat: 4.047032974135707 (+0.1963664511559715) | > avg_loss_mel: 16.62357468672203 (+0.03804603093107772) | > avg_loss_duration: 1.5272108383581673 (+0.0001601282979404317) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.391783351629552 (+0.2201633184728493) | > avg_grad_norm_1: tensor(10905.7324, device='cuda:0') (+tensor(2308.3281, device='cuda:0'))  > EPOCH: 1365/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:16:25)   --> TIME: 2025-02-25 22:16:36 -- STEP: 19/72 -- GLOBAL_STEP: 1098300 | > loss_disc: 2.485157012939453 (2.4851124663101998) | > loss_disc_real_0: 0.11521226167678833 (0.07061080713021128) | > loss_disc_real_1: 0.2376473844051361 (0.21305378957798607) | > loss_disc_real_2: 0.21421433985233307 (0.2200662650560078) | > loss_disc_real_3: 0.23286095261573792 (0.23487334110234914) | > loss_disc_real_4: 0.24357672035694122 (0.2417970409518794) | > loss_disc_real_5: 0.2454787790775299 (0.24533616084801524) | > loss_0: 2.485157012939453 (2.4851124663101998) | > grad_norm_0: tensor(242.5562, device='cuda:0') (tensor(281.2821, device='cuda:0')) | > loss_gen: 2.3294930458068848 (2.2576787220804317) | > loss_kl: 1.9542306661605835 (1.9031545300232737) | > loss_feat: 3.864457607269287 (4.33352962293123) | > loss_mel: 15.919611930847168 (16.502955988833783) | > loss_duration: 1.4753124713897705 (1.4774858135926097) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.543106079101562 (26.474804426494398) | > grad_norm_1: tensor(6660.8022, device='cuda:0') (tensor(10495.7275, device='cuda:0')) | > current_lr_0: 0.00016862573977109997 | > current_lr_1: 0.00016862573977109997 | > step_time: 0.4407 (0.4561905108000103) | > loader_time: 0.0069 (0.006541264684576737)  --> TIME: 2025-02-25 22:16:59 -- STEP: 69/72 -- GLOBAL_STEP: 1098350 | > loss_disc: 2.6387181282043457 (2.501143275827601) | > loss_disc_real_0: 0.08719571679830551 (0.08317228919569995) | > loss_disc_real_1: 0.20277726650238037 (0.220400202749432) | > loss_disc_real_2: 0.21494930982589722 (0.22716508676176486) | > loss_disc_real_3: 0.22381311655044556 (0.23213366991367893) | > loss_disc_real_4: 0.23536688089370728 (0.24369334937005804) | > loss_disc_real_5: 0.2551836669445038 (0.24532103365746097) | > loss_0: 2.6387181282043457 (2.501143275827601) | > grad_norm_0: tensor(273.0229, device='cuda:0') (tensor(322.0753, device='cuda:0')) | > loss_gen: 1.9477734565734863 (2.2639102814854053) | > loss_kl: 2.1804776191711426 (1.9929304865823276) | > loss_feat: 3.854994535446167 (4.231142911358158) | > loss_mel: 16.7449951171875 (16.542431513468426) | > loss_duration: 1.6280169486999512 (1.5227268968803296) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.356258392333984 (26.553141994752746) | > grad_norm_1: tensor(10026.1953, device='cuda:0') (tensor(10420.5811, device='cuda:0')) | > current_lr_0: 0.00016862573977109997 | > current_lr_1: 0.00016862573977109997 | > step_time: 0.5062 (0.4540511663409247) | > loader_time: 0.0125 (0.008448016816291254) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850150618754642 (-5.806331903162286e-05) | > avg_step_time: 0.4552660794325278 (+0.004578536664935906) | > avg_loss_disc: 2.5034320421621827 (-0.06453356608538607) | > avg_loss_disc_real_0: 0.08292428063045083 (-0.007559255500074857) | > avg_loss_disc_real_1: 0.2208309744445371 (-0.017095715856887933) | > avg_loss_disc_real_2: 0.22602215283353563 (-0.014612163666268468) | > avg_loss_disc_real_3: 0.2324407803760448 (-0.003194689121044869) | > avg_loss_disc_real_4: 0.2433636465542753 (+0.004882171330317653) | > avg_loss_disc_real_5: 0.24513533837358717 (-0.0016568804710683782) | > avg_loss_0: 2.5034320421621827 (-0.06453356608538607) | > avg_grad_norm_0: tensor(326.4703, device='cuda:0') (tensor(-49.9686, device='cuda:0')) | > avg_loss_gen: 2.2622368151033423 (+0.06116780596719673) | > avg_loss_kl: 1.9924656992227259 (-0.0004303169922090788) | > avg_loss_feat: 4.2287845678732445 (+0.18175159373753758) | > avg_loss_mel: 16.51750235490396 (-0.10607233181806919) | > avg_loss_duration: 1.5236714944033556 (-0.0035393439548117023) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.524660835803395 (+0.13287748417384293) | > avg_grad_norm_1: tensor(10433.5762, device='cuda:0') (tensor(-472.1562, device='cuda:0'))  > EPOCH: 1366/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:17:02)   --> TIME: 2025-02-25 22:17:25 -- STEP: 47/72 -- GLOBAL_STEP: 1098400 | > loss_disc: 2.6691081523895264 (2.5134224333661663) | > loss_disc_real_0: 0.0633205771446228 (0.0909379210243834) | > loss_disc_real_1: 0.19808654487133026 (0.21860869989750234) | > loss_disc_real_2: 0.2367817461490631 (0.22799359230284996) | > loss_disc_real_3: 0.22820627689361572 (0.22830283990565767) | > loss_disc_real_4: 0.2486991584300995 (0.23539703450304397) | > loss_disc_real_5: 0.2250029742717743 (0.24279008869160998) | > loss_0: 2.6691081523895264 (2.5134224333661663) | > grad_norm_0: tensor(422.7658, device='cuda:0') (tensor(375.6540, device='cuda:0')) | > loss_gen: 1.9991787672042847 (2.2529685091465073) | > loss_kl: 2.1807687282562256 (1.9753391438342156) | > loss_feat: 4.357775688171387 (4.177122902362905) | > loss_mel: 16.818065643310547 (16.361769615335664) | > loss_duration: 1.5532995462417603 (1.5018358864682788) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.909086227416992 (26.2690361104113) | > grad_norm_1: tensor(14534.7871, device='cuda:0') (tensor(11579.2695, device='cuda:0')) | > current_lr_0: 0.00016860466155362857 | > current_lr_1: 0.00016860466155362857 | > step_time: 0.45 (0.43828400652459326) | > loader_time: 0.0086 (0.007617184456358565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573525388475875 (+7.201920092945492e-05) | > avg_step_time: 0.4493605015983045 (-0.005905577834223297) | > avg_loss_disc: 2.517141970110611 (+0.013709927948428291) | > avg_loss_disc_real_0: 0.09382519566676989 (+0.010900915036319056) | > avg_loss_disc_real_1: 0.21987997499150289 (-0.0009509994530342014) | > avg_loss_disc_real_2: 0.22529023633876316 (-0.0007319164947724721) | > avg_loss_disc_real_3: 0.2283684451815108 (-0.0040723351945339825) | > avg_loss_disc_real_4: 0.2353336945386 (-0.008029952015675301) | > avg_loss_disc_real_5: 0.24222913279499805 (-0.002906205578589116) | > avg_loss_0: 2.517141970110611 (+0.013709927948428291) | > avg_grad_norm_0: tensor(357.9111, device='cuda:0') (+tensor(31.4408, device='cuda:0')) | > avg_loss_gen: 2.261746122803487 (-0.000490692299855322) | > avg_loss_kl: 2.0108723103160586 (+0.018406611093332748) | > avg_loss_feat: 4.193170231832588 (-0.03561433604065645) | > avg_loss_mel: 16.43122601844895 (-0.0862763364550112) | > avg_loss_duration: 1.524925210106541 (+0.0012537157031853319) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.42193993044571 (-0.10272090535768541) | > avg_grad_norm_1: tensor(11083.7041, device='cuda:0') (+tensor(650.1279, device='cuda:0'))  > EPOCH: 1367/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:17:38)   --> TIME: 2025-02-25 22:17:52 -- STEP: 25/72 -- GLOBAL_STEP: 1098450 | > loss_disc: 2.354860782623291 (2.549714670181274) | > loss_disc_real_0: 0.1997683197259903 (0.09235223323106766) | > loss_disc_real_1: 0.20669832825660706 (0.2204568886756897) | > loss_disc_real_2: 0.21371184289455414 (0.2056579875946045) | > loss_disc_real_3: 0.21695640683174133 (0.22023678123950957) | > loss_disc_real_4: 0.22341059148311615 (0.2280240100622177) | > loss_disc_real_5: 0.25152215361595154 (0.24358343839645386) | > loss_0: 2.354860782623291 (2.549714670181274) | > grad_norm_0: tensor(547.6101, device='cuda:0') (tensor(335.1200, device='cuda:0')) | > loss_gen: 2.8106634616851807 (2.233424596786499) | > loss_kl: 1.940269112586975 (2.022810640335083) | > loss_feat: 4.81886625289917 (4.474783735275268) | > loss_mel: 16.459964752197266 (16.79226142883301) | > loss_duration: 1.4924345016479492 (1.4831083345413207) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.522197723388672 (27.006388702392577) | > grad_norm_1: tensor(10185.8184, device='cuda:0') (tensor(10533.3672, device='cuda:0')) | > current_lr_0: 0.00016858358597093435 | > current_lr_1: 0.00016858358597093435 | > step_time: 0.4205 (0.43373879432678225) | > loader_time: 0.0063 (0.006859359741210938) --> EVAL PERFORMANCE | > avg_loader_time: 0.00853204391372036 (-4.148147475551561e-05) | > avg_step_time: 0.45241604388599665 (+0.00305554228769217) | > avg_loss_disc: 2.5891226647605357 (+0.07198069464992463) | > avg_loss_disc_real_0: 0.1412348151731659 (+0.04740961950639602) | > avg_loss_disc_real_1: 0.22300917707698442 (+0.003129202085481536) | > avg_loss_disc_real_2: 0.21675367695345005 (-0.008536559385313114) | > avg_loss_disc_real_3: 0.23795688949840169 (+0.009588444316890876) | > avg_loss_disc_real_4: 0.2412421443092991 (+0.005908449770699098) | > avg_loss_disc_real_5: 0.24896952929631086 (+0.006740396501312806) | > avg_loss_0: 2.5891226647605357 (+0.07198069464992463) | > avg_grad_norm_0: tensor(336.1899, device='cuda:0') (tensor(-21.7212, device='cuda:0')) | > avg_loss_gen: 2.3113794125301736 (+0.04963328972668668) | > avg_loss_kl: 2.06034298681877 (+0.04947067650271153) | > avg_loss_feat: 4.249771826703784 (+0.05660159487119554) | > avg_loss_mel: 16.68653082511794 (+0.2553048066689918) | > avg_loss_duration: 1.527707687565978 (+0.0027824774594371604) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.835732634638397 (+0.4137927041926872) | > avg_grad_norm_1: tensor(9628.4102, device='cuda:0') (tensor(-1455.2939, device='cuda:0'))  > EPOCH: 1368/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:18:15)   --> TIME: 2025-02-25 22:18:18 -- STEP: 3/72 -- GLOBAL_STEP: 1098500 | > loss_disc: 2.491201639175415 (2.7179953257242837) | > loss_disc_real_0: 0.1273995339870453 (0.32197325428326923) | > loss_disc_real_1: 0.24827124178409576 (0.2277410626411438) | > loss_disc_real_2: 0.23333890736103058 (0.23058452208836874) | > loss_disc_real_3: 0.20580993592739105 (0.21703071892261505) | > loss_disc_real_4: 0.2799893915653229 (0.27492010593414307) | > loss_disc_real_5: 0.2851981818675995 (0.2877030670642853) | > loss_0: 2.491201639175415 (2.7179953257242837) | > grad_norm_0: tensor(88.7901, device='cuda:0') (tensor(420.6952, device='cuda:0')) | > loss_gen: 2.4584317207336426 (2.2774442036946616) | > loss_kl: 2.5235183238983154 (2.318842967351278) | > loss_feat: 4.444302558898926 (4.054844617843628) | > loss_mel: 18.431926727294922 (18.513621648152668) | > loss_duration: 1.5689928531646729 (1.539106289545695) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.42717170715332 (28.70385996500651) | > grad_norm_1: tensor(3650.7922, device='cuda:0') (tensor(2732.8760, device='cuda:0')) | > current_lr_0: 0.00016856251302268797 | > current_lr_1: 0.00016856251302268797 | > step_time: 0.4208 (0.4277242024739583) | > loader_time: 0.0117 (0.007970889409383139)  --> TIME: 2025-02-25 22:18:42 -- STEP: 53/72 -- GLOBAL_STEP: 1098550 | > loss_disc: 2.8324594497680664 (2.649940054371672) | > loss_disc_real_0: 0.19820456206798553 (0.15159753358589032) | > loss_disc_real_1: 0.3676256835460663 (0.2298981987361638) | > loss_disc_real_2: 0.24121502041816711 (0.22553631944476432) | > loss_disc_real_3: 0.27441397309303284 (0.23656561520864378) | > loss_disc_real_4: 0.28067952394485474 (0.24408841611079446) | > loss_disc_real_5: 0.23925232887268066 (0.24566830216713678) | > loss_0: 2.8324594497680664 (2.649940054371672) | > grad_norm_0: tensor(146.2451, device='cuda:0') (tensor(293.2997, device='cuda:0')) | > loss_gen: 2.4353487491607666 (2.17933742055353) | > loss_kl: 2.101041078567505 (2.181308638374761) | > loss_feat: 3.203873634338379 (3.9776671112708324) | > loss_mel: 15.929019927978516 (17.069261946768126) | > loss_duration: 1.5707415342330933 (1.5299498697496807) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.24002456665039 (26.937524939483065) | > grad_norm_1: tensor(5241.7397, device='cuda:0') (tensor(7726.9976, device='cuda:0')) | > current_lr_0: 0.00016856251302268797 | > current_lr_1: 0.00016856251302268797 | > step_time: 0.4471 (0.4487674146328332) | > loader_time: 0.0089 (0.0076784772692986255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008506042856565664 (-2.6001057154695362e-05) | > avg_step_time: 0.45775396723142814 (+0.005337923345431483) | > avg_loss_disc: 2.6858078056657817 (+0.09668514090524605) | > avg_loss_disc_real_0: 0.1819736641687407 (+0.04073884899557478) | > avg_loss_disc_real_1: 0.23474442171798626 (+0.01173524464100184) | > avg_loss_disc_real_2: 0.22898370909019256 (+0.012230032136742508) | > avg_loss_disc_real_3: 0.2366714467045287 (-0.0012854427938729929) | > avg_loss_disc_real_4: 0.2410025800197897 (-0.0002395642895094141) | > avg_loss_disc_real_5: 0.24636771498431628 (-0.002601814311994577) | > avg_loss_0: 2.6858078056657817 (+0.09668514090524605) | > avg_grad_norm_0: tensor(311.5784, device='cuda:0') (tensor(-24.6115, device='cuda:0')) | > avg_loss_gen: 2.1640196870750095 (-0.14735972545516418) | > avg_loss_kl: 2.1883892257448663 (+0.12804623892609612) | > avg_loss_feat: 3.8868835274602325 (-0.36288829924355115) | > avg_loss_mel: 16.962550915462856 (+0.2760200903449146) | > avg_loss_duration: 1.5425209176372472 (+0.014813230071269112) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.74436426834321 (-0.09136836629518541) | > avg_grad_norm_1: tensor(7045.0796, device='cuda:0') (tensor(-2583.3306, device='cuda:0'))  > EPOCH: 1369/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:18:52)   --> TIME: 2025-02-25 22:19:08 -- STEP: 31/72 -- GLOBAL_STEP: 1098600 | > loss_disc: 2.6892709732055664 (2.621497500327326) | > loss_disc_real_0: 0.12648680806159973 (0.11880783544432733) | > loss_disc_real_1: 0.23459891974925995 (0.234588467786389) | > loss_disc_real_2: 0.2422192245721817 (0.23461504136362382) | > loss_disc_real_3: 0.265781044960022 (0.22085756111529567) | > loss_disc_real_4: 0.2917431890964508 (0.26207727145764137) | > loss_disc_real_5: 0.23860134184360504 (0.23331651476121718) | > loss_0: 2.6892709732055664 (2.621497500327326) | > grad_norm_0: tensor(235.7685, device='cuda:0') (tensor(205.5833, device='cuda:0')) | > loss_gen: 1.9236258268356323 (2.2136347986036724) | > loss_kl: 2.3991312980651855 (2.3831700317321287) | > loss_feat: 3.2900197505950928 (4.202601894255609) | > loss_mel: 17.88043212890625 (18.060592559076124) | > loss_duration: 1.5505037307739258 (1.5519010943751181) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.043712615966797 (28.411900428033643) | > grad_norm_1: tensor(11046.9219, device='cuda:0') (tensor(4818.4380, device='cuda:0')) | > current_lr_0: 0.00016854144270856013 | > current_lr_1: 0.00016854144270856013 | > step_time: 0.4329 (0.43304906352873773) | > loader_time: 0.0081 (0.007067072776056105) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852085838855152 (+1.4815531985856029e-05) | > avg_step_time: 0.45110982236727865 (-0.006644144864149493) | > avg_loss_disc: 2.61823203865911 (-0.06757576700667167) | > avg_loss_disc_real_0: 0.1233494357323982 (-0.05862422843634249) | > avg_loss_disc_real_1: 0.22779802033598995 (-0.006946401381996314) | > avg_loss_disc_real_2: 0.23269824221940108 (+0.003714533129208525) | > avg_loss_disc_real_3: 0.22222597414339093 (-0.014445472561137762) | > avg_loss_disc_real_4: 0.24752825183767668 (+0.006525671817886991) | > avg_loss_disc_real_5: 0.2422687612788778 (-0.004098953705438474) | > avg_loss_0: 2.61823203865911 (-0.06757576700667167) | > avg_grad_norm_0: tensor(217.7810, device='cuda:0') (tensor(-93.7974, device='cuda:0')) | > avg_loss_gen: 2.18149987576713 (+0.017480188692120535) | > avg_loss_kl: 2.305667462483257 (+0.11727823673839088) | > avg_loss_feat: 4.017858649643373 (+0.13097512218314078) | > avg_loss_mel: 17.606595254280197 (+0.6440443388173414) | > avg_loss_duration: 1.568623445403408 (+0.026102527766160843) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.680244660713303 (+0.9358803923700911) | > avg_grad_norm_1: tensor(6899.8350, device='cuda:0') (tensor(-145.2446, device='cuda:0'))  > EPOCH: 1370/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:19:29)   --> TIME: 2025-02-25 22:19:35 -- STEP: 9/72 -- GLOBAL_STEP: 1098650 | > loss_disc: 2.515103340148926 (2.4708624415927463) | > loss_disc_real_0: 0.043309956789016724 (0.08312144296036826) | > loss_disc_real_1: 0.5338088870048523 (0.28404562837547725) | > loss_disc_real_2: 0.23120659589767456 (0.19822133001354006) | > loss_disc_real_3: 0.1992236226797104 (0.20702982445557913) | > loss_disc_real_4: 0.20364466309547424 (0.2409110880560345) | > loss_disc_real_5: 0.22419221699237823 (0.23664966556761) | > loss_0: 2.515103340148926 (2.4708624415927463) | > grad_norm_0: tensor(134.3792, device='cuda:0') (tensor(170.2598, device='cuda:0')) | > loss_gen: 2.632716655731201 (2.392388423283895) | > loss_kl: 2.465319871902466 (2.555947224299113) | > loss_feat: 4.685918807983398 (4.46157259411282) | > loss_mel: 20.16098976135254 (20.711332745022244) | > loss_duration: 1.5497376918792725 (1.5573190848032634) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.49468421936035 (31.678560256958008) | > grad_norm_1: tensor(4314.2148, device='cuda:0') (tensor(7869.0366, device='cuda:0')) | > current_lr_0: 0.00016852037502822155 | > current_lr_1: 0.00016852037502822155 | > step_time: 0.4456 (0.4327315224541558) | > loader_time: 0.0056 (0.006471421983506944)  --> TIME: 2025-02-25 22:19:58 -- STEP: 59/72 -- GLOBAL_STEP: 1098700 | > loss_disc: 2.6283600330352783 (2.711824958607303) | > loss_disc_real_0: 0.1152835339307785 (0.14550798796748718) | > loss_disc_real_1: 0.23414190113544464 (0.2555212297682034) | > loss_disc_real_2: 0.24073173105716705 (0.25051478144981093) | > loss_disc_real_3: 0.23921923339366913 (0.2451311021538104) | > loss_disc_real_4: 0.2384559065103531 (0.24750409161640427) | > loss_disc_real_5: 0.2217261642217636 (0.24874029851565924) | > loss_0: 2.6283600330352783 (2.711824958607303) | > grad_norm_0: tensor(32.4520, device='cuda:0') (tensor(279.0216, device='cuda:0')) | > loss_gen: 1.883223533630371 (2.115066257573791) | > loss_kl: 2.238698959350586 (2.328475572295108) | > loss_feat: 3.311448812484741 (3.754576909340034) | > loss_mel: 16.85713005065918 (18.598453667204268) | > loss_duration: 1.567863941192627 (1.5768833079580533) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.85836410522461 (28.373455855805997) | > grad_norm_1: tensor(98.1494, device='cuda:0') (tensor(8022.1895, device='cuda:0')) | > current_lr_0: 0.00016852037502822155 | > current_lr_1: 0.00016852037502822155 | > step_time: 0.4617 (0.44260780690080026) | > loader_time: 0.0097 (0.007984549312268276) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547997810471226 (+2.7139421919706447e-05) | > avg_step_time: 0.4524317358581113 (+0.0013219134908326602) | > avg_loss_disc: 2.71534953654652 (+0.09711749788740986) | > avg_loss_disc_real_0: 0.14444472236742442 (+0.021095286635026217) | > avg_loss_disc_real_1: 0.2523112433477187 (+0.024513223011728758) | > avg_loss_disc_real_2: 0.2506705030379161 (+0.017972260818515023) | > avg_loss_disc_real_3: 0.24601973131508895 (+0.023793757171698016) | > avg_loss_disc_real_4: 0.24769717349972523 (+0.00016892166204854342) | > avg_loss_disc_real_5: 0.24879804316540838 (+0.006529281886530569) | > avg_loss_0: 2.71534953654652 (+0.09711749788740986) | > avg_grad_norm_0: tensor(269.1357, device='cuda:0') (+tensor(51.3547, device='cuda:0')) | > avg_loss_gen: 2.1124066601336846 (-0.06909321563344539) | > avg_loss_kl: 2.458264310595015 (+0.15259684811175767) | > avg_loss_feat: 3.7669101466595287 (-0.2509485029838445) | > avg_loss_mel: 18.73607685196568 (+1.1294815976854835) | > avg_loss_duration: 1.5914069145498142 (+0.022783469146406166) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.665065013187032 (+0.9848203524737293) | > avg_grad_norm_1: tensor(8138.0332, device='cuda:0') (+tensor(1238.1982, device='cuda:0'))  > EPOCH: 1371/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:20:06)   --> TIME: 2025-02-25 22:20:25 -- STEP: 37/72 -- GLOBAL_STEP: 1098750 | > loss_disc: 2.568808078765869 (2.5384662086899215) | > loss_disc_real_0: 0.1063358262181282 (0.09696185649246783) | > loss_disc_real_1: 0.3468274772167206 (0.22352771541556796) | > loss_disc_real_2: 0.30120280385017395 (0.2339805561948467) | > loss_disc_real_3: 0.21953843533992767 (0.23939362973780245) | > loss_disc_real_4: 0.2595883309841156 (0.24754128182256543) | > loss_disc_real_5: 0.2731817960739136 (0.25381672422628143) | > loss_0: 2.568808078765869 (2.5384662086899215) | > grad_norm_0: tensor(140.4964, device='cuda:0') (tensor(235.2452, device='cuda:0')) | > loss_gen: 2.361506938934326 (2.2649261790352906) | > loss_kl: 1.9383987188339233 (2.1515208708273397) | > loss_feat: 3.7211532592773438 (4.133271681295861) | > loss_mel: 16.85762596130371 (17.686422167597595) | > loss_duration: 1.5574791431427002 (1.523584185419856) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.43616485595703 (27.759725261378932) | > grad_norm_1: tensor(9099.5947, device='cuda:0') (tensor(7699.0635, device='cuda:0')) | > current_lr_0: 0.00016849930998134302 | > current_lr_1: 0.00016849930998134302 | > step_time: 0.4368 (0.4468877186646333) | > loader_time: 0.008 (0.007125777167242927) --> EVAL PERFORMANCE | > avg_loader_time: 0.008589724419822156 (+4.17266093509297e-05) | > avg_step_time: 0.4566593035845689 (+0.004227567726457604) | > avg_loss_disc: 2.5455076929549096 (-0.16984184359161025) | > avg_loss_disc_real_0: 0.08910941975322408 (-0.055335302614200346) | > avg_loss_disc_real_1: 0.20987879558348319 (-0.04243244776423552) | > avg_loss_disc_real_2: 0.22669991808877865 (-0.023970584949137452) | > avg_loss_disc_real_3: 0.23079356236357085 (-0.015226168951518099) | > avg_loss_disc_real_4: 0.24211874247436793 (-0.005578431025357294) | > avg_loss_disc_real_5: 0.24644403033693066 (-0.0023540128284777195) | > avg_loss_0: 2.5455076929549096 (-0.16984184359161025) | > avg_grad_norm_0: tensor(286.0396, device='cuda:0') (+tensor(16.9039, device='cuda:0')) | > avg_loss_gen: 2.1695880184710865 (+0.0571813583374019) | > avg_loss_kl: 2.1389957666397095 (-0.31926854395530535) | > avg_loss_feat: 4.055009314711668 (+0.28809916805213964) | > avg_loss_mel: 17.321654400355385 (-1.414422451610296) | > avg_loss_duration: 1.5486529612205395 (-0.042753953329274674) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.23390033883108 (-1.4311646743559514) | > avg_grad_norm_1: tensor(8685.5498, device='cuda:0') (+tensor(547.5166, device='cuda:0'))  > EPOCH: 1372/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:20:43)   --> TIME: 2025-02-25 22:20:51 -- STEP: 15/72 -- GLOBAL_STEP: 1098800 | > loss_disc: 2.5460379123687744 (2.5381635348002116) | > loss_disc_real_0: 0.0983283519744873 (0.10267206554611524) | > loss_disc_real_1: 0.3365309536457062 (0.254935363928477) | > loss_disc_real_2: 0.2965153157711029 (0.24028337796529134) | > loss_disc_real_3: 0.2358008325099945 (0.23989710013071697) | > loss_disc_real_4: 0.2445993274450302 (0.23885338207085927) | > loss_disc_real_5: 0.22943706810474396 (0.23810937702655793) | > loss_0: 2.5460379123687744 (2.5381635348002116) | > grad_norm_0: tensor(204.7596, device='cuda:0') (tensor(318.5393, device='cuda:0')) | > loss_gen: 2.509758949279785 (2.2136622587839763) | > loss_kl: 1.974482536315918 (1.9186285734176636) | > loss_feat: 4.061020374298096 (3.954332129160563) | > loss_mel: 17.132814407348633 (16.60982519785563) | > loss_duration: 1.4880242347717285 (1.488275702794393) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.166101455688477 (26.18472391764323) | > grad_norm_1: tensor(3948.7993, device='cuda:0') (tensor(8227.3633, device='cuda:0')) | > current_lr_0: 0.00016847824756759534 | > current_lr_1: 0.00016847824756759534 | > step_time: 0.4316 (0.43701416651407876) | > loader_time: 0.0069 (0.006782372792561849)  --> TIME: 2025-02-25 22:21:15 -- STEP: 65/72 -- GLOBAL_STEP: 1098850 | > loss_disc: 2.452101707458496 (2.564762122814471) | > loss_disc_real_0: 0.0985502153635025 (0.11233164467490636) | > loss_disc_real_1: 0.2245538830757141 (0.24227188435884622) | > loss_disc_real_2: 0.23283441364765167 (0.25516073543291823) | > loss_disc_real_3: 0.2348930835723877 (0.2300939910686933) | > loss_disc_real_4: 0.22371067106723785 (0.24086789374168102) | > loss_disc_real_5: 0.2268880307674408 (0.24161917567253113) | > loss_0: 2.452101707458496 (2.564762122814471) | > grad_norm_0: tensor(156.3813, device='cuda:0') (tensor(315.1847, device='cuda:0')) | > loss_gen: 2.0821480751037598 (2.2381609990046574) | > loss_kl: 2.3036625385284424 (1.9899701375227707) | > loss_feat: 3.791393280029297 (4.076108752764188) | > loss_mel: 16.664386749267578 (16.648988283597507) | > loss_duration: 1.591629981994629 (1.5246501115652231) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.43321990966797 (26.47787810105544) | > grad_norm_1: tensor(10712.5859, device='cuda:0') (tensor(7874.3623, device='cuda:0')) | > current_lr_0: 0.00016847824756759534 | > current_lr_1: 0.00016847824756759534 | > step_time: 0.4869 (0.4499308622800387) | > loader_time: 0.0117 (0.008385845331045298) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865141102965449 (+6.168660983233397e-05) | > avg_step_time: 0.4545149030819745 (-0.0021444005025944013) | > avg_loss_disc: 2.559171045330208 (+0.013663352375298476) | > avg_loss_disc_real_0: 0.10776834069213397 (+0.01865892093890989) | > avg_loss_disc_real_1: 0.2398424205225958 (+0.029963624939112615) | > avg_loss_disc_real_2: 0.2524978224660309 (+0.025797904377252268) | > avg_loss_disc_real_3: 0.22908603893199436 (-0.0017075234315764853) | > avg_loss_disc_real_4: 0.23987289780462293 (-0.0022458446697450063) | > avg_loss_disc_real_5: 0.24053231931068528 (-0.005911711026245381) | > avg_loss_0: 2.559171045330208 (+0.013663352375298476) | > avg_grad_norm_0: tensor(305.1516, device='cuda:0') (+tensor(19.1120, device='cuda:0')) | > avg_loss_gen: 2.226314687393081 (+0.056726668921994516) | > avg_loss_kl: 2.0015679429954205 (-0.13742782364428896) | > avg_loss_feat: 4.082612356669466 (+0.02760304195779728) | > avg_loss_mel: 16.663994023497676 (-0.6576603768577094) | > avg_loss_duration: 1.5304929101970834 (-0.01816005102345608) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.50498175285232 (-0.7289185859787608) | > avg_grad_norm_1: tensor(8010.4561, device='cuda:0') (tensor(-675.0938, device='cuda:0'))  > EPOCH: 1373/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:21:20)   --> TIME: 2025-02-25 22:21:41 -- STEP: 43/72 -- GLOBAL_STEP: 1098900 | > loss_disc: 2.4296388626098633 (2.653668902641119) | > loss_disc_real_0: 0.06077516824007034 (0.21957000067760782) | > loss_disc_real_1: 0.19752009212970734 (0.20769576728343964) | > loss_disc_real_2: 0.22327464818954468 (0.22661014802234117) | > loss_disc_real_3: 0.21861660480499268 (0.22586686424044675) | > loss_disc_real_4: 0.23957455158233643 (0.2391556761292524) | > loss_disc_real_5: 0.25033456087112427 (0.24150027092113052) | > loss_0: 2.4296388626098633 (2.653668902641119) | > grad_norm_0: tensor(153.7514, device='cuda:0') (tensor(354.6890, device='cuda:0')) | > loss_gen: 2.245145082473755 (2.160664295041284) | > loss_kl: 1.9493472576141357 (1.9549736477607904) | > loss_feat: 4.261356353759766 (3.8283889404563016) | > loss_mel: 16.95411491394043 (16.83855411618255) | > loss_duration: 1.5426275730133057 (1.5128597104272177) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.952590942382812 (26.29544098432674) | > grad_norm_1: tensor(9193.3701, device='cuda:0') (tensor(7132.8911, device='cuda:0')) | > current_lr_0: 0.00016845718778664938 | > current_lr_1: 0.00016845718778664938 | > step_time: 0.4492 (0.4379942029021507) | > loader_time: 0.0095 (0.007370616114416788) --> EVAL PERFORMANCE | > avg_loader_time: 0.008529770542198504 (-0.00012164048745598628) | > avg_step_time: 0.453884739271352 (-0.0006301638106225305) | > avg_loss_disc: 2.7014420737682934 (+0.14227102843808526) | > avg_loss_disc_real_0: 0.2445872879364121 (+0.13681894724427812) | > avg_loss_disc_real_1: 0.2116518369023229 (-0.028190583620272908) | > avg_loss_disc_real_2: 0.23505858825126164 (-0.017439234214769284) | > avg_loss_disc_real_3: 0.23345253081388875 (+0.004366491881894391) | > avg_loss_disc_real_4: 0.24412797454377294 (+0.0042550767391500155) | > avg_loss_disc_real_5: 0.24037497207312516 (-0.0001573472375601126) | > avg_loss_0: 2.7014420737682934 (+0.14227102843808526) | > avg_grad_norm_0: tensor(348.0154, device='cuda:0') (+tensor(42.8639, device='cuda:0')) | > avg_loss_gen: 2.162508877230362 (-0.06380581016271902) | > avg_loss_kl: 1.9977128589656992 (-0.003855084029721345) | > avg_loss_feat: 3.6856802483679543 (-0.3969321083015114) | > avg_loss_mel: 16.773215844597615 (+0.10922182109993983) | > avg_loss_duration: 1.5343629054620231 (+0.0038699952649396963) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.15348085215394 (-0.3515009006983796) | > avg_grad_norm_1: tensor(6772.4658, device='cuda:0') (tensor(-1237.9902, device='cuda:0'))  > EPOCH: 1374/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:21:56)   --> TIME: 2025-02-25 22:22:08 -- STEP: 21/72 -- GLOBAL_STEP: 1098950 | > loss_disc: 2.930988311767578 (2.705552169254848) | > loss_disc_real_0: 0.4627212882041931 (0.29017274649370284) | > loss_disc_real_1: 0.36011239886283875 (0.22242565027305058) | > loss_disc_real_2: 0.20475934445858002 (0.2108712430511202) | > loss_disc_real_3: 0.2165997475385666 (0.23059954103969393) | > loss_disc_real_4: 0.2472698837518692 (0.22996979100363596) | > loss_disc_real_5: 0.23175698518753052 (0.23201884684108554) | > loss_0: 2.930988311767578 (2.705552169254848) | > grad_norm_0: tensor(472.2651, device='cuda:0') (tensor(349.8871, device='cuda:0')) | > loss_gen: 2.4291653633117676 (2.058869401613871) | > loss_kl: 2.847275733947754 (2.553254417010716) | > loss_feat: 4.085178375244141 (3.635114249728975) | > loss_mel: 20.063539505004883 (19.643852642604283) | > loss_duration: 1.6749999523162842 (1.6549514759154547) | > amp_scaler: 8.0 (8.0) | > loss_1: 31.10015869140625 (29.54604230608259) | > grad_norm_1: tensor(4497.7852, device='cuda:0') (tensor(6973.7583, device='cuda:0')) | > current_lr_0: 0.00016843613063817603 | > current_lr_1: 0.00016843613063817603 | > step_time: 0.4306 (0.43296600523449125) | > loader_time: 0.0066 (0.007123640605381557)  --> TIME: 2025-02-25 22:22:32 -- STEP: 71/72 -- GLOBAL_STEP: 1099000 | > loss_disc: 2.7586891651153564 (2.6750846009858895) | > loss_disc_real_0: 0.24635344743728638 (0.1886544514867202) | > loss_disc_real_1: 0.279404878616333 (0.22329941853671006) | > loss_disc_real_2: 0.19128091633319855 (0.23526698487325454) | > loss_disc_real_3: 0.2799449861049652 (0.23464529258264621) | > loss_disc_real_4: 0.25768381357192993 (0.24258862750630983) | > loss_disc_real_5: 0.24755506217479706 (0.23829340074263827) | > loss_0: 2.7586891651153564 (2.6750846009858895) | > grad_norm_0: tensor(499.6745, device='cuda:0') (tensor(275.7405, device='cuda:0')) | > loss_gen: 2.657674789428711 (2.180833158358721) | > loss_kl: 2.062103033065796 (2.3315853165908598) | > loss_feat: 3.831685781478882 (3.9177419904252173) | > loss_mel: 18.220165252685547 (18.04438944265876) | > loss_duration: 1.5689945220947266 (1.6077907370849391) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.34062385559082 (28.08234067030356) | > grad_norm_1: tensor(2430.9253, device='cuda:0') (tensor(5841.1289, device='cuda:0')) | > current_lr_0: 0.00016843613063817603 | > current_lr_1: 0.00016843613063817603 | > step_time: 0.4723 (0.4542790668111452) | > loader_time: 0.008 (0.008776580783682812) --> EVAL PERFORMANCE | > avg_loader_time: 0.008776580783682812 (+0.00024681024148430833) | > avg_step_time: 0.4542790668111452 (+0.00039432753979323776) | > avg_loss_disc: 2.6750846009858895 (-0.026357472782403857) | > avg_loss_disc_real_0: 0.1886544514867202 (-0.05593283644969191) | > avg_loss_disc_real_1: 0.22329941853671006 (+0.011647581634387172) | > avg_loss_disc_real_2: 0.23526698487325454 (+0.0002083966219929012) | > avg_loss_disc_real_3: 0.23464529258264621 (+0.001192761768757461) | > avg_loss_disc_real_4: 0.24258862750630983 (-0.0015393470374631124) | > avg_loss_disc_real_5: 0.23829340074263827 (-0.002081571330486892) | > avg_loss_0: 2.6750846009858895 (-0.026357472782403857) | > avg_grad_norm_0: tensor(275.7405, device='cuda:0') (tensor(-72.2749, device='cuda:0')) | > avg_loss_gen: 2.180833158358721 (+0.018324281128359043) | > avg_loss_kl: 2.3315853165908598 (+0.3338724576251606) | > avg_loss_feat: 3.9177419904252173 (+0.23206174205726304) | > avg_loss_mel: 18.04438944265876 (+1.271173598061143) | > avg_loss_duration: 1.6077907370849391 (+0.073427831622916) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.08234067030356 (+1.9288598181496184) | > avg_grad_norm_1: tensor(5841.1289, device='cuda:0') (tensor(-931.3369, device='cuda:0'))  > EPOCH: 1375/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:22:34)   --> TIME: 2025-02-25 22:22:58 -- STEP: 49/72 -- GLOBAL_STEP: 1099050 | > loss_disc: 2.467193841934204 (2.650867462158203) | > loss_disc_real_0: 0.06471208482980728 (0.12740270132008868) | > loss_disc_real_1: 0.1277695894241333 (0.23347251193255794) | > loss_disc_real_2: 0.21830704808235168 (0.2217550657841624) | > loss_disc_real_3: 0.277271032333374 (0.24235284571744958) | > loss_disc_real_4: 0.22891873121261597 (0.2430189778001941) | > loss_disc_real_5: 0.2316921204328537 (0.243023189354916) | > loss_0: 2.467193841934204 (2.650867462158203) | > grad_norm_0: tensor(253.3307, device='cuda:0') (tensor(247.4125, device='cuda:0')) | > loss_gen: 2.0515758991241455 (2.1133957298434503) | > loss_kl: 2.047394275665283 (2.0595110216919257) | > loss_feat: 4.577273368835449 (3.8643909181867326) | > loss_mel: 18.17972755432129 (17.116407297095474) | > loss_duration: 1.5737638473510742 (1.519801290667787) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.429737091064453 (26.673506094484914) | > grad_norm_1: tensor(6559.9434, device='cuda:0') (tensor(7202.9229, device='cuda:0')) | > current_lr_0: 0.00016841507612184626 | > current_lr_1: 0.00016841507612184626 | > step_time: 0.4472 (0.4448782083939533) | > loader_time: 0.0098 (0.0076923370361328125) --> EVAL PERFORMANCE | > avg_loader_time: 0.00862332800744285 (-0.0001532527762399625) | > avg_step_time: 0.4566690787463121 (+0.002390011935166858) | > avg_loss_disc: 2.6444668400455527 (-0.030617760940336858) | > avg_loss_disc_real_0: 0.12253742491905123 (-0.06611702656766896) | > avg_loss_disc_real_1: 0.22567268104200633 (+0.002373262505296264) | > avg_loss_disc_real_2: 0.22706819420129482 (-0.008198790671959721) | > avg_loss_disc_real_3: 0.24694549092944243 (+0.012300198346796215) | > avg_loss_disc_real_4: 0.24601759247376886 (+0.003428964967459036) | > avg_loss_disc_real_5: 0.24514449356307447 (+0.006851092820436194) | > avg_loss_0: 2.6444668400455527 (-0.030617760940336858) | > avg_grad_norm_0: tensor(252.7230, device='cuda:0') (tensor(-23.0175, device='cuda:0')) | > avg_loss_gen: 2.098165679985368 (-0.0826674783733532) | > avg_loss_kl: 2.0779906931057783 (-0.25359462348508144) | > avg_loss_feat: 3.7935794776594136 (-0.1241625127658037) | > avg_loss_mel: 17.045088271020163 (-0.9993011716385958) | > avg_loss_duration: 1.53627284144012 (-0.07151789564481903) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.55109684903857 (-1.5312438212649901) | > avg_grad_norm_1: tensor(7225.1499, device='cuda:0') (+tensor(1384.0210, device='cuda:0'))  > EPOCH: 1376/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:23:11)   --> TIME: 2025-02-25 22:23:25 -- STEP: 27/72 -- GLOBAL_STEP: 1099100 | > loss_disc: 2.5953495502471924 (2.68401621889185) | > loss_disc_real_0: 0.06337079405784607 (0.10027118482523495) | > loss_disc_real_1: 0.23311133682727814 (0.22712386482291752) | > loss_disc_real_2: 0.20066359639167786 (0.24366115916658335) | > loss_disc_real_3: 0.24347323179244995 (0.24696080497017614) | > loss_disc_real_4: 0.22932828962802887 (0.24424762747905873) | > loss_disc_real_5: 0.2578982412815094 (0.2526441255101451) | > loss_0: 2.5953495502471924 (2.68401621889185) | > grad_norm_0: tensor(185.1881, device='cuda:0') (tensor(258.5686, device='cuda:0')) | > loss_gen: 2.0081787109375 (2.0392695179692013) | > loss_kl: 2.6977646350860596 (2.4439000818464502) | > loss_feat: 4.044600486755371 (3.9018333664646856) | > loss_mel: 19.71416473388672 (18.681162657561124) | > loss_duration: 1.6721441745758057 (1.5689766274558172) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.136852264404297 (28.635142079106085) | > grad_norm_1: tensor(9352.4346, device='cuda:0') (tensor(8685.4922, device='cuda:0')) | > current_lr_0: 0.00016839402423733102 | > current_lr_1: 0.00016839402423733102 | > step_time: 0.4355 (0.43371463704992225) | > loader_time: 0.0087 (0.006809534849943938) --> EVAL PERFORMANCE | > avg_loader_time: 0.008667495888723457 (+4.416788128060699e-05) | > avg_step_time: 0.4546448210595359 (-0.0020242576867761652) | > avg_loss_disc: 2.656934090063606 (+0.012467250018053377) | > avg_loss_disc_real_0: 0.13555216946652232 (+0.013014744547471091) | > avg_loss_disc_real_1: 0.22064074391210584 (-0.005031937129900493) | > avg_loss_disc_real_2: 0.23527444248468105 (+0.00820624828338623) | > avg_loss_disc_real_3: 0.24139432525131063 (-0.0055511656781317975) | > avg_loss_disc_real_4: 0.2455654973295373 (-0.00045209514423155683) | > avg_loss_disc_real_5: 0.24810821489549015 (+0.002963721332415681) | > avg_loss_0: 2.656934090063606 (+0.012467250018053377) | > avg_grad_norm_0: tensor(273.4525, device='cuda:0') (+tensor(20.7296, device='cuda:0')) | > avg_loss_gen: 2.04707396869928 (-0.051091711286087804) | > avg_loss_kl: 2.3395753544820868 (+0.2615846613763084) | > avg_loss_feat: 3.768919246297487 (-0.02466023136192641) | > avg_loss_mel: 18.0906142919836 (+1.045526020963436) | > avg_loss_duration: 1.59105319708166 (+0.054780355641539824) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.83723592086577 (+1.2861390718272006) | > avg_grad_norm_1: tensor(7710.3530, device='cuda:0') (+tensor(485.2031, device='cuda:0'))  > EPOCH: 1377/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:23:47)   --> TIME: 2025-02-25 22:23:52 -- STEP: 5/72 -- GLOBAL_STEP: 1099150 | > loss_disc: 2.5058393478393555 (2.6136446952819825) | > loss_disc_real_0: 0.13442054390907288 (0.1407068058848381) | > loss_disc_real_1: 0.23030154407024384 (0.2170976758003235) | > loss_disc_real_2: 0.23557038605213165 (0.23037049770355225) | > loss_disc_real_3: 0.22564944624900818 (0.23123387694358827) | > loss_disc_real_4: 0.2439574897289276 (0.23773572444915772) | > loss_disc_real_5: 0.2570524215698242 (0.23931665420532228) | > loss_0: 2.5058393478393555 (2.6136446952819825) | > grad_norm_0: tensor(229.7407, device='cuda:0') (tensor(105.5020, device='cuda:0')) | > loss_gen: 2.389470100402832 (2.056562328338623) | > loss_kl: 1.7812726497650146 (1.8672292947769165) | > loss_feat: 4.272547245025635 (3.6858831882476806) | > loss_mel: 16.45424461364746 (16.809831619262695) | > loss_duration: 1.4906303882598877 (1.5143591642379761) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.388164520263672 (25.933866119384767) | > grad_norm_1: tensor(4790.0503, device='cuda:0') (tensor(5923.9785, device='cuda:0')) | > current_lr_0: 0.00016837297498430135 | > current_lr_1: 0.00016837297498430135 | > step_time: 0.4181 (0.4248312473297119) | > loader_time: 0.0062 (0.006278181076049804)  --> TIME: 2025-02-25 22:24:14 -- STEP: 55/72 -- GLOBAL_STEP: 1099200 | > loss_disc: 2.5766215324401855 (2.607173841649837) | > loss_disc_real_0: 0.09265384078025818 (0.12651162391359158) | > loss_disc_real_1: 0.24957537651062012 (0.2201056510210037) | > loss_disc_real_2: 0.2347605675458908 (0.24253655997189608) | > loss_disc_real_3: 0.2460297793149948 (0.23874127052047037) | > loss_disc_real_4: 0.24808835983276367 (0.24700493270700627) | > loss_disc_real_5: 0.2660863697528839 (0.24903376617214895) | > loss_0: 2.5766215324401855 (2.607173841649837) | > grad_norm_0: tensor(158.1684, device='cuda:0') (tensor(286.9386, device='cuda:0')) | > loss_gen: 2.1818408966064453 (2.1618327682668506) | > loss_kl: 2.47164249420166 (2.0970529209483755) | > loss_feat: 4.055774211883545 (3.9276467149907894) | > loss_mel: 19.158672332763672 (17.395003752274945) | > loss_duration: 1.6356124877929688 (1.5480013825676657) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.503541946411133 (27.12953758239746) | > grad_norm_1: tensor(10005.0713, device='cuda:0') (tensor(7331.0444, device='cuda:0')) | > current_lr_0: 0.00016837297498430135 | > current_lr_1: 0.00016837297498430135 | > step_time: 0.4719 (0.43963967236605556) | > loader_time: 0.0093 (0.007812513004649769) --> EVAL PERFORMANCE | > avg_loader_time: 0.008563582326324893 (-0.0001039135623985641) | > avg_step_time: 0.4497987317367339 (-0.004846089322802005) | > avg_loss_disc: 2.6107275788213173 (-0.046206511242288695) | > avg_loss_disc_real_0: 0.13182288597167383 (-0.0037292834948484954) | > avg_loss_disc_real_1: 0.22124106101167035 (+0.0006003170995645124) | > avg_loss_disc_real_2: 0.23879540491272028 (+0.0035209624280392315) | > avg_loss_disc_real_3: 0.23626274898858138 (-0.005131576262729254) | > avg_loss_disc_real_4: 0.24486127495765686 (-0.0007042223718804475) | > avg_loss_disc_real_5: 0.24697524938784854 (-0.001132965507641609) | > avg_loss_0: 2.6107275788213173 (-0.046206511242288695) | > avg_grad_norm_0: tensor(292.0679, device='cuda:0') (+tensor(18.6154, device='cuda:0')) | > avg_loss_gen: 2.148673860120101 (+0.10159989142082093) | > avg_loss_kl: 2.16756762249369 (-0.1720077319883968) | > avg_loss_feat: 3.8830944920929387 (+0.11417524579545146) | > avg_loss_mel: 17.64873966700594 (-0.4418746249776575) | > avg_loss_duration: 1.5622566417909005 (-0.028796555290759418) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.410332317083654 (-0.4269036037821152) | > avg_grad_norm_1: tensor(7559.9199, device='cuda:0') (tensor(-150.4331, device='cuda:0'))  > EPOCH: 1378/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:24:24)   --> TIME: 2025-02-25 22:24:41 -- STEP: 33/72 -- GLOBAL_STEP: 1099250 | > loss_disc: 2.5501232147216797 (2.603553360158747) | > loss_disc_real_0: 0.16339340806007385 (0.13070283028664012) | > loss_disc_real_1: 0.23350857198238373 (0.21440760723569177) | > loss_disc_real_2: 0.17799688875675201 (0.23308993830825342) | > loss_disc_real_3: 0.1950390338897705 (0.23349116110440457) | > loss_disc_real_4: 0.2317858785390854 (0.2462790355537877) | > loss_disc_real_5: 0.23935997486114502 (0.24719592341870972) | > loss_0: 2.5501232147216797 (2.603553360158747) | > grad_norm_0: tensor(301.7352, device='cuda:0') (tensor(232.9715, device='cuda:0')) | > loss_gen: 2.2667078971862793 (2.1876921978863804) | > loss_kl: 2.0181620121002197 (2.021851716619549) | > loss_feat: 4.431996822357178 (4.067142183130436) | > loss_mel: 17.42375373840332 (17.41675845059482) | > loss_duration: 1.487635850906372 (1.511152415564566) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.628257751464844 (27.20459695295854) | > grad_norm_1: tensor(5003.5391, device='cuda:0') (tensor(6190.3042, device='cuda:0')) | > current_lr_0: 0.00016835192836242832 | > current_lr_1: 0.00016835192836242832 | > step_time: 0.4381 (0.4362983703613281) | > loader_time: 0.008 (0.006911667910489169) --> EVAL PERFORMANCE | > avg_loader_time: 0.008519558839395012 (-4.402348692988077e-05) | > avg_step_time: 0.45423655106987754 (+0.004437819333143633) | > avg_loss_disc: 2.6169051620322206 (+0.006177583210903226) | > avg_loss_disc_real_0: 0.13479475729482276 (+0.002971871323148939) | > avg_loss_disc_real_1: 0.2417568722870988 (+0.020515811275428453) | > avg_loss_disc_real_2: 0.23061430202403538 (-0.008181102888684899) | > avg_loss_disc_real_3: 0.23297018358405208 (-0.0032925654045292962) | > avg_loss_disc_real_4: 0.2458841626493024 (+0.0010228876916455465) | > avg_loss_disc_real_5: 0.24613595806377034 (-0.0008392913240782007) | > avg_loss_0: 2.6169051620322206 (+0.006177583210903226) | > avg_grad_norm_0: tensor(258.7324, device='cuda:0') (tensor(-33.3355, device='cuda:0')) | > avg_loss_gen: 2.1981944399820255 (+0.049520579861924485) | > avg_loss_kl: 2.1021135995085802 (-0.06545402298510972) | > avg_loss_feat: 3.9584377819383647 (+0.07534328984542604) | > avg_loss_mel: 17.3964912924968 (-0.25224837450913995) | > avg_loss_duration: 1.54718300826113 (-0.015073633529770536) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.202420167520014 (-0.20791214956364001) | > avg_grad_norm_1: tensor(6077.2217, device='cuda:0') (tensor(-1482.6982, device='cuda:0'))  > EPOCH: 1379/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:25:01)   --> TIME: 2025-02-25 22:25:08 -- STEP: 11/72 -- GLOBAL_STEP: 1099300 | > loss_disc: 2.6072134971618652 (2.68389968438582) | > loss_disc_real_0: 0.16555088758468628 (0.24647216227921573) | > loss_disc_real_1: 0.2204170823097229 (0.18891122124411844) | > loss_disc_real_2: 0.1832006573677063 (0.23680780150673605) | > loss_disc_real_3: 0.24722498655319214 (0.2289475216106935) | > loss_disc_real_4: 0.2523861825466156 (0.2455419193614613) | > loss_disc_real_5: 0.2602604925632477 (0.2488457899202) | > loss_0: 2.6072134971618652 (2.68389968438582) | > grad_norm_0: tensor(566.7344, device='cuda:0') (tensor(332.4393, device='cuda:0')) | > loss_gen: 1.8205775022506714 (2.1045331087979404) | > loss_kl: 2.0113444328308105 (1.9342586994171143) | > loss_feat: 3.3618946075439453 (3.651091055436568) | > loss_mel: 17.71663475036621 (17.208548285744406) | > loss_duration: 1.487718105316162 (1.4991003708405928) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.398170471191406 (26.397531509399414) | > grad_norm_1: tensor(10158.0898, device='cuda:0') (tensor(7273.4873, device='cuda:0')) | > current_lr_0: 0.000168330884371383 | > current_lr_1: 0.000168330884371383 | > step_time: 0.431 (0.4293803735212846) | > loader_time: 0.0056 (0.006172158501364968)  --> TIME: 2025-02-25 22:25:31 -- STEP: 61/72 -- GLOBAL_STEP: 1099350 | > loss_disc: 2.7023439407348633 (2.6694345904178305) | > loss_disc_real_0: 0.1571112871170044 (0.17456020824000484) | > loss_disc_real_1: 0.2032966911792755 (0.21076471346323608) | > loss_disc_real_2: 0.22521574795246124 (0.256512145038511) | > loss_disc_real_3: 0.24290403723716736 (0.2359681244267792) | > loss_disc_real_4: 0.26011332869529724 (0.24785457209485473) | > loss_disc_real_5: 0.2498818337917328 (0.25090332817835875) | > loss_0: 2.7023439407348633 (2.6694345904178305) | > grad_norm_0: tensor(101.2810, device='cuda:0') (tensor(269.6550, device='cuda:0')) | > loss_gen: 1.8910324573516846 (2.0474049888673376) | > loss_kl: 1.9962319135665894 (1.9788312149829552) | > loss_feat: 3.738884687423706 (3.6423150750457265) | > loss_mel: 17.399396896362305 (16.92274428195641) | > loss_duration: 1.5683557987213135 (1.5254213477744432) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.593902587890625 (26.116716728835808) | > grad_norm_1: tensor(321.9736, device='cuda:0') (tensor(7689.5273, device='cuda:0')) | > current_lr_0: 0.000168330884371383 | > current_lr_1: 0.000168330884371383 | > step_time: 0.4889 (0.44694821560969117) | > loader_time: 0.0105 (0.007999291185472832) --> EVAL PERFORMANCE | > avg_loader_time: 0.008471106139706894 (-4.8452699688117384e-05) | > avg_step_time: 0.4559490076253112 (+0.0017124565554336724) | > avg_loss_disc: 2.6626476234113667 (+0.04574246137914617) | > avg_loss_disc_real_0: 0.1822404395001875 (+0.04744568220536474) | > avg_loss_disc_real_1: 0.2130123345364987 (-0.028744537750600097) | > avg_loss_disc_real_2: 0.2540404242528996 (+0.0234261222288642) | > avg_loss_disc_real_3: 0.23679567810515284 (+0.0038254945211007585) | > avg_loss_disc_real_4: 0.2465867343502985 (+0.0007025717009960797) | > avg_loss_disc_real_5: 0.24854412695891412 (+0.0024081688951437807) | > avg_loss_0: 2.6626476234113667 (+0.04574246137914617) | > avg_grad_norm_0: tensor(272.6265, device='cuda:0') (+tensor(13.8941, device='cuda:0')) | > avg_loss_gen: 2.0625767288073686 (-0.13561771117465682) | > avg_loss_kl: 2.037338493575512 (-0.06477510593306812) | > avg_loss_feat: 3.6236688587027537 (-0.334768923235611) | > avg_loss_mel: 17.13910833546813 (-0.2573829570286712) | > avg_loss_duration: 1.5376246160184834 (-0.00955839224264654) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.40031685627682 (-0.8021033112431937) | > avg_grad_norm_1: tensor(7570.6162, device='cuda:0') (+tensor(1493.3945, device='cuda:0'))  > EPOCH: 1380/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:25:38)   --> TIME: 2025-02-25 22:25:58 -- STEP: 39/72 -- GLOBAL_STEP: 1099400 | > loss_disc: 2.3709616661071777 (2.610041740613106) | > loss_disc_real_0: 0.07275474071502686 (0.1105616049697766) | > loss_disc_real_1: 0.19863645732402802 (0.21148596169092718) | > loss_disc_real_2: 0.19628725945949554 (0.21972108307557228) | > loss_disc_real_3: 0.2333696484565735 (0.2398359049589206) | > loss_disc_real_4: 0.22713065147399902 (0.2467877387236326) | > loss_disc_real_5: 0.23423530161380768 (0.24876888898702768) | > loss_0: 2.3709616661071777 (2.610041740613106) | > grad_norm_0: tensor(222.4804, device='cuda:0') (tensor(229.1567, device='cuda:0')) | > loss_gen: 2.2519724369049072 (2.168951789538065) | > loss_kl: 2.177812099456787 (2.0512831119390635) | > loss_feat: 4.218068599700928 (4.1797879964877405) | > loss_mel: 16.718509674072266 (17.24495415809827) | > loss_duration: 1.53163743019104 (1.508223347174815) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.898000717163086 (27.153200296255257) | > grad_norm_1: tensor(7018.2227, device='cuda:0') (tensor(5650.1748, device='cuda:0')) | > current_lr_0: 0.00016830984301083657 | > current_lr_1: 0.00016830984301083657 | > step_time: 0.4812 (0.43730810361030775) | > loader_time: 0.0084 (0.007533739774655073) --> EVAL PERFORMANCE | > avg_loader_time: 0.008751983373937471 (+0.00028087723423057646) | > avg_step_time: 0.45093216358775823 (-0.005016844037552981) | > avg_loss_disc: 2.6567806996090306 (-0.00586692380233611) | > avg_loss_disc_real_0: 0.11865152780648688 (-0.06358891169370062) | > avg_loss_disc_real_1: 0.23947459333379503 (+0.02646225879729633) | > avg_loss_disc_real_2: 0.22709491806970517 (-0.026945506183194412) | > avg_loss_disc_real_3: 0.24127699959446008 (+0.004481321489307244) | > avg_loss_disc_real_4: 0.24978343093059432 (+0.003196696580295838) | > avg_loss_disc_real_5: 0.24936585464108157 (+0.0008217276821674557) | > avg_loss_0: 2.6567806996090306 (-0.00586692380233611) | > avg_grad_norm_0: tensor(241.7605, device='cuda:0') (tensor(-30.8660, device='cuda:0')) | > avg_loss_gen: 2.1612137572866086 (+0.09863702847923994) | > avg_loss_kl: 2.3953891237017135 (+0.3580506301262014) | > avg_loss_feat: 4.041916061455094 (+0.4182472027523403) | > avg_loss_mel: 18.508950569260282 (+1.369842233792152) | > avg_loss_duration: 1.5574280409745762 (+0.019803424956092774) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.664897462012064 (+2.264580605735244) | > avg_grad_norm_1: tensor(5818.9189, device='cuda:0') (tensor(-1751.6973, device='cuda:0'))  > EPOCH: 1381/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:26:15)   --> TIME: 2025-02-25 22:26:24 -- STEP: 17/72 -- GLOBAL_STEP: 1099450 | > loss_disc: 2.823303461074829 (2.4667078887715057) | > loss_disc_real_0: 0.024524271488189697 (0.05616558288388392) | > loss_disc_real_1: 0.3022315800189972 (0.2205992612768622) | > loss_disc_real_2: 0.29737767577171326 (0.2266258088981404) | > loss_disc_real_3: 0.5109796524047852 (0.2695985690635793) | > loss_disc_real_4: 0.3250671923160553 (0.24829307373832255) | > loss_disc_real_5: 0.2949391007423401 (0.2328428030014038) | > loss_0: 2.823303461074829 (2.4667078887715057) | > grad_norm_0: tensor(321.9579, device='cuda:0') (tensor(205.3920, device='cuda:0')) | > loss_gen: 2.8025338649749756 (2.394233970081105) | > loss_kl: 2.408663749694824 (2.4715919494628906) | > loss_feat: 4.1655049324035645 (4.430949028800516) | > loss_mel: 19.47212791442871 (18.67743121876436) | > loss_duration: 1.5449618101119995 (1.5443592562394983) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.3937931060791 (29.51856568280388) | > grad_norm_1: tensor(14975.4912, device='cuda:0') (tensor(8403.0439, device='cuda:0')) | > current_lr_0: 0.0001682888042804602 | > current_lr_1: 0.0001682888042804602 | > step_time: 0.4377 (0.4271519324358772) | > loader_time: 0.0074 (0.006925049950094784)  --> TIME: 2025-02-25 22:26:48 -- STEP: 67/72 -- GLOBAL_STEP: 1099500 | > loss_disc: 2.531982421875 (2.5005902745830477) | > loss_disc_real_0: 0.0929298847913742 (0.08307141271322521) | > loss_disc_real_1: 0.1995176523923874 (0.2214502500953959) | > loss_disc_real_2: 0.18446126580238342 (0.2166423110374764) | > loss_disc_real_3: 0.22845229506492615 (0.2631865437351056) | > loss_disc_real_4: 0.25289902091026306 (0.2561627842152297) | > loss_disc_real_5: 0.24082131683826447 (0.2534539490493375) | > loss_0: 2.531982421875 (2.5005902745830477) | > grad_norm_0: tensor(97.7804, device='cuda:0') (tensor(248.0549, device='cuda:0')) | > loss_gen: 2.1344261169433594 (2.330521820196465) | > loss_kl: 2.1020140647888184 (2.2595802296453447) | > loss_feat: 4.085500717163086 (4.234833375731511) | > loss_mel: 17.565959930419922 (17.68606713992446) | > loss_duration: 1.6254924535751343 (1.5574024969072484) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.51339340209961 (28.06840500902774) | > grad_norm_1: tensor(13332.6230, device='cuda:0') (tensor(8691.3428, device='cuda:0')) | > current_lr_0: 0.0001682888042804602 | > current_lr_1: 0.0001682888042804602 | > step_time: 0.4843 (0.4530108295269866) | > loader_time: 0.0118 (0.00843669051554666) --> EVAL PERFORMANCE | > avg_loader_time: 0.00859948279152454 (-0.00015250058241293024) | > avg_step_time: 0.45585286785179463 (+0.004920704264036402) | > avg_loss_disc: 2.505623474927016 (-0.15115722468201476) | > avg_loss_disc_real_0: 0.08419710360992122 (-0.034454424196565664) | > avg_loss_disc_real_1: 0.22228519765424057 (-0.017189395679554464) | > avg_loss_disc_real_2: 0.21745564715123514 (-0.009639270918470028) | > avg_loss_disc_real_3: 0.26108996045421545 (+0.01981296085975537) | > avg_loss_disc_real_4: 0.25466752681933663 (+0.004884095888742307) | > avg_loss_disc_real_5: 0.25263632414206644 (+0.0032704695009848628) | > avg_loss_0: 2.505623474927016 (-0.15115722468201476) | > avg_grad_norm_0: tensor(241.6682, device='cuda:0') (tensor(-0.0923, device='cuda:0')) | > avg_loss_gen: 2.3153339322184174 (+0.15412017493180885) | > avg_loss_kl: 2.2545641089828923 (-0.14082501471882125) | > avg_loss_feat: 4.213552542135749 (+0.17163648068065473) | > avg_loss_mel: 17.653078200111928 (-0.8558723691483543) | > avg_loss_duration: 1.5589859317725814 (+0.0015578907980051682) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.995514641345387 (-0.6693828206666765) | > avg_grad_norm_1: tensor(8611.5283, device='cuda:0') (+tensor(2792.6094, device='cuda:0'))  > EPOCH: 1382/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:26:52)   --> TIME: 2025-02-25 22:27:14 -- STEP: 45/72 -- GLOBAL_STEP: 1099550 | > loss_disc: 2.659562110900879 (2.6984953721364335) | > loss_disc_real_0: 0.20555564761161804 (0.19236377365887164) | > loss_disc_real_1: 0.31340768933296204 (0.22160100158717896) | > loss_disc_real_2: 0.2568417191505432 (0.2311024417479833) | > loss_disc_real_3: 0.23227958381175995 (0.24088735116852655) | > loss_disc_real_4: 0.23782168328762054 (0.23731042908297645) | > loss_disc_real_5: 0.22708937525749207 (0.2437204556332694) | > loss_0: 2.659562110900879 (2.6984953721364335) | > grad_norm_0: tensor(504.9415, device='cuda:0') (tensor(323.9262, device='cuda:0')) | > loss_gen: 2.2404963970184326 (2.112014659245809) | > loss_kl: 2.1587541103363037 (2.0200162967046094) | > loss_feat: 3.8091495037078857 (3.7028569327460397) | > loss_mel: 17.51920509338379 (16.886414824591743) | > loss_duration: 1.5537753105163574 (1.5124533255894979) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.28137969970703 (26.23375587463379) | > grad_norm_1: tensor(4070.5295, device='cuda:0') (tensor(6781.8496, device='cuda:0')) | > current_lr_0: 0.00016826776817992515 | > current_lr_1: 0.00016826776817992515 | > step_time: 0.4614 (0.43888324631585013) | > loader_time: 0.0106 (0.007526784472995334) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542705589616803 (-5.677720190773797e-05) | > avg_step_time: 0.45156646110642124 (-0.0042864067453733945) | > avg_loss_disc: 2.675532260411222 (+0.16990878548420607) | > avg_loss_disc_real_0: 0.18589150766566606 (+0.10169440405574484) | > avg_loss_disc_real_1: 0.22450725262013962 (+0.0022220549658990563) | > avg_loss_disc_real_2: 0.23306829450835645 (+0.015612647357121312) | > avg_loss_disc_real_3: 0.2403017481867696 (-0.020788212267445844) | > avg_loss_disc_real_4: 0.24002332964413603 (-0.014644197175200602) | > avg_loss_disc_real_5: 0.24522165303498927 (-0.007414671107077164) | > avg_loss_0: 2.675532260411222 (+0.16990878548420607) | > avg_grad_norm_0: tensor(320.9161, device='cuda:0') (+tensor(79.2479, device='cuda:0')) | > avg_loss_gen: 2.117996068067953 (-0.19733786415046461) | > avg_loss_kl: 2.0430853450802005 (-0.21147876390269182) | > avg_loss_feat: 3.663392271794064 (-0.5501602703416846) | > avg_loss_mel: 16.76248990985709 (-0.8905882902548363) | > avg_loss_duration: 1.5345180672658998 (-0.02446786450668159) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.12148150591783 (-1.8740331354275561) | > avg_grad_norm_1: tensor(6870.2500, device='cuda:0') (tensor(-1741.2783, device='cuda:0'))  > EPOCH: 1383/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:27:29)   --> TIME: 2025-02-25 22:27:41 -- STEP: 23/72 -- GLOBAL_STEP: 1099600 | > loss_disc: 2.5913619995117188 (2.589164391807888) | > loss_disc_real_0: 0.24050915241241455 (0.13872305562962656) | > loss_disc_real_1: 0.19237421452999115 (0.22843452704989392) | > loss_disc_real_2: 0.25216659903526306 (0.23211958032587302) | > loss_disc_real_3: 0.24618329107761383 (0.2357251851455025) | > loss_disc_real_4: 0.2577912509441376 (0.25212060562942346) | > loss_disc_real_5: 0.25070032477378845 (0.24364426926426266) | > loss_0: 2.5913619995117188 (2.589164391807888) | > grad_norm_0: tensor(322.8681, device='cuda:0') (tensor(274.7574, device='cuda:0')) | > loss_gen: 2.4083268642425537 (2.136398382808851) | > loss_kl: 1.9599242210388184 (2.035214771395144) | > loss_feat: 3.743192672729492 (3.824388265609741) | > loss_mel: 15.756468772888184 (17.436735360518746) | > loss_duration: 1.5114755630493164 (1.4913111614144368) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.37938690185547 (26.92404796766198) | > grad_norm_1: tensor(1709.7329, device='cuda:0') (tensor(7345.4165, device='cuda:0')) | > current_lr_0: 0.00016824673470890265 | > current_lr_1: 0.00016824673470890265 | > step_time: 0.4461 (0.4320706491884978) | > loader_time: 0.0063 (0.006740891415139903) --> EVAL PERFORMANCE | > avg_loader_time: 0.008571074042521732 (+2.8368452904929675e-05) | > avg_step_time: 0.4509420327737298 (-0.0006244283326914402) | > avg_loss_disc: 2.661647954457243 (-0.013884305953979048) | > avg_loss_disc_real_0: 0.12808026027091793 (-0.05781124739474813) | > avg_loss_disc_real_1: 0.21526502429599492 (-0.009242228324144702) | > avg_loss_disc_real_2: 0.23674306861111816 (+0.0036747741027617076) | > avg_loss_disc_real_3: 0.23361368792157777 (-0.006688060265191836) | > avg_loss_disc_real_4: 0.245943084359169 (+0.005919754715032977) | > avg_loss_disc_real_5: 0.2506537282131089 (+0.005432075178119611) | > avg_loss_0: 2.661647954457243 (-0.013884305953979048) | > avg_grad_norm_0: tensor(272.8500, device='cuda:0') (tensor(-48.0662, device='cuda:0')) | > avg_loss_gen: 2.0608842121043676 (-0.057111855963585256) | > avg_loss_kl: 2.0296963520453 (-0.013388993034900487) | > avg_loss_feat: 3.826160843943206 (+0.1627685721491421) | > avg_loss_mel: 16.938266539237866 (+0.1757766293807741) | > avg_loss_duration: 1.5327792906425368 (-0.0017387766233629698) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.387787268195357 (+0.26630576227752556) | > avg_grad_norm_1: tensor(6167.4712, device='cuda:0') (tensor(-702.7788, device='cuda:0'))  > EPOCH: 1384/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:28:05)   --> TIME: 2025-02-25 22:28:08 -- STEP: 1/72 -- GLOBAL_STEP: 1099650 | > loss_disc: 2.4375622272491455 (2.4375622272491455) | > loss_disc_real_0: 0.050404056906700134 (0.050404056906700134) | > loss_disc_real_1: 0.23195840418338776 (0.23195840418338776) | > loss_disc_real_2: 0.18601518869400024 (0.18601518869400024) | > loss_disc_real_3: 0.2568170726299286 (0.2568170726299286) | > loss_disc_real_4: 0.2507076859474182 (0.2507076859474182) | > loss_disc_real_5: 0.2481384426355362 (0.2481384426355362) | > loss_0: 2.4375622272491455 (2.4375622272491455) | > grad_norm_0: tensor(53.0455, device='cuda:0') (tensor(53.0455, device='cuda:0')) | > loss_gen: 2.341939926147461 (2.341939926147461) | > loss_kl: 2.070875644683838 (2.070875644683838) | > loss_feat: 4.368210315704346 (4.368210315704346) | > loss_mel: 17.237295150756836 (17.237295150756836) | > loss_duration: 1.4986532926559448 (1.4986532926559448) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.5169734954834 (27.5169734954834) | > grad_norm_1: tensor(2196.9656, device='cuda:0') (tensor(2196.9656, device='cuda:0')) | > current_lr_0: 0.00016822570386706404 | > current_lr_1: 0.00016822570386706404 | > step_time: 0.4322 (0.4322075843811035) | > loader_time: 0.006 (0.006048440933227539)  --> TIME: 2025-02-25 22:28:31 -- STEP: 51/72 -- GLOBAL_STEP: 1099700 | > loss_disc: 2.653824806213379 (2.676679527058321) | > loss_disc_real_0: 0.10446610301733017 (0.15759380388201444) | > loss_disc_real_1: 0.22105064988136292 (0.22063792745272318) | > loss_disc_real_2: 0.21977391839027405 (0.22878426108874528) | > loss_disc_real_3: 0.20858833193778992 (0.23784540389098374) | > loss_disc_real_4: 0.2150205373764038 (0.2406118404047162) | > loss_disc_real_5: 0.23995237052440643 (0.24450859047618567) | > loss_0: 2.653824806213379 (2.676679527058321) | > grad_norm_0: tensor(13.7514, device='cuda:0') (tensor(256.0629, device='cuda:0')) | > loss_gen: 2.107851505279541 (2.051087236871907) | > loss_kl: 1.9756742715835571 (2.0292138936472877) | > loss_feat: 3.5367207527160645 (3.6943849722544346) | > loss_mel: 15.76478385925293 (16.919463269850784) | > loss_duration: 1.5669171810150146 (1.5212307771046956) | > amp_scaler: 16.0 (10.196078431372548) | > loss_1: 24.951948165893555 (26.215380238551717) | > grad_norm_1: tensor(7430.9551, device='cuda:0') (tensor(5832.2476, device='cuda:0')) | > current_lr_0: 0.00016822570386706404 | > current_lr_1: 0.00016822570386706404 | > step_time: 0.4401 (0.4508341761196361) | > loader_time: 0.0089 (0.007662791831820619) --> EVAL PERFORMANCE | > avg_loader_time: 0.008537376430672658 (-3.369761184907408e-05) | > avg_step_time: 0.4604427478682827 (+0.009500715094552914) | > avg_loss_disc: 2.6646192476783 (+0.0029712932210572696) | > avg_loss_disc_real_0: 0.15707363104316546 (+0.028993370772247534) | > avg_loss_disc_real_1: 0.22103183139377916 (+0.005766807097784238) | > avg_loss_disc_real_2: 0.2306754043404485 (-0.006067664270669654) | > avg_loss_disc_real_3: 0.23651788205328123 (+0.0029041941317034525) | > avg_loss_disc_real_4: 0.24456785913084594 (-0.0013752252283230626) | > avg_loss_disc_real_5: 0.24602112719710445 (-0.004632601016004434) | > avg_loss_0: 2.6646192476783 (+0.0029712932210572696) | > avg_grad_norm_0: tensor(257.4111, device='cuda:0') (tensor(-15.4389, device='cuda:0')) | > avg_loss_gen: 2.067025738702694 (+0.00614152659832623) | > avg_loss_kl: 2.0603548523405912 (+0.03065850029529127) | > avg_loss_feat: 3.6626215719840896 (-0.16353927195911666) | > avg_loss_mel: 16.865496662301076 (-0.07276987693678905) | > avg_loss_duration: 1.5374874078052145 (+0.004708117162677716) | > avg_amp_scaler: 11.830985915492958 (+3.830985915492958) | > avg_loss_1: 26.192986233133663 (-0.194801035061694) | > avg_grad_norm_1: tensor(5807.6519, device='cuda:0') (tensor(-359.8193, device='cuda:0'))  > EPOCH: 1385/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:28:43)   --> TIME: 2025-02-25 22:28:58 -- STEP: 29/72 -- GLOBAL_STEP: 1099750 | > loss_disc: 2.491445779800415 (2.654244061174064) | > loss_disc_real_0: 0.16577669978141785 (0.06214904052944019) | > loss_disc_real_1: 0.1879439502954483 (0.22649392948068423) | > loss_disc_real_2: 0.15279342234134674 (0.29811153175501987) | > loss_disc_real_3: 0.1962442696094513 (0.27226068850221313) | > loss_disc_real_4: 0.220823273062706 (0.2552785744954801) | > loss_disc_real_5: 0.2166944295167923 (0.24766412325974166) | > loss_0: 2.491445779800415 (2.654244061174064) | > grad_norm_0: tensor(281.0159, device='cuda:0') (tensor(278.5310, device='cuda:0')) | > loss_gen: 1.9460610151290894 (2.2220155987246284) | > loss_kl: 2.154035806655884 (2.022886901066221) | > loss_feat: 3.6138927936553955 (4.1347114382119) | > loss_mel: 17.08283233642578 (16.878446874947382) | > loss_duration: 1.5243439674377441 (1.5024435931238636) | > amp_scaler: 8.0 (12.689655172413794) | > loss_1: 26.321165084838867 (26.76050429508604) | > grad_norm_1: tensor(8039.1519, device='cuda:0') (tensor(5050.9692, device='cuda:0')) | > current_lr_0: 0.00016820467565408065 | > current_lr_1: 0.00016820467565408065 | > step_time: 0.4314 (0.4389692421617179) | > loader_time: 0.0078 (0.007173118920161806) --> EVAL PERFORMANCE | > avg_loader_time: 0.008675632342486314 (+0.00013825591181365615) | > avg_step_time: 0.45511094617172027 (-0.005331801696562444) | > avg_loss_disc: 2.6324508156574944 (-0.03216843202080577) | > avg_loss_disc_real_0: 0.11613404283850966 (-0.040939588204655805) | > avg_loss_disc_real_1: 0.2350343341978503 (+0.014002502804071143) | > avg_loss_disc_real_2: 0.25701786493751366 (+0.02634246059706516) | > avg_loss_disc_real_3: 0.24866427439199373 (+0.012146392338712508) | > avg_loss_disc_real_4: 0.24850363840519551 (+0.003935779274349571) | > avg_loss_disc_real_5: 0.24841416910500594 (+0.0023930419079014886) | > avg_loss_0: 2.6324508156574944 (-0.03216843202080577) | > avg_grad_norm_0: tensor(245.3726, device='cuda:0') (tensor(-12.0385, device='cuda:0')) | > avg_loss_gen: 2.193597071607349 (+0.12657133290465516) | > avg_loss_kl: 2.2225551151893503 (+0.1622002628487591) | > avg_loss_feat: 3.9512795663215763 (+0.28865799433748673) | > avg_loss_mel: 17.435573013735482 (+0.5700763514344054) | > avg_loss_duration: 1.5530140299192616 (+0.015526622114047095) | > avg_amp_scaler: 9.915492957746478 (-1.9154929577464799) | > avg_loss_1: 27.356018764871948 (+1.1630325317382848) | > avg_grad_norm_1: tensor(5516.7500, device='cuda:0') (tensor(-290.9019, device='cuda:0'))  > EPOCH: 1386/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:29:19)   --> TIME: 2025-02-25 22:29:25 -- STEP: 7/72 -- GLOBAL_STEP: 1099800 | > loss_disc: 2.5727765560150146 (2.8177370003291538) | > loss_disc_real_0: 0.16837911307811737 (0.38112115008490427) | > loss_disc_real_1: 0.5308868288993835 (0.3773980438709259) | > loss_disc_real_2: 0.21138715744018555 (0.22616605034896306) | > loss_disc_real_3: 0.23437726497650146 (0.232072080884661) | > loss_disc_real_4: 0.23134271800518036 (0.24295289388724736) | > loss_disc_real_5: 0.21647422015666962 (0.22954119316169194) | > loss_0: 2.5727765560150146 (2.8177370003291538) | > grad_norm_0: tensor(59.1895, device='cuda:0') (tensor(339.2598, device='cuda:0')) | > loss_gen: 2.487375259399414 (2.4960267543792725) | > loss_kl: 2.260388135910034 (2.047235914639064) | > loss_feat: 3.989218235015869 (3.5024715832301547) | > loss_mel: 18.49462890625 (17.36634281703404) | > loss_duration: 1.4897781610488892 (1.5128186770847865) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.721389770507812 (26.924895422799246) | > grad_norm_1: tensor(5225.0464, device='cuda:0') (tensor(3696.4956, device='cuda:0')) | > current_lr_0: 0.0001681836500696239 | > current_lr_1: 0.0001681836500696239 | > step_time: 0.4306 (0.4309884820665632) | > loader_time: 0.0068 (0.006425789424351284)  --> TIME: 2025-02-25 22:29:48 -- STEP: 57/72 -- GLOBAL_STEP: 1099850 | > loss_disc: 2.587477207183838 (2.6860095701719584) | > loss_disc_real_0: 0.1556546688079834 (0.18148691350953622) | > loss_disc_real_1: 0.22590157389640808 (0.2487331051052661) | > loss_disc_real_2: 0.23020166158676147 (0.23372324438471542) | > loss_disc_real_3: 0.22766385972499847 (0.2374475287240848) | > loss_disc_real_4: 0.24205876886844635 (0.24551752810938315) | > loss_disc_real_5: 0.25012946128845215 (0.24716858931800775) | > loss_0: 2.587477207183838 (2.6860095701719584) | > grad_norm_0: tensor(417.8569, device='cuda:0') (tensor(246.9990, device='cuda:0')) | > loss_gen: 1.9908400774002075 (2.0560184528953167) | > loss_kl: 2.0827994346618652 (2.0691323050281456) | > loss_feat: 3.671518325805664 (3.566550585261562) | > loss_mel: 17.1046142578125 (16.961233072113565) | > loss_duration: 1.575728178024292 (1.5270526952910841) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.425500869750977 (26.17998708758438) | > grad_norm_1: tensor(13367.2480, device='cuda:0') (tensor(8540.8223, device='cuda:0')) | > current_lr_0: 0.0001681836500696239 | > current_lr_1: 0.0001681836500696239 | > step_time: 0.4514 (0.443904951999062) | > loader_time: 0.0096 (0.008048116115101596) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867458464394153 (-1.047698544785064e-06) | > avg_step_time: 0.4537127756736648 (-0.0013981704980554621) | > avg_loss_disc: 2.6734925458129024 (+0.04104173015540802) | > avg_loss_disc_real_0: 0.17069731049344572 (+0.05456326765493606) | > avg_loss_disc_real_1: 0.24184945792379503 (+0.006815123725944727) | > avg_loss_disc_real_2: 0.2323633035723592 (-0.024654561365154454) | > avg_loss_disc_real_3: 0.23565308539800242 (-0.013011188993991313) | > avg_loss_disc_real_4: 0.2456140828804231 (-0.002889555524772408) | > avg_loss_disc_real_5: 0.24751989312574896 (-0.000894275979256981) | > avg_loss_0: 2.6734925458129024 (+0.04104173015540802) | > avg_grad_norm_0: tensor(264.0602, device='cuda:0') (+tensor(18.6875, device='cuda:0')) | > avg_loss_gen: 2.040976969289109 (-0.15262010231823986) | > avg_loss_kl: 2.084550422681889 (-0.13800469250746117) | > avg_loss_feat: 3.574017091536186 (-0.3772624747853901) | > avg_loss_mel: 16.93546524853773 (-0.5001077651977504) | > avg_loss_duration: 1.538322925567627 (-0.014691104351634676) | > avg_amp_scaler: 8.0 (-1.915492957746478) | > avg_loss_1: 26.173332671044577 (-1.182686093827371) | > avg_grad_norm_1: tensor(8622.6904, device='cuda:0') (+tensor(3105.9404, device='cuda:0'))  > EPOCH: 1387/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:29:56)   --> TIME: 2025-02-25 22:30:14 -- STEP: 35/72 -- GLOBAL_STEP: 1099900 | > loss_disc: 2.513540744781494 (2.5736554350171765) | > loss_disc_real_0: 0.1118447333574295 (0.11283721242632185) | > loss_disc_real_1: 0.2178182750940323 (0.2123205359492983) | > loss_disc_real_2: 0.2297552227973938 (0.2358525093112673) | > loss_disc_real_3: 0.2347404956817627 (0.24033723090376172) | > loss_disc_real_4: 0.25630009174346924 (0.24961032143660955) | > loss_disc_real_5: 0.2993878722190857 (0.2577616091285434) | > loss_0: 2.513540744781494 (2.5736554350171765) | > grad_norm_0: tensor(173.4614, device='cuda:0') (tensor(337.3174, device='cuda:0')) | > loss_gen: 2.277395009994507 (2.1070568016597195) | > loss_kl: 2.090874195098877 (1.9368701321738107) | > loss_feat: 3.7360610961914062 (3.8451846258980886) | > loss_mel: 16.432600021362305 (16.712320164271762) | > loss_duration: 1.537988543510437 (1.4941305501120432) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.074918746948242 (26.095562362670897) | > grad_norm_1: tensor(12738.3740, device='cuda:0') (tensor(11336.6855, device='cuda:0')) | > current_lr_0: 0.0001681626271133652 | > current_lr_1: 0.0001681626271133652 | > step_time: 0.4348 (0.4338080406188965) | > loader_time: 0.0085 (0.00709881101335798) --> EVAL PERFORMANCE | > avg_loader_time: 0.008659933654355332 (-1.4650989586197441e-05) | > avg_step_time: 0.4517539084797174 (-0.001958867193947389) | > avg_loss_disc: 2.571791766395031 (-0.10170077941787126) | > avg_loss_disc_real_0: 0.11579894814902628 (-0.054898362344419444) | > avg_loss_disc_real_1: 0.21585268936526608 (-0.02599676855852895) | > avg_loss_disc_real_2: 0.23318628819895462 (+0.0008229846265954133) | > avg_loss_disc_real_3: 0.2329136225539194 (-0.0027394628440830238) | > avg_loss_disc_real_4: 0.2452162693084126 (-0.00039781357201051915) | > avg_loss_disc_real_5: 0.24955574979244832 (+0.002035856666699365) | > avg_loss_0: 2.571791766395031 (-0.10170077941787126) | > avg_grad_norm_0: tensor(351.2604, device='cuda:0') (+tensor(87.2003, device='cuda:0')) | > avg_loss_gen: 2.094964301082449 (+0.0539873317933397) | > avg_loss_kl: 1.9983981763812857 (-0.08615224630060347) | > avg_loss_feat: 3.820715652385228 (+0.24669856084904174) | > avg_loss_mel: 16.68312455566836 (-0.252340692869371) | > avg_loss_duration: 1.5276186986708304 (-0.01070422689679651) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.124821407694213 (-0.048511263350363265) | > avg_grad_norm_1: tensor(11255.0469, device='cuda:0') (+tensor(2632.3564, device='cuda:0'))  > EPOCH: 1388/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:30:33)   --> TIME: 2025-02-25 22:30:41 -- STEP: 13/72 -- GLOBAL_STEP: 1099950 | > loss_disc: 2.5356874465942383 (2.5342848117534933) | > loss_disc_real_0: 0.13451050221920013 (0.11341428814026025) | > loss_disc_real_1: 0.20515094697475433 (0.21687205479695246) | > loss_disc_real_2: 0.206463024020195 (0.23220659792423248) | > loss_disc_real_3: 0.18559451401233673 (0.2395270191706144) | > loss_disc_real_4: 0.2274647206068039 (0.2462090700864792) | > loss_disc_real_5: 0.24826562404632568 (0.24021616463477796) | > loss_0: 2.5356874465942383 (2.5342848117534933) | > grad_norm_0: tensor(378.3082, device='cuda:0') (tensor(324.6688, device='cuda:0')) | > loss_gen: 1.991021752357483 (2.1678571425951443) | > loss_kl: 2.0282700061798096 (1.8783304691314697) | > loss_feat: 4.037989139556885 (3.9505633207467885) | > loss_mel: 16.273269653320312 (16.474880071786735) | > loss_duration: 1.4201664924621582 (1.4693811123187726) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.750717163085938 (25.941012455866886) | > grad_norm_1: tensor(10801.5273, device='cuda:0') (tensor(10554.5781, device='cuda:0')) | > current_lr_0: 0.000168141606784976 | > current_lr_1: 0.000168141606784976 | > step_time: 0.4431 (0.4252992593325101) | > loader_time: 0.0058 (0.006342649459838867)  --> TIME: 2025-02-25 22:31:04 -- STEP: 63/72 -- GLOBAL_STEP: 1100000 | > loss_disc: 2.4962637424468994 (2.556654725755964) | > loss_disc_real_0: 0.15334613621234894 (0.1148799778450103) | > loss_disc_real_1: 0.2158443182706833 (0.21596292087009974) | > loss_disc_real_2: 0.22900651395320892 (0.23307221725819602) | > loss_disc_real_3: 0.2549692988395691 (0.23454626283002278) | > loss_disc_real_4: 0.24959449470043182 (0.24563533848240263) | > loss_disc_real_5: 0.24873791635036469 (0.24668375130683656) | > loss_0: 2.4962637424468994 (2.556654725755964) | > grad_norm_0: tensor(355.4293, device='cuda:0') (tensor(352.0938, device='cuda:0')) | > loss_gen: 2.244903802871704 (2.1413129170735665) | > loss_kl: 2.035775899887085 (1.9444739705040341) | > loss_feat: 3.494617462158203 (3.8827229681469144) | > loss_mel: 16.741621017456055 (16.38615957895914) | > loss_duration: 1.5756597518920898 (1.5145516073892986) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.092578887939453 (25.86922115749783) | > grad_norm_1: tensor(11013.1992, device='cuda:0') (tensor(11610.8818, device='cuda:0')) | > current_lr_0: 0.000168141606784976 | > current_lr_1: 0.000168141606784976 | > step_time: 0.4843 (0.4537102532765222) | > loader_time: 0.0114 (0.008151841542077444) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1100000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008545590118623115 (-0.00011434353573221666) | > avg_step_time: 0.458254337310791 (+0.006500428831073601) | > avg_loss_disc: 2.555938760999223 (-0.01585300539580814) | > avg_loss_disc_real_0: 0.11554708058985186 (-0.00025186755917441794) | > avg_loss_disc_real_1: 0.21550146794654956 (-0.0003512214187165186) | > avg_loss_disc_real_2: 0.23258317881067034 (-0.0006031093882842842) | > avg_loss_disc_real_3: 0.2335097311248242 (+0.0005961085709048075) | > avg_loss_disc_real_4: 0.24489742937222334 (-0.00031883993618925244) | > avg_loss_disc_real_5: 0.24635606479476874 (-0.003199684997679586) | > avg_loss_0: 2.555938760999223 (-0.01585300539580814) | > avg_grad_norm_0: tensor(360.0309, device='cuda:0') (+tensor(8.7705, device='cuda:0')) | > avg_loss_gen: 2.140246606208908 (+0.04528230512645903) | > avg_loss_kl: 1.9615487934837879 (-0.03684938289749784) | > avg_loss_feat: 3.883025955146467 (+0.06231030276123928) | > avg_loss_mel: 16.43184628285152 (-0.25127827281684034) | > avg_loss_duration: 1.522889666154351 (-0.004729032516479492) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.93955738443724 (-0.18526402325697333) | > avg_grad_norm_1: tensor(11727.4160, device='cuda:0') (+tensor(472.3691, device='cuda:0'))  > EPOCH: 1389/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:31:13)   --> TIME: 2025-02-25 22:31:33 -- STEP: 41/72 -- GLOBAL_STEP: 1100050 | > loss_disc: 2.536907434463501 (2.552901052847141) | > loss_disc_real_0: 0.11891579627990723 (0.11691016249540376) | > loss_disc_real_1: 0.2195967584848404 (0.21938367769485567) | > loss_disc_real_2: 0.23209460079669952 (0.2307577256749316) | > loss_disc_real_3: 0.21795587241649628 (0.23006567031871983) | > loss_disc_real_4: 0.23566928505897522 (0.24364111263577531) | > loss_disc_real_5: 0.21473336219787598 (0.24502911327815638) | > loss_0: 2.536907434463501 (2.552901052847141) | > grad_norm_0: tensor(120.6622, device='cuda:0') (tensor(374.6550, device='cuda:0')) | > loss_gen: 2.1402342319488525 (2.1683135556011663) | > loss_kl: 2.0045759677886963 (1.9138634524694302) | > loss_feat: 3.913356065750122 (3.9701717655833173) | > loss_mel: 16.914743423461914 (16.518526100530853) | > loss_duration: 1.5171263217926025 (1.493034575043655) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.490036010742188 (26.063909344556855) | > grad_norm_1: tensor(8805.0771, device='cuda:0') (tensor(11515.1943, device='cuda:0')) | > current_lr_0: 0.0001681205890841279 | > current_lr_1: 0.0001681205890841279 | > step_time: 0.4427 (0.4351991385948367) | > loader_time: 0.0093 (0.007231311100285228) --> EVAL PERFORMANCE | > avg_loader_time: 0.008473658225905727 (-7.193189271738863e-05) | > avg_step_time: 0.4534805626936362 (-0.004773774617154836) | > avg_loss_disc: 2.561516640891491 (+0.0055778798922681005) | > avg_loss_disc_real_0: 0.11634634298757768 (+0.0007992623977258195) | > avg_loss_disc_real_1: 0.21951556205749512 (+0.004014094110945554) | > avg_loss_disc_real_2: 0.23335675601388367 (+0.0007735772032133326) | > avg_loss_disc_real_3: 0.232261179828308 (-0.0012485512965162149) | > avg_loss_disc_real_4: 0.24513527646031177 (+0.00023784708808843869) | > avg_loss_disc_real_5: 0.24615861496455232 (-0.00019744983021641183) | > avg_loss_0: 2.561516640891491 (+0.0055778798922681005) | > avg_grad_norm_0: tensor(354.0718, device='cuda:0') (tensor(-5.9591, device='cuda:0')) | > avg_loss_gen: 2.1568771634303356 (+0.01663055722142781) | > avg_loss_kl: 1.967911292129839 (+0.006362498646051096) | > avg_loss_feat: 3.900780479672929 (+0.017754524526461868) | > avg_loss_mel: 16.399451954264034 (-0.03239432858748614) | > avg_loss_duration: 1.5225008440689303 (-0.0003888220854206814) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.947521612677775 (+0.00796422824053522) | > avg_grad_norm_1: tensor(11090.0127, device='cuda:0') (tensor(-637.4033, device='cuda:0'))  > EPOCH: 1390/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:31:50)   --> TIME: 2025-02-25 22:32:00 -- STEP: 19/72 -- GLOBAL_STEP: 1100100 | > loss_disc: 2.490684986114502 (2.557254352067646) | > loss_disc_real_0: 0.13508707284927368 (0.10807904208961286) | > loss_disc_real_1: 0.18093150854110718 (0.2192856230233845) | > loss_disc_real_2: 0.21272552013397217 (0.22934217751026154) | > loss_disc_real_3: 0.21649934351444244 (0.22911954082940755) | > loss_disc_real_4: 0.2328924536705017 (0.24031141870900205) | > loss_disc_real_5: 0.23862355947494507 (0.24345692913783223) | > loss_0: 2.490684986114502 (2.557254352067646) | > grad_norm_0: tensor(525.0812, device='cuda:0') (tensor(345.2621, device='cuda:0')) | > loss_gen: 2.1523611545562744 (2.1603398887734664) | > loss_kl: 1.8696341514587402 (1.859176792596516) | > loss_feat: 4.086881637573242 (3.943039768620541) | > loss_mel: 17.142301559448242 (16.638887907329362) | > loss_duration: 1.461484670639038 (1.4724808680383783) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.712663650512695 (26.073925118697318) | > grad_norm_1: tensor(16193.2686, device='cuda:0') (tensor(11973.2695, device='cuda:0')) | > current_lr_0: 0.00016809957401049238 | > current_lr_1: 0.00016809957401049238 | > step_time: 0.4333 (0.42931266834861354) | > loader_time: 0.006 (0.006853429894698293)  --> TIME: 2025-02-25 22:32:24 -- STEP: 69/72 -- GLOBAL_STEP: 1100150 | > loss_disc: 2.6411070823669434 (2.5644511623658994) | > loss_disc_real_0: 0.13671360909938812 (0.11259559498748918) | > loss_disc_real_1: 0.2387089878320694 (0.2177729811789333) | > loss_disc_real_2: 0.23405703902244568 (0.2292490195536959) | > loss_disc_real_3: 0.24910327792167664 (0.23060936513154404) | > loss_disc_real_4: 0.23392415046691895 (0.24346802092116812) | > loss_disc_real_5: 0.24130792915821075 (0.24548291101835776) | > loss_0: 2.6411070823669434 (2.5644511623658994) | > grad_norm_0: tensor(417.5570, device='cuda:0') (tensor(357.3200, device='cuda:0')) | > loss_gen: 2.0017826557159424 (2.1596346817154823) | > loss_kl: 2.2570478916168213 (1.9798733306967693) | > loss_feat: 3.63071870803833 (3.959173706994538) | > loss_mel: 16.464534759521484 (16.45323012531666) | > loss_duration: 1.6265416145324707 (1.5208210184954214) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.980627059936523 (26.072732870129567) | > grad_norm_1: tensor(14342.4346, device='cuda:0') (tensor(11862.1875, device='cuda:0')) | > current_lr_0: 0.00016809957401049238 | > current_lr_1: 0.00016809957401049238 | > step_time: 0.5345 (0.4501833881157032) | > loader_time: 0.013 (0.008600901866304703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647250457548761 (+0.0001735922316430348) | > avg_step_time: 0.45251738857215557 (-0.0009631741214806144) | > avg_loss_disc: 2.5656695936767147 (+0.00415295278522354) | > avg_loss_disc_real_0: 0.11378403836992425 (-0.0025623046176534298) | > avg_loss_disc_real_1: 0.21761774953822016 (-0.001897812519274955) | > avg_loss_disc_real_2: 0.22905951969220606 (-0.004297236321677611) | > avg_loss_disc_real_3: 0.23076778209545243 (-0.0014933977328555614) | > avg_loss_disc_real_4: 0.24334366388723883 (-0.0017916125730729482) | > avg_loss_disc_real_5: 0.24537748392199127 (-0.0007811310425610563) | > avg_loss_0: 2.5656695936767147 (+0.00415295278522354) | > avg_grad_norm_0: tensor(355.6252, device='cuda:0') (+tensor(1.5534, device='cuda:0')) | > avg_loss_gen: 2.1618564212825944 (+0.004979257852258723) | > avg_loss_kl: 1.9827718163879824 (+0.014860524258143437) | > avg_loss_feat: 3.965943934212266 (+0.06516345453933692) | > avg_loss_mel: 16.453056416041402 (+0.05360446177736833) | > avg_loss_duration: 1.5215473191838869 (-0.0009535248850434108) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.085175930614195 (+0.1376543179364198) | > avg_grad_norm_1: tensor(11838.6855, device='cuda:0') (+tensor(748.6729, device='cuda:0'))  > EPOCH: 1391/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:32:27)   --> TIME: 2025-02-25 22:32:50 -- STEP: 47/72 -- GLOBAL_STEP: 1100200 | > loss_disc: 2.528419256210327 (2.5730448377893325) | > loss_disc_real_0: 0.2031261920928955 (0.12020714359080538) | > loss_disc_real_1: 0.2507619261741638 (0.2177103891017589) | > loss_disc_real_2: 0.2176806926727295 (0.22624928615194687) | > loss_disc_real_3: 0.21749401092529297 (0.23221457860571273) | > loss_disc_real_4: 0.23816709220409393 (0.2434557723872205) | > loss_disc_real_5: 0.2403104603290558 (0.24659569085912503) | > loss_0: 2.528419256210327 (2.5730448377893325) | > grad_norm_0: tensor(429.3118, device='cuda:0') (tensor(363.3818, device='cuda:0')) | > loss_gen: 2.3415911197662354 (2.1668228291450657) | > loss_kl: 2.0027782917022705 (1.9523859328411994) | > loss_feat: 3.810622453689575 (3.9895764766855444) | > loss_mel: 16.810218811035156 (16.62967170553005) | > loss_duration: 1.5537996292114258 (1.4952862490998937) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.51900863647461 (26.233743018292365) | > grad_norm_1: tensor(9422.5361, device='cuda:0') (tensor(11650.8984, device='cuda:0')) | > current_lr_0: 0.00016807856156374106 | > current_lr_1: 0.00016807856156374106 | > step_time: 0.4474 (0.44182677979164936) | > loader_time: 0.0088 (0.007619594005828208) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585688094018216 (-6.156236353054532e-05) | > avg_step_time: 0.4536989406800606 (+0.0011815521079050484) | > avg_loss_disc: 2.5788197517395024 (+0.013150158062787742) | > avg_loss_disc_real_0: 0.12075148071621505 (+0.006967442346290806) | > avg_loss_disc_real_1: 0.2194097512204882 (+0.0017920016822680351) | > avg_loss_disc_real_2: 0.22600025966973372 (-0.0030592600224723376) | > avg_loss_disc_real_3: 0.23122043697766856 (+0.00045265488221613026) | > avg_loss_disc_real_4: 0.24355052054767878 (+0.00020685666043995798) | > avg_loss_disc_real_5: 0.24632423108732196 (+0.0009467471653306914) | > avg_loss_0: 2.5788197517395024 (+0.013150158062787742) | > avg_grad_norm_0: tensor(353.4378, device='cuda:0') (tensor(-2.1874, device='cuda:0')) | > avg_loss_gen: 2.1539554579157234 (-0.007900963366870961) | > avg_loss_kl: 1.9965483856872774 (+0.01377656929929505) | > avg_loss_feat: 3.9197997543173777 (-0.04614417989488828) | > avg_loss_mel: 16.488388746557106 (+0.03533233051570406) | > avg_loss_duration: 1.5191843106713094 (-0.0023630085125774247) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.077876534260493 (-0.007299396353701582) | > avg_grad_norm_1: tensor(11328.3193, device='cuda:0') (tensor(-510.3662, device='cuda:0'))  > EPOCH: 1392/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:33:04)   --> TIME: 2025-02-25 22:33:17 -- STEP: 25/72 -- GLOBAL_STEP: 1100250 | > loss_disc: 2.542804718017578 (2.6125065898895268) | > loss_disc_real_0: 0.11749637871980667 (0.12319721028208733) | > loss_disc_real_1: 0.24143683910369873 (0.2470930516719819) | > loss_disc_real_2: 0.22047559916973114 (0.23022553205490112) | > loss_disc_real_3: 0.21861401200294495 (0.2356431668996811) | > loss_disc_real_4: 0.23930497467517853 (0.2403386116027832) | > loss_disc_real_5: 0.25736203789711 (0.24329858541488647) | > loss_0: 2.542804718017578 (2.6125065898895268) | > grad_norm_0: tensor(410.7056, device='cuda:0') (tensor(350.9021, device='cuda:0')) | > loss_gen: 2.0557971000671387 (2.161415681838989) | > loss_kl: 2.1297190189361572 (1.9290398502349853) | > loss_feat: 3.640428066253662 (3.969373207092285) | > loss_mel: 15.521011352539062 (16.511519699096684) | > loss_duration: 1.4887101650238037 (1.4748304843902589) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.83566665649414 (26.046178894042967) | > grad_norm_1: tensor(12918.8994, device='cuda:0') (tensor(11219.4199, device='cuda:0')) | > current_lr_0: 0.0001680575517435456 | > current_lr_1: 0.0001680575517435456 | > step_time: 0.426 (0.43674650192260744) | > loader_time: 0.0073 (0.006943845748901367) --> EVAL PERFORMANCE | > avg_loader_time: 0.008664957234557246 (+7.926914053902993e-05) | > avg_step_time: 0.4538394397413227 (+0.00014049906126206713) | > avg_loss_disc: 2.594776549809416 (+0.01595679806991379) | > avg_loss_disc_real_0: 0.12346399958494683 (+0.002712518868731778) | > avg_loss_disc_real_1: 0.23054798472095545 (+0.011138233500467248) | > avg_loss_disc_real_2: 0.2316558732533119 (+0.0056556135835781896) | > avg_loss_disc_real_3: 0.2322799842542326 (+0.0010595472765640435) | > avg_loss_disc_real_4: 0.2412809582663254 (-0.0022695622813533878) | > avg_loss_disc_real_5: 0.2440848900398738 (-0.0022393410474481623) | > avg_loss_0: 2.594776549809416 (+0.01595679806991379) | > avg_grad_norm_0: tensor(365.7086, device='cuda:0') (+tensor(12.2708, device='cuda:0')) | > avg_loss_gen: 2.1593647255024435 (+0.005409267586720112) | > avg_loss_kl: 1.999697164750435 (+0.0031487790631574697) | > avg_loss_feat: 3.921211450872287 (+0.0014116965549093585) | > avg_loss_mel: 16.543180089601325 (+0.05479134304421862) | > avg_loss_duration: 1.5222923839595954 (+0.003108073288285995) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.145745908710317 (+0.067869374449824) | > avg_grad_norm_1: tensor(11492.0947, device='cuda:0') (+tensor(163.7754, device='cuda:0'))  > EPOCH: 1393/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:33:41)   --> TIME: 2025-02-25 22:33:44 -- STEP: 3/72 -- GLOBAL_STEP: 1100300 | > loss_disc: 2.4849538803100586 (2.545365571975708) | > loss_disc_real_0: 0.12949927151203156 (0.09671588242053986) | > loss_disc_real_1: 0.20346660912036896 (0.19246646265188852) | > loss_disc_real_2: 0.2453983575105667 (0.23831519981225333) | > loss_disc_real_3: 0.2377013862133026 (0.2322750041882197) | > loss_disc_real_4: 0.2477749139070511 (0.2429913729429245) | > loss_disc_real_5: 0.2522103488445282 (0.24126712481180826) | > loss_0: 2.4849538803100586 (2.545365571975708) | > grad_norm_0: tensor(236.5777, device='cuda:0') (tensor(355.3547, device='cuda:0')) | > loss_gen: 2.272437572479248 (2.1150041818618774) | > loss_kl: 1.792392373085022 (1.7700964609781902) | > loss_feat: 3.635833740234375 (3.8337029616038003) | > loss_mel: 14.552691459655762 (15.825425783793131) | > loss_duration: 1.4807974100112915 (1.4824556112289429) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.73415184020996 (25.026684443155926) | > grad_norm_1: tensor(9251.9033, device='cuda:0') (tensor(11831.6904, device='cuda:0')) | > current_lr_0: 0.00016803654454957765 | > current_lr_1: 0.00016803654454957765 | > step_time: 0.4442 (0.4375738302866618) | > loader_time: 0.0057 (0.00533143679300944)  --> TIME: 2025-02-25 22:34:06 -- STEP: 53/72 -- GLOBAL_STEP: 1100350 | > loss_disc: 2.6667895317077637 (2.5620802078606943) | > loss_disc_real_0: 0.07599513232707977 (0.1137452188808963) | > loss_disc_real_1: 0.24138078093528748 (0.21983672059931844) | > loss_disc_real_2: 0.23399296402931213 (0.23016470867507863) | > loss_disc_real_3: 0.26187002658843994 (0.23149168069632547) | > loss_disc_real_4: 0.2666667401790619 (0.24294964772350383) | > loss_disc_real_5: 0.2601165771484375 (0.24467099275229112) | > loss_0: 2.6667895317077637 (2.5620802078606943) | > grad_norm_0: tensor(217.1125, device='cuda:0') (tensor(354.0784, device='cuda:0')) | > loss_gen: 2.088184118270874 (2.188808569368327) | > loss_kl: 2.1998257637023926 (1.9517665224255256) | > loss_feat: 3.9421024322509766 (4.059796936107131) | > loss_mel: 16.130104064941406 (16.513236927536294) | > loss_duration: 1.567138671875 (1.5070815626180396) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.92735481262207 (26.22069045732606) | > grad_norm_1: tensor(10097.2041, device='cuda:0') (tensor(12229.1191, device='cuda:0')) | > current_lr_0: 0.00016803654454957765 | > current_lr_1: 0.00016803654454957765 | > step_time: 0.4376 (0.4381625067512944) | > loader_time: 0.0089 (0.007668841559931917) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514092002116459 (-0.00015086523244078713) | > avg_step_time: 0.45044857347515266 (-0.003390866266170023) | > avg_loss_disc: 2.569229159556644 (-0.025547390252772306) | > avg_loss_disc_real_0: 0.11796490554238709 (-0.0054990940425597434) | > avg_loss_disc_real_1: 0.21840610596495616 (-0.012141878755999286) | > avg_loss_disc_real_2: 0.23039451682231796 (-0.001261356430993954) | > avg_loss_disc_real_3: 0.232715260394862 (+0.0004352761406294092) | > avg_loss_disc_real_4: 0.242883992027229 (+0.0016030337609035938) | > avg_loss_disc_real_5: 0.24406475353408869 (-2.013650578511217e-05) | > avg_loss_0: 2.569229159556644 (-0.025547390252772306) | > avg_grad_norm_0: tensor(350.4063, device='cuda:0') (tensor(-15.3022, device='cuda:0')) | > avg_loss_gen: 2.17925122086431 (+0.019886495361866263) | > avg_loss_kl: 1.991904638182949 (-0.007792526567485858) | > avg_loss_feat: 4.0048716974930025 (+0.08366024662071547) | > avg_loss_mel: 16.561803656564624 (+0.018623566963299254) | > avg_loss_duration: 1.5235566035122936 (+0.0012642195526981759) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.26138778471611 (+0.11564187600579245) | > avg_grad_norm_1: tensor(11968.2344, device='cuda:0') (+tensor(476.1396, device='cuda:0'))  > EPOCH: 1394/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:34:17)   --> TIME: 2025-02-25 22:34:33 -- STEP: 31/72 -- GLOBAL_STEP: 1100400 | > loss_disc: 2.8722965717315674 (2.7858121241292646) | > loss_disc_real_0: 0.37979578971862793 (0.21578258420190505) | > loss_disc_real_1: 0.2622297704219818 (0.25016779957279084) | > loss_disc_real_2: 0.18043656647205353 (0.23038639320481208) | > loss_disc_real_3: 0.29977625608444214 (0.24128750783781852) | > loss_disc_real_4: 0.27980098128318787 (0.244302743384915) | > loss_disc_real_5: 0.24562180042266846 (0.244750369460352) | > loss_0: 2.8722965717315674 (2.7858121241292646) | > grad_norm_0: tensor(563.3478, device='cuda:0') (tensor(412.9869, device='cuda:0')) | > loss_gen: 2.2314963340759277 (2.07169423949334) | > loss_kl: 2.376906156539917 (2.075077745222277) | > loss_feat: 3.7763891220092773 (3.7192967399474113) | > loss_mel: 17.555143356323242 (16.8341586512904) | > loss_duration: 1.5587239265441895 (1.5070417657975228) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.498659133911133 (26.207269237887473) | > grad_norm_1: tensor(624.4122, device='cuda:0') (tensor(7790.9077, device='cuda:0')) | > current_lr_0: 0.00016801553998150895 | > current_lr_1: 0.00016801553998150895 | > step_time: 0.4403 (0.44030460234611263) | > loader_time: 0.0086 (0.007307360249180947) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693698426367533 (+0.00017960642425107406) | > avg_step_time: 0.45587919463574045 (+0.00543062116058779) | > avg_loss_disc: 2.727157377860915 (+0.15792821830427117) | > avg_loss_disc_real_0: 0.19694134697947704 (+0.07897644143708996) | > avg_loss_disc_real_1: 0.23791368095807627 (+0.019507574993120114) | > avg_loss_disc_real_2: 0.23158126429352963 (+0.0011867474712116688) | > avg_loss_disc_real_3: 0.23584815901769718 (+0.0031328986228351674) | > avg_loss_disc_real_4: 0.24232407438922937 (-0.0005599176379996185) | > avg_loss_disc_real_5: 0.24526920704774455 (+0.001204453513655862) | > avg_loss_0: 2.727157377860915 (+0.15792821830427117) | > avg_grad_norm_0: tensor(337.4652, device='cuda:0') (tensor(-12.9411, device='cuda:0')) | > avg_loss_gen: 2.1235777374724276 (-0.05567348339188216) | > avg_loss_kl: 2.3857297511167928 (+0.3938251129338437) | > avg_loss_feat: 3.819610296840399 (-0.18526140065260366) | > avg_loss_mel: 18.04179491123682 (+1.4799912546721963) | > avg_loss_duration: 1.5724732842243894 (+0.04891668071209576) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.943186155507263 (+1.681798370791153) | > avg_grad_norm_1: tensor(6394.7100, device='cuda:0') (tensor(-5573.5244, device='cuda:0'))  > EPOCH: 1395/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:34:54)   --> TIME: 2025-02-25 22:35:00 -- STEP: 9/72 -- GLOBAL_STEP: 1100450 | > loss_disc: 2.795017719268799 (2.6423556804656982) | > loss_disc_real_0: 0.24613940715789795 (0.11602828941411442) | > loss_disc_real_1: 0.20054295659065247 (0.22704360054598915) | > loss_disc_real_2: 0.18544629216194153 (0.23690538439485762) | > loss_disc_real_3: 0.24109911918640137 (0.24894649783770242) | > loss_disc_real_4: 0.2428663969039917 (0.25665084852112663) | > loss_disc_real_5: 0.25100529193878174 (0.26036622954739463) | > loss_0: 2.795017719268799 (2.6423556804656982) | > grad_norm_0: tensor(515.5062, device='cuda:0') (tensor(354.9960, device='cuda:0')) | > loss_gen: 1.8984013795852661 (2.2101822826597424) | > loss_kl: 2.116910696029663 (2.2358668910132513) | > loss_feat: 3.32549786567688 (3.9563611348470054) | > loss_mel: 17.492412567138672 (17.754312091403538) | > loss_duration: 1.5094127655029297 (1.5290531184938219) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.342636108398438 (27.68577554490831) | > grad_norm_1: tensor(4645.4155, device='cuda:0') (tensor(11421.5039, device='cuda:0')) | > current_lr_0: 0.00016799453803901124 | > current_lr_1: 0.00016799453803901124 | > step_time: 0.4361 (0.43219417995876735) | > loader_time: 0.0056 (0.006452719370524089)  --> TIME: 2025-02-25 22:35:24 -- STEP: 59/72 -- GLOBAL_STEP: 1100500 | > loss_disc: 2.214810371398926 (2.5818000243882007) | > loss_disc_real_0: 0.05798829346895218 (0.11520806857842507) | > loss_disc_real_1: 0.20153547823429108 (0.2209823624057285) | > loss_disc_real_2: 0.24279557168483734 (0.2300253773139695) | > loss_disc_real_3: 0.19199368357658386 (0.23864147294375856) | > loss_disc_real_4: 0.26188263297080994 (0.2393799052905228) | > loss_disc_real_5: 0.25986406207084656 (0.24609809431989313) | > loss_0: 2.214810371398926 (2.5818000243882007) | > grad_norm_0: tensor(77.9519, device='cuda:0') (tensor(300.6008, device='cuda:0')) | > loss_gen: 2.7384369373321533 (2.295411881753954) | > loss_kl: 2.275596857070923 (2.1426518973657633) | > loss_feat: 5.101487159729004 (4.245334079710103) | > loss_mel: 17.0804500579834 (17.261929059432724) | > loss_duration: 1.5566259622573853 (1.5379197395454018) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.75259780883789 (27.483246625480007) | > grad_norm_1: tensor(4500.2607, device='cuda:0') (tensor(7076.2212, device='cuda:0')) | > current_lr_0: 0.00016799453803901124 | > current_lr_1: 0.00016799453803901124 | > step_time: 0.4582 (0.4468047982555325) | > loader_time: 0.0097 (0.008073358212487175) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861544340429172 (-7.82550220758127e-05) | > avg_step_time: 0.45562523855289944 (-0.00025395608284101145) | > avg_loss_disc: 2.5790360007487556 (-0.14812137711215945) | > avg_loss_disc_real_0: 0.1072838680005409 (-0.08965747897893614) | > avg_loss_disc_real_1: 0.2224555941114963 (-0.015458086846579983) | > avg_loss_disc_real_2: 0.23324503382326853 (+0.001663769529738901) | > avg_loss_disc_real_3: 0.2344595926328444 (-0.0013885663848527685) | > avg_loss_disc_real_4: 0.239775408741454 (-0.0025486656477753666) | > avg_loss_disc_real_5: 0.2446820236427683 (-0.0005871834049762381) | > avg_loss_0: 2.5790360007487556 (-0.14812137711215945) | > avg_grad_norm_0: tensor(288.8957, device='cuda:0') (tensor(-48.5695, device='cuda:0')) | > avg_loss_gen: 2.2839293547079595 (+0.16035161723553193) | > avg_loss_kl: 2.1474076425525497 (-0.23832210856424307) | > avg_loss_feat: 4.244152082523829 (+0.4245417856834304) | > avg_loss_mel: 17.17999768592942 (-0.8617972253074022) | > avg_loss_duration: 1.5468603456524057 (-0.025612938571983657) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.402347108008158 (-0.5408390474991052) | > avg_grad_norm_1: tensor(7156.5190, device='cuda:0') (+tensor(761.8091, device='cuda:0'))  > EPOCH: 1396/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:35:31)   --> TIME: 2025-02-25 22:35:50 -- STEP: 37/72 -- GLOBAL_STEP: 1100550 | > loss_disc: 2.55167555809021 (2.5368693197095715) | > loss_disc_real_0: 0.07483135163784027 (0.11834740794792369) | > loss_disc_real_1: 0.18071182072162628 (0.22003683608931465) | > loss_disc_real_2: 0.22115422785282135 (0.22362658864743) | > loss_disc_real_3: 0.18977761268615723 (0.23002514605586594) | > loss_disc_real_4: 0.22954988479614258 (0.23280536927081444) | > loss_disc_real_5: 0.21407827734947205 (0.24624457069345432) | > loss_0: 2.55167555809021 (2.5368693197095715) | > grad_norm_0: tensor(268.9232, device='cuda:0') (tensor(279.8534, device='cuda:0')) | > loss_gen: 2.038032054901123 (2.331410714097925) | > loss_kl: 2.108656167984009 (1.9756002490584914) | > loss_feat: 4.063285827636719 (4.310565755173966) | > loss_mel: 15.951385498046875 (16.69834294190278) | > loss_duration: 1.5459964275360107 (1.5029200347694192) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.707355499267578 (26.81883971755569) | > grad_norm_1: tensor(9519.8467, device='cuda:0') (tensor(6696.4336, device='cuda:0')) | > current_lr_0: 0.00016797353872175636 | > current_lr_1: 0.00016797353872175636 | > step_time: 0.4751 (0.4372335446847452) | > loader_time: 0.008 (0.007201594275397224) --> EVAL PERFORMANCE | > avg_loader_time: 0.00867903736275686 (+6.359395846513953e-05) | > avg_step_time: 0.45165716762274083 (-0.003968070930158607) | > avg_loss_disc: 2.515331483223069 (-0.06370451752568673) | > avg_loss_disc_real_0: 0.10253052909294484 (-0.004753338907596064) | > avg_loss_disc_real_1: 0.21933633131040653 (-0.0031192628010897616) | > avg_loss_disc_real_2: 0.22259525220159074 (-0.01064978162167779) | > avg_loss_disc_real_3: 0.22908259193662187 (-0.005377000696222545) | > avg_loss_disc_real_4: 0.2357845083928444 (-0.003990900348609605) | > avg_loss_disc_real_5: 0.2458004491849685 (+0.0011184255422001799) | > avg_loss_0: 2.515331483223069 (-0.06370451752568673) | > avg_grad_norm_0: tensor(265.7215, device='cuda:0') (tensor(-23.1742, device='cuda:0')) | > avg_loss_gen: 2.2996766483280027 (+0.015747293620043123) | > avg_loss_kl: 2.0067382547217356 (-0.14066938783081406) | > avg_loss_feat: 4.282817749909952 (+0.038665667386122315) | > avg_loss_mel: 16.625430013092473 (-0.5545676728369457) | > avg_loss_duration: 1.5339648203111031 (-0.01289552534130256) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.748627434314137 (-0.6537196736940203) | > avg_grad_norm_1: tensor(6741.8545, device='cuda:0') (tensor(-414.6646, device='cuda:0'))  > EPOCH: 1397/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:36:08)   --> TIME: 2025-02-25 22:36:17 -- STEP: 15/72 -- GLOBAL_STEP: 1100600 | > loss_disc: 2.478872060775757 (2.457378689448039) | > loss_disc_real_0: 0.09609080851078033 (0.05793668900926908) | > loss_disc_real_1: 0.21431194245815277 (0.21581318378448486) | > loss_disc_real_2: 0.22380375862121582 (0.214987243215243) | > loss_disc_real_3: 0.22524429857730865 (0.2284851372241974) | > loss_disc_real_4: 0.23682288825511932 (0.23934679528077443) | > loss_disc_real_5: 0.2488633543252945 (0.252177823583285) | > loss_0: 2.478872060775757 (2.457378689448039) | > grad_norm_0: tensor(287.8770, device='cuda:0') (tensor(241.5141, device='cuda:0')) | > loss_gen: 2.4448070526123047 (2.340492057800293) | > loss_kl: 1.9394358396530151 (1.8573545773824056) | > loss_feat: 4.508556842803955 (4.546898174285888) | > loss_mel: 17.158531188964844 (16.763981691996257) | > loss_duration: 1.4662766456604004 (1.4692951122919717) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.517606735229492 (26.978021113077798) | > grad_norm_1: tensor(3652.2329, device='cuda:0') (tensor(7859.7969, device='cuda:0')) | > current_lr_0: 0.00016795254202941614 | > current_lr_1: 0.00016795254202941614 | > step_time: 0.4502 (0.42885921796162924) | > loader_time: 0.0073 (0.006696462631225586)  --> TIME: 2025-02-25 22:36:41 -- STEP: 65/72 -- GLOBAL_STEP: 1100650 | > loss_disc: 2.73781156539917 (2.4757852407602154) | > loss_disc_real_0: 0.06367801129817963 (0.07570988598924414) | > loss_disc_real_1: 0.2483856976032257 (0.217125371327767) | > loss_disc_real_2: 0.18061070144176483 (0.2220667036680075) | > loss_disc_real_3: 0.23071736097335815 (0.23068690941883968) | > loss_disc_real_4: 0.27931371331214905 (0.24180954625973336) | > loss_disc_real_5: 0.24250206351280212 (0.24632561734089484) | > loss_0: 2.73781156539917 (2.4757852407602154) | > grad_norm_0: tensor(280.2599, device='cuda:0') (tensor(236.2879, device='cuda:0')) | > loss_gen: 1.9717414379119873 (2.3416098227867708) | > loss_kl: 2.1413562297821045 (1.945824069243211) | > loss_feat: 4.075216770172119 (4.468045803216786) | > loss_mel: 17.013948440551758 (16.722690156789923) | > loss_duration: 1.601891279220581 (1.5175961714524488) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.804153442382812 (26.99576571537898) | > grad_norm_1: tensor(10661.3379, device='cuda:0') (tensor(7451.0430, device='cuda:0')) | > current_lr_0: 0.00016795254202941614 | > current_lr_1: 0.00016795254202941614 | > step_time: 0.5039 (0.45468422082754284) | > loader_time: 0.0118 (0.008400557591364937) --> EVAL PERFORMANCE | > avg_loader_time: 0.008673845882147133 (-5.191480609727239e-06) | > avg_step_time: 0.45884639780286335 (+0.007189230180122519) | > avg_loss_disc: 2.4786843346877827 (-0.03664714853528617) | > avg_loss_disc_real_0: 0.07931613491874343 (-0.023214394174201416) | > avg_loss_disc_real_1: 0.21935031930325735 (+1.3987992850822417e-05) | > avg_loss_disc_real_2: 0.22157100451664186 (-0.0010242476849488813) | > avg_loss_disc_real_3: 0.2302031783570706 (+0.0011205864204487381) | > avg_loss_disc_real_4: 0.24148417862368302 (+0.005699670230838616) | > avg_loss_disc_real_5: 0.24652329632933712 (+0.0007228471443686268) | > avg_loss_0: 2.4786843346877827 (-0.03664714853528617) | > avg_grad_norm_0: tensor(233.5865, device='cuda:0') (tensor(-32.1350, device='cuda:0')) | > avg_loss_gen: 2.3439648487198514 (+0.044288200391848775) | > avg_loss_kl: 1.959175579984423 (-0.04756267473731257) | > avg_loss_feat: 4.4254212245135225 (+0.14260347460357092) | > avg_loss_mel: 16.739928581345247 (+0.11449856825277394) | > avg_loss_duration: 1.523741579391587 (-0.010223240919516208) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.99223144961075 (+0.24360401529661146) | > avg_grad_norm_1: tensor(7360.2471, device='cuda:0') (+tensor(618.3926, device='cuda:0'))  > EPOCH: 1398/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:36:45)   --> TIME: 2025-02-25 22:37:07 -- STEP: 43/72 -- GLOBAL_STEP: 1100700 | > loss_disc: 2.500223159790039 (2.5047736057015357) | > loss_disc_real_0: 0.08316687494516373 (0.101517278441163) | > loss_disc_real_1: 0.23025061190128326 (0.22553741204184155) | > loss_disc_real_2: 0.22859109938144684 (0.22188903807207597) | > loss_disc_real_3: 0.26220282912254333 (0.23148592474848725) | > loss_disc_real_4: 0.21050991117954254 (0.2431822218174158) | > loss_disc_real_5: 0.24898670613765717 (0.2435575910778933) | > loss_0: 2.500223159790039 (2.5047736057015357) | > grad_norm_0: tensor(214.8478, device='cuda:0') (tensor(254.0695, device='cuda:0')) | > loss_gen: 2.2981197834014893 (2.330801858458408) | > loss_kl: 1.9492565393447876 (1.908281123915384) | > loss_feat: 4.885732650756836 (4.433151461357294) | > loss_mel: 16.14900016784668 (16.70823813593665) | > loss_duration: 1.523167371749878 (1.4959134861480359) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.80527687072754 (26.876385844031045) | > grad_norm_1: tensor(3020.7151, device='cuda:0') (tensor(6963.4731, device='cuda:0')) | > current_lr_0: 0.00016793154796166244 | > current_lr_1: 0.00016793154796166244 | > step_time: 0.4472 (0.4356861946194671) | > loader_time: 0.0092 (0.007400401802950127) --> EVAL PERFORMANCE | > avg_loader_time: 0.008537047345873336 (-0.00013679853627379668) | > avg_step_time: 0.4490945507103289 (-0.00975184709253446) | > avg_loss_disc: 2.525752332848563 (+0.047067998160780444) | > avg_loss_disc_real_0: 0.10991969179939218 (+0.030603556880648752) | > avg_loss_disc_real_1: 0.2234139616640521 (+0.004063642360794761) | > avg_loss_disc_real_2: 0.2267110761202557 (+0.005140071603613849) | > avg_loss_disc_real_3: 0.23060514347653993 (+0.00040196511946932745) | > avg_loss_disc_real_4: 0.2366885971015608 (-0.0047955815221222275) | > avg_loss_disc_real_5: 0.2442269973771673 (-0.0022962989521698174) | > avg_loss_0: 2.525752332848563 (+0.047067998160780444) | > avg_grad_norm_0: tensor(243.8138, device='cuda:0') (+tensor(10.2273, device='cuda:0')) | > avg_loss_gen: 2.3177204719731503 (-0.026244376746701104) | > avg_loss_kl: 1.9675605582519315 (+0.008384978267508503) | > avg_loss_feat: 4.328733719570536 (-0.09668750494298628) | > avg_loss_mel: 16.635578934575477 (-0.10434964676976932) | > avg_loss_duration: 1.5227630222347421 (-0.000978557156844806) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.772356490014303 (-0.21987495959644576) | > avg_grad_norm_1: tensor(6659.8550, device='cuda:0') (tensor(-700.3921, device='cuda:0'))  > EPOCH: 1399/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:37:22)   --> TIME: 2025-02-25 22:37:33 -- STEP: 21/72 -- GLOBAL_STEP: 1100750 | > loss_disc: 2.699763774871826 (2.474001895813714) | > loss_disc_real_0: 0.0475383996963501 (0.06612905469678697) | > loss_disc_real_1: 0.20290884375572205 (0.21665343713192714) | > loss_disc_real_2: 0.21872112154960632 (0.22612910469373068) | > loss_disc_real_3: 0.2220763862133026 (0.23346164609704698) | > loss_disc_real_4: 0.21457117795944214 (0.2371658519619987) | > loss_disc_real_5: 0.23874042928218842 (0.23869073816708156) | > loss_0: 2.699763774871826 (2.474001895813714) | > grad_norm_0: tensor(137.3844, device='cuda:0') (tensor(184.3833, device='cuda:0')) | > loss_gen: 1.925499677658081 (2.313838867914109) | > loss_kl: 1.783164381980896 (1.9030945244289579) | > loss_feat: 3.7767574787139893 (4.484566790716989) | > loss_mel: 15.781590461730957 (16.901698884509855) | > loss_duration: 1.5074563026428223 (1.4746194112868536) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.77446746826172 (27.07781828017462) | > grad_norm_1: tensor(11563.9893, device='cuda:0') (tensor(7220.8364, device='cuda:0')) | > current_lr_0: 0.00016791055651816723 | > current_lr_1: 0.00016791055651816723 | > step_time: 0.427 (0.43283834911528085) | > loader_time: 0.0084 (0.006661471866426014)  --> TIME: 2025-02-25 22:37:57 -- STEP: 71/72 -- GLOBAL_STEP: 1100800 | > loss_disc: 2.5157246589660645 (2.491047580477217) | > loss_disc_real_0: 0.1216576099395752 (0.07611293702478143) | > loss_disc_real_1: 0.2637595534324646 (0.21863075671061666) | > loss_disc_real_2: 0.223059743642807 (0.23366882368712358) | > loss_disc_real_3: 0.2207658588886261 (0.23247244975096742) | > loss_disc_real_4: 0.2540672719478607 (0.23676483060272646) | > loss_disc_real_5: 0.2065865397453308 (0.24330000768245105) | > loss_0: 2.5157246589660645 (2.491047580477217) | > grad_norm_0: tensor(157.6602, device='cuda:0') (tensor(209.6109, device='cuda:0')) | > loss_gen: 2.5753278732299805 (2.326242273961995) | > loss_kl: 2.0455634593963623 (1.992250706108523) | > loss_feat: 4.572493553161621 (4.403691268303025) | > loss_mel: 16.14043617248535 (16.617574060466932) | > loss_duration: 1.5351135730743408 (1.523496169439504) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.868934631347656 (26.86325433220662) | > grad_norm_1: tensor(1556.0569, device='cuda:0') (tensor(7233.1631, device='cuda:0')) | > current_lr_0: 0.00016791055651816723 | > current_lr_1: 0.00016791055651816723 | > step_time: 0.4897 (0.45166910869974486) | > loader_time: 0.008 (0.008554985825444611) --> EVAL PERFORMANCE | > avg_loader_time: 0.008554985825444611 (+1.793847957127538e-05) | > avg_step_time: 0.45166910869974486 (+0.002574557989415971) | > avg_loss_disc: 2.491047580477217 (-0.03470475237134618) | > avg_loss_disc_real_0: 0.07611293702478143 (-0.033806754774610745) | > avg_loss_disc_real_1: 0.21863075671061666 (-0.004783204953435455) | > avg_loss_disc_real_2: 0.23366882368712358 (+0.006957747566867872) | > avg_loss_disc_real_3: 0.23247244975096742 (+0.0018673062744274938) | > avg_loss_disc_real_4: 0.23676483060272646 (+7.62335011656734e-05) | > avg_loss_disc_real_5: 0.24330000768245105 (-0.0009269896947162459) | > avg_loss_0: 2.491047580477217 (-0.03470475237134618) | > avg_grad_norm_0: tensor(209.6109, device='cuda:0') (tensor(-34.2030, device='cuda:0')) | > avg_loss_gen: 2.326242273961995 (+0.008521801988844668) | > avg_loss_kl: 1.992250706108523 (+0.024690147856591516) | > avg_loss_feat: 4.403691268303025 (+0.07495754873248917) | > avg_loss_mel: 16.617574060466932 (-0.01800487410854501) | > avg_loss_duration: 1.523496169439504 (+0.0007331472047618082) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.86325433220662 (+0.09089784219231589) | > avg_grad_norm_1: tensor(7233.1631, device='cuda:0') (+tensor(573.3081, device='cuda:0'))  > EPOCH: 1400/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:37:59)   --> TIME: 2025-02-25 22:38:23 -- STEP: 49/72 -- GLOBAL_STEP: 1100850 | > loss_disc: 2.6222987174987793 (2.5880127780291504) | > loss_disc_real_0: 0.11086045205593109 (0.1594547537364522) | > loss_disc_real_1: 0.2405073642730713 (0.22250050427962323) | > loss_disc_real_2: 0.09519167989492416 (0.2470680875133495) | > loss_disc_real_3: 0.19703516364097595 (0.2333304407645245) | > loss_disc_real_4: 0.21275879442691803 (0.23349858242638258) | > loss_disc_real_5: 0.23314103484153748 (0.24106981498854502) | > loss_0: 2.6222987174987793 (2.5880127780291504) | > grad_norm_0: tensor(277.1867, device='cuda:0') (tensor(284.1964, device='cuda:0')) | > loss_gen: 1.9963161945343018 (2.3584060279690497) | > loss_kl: 2.099083662033081 (2.005263238537068) | > loss_feat: 4.38942289352417 (4.317308571873878) | > loss_mel: 17.454824447631836 (16.837443079267224) | > loss_duration: 1.5727581977844238 (1.5063165450582696) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.512405395507812 (27.024737416481486) | > grad_norm_1: tensor(9299.8809, device='cuda:0') (tensor(5773.9004, device='cuda:0')) | > current_lr_0: 0.00016788956769860246 | > current_lr_1: 0.00016788956769860246 | > step_time: 0.442 (0.43910940325990017) | > loader_time: 0.0094 (0.007614038428481744) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611427226536711 (+5.6441401092099594e-05) | > avg_step_time: 0.4519020033554292 (+0.00023289465568432366) | > avg_loss_disc: 2.5876896750759073 (+0.09664209459869033) | > avg_loss_disc_real_0: 0.1568134349941368 (+0.08070049796935537) | > avg_loss_disc_real_1: 0.22819568426676198 (+0.009564927556145325) | > avg_loss_disc_real_2: 0.2436733816710996 (+0.010004557983976009) | > avg_loss_disc_real_3: 0.233809287069549 (+0.0013368373185815852) | > avg_loss_disc_real_4: 0.233163896580817 (-0.0036009340219094743) | > avg_loss_disc_real_5: 0.24033416136049887 (-0.0029658463219521813) | > avg_loss_0: 2.5876896750759073 (+0.09664209459869033) | > avg_grad_norm_0: tensor(280.5499, device='cuda:0') (+tensor(70.9390, device='cuda:0')) | > avg_loss_gen: 2.350430538956549 (+0.024188264994553954) | > avg_loss_kl: 2.0524331834954266 (+0.06018247738690352) | > avg_loss_feat: 4.195930067922028 (-0.20776120038099766) | > avg_loss_mel: 16.794380913318044 (+0.17680685285111153) | > avg_loss_duration: 1.526874691667691 (+0.0033785222281870197) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.920049344989614 (+0.05679501278299526) | > avg_grad_norm_1: tensor(5849.5283, device='cuda:0') (tensor(-1383.6348, device='cuda:0'))  > EPOCH: 1401/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:38:36)   --> TIME: 2025-02-25 22:38:50 -- STEP: 27/72 -- GLOBAL_STEP: 1100900 | > loss_disc: 2.689044713973999 (2.698731272308914) | > loss_disc_real_0: 0.11598573625087738 (0.15412256756314519) | > loss_disc_real_1: 0.1969922035932541 (0.23883601264269264) | > loss_disc_real_2: 0.2095230370759964 (0.23462889205526422) | > loss_disc_real_3: 0.23048986494541168 (0.2333929196551994) | > loss_disc_real_4: 0.213958278298378 (0.252496694525083) | > loss_disc_real_5: 0.23227104544639587 (0.2512233836783303) | > loss_0: 2.689044713973999 (2.698731272308914) | > grad_norm_0: tensor(368.3498, device='cuda:0') (tensor(252.6919, device='cuda:0')) | > loss_gen: 1.75376558303833 (2.054690029886034) | > loss_kl: 2.1679306030273438 (2.1098212930891247) | > loss_feat: 3.5426974296569824 (3.687602087303444) | > loss_mel: 16.653736114501953 (17.199280067726416) | > loss_duration: 1.5085599422454834 (1.498462681417112) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.626689910888672 (26.549855691415292) | > grad_norm_1: tensor(8161.4014, device='cuda:0') (tensor(9130.4619, device='cuda:0')) | > current_lr_0: 0.00016786858150264012 | > current_lr_1: 0.00016786858150264012 | > step_time: 0.432 (0.43598401104962387) | > loader_time: 0.0084 (0.007000287373860677) --> EVAL PERFORMANCE | > avg_loader_time: 0.008593539117087783 (-1.7888109448927977e-05) | > avg_step_time: 0.45195201417090186 (+5.0010815472678605e-05) | > avg_loss_disc: 2.673905637902273 (+0.0862159628263659) | > avg_loss_disc_real_0: 0.1400866652773299 (-0.016726769716806894) | > avg_loss_disc_real_1: 0.2313010900897879 (+0.003105405823025903) | > avg_loss_disc_real_2: 0.2434136875918214 (-0.0002596940792781999) | > avg_loss_disc_real_3: 0.23319891724787967 (-0.0006103698216693432) | > avg_loss_disc_real_4: 0.24134732384077265 (+0.008183427259955661) | > avg_loss_disc_real_5: 0.24820789442935456 (+0.007873733068855693) | > avg_loss_0: 2.673905637902273 (+0.0862159628263659) | > avg_grad_norm_0: tensor(314.3991, device='cuda:0') (+tensor(33.8492, device='cuda:0')) | > avg_loss_gen: 2.016088346360435 (-0.3343421925961141) | > avg_loss_kl: 2.1040884760064125 (+0.051655292510985884) | > avg_loss_feat: 3.60224874926285 (-0.5936813186591778) | > avg_loss_mel: 16.8593901513328 (+0.06500923801475622) | > avg_loss_duration: 1.5338088559432768 (+0.006934164275585797) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.115624306907115 (-0.8044250380824991) | > avg_grad_norm_1: tensor(10328.8389, device='cuda:0') (+tensor(4479.3105, device='cuda:0'))  > EPOCH: 1402/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:39:12)   --> TIME: 2025-02-25 22:39:17 -- STEP: 5/72 -- GLOBAL_STEP: 1100950 | > loss_disc: 2.5232090950012207 (2.586991882324219) | > loss_disc_real_0: 0.11237262189388275 (0.13494397401809693) | > loss_disc_real_1: 0.21885453164577484 (0.23181842863559723) | > loss_disc_real_2: 0.20244458317756653 (0.22974208891391754) | > loss_disc_real_3: 0.24491974711418152 (0.23850911855697632) | > loss_disc_real_4: 0.2578750252723694 (0.24178216755390167) | > loss_disc_real_5: 0.24421705305576324 (0.23759728372097016) | > loss_0: 2.5232090950012207 (2.586991882324219) | > grad_norm_0: tensor(94.7007, device='cuda:0') (tensor(450.5883, device='cuda:0')) | > loss_gen: 2.136331796646118 (2.1334289073944093) | > loss_kl: 1.8488532304763794 (1.849884033203125) | > loss_feat: 4.019689559936523 (3.935091829299927) | > loss_mel: 16.301279067993164 (16.658359146118165) | > loss_duration: 1.4815298318862915 (1.4852949619293212) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.787683486938477 (26.06205825805664) | > grad_norm_1: tensor(15887.9805, device='cuda:0') (tensor(16031.3848, device='cuda:0')) | > current_lr_0: 0.0001678475979299523 | > current_lr_1: 0.0001678475979299523 | > step_time: 0.434 (0.4369230270385742) | > loader_time: 0.0052 (0.006624650955200195)  --> TIME: 2025-02-25 22:39:40 -- STEP: 55/72 -- GLOBAL_STEP: 1101000 | > loss_disc: 2.562680959701538 (2.6003261002627287) | > loss_disc_real_0: 0.11036572605371475 (0.12554478008638728) | > loss_disc_real_1: 0.2388918399810791 (0.22464337105100804) | > loss_disc_real_2: 0.22593218088150024 (0.2382143651897257) | > loss_disc_real_3: 0.22598908841609955 (0.23608984784646467) | > loss_disc_real_4: 0.23719336092472076 (0.24061510589989749) | > loss_disc_real_5: 0.24380894005298615 (0.24136145196177744) | > loss_0: 2.562680959701538 (2.6003261002627287) | > grad_norm_0: tensor(355.1552, device='cuda:0') (tensor(382.7851, device='cuda:0')) | > loss_gen: 1.970937728881836 (2.0803765015168616) | > loss_kl: 2.0138394832611084 (1.9980417880144987) | > loss_feat: 3.4723474979400635 (3.8102590690959586) | > loss_mel: 16.475547790527344 (16.365453303943983) | > loss_duration: 1.5727999210357666 (1.5072644732215188) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.50547218322754 (25.761394812844017) | > grad_norm_1: tensor(8894.5586, device='cuda:0') (tensor(12459.9414, device='cuda:0')) | > current_lr_0: 0.0001678475979299523 | > current_lr_1: 0.0001678475979299523 | > step_time: 0.4609 (0.4438131115653298) | > loader_time: 0.0093 (0.007891429554332386) --> EVAL PERFORMANCE | > avg_loader_time: 0.008621810187756176 (+2.8271070668392864e-05) | > avg_step_time: 0.45498541375281104 (+0.0030333995819091797) | > avg_loss_disc: 2.6009255832349747 (-0.07298005466729851) | > avg_loss_disc_real_0: 0.12605409471082016 (-0.01403257056650975) | > avg_loss_disc_real_1: 0.22355924294867985 (-0.007741847141108038) | > avg_loss_disc_real_2: 0.23784530267749035 (-0.005568384914331037) | > avg_loss_disc_real_3: 0.23593387721290052 (+0.00273495996502085) | > avg_loss_disc_real_4: 0.2423886621082333 (+0.0010413382674606397) | > avg_loss_disc_real_5: 0.2430452200728403 (-0.0051626743565142685) | > avg_loss_0: 2.6009255832349747 (-0.07298005466729851) | > avg_grad_norm_0: tensor(379.7222, device='cuda:0') (+tensor(65.3231, device='cuda:0')) | > avg_loss_gen: 2.08240524647941 (+0.06631690011897495) | > avg_loss_kl: 2.018538307136212 (-0.0855501688702005) | > avg_loss_feat: 3.7848045255096867 (+0.18255577624683683) | > avg_loss_mel: 16.4037132397504 (-0.4556769115824011) | > avg_loss_duration: 1.5215977195283057 (-0.012211136414971069) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.811058823491486 (-0.3045654834156295) | > avg_grad_norm_1: tensor(12450.8965, device='cuda:0') (+tensor(2122.0576, device='cuda:0'))  > EPOCH: 1403/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:39:49)   --> TIME: 2025-02-25 22:40:06 -- STEP: 33/72 -- GLOBAL_STEP: 1101050 | > loss_disc: 2.5334925651550293 (2.5601593653361) | > loss_disc_real_0: 0.17540854215621948 (0.10906732895157555) | > loss_disc_real_1: 0.18720237910747528 (0.21366069488453143) | > loss_disc_real_2: 0.21531742811203003 (0.2248534939505837) | > loss_disc_real_3: 0.2223745882511139 (0.22786009898691467) | > loss_disc_real_4: 0.2450444996356964 (0.2380161854353818) | > loss_disc_real_5: 0.2488229125738144 (0.24090123718435114) | > loss_0: 2.5334925651550293 (2.5601593653361) | > grad_norm_0: tensor(917.1980, device='cuda:0') (tensor(351.8916, device='cuda:0')) | > loss_gen: 2.1159706115722656 (2.111751563621289) | > loss_kl: 1.9596086740493774 (1.9165624235615586) | > loss_feat: 4.163961887359619 (4.023023301904851) | > loss_mel: 16.27269744873047 (16.37259601824212) | > loss_duration: 1.4979140758514404 (1.4779907103740808) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.010150909423828 (25.901924248897668) | > grad_norm_1: tensor(15646.2207, device='cuda:0') (tensor(12982.6201, device='cuda:0')) | > current_lr_0: 0.00016782661698021105 | > current_lr_1: 0.00016782661698021105 | > step_time: 0.433 (0.4342505498365922) | > loader_time: 0.0076 (0.007048715244640003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008533806868002449 (-8.800331975372666e-05) | > avg_step_time: 0.4512306267107037 (-0.003754787042107355) | > avg_loss_disc: 2.571500187188807 (-0.029425396046167673) | > avg_loss_disc_real_0: 0.11454739857097747 (-0.01150669613984269) | > avg_loss_disc_real_1: 0.21804822225805742 (-0.005511020690622426) | > avg_loss_disc_real_2: 0.22889201175159132 (-0.008953290925899027) | > avg_loss_disc_real_3: 0.23003187712649226 (-0.00590200008640826) | > avg_loss_disc_real_4: 0.2386727253316154 (-0.003715936776617884) | > avg_loss_disc_real_5: 0.24460661893999072 (+0.0015613988671504264) | > avg_loss_0: 2.571500187188807 (-0.029425396046167673) | > avg_grad_norm_0: tensor(387.6954, device='cuda:0') (+tensor(7.9732, device='cuda:0')) | > avg_loss_gen: 2.12459370955615 (+0.04218846307673996) | > avg_loss_kl: 1.9807611989303373 (-0.037777108205874654) | > avg_loss_feat: 3.947869512396799 (+0.16306498688711235) | > avg_loss_mel: 16.424480881489508 (+0.020767641739109166) | > avg_loss_duration: 1.5180593158157778 (-0.0035384037125278933) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.995764638336613 (+0.1847058148451275) | > avg_grad_norm_1: tensor(13017.5811, device='cuda:0') (+tensor(566.6846, device='cuda:0'))  > EPOCH: 1404/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:40:26)   --> TIME: 2025-02-25 22:40:33 -- STEP: 11/72 -- GLOBAL_STEP: 1101100 | > loss_disc: 2.6982622146606445 (2.5674022327769888) | > loss_disc_real_0: 0.11661280691623688 (0.11326207491484555) | > loss_disc_real_1: 0.22756946086883545 (0.2172601575201208) | > loss_disc_real_2: 0.2285502403974533 (0.2323340814221989) | > loss_disc_real_3: 0.2491443008184433 (0.2383834719657898) | > loss_disc_real_4: 0.24659046530723572 (0.24746783755042337) | > loss_disc_real_5: 0.24289973080158234 (0.24005742912942712) | > loss_0: 2.6982622146606445 (2.5674022327769888) | > grad_norm_0: tensor(528.2756, device='cuda:0') (tensor(380.6898, device='cuda:0')) | > loss_gen: 1.922067403793335 (2.1684256466952236) | > loss_kl: 1.9328471422195435 (1.8541812679984353) | > loss_feat: 3.865264654159546 (4.088726087050005) | > loss_mel: 16.512052536010742 (16.54654745622115) | > loss_duration: 1.445014476776123 (1.4683158939534968) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.677248001098633 (26.126196947964754) | > grad_norm_1: tensor(11187.3994, device='cuda:0') (tensor(13181.3398, device='cuda:0')) | > current_lr_0: 0.0001678056386530885 | > current_lr_1: 0.0001678056386530885 | > step_time: 0.4377 (0.43489584055813874) | > loader_time: 0.0071 (0.0064719807017933235)  --> TIME: 2025-02-25 22:40:56 -- STEP: 61/72 -- GLOBAL_STEP: 1101150 | > loss_disc: 2.4904184341430664 (2.584775162524864) | > loss_disc_real_0: 0.06698731333017349 (0.11786005902485769) | > loss_disc_real_1: 0.22651594877243042 (0.21698279947531027) | > loss_disc_real_2: 0.2699066698551178 (0.23394168571370547) | > loss_disc_real_3: 0.22511720657348633 (0.23148508663060235) | > loss_disc_real_4: 0.23933382332324982 (0.24324802907763934) | > loss_disc_real_5: 0.2699417471885681 (0.2464999957162826) | > loss_0: 2.4904184341430664 (2.584775162524864) | > grad_norm_0: tensor(179.2248, device='cuda:0') (tensor(373.7096, device='cuda:0')) | > loss_gen: 2.2116079330444336 (2.1678149094347092) | > loss_kl: 2.0787298679351807 (1.9826311048914174) | > loss_feat: 4.02100944519043 (4.010287085517507) | > loss_mel: 16.250930786132812 (16.394882123978412) | > loss_duration: 1.560119390487671 (1.5076346983675097) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.122398376464844 (26.063249869424787) | > grad_norm_1: tensor(13059.9102, device='cuda:0') (tensor(12685.3594, device='cuda:0')) | > current_lr_0: 0.0001678056386530885 | > current_lr_1: 0.0001678056386530885 | > step_time: 0.4572 (0.4456067984221411) | > loader_time: 0.0108 (0.00815715555284844) --> EVAL PERFORMANCE | > avg_loader_time: 0.008627841170404998 (+9.403430240254897e-05) | > avg_step_time: 0.4546509695724702 (+0.003420342861766512) | > avg_loss_disc: 2.5842736371805963 (+0.01277344999178931) | > avg_loss_disc_real_0: 0.1191575899720192 (+0.004610191401041727) | > avg_loss_disc_real_1: 0.2168905579707992 (-0.00115766428725822) | > avg_loss_disc_real_2: 0.23139567820119186 (+0.002503666449600539) | > avg_loss_disc_real_3: 0.23183684093011936 (+0.001804963803627102) | > avg_loss_disc_real_4: 0.24311953866985483 (+0.00444681333823943) | > avg_loss_disc_real_5: 0.24602572724852764 (+0.0014191083085369205) | > avg_loss_0: 2.5842736371805963 (+0.01277344999178931) | > avg_grad_norm_0: tensor(380.7124, device='cuda:0') (tensor(-6.9829, device='cuda:0')) | > avg_loss_gen: 2.167135663435493 (+0.04254195387934301) | > avg_loss_kl: 1.995688450168556 (+0.014927251238218586) | > avg_loss_feat: 3.9800733714036522 (+0.032203859006853186) | > avg_loss_mel: 16.43250748808955 (+0.008026606600040509) | > avg_loss_duration: 1.5187807150290047 (+0.000721399213226892) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.094185654546173 (+0.09842101620956001) | > avg_grad_norm_1: tensor(12628.2041, device='cuda:0') (tensor(-389.3770, device='cuda:0'))  > EPOCH: 1405/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:41:03)   --> TIME: 2025-02-25 22:41:23 -- STEP: 39/72 -- GLOBAL_STEP: 1101200 | > loss_disc: 2.5506722927093506 (2.574982514748207) | > loss_disc_real_0: 0.1354023516178131 (0.12017006255113162) | > loss_disc_real_1: 0.1852828860282898 (0.21582921689901596) | > loss_disc_real_2: 0.25314515829086304 (0.22411779715464666) | > loss_disc_real_3: 0.21841561794281006 (0.22921958986001137) | > loss_disc_real_4: 0.24395912885665894 (0.23864767604913467) | > loss_disc_real_5: 0.236125186085701 (0.2429904697033075) | > loss_0: 2.5506722927093506 (2.574982514748207) | > grad_norm_0: tensor(486.8429, device='cuda:0') (tensor(366.6599, device='cuda:0')) | > loss_gen: 2.121264696121216 (2.161546795796125) | > loss_kl: 2.143806219100952 (1.95904886111235) | > loss_feat: 3.950906276702881 (4.04688036136138) | > loss_mel: 17.26852798461914 (16.665975595131894) | > loss_duration: 1.524388074874878 (1.4881389874678392) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.008892059326172 (26.321590717022236) | > grad_norm_1: tensor(10847.3633, device='cuda:0') (tensor(12101.1328, device='cuda:0')) | > current_lr_0: 0.00016778466294825687 | > current_lr_1: 0.00016778466294825687 | > step_time: 0.4522 (0.4415205625387338) | > loader_time: 0.0105 (0.007485817640255659) --> EVAL PERFORMANCE | > avg_loader_time: 0.008642394777754662 (+1.45536073496641e-05) | > avg_step_time: 0.4553161943462533 (+0.000665224773783113) | > avg_loss_disc: 2.584120535514725 (-0.00015310166587134688) | > avg_loss_disc_real_0: 0.11623540667580887 (-0.002922183296210329) | > avg_loss_disc_real_1: 0.21774796868713808 (+0.00085741071633888) | > avg_loss_disc_real_2: 0.22786028410347414 (-0.00353539409771772) | > avg_loss_disc_real_3: 0.2315144849495149 (-0.00032235598060445514) | > avg_loss_disc_real_4: 0.24044003641941178 (-0.0026795022504430555) | > avg_loss_disc_real_5: 0.24226185532522873 (-0.0037638719232989115) | > avg_loss_0: 2.584120535514725 (-0.00015310166587134688) | > avg_grad_norm_0: tensor(375.9458, device='cuda:0') (tensor(-4.7666, device='cuda:0')) | > avg_loss_gen: 2.1413088482870184 (-0.025826815148474402) | > avg_loss_kl: 2.006402247388598 (+0.010713797220042176) | > avg_loss_feat: 3.99460923503822 (+0.014535863634567825) | > avg_loss_mel: 16.60552854940925 (+0.17302106131970163) | > avg_loss_duration: 1.5204665828758561 (+0.0016858678468514299) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.26831546299894 (+0.1741298084527685) | > avg_grad_norm_1: tensor(12029.3232, device='cuda:0') (tensor(-598.8809, device='cuda:0'))  > EPOCH: 1406/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:41:40)   --> TIME: 2025-02-25 22:41:50 -- STEP: 17/72 -- GLOBAL_STEP: 1101250 | > loss_disc: 2.6317169666290283 (2.56713324434617) | > loss_disc_real_0: 0.10498549044132233 (0.1182911102386082) | > loss_disc_real_1: 0.2233114093542099 (0.2167908531777999) | > loss_disc_real_2: 0.2649375796318054 (0.22878983704482808) | > loss_disc_real_3: 0.2520521581172943 (0.23488622465554407) | > loss_disc_real_4: 0.27230170369148254 (0.23874171341166778) | > loss_disc_real_5: 0.26097211241722107 (0.2481934016241747) | > loss_0: 2.6317169666290283 (2.56713324434617) | > grad_norm_0: tensor(486.5775, device='cuda:0') (tensor(388.5795, device='cuda:0')) | > loss_gen: 2.0336356163024902 (2.164369926733129) | > loss_kl: 2.104295253753662 (1.9474459816427792) | > loss_feat: 3.71577787399292 (4.051273977055269) | > loss_mel: 17.064884185791016 (16.683027099160586) | > loss_duration: 1.5003451108932495 (1.4774131704779232) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.41893768310547 (26.323530309340534) | > grad_norm_1: tensor(12068.7275, device='cuda:0') (tensor(11153.9092, device='cuda:0')) | > current_lr_0: 0.00016776368986538834 | > current_lr_1: 0.00016776368986538834 | > step_time: 0.4415 (0.43464138928581686) | > loader_time: 0.007 (0.007190746419570025)  --> TIME: 2025-02-25 22:42:13 -- STEP: 67/72 -- GLOBAL_STEP: 1101300 | > loss_disc: 2.7214818000793457 (2.5902959624333164) | > loss_disc_real_0: 0.12077973783016205 (0.12841314277542173) | > loss_disc_real_1: 0.21072423458099365 (0.2191369302237212) | > loss_disc_real_2: 0.22413045167922974 (0.2304378345386306) | > loss_disc_real_3: 0.27025166153907776 (0.23266396371286308) | > loss_disc_real_4: 0.24216622114181519 (0.24113233298508088) | > loss_disc_real_5: 0.23560838401317596 (0.24730667316201907) | > loss_0: 2.7214818000793457 (2.5902959624333164) | > grad_norm_0: tensor(662.3892, device='cuda:0') (tensor(381.1749, device='cuda:0')) | > loss_gen: 1.829186201095581 (2.159662445979331) | > loss_kl: 2.084585428237915 (2.03199675723688) | > loss_feat: 3.8013601303100586 (3.9682866174783262) | > loss_mel: 16.77918243408203 (16.640767154408923) | > loss_duration: 1.6126503944396973 (1.5212469225499168) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.106962203979492 (26.32195993679673) | > grad_norm_1: tensor(14213.7305, device='cuda:0') (tensor(10591.0508, device='cuda:0')) | > current_lr_0: 0.00016776368986538834 | > current_lr_1: 0.00016776368986538834 | > step_time: 0.4876 (0.44643918791813636) | > loader_time: 0.0123 (0.0085933777823377) --> EVAL PERFORMANCE | > avg_loader_time: 0.008768642452401174 (+0.00012624767464651174) | > avg_step_time: 0.44979995740971096 (-0.005516236936542351) | > avg_loss_disc: 2.589373249403187 (+0.005252713888462157) | > avg_loss_disc_real_0: 0.12736716887480778 (+0.011131762198998918) | > avg_loss_disc_real_1: 0.21919520891888042 (+0.001447240231742336) | > avg_loss_disc_real_2: 0.23014123003247758 (+0.002280945929003436) | > avg_loss_disc_real_3: 0.23216936831742946 (+0.0006548833679145549) | > avg_loss_disc_real_4: 0.24025512203364305 (-0.00018491438576873076) | > avg_loss_disc_real_5: 0.24614282833858275 (+0.003880973013354022) | > avg_loss_0: 2.589373249403187 (+0.005252713888462157) | > avg_grad_norm_0: tensor(376.3698, device='cuda:0') (+tensor(0.4240, device='cuda:0')) | > avg_loss_gen: 2.1599473583866167 (+0.01863851009959827) | > avg_loss_kl: 2.0355098566538854 (+0.029107609265287326) | > avg_loss_feat: 3.968505869449023 (-0.026103365589197125) | > avg_loss_mel: 16.63931200537883 (+0.03378345596957999) | > avg_loss_duration: 1.5244243329679463 (+0.0039577500920902065) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.327699446342365 (+0.0593839833434231) | > avg_grad_norm_1: tensor(10508.9434, device='cuda:0') (tensor(-1520.3799, device='cuda:0'))  > EPOCH: 1407/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:42:17)   --> TIME: 2025-02-25 22:42:40 -- STEP: 45/72 -- GLOBAL_STEP: 1101350 | > loss_disc: 2.9077320098876953 (2.7073150740729446) | > loss_disc_real_0: 0.12036088854074478 (0.18875841009947988) | > loss_disc_real_1: 0.2446647435426712 (0.22613375816080306) | > loss_disc_real_2: 0.29423201084136963 (0.2356073564953274) | > loss_disc_real_3: 0.2642926275730133 (0.25161382059256227) | > loss_disc_real_4: 0.22981493175029755 (0.25473324159781136) | > loss_disc_real_5: 0.2519063651561737 (0.24414171477158864) | > loss_0: 2.9077320098876953 (2.7073150740729446) | > grad_norm_0: tensor(182.4072, device='cuda:0') (tensor(284.3350, device='cuda:0')) | > loss_gen: 1.9166203737258911 (2.221231047312419) | > loss_kl: 3.1744229793548584 (2.185696013768514) | > loss_feat: 3.6448569297790527 (4.035461870829265) | > loss_mel: 22.286943435668945 (17.407023959689674) | > loss_duration: 1.9528499841690063 (1.5337960349188908) | > amp_scaler: 8.0 (8.0) | > loss_1: 32.97569274902344 (27.383208804660374) | > grad_norm_1: tensor(1242.0292, device='cuda:0') (tensor(6779.4619, device='cuda:0')) | > current_lr_0: 0.00016774271940415517 | > current_lr_1: 0.00016774271940415517 | > step_time: 0.4467 (0.44059830771552194) | > loader_time: 0.0085 (0.007326428095499674) --> EVAL PERFORMANCE | > avg_loader_time: 0.008458248326476191 (-0.00031039412592498254) | > avg_step_time: 0.4553733745091398 (+0.0055734170994288235) | > avg_loss_disc: 2.67993967298051 (+0.09056642357732292) | > avg_loss_disc_real_0: 0.16220329082767732 (+0.03483612195286953) | > avg_loss_disc_real_1: 0.2241429071611082 (+0.004947698242227794) | > avg_loss_disc_real_2: 0.23642716168517797 (+0.006285931652700388) | > avg_loss_disc_real_3: 0.24661238218697015 (+0.014443013869540694) | > avg_loss_disc_real_4: 0.25844956115937573 (+0.01819443912573268) | > avg_loss_disc_real_5: 0.2495513839620939 (+0.003408555623511139) | > avg_loss_0: 2.67993967298051 (+0.09056642357732292) | > avg_grad_norm_0: tensor(279.9259, device='cuda:0') (tensor(-96.4439, device='cuda:0')) | > avg_loss_gen: 2.2061673570686664 (+0.046219998682049734) | > avg_loss_kl: 2.4078436603008853 (+0.37233380364699986) | > avg_loss_feat: 4.003108541730424 (+0.03460267228140124) | > avg_loss_mel: 18.067411288409172 (+1.4280992830303418) | > avg_loss_duration: 1.6314636838268226 (+0.10703935085887628) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.315994370151575 (+1.9882949238092102) | > avg_grad_norm_1: tensor(6891.3760, device='cuda:0') (tensor(-3617.5674, device='cuda:0'))  > EPOCH: 1408/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:42:54)   --> TIME: 2025-02-25 22:43:07 -- STEP: 23/72 -- GLOBAL_STEP: 1101400 | > loss_disc: 2.6036484241485596 (2.4621681026790454) | > loss_disc_real_0: 0.07576269656419754 (0.07888945929058222) | > loss_disc_real_1: 0.20232082903385162 (0.2042053063278613) | > loss_disc_real_2: 0.19897626340389252 (0.22339605896369272) | > loss_disc_real_3: 0.2064356952905655 (0.22838885304720505) | > loss_disc_real_4: 0.2160850614309311 (0.23735499900320303) | > loss_disc_real_5: 0.25701650977134705 (0.25027786584004114) | > loss_0: 2.6036484241485596 (2.4621681026790454) | > grad_norm_0: tensor(214.3052, device='cuda:0') (tensor(283.5379, device='cuda:0')) | > loss_gen: 2.1061787605285645 (2.3035145065058833) | > loss_kl: 1.904497742652893 (2.1052758693695073) | > loss_feat: 3.6561434268951416 (4.314547932666281) | > loss_mel: 16.146461486816406 (17.330799683280617) | > loss_duration: 1.5365968942642212 (1.545535745828048) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.349878311157227 (27.59967372728431) | > grad_norm_1: tensor(11782.7383, device='cuda:0') (tensor(8466.0420, device='cuda:0')) | > current_lr_0: 0.00016772175156422965 | > current_lr_1: 0.00016772175156422965 | > step_time: 0.4289 (0.4359842590663744) | > loader_time: 0.0066 (0.006681214208188264) --> EVAL PERFORMANCE | > avg_loader_time: 0.008549045509016012 (+9.079718253982016e-05) | > avg_step_time: 0.4544994360964063 (-0.0008739384127334771) | > avg_loss_disc: 2.4976809461351848 (-0.1822587268453253) | > avg_loss_disc_real_0: 0.09936913168451315 (-0.06283415914316416) | > avg_loss_disc_real_1: 0.2151726805408236 (-0.0089702266202846) | > avg_loss_disc_real_2: 0.2261317943183469 (-0.010295367366831065) | > avg_loss_disc_real_3: 0.22964907762869982 (-0.01696330455827033) | > avg_loss_disc_real_4: 0.23784557740453263 (-0.020603983754843097) | > avg_loss_disc_real_5: 0.2448827473210617 (-0.004668636641032203) | > avg_loss_0: 2.4976809461351848 (-0.1822587268453253) | > avg_grad_norm_0: tensor(290.3533, device='cuda:0') (+tensor(10.4275, device='cuda:0')) | > avg_loss_gen: 2.29126632381493 (+0.08509896674626338) | > avg_loss_kl: 2.1088252420156772 (-0.29901841828520803) | > avg_loss_feat: 4.166655506886227 (+0.16354696515580258) | > avg_loss_mel: 16.93848323822023 (-1.128928050188943) | > avg_loss_duration: 1.5636028051376343 (-0.06786087868918833) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.0688331227907 (-1.247161247360875) | > avg_grad_norm_1: tensor(8359.8232, device='cuda:0') (+tensor(1468.4473, device='cuda:0'))  > EPOCH: 1409/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:43:31)   --> TIME: 2025-02-25 22:43:34 -- STEP: 1/72 -- GLOBAL_STEP: 1101450 | > loss_disc: 2.539290189743042 (2.539290189743042) | > loss_disc_real_0: 0.2854039967060089 (0.2854039967060089) | > loss_disc_real_1: 0.22956347465515137 (0.22956347465515137) | > loss_disc_real_2: 0.20432662963867188 (0.20432662963867188) | > loss_disc_real_3: 0.285775363445282 (0.285775363445282) | > loss_disc_real_4: 0.258504182100296 (0.258504182100296) | > loss_disc_real_5: 0.24547885358333588 (0.24547885358333588) | > loss_0: 2.539290189743042 (2.539290189743042) | > grad_norm_0: tensor(745.0856, device='cuda:0') (tensor(745.0856, device='cuda:0')) | > loss_gen: 2.5630714893341064 (2.5630714893341064) | > loss_kl: 1.8583369255065918 (1.8583369255065918) | > loss_feat: 4.803923606872559 (4.803923606872559) | > loss_mel: 17.875146865844727 (17.875146865844727) | > loss_duration: 1.5176724195480347 (1.5176724195480347) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.61815071105957 (28.61815071105957) | > grad_norm_1: tensor(6082.2480, device='cuda:0') (tensor(6082.2480, device='cuda:0')) | > current_lr_0: 0.00016770078634528412 | > current_lr_1: 0.00016770078634528412 | > step_time: 0.4193 (0.4192991256713867) | > loader_time: 0.0061 (0.006055355072021484)  --> TIME: 2025-02-25 22:43:56 -- STEP: 51/72 -- GLOBAL_STEP: 1101500 | > loss_disc: 2.5105302333831787 (2.6113223655551088) | > loss_disc_real_0: 0.11983408778905869 (0.15336294282300797) | > loss_disc_real_1: 0.27352675795555115 (0.23247728452962987) | > loss_disc_real_2: 0.2788175940513611 (0.22945833074695923) | > loss_disc_real_3: 0.24774980545043945 (0.23174210973814421) | > loss_disc_real_4: 0.24061265587806702 (0.23920388665853762) | > loss_disc_real_5: 0.24203258752822876 (0.24431692940347335) | > loss_0: 2.5105302333831787 (2.6113223655551088) | > grad_norm_0: tensor(787.6345, device='cuda:0') (tensor(320.4301, device='cuda:0')) | > loss_gen: 2.1579885482788086 (2.2737281462725476) | > loss_kl: 1.8959349393844604 (2.036332420274323) | > loss_feat: 3.766638994216919 (4.126598072986975) | > loss_mel: 16.208805084228516 (16.846843177197016) | > loss_duration: 1.5675780773162842 (1.5196003399643243) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.596946716308594 (26.80310219409419) | > grad_norm_1: tensor(12323.1436, device='cuda:0') (tensor(6628.8467, device='cuda:0')) | > current_lr_0: 0.00016770078634528412 | > current_lr_1: 0.00016770078634528412 | > step_time: 0.4581 (0.43861836078120214) | > loader_time: 0.0088 (0.007672440771963082) --> EVAL PERFORMANCE | > avg_loader_time: 0.008523178772187567 (-2.586673682844487e-05) | > avg_step_time: 0.45046267039339305 (-0.0040367657030132564) | > avg_loss_disc: 2.6023393214588433 (+0.10465837532365851) | > avg_loss_disc_real_0: 0.13959516658329627 (+0.04022603489878311) | > avg_loss_disc_real_1: 0.23174164307789064 (+0.016568962537067022) | > avg_loss_disc_real_2: 0.23261681230555117 (+0.006485017987204272) | > avg_loss_disc_real_3: 0.23253431874261776 (+0.0028852411139179412) | > avg_loss_disc_real_4: 0.2391971234284656 (+0.0013515460239329757) | > avg_loss_disc_real_5: 0.2459238328984086 (+0.0010410855773469174) | > avg_loss_0: 2.6023393214588433 (+0.10465837532365851) | > avg_grad_norm_0: tensor(328.1056, device='cuda:0') (+tensor(37.7523, device='cuda:0')) | > avg_loss_gen: 2.2383125063399203 (-0.05295381747500949) | > avg_loss_kl: 2.061710898305329 (-0.04711434371034828) | > avg_loss_feat: 4.068951902255204 (-0.09770360463102268) | > avg_loss_mel: 16.81351331254125 (-0.12496992567897891) | > avg_loss_duration: 1.5361074998345177 (-0.027495305303116613) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.718596149498307 (-0.3502369732923931) | > avg_grad_norm_1: tensor(7579.8667, device='cuda:0') (tensor(-779.9565, device='cuda:0'))  > EPOCH: 1410/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:44:08)   --> TIME: 2025-02-25 22:44:22 -- STEP: 29/72 -- GLOBAL_STEP: 1101550 | > loss_disc: 2.630128860473633 (2.5196080947744433) | > loss_disc_real_0: 0.06205900013446808 (0.09443487792179503) | > loss_disc_real_1: 0.26355692744255066 (0.20946389231188545) | > loss_disc_real_2: 0.2407447248697281 (0.23628769866351423) | > loss_disc_real_3: 0.2514367401599884 (0.2315856464977922) | > loss_disc_real_4: 0.2624461352825165 (0.24147905203802833) | > loss_disc_real_5: 0.26000455021858215 (0.2451401625213952) | > loss_0: 2.630128860473633 (2.5196080947744433) | > grad_norm_0: tensor(183.7004, device='cuda:0') (tensor(359.7956, device='cuda:0')) | > loss_gen: 2.2423031330108643 (2.236526682459075) | > loss_kl: 2.1579103469848633 (1.8926401960438695) | > loss_feat: 3.839651346206665 (4.2175210673233545) | > loss_mel: 17.325239181518555 (16.771767090106835) | > loss_duration: 1.5163875818252563 (1.4843596343336434) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.081491470336914 (26.602814509950836) | > grad_norm_1: tensor(11118.5938, device='cuda:0') (tensor(9817.5820, device='cuda:0')) | > current_lr_0: 0.00016767982374699095 | > current_lr_1: 0.00016767982374699095 | > step_time: 0.4322 (0.42666493613144446) | > loader_time: 0.0079 (0.006827847710971175) --> EVAL PERFORMANCE | > avg_loader_time: 0.008505985770427006 (-1.7193001760560864e-05) | > avg_step_time: 0.4465252916577836 (-0.003937378735609454) | > avg_loss_disc: 2.5195947902303355 (-0.08274453122850778) | > avg_loss_disc_real_0: 0.09490994143653923 (-0.04468522514675703) | > avg_loss_disc_real_1: 0.21565502735090927 (-0.016086615726981363) | > avg_loss_disc_real_2: 0.231565600549671 (-0.0010512117558801604) | > avg_loss_disc_real_3: 0.23210900329368214 (-0.00042531544893562034) | > avg_loss_disc_real_4: 0.2455189935338329 (+0.006321870105367305) | > avg_loss_disc_real_5: 0.247260633908527 (+0.0013368010101184047) | > avg_loss_0: 2.5195947902303355 (-0.08274453122850778) | > avg_grad_norm_0: tensor(340.2245, device='cuda:0') (+tensor(12.1189, device='cuda:0')) | > avg_loss_gen: 2.2501080791715165 (+0.011795572831596157) | > avg_loss_kl: 1.9707089528231554 (-0.09100194548217355) | > avg_loss_feat: 4.180734805657829 (+0.11178290340262453) | > avg_loss_mel: 16.60082036676542 (-0.21269294577583153) | > avg_loss_duration: 1.5251481398730211 (-0.010959359961496551) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.52752030063683 (-0.19107584886147677) | > avg_grad_norm_1: tensor(9339.1934, device='cuda:0') (+tensor(1759.3267, device='cuda:0'))  > EPOCH: 1411/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:44:44)   --> TIME: 2025-02-25 22:44:49 -- STEP: 7/72 -- GLOBAL_STEP: 1101600 | > loss_disc: 2.9079113006591797 (2.8017084257943288) | > loss_disc_real_0: 0.429726243019104 (0.20565651570047652) | > loss_disc_real_1: 0.217066690325737 (0.1806415713259152) | > loss_disc_real_2: 0.20117896795272827 (0.31188067368098665) | > loss_disc_real_3: 0.20181381702423096 (0.2046173300061907) | > loss_disc_real_4: 0.2588950991630554 (0.21693719072001322) | > loss_disc_real_5: 0.23570604622364044 (0.22801934608391353) | > loss_0: 2.9079113006591797 (2.8017084257943288) | > grad_norm_0: tensor(801.8833, device='cuda:0') (tensor(508.1643, device='cuda:0')) | > loss_gen: 2.266181468963623 (2.1664033106395175) | > loss_kl: 2.0456135272979736 (1.8745674405779158) | > loss_feat: 3.753248453140259 (4.19539441381182) | > loss_mel: 16.510866165161133 (16.739789417811803) | > loss_duration: 1.4433668851852417 (1.4762324094772339) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.019277572631836 (26.4523868560791) | > grad_norm_1: tensor(4178.5181, device='cuda:0') (tensor(7349.4561, device='cuda:0')) | > current_lr_0: 0.00016765886376902256 | > current_lr_1: 0.00016765886376902256 | > step_time: 0.4286 (0.4248090812138149) | > loader_time: 0.0056 (0.006281886781964984)  --> TIME: 2025-02-25 22:45:12 -- STEP: 57/72 -- GLOBAL_STEP: 1101650 | > loss_disc: 3.0253920555114746 (2.6867315727367735) | > loss_disc_real_0: 0.5317556262016296 (0.18471030524948187) | > loss_disc_real_1: 0.11376834660768509 (0.2245055097236968) | > loss_disc_real_2: 0.2178802192211151 (0.24281874455903707) | > loss_disc_real_3: 0.17713864147663116 (0.22423818472184634) | > loss_disc_real_4: 0.2098427265882492 (0.23418656618971573) | > loss_disc_real_5: 0.22537299990653992 (0.2433007369961655) | > loss_0: 3.0253920555114746 (2.6867315727367735) | > grad_norm_0: tensor(802.7698, device='cuda:0') (tensor(421.2036, device='cuda:0')) | > loss_gen: 1.8086906671524048 (2.1297593158588075) | > loss_kl: 2.11761212348938 (2.0291458347387485) | > loss_feat: 3.3682284355163574 (3.8595916221016333) | > loss_mel: 16.8996639251709 (16.760129175688093) | > loss_duration: 1.5868403911590576 (1.5148927307965463) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.781036376953125 (26.293518668726872) | > grad_norm_1: tensor(3475.2937, device='cuda:0') (tensor(8014.5601, device='cuda:0')) | > current_lr_0: 0.00016765886376902256 | > current_lr_1: 0.00016765886376902256 | > step_time: 0.4705 (0.4432142115475839) | > loader_time: 0.0095 (0.007873656456930596) --> EVAL PERFORMANCE | > avg_loader_time: 0.008490273650263395 (-1.5712120163611024e-05) | > avg_step_time: 0.4508297544130137 (+0.00430446275523011) | > avg_loss_disc: 2.708471694462736 (+0.18887690423240056) | > avg_loss_disc_real_0: 0.19995673682907938 (+0.10504679539254015) | > avg_loss_disc_real_1: 0.221161473909734 (+0.005506446558824735) | > avg_loss_disc_real_2: 0.24049774211057476 (+0.008932141560903745) | > avg_loss_disc_real_3: 0.2205463364510469 (-0.011562666842635239) | > avg_loss_disc_real_4: 0.23276696070818834 (-0.012752032825644577) | > avg_loss_disc_real_5: 0.24463636984287854 (-0.002624264065648474) | > avg_loss_0: 2.708471694462736 (+0.18887690423240056) | > avg_grad_norm_0: tensor(419.0884, device='cuda:0') (+tensor(78.8639, device='cuda:0')) | > avg_loss_gen: 2.1253588753686827 (-0.12474920380283372) | > avg_loss_kl: 2.050026494012753 (+0.07931754118959744) | > avg_loss_feat: 3.792244639195187 (-0.38849016646264145) | > avg_loss_mel: 16.74699271564753 (+0.14617234888211073) | > avg_loss_duration: 1.5275273155158675 (+0.0023791756428463984) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.24215005149304 (-0.2853702491437886) | > avg_grad_norm_1: tensor(8117.1348, device='cuda:0') (tensor(-1222.0586, device='cuda:0'))  > EPOCH: 1412/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:45:21)   --> TIME: 2025-02-25 22:45:39 -- STEP: 35/72 -- GLOBAL_STEP: 1101700 | > loss_disc: 2.581671953201294 (2.5875881603785924) | > loss_disc_real_0: 0.06819663941860199 (0.12988077726747307) | > loss_disc_real_1: 0.24364624917507172 (0.21861061134508677) | > loss_disc_real_2: 0.20355355739593506 (0.21906607768365316) | > loss_disc_real_3: 0.28431594371795654 (0.22900406462805611) | > loss_disc_real_4: 0.21088145673274994 (0.219832900592259) | > loss_disc_real_5: 0.2529396414756775 (0.23777868236814226) | > loss_0: 2.581671953201294 (2.5875881603785924) | > grad_norm_0: tensor(259.3992, device='cuda:0') (tensor(284.4021, device='cuda:0')) | > loss_gen: 2.2072041034698486 (2.206851032802036) | > loss_kl: 2.0739386081695557 (2.0430296795708784) | > loss_feat: 4.109883785247803 (4.23746715954372) | > loss_mel: 16.322463989257812 (16.966580799647733) | > loss_duration: 1.517776608467102 (1.4950308322906494) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.23126792907715 (26.94895918709891) | > grad_norm_1: tensor(4563.3652, device='cuda:0') (tensor(5363.7852, device='cuda:0')) | > current_lr_0: 0.00016763790641105143 | > current_lr_1: 0.00016763790641105143 | > step_time: 0.4333 (0.43594423021589007) | > loader_time: 0.0084 (0.007159859793526786) --> EVAL PERFORMANCE | > avg_loader_time: 0.008673980202473385 (+0.00018370655220999003) | > avg_step_time: 0.45333284055682976 (+0.002503086143816058) | > avg_loss_disc: 2.5598708676620268 (-0.1486008268007093) | > avg_loss_disc_real_0: 0.10952725447714329 (-0.0904294823519361) | > avg_loss_disc_real_1: 0.2215649547501349 (+0.0004034808404008994) | > avg_loss_disc_real_2: 0.2177848636474408 (-0.022712878463133968) | > avg_loss_disc_real_3: 0.23183101415634155 (+0.011284677705294649) | > avg_loss_disc_real_4: 0.2240281575162646 (-0.008738803191923744) | > avg_loss_disc_real_5: 0.2419206208326447 (-0.002715749010233831) | > avg_loss_0: 2.5598708676620268 (-0.1486008268007093) | > avg_grad_norm_0: tensor(269.1329, device='cuda:0') (tensor(-149.9555, device='cuda:0')) | > avg_loss_gen: 2.2217828891646696 (+0.09642401379598686) | > avg_loss_kl: 2.0626141454132516 (+0.012587651400498778) | > avg_loss_feat: 4.20585045680194 (+0.41360581760675297) | > avg_loss_mel: 16.76552984748088 (+0.018537131833351594) | > avg_loss_duration: 1.528735894552419 (+0.0012085790365514715) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.784512989957566 (+0.5423629384645245) | > avg_grad_norm_1: tensor(6072.4194, device='cuda:0') (tensor(-2044.7153, device='cuda:0'))  > EPOCH: 1413/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:45:58)   --> TIME: 2025-02-25 22:46:05 -- STEP: 13/72 -- GLOBAL_STEP: 1101750 | > loss_disc: 2.579575300216675 (2.6137923827538123) | > loss_disc_real_0: 0.1636938750743866 (0.1564094415650918) | > loss_disc_real_1: 0.2915131151676178 (0.2283897606226114) | > loss_disc_real_2: 0.1910431683063507 (0.22760145251567548) | > loss_disc_real_3: 0.19955986738204956 (0.22561790851446298) | > loss_disc_real_4: 0.2229180932044983 (0.25773934561472667) | > loss_disc_real_5: 0.2488185167312622 (0.2577941176983026) | > loss_0: 2.579575300216675 (2.6137923827538123) | > grad_norm_0: tensor(44.9360, device='cuda:0') (tensor(205.1560, device='cuda:0')) | > loss_gen: 2.1127755641937256 (2.268417615156907) | > loss_kl: 1.895636796951294 (1.9026775818604689) | > loss_feat: 3.635143995285034 (4.025979830668523) | > loss_mel: 16.031091690063477 (16.817262942974384) | > loss_duration: 1.4250359535217285 (1.4694477228017955) | > amp_scaler: 8.0 (9.23076923076923) | > loss_1: 25.09968376159668 (26.483785922710712) | > grad_norm_1: tensor(8022.5234, device='cuda:0') (tensor(7544.8042, device='cuda:0')) | > current_lr_0: 0.00016761695167275003 | > current_lr_1: 0.00016761695167275003 | > step_time: 0.4658 (0.42434070660517764) | > loader_time: 0.0061 (0.006685385337242713)  --> TIME: 2025-02-25 22:46:28 -- STEP: 63/72 -- GLOBAL_STEP: 1101800 | > loss_disc: 2.5939459800720215 (2.6238501563904784) | > loss_disc_real_0: 0.19470325112342834 (0.14535321182911357) | > loss_disc_real_1: 0.22124971449375153 (0.22709717128484969) | > loss_disc_real_2: 0.22615064680576324 (0.23286148314438168) | > loss_disc_real_3: 0.2235831618309021 (0.23339954917393033) | > loss_disc_real_4: 0.23389682173728943 (0.2453908364451121) | > loss_disc_real_5: 0.2516932189464569 (0.24531297977008518) | > loss_0: 2.5939459800720215 (2.6238501563904784) | > grad_norm_0: tensor(346.7830, device='cuda:0') (tensor(239.4711, device='cuda:0')) | > loss_gen: 2.390028953552246 (2.1754894256591792) | > loss_kl: 2.3793106079101562 (2.1902789388384143) | > loss_feat: 3.978581190109253 (3.951336342190939) | > loss_mel: 17.49142837524414 (17.160604840233212) | > loss_duration: 1.5805041790008545 (1.5319071667534965) | > amp_scaler: 8.0 (8.25396825396825) | > loss_1: 27.819852828979492 (27.00961679125589) | > grad_norm_1: tensor(7869.9414, device='cuda:0') (tensor(7328.5781, device='cuda:0')) | > current_lr_0: 0.00016761695167275003 | > current_lr_1: 0.00016761695167275003 | > step_time: 0.4859 (0.44026486078898114) | > loader_time: 0.0112 (0.008124847260732499) --> EVAL PERFORMANCE | > avg_loader_time: 0.008505676833676621 (-0.00016830336879676382) | > avg_step_time: 0.44674313236290303 (-0.0065897081939267355) | > avg_loss_disc: 2.623328998055257 (+0.06345813039323023) | > avg_loss_disc_real_0: 0.14766729261044043 (+0.03814003813329714) | > avg_loss_disc_real_1: 0.22416419346987365 (+0.0025992387197387445) | > avg_loss_disc_real_2: 0.2319660289606578 (+0.014181165313217003) | > avg_loss_disc_real_3: 0.23145825909057133 (-0.000372755065770225) | > avg_loss_disc_real_4: 0.2443424127471279 (+0.020314255230863304) | > avg_loss_disc_real_5: 0.2452223002910614 (+0.0033016794584166953) | > avg_loss_0: 2.623328998055257 (+0.06345813039323023) | > avg_grad_norm_0: tensor(245.3893, device='cuda:0') (tensor(-23.7436, device='cuda:0')) | > avg_loss_gen: 2.1611643542706123 (-0.06061853489405733) | > avg_loss_kl: 2.2026964543570937 (+0.1400823089438421) | > avg_loss_feat: 3.9062910147116217 (-0.2995594420903185) | > avg_loss_mel: 17.14241088276178 (+0.3768810352808991) | > avg_loss_duration: 1.5402347621783403 (+0.011498867625921338) | > avg_amp_scaler: 8.225352112676052 (+0.22535211267605249) | > avg_loss_1: 26.952797500180527 (+0.16828451022296065) | > avg_grad_norm_1: tensor(7608.7886, device='cuda:0') (+tensor(1536.3691, device='cuda:0'))  > EPOCH: 1414/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:46:34)   --> TIME: 2025-02-25 22:46:54 -- STEP: 41/72 -- GLOBAL_STEP: 1101850 | > loss_disc: 2.6805641651153564 (2.581315104554339) | > loss_disc_real_0: 0.1917496621608734 (0.11356743915778834) | > loss_disc_real_1: 0.22307047247886658 (0.22580021329042388) | > loss_disc_real_2: 0.23797911405563354 (0.2386071016875709) | > loss_disc_real_3: 0.23996038734912872 (0.23256758764022734) | > loss_disc_real_4: 0.2635679841041565 (0.24579299186787953) | > loss_disc_real_5: 0.26310276985168457 (0.25059529848215056) | > loss_0: 2.6805641651153564 (2.581315104554339) | > grad_norm_0: tensor(641.7920, device='cuda:0') (tensor(337.0931, device='cuda:0')) | > loss_gen: 2.177603244781494 (2.1461547932973724) | > loss_kl: 2.0224742889404297 (2.0193544219179844) | > loss_feat: 4.132186412811279 (3.971729883333532) | > loss_mel: 16.41758918762207 (16.996896162265685) | > loss_duration: 1.5330950021743774 (1.5028269610753873) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.282947540283203 (26.636962285855923) | > grad_norm_1: tensor(15130.4551, device='cuda:0') (tensor(11827.1367, device='cuda:0')) | > current_lr_0: 0.00016759599955379093 | > current_lr_1: 0.00016759599955379093 | > step_time: 0.4459 (0.4314941545812095) | > loader_time: 0.0096 (0.007396064153531703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573558968557438 (+6.78821348808166e-05) | > avg_step_time: 0.4552790816401092 (+0.00853594927720619) | > avg_loss_disc: 2.6102093441385614 (-0.013119653916695562) | > avg_loss_disc_real_0: 0.10678544750725719 (-0.04088184510318324) | > avg_loss_disc_real_1: 0.2531032986204389 (+0.028939105150565247) | > avg_loss_disc_real_2: 0.22892581975795853 (-0.003040209202699262) | > avg_loss_disc_real_3: 0.23293448522896834 (+0.0014762261383970132) | > avg_loss_disc_real_4: 0.2531810289537403 (+0.008838616206612387) | > avg_loss_disc_real_5: 0.24722535467483628 (+0.002003054383774877) | > avg_loss_0: 2.6102093441385614 (-0.013119653916695562) | > avg_grad_norm_0: tensor(374.0530, device='cuda:0') (+tensor(128.6637, device='cuda:0')) | > avg_loss_gen: 2.1545631365037283 (-0.006601217766883938) | > avg_loss_kl: 2.0541391557371114 (-0.14855729861998235) | > avg_loss_feat: 3.9623026545618623 (+0.0560116398502406) | > avg_loss_mel: 16.811560214405326 (-0.3308506683564545) | > avg_loss_duration: 1.5296257952569237 (-0.010608966921416663) | > avg_amp_scaler: 8.0 (-0.22535211267605249) | > avg_loss_1: 26.51219102026711 (-0.4406064799134164) | > avg_grad_norm_1: tensor(12828.2666, device='cuda:0') (+tensor(5219.4780, device='cuda:0'))  > EPOCH: 1415/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:47:11)   --> TIME: 2025-02-25 22:47:21 -- STEP: 19/72 -- GLOBAL_STEP: 1101900 | > loss_disc: 2.513239860534668 (2.5952110792461194) | > loss_disc_real_0: 0.0803660973906517 (0.11430996656417847) | > loss_disc_real_1: 0.23556330800056458 (0.2168370767643577) | > loss_disc_real_2: 0.23609128594398499 (0.22363141884929255) | > loss_disc_real_3: 0.2130783200263977 (0.22988772705981605) | > loss_disc_real_4: 0.22760973870754242 (0.22209494600170537) | > loss_disc_real_5: 0.23555628955364227 (0.2440414420868221) | > loss_0: 2.513239860534668 (2.5952110792461194) | > grad_norm_0: tensor(185.2214, device='cuda:0') (tensor(433.0600, device='cuda:0')) | > loss_gen: 2.213575839996338 (2.087405938851206) | > loss_kl: 1.973113775253296 (1.968701764156944) | > loss_feat: 4.418463706970215 (3.959163703416523) | > loss_mel: 16.548587799072266 (16.737803910907946) | > loss_duration: 1.467064619064331 (1.4728884383251792) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.620805740356445 (26.225963592529297) | > grad_norm_1: tensor(14188.7998, device='cuda:0') (tensor(13367.2725, device='cuda:0')) | > current_lr_0: 0.0001675750500538467 | > current_lr_1: 0.0001675750500538467 | > step_time: 0.4218 (0.43221207668906764) | > loader_time: 0.0059 (0.007023723501908152)  --> TIME: 2025-02-25 22:47:45 -- STEP: 69/72 -- GLOBAL_STEP: 1101950 | > loss_disc: 2.5628345012664795 (2.5809175968170166) | > loss_disc_real_0: 0.08531685918569565 (0.11364833870227786) | > loss_disc_real_1: 0.2049824595451355 (0.22116705213767895) | > loss_disc_real_2: 0.2116696685552597 (0.23184816496095795) | > loss_disc_real_3: 0.2236337959766388 (0.23191682195317917) | > loss_disc_real_4: 0.23373615741729736 (0.23587200736653977) | > loss_disc_real_5: 0.23382318019866943 (0.24758707677972489) | > loss_0: 2.5628345012664795 (2.5809175968170166) | > grad_norm_0: tensor(152.2313, device='cuda:0') (tensor(418.1342, device='cuda:0')) | > loss_gen: 2.0853569507598877 (2.1278682688008184) | > loss_kl: 2.2331295013427734 (1.996690881425056) | > loss_feat: 4.315064907073975 (3.900021573771601) | > loss_mel: 17.417776107788086 (16.54922272502512) | > loss_duration: 1.6159030199050903 (1.5181423183800518) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.667232513427734 (26.09194581404976) | > grad_norm_1: tensor(10264.7471, device='cuda:0') (tensor(12914.9600, device='cuda:0')) | > current_lr_0: 0.0001675750500538467 | > current_lr_1: 0.0001675750500538467 | > step_time: 0.5309 (0.4503375792848891) | > loader_time: 0.0126 (0.008679780407228334) --> EVAL PERFORMANCE | > avg_loader_time: 0.0087359455269827 (+0.0001623865584252631) | > avg_step_time: 0.4511911902629154 (-0.004087891377193842) | > avg_loss_disc: 2.5807528932329635 (-0.029456450905597897) | > avg_loss_disc_real_0: 0.11367678631779174 (+0.006891338810534553) | > avg_loss_disc_real_1: 0.22064236352141475 (-0.032460935099024146) | > avg_loss_disc_real_2: 0.23118288395270495 (+0.0022570641947464165) | > avg_loss_disc_real_3: 0.2317127914915622 (-0.001221693737406132) | > avg_loss_disc_real_4: 0.23575846158282857 (-0.017422567370911718) | > avg_loss_disc_real_5: 0.24744453333633046 (+0.0002191786614941793) | > avg_loss_0: 2.5807528932329635 (-0.029456450905597897) | > avg_grad_norm_0: tensor(422.3799, device='cuda:0') (+tensor(48.3269, device='cuda:0')) | > avg_loss_gen: 2.12561727241731 (-0.028945864086418194) | > avg_loss_kl: 2.0002190848471413 (-0.053920070889970084) | > avg_loss_feat: 3.903761793190325 (-0.05854086137153747) | > avg_loss_mel: 16.549601648894832 (-0.26195856551049346) | > avg_loss_duration: 1.5190751804432399 (-0.0105506148136838) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.098275010014923 (-0.41391601025218705) | > avg_grad_norm_1: tensor(13000.2666, device='cuda:0') (+tensor(172., device='cuda:0'))  > EPOCH: 1416/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:47:48)   --> TIME: 2025-02-25 22:48:11 -- STEP: 47/72 -- GLOBAL_STEP: 1102000 | > loss_disc: 2.4537405967712402 (2.5897636870120437) | > loss_disc_real_0: 0.10721323639154434 (0.12527496843261926) | > loss_disc_real_1: 0.24659769237041473 (0.2115154076129832) | > loss_disc_real_2: 0.22642937302589417 (0.221799336215283) | > loss_disc_real_3: 0.22751988470554352 (0.22962125311506557) | > loss_disc_real_4: 0.23291727900505066 (0.2412424588457067) | > loss_disc_real_5: 0.23385639488697052 (0.24765756662855756) | > loss_0: 2.4537405967712402 (2.5897636870120437) | > grad_norm_0: tensor(255.8905, device='cuda:0') (tensor(421.3443, device='cuda:0')) | > loss_gen: 2.2874693870544434 (2.1258136921740585) | > loss_kl: 2.152273416519165 (1.9460605281464598) | > loss_feat: 4.207928657531738 (3.982119012386241) | > loss_mel: 16.62689208984375 (16.635838650642558) | > loss_duration: 1.5322002172470093 (1.493827817287851) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.806764602661133 (26.183659939055747) | > grad_norm_1: tensor(12658.5117, device='cuda:0') (tensor(12071.0127, device='cuda:0')) | > current_lr_0: 0.00016755410317258995 | > current_lr_1: 0.00016755410317258995 | > step_time: 0.4546 (0.43821287155151367) | > loader_time: 0.0089 (0.007712348978570167) --> EVAL PERFORMANCE | > avg_loader_time: 0.008679064226822115 (-5.688130016058557e-05) | > avg_step_time: 0.44857351880677987 (-0.0026176714561355063) | > avg_loss_disc: 2.5941591329977545 (+0.01340623976479094) | > avg_loss_disc_real_0: 0.12324312906449952 (+0.009566342746707784) | > avg_loss_disc_real_1: 0.21525994027164622 (-0.0053824232497685365) | > avg_loss_disc_real_2: 0.22533305723902206 (-0.005849826713682893) | > avg_loss_disc_real_3: 0.23185155551198502 (+0.00013876402042281577) | > avg_loss_disc_real_4: 0.24404338002204895 (+0.008284918439220385) | > avg_loss_disc_real_5: 0.24769468332680178 (+0.00025014999047132114) | > avg_loss_0: 2.5941591329977545 (+0.01340623976479094) | > avg_grad_norm_0: tensor(409.8944, device='cuda:0') (tensor(-12.4854, device='cuda:0')) | > avg_loss_gen: 2.1335213201146717 (+0.007904047697361527) | > avg_loss_kl: 1.9942587496529163 (-0.005960335194225008) | > avg_loss_feat: 3.941962242126465 (+0.03820044893614005) | > avg_loss_mel: 16.56707910081031 (+0.01747745191547878) | > avg_loss_duration: 1.5199281716010937 (+0.0008529911578538663) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.15674983279806 (+0.058474822783136204) | > avg_grad_norm_1: tensor(11937.7197, device='cuda:0') (tensor(-1062.5469, device='cuda:0'))  > EPOCH: 1417/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:48:24)   --> TIME: 2025-02-25 22:48:37 -- STEP: 25/72 -- GLOBAL_STEP: 1102050 | > loss_disc: 2.56480073928833 (2.550886325836182) | > loss_disc_real_0: 0.09330730140209198 (0.12122137725353241) | > loss_disc_real_1: 0.1944136917591095 (0.2124962776899338) | > loss_disc_real_2: 0.20940665900707245 (0.22839626729488371) | > loss_disc_real_3: 0.21152718365192413 (0.22991931617259978) | > loss_disc_real_4: 0.22588717937469482 (0.2397706115245819) | > loss_disc_real_5: 0.2346869707107544 (0.24418602168560027) | > loss_0: 2.56480073928833 (2.550886325836182) | > grad_norm_0: tensor(314.0788, device='cuda:0') (tensor(397.1566, device='cuda:0')) | > loss_gen: 1.979312777519226 (2.155481166839599) | > loss_kl: 1.9182244539260864 (1.9120887899398804) | > loss_feat: 4.147629261016846 (4.037598466873169) | > loss_mel: 15.993410110473633 (16.409321327209472) | > loss_duration: 1.4877142906188965 (1.4711527490615846) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.526290893554688 (25.985642547607423) | > grad_norm_1: tensor(12102.5654, device='cuda:0') (tensor(12381.3213, device='cuda:0')) | > current_lr_0: 0.00016753315890969338 | > current_lr_1: 0.00016753315890969338 | > step_time: 0.4288 (0.4298325729370117) | > loader_time: 0.0081 (0.0066423511505126955) --> EVAL PERFORMANCE | > avg_loader_time: 0.008505958906361757 (-0.0001731053204603577) | > avg_step_time: 0.4497373641376764 (+0.001163845330896529) | > avg_loss_disc: 2.6109803763913435 (+0.016821243393589036) | > avg_loss_disc_real_0: 0.12515811249613762 (+0.0019149834316380954) | > avg_loss_disc_real_1: 0.22042958627284412 (+0.005169646001197903) | > avg_loss_disc_real_2: 0.2333650809358543 (+0.008032023696832258) | > avg_loss_disc_real_3: 0.23160365328822338 (-0.0002479022237616424) | > avg_loss_disc_real_4: 0.2375377632782493 (-0.006505616743799653) | > avg_loss_disc_real_5: 0.2439218753240478 (-0.00377280800275398) | > avg_loss_0: 2.6109803763913435 (+0.016821243393589036) | > avg_grad_norm_0: tensor(395.8446, device='cuda:0') (tensor(-14.0498, device='cuda:0')) | > avg_loss_gen: 2.0862456728035297 (-0.04727564731114198) | > avg_loss_kl: 2.017090523746652 (+0.022831774093735602) | > avg_loss_feat: 3.89117019948825 (-0.050792042638214685) | > avg_loss_mel: 16.49668185811647 (-0.07039724269383996) | > avg_loss_duration: 1.5205258399667874 (+0.0005976683656936377) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.011714075652645 (-0.14503575714541483) | > avg_grad_norm_1: tensor(11856.8145, device='cuda:0') (tensor(-80.9053, device='cuda:0'))  > EPOCH: 1418/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:49:01)   --> TIME: 2025-02-25 22:49:04 -- STEP: 3/72 -- GLOBAL_STEP: 1102100 | > loss_disc: 3.2146735191345215 (2.8333216508229575) | > loss_disc_real_0: 0.20957797765731812 (0.1099397415916125) | > loss_disc_real_1: 0.17730440199375153 (0.17330039044221243) | > loss_disc_real_2: 0.20958803594112396 (0.2223072350025177) | > loss_disc_real_3: 0.3376466929912567 (0.33009562889734906) | > loss_disc_real_4: 0.2099762260913849 (0.22023572524388632) | > loss_disc_real_5: 0.24402518570423126 (0.2469829519589742) | > loss_0: 3.2146735191345215 (2.8333216508229575) | > grad_norm_0: tensor(958.7934, device='cuda:0') (tensor(585.8549, device='cuda:0')) | > loss_gen: 1.6808080673217773 (2.013878266016642) | > loss_kl: 1.9006946086883545 (1.9626094500223796) | > loss_feat: 3.7507357597351074 (3.9092840353647866) | > loss_mel: 16.85322380065918 (16.90289243062337) | > loss_duration: 1.488829255104065 (1.4827642838160198) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.674291610717773 (26.271427790323894) | > grad_norm_1: tensor(3204.2791, device='cuda:0') (tensor(9151.5645, device='cuda:0')) | > current_lr_0: 0.00016751221726482967 | > current_lr_1: 0.00016751221726482967 | > step_time: 0.4135 (0.4143830140431722) | > loader_time: 0.0084 (0.006651719411214192)  --> TIME: 2025-02-25 22:49:27 -- STEP: 53/72 -- GLOBAL_STEP: 1102150 | > loss_disc: 2.515252113342285 (2.727736383114221) | > loss_disc_real_0: 0.09949205815792084 (0.16831361762476418) | > loss_disc_real_1: 0.21715666353702545 (0.2192029889743283) | > loss_disc_real_2: 0.1833602339029312 (0.21657535040153647) | > loss_disc_real_3: 0.2626792788505554 (0.24472686170407063) | > loss_disc_real_4: 0.23932518064975739 (0.2294853501162439) | > loss_disc_real_5: 0.22730691730976105 (0.24151286904541952) | > loss_0: 2.515252113342285 (2.727736383114221) | > grad_norm_0: tensor(396.2616, device='cuda:0') (tensor(405.2986, device='cuda:0')) | > loss_gen: 2.375232219696045 (2.0131307255547) | > loss_kl: 2.6220760345458984 (2.157890515507392) | > loss_feat: 4.315850257873535 (3.8606539042490833) | > loss_mel: 18.130338668823242 (17.17458249937814) | > loss_duration: 1.6226027011871338 (1.5204613028832201) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.066099166870117 (26.726719010551022) | > grad_norm_1: tensor(18350.9648, device='cuda:0') (tensor(9132.3730, device='cuda:0')) | > current_lr_0: 0.00016751221726482967 | > current_lr_1: 0.00016751221726482967 | > step_time: 0.4517 (0.4492990250857371) | > loader_time: 0.0091 (0.0076472039492625115) --> EVAL PERFORMANCE | > avg_loader_time: 0.00849170080372985 (-1.4258102631907812e-05) | > avg_step_time: 0.4572237813976449 (+0.007486417259968514) | > avg_loss_disc: 2.7180358927014847 (+0.10705551631014121) | > avg_loss_disc_real_0: 0.1577242774967576 (+0.032566165000619984) | > avg_loss_disc_real_1: 0.2190339810831446 (-0.001395605189699528) | > avg_loss_disc_real_2: 0.22096355406331344 (-0.012401526872540869) | > avg_loss_disc_real_3: 0.24424299115026502 (+0.012639337862041633) | > avg_loss_disc_real_4: 0.23077719333306165 (-0.006760569945187644) | > avg_loss_disc_real_5: 0.24158660165021117 (-0.002335273673836624) | > avg_loss_0: 2.7180358927014847 (+0.10705551631014121) | > avg_grad_norm_0: tensor(402.6906, device='cuda:0') (+tensor(6.8459, device='cuda:0')) | > avg_loss_gen: 2.0149291166117496 (-0.07131655619178012) | > avg_loss_kl: 2.2229430087855166 (+0.20585248503886477) | > avg_loss_feat: 3.8704620918757477 (-0.020708107612502413) | > avg_loss_mel: 17.328425984987078 (+0.831744126870607) | > avg_loss_duration: 1.5425823742235212 (+0.022056534256733817) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.979342769569076 (+0.9676286939164314) | > avg_grad_norm_1: tensor(9818.8057, device='cuda:0') (tensor(-2038.0088, device='cuda:0'))  > EPOCH: 1419/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:49:38)   --> TIME: 2025-02-25 22:49:53 -- STEP: 31/72 -- GLOBAL_STEP: 1102200 | > loss_disc: 2.9631948471069336 (2.6975516657675462) | > loss_disc_real_0: 0.3007110357284546 (0.14980041199634153) | > loss_disc_real_1: 0.22774963080883026 (0.2648198198406928) | > loss_disc_real_2: 0.22140155732631683 (0.23215539657300518) | > loss_disc_real_3: 0.22623640298843384 (0.2367042524199332) | > loss_disc_real_4: 0.2422744780778885 (0.24673381976542935) | > loss_disc_real_5: 0.25079345703125 (0.25052290962588397) | > loss_0: 2.9631948471069336 (2.6975516657675462) | > grad_norm_0: tensor(851.1175, device='cuda:0') (tensor(377.8434, device='cuda:0')) | > loss_gen: 1.7121800184249878 (2.1378177596676733) | > loss_kl: 2.121156692504883 (2.4283789088649135) | > loss_feat: 3.344355821609497 (3.8812746463283414) | > loss_mel: 17.802490234375 (18.65263274408155) | > loss_duration: 1.538942575454712 (1.541914286152009) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.519123077392578 (28.642018533522084) | > grad_norm_1: tensor(554.8956, device='cuda:0') (tensor(10572.7900, device='cuda:0')) | > current_lr_0: 0.00016749127823767156 | > current_lr_1: 0.00016749127823767156 | > step_time: 0.4436 (0.4329777609917425) | > loader_time: 0.0088 (0.006892681121826172) --> EVAL PERFORMANCE | > avg_loader_time: 0.008524142520528443 (+3.244171679859381e-05) | > avg_step_time: 0.4511401485389387 (-0.006083632858706223) | > avg_loss_disc: 2.7019811482496667 (-0.01605474445181798) | > avg_loss_disc_real_0: 0.13719316383062957 (-0.02053111366612803) | > avg_loss_disc_real_1: 0.24509627372026443 (+0.026062292637119844) | > avg_loss_disc_real_2: 0.250456743047271 (+0.029493188983957558) | > avg_loss_disc_real_3: 0.2400175345615602 (-0.0042254565887048234) | > avg_loss_disc_real_4: 0.25299842466770756 (+0.022221231334645902) | > avg_loss_disc_real_5: 0.2549749048243105 (+0.013388303174099336) | > avg_loss_0: 2.7019811482496667 (-0.01605474445181798) | > avg_grad_norm_0: tensor(345.0986, device='cuda:0') (tensor(-57.5920, device='cuda:0')) | > avg_loss_gen: 2.170179017832582 (+0.1552499012208326) | > avg_loss_kl: 2.4814103908941787 (+0.25846738210866205) | > avg_loss_feat: 4.011502937531807 (+0.14104084565605923) | > avg_loss_mel: 19.197722770798368 (+1.86929678581129) | > avg_loss_duration: 1.5914912358136242 (+0.04890886159010299) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.45230658625213 (+2.4729638166830554) | > avg_grad_norm_1: tensor(8882.9277, device='cuda:0') (tensor(-935.8779, device='cuda:0'))  > EPOCH: 1420/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:50:14)   --> TIME: 2025-02-25 22:50:20 -- STEP: 9/72 -- GLOBAL_STEP: 1102250 | > loss_disc: 2.6686105728149414 (2.6042805777655706) | > loss_disc_real_0: 0.1412689983844757 (0.11306140199303627) | > loss_disc_real_1: 0.23134493827819824 (0.26178038948112065) | > loss_disc_real_2: 0.23262320458889008 (0.1809055209159851) | > loss_disc_real_3: 0.24425525963306427 (0.22475272748205397) | > loss_disc_real_4: 0.23042352497577667 (0.20907239119211832) | > loss_disc_real_5: 0.26318061351776123 (0.23834777375062308) | > loss_0: 2.6686105728149414 (2.6042805777655706) | > grad_norm_0: tensor(525.4512, device='cuda:0') (tensor(233.0407, device='cuda:0')) | > loss_gen: 1.903441309928894 (2.135515226258172) | > loss_kl: 2.0917460918426514 (2.040872666570875) | > loss_feat: 3.5388646125793457 (3.9447861777411566) | > loss_mel: 17.98300552368164 (17.800726148817276) | > loss_duration: 1.5127061605453491 (1.5293881230884128) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.02976417541504 (27.451288647121853) | > grad_norm_1: tensor(5448.5459, device='cuda:0') (tensor(10973.2549, device='cuda:0')) | > current_lr_0: 0.00016747034182789186 | > current_lr_1: 0.00016747034182789186 | > step_time: 0.441 (0.4273677666982015) | > loader_time: 0.0056 (0.006305827034844292)  --> TIME: 2025-02-25 22:50:43 -- STEP: 59/72 -- GLOBAL_STEP: 1102300 | > loss_disc: 3.15151047706604 (2.5995316586251986) | > loss_disc_real_0: 0.19867345690727234 (0.11482805973392422) | > loss_disc_real_1: 0.21114136278629303 (0.21837055455830137) | > loss_disc_real_2: 0.24196556210517883 (0.2344110499499208) | > loss_disc_real_3: 0.24464261531829834 (0.23205776790441093) | > loss_disc_real_4: 0.2620098888874054 (0.24163673969648652) | > loss_disc_real_5: 0.25747033953666687 (0.25037492735911215) | > loss_0: 3.15151047706604 (2.5995316586251986) | > grad_norm_0: tensor(718.9613, device='cuda:0') (tensor(371.8542, device='cuda:0')) | > loss_gen: 1.5478203296661377 (2.132636440002313) | > loss_kl: 1.9624758958816528 (2.033648222179736) | > loss_feat: 3.2525622844696045 (3.9388966641183627) | > loss_mel: 16.783283233642578 (17.14101875434487) | > loss_duration: 1.5790073871612549 (1.533609044753899) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.125150680541992 (26.77980911125571) | > grad_norm_1: tensor(3087.0720, device='cuda:0') (tensor(12846.7266, device='cuda:0')) | > current_lr_0: 0.00016747034182789186 | > current_lr_1: 0.00016747034182789186 | > step_time: 0.4672 (0.442376262050564) | > loader_time: 0.0099 (0.008001695244999255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008564720691089898 (+4.057817056145503e-05) | > avg_step_time: 0.4516722014252569 (+0.0005320528863181906) | > avg_loss_disc: 2.6155981681716276 (-0.08638298007803913) | > avg_loss_disc_real_0: 0.12774472704655687 (-0.0094484367840727) | > avg_loss_disc_real_1: 0.22038861227707124 (-0.024707661443193196) | > avg_loss_disc_real_2: 0.23023964569602215 (-0.020217097351248853) | > avg_loss_disc_real_3: 0.23035775964528743 (-0.009659774916272762) | > avg_loss_disc_real_4: 0.24259104086479671 (-0.010407383802910841) | > avg_loss_disc_real_5: 0.2498139590024947 (-0.00516094582181581) | > avg_loss_0: 2.6155981681716276 (-0.08638298007803913) | > avg_grad_norm_0: tensor(386.2013, device='cuda:0') (+tensor(41.1028, device='cuda:0')) | > avg_loss_gen: 2.1292836061665716 (-0.04089541166601052) | > avg_loss_kl: 2.0597587857447874 (-0.42165160514939126) | > avg_loss_feat: 3.873003603706897 (-0.13849933382490986) | > avg_loss_mel: 17.066153821810865 (-2.1315689489875034) | > avg_loss_duration: 1.5441352884534378 (-0.047355947360186335) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.67233504711742 (-2.77997153913471) | > avg_grad_norm_1: tensor(12464.1230, device='cuda:0') (+tensor(3581.1953, device='cuda:0'))  > EPOCH: 1421/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:50:51)   --> TIME: 2025-02-25 22:51:09 -- STEP: 37/72 -- GLOBAL_STEP: 1102350 | > loss_disc: 2.44045352935791 (2.566291763975814) | > loss_disc_real_0: 0.07122962176799774 (0.09719343126021526) | > loss_disc_real_1: 0.35306087136268616 (0.25952027013172974) | > loss_disc_real_2: 0.19332048296928406 (0.22018666766785286) | > loss_disc_real_3: 0.2841712236404419 (0.23676483051196948) | > loss_disc_real_4: 0.3160870373249054 (0.24405544714347735) | > loss_disc_real_5: 0.21584168076515198 (0.23667883188337893) | > loss_0: 2.44045352935791 (2.566291763975814) | > grad_norm_0: tensor(191.3906, device='cuda:0') (tensor(248.9347, device='cuda:0')) | > loss_gen: 2.616595506668091 (2.312452490265305) | > loss_kl: 1.9046694040298462 (1.9369001324112352) | > loss_feat: 4.725105285644531 (4.334677979752824) | > loss_mel: 16.776639938354492 (16.779918722204265) | > loss_duration: 1.5456057786941528 (1.4998545711104934) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.56861686706543 (26.863803605775573) | > grad_norm_1: tensor(6695.2461, device='cuda:0') (tensor(5588.8882, device='cuda:0')) | > current_lr_0: 0.00016744940803516337 | > current_lr_1: 0.00016744940803516337 | > step_time: 0.4647 (0.43480061195992137) | > loader_time: 0.0082 (0.007163962802371463) --> EVAL PERFORMANCE | > avg_loader_time: 0.00876606350213709 (+0.0002013428110471914) | > avg_step_time: 0.4576152479144889 (+0.005943046489231996) | > avg_loss_disc: 2.6613026001084026 (+0.04570443193677498) | > avg_loss_disc_real_0: 0.13933582087112978 (+0.011591093824572907) | > avg_loss_disc_real_1: 0.24132765241911713 (+0.020939040142045895) | > avg_loss_disc_real_2: 0.2254650897123444 (-0.004774555983677736) | > avg_loss_disc_real_3: 0.23794418342516455 (+0.007586423779877116) | > avg_loss_disc_real_4: 0.24279545352492535 (+0.00020441266012863335) | > avg_loss_disc_real_5: 0.23650831760654986 (-0.01330564139594484) | > avg_loss_0: 2.6613026001084026 (+0.04570443193677498) | > avg_grad_norm_0: tensor(328.2353, device='cuda:0') (tensor(-57.9660, device='cuda:0')) | > avg_loss_gen: 2.1845126874010337 (+0.05522908123446202) | > avg_loss_kl: 2.002337737822197 (-0.05742104792259051) | > avg_loss_feat: 3.9977842720461565 (+0.12478066833925938) | > avg_loss_mel: 16.773831004827795 (-0.29232281698307006) | > avg_loss_duration: 1.5320794045085637 (-0.012055883944874113) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.49054513850682 (-0.1817899086106003) | > avg_grad_norm_1: tensor(7098.0762, device='cuda:0') (tensor(-5366.0469, device='cuda:0'))  > EPOCH: 1422/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:51:28)   --> TIME: 2025-02-25 22:51:37 -- STEP: 15/72 -- GLOBAL_STEP: 1102400 | > loss_disc: 2.715514659881592 (2.7434810161590577) | > loss_disc_real_0: 0.14895585179328918 (0.25076513836781184) | > loss_disc_real_1: 0.25583916902542114 (0.22570393979549408) | > loss_disc_real_2: 0.2541872262954712 (0.23673667907714843) | > loss_disc_real_3: 0.2468206286430359 (0.2443583736817042) | > loss_disc_real_4: 0.2583253085613251 (0.24968080818653107) | > loss_disc_real_5: 0.2547302544116974 (0.24927332897981008) | > loss_0: 2.715514659881592 (2.7434810161590577) | > grad_norm_0: tensor(278.0681, device='cuda:0') (tensor(359.1277, device='cuda:0')) | > loss_gen: 2.1890451908111572 (2.1593208074569703) | > loss_kl: 3.840792179107666 (3.028148953119914) | > loss_feat: 4.163409233093262 (3.727620315551758) | > loss_mel: 22.469379425048828 (19.62419891357422) | > loss_duration: 1.6313883066177368 (1.5556953112284342) | > amp_scaler: 8.0 (8.0) | > loss_1: 34.29401779174805 (30.094984817504884) | > grad_norm_1: tensor(1325.9321, device='cuda:0') (tensor(5561.3291, device='cuda:0')) | > current_lr_0: 0.00016742847685915897 | > current_lr_1: 0.00016742847685915897 | > step_time: 0.4207 (0.43307695388793943) | > loader_time: 0.0057 (0.006724723180135091)  --> TIME: 2025-02-25 22:52:00 -- STEP: 65/72 -- GLOBAL_STEP: 1102450 | > loss_disc: 3.0444538593292236 (2.647236119783841) | > loss_disc_real_0: 0.09095144271850586 (0.19045571799461652) | > loss_disc_real_1: 0.24388229846954346 (0.24384483717955074) | > loss_disc_real_2: 0.26819032430648804 (0.23503582844367393) | > loss_disc_real_3: 0.23394206166267395 (0.23196844940002148) | > loss_disc_real_4: 0.2002333551645279 (0.23725371314929083) | > loss_disc_real_5: 0.1995377391576767 (0.24447474319201254) | > loss_0: 3.0444538593292236 (2.647236119783841) | > grad_norm_0: tensor(186.1346, device='cuda:0') (tensor(301.6429, device='cuda:0')) | > loss_gen: 1.6023932695388794 (2.26379094674037) | > loss_kl: 2.5541093349456787 (2.5788229355445265) | > loss_feat: 3.5714027881622314 (3.9924642856304486) | > loss_mel: 17.304462432861328 (18.390358660771298) | > loss_duration: 1.6278996467590332 (1.5748723286848803) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.660266876220703 (28.80030925457294) | > grad_norm_1: tensor(5444.2256, device='cuda:0') (tensor(4911.3198, device='cuda:0')) | > current_lr_0: 0.00016742847685915897 | > current_lr_1: 0.00016742847685915897 | > step_time: 0.5057 (0.4470713908855732) | > loader_time: 0.0118 (0.008314569179828351) --> EVAL PERFORMANCE | > avg_loader_time: 0.008586782804677184 (-0.0001792806974599062) | > avg_step_time: 0.4521485315242284 (-0.005466716390260473) | > avg_loss_disc: 2.6565379693474562 (-0.004764630760946353) | > avg_loss_disc_real_0: 0.18353888518373726 (+0.04420306431260748) | > avg_loss_disc_real_1: 0.2401842025803848 (-0.0011434498387323244) | > avg_loss_disc_real_2: 0.23482082679238117 (+0.009355737080036758) | > avg_loss_disc_real_3: 0.2329606336187309 (-0.004983549806433646) | > avg_loss_disc_real_4: 0.2344957474671619 (-0.00829970605776345) | > avg_loss_disc_real_5: 0.24211862032682127 (+0.0056103027202714095) | > avg_loss_0: 2.6565379693474562 (-0.004764630760946353) | > avg_grad_norm_0: tensor(294.6418, device='cuda:0') (tensor(-33.5935, device='cuda:0')) | > avg_loss_gen: 2.23404188223288 (+0.04952919483184637) | > avg_loss_kl: 2.5636591071813872 (+0.5613213693591903) | > avg_loss_feat: 3.9746671931844384 (-0.0231170788617181) | > avg_loss_mel: 18.360482484522 (+1.5866514796942042) | > avg_loss_duration: 1.5789142192249568 (+0.04683481471639306) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.71176499380192 (+2.2212198552951) | > avg_grad_norm_1: tensor(5150.1777, device='cuda:0') (tensor(-1947.8984, device='cuda:0'))  > EPOCH: 1423/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:52:05)   --> TIME: 2025-02-25 22:52:26 -- STEP: 43/72 -- GLOBAL_STEP: 1102500 | > loss_disc: 2.8029375076293945 (2.6934543875760806) | > loss_disc_real_0: 0.3437073230743408 (0.16640632807515388) | > loss_disc_real_1: 0.22601144015789032 (0.22327476393344792) | > loss_disc_real_2: 0.22707921266555786 (0.2449909275354341) | > loss_disc_real_3: 0.19984643161296844 (0.2500945069762163) | > loss_disc_real_4: 0.2162744402885437 (0.25134496252204086) | > loss_disc_real_5: 0.24712345004081726 (0.24508334037869475) | > loss_0: 2.8029375076293945 (2.6934543875760806) | > grad_norm_0: tensor(27.2836, device='cuda:0') (tensor(258.1871, device='cuda:0')) | > loss_gen: 1.8244377374649048 (2.132490363231925) | > loss_kl: 2.12150239944458 (2.08568719930427) | > loss_feat: 3.4059436321258545 (3.9688720536786457) | > loss_mel: 16.456689834594727 (17.11986537312353) | > loss_duration: 1.5397448539733887 (1.5169584501621336) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.348318099975586 (26.823873253755792) | > grad_norm_1: tensor(295.6537, device='cuda:0') (tensor(6104.7139, device='cuda:0')) | > current_lr_0: 0.00016740754829955157 | > current_lr_1: 0.00016740754829955157 | > step_time: 0.4521 (0.4355486880901248) | > loader_time: 0.0095 (0.007474988005882086) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596829965081014 (+1.0047160403829983e-05) | > avg_step_time: 0.45080531482965175 (-0.0013432166945766544) | > avg_loss_disc: 2.6952739937204706 (+0.03873602437301438) | > avg_loss_disc_real_0: 0.16818887331116367 (-0.015350011872573593) | > avg_loss_disc_real_1: 0.22866586690217677 (-0.011518335678208036) | > avg_loss_disc_real_2: 0.24123784207122426 (+0.006417015278843086) | > avg_loss_disc_real_3: 0.24481157538756518 (+0.011850941768834278) | > avg_loss_disc_real_4: 0.24788350538468687 (+0.013387757917524973) | > avg_loss_disc_real_5: 0.24193225066426774 (-0.0001863696625535316) | > avg_loss_0: 2.6952739937204706 (+0.03873602437301438) | > avg_grad_norm_0: tensor(194.1983, device='cuda:0') (tensor(-100.4435, device='cuda:0')) | > avg_loss_gen: 2.0962638049058513 (-0.13777807732702874) | > avg_loss_kl: 2.2082657545385227 (-0.35539335264286454) | > avg_loss_feat: 3.8947018905424735 (-0.07996530264196489) | > avg_loss_mel: 17.518412872099553 (-0.8420696124224456) | > avg_loss_duration: 1.5534997977001566 (-0.025414421524800224) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.271144114749532 (-1.4406208790523891) | > avg_grad_norm_1: tensor(4428.4692, device='cuda:0') (tensor(-721.7085, device='cuda:0'))  > EPOCH: 1424/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:52:41)   --> TIME: 2025-02-25 22:52:53 -- STEP: 21/72 -- GLOBAL_STEP: 1102550 | > loss_disc: 2.524467945098877 (2.4851468404134116) | > loss_disc_real_0: 0.1333475112915039 (0.08671792756233897) | > loss_disc_real_1: 0.28799620270729065 (0.22551584385690235) | > loss_disc_real_2: 0.21653962135314941 (0.22528437205723353) | > loss_disc_real_3: 0.23209428787231445 (0.22674596522535598) | > loss_disc_real_4: 0.24621790647506714 (0.24492586226690383) | > loss_disc_real_5: 0.24703307449817657 (0.23652830294200353) | > loss_0: 2.524467945098877 (2.4851468404134116) | > grad_norm_0: tensor(57.1430, device='cuda:0') (tensor(148.7803, device='cuda:0')) | > loss_gen: 2.286346197128296 (2.303154956726801) | > loss_kl: 2.0272397994995117 (2.086303665524438) | > loss_feat: 4.315187931060791 (4.315225941794259) | > loss_mel: 17.531356811523438 (17.7359135945638) | > loss_duration: 1.5229572057724 (1.5292586769376482) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.683088302612305 (27.969856988816034) | > grad_norm_1: tensor(9708.8096, device='cuda:0') (tensor(5760.1367, device='cuda:0')) | > current_lr_0: 0.0001673866223560141 | > current_lr_1: 0.0001673866223560141 | > step_time: 0.4277 (0.4322390102204822) | > loader_time: 0.0061 (0.006692647933959961)  --> TIME: 2025-02-25 22:53:16 -- STEP: 71/72 -- GLOBAL_STEP: 1102600 | > loss_disc: 2.937631607055664 (2.562825340620229) | > loss_disc_real_0: 0.06411996483802795 (0.10656264177720312) | > loss_disc_real_1: 0.20742079615592957 (0.2151867111384029) | > loss_disc_real_2: 0.2170354574918747 (0.2318300271118191) | > loss_disc_real_3: 0.20886299014091492 (0.22236891090869904) | > loss_disc_real_4: 0.2156670242547989 (0.24709880142144755) | > loss_disc_real_5: 0.24741719663143158 (0.2398528989771722) | > loss_0: 2.937631607055664 (2.562825340620229) | > grad_norm_0: tensor(92.3161, device='cuda:0') (tensor(225.8118, device='cuda:0')) | > loss_gen: 1.667628526687622 (2.184621231656679) | > loss_kl: 2.0046281814575195 (2.055115409300361) | > loss_feat: 3.1155271530151367 (4.064084526518701) | > loss_mel: 17.19561004638672 (17.317217276129924) | > loss_duration: 1.5521140098571777 (1.5489378160154317) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.53550910949707 (27.16997635532433) | > grad_norm_1: tensor(4164.0015, device='cuda:0') (tensor(6735.3247, device='cuda:0')) | > current_lr_0: 0.0001673866223560141 | > current_lr_1: 0.0001673866223560141 | > step_time: 0.4952 (0.4514672084593437) | > loader_time: 0.0079 (0.008516667594372385) --> EVAL PERFORMANCE | > avg_loader_time: 0.008516667594372385 (-8.016237070862867e-05) | > avg_step_time: 0.4514672084593437 (+0.0006618936296919342) | > avg_loss_disc: 2.562825340620229 (-0.13244865310024156) | > avg_loss_disc_real_0: 0.10656264177720312 (-0.061626231533960546) | > avg_loss_disc_real_1: 0.2151867111384029 (-0.01347915576377387) | > avg_loss_disc_real_2: 0.2318300271118191 (-0.009407814959405153) | > avg_loss_disc_real_3: 0.22236891090869904 (-0.022442664478866142) | > avg_loss_disc_real_4: 0.24709880142144755 (-0.0007847039632393193) | > avg_loss_disc_real_5: 0.2398528989771722 (-0.0020793516870955264) | > avg_loss_0: 2.562825340620229 (-0.13244865310024156) | > avg_grad_norm_0: tensor(225.8118, device='cuda:0') (+tensor(31.6135, device='cuda:0')) | > avg_loss_gen: 2.184621231656679 (+0.0883574267508278) | > avg_loss_kl: 2.055115409300361 (-0.15315034523816173) | > avg_loss_feat: 4.064084526518701 (+0.16938263597622738) | > avg_loss_mel: 17.317217276129924 (-0.20119559596962944) | > avg_loss_duration: 1.5489378160154317 (-0.004561981684724881) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.16997635532433 (-0.10116775942520206) | > avg_grad_norm_1: tensor(6735.3247, device='cuda:0') (+tensor(2306.8555, device='cuda:0'))  > EPOCH: 1425/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:53:18)   --> TIME: 2025-02-25 22:53:43 -- STEP: 49/72 -- GLOBAL_STEP: 1102650 | > loss_disc: 2.783240795135498 (2.6730405505822628) | > loss_disc_real_0: 0.14019961655139923 (0.20478599989900784) | > loss_disc_real_1: 0.19541993737220764 (0.22020630120318763) | > loss_disc_real_2: 0.3044445216655731 (0.23349815819944655) | > loss_disc_real_3: 0.2404799461364746 (0.23089804545957215) | > loss_disc_real_4: 0.3298669755458832 (0.23993266267435892) | > loss_disc_real_5: 0.2747887372970581 (0.24748968378621705) | > loss_0: 2.783240795135498 (2.6730405505822628) | > grad_norm_0: tensor(380.1436, device='cuda:0') (tensor(255.7804, device='cuda:0')) | > loss_gen: 2.098010778427124 (2.15467398993823) | > loss_kl: 1.9562169313430786 (1.9276067383435307) | > loss_feat: 4.393814563751221 (3.8254937687698676) | > loss_mel: 16.148536682128906 (16.758064114317598) | > loss_duration: 1.5883653163909912 (1.51104850428445) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.18494415283203 (26.17688696725028) | > grad_norm_1: tensor(2683.8350, device='cuda:0') (tensor(4853.9624, device='cuda:0')) | > current_lr_0: 0.0001673656990282196 | > current_lr_1: 0.0001673656990282196 | > step_time: 0.4335 (0.4493928101598) | > loader_time: 0.0095 (0.007602521351405552) --> EVAL PERFORMANCE | > avg_loader_time: 0.008556292090617435 (+3.962449624505052e-05) | > avg_step_time: 0.45727864789291167 (+0.005811439433567989) | > avg_loss_disc: 2.6538084560716655 (+0.09098311545143645) | > avg_loss_disc_real_0: 0.1677349805569565 (+0.061172338779753366) | > avg_loss_disc_real_1: 0.21954646271089434 (+0.004359751572491438) | > avg_loss_disc_real_2: 0.24630820037613452 (+0.014478173264315414) | > avg_loss_disc_real_3: 0.22900975736933696 (+0.0066408464606379225) | > avg_loss_disc_real_4: 0.23931890852014784 (-0.007779892901299712) | > avg_loss_disc_real_5: 0.2497036207309911 (+0.009850721753818875) | > avg_loss_0: 2.6538084560716655 (+0.09098311545143645) | > avg_grad_norm_0: tensor(236.2913, device='cuda:0') (+tensor(10.4796, device='cuda:0')) | > avg_loss_gen: 2.1690655023279324 (-0.015555729328746715) | > avg_loss_kl: 1.9878887257105868 (-0.06722668358977413) | > avg_loss_feat: 3.898092256465428 (-0.16599227005327277) | > avg_loss_mel: 16.845454820444886 (-0.47176245568503816) | > avg_loss_duration: 1.5319002816374874 (-0.017037534377944308) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.432401469056035 (-0.7375748862682947) | > avg_grad_norm_1: tensor(5386.2891, device='cuda:0') (tensor(-1349.0356, device='cuda:0'))  > EPOCH: 1426/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:53:55)   --> TIME: 2025-02-25 22:54:09 -- STEP: 27/72 -- GLOBAL_STEP: 1102700 | > loss_disc: 2.631016731262207 (2.7061092058817544) | > loss_disc_real_0: 0.2639395594596863 (0.14543593481734945) | > loss_disc_real_1: 0.1826978325843811 (0.21059359775649178) | > loss_disc_real_2: 0.21685156226158142 (0.22208373414145577) | > loss_disc_real_3: 0.24027805030345917 (0.22258953529375572) | > loss_disc_real_4: 0.23856133222579956 (0.26679538079985865) | > loss_disc_real_5: 0.2592547535896301 (0.24291522359406506) | > loss_0: 2.631016731262207 (2.7061092058817544) | > grad_norm_0: tensor(32.8391, device='cuda:0') (tensor(213.9665, device='cuda:0')) | > loss_gen: 2.5013060569763184 (2.0828831593195596) | > loss_kl: 1.9004007577896118 (1.961364260426274) | > loss_feat: 4.15312385559082 (3.9409370863879167) | > loss_mel: 17.30533790588379 (17.170834435356987) | > loss_duration: 1.481764316558838 (1.495756255255805) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.34193229675293 (26.651775218822337) | > grad_norm_1: tensor(4838.1831, device='cuda:0') (tensor(6714.6279, device='cuda:0')) | > current_lr_0: 0.00016734477831584107 | > current_lr_1: 0.00016734477831584107 | > step_time: 0.4465 (0.43621438520926015) | > loader_time: 0.0081 (0.006927022227534541) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863015483802473 (+7.386274740729497e-05) | > avg_step_time: 0.45266079902648926 (-0.004617848866422414) | > avg_loss_disc: 2.6907440541495737 (+0.03693559807790825) | > avg_loss_disc_real_0: 0.18931313347018938 (+0.021578152913232895) | > avg_loss_disc_real_1: 0.21551054023521046 (-0.0040359224756838785) | > avg_loss_disc_real_2: 0.23074059776017364 (-0.01556760261596088) | > avg_loss_disc_real_3: 0.23127684165054643 (+0.002267084281209475) | > avg_loss_disc_real_4: 0.2532931564139648 (+0.013974247893816988) | > avg_loss_disc_real_5: 0.24601063468086887 (-0.003692986050122221) | > avg_loss_0: 2.6907440541495737 (+0.03693559807790825) | > avg_grad_norm_0: tensor(235.4545, device='cuda:0') (tensor(-0.8368, device='cuda:0')) | > avg_loss_gen: 2.1054668443303703 (-0.06359865799756204) | > avg_loss_kl: 2.052824171496109 (+0.06493544578552224) | > avg_loss_feat: 3.759352059431479 (-0.13874019703394902) | > avg_loss_mel: 17.276961407191315 (+0.43150658674642983) | > avg_loss_duration: 1.5863504275469713 (+0.05445014590948394) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.780954817651022 (+0.34855334859498655) | > avg_grad_norm_1: tensor(5817.0503, device='cuda:0') (+tensor(430.7612, device='cuda:0'))  > EPOCH: 1427/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:54:32)   --> TIME: 2025-02-25 22:54:36 -- STEP: 5/72 -- GLOBAL_STEP: 1102750 | > loss_disc: 2.3846113681793213 (2.68249626159668) | > loss_disc_real_0: 0.11985836923122406 (0.13326777666807174) | > loss_disc_real_1: 0.22559715807437897 (0.21117625534534454) | > loss_disc_real_2: 0.25316575169563293 (0.2750564396381378) | > loss_disc_real_3: 0.23205867409706116 (0.2561413526535034) | > loss_disc_real_4: 0.22349749505519867 (0.2216633141040802) | > loss_disc_real_5: 0.23549450933933258 (0.22173003256320953) | > loss_0: 2.3846113681793213 (2.68249626159668) | > grad_norm_0: tensor(137.9021, device='cuda:0') (tensor(192.6506, device='cuda:0')) | > loss_gen: 2.416529893875122 (2.005288577079773) | > loss_kl: 1.8311536312103271 (1.8975989103317261) | > loss_feat: 4.0533342361450195 (3.707242727279663) | > loss_mel: 16.568347930908203 (16.978047943115236) | > loss_duration: 1.4935047626495361 (1.5413838863372802) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.362871170043945 (26.129562759399413) | > grad_norm_1: tensor(2533.6367, device='cuda:0') (tensor(4411.0488, device='cuda:0')) | > current_lr_0: 0.0001673238602185516 | > current_lr_1: 0.0001673238602185516 | > step_time: 0.432 (0.4316103935241699) | > loader_time: 0.0069 (0.006386661529541015)  --> TIME: 2025-02-25 22:54:59 -- STEP: 55/72 -- GLOBAL_STEP: 1102800 | > loss_disc: 2.8488802909851074 (2.691861668500033) | > loss_disc_real_0: 0.07519599795341492 (0.14806790521199065) | > loss_disc_real_1: 0.19770225882530212 (0.24693042066964238) | > loss_disc_real_2: 0.14326323568820953 (0.25438955859704454) | > loss_disc_real_3: 0.14044244587421417 (0.23958705419843848) | > loss_disc_real_4: 0.24086208641529083 (0.24250692400065335) | > loss_disc_real_5: 0.24060317873954773 (0.24174824059009553) | > loss_0: 2.8488802909851074 (2.691861668500033) | > grad_norm_0: tensor(256.1393, device='cuda:0') (tensor(246.7269, device='cuda:0')) | > loss_gen: 1.5226432085037231 (2.1010171283375128) | > loss_kl: 2.1164190769195557 (2.0947755011645226) | > loss_feat: 3.0875396728515625 (3.822978297146884) | > loss_mel: 16.931812286376953 (17.228765383633693) | > loss_duration: 1.5981395244598389 (1.5481414621526546) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.256553649902344 (26.795678121393376) | > grad_norm_1: tensor(5332.2681, device='cuda:0') (tensor(5837.1738, device='cuda:0')) | > current_lr_0: 0.0001673238602185516 | > current_lr_1: 0.0001673238602185516 | > step_time: 0.4518 (0.440004639192061) | > loader_time: 0.0092 (0.008006199923428622) --> EVAL PERFORMANCE | > avg_loader_time: 0.008716741078336475 (+8.658624031174407e-05) | > avg_step_time: 0.45091567240970243 (-0.0017451266167868296) | > avg_loss_disc: 2.7112918739587486 (+0.02054781980917486) | > avg_loss_disc_real_0: 0.17077838750162605 (-0.018534745968563332) | > avg_loss_disc_real_1: 0.2396222900756648 (+0.024111749840454333) | > avg_loss_disc_real_2: 0.24617860602660918 (+0.015438008266435543) | > avg_loss_disc_real_3: 0.23672953394936844 (+0.005452692298822004) | > avg_loss_disc_real_4: 0.24206954933388133 (-0.011223607080083492) | > avg_loss_disc_real_5: 0.24083924377468272 (-0.005171390906186152) | > avg_loss_0: 2.7112918739587486 (+0.02054781980917486) | > avg_grad_norm_0: tensor(241.7234, device='cuda:0') (+tensor(6.2688, device='cuda:0')) | > avg_loss_gen: 2.0655850558213786 (-0.03988178850899171) | > avg_loss_kl: 2.112454456342778 (+0.05963028484666877) | > avg_loss_feat: 3.7482885508470134 (-0.011063508584465698) | > avg_loss_mel: 17.19862301248899 (-0.07833839470232462) | > avg_loss_duration: 1.557075794313995 (-0.0292746332329763) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.682027118306763 (-0.09892769934425871) | > avg_grad_norm_1: tensor(5626.8291, device='cuda:0') (tensor(-190.2212, device='cuda:0'))  > EPOCH: 1428/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:55:08)   --> TIME: 2025-02-25 22:55:26 -- STEP: 33/72 -- GLOBAL_STEP: 1102850 | > loss_disc: 2.5501134395599365 (2.645824417923436) | > loss_disc_real_0: 0.07345984876155853 (0.1040652867293719) | > loss_disc_real_1: 0.22411754727363586 (0.2273981530557979) | > loss_disc_real_2: 0.2288515567779541 (0.24548297036777844) | > loss_disc_real_3: 0.2465227097272873 (0.2438941327008334) | > loss_disc_real_4: 0.22846104204654694 (0.23652367203524619) | > loss_disc_real_5: 0.23717337846755981 (0.24625450688781161) | > loss_0: 2.5501134395599365 (2.645824417923436) | > grad_norm_0: tensor(320.2993, device='cuda:0') (tensor(250.0313, device='cuda:0')) | > loss_gen: 2.2078237533569336 (2.0898812973138057) | > loss_kl: 1.8953335285186768 (2.011457150632685) | > loss_feat: 3.545145273208618 (4.026206861842763) | > loss_mel: 16.575014114379883 (16.925233754244722) | > loss_duration: 1.4936938285827637 (1.5017233870246194) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.717010498046875 (26.554502140391957) | > grad_norm_1: tensor(6869.3242, device='cuda:0') (tensor(5394.9463, device='cuda:0')) | > current_lr_0: 0.00016730294473602428 | > current_lr_1: 0.00016730294473602428 | > step_time: 0.4458 (0.4517249483050722) | > loader_time: 0.0087 (0.007136222087975704) --> EVAL PERFORMANCE | > avg_loader_time: 0.008713305835992518 (-3.4352423439569513e-06) | > avg_step_time: 0.4624575359720579 (+0.011541863562355481) | > avg_loss_disc: 2.639923760588739 (-0.07136811337000948) | > avg_loss_disc_real_0: 0.11397471961001275 (-0.0568036678916133) | > avg_loss_disc_real_1: 0.2279615422159853 (-0.011660747859679493) | > avg_loss_disc_real_2: 0.24214253375228023 (-0.00403607227432895) | > avg_loss_disc_real_3: 0.23991529643535614 (+0.003185762485987703) | > avg_loss_disc_real_4: 0.2448064882150838 (+0.0027369388812024664) | > avg_loss_disc_real_5: 0.24595636842956006 (+0.0051171246548773475) | > avg_loss_0: 2.639923760588739 (-0.07136811337000948) | > avg_grad_norm_0: tensor(247.9197, device='cuda:0') (+tensor(6.1964, device='cuda:0')) | > avg_loss_gen: 2.0916329504738385 (+0.026047894652459824) | > avg_loss_kl: 2.4171342228499935 (+0.3046797665072156) | > avg_loss_feat: 3.9255229217905394 (+0.177234370943526) | > avg_loss_mel: 17.893007869451825 (+0.6943848569628344) | > avg_loss_duration: 1.5733791354676367 (+0.016303341153641737) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.900676942207443 (+1.2186498239006802) | > avg_grad_norm_1: tensor(6723.4922, device='cuda:0') (+tensor(1096.6631, device='cuda:0'))  > EPOCH: 1429/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:55:46)   --> TIME: 2025-02-25 22:55:53 -- STEP: 11/72 -- GLOBAL_STEP: 1102900 | > loss_disc: 2.600795030593872 (2.4791073365644976) | > loss_disc_real_0: 0.050351522862911224 (0.07694388790564104) | > loss_disc_real_1: 0.22034099698066711 (0.23417832499200647) | > loss_disc_real_2: 0.22536210715770721 (0.22642510452053763) | > loss_disc_real_3: 0.20325790345668793 (0.227793581106446) | > loss_disc_real_4: 0.24954892694950104 (0.2270055508071726) | > loss_disc_real_5: 0.2682695686817169 (0.24207929183136334) | > loss_0: 2.600795030593872 (2.4791073365644976) | > grad_norm_0: tensor(102.6279, device='cuda:0') (tensor(109.4940, device='cuda:0')) | > loss_gen: 2.0018508434295654 (2.3664416074752808) | > loss_kl: 2.7989799976348877 (2.5503513379530474) | > loss_feat: 3.775249719619751 (4.582786711779508) | > loss_mel: 19.379940032958984 (19.204281200062145) | > loss_duration: 1.5476174354553223 (1.5857748876918445) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.503637313842773 (30.289635918357156) | > grad_norm_1: tensor(6580.6855, device='cuda:0') (tensor(3327.3374, device='cuda:0')) | > current_lr_0: 0.00016728203186793227 | > current_lr_1: 0.00016728203186793227 | > step_time: 0.4323 (0.4367519942196933) | > loader_time: 0.0059 (0.0065679766915061255)  --> TIME: 2025-02-25 22:56:16 -- STEP: 61/72 -- GLOBAL_STEP: 1102950 | > loss_disc: 2.604654550552368 (2.627283756850196) | > loss_disc_real_0: 0.10381990671157837 (0.147528133981052) | > loss_disc_real_1: 0.2846295237541199 (0.22364310065253837) | > loss_disc_real_2: 0.4038122594356537 (0.24222188821581545) | > loss_disc_real_3: 0.23564916849136353 (0.23344502170554926) | > loss_disc_real_4: 0.2445731908082962 (0.24383141246975445) | > loss_disc_real_5: 0.2533051371574402 (0.24816719821242036) | > loss_0: 2.604654550552368 (2.627283756850196) | > grad_norm_0: tensor(415.8979, device='cuda:0') (tensor(247.0279, device='cuda:0')) | > loss_gen: 2.3850650787353516 (2.184641093504234) | > loss_kl: 2.043077230453491 (2.264918296063532) | > loss_feat: 4.314904689788818 (3.9527547554891624) | > loss_mel: 16.18180274963379 (17.694964846626657) | > loss_duration: 1.592313528060913 (1.5599996398706906) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.517162322998047 (27.657278842613344) | > grad_norm_1: tensor(11449.9395, device='cuda:0') (tensor(6163.5962, device='cuda:0')) | > current_lr_0: 0.00016728203186793227 | > current_lr_1: 0.00016728203186793227 | > step_time: 0.4567 (0.4531401532595275) | > loader_time: 0.0108 (0.008275383808573738) --> EVAL PERFORMANCE | > avg_loader_time: 0.00874281601167061 (+2.951017567809182e-05) | > avg_step_time: 0.45909593810497873 (-0.00336159786707918) | > avg_loss_disc: 2.6207342248567396 (-0.019189535731999552) | > avg_loss_disc_real_0: 0.14554753338157292 (+0.03157281377156017) | > avg_loss_disc_real_1: 0.22214557440348073 (-0.005815967812504569) | > avg_loss_disc_real_2: 0.23848040989587005 (-0.003662123856410182) | > avg_loss_disc_real_3: 0.2311731644079719 (-0.008742132027384247) | > avg_loss_disc_real_4: 0.24354022573417342 (-0.001266262480910385) | > avg_loss_disc_real_5: 0.2477772013402321 (+0.0018208329106720322) | > avg_loss_0: 2.6207342248567396 (-0.019189535731999552) | > avg_grad_norm_0: tensor(229.1640, device='cuda:0') (tensor(-18.7558, device='cuda:0')) | > avg_loss_gen: 2.1907046126647742 (+0.09907166219093577) | > avg_loss_kl: 2.253681710068608 (-0.16345251278138528) | > avg_loss_feat: 3.9410838442788996 (+0.015560922488360163) | > avg_loss_mel: 17.64251683463513 (-0.25049103481669377) | > avg_loss_duration: 1.5664094602557974 (-0.006969675211839332) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.59439664491465 (-0.30628029729279405) | > avg_grad_norm_1: tensor(6139.3345, device='cuda:0') (tensor(-584.1577, device='cuda:0'))  > EPOCH: 1430/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:56:23)   --> TIME: 2025-02-25 22:56:43 -- STEP: 39/72 -- GLOBAL_STEP: 1103000 | > loss_disc: 2.404860258102417 (2.607930354582958) | > loss_disc_real_0: 0.04600738734006882 (0.07482794371361916) | > loss_disc_real_1: 0.2536713480949402 (0.20923856321053627) | > loss_disc_real_2: 0.3214530944824219 (0.2679246683151294) | > loss_disc_real_3: 0.26119306683540344 (0.24656986578916892) | > loss_disc_real_4: 0.2770954370498657 (0.25896206498146057) | > loss_disc_real_5: 0.2541433870792389 (0.24840867634002978) | > loss_0: 2.404860258102417 (2.607930354582958) | > grad_norm_0: tensor(328.5005, device='cuda:0') (tensor(312.5458, device='cuda:0')) | > loss_gen: 2.4067933559417725 (2.1327668306155085) | > loss_kl: 1.9186159372329712 (1.9683880316905487) | > loss_feat: 4.074869632720947 (4.003316903725647) | > loss_mel: 17.126161575317383 (16.741555434006916) | > loss_duration: 1.5358946323394775 (1.5083118829971704) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.062335968017578 (26.354339061639248) | > grad_norm_1: tensor(11789.1846, device='cuda:0') (tensor(9251.3164, device='cuda:0')) | > current_lr_0: 0.00016726112161394877 | > current_lr_1: 0.00016726112161394877 | > step_time: 0.4476 (0.4373983725523337) | > loader_time: 0.0083 (0.007347485958001552) --> EVAL PERFORMANCE | > avg_loader_time: 0.00861398938676001 (-0.00012882662491059925) | > avg_step_time: 0.4528034740770367 (-0.006292464027942035) | > avg_loss_disc: 2.569854356873204 (-0.05087986798353539) | > avg_loss_disc_real_0: 0.0776081828426727 (-0.06793935053890023) | > avg_loss_disc_real_1: 0.2133853876254928 (-0.008760186777987927) | > avg_loss_disc_real_2: 0.25316998677354463 (+0.014689576877674582) | > avg_loss_disc_real_3: 0.24160214398108737 (+0.010428979573115477) | > avg_loss_disc_real_4: 0.24906206529744915 (+0.005521839563275732) | > avg_loss_disc_real_5: 0.24644339399438508 (-0.00133380734584701) | > avg_loss_0: 2.569854356873204 (-0.05087986798353539) | > avg_grad_norm_0: tensor(323.5656, device='cuda:0') (+tensor(94.4017, device='cuda:0')) | > avg_loss_gen: 2.150672625487958 (-0.04003198717681622) | > avg_loss_kl: 2.000443844728067 (-0.2532378653405414) | > avg_loss_feat: 4.028101205825805 (+0.0870173615469052) | > avg_loss_mel: 16.68062310823253 (-0.9618937264026002) | > avg_loss_duration: 1.536167670303667 (-0.030241789952130338) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.396008558676275 (-1.1983880862383742) | > avg_grad_norm_1: tensor(9536.2402, device='cuda:0') (+tensor(3396.9058, device='cuda:0'))  > EPOCH: 1431/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:57:00)   --> TIME: 2025-02-25 22:57:09 -- STEP: 17/72 -- GLOBAL_STEP: 1103050 | > loss_disc: 2.4176743030548096 (2.504718542098999) | > loss_disc_real_0: 0.07469610869884491 (0.07107571906903211) | > loss_disc_real_1: 0.2076134830713272 (0.20476976650602677) | > loss_disc_real_2: 0.17932933568954468 (0.20430080154362848) | > loss_disc_real_3: 0.21799905598163605 (0.22292315083391526) | > loss_disc_real_4: 0.21523547172546387 (0.22386890562141643) | > loss_disc_real_5: 0.24048706889152527 (0.24292470076504877) | > loss_0: 2.4176743030548096 (2.504718542098999) | > grad_norm_0: tensor(130.3131, device='cuda:0') (tensor(275.7141, device='cuda:0')) | > loss_gen: 2.2980453968048096 (2.1776771194794597) | > loss_kl: 1.964320421218872 (1.891331350102144) | > loss_feat: 4.0961737632751465 (4.279325765721938) | > loss_mel: 16.823684692382812 (16.852800930247586) | > loss_duration: 1.5334408283233643 (1.4924101549036362) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.715665817260742 (26.693545060999252) | > grad_norm_1: tensor(7459.0029, device='cuda:0') (tensor(11031.0635, device='cuda:0')) | > current_lr_0: 0.000167240213973747 | > current_lr_1: 0.000167240213973747 | > step_time: 0.4272 (0.42697252946741443) | > loader_time: 0.007 (0.006671793320599724)  --> TIME: 2025-02-25 22:57:34 -- STEP: 67/72 -- GLOBAL_STEP: 1103100 | > loss_disc: 2.5954012870788574 (2.5049069603877294) | > loss_disc_real_0: 0.0771060660481453 (0.08204690311382067) | > loss_disc_real_1: 0.20058777928352356 (0.2157848613030875) | > loss_disc_real_2: 0.20653927326202393 (0.2092225685048459) | > loss_disc_real_3: 0.22744996845722198 (0.22602679391405475) | > loss_disc_real_4: 0.23260371387004852 (0.22678108713520106) | > loss_disc_real_5: 0.24075239896774292 (0.24747128584491673) | > loss_0: 2.5954012870788574 (2.5049069603877294) | > grad_norm_0: tensor(453.3544, device='cuda:0') (tensor(317.8036, device='cuda:0')) | > loss_gen: 2.1681127548217773 (2.2035011515688536) | > loss_kl: 1.9942914247512817 (1.9601240727438856) | > loss_feat: 4.608855247497559 (4.231073884821652) | > loss_mel: 17.67116355895996 (16.671337796680966) | > loss_duration: 1.6015112400054932 (1.5273946185610188) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.043935775756836 (26.59343147277832) | > grad_norm_1: tensor(9868.7920, device='cuda:0') (tensor(10759.3457, device='cuda:0')) | > current_lr_0: 0.000167240213973747 | > current_lr_1: 0.000167240213973747 | > step_time: 0.5204 (0.4575854557663647) | > loader_time: 0.012 (0.008437192262108647) --> EVAL PERFORMANCE | > avg_loader_time: 0.008630151480016574 (+1.616209325656405e-05) | > avg_step_time: 0.46175810316918603 (+0.008954629092149335) | > avg_loss_disc: 2.498683059719248 (-0.0711712971539562) | > avg_loss_disc_real_0: 0.082591766412829 (+0.004983583570156311) | > avg_loss_disc_real_1: 0.2152900945552638 (+0.0019047069297709884) | > avg_loss_disc_real_2: 0.20990363560931782 (-0.04326635116422681) | > avg_loss_disc_real_3: 0.22579864486002585 (-0.01580349912106152) | > avg_loss_disc_real_4: 0.22705812731259306 (-0.022003937984856087) | > avg_loss_disc_real_5: 0.24707385016159272 (+0.0006304561672076381) | > avg_loss_0: 2.498683059719248 (-0.0711712971539562) | > avg_grad_norm_0: tensor(304.7115, device='cuda:0') (tensor(-18.8541, device='cuda:0')) | > avg_loss_gen: 2.217455299807266 (+0.06678267431930784) | > avg_loss_kl: 1.967135041532382 (-0.033308803195684744) | > avg_loss_feat: 4.229442791200022 (+0.20134158537421687) | > avg_loss_mel: 16.643114345174443 (-0.03750876305808859) | > avg_loss_duration: 1.5309751369583775 (-0.005192533345289574) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.588122569339376 (+0.19211401066310074) | > avg_grad_norm_1: tensor(10578.6123, device='cuda:0') (+tensor(1042.3721, device='cuda:0'))  > EPOCH: 1432/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:57:38)   --> TIME: 2025-02-25 22:58:00 -- STEP: 45/72 -- GLOBAL_STEP: 1103150 | > loss_disc: 2.4203357696533203 (2.5562408023410383) | > loss_disc_real_0: 0.12771488726139069 (0.08388382519284883) | > loss_disc_real_1: 0.23822373151779175 (0.22323075400458442) | > loss_disc_real_2: 0.22310972213745117 (0.22277330822414823) | > loss_disc_real_3: 0.25666648149490356 (0.23749447796079848) | > loss_disc_real_4: 0.26369982957839966 (0.22471853726440005) | > loss_disc_real_5: 0.26121485233306885 (0.24356379906336467) | > loss_0: 2.4203357696533203 (2.5562408023410383) | > grad_norm_0: tensor(513.2798, device='cuda:0') (tensor(366.6441, device='cuda:0')) | > loss_gen: 2.46768856048584 (2.206762231720819) | > loss_kl: 2.032332420349121 (1.9093947304619683) | > loss_feat: 3.938753128051758 (4.269975736406114) | > loss_mel: 16.92363166809082 (16.55812907748752) | > loss_duration: 1.5343945026397705 (1.5001129574245877) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.89680290222168 (26.444374720255535) | > grad_norm_1: tensor(9987.8906, device='cuda:0') (tensor(11295.4756, device='cuda:0')) | > current_lr_0: 0.00016721930894700028 | > current_lr_1: 0.00016721930894700028 | > step_time: 0.4635 (0.445080746544732) | > loader_time: 0.0087 (0.007540999518500434) --> EVAL PERFORMANCE | > avg_loader_time: 0.008612914824149978 (-1.7236655866595743e-05) | > avg_step_time: 0.457803531431816 (-0.003954571737370016) | > avg_loss_disc: 2.5440587225094657 (+0.04537566279021776) | > avg_loss_disc_real_0: 0.08913708929444703 (+0.006545322881618021) | > avg_loss_disc_real_1: 0.220944044455676 (+0.0056539499004122085) | > avg_loss_disc_real_2: 0.223359475043458 (+0.013455839434140165) | > avg_loss_disc_real_3: 0.23432745270326105 (+0.008528807843235203) | > avg_loss_disc_real_4: 0.22999856190782197 (+0.0029404345952289135) | > avg_loss_disc_real_5: 0.24462304421713654 (-0.0024508059444561803) | > avg_loss_0: 2.5440587225094657 (+0.04537566279021776) | > avg_grad_norm_0: tensor(358.6851, device='cuda:0') (+tensor(53.9735, device='cuda:0')) | > avg_loss_gen: 2.2071999731198173 (-0.010255326687448552) | > avg_loss_kl: 1.9536477193026476 (-0.013487322229734477) | > avg_loss_feat: 4.17649680460003 (-0.05294598659999128) | > avg_loss_mel: 16.538436728464045 (-0.1046776167103971) | > avg_loss_duration: 1.5261304680730257 (-0.004844668885351844) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.401911668374506 (-0.1862109009648698) | > avg_grad_norm_1: tensor(10886.6807, device='cuda:0') (+tensor(308.0684, device='cuda:0'))  > EPOCH: 1433/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:58:15)   --> TIME: 2025-02-25 22:58:27 -- STEP: 23/72 -- GLOBAL_STEP: 1103200 | > loss_disc: 2.474235773086548 (2.4818795141966445) | > loss_disc_real_0: 0.06696732342243195 (0.08107587597940279) | > loss_disc_real_1: 0.25327029824256897 (0.21502118719660718) | > loss_disc_real_2: 0.22499339282512665 (0.228866634161576) | > loss_disc_real_3: 0.2458520233631134 (0.2295928435481113) | > loss_disc_real_4: 0.26379039883613586 (0.23951514000477997) | > loss_disc_real_5: 0.25634267926216125 (0.24304015221803085) | > loss_0: 2.474235773086548 (2.4818795141966445) | > grad_norm_0: tensor(139.2289, device='cuda:0') (tensor(286.9332, device='cuda:0')) | > loss_gen: 2.3076553344726562 (2.257687459821286) | > loss_kl: 1.6051075458526611 (1.8672562474789827) | > loss_feat: 3.691209554672241 (4.197828085526176) | > loss_mel: 16.174985885620117 (16.29242863862411) | > loss_duration: 1.5140063762664795 (1.4788511628689973) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.2929630279541 (26.094051444012187) | > grad_norm_1: tensor(8057.7339, device='cuda:0') (tensor(9075.0596, device='cuda:0')) | > current_lr_0: 0.0001671984065333819 | > current_lr_1: 0.0001671984065333819 | > step_time: 0.4441 (0.44169963961062225) | > loader_time: 0.0065 (0.006800112516983696) --> EVAL PERFORMANCE | > avg_loader_time: 0.008769136079600158 (+0.0001562212554501799) | > avg_step_time: 0.46124195716750455 (+0.003438425735688533) | > avg_loss_disc: 2.5255607013971026 (-0.018498021112363094) | > avg_loss_disc_real_0: 0.09400344891867167 (+0.004866359624224645) | > avg_loss_disc_real_1: 0.2219314046309028 (+0.000987360175226798) | > avg_loss_disc_real_2: 0.23031459077143332 (+0.006955115727975331) | > avg_loss_disc_real_3: 0.22692877159152233 (-0.00739868111173872) | > avg_loss_disc_real_4: 0.23966833424400275 (+0.00966977233618077) | > avg_loss_disc_real_5: 0.24457872228723176 (-4.4321929904778123e-05) | > avg_loss_0: 2.5255607013971026 (-0.018498021112363094) | > avg_grad_norm_0: tensor(291.3321, device='cuda:0') (tensor(-67.3530, device='cuda:0')) | > avg_loss_gen: 2.2494268518098637 (+0.04222687869004638) | > avg_loss_kl: 1.9490575991885764 (-0.004590120114071228) | > avg_loss_feat: 4.211882853172196 (+0.03538604857216576) | > avg_loss_mel: 16.39161410801847 (-0.14682262044557604) | > avg_loss_duration: 1.523697252004919 (-0.0024332160681066473) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.3256785701698 (-0.0762330982047068) | > avg_grad_norm_1: tensor(8419.5088, device='cuda:0') (tensor(-2467.1719, device='cuda:0'))  > EPOCH: 1434/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:58:52)   --> TIME: 2025-02-25 22:58:55 -- STEP: 1/72 -- GLOBAL_STEP: 1103250 | > loss_disc: 2.5978128910064697 (2.5978128910064697) | > loss_disc_real_0: 0.12383938580751419 (0.12383938580751419) | > loss_disc_real_1: 0.2065628319978714 (0.2065628319978714) | > loss_disc_real_2: 0.22321031987667084 (0.22321031987667084) | > loss_disc_real_3: 0.2386939376592636 (0.2386939376592636) | > loss_disc_real_4: 0.23644468188285828 (0.23644468188285828) | > loss_disc_real_5: 0.22768515348434448 (0.22768515348434448) | > loss_0: 2.5978128910064697 (2.5978128910064697) | > grad_norm_0: tensor(405.8098, device='cuda:0') (tensor(405.8098, device='cuda:0')) | > loss_gen: 2.351830244064331 (2.351830244064331) | > loss_kl: 2.0122809410095215 (2.0122809410095215) | > loss_feat: 4.64658260345459 (4.64658260345459) | > loss_mel: 16.420242309570312 (16.420242309570312) | > loss_duration: 1.4902784824371338 (1.4902784824371338) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.921213150024414 (26.921213150024414) | > grad_norm_1: tensor(6673.4224, device='cuda:0') (tensor(6673.4224, device='cuda:0')) | > current_lr_0: 0.0001671775067325652 | > current_lr_1: 0.0001671775067325652 | > step_time: 0.4399 (0.43993544578552246) | > loader_time: 0.0059 (0.005902290344238281)  --> TIME: 2025-02-25 22:59:18 -- STEP: 51/72 -- GLOBAL_STEP: 1103300 | > loss_disc: 2.2857770919799805 (2.5163332153769105) | > loss_disc_real_0: 0.051800284534692764 (0.09981983279188474) | > loss_disc_real_1: 0.21722330152988434 (0.2161320272613974) | > loss_disc_real_2: 0.21048316359519958 (0.22988423266831567) | > loss_disc_real_3: 0.2081633359193802 (0.23256579275224723) | > loss_disc_real_4: 0.24649430811405182 (0.2399962859995225) | > loss_disc_real_5: 0.21840375661849976 (0.24613294911150838) | > loss_0: 2.2857770919799805 (2.5163332153769105) | > grad_norm_0: tensor(145.4986, device='cuda:0') (tensor(324.2971, device='cuda:0')) | > loss_gen: 2.568077325820923 (2.2825027166628375) | > loss_kl: 1.946719765663147 (1.9005670781229056) | > loss_feat: 4.972358703613281 (4.279580434163413) | > loss_mel: 17.040889739990234 (16.511413854711197) | > loss_duration: 1.5458818674087524 (1.5017082060084623) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.07392692565918 (26.475772034888173) | > grad_norm_1: tensor(8991.5723, device='cuda:0') (tensor(8739.3213, device='cuda:0')) | > current_lr_0: 0.0001671775067325652 | > current_lr_1: 0.0001671775067325652 | > step_time: 0.4483 (0.445051394256891) | > loader_time: 0.0092 (0.007987433788823148) --> EVAL PERFORMANCE | > avg_loader_time: 0.008832857642375251 (+6.372156277509311e-05) | > avg_step_time: 0.46384653910784657 (+0.002604581940342021) | > avg_loss_disc: 2.52265262267959 (-0.002908078717512641) | > avg_loss_disc_real_0: 0.09559708886163336 (+0.0015936399429616888) | > avg_loss_disc_real_1: 0.21779444498915068 (-0.004136959641752119) | > avg_loss_disc_real_2: 0.2258822498187213 (-0.004432340952712027) | > avg_loss_disc_real_3: 0.2306458851401235 (+0.0037171135486011586) | > avg_loss_disc_real_4: 0.235476338947323 (-0.00419199529667974) | > avg_loss_disc_real_5: 0.24532288207974232 (+0.0007441597925105514) | > avg_loss_0: 2.52265262267959 (-0.002908078717512641) | > avg_grad_norm_0: tensor(330.9797, device='cuda:0') (+tensor(39.6476, device='cuda:0')) | > avg_loss_gen: 2.264834783446621 (+0.015407931636757244) | > avg_loss_kl: 1.9546485417325732 (+0.005590942543996835) | > avg_loss_feat: 4.255606577429975 (+0.04372372425777904) | > avg_loss_mel: 16.589373266193235 (+0.1977591581747653) | > avg_loss_duration: 1.5207197901228784 (-0.002977461882040622) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.58518283467897 (+0.25950426450917163) | > avg_grad_norm_1: tensor(9111.2148, device='cuda:0') (+tensor(691.7061, device='cuda:0'))  > EPOCH: 1435/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 22:59:30)   --> TIME: 2025-02-25 22:59:45 -- STEP: 29/72 -- GLOBAL_STEP: 1103350 | > loss_disc: 2.42753267288208 (2.5033676953151307) | > loss_disc_real_0: 0.06247081235051155 (0.08695795528333762) | > loss_disc_real_1: 0.2604588270187378 (0.21839824113352546) | > loss_disc_real_2: 0.21153680980205536 (0.22529087395503603) | > loss_disc_real_3: 0.2629826068878174 (0.22939113655994678) | > loss_disc_real_4: 0.24017050862312317 (0.23896806301741763) | > loss_disc_real_5: 0.265388160943985 (0.24573613343567685) | > loss_0: 2.42753267288208 (2.5033676953151307) | > grad_norm_0: tensor(92.6470, device='cuda:0') (tensor(328.9303, device='cuda:0')) | > loss_gen: 2.538148880004883 (2.3053198230677636) | > loss_kl: 1.8468403816223145 (1.8897764970516335) | > loss_feat: 4.63884973526001 (4.436290666974826) | > loss_mel: 17.58073616027832 (16.628217993111438) | > loss_duration: 1.4960893392562866 (1.4795818452177376) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.100664138793945 (26.739186385582233) | > grad_norm_1: tensor(10186.4980, device='cuda:0') (tensor(10211.4785, device='cuda:0')) | > current_lr_0: 0.00016715660954422365 | > current_lr_1: 0.00016715660954422365 | > step_time: 0.4496 (0.4420592784881592) | > loader_time: 0.0078 (0.006948010674838362) --> EVAL PERFORMANCE | > avg_loader_time: 0.008629271681879607 (-0.000203585960495644) | > avg_step_time: 0.46038605797458704 (-0.0034604811332595298) | > avg_loss_disc: 2.6487089680953755 (+0.12605634541578548) | > avg_loss_disc_real_0: 0.14270611133583835 (+0.047109022474204995) | > avg_loss_disc_real_1: 0.2378600388765335 (+0.020065593887382827) | > avg_loss_disc_real_2: 0.21457573897402052 (-0.011306510844700773) | > avg_loss_disc_real_3: 0.23608374406754132 (+0.005437858927417827) | > avg_loss_disc_real_4: 0.24247006589258221 (+0.00699372694525921) | > avg_loss_disc_real_5: 0.23976984452193892 (-0.0055530375578033975) | > avg_loss_0: 2.6487089680953755 (+0.12605634541578548) | > avg_grad_norm_0: tensor(349.7057, device='cuda:0') (+tensor(18.7260, device='cuda:0')) | > avg_loss_gen: 2.167312049529921 (-0.09752273391669997) | > avg_loss_kl: 2.1492117458665874 (+0.19456320413401418) | > avg_loss_feat: 4.118748510387584 (-0.13685806704239134) | > avg_loss_mel: 17.262341190391858 (+0.6729679241986233) | > avg_loss_duration: 1.5353817620747525 (+0.014661971951874131) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.232995127288387 (+0.6478122926094159) | > avg_grad_norm_1: tensor(8758.8535, device='cuda:0') (tensor(-352.3613, device='cuda:0'))  > EPOCH: 1436/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:00:07)   --> TIME: 2025-02-25 23:00:12 -- STEP: 7/72 -- GLOBAL_STEP: 1103400 | > loss_disc: 2.522369146347046 (2.7213193348475864) | > loss_disc_real_0: 0.13278333842754364 (0.15411837505442755) | > loss_disc_real_1: 0.23719125986099243 (0.23492535310132162) | > loss_disc_real_2: 0.20045258104801178 (0.2330548827137266) | > loss_disc_real_3: 0.2835029363632202 (0.25829086346285685) | > loss_disc_real_4: 0.2946174442768097 (0.24700259949479783) | > loss_disc_real_5: 0.2485271692276001 (0.25579204303877695) | > loss_0: 2.522369146347046 (2.7213193348475864) | > grad_norm_0: tensor(353.4919, device='cuda:0') (tensor(383.6990, device='cuda:0')) | > loss_gen: 2.2629120349884033 (2.0830631937299455) | > loss_kl: 2.1049463748931885 (2.2066058090754916) | > loss_feat: 4.100725173950195 (3.9114242621830533) | > loss_mel: 17.256681442260742 (18.31656401497977) | > loss_duration: 1.4789950847625732 (1.5115962879998344) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.204259872436523 (28.029253278459823) | > grad_norm_1: tensor(15340.3496, device='cuda:0') (tensor(9836.8418, device='cuda:0')) | > current_lr_0: 0.0001671357149680306 | > current_lr_1: 0.0001671357149680306 | > step_time: 0.429 (0.4345181328909738) | > loader_time: 0.0056 (0.006384645189557757)  --> TIME: 2025-02-25 23:00:36 -- STEP: 57/72 -- GLOBAL_STEP: 1103450 | > loss_disc: 2.5379581451416016 (2.6423306548804564) | > loss_disc_real_0: 0.06277337670326233 (0.11496032591451678) | > loss_disc_real_1: 0.3664343059062958 (0.23445331213767068) | > loss_disc_real_2: 0.23202680051326752 (0.22999700407187143) | > loss_disc_real_3: 0.28989389538764954 (0.23843410402013546) | > loss_disc_real_4: 0.2524760067462921 (0.24596331882895084) | > loss_disc_real_5: 0.2646348476409912 (0.24801608867812577) | > loss_0: 2.5379581451416016 (2.6423306548804564) | > grad_norm_0: tensor(30.9654, device='cuda:0') (tensor(424.6716, device='cuda:0')) | > loss_gen: 2.1880040168762207 (2.0377494912398495) | > loss_kl: 2.1037092208862305 (2.0424550813541082) | > loss_feat: 4.368495464324951 (3.8984725726278207) | > loss_mel: 16.302072525024414 (17.057876603645187) | > loss_duration: 1.5746119022369385 (1.5191861391067505) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.53689193725586 (26.555739720662434) | > grad_norm_1: tensor(13696.0039, device='cuda:0') (tensor(12920.7158, device='cuda:0')) | > current_lr_0: 0.0001671357149680306 | > current_lr_1: 0.0001671357149680306 | > step_time: 0.4667 (0.4483310465227094) | > loader_time: 0.0098 (0.008110230429130687) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738312922732928 (+0.00010904124085332062) | > avg_step_time: 0.4580192532337887 (-0.0023668047407983117) | > avg_loss_disc: 2.646937219189926 (-0.0017717489054493996) | > avg_loss_disc_real_0: 0.11593447688599708 (-0.026771634449841278) | > avg_loss_disc_real_1: 0.22666024700017043 (-0.01119979187636308) | > avg_loss_disc_real_2: 0.2294711281715984 (+0.01489538919757788) | > avg_loss_disc_real_3: 0.23517436649597867 (-0.0009093775715626473) | > avg_loss_disc_real_4: 0.24391439703988357 (+0.0014443311473013587) | > avg_loss_disc_real_5: 0.24635589185734869 (+0.006586047335409767) | > avg_loss_0: 2.646937219189926 (-0.0017717489054493996) | > avg_grad_norm_0: tensor(433.2370, device='cuda:0') (+tensor(83.5313, device='cuda:0')) | > avg_loss_gen: 2.0188899006642087 (-0.14842214886571226) | > avg_loss_kl: 2.0584432323214035 (-0.0907685135451839) | > avg_loss_feat: 3.8694926080569414 (-0.24925590233064243) | > avg_loss_mel: 16.98158239982497 (-0.2807587905668889) | > avg_loss_duration: 1.5306567994641587 (-0.0047249626105938525) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.459064832875427 (-0.7739302944129598) | > avg_grad_norm_1: tensor(13269.8408, device='cuda:0') (+tensor(4510.9873, device='cuda:0'))  > EPOCH: 1437/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:00:45)   --> TIME: 2025-02-25 23:01:02 -- STEP: 35/72 -- GLOBAL_STEP: 1103500 | > loss_disc: 2.5501363277435303 (2.6083514622279576) | > loss_disc_real_0: 0.2251463234424591 (0.09834073558449745) | > loss_disc_real_1: 0.2384369671344757 (0.21647948111806598) | > loss_disc_real_2: 0.23045219480991364 (0.23572621302945274) | > loss_disc_real_3: 0.22983159124851227 (0.23374211405004774) | > loss_disc_real_4: 0.23730382323265076 (0.2449109239237649) | > loss_disc_real_5: 0.24682700634002686 (0.2433867816414152) | > loss_0: 2.5501363277435303 (2.6083514622279576) | > grad_norm_0: tensor(330.7963, device='cuda:0') (tensor(480.0309, device='cuda:0')) | > loss_gen: 2.3414723873138428 (2.0737221990312853) | > loss_kl: 1.9203898906707764 (1.8673538650785173) | > loss_feat: 3.99942684173584 (3.9735971791403633) | > loss_mel: 16.607460021972656 (16.600560814993724) | > loss_duration: 1.5240678787231445 (1.4858734607696533) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.392818450927734 (26.001107352120535) | > grad_norm_1: tensor(12954.6055, device='cuda:0') (tensor(15264.7041, device='cuda:0')) | > current_lr_0: 0.0001671148230036596 | > current_lr_1: 0.0001671148230036596 | > step_time: 0.4391 (0.43861237253461566) | > loader_time: 0.0082 (0.007368367058890207) --> EVAL PERFORMANCE | > avg_loader_time: 0.008828213517095 (+8.990059436207211e-05) | > avg_step_time: 0.4553773067366909 (-0.0026419464970978135) | > avg_loss_disc: 2.6613355791065056 (+0.014398359916579562) | > avg_loss_disc_real_0: 0.11278225156203123 (-0.0031522253239658488) | > avg_loss_disc_real_1: 0.24116022087318797 (+0.014499973873017546) | > avg_loss_disc_real_2: 0.23630890124280687 (+0.006837773071208475) | > avg_loss_disc_real_3: 0.22415441106742537 (-0.011019955428553302) | > avg_loss_disc_real_4: 0.23933208891203706 (-0.004582308127846518) | > avg_loss_disc_real_5: 0.24259902033167827 (-0.003756871525670419) | > avg_loss_0: 2.6613355791065056 (+0.014398359916579562) | > avg_grad_norm_0: tensor(473.5984, device='cuda:0') (+tensor(40.3614, device='cuda:0')) | > avg_loss_gen: 2.0529727818260737 (+0.034082881161864975) | > avg_loss_kl: 1.9526494472799167 (-0.10579378504148673) | > avg_loss_feat: 3.853621472775097 (-0.015871135281844584) | > avg_loss_mel: 16.524944412876184 (-0.45663798694878466) | > avg_loss_duration: 1.5220956130766532 (-0.00856118638750547) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.906283580081563 (-0.5527812527938636) | > avg_grad_norm_1: tensor(14276.8525, device='cuda:0') (+tensor(1007.0117, device='cuda:0'))  > EPOCH: 1438/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:01:21)   --> TIME: 2025-02-25 23:01:29 -- STEP: 13/72 -- GLOBAL_STEP: 1103550 | > loss_disc: 2.6587066650390625 (2.6511668241941004) | > loss_disc_real_0: 0.19713404774665833 (0.17564604259454286) | > loss_disc_real_1: 0.30158981680870056 (0.22840951153865227) | > loss_disc_real_2: 0.2356031984090805 (0.22427644637914804) | > loss_disc_real_3: 0.24427534639835358 (0.23730914753216964) | > loss_disc_real_4: 0.23860692977905273 (0.2409408986568451) | > loss_disc_real_5: 0.2387828230857849 (0.2410390984553557) | > loss_0: 2.6587066650390625 (2.6511668241941004) | > grad_norm_0: tensor(528.7463, device='cuda:0') (tensor(414.9087, device='cuda:0')) | > loss_gen: 2.588132381439209 (2.1296075399105368) | > loss_kl: 1.9105197191238403 (1.9010462485826933) | > loss_feat: 4.044061183929443 (3.7278016347151537) | > loss_mel: 15.363260269165039 (16.262850761413574) | > loss_duration: 1.4266986846923828 (1.4676710367202759) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.332672119140625 (25.488976551936222) | > grad_norm_1: tensor(9557.0850, device='cuda:0') (tensor(11731.8975, device='cuda:0')) | > current_lr_0: 0.00016709393365078415 | > current_lr_1: 0.00016709393365078415 | > step_time: 0.4643 (0.43230412556574893) | > loader_time: 0.006 (0.006707925062913161)  --> TIME: 2025-02-25 23:01:53 -- STEP: 63/72 -- GLOBAL_STEP: 1103600 | > loss_disc: 2.6184582710266113 (2.628644390711709) | > loss_disc_real_0: 0.10177292674779892 (0.13370827952074632) | > loss_disc_real_1: 0.22714200615882874 (0.23944179335283852) | > loss_disc_real_2: 0.2556154429912567 (0.23972562404852066) | > loss_disc_real_3: 0.2453444004058838 (0.2518064637505819) | > loss_disc_real_4: 0.25851860642433167 (0.2520501100354724) | > loss_disc_real_5: 0.2522737383842468 (0.25053037159026614) | > loss_0: 2.6184582710266113 (2.628644390711709) | > grad_norm_0: tensor(345.2722, device='cuda:0') (tensor(430.7489, device='cuda:0')) | > loss_gen: 2.1006431579589844 (2.182781287602015) | > loss_kl: 2.0326735973358154 (1.9661586227871122) | > loss_feat: 3.675016164779663 (3.840985453318036) | > loss_mel: 15.619266510009766 (16.499593568226643) | > loss_duration: 1.5583488941192627 (1.5121574061257501) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.98594856262207 (26.00167619614374) | > grad_norm_1: tensor(11406.3926, device='cuda:0') (tensor(12840.0078, device='cuda:0')) | > current_lr_0: 0.00016709393365078415 | > current_lr_1: 0.00016709393365078415 | > step_time: 0.4903 (0.45909846396673293) | > loader_time: 0.0118 (0.008299430211385092) --> EVAL PERFORMANCE | > avg_loader_time: 0.00880041928358481 (-2.779423351018974e-05) | > avg_step_time: 0.46514304926697636 (+0.009765742530285448) | > avg_loss_disc: 2.624028947991385 (-0.03730663111512067) | > avg_loss_disc_real_0: 0.13145354505575885 (+0.01867129349372762) | > avg_loss_disc_real_1: 0.2363846889683898 (-0.00477553190479818) | > avg_loss_disc_real_2: 0.23863699822358683 (+0.002328096980779959) | > avg_loss_disc_real_3: 0.25059029837729224 (+0.026435887309866873) | > avg_loss_disc_real_4: 0.250783020761651 (+0.011450931849613932) | > avg_loss_disc_real_5: 0.25011080734326807 (+0.0075117870115898044) | > avg_loss_0: 2.624028947991385 (-0.03730663111512067) | > avg_grad_norm_0: tensor(438.4780, device='cuda:0') (tensor(-35.1203, device='cuda:0')) | > avg_loss_gen: 2.1741400537356514 (+0.12116727190957777) | > avg_loss_kl: 1.9804767034423183 (+0.02782725616240156) | > avg_loss_feat: 3.838566333475247 (-0.015055139299849696) | > avg_loss_mel: 16.543476198760555 (+0.018531785884370322) | > avg_loss_duration: 1.5209553443210229 (-0.001140268755630336) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.05761452795754 (+0.15133094787597656) | > avg_grad_norm_1: tensor(13056.2832, device='cuda:0') (tensor(-1220.5693, device='cuda:0'))  > EPOCH: 1439/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:01:59)   --> TIME: 2025-02-25 23:02:20 -- STEP: 41/72 -- GLOBAL_STEP: 1103650 | > loss_disc: 2.5595757961273193 (2.566394503523664) | > loss_disc_real_0: 0.06920185685157776 (0.11407833291990001) | > loss_disc_real_1: 0.19675105810165405 (0.21571715975680003) | > loss_disc_real_2: 0.21790184080600739 (0.2278022602563951) | > loss_disc_real_3: 0.23821237683296204 (0.23276304035651973) | > loss_disc_real_4: 0.24155910313129425 (0.2439879364356762) | > loss_disc_real_5: 0.23742422461509705 (0.2425177300848612) | > loss_0: 2.5595757961273193 (2.566394503523664) | > grad_norm_0: tensor(288.7824, device='cuda:0') (tensor(480.7798, device='cuda:0')) | > loss_gen: 2.017186164855957 (2.1452955094779416) | > loss_kl: 2.0571999549865723 (1.9314267809798078) | > loss_feat: 4.174829483032227 (4.041396722560974) | > loss_mel: 16.55341339111328 (16.500522287880504) | > loss_duration: 1.5089967250823975 (1.4882370495214694) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.31162452697754 (26.106878234118952) | > grad_norm_1: tensor(17706.9355, device='cuda:0') (tensor(13630.6807, device='cuda:0')) | > current_lr_0: 0.00016707304690907778 | > current_lr_1: 0.00016707304690907778 | > step_time: 0.4802 (0.45188680509241613) | > loader_time: 0.0095 (0.007796967901834627) --> EVAL PERFORMANCE | > avg_loader_time: 0.008971123628213371 (+0.00017070434462856075) | > avg_step_time: 0.4644656114175286 (-0.0006774378494477373) | > avg_loss_disc: 2.5784739373435435 (-0.04555501064784151) | > avg_loss_disc_real_0: 0.12646390067439683 (-0.004989644381362013) | > avg_loss_disc_real_1: 0.21410315998003515 (-0.022281528988354643) | > avg_loss_disc_real_2: 0.23357052198598083 (-0.005066476237606005) | > avg_loss_disc_real_3: 0.2334183649697774 (-0.017171933407514856) | > avg_loss_disc_real_4: 0.241025297994345 (-0.009757722767305993) | > avg_loss_disc_real_5: 0.23996393592424795 (-0.010146871419020126) | > avg_loss_0: 2.5784739373435435 (-0.04555501064784151) | > avg_grad_norm_0: tensor(482.2221, device='cuda:0') (+tensor(43.7441, device='cuda:0')) | > avg_loss_gen: 2.1831856596637778 (+0.00904560592812631) | > avg_loss_kl: 1.9974810271195962 (+0.01700432367727789) | > avg_loss_feat: 4.083776302740607 (+0.24520996926535998) | > avg_loss_mel: 16.66245801683883 (+0.11898181807827513) | > avg_loss_duration: 1.5227049253356288 (+0.0017495810146059831) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.449605861180267 (+0.3919913332227267) | > avg_grad_norm_1: tensor(11888.6016, device='cuda:0') (tensor(-1167.6816, device='cuda:0'))  > EPOCH: 1440/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:02:37)   --> TIME: 2025-02-25 23:02:48 -- STEP: 19/72 -- GLOBAL_STEP: 1103700 | > loss_disc: 2.718064546585083 (2.728701026816117) | > loss_disc_real_0: 0.08455546945333481 (0.20982705037060537) | > loss_disc_real_1: 0.32966479659080505 (0.27146210443032415) | > loss_disc_real_2: 0.238042950630188 (0.21925131466827893) | > loss_disc_real_3: 0.23582713305950165 (0.23003978556708285) | > loss_disc_real_4: 0.20820052921772003 (0.25415432845291336) | > loss_disc_real_5: 0.22277839481830597 (0.2460676321857854) | > loss_0: 2.718064546585083 (2.728701026816117) | > grad_norm_0: tensor(49.0444, device='cuda:0') (tensor(443.7066, device='cuda:0')) | > loss_gen: 2.014488697052002 (2.2519267044569315) | > loss_kl: 2.2826297283172607 (2.1179479297838713) | > loss_feat: 3.2751502990722656 (3.8923990224537097) | > loss_mel: 16.330547332763672 (17.420400519120065) | > loss_duration: 1.5013623237609863 (1.503675900007549) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.404178619384766 (27.186350270321494) | > grad_norm_1: tensor(7996.4146, device='cuda:0') (tensor(7741.1030, device='cuda:0')) | > current_lr_0: 0.00016705216277821414 | > current_lr_1: 0.00016705216277821414 | > step_time: 0.4581 (0.44743344658299494) | > loader_time: 0.0067 (0.006972074508666992)  --> TIME: 2025-02-25 23:03:12 -- STEP: 69/72 -- GLOBAL_STEP: 1103750 | > loss_disc: 3.00281023979187 (2.7048421700795493) | > loss_disc_real_0: 0.3710259199142456 (0.1900736684708491) | > loss_disc_real_1: 0.2371467798948288 (0.2310372613694357) | > loss_disc_real_2: 0.2527616322040558 (0.23622432772232138) | > loss_disc_real_3: 0.22728624939918518 (0.2355584361846896) | > loss_disc_real_4: 0.2153242975473404 (0.24548113259716312) | > loss_disc_real_5: 0.2312295287847519 (0.24398999261683313) | > loss_0: 3.00281023979187 (2.7048421700795493) | > grad_norm_0: tensor(495.4688, device='cuda:0') (tensor(367.6872, device='cuda:0')) | > loss_gen: 2.0241951942443848 (2.210784799810769) | > loss_kl: 2.778172254562378 (2.123270575551019) | > loss_feat: 3.6059391498565674 (3.8723287789717964) | > loss_mel: 17.90267562866211 (17.188637595245797) | > loss_duration: 1.7229118347167969 (1.5435431176337644) | > amp_scaler: 8.0 (8.46376811594203) | > loss_1: 28.033893585205078 (26.938564908677254) | > grad_norm_1: tensor(12253.7012, device='cuda:0') (tensor(7398.8979, device='cuda:0')) | > current_lr_0: 0.00016705216277821414 | > current_lr_1: 0.00016705216277821414 | > step_time: 0.5237 (0.4685140658115995) | > loader_time: 0.0127 (0.008864979813064352) --> EVAL PERFORMANCE | > avg_loader_time: 0.008915975060261469 (-5.514856795190218e-05) | > avg_step_time: 0.46983456275832486 (+0.005368951340796235) | > avg_loss_disc: 2.7054514784208488 (+0.1269775410773053) | > avg_loss_disc_real_0: 0.18944562105855467 (+0.06298172038415784) | > avg_loss_disc_real_1: 0.23100405291352474 (+0.016900892933489592) | > avg_loss_disc_real_2: 0.23677767781724393 (+0.0032071558312631032) | > avg_loss_disc_real_3: 0.23555000881913682 (+0.0021316438493594325) | > avg_loss_disc_real_4: 0.24553807415592838 (+0.004512776161583382) | > avg_loss_disc_real_5: 0.24450977251563275 (+0.0045458365913848) | > avg_loss_0: 2.7054514784208488 (+0.1269775410773053) | > avg_grad_norm_0: tensor(363.2847, device='cuda:0') (tensor(-118.9374, device='cuda:0')) | > avg_loss_gen: 2.202365327888811 (+0.019179668225033364) | > avg_loss_kl: 2.139673741770462 (+0.14219271465086591) | > avg_loss_feat: 3.8683100552626057 (-0.21546624747800136) | > avg_loss_mel: 17.285978223236512 (+0.6235202063976821) | > avg_loss_duration: 1.5469372473971945 (+0.02423232206156567) | > avg_amp_scaler: 8.450704225352112 (+0.4507042253521121) | > avg_loss_1: 27.043264657678737 (+0.5936587964984703) | > avg_grad_norm_1: tensor(7511.9541, device='cuda:0') (tensor(-4376.6475, device='cuda:0'))  > EPOCH: 1441/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:03:15)   --> TIME: 2025-02-25 23:03:39 -- STEP: 47/72 -- GLOBAL_STEP: 1103800 | > loss_disc: 2.979495048522949 (2.7671089527454793) | > loss_disc_real_0: 0.30385154485702515 (0.23931684979098913) | > loss_disc_real_1: 0.11661656200885773 (0.21608148388406065) | > loss_disc_real_2: 0.22981058061122894 (0.22116460635306986) | > loss_disc_real_3: 0.29628825187683105 (0.23659133720905223) | > loss_disc_real_4: 0.30555260181427 (0.23551526190118588) | > loss_disc_real_5: 0.2625679075717926 (0.24306078603927125) | > loss_0: 2.979495048522949 (2.7671089527454793) | > grad_norm_0: tensor(530.9743, device='cuda:0') (tensor(284.0852, device='cuda:0')) | > loss_gen: 1.9695218801498413 (2.107956675772971) | > loss_kl: 2.2000927925109863 (2.130710764134184) | > loss_feat: 3.334866523742676 (3.8585387240064906) | > loss_mel: 17.340063095092773 (17.7086761961592) | > loss_duration: 1.588571310043335 (1.525365801567727) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.433116912841797 (27.331248466004716) | > grad_norm_1: tensor(5392.9814, device='cuda:0') (tensor(5418.6899, device='cuda:0')) | > current_lr_0: 0.00016703128125786684 | > current_lr_1: 0.00016703128125786684 | > step_time: 0.4607 (0.45066608266627534) | > loader_time: 0.0092 (0.0079289750849947) --> EVAL PERFORMANCE | > avg_loader_time: 0.008895957973641408 (-2.001708662006073e-05) | > avg_step_time: 0.4634451731829576 (-0.006389389575367277) | > avg_loss_disc: 2.7706120752952477 (+0.0651605968743989) | > avg_loss_disc_real_0: 0.21354381236392012 (+0.024098191305365446) | > avg_loss_disc_real_1: 0.22160200795657198 (-0.009402044956952765) | > avg_loss_disc_real_2: 0.23696990835834558 (+0.00019223054110165116) | > avg_loss_disc_real_3: 0.24185214286119167 (+0.006302134042054847) | > avg_loss_disc_real_4: 0.2377431673902861 (-0.00779490676564229) | > avg_loss_disc_real_5: 0.2426726049940351 (-0.0018371675215976546) | > avg_loss_0: 2.7706120752952477 (+0.0651605968743989) | > avg_grad_norm_0: tensor(283.4648, device='cuda:0') (tensor(-79.8199, device='cuda:0')) | > avg_loss_gen: 2.084259499966259 (-0.1181058279225522) | > avg_loss_kl: 2.1384176972886197 (-0.0012560444818423555) | > avg_loss_feat: 3.7535144342503077 (-0.11479562101229801) | > avg_loss_mel: 17.441356376862856 (+0.15537815362634433) | > avg_loss_duration: 1.5442592677935747 (-0.002677979603619818) | > avg_amp_scaler: 8.0 (-0.4507042253521121) | > avg_loss_1: 26.961807358432825 (-0.08145729924591194) | > avg_grad_norm_1: tensor(5371.7261, device='cuda:0') (tensor(-2140.2280, device='cuda:0'))  > EPOCH: 1442/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:03:53)   --> TIME: 2025-02-25 23:04:07 -- STEP: 25/72 -- GLOBAL_STEP: 1103850 | > loss_disc: 2.580141544342041 (2.6448430728912355) | > loss_disc_real_0: 0.19434033334255219 (0.19015952110290527) | > loss_disc_real_1: 0.2175186276435852 (0.22329648494720458) | > loss_disc_real_2: 0.24722982943058014 (0.2268821769952774) | > loss_disc_real_3: 0.23192444443702698 (0.2401399505138397) | > loss_disc_real_4: 0.2684333026409149 (0.2399088478088379) | > loss_disc_real_5: 0.2806100845336914 (0.2441363078355789) | > loss_0: 2.580141544342041 (2.6448430728912355) | > grad_norm_0: tensor(578.4354, device='cuda:0') (tensor(295.4273, device='cuda:0')) | > loss_gen: 2.4892969131469727 (2.2063297748565676) | > loss_kl: 2.0544471740722656 (1.9875065279006958) | > loss_feat: 4.07918643951416 (3.9336502933502198) | > loss_mel: 17.6444149017334 (16.704150352478027) | > loss_duration: 1.488835096359253 (1.4916389226913451) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.756179809570312 (26.323275909423828) | > grad_norm_1: tensor(5092.1279, device='cuda:0') (tensor(5714.1782, device='cuda:0')) | > current_lr_0: 0.0001670104023477096 | > current_lr_1: 0.0001670104023477096 | > step_time: 0.4444 (0.4409883403778076) | > loader_time: 0.0074 (0.007027931213378906) --> EVAL PERFORMANCE | > avg_loader_time: 0.008682936010226397 (-0.0002130219634150117) | > avg_step_time: 0.4588760422988677 (-0.004569130884089889) | > avg_loss_disc: 2.6680601616980324 (-0.10255191359721527) | > avg_loss_disc_real_0: 0.16339256729878168 (-0.05015124506513843) | > avg_loss_disc_real_1: 0.22082854072812577 (-0.0007734672284462063) | > avg_loss_disc_real_2: 0.22907804183557 (-0.00789186652277557) | > avg_loss_disc_real_3: 0.23432727032144304 (-0.007524872539748623) | > avg_loss_disc_real_4: 0.2399851534987839 (+0.002241986108497823) | > avg_loss_disc_real_5: 0.24383091443861035 (+0.0011583094445752617) | > avg_loss_0: 2.6680601616980324 (-0.10255191359721527) | > avg_grad_norm_0: tensor(267.6989, device='cuda:0') (tensor(-15.7659, device='cuda:0')) | > avg_loss_gen: 2.112161156157373 (+0.027901656191114288) | > avg_loss_kl: 2.081162217637185 (-0.057255479651434626) | > avg_loss_feat: 3.8150125187887274 (+0.06149808453841965) | > avg_loss_mel: 16.83872870324363 (-0.602627673619228) | > avg_loss_duration: 1.5435346952626403 (-0.0007245725309343864) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.39059939854582 (-0.5712079598870048) | > avg_grad_norm_1: tensor(6199.6685, device='cuda:0') (+tensor(827.9424, device='cuda:0'))  > EPOCH: 1443/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:04:31)   --> TIME: 2025-02-25 23:04:34 -- STEP: 3/72 -- GLOBAL_STEP: 1103900 | > loss_disc: 2.7385709285736084 (2.8718499342600503) | > loss_disc_real_0: 0.3882763385772705 (0.5032928586006165) | > loss_disc_real_1: 0.22857816517353058 (0.24833446244398752) | > loss_disc_real_2: 0.22777040302753448 (0.22345595061779022) | > loss_disc_real_3: 0.25829049944877625 (0.26317350069681805) | > loss_disc_real_4: 0.22139886021614075 (0.22865324219067892) | > loss_disc_real_5: 0.21583431959152222 (0.2278884400924047) | > loss_0: 2.7385709285736084 (2.8718499342600503) | > grad_norm_0: tensor(464.4410, device='cuda:0') (tensor(418.0689, device='cuda:0')) | > loss_gen: 2.412158250808716 (2.4736098448435464) | > loss_kl: 1.8730777502059937 (1.940814773241679) | > loss_feat: 4.3869547843933105 (3.8937564690907798) | > loss_mel: 16.910974502563477 (16.85533269246419) | > loss_duration: 1.5402107238769531 (1.51966389020284) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.123376846313477 (26.683177312215168) | > grad_norm_1: tensor(2859.3748, device='cuda:0') (tensor(2663.5493, device='cuda:0')) | > current_lr_0: 0.00016698952604741614 | > current_lr_1: 0.00016698952604741614 | > step_time: 0.4282 (0.44394977887471515) | > loader_time: 0.0084 (0.0066832701365153)  --> TIME: 2025-02-25 23:04:57 -- STEP: 53/72 -- GLOBAL_STEP: 1103950 | > loss_disc: 2.78105092048645 (2.6717647651456433) | > loss_disc_real_0: 0.036031484603881836 (0.18908628146603423) | > loss_disc_real_1: 0.14112479984760284 (0.22241689798966893) | > loss_disc_real_2: 0.3044641315937042 (0.23186829742395654) | > loss_disc_real_3: 0.23306915163993835 (0.23884170207212557) | > loss_disc_real_4: 0.3220060467720032 (0.24284333187454152) | > loss_disc_real_5: 0.2846590578556061 (0.24400135368671058) | > loss_0: 2.78105092048645 (2.6717647651456433) | > grad_norm_0: tensor(258.6238, device='cuda:0') (tensor(185.9535, device='cuda:0')) | > loss_gen: 2.13423490524292 (2.149594149499569) | > loss_kl: 3.6312520503997803 (2.170624813943537) | > loss_feat: 4.274831295013428 (3.9117196820816904) | > loss_mel: 22.498647689819336 (17.433769118111094) | > loss_duration: 1.703343152999878 (1.5331102002341792) | > amp_scaler: 8.0 (8.0) | > loss_1: 34.2423095703125 (27.198818134811688) | > grad_norm_1: tensor(14850.4180, device='cuda:0') (tensor(5509.4155, device='cuda:0')) | > current_lr_0: 0.00016698952604741614 | > current_lr_1: 0.00016698952604741614 | > step_time: 0.4588 (0.4460172248336504) | > loader_time: 0.009 (0.007872167623267983) --> EVAL PERFORMANCE | > avg_loader_time: 0.008721237451257842 (+3.8301441031445005e-05) | > avg_step_time: 0.45776450801903096 (-0.0011115342798367345) | > avg_loss_disc: 2.6903214018109813 (+0.022261240112948943) | > avg_loss_disc_real_0: 0.17849862638493658 (+0.015106059086154894) | > avg_loss_disc_real_1: 0.23794759222319428 (+0.01711905149506851) | > avg_loss_disc_real_2: 0.23617270609862367 (+0.007094664263053663) | > avg_loss_disc_real_3: 0.2428712588800511 (+0.008543988558608051) | > avg_loss_disc_real_4: 0.23998012886920445 (-5.024629579464257e-06) | > avg_loss_disc_real_5: 0.24642100132686992 (+0.0025900868882595685) | > avg_loss_0: 2.6903214018109813 (+0.022261240112948943) | > avg_grad_norm_0: tensor(217.5892, device='cuda:0') (tensor(-50.1097, device='cuda:0')) | > avg_loss_gen: 2.159994563586275 (+0.047833407428901964) | > avg_loss_kl: 2.3458840981335696 (+0.26472188049638445) | > avg_loss_feat: 3.8824974516747703 (+0.06748493288604296) | > avg_loss_mel: 18.032486740971954 (+1.1937580377283261) | > avg_loss_duration: 1.5625988429700826 (+0.019064147707442247) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.983461890422124 (+1.5928624918763035) | > avg_grad_norm_1: tensor(5635.6060, device='cuda:0') (tensor(-564.0625, device='cuda:0'))  > EPOCH: 1444/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:05:08)   --> TIME: 2025-02-25 23:05:24 -- STEP: 31/72 -- GLOBAL_STEP: 1104000 | > loss_disc: 2.5390048027038574 (2.511236536887384) | > loss_disc_real_0: 0.04814669489860535 (0.08712705813588635) | > loss_disc_real_1: 0.21631401777267456 (0.2358356483521) | > loss_disc_real_2: 0.23175550997257233 (0.22534303030660074) | > loss_disc_real_3: 0.2488081157207489 (0.22868458638268133) | > loss_disc_real_4: 0.275409460067749 (0.240847090559621) | > loss_disc_real_5: 0.26591458916664124 (0.23815352974399442) | > loss_0: 2.5390048027038574 (2.511236536887384) | > grad_norm_0: tensor(394.0359, device='cuda:0') (tensor(210.2177, device='cuda:0')) | > loss_gen: 2.255786418914795 (2.1924462279965806) | > loss_kl: 2.2689242362976074 (2.18098497006201) | > loss_feat: 3.9103126525878906 (4.089316837249264) | > loss_mel: 17.788965225219727 (17.844485375189016) | > loss_duration: 1.526871681213379 (1.5156590169475925) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.75086212158203 (27.822892281316943) | > grad_norm_1: tensor(14021.5986, device='cuda:0') (tensor(7352.0835, device='cuda:0')) | > current_lr_0: 0.0001669686523566602 | > current_lr_1: 0.0001669686523566602 | > step_time: 0.4478 (0.4439527911524619) | > loader_time: 0.0089 (0.007190265963154454) --> EVAL PERFORMANCE | > avg_loader_time: 0.008746244537998254 (+2.500708674041223e-05) | > avg_step_time: 0.4613821875881141 (+0.0036176795690831542) | > avg_loss_disc: 2.574299251529532 (-0.11602215028144913) | > avg_loss_disc_real_0: 0.08845310290934334 (-0.09004552347559323) | > avg_loss_disc_real_1: 0.25947947884109657 (+0.021531886617902285) | > avg_loss_disc_real_2: 0.23564116761718 (-0.0005315384814436885) | > avg_loss_disc_real_3: 0.23203531264419286 (-0.010835946235858235) | > avg_loss_disc_real_4: 0.2374984733235668 (-0.0024816555456376455) | > avg_loss_disc_real_5: 0.2372673705849849 (-0.009153630741885022) | > avg_loss_0: 2.574299251529532 (-0.11602215028144913) | > avg_grad_norm_0: tensor(266.4148, device='cuda:0') (+tensor(48.8256, device='cuda:0')) | > avg_loss_gen: 2.1817073939551768 (+0.021712830368901592) | > avg_loss_kl: 2.1557267299840155 (-0.19015736814955408) | > avg_loss_feat: 4.078334878867783 (+0.19583742719301256) | > avg_loss_mel: 17.39742211892571 (-0.6350646220462437) | > avg_loss_duration: 1.5456033038421415 (-0.01699553912794105) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.358794333229604 (-0.6246675571925202) | > avg_grad_norm_1: tensor(8493.1377, device='cuda:0') (+tensor(2857.5317, device='cuda:0'))  > EPOCH: 1445/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:05:45)   --> TIME: 2025-02-25 23:05:52 -- STEP: 9/72 -- GLOBAL_STEP: 1104050 | > loss_disc: 2.5388545989990234 (2.6477297147115073) | > loss_disc_real_0: 0.038360293954610825 (0.1037983385225137) | > loss_disc_real_1: 0.3578369915485382 (0.30696334607071346) | > loss_disc_real_2: 0.2536909282207489 (0.2587309082349141) | > loss_disc_real_3: 0.22829295694828033 (0.25443849960962933) | > loss_disc_real_4: 0.2428724616765976 (0.25374379257361096) | > loss_disc_real_5: 0.24315327405929565 (0.22880632181962332) | > loss_0: 2.5388545989990234 (2.6477297147115073) | > grad_norm_0: tensor(66.5350, device='cuda:0') (tensor(376.5238, device='cuda:0')) | > loss_gen: 2.2952237129211426 (2.2341016663445368) | > loss_kl: 1.7828848361968994 (1.823750654856364) | > loss_feat: 4.796760559082031 (4.112436718410915) | > loss_mel: 17.114482879638672 (16.97763951619466) | > loss_duration: 1.518456220626831 (1.4980034960640802) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.5078067779541 (26.64593251546224) | > grad_norm_1: tensor(11818.4131, device='cuda:0') (tensor(11606.2305, device='cuda:0')) | > current_lr_0: 0.00016694778127511562 | > current_lr_1: 0.00016694778127511562 | > step_time: 0.4422 (0.4376668400234646) | > loader_time: 0.0058 (0.0068086253272162545)  --> TIME: 2025-02-25 23:06:15 -- STEP: 59/72 -- GLOBAL_STEP: 1104100 | > loss_disc: 2.4986796379089355 (2.5671819104986677) | > loss_disc_real_0: 0.08072198182344437 (0.08063763196943172) | > loss_disc_real_1: 0.20794734358787537 (0.22425721838312634) | > loss_disc_real_2: 0.251987099647522 (0.2708288148298102) | > loss_disc_real_3: 0.21516093611717224 (0.2331548343270512) | > loss_disc_real_4: 0.23207367956638336 (0.23163330277143898) | > loss_disc_real_5: 0.23310792446136475 (0.24375592652013747) | > loss_0: 2.4986796379089355 (2.5671819104986677) | > grad_norm_0: tensor(461.2169, device='cuda:0') (tensor(352.6823, device='cuda:0')) | > loss_gen: 1.9825236797332764 (2.211382122363075) | > loss_kl: 1.8800113201141357 (1.9701595043731948) | > loss_feat: 3.6306662559509277 (4.178230641251903) | > loss_mel: 16.59190559387207 (16.718145936222403) | > loss_duration: 1.534857988357544 (1.5194905208329024) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.619964599609375 (26.597408844252765) | > grad_norm_1: tensor(13625.7061, device='cuda:0') (tensor(11759.2861, device='cuda:0')) | > current_lr_0: 0.00016694778127511562 | > current_lr_1: 0.00016694778127511562 | > step_time: 0.4746 (0.4501060025166657) | > loader_time: 0.0099 (0.008350432929346115) --> EVAL PERFORMANCE | > avg_loader_time: 0.008886847697513203 (+0.00014060315951494942) | > avg_step_time: 0.45776216077132964 (-0.003620026816784472) | > avg_loss_disc: 2.5620085924444065 (-0.012290659085125721) | > avg_loss_disc_real_0: 0.08080577089543076 (-0.007647332013912586) | > avg_loss_disc_real_1: 0.22514428806976533 (-0.03433519077133124) | > avg_loss_disc_real_2: 0.2663600178251804 (+0.030718850208000387) | > avg_loss_disc_real_3: 0.23267386916657568 (+0.000638556522382816) | > avg_loss_disc_real_4: 0.2342586681036882 (-0.0032398052198786) | > avg_loss_disc_real_5: 0.24464664891572066 (+0.007379278330735761) | > avg_loss_0: 2.5620085924444065 (-0.012290659085125721) | > avg_grad_norm_0: tensor(347.3921, device='cuda:0') (+tensor(80.9773, device='cuda:0')) | > avg_loss_gen: 2.2158606992640975 (+0.03415330530892069) | > avg_loss_kl: 1.9906526065208543 (-0.1650741234631612) | > avg_loss_feat: 4.154832544461103 (+0.07649766559331983) | > avg_loss_mel: 16.717060572664504 (-0.6803615462612065) | > avg_loss_duration: 1.5307897581181056 (-0.014813545724035926) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.609196219645757 (-0.749598113583847) | > avg_grad_norm_1: tensor(11486.3838, device='cuda:0') (+tensor(2993.2461, device='cuda:0'))  > EPOCH: 1446/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:06:23)   --> TIME: 2025-02-25 23:06:42 -- STEP: 37/72 -- GLOBAL_STEP: 1104150 | > loss_disc: 2.495090961456299 (2.5066414008269438) | > loss_disc_real_0: 0.05501273274421692 (0.09032919388767835) | > loss_disc_real_1: 0.21552161872386932 (0.21967063884477359) | > loss_disc_real_2: 0.24751880764961243 (0.2445403212631071) | > loss_disc_real_3: 0.2293933480978012 (0.23279920784202782) | > loss_disc_real_4: 0.23589029908180237 (0.24024488313778028) | > loss_disc_real_5: 0.2557438015937805 (0.2447993553973533) | > loss_0: 2.495090961456299 (2.5066414008269438) | > grad_norm_0: tensor(289.8584, device='cuda:0') (tensor(341.6624, device='cuda:0')) | > loss_gen: 2.2504405975341797 (2.303579807281494) | > loss_kl: 1.9931844472885132 (1.930479861594535) | > loss_feat: 4.198774814605713 (4.295243392119536) | > loss_mel: 16.313194274902344 (16.621342530121677) | > loss_duration: 1.523285984992981 (1.4916906131280434) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.278881072998047 (26.642336046373522) | > grad_norm_1: tensor(11138.6631, device='cuda:0') (tensor(9617.1523, device='cuda:0')) | > current_lr_0: 0.00016692691280245623 | > current_lr_1: 0.00016692691280245623 | > step_time: 0.5005 (0.44612331648130676) | > loader_time: 0.0081 (0.007378069130150047) --> EVAL PERFORMANCE | > avg_loader_time: 0.00873102604503363 (-0.00015582165247957254) | > avg_step_time: 0.4624208094368518 (+0.00465864866552218) | > avg_loss_disc: 2.505557073673732 (-0.05645151877067445) | > avg_loss_disc_real_0: 0.09045702759438837 (+0.009651256698957611) | > avg_loss_disc_real_1: 0.22114875954641422 (-0.003995528523351111) | > avg_loss_disc_real_2: 0.23734063371806077 (-0.0290193841071196) | > avg_loss_disc_real_3: 0.2319034434540171 (-0.0007704257125585867) | > avg_loss_disc_real_4: 0.23983468451130557 (+0.0055760164076173735) | > avg_loss_disc_real_5: 0.24419822764228766 (-0.00044842127343300286) | > avg_loss_0: 2.505557073673732 (-0.05645151877067445) | > avg_grad_norm_0: tensor(342.5147, device='cuda:0') (tensor(-4.8773, device='cuda:0')) | > avg_loss_gen: 2.278875035299382 (+0.06301433603528439) | > avg_loss_kl: 1.967619778404773 (-0.023032828116081205) | > avg_loss_feat: 4.252993879183917 (+0.09816133472281408) | > avg_loss_mel: 16.541212968423334 (-0.17584760424117007) | > avg_loss_duration: 1.5252547196938957 (-0.005535038424209882) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.565956303771113 (-0.043239915874643486) | > avg_grad_norm_1: tensor(9619.7422, device='cuda:0') (tensor(-1866.6416, device='cuda:0'))  > EPOCH: 1447/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:07:01)   --> TIME: 2025-02-25 23:07:09 -- STEP: 15/72 -- GLOBAL_STEP: 1104200 | > loss_disc: 2.347752571105957 (2.488477087020874) | > loss_disc_real_0: 0.1162601113319397 (0.0853265273074309) | > loss_disc_real_1: 0.2321750521659851 (0.21864575843016307) | > loss_disc_real_2: 0.20542141795158386 (0.23153058489163716) | > loss_disc_real_3: 0.23059159517288208 (0.23453554411729177) | > loss_disc_real_4: 0.24134483933448792 (0.24679274658362071) | > loss_disc_real_5: 0.23697561025619507 (0.24150305688381196) | > loss_0: 2.347752571105957 (2.488477087020874) | > grad_norm_0: tensor(282.1558, device='cuda:0') (tensor(306.5150, device='cuda:0')) | > loss_gen: 2.6592459678649902 (2.3012390613555906) | > loss_kl: 1.9134166240692139 (1.895139177640279) | > loss_feat: 4.84796142578125 (4.416671291987101) | > loss_mel: 16.726943969726562 (16.6951384862264) | > loss_duration: 1.4893593788146973 (1.4697616259256998) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.63692855834961 (26.77795015970866) | > grad_norm_1: tensor(5831.0566, device='cuda:0') (tensor(9223.2793, device='cuda:0')) | > current_lr_0: 0.0001669060469383559 | > current_lr_1: 0.0001669060469383559 | > step_time: 0.435 (0.44064931869506835) | > loader_time: 0.0062 (0.0067026774088541664)  --> TIME: 2025-02-25 23:07:33 -- STEP: 65/72 -- GLOBAL_STEP: 1104250 | > loss_disc: 2.5365076065063477 (2.516718593010536) | > loss_disc_real_0: 0.10498948395252228 (0.10083406395637072) | > loss_disc_real_1: 0.21417945623397827 (0.22155498701792498) | > loss_disc_real_2: 0.24695473909378052 (0.2297732094159493) | > loss_disc_real_3: 0.24785177409648895 (0.2367926081785789) | > loss_disc_real_4: 0.2383262664079666 (0.24473153215188245) | > loss_disc_real_5: 0.24054571986198425 (0.2464013709471776) | > loss_0: 2.5365076065063477 (2.516718593010536) | > grad_norm_0: tensor(284.2147, device='cuda:0') (tensor(327.2704, device='cuda:0')) | > loss_gen: 1.9963483810424805 (2.283011346596937) | > loss_kl: 2.1705005168914795 (1.9644701022368212) | > loss_feat: 3.4448373317718506 (4.245980292100171) | > loss_mel: 16.806516647338867 (16.49053200941819) | > loss_duration: 1.5949923992156982 (1.5141687539907602) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.013193130493164 (26.4981626363901) | > grad_norm_1: tensor(12179.9932, device='cuda:0') (tensor(9019.7217, device='cuda:0')) | > current_lr_0: 0.0001669060469383559 | > current_lr_1: 0.0001669060469383559 | > step_time: 0.5104 (0.45348600974449743) | > loader_time: 0.0117 (0.00833586546090933) --> EVAL PERFORMANCE | > avg_loader_time: 0.008595409527630873 (-0.00013561651740275765) | > avg_step_time: 0.45854739404060474 (-0.003873415396247082) | > avg_loss_disc: 2.5153206469307485 (+0.009763573257016489) | > avg_loss_disc_real_0: 0.10023012473969392 (+0.009773097145305554) | > avg_loss_disc_real_1: 0.2212970598902501 (+0.00014830034383586654) | > avg_loss_disc_real_2: 0.2293571844487123 (-0.007983449269348464) | > avg_loss_disc_real_3: 0.23641290484179914 (+0.004509461387782049) | > avg_loss_disc_real_4: 0.24374443481505756 (+0.003909750303751985) | > avg_loss_disc_real_5: 0.24644412792904277 (+0.0022459002867551148) | > avg_loss_0: 2.5153206469307485 (+0.009763573257016489) | > avg_grad_norm_0: tensor(331.3597, device='cuda:0') (tensor(-11.1550, device='cuda:0')) | > avg_loss_gen: 2.2746346282287373 (-0.004240407070644547) | > avg_loss_kl: 1.9732045724358358 (+0.005584794031062712) | > avg_loss_feat: 4.218356639566554 (-0.034637239617362425) | > avg_loss_mel: 16.494442469637153 (-0.04677049878618078) | > avg_loss_duration: 1.5203739522208632 (-0.004880767473032455) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.48101234436035 (-0.0849439594107615) | > avg_grad_norm_1: tensor(9007.0410, device='cuda:0') (tensor(-612.7012, device='cuda:0'))  > EPOCH: 1448/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:07:38)   --> TIME: 2025-02-25 23:07:59 -- STEP: 43/72 -- GLOBAL_STEP: 1104300 | > loss_disc: 2.7710912227630615 (2.607000628183055) | > loss_disc_real_0: 0.09035893529653549 (0.1224848458240198) | > loss_disc_real_1: 0.3088185787200928 (0.22204100236643193) | > loss_disc_real_2: 0.22833020985126495 (0.24597558717048437) | > loss_disc_real_3: 0.23779642581939697 (0.22055976491334828) | > loss_disc_real_4: 0.2791571319103241 (0.2262551295895909) | > loss_disc_real_5: 0.22880354523658752 (0.23853889523550523) | > loss_0: 2.7710912227630615 (2.607000628183055) | > grad_norm_0: tensor(195.1377, device='cuda:0') (tensor(281.4830, device='cuda:0')) | > loss_gen: 2.080869674682617 (2.24429543628249) | > loss_kl: 2.4551711082458496 (2.4344714076019995) | > loss_feat: 3.832122325897217 (4.391341858131941) | > loss_mel: 18.599342346191406 (17.916971162308098) | > loss_duration: 1.585594892501831 (1.5880644044210745) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.553098678588867 (28.575144213299417) | > grad_norm_1: tensor(10519.3916, device='cuda:0') (tensor(7828.5137, device='cuda:0')) | > current_lr_0: 0.0001668851836824886 | > current_lr_1: 0.0001668851836824886 | > step_time: 0.4608 (0.44288989000542217) | > loader_time: 0.0094 (0.007616675177285838) --> EVAL PERFORMANCE | > avg_loader_time: 0.008816101181675013 (+0.00022069165404414032) | > avg_step_time: 0.45853166513040033 (-1.5728910204404478e-05) | > avg_loss_disc: 2.641989902711251 (+0.12666925578050225) | > avg_loss_disc_real_0: 0.13715247400629685 (+0.03692234926660293) | > avg_loss_disc_real_1: 0.23009907915978364 (+0.008802019269533556) | > avg_loss_disc_real_2: 0.23586720432823807 (+0.0065100198795257636) | > avg_loss_disc_real_3: 0.22769059545137513 (-0.00872230939042401) | > avg_loss_disc_real_4: 0.23429093247568103 (-0.00945350233937653) | > avg_loss_disc_real_5: 0.23984485759701527 (-0.006599270332027507) | > avg_loss_0: 2.641989902711251 (+0.12666925578050225) | > avg_grad_norm_0: tensor(308.6907, device='cuda:0') (tensor(-22.6691, device='cuda:0')) | > avg_loss_gen: 2.1659121630896983 (-0.10872246513903905) | > avg_loss_kl: 2.3830352652240805 (+0.4098306927882447) | > avg_loss_feat: 4.120042139375713 (-0.09831450019084098) | > avg_loss_mel: 17.93991468993711 (+1.4454722202999548) | > avg_loss_duration: 1.5924385517415867 (+0.07206459952072342) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.20134270359093 (+1.72033035923058) | > avg_grad_norm_1: tensor(7607.8730, device='cuda:0') (tensor(-1399.1680, device='cuda:0'))  > EPOCH: 1449/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:08:15)   --> TIME: 2025-02-25 23:08:27 -- STEP: 21/72 -- GLOBAL_STEP: 1104350 | > loss_disc: 2.5320992469787598 (2.8534309523446217) | > loss_disc_real_0: 0.22132405638694763 (0.20760863416251682) | > loss_disc_real_1: 0.21308836340904236 (0.24071334515299117) | > loss_disc_real_2: 0.20504899322986603 (0.24819441920235044) | > loss_disc_real_3: 0.20888109505176544 (0.25084078524793896) | > loss_disc_real_4: 0.238785520195961 (0.2570334126551946) | > loss_disc_real_5: 0.2529546022415161 (0.2577855402515048) | > loss_0: 2.5320992469787598 (2.8534309523446217) | > grad_norm_0: tensor(141.0789, device='cuda:0') (tensor(343.1601, device='cuda:0')) | > loss_gen: 2.235184907913208 (2.041032870610555) | > loss_kl: 3.0719659328460693 (2.8618599460238503) | > loss_feat: 3.895443916320801 (3.6493858723413375) | > loss_mel: 23.51594352722168 (20.929257347470237) | > loss_duration: 1.7433788776397705 (1.6953410704930623) | > amp_scaler: 8.0 (8.0) | > loss_1: 34.4619140625 (31.176876976376487) | > grad_norm_1: tensor(2655.4746, device='cuda:0') (tensor(7937.5864, device='cuda:0')) | > current_lr_0: 0.0001668643230345283 | > current_lr_1: 0.0001668643230345283 | > step_time: 0.4481 (0.4484592392331078) | > loader_time: 0.0066 (0.007058711279006232)  --> TIME: 2025-02-25 23:08:51 -- STEP: 71/72 -- GLOBAL_STEP: 1104400 | > loss_disc: 2.7599542140960693 (2.6410058887911516) | > loss_disc_real_0: 0.03574587032198906 (0.1490786639632474) | > loss_disc_real_1: 0.6227205395698547 (0.23029775182965775) | > loss_disc_real_2: 0.2526952922344208 (0.23551207780838013) | > loss_disc_real_3: 0.1956140100955963 (0.23947815848907955) | > loss_disc_real_4: 0.24060232937335968 (0.24934265689111101) | > loss_disc_real_5: 0.21350710093975067 (0.24320659650043702) | > loss_0: 2.7599542140960693 (2.6410058887911516) | > grad_norm_0: tensor(249.9124, device='cuda:0') (tensor(311.0626, device='cuda:0')) | > loss_gen: 2.657740354537964 (2.1700997083959446) | > loss_kl: 2.015930652618408 (2.4532926401621857) | > loss_feat: 4.662628173828125 (3.929100019831053) | > loss_mel: 16.897689819335938 (18.89931612955013) | > loss_duration: 1.5517815351486206 (1.6305047072155372) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.785768508911133 (29.082313027180415) | > grad_norm_1: tensor(7591.5591, device='cuda:0') (tensor(7651.6636, device='cuda:0')) | > current_lr_0: 0.0001668643230345283 | > current_lr_1: 0.0001668643230345283 | > step_time: 0.485 (0.45782744044988927) | > loader_time: 0.0078 (0.008688906548728404) --> EVAL PERFORMANCE | > avg_loader_time: 0.008688906548728404 (-0.00012719463294660893) | > avg_step_time: 0.45782744044988927 (-0.0007042246805110675) | > avg_loss_disc: 2.6410058887911516 (-0.0009840139200991693) | > avg_loss_disc_real_0: 0.1490786639632474 (+0.01192618995695055) | > avg_loss_disc_real_1: 0.23029775182965775 (+0.00019867266987411147) | > avg_loss_disc_real_2: 0.23551207780838013 (-0.00035512651985794563) | > avg_loss_disc_real_3: 0.23947815848907955 (+0.011787563037704424) | > avg_loss_disc_real_4: 0.24934265689111101 (+0.015051724415429985) | > avg_loss_disc_real_5: 0.24320659650043702 (+0.003361738903421757) | > avg_loss_0: 2.6410058887911516 (-0.0009840139200991693) | > avg_grad_norm_0: tensor(311.0626, device='cuda:0') (+tensor(2.3719, device='cuda:0')) | > avg_loss_gen: 2.1700997083959446 (+0.004187545306246321) | > avg_loss_kl: 2.4532926401621857 (+0.07025737493810524) | > avg_loss_feat: 3.929100019831053 (-0.19094211954466056) | > avg_loss_mel: 18.89931612955013 (+0.9594014396130213) | > avg_loss_duration: 1.6305047072155372 (+0.038066155473950536) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.082313027180415 (+0.8809703235894837) | > avg_grad_norm_1: tensor(7651.6636, device='cuda:0') (+tensor(43.7905, device='cuda:0'))  > EPOCH: 1450/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:08:53)   --> TIME: 2025-02-25 23:09:17 -- STEP: 49/72 -- GLOBAL_STEP: 1104450 | > loss_disc: 2.4985995292663574 (2.751576141435273) | > loss_disc_real_0: 0.0989961251616478 (0.15622742991058194) | > loss_disc_real_1: 0.2349957525730133 (0.3179652590532692) | > loss_disc_real_2: 0.23807314038276672 (0.24915044678717244) | > loss_disc_real_3: 0.21836301684379578 (0.21254470062499142) | > loss_disc_real_4: 0.23834766447544098 (0.2395882478782109) | > loss_disc_real_5: 0.24689528346061707 (0.24433066072512646) | > loss_0: 2.4985995292663574 (2.751576141435273) | > grad_norm_0: tensor(128.0031, device='cuda:0') (tensor(338.7770, device='cuda:0')) | > loss_gen: 1.9858381748199463 (2.173678451654862) | > loss_kl: 1.9509390592575073 (2.006011982353366) | > loss_feat: 3.594205141067505 (3.8061845789150324) | > loss_mel: 17.197391510009766 (16.863225742262237) | > loss_duration: 1.5986924171447754 (1.5270957679164654) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.327064514160156 (26.37619653040049) | > grad_norm_1: tensor(3516.1797, device='cuda:0') (tensor(6950.4194, device='cuda:0')) | > current_lr_0: 0.00016684346499414897 | > current_lr_1: 0.00016684346499414897 | > step_time: 0.4341 (0.4418055281347158) | > loader_time: 0.0093 (0.00794048698580995) --> EVAL PERFORMANCE | > avg_loader_time: 0.008765119901845152 (+7.621335311674797e-05) | > avg_step_time: 0.4511287749653131 (-0.0066986654845761495) | > avg_loss_disc: 2.7078399624623044 (+0.0668340736711528) | > avg_loss_disc_real_0: 0.1478092183529491 (-0.001269445610298292) | > avg_loss_disc_real_1: 0.291697587975314 (+0.06139983614565622) | > avg_loss_disc_real_2: 0.24517052715093318 (+0.009658449342553055) | > avg_loss_disc_real_3: 0.2178460019575038 (-0.02163215653157574) | > avg_loss_disc_real_4: 0.2436439833590682 (-0.005698673532042814) | > avg_loss_disc_real_5: 0.2436457830415645 (+0.00043918654112748423) | > avg_loss_0: 2.7078399624623044 (+0.0668340736711528) | > avg_grad_norm_0: tensor(327.4529, device='cuda:0') (+tensor(16.3902, device='cuda:0')) | > avg_loss_gen: 2.151349166749228 (-0.018750541646716634) | > avg_loss_kl: 2.0357995369064983 (-0.41749310325568745) | > avg_loss_feat: 3.7708343418551165 (-0.15826567797593638) | > avg_loss_mel: 16.815241249514298 (-2.084074880035832) | > avg_loss_duration: 1.5425965282279002 (-0.08790817898763703) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.315820828290057 (-2.766492198890358) | > avg_grad_norm_1: tensor(6280.5952, device='cuda:0') (tensor(-1371.0684, device='cuda:0'))  > EPOCH: 1451/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:09:30)   --> TIME: 2025-02-25 23:09:44 -- STEP: 27/72 -- GLOBAL_STEP: 1104500 | > loss_disc: 2.6785895824432373 (2.654301634541264) | > loss_disc_real_0: 0.25406962633132935 (0.1269448726541466) | > loss_disc_real_1: 0.19909875094890594 (0.22740136252509224) | > loss_disc_real_2: 0.22597740590572357 (0.24170965728936372) | > loss_disc_real_3: 0.20844760537147522 (0.23108735846148598) | > loss_disc_real_4: 0.18218612670898438 (0.2469442431573514) | > loss_disc_real_5: 0.2455618977546692 (0.252836486255681) | > loss_0: 2.6785895824432373 (2.654301634541264) | > grad_norm_0: tensor(285.9739, device='cuda:0') (tensor(296.5317, device='cuda:0')) | > loss_gen: 2.3106956481933594 (2.115563931288542) | > loss_kl: 1.9979174137115479 (1.902722160021464) | > loss_feat: 4.089348793029785 (3.9923245730223478) | > loss_mel: 16.715808868408203 (16.746178485729075) | > loss_duration: 1.497750163078308 (1.4906813127023202) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.61151885986328 (26.24747014928747) | > grad_norm_1: tensor(5267.3687, device='cuda:0') (tensor(7077.4072, device='cuda:0')) | > current_lr_0: 0.00016682260956102469 | > current_lr_1: 0.00016682260956102469 | > step_time: 0.4533 (0.4338436833134404) | > loader_time: 0.0077 (0.0067958831787109375) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568125711360447 (-0.00019699419048470508) | > avg_step_time: 0.46267341560041403 (+0.011544640635100911) | > avg_loss_disc: 2.653662916640161 (-0.054177045822143555) | > avg_loss_disc_real_0: 0.1606189116940532 (+0.0128096933411041) | > avg_loss_disc_real_1: 0.22484328965066183 (-0.06685429832465214) | > avg_loss_disc_real_2: 0.240358163353423 (-0.004812363797510183) | > avg_loss_disc_real_3: 0.22870890565321478 (+0.010862903695710968) | > avg_loss_disc_real_4: 0.2418838814950325 (-0.0017601018640356902) | > avg_loss_disc_real_5: 0.24846424980902326 (+0.004818466767458757) | > avg_loss_0: 2.653662916640161 (-0.054177045822143555) | > avg_grad_norm_0: tensor(289.7792, device='cuda:0') (tensor(-37.6737, device='cuda:0')) | > avg_loss_gen: 2.0814436959548748 (-0.06990547079435316) | > avg_loss_kl: 1.9776357429128297 (-0.058163793993668556) | > avg_loss_feat: 3.737008917499596 (-0.03382542435552027) | > avg_loss_mel: 16.664601366284877 (-0.15063988322942024) | > avg_loss_duration: 1.530335545539856 (-0.012260982688044209) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.99102511876066 (-0.32479570952939696) | > avg_grad_norm_1: tensor(6694.5591, device='cuda:0') (+tensor(413.9639, device='cuda:0'))  > EPOCH: 1452/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:10:07)   --> TIME: 2025-02-25 23:10:11 -- STEP: 5/72 -- GLOBAL_STEP: 1104550 | > loss_disc: 2.4093897342681885 (2.6526432991027833) | > loss_disc_real_0: 0.15287494659423828 (0.28675360679626466) | > loss_disc_real_1: 0.2393142580986023 (0.24319524466991424) | > loss_disc_real_2: 0.2503204345703125 (0.2514610171318054) | > loss_disc_real_3: 0.244438037276268 (0.26124952733516693) | > loss_disc_real_4: 0.25547757744789124 (0.2510441571474075) | > loss_disc_real_5: 0.2474004477262497 (0.25088785886764525) | > loss_0: 2.4093897342681885 (2.6526432991027833) | > grad_norm_0: tensor(328.7703, device='cuda:0') (tensor(381.9695, device='cuda:0')) | > loss_gen: 2.5023727416992188 (2.390834355354309) | > loss_kl: 1.8384870290756226 (1.8087064981460572) | > loss_feat: 4.171600341796875 (3.94819712638855) | > loss_mel: 16.492643356323242 (16.715515899658204) | > loss_duration: 1.4670597314834595 (1.4897985219955445) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.472164154052734 (26.353052139282227) | > grad_norm_1: tensor(3222.4243, device='cuda:0') (tensor(4379.0562, device='cuda:0')) | > current_lr_0: 0.00016680175673482956 | > current_lr_1: 0.00016680175673482956 | > step_time: 0.4347 (0.4347411632537842) | > loader_time: 0.0056 (0.006420516967773437)  --> TIME: 2025-02-25 23:10:34 -- STEP: 55/72 -- GLOBAL_STEP: 1104600 | > loss_disc: 2.804218292236328 (2.640939794887196) | > loss_disc_real_0: 0.06727124005556107 (0.13860307525504717) | > loss_disc_real_1: 0.19818295538425446 (0.22263737185434862) | > loss_disc_real_2: 0.2787986099720001 (0.2364410332658074) | > loss_disc_real_3: 0.2849940061569214 (0.23704578253355893) | > loss_disc_real_4: 0.2702830135822296 (0.24191365296190434) | > loss_disc_real_5: 0.2662871479988098 (0.2424204856157303) | > loss_0: 2.804218292236328 (2.640939794887196) | > grad_norm_0: tensor(310.6352, device='cuda:0') (tensor(273.1390, device='cuda:0')) | > loss_gen: 2.009021282196045 (2.0965303637764667) | > loss_kl: 2.1352367401123047 (2.000915024497292) | > loss_feat: 3.4427263736724854 (3.9339472857388595) | > loss_mel: 16.47195053100586 (16.763104352084074) | > loss_duration: 1.5903379917144775 (1.5200231248682194) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.649272918701172 (26.314520159634675) | > grad_norm_1: tensor(11988.8389, device='cuda:0') (tensor(7933.1338, device='cuda:0')) | > current_lr_0: 0.00016680175673482956 | > current_lr_1: 0.00016680175673482956 | > step_time: 0.4629 (0.44601713960820977) | > loader_time: 0.0093 (0.007887194373390893) --> EVAL PERFORMANCE | > avg_loader_time: 0.008616494460844662 (+4.8368749484214296e-05) | > avg_step_time: 0.45440987130285987 (-0.00826354429755416) | > avg_loss_disc: 2.662611739736208 (+0.008948823096047054) | > avg_loss_disc_real_0: 0.15130747141132891 (-0.009311440282724293) | > avg_loss_disc_real_1: 0.2229038073982991 (-0.0019394822523627342) | > avg_loss_disc_real_2: 0.2404047924448067 (+4.662909138369864e-05) | > avg_loss_disc_real_3: 0.2364177449908055 (+0.007708839337590728) | > avg_loss_disc_real_4: 0.24407932057347095 (+0.0021954390784384437) | > avg_loss_disc_real_5: 0.2446842136937128 (-0.00378003611531047) | > avg_loss_0: 2.662611739736208 (+0.008948823096047054) | > avg_grad_norm_0: tensor(280.0779, device='cuda:0') (tensor(-9.7012, device='cuda:0')) | > avg_loss_gen: 2.0851981488751696 (+0.0037544529202948596) | > avg_loss_kl: 2.0239020586013785 (+0.046266315688548776) | > avg_loss_feat: 3.8482976295578655 (+0.11128871205826929) | > avg_loss_mel: 16.703436851501465 (+0.03883548521658753) | > avg_loss_duration: 1.532879354248584 (+0.002543808708727946) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.19371406125351 (+0.2026889424928484) | > avg_grad_norm_1: tensor(7587.8687, device='cuda:0') (+tensor(893.3096, device='cuda:0'))  > EPOCH: 1453/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:10:44)   --> TIME: 2025-02-25 23:11:01 -- STEP: 33/72 -- GLOBAL_STEP: 1104650 | > loss_disc: 2.7884585857391357 (2.6384760899977255) | > loss_disc_real_0: 0.04230194911360741 (0.1341665199760235) | > loss_disc_real_1: 0.26293855905532837 (0.22688206669056055) | > loss_disc_real_2: 0.22265975177288055 (0.23068089647726578) | > loss_disc_real_3: 0.23869580030441284 (0.24062080907099176) | > loss_disc_real_4: 0.21878357231616974 (0.23914384570988742) | > loss_disc_real_5: 0.25396013259887695 (0.24368550651001208) | > loss_0: 2.7884585857391357 (2.6384760899977255) | > grad_norm_0: tensor(469.5596, device='cuda:0') (tensor(268.6392, device='cuda:0')) | > loss_gen: 1.679257869720459 (2.1336676279703775) | > loss_kl: 2.357046127319336 (2.355561057726542) | > loss_feat: 4.041269302368164 (4.039259679389723) | > loss_mel: 17.533823013305664 (18.1470317551584) | > loss_duration: 1.5354019403457642 (1.5854622270121719) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.146799087524414 (28.260982397830848) | > grad_norm_1: tensor(10833.1914, device='cuda:0') (tensor(6702.2568, device='cuda:0')) | > current_lr_0: 0.0001667809065152377 | > current_lr_1: 0.0001667809065152377 | > step_time: 0.4375 (0.43343866232669714) | > loader_time: 0.0076 (0.007030357014049183) --> EVAL PERFORMANCE | > avg_loader_time: 0.008580473107351383 (-3.6021353493278715e-05) | > avg_step_time: 0.4512121039377132 (-0.003197767365146653) | > avg_loss_disc: 2.630697932041867 (-0.031913807694341045) | > avg_loss_disc_real_0: 0.11756194524571932 (-0.03374552616560959) | > avg_loss_disc_real_1: 0.23236549866031592 (+0.00946169126201682) | > avg_loss_disc_real_2: 0.22836527941932142 (-0.012039513025485282) | > avg_loss_disc_real_3: 0.23834360275470035 (+0.0019258577638948393) | > avg_loss_disc_real_4: 0.2361654328628325 (-0.007913887710638445) | > avg_loss_disc_real_5: 0.24281116081795223 (-0.0018730528757605613) | > avg_loss_0: 2.630697932041867 (-0.031913807694341045) | > avg_grad_norm_0: tensor(251.3970, device='cuda:0') (tensor(-28.6809, device='cuda:0')) | > avg_loss_gen: 2.1150302400051717 (+0.02983209113000207) | > avg_loss_kl: 2.2423188602420647 (+0.21841680164068622) | > avg_loss_feat: 3.9490363698610116 (+0.10073874030314611) | > avg_loss_mel: 17.5009653198887 (+0.7975284683872346) | > avg_loss_duration: 1.5785547444518184 (+0.04567539020323452) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.385905574744857 (+1.1921915134913483) | > avg_grad_norm_1: tensor(6557.2056, device='cuda:0') (tensor(-1030.6631, device='cuda:0'))  > EPOCH: 1454/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:11:21)   --> TIME: 2025-02-25 23:11:27 -- STEP: 11/72 -- GLOBAL_STEP: 1104700 | > loss_disc: 2.4406652450561523 (2.552537376230413) | > loss_disc_real_0: 0.07751673460006714 (0.10667698153040626) | > loss_disc_real_1: 0.23588894307613373 (0.2382041270082647) | > loss_disc_real_2: 0.24980761110782623 (0.23718395693735642) | > loss_disc_real_3: 0.24485129117965698 (0.23626570809971204) | > loss_disc_real_4: 0.24744045734405518 (0.23821828175674786) | > loss_disc_real_5: 0.2533045709133148 (0.25422037866982544) | > loss_0: 2.4406652450561523 (2.552537376230413) | > grad_norm_0: tensor(125.7994, device='cuda:0') (tensor(232.3708, device='cuda:0')) | > loss_gen: 2.3259646892547607 (2.2999880097129126) | > loss_kl: 1.980514407157898 (1.9998803788965398) | > loss_feat: 4.148622035980225 (4.173386313698509) | > loss_mel: 16.683691024780273 (16.759040225635875) | > loss_duration: 1.4730207920074463 (1.4878380406986584) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.611812591552734 (26.720132654363457) | > grad_norm_1: tensor(5362.0098, device='cuda:0') (tensor(5731.6040, device='cuda:0')) | > current_lr_0: 0.0001667600589019233 | > current_lr_1: 0.0001667600589019233 | > step_time: 0.4314 (0.42805706370960583) | > loader_time: 0.0059 (0.0064018639651211824)  --> TIME: 2025-02-25 23:11:51 -- STEP: 61/72 -- GLOBAL_STEP: 1104750 | > loss_disc: 2.816676616668701 (2.5478903465583675) | > loss_disc_real_0: 0.26060420274734497 (0.12275896196971174) | > loss_disc_real_1: 0.2902587056159973 (0.22395054385310315) | > loss_disc_real_2: 0.2662222683429718 (0.23299593954789835) | > loss_disc_real_3: 0.24815846979618073 (0.23133770755079927) | > loss_disc_real_4: 0.2674032151699066 (0.2357125497255169) | > loss_disc_real_5: 0.2509014308452606 (0.2477993501014397) | > loss_0: 2.816676616668701 (2.5478903465583675) | > grad_norm_0: tensor(636.4908, device='cuda:0') (tensor(216.0882, device='cuda:0')) | > loss_gen: 2.1135075092315674 (2.2783207991084113) | > loss_kl: 2.1803698539733887 (2.059565079016763) | > loss_feat: 2.8997859954833984 (4.108814880496164) | > loss_mel: 16.57432746887207 (16.715545279080754) | > loss_duration: 1.5855008363723755 (1.522189064104049) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.353492736816406 (26.68443492201508) | > grad_norm_1: tensor(3290.5491, device='cuda:0') (tensor(5563.5669, device='cuda:0')) | > current_lr_0: 0.0001667600589019233 | > current_lr_1: 0.0001667600589019233 | > step_time: 0.4575 (0.45157638143320555) | > loader_time: 0.0108 (0.008189193537977875) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653009441536916 (+7.25363341855332e-05) | > avg_step_time: 0.45966635959249147 (+0.008454255654778253) | > avg_loss_disc: 2.5766319926355927 (-0.05406593940627413) | > avg_loss_disc_real_0: 0.12947077010299118 (+0.011908824857271863) | > avg_loss_disc_real_1: 0.2309215226223771 (-0.0014439760379388045) | > avg_loss_disc_real_2: 0.2343900140742181 (+0.006024734654896696) | > avg_loss_disc_real_3: 0.23463334457975038 (-0.0037102581749499652) | > avg_loss_disc_real_4: 0.23933403068025347 (+0.0031685978174209595) | > avg_loss_disc_real_5: 0.24791542241271114 (+0.005104261594758908) | > avg_loss_0: 2.5766319926355927 (-0.05406593940627413) | > avg_grad_norm_0: tensor(229.4826, device='cuda:0') (tensor(-21.9144, device='cuda:0')) | > avg_loss_gen: 2.2386731312308514 (+0.12364289122567973) | > avg_loss_kl: 2.0894729923194553 (-0.15284586792260946) | > avg_loss_feat: 4.011454192685409 (+0.06241782282439701) | > avg_loss_mel: 16.773162250787443 (-0.727803069101256) | > avg_loss_duration: 1.5331582233939371 (-0.04539652105788128) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.645920659454777 (-0.7399849152900799) | > avg_grad_norm_1: tensor(6008.2734, device='cuda:0') (tensor(-548.9321, device='cuda:0'))  > EPOCH: 1455/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:11:58)   --> TIME: 2025-02-25 23:12:18 -- STEP: 39/72 -- GLOBAL_STEP: 1104800 | > loss_disc: 2.5844407081604004 (2.5936309496561685) | > loss_disc_real_0: 0.16427671909332275 (0.1134909015053358) | > loss_disc_real_1: 0.23431670665740967 (0.21950576970210442) | > loss_disc_real_2: 0.18808455765247345 (0.21776934159107697) | > loss_disc_real_3: 0.2216174453496933 (0.23209401544852135) | > loss_disc_real_4: 0.25257882475852966 (0.2381997918471312) | > loss_disc_real_5: 0.24439634382724762 (0.2410917049035048) | > loss_0: 2.5844407081604004 (2.5936309496561685) | > grad_norm_0: tensor(27.9333, device='cuda:0') (tensor(266.5557, device='cuda:0')) | > loss_gen: 2.598351001739502 (2.1010461159241505) | > loss_kl: 3.01301908493042 (2.1291145667051663) | > loss_feat: 4.063204288482666 (4.0633246042789555) | > loss_mel: 19.423433303833008 (17.188869207333287) | > loss_duration: 1.6103832721710205 (1.5056212376325557) | > amp_scaler: 8.0 (8.0) | > loss_1: 30.708391189575195 (26.987975927499626) | > grad_norm_1: tensor(5543.0693, device='cuda:0') (tensor(9304.0732, device='cuda:0')) | > current_lr_0: 0.00016673921389456057 | > current_lr_1: 0.00016673921389456057 | > step_time: 0.454 (0.4397116868923872) | > loader_time: 0.0083 (0.007341079222850311) --> EVAL PERFORMANCE | > avg_loader_time: 0.008686391400619293 (+3.338195908237675e-05) | > avg_step_time: 0.4536344199113443 (-0.00603193968114718) | > avg_loss_disc: 2.614756896462239 (+0.03812490382664624) | > avg_loss_disc_real_0: 0.12577235310430257 (-0.003698416998688614) | > avg_loss_disc_real_1: 0.22852675259952815 (-0.0023947700228489632) | > avg_loss_disc_real_2: 0.23174659970780492 (-0.002643414366413188) | > avg_loss_disc_real_3: 0.23171417708967773 (-0.0029191674900726494) | > avg_loss_disc_real_4: 0.23665791561066266 (-0.002676115069590812) | > avg_loss_disc_real_5: 0.24008332633636367 (-0.007832096076347467) | > avg_loss_0: 2.614756896462239 (+0.03812490382664624) | > avg_grad_norm_0: tensor(273.7487, device='cuda:0') (+tensor(44.2661, device='cuda:0')) | > avg_loss_gen: 2.134978020694894 (-0.10369511053595737) | > avg_loss_kl: 2.701526507525377 (+0.6120535152059219) | > avg_loss_feat: 4.068566765583738 (+0.057112572898329717) | > avg_loss_mel: 19.12723615135945 (+2.3540739005720077) | > avg_loss_duration: 1.6243731505434273 (+0.0912149271494902) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 29.656680711558167 (+3.0107600521033895) | > avg_grad_norm_1: tensor(7980.8184, device='cuda:0') (+tensor(1972.5449, device='cuda:0'))  > EPOCH: 1456/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:12:35)   --> TIME: 2025-02-25 23:12:44 -- STEP: 17/72 -- GLOBAL_STEP: 1104850 | > loss_disc: 2.7342777252197266 (2.7440908796647014) | > loss_disc_real_0: 0.27997198700904846 (0.09511964439469225) | > loss_disc_real_1: 0.25160375237464905 (0.27538668846382813) | > loss_disc_real_2: 0.1874333620071411 (0.21962288810926325) | > loss_disc_real_3: 0.2211260050535202 (0.25423893507789164) | > loss_disc_real_4: 0.2706282436847687 (0.1988324260010439) | > loss_disc_real_5: 0.26387372612953186 (0.2303028273231843) | > loss_0: 2.7342777252197266 (2.7440908796647014) | > grad_norm_0: tensor(464.3743, device='cuda:0') (tensor(242.7349, device='cuda:0')) | > loss_gen: 2.0474724769592285 (2.032796621322632) | > loss_kl: 2.0357704162597656 (2.161061076556935) | > loss_feat: 3.366441011428833 (3.935274166219375) | > loss_mel: 18.92243003845215 (17.89851480371812) | > loss_duration: 1.5409120321273804 (1.5243963353774126) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.913026809692383 (27.552042792825137) | > grad_norm_1: tensor(6776.4170, device='cuda:0') (tensor(8075.0127, device='cuda:0')) | > current_lr_0: 0.00016671837149282375 | > current_lr_1: 0.00016671837149282375 | > step_time: 0.4374 (0.433945894241333) | > loader_time: 0.0061 (0.006519065183751723)  --> TIME: 2025-02-25 23:13:08 -- STEP: 67/72 -- GLOBAL_STEP: 1104900 | > loss_disc: 2.7942121028900146 (2.652528741466466) | > loss_disc_real_0: 0.46126309037208557 (0.14447695845321037) | > loss_disc_real_1: 0.32297345995903015 (0.23831040868118622) | > loss_disc_real_2: 0.27703461050987244 (0.22543604249384866) | > loss_disc_real_3: 0.26834726333618164 (0.2332810655013839) | > loss_disc_real_4: 0.2583632171154022 (0.2319946609326263) | > loss_disc_real_5: 0.22026100754737854 (0.24518057214680003) | > loss_0: 2.7942121028900146 (2.652528741466466) | > grad_norm_0: tensor(326.8720, device='cuda:0') (tensor(263.2328, device='cuda:0')) | > loss_gen: 3.142491102218628 (2.1566680651992116) | > loss_kl: 2.0730514526367188 (2.111268803254881) | > loss_feat: 3.479050636291504 (3.8193053665445813) | > loss_mel: 16.3076171875 (17.241298405092156) | > loss_duration: 1.6172995567321777 (1.5506674656227453) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.619508743286133 (26.879208095038116) | > grad_norm_1: tensor(1839.9381, device='cuda:0') (tensor(8497.6094, device='cuda:0')) | > current_lr_0: 0.00016671837149282375 | > current_lr_1: 0.00016671837149282375 | > step_time: 0.5175 (0.4496052621015862) | > loader_time: 0.0119 (0.008369659309956565) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854310519258741 (-0.0001432862080318828) | > avg_step_time: 0.4530440720034317 (-0.0005903479079125695) | > avg_loss_disc: 2.6732462191245925 (+0.058489322662353516) | > avg_loss_disc_real_0: 0.16688563942279608 (+0.04111328631849351) | > avg_loss_disc_real_1: 0.2451235517649583 (+0.016596799165430137) | > avg_loss_disc_real_2: 0.22926435965887257 (-0.002482240048932355) | > avg_loss_disc_real_3: 0.23727868539346775 (+0.005564508303790017) | > avg_loss_disc_real_4: 0.23407036779632032 (-0.002587547814342339) | > avg_loss_disc_real_5: 0.24445138158093035 (+0.004368055244566682) | > avg_loss_0: 2.6732462191245925 (+0.058489322662353516) | > avg_grad_norm_0: tensor(271.8489, device='cuda:0') (tensor(-1.8998, device='cuda:0')) | > avg_loss_gen: 2.2075770203496368 (+0.0725989996547427) | > avg_loss_kl: 2.110632280228842 (-0.590894227296535) | > avg_loss_feat: 3.79876515898906 (-0.2698016065946782) | > avg_loss_mel: 17.212835029816965 (-1.914401121542486) | > avg_loss_duration: 1.5530930421721767 (-0.07128010837125065) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.882902521482656 (-2.773778190075511) | > avg_grad_norm_1: tensor(8124.8784, device='cuda:0') (+tensor(144.0601, device='cuda:0'))  > EPOCH: 1457/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:13:12)   --> TIME: 2025-02-25 23:13:34 -- STEP: 45/72 -- GLOBAL_STEP: 1104950 | > loss_disc: 2.516263484954834 (2.7094768736097548) | > loss_disc_real_0: 0.3050422668457031 (0.2024937913649612) | > loss_disc_real_1: 0.21006126701831818 (0.23006511198149787) | > loss_disc_real_2: 0.2460440695285797 (0.2344145874182383) | > loss_disc_real_3: 0.2713282108306885 (0.23358873029549918) | > loss_disc_real_4: 0.22786474227905273 (0.2512367294894324) | > loss_disc_real_5: 0.2744539976119995 (0.23830354544851515) | > loss_0: 2.516263484954834 (2.7094768736097548) | > grad_norm_0: tensor(505.8116, device='cuda:0') (tensor(230.6781, device='cuda:0')) | > loss_gen: 2.6626827716827393 (2.077362982432047) | > loss_kl: 1.974291205406189 (1.9734744177924262) | > loss_feat: 3.9953837394714355 (3.7577389081319175) | > loss_mel: 17.17824935913086 (16.83985790676541) | > loss_duration: 1.554235577583313 (1.5097729576958554) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.364843368530273 (26.158207024468314) | > grad_norm_1: tensor(3189.2295, device='cuda:0') (tensor(6158.8945, device='cuda:0')) | > current_lr_0: 0.00016669753169638713 | > current_lr_1: 0.00016669753169638713 | > step_time: 0.4808 (0.45121537314520943) | > loader_time: 0.0094 (0.007678922017415365) --> EVAL PERFORMANCE | > avg_loader_time: 0.008729988420513314 (+0.00018688322792590438) | > avg_step_time: 0.45915117062313454 (+0.006107098619702822) | > avg_loss_disc: 2.6732977511177602 (+5.15319931677638e-05) | > avg_loss_disc_real_0: 0.1941779463643759 (+0.02729230694157983) | > avg_loss_disc_real_1: 0.22486097544011935 (-0.020262576324838938) | > avg_loss_disc_real_2: 0.2320442686618214 (+0.0027799090029488327) | > avg_loss_disc_real_3: 0.23435453524891758 (-0.002924150144550164) | > avg_loss_disc_real_4: 0.2407083435797355 (+0.006637975783415184) | > avg_loss_disc_real_5: 0.2443262710117958 (-0.00012511056913455665) | > avg_loss_0: 2.6732977511177602 (+5.15319931677638e-05) | > avg_grad_norm_0: tensor(232.3263, device='cuda:0') (tensor(-39.5225, device='cuda:0')) | > avg_loss_gen: 2.126584543308742 (-0.08099247704089496) | > avg_loss_kl: 2.016010447287224 (-0.09462183294161841) | > avg_loss_feat: 3.80124698222523 (+0.0024818232361698556) | > avg_loss_mel: 16.863606399213765 (-0.3492286306032) | > avg_loss_duration: 1.5346523472960567 (-0.01844069487611999) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.342100707577988 (-0.5408018139046682) | > avg_grad_norm_1: tensor(5730.8550, device='cuda:0') (tensor(-2394.0234, device='cuda:0'))  > EPOCH: 1458/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:13:49)   --> TIME: 2025-02-25 23:14:01 -- STEP: 23/72 -- GLOBAL_STEP: 1105000 | > loss_disc: 2.7715444564819336 (2.5761668578438135) | > loss_disc_real_0: 0.14280852675437927 (0.12938076184819575) | > loss_disc_real_1: 0.17324459552764893 (0.22216426807901132) | > loss_disc_real_2: 0.24279718101024628 (0.23376423703587573) | > loss_disc_real_3: 0.25311270356178284 (0.2299760735553244) | > loss_disc_real_4: 0.25178369879722595 (0.22984144221181455) | > loss_disc_real_5: 0.25857219099998474 (0.24611064986042355) | > loss_0: 2.7715444564819336 (2.5761668578438135) | > grad_norm_0: tensor(173.9807, device='cuda:0') (tensor(225.5733, device='cuda:0')) | > loss_gen: 1.7649753093719482 (2.1982686104981792) | > loss_kl: 2.3821306228637695 (2.0252301952113276) | > loss_feat: 3.6320717334747314 (4.068567472955458) | > loss_mel: 18.734106063842773 (17.31369690273119) | > loss_duration: 1.5959763526916504 (1.5041273573170537) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.1092586517334 (27.109890481700067) | > grad_norm_1: tensor(3537.9846, device='cuda:0') (tensor(5466.8379, device='cuda:0')) | > current_lr_0: 0.00016667669450492507 | > current_lr_1: 0.00016667669450492507 | > step_time: 0.4361 (0.4329200620236604) | > loader_time: 0.0075 (0.006773420002149499) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1105000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008565093429995254 (-0.00016489499051806043) | > avg_step_time: 0.45215904208975777 (-0.0069921285333767735) | > avg_loss_disc: 2.6412962490404155 (-0.03200150207734476) | > avg_loss_disc_real_0: 0.13675566786296767 (-0.05742227850140824) | > avg_loss_disc_real_1: 0.23555882778805745 (+0.010697852347938103) | > avg_loss_disc_real_2: 0.23773127878215952 (+0.005687010120338121) | > avg_loss_disc_real_3: 0.2339817740967576 (-0.0003727611521599783) | > avg_loss_disc_real_4: 0.24014538064808913 (-0.0005629629316463713) | > avg_loss_disc_real_5: 0.24616233710671814 (+0.001836066094922345) | > avg_loss_0: 2.6412962490404155 (-0.03200150207734476) | > avg_grad_norm_0: tensor(232.6961, device='cuda:0') (+tensor(0.3697, device='cuda:0')) | > avg_loss_gen: 2.162262563974085 (+0.03567802066534309) | > avg_loss_kl: 2.176491915340155 (+0.1604814680529314) | > avg_loss_feat: 3.913445385409074 (+0.11219840318384389) | > avg_loss_mel: 17.632544436924896 (+0.7689380377111306) | > avg_loss_duration: 1.561937840891556 (+0.027285493595499366) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.44668224495901 (+1.1045815373810228) | > avg_grad_norm_1: tensor(6316.4263, device='cuda:0') (+tensor(585.5713, device='cuda:0'))  > EPOCH: 1459/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:14:28)   --> TIME: 2025-02-25 23:14:31 -- STEP: 1/72 -- GLOBAL_STEP: 1105050 | > loss_disc: 2.839411973953247 (2.839411973953247) | > loss_disc_real_0: 0.10196641832590103 (0.10196641832590103) | > loss_disc_real_1: 0.22040100395679474 (0.22040100395679474) | > loss_disc_real_2: 0.48098620772361755 (0.48098620772361755) | > loss_disc_real_3: 0.2469249963760376 (0.2469249963760376) | > loss_disc_real_4: 0.2666911780834198 (0.2666911780834198) | > loss_disc_real_5: 0.2872561514377594 (0.2872561514377594) | > loss_0: 2.839411973953247 (2.839411973953247) | > grad_norm_0: tensor(201.1621, device='cuda:0') (tensor(201.1621, device='cuda:0')) | > loss_gen: 2.0807971954345703 (2.0807971954345703) | > loss_kl: 2.1479203701019287 (2.1479203701019287) | > loss_feat: 3.451892614364624 (3.451892614364624) | > loss_mel: 19.10137939453125 (19.10137939453125) | > loss_duration: 1.538809895515442 (1.538809895515442) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.32080078125 (28.32080078125) | > grad_norm_1: tensor(3175.4021, device='cuda:0') (tensor(3175.4021, device='cuda:0')) | > current_lr_0: 0.00016665585991811196 | > current_lr_1: 0.00016665585991811196 | > step_time: 0.4331 (0.4330744743347168) | > loader_time: 0.0072 (0.007237911224365234)  --> TIME: 2025-02-25 23:14:54 -- STEP: 51/72 -- GLOBAL_STEP: 1105100 | > loss_disc: 2.715125322341919 (2.7356230184143664) | > loss_disc_real_0: 0.1570511758327484 (0.1943367531559631) | > loss_disc_real_1: 0.21658433973789215 (0.23272850671235254) | > loss_disc_real_2: 0.23766553401947021 (0.240600673883569) | > loss_disc_real_3: 0.24054975807666779 (0.23396538694699606) | > loss_disc_real_4: 0.2406003773212433 (0.24261607580325176) | > loss_disc_real_5: 0.23274996876716614 (0.24885585524287882) | > loss_0: 2.715125322341919 (2.7356230184143664) | > grad_norm_0: tensor(98.5749, device='cuda:0') (tensor(134.5412, device='cuda:0')) | > loss_gen: 1.839442253112793 (2.0377177257163837) | > loss_kl: 2.164933919906616 (2.236839009266274) | > loss_feat: 3.1065123081207275 (3.6646754414427516) | > loss_mel: 16.670265197753906 (18.149134280634865) | > loss_duration: 1.584421157836914 (1.572656746004142) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.365575790405273 (27.66102304645613) | > grad_norm_1: tensor(205.4491, device='cuda:0') (tensor(2809.0344, device='cuda:0')) | > current_lr_0: 0.00016665585991811196 | > current_lr_1: 0.00016665585991811196 | > step_time: 0.4481 (0.45208050690445245) | > loader_time: 0.0094 (0.007712046305338542) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863464449492978 (+6.95510649345258e-05) | > avg_step_time: 0.45815122631234184 (+0.005992184222584074) | > avg_loss_disc: 2.7200178529175236 (+0.07872160387710814) | > avg_loss_disc_real_0: 0.18439185270436212 (+0.04763618484139445) | > avg_loss_disc_real_1: 0.23668002830424778 (+0.0011212005161903293) | > avg_loss_disc_real_2: 0.2421166905215089 (+0.004385411739349365) | > avg_loss_disc_real_3: 0.23572184901002427 (+0.0017400749132666649) | > avg_loss_disc_real_4: 0.24345959030406572 (+0.003314209655976591) | > avg_loss_disc_real_5: 0.24962154009812315 (+0.003459202991405008) | > avg_loss_0: 2.7200178529175236 (+0.07872160387710814) | > avg_grad_norm_0: tensor(139.8764, device='cuda:0') (tensor(-92.8197, device='cuda:0')) | > avg_loss_gen: 2.073034830496343 (-0.08922773347774182) | > avg_loss_kl: 2.2710310036027934 (+0.09453908826263824) | > avg_loss_feat: 3.65825505323813 (-0.2551903321709439) | > avg_loss_mel: 18.12089003979321 (+0.4883456028683142) | > avg_loss_duration: 1.5835209530843815 (+0.021583112192825382) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.706731769400584 (+0.26004952444157325) | > avg_grad_norm_1: tensor(3748.4431, device='cuda:0') (tensor(-2567.9832, device='cuda:0'))  > EPOCH: 1460/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:15:06)   --> TIME: 2025-02-25 23:15:20 -- STEP: 29/72 -- GLOBAL_STEP: 1105150 | > loss_disc: 2.526637315750122 (2.607249169514097) | > loss_disc_real_0: 0.11896419525146484 (0.1572479430971475) | > loss_disc_real_1: 0.21191762387752533 (0.22036139410117578) | > loss_disc_real_2: 0.2394961714744568 (0.23129281915467362) | > loss_disc_real_3: 0.23979440331459045 (0.23176110407401776) | > loss_disc_real_4: 0.24758726358413696 (0.24299379566620136) | > loss_disc_real_5: 0.25888216495513916 (0.24210059077575288) | > loss_0: 2.526637315750122 (2.607249169514097) | > grad_norm_0: tensor(177.4673, device='cuda:0') (tensor(210.8364, device='cuda:0')) | > loss_gen: 2.053770065307617 (2.067181032279442) | > loss_kl: 1.7808911800384521 (1.9336919579012641) | > loss_feat: 3.5127291679382324 (3.7458396122373383) | > loss_mel: 16.73324966430664 (17.324022227320178) | > loss_duration: 1.4952529668807983 (1.501578742060168) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.57589340209961 (26.572313900651604) | > grad_norm_1: tensor(9881.5479, device='cuda:0') (tensor(6189.0371, device='cuda:0')) | > current_lr_0: 0.00016663502793562218 | > current_lr_1: 0.00016663502793562218 | > step_time: 0.439 (0.43557819826849575) | > loader_time: 0.0078 (0.006988393849339978) --> EVAL PERFORMANCE | > avg_loader_time: 0.008651951668967666 (+1.7307174037885925e-05) | > avg_step_time: 0.45215311520536183 (-0.005998111106980009) | > avg_loss_disc: 2.579708519116254 (-0.14030933380126953) | > avg_loss_disc_real_0: 0.1325310270866038 (-0.051860825617758316) | > avg_loss_disc_real_1: 0.23077413061974753 (-0.005905897684500255) | > avg_loss_disc_real_2: 0.23001585376094763 (-0.012100836760561257) | > avg_loss_disc_real_3: 0.2294387269607732 (-0.0062831220492510775) | > avg_loss_disc_real_4: 0.24088737893272455 (-0.002572211371341171) | > avg_loss_disc_real_5: 0.24287232588714278 (-0.0067492142109803754) | > avg_loss_0: 2.579708519116254 (-0.14030933380126953) | > avg_grad_norm_0: tensor(186.8408, device='cuda:0') (+tensor(46.9645, device='cuda:0')) | > avg_loss_gen: 2.11020970176643 (+0.03717487127008701) | > avg_loss_kl: 2.004951079126815 (-0.2660799244759784) | > avg_loss_feat: 3.828471798292348 (+0.17021674505421824) | > avg_loss_mel: 16.90461765880317 (-1.2162723809900413) | > avg_loss_duration: 1.537585501939478 (-0.045935451144903494) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.385835943087724 (-1.32089582631286) | > avg_grad_norm_1: tensor(5610.0562, device='cuda:0') (+tensor(1861.6130, device='cuda:0'))  > EPOCH: 1461/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:15:42)   --> TIME: 2025-02-25 23:15:47 -- STEP: 7/72 -- GLOBAL_STEP: 1105200 | > loss_disc: 2.5952951908111572 (2.6153263364519392) | > loss_disc_real_0: 0.2454744577407837 (0.11160419508814812) | > loss_disc_real_1: 0.2405189722776413 (0.21330119456563676) | > loss_disc_real_2: 0.22640621662139893 (0.23420565043176925) | > loss_disc_real_3: 0.24593131244182587 (0.2442213339465005) | > loss_disc_real_4: 0.26072630286216736 (0.24179520990167344) | > loss_disc_real_5: 0.24342411756515503 (0.2400879923786436) | > loss_0: 2.5952951908111572 (2.6153263364519392) | > grad_norm_0: tensor(48.3516, device='cuda:0') (tensor(180.6909, device='cuda:0')) | > loss_gen: 2.1398844718933105 (2.1112375940595354) | > loss_kl: 1.928307056427002 (1.8792546817234583) | > loss_feat: 3.491950750350952 (3.8983102525983537) | > loss_mel: 16.86102294921875 (16.953396933419363) | > loss_duration: 1.4707014560699463 (1.5028966324669975) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.89186668395996 (26.345096043178014) | > grad_norm_1: tensor(8874.6787, device='cuda:0') (tensor(8806.2461, device='cuda:0')) | > current_lr_0: 0.00016661419855713022 | > current_lr_1: 0.00016661419855713022 | > step_time: 0.4324 (0.43008467129298616) | > loader_time: 0.0059 (0.006699698311941964)  --> TIME: 2025-02-25 23:16:10 -- STEP: 57/72 -- GLOBAL_STEP: 1105250 | > loss_disc: 2.5013790130615234 (2.62126098181072) | > loss_disc_real_0: 0.06673891842365265 (0.13141877955773426) | > loss_disc_real_1: 0.23577144742012024 (0.2539079385368447) | > loss_disc_real_2: 0.30602580308914185 (0.2369508230895327) | > loss_disc_real_3: 0.24512824416160583 (0.23772191321640684) | > loss_disc_real_4: 0.2518324851989746 (0.24351363040898977) | > loss_disc_real_5: 0.2614505887031555 (0.2455662635334751) | > loss_0: 2.5013790130615234 (2.62126098181072) | > grad_norm_0: tensor(185.6975, device='cuda:0') (tensor(193.2692, device='cuda:0')) | > loss_gen: 2.3603382110595703 (2.2205223242441816) | > loss_kl: 2.129080057144165 (2.003523774314345) | > loss_feat: 4.476804733276367 (3.921027145887676) | > loss_mel: 16.27044677734375 (17.11013517881694) | > loss_duration: 1.5787817239761353 (1.5243763212572061) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.81545066833496 (26.779584616945503) | > grad_norm_1: tensor(2822.4409, device='cuda:0') (tensor(6180.3491, device='cuda:0')) | > current_lr_0: 0.00016661419855713022 | > current_lr_1: 0.00016661419855713022 | > step_time: 0.4483 (0.441702219477871) | > loader_time: 0.0099 (0.008022308349609375) --> EVAL PERFORMANCE | > avg_loader_time: 0.008684752692638988 (+3.280102367132255e-05) | > avg_step_time: 0.45007112328435334 (-0.002081991921008497) | > avg_loss_disc: 2.6280596021195533 (+0.048351083003299156) | > avg_loss_disc_real_0: 0.12201378289872497 (-0.010517244187878833) | > avg_loss_disc_real_1: 0.24678444337676952 (+0.016010312757022) | > avg_loss_disc_real_2: 0.24812066429097887 (+0.01810481053003124) | > avg_loss_disc_real_3: 0.23327613779356782 (+0.0038374108327946244) | > avg_loss_disc_real_4: 0.24645715490193434 (+0.005569775969209795) | > avg_loss_disc_real_5: 0.24418529826150814 (+0.0013129723743653676) | > avg_loss_0: 2.6280596021195533 (+0.048351083003299156) | > avg_grad_norm_0: tensor(211.2395, device='cuda:0') (+tensor(24.3987, device='cuda:0')) | > avg_loss_gen: 2.19912898204696 (+0.08891928028053009) | > avg_loss_kl: 2.033823401155605 (+0.028872322028790087) | > avg_loss_feat: 3.9148401811089313 (+0.0863683828165831) | > avg_loss_mel: 17.035546934100942 (+0.13092927529777398) | > avg_loss_duration: 1.536397312728452 (-0.0011881892110259251) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.71973682457293 (+0.33390088148520647) | > avg_grad_norm_1: tensor(6141.3740, device='cuda:0') (+tensor(531.3179, device='cuda:0'))  > EPOCH: 1462/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:16:19)   --> TIME: 2025-02-25 23:16:37 -- STEP: 35/72 -- GLOBAL_STEP: 1105300 | > loss_disc: 2.7109856605529785 (2.7223965031760087) | > loss_disc_real_0: 0.21669140458106995 (0.16695764692766327) | > loss_disc_real_1: 0.20525439083576202 (0.22141378862517222) | > loss_disc_real_2: 0.27961841225624084 (0.240957054070064) | > loss_disc_real_3: 0.2425142079591751 (0.2459543662411826) | > loss_disc_real_4: 0.25023525953292847 (0.2364924384014947) | > loss_disc_real_5: 0.2653661072254181 (0.223124355503491) | > loss_0: 2.7109856605529785 (2.7223965031760087) | > grad_norm_0: tensor(78.1178, device='cuda:0') (tensor(218.8436, device='cuda:0')) | > loss_gen: 2.3801095485687256 (2.051291264806475) | > loss_kl: 2.1028926372528076 (1.9357479197638376) | > loss_feat: 3.799550771713257 (3.7937442030225483) | > loss_mel: 16.971111297607422 (16.70273745400565) | > loss_duration: 1.5480015277862549 (1.491949646813529) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.801666259765625 (25.97547013419015) | > grad_norm_1: tensor(5354.6123, device='cuda:0') (tensor(7631.3564, device='cuda:0')) | > current_lr_0: 0.00016659337178231057 | > current_lr_1: 0.00016659337178231057 | > step_time: 0.8875 (0.4474742344447545) | > loader_time: 0.0086 (0.0069530895778111045) --> EVAL PERFORMANCE | > avg_loader_time: 0.00843825138790507 (-0.0002465013047339183) | > avg_step_time: 0.4579886483474517 (+0.00791752506309834) | > avg_loss_disc: 2.6494575755696905 (+0.021397973450137275) | > avg_loss_disc_real_0: 0.1507728600376089 (+0.028759077138883926) | > avg_loss_disc_real_1: 0.22164569606243728 (-0.025138747314332244) | > avg_loss_disc_real_2: 0.23544777552006949 (-0.012672888770909385) | > avg_loss_disc_real_3: 0.23811153228014287 (+0.004835394486575051) | > avg_loss_disc_real_4: 0.23895494870736567 (-0.007502206194568678) | > avg_loss_disc_real_5: 0.23623845883658234 (-0.0079468394249258) | > avg_loss_0: 2.6494575755696905 (+0.021397973450137275) | > avg_grad_norm_0: tensor(217.8204, device='cuda:0') (+tensor(6.5808, device='cuda:0')) | > avg_loss_gen: 2.0951687634830733 (-0.10396021856388682) | > avg_loss_kl: 1.9876742413346196 (-0.04614915982098555) | > avg_loss_feat: 3.805100296584653 (-0.10973988452427808) | > avg_loss_mel: 16.62678584246569 (-0.4087610916352524) | > avg_loss_duration: 1.5285028655764084 (-0.007894447152043682) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.043231937247263 (-0.6765048873256667) | > avg_grad_norm_1: tensor(6492.7329, device='cuda:0') (+tensor(351.3589, device='cuda:0'))  > EPOCH: 1463/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:16:56)   --> TIME: 2025-02-25 23:17:04 -- STEP: 13/72 -- GLOBAL_STEP: 1105350 | > loss_disc: 2.4995951652526855 (2.4707155411060038) | > loss_disc_real_0: 0.0791238397359848 (0.09133075320949921) | > loss_disc_real_1: 0.18931268155574799 (0.22039359349470872) | > loss_disc_real_2: 0.2373550683259964 (0.22793479149158186) | > loss_disc_real_3: 0.21196596324443817 (0.23539777214710528) | > loss_disc_real_4: 0.2290409356355667 (0.23153970333246085) | > loss_disc_real_5: 0.24730879068374634 (0.2478798650778257) | > loss_0: 2.4995951652526855 (2.4707155411060038) | > grad_norm_0: tensor(351.4951, device='cuda:0') (tensor(235.7797, device='cuda:0')) | > loss_gen: 2.1500778198242188 (2.279358203594501) | > loss_kl: 1.755291223526001 (1.8705029579309316) | > loss_feat: 4.122962474822998 (4.280666076219999) | > loss_mel: 16.332237243652344 (16.70333018669715) | > loss_duration: 1.4264817237854004 (1.475718149772057) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.787050247192383 (26.60957571176382) | > grad_norm_1: tensor(8819.0547, device='cuda:0') (tensor(6130.8481, device='cuda:0')) | > current_lr_0: 0.00016657254761083778 | > current_lr_1: 0.00016657254761083778 | > step_time: 0.4531 (0.4303536048302284) | > loader_time: 0.0058 (0.006755150281465971)  --> TIME: 2025-02-25 23:17:27 -- STEP: 63/72 -- GLOBAL_STEP: 1105400 | > loss_disc: 2.6294808387756348 (2.463681905988664) | > loss_disc_real_0: 0.04734840244054794 (0.0870639355409713) | > loss_disc_real_1: 0.2343965470790863 (0.21505147124093676) | > loss_disc_real_2: 0.1483943611383438 (0.2262329856554667) | > loss_disc_real_3: 0.243774875998497 (0.2303699664180241) | > loss_disc_real_4: 0.2200324684381485 (0.23817567905736348) | > loss_disc_real_5: 0.2388908416032791 (0.24529985823328532) | > loss_0: 2.6294808387756348 (2.463681905988664) | > grad_norm_0: tensor(132.9769, device='cuda:0') (tensor(196.9482, device='cuda:0')) | > loss_gen: 2.0614888668060303 (2.287585773165264) | > loss_kl: 2.0812900066375732 (1.916166076584468) | > loss_feat: 4.692997932434082 (4.254899997559803) | > loss_mel: 17.04511260986328 (16.544069123646562) | > loss_duration: 1.5645089149475098 (1.5135393407609727) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.445398330688477 (26.516260298471604) | > grad_norm_1: tensor(4824.9424, device='cuda:0') (tensor(6588.8799, device='cuda:0')) | > current_lr_0: 0.00016657254761083778 | > current_lr_1: 0.00016657254761083778 | > step_time: 0.5048 (0.44642423069666304) | > loader_time: 0.0111 (0.00819873052930075) --> EVAL PERFORMANCE | > avg_loader_time: 0.008581282387317066 (+0.00014303099941199646) | > avg_step_time: 0.4521250691212399 (-0.005863579226211779) | > avg_loss_disc: 2.4741461277008066 (-0.17531144786888397) | > avg_loss_disc_real_0: 0.09530624873201612 (-0.05546661130559277) | > avg_loss_disc_real_1: 0.21562215706831972 (-0.006023538994117561) | > avg_loss_disc_real_2: 0.22081126923292455 (-0.01463650628714494) | > avg_loss_disc_real_3: 0.2295601164371195 (-0.00855141584302338) | > avg_loss_disc_real_4: 0.23419018614460047 (-0.004764762562765201) | > avg_loss_disc_real_5: 0.24480558784914688 (+0.00856712901256454) | > avg_loss_0: 2.4741461277008066 (-0.17531144786888397) | > avg_grad_norm_0: tensor(217.9468, device='cuda:0') (+tensor(0.1265, device='cuda:0')) | > avg_loss_gen: 2.270026596499161 (+0.17485783301608748) | > avg_loss_kl: 1.9315689449578943 (-0.05610529637672523) | > avg_loss_feat: 4.213066013766005 (+0.40796571718135155) | > avg_loss_mel: 16.55618801922865 (-0.07059782323704056) | > avg_loss_duration: 1.522172513142438 (-0.006330352433970399) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.493022112779215 (+0.4497901755319518) | > avg_grad_norm_1: tensor(6638.1372, device='cuda:0') (+tensor(145.4043, device='cuda:0'))  > EPOCH: 1464/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:17:33)   --> TIME: 2025-02-25 23:17:53 -- STEP: 41/72 -- GLOBAL_STEP: 1105450 | > loss_disc: 2.654397964477539 (2.7232248259753717) | > loss_disc_real_0: 0.05146978050470352 (0.1795024478399172) | > loss_disc_real_1: 0.18680541217327118 (0.24500689673714526) | > loss_disc_real_2: 0.118374764919281 (0.25574361660131595) | > loss_disc_real_3: 0.2230861932039261 (0.22530936013634612) | > loss_disc_real_4: 0.18897520005702972 (0.24105422889314046) | > loss_disc_real_5: 0.22912590205669403 (0.23985622096352466) | > loss_0: 2.654397964477539 (2.7232248259753717) | > grad_norm_0: tensor(80.8937, device='cuda:0') (tensor(239.0323, device='cuda:0')) | > loss_gen: 1.674404263496399 (2.127008304363344) | > loss_kl: 3.833824634552002 (2.1709653458944183) | > loss_feat: 4.0030717849731445 (3.863997825762121) | > loss_mel: 25.96592903137207 (17.423426348988606) | > loss_duration: 1.6494348049163818 (1.510771088483857) | > amp_scaler: 8.0 (8.0) | > loss_1: 37.12666320800781 (27.096169029794087) | > grad_norm_1: tensor(7190.8037, device='cuda:0') (tensor(5750.2612, device='cuda:0')) | > current_lr_0: 0.00016655172604238642 | > current_lr_1: 0.00016655172604238642 | > step_time: 0.4391 (0.43677232905132013) | > loader_time: 0.0089 (0.007131739360530202) --> EVAL PERFORMANCE | > avg_loader_time: 0.008472744847687199 (-0.0001085375396298674) | > avg_step_time: 0.45018662197489134 (-0.0019384471463485542) | > avg_loss_disc: 2.6706957951397965 (+0.19654966743898994) | > avg_loss_disc_real_0: 0.16483204092987827 (+0.06952579219786215) | > avg_loss_disc_real_1: 0.23592935796354858 (+0.02030720089522886) | > avg_loss_disc_real_2: 0.24222158525191562 (+0.02141031601899107) | > avg_loss_disc_real_3: 0.23021980123200886 (+0.0006596847948893703) | > avg_loss_disc_real_4: 0.23905183248956438 (+0.004861646344963916) | > avg_loss_disc_real_5: 0.23747053532533244 (-0.007335052523814445) | > avg_loss_0: 2.6706957951397965 (+0.19654966743898994) | > avg_grad_norm_0: tensor(237.1107, device='cuda:0') (+tensor(19.1639, device='cuda:0')) | > avg_loss_gen: 2.1650907892576408 (-0.10493580724152007) | > avg_loss_kl: 2.3857637425543565 (+0.4541947975964622) | > avg_loss_feat: 3.972287292211828 (-0.24077872155417657) | > avg_loss_mel: 18.668941242594116 (+2.1127532233654662) | > avg_loss_duration: 1.5564297756678622 (+0.03425726252542427) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.748512778483647 (+2.255490665704432) | > avg_grad_norm_1: tensor(6605.5132, device='cuda:0') (tensor(-32.6240, device='cuda:0'))  > EPOCH: 1465/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:18:09)   --> TIME: 2025-02-25 23:18:20 -- STEP: 19/72 -- GLOBAL_STEP: 1105500 | > loss_disc: 2.413681745529175 (2.487570461473967) | > loss_disc_real_0: 0.06580249220132828 (0.06932193021240987) | > loss_disc_real_1: 0.1712275743484497 (0.1989647059848434) | > loss_disc_real_2: 0.21478907763957977 (0.22874438213674644) | > loss_disc_real_3: 0.22147680819034576 (0.20561020546837858) | > loss_disc_real_4: 0.20875538885593414 (0.20762842501464643) | > loss_disc_real_5: 0.2178824245929718 (0.24419963516687093) | > loss_0: 2.413681745529175 (2.487570461473967) | > grad_norm_0: tensor(178.3100, device='cuda:0') (tensor(196.7094, device='cuda:0')) | > loss_gen: 2.067182779312134 (2.1840136239403174) | > loss_kl: 1.9485313892364502 (2.106360448034186) | > loss_feat: 4.418693542480469 (4.313690963544343) | > loss_mel: 17.62247657775879 (18.21432746084113) | > loss_duration: 1.4979777336120605 (1.5119005567149113) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.554861068725586 (28.330293053074886) | > grad_norm_1: tensor(8264.0381, device='cuda:0') (tensor(7499.7188, device='cuda:0')) | > current_lr_0: 0.0001665309070766311 | > current_lr_1: 0.0001665309070766311 | > step_time: 0.4384 (0.4281800044210334) | > loader_time: 0.0069 (0.006831846739116468)  --> TIME: 2025-02-25 23:18:43 -- STEP: 69/72 -- GLOBAL_STEP: 1105550 | > loss_disc: 2.3536550998687744 (2.4811744482620908) | > loss_disc_real_0: 0.06601765751838684 (0.074922825007335) | > loss_disc_real_1: 0.2391602247953415 (0.21357960037995075) | > loss_disc_real_2: 0.2269437164068222 (0.23352971867374753) | > loss_disc_real_3: 0.2537834048271179 (0.22328428837700168) | > loss_disc_real_4: 0.26637306809425354 (0.23438385193762573) | > loss_disc_real_5: 0.25659751892089844 (0.24569931259189826) | > loss_0: 2.3536550998687744 (2.4811744482620908) | > grad_norm_0: tensor(87.8110, device='cuda:0') (tensor(196.7233, device='cuda:0')) | > loss_gen: 2.488677978515625 (2.243212292159814) | > loss_kl: 2.0918281078338623 (2.0342702019041856) | > loss_feat: 4.359419822692871 (4.205549164094787) | > loss_mel: 17.307226181030273 (17.313638120457746) | > loss_duration: 1.6464345455169678 (1.5415209808211392) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.893587112426758 (27.338190880374633) | > grad_norm_1: tensor(7240.6929, device='cuda:0') (tensor(7199.3413, device='cuda:0')) | > current_lr_0: 0.0001665309070766311 | > current_lr_1: 0.0001665309070766311 | > step_time: 0.5252 (0.4483696723329848) | > loader_time: 0.0126 (0.008589354114256043) --> EVAL PERFORMANCE | > avg_loader_time: 0.008638784918986576 (+0.0001660400712993771) | > avg_step_time: 0.449707309964677 (-0.0004793120102143389) | > avg_loss_disc: 2.4788050617970216 (-0.19189073334277484) | > avg_loss_disc_real_0: 0.07458586151331245 (-0.09024617941656582) | > avg_loss_disc_real_1: 0.21303659436148656 (-0.022892763602062022) | > avg_loss_disc_real_2: 0.2327649626933353 (-0.009456622558580319) | > avg_loss_disc_real_3: 0.22302984288880523 (-0.0071899583432036285) | > avg_loss_disc_real_4: 0.2339283546931307 (-0.005123477796433684) | > avg_loss_disc_real_5: 0.2453781007461145 (+0.007907565420782053) | > avg_loss_0: 2.4788050617970216 (-0.19189073334277484) | > avg_grad_norm_0: tensor(193.9555, device='cuda:0') (tensor(-43.1552, device='cuda:0')) | > avg_loss_gen: 2.2436268699001265 (+0.07853608064248574) | > avg_loss_kl: 2.032190153296564 (-0.3535735892577927) | > avg_loss_feat: 4.222004373308638 (+0.24971708109681012) | > avg_loss_mel: 17.306470427714604 (-1.3624708148795115) | > avg_loss_duration: 1.5421990424814354 (-0.01423073318642687) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.34649099430568 (-1.4020217841779683) | > avg_grad_norm_1: tensor(7146.3950, device='cuda:0') (+tensor(540.8818, device='cuda:0'))  > EPOCH: 1466/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:18:46)   --> TIME: 2025-02-25 23:19:10 -- STEP: 47/72 -- GLOBAL_STEP: 1105600 | > loss_disc: 2.295872688293457 (2.507576115587924) | > loss_disc_real_0: 0.08118831366300583 (0.08728065460603288) | > loss_disc_real_1: 0.2196897715330124 (0.22035143889011222) | > loss_disc_real_2: 0.22802400588989258 (0.23605200711716998) | > loss_disc_real_3: 0.2011016607284546 (0.2319468392970714) | > loss_disc_real_4: 0.22236205637454987 (0.23766029832210947) | > loss_disc_real_5: 0.2246870994567871 (0.24095525227962655) | > loss_0: 2.295872688293457 (2.507576115587924) | > grad_norm_0: tensor(148.8376, device='cuda:0') (tensor(267.0002, device='cuda:0')) | > loss_gen: 2.6492228507995605 (2.2212644815444946) | > loss_kl: 2.084674835205078 (1.9014508014029645) | > loss_feat: 5.287685871124268 (4.224137762759596) | > loss_mel: 17.030803680419922 (16.734308871817085) | > loss_duration: 1.5369268655776978 (1.503496639272) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.589313507080078 (26.5846587647783) | > grad_norm_1: tensor(5035.8970, device='cuda:0') (tensor(8174.8408, device='cuda:0')) | > current_lr_0: 0.00016651009071324652 | > current_lr_1: 0.00016651009071324652 | > step_time: 0.4436 (0.4468669333356492) | > loader_time: 0.0086 (0.007420752910857505) --> EVAL PERFORMANCE | > avg_loader_time: 0.008469427135628713 (-0.00016935778335786314) | > avg_step_time: 0.4557867016590817 (+0.00607939169440469) | > avg_loss_disc: 2.5058080512033376 (+0.027002989406315958) | > avg_loss_disc_real_0: 0.08728248080317404 (+0.012696619289861585) | > avg_loss_disc_real_1: 0.22109918506212636 (+0.008062590700639805) | > avg_loss_disc_real_2: 0.23575427792441678 (+0.0029893152310814797) | > avg_loss_disc_real_3: 0.2324055792160437 (+0.009375736327238482) | > avg_loss_disc_real_4: 0.24252300950842845 (+0.008594654815297753) | > avg_loss_disc_real_5: 0.2452581520651428 (-0.0001199486809717043) | > avg_loss_0: 2.5058080512033376 (+0.027002989406315958) | > avg_grad_norm_0: tensor(261.9905, device='cuda:0') (+tensor(68.0350, device='cuda:0')) | > avg_loss_gen: 2.2304566880347028 (-0.013170181865423736) | > avg_loss_kl: 1.9275065475786235 (-0.10468360571794033) | > avg_loss_feat: 4.163023851287199 (-0.05898052202143944) | > avg_loss_mel: 16.628895370053584 (-0.67757505766102) | > avg_loss_duration: 1.5258552322924979 (-0.01634381018893749) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.475737746332733 (-0.8707532479729458) | > avg_grad_norm_1: tensor(7906.1812, device='cuda:0') (+tensor(759.7861, device='cuda:0'))  > EPOCH: 1467/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:19:23)   --> TIME: 2025-02-25 23:19:36 -- STEP: 25/72 -- GLOBAL_STEP: 1105650 | > loss_disc: 2.4255285263061523 (2.475042247772217) | > loss_disc_real_0: 0.0782313272356987 (0.092006366699934) | > loss_disc_real_1: 0.2719569206237793 (0.22328053176403045) | > loss_disc_real_2: 0.24343474209308624 (0.23048815548419951) | > loss_disc_real_3: 0.23768943548202515 (0.22355140209198) | > loss_disc_real_4: 0.2654981315135956 (0.23038588345050812) | > loss_disc_real_5: 0.27346956729888916 (0.2440647917985916) | > loss_0: 2.4255285263061523 (2.475042247772217) | > grad_norm_0: tensor(43.3419, device='cuda:0') (tensor(263.8114, device='cuda:0')) | > loss_gen: 2.423757314682007 (2.2743928861618032) | > loss_kl: 1.8434054851531982 (1.8117704725265502) | > loss_feat: 4.710714817047119 (4.3496349334716795) | > loss_mel: 16.97508430480957 (16.6968588256836) | > loss_duration: 1.476454496383667 (1.4790776681900024) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.42941665649414 (26.611735000610352) | > grad_norm_1: tensor(7230.4219, device='cuda:0') (tensor(7506.3779, device='cuda:0')) | > current_lr_0: 0.00016648927695190734 | > current_lr_1: 0.00016648927695190734 | > step_time: 0.434 (0.4325905704498291) | > loader_time: 0.0066 (0.0071267223358154295) --> EVAL PERFORMANCE | > avg_loader_time: 0.008700021555725959 (+0.00023059442009724582) | > avg_step_time: 0.4525156390499061 (-0.00327106260917559) | > avg_loss_disc: 2.616246589472596 (+0.11043853826925831) | > avg_loss_disc_real_0: 0.17023410154900082 (+0.08295162074582678) | > avg_loss_disc_real_1: 0.22041448887804865 (-0.0006846961840777099) | > avg_loss_disc_real_2: 0.22744482650723255 (-0.008309451417184227) | > avg_loss_disc_real_3: 0.22078691171088688 (-0.011618667505156832) | > avg_loss_disc_real_4: 0.23775051198375058 (-0.004772497524677871) | > avg_loss_disc_real_5: 0.24957693882391477 (+0.004318786758771981) | > avg_loss_0: 2.616246589472596 (+0.11043853826925831) | > avg_grad_norm_0: tensor(283.6658, device='cuda:0') (+tensor(21.6753, device='cuda:0')) | > avg_loss_gen: 2.1630440799283304 (-0.0674126081063724) | > avg_loss_kl: 1.9609897909030107 (+0.033483243324387235) | > avg_loss_feat: 4.017665681704669 (-0.14535816958253012) | > avg_loss_mel: 16.88439123395464 (+0.25549586390105716) | > avg_loss_duration: 1.5475468014327574 (+0.021691569140259492) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.573637626540492 (+0.09789988020775908) | > avg_grad_norm_1: tensor(6418.5405, device='cuda:0') (tensor(-1487.6406, device='cuda:0'))  > EPOCH: 1468/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:20:00)   --> TIME: 2025-02-25 23:20:03 -- STEP: 3/72 -- GLOBAL_STEP: 1105700 | > loss_disc: 2.9774789810180664 (2.8714640935262046) | > loss_disc_real_0: 0.2940741777420044 (0.23699932297070822) | > loss_disc_real_1: 0.18165013194084167 (0.23342527449131012) | > loss_disc_real_2: 0.22093190252780914 (0.23160849511623383) | > loss_disc_real_3: 0.20893286168575287 (0.21687174836794534) | > loss_disc_real_4: 0.2602473497390747 (0.24838274220625559) | > loss_disc_real_5: 0.24022267758846283 (0.24427280823389688) | > loss_0: 2.9774789810180664 (2.8714640935262046) | > grad_norm_0: tensor(325.5186, device='cuda:0') (tensor(121.0857, device='cuda:0')) | > loss_gen: 1.9918140172958374 (1.9173025687535603) | > loss_kl: 2.1360361576080322 (2.303649663925171) | > loss_feat: 3.643833637237549 (3.302953561147054) | > loss_mel: 18.798690795898438 (18.28144709269206) | > loss_duration: 1.6275825500488281 (1.6071890989939372) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.197956085205078 (27.41254170735677) | > grad_norm_1: tensor(3520.5596, device='cuda:0') (tensor(1651.4355, device='cuda:0')) | > current_lr_0: 0.00016646846579228836 | > current_lr_1: 0.00016646846579228836 | > step_time: 0.4654 (0.4414211908976237) | > loader_time: 0.006 (0.006609042485555013)  --> TIME: 2025-02-25 23:20:26 -- STEP: 53/72 -- GLOBAL_STEP: 1105750 | > loss_disc: 2.6471550464630127 (2.6360080152187697) | > loss_disc_real_0: 0.26980137825012207 (0.16885287384941894) | > loss_disc_real_1: 0.23833078145980835 (0.2203922876209583) | > loss_disc_real_2: 0.2096107453107834 (0.24033963792728927) | > loss_disc_real_3: 0.1902341991662979 (0.23066075174313672) | > loss_disc_real_4: 0.20682522654533386 (0.2463090605330917) | > loss_disc_real_5: 0.2205294519662857 (0.2459106327227826) | > loss_0: 2.6471550464630127 (2.6360080152187697) | > grad_norm_0: tensor(190.8084, device='cuda:0') (tensor(231.7209, device='cuda:0')) | > loss_gen: 2.077230453491211 (2.1489523761677307) | > loss_kl: 2.052999496459961 (2.135078457166563) | > loss_feat: 3.740781545639038 (3.9809274943369752) | > loss_mel: 16.888036727905273 (17.650166853418888) | > loss_duration: 1.5913012027740479 (1.5505289469125136) | > amp_scaler: 16.0 (8.905660377358492) | > loss_1: 26.35034942626953 (27.465654301193524) | > grad_norm_1: tensor(7865.1440, device='cuda:0') (tensor(3876.7180, device='cuda:0')) | > current_lr_0: 0.00016646846579228836 | > current_lr_1: 0.00016646846579228836 | > step_time: 0.4497 (0.4394901023720795) | > loader_time: 0.0091 (0.007958704570554335) --> EVAL PERFORMANCE | > avg_loader_time: 0.008687533123392453 (-1.2488432333505536e-05) | > avg_step_time: 0.45116995421933453 (-0.0013456848305715696) | > avg_loss_disc: 2.632386281456745 (+0.016139691984148996) | > avg_loss_disc_real_0: 0.15699987496498605 (-0.013234226584014769) | > avg_loss_disc_real_1: 0.21482738453737446 (-0.005587104340674193) | > avg_loss_disc_real_2: 0.240226181669974 (+0.012781355162741453) | > avg_loss_disc_real_3: 0.2372159093198642 (+0.016428997608977314) | > avg_loss_disc_real_4: 0.2483623829525961 (+0.010611870968845527) | > avg_loss_disc_real_5: 0.24791783388231842 (-0.001659104941596351) | > avg_loss_0: 2.632386281456745 (+0.016139691984148996) | > avg_grad_norm_0: tensor(252.4211, device='cuda:0') (tensor(-31.2447, device='cuda:0')) | > avg_loss_gen: 2.1296987651099633 (-0.03334531481836711) | > avg_loss_kl: 2.1135901699603434 (+0.1526003790573327) | > avg_loss_feat: 3.96048167725684 (-0.057184004447828674) | > avg_loss_mel: 17.41594295770349 (+0.5315517237488478) | > avg_loss_duration: 1.5590104250840737 (+0.011463623651316368) | > avg_amp_scaler: 10.704225352112676 (+2.704225352112676) | > avg_loss_1: 27.1787242352123 (+0.6050866086718081) | > avg_grad_norm_1: tensor(4521.5083, device='cuda:0') (tensor(-1897.0322, device='cuda:0'))  > EPOCH: 1469/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:20:36)   --> TIME: 2025-02-25 23:20:53 -- STEP: 31/72 -- GLOBAL_STEP: 1105800 | > loss_disc: 2.3921265602111816 (2.6484269942006757) | > loss_disc_real_0: 0.05213932320475578 (0.16723400354385376) | > loss_disc_real_1: 0.21690519154071808 (0.23138583619748393) | > loss_disc_real_2: 0.24432778358459473 (0.24089671046503128) | > loss_disc_real_3: 0.2535378634929657 (0.23191735292634658) | > loss_disc_real_4: 0.26625385880470276 (0.2469335727153286) | > loss_disc_real_5: 0.2524941861629486 (0.24224527612809213) | > loss_0: 2.3921265602111816 (2.6484269942006757) | > grad_norm_0: tensor(129.2862, device='cuda:0') (tensor(305.6573, device='cuda:0')) | > loss_gen: 2.3656883239746094 (2.088373203431406) | > loss_kl: 2.5914597511291504 (2.1709573461163427) | > loss_feat: 4.352383613586426 (3.893598364245507) | > loss_mel: 17.807849884033203 (17.768991101172663) | > loss_duration: 1.5842301845550537 (1.5406683952577653) | > amp_scaler: 16.0 (16.0) | > loss_1: 28.70161247253418 (27.462588279478013) | > grad_norm_1: tensor(5904.6807, device='cuda:0') (tensor(6861.6777, device='cuda:0')) | > current_lr_0: 0.0001664476572340643 | > current_lr_1: 0.0001664476572340643 | > step_time: 0.4592 (0.4534707915398382) | > loader_time: 0.0087 (0.007084354277580015) --> EVAL PERFORMANCE | > avg_loader_time: 0.008693365983560053 (+5.83286016759936e-06) | > avg_step_time: 0.46361837252764637 (+0.012448418308311837) | > avg_loss_disc: 2.604316792017977 (-0.02806948943876808) | > avg_loss_disc_real_0: 0.1512804236525381 (-0.005719451312447943) | > avg_loss_disc_real_1: 0.22437606940806748 (+0.00954868487069302) | > avg_loss_disc_real_2: 0.23924891209938157 (-0.0009772695705924372) | > avg_loss_disc_real_3: 0.22592977151064805 (-0.01128613780921614) | > avg_loss_disc_real_4: 0.24277986921894718 (-0.0055825137336489306) | > avg_loss_disc_real_5: 0.243935066419588 (-0.003982767462730408) | > avg_loss_0: 2.604316792017977 (-0.02806948943876808) | > avg_grad_norm_0: tensor(328.0701, device='cuda:0') (+tensor(75.6490, device='cuda:0')) | > avg_loss_gen: 2.1535121739750167 (+0.023813408865053454) | > avg_loss_kl: 2.200782915236244 (+0.08719274527590049) | > avg_loss_feat: 3.9925968949223907 (+0.03211521766555059) | > avg_loss_mel: 17.600100436680755 (+0.18415747897726575) | > avg_loss_duration: 1.565253731230615 (+0.006243306146541228) | > avg_amp_scaler: 12.732394366197184 (+2.028169014084508) | > avg_loss_1: 27.51224617219307 (+0.3335219369807696) | > avg_grad_norm_1: tensor(6707.2085, device='cuda:0') (+tensor(2185.7002, device='cuda:0'))  > EPOCH: 1470/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:21:14)   --> TIME: 2025-02-25 23:21:20 -- STEP: 9/72 -- GLOBAL_STEP: 1105850 | > loss_disc: 2.6556925773620605 (2.6705447832743325) | > loss_disc_real_0: 0.14904239773750305 (0.20082295023732716) | > loss_disc_real_1: 0.16822463274002075 (0.21779056969616148) | > loss_disc_real_2: 0.1811559796333313 (0.2543158034483592) | > loss_disc_real_3: 0.21052956581115723 (0.24986276030540466) | > loss_disc_real_4: 0.24932284653186798 (0.24412131309509277) | > loss_disc_real_5: 0.24926722049713135 (0.24536235630512238) | > loss_0: 2.6556925773620605 (2.6705447832743325) | > grad_norm_0: tensor(432.1806, device='cuda:0') (tensor(313.9884, device='cuda:0')) | > loss_gen: 1.7326602935791016 (2.319290386305915) | > loss_kl: 1.9254120588302612 (1.9123589727613661) | > loss_feat: 3.6716814041137695 (3.9810794724358454) | > loss_mel: 17.047597885131836 (16.737746662563747) | > loss_duration: 1.5153521299362183 (1.5071603059768677) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.892704010009766 (26.4576358795166) | > grad_norm_1: tensor(10593.8262, device='cuda:0') (tensor(6526.1025, device='cuda:0')) | > current_lr_0: 0.00016642685127691003 | > current_lr_1: 0.00016642685127691003 | > step_time: 0.4364 (0.43784207767910427) | > loader_time: 0.006 (0.006776544782850478)  --> TIME: 2025-02-25 23:21:43 -- STEP: 59/72 -- GLOBAL_STEP: 1105900 | > loss_disc: 2.35567045211792 (2.633834438808894) | > loss_disc_real_0: 0.121075838804245 (0.13877499955943076) | > loss_disc_real_1: 0.23489242792129517 (0.263015483016685) | > loss_disc_real_2: 0.22959227859973907 (0.24766723660089204) | > loss_disc_real_3: 0.206556037068367 (0.23317689031867658) | > loss_disc_real_4: 0.24676349759101868 (0.2437561641810304) | > loss_disc_real_5: 0.2446122169494629 (0.24653258465104183) | > loss_0: 2.35567045211792 (2.633834438808894) | > grad_norm_0: tensor(99.0065, device='cuda:0') (tensor(257.0945, device='cuda:0')) | > loss_gen: 2.578230381011963 (2.2090139429448006) | > loss_kl: 2.00235652923584 (1.9738988977367595) | > loss_feat: 4.316426753997803 (3.9141569056753385) | > loss_mel: 16.908321380615234 (16.755508616819213) | > loss_duration: 1.560775637626648 (1.5206678034895558) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.36610984802246 (26.373246144440216) | > grad_norm_1: tensor(1993.3783, device='cuda:0') (tensor(6127.0859, device='cuda:0')) | > current_lr_0: 0.00016642685127691003 | > current_lr_1: 0.00016642685127691003 | > step_time: 0.4746 (0.4515917584047479) | > loader_time: 0.0101 (0.00817977371862379) --> EVAL PERFORMANCE | > avg_loader_time: 0.008769555830619705 (+7.61898470596524e-05) | > avg_step_time: 0.4600229800586969 (-0.0035953924689494654) | > avg_loss_disc: 2.6066183573763135 (+0.002301565358336699) | > avg_loss_disc_real_0: 0.12656093567190035 (-0.02471948798063775) | > avg_loss_disc_real_1: 0.25528573727523785 (+0.030909667867170365) | > avg_loss_disc_real_2: 0.2508413103264822 (+0.011592398227100648) | > avg_loss_disc_real_3: 0.230749968701685 (+0.0048201971910369346) | > avg_loss_disc_real_4: 0.24411559713558412 (+0.0013357279166369418) | > avg_loss_disc_real_5: 0.24537914970391234 (+0.0014440832843243268) | > avg_loss_0: 2.6066183573763135 (+0.002301565358336699) | > avg_grad_norm_0: tensor(230.7010, device='cuda:0') (tensor(-97.3691, device='cuda:0')) | > avg_loss_gen: 2.22443019336378 (+0.0709180193887633) | > avg_loss_kl: 1.9978661184579554 (-0.20291679677828856) | > avg_loss_feat: 3.972779864996252 (-0.019817029926138652) | > avg_loss_mel: 16.789010880698616 (-0.8110895559821394) | > avg_loss_duration: 1.5308900047355976 (-0.0343637264950174) | > avg_amp_scaler: 8.0 (-4.732394366197184) | > avg_loss_1: 26.51497703874615 (-0.9972691334469204) | > avg_grad_norm_1: tensor(5980.8989, device='cuda:0') (tensor(-726.3096, device='cuda:0'))  > EPOCH: 1471/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:21:51)   --> TIME: 2025-02-25 23:22:10 -- STEP: 37/72 -- GLOBAL_STEP: 1105950 | > loss_disc: 2.818073034286499 (2.6912970027408085) | > loss_disc_real_0: 0.17912231385707855 (0.13615820748177734) | > loss_disc_real_1: 0.2180459201335907 (0.21311393017704422) | > loss_disc_real_2: 0.2153662145137787 (0.2650379176075394) | > loss_disc_real_3: 0.20263159275054932 (0.24675208813435323) | > loss_disc_real_4: 0.2387506067752838 (0.2504074948059546) | > loss_disc_real_5: 0.24681074917316437 (0.24667402417273135) | > loss_0: 2.818073034286499 (2.6912970027408085) | > grad_norm_0: tensor(660.1397, device='cuda:0') (tensor(344.2189, device='cuda:0')) | > loss_gen: 1.76558256149292 (2.075488860542709) | > loss_kl: 1.8729848861694336 (1.9213185342582497) | > loss_feat: 3.729433536529541 (3.8093731532225736) | > loss_mel: 15.982218742370605 (16.46074181634026) | > loss_duration: 1.5140420198440552 (1.4895656882105648) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.864261627197266 (25.75648797524942) | > grad_norm_1: tensor(6820.2505, device='cuda:0') (tensor(8340.6777, device='cuda:0')) | > current_lr_0: 0.0001664060479205004 | > current_lr_1: 0.0001664060479205004 | > step_time: 0.4546 (0.4440143752742458) | > loader_time: 0.0083 (0.0075864856307571) --> EVAL PERFORMANCE | > avg_loader_time: 0.00894629115789709 (+0.0001767353272773848) | > avg_step_time: 0.45828384412846096 (-0.0017391359302359466) | > avg_loss_disc: 2.689232544160225 (+0.08261418678391141) | > avg_loss_disc_real_0: 0.14020173471998165 (+0.013640799048081298) | > avg_loss_disc_real_1: 0.2181097153626697 (-0.037176021912568136) | > avg_loss_disc_real_2: 0.24482844392178763 (-0.006012866404694589) | > avg_loss_disc_real_3: 0.2350604064028028 (+0.004310437701117803) | > avg_loss_disc_real_4: 0.24242567344450613 (-0.0016899236910779825) | > avg_loss_disc_real_5: 0.24398187474465707 (-0.0013972749592552625) | > avg_loss_0: 2.689232544160225 (+0.08261418678391141) | > avg_grad_norm_0: tensor(338.2707, device='cuda:0') (+tensor(107.5697, device='cuda:0')) | > avg_loss_gen: 2.0264010496542477 (-0.19802914370953228) | > avg_loss_kl: 1.980380115374713 (-0.01748600308324244) | > avg_loss_feat: 3.728747760745841 (-0.24403210425041122) | > avg_loss_mel: 16.573391793479377 (-0.21561908721923828) | > avg_loss_duration: 1.5265482311517418 (-0.00434177358385579) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 25.83546872206137 (-0.6795083166847782) | > avg_grad_norm_1: tensor(8265.5156, device='cuda:0') (+tensor(2284.6167, device='cuda:0'))  > EPOCH: 1472/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:22:29)   --> TIME: 2025-02-25 23:22:38 -- STEP: 15/72 -- GLOBAL_STEP: 1106000 | > loss_disc: 2.443089008331299 (2.579397169748942) | > loss_disc_real_0: 0.11179430782794952 (0.1377648378411929) | > loss_disc_real_1: 0.22105634212493896 (0.21768031219641368) | > loss_disc_real_2: 0.22882424294948578 (0.2329224665959676) | > loss_disc_real_3: 0.23551538586616516 (0.23719827930132548) | > loss_disc_real_4: 0.24399836361408234 (0.24754753510157274) | > loss_disc_real_5: 0.2507327198982239 (0.250505659977595) | > loss_0: 2.443089008331299 (2.579397169748942) | > grad_norm_0: tensor(287.7390, device='cuda:0') (tensor(226.6472, device='cuda:0')) | > loss_gen: 2.496756076812744 (2.153607455889384) | > loss_kl: 2.0044212341308594 (1.8557227849960327) | > loss_feat: 4.663133144378662 (3.899509572982788) | > loss_mel: 17.576669692993164 (16.750630823771157) | > loss_duration: 1.473372220993042 (1.4833720207214356) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.214351654052734 (26.142842356363932) | > grad_norm_1: tensor(1998.2231, device='cuda:0') (tensor(7427.3301, device='cuda:0')) | > current_lr_0: 0.00016638524716451034 | > current_lr_1: 0.00016638524716451034 | > step_time: 0.4323 (0.43943328857421876) | > loader_time: 0.006 (0.006561136245727539)  --> TIME: 2025-02-25 23:23:01 -- STEP: 65/72 -- GLOBAL_STEP: 1106050 | > loss_disc: 3.030045509338379 (2.5443912909581106) | > loss_disc_real_0: 0.04069952666759491 (0.11610106212588456) | > loss_disc_real_1: 0.28346121311187744 (0.2232620908663823) | > loss_disc_real_2: 0.2546660006046295 (0.23275893789071303) | > loss_disc_real_3: 0.30474233627319336 (0.23765534277145678) | > loss_disc_real_4: 0.3220261335372925 (0.24689133236041436) | > loss_disc_real_5: 0.273752361536026 (0.24811645448207856) | > loss_0: 3.030045509338379 (2.5443912909581106) | > grad_norm_0: tensor(187.2070, device='cuda:0') (tensor(218.7736, device='cuda:0')) | > loss_gen: 1.8271161317825317 (2.2344855253513045) | > loss_kl: 2.232728958129883 (1.9349901896256667) | > loss_feat: 2.8494224548339844 (4.063450714258048) | > loss_mel: 16.79347801208496 (16.69326796898475) | > loss_duration: 1.599153757095337 (1.5185069762743437) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.301898956298828 (26.444701150747445) | > grad_norm_1: tensor(6098.1270, device='cuda:0') (tensor(5939.4697, device='cuda:0')) | > current_lr_0: 0.00016638524716451034 | > current_lr_1: 0.00016638524716451034 | > step_time: 0.4958 (0.4503406011141263) | > loader_time: 0.0116 (0.008377530024601863) --> EVAL PERFORMANCE | > avg_loader_time: 0.008650914044447348 (-0.0002953771134497421) | > avg_step_time: 0.45446751487087195 (-0.003816329257589013) | > avg_loss_disc: 2.537700112436858 (-0.15153243172336683) | > avg_loss_disc_real_0: 0.11222045894869614 (-0.027981275771285508) | > avg_loss_disc_real_1: 0.22241870576227216 (+0.004308990399602447) | > avg_loss_disc_real_2: 0.23280040353116854 (-0.012028040390619082) | > avg_loss_disc_real_3: 0.23679150640964508 (+0.00173110000684229) | > avg_loss_disc_real_4: 0.24700503974733218 (+0.004579366302826043) | > avg_loss_disc_real_5: 0.247736856341362 (+0.003754981596704926) | > avg_loss_0: 2.537700112436858 (-0.15153243172336683) | > avg_grad_norm_0: tensor(218.9693, device='cuda:0') (tensor(-119.3014, device='cuda:0')) | > avg_loss_gen: 2.232576133499683 (+0.20617508384543548) | > avg_loss_kl: 1.9494698484179 (-0.03091026695681287) | > avg_loss_feat: 4.0689444676251485 (+0.3401967068793077) | > avg_loss_mel: 16.698760690823406 (+0.12536889734402834) | > avg_loss_duration: 1.524405472715136 (-0.0021427584366058294) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.474156419995804 (+0.6386876979344329) | > avg_grad_norm_1: tensor(5987.7656, device='cuda:0') (tensor(-2277.7500, device='cuda:0'))  > EPOCH: 1473/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:23:06)   --> TIME: 2025-02-25 23:23:28 -- STEP: 43/72 -- GLOBAL_STEP: 1106100 | > loss_disc: 2.5576493740081787 (2.55946607367937) | > loss_disc_real_0: 0.07112788408994675 (0.11733309300832968) | > loss_disc_real_1: 0.22136415541172028 (0.22588973440403162) | > loss_disc_real_2: 0.23768304288387299 (0.23190537580223972) | > loss_disc_real_3: 0.23451055586338043 (0.22933829489142396) | > loss_disc_real_4: 0.20584873855113983 (0.23764441387597904) | > loss_disc_real_5: 0.241307333111763 (0.24007327265517656) | > loss_0: 2.5576493740081787 (2.55946607367937) | > grad_norm_0: tensor(359.6030, device='cuda:0') (tensor(261.3684, device='cuda:0')) | > loss_gen: 2.126941204071045 (2.225977193477542) | > loss_kl: 2.002577066421509 (1.9022894238316737) | > loss_feat: 3.835829734802246 (4.147048379099645) | > loss_mel: 16.04669189453125 (16.76738747885061) | > loss_duration: 1.5307657718658447 (1.4982565086941393) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.542804718017578 (26.54095853761184) | > grad_norm_1: tensor(8666.5947, device='cuda:0') (tensor(6799.4702, device='cuda:0')) | > current_lr_0: 0.00016636444900861478 | > current_lr_1: 0.00016636444900861478 | > step_time: 0.4793 (0.44361604091733003) | > loader_time: 0.0097 (0.0074904464011968565) --> EVAL PERFORMANCE | > avg_loader_time: 0.008727950109562404 (+7.703606511505652e-05) | > avg_step_time: 0.46124141988619954 (+0.006773905015327597) | > avg_loss_disc: 2.564555668495071 (+0.026855556058213104) | > avg_loss_disc_real_0: 0.10697908675901485 (-0.005241372189681293) | > avg_loss_disc_real_1: 0.22456350041107392 (+0.0021447946488017677) | > avg_loss_disc_real_2: 0.23291172590893758 (+0.00011132237776903531) | > avg_loss_disc_real_3: 0.2326558272183781 (-0.004135679191266978) | > avg_loss_disc_real_4: 0.2385621964931488 (-0.008442843254183374) | > avg_loss_disc_real_5: 0.24111662574217352 (-0.006620230599188481) | > avg_loss_0: 2.564555668495071 (+0.026855556058213104) | > avg_grad_norm_0: tensor(256.1569, device='cuda:0') (+tensor(37.1876, device='cuda:0')) | > avg_loss_gen: 2.2184224447733922 (-0.01415368872629097) | > avg_loss_kl: 1.958029243308054 (+0.008559394890153893) | > avg_loss_feat: 4.123866228990152 (+0.05492176136500326) | > avg_loss_mel: 16.683205456800863 (-0.015555234022542663) | > avg_loss_duration: 1.5245417557971581 (+0.00013628308202218165) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.50806486102896 (+0.03390844103315516) | > avg_grad_norm_1: tensor(7242.4282, device='cuda:0') (+tensor(1254.6626, device='cuda:0'))  > EPOCH: 1474/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:23:43)   --> TIME: 2025-02-25 23:23:55 -- STEP: 21/72 -- GLOBAL_STEP: 1106150 | > loss_disc: 2.404698133468628 (2.5493252958570207) | > loss_disc_real_0: 0.03827446326613426 (0.12128137477806636) | > loss_disc_real_1: 0.21703305840492249 (0.2101190366915294) | > loss_disc_real_2: 0.23296645283699036 (0.2406741976737976) | > loss_disc_real_3: 0.20942573249340057 (0.24624221097855342) | > loss_disc_real_4: 0.22712482511997223 (0.24085775514443716) | > loss_disc_real_5: 0.21792735159397125 (0.23517647882302603) | > loss_0: 2.404698133468628 (2.5493252958570207) | > grad_norm_0: tensor(58.4315, device='cuda:0') (tensor(272.0395, device='cuda:0')) | > loss_gen: 2.3502626419067383 (2.299667029153733) | > loss_kl: 1.7365154027938843 (1.838868674777803) | > loss_feat: 4.782754421234131 (4.221954073224749) | > loss_mel: 16.47394371032715 (16.648818833487375) | > loss_duration: 1.4945852756500244 (1.4813169638315837) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.83806037902832 (26.490625472295854) | > grad_norm_1: tensor(3722.7102, device='cuda:0') (tensor(6452.6279, device='cuda:0')) | > current_lr_0: 0.00016634365345248868 | > current_lr_1: 0.00016634365345248868 | > step_time: 0.455 (0.44486412547883536) | > loader_time: 0.0068 (0.006711165110270183)  --> TIME: 2025-02-25 23:24:20 -- STEP: 71/72 -- GLOBAL_STEP: 1106200 | > loss_disc: 2.841041088104248 (2.5674927704770796) | > loss_disc_real_0: 0.04009290039539337 (0.09269480018968314) | > loss_disc_real_1: 0.26115748286247253 (0.22795895140775493) | > loss_disc_real_2: 0.25834035873413086 (0.23695558519430562) | > loss_disc_real_3: 0.243036687374115 (0.23571789537517118) | > loss_disc_real_4: 0.2594682276248932 (0.23337973780195478) | > loss_disc_real_5: 0.24843254685401917 (0.24064669432774397) | > loss_0: 2.841041088104248 (2.5674927704770796) | > grad_norm_0: tensor(180.5268, device='cuda:0') (tensor(246.4526, device='cuda:0')) | > loss_gen: 1.863007664680481 (2.2255249795779375) | > loss_kl: 1.9089930057525635 (1.9459313573971602) | > loss_feat: 3.282526731491089 (4.232013497554078) | > loss_mel: 14.893861770629883 (16.603632953804983) | > loss_duration: 1.5271061658859253 (1.5234639879683374) | > amp_scaler: 8.0 (8.0) | > loss_1: 23.475494384765625 (26.530566712500345) | > grad_norm_1: tensor(8917.5928, device='cuda:0') (tensor(7435.3730, device='cuda:0')) | > current_lr_0: 0.00016634365345248868 | > current_lr_1: 0.00016634365345248868 | > step_time: 0.5072 (0.4666629307706591) | > loader_time: 0.0081 (0.00880353215714576) --> EVAL PERFORMANCE | > avg_loader_time: 0.00880353215714576 (+7.558204758335504e-05) | > avg_step_time: 0.4666629307706591 (+0.005421510884459579) | > avg_loss_disc: 2.5674927704770796 (+0.002937101982008361) | > avg_loss_disc_real_0: 0.09269480018968314 (-0.014284286569331714) | > avg_loss_disc_real_1: 0.22795895140775493 (+0.00339545099668101) | > avg_loss_disc_real_2: 0.23695558519430562 (+0.0040438592853680455) | > avg_loss_disc_real_3: 0.23571789537517118 (+0.0030620681567930796) | > avg_loss_disc_real_4: 0.23337973780195478 (-0.00518245869119402) | > avg_loss_disc_real_5: 0.24064669432774397 (-0.000469931414429553) | > avg_loss_0: 2.5674927704770796 (+0.002937101982008361) | > avg_grad_norm_0: tensor(246.4526, device='cuda:0') (tensor(-9.7043, device='cuda:0')) | > avg_loss_gen: 2.2255249795779375 (+0.007102534804545257) | > avg_loss_kl: 1.9459313573971602 (-0.01209788591089378) | > avg_loss_feat: 4.232013497554078 (+0.10814726856392642) | > avg_loss_mel: 16.603632953804983 (-0.07957250299588026) | > avg_loss_duration: 1.5234639879683374 (-0.0010777678288207415) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.530566712500345 (+0.022501851471385947) | > avg_grad_norm_1: tensor(7435.3730, device='cuda:0') (+tensor(192.9448, device='cuda:0'))  > EPOCH: 1475/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:24:21)   --> TIME: 2025-02-25 23:24:46 -- STEP: 49/72 -- GLOBAL_STEP: 1106250 | > loss_disc: 2.356485366821289 (2.531521704732155) | > loss_disc_real_0: 0.10982371866703033 (0.08246757888368197) | > loss_disc_real_1: 0.18974865972995758 (0.21089009332413577) | > loss_disc_real_2: 0.2755008935928345 (0.20675122220905454) | > loss_disc_real_3: 0.19141213595867157 (0.23776435912871846) | > loss_disc_real_4: 0.2363341599702835 (0.25728217132237485) | > loss_disc_real_5: 0.22758762538433075 (0.2525871396064759) | > loss_0: 2.356485366821289 (2.531521704732155) | > grad_norm_0: tensor(245.5493, device='cuda:0') (tensor(244.2370, device='cuda:0')) | > loss_gen: 2.5955471992492676 (2.259778596916978) | > loss_kl: 1.9884968996047974 (1.9092994782389427) | > loss_feat: 4.83138370513916 (4.300895510887613) | > loss_mel: 17.027191162109375 (16.506238158868285) | > loss_duration: 1.578176498413086 (1.4993817879229177) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.020795822143555 (26.4755934111926) | > grad_norm_1: tensor(3337.6812, device='cuda:0') (tensor(6611.8296, device='cuda:0')) | > current_lr_0: 0.00016632286049580712 | > current_lr_1: 0.00016632286049580712 | > step_time: 0.459 (0.45056594634542657) | > loader_time: 0.0098 (0.00792311648933255) --> EVAL PERFORMANCE | > avg_loader_time: 0.008802837049457385 (-6.951076883740526e-07) | > avg_step_time: 0.46155819758562977 (-0.005104733185029353) | > avg_loss_disc: 2.519316589328605 (-0.04817618114847466) | > avg_loss_disc_real_0: 0.08257683502956176 (-0.010117965160121381) | > avg_loss_disc_real_1: 0.21309284341167395 (-0.014866107996080985) | > avg_loss_disc_real_2: 0.21420311067305814 (-0.022752474521247484) | > avg_loss_disc_real_3: 0.23463969877068425 (-0.0010781966044869362) | > avg_loss_disc_real_4: 0.25363466269533397 (+0.020254924893379184) | > avg_loss_disc_real_5: 0.25075588700637014 (+0.010109192678626172) | > avg_loss_0: 2.519316589328605 (-0.04817618114847466) | > avg_grad_norm_0: tensor(248.7128, device='cuda:0') (+tensor(2.2602, device='cuda:0')) | > avg_loss_gen: 2.259688379059376 (+0.03416339948143854) | > avg_loss_kl: 1.959907870897105 (+0.013976513499944776) | > avg_loss_feat: 4.254544980089431 (+0.022531482535352865) | > avg_loss_mel: 16.554587847749954 (-0.04904510605502921) | > avg_loss_duration: 1.5197322586892357 (-0.003731729279101703) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.54846110814054 (+0.01789439564019446) | > avg_grad_norm_1: tensor(6607.3936, device='cuda:0') (tensor(-827.9795, device='cuda:0'))  > EPOCH: 1476/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:24:59)   --> TIME: 2025-02-25 23:25:14 -- STEP: 27/72 -- GLOBAL_STEP: 1106300 | > loss_disc: 2.6334052085876465 (2.5167937720263445) | > loss_disc_real_0: 0.054061513394117355 (0.07444610467387569) | > loss_disc_real_1: 0.16497904062271118 (0.22662297277538865) | > loss_disc_real_2: 0.128151074051857 (0.23377615875667995) | > loss_disc_real_3: 0.19273002445697784 (0.2411946483232357) | > loss_disc_real_4: 0.1999710500240326 (0.26218781703048283) | > loss_disc_real_5: 0.254180908203125 (0.2471914561810317) | > loss_0: 2.6334052085876465 (2.5167937720263445) | > grad_norm_0: tensor(242.0584, device='cuda:0') (tensor(236.6580, device='cuda:0')) | > loss_gen: 1.9484226703643799 (2.3066222623542507) | > loss_kl: 2.0215961933135986 (1.9102753709863733) | > loss_feat: 4.283344268798828 (4.379668527179295) | > loss_mel: 16.6840763092041 (16.660415119594997) | > loss_duration: 1.4817533493041992 (1.477945539686415) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.419193267822266 (26.734927142107928) | > grad_norm_1: tensor(6959.5259, device='cuda:0') (tensor(6505.4111, device='cuda:0')) | > current_lr_0: 0.00016630207013824513 | > current_lr_1: 0.00016630207013824513 | > step_time: 0.4447 (0.4619629736299868) | > loader_time: 0.0086 (0.0072595543331570095) --> EVAL PERFORMANCE | > avg_loader_time: 0.008732795715332033 (-7.004133412535224e-05) | > avg_step_time: 0.46419389818755674 (+0.0026357006019269713) | > avg_loss_disc: 2.520927395619137 (+0.0016108062905320075) | > avg_loss_disc_real_0: 0.09381417345098207 (+0.011237338421420315) | > avg_loss_disc_real_1: 0.22781948997101314 (+0.014726646559339196) | > avg_loss_disc_real_2: 0.22470250704758604 (+0.010499396374527903) | > avg_loss_disc_real_3: 0.22974687164098445 (-0.004892827129699795) | > avg_loss_disc_real_4: 0.23920185066444774 (-0.014432812030886227) | > avg_loss_disc_real_5: 0.24803400270535914 (-0.0027218843010110017) | > avg_loss_0: 2.520927395619137 (+0.0016108062905320075) | > avg_grad_norm_0: tensor(243.1705, device='cuda:0') (tensor(-5.5423, device='cuda:0')) | > avg_loss_gen: 2.273678769528027 (+0.013990390468650915) | > avg_loss_kl: 2.0009191523135548 (+0.04101128141644983) | > avg_loss_feat: 4.2043349574989 (-0.05021002259053109) | > avg_loss_mel: 16.537859930119044 (-0.016727917630909417) | > avg_loss_duration: 1.5221717005044642 (+0.0024394418152284825) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.538964661074356 (-0.009496447066183578) | > avg_grad_norm_1: tensor(6111.7568, device='cuda:0') (tensor(-495.6367, device='cuda:0'))  > EPOCH: 1477/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:25:36)   --> TIME: 2025-02-25 23:25:41 -- STEP: 5/72 -- GLOBAL_STEP: 1106350 | > loss_disc: 2.52262282371521 (2.426973819732666) | > loss_disc_real_0: 0.07664231210947037 (0.0579386867582798) | > loss_disc_real_1: 0.23574860394001007 (0.22833161652088166) | > loss_disc_real_2: 0.28902167081832886 (0.24257002472877504) | > loss_disc_real_3: 0.2640933096408844 (0.247430282831192) | > loss_disc_real_4: 0.26711466908454895 (0.2413702577352524) | > loss_disc_real_5: 0.2562660872936249 (0.23707571029663085) | > loss_0: 2.52262282371521 (2.426973819732666) | > grad_norm_0: tensor(195.6101, device='cuda:0') (tensor(189.5342, device='cuda:0')) | > loss_gen: 2.496598243713379 (2.456126499176025) | > loss_kl: 2.0042269229888916 (1.846146821975708) | > loss_feat: 4.573457717895508 (4.617862129211426) | > loss_mel: 17.332469940185547 (17.009911346435548) | > loss_duration: 1.4735143184661865 (1.4789143562316895) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.880268096923828 (27.40896110534668) | > grad_norm_1: tensor(2614.4844, device='cuda:0') (tensor(5902.8701, device='cuda:0')) | > current_lr_0: 0.00016628128237947785 | > current_lr_1: 0.00016628128237947785 | > step_time: 0.4255 (0.432017183303833) | > loader_time: 0.0096 (0.006599950790405274)  --> TIME: 2025-02-25 23:26:04 -- STEP: 55/72 -- GLOBAL_STEP: 1106400 | > loss_disc: 2.6882123947143555 (2.475894784927368) | > loss_disc_real_0: 0.055716194212436676 (0.08539648232134905) | > loss_disc_real_1: 0.21438707411289215 (0.21927935372699392) | > loss_disc_real_2: 0.2150978296995163 (0.224959151040424) | > loss_disc_real_3: 0.26237425208091736 (0.23194161626425655) | > loss_disc_real_4: 0.2728783190250397 (0.24301926033063367) | > loss_disc_real_5: 0.25717106461524963 (0.24600069631229748) | > loss_0: 2.6882123947143555 (2.475894784927368) | > grad_norm_0: tensor(160.4985, device='cuda:0') (tensor(236.1171, device='cuda:0')) | > loss_gen: 2.0667471885681152 (2.35448093847795) | > loss_kl: 2.2221343517303467 (1.9682410803708164) | > loss_feat: 3.7710001468658447 (4.407853308590975) | > loss_mel: 16.221271514892578 (16.64855046705766) | > loss_duration: 1.5784895420074463 (1.5062091437253085) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.859642028808594 (26.885334847190162) | > grad_norm_1: tensor(6797.7251, device='cuda:0') (tensor(6352.3926, device='cuda:0')) | > current_lr_0: 0.00016628128237947785 | > current_lr_1: 0.00016628128237947785 | > step_time: 0.4726 (0.4432562524622137) | > loader_time: 0.0092 (0.007938944209705699) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640020665988115 (-9.277504934391764e-05) | > avg_step_time: 0.4540617499552982 (-0.010132148232258553) | > avg_loss_disc: 2.47716900664316 (-0.043758388975976725) | > avg_loss_disc_real_0: 0.08545834287791185 (-0.008355830573070225) | > avg_loss_disc_real_1: 0.22010832101526395 (-0.0077111689557491925) | > avg_loss_disc_real_2: 0.22316644036434066 (-0.0015360666832453795) | > avg_loss_disc_real_3: 0.23167555894650205 (+0.0019286873055175957) | > avg_loss_disc_real_4: 0.24033085959897915 (+0.0011290089345314114) | > avg_loss_disc_real_5: 0.2446123592450585 (-0.003421643460300633) | > avg_loss_0: 2.47716900664316 (-0.043758388975976725) | > avg_grad_norm_0: tensor(238.7514, device='cuda:0') (tensor(-4.4191, device='cuda:0')) | > avg_loss_gen: 2.3405371686102647 (+0.06685839908223778) | > avg_loss_kl: 1.9972091896433226 (-0.0037099626702321586) | > avg_loss_feat: 4.382348906825966 (+0.17801394932706582) | > avg_loss_mel: 16.681153875001723 (+0.14329394488267866) | > avg_loss_duration: 1.5222858828558048 (+0.00011418235134064147) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.923534930591853 (+0.3845702695174964) | > avg_grad_norm_1: tensor(6471.1353, device='cuda:0') (+tensor(359.3784, device='cuda:0'))  > EPOCH: 1478/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:26:13)   --> TIME: 2025-02-25 23:26:30 -- STEP: 33/72 -- GLOBAL_STEP: 1106450 | > loss_disc: 2.5030055046081543 (2.477692062204534) | > loss_disc_real_0: 0.10739053785800934 (0.08299417319622904) | > loss_disc_real_1: 0.2734631597995758 (0.2167376239191402) | > loss_disc_real_2: 0.24703019857406616 (0.2270331811724287) | > loss_disc_real_3: 0.2432764768600464 (0.2299502103617697) | > loss_disc_real_4: 0.24526731669902802 (0.24458114125511862) | > loss_disc_real_5: 0.2309587597846985 (0.24762607343269116) | > loss_0: 2.5030055046081543 (2.477692062204534) | > grad_norm_0: tensor(165.4571, device='cuda:0') (tensor(214.5359, device='cuda:0')) | > loss_gen: 2.480802297592163 (2.376522746953097) | > loss_kl: 2.0268781185150146 (1.9277023871739705) | > loss_feat: 4.597906589508057 (4.421640937978571) | > loss_mel: 16.562664031982422 (16.808641809405703) | > loss_duration: 1.4899460077285767 (1.4824367039131396) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.1581974029541 (27.016944885253906) | > grad_norm_1: tensor(2790.9854, device='cuda:0') (tensor(6904.1802, device='cuda:0')) | > current_lr_0: 0.0001662604972191804 | > current_lr_1: 0.0001662604972191804 | > step_time: 0.4541 (0.43508039821277966) | > loader_time: 0.008 (0.006972493547381777) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534105730728365 (-0.00010591493525975021) | > avg_step_time: 0.45211512605908893 (-0.0019466238962092541) | > avg_loss_disc: 2.502295363117272 (+0.025126356474111766) | > avg_loss_disc_real_0: 0.08356388054893052 (-0.0018944623289813278) | > avg_loss_disc_real_1: 0.21680980501040606 (-0.003298516004857893) | > avg_loss_disc_real_2: 0.22600490127650785 (+0.00283846091216719) | > avg_loss_disc_real_3: 0.23007890687022411 (-0.0015966520762779324) | > avg_loss_disc_real_4: 0.24000692262615955 (-0.00032393697281959954) | > avg_loss_disc_real_5: 0.24459354810311762 (-1.881114194088318e-05) | > avg_loss_0: 2.502295363117272 (+0.025126356474111766) | > avg_grad_norm_0: tensor(229.5732, device='cuda:0') (tensor(-9.1781, device='cuda:0')) | > avg_loss_gen: 2.314684867858887 (-0.025852300751377566) | > avg_loss_kl: 1.999953802202789 (+0.0027446125594663418) | > avg_loss_feat: 4.3361371940290425 (-0.046211712796923266) | > avg_loss_mel: 16.81266540204974 (+0.1315115270480156) | > avg_loss_duration: 1.5226786489218054 (+0.0003927660660005827) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.986120035950567 (+0.06258510535871409) | > avg_grad_norm_1: tensor(7004.5947, device='cuda:0') (+tensor(533.4595, device='cuda:0'))  > EPOCH: 1479/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:26:50)   --> TIME: 2025-02-25 23:26:57 -- STEP: 11/72 -- GLOBAL_STEP: 1106500 | > loss_disc: 2.6457293033599854 (2.4335887865586714) | > loss_disc_real_0: 0.046293437480926514 (0.07640294256535443) | > loss_disc_real_1: 0.2100432962179184 (0.21849529174241153) | > loss_disc_real_2: 0.24080830812454224 (0.2340869212692434) | > loss_disc_real_3: 0.23451310396194458 (0.23366554352370175) | > loss_disc_real_4: 0.23654375970363617 (0.23819067532365973) | > loss_disc_real_5: 0.2310509830713272 (0.2388950681144541) | > loss_0: 2.6457293033599854 (2.4335887865586714) | > grad_norm_0: tensor(259.7852, device='cuda:0') (tensor(229.4760, device='cuda:0')) | > loss_gen: 2.034630060195923 (2.4129520004445855) | > loss_kl: 2.0435800552368164 (1.929037419232455) | > loss_feat: 3.9295246601104736 (4.546978907151655) | > loss_mel: 17.051420211791992 (16.867987806146797) | > loss_duration: 1.477919101715088 (1.4814326438036831) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.537073135375977 (27.23838858170943) | > grad_norm_1: tensor(8969.4951, device='cuda:0') (tensor(6740.2827, device='cuda:0')) | > current_lr_0: 0.000166239714657028 | > current_lr_1: 0.000166239714657028 | > step_time: 0.4279 (0.4271403442729603) | > loader_time: 0.0057 (0.006500330838290128)  --> TIME: 2025-02-25 23:27:20 -- STEP: 61/72 -- GLOBAL_STEP: 1106550 | > loss_disc: 2.5153968334198 (2.537591625432499) | > loss_disc_real_0: 0.15696072578430176 (0.11795183631484629) | > loss_disc_real_1: 0.17334328591823578 (0.2221488639956615) | > loss_disc_real_2: 0.189468652009964 (0.23325810153953364) | > loss_disc_real_3: 0.1917036920785904 (0.23354765259828725) | > loss_disc_real_4: 0.2260054349899292 (0.24093902306478532) | > loss_disc_real_5: 0.2245485931634903 (0.24367079212040196) | > loss_0: 2.5153968334198 (2.537591625432499) | > grad_norm_0: tensor(555.5862, device='cuda:0') (tensor(259.0201, device='cuda:0')) | > loss_gen: 2.1507623195648193 (2.3309999567563415) | > loss_kl: 2.265537977218628 (2.0194797105476514) | > loss_feat: 4.191826820373535 (4.283351960729382) | > loss_mel: 16.28668785095215 (16.76526893553185) | > loss_duration: 1.5655908584594727 (1.5157227907024446) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.460407257080078 (26.91482328196041) | > grad_norm_1: tensor(3737.2913, device='cuda:0') (tensor(6309.4165, device='cuda:0')) | > current_lr_0: 0.000166239714657028 | > current_lr_1: 0.000166239714657028 | > step_time: 0.457 (0.4439434575252846) | > loader_time: 0.011 (0.00828343532124504) --> EVAL PERFORMANCE | > avg_loader_time: 0.008743121590412838 (+0.0002090158596844728) | > avg_step_time: 0.45352895159116935 (+0.001413825532080415) | > avg_loss_disc: 2.5372155854399776 (+0.03492022232270564) | > avg_loss_disc_real_0: 0.1127652856980411 (+0.029201405149110585) | > avg_loss_disc_real_1: 0.22114602612777495 (+0.004336221117368894) | > avg_loss_disc_real_2: 0.22857925085954264 (+0.0025743495830347907) | > avg_loss_disc_real_3: 0.23212073533467845 (+0.0020418284644543316) | > avg_loss_disc_real_4: 0.23969098086088475 (-0.000315941765274802) | > avg_loss_disc_real_5: 0.2441921311784798 (-0.0004014169246378263) | > avg_loss_0: 2.5372155854399776 (+0.03492022232270564) | > avg_grad_norm_0: tensor(250.3718, device='cuda:0') (+tensor(20.7985, device='cuda:0')) | > avg_loss_gen: 2.3142860708102373 (-0.0003987970486498682) | > avg_loss_kl: 2.040084447659238 (+0.040130645456448955) | > avg_loss_feat: 4.259367640589326 (-0.07676955343971681) | > avg_loss_mel: 16.712533923941585 (-0.10013147810815326) | > avg_loss_duration: 1.5256821773421596 (+0.003003528420354229) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.85195428552762 (-0.1341657504229481) | > avg_grad_norm_1: tensor(6413.9502, device='cuda:0') (tensor(-590.6445, device='cuda:0'))  > EPOCH: 1480/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:27:27)   --> TIME: 2025-02-25 23:27:47 -- STEP: 39/72 -- GLOBAL_STEP: 1106600 | > loss_disc: 2.7154715061187744 (2.6455878783494997) | > loss_disc_real_0: 0.05296500772237778 (0.1186477723889626) | > loss_disc_real_1: 0.22403055429458618 (0.22099841729952738) | > loss_disc_real_2: 0.1799701303243637 (0.2946137356070371) | > loss_disc_real_3: 0.2067386507987976 (0.23355680322035766) | > loss_disc_real_4: 0.24088707566261292 (0.26142201171471524) | > loss_disc_real_5: 0.21394413709640503 (0.2500068919780926) | > loss_0: 2.7154715061187744 (2.6455878783494997) | > grad_norm_0: tensor(75.8915, device='cuda:0') (tensor(288.9377, device='cuda:0')) | > loss_gen: 1.7540134191513062 (2.3314752028538632) | > loss_kl: 2.051935911178589 (1.9962826417042658) | > loss_feat: 3.7900705337524414 (4.364386705251839) | > loss_mel: 16.490062713623047 (16.667737863002674) | > loss_duration: 1.5335010290145874 (1.48876542922778) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.619583129882812 (26.84864768003806) | > grad_norm_1: tensor(5425.8247, device='cuda:0') (tensor(7113.5083, device='cuda:0')) | > current_lr_0: 0.00016621893469269587 | > current_lr_1: 0.00016621893469269587 | > step_time: 0.4812 (0.4436469933925531) | > loader_time: 0.0084 (0.007297577002109625) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614425927820343 (-0.0001286956625924946) | > avg_step_time: 0.4572999510966556 (+0.003770999505486261) | > avg_loss_disc: 2.63873806469877 (+0.10152247925879232) | > avg_loss_disc_real_0: 0.11274913939791666 (-1.6146300124447732e-05) | > avg_loss_disc_real_1: 0.22252729335721114 (+0.0013812672294361839) | > avg_loss_disc_real_2: 0.2630905887820351 (+0.034511337922492474) | > avg_loss_disc_real_3: 0.2370125284077416 (+0.004891793073063144) | > avg_loss_disc_real_4: 0.25395628998816855 (+0.014265309127283798) | > avg_loss_disc_real_5: 0.2535458889225839 (+0.009353757744104102) | > avg_loss_0: 2.63873806469877 (+0.10152247925879232) | > avg_grad_norm_0: tensor(285.0024, device='cuda:0') (+tensor(34.6306, device='cuda:0')) | > avg_loss_gen: 2.280921333272692 (-0.03336473753754543) | > avg_loss_kl: 2.0516059851982225 (+0.01152153753898455) | > avg_loss_feat: 4.2441932349137845 (-0.01517440567554118) | > avg_loss_mel: 16.680280457080244 (-0.03225346686134145) | > avg_loss_duration: 1.5213322118974069 (-0.004349965444752746) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.77833315352319 (-0.07362113200442977) | > avg_grad_norm_1: tensor(6466.1978, device='cuda:0') (+tensor(52.2476, device='cuda:0'))  > EPOCH: 1481/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:28:04)   --> TIME: 2025-02-25 23:28:14 -- STEP: 17/72 -- GLOBAL_STEP: 1106650 | > loss_disc: 3.1622233390808105 (2.603121603236479) | > loss_disc_real_0: 0.25930365920066833 (0.14270236124010646) | > loss_disc_real_1: 0.35123908519744873 (0.24790274308008306) | > loss_disc_real_2: 0.11433732509613037 (0.21550578404875362) | > loss_disc_real_3: 0.12392295151948929 (0.2236921695225379) | > loss_disc_real_4: 0.2827746570110321 (0.22160080338225646) | > loss_disc_real_5: 0.25674423575401306 (0.2411350379971897) | > loss_0: 3.1622233390808105 (2.603121603236479) | > grad_norm_0: tensor(331.2721, device='cuda:0') (tensor(336.1156, device='cuda:0')) | > loss_gen: 1.6488183736801147 (2.225074803127962) | > loss_kl: 2.182931661605835 (1.9493701387854183) | > loss_feat: 3.7160263061523438 (4.106341067482443) | > loss_mel: 17.424232482910156 (16.622029697193817) | > loss_duration: 1.5085344314575195 (1.4772191258037792) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.480545043945312 (26.380034615011777) | > grad_norm_1: tensor(723.2421, device='cuda:0') (tensor(6948.6025, device='cuda:0')) | > current_lr_0: 0.0001661981573258593 | > current_lr_1: 0.0001661981573258593 | > step_time: 0.4363 (0.43429071762982535) | > loader_time: 0.0061 (0.006781620137831744)  --> TIME: 2025-02-25 23:28:37 -- STEP: 67/72 -- GLOBAL_STEP: 1106700 | > loss_disc: 2.7057998180389404 (2.660708395402822) | > loss_disc_real_0: 0.15826809406280518 (0.16082071846545637) | > loss_disc_real_1: 0.2641354203224182 (0.24547614913378188) | > loss_disc_real_2: 0.2994265854358673 (0.2178041174340604) | > loss_disc_real_3: 0.2722344398498535 (0.22111814513580122) | > loss_disc_real_4: 0.22952431440353394 (0.23769058554030176) | > loss_disc_real_5: 0.2609117031097412 (0.24706176368158256) | > loss_0: 2.7057998180389404 (2.660708395402822) | > grad_norm_0: tensor(765.2798, device='cuda:0') (tensor(312.5650, device='cuda:0')) | > loss_gen: 2.154160976409912 (2.1546690054793856) | > loss_kl: 2.177441358566284 (2.150394329384192) | > loss_feat: 3.8872199058532715 (3.9281381101750616) | > loss_mel: 17.26759910583496 (16.994019422958143) | > loss_duration: 1.6253941059112549 (1.5810741026010087) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.11181640625 (26.808294894090338) | > grad_norm_1: tensor(10189.1211, device='cuda:0') (tensor(6688.3818, device='cuda:0')) | > current_lr_0: 0.0001661981573258593 | > current_lr_1: 0.0001661981573258593 | > step_time: 0.4824 (0.4477458676295494) | > loader_time: 0.0121 (0.00848150609144524) --> EVAL PERFORMANCE | > avg_loader_time: 0.008653328452311771 (+3.89025244914281e-05) | > avg_step_time: 0.4506775829154001 (-0.0066223681812555) | > avg_loss_disc: 2.667849312365894 (+0.02911124766712403) | > avg_loss_disc_real_0: 0.1570173163019436 (+0.04426817690402694) | > avg_loss_disc_real_1: 0.2468048159925031 (+0.024277522635291954) | > avg_loss_disc_real_2: 0.2245712846937314 (-0.03851930408830373) | > avg_loss_disc_real_3: 0.223904104404886 (-0.013108424002855584) | > avg_loss_disc_real_4: 0.23733411981186397 (-0.016622170176304574) | > avg_loss_disc_real_5: 0.2471865241376447 (-0.006359364784939203) | > avg_loss_0: 2.667849312365894 (+0.02911124766712403) | > avg_grad_norm_0: tensor(312.3265, device='cuda:0') (+tensor(27.3242, device='cuda:0')) | > avg_loss_gen: 2.14734341225154 (-0.13357792102115207) | > avg_loss_kl: 2.151601173508335 (+0.09999518831011267) | > avg_loss_feat: 3.9044969753480294 (-0.3396962595657551) | > avg_loss_mel: 16.98609689255835 (+0.305816435478107) | > avg_loss_duration: 1.5820004721762428 (+0.060668260278835984) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.771538828460262 (-0.006794325062926987) | > avg_grad_norm_1: tensor(6767.6616, device='cuda:0') (+tensor(301.4639, device='cuda:0'))  > EPOCH: 1482/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:28:41)   --> TIME: 2025-02-25 23:29:03 -- STEP: 45/72 -- GLOBAL_STEP: 1106750 | > loss_disc: 2.56466007232666 (2.7320798185136588) | > loss_disc_real_0: 0.08981627225875854 (0.19673481856783231) | > loss_disc_real_1: 0.2127280980348587 (0.24474796073304278) | > loss_disc_real_2: 0.21363483369350433 (0.25652134550942324) | > loss_disc_real_3: 0.22446244955062866 (0.23935681084791818) | > loss_disc_real_4: 0.24795301258563995 (0.24565285212463803) | > loss_disc_real_5: 0.262743204832077 (0.24931917190551758) | > loss_0: 2.56466007232666 (2.7320798185136588) | > grad_norm_0: tensor(60.4706, device='cuda:0') (tensor(357.2725, device='cuda:0')) | > loss_gen: 2.127943992614746 (2.140917992591858) | > loss_kl: 2.3138036727905273 (2.203711165322198) | > loss_feat: 3.8080732822418213 (3.748309744728936) | > loss_mel: 16.871017456054688 (17.856923972235787) | > loss_duration: 1.5555715560913086 (1.5265107419755723) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.676410675048828 (27.476373842027453) | > grad_norm_1: tensor(4514.0454, device='cuda:0') (tensor(7317.8760, device='cuda:0')) | > current_lr_0: 0.00016617738255619356 | > current_lr_1: 0.00016617738255619356 | > step_time: 0.4511 (0.4363433043162028) | > loader_time: 0.0089 (0.0074545542399088545) --> EVAL PERFORMANCE | > avg_loader_time: 0.008571121054635924 (-8.220739767584792e-05) | > avg_step_time: 0.45519356660439936 (+0.004515983688999248) | > avg_loss_disc: 2.686366537926902 (+0.01851722556100821) | > avg_loss_disc_real_0: 0.16748433449948338 (+0.010467018197539785) | > avg_loss_disc_real_1: 0.23488406343779095 (-0.011920752554712144) | > avg_loss_disc_real_2: 0.24664057602345113 (+0.022069291329719742) | > avg_loss_disc_real_3: 0.23527853631637466 (+0.011374431911488653) | > avg_loss_disc_real_4: 0.2417123640926791 (+0.0043782442808151245) | > avg_loss_disc_real_5: 0.25022458967188715 (+0.0030380655342424623) | > avg_loss_0: 2.686366537926902 (+0.01851722556100821) | > avg_grad_norm_0: tensor(291.7830, device='cuda:0') (tensor(-20.5436, device='cuda:0')) | > avg_loss_gen: 2.145218244740661 (-0.002125167510878878) | > avg_loss_kl: 2.1776767935551393 (+0.026075620046804193) | > avg_loss_feat: 3.7906356327970263 (-0.11386134255100311) | > avg_loss_mel: 17.478928861483723 (+0.4928319689253726) | > avg_loss_duration: 1.544522154499108 (-0.03747831767713494) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.136981964111325 (+0.3654431356510628) | > avg_grad_norm_1: tensor(6623.3457, device='cuda:0') (tensor(-144.3159, device='cuda:0'))  > EPOCH: 1483/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:29:18)   --> TIME: 2025-02-25 23:29:30 -- STEP: 23/72 -- GLOBAL_STEP: 1106800 | > loss_disc: 2.463151454925537 (2.6747820791990864) | > loss_disc_real_0: 0.10917671024799347 (0.12134553087146385) | > loss_disc_real_1: 0.2562410831451416 (0.26650837582090625) | > loss_disc_real_2: 0.24402378499507904 (0.25781180547631305) | > loss_disc_real_3: 0.19937261939048767 (0.23276238273019376) | > loss_disc_real_4: 0.22564272582530975 (0.2468653470277786) | > loss_disc_real_5: 0.23966240882873535 (0.24671806848567465) | > loss_0: 2.463151454925537 (2.6747820791990864) | > grad_norm_0: tensor(85.5314, device='cuda:0') (tensor(278.2970, device='cuda:0')) | > loss_gen: 2.1071414947509766 (2.126719018687373) | > loss_kl: 2.156918525695801 (1.9667954600375632) | > loss_feat: 3.855891704559326 (3.812031787374745) | > loss_mel: 16.151647567749023 (17.07225932245669) | > loss_duration: 1.5100059509277344 (1.4863269691881926) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.781604766845703 (26.464132391888164) | > grad_norm_1: tensor(13578.5684, device='cuda:0') (tensor(7162.0015, device='cuda:0')) | > current_lr_0: 0.000166156610383374 | > current_lr_1: 0.000166156610383374 | > step_time: 0.4442 (0.4312708792479142) | > loader_time: 0.0065 (0.006952078446097996) --> EVAL PERFORMANCE | > avg_loader_time: 0.008686676831312587 (+0.00011555577667666378) | > avg_step_time: 0.45054140561063527 (-0.004652160993764087) | > avg_loss_disc: 2.6817415331450984 (-0.004625004781803721) | > avg_loss_disc_real_0: 0.1610788594878895 (-0.006405475011593886) | > avg_loss_disc_real_1: 0.24775275839886204 (+0.012868694961071098) | > avg_loss_disc_real_2: 0.24433999540100634 (-0.0023005806224447867) | > avg_loss_disc_real_3: 0.23302810670624316 (-0.002250429610131499) | > avg_loss_disc_real_4: 0.24550453541984021 (+0.0037921713271611146) | > avg_loss_disc_real_5: 0.24592950163592756 (-0.004295088035959599) | > avg_loss_0: 2.6817415331450984 (-0.004625004781803721) | > avg_grad_norm_0: tensor(222.9083, device='cuda:0') (tensor(-68.8746, device='cuda:0')) | > avg_loss_gen: 2.1099458210904833 (-0.035272423650177576) | > avg_loss_kl: 2.108339210631143 (-0.06933758292399617) | > avg_loss_feat: 3.677448672308049 (-0.11318696048897747) | > avg_loss_mel: 17.09335281479527 (-0.3855760466884526) | > avg_loss_duration: 1.5400609164170815 (-0.004461238082026364) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.529147430205008 (-0.6078345339063169) | > avg_grad_norm_1: tensor(6004.4946, device='cuda:0') (tensor(-618.8511, device='cuda:0'))  > EPOCH: 1484/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:29:55)   --> TIME: 2025-02-25 23:29:57 -- STEP: 1/72 -- GLOBAL_STEP: 1106850 | > loss_disc: 2.2218730449676514 (2.2218730449676514) | > loss_disc_real_0: 0.06403577327728271 (0.06403577327728271) | > loss_disc_real_1: 0.22690916061401367 (0.22690916061401367) | > loss_disc_real_2: 0.23252977430820465 (0.23252977430820465) | > loss_disc_real_3: 0.23092769086360931 (0.23092769086360931) | > loss_disc_real_4: 0.24435459077358246 (0.24435459077358246) | > loss_disc_real_5: 0.23781141638755798 (0.23781141638755798) | > loss_0: 2.2218730449676514 (2.2218730449676514) | > grad_norm_0: tensor(6.5578, device='cuda:0') (tensor(6.5578, device='cuda:0')) | > loss_gen: 2.9772021770477295 (2.9772021770477295) | > loss_kl: 2.0098111629486084 (2.0098111629486084) | > loss_feat: 5.403396129608154 (5.403396129608154) | > loss_mel: 17.303173065185547 (17.303173065185547) | > loss_duration: 1.504915475845337 (1.504915475845337) | > amp_scaler: 8.0 (8.0) | > loss_1: 29.198497772216797 (29.198497772216797) | > grad_norm_1: tensor(6187.9292, device='cuda:0') (tensor(6187.9292, device='cuda:0')) | > current_lr_0: 0.00016613584080707607 | > current_lr_1: 0.00016613584080707607 | > step_time: 0.4279 (0.4278721809387207) | > loader_time: 0.0047 (0.004680633544921875)  --> TIME: 2025-02-25 23:30:20 -- STEP: 51/72 -- GLOBAL_STEP: 1106900 | > loss_disc: 2.7399518489837646 (2.671632462856816) | > loss_disc_real_0: 0.0938539206981659 (0.08337855985497727) | > loss_disc_real_1: 0.2496371567249298 (0.23164243353348152) | > loss_disc_real_2: 0.2275518923997879 (0.24944264222593876) | > loss_disc_real_3: 0.2581712305545807 (0.24808420007135368) | > loss_disc_real_4: 0.2463764101266861 (0.2508721725613463) | > loss_disc_real_5: 0.25695833563804626 (0.24223433084347668) | > loss_0: 2.7399518489837646 (2.671632462856816) | > grad_norm_0: tensor(469.9862, device='cuda:0') (tensor(344.2845, device='cuda:0')) | > loss_gen: 1.8455982208251953 (2.123160203297934) | > loss_kl: 1.8870900869369507 (2.0195334635528863) | > loss_feat: 3.3360602855682373 (4.156572215697346) | > loss_mel: 15.489165306091309 (16.91561846639596) | > loss_duration: 1.5466656684875488 (1.511301412301905) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.104578018188477 (26.726185555551567) | > grad_norm_1: tensor(12651.3213, device='cuda:0') (tensor(10338.4062, device='cuda:0')) | > current_lr_0: 0.00016613584080707607 | > current_lr_1: 0.00016613584080707607 | > step_time: 0.4414 (0.43732137306063784) | > loader_time: 0.0091 (0.007582706563612994) --> EVAL PERFORMANCE | > avg_loader_time: 0.00846979651652591 (-0.00021688031478667678) | > avg_step_time: 0.44819171327940177 (-0.0023496923312334994) | > avg_loss_disc: 2.626773307021235 (-0.05496822612386332) | > avg_loss_disc_real_0: 0.0842637533820431 (-0.0768151061058464) | > avg_loss_disc_real_1: 0.22716823073340134 (-0.020584527665460706) | > avg_loss_disc_real_2: 0.240307296035995 (-0.004032699365011355) | > avg_loss_disc_real_3: 0.24269897690121567 (+0.009670870194972508) | > avg_loss_disc_real_4: 0.2458629692104501 (+0.00035843379060987446) | > avg_loss_disc_real_5: 0.24276996855164917 (-0.003159533084278382) | > avg_loss_0: 2.626773307021235 (-0.05496822612386332) | > avg_grad_norm_0: tensor(336.7536, device='cuda:0') (+tensor(113.8453, device='cuda:0')) | > avg_loss_gen: 2.135856537751749 (+0.025910716661265543) | > avg_loss_kl: 2.0412703198446356 (-0.06706889078650757) | > avg_loss_feat: 4.129718854393759 (+0.45227018208571) | > avg_loss_mel: 16.799820805939152 (-0.2935320088561184) | > avg_loss_duration: 1.5276823883325281 (-0.01237852808455342) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.63434880216357 (+0.10520137195856094) | > avg_grad_norm_1: tensor(10178.9893, device='cuda:0') (+tensor(4174.4946, device='cuda:0'))  > EPOCH: 1485/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:30:31)   --> TIME: 2025-02-25 23:30:46 -- STEP: 29/72 -- GLOBAL_STEP: 1106950 | > loss_disc: 2.5692381858825684 (2.4736983282812712) | > loss_disc_real_0: 0.07001438736915588 (0.08420282859226753) | > loss_disc_real_1: 0.2379305362701416 (0.20884445772088808) | > loss_disc_real_2: 0.18106617033481598 (0.211476665118645) | > loss_disc_real_3: 0.23429402709007263 (0.22358536257826048) | > loss_disc_real_4: 0.2305269092321396 (0.2302165499021267) | > loss_disc_real_5: 0.23841659724712372 (0.23885356660546928) | > loss_0: 2.5692381858825684 (2.4736983282812712) | > grad_norm_0: tensor(202.4802, device='cuda:0') (tensor(300.9651, device='cuda:0')) | > loss_gen: 2.0877490043640137 (2.214752028728354) | > loss_kl: 1.958570122718811 (1.9089872467106785) | > loss_feat: 4.232170104980469 (4.251674545222316) | > loss_mel: 16.39154815673828 (16.482207988870556) | > loss_duration: 1.4919102191925049 (1.4783451351626167) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.161949157714844 (26.335967228330414) | > grad_norm_1: tensor(10039.5869, device='cuda:0') (tensor(8794.2832, device='cuda:0')) | > current_lr_0: 0.00016611507382697518 | > current_lr_1: 0.00016611507382697518 | > step_time: 0.4342 (0.4304814996390507) | > loader_time: 0.0077 (0.00721736611991093) --> EVAL PERFORMANCE | > avg_loader_time: 0.00872836986058195 (+0.00025857334405603866) | > avg_step_time: 0.4556612431163519 (+0.007469529836950151) | > avg_loss_disc: 2.5109645783061714 (-0.11580872871506376) | > avg_loss_disc_real_0: 0.09112380161671571 (+0.006860048234672608) | > avg_loss_disc_real_1: 0.21821127454159964 (-0.008956956191801702) | > avg_loss_disc_real_2: 0.21799923122768675 (-0.022308064808308242) | > avg_loss_disc_real_3: 0.22656340246469203 (-0.016135574436523636) | > avg_loss_disc_real_4: 0.2349637767798464 (-0.010899192430603694) | > avg_loss_disc_real_5: 0.24480781651718517 (+0.002037847965535994) | > avg_loss_0: 2.5109645783061714 (-0.11580872871506376) | > avg_grad_norm_0: tensor(289.5617, device='cuda:0') (tensor(-47.1919, device='cuda:0')) | > avg_loss_gen: 2.20940851661521 (+0.0735519788634611) | > avg_loss_kl: 1.9761050670919285 (-0.06516525275270713) | > avg_loss_feat: 4.13110661506653 (+0.0013877606727712433) | > avg_loss_mel: 16.430003125902633 (-0.36981768003651894) | > avg_loss_duration: 1.5187808594233556 (-0.008901528909172507) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.265404392296162 (-0.3689444098674066) | > avg_grad_norm_1: tensor(8806.6992, device='cuda:0') (tensor(-1372.2900, device='cuda:0'))  > EPOCH: 1486/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:31:08)   --> TIME: 2025-02-25 23:31:13 -- STEP: 7/72 -- GLOBAL_STEP: 1107000 | > loss_disc: 2.465501308441162 (2.4881344522748674) | > loss_disc_real_0: 0.10939735174179077 (0.08975051237004143) | > loss_disc_real_1: 0.18883921205997467 (0.20612883354936326) | > loss_disc_real_2: 0.20540085434913635 (0.2213618414742606) | > loss_disc_real_3: 0.216617152094841 (0.22996306206498826) | > loss_disc_real_4: 0.20683100819587708 (0.2347601396696908) | > loss_disc_real_5: 0.22130921483039856 (0.24346137472561427) | > loss_0: 2.465501308441162 (2.4881344522748674) | > grad_norm_0: tensor(346.9059, device='cuda:0') (tensor(271.8842, device='cuda:0')) | > loss_gen: 2.2356066703796387 (2.244380235671997) | > loss_kl: 1.9382814168930054 (1.8390031371797835) | > loss_feat: 4.680214881896973 (4.129420450755528) | > loss_mel: 16.92050552368164 (16.351362773350306) | > loss_duration: 1.4584308862686157 (1.4744387524468558) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.23303985595703 (26.038605553763254) | > grad_norm_1: tensor(8799.3623, device='cuda:0') (tensor(8140.3984, device='cuda:0')) | > current_lr_0: 0.0001660943094427468 | > current_lr_1: 0.0001660943094427468 | > step_time: 0.428 (0.42976304462977816) | > loader_time: 0.0085 (0.0063402652740478516)  --> TIME: 2025-02-25 23:31:36 -- STEP: 57/72 -- GLOBAL_STEP: 1107050 | > loss_disc: 2.4752635955810547 (2.5045953382525523) | > loss_disc_real_0: 0.06823638826608658 (0.09455846269664012) | > loss_disc_real_1: 0.1852235049009323 (0.21441062186893664) | > loss_disc_real_2: 0.21478688716888428 (0.22311425444326902) | > loss_disc_real_3: 0.25627565383911133 (0.2309061589471081) | > loss_disc_real_4: 0.24358238279819489 (0.2379944193781468) | > loss_disc_real_5: 0.2416887879371643 (0.24420683870190069) | > loss_0: 2.4752635955810547 (2.5045953382525523) | > grad_norm_0: tensor(190.2745, device='cuda:0') (tensor(304.8882, device='cuda:0')) | > loss_gen: 2.147045612335205 (2.226955231867339) | > loss_kl: 1.8439855575561523 (1.931849759921693) | > loss_feat: 4.168590545654297 (4.134846289952594) | > loss_mel: 16.225019454956055 (16.426261751275312) | > loss_duration: 1.5449388027191162 (1.5056074021155375) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.92957878112793 (26.22552031801458) | > grad_norm_1: tensor(10339.0098, device='cuda:0') (tensor(8469.6035, device='cuda:0')) | > current_lr_0: 0.0001660943094427468 | > current_lr_1: 0.0001660943094427468 | > step_time: 0.482 (0.44138842298273456) | > loader_time: 0.0097 (0.007961611998708623) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584781431815994 (-0.00014358842876595508) | > avg_step_time: 0.4500672917970469 (-0.005593951319305013) | > avg_loss_disc: 2.5090047574379066 (-0.001959820868264739) | > avg_loss_disc_real_0: 0.09588328386906167 (+0.004759482252345959) | > avg_loss_disc_real_1: 0.21633809720966177 (-0.0018731773319378697) | > avg_loss_disc_real_2: 0.22361591626221025 (+0.005616685034523505) | > avg_loss_disc_real_3: 0.23087750291320638 (+0.004314100448514352) | > avg_loss_disc_real_4: 0.23631096578819652 (+0.0013471890083501248) | > avg_loss_disc_real_5: 0.24497026226050417 (+0.00016244574331900075) | > avg_loss_0: 2.5090047574379066 (-0.001959820868264739) | > avg_grad_norm_0: tensor(310.0005, device='cuda:0') (+tensor(20.4388, device='cuda:0')) | > avg_loss_gen: 2.2237427604030557 (+0.014334243787845757) | > avg_loss_kl: 1.9692784346325296 (-0.0068266324593988514) | > avg_loss_feat: 4.0819245560068484 (-0.04918205905968165) | > avg_loss_mel: 16.381591367049953 (-0.04841175885268001) | > avg_loss_duration: 1.518756369469871 (-2.4489953484696514e-05) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.175293452303176 (-0.0901109399929858) | > avg_grad_norm_1: tensor(8515.3398, device='cuda:0') (tensor(-291.3594, device='cuda:0'))  > EPOCH: 1487/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:31:45)   --> TIME: 2025-02-25 23:32:02 -- STEP: 35/72 -- GLOBAL_STEP: 1107100 | > loss_disc: 2.581495523452759 (2.5128474712371824) | > loss_disc_real_0: 0.11291868984699249 (0.09729688391089439) | > loss_disc_real_1: 0.2198387086391449 (0.21995964050292968) | > loss_disc_real_2: 0.24677008390426636 (0.2262449085712433) | > loss_disc_real_3: 0.26593512296676636 (0.22860954872199468) | > loss_disc_real_4: 0.2682039737701416 (0.23829351408141) | > loss_disc_real_5: 0.2739384174346924 (0.2453679152897426) | > loss_0: 2.581495523452759 (2.5128474712371824) | > grad_norm_0: tensor(202.1726, device='cuda:0') (tensor(319.2198, device='cuda:0')) | > loss_gen: 2.2827892303466797 (2.255145113808768) | > loss_kl: 2.0527822971343994 (1.9008318935121808) | > loss_feat: 3.739264965057373 (4.19253178324018) | > loss_mel: 16.77328872680664 (16.414402335030694) | > loss_duration: 1.5397568941116333 (1.4787017345428466) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.387882232666016 (26.241612570626394) | > grad_norm_1: tensor(6516.2632, device='cuda:0') (tensor(8543.8682, device='cuda:0')) | > current_lr_0: 0.00016607354765406644 | > current_lr_1: 0.00016607354765406644 | > step_time: 0.4583 (0.43486874444144114) | > loader_time: 0.0085 (0.007112033026559012) --> EVAL PERFORMANCE | > avg_loader_time: 0.008597420974516533 (+1.2639542700539075e-05) | > avg_step_time: 0.45295093764721506 (+0.0028836458501681483) | > avg_loss_disc: 2.5249545809248803 (+0.015949823486973713) | > avg_loss_disc_real_0: 0.09842990433246317 (+0.002546620463401497) | > avg_loss_disc_real_1: 0.22343581075399693 (+0.007097713544335166) | > avg_loss_disc_real_2: 0.22622458783673569 (+0.002608671574525434) | > avg_loss_disc_real_3: 0.22946126717077175 (-0.0014162357424346295) | > avg_loss_disc_real_4: 0.23932531035282242 (+0.0030143445646259037) | > avg_loss_disc_real_5: 0.24669275716157027 (+0.0017224949010661017) | > avg_loss_0: 2.5249545809248803 (+0.015949823486973713) | > avg_grad_norm_0: tensor(316.4592, device='cuda:0') (+tensor(6.4587, device='cuda:0')) | > avg_loss_gen: 2.241175844635761 (+0.017433084232705287) | > avg_loss_kl: 1.962924320932845 (-0.006354113699684527) | > avg_loss_feat: 4.1254330890279425 (+0.04350853302109403) | > avg_loss_mel: 16.431324945369244 (+0.04973357831929093) | > avg_loss_duration: 1.516147166910306 (-0.0026092025595649826) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.277005182185643 (+0.10171172988246724) | > avg_grad_norm_1: tensor(8683.2881, device='cuda:0') (+tensor(167.9482, device='cuda:0'))  > EPOCH: 1488/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:32:21)   --> TIME: 2025-02-25 23:32:29 -- STEP: 13/72 -- GLOBAL_STEP: 1107150 | > loss_disc: 2.558928966522217 (2.5088692078223596) | > loss_disc_real_0: 0.1084306612610817 (0.10550549311133531) | > loss_disc_real_1: 0.2077563852071762 (0.21593370919044202) | > loss_disc_real_2: 0.23767371475696564 (0.22840424455129182) | > loss_disc_real_3: 0.2272651046514511 (0.23916355004677406) | > loss_disc_real_4: 0.254957377910614 (0.2407474506359834) | > loss_disc_real_5: 0.2379463165998459 (0.24753663058464342) | > loss_0: 2.558928966522217 (2.5088692078223596) | > grad_norm_0: tensor(239.1804, device='cuda:0') (tensor(312.6751, device='cuda:0')) | > loss_gen: 2.281961441040039 (2.2958873418661265) | > loss_kl: 1.9320807456970215 (1.8658149609198937) | > loss_feat: 4.067636489868164 (4.238771310219398) | > loss_mel: 16.08108901977539 (16.72585575397198) | > loss_duration: 1.4200952053070068 (1.467634366108821) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.78286361694336 (26.59396406320425) | > grad_norm_1: tensor(3902.1260, device='cuda:0') (tensor(7957.4131, device='cuda:0')) | > current_lr_0: 0.00016605278846060968 | > current_lr_1: 0.00016605278846060968 | > step_time: 0.4533 (0.4318408232468825) | > loader_time: 0.0058 (0.0066425066727858325)  --> TIME: 2025-02-25 23:32:53 -- STEP: 63/72 -- GLOBAL_STEP: 1107200 | > loss_disc: 2.9208037853240967 (2.5615844537341403) | > loss_disc_real_0: 0.44959476590156555 (0.13469861240850556) | > loss_disc_real_1: 0.11278019845485687 (0.21229076420977003) | > loss_disc_real_2: 0.2050572633743286 (0.2265158241230344) | > loss_disc_real_3: 0.23198294639587402 (0.23246706785663726) | > loss_disc_real_4: 0.23244008421897888 (0.23852896004442184) | > loss_disc_real_5: 0.2294057011604309 (0.24510394935570065) | > loss_0: 2.9208037853240967 (2.5615844537341403) | > grad_norm_0: tensor(509.7133, device='cuda:0') (tensor(332.7889, device='cuda:0')) | > loss_gen: 2.0359549522399902 (2.2329737648131354) | > loss_kl: 2.112196683883667 (2.003312816695561) | > loss_feat: 3.57159686088562 (4.065099485336787) | > loss_mel: 16.850601196289062 (16.554664626954093) | > loss_duration: 1.543286919593811 (1.5094580574641148) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.113637924194336 (26.365508760724747) | > grad_norm_1: tensor(5086.0737, device='cuda:0') (tensor(7504.0112, device='cuda:0')) | > current_lr_0: 0.00016605278846060968 | > current_lr_1: 0.00016605278846060968 | > step_time: 0.5052 (0.4539017715151348) | > loader_time: 0.0113 (0.008199502551366412) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599798444291236 (+2.3774697747031004e-06) | > avg_step_time: 0.4597468980601136 (+0.0067959604128985385) | > avg_loss_disc: 2.5767970790325756 (+0.051842498107695256) | > avg_loss_disc_real_0: 0.1398427786016968 (+0.04141287426923364) | > avg_loss_disc_real_1: 0.21511677673584978 (-0.008319034018147153) | > avg_loss_disc_real_2: 0.22617866056905667 (-4.5927267679019e-05) | > avg_loss_disc_real_3: 0.2320071536470467 (+0.0025458864762749334) | > avg_loss_disc_real_4: 0.23867346514278734 (-0.0006518452100350847) | > avg_loss_disc_real_5: 0.24422704522878352 (-0.002465711932786746) | > avg_loss_0: 2.5767970790325756 (+0.051842498107695256) | > avg_grad_norm_0: tensor(330.4673, device='cuda:0') (+tensor(14.0081, device='cuda:0')) | > avg_loss_gen: 2.21390532775664 (-0.027270516879120965) | > avg_loss_kl: 2.0234440380418803 (+0.060519717109035254) | > avg_loss_feat: 4.010485706195024 (-0.11494738283291817) | > avg_loss_mel: 16.579335212707523 (+0.14801026733827882) | > avg_loss_duration: 1.5181524132339044 (+0.002005246323598442) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.34532278356418 (+0.06831760137853493) | > avg_grad_norm_1: tensor(7570.6328, device='cuda:0') (tensor(-1112.6553, device='cuda:0'))  > EPOCH: 1489/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:32:59)   --> TIME: 2025-02-25 23:33:19 -- STEP: 41/72 -- GLOBAL_STEP: 1107250 | > loss_disc: 2.4006919860839844 (2.53101729765171) | > loss_disc_real_0: 0.12654295563697815 (0.10432450073521311) | > loss_disc_real_1: 0.2108507603406906 (0.22005319631681208) | > loss_disc_real_2: 0.2292722463607788 (0.23030991488840521) | > loss_disc_real_3: 0.25179073214530945 (0.22852987051010132) | > loss_disc_real_4: 0.2517663836479187 (0.23925744542261448) | > loss_disc_real_5: 0.25505709648132324 (0.24383485135508748) | > loss_0: 2.4006919860839844 (2.53101729765171) | > grad_norm_0: tensor(233.8960, device='cuda:0') (tensor(332.3528, device='cuda:0')) | > loss_gen: 2.6154110431671143 (2.2200380447434216) | > loss_kl: 1.8965057134628296 (2.0199242103390582) | > loss_feat: 4.219749927520752 (4.133882208568292) | > loss_mel: 17.1804141998291 (16.613364731393208) | > loss_duration: 1.5295844078063965 (1.4926162085882049) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.441665649414062 (26.47982541526236) | > grad_norm_1: tensor(5174.0757, device='cuda:0') (tensor(8068.5752, device='cuda:0')) | > current_lr_0: 0.0001660320318620521 | > current_lr_1: 0.0001660320318620521 | > step_time: 0.4908 (0.43791322010319406) | > loader_time: 0.0095 (0.007348589780854016) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573783955103914 (-2.6014489187322146e-05) | > avg_step_time: 0.4499230317666497 (-0.009823866293463879) | > avg_loss_disc: 2.547539687492478 (-0.029257391540097544) | > avg_loss_disc_real_0: 0.11016440716847567 (-0.029678371433221135) | > avg_loss_disc_real_1: 0.22123854859194286 (+0.006121771856093083) | > avg_loss_disc_real_2: 0.22824872232658763 (+0.0020700617575309666) | > avg_loss_disc_real_3: 0.22926095484847753 (-0.0027461987985691605) | > avg_loss_disc_real_4: 0.2394645931855054 (+0.0007911280427180534) | > avg_loss_disc_real_5: 0.24525160499861542 (+0.0010245597698319009) | > avg_loss_0: 2.547539687492478 (-0.029257391540097544) | > avg_grad_norm_0: tensor(337.3020, device='cuda:0') (+tensor(6.8347, device='cuda:0')) | > avg_loss_gen: 2.216758889211736 (+0.0028535614550957433) | > avg_loss_kl: 2.0584253996190904 (+0.0349813615772101) | > avg_loss_feat: 4.084305044630884 (+0.07381933843586008) | > avg_loss_mel: 16.604762010171374 (+0.025426797463850903) | > avg_loss_duration: 1.5203620262548958 (+0.00220961302099143) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.48461347230723 (+0.13929068874305273) | > avg_grad_norm_1: tensor(8066.1973, device='cuda:0') (+tensor(495.5645, device='cuda:0'))  > EPOCH: 1490/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:33:35)   --> TIME: 2025-02-25 23:33:46 -- STEP: 19/72 -- GLOBAL_STEP: 1107300 | > loss_disc: 2.770083427429199 (2.7848305451242545) | > loss_disc_real_0: 0.3152487874031067 (0.17769957294589594) | > loss_disc_real_1: 0.1938444823026657 (0.19581461344894616) | > loss_disc_real_2: 0.38794559240341187 (0.3481012284755707) | > loss_disc_real_3: 0.2754167914390564 (0.2282115228866276) | > loss_disc_real_4: 0.36111754179000854 (0.30871717239681046) | > loss_disc_real_5: 0.2895643711090088 (0.2525378889159152) | > loss_0: 2.770083427429199 (2.7848305451242545) | > grad_norm_0: tensor(625.4254, device='cuda:0') (tensor(320.3703, device='cuda:0')) | > loss_gen: 2.8321444988250732 (2.5060681920302543) | > loss_kl: 2.345980167388916 (2.104123504538285) | > loss_feat: 3.7327702045440674 (4.297872217077958) | > loss_mel: 17.076257705688477 (17.252522368180124) | > loss_duration: 1.6101839542388916 (1.4970093149887889) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.597335815429688 (27.657595785040606) | > grad_norm_1: tensor(1417.6006, device='cuda:0') (tensor(4705.3794, device='cuda:0')) | > current_lr_0: 0.00016601127785806932 | > current_lr_1: 0.00016601127785806932 | > step_time: 0.4197 (0.4286604053095767) | > loader_time: 0.006 (0.006529808044433594)  --> TIME: 2025-02-25 23:34:09 -- STEP: 69/72 -- GLOBAL_STEP: 1107350 | > loss_disc: 2.6409592628479004 (2.6851691577745522) | > loss_disc_real_0: 0.18781757354736328 (0.13987140508665555) | > loss_disc_real_1: 0.2484709918498993 (0.23440832137197687) | > loss_disc_real_2: 0.245000958442688 (0.26980749247730634) | > loss_disc_real_3: 0.22326625883579254 (0.2328296619049017) | > loss_disc_real_4: 0.2415655255317688 (0.2582219713839933) | > loss_disc_real_5: 0.2451896220445633 (0.24682003475617673) | > loss_0: 2.6409592628479004 (2.6851691577745522) | > grad_norm_0: tensor(120.9178, device='cuda:0') (tensor(254.5103, device='cuda:0')) | > loss_gen: 2.331280469894409 (2.3155655670857085) | > loss_kl: 2.20318341255188 (2.201781606328659) | > loss_feat: 3.514644145965576 (4.017601299976959) | > loss_mel: 16.879791259765625 (17.07904016798821) | > loss_duration: 1.6322269439697266 (1.5429775490277056) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.561124801635742 (27.156966306161188) | > grad_norm_1: tensor(6658.4038, device='cuda:0') (tensor(4195.1279, device='cuda:0')) | > current_lr_0: 0.00016601127785806932 | > current_lr_1: 0.00016601127785806932 | > step_time: 0.5333 (0.4483101713484612) | > loader_time: 0.0126 (0.008485859718875607) --> EVAL PERFORMANCE | > avg_loader_time: 0.008541631027006766 (-3.2152928097148314e-05) | > avg_step_time: 0.44989678893290774 (-2.6242833741974003e-05) | > avg_loss_disc: 2.682219300471561 (+0.13467961297908282) | > avg_loss_disc_real_0: 0.1386100324526639 (+0.02844562528418823) | > avg_loss_disc_real_1: 0.23420233226997753 (+0.012963783678034663) | > avg_loss_disc_real_2: 0.2690578153435612 (+0.04080909301697358) | > avg_loss_disc_real_3: 0.23249282719383776 (+0.003231872345360237) | > avg_loss_disc_real_4: 0.25733091398863733 (+0.01786632080313194) | > avg_loss_disc_real_5: 0.24655175104107654 (+0.00130014604246112) | > avg_loss_0: 2.682219300471561 (+0.13467961297908282) | > avg_grad_norm_0: tensor(254.2912, device='cuda:0') (tensor(-83.0107, device='cuda:0')) | > avg_loss_gen: 2.312093095040657 (+0.09533420582892127) | > avg_loss_kl: 2.1988864549448786 (+0.14046105532578812) | > avg_loss_feat: 4.020430752928829 (-0.06387429170205561) | > avg_loss_mel: 17.067523311561263 (+0.46276130138988947) | > avg_loss_duration: 1.5430814380377111 (+0.022719411782815335) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 27.142015188512666 (+0.6574017162054346) | > avg_grad_norm_1: tensor(4475.1592, device='cuda:0') (tensor(-3591.0381, device='cuda:0'))  > EPOCH: 1491/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:34:12)   --> TIME: 2025-02-25 23:34:35 -- STEP: 47/72 -- GLOBAL_STEP: 1107400 | > loss_disc: 2.598416328430176 (2.701150802855796) | > loss_disc_real_0: 0.06297406554222107 (0.16632594865687347) | > loss_disc_real_1: 0.2397945374250412 (0.24199997427615713) | > loss_disc_real_2: 0.2493201643228531 (0.24366158754267592) | > loss_disc_real_3: 0.24331212043762207 (0.23836142870974034) | > loss_disc_real_4: 0.26083171367645264 (0.2493181415694826) | > loss_disc_real_5: 0.2873169481754303 (0.24712827421249234) | > loss_0: 2.598416328430176 (2.701150802855796) | > grad_norm_0: tensor(201.0185, device='cuda:0') (tensor(297.0503, device='cuda:0')) | > loss_gen: 2.031003713607788 (2.1491863829024296) | > loss_kl: 2.2943906784057617 (2.418625661667357) | > loss_feat: 3.8436813354492188 (3.800383334464215) | > loss_mel: 18.176847457885742 (18.665961671382824) | > loss_duration: 1.6047959327697754 (1.6284231784495902) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.95071792602539 (28.662580124875333) | > grad_norm_1: tensor(9504.7148, device='cuda:0') (tensor(6254.9473, device='cuda:0')) | > current_lr_0: 0.00016599052644833705 | > current_lr_1: 0.00016599052644833705 | > step_time: 0.4495 (0.449555219487941) | > loader_time: 0.0091 (0.007897894433204164) --> EVAL PERFORMANCE | > avg_loader_time: 0.008792907419339033 (+0.0002512763923322673) | > avg_step_time: 0.4582708352048632 (+0.008374046271955482) | > avg_loss_disc: 2.6639860180062307 (-0.01823328246533018) | > avg_loss_disc_real_0: 0.14612549884428436 (+0.007515466391620457) | > avg_loss_disc_real_1: 0.2356942314916933 (+0.0014918992217157756) | > avg_loss_disc_real_2: 0.24359064370813505 (-0.025467171635426167) | > avg_loss_disc_real_3: 0.23430326937789647 (+0.0018104421840587082) | > avg_loss_disc_real_4: 0.2470706894364156 (-0.010260224552221725) | > avg_loss_disc_real_5: 0.2508347338773835 (+0.004282982836306931) | > avg_loss_0: 2.6639860180062307 (-0.01823328246533018) | > avg_grad_norm_0: tensor(274.8914, device='cuda:0') (+tensor(20.6002, device='cuda:0')) | > avg_loss_gen: 2.1498088685559558 (-0.1622842264847013) | > avg_loss_kl: 2.3862151411217694 (+0.1873286861768908) | > avg_loss_feat: 3.7962247317945454 (-0.22420602113428334) | > avg_loss_mel: 18.291178219754926 (+1.2236549081936623) | > avg_loss_duration: 1.6174825043745444 (+0.0744010663368333) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 28.24090944209569 (+1.0988942535830226) | > avg_grad_norm_1: tensor(5777.1885, device='cuda:0') (+tensor(1302.0293, device='cuda:0'))  > EPOCH: 1492/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:34:49)   --> TIME: 2025-02-25 23:35:02 -- STEP: 25/72 -- GLOBAL_STEP: 1107450 | > loss_disc: 2.3365604877471924 (2.522210092544556) | > loss_disc_real_0: 0.05386016517877579 (0.06868613585829733) | > loss_disc_real_1: 0.16341039538383484 (0.2208511561155319) | > loss_disc_real_2: 0.19328546524047852 (0.22443708300590515) | > loss_disc_real_3: 0.16632449626922607 (0.24528461039066315) | > loss_disc_real_4: 0.23547296226024628 (0.24853353083133697) | > loss_disc_real_5: 0.24504823982715607 (0.2481343364715576) | > loss_0: 2.3365604877471924 (2.522210092544556) | > grad_norm_0: tensor(21.6670, device='cuda:0') (tensor(221.3320, device='cuda:0')) | > loss_gen: 2.4413845539093018 (2.2435738849639892) | > loss_kl: 1.8508607149124146 (1.897079243659973) | > loss_feat: 4.807378768920898 (4.271049222946166) | > loss_mel: 16.607730865478516 (16.86116123199463) | > loss_duration: 1.4777209758758545 (1.488732705116272) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.185075759887695 (26.76159637451172) | > grad_norm_1: tensor(3318.2112, device='cuda:0') (tensor(5946.0801, device='cuda:0')) | > current_lr_0: 0.000165969777632531 | > current_lr_1: 0.000165969777632531 | > step_time: 0.4387 (0.4346965980529785) | > loader_time: 0.0077 (0.006784038543701172) --> EVAL PERFORMANCE | > avg_loader_time: 0.008600382737710445 (-0.00019252468162858812) | > avg_step_time: 0.45341015533662177 (-0.004860679868241458) | > avg_loss_disc: 2.5535907711781243 (-0.11039524682810642) | > avg_loss_disc_real_0: 0.08290359362120356 (-0.0632219052230808) | > avg_loss_disc_real_1: 0.2243424314008632 (-0.011351800090830089) | > avg_loss_disc_real_2: 0.22502930735198545 (-0.018561336356149594) | > avg_loss_disc_real_3: 0.2285035292447453 (-0.005799740133151182) | > avg_loss_disc_real_4: 0.23798391642704816 (-0.009086773009367444) | > avg_loss_disc_real_5: 0.24633394412591425 (-0.004500789751469225) | > avg_loss_0: 2.5535907711781243 (-0.11039524682810642) | > avg_grad_norm_0: tensor(231.5151, device='cuda:0') (tensor(-43.3764, device='cuda:0')) | > avg_loss_gen: 2.1982032262103655 (+0.04839435765440969) | > avg_loss_kl: 1.970090753595594 (-0.4161243875261753) | > avg_loss_feat: 4.174134983143338 (+0.3779102513487924) | > avg_loss_mel: 16.63637726743457 (-1.6548009523203575) | > avg_loss_duration: 1.5271789926878163 (-0.09030351168672812) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.505985206281636 (-1.7349242358140522) | > avg_grad_norm_1: tensor(6195.3433, device='cuda:0') (+tensor(418.1548, device='cuda:0'))  > EPOCH: 1493/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:35:26)   --> TIME: 2025-02-25 23:35:29 -- STEP: 3/72 -- GLOBAL_STEP: 1107500 | > loss_disc: 2.328023910522461 (2.474876801172892) | > loss_disc_real_0: 0.08196298778057098 (0.1270693118373553) | > loss_disc_real_1: 0.2079516500234604 (0.20668113231658936) | > loss_disc_real_2: 0.21610283851623535 (0.22646590570608774) | > loss_disc_real_3: 0.20377576351165771 (0.21125587821006775) | > loss_disc_real_4: 0.19000698626041412 (0.19734645386536917) | > loss_disc_real_5: 0.22139661014080048 (0.22961472968260446) | > loss_0: 2.328023910522461 (2.474876801172892) | > grad_norm_0: tensor(50.8547, device='cuda:0') (tensor(265.8963, device='cuda:0')) | > loss_gen: 2.412255048751831 (2.356825510660807) | > loss_kl: 1.8965628147125244 (1.8705292542775471) | > loss_feat: 4.597738265991211 (4.365386327107747) | > loss_mel: 16.794660568237305 (16.824085235595703) | > loss_duration: 1.4886741638183594 (1.481586217880249) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.189889907836914 (26.898412068684895) | > grad_norm_1: tensor(8919.6826, device='cuda:0') (tensor(7858.5127, device='cuda:0')) | > current_lr_0: 0.00016594903141032693 | > current_lr_1: 0.00016594903141032693 | > step_time: 0.4342 (0.43312811851501465) | > loader_time: 0.0065 (0.005906581878662109)  --> TIME: 2025-02-25 23:35:52 -- STEP: 53/72 -- GLOBAL_STEP: 1107550 | > loss_disc: 2.454362154006958 (2.481799858920979) | > loss_disc_real_0: 0.10243936628103256 (0.09449330478344324) | > loss_disc_real_1: 0.2112547755241394 (0.2175922098587144) | > loss_disc_real_2: 0.22638531029224396 (0.2309674232635858) | > loss_disc_real_3: 0.2147120237350464 (0.23040548079418685) | > loss_disc_real_4: 0.23720158636569977 (0.23911857436288078) | > loss_disc_real_5: 0.23121528327465057 (0.24645927042331336) | > loss_0: 2.454362154006958 (2.481799858920979) | > grad_norm_0: tensor(249.1298, device='cuda:0') (tensor(224.1302, device='cuda:0')) | > loss_gen: 2.4307868480682373 (2.324607430763965) | > loss_kl: 1.9532876014709473 (1.9066367576707084) | > loss_feat: 4.370697021484375 (4.211953752445726) | > loss_mel: 15.940370559692383 (16.419581953084684) | > loss_duration: 1.542643427848816 (1.5001789308943838) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.237783432006836 (26.362958836105634) | > grad_norm_1: tensor(2971.1460, device='cuda:0') (tensor(6112.2109, device='cuda:0')) | > current_lr_0: 0.00016594903141032693 | > current_lr_1: 0.00016594903141032693 | > step_time: 0.445 (0.44207330919661614) | > loader_time: 0.0091 (0.007715693059957253) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534381087397189 (-6.600165031325593e-05) | > avg_step_time: 0.4539705296637307 (+0.0005603743271089257) | > avg_loss_disc: 2.5163580061684194 (-0.03723276500970485) | > avg_loss_disc_real_0: 0.12808257677185703 (+0.04517898315065347) | > avg_loss_disc_real_1: 0.22091493073483587 (-0.0034275006660273444) | > avg_loss_disc_real_2: 0.23068365811462133 (+0.0056543507626358736) | > avg_loss_disc_real_3: 0.2286826256715076 (+0.00017909642676230964) | > avg_loss_disc_real_4: 0.24105310398088375 (+0.0030691875538355895) | > avg_loss_disc_real_5: 0.24819902278168102 (+0.0018650786557667665) | > avg_loss_0: 2.5163580061684194 (-0.03723276500970485) | > avg_grad_norm_0: tensor(261.9912, device='cuda:0') (+tensor(30.4761, device='cuda:0')) | > avg_loss_gen: 2.3107820396691983 (+0.1125788134588328) | > avg_loss_kl: 1.9449356022015425 (-0.025155151394051556) | > avg_loss_feat: 4.096876308951579 (-0.07725867419175891) | > avg_loss_mel: 16.42473422305685 (-0.21164304437771975) | > avg_loss_duration: 1.5180419565926135 (-0.009137036095202777) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.295370128792776 (-0.21061507748886044) | > avg_grad_norm_1: tensor(5813.3604, device='cuda:0') (tensor(-381.9829, device='cuda:0'))  > EPOCH: 1494/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:36:03)   --> TIME: 2025-02-25 23:36:19 -- STEP: 31/72 -- GLOBAL_STEP: 1107600 | > loss_disc: 2.6746749877929688 (2.661282754713489) | > loss_disc_real_0: 0.23629361391067505 (0.1828227958852245) | > loss_disc_real_1: 0.2323031723499298 (0.2433985202543197) | > loss_disc_real_2: 0.23475831747055054 (0.23314196544308816) | > loss_disc_real_3: 0.23528814315795898 (0.2404871174404698) | > loss_disc_real_4: 0.23443882167339325 (0.24818661616694543) | > loss_disc_real_5: 0.24445085227489471 (0.24442644128876348) | > loss_0: 2.6746749877929688 (2.661282754713489) | > grad_norm_0: tensor(94.9883, device='cuda:0') (tensor(284.2058, device='cuda:0')) | > loss_gen: 2.3179616928100586 (2.2018727756315664) | > loss_kl: 2.448606491088867 (2.062440726064867) | > loss_feat: 3.692876100540161 (3.9411842669210126) | > loss_mel: 16.689382553100586 (17.12451381068076) | > loss_duration: 1.5708386898040771 (1.494704265748301) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.71966552734375 (26.824716137301536) | > grad_norm_1: tensor(6095.7627, device='cuda:0') (tensor(6499.8169, device='cuda:0')) | > current_lr_0: 0.00016592828778140063 | > current_lr_1: 0.00016592828778140063 | > step_time: 0.4443 (0.4366027155230122) | > loader_time: 0.0091 (0.007084331204814296) --> EVAL PERFORMANCE | > avg_loader_time: 0.008803908254059264 (+0.0002695271666620751) | > avg_step_time: 0.4526804803122937 (-0.001290049351437006) | > avg_loss_disc: 2.6559684142260482 (+0.1396104080576288) | > avg_loss_disc_real_0: 0.15088669903261565 (+0.02280412226075862) | > avg_loss_disc_real_1: 0.24276781879680256 (+0.021852888061966697) | > avg_loss_disc_real_2: 0.23061999447748693 (-6.366363713439238e-05) | > avg_loss_disc_real_3: 0.2355575248808928 (+0.0068748992093851935) | > avg_loss_disc_real_4: 0.24414828671535976 (+0.0030951827344760097) | > avg_loss_disc_real_5: 0.24409166692008435 (-0.004107355861596662) | > avg_loss_0: 2.6559684142260482 (+0.1396104080576288) | > avg_grad_norm_0: tensor(247.8031, device='cuda:0') (tensor(-14.1881, device='cuda:0')) | > avg_loss_gen: 2.1331395851054653 (-0.17764245456373295) | > avg_loss_kl: 2.1851887451091283 (+0.24025314290758581) | > avg_loss_feat: 3.864069938659668 (-0.23280637029191098) | > avg_loss_mel: 17.167105137462354 (+0.7423709144055053) | > avg_loss_duration: 1.5440190946552115 (+0.025977138062597938) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.893522772990487 (+0.598152644197711) | > avg_grad_norm_1: tensor(5991.7759, device='cuda:0') (+tensor(178.4155, device='cuda:0'))  > EPOCH: 1495/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:36:40)   --> TIME: 2025-02-25 23:36:46 -- STEP: 9/72 -- GLOBAL_STEP: 1107650 | > loss_disc: 2.7088241577148438 (2.7032660643259683) | > loss_disc_real_0: 0.27623170614242554 (0.3674694862630632) | > loss_disc_real_1: 0.21404387056827545 (0.2240209182103475) | > loss_disc_real_2: 0.2068624347448349 (0.2192657838265101) | > loss_disc_real_3: 0.2574669420719147 (0.21011961499849954) | > loss_disc_real_4: 0.21282029151916504 (0.209701269865036) | > loss_disc_real_5: 0.2271084487438202 (0.2165616038772795) | > loss_0: 2.7088241577148438 (2.7032660643259683) | > grad_norm_0: tensor(306.7609, device='cuda:0') (tensor(323.3916, device='cuda:0')) | > loss_gen: 2.0050878524780273 (2.3756260342068143) | > loss_kl: 2.160940170288086 (2.231169435713026) | > loss_feat: 3.6246585845947266 (3.9026522636413574) | > loss_mel: 17.378110885620117 (17.625282923380535) | > loss_duration: 1.5166432857513428 (1.5086860921647813) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.685440063476562 (27.643417358398438) | > grad_norm_1: tensor(6426.5288, device='cuda:0') (tensor(3291.8972, device='cuda:0')) | > current_lr_0: 0.00016590754674542795 | > current_lr_1: 0.00016590754674542795 | > step_time: 0.4251 (0.4301968150668674) | > loader_time: 0.0056 (0.006273614035712348)  --> TIME: 2025-02-25 23:37:10 -- STEP: 59/72 -- GLOBAL_STEP: 1107700 | > loss_disc: 2.8059606552124023 (2.6681146136784957) | > loss_disc_real_0: 0.03753777593374252 (0.18162027885348117) | > loss_disc_real_1: 0.2399291694164276 (0.21142770728822483) | > loss_disc_real_2: 0.2494077980518341 (0.24734970851469848) | > loss_disc_real_3: 0.23757195472717285 (0.2240387466499361) | > loss_disc_real_4: 0.25041714310646057 (0.24781436106916196) | > loss_disc_real_5: 0.22944249212741852 (0.24010452981722558) | > loss_0: 2.8059606552124023 (2.6681146136784957) | > grad_norm_0: tensor(86.3609, device='cuda:0') (tensor(277.1767, device='cuda:0')) | > loss_gen: 1.7111053466796875 (2.1844089091834364) | > loss_kl: 1.9242327213287354 (2.0820633698317965) | > loss_feat: 3.697105646133423 (3.9487834259615107) | > loss_mel: 16.857948303222656 (17.079051373368603) | > loss_duration: 1.5519609451293945 (1.5236343812134305) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.742351531982422 (26.817941568665585) | > grad_norm_1: tensor(4019.0120, device='cuda:0') (tensor(5093.1182, device='cuda:0')) | > current_lr_0: 0.00016590754674542795 | > current_lr_1: 0.00016590754674542795 | > step_time: 0.48 (0.4531594898741124) | > loader_time: 0.0101 (0.008290816161592128) --> EVAL PERFORMANCE | > avg_loader_time: 0.00882392534067933 (+2.0017086620065933e-05) | > avg_step_time: 0.4608540031271921 (+0.008173522814898415) | > avg_loss_disc: 2.652571694951662 (-0.0033967192743862284) | > avg_loss_disc_real_0: 0.17184457235353098 (+0.020957873320915338) | > avg_loss_disc_real_1: 0.21350488024698178 (-0.02926293854982079) | > avg_loss_disc_real_2: 0.24669875687276813 (+0.016078762395281193) | > avg_loss_disc_real_3: 0.2240819297206234 (-0.011475595160269386) | > avg_loss_disc_real_4: 0.24603236057388944 (+0.001884073858529678) | > avg_loss_disc_real_5: 0.24138718192846004 (-0.0027044849916243174) | > avg_loss_0: 2.652571694951662 (-0.0033967192743862284) | > avg_grad_norm_0: tensor(276.6878, device='cuda:0') (+tensor(28.8848, device='cuda:0')) | > avg_loss_gen: 2.1904490061209225 (+0.05730942101545722) | > avg_loss_kl: 2.090315299974361 (-0.09487344513476748) | > avg_loss_feat: 3.935342466327506 (+0.07127252766783787) | > avg_loss_mel: 16.99135823316977 (-0.1757469042925841) | > avg_loss_duration: 1.5337473036537708 (-0.010271791001440711) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.741212361295457 (-0.15231041169503) | > avg_grad_norm_1: tensor(4798.2686, device='cuda:0') (tensor(-1193.5073, device='cuda:0'))  > EPOCH: 1496/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:37:17)   --> TIME: 2025-02-25 23:37:37 -- STEP: 37/72 -- GLOBAL_STEP: 1107750 | > loss_disc: 2.6239328384399414 (2.5831959118714205) | > loss_disc_real_0: 0.11966461688280106 (0.12659412129102526) | > loss_disc_real_1: 0.28352174162864685 (0.21241445718584834) | > loss_disc_real_2: 0.23268528282642365 (0.21460808128923983) | > loss_disc_real_3: 0.2454162836074829 (0.22919121505440893) | > loss_disc_real_4: 0.2519517242908478 (0.2302594837304708) | > loss_disc_real_5: 0.2660110294818878 (0.24513836003638603) | > loss_0: 2.6239328384399414 (2.5831959118714205) | > grad_norm_0: tensor(67.8401, device='cuda:0') (tensor(361.4308, device='cuda:0')) | > loss_gen: 2.1894352436065674 (2.0901201415706323) | > loss_kl: 1.815295934677124 (1.907446516526712) | > loss_feat: 3.5095624923706055 (3.9291979042259424) | > loss_mel: 15.609607696533203 (16.6417240452122) | > loss_duration: 1.5053167343139648 (1.4885273011955056) | > amp_scaler: 8.0 (8.0) | > loss_1: 24.62921905517578 (26.05701596028096) | > grad_norm_1: tensor(10402.8711, device='cuda:0') (tensor(9209.1387, device='cuda:0')) | > current_lr_0: 0.00016588680830208476 | > current_lr_1: 0.00016588680830208476 | > step_time: 0.4941 (0.44847012210536646) | > loader_time: 0.0084 (0.007422234560992266) --> EVAL PERFORMANCE | > avg_loader_time: 0.008814858718657158 (-9.0666220221719e-06) | > avg_step_time: 0.4658779560680121 (+0.005023952940819976) | > avg_loss_disc: 2.597086510188143 (-0.05548518476351916) | > avg_loss_disc_real_0: 0.12277724404989833 (-0.049067328303632654) | > avg_loss_disc_real_1: 0.21259260954151687 (-0.0009122707054649015) | > avg_loss_disc_real_2: 0.22312606787177877 (-0.023572689000989355) | > avg_loss_disc_real_3: 0.23030276260745358 (+0.006220832886830174) | > avg_loss_disc_real_4: 0.23619750792711552 (-0.009834852646773917) | > avg_loss_disc_real_5: 0.24465748982530244 (+0.003270307896842406) | > avg_loss_0: 2.597086510188143 (-0.05548518476351916) | > avg_grad_norm_0: tensor(301.3963, device='cuda:0') (+tensor(24.7085, device='cuda:0')) | > avg_loss_gen: 2.0869043259553504 (-0.10354468016557217) | > avg_loss_kl: 1.9692313318521204 (-0.12108396812224043) | > avg_loss_feat: 3.8475800198568426 (-0.08776244647066322) | > avg_loss_mel: 16.587649734926895 (-0.40370849824287447) | > avg_loss_duration: 1.523049492231557 (-0.010697811422213821) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.01441493504484 (-0.7267974262506165) | > avg_grad_norm_1: tensor(7722.3872, device='cuda:0') (+tensor(2924.1187, device='cuda:0'))  > EPOCH: 1497/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:37:55)   --> TIME: 2025-02-25 23:38:04 -- STEP: 15/72 -- GLOBAL_STEP: 1107800 | > loss_disc: 2.737356185913086 (2.6492709000905355) | > loss_disc_real_0: 0.046125102788209915 (0.14263822014133135) | > loss_disc_real_1: 0.3938210606575012 (0.24926733275254567) | > loss_disc_real_2: 0.2252066284418106 (0.22703824440638223) | > loss_disc_real_3: 0.21663589775562286 (0.2175336480140686) | > loss_disc_real_4: 0.2553408443927765 (0.24469871719678243) | > loss_disc_real_5: 0.21519026160240173 (0.23593251605828602) | > loss_0: 2.737356185913086 (2.6492709000905355) | > grad_norm_0: tensor(249.8602, device='cuda:0') (tensor(382.6851, device='cuda:0')) | > loss_gen: 1.9786056280136108 (2.0712656736373902) | > loss_kl: 1.979805827140808 (2.075966930389404) | > loss_feat: 3.9833312034606934 (3.832327874501546) | > loss_mel: 18.653947830200195 (18.050306701660165) | > loss_duration: 1.5048421621322632 (1.4945465882619222) | > amp_scaler: 8.0 (8.0) | > loss_1: 28.10053253173828 (27.524413553873696) | > grad_norm_1: tensor(11302.5693, device='cuda:0') (tensor(7749.2295, device='cuda:0')) | > current_lr_0: 0.00016586607245104698 | > current_lr_1: 0.00016586607245104698 | > step_time: 0.4416 (0.4491230646769206) | > loader_time: 0.0063 (0.0069538593292236325)  --> TIME: 2025-02-25 23:38:28 -- STEP: 65/72 -- GLOBAL_STEP: 1107850 | > loss_disc: 3.3643219470977783 (2.6886216493753285) | > loss_disc_real_0: 0.07356441766023636 (0.13733436084137515) | > loss_disc_real_1: 0.08868227899074554 (0.20834218659079992) | > loss_disc_real_2: 0.15249861776828766 (0.23338239754621798) | > loss_disc_real_3: 0.14688126742839813 (0.22106761370713895) | > loss_disc_real_4: 0.25774329900741577 (0.24906114339828492) | > loss_disc_real_5: 0.22059498727321625 (0.23768131389067723) | > loss_0: 3.3643219470977783 (2.6886216493753285) | > grad_norm_0: tensor(351.4109, device='cuda:0') (tensor(345.1515, device='cuda:0')) | > loss_gen: 1.2705276012420654 (2.0845549234977137) | > loss_kl: 2.308755397796631 (2.1355157962212186) | > loss_feat: 3.0204174518585205 (3.8978899735670822) | > loss_mel: 16.413652420043945 (17.302552325908962) | > loss_duration: 1.621842384338379 (1.5437172211133516) | > amp_scaler: 8.0 (10.092307692307692) | > loss_1: 24.63519287109375 (26.96423014127291) | > grad_norm_1: tensor(11224.2109, device='cuda:0') (tensor(7176.2725, device='cuda:0')) | > current_lr_0: 0.00016586607245104698 | > current_lr_1: 0.00016586607245104698 | > step_time: 0.5385 (0.4623919413639949) | > loader_time: 0.012 (0.008582698381864108) --> EVAL PERFORMANCE | > avg_loader_time: 0.00885290159306056 (+3.80428744034024e-05) | > avg_step_time: 0.46703990412430024 (+0.0011619480562881646) | > avg_loss_disc: 2.6937518791413644 (+0.09666536895322153) | > avg_loss_disc_real_0: 0.1293045199200721 (+0.006527275870173779) | > avg_loss_disc_real_1: 0.20352744169428316 (-0.009065167847233713) | > avg_loss_disc_real_2: 0.2298212734536386 (+0.00669520558185982) | > avg_loss_disc_real_3: 0.2182450108754803 (-0.012057751731973293) | > avg_loss_disc_real_4: 0.2504757300648891 (+0.014278222137773577) | > avg_loss_disc_real_5: 0.2339252510540922 (-0.01073223877121024) | > avg_loss_0: 2.6937518791413644 (+0.09666536895322153) | > avg_grad_norm_0: tensor(347.0433, device='cuda:0') (+tensor(45.6470, device='cuda:0')) | > avg_loss_gen: 2.058607215612707 (-0.02829711034264326) | > avg_loss_kl: 2.147993235520913 (+0.17876190366879263) | > avg_loss_feat: 3.886227077161762 (+0.038647057304919574) | > avg_loss_mel: 17.288448280012112 (+0.7007985450852168) | > avg_loss_duration: 1.5490144021074537 (+0.025964909875896725) | > avg_amp_scaler: 9.915492957746478 (+1.915492957746478) | > avg_loss_1: 26.930290114711706 (+0.915875179666866) | > avg_grad_norm_1: tensor(7996.4585, device='cuda:0') (+tensor(274.0713, device='cuda:0'))  > EPOCH: 1498/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:38:34)   --> TIME: 2025-02-25 23:38:55 -- STEP: 43/72 -- GLOBAL_STEP: 1107900 | > loss_disc: 2.3112711906433105 (2.6649068344471063) | > loss_disc_real_0: 0.0902676209807396 (0.14698040875238044) | > loss_disc_real_1: 0.19827266037464142 (0.23275228225907615) | > loss_disc_real_2: 0.26874998211860657 (0.25385026564431745) | > loss_disc_real_3: 0.2195686399936676 (0.24569664895534515) | > loss_disc_real_4: 0.2363930493593216 (0.24507293001163838) | > loss_disc_real_5: 0.2494296133518219 (0.24339380998944127) | > loss_0: 2.3112711906433105 (2.6649068344471063) | > grad_norm_0: tensor(129.4237, device='cuda:0') (tensor(281.7062, device='cuda:0')) | > loss_gen: 2.687251567840576 (2.1442052985346587) | > loss_kl: 2.2447333335876465 (2.3409498070561607) | > loss_feat: 4.566197395324707 (3.835603869238565) | > loss_mel: 16.539403915405273 (17.986118227936498) | > loss_duration: 1.5864630937576294 (1.5737424700759177) | > amp_scaler: 8.0 (8.0) | > loss_1: 27.62405014038086 (27.88061927085699) | > grad_norm_1: tensor(3592.7612, device='cuda:0') (tensor(9447.0889, device='cuda:0')) | > current_lr_0: 0.0001658453391919906 | > current_lr_1: 0.0001658453391919906 | > step_time: 0.4606 (0.4447507248368374) | > loader_time: 0.0094 (0.00763069751650788) --> EVAL PERFORMANCE | > avg_loader_time: 0.008711693992077465 (-0.00014120760098309572) | > avg_step_time: 0.456249260566604 (-0.010790643557696267) | > avg_loss_disc: 2.673736078638426 (-0.020015800502938497) | > avg_loss_disc_real_0: 0.14840590169417187 (+0.019101381774099757) | > avg_loss_disc_real_1: 0.23455514034754793 (+0.031027698653264768) | > avg_loss_disc_real_2: 0.23601339579048292 (+0.0061921223368443246) | > avg_loss_disc_real_3: 0.2514545371834661 (+0.03320952630798582) | > avg_loss_disc_real_4: 0.2382898062047824 (-0.012185923860106695) | > avg_loss_disc_real_5: 0.24090450603357502 (+0.0069792549794828185) | > avg_loss_0: 2.673736078638426 (-0.020015800502938497) | > avg_grad_norm_0: tensor(273.8241, device='cuda:0') (tensor(-73.2191, device='cuda:0')) | > avg_loss_gen: 2.1394415220744167 (+0.08083430646170964) | > avg_loss_kl: 2.282170749046433 (+0.1341775135255201) | > avg_loss_feat: 3.842370500027294 (-0.04385657713446811) | > avg_loss_mel: 17.63291322681265 (+0.34446494680053874) | > avg_loss_duration: 1.5794684467181352 (+0.030454044610681574) | > avg_amp_scaler: 8.0 (-1.915492957746478) | > avg_loss_1: 27.476364108878123 (+0.5460739941664166) | > avg_grad_norm_1: tensor(8168.7148, device='cuda:0') (+tensor(172.2563, device='cuda:0'))  > EPOCH: 1499/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 23:39:11)   --> TIME: 2025-02-25 23:39:22 -- STEP: 21/72 -- GLOBAL_STEP: 1107950 | > loss_disc: 2.5648956298828125 (2.5987246263594854) | > loss_disc_real_0: 0.03132210671901703 (0.09911684975737617) | > loss_disc_real_1: 0.23729734122753143 (0.2275272352354867) | > loss_disc_real_2: 0.18510743975639343 (0.22551387761320388) | > loss_disc_real_3: 0.33709368109703064 (0.2397505548738298) | > loss_disc_real_4: 0.21819022297859192 (0.23396931304818108) | > loss_disc_real_5: 0.2660888433456421 (0.24893278876940408) | > loss_0: 2.5648956298828125 (2.5987246263594854) | > grad_norm_0: tensor(404.1685, device='cuda:0') (tensor(191.6646, device='cuda:0')) | > loss_gen: 1.8837838172912598 (2.119734917368208) | > loss_kl: 1.851718544960022 (1.9435021877288818) | > loss_feat: 3.7866339683532715 (4.033410038266863) | > loss_mel: 17.26854133605957 (17.15395282563709) | > loss_duration: 1.5141689777374268 (1.4936223313921975) | > amp_scaler: 8.0 (8.0) | > loss_1: 26.304847717285156 (26.744222005208332) | > grad_norm_1: tensor(16884.1621, device='cuda:0') (tensor(5277.9155, device='cuda:0')) | > current_lr_0: 0.0001658246085245916 | > current_lr_1: 0.0001658246085245916 | > step_time: 0.4329 (0.43618206750778926) | > loader_time: 0.0062 (0.006715240932646252)  --> TIME: 2025-02-25 23:39:46 -- STEP: 71/72 -- GLOBAL_STEP: 1108000 | > loss_disc: 2.6482934951782227 (2.6465414678546746) | > loss_disc_real_0: 0.04069184511899948 (0.1120005749480825) | > loss_disc_real_1: 0.288603276014328 (0.22417305451883396) | > loss_disc_real_2: 0.24473102390766144 (0.22547829549917034) | > loss_disc_real_3: 0.19085857272148132 (0.24048316646629656) | > loss_disc_real_4: 0.308353990316391 (0.23926305665936268) | > loss_disc_real_5: 0.24210160970687866 (0.24277425085155063) | > loss_0: 2.6482934951782227 (2.6465414678546746) | > grad_norm_0: tensor(405.4914, device='cuda:0') (tensor(232.9472, device='cuda:0')) | > loss_gen: 2.3150534629821777 (2.022544493137951) | > loss_kl: 1.7747721672058105 (2.0732508864201282) | > loss_feat: 4.074528694152832 (3.833809970130383) | > loss_mel: 15.819372177124023 (17.137904261199527) | > loss_duration: 1.534381628036499 (1.547701229511852) | > amp_scaler: 8.0 (8.0) | > loss_1: 25.518108367919922 (26.615210761486644) | > grad_norm_1: tensor(17532.5293, device='cuda:0') (tensor(7072.2749, device='cuda:0')) | > current_lr_0: 0.0001658246085245916 | > current_lr_1: 0.0001658246085245916 | > step_time: 0.4978 (0.4546609025605967) | > loader_time: 0.0077 (0.008553085192828115) --> EVAL PERFORMANCE | > avg_loader_time: 0.008553085192828115 (-0.00015860879924935005) | > avg_step_time: 0.4546609025605967 (-0.0015883580060072622) | > avg_loss_disc: 2.6465414678546746 (-0.027194610783751294) | > avg_loss_disc_real_0: 0.1120005749480825 (-0.03640532674608936) | > avg_loss_disc_real_1: 0.22417305451883396 (-0.010382085828713972) | > avg_loss_disc_real_2: 0.22547829549917034 (-0.010535100291312577) | > avg_loss_disc_real_3: 0.24048316646629656 (-0.010971370717169554) | > avg_loss_disc_real_4: 0.23926305665936268 (+0.0009732504545802712) | > avg_loss_disc_real_5: 0.24277425085155063 (+0.0018697448179756104) | > avg_loss_0: 2.6465414678546746 (-0.027194610783751294) | > avg_grad_norm_0: tensor(232.9472, device='cuda:0') (tensor(-40.8770, device='cuda:0')) | > avg_loss_gen: 2.022544493137951 (-0.11689702893646592) | > avg_loss_kl: 2.0732508864201282 (-0.2089198626263049) | > avg_loss_feat: 3.833809970130383 (-0.008560529896910918) | > avg_loss_mel: 17.137904261199527 (-0.495008965613124) | > avg_loss_duration: 1.547701229511852 (-0.03176721720628328) | > avg_amp_scaler: 8.0 (+0.0) | > avg_loss_1: 26.615210761486644 (-0.861153347391479) | > avg_grad_norm_1: tensor(7072.2749, device='cuda:0') (tensor(-1096.4399, device='cuda:0'))