> Training Environment: | > Backend: Torch | > Mixed precision: True | > Precision: fp16 | > Current device: 0 | > Num. of GPUs: 1 | > Num. of CPUs: 12 | > Num. of Torch Threads: 6 | > Torch seed: 54321 | > Torch CUDNN: True | > Torch CUDNN deterministic: False | > Torch CUDNN benchmark: False | > Torch TF32 MatMul: False > Start Tensorboard: tensorboard --logdir=voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a > Restoring from model_file.pth ... > Restoring Model... > Partial model initialization... | > Layer missing in the model definition: posterior_encoder.enc.in_layers.0.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.0.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.1.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.1.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.2.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.2.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.3.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.3.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.4.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.4.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.5.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.5.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.6.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.6.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.7.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.7.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.8.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.8.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.9.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.9.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.10.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.10.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.11.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.11.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.12.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.12.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.13.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.13.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.14.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.14.weight_v | > Layer missing in the model definition: posterior_encoder.enc.in_layers.15.weight_g | > Layer missing in the model definition: posterior_encoder.enc.in_layers.15.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.4.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.4.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.5.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.5.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.6.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.6.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.7.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.7.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.8.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.8.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.9.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.9.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.10.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.10.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.11.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.11.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.12.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.12.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.13.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.13.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.14.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.14.weight_v | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.15.weight_g | > Layer missing in the model definition: posterior_encoder.enc.res_skip_layers.15.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.0.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.0.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.0.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.1.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.1.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.1.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.2.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.2.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.2.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.0.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.0.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.1.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.1.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.2.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.2.weight_v | > Layer missing in the model definition: flow.flows.3.enc.in_layers.3.weight_g | > Layer missing in the model definition: flow.flows.3.enc.in_layers.3.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.0.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.0.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.1.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.1.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.2.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.2.weight_v | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.3.weight_g | > Layer missing in the model definition: flow.flows.3.enc.res_skip_layers.3.weight_v | > Layer missing in the model definition: waveform_decoder.ups.0.weight_g | > Layer missing in the model definition: waveform_decoder.ups.0.weight_v | > Layer missing in the model definition: waveform_decoder.ups.1.weight_g | > Layer missing in the model definition: waveform_decoder.ups.1.weight_v | > Layer missing in the model definition: waveform_decoder.ups.2.weight_g | > Layer missing in the model definition: waveform_decoder.ups.2.weight_v | > Layer missing in the model definition: waveform_decoder.ups.3.weight_g | > Layer missing in the model definition: waveform_decoder.ups.3.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.0.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.1.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.2.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.3.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.4.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.5.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.6.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.7.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.8.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.9.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.10.convs2.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs1.2.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.0.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.0.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.1.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.1.weight_v | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.2.weight_g | > Layer missing in the model definition: waveform_decoder.resblocks.11.convs2.2.weight_v | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.6.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.6.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.7.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.7.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.8.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.8.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.9.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.9.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.10.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.10.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.11.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.11.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.12.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.12.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.13.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.13.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.14.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.14.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.15.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.in_layers.15.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.6.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.6.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.7.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.7.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.8.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.8.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.9.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.9.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.10.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.10.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.11.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.11.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.12.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.12.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.13.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.13.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.14.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.14.parametrizations.weight.original1 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.15.parametrizations.weight.original0 | > Layer missing in the checkpoint: posterior_encoder.enc.res_skip_layers.15.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.0.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.1.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.2.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.in_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: flow.flows.3.enc.res_skip_layers.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.ups.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.ups.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.0.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.1.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.2.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.3.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.4.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.5.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.6.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.7.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.8.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.9.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.10.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs1.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: waveform_decoder.resblocks.11.convs2.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.0.bias | > Layer missing in the checkpoint: disc.nets.0.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.1.bias | > Layer missing in the checkpoint: disc.nets.0.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.2.bias | > Layer missing in the checkpoint: disc.nets.0.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.3.bias | > Layer missing in the checkpoint: disc.nets.0.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.4.bias | > Layer missing in the checkpoint: disc.nets.0.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.convs.5.bias | > Layer missing in the checkpoint: disc.nets.0.convs.5.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.convs.5.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.0.conv_post.bias | > Layer missing in the checkpoint: disc.nets.0.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.0.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.0.bias | > Layer missing in the checkpoint: disc.nets.1.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.1.bias | > Layer missing in the checkpoint: disc.nets.1.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.2.bias | > Layer missing in the checkpoint: disc.nets.1.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.3.bias | > Layer missing in the checkpoint: disc.nets.1.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.convs.4.bias | > Layer missing in the checkpoint: disc.nets.1.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.1.conv_post.bias | > Layer missing in the checkpoint: disc.nets.1.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.1.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.0.bias | > Layer missing in the checkpoint: disc.nets.2.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.1.bias | > Layer missing in the checkpoint: disc.nets.2.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.2.bias | > Layer missing in the checkpoint: disc.nets.2.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.3.bias | > Layer missing in the checkpoint: disc.nets.2.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.convs.4.bias | > Layer missing in the checkpoint: disc.nets.2.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.2.conv_post.bias | > Layer missing in the checkpoint: disc.nets.2.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.2.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.0.bias | > Layer missing in the checkpoint: disc.nets.3.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.1.bias | > Layer missing in the checkpoint: disc.nets.3.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.2.bias | > Layer missing in the checkpoint: disc.nets.3.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.3.bias | > Layer missing in the checkpoint: disc.nets.3.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.convs.4.bias | > Layer missing in the checkpoint: disc.nets.3.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.3.conv_post.bias | > Layer missing in the checkpoint: disc.nets.3.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.3.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.0.bias | > Layer missing in the checkpoint: disc.nets.4.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.1.bias | > Layer missing in the checkpoint: disc.nets.4.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.2.bias | > Layer missing in the checkpoint: disc.nets.4.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.3.bias | > Layer missing in the checkpoint: disc.nets.4.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.convs.4.bias | > Layer missing in the checkpoint: disc.nets.4.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.4.conv_post.bias | > Layer missing in the checkpoint: disc.nets.4.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.4.conv_post.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.0.bias | > Layer missing in the checkpoint: disc.nets.5.convs.0.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.0.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.1.bias | > Layer missing in the checkpoint: disc.nets.5.convs.1.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.1.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.2.bias | > Layer missing in the checkpoint: disc.nets.5.convs.2.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.2.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.3.bias | > Layer missing in the checkpoint: disc.nets.5.convs.3.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.3.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.convs.4.bias | > Layer missing in the checkpoint: disc.nets.5.convs.4.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.convs.4.parametrizations.weight.original1 | > Layer missing in the checkpoint: disc.nets.5.conv_post.bias | > Layer missing in the checkpoint: disc.nets.5.conv_post.parametrizations.weight.original0 | > Layer missing in the checkpoint: disc.nets.5.conv_post.parametrizations.weight.original1 | > Layer dimention missmatch between model definition and checkpoint: text_encoder.emb.weight | > 557 / 949 layers are restored. > Model restored from step 1000000 > Model has 83047468 parameters  > EPOCH: 0/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:10:48)   --> TIME: 2025-02-25 08:11:16 -- STEP: 49/72 -- GLOBAL_STEP: 1000050 | > loss_disc: 2.9825031757354736 (3.2188744788267174) | > loss_disc_real_0: 0.25394102931022644 (0.29492562142561896) | > loss_disc_real_1: 0.25017398595809937 (0.2998270029012038) | > loss_disc_real_2: 0.22751484811306 (0.29802338064325107) | > loss_disc_real_3: 0.2826526463031769 (0.3007126067365919) | > loss_disc_real_4: 0.26421108841896057 (0.30721953146311703) | > loss_disc_real_5: 0.23740074038505554 (0.30391768442124734) | > loss_0: 2.9825031757354736 (3.2188744788267174) | > grad_norm_0: tensor(0.9411, device='cuda:0') (tensor(1.4486, device='cuda:0')) | > loss_gen: 1.522012710571289 (1.7242362146474877) | > loss_kl: 3.9697844982147217 (6.595462774743839) | > loss_feat: 0.35030442476272583 (0.3189420134437327) | > loss_mel: 21.26411247253418 (23.159969485535914) | > loss_duration: 1.8520582914352417 (1.9332660023047) | > amp_scaler: 256.0 (882.9387755102041) | > loss_1: 28.95827293395996 (33.73187649006747) | > grad_norm_1: tensor(141.9976, device='cuda:0') (tensor(158.1107, device='cuda:0')) | > current_lr_0: 0.0002 | > current_lr_1: 0.0002 | > step_time: 0.469 (0.4867572833080681) | > loader_time: 0.0096 (0.008112372184286312) --> EVAL PERFORMANCE | > avg_loader_time: 0.00901026121327575 (+0) | > avg_step_time: 0.4985209955296046 (+0) | > avg_loss_disc: 3.1403814436684194 (+0) | > avg_loss_disc_real_0: 0.2810367999572149 (+0) | > avg_loss_disc_real_1: 0.2837229065701995 (+0) | > avg_loss_disc_real_2: 0.28172466316273514 (+0) | > avg_loss_disc_real_3: 0.28414903129910085 (+0) | > avg_loss_disc_real_4: 0.2883562760453829 (+0) | > avg_loss_disc_real_5: 0.2869413927407332 (+0) | > avg_loss_0: 3.1403814436684194 (+0) | > avg_grad_norm_0: tensor(1.7452, device='cuda:0') (+0) | > avg_loss_gen: 1.670924420927612 (+0) | > avg_loss_kl: 5.7890785781430525 (+0) | > avg_loss_feat: 0.3709548791529427 (+0) | > avg_loss_mel: 22.693415440304175 (+0) | > avg_loss_duration: 1.8996801141282202 (+0) | > avg_amp_scaler: 688.6760563380282 (+0) | > avg_loss_1: 32.42405346077934 (+0) | > avg_grad_norm_1: tensor(175.1228, device='cuda:0') (+0) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000073.pth  > EPOCH: 1/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:11:32)   --> TIME: 2025-02-25 08:11:46 -- STEP: 27/72 -- GLOBAL_STEP: 1000100 | > loss_disc: 2.925985813140869 (2.9307760927412243) | > loss_disc_real_0: 0.2283693552017212 (0.24704700266873394) | > loss_disc_real_1: 0.2995283603668213 (0.21307367231282923) | > loss_disc_real_2: 0.26690712571144104 (0.24293037503957748) | > loss_disc_real_3: 0.16060686111450195 (0.24704200884810198) | > loss_disc_real_4: 0.26388227939605713 (0.2572009099854364) | > loss_disc_real_5: 0.2014242261648178 (0.24466190459551634) | > loss_0: 2.925985813140869 (2.9307760927412243) | > grad_norm_0: tensor(7.6913, device='cuda:0') (tensor(11.0084, device='cuda:0')) | > loss_gen: 1.581876277923584 (1.622780817526358) | > loss_kl: 3.5429091453552246 (3.8475298086802163) | > loss_feat: 0.789517343044281 (0.8173273007074991) | > loss_mel: 23.02496910095215 (21.421504621152526) | > loss_duration: 1.8326542377471924 (1.836715203744394) | > amp_scaler: 256.0 (256.0) | > loss_1: 30.771926879882812 (29.545857747395832) | > grad_norm_1: tensor(108.3841, device='cuda:0') (tensor(147.1433, device='cuda:0')) | > current_lr_0: 0.000199975 | > current_lr_1: 0.000199975 | > step_time: 0.4305 (0.42321381745515047) | > loader_time: 0.0081 (0.006800201204087999) --> EVAL PERFORMANCE | > avg_loader_time: 0.008423361979739766 (-0.0005868992335359841) | > avg_step_time: 0.44578703356460786 (-0.052733961964996745) | > avg_loss_disc: 2.935559648863027 (-0.20482179480539253) | > avg_loss_disc_real_0: 0.24788679539317815 (-0.03315000456403677) | > avg_loss_disc_real_1: 0.22574999469371748 (-0.057972911876482014) | > avg_loss_disc_real_2: 0.24085514981981734 (-0.0408695133429178) | > avg_loss_disc_real_3: 0.24977426472264277 (-0.03437476657645808) | > avg_loss_disc_real_4: 0.24848778566844026 (-0.03986849037694262) | > avg_loss_disc_real_5: 0.25090073154006204 (-0.036040661200671176) | > avg_loss_0: 2.935559648863027 (-0.20482179480539253) | > avg_grad_norm_0: tensor(11.2242, device='cuda:0') (+tensor(9.4790, device='cuda:0')) | > avg_loss_gen: 1.6193270632918453 (-0.05159735763576667) | > avg_loss_kl: 3.6470820635137424 (-2.14199651462931) | > avg_loss_feat: 0.8079535020908839 (+0.43699862293794123) | > avg_loss_mel: 21.401046457424968 (-1.292368982879207) | > avg_loss_duration: 1.8941579402332576 (-0.005522173894962679) | > avg_amp_scaler: 256.0 (-432.67605633802816) | > avg_loss_1: 29.369566930851466 (-3.0544865299278747) | > avg_grad_norm_1: tensor(173.4342, device='cuda:0') (tensor(-1.6886, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000145.pth  > EPOCH: 2/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:12:11)   --> TIME: 2025-02-25 08:12:15 -- STEP: 5/72 -- GLOBAL_STEP: 1000150 | > loss_disc: 2.957272529602051 (2.9675930023193358) | > loss_disc_real_0: 0.2359703928232193 (0.245746248960495) | > loss_disc_real_1: 0.2659362554550171 (0.23980973660945892) | > loss_disc_real_2: 0.2497364580631256 (0.2524750679731369) | > loss_disc_real_3: 0.24600060284137726 (0.242550528049469) | > loss_disc_real_4: 0.37197819352149963 (0.2659307271242142) | > loss_disc_real_5: 0.2499941885471344 (0.25404737889766693) | > loss_0: 2.957272529602051 (2.9675930023193358) | > grad_norm_0: tensor(4.4228, device='cuda:0') (tensor(6.1251, device='cuda:0')) | > loss_gen: 1.4387426376342773 (1.5379149913787842) | > loss_kl: 3.8326852321624756 (3.602376699447632) | > loss_feat: 0.7235944867134094 (0.7265236973762512) | > loss_mel: 22.401273727416992 (21.876456451416015) | > loss_duration: 1.9294607639312744 (1.9299361944198608) | > amp_scaler: 256.0 (256.0) | > loss_1: 30.32575798034668 (29.67320785522461) | > grad_norm_1: tensor(374.4723, device='cuda:0') (tensor(278.3387, device='cuda:0')) | > current_lr_0: 0.000199950003125 | > current_lr_1: 0.000199950003125 | > step_time: 0.4118 (0.4143852233886719) | > loader_time: 0.0059 (0.007408428192138672)  --> TIME: 2025-02-25 08:12:38 -- STEP: 55/72 -- GLOBAL_STEP: 1000200 | > loss_disc: 2.9496872425079346 (2.9714270895177664) | > loss_disc_real_0: 0.23269852995872498 (0.247294771400365) | > loss_disc_real_1: 0.2480759471654892 (0.24936744652011178) | > loss_disc_real_2: 0.25224682688713074 (0.25010719326409425) | > loss_disc_real_3: 0.2053726762533188 (0.24796133908358486) | > loss_disc_real_4: 0.2475266456604004 (0.24789060977372257) | > loss_disc_real_5: 0.21277084946632385 (0.2479921517047015) | > loss_0: 2.9496872425079346 (2.9714270895177664) | > grad_norm_0: tensor(2.1139, device='cuda:0') (tensor(6.5297, device='cuda:0')) | > loss_gen: 1.3628805875778198 (1.5691591999747536) | > loss_kl: 3.1818618774414062 (3.311189543117176) | > loss_feat: 0.6071171164512634 (0.693370153687217) | > loss_mel: 22.24699592590332 (21.16485925154253) | > loss_duration: 1.9517102241516113 (1.920708120952953) | > amp_scaler: 256.0 (256.0) | > loss_1: 29.35056495666504 (28.65928622159091) | > grad_norm_1: tensor(62.5079, device='cuda:0') (tensor(205.5087, device='cuda:0')) | > current_lr_0: 0.000199950003125 | > current_lr_1: 0.000199950003125 | > step_time: 0.4633 (0.4420865665782582) | > loader_time: 0.0092 (0.007749860936945135) --> EVAL PERFORMANCE | > avg_loader_time: 0.008473221684845397 (+4.985970510563119e-05) | > avg_step_time: 0.45470458353069465 (+0.00891754996608679) | > avg_loss_disc: 2.9702808689063698 (+0.03472122004334288) | > avg_loss_disc_real_0: 0.24721921854455706 (-0.0006675768486210931) | > avg_loss_disc_real_1: 0.2482473801978877 (+0.022497385504170214) | > avg_loss_disc_real_2: 0.250255586395801 (+0.009400436575983634) | > avg_loss_disc_real_3: 0.24880759250110304 (-0.0009666722215397328) | > avg_loss_disc_real_4: 0.24763951419104993 (-0.0008482714773903333) | > avg_loss_disc_real_5: 0.24635876022594075 (-0.00454197131412129) | > avg_loss_0: 2.9702808689063698 (+0.03472122004334288) | > avg_grad_norm_0: tensor(6.5843, device='cuda:0') (tensor(-4.6399, device='cuda:0')) | > avg_loss_gen: 1.5697072962640037 (-0.049619767027841544) | > avg_loss_kl: 3.2820434234511677 (-0.3650386400625747) | > avg_loss_feat: 0.6836673575387874 (-0.12428614455209652) | > avg_loss_mel: 21.23391457678567 (-0.16713188063929962) | > avg_loss_duration: 1.9250469560354528 (+0.030889015802195274) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 28.69437956474197 (-0.6751873661094976) | > avg_grad_norm_1: tensor(225.2051, device='cuda:0') (+tensor(51.7709, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000217.pth  > EPOCH: 3/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:12:50)   --> TIME: 2025-02-25 08:13:06 -- STEP: 33/72 -- GLOBAL_STEP: 1000250 | > loss_disc: 2.995230197906494 (2.9239247928966177) | > loss_disc_real_0: 0.2960442304611206 (0.25134173848412256) | > loss_disc_real_1: 0.39282822608947754 (0.24443596962726477) | > loss_disc_real_2: 0.20393028855323792 (0.24824705268397476) | > loss_disc_real_3: 0.2923261821269989 (0.24795765813552972) | > loss_disc_real_4: 0.3965698480606079 (0.2557710047924157) | > loss_disc_real_5: 0.28516554832458496 (0.23493806837183057) | > loss_0: 2.995230197906494 (2.9239247928966177) | > grad_norm_0: tensor(15.1858, device='cuda:0') (tensor(5.6415, device='cuda:0')) | > loss_gen: 1.662196159362793 (1.605379758459149) | > loss_kl: 2.84491229057312 (3.160514484752308) | > loss_feat: 0.8706418871879578 (0.7842760989160249) | > loss_mel: 19.98088264465332 (21.18256158539743) | > loss_duration: 1.899662733078003 (1.9039002837556782) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.258296966552734 (28.63663239912553) | > grad_norm_1: tensor(79.2487, device='cuda:0') (tensor(196.5991, device='cuda:0')) | > current_lr_0: 0.00019992500937460937 | > current_lr_1: 0.00019992500937460937 | > step_time: 0.4304 (0.42303753621650464) | > loader_time: 0.0078 (0.006899826454393791) --> EVAL PERFORMANCE | > avg_loader_time: 0.008448533608879842 (-2.4688075965555167e-05) | > avg_step_time: 0.4434272369868319 (-0.011277346543862754) | > avg_loss_disc: 2.9282614110221323 (-0.04201945788423744) | > avg_loss_disc_real_0: 0.24977532385940282 (+0.0025561053148457635) | > avg_loss_disc_real_1: 0.24855356065320297 (+0.0003061804553152747) | > avg_loss_disc_real_2: 0.2500136068589251 (-0.00024197953687588036) | > avg_loss_disc_real_3: 0.2461773285143812 (-0.002630263986721837) | > avg_loss_disc_real_4: 0.24869454389726606 (+0.0010550297062161262) | > avg_loss_disc_real_5: 0.23802789458086793 (-0.008330865645072821) | > avg_loss_0: 2.9282614110221323 (-0.04201945788423744) | > avg_grad_norm_0: tensor(5.4481, device='cuda:0') (tensor(-1.1362, device='cuda:0')) | > avg_loss_gen: 1.6043520024125004 (+0.034644706148496685) | > avg_loss_kl: 3.072064967222617 (-0.20997845622855094) | > avg_loss_feat: 0.7786891426838619 (+0.09502178514507453) | > avg_loss_mel: 20.817215127004705 (-0.41669944978096396) | > avg_loss_duration: 1.9170460768148934 (-0.008000879220559476) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 28.189367348039653 (-0.5050122167023154) | > avg_grad_norm_1: tensor(183.3010, device='cuda:0') (tensor(-41.9041, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000289.pth  > EPOCH: 4/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:13:29)   --> TIME: 2025-02-25 08:13:36 -- STEP: 11/72 -- GLOBAL_STEP: 1000300 | > loss_disc: 2.879227638244629 (2.902175881645896) | > loss_disc_real_0: 0.23267687857151031 (0.24812510203231464) | > loss_disc_real_1: 0.2110191434621811 (0.24586922065778213) | > loss_disc_real_2: 0.2280118316411972 (0.24365519799969412) | > loss_disc_real_3: 0.25330987572669983 (0.23963796279647134) | > loss_disc_real_4: 0.2743997275829315 (0.24286146461963654) | > loss_disc_real_5: 0.19379922747612 (0.2390293600884351) | > loss_0: 2.879227638244629 (2.902175881645896) | > grad_norm_0: tensor(3.5757, device='cuda:0') (tensor(2.4715, device='cuda:0')) | > loss_gen: 1.762873649597168 (1.6156580664894797) | > loss_kl: 3.112015724182129 (3.026738491925326) | > loss_feat: 0.9788455963134766 (0.9223772395740856) | > loss_mel: 21.10115623474121 (20.705883026123047) | > loss_duration: 1.8683559894561768 (1.9002541953867131) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.8232479095459 (28.170911268754438) | > grad_norm_1: tensor(231.1476, device='cuda:0') (tensor(144.6830, device='cuda:0')) | > current_lr_0: 0.00019990001874843754 | > current_lr_1: 0.00019990001874843754 | > step_time: 0.412 (0.41687536239624023) | > loader_time: 0.0055 (0.006056547164916992)  --> TIME: 2025-02-25 08:13:59 -- STEP: 61/72 -- GLOBAL_STEP: 1000350 | > loss_disc: 2.9949309825897217 (2.9275278732424876) | > loss_disc_real_0: 0.19715148210525513 (0.24802992334131335) | > loss_disc_real_1: 0.1990196853876114 (0.24823996302534324) | > loss_disc_real_2: 0.18832583725452423 (0.24527351729205396) | > loss_disc_real_3: 0.26448971033096313 (0.24341604748710258) | > loss_disc_real_4: 0.2670530080795288 (0.24608897600994736) | > loss_disc_real_5: 0.21023260056972504 (0.2456031514972937) | > loss_0: 2.9949309825897217 (2.9275278732424876) | > grad_norm_0: tensor(7.2462, device='cuda:0') (tensor(3.8344, device='cuda:0')) | > loss_gen: 1.538955569267273 (1.6051671661314417) | > loss_kl: 2.8710379600524902 (2.917222882880539) | > loss_feat: 0.8065652847290039 (0.8815001894216068) | > loss_mel: 20.197891235351562 (20.596212887373124) | > loss_duration: 1.8930835723876953 (1.8882950133964664) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.307533264160156 (27.88839834244525) | > grad_norm_1: tensor(309.7876, device='cuda:0') (tensor(183.4985, device='cuda:0')) | > current_lr_0: 0.00019990001874843754 | > current_lr_1: 0.00019990001874843754 | > step_time: 0.4916 (0.4392431055913206) | > loader_time: 0.0108 (0.00774496500609351) --> EVAL PERFORMANCE | > avg_loader_time: 0.008269555132154008 (-0.00017897847672583393) | > avg_step_time: 0.45216000248009053 (+0.008732765493258632) | > avg_loss_disc: 2.9316021925966504 (+0.0033407815745181146) | > avg_loss_disc_real_0: 0.2479266923078349 (-0.0018486315515679197) | > avg_loss_disc_real_1: 0.24831844971213543 (-0.00023511094106754005) | > avg_loss_disc_real_2: 0.24607928642924404 (-0.00393432042968106) | > avg_loss_disc_real_3: 0.24283116846017436 (-0.003346160054206848) | > avg_loss_disc_real_4: 0.24561430723734304 (-0.003080236659923019) | > avg_loss_disc_real_5: 0.24668525872935712 (+0.008657364148489194) | > avg_loss_0: 2.9316021925966504 (+0.0033407815745181146) | > avg_grad_norm_0: tensor(3.7589, device='cuda:0') (tensor(-1.6892, device='cuda:0')) | > avg_loss_gen: 1.6033664908207639 (-0.0009855115917365431) | > avg_loss_kl: 2.912233651523859 (-0.15983131569875786) | > avg_loss_feat: 0.8629949571381154 (+0.08430581445425345) | > avg_loss_mel: 20.59236462015501 (-0.22485050684969465) | > avg_loss_duration: 1.8902590644191688 (-0.026787012395724563) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.861218949438822 (-0.3281483986008311) | > avg_grad_norm_1: tensor(185.9581, device='cuda:0') (+tensor(2.6571, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000361.pth  > EPOCH: 5/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:14:09)   --> TIME: 2025-02-25 08:14:28 -- STEP: 39/72 -- GLOBAL_STEP: 1000400 | > loss_disc: 2.90788197517395 (2.928270584497696) | > loss_disc_real_0: 0.24960829317569733 (0.24170295550273016) | > loss_disc_real_1: 0.2572973668575287 (0.2436230717561184) | > loss_disc_real_2: 0.2461818903684616 (0.24636340523377442) | > loss_disc_real_3: 0.22269019484519958 (0.240693449591979) | > loss_disc_real_4: 0.2480708658695221 (0.2436519910891851) | > loss_disc_real_5: 0.2448655068874359 (0.24569719533125559) | > loss_0: 2.90788197517395 (2.928270584497696) | > grad_norm_0: tensor(1.9227, device='cuda:0') (tensor(5.0588, device='cuda:0')) | > loss_gen: 1.5842304229736328 (1.5921040100929065) | > loss_kl: 2.9740002155303955 (2.9450195263593617) | > loss_feat: 0.935753345489502 (0.8882377468622648) | > loss_mel: 20.40302276611328 (20.475824698423732) | > loss_duration: 1.852944254875183 (1.858114936412909) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.749950408935547 (27.759301063341972) | > grad_norm_1: tensor(208.6264, device='cuda:0') (tensor(185.2366, device='cuda:0')) | > current_lr_0: 0.00019987503124609398 | > current_lr_1: 0.00019987503124609398 | > step_time: 0.4458 (0.42758582799862593) | > loader_time: 0.0079 (0.007141602345002003) --> EVAL PERFORMANCE | > avg_loader_time: 0.008404939946993975 (+0.00013538481483996688) | > avg_step_time: 0.4467850369466862 (-0.00537496553340433) | > avg_loss_disc: 2.9441230800789846 (+0.012520887482334153) | > avg_loss_disc_real_0: 0.2441791142796127 (-0.0037475780282222038) | > avg_loss_disc_real_1: 0.25010623940279786 (+0.001787789690662428) | > avg_loss_disc_real_2: 0.24738614450038318 (+0.0013068580711391442) | > avg_loss_disc_real_3: 0.24494413569779463 (+0.002112967237620278) | > avg_loss_disc_real_4: 0.2478619782437741 (+0.0022476710064310568) | > avg_loss_disc_real_5: 0.24742765535771008 (+0.0007423966283529593) | > avg_loss_0: 2.9441230800789846 (+0.012520887482334153) | > avg_grad_norm_0: tensor(5.3387, device='cuda:0') (+tensor(1.5798, device='cuda:0')) | > avg_loss_gen: 1.6001049867818053 (-0.0032615040389585293) | > avg_loss_kl: 2.9171111281488975 (+0.004877476625038568) | > avg_loss_feat: 0.8564649257861393 (-0.006530031351976051) | > avg_loss_mel: 20.468097767359772 (-0.12426685279523753) | > avg_loss_duration: 1.8697114192264181 (-0.020547645192750652) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.711490335598796 (-0.14972861384002556) | > avg_grad_norm_1: tensor(207.6699, device='cuda:0') (+tensor(21.7118, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000433.pth  > EPOCH: 6/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:14:48)   --> TIME: 2025-02-25 08:14:57 -- STEP: 17/72 -- GLOBAL_STEP: 1000450 | > loss_disc: 2.9398820400238037 (2.9305135642780975) | > loss_disc_real_0: 0.29226112365722656 (0.24547380121315227) | > loss_disc_real_1: 0.2757374346256256 (0.24457197592538946) | > loss_disc_real_2: 0.2659173607826233 (0.24676142895922942) | > loss_disc_real_3: 0.27626511454582214 (0.24606756427708795) | > loss_disc_real_4: 0.2918429672718048 (0.24552355005460627) | > loss_disc_real_5: 0.2863673269748688 (0.24438966021818273) | > loss_0: 2.9398820400238037 (2.9305135642780975) | > grad_norm_0: tensor(4.2697, device='cuda:0') (tensor(3.5832, device='cuda:0')) | > loss_gen: 1.5584704875946045 (1.579591288286097) | > loss_kl: 3.1111977100372314 (2.8222697061650894) | > loss_feat: 0.6928841471672058 (0.834976918557111) | > loss_mel: 20.777359008789062 (20.34934358035817) | > loss_duration: 1.8818970918655396 (1.8357375018736894) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.021806716918945 (27.421918981215534) | > grad_norm_1: tensor(239.1420, device='cuda:0') (tensor(195.1130, device='cuda:0')) | > current_lr_0: 0.0001998500468671882 | > current_lr_1: 0.0001998500468671882 | > step_time: 0.4185 (0.4211083720712101) | > loader_time: 0.0061 (0.006507046082440545)  --> TIME: 2025-02-25 08:15:21 -- STEP: 67/72 -- GLOBAL_STEP: 1000500 | > loss_disc: 2.966071844100952 (2.9377699182994337) | > loss_disc_real_0: 0.19781923294067383 (0.2453056179098229) | > loss_disc_real_1: 0.20561492443084717 (0.24635118913294665) | > loss_disc_real_2: 0.24311192333698273 (0.24661420130017978) | > loss_disc_real_3: 0.24655815958976746 (0.24587138041631498) | > loss_disc_real_4: 0.21554210782051086 (0.24345664613282503) | > loss_disc_real_5: 0.23191604018211365 (0.24542015501812323) | > loss_0: 2.966071844100952 (2.9377699182994337) | > grad_norm_0: tensor(4.2959, device='cuda:0') (tensor(5.0403, device='cuda:0')) | > loss_gen: 1.38261878490448 (1.5883239187411409) | > loss_kl: 2.8736062049865723 (2.8745713091608303) | > loss_feat: 0.6620239615440369 (0.8715708211286745) | > loss_mel: 20.178743362426758 (20.38734658084699) | > loss_duration: 1.8832614421844482 (1.8419423726067614) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.980253219604492 (27.563755063868282) | > grad_norm_1: tensor(303.9269, device='cuda:0') (tensor(205.5945, device='cuda:0')) | > current_lr_0: 0.0001998500468671882 | > current_lr_1: 0.0001998500468671882 | > step_time: 0.5439 (0.44295274321712663) | > loader_time: 0.0121 (0.00807895945079291) --> EVAL PERFORMANCE | > avg_loader_time: 0.008262546969131685 (-0.00014239297786228927) | > avg_step_time: 0.44863293876110666 (+0.0018479018144204606) | > avg_loss_disc: 2.9382684533025185 (-0.0058546267764660875) | > avg_loss_disc_real_0: 0.2451584871176263 (+0.000979372838013609) | > avg_loss_disc_real_1: 0.24566233850700755 (-0.004443900895790304) | > avg_loss_disc_real_2: 0.2461598236376131 (-0.0012263208627700806) | > avg_loss_disc_real_3: 0.24536493672451504 (+0.0004208010267204021) | > avg_loss_disc_real_4: 0.24385106605543216 (-0.004010912188341931) | > avg_loss_disc_real_5: 0.2450024556945747 (-0.002425199663135369) | > avg_loss_0: 2.9382684533025185 (-0.0058546267764660875) | > avg_grad_norm_0: tensor(5.0772, device='cuda:0') (tensor(-0.2615, device='cuda:0')) | > avg_loss_gen: 1.5883359606836884 (-0.01176902609811692) | > avg_loss_kl: 2.8649174193261375 (-0.05219370882275998) | > avg_loss_feat: 0.8726533852832418 (+0.01618845949710246) | > avg_loss_mel: 20.36292452207754 (-0.10517324528223071) | > avg_loss_duration: 1.8423974211786833 (-0.027313998047734822) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.531228804252518 (-0.18026153134627876) | > avg_grad_norm_1: tensor(208.9674, device='cuda:0') (+tensor(1.2975, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000505.pth  > EPOCH: 7/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:15:27)   --> TIME: 2025-02-25 08:15:49 -- STEP: 45/72 -- GLOBAL_STEP: 1000550 | > loss_disc: 2.9972050189971924 (2.9300031661987305) | > loss_disc_real_0: 0.23143596947193146 (0.2461431662241618) | > loss_disc_real_1: 0.22900594770908356 (0.24338801370726693) | > loss_disc_real_2: 0.2566421329975128 (0.24632740848594242) | > loss_disc_real_3: 0.27437376976013184 (0.24567623701360491) | > loss_disc_real_4: 0.33497583866119385 (0.24791279170248243) | > loss_disc_real_5: 0.33535492420196533 (0.24807819128036498) | > loss_0: 2.9972050189971924 (2.9300031661987305) | > grad_norm_0: tensor(3.2957, device='cuda:0') (tensor(5.0880, device='cuda:0')) | > loss_gen: 1.4765925407409668 (1.6007155100504558) | > loss_kl: 2.767991304397583 (2.882087177700467) | > loss_feat: 0.7573830485343933 (0.934130068620046) | > loss_mel: 19.508838653564453 (20.08430035909017) | > loss_duration: 1.851638674736023 (1.8174825562371149) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.362445831298828 (27.31871566772461) | > grad_norm_1: tensor(147.1764, device='cuda:0') (tensor(241.4984, device='cuda:0')) | > current_lr_0: 0.00019982506561132978 | > current_lr_1: 0.00019982506561132978 | > step_time: 0.4722 (0.4402787950303819) | > loader_time: 0.0085 (0.007095162073771159) --> EVAL PERFORMANCE | > avg_loader_time: 0.008433187511605277 (+0.00017064054247359177) | > avg_step_time: 0.45163247283075897 (+0.00299953406965231) | > avg_loss_disc: 2.9271783022813396 (-0.01109015102117894) | > avg_loss_disc_real_0: 0.24753031344480916 (+0.0023718263271828577) | > avg_loss_disc_real_1: 0.24352492476013346 (-0.002137413746874095) | > avg_loss_disc_real_2: 0.24574485652043787 (-0.00041496711717523005) | > avg_loss_disc_real_3: 0.24520082813753208 (-0.00016410858698295838) | > avg_loss_disc_real_4: 0.24561880780777462 (+0.0017677417523424555) | > avg_loss_disc_real_5: 0.24555425912561551 (+0.0005518034310408038) | > avg_loss_0: 2.9271783022813396 (-0.01109015102117894) | > avg_grad_norm_0: tensor(5.1919, device='cuda:0') (+tensor(0.1147, device='cuda:0')) | > avg_loss_gen: 1.6024325884563821 (+0.014096627772693715) | > avg_loss_kl: 2.830833784291442 (-0.03408363503469536) | > avg_loss_feat: 0.92838226573568 (+0.05572888045243818) | > avg_loss_mel: 20.030793525803258 (-0.3321309962742838) | > avg_loss_duration: 1.825426073141501 (-0.016971348037182254) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.217868375106598 (-0.31336042914592) | > avg_grad_norm_1: tensor(236.6461, device='cuda:0') (+tensor(27.6787, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000577.pth  > EPOCH: 8/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:16:06)   --> TIME: 2025-02-25 08:16:18 -- STEP: 23/72 -- GLOBAL_STEP: 1000600 | > loss_disc: 2.8464109897613525 (2.9138681370279063) | > loss_disc_real_0: 0.24167530238628387 (0.24325829873914304) | > loss_disc_real_1: 0.22613608837127686 (0.2467479265254477) | > loss_disc_real_2: 0.23937265574932098 (0.24493600233741428) | > loss_disc_real_3: 0.24034398794174194 (0.245130448885586) | > loss_disc_real_4: 0.27436548471450806 (0.24418555718401205) | > loss_disc_real_5: 0.2806076407432556 (0.24111954803052155) | > loss_0: 2.8464109897613525 (2.9138681370279063) | > grad_norm_0: tensor(2.6419, device='cuda:0') (tensor(5.3694, device='cuda:0')) | > loss_gen: 1.6769295930862427 (1.6183695793151855) | > loss_kl: 2.6130993366241455 (2.827208767766538) | > loss_feat: 0.9449037313461304 (0.9738812420679175) | > loss_mel: 18.87936782836914 (20.06368637084961) | > loss_duration: 1.828916072845459 (1.803760544113491) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.94321632385254 (27.286906283834707) | > grad_norm_1: tensor(212.1963, device='cuda:0') (tensor(205.3542, device='cuda:0')) | > current_lr_0: 0.00019980008747812837 | > current_lr_1: 0.00019980008747812837 | > step_time: 0.4296 (0.424426503803419) | > loader_time: 0.0068 (0.00653417214103367) --> EVAL PERFORMANCE | > avg_loader_time: 0.008400698782692492 (-3.248872891278495e-05) | > avg_step_time: 0.4488682511826636 (-0.0027642216480953663) | > avg_loss_disc: 2.9093105289298045 (-0.01786777335153511) | > avg_loss_disc_real_0: 0.24363864349647307 (-0.0038916699483360906) | > avg_loss_disc_real_1: 0.24503066317296365 (+0.0015057384128301932) | > avg_loss_disc_real_2: 0.24405031682739795 (-0.001694539693039926) | > avg_loss_disc_real_3: 0.24530617032252566 (+0.00010534218499358428) | > avg_loss_disc_real_4: 0.2442776814732753 (-0.0013411263344993152) | > avg_loss_disc_real_5: 0.24436620906205245 (-0.0011880500635630675) | > avg_loss_0: 2.9093105289298045 (-0.01786777335153511) | > avg_grad_norm_0: tensor(5.4159, device='cuda:0') (+tensor(0.2240, device='cuda:0')) | > avg_loss_gen: 1.628979313541466 (+0.026546725085083933) | > avg_loss_kl: 2.7761729435181954 (-0.05466084077324673) | > avg_loss_feat: 1.0186714588756292 (+0.09028919313994921) | > avg_loss_mel: 20.05015007878693 (+0.019356552983673225) | > avg_loss_duration: 1.8126053491108853 (-0.012820724030615738) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.286579212672272 (+0.06871083756567486) | > avg_grad_norm_1: tensor(229.4813, device='cuda:0') (tensor(-7.1648, device='cuda:0'))  > EPOCH: 9/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:16:43)   --> TIME: 2025-02-25 08:16:45 -- STEP: 1/72 -- GLOBAL_STEP: 1000650 | > loss_disc: 2.8769631385803223 (2.8769631385803223) | > loss_disc_real_0: 0.21832013130187988 (0.21832013130187988) | > loss_disc_real_1: 0.2879323959350586 (0.2879323959350586) | > loss_disc_real_2: 0.26962506771087646 (0.26962506771087646) | > loss_disc_real_3: 0.20471589267253876 (0.20471589267253876) | > loss_disc_real_4: 0.2201765775680542 (0.2201765775680542) | > loss_disc_real_5: 0.24503391981124878 (0.24503391981124878) | > loss_0: 2.8769631385803223 (2.8769631385803223) | > grad_norm_0: tensor(5.0745, device='cuda:0') (tensor(5.0745, device='cuda:0')) | > loss_gen: 1.6040257215499878 (1.6040257215499878) | > loss_kl: 2.6145920753479004 (2.6145920753479004) | > loss_feat: 1.1193921566009521 (1.1193921566009521) | > loss_mel: 20.217857360839844 (20.217857360839844) | > loss_duration: 1.8253167867660522 (1.8253167867660522) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.381183624267578 (27.381183624267578) | > grad_norm_1: tensor(452.1778, device='cuda:0') (tensor(452.1778, device='cuda:0')) | > current_lr_0: 0.0001997751124671936 | > current_lr_1: 0.0001997751124671936 | > step_time: 0.4185 (0.4185006618499756) | > loader_time: 0.0057 (0.0056955814361572266)  --> TIME: 2025-02-25 08:17:07 -- STEP: 51/72 -- GLOBAL_STEP: 1000700 | > loss_disc: 2.9296107292175293 (2.91784771750955) | > loss_disc_real_0: 0.2645227909088135 (0.25209840752330487) | > loss_disc_real_1: 0.2308952957391739 (0.24437146619254468) | > loss_disc_real_2: 0.28264302015304565 (0.2506018593615176) | > loss_disc_real_3: 0.2688547670841217 (0.24860463890374873) | > loss_disc_real_4: 0.281634122133255 (0.24531287626892911) | > loss_disc_real_5: 0.22337229549884796 (0.24581001406791164) | > loss_0: 2.9296107292175293 (2.91784771750955) | > grad_norm_0: tensor(3.5661, device='cuda:0') (tensor(7.2315, device='cuda:0')) | > loss_gen: 1.7124643325805664 (1.649853460928973) | > loss_kl: 2.73689603805542 (2.816495988883225) | > loss_feat: 0.9427319765090942 (1.0305962176883923) | > loss_mel: 19.385393142700195 (20.07251219655953) | > loss_duration: 1.8310108184814453 (1.8051764474195593) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.608495712280273 (27.374634499643363) | > grad_norm_1: tensor(198.5641, device='cuda:0') (tensor(243.3296, device='cuda:0')) | > current_lr_0: 0.0001997751124671936 | > current_lr_1: 0.0001997751124671936 | > step_time: 0.4582 (0.43444059409347235) | > loader_time: 0.0089 (0.007505533741969688) --> EVAL PERFORMANCE | > avg_loader_time: 0.008385356043426084 (-1.5342739266408292e-05) | > avg_step_time: 0.454880523010039 (+0.006012271827375404) | > avg_loss_disc: 2.916189825031119 (+0.006879296101314658) | > avg_loss_disc_real_0: 0.25026441816712786 (+0.006625774670654783) | > avg_loss_disc_real_1: 0.24393459713794816 (-0.0010960660350154972) | > avg_loss_disc_real_2: 0.2500373620382497 (+0.005987045210851777) | > avg_loss_disc_real_3: 0.24862659439234666 (+0.0033204240698209986) | > avg_loss_disc_real_4: 0.2461848410082535 (+0.0019071595349781922) | > avg_loss_disc_real_5: 0.247456410401304 (+0.003090201339251558) | > avg_loss_0: 2.916189825031119 (+0.006879296101314658) | > avg_grad_norm_0: tensor(6.8484, device='cuda:0') (+tensor(1.4324, device='cuda:0')) | > avg_loss_gen: 1.6439085325724643 (+0.014929219030998198) | > avg_loss_kl: 2.7870324296011058 (+0.010859486082910319) | > avg_loss_feat: 1.0238529863491868 (+0.005181527473557601) | > avg_loss_mel: 20.1327751589493 (+0.08262508016236936) | > avg_loss_duration: 1.8063769256564932 (-0.006228423454392118) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.393946150658834 (+0.10736693798656205) | > avg_grad_norm_1: tensor(249.6992, device='cuda:0') (+tensor(20.2178, device='cuda:0'))  > EPOCH: 10/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:17:19)   --> TIME: 2025-02-25 08:17:34 -- STEP: 29/72 -- GLOBAL_STEP: 1000750 | > loss_disc: 2.8289976119995117 (2.8804984174925705) | > loss_disc_real_0: 0.2652198076248169 (0.25028102202662106) | > loss_disc_real_1: 0.17679286003112793 (0.24071406752898775) | > loss_disc_real_2: 0.22767159342765808 (0.24491295167084398) | > loss_disc_real_3: 0.24257221817970276 (0.24743170162727093) | > loss_disc_real_4: 0.20909617841243744 (0.24212476662520704) | > loss_disc_real_5: 0.20427633821964264 (0.2409916951738555) | > loss_0: 2.8289976119995117 (2.8804984174925705) | > grad_norm_0: tensor(7.4484, device='cuda:0') (tensor(7.8095, device='cuda:0')) | > loss_gen: 1.6185669898986816 (1.6746325986138706) | > loss_kl: 3.2731237411499023 (2.906676095107506) | > loss_feat: 1.2397093772888184 (1.165622024700559) | > loss_mel: 21.1178035736084 (20.386256777007006) | > loss_duration: 1.7780410051345825 (1.7875927275624768) | > amp_scaler: 256.0 (256.0) | > loss_1: 29.027244567871094 (27.920780116114123) | > grad_norm_1: tensor(295.2816, device='cuda:0') (tensor(237.2145, device='cuda:0')) | > current_lr_0: 0.00019975014057813518 | > current_lr_1: 0.00019975014057813518 | > step_time: 0.4281 (0.4391281111487027) | > loader_time: 0.0076 (0.006659367988849509) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640114690216494 (+0.0002547586467904102) | > avg_step_time: 0.4522293890026254 (-0.0026511340074136247) | > avg_loss_disc: 2.885640520445058 (-0.030549304586060977) | > avg_loss_disc_real_0: 0.24804982291141026 (-0.0022145952557175996) | > avg_loss_disc_real_1: 0.24293688527295287 (-0.000997711864995282) | > avg_loss_disc_real_2: 0.24643660410189291 (-0.0036007579363568076) | > avg_loss_disc_real_3: 0.24562926359579595 (-0.0029973307965507068) | > avg_loss_disc_real_4: 0.24302487935818418 (-0.0031599616500693206) | > avg_loss_disc_real_5: 0.24211507699858975 (-0.0053413334027142545) | > avg_loss_0: 2.885640520445058 (-0.030549304586060977) | > avg_grad_norm_0: tensor(6.6878, device='cuda:0') (tensor(-0.1605, device='cuda:0')) | > avg_loss_gen: 1.669094099125392 (+0.025185566552927696) | > avg_loss_kl: 2.839984806490616 (+0.052952376889510155) | > avg_loss_feat: 1.1472828707224882 (+0.12342988437330149) | > avg_loss_mel: 20.204558547113987 (+0.0717833881646861) | > avg_loss_duration: 1.8000499329096835 (-0.006326992746809701) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.660970204313035 (+0.26702405365420034) | > avg_grad_norm_1: tensor(224.7787, device='cuda:0') (tensor(-24.9205, device='cuda:0'))  > EPOCH: 11/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:17:56)   --> TIME: 2025-02-25 08:18:01 -- STEP: 7/72 -- GLOBAL_STEP: 1000800 | > loss_disc: 3.022501230239868 (2.9057939052581787) | > loss_disc_real_0: 0.41650375723838806 (0.25090691447257996) | > loss_disc_real_1: 0.34224194288253784 (0.2608590636934553) | > loss_disc_real_2: 0.31523391604423523 (0.2509922427790506) | > loss_disc_real_3: 0.26713624596595764 (0.2468951599938529) | > loss_disc_real_4: 0.2502875328063965 (0.25770252517291475) | > loss_disc_real_5: 0.3119344413280487 (0.24473634362220764) | > loss_0: 3.022501230239868 (2.9057939052581787) | > grad_norm_0: tensor(15.6255, device='cuda:0') (tensor(10.2598, device='cuda:0')) | > loss_gen: 1.9256906509399414 (1.7630830662591117) | > loss_kl: 2.5568668842315674 (2.8391561848776683) | > loss_feat: 0.9208410978317261 (1.1693613018308366) | > loss_mel: 19.406442642211914 (19.87429509844099) | > loss_duration: 1.7682902812957764 (1.792463711329869) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.5781307220459 (27.43835994175502) | > grad_norm_1: tensor(179.4094, device='cuda:0') (tensor(292.6533, device='cuda:0')) | > current_lr_0: 0.00019972517181056292 | > current_lr_1: 0.00019972517181056292 | > step_time: 0.4171 (0.41687491961887907) | > loader_time: 0.0069 (0.006185293197631836)  --> TIME: 2025-02-25 08:18:23 -- STEP: 57/72 -- GLOBAL_STEP: 1000850 | > loss_disc: 2.89868426322937 (2.8832635210271467) | > loss_disc_real_0: 0.2129419445991516 (0.24560617433305373) | > loss_disc_real_1: 0.23200975358486176 (0.23927714910946393) | > loss_disc_real_2: 0.2414505034685135 (0.24273562418264255) | > loss_disc_real_3: 0.27278319001197815 (0.24064500446905168) | > loss_disc_real_4: 0.21306613087654114 (0.23999869039184168) | > loss_disc_real_5: 0.2478225976228714 (0.24174921659001133) | > loss_0: 2.89868426322937 (2.8832635210271467) | > grad_norm_0: tensor(3.6950, device='cuda:0') (tensor(9.2457, device='cuda:0')) | > loss_gen: 1.5882189273834229 (1.6605840524037678) | > loss_kl: 2.4520556926727295 (2.6633925730722) | > loss_feat: 1.018439531326294 (1.1950912977519788) | > loss_mel: 19.213224411010742 (19.642249659488083) | > loss_duration: 1.7990543842315674 (1.7815925008372258) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.070993423461914 (26.942910177665844) | > grad_norm_1: tensor(220.1996, device='cuda:0') (tensor(242.6119, device='cuda:0')) | > current_lr_0: 0.00019972517181056292 | > current_lr_1: 0.00019972517181056292 | > step_time: 0.451 (0.43276616146689967) | > loader_time: 0.0093 (0.0077864914609674825) --> EVAL PERFORMANCE | > avg_loader_time: 0.008427807982538787 (-0.000212306707677707) | > avg_step_time: 0.45022679382646585 (-0.0020025951761595273) | > avg_loss_disc: 2.8900426911636137 (+0.0044021707185555314) | > avg_loss_disc_real_0: 0.24702910827079289 (-0.0010207146406173706) | > avg_loss_disc_real_1: 0.24157449130860853 (-0.001362393964344344) | > avg_loss_disc_real_2: 0.24327565341348378 (-0.003160950688409131) | > avg_loss_disc_real_3: 0.2416333709384354 (-0.003995892657360556) | > avg_loss_disc_real_4: 0.2420301720709868 (-0.0009947072871973883) | > avg_loss_disc_real_5: 0.24228711741071352 (+0.00017204041212376397) | > avg_loss_0: 2.8900426911636137 (+0.0044021707185555314) | > avg_grad_norm_0: tensor(8.4805, device='cuda:0') (+tensor(1.7927, device='cuda:0')) | > avg_loss_gen: 1.6651574473985484 (-0.003936651726843543) | > avg_loss_kl: 2.6701318243859524 (-0.16985298210466349) | > avg_loss_feat: 1.1840431186514841 (+0.036760247928995904) | > avg_loss_mel: 19.74468499841825 (-0.45987354869573593) | > avg_loss_duration: 1.7837246925058499 (-0.016325240403833652) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.047742172026297 (-0.6132280322867381) | > avg_grad_norm_1: tensor(240.1803, device='cuda:0') (+tensor(15.4016, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1000865.pth  > EPOCH: 12/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:18:35)   --> TIME: 2025-02-25 08:18:52 -- STEP: 35/72 -- GLOBAL_STEP: 1000900 | > loss_disc: 2.7877750396728516 (2.8686034611293247) | > loss_disc_real_0: 0.18705663084983826 (0.2402053062404905) | > loss_disc_real_1: 0.15801887214183807 (0.24070787600108556) | > loss_disc_real_2: 0.20947931706905365 (0.24327957119260515) | > loss_disc_real_3: 0.25703567266464233 (0.24586929295744214) | > loss_disc_real_4: 0.20974689722061157 (0.2456662280218942) | > loss_disc_real_5: 0.23456788063049316 (0.24148969990866526) | > loss_0: 2.7877750396728516 (2.8686034611293247) | > grad_norm_0: tensor(14.2712, device='cuda:0') (tensor(7.3042, device='cuda:0')) | > loss_gen: 1.5024340152740479 (1.6767275026866368) | > loss_kl: 2.979099750518799 (2.8799008641924178) | > loss_feat: 1.530431866645813 (1.2490832941872734) | > loss_mel: 20.60323715209961 (19.93271484375) | > loss_duration: 1.7908602952957153 (1.7640734672546388) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.406063079833984 (27.50249993460519) | > grad_norm_1: tensor(331.6520, device='cuda:0') (tensor(223.0181, device='cuda:0')) | > current_lr_0: 0.0001997002061640866 | > current_lr_1: 0.0001997002061640866 | > step_time: 0.4292 (0.4262429101126535) | > loader_time: 0.008 (0.006985133034842355) --> EVAL PERFORMANCE | > avg_loader_time: 0.008422898574614191 (-4.9094079245960015e-06) | > avg_step_time: 0.45759253770532743 (+0.007365743878861575) | > avg_loss_disc: 2.885740746914501 (-0.004301944249112566) | > avg_loss_disc_real_0: 0.24375186734636065 (-0.00327724092443224) | > avg_loss_disc_real_1: 0.24129905614634634 (-0.0002754351622621942) | > avg_loss_disc_real_2: 0.24566206797747545 (+0.0023864145639916623) | > avg_loss_disc_real_3: 0.24597520454668662 (+0.0043418336082512166) | > avg_loss_disc_real_4: 0.24588175621670735 (+0.0038515841457205657) | > avg_loss_disc_real_5: 0.24239242328724392 (+0.0001053058765304038) | > avg_loss_0: 2.885740746914501 (-0.004301944249112566) | > avg_grad_norm_0: tensor(7.0046, device='cuda:0') (tensor(-1.4759, device='cuda:0')) | > avg_loss_gen: 1.6706872842681237 (+0.005529836869575311) | > avg_loss_kl: 2.7855866056093026 (+0.11545478122335018) | > avg_loss_feat: 1.2023540013272997 (+0.0183108826758156) | > avg_loss_mel: 19.925112039270534 (+0.18042704085228323) | > avg_loss_duration: 1.776716521088506 (-0.007008171417343778) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.36045643981074 (+0.3127142677844432) | > avg_grad_norm_1: tensor(228.9953, device='cuda:0') (tensor(-11.1849, device='cuda:0'))  > EPOCH: 13/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:19:12)   --> TIME: 2025-02-25 08:19:19 -- STEP: 13/72 -- GLOBAL_STEP: 1000950 | > loss_disc: 2.9315714836120605 (2.859658663089459) | > loss_disc_real_0: 0.3524682819843292 (0.24717491062787864) | > loss_disc_real_1: 0.3101600110530853 (0.24358717294839713) | > loss_disc_real_2: 0.2592356503009796 (0.24688566762667435) | > loss_disc_real_3: 0.28543463349342346 (0.24767281458928034) | > loss_disc_real_4: 0.3111870586872101 (0.2436544127189196) | > loss_disc_real_5: 0.3164878785610199 (0.24179265934687394) | > loss_0: 2.9315714836120605 (2.859658663089459) | > grad_norm_0: tensor(10.1561, device='cuda:0') (tensor(7.3742, device='cuda:0')) | > loss_gen: 1.775195598602295 (1.7189474014135508) | > loss_kl: 2.8190438747406006 (2.7079458970289965) | > loss_feat: 1.055268406867981 (1.261477727156419) | > loss_mel: 18.10359001159668 (20.316599185650166) | > loss_duration: 1.748166561126709 (1.7785894137162428) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.501264572143555 (27.783559652475212) | > grad_norm_1: tensor(146.8176, device='cuda:0') (tensor(198.4345, device='cuda:0')) | > current_lr_0: 0.00019967524363831608 | > current_lr_1: 0.00019967524363831608 | > step_time: 0.4553 (0.4244197148543138) | > loader_time: 0.0114 (0.0070725954495943506)  --> TIME: 2025-02-25 08:19:43 -- STEP: 63/72 -- GLOBAL_STEP: 1001000 | > loss_disc: 2.9039359092712402 (2.876328426694113) | > loss_disc_real_0: 0.19481578469276428 (0.24034472330222054) | > loss_disc_real_1: 0.20156902074813843 (0.24279885774567014) | > loss_disc_real_2: 0.19965270161628723 (0.2450341416729821) | > loss_disc_real_3: 0.21853460371494293 (0.2426593710505773) | > loss_disc_real_4: 0.25568804144859314 (0.24422236354578108) | > loss_disc_real_5: 0.25536811351776123 (0.24431820615889535) | > loss_0: 2.9039359092712402 (2.876328426694113) | > grad_norm_0: tensor(7.1405, device='cuda:0') (tensor(7.2322, device='cuda:0')) | > loss_gen: 1.5807216167449951 (1.6851773999986195) | > loss_kl: 2.6276352405548096 (2.7272086408403187) | > loss_feat: 1.294409990310669 (1.2512692705033317) | > loss_mel: 19.652387619018555 (20.016976765223905) | > loss_duration: 1.750496506690979 (1.7735278606414795) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.905649185180664 (27.454159872872488) | > grad_norm_1: tensor(159.6012, device='cuda:0') (tensor(197.0115, device='cuda:0')) | > current_lr_0: 0.00019967524363831608 | > current_lr_1: 0.00019967524363831608 | > step_time: 0.4941 (0.4467459860302153) | > loader_time: 0.0116 (0.008144367308843704) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515069182489959 (+9.217060787576788e-05) | > avg_step_time: 0.45296536029224665 (-0.004627177413080774) | > avg_loss_disc: 2.877741343538526 (-0.007999403375975156) | > avg_loss_disc_real_0: 0.2416839341462498 (-0.0020679332001108386) | > avg_loss_disc_real_1: 0.241957415367516 (+0.0006583592211696632) | > avg_loss_disc_real_2: 0.24446731197162413 (-0.0011947560058513107) | > avg_loss_disc_real_3: 0.24346530773270297 (-0.002509896813983642) | > avg_loss_disc_real_4: 0.24341142702270563 (-0.0024703291940017247) | > avg_loss_disc_real_5: 0.24444520914218795 (+0.0020527858549440336) | > avg_loss_0: 2.877741343538526 (-0.007999403375975156) | > avg_grad_norm_0: tensor(7.1236, device='cuda:0') (+tensor(0.1190, device='cuda:0')) | > avg_loss_gen: 1.684932341038341 (+0.014245056770217257) | > avg_loss_kl: 2.713794449685325 (-0.07179215592397759) | > avg_loss_feat: 1.2489054740314751 (+0.04655147270417537) | > avg_loss_mel: 19.97574373701928 (+0.0506316977487451) | > avg_loss_duration: 1.776198706156771 (-0.0005178149317350833) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.399574682746135 (+0.039118242935394676) | > avg_grad_norm_1: tensor(202.5832, device='cuda:0') (tensor(-26.4121, device='cuda:0'))  > EPOCH: 14/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:19:48)   --> TIME: 2025-02-25 08:20:08 -- STEP: 41/72 -- GLOBAL_STEP: 1001050 | > loss_disc: 2.977388858795166 (2.878531293171208) | > loss_disc_real_0: 0.25983405113220215 (0.2464263141155243) | > loss_disc_real_1: 0.191773921251297 (0.24152139229018513) | > loss_disc_real_2: 0.2701978087425232 (0.24393404257006762) | > loss_disc_real_3: 0.27118754386901855 (0.24190911687001948) | > loss_disc_real_4: 0.2590712904930115 (0.24080294952159975) | > loss_disc_real_5: 0.2845865488052368 (0.2433111318727819) | > loss_0: 2.977388858795166 (2.878531293171208) | > grad_norm_0: tensor(8.9156, device='cuda:0') (tensor(7.1205, device='cuda:0')) | > loss_gen: 1.6471076011657715 (1.6864519439092496) | > loss_kl: 2.553969383239746 (2.7621085876371803) | > loss_feat: 1.1011830568313599 (1.312230002589342) | > loss_mel: 18.994234085083008 (19.960613064649625) | > loss_duration: 1.7527351379394531 (1.7656976653308403) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.049230575561523 (27.48710143856886) | > grad_norm_1: tensor(132.5952, device='cuda:0') (tensor(205.4990, device='cuda:0')) | > current_lr_0: 0.0001996502842328613 | > current_lr_1: 0.0001996502842328613 | > step_time: 0.4414 (0.4329438674740675) | > loader_time: 0.0089 (0.007291270465385623) --> EVAL PERFORMANCE | > avg_loader_time: 0.00846931967936771 (-4.574950312224817e-05) | > avg_step_time: 0.44765994246576873 (-0.005305417826477921) | > avg_loss_disc: 2.885775539237009 (+0.008034195698483071) | > avg_loss_disc_real_0: 0.24507415882298644 (+0.003390224676736636) | > avg_loss_disc_real_1: 0.2413995689909223 (-0.0005578463765937014) | > avg_loss_disc_real_2: 0.24442541599273682 (-4.189597888731855e-05) | > avg_loss_disc_real_3: 0.2437166735320024 (+0.0002513657992994356) | > avg_loss_disc_real_4: 0.24321623104558865 (-0.00019519597711697978) | > avg_loss_disc_real_5: 0.2426734900390598 (-0.0017717191031281498) | > avg_loss_0: 2.885775539237009 (+0.008034195698483071) | > avg_grad_norm_0: tensor(6.7890, device='cuda:0') (tensor(-0.3345, device='cuda:0')) | > avg_loss_gen: 1.678617450552927 (-0.006314890485414004) | > avg_loss_kl: 2.704093714834939 (-0.009700734850385917) | > avg_loss_feat: 1.2663006589446266 (+0.017395184913151507) | > avg_loss_mel: 19.912659443600077 (-0.06308429341920174) | > avg_loss_duration: 1.7729368562429724 (-0.003261849913798631) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.334608266051386 (-0.06496641669474812) | > avg_grad_norm_1: tensor(202.0191, device='cuda:0') (tensor(-0.5641, device='cuda:0'))  > EPOCH: 15/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:20:25)   --> TIME: 2025-02-25 08:20:35 -- STEP: 19/72 -- GLOBAL_STEP: 1001100 | > loss_disc: 2.9587786197662354 (2.891740736208464) | > loss_disc_real_0: 0.27034133672714233 (0.24588708579540253) | > loss_disc_real_1: 0.34924983978271484 (0.24602184562306656) | > loss_disc_real_2: 0.2560771703720093 (0.243091513451777) | > loss_disc_real_3: 0.27094972133636475 (0.24866846988075658) | > loss_disc_real_4: 0.32028138637542725 (0.24683895001285955) | > loss_disc_real_5: 0.31681591272354126 (0.24171258117023267) | > loss_0: 2.9587786197662354 (2.891740736208464) | > grad_norm_0: tensor(6.8855, device='cuda:0') (tensor(7.5957, device='cuda:0')) | > loss_gen: 1.734466791152954 (1.7025118187854165) | > loss_kl: 2.6958789825439453 (2.621646655233283) | > loss_feat: 1.0892359018325806 (1.2940272657494796) | > loss_mel: 19.261457443237305 (20.187122746517783) | > loss_duration: 1.712044596672058 (1.7523123590569747) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.49308204650879 (27.557620901810495) | > grad_norm_1: tensor(70.3258, device='cuda:0') (tensor(167.9343, device='cuda:0')) | > current_lr_0: 0.00019962532794733217 | > current_lr_1: 0.00019962532794733217 | > step_time: 0.4331 (0.4235148555354068) | > loader_time: 0.0066 (0.006843265734220806)  --> TIME: 2025-02-25 08:20:58 -- STEP: 69/72 -- GLOBAL_STEP: 1001150 | > loss_disc: 2.8806498050689697 (2.8952288282090337) | > loss_disc_real_0: 0.2649874687194824 (0.24395019420679065) | > loss_disc_real_1: 0.26336660981178284 (0.24373457850753397) | > loss_disc_real_2: 0.24689912796020508 (0.2459157549816629) | > loss_disc_real_3: 0.24482598900794983 (0.24489527012127033) | > loss_disc_real_4: 0.25887614488601685 (0.2448471415302028) | > loss_disc_real_5: 0.23329280316829681 (0.24248343878898068) | > loss_0: 2.8806498050689697 (2.8952288282090337) | > grad_norm_0: tensor(2.8627, device='cuda:0') (tensor(5.9190, device='cuda:0')) | > loss_gen: 1.6929004192352295 (1.6703599501347197) | > loss_kl: 2.634803056716919 (2.6027731930000195) | > loss_feat: 1.1456425189971924 (1.2149052343506739) | > loss_mel: 19.230792999267578 (19.65604370227758) | > loss_duration: 1.8460369110107422 (1.763031285742055) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.550174713134766 (26.90711333786232) | > grad_norm_1: tensor(278.4644, device='cuda:0') (tensor(227.6208, device='cuda:0')) | > current_lr_0: 0.00019962532794733217 | > current_lr_1: 0.00019962532794733217 | > step_time: 0.5385 (0.44761247911315033) | > loader_time: 0.0126 (0.008410688759624094) --> EVAL PERFORMANCE | > avg_loader_time: 0.008482197640647351 (+1.2877961279640637e-05) | > avg_step_time: 0.4496731355156697 (+0.002013193049900941) | > avg_loss_disc: 2.8944109695058473 (+0.008635430268838284) | > avg_loss_disc_real_0: 0.24340019805330626 (-0.0016739607696801828) | > avg_loss_disc_real_1: 0.24362085444826476 (+0.002221285457342459) | > avg_loss_disc_real_2: 0.24575299696183542 (+0.0013275809690986018) | > avg_loss_disc_real_3: 0.24498131367522227 (+0.001264640143219864) | > avg_loss_disc_real_4: 0.24501936389526852 (+0.001803132849679867) | > avg_loss_disc_real_5: 0.24291912581719144 (+0.0002456357781316365) | > avg_loss_0: 2.8944109695058473 (+0.008635430268838284) | > avg_grad_norm_0: tensor(5.8704, device='cuda:0') (tensor(-0.9186, device='cuda:0')) | > avg_loss_gen: 1.6680736239527312 (-0.010543826600195816) | > avg_loss_kl: 2.6072503311533324 (-0.09684338368160672) | > avg_loss_feat: 1.2130633743715953 (-0.05323728457303134) | > avg_loss_mel: 19.660011989969597 (-0.25264745363048036) | > avg_loss_duration: 1.762927406270739 (-0.010009449972233408) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.911326690458914 (-0.4232815755924726) | > avg_grad_norm_1: tensor(226.3823, device='cuda:0') (+tensor(24.3631, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001153.pth  > EPOCH: 16/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:21:04)   --> TIME: 2025-02-25 08:21:27 -- STEP: 47/72 -- GLOBAL_STEP: 1001200 | > loss_disc: 2.8801558017730713 (2.89119850321019) | > loss_disc_real_0: 0.22632938623428345 (0.24236279662619245) | > loss_disc_real_1: 0.24657349288463593 (0.2369601436434908) | > loss_disc_real_2: 0.2877485454082489 (0.24102419646496467) | > loss_disc_real_3: 0.2370375245809555 (0.24164117302032226) | > loss_disc_real_4: 0.2510968744754791 (0.24444667551111668) | > loss_disc_real_5: 0.21609003841876984 (0.2449251632107065) | > loss_0: 2.8801558017730713 (2.89119850321019) | > grad_norm_0: tensor(3.4926, device='cuda:0') (tensor(7.1661, device='cuda:0')) | > loss_gen: 1.5476938486099243 (1.6523948755670101) | > loss_kl: 2.5381743907928467 (2.70546947134302) | > loss_feat: 1.0153754949569702 (1.2175817299396432) | > loss_mel: 19.034391403198242 (19.714518689094707) | > loss_duration: 1.774019479751587 (1.752899134412725) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.909652709960938 (27.042863724079538) | > grad_norm_1: tensor(87.6172, device='cuda:0') (tensor(227.5675, device='cuda:0')) | > current_lr_0: 0.00019960037478133875 | > current_lr_1: 0.00019960037478133875 | > step_time: 0.4417 (0.43663811683654785) | > loader_time: 0.0088 (0.007510621496971617) --> EVAL PERFORMANCE | > avg_loader_time: 0.008463342424849389 (-1.8855215797962746e-05) | > avg_step_time: 0.45154077570203327 (+0.001867640186363595) | > avg_loss_disc: 2.887251558438154 (-0.007159411067693355) | > avg_loss_disc_real_0: 0.24612293633776652 (+0.0027227382844602632) | > avg_loss_disc_real_1: 0.23776750612846562 (-0.00585334831979914) | > avg_loss_disc_real_2: 0.24274722476240615 (-0.0030057721994292685) | > avg_loss_disc_real_3: 0.24198563417918245 (-0.0029956794960398214) | > avg_loss_disc_real_4: 0.24403912374671077 (-0.0009802401485577428) | > avg_loss_disc_real_5: 0.24497131394668364 (+0.002052188129492205) | > avg_loss_0: 2.887251558438154 (-0.007159411067693355) | > avg_grad_norm_0: tensor(8.2155, device='cuda:0') (+tensor(2.3451, device='cuda:0')) | > avg_loss_gen: 1.670580568447919 (+0.002506944495187735) | > avg_loss_kl: 2.681766782008426 (+0.07451645085509373) | > avg_loss_feat: 1.2062283981014306 (-0.0068349762701647165) | > avg_loss_mel: 19.66550966719506 (+0.005497677225463349) | > avg_loss_duration: 1.7629009585984996 (-2.6447672239360287e-05) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.98698621400645 (+0.07565952354753591) | > avg_grad_norm_1: tensor(234.5394, device='cuda:0') (+tensor(8.1571, device='cuda:0'))  > EPOCH: 17/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:21:40)   --> TIME: 2025-02-25 08:21:53 -- STEP: 25/72 -- GLOBAL_STEP: 1001250 | > loss_disc: 2.790039539337158 (2.8882436561584472) | > loss_disc_real_0: 0.23388923704624176 (0.24141636848449707) | > loss_disc_real_1: 0.27879729866981506 (0.2392246311903) | > loss_disc_real_2: 0.2542441487312317 (0.24435311675071716) | > loss_disc_real_3: 0.2034023553133011 (0.2452193319797516) | > loss_disc_real_4: 0.2253572791814804 (0.24818501949310304) | > loss_disc_real_5: 0.26610299944877625 (0.2477766352891922) | > loss_0: 2.790039539337158 (2.8882436561584472) | > grad_norm_0: tensor(4.4893, device='cuda:0') (tensor(6.9446, device='cuda:0')) | > loss_gen: 1.4962618350982666 (1.685491557121277) | > loss_kl: 2.8698010444641113 (2.6294270038604735) | > loss_feat: 1.403889536857605 (1.287045631408691) | > loss_mel: 20.34833526611328 (19.278455886840824) | > loss_duration: 1.7233526706695557 (1.7481592416763305) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.84164047241211 (26.628579177856444) | > grad_norm_1: tensor(396.0205, device='cuda:0') (tensor(254.9120, device='cuda:0')) | > current_lr_0: 0.00019957542473449108 | > current_lr_1: 0.00019957542473449108 | > step_time: 0.4222 (0.4245296096801758) | > loader_time: 0.0071 (0.00649531364440918) --> EVAL PERFORMANCE | > avg_loader_time: 0.008334824736689176 (-0.0001285176881602127) | > avg_step_time: 0.45264697074890137 (+0.001106195046868097) | > avg_loss_disc: 2.8851246531580537 (-0.0021269052801002886) | > avg_loss_disc_real_0: 0.24287784057603756 (-0.003245095761728961) | > avg_loss_disc_real_1: 0.24051104087225147 (+0.002743534743785858) | > avg_loss_disc_real_2: 0.24369787456284106 (+0.0009506498004349107) | > avg_loss_disc_real_3: 0.24398707378078516 (+0.002001439601602706) | > avg_loss_disc_real_4: 0.24507518385497618 (+0.001036060108265402) | > avg_loss_disc_real_5: 0.24514119935707307 (+0.0001698854103894254) | > avg_loss_0: 2.8851246531580537 (-0.0021269052801002886) | > avg_grad_norm_0: tensor(6.5905, device='cuda:0') (tensor(-1.6250, device='cuda:0')) | > avg_loss_gen: 1.6853060755931155 (+0.014725507145196604) | > avg_loss_kl: 2.6858854327403323 (+0.004118650731906204) | > avg_loss_feat: 1.2748170201207552 (+0.06858862201932459) | > avg_loss_mel: 19.55238898371308 (-0.11312068348198068) | > avg_loss_duration: 1.7624613617507505 (-0.0004395968477490797) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.960858842016943 (-0.026127371989506543) | > avg_grad_norm_1: tensor(246.7722, device='cuda:0') (+tensor(12.2328, device='cuda:0'))  > EPOCH: 18/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:22:17)   --> TIME: 2025-02-25 08:22:20 -- STEP: 3/72 -- GLOBAL_STEP: 1001300 | > loss_disc: 3.022946834564209 (2.973973035812378) | > loss_disc_real_0: 0.21725699305534363 (0.2384794851144155) | > loss_disc_real_1: 0.19852706789970398 (0.26173823078473407) | > loss_disc_real_2: 0.24702955782413483 (0.2474996397892634) | > loss_disc_real_3: 0.29018011689186096 (0.27646850049495697) | > loss_disc_real_4: 0.29478195309638977 (0.28743542234102887) | > loss_disc_real_5: 0.32211995124816895 (0.27543029685815174) | > loss_0: 3.022946834564209 (2.973973035812378) | > grad_norm_0: tensor(3.5352, device='cuda:0') (tensor(4.5872, device='cuda:0')) | > loss_gen: 1.5852434635162354 (1.7098170916239421) | > loss_kl: 2.736971139907837 (2.7424144744873047) | > loss_feat: 1.158319115638733 (1.158014436562856) | > loss_mel: 19.42810821533203 (19.05272674560547) | > loss_duration: 1.7827937602996826 (1.7700860897699993) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.691434860229492 (26.433059056599934) | > grad_norm_1: tensor(352.0777, device='cuda:0') (tensor(311.0700, device='cuda:0')) | > current_lr_0: 0.00019955047780639926 | > current_lr_1: 0.00019955047780639926 | > step_time: 0.4377 (0.43405946095784503) | > loader_time: 0.0056 (0.006072282791137695)  --> TIME: 2025-02-25 08:22:43 -- STEP: 53/72 -- GLOBAL_STEP: 1001350 | > loss_disc: 2.9231977462768555 (2.8913787041070327) | > loss_disc_real_0: 0.3182792067527771 (0.24189460755519146) | > loss_disc_real_1: 0.21800488233566284 (0.2383382224811698) | > loss_disc_real_2: 0.19074992835521698 (0.2451438288081367) | > loss_disc_real_3: 0.24464261531829834 (0.2444407231402847) | > loss_disc_real_4: 0.25461989641189575 (0.2451840048691012) | > loss_disc_real_5: 0.22932298481464386 (0.2429240306593337) | > loss_0: 2.9231977462768555 (2.8913787041070327) | > grad_norm_0: tensor(7.9937, device='cuda:0') (tensor(6.5563, device='cuda:0')) | > loss_gen: 1.7604336738586426 (1.663887914621605) | > loss_kl: 2.6514511108398438 (2.5678084886298995) | > loss_feat: 1.0244704484939575 (1.2207508818158561) | > loss_mel: 17.696496963500977 (19.30402539811044) | > loss_duration: 1.8110160827636719 (1.7488588926927098) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.943866729736328 (26.50533150726894) | > grad_norm_1: tensor(103.9918, device='cuda:0') (tensor(242.3885, device='cuda:0')) | > current_lr_0: 0.00019955047780639926 | > current_lr_1: 0.00019955047780639926 | > step_time: 0.476 (0.4415345012017016) | > loader_time: 0.0093 (0.007609421352170549) --> EVAL PERFORMANCE | > avg_loader_time: 0.008441867962689467 (+0.00010704322600029077) | > avg_step_time: 0.4570131906321351 (+0.004366219883233713) | > avg_loss_disc: 2.8850599544149054 (-6.469874314829482e-05) | > avg_loss_disc_real_0: 0.2408868305280175 (-0.0019910100480200754) | > avg_loss_disc_real_1: 0.23664685307254255 (-0.003864187799708929) | > avg_loss_disc_real_2: 0.2445809673255598 (+0.0008830927627187513) | > avg_loss_disc_real_3: 0.24319868524309615 (-0.0007883885376890054) | > avg_loss_disc_real_4: 0.24389103537713977 (-0.0011841484778364053) | > avg_loss_disc_real_5: 0.24323827951726779 (-0.0019029198398052838) | > avg_loss_0: 2.8850599544149054 (-6.469874314829482e-05) | > avg_grad_norm_0: tensor(6.4609, device='cuda:0') (tensor(-0.1297, device='cuda:0')) | > avg_loss_gen: 1.664563251213289 (-0.02074282437982644) | > avg_loss_kl: 2.582585391863971 (-0.10330004087636135) | > avg_loss_feat: 1.2314131755224416 (-0.04340384459831359) | > avg_loss_mel: 19.358875220930067 (-0.19351376278301302) | > avg_loss_duration: 1.7536705933826073 (-0.008790768368143187) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.591107543085663 (-0.36975129893128056) | > avg_grad_norm_1: tensor(246.4767, device='cuda:0') (tensor(-0.2955, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001369.pth  > EPOCH: 19/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:22:57)   --> TIME: 2025-02-25 08:23:12 -- STEP: 31/72 -- GLOBAL_STEP: 1001400 | > loss_disc: 2.9657487869262695 (2.8861963748931885) | > loss_disc_real_0: 0.3554186522960663 (0.2408942493700212) | > loss_disc_real_1: 0.420213907957077 (0.24729587570313485) | > loss_disc_real_2: 0.24035559594631195 (0.24597587123993905) | > loss_disc_real_3: 0.23452018201351166 (0.24270666366623295) | > loss_disc_real_4: 0.22193938493728638 (0.2465059281356873) | > loss_disc_real_5: 0.21424835920333862 (0.24475495036571257) | > loss_0: 2.9657487869262695 (2.8861963748931885) | > grad_norm_0: tensor(17.0801, device='cuda:0') (tensor(7.9736, device='cuda:0')) | > loss_gen: 1.712208867073059 (1.7029488240518877) | > loss_kl: 2.682657480239868 (2.6941521167755127) | > loss_feat: 1.3996262550354004 (1.4023534482525242) | > loss_mel: 20.190542221069336 (19.872536382367535) | > loss_duration: 1.7450640201568604 (1.7386954369083527) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.730100631713867 (27.41068636986517) | > grad_norm_1: tensor(337.0285, device='cuda:0') (tensor(212.1895, device='cuda:0')) | > current_lr_0: 0.00019952553399667344 | > current_lr_1: 0.00019952553399667344 | > step_time: 0.448 (0.4353668612818564) | > loader_time: 0.0084 (0.0069086551666259766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008506358509332361 (+6.449054664289453e-05) | > avg_step_time: 0.45076894760131836 (-0.006244243030816721) | > avg_loss_disc: 2.9073612723552005 (+0.022301317940295107) | > avg_loss_disc_real_0: 0.24147499310718457 (+0.0005881625791670786) | > avg_loss_disc_real_1: 0.2435164761165498 (+0.006869623044007261) | > avg_loss_disc_real_2: 0.24652379079603812 (+0.0019428234704783054) | > avg_loss_disc_real_3: 0.2444078842519035 (+0.0012091990088073379) | > avg_loss_disc_real_4: 0.2468671998087789 (+0.0029761644316391167) | > avg_loss_disc_real_5: 0.2469807806569086 (+0.003742501139640808) | > avg_loss_0: 2.9073612723552005 (+0.022301317940295107) | > avg_grad_norm_0: tensor(6.9748, device='cuda:0') (+tensor(0.5139, device='cuda:0')) | > avg_loss_gen: 1.664663877285702 (+0.00010062607241301258) | > avg_loss_kl: 2.6745133299223136 (+0.09192793805834265) | > avg_loss_feat: 1.2391708652738116 (+0.007757689751370034) | > avg_loss_mel: 19.71130373780156 (+0.3524285168714947) | > avg_loss_duration: 1.758972812706316 (+0.005302219323708668) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 27.048624683433854 (+0.457517140348191) | > avg_grad_norm_1: tensor(186.7496, device='cuda:0') (tensor(-59.7271, device='cuda:0'))  > EPOCH: 20/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:23:33)   --> TIME: 2025-02-25 08:23:39 -- STEP: 9/72 -- GLOBAL_STEP: 1001450 | > loss_disc: 2.9057536125183105 (2.8519533475240073) | > loss_disc_real_0: 0.19697996973991394 (0.23669269680976868) | > loss_disc_real_1: 0.2975824177265167 (0.24019617007838356) | > loss_disc_real_2: 0.2127843052148819 (0.23714648849434322) | > loss_disc_real_3: 0.18719737231731415 (0.23879917959372202) | > loss_disc_real_4: 0.24907398223876953 (0.2450561308198505) | > loss_disc_real_5: 0.27047187089920044 (0.24283565249707964) | > loss_0: 2.9057536125183105 (2.8519533475240073) | > grad_norm_0: tensor(8.0786, device='cuda:0') (tensor(6.7126, device='cuda:0')) | > loss_gen: 1.6266931295394897 (1.6845524046156142) | > loss_kl: 2.3659915924072266 (2.481327348285251) | > loss_feat: 1.4207643270492554 (1.306417531437344) | > loss_mel: 20.02596092224121 (19.61341094970703) | > loss_duration: 1.759803056716919 (1.7548447317547269) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.199214935302734 (26.840553283691406) | > grad_norm_1: tensor(56.8906, device='cuda:0') (tensor(197.0267, device='cuda:0')) | > current_lr_0: 0.00019950059330492385 | > current_lr_1: 0.00019950059330492385 | > step_time: 0.4221 (0.4261559910244412) | > loader_time: 0.0068 (0.006579902436998155)  --> TIME: 2025-02-25 08:24:02 -- STEP: 59/72 -- GLOBAL_STEP: 1001500 | > loss_disc: 2.850343942642212 (2.903893834453518) | > loss_disc_real_0: 0.2358885109424591 (0.2425661152702267) | > loss_disc_real_1: 0.16816596686840057 (0.2410505739814144) | > loss_disc_real_2: 0.2285175770521164 (0.24695299717329317) | > loss_disc_real_3: 0.20940260589122772 (0.2452142031516059) | > loss_disc_real_4: 0.2107834815979004 (0.24606611036648185) | > loss_disc_real_5: 0.22869591414928436 (0.24516853210279496) | > loss_0: 2.850343942642212 (2.903893834453518) | > grad_norm_0: tensor(9.7535, device='cuda:0') (tensor(6.2345, device='cuda:0')) | > loss_gen: 1.7142466306686401 (1.660745638912007) | > loss_kl: 2.4597110748291016 (2.6089320990998868) | > loss_feat: 1.3537002801895142 (1.196908824524637) | > loss_mel: 19.05359649658203 (19.6433622069278) | > loss_duration: 1.750388503074646 (1.752715280500509) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.33164405822754 (26.862664109569486) | > grad_norm_1: tensor(345.9215, device='cuda:0') (tensor(201.6814, device='cuda:0')) | > current_lr_0: 0.00019950059330492385 | > current_lr_1: 0.00019950059330492385 | > step_time: 0.4598 (0.4391410876128633) | > loader_time: 0.0097 (0.007933745949955308) --> EVAL PERFORMANCE | > avg_loader_time: 0.008486969370237537 (-1.9389139094824065e-05) | > avg_step_time: 0.44967846467461386 (-0.0010904829267044946) | > avg_loss_disc: 2.913110464391574 (+0.005749192036373696) | > avg_loss_disc_real_0: 0.24356493752607158 (+0.0020899444188870187) | > avg_loss_disc_real_1: 0.24344738058640925 (-6.909553014056158e-05) | > avg_loss_disc_real_2: 0.24606376088840862 (-0.0004600299076295) | > avg_loss_disc_real_3: 0.246014116515576 (+0.0016062322636725135) | > avg_loss_disc_real_4: 0.24706691566487432 (+0.00019971585609543374) | > avg_loss_disc_real_5: 0.24677321524687215 (-0.00020756541003644213) | > avg_loss_0: 2.913110464391574 (+0.005749192036373696) | > avg_grad_norm_0: tensor(5.9363, device='cuda:0') (tensor(-1.0385, device='cuda:0')) | > avg_loss_gen: 1.6528658161700611 (-0.011798061115640968) | > avg_loss_kl: 2.606387517821621 (-0.06812581210069268) | > avg_loss_feat: 1.1746448370772349 (-0.06452602819657671) | > avg_loss_mel: 19.608869848117024 (-0.10243388968453715) | > avg_loss_duration: 1.7544645759421336 (-0.004508236764182438) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.797232614436616 (-0.2513920689972373) | > avg_grad_norm_1: tensor(218.1075, device='cuda:0') (+tensor(31.3579, device='cuda:0'))  > EPOCH: 21/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:24:10)   --> TIME: 2025-02-25 08:24:28 -- STEP: 37/72 -- GLOBAL_STEP: 1001550 | > loss_disc: 2.9448161125183105 (2.8914007302877067) | > loss_disc_real_0: 0.28195828199386597 (0.24145464075578227) | > loss_disc_real_1: 0.3972407877445221 (0.24000603927148356) | > loss_disc_real_2: 0.23614472150802612 (0.24690183795787193) | > loss_disc_real_3: 0.2474469393491745 (0.24386362368996078) | > loss_disc_real_4: 0.24078160524368286 (0.24511757291652062) | > loss_disc_real_5: 0.268418550491333 (0.24630653254083684) | > loss_0: 2.9448161125183105 (2.8914007302877067) | > grad_norm_0: tensor(15.5872, device='cuda:0') (tensor(4.6089, device='cuda:0')) | > loss_gen: 1.8741023540496826 (1.660048181946213) | > loss_kl: 2.743349552154541 (2.611531502491719) | > loss_feat: 1.009792447090149 (1.2089776219548405) | > loss_mel: 19.08313751220703 (19.38737389848039) | > loss_duration: 1.7342770099639893 (1.732447582322198) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.444658279418945 (26.60037865509858) | > grad_norm_1: tensor(91.5304, device='cuda:0') (tensor(217.5773, device='cuda:0')) | > current_lr_0: 0.00019947565573076072 | > current_lr_1: 0.00019947565573076072 | > step_time: 0.4424 (0.42828393626857447) | > loader_time: 0.0086 (0.007540071332776868) --> EVAL PERFORMANCE | > avg_loader_time: 0.008637079050843143 (+0.00015010968060560556) | > avg_step_time: 0.44622098895865425 (-0.0034574757159596126) | > avg_loss_disc: 2.8921016303586287 (-0.02100883403294551) | > avg_loss_disc_real_0: 0.24245233225150847 (-0.001112605274563111) | > avg_loss_disc_real_1: 0.24015515476045474 (-0.003292225825954509) | > avg_loss_disc_real_2: 0.24692973494529724 (+0.0008659740568886243) | > avg_loss_disc_real_3: 0.24394275101137833 (-0.0020713655041976753) | > avg_loss_disc_real_4: 0.24475297877486324 (-0.0023139368900110813) | > avg_loss_disc_real_5: 0.2461578434621784 (-0.0006153717846937579) | > avg_loss_0: 2.8921016303586287 (-0.02100883403294551) | > avg_grad_norm_0: tensor(5.5973, device='cuda:0') (tensor(-0.3390, device='cuda:0')) | > avg_loss_gen: 1.6586350840581974 (+0.005769267888136254) | > avg_loss_kl: 2.5842671495088387 (-0.02212036831278219) | > avg_loss_feat: 1.1947253321258118 (+0.020080495048576896) | > avg_loss_mel: 19.36354575358646 (-0.24532409453056303) | > avg_loss_duration: 1.7461506346581688 (-0.008313941283964787) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.547323898530344 (-0.2499087159062725) | > avg_grad_norm_1: tensor(240.3257, device='cuda:0') (+tensor(22.2182, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001585.pth  > EPOCH: 22/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:24:48)   --> TIME: 2025-02-25 08:24:56 -- STEP: 15/72 -- GLOBAL_STEP: 1001600 | > loss_disc: 2.9230260848999023 (2.9044661362965902) | > loss_disc_real_0: 0.26215627789497375 (0.250490923722585) | > loss_disc_real_1: 0.19402411580085754 (0.22839810848236083) | > loss_disc_real_2: 0.26670971512794495 (0.2391862432161967) | > loss_disc_real_3: 0.25199875235557556 (0.24446295499801635) | > loss_disc_real_4: 0.2547849714756012 (0.24881505270799) | > loss_disc_real_5: 0.256616473197937 (0.24873322943846385) | > loss_0: 2.9230260848999023 (2.9044661362965902) | > grad_norm_0: tensor(6.9145, device='cuda:0') (tensor(6.3935, device='cuda:0')) | > loss_gen: 1.6616921424865723 (1.6617403745651245) | > loss_kl: 2.4423415660858154 (2.540050919850667) | > loss_feat: 1.0093644857406616 (1.230627751350403) | > loss_mel: 19.70926284790039 (19.273007202148438) | > loss_duration: 1.697892665863037 (1.7411004940668742) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.520553588867188 (26.446527099609376) | > grad_norm_1: tensor(317.4110, device='cuda:0') (tensor(314.8491, device='cuda:0')) | > current_lr_0: 0.00019945072127379438 | > current_lr_1: 0.00019945072127379438 | > step_time: 0.4238 (0.42617629369099935) | > loader_time: 0.0058 (0.006667598088582357)  --> TIME: 2025-02-25 08:25:20 -- STEP: 65/72 -- GLOBAL_STEP: 1001650 | > loss_disc: 2.921804189682007 (2.906099653244018) | > loss_disc_real_0: 0.30117878317832947 (0.24386619444076832) | > loss_disc_real_1: 0.28007960319519043 (0.23443187394967446) | > loss_disc_real_2: 0.2356089949607849 (0.24304747306383573) | > loss_disc_real_3: 0.2165457010269165 (0.24201442209573892) | > loss_disc_real_4: 0.23861585557460785 (0.24585791069727678) | > loss_disc_real_5: 0.22613444924354553 (0.24501121090008662) | > loss_0: 2.921804189682007 (2.906099653244018) | > grad_norm_0: tensor(5.7179, device='cuda:0') (tensor(7.2561, device='cuda:0')) | > loss_gen: 1.6835285425186157 (1.644925198188195) | > loss_kl: 2.5612876415252686 (2.5255652390993557) | > loss_feat: 1.1194744110107422 (1.1925956010818481) | > loss_mel: 19.547809600830078 (18.938758351252638) | > loss_duration: 1.7696552276611328 (1.7377421709207388) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.68175506591797 (26.039586844811073) | > grad_norm_1: tensor(497.3328, device='cuda:0') (tensor(277.7334, device='cuda:0')) | > current_lr_0: 0.00019945072127379438 | > current_lr_1: 0.00019945072127379438 | > step_time: 0.4902 (0.4549895506638747) | > loader_time: 0.0115 (0.008243193993201626) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850734912173849 (-0.0001297299291046529) | > avg_step_time: 0.460912586937488 (+0.014691597978833737) | > avg_loss_disc: 2.905481429167196 (+0.013379798808567323) | > avg_loss_disc_real_0: 0.24452441280156795 (+0.002072080550059474) | > avg_loss_disc_real_1: 0.2356960837480048 (-0.004459071012449939) | > avg_loss_disc_real_2: 0.2435456266705419 (-0.003384108274755354) | > avg_loss_disc_real_3: 0.2420432443769885 (-0.0018995066343898415) | > avg_loss_disc_real_4: 0.24546054068585516 (+0.0007075619109919162) | > avg_loss_disc_real_5: 0.2446866297805813 (-0.0014712136815970833) | > avg_loss_0: 2.905481429167196 (+0.013379798808567323) | > avg_grad_norm_0: tensor(7.1260, device='cuda:0') (+tensor(1.5287, device='cuda:0')) | > avg_loss_gen: 1.6456351532063014 (-0.012999930851895991) | > avg_loss_kl: 2.5346831200828013 (-0.04958402942603746) | > avg_loss_feat: 1.1953330862690026 (+0.0006077541431908084) | > avg_loss_mel: 19.04634712111783 (-0.31719863246863156) | > avg_loss_duration: 1.740200036008593 (-0.005950598649575811) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.162198778609156 (-0.38512511992118803) | > avg_grad_norm_1: tensor(291.7075, device='cuda:0') (+tensor(51.3817, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1001657.pth  > EPOCH: 23/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:25:28)   --> TIME: 2025-02-25 08:25:49 -- STEP: 43/72 -- GLOBAL_STEP: 1001700 | > loss_disc: 2.9389889240264893 (2.9011113643646236) | > loss_disc_real_0: 0.18256796896457672 (0.2411255053309507) | > loss_disc_real_1: 0.21409055590629578 (0.23725411607775576) | > loss_disc_real_2: 0.2704111933708191 (0.24729260005230128) | > loss_disc_real_3: 0.27913230657577515 (0.24738186909708865) | > loss_disc_real_4: 0.2609252631664276 (0.24790116035661033) | > loss_disc_real_5: 0.28942105174064636 (0.24735097587108612) | > loss_0: 2.9389889240264893 (2.9011113643646236) | > grad_norm_0: tensor(6.4940, device='cuda:0') (tensor(5.9958, device='cuda:0')) | > loss_gen: 1.523191213607788 (1.6604462501614592) | > loss_kl: 2.463573932647705 (2.6068627778873887) | > loss_feat: 1.3138432502746582 (1.2005345447118891) | > loss_mel: 18.837017059326172 (19.157601733540382) | > loss_duration: 1.7183700799942017 (1.72453605019769) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.855995178222656 (26.349981352340343) | > grad_norm_1: tensor(369.1953, device='cuda:0') (tensor(254.5417, device='cuda:0')) | > current_lr_0: 0.00019942578993363514 | > current_lr_1: 0.00019942578993363514 | > step_time: 0.4554 (0.4330055270084115) | > loader_time: 0.0096 (0.0073826701142067136) --> EVAL PERFORMANCE | > avg_loader_time: 0.008522933637592154 (+1.5584515853664382e-05) | > avg_step_time: 0.45705856403834383 (-0.0038540228991441605) | > avg_loss_disc: 2.9063831416653914 (+0.0009017124981953728) | > avg_loss_disc_real_0: 0.2417506029908086 (-0.0027738098107593367) | > avg_loss_disc_real_1: 0.2354227019027925 (-0.000273381845212306) | > avg_loss_disc_real_2: 0.24400462441041437 (+0.0004589977398724854) | > avg_loss_disc_real_3: 0.24621158586421482 (+0.004168341487226335) | > avg_loss_disc_real_4: 0.24701110955695033 (+0.0015505688710951704) | > avg_loss_disc_real_5: 0.2468491983665547 (+0.0021625685859733845) | > avg_loss_0: 2.9063831416653914 (+0.0009017124981953728) | > avg_grad_norm_0: tensor(6.3835, device='cuda:0') (tensor(-0.7425, device='cuda:0')) | > avg_loss_gen: 1.648840388781588 (+0.0032052355752865136) | > avg_loss_kl: 2.567896611253979 (+0.03321349117117789) | > avg_loss_feat: 1.1878893308236567 (-0.007443755445345923) | > avg_loss_mel: 19.235340763145768 (+0.18899364202793834) | > avg_loss_duration: 1.733704793621117 (-0.006495242387476008) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.373672001798386 (+0.21147322318923045) | > avg_grad_norm_1: tensor(279.7248, device='cuda:0') (tensor(-11.9827, device='cuda:0'))  > EPOCH: 24/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:26:05)   --> TIME: 2025-02-25 08:26:15 -- STEP: 21/72 -- GLOBAL_STEP: 1001750 | > loss_disc: 2.8914008140563965 (2.894137461980184) | > loss_disc_real_0: 0.23588694632053375 (0.24143440240905398) | > loss_disc_real_1: 0.26899799704551697 (0.2443155717282068) | > loss_disc_real_2: 0.2539578378200531 (0.24783170010362351) | > loss_disc_real_3: 0.24179720878601074 (0.2451774598587127) | > loss_disc_real_4: 0.23163126409053802 (0.2511644583372843) | > loss_disc_real_5: 0.25078728795051575 (0.24738918173880803) | > loss_0: 2.8914008140563965 (2.894137461980184) | > grad_norm_0: tensor(4.5616, device='cuda:0') (tensor(7.9450, device='cuda:0')) | > loss_gen: 1.5127112865447998 (1.654058882168361) | > loss_kl: 2.279688835144043 (2.4455625670296803) | > loss_feat: 1.0835776329040527 (1.2152665853500366) | > loss_mel: 18.296741485595703 (19.25797798520043) | > loss_duration: 1.752542495727539 (1.717810034751892) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.925260543823242 (26.290676026117232) | > grad_norm_1: tensor(280.0089, device='cuda:0') (tensor(312.6554, device='cuda:0')) | > current_lr_0: 0.00019940086170989343 | > current_lr_1: 0.00019940086170989343 | > step_time: 0.4277 (0.42572488103594097) | > loader_time: 0.0077 (0.006925696418398903)  --> TIME: 2025-02-25 08:26:39 -- STEP: 71/72 -- GLOBAL_STEP: 1001800 | > loss_disc: 2.7988154888153076 (2.909002508915646) | > loss_disc_real_0: 0.18274395167827606 (0.2420362695841722) | > loss_disc_real_1: 0.2537137567996979 (0.24012519293267962) | > loss_disc_real_2: 0.21837392449378967 (0.2450858241235706) | > loss_disc_real_3: 0.2159198671579361 (0.24332928804444595) | > loss_disc_real_4: 0.24884147942066193 (0.2466041926347034) | > loss_disc_real_5: 0.2403886467218399 (0.24577752931017272) | > loss_0: 2.7988154888153076 (2.909002508915646) | > grad_norm_0: tensor(6.7448, device='cuda:0') (tensor(6.8917, device='cuda:0')) | > loss_gen: 1.405028223991394 (1.6372650777789908) | > loss_kl: 2.322584390640259 (2.579905849107554) | > loss_feat: 1.049839735031128 (1.1836473589212122) | > loss_mel: 18.84657859802246 (19.19130070108763) | > loss_duration: 1.7304630279541016 (1.7336897531025846) | > amp_scaler: 128.0 (227.1549295774648) | > loss_1: 25.3544921875 (26.325808860886262) | > grad_norm_1: tensor(144.4796, device='cuda:0') (tensor(299.6810, device='cuda:0')) | > current_lr_0: 0.00019940086170989343 | > current_lr_1: 0.00019940086170989343 | > step_time: 0.4663 (0.4488499634702441) | > loader_time: 0.0078 (0.008547447097133582) --> EVAL PERFORMANCE | > avg_loader_time: 0.008547447097133582 (+2.4513459541427793e-05) | > avg_step_time: 0.4488499634702441 (-0.008208600568099744) | > avg_loss_disc: 2.909002508915646 (+0.0026193672502548004) | > avg_loss_disc_real_0: 0.2420362695841722 (+0.00028566659336359823) | > avg_loss_disc_real_1: 0.24012519293267962 (+0.004702491029887124) | > avg_loss_disc_real_2: 0.2450858241235706 (+0.0010811997131562334) | > avg_loss_disc_real_3: 0.24332928804444595 (-0.00288229781976887) | > avg_loss_disc_real_4: 0.2466041926347034 (-0.0004069169222469249) | > avg_loss_disc_real_5: 0.24577752931017272 (-0.0010716690563819797) | > avg_loss_0: 2.909002508915646 (+0.0026193672502548004) | > avg_grad_norm_0: tensor(6.8917, device='cuda:0') (+tensor(0.5082, device='cuda:0')) | > avg_loss_gen: 1.6372650777789908 (-0.011575311002597122) | > avg_loss_kl: 2.579905849107554 (+0.012009237853574994) | > avg_loss_feat: 1.1836473589212122 (-0.0042419719024444635) | > avg_loss_mel: 19.19130070108763 (-0.04404006205813715) | > avg_loss_duration: 1.7336897531025846 (-1.50405185324054e-05) | > avg_amp_scaler: 227.1549295774648 (-28.8450704225352) | > avg_loss_1: 26.325808860886262 (-0.04786314091212418) | > avg_grad_norm_1: tensor(299.6810, device='cuda:0') (+tensor(19.9562, device='cuda:0'))  > EPOCH: 25/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:26:41)   --> TIME: 2025-02-25 08:27:05 -- STEP: 49/72 -- GLOBAL_STEP: 1001850 | > loss_disc: 2.8556389808654785 (2.895669635461301) | > loss_disc_real_0: 0.18077844381332397 (0.2396995613769609) | > loss_disc_real_1: 0.27560096979141235 (0.23867878196190814) | > loss_disc_real_2: 0.2606188654899597 (0.24328728750043985) | > loss_disc_real_3: 0.25425902009010315 (0.24187231489590236) | > loss_disc_real_4: 0.27478623390197754 (0.24526824482849666) | > loss_disc_real_5: 0.2694644033908844 (0.24516464587377043) | > loss_0: 2.8556389808654785 (2.895669635461301) | > grad_norm_0: tensor(8.2706, device='cuda:0') (tensor(7.1980, device='cuda:0')) | > loss_gen: 1.6491925716400146 (1.6565431551057466) | > loss_kl: 2.399681806564331 (2.750666000405136) | > loss_feat: 1.2994184494018555 (1.2535591356608329) | > loss_mel: 17.94442367553711 (19.61210262532137) | > loss_duration: 1.8006691932678223 (1.7377094857546749) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.093385696411133 (27.01058037427007) | > grad_norm_1: tensor(125.5050, device='cuda:0') (tensor(204.5266, device='cuda:0')) | > current_lr_0: 0.0001993759366021797 | > current_lr_1: 0.0001993759366021797 | > step_time: 0.4408 (0.4379293237413679) | > loader_time: 0.0095 (0.007641982059089505) --> EVAL PERFORMANCE | > avg_loader_time: 0.00854810526673223 (+6.581695986482283e-07) | > avg_step_time: 0.449549332470961 (+0.0006993690007169318) | > avg_loss_disc: 2.8968704653457857 (-0.012132043569860418) | > avg_loss_disc_real_0: 0.2431541687585938 (+0.0011178991744215938) | > avg_loss_disc_real_1: 0.23700751091392946 (-0.003117682018750151) | > avg_loss_disc_real_2: 0.24426208835252575 (-0.000823735771044859) | > avg_loss_disc_real_3: 0.24248939332827715 (-0.0008398947161687986) | > avg_loss_disc_real_4: 0.24520270694309557 (-0.0014014856916078344) | > avg_loss_disc_real_5: 0.2454278827972815 (-0.00034964651289121074) | > avg_loss_0: 2.8968704653457857 (-0.012132043569860418) | > avg_grad_norm_0: tensor(7.0618, device='cuda:0') (+tensor(0.1701, device='cuda:0')) | > avg_loss_gen: 1.658010931082175 (+0.020745853303184125) | > avg_loss_kl: 2.6929909074810188 (+0.11308505837346461) | > avg_loss_feat: 1.2413004023928036 (+0.05765304347159139) | > avg_loss_mel: 19.55610339742311 (+0.3648026963354809) | > avg_loss_duration: 1.740564715694374 (+0.006874962591789435) | > avg_amp_scaler: 128.0 (-99.1549295774648) | > avg_loss_1: 26.888970334764938 (+0.5631614738786759) | > avg_grad_norm_1: tensor(216.2965, device='cuda:0') (tensor(-83.3845, device='cuda:0'))  > EPOCH: 26/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:27:18)   --> TIME: 2025-02-25 08:27:31 -- STEP: 27/72 -- GLOBAL_STEP: 1001900 | > loss_disc: 2.8695228099823 (2.904220607545641) | > loss_disc_real_0: 0.26708269119262695 (0.2392360700501336) | > loss_disc_real_1: 0.20630159974098206 (0.24170565053268714) | > loss_disc_real_2: 0.22509101033210754 (0.24667198956012723) | > loss_disc_real_3: 0.21079277992248535 (0.2459065682358212) | > loss_disc_real_4: 0.21353182196617126 (0.2432436529133055) | > loss_disc_real_5: 0.21402543783187866 (0.24523563462275047) | > loss_0: 2.8695228099823 (2.904220607545641) | > grad_norm_0: tensor(4.0151, device='cuda:0') (tensor(6.3086, device='cuda:0')) | > loss_gen: 1.6267127990722656 (1.6586766331284135) | > loss_kl: 2.5772647857666016 (2.592590888341268) | > loss_feat: 1.3629419803619385 (1.2265390599215473) | > loss_mel: 19.445457458496094 (19.550475791648584) | > loss_duration: 1.7088394165039062 (1.7162652368898745) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.721216201782227 (26.744547808611834) | > grad_norm_1: tensor(293.4725, device='cuda:0') (tensor(185.3245, device='cuda:0')) | > current_lr_0: 0.00019935101461010442 | > current_lr_1: 0.00019935101461010442 | > step_time: 0.4334 (0.42764713146068434) | > loader_time: 0.0084 (0.0068843188109221284) --> EVAL PERFORMANCE | > avg_loader_time: 0.00844210638126857 (-0.00010599888546366024) | > avg_step_time: 0.44634522182840697 (-0.0032041106425540478) | > avg_loss_disc: 2.912003010091648 (+0.01513254474586212) | > avg_loss_disc_real_0: 0.24467630537462906 (+0.0015221366160352579) | > avg_loss_disc_real_1: 0.2403399060729524 (+0.0033323951590229217) | > avg_loss_disc_real_2: 0.24538843354708711 (+0.0011263451945613678) | > avg_loss_disc_real_3: 0.2452800800682793 (+0.0027906867400021573) | > avg_loss_disc_real_4: 0.24644167553371107 (+0.001238968590615508) | > avg_loss_disc_real_5: 0.2465311200685904 (+0.001103237271308899) | > avg_loss_0: 2.912003010091648 (+0.01513254474586212) | > avg_grad_norm_0: tensor(6.5527, device='cuda:0') (tensor(-0.5092, device='cuda:0')) | > avg_loss_gen: 1.6564736567752463 (-0.0015372743069286265) | > avg_loss_kl: 2.5982511580829883 (-0.09473974939803043) | > avg_loss_feat: 1.2037514305450552 (-0.037548971847748414) | > avg_loss_mel: 19.39437296692754 (-0.16173043049557023) | > avg_loss_duration: 1.7291814743633 (-0.011383241331073934) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.582030658990565 (-0.3069396757743732) | > avg_grad_norm_1: tensor(203.6570, device='cuda:0') (tensor(-12.6396, device='cuda:0'))  > EPOCH: 27/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:27:54)   --> TIME: 2025-02-25 08:27:58 -- STEP: 5/72 -- GLOBAL_STEP: 1001950 | > loss_disc: 2.880256175994873 (2.9262388229370115) | > loss_disc_real_0: 0.20891675353050232 (0.25772765576839446) | > loss_disc_real_1: 0.26630711555480957 (0.2540987581014633) | > loss_disc_real_2: 0.18986926972866058 (0.2537607401609421) | > loss_disc_real_3: 0.23387284576892853 (0.26375583112239837) | > loss_disc_real_4: 0.24224036931991577 (0.2527202367782593) | > loss_disc_real_5: 0.2546103894710541 (0.2588008642196655) | > loss_0: 2.880256175994873 (2.9262388229370115) | > grad_norm_0: tensor(4.0717, device='cuda:0') (tensor(6.6429, device='cuda:0')) | > loss_gen: 1.6446689367294312 (1.6761852979660035) | > loss_kl: 2.6150712966918945 (2.671172571182251) | > loss_feat: 1.2884923219680786 (1.2545649766921998) | > loss_mel: 19.72877311706543 (19.12577590942383) | > loss_duration: 1.680324673652649 (1.735061240196228) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.95732879638672 (26.46275978088379) | > grad_norm_1: tensor(410.9273, device='cuda:0') (tensor(354.3331, device='cuda:0')) | > current_lr_0: 0.00019932609573327815 | > current_lr_1: 0.00019932609573327815 | > step_time: 0.4233 (0.42119503021240234) | > loader_time: 0.006 (0.006618404388427734)  --> TIME: 2025-02-25 08:28:20 -- STEP: 55/72 -- GLOBAL_STEP: 1002000 | > loss_disc: 3.0089752674102783 (2.907105168429288) | > loss_disc_real_0: 0.3770425021648407 (0.2445363919843327) | > loss_disc_real_1: 0.25121235847473145 (0.24085338684645566) | > loss_disc_real_2: 0.27587640285491943 (0.24784060201861638) | > loss_disc_real_3: 0.2669304311275482 (0.24532388069412925) | > loss_disc_real_4: 0.2648363709449768 (0.246510297330943) | > loss_disc_real_5: 0.2876167595386505 (0.2465935157103972) | > loss_0: 3.0089752674102783 (2.907105168429288) | > grad_norm_0: tensor(8.9664, device='cuda:0') (tensor(6.0818, device='cuda:0')) | > loss_gen: 1.7046538591384888 (1.6561946218663997) | > loss_kl: 2.5846939086914062 (2.664400477842851) | > loss_feat: 0.970173180103302 (1.2314907930114052) | > loss_mel: 19.04538917541504 (19.01426089893688) | > loss_duration: 1.7613420486450195 (1.725287628173828) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.066253662109375 (26.29163440357555) | > grad_norm_1: tensor(171.7437, device='cuda:0') (tensor(224.7897, device='cuda:0')) | > current_lr_0: 0.00019932609573327815 | > current_lr_1: 0.00019932609573327815 | > step_time: 0.459 (0.4345296079462225) | > loader_time: 0.0093 (0.007889921014959162) --> EVAL PERFORMANCE | > avg_loader_time: 0.008554230273609428 (+0.00011212389234085789) | > avg_step_time: 0.44428781052710303 (-0.0020574113013039352) | > avg_loss_disc: 2.8994698457314936 (-0.01253316436015428) | > avg_loss_disc_real_0: 0.24434050141086042 (-0.00033580396376864385) | > avg_loss_disc_real_1: 0.24105809104274695 (+0.0007181849697945608) | > avg_loss_disc_real_2: 0.2476593358835704 (+0.002270902336483277) | > avg_loss_disc_real_3: 0.24421324171650577 (-0.0010668383517735414) | > avg_loss_disc_real_4: 0.24537062707921148 (-0.0010710484544995957) | > avg_loss_disc_real_5: 0.24581523827264007 (-0.000715881795950335) | > avg_loss_0: 2.8994698457314936 (-0.01253316436015428) | > avg_grad_norm_0: tensor(5.9078, device='cuda:0') (tensor(-0.6449, device='cuda:0')) | > avg_loss_gen: 1.658694346186141 (+0.002220689410894705) | > avg_loss_kl: 2.6509465331762607 (+0.05269537509327238) | > avg_loss_feat: 1.2220029822537597 (+0.018251551708704516) | > avg_loss_mel: 19.1393998105761 (-0.2549731563514399) | > avg_loss_duration: 1.7305336871617276 (+0.0013522127984275656) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.40157734508246 (-0.1804533139081066) | > avg_grad_norm_1: tensor(230.7323, device='cuda:0') (+tensor(27.0753, device='cuda:0'))  > EPOCH: 28/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:28:30)   --> TIME: 2025-02-25 08:28:47 -- STEP: 33/72 -- GLOBAL_STEP: 1002050 | > loss_disc: 2.976036310195923 (2.9309029145674272) | > loss_disc_real_0: 0.20587162673473358 (0.23745529082688419) | > loss_disc_real_1: 0.1931743323802948 (0.23799870253512354) | > loss_disc_real_2: 0.27978792786598206 (0.24669118180419458) | > loss_disc_real_3: 0.28441089391708374 (0.2463208558884534) | > loss_disc_real_4: 0.2771759033203125 (0.24800060237898972) | > loss_disc_real_5: 0.2335912138223648 (0.24903935781030945) | > loss_0: 2.976036310195923 (2.9309029145674272) | > grad_norm_0: tensor(5.0772, device='cuda:0') (tensor(6.7029, device='cuda:0')) | > loss_gen: 1.6068952083587646 (1.6447518955577503) | > loss_kl: 2.9824321269989014 (2.6017336267413516) | > loss_feat: 1.1848703622817993 (1.2218602986046765) | > loss_mel: 18.921384811401367 (19.441229444561582) | > loss_duration: 1.7305535078048706 (1.7117823145606301) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.426136016845703 (26.621357599894207) | > grad_norm_1: tensor(148.0924, device='cuda:0') (tensor(181.7621, device='cuda:0')) | > current_lr_0: 0.0001993011799713115 | > current_lr_1: 0.0001993011799713115 | > step_time: 0.4367 (0.44811372323469684) | > loader_time: 0.0076 (0.00698749946825432) --> EVAL PERFORMANCE | > avg_loader_time: 0.008497459787718008 (-5.677048589142024e-05) | > avg_step_time: 0.4559218984254649 (+0.011634087898361845) | > avg_loss_disc: 2.916231155395508 (+0.016761309664014234) | > avg_loss_disc_real_0: 0.2406132393319842 (-0.00372726207887622) | > avg_loss_disc_real_1: 0.24099179881979044 (-6.629222295650994e-05) | > avg_loss_disc_real_2: 0.2462198795147345 (-0.0014394563688358841) | > avg_loss_disc_real_3: 0.24556239893738652 (+0.0013491572208807479) | > avg_loss_disc_real_4: 0.24798361852135456 (+0.002612991442143081) | > avg_loss_disc_real_5: 0.24840382413125375 (+0.0025885858586136845) | > avg_loss_0: 2.916231155395508 (+0.016761309664014234) | > avg_grad_norm_0: tensor(6.1530, device='cuda:0') (+tensor(0.2453, device='cuda:0')) | > avg_loss_gen: 1.655425572059524 (-0.00326877412661708) | > avg_loss_kl: 2.609278057662534 (-0.041668475513726655) | > avg_loss_feat: 1.2108437259432296 (-0.011159256310530052) | > avg_loss_mel: 19.29311628744636 (+0.15371647687025813) | > avg_loss_duration: 1.7276055275554387 (-0.0029281596062888937) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.49626922607422 (+0.09469188099176051) | > avg_grad_norm_1: tensor(211.5025, device='cuda:0') (tensor(-19.2298, device='cuda:0'))  > EPOCH: 29/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:29:07)   --> TIME: 2025-02-25 08:29:13 -- STEP: 11/72 -- GLOBAL_STEP: 1002100 | > loss_disc: 2.8044023513793945 (2.8665858182040127) | > loss_disc_real_0: 0.19525587558746338 (0.23262278464707462) | > loss_disc_real_1: 0.2704111635684967 (0.24143631078980185) | > loss_disc_real_2: 0.2375972718000412 (0.2398239726370031) | > loss_disc_real_3: 0.24933883547782898 (0.23597031154415823) | > loss_disc_real_4: 0.2570349872112274 (0.2354002743959427) | > loss_disc_real_5: 0.2277580052614212 (0.24056207456372) | > loss_0: 2.8044023513793945 (2.8665858182040127) | > grad_norm_0: tensor(5.0177, device='cuda:0') (tensor(6.4595, device='cuda:0')) | > loss_gen: 1.69826078414917 (1.6630829247561367) | > loss_kl: 2.561954975128174 (2.5437773140993984) | > loss_feat: 1.3805233240127563 (1.3183138804002241) | > loss_mel: 19.541391372680664 (19.84244138544256) | > loss_duration: 1.7258974313735962 (1.7264876040545376) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.90802764892578 (27.094103032892402) | > grad_norm_1: tensor(323.5125, device='cuda:0') (tensor(205.3059, device='cuda:0')) | > current_lr_0: 0.00019927626732381507 | > current_lr_1: 0.00019927626732381507 | > step_time: 0.4236 (0.4238014437935569) | > loader_time: 0.0062 (0.0062371384013782845)  --> TIME: 2025-02-25 08:29:36 -- STEP: 61/72 -- GLOBAL_STEP: 1002150 | > loss_disc: 2.961754322052002 (2.8894941142347994) | > loss_disc_real_0: 0.15305183827877045 (0.23720981034099078) | > loss_disc_real_1: 0.16426782310009003 (0.23510585163460404) | > loss_disc_real_2: 0.22591879963874817 (0.24480328989810632) | > loss_disc_real_3: 0.18942636251449585 (0.24150941801852868) | > loss_disc_real_4: 0.21312831342220306 (0.24316920242348655) | > loss_disc_real_5: 0.22157585620880127 (0.24319493672886833) | > loss_0: 2.961754322052002 (2.8894941142347994) | > grad_norm_0: tensor(15.1715, device='cuda:0') (tensor(6.0190, device='cuda:0')) | > loss_gen: 1.4736782312393188 (1.6540922375976062) | > loss_kl: 2.588895797729492 (2.545302051012633) | > loss_feat: 1.4081127643585205 (1.2811126992350723) | > loss_mel: 17.904130935668945 (18.973327167698596) | > loss_duration: 1.7446707487106323 (1.7199998664074256) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.119489669799805 (26.17383390958192) | > grad_norm_1: tensor(419.5436, device='cuda:0') (tensor(252.9745, device='cuda:0')) | > current_lr_0: 0.00019927626732381507 | > current_lr_1: 0.00019927626732381507 | > step_time: 0.4553 (0.44164360155824756) | > loader_time: 0.0103 (0.008093755753313909) --> EVAL PERFORMANCE | > avg_loader_time: 0.008529233260893486 (+3.177347317547853e-05) | > avg_step_time: 0.447637913932263 (-0.00828398449320189) | > avg_loss_disc: 2.8930417752601727 (-0.023189380135335114) | > avg_loss_disc_real_0: 0.23991943685941294 (-0.0006938024725712533) | > avg_loss_disc_real_1: 0.2363490254526407 (-0.004642773367149744) | > avg_loss_disc_real_2: 0.2443354113001219 (-0.0018844682146126057) | > avg_loss_disc_real_3: 0.24259746137639165 (-0.0029649375609948625) | > avg_loss_disc_real_4: 0.24398707126227903 (-0.003996547259075528) | > avg_loss_disc_real_5: 0.24447126119909152 (-0.003932562932162237) | > avg_loss_0: 2.8930417752601727 (-0.023189380135335114) | > avg_grad_norm_0: tensor(6.6523, device='cuda:0') (+tensor(0.4992, device='cuda:0')) | > avg_loss_gen: 1.664705002811593 (+0.009279430752069162) | > avg_loss_kl: 2.5548539329582534 (-0.05442412470428071) | > avg_loss_feat: 1.2804931998252873 (+0.06964947388205767) | > avg_loss_mel: 19.107457093789545 (-0.18565919365681438) | > avg_loss_duration: 1.7235060225070362 (-0.0040995050484025075) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.331015170460017 (-0.16525405561420214) | > avg_grad_norm_1: tensor(253.0182, device='cuda:0') (+tensor(41.5157, device='cuda:0'))  > EPOCH: 30/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:29:43)   --> TIME: 2025-02-25 08:30:02 -- STEP: 39/72 -- GLOBAL_STEP: 1002200 | > loss_disc: 2.9051389694213867 (2.9101743392455273) | > loss_disc_real_0: 0.20175959169864655 (0.23791853701457) | > loss_disc_real_1: 0.17041468620300293 (0.238667294383049) | > loss_disc_real_2: 0.26307928562164307 (0.2494499786542012) | > loss_disc_real_3: 0.27091455459594727 (0.24691254435441432) | > loss_disc_real_4: 0.26733770966529846 (0.2487466656244718) | > loss_disc_real_5: 0.26704129576683044 (0.24769389858612648) | > loss_0: 2.9051389694213867 (2.9101743392455273) | > grad_norm_0: tensor(5.9875, device='cuda:0') (tensor(5.8237, device='cuda:0')) | > loss_gen: 1.633084774017334 (1.6568134686885736) | > loss_kl: 2.701265335083008 (2.5768408958728495) | > loss_feat: 1.2087936401367188 (1.250619414525154) | > loss_mel: 18.595983505249023 (19.05056200271998) | > loss_duration: 1.7110475301742554 (1.7184196068690374) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.850175857543945 (26.253255795209835) | > grad_norm_1: tensor(147.7195, device='cuda:0') (tensor(230.1922, device='cuda:0')) | > current_lr_0: 0.00019925135779039958 | > current_lr_1: 0.00019925135779039958 | > step_time: 0.4368 (0.4307803740868202) | > loader_time: 0.0081 (0.007277464255308494) --> EVAL PERFORMANCE | > avg_loader_time: 0.008515730710096762 (-1.3502550796724508e-05) | > avg_step_time: 0.44476431188449056 (-0.0028736020477724278) | > avg_loss_disc: 2.903841475365867 (+0.01079970010569431) | > avg_loss_disc_real_0: 0.2404928488630644 (+0.0005734120036514634) | > avg_loss_disc_real_1: 0.24171439694686675 (+0.005365371494226057) | > avg_loss_disc_real_2: 0.2464573924810114 (+0.002121981180889493) | > avg_loss_disc_real_3: 0.24473163464539488 (+0.0021341732690032256) | > avg_loss_disc_real_4: 0.2472165606811013 (+0.0032294894188222567) | > avg_loss_disc_real_5: 0.24697495514238385 (+0.0025036939432923344) | > avg_loss_0: 2.903841475365867 (+0.01079970010569431) | > avg_grad_norm_0: tensor(6.3135, device='cuda:0') (tensor(-0.3388, device='cuda:0')) | > avg_loss_gen: 1.6598677467292464 (-0.0048372560823466415) | > avg_loss_kl: 2.5550847456488803 (+0.00023081269062696919) | > avg_loss_feat: 1.2457197971746956 (-0.0347734026505917) | > avg_loss_mel: 18.97614307134923 (-0.1313140224403142) | > avg_loss_duration: 1.727341161647313 (+0.0038351391402768886) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.16415668541277 (-0.16685848504724632) | > avg_grad_norm_1: tensor(245.1848, device='cuda:0') (tensor(-7.8333, device='cuda:0'))  > EPOCH: 31/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:30:19)   --> TIME: 2025-02-25 08:30:29 -- STEP: 17/72 -- GLOBAL_STEP: 1002250 | > loss_disc: 2.864018440246582 (2.9019783524905933) | > loss_disc_real_0: 0.29712992906570435 (0.24934719327618093) | > loss_disc_real_1: 0.2080416977405548 (0.23401911206105175) | > loss_disc_real_2: 0.2316843420267105 (0.24270843407687018) | > loss_disc_real_3: 0.23602695763111115 (0.2472345522221397) | > loss_disc_real_4: 0.24820038676261902 (0.24654424891752355) | > loss_disc_real_5: 0.2626597583293915 (0.24605477382155025) | > loss_0: 2.864018440246582 (2.9019783524905933) | > grad_norm_0: tensor(7.9901, device='cuda:0') (tensor(6.2393, device='cuda:0')) | > loss_gen: 1.6216422319412231 (1.6755483781590181) | > loss_kl: 2.676908254623413 (2.5828358005074894) | > loss_feat: 1.4776495695114136 (1.30576790080351) | > loss_mel: 20.867889404296875 (19.58941526973949) | > loss_duration: 1.733891248703003 (1.7103996767717249) | > amp_scaler: 128.0 (128.0) | > loss_1: 28.377981185913086 (26.863967110128964) | > grad_norm_1: tensor(443.6864, device='cuda:0') (tensor(221.4569, device='cuda:0')) | > current_lr_0: 0.00019922645137067577 | > current_lr_1: 0.00019922645137067577 | > step_time: 0.425 (0.4569481681374943) | > loader_time: 0.006 (0.006879834567799288)  --> TIME: 2025-02-25 08:30:52 -- STEP: 67/72 -- GLOBAL_STEP: 1002300 | > loss_disc: 2.9482715129852295 (2.903370501390144) | > loss_disc_real_0: 0.2792314291000366 (0.24132370125891558) | > loss_disc_real_1: 0.24953202903270721 (0.23955076198969313) | > loss_disc_real_2: 0.26623016595840454 (0.24525578653634483) | > loss_disc_real_3: 0.24527955055236816 (0.24463464564351894) | > loss_disc_real_4: 0.254688560962677 (0.24538969103969746) | > loss_disc_real_5: 0.2742651402950287 (0.24757257637692925) | > loss_0: 2.9482715129852295 (2.903370501390144) | > grad_norm_0: tensor(5.1398, device='cuda:0') (tensor(6.8617, device='cuda:0')) | > loss_gen: 1.4519479274749756 (1.6653758127297928) | > loss_kl: 2.39963960647583 (2.5770961846878278) | > loss_feat: 1.0688042640686035 (1.2812355836825586) | > loss_mel: 18.657459259033203 (19.282149272178536) | > loss_duration: 1.7560886144638062 (1.7200310052330814) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.333940505981445 (26.525887816699584) | > grad_norm_1: tensor(189.3038, device='cuda:0') (tensor(203.0663, device='cuda:0')) | > current_lr_0: 0.00019922645137067577 | > current_lr_1: 0.00019922645137067577 | > step_time: 0.4928 (0.4522637395716425) | > loader_time: 0.0117 (0.00831432840717373) --> EVAL PERFORMANCE | > avg_loader_time: 0.008490837795633668 (-2.4892914463094107e-05) | > avg_step_time: 0.45534938154086263 (+0.01058506965637207) | > avg_loss_disc: 2.906167181444839 (+0.0023257060789720896) | > avg_loss_disc_real_0: 0.24016203322041202 (-0.00033081564265238783) | > avg_loss_disc_real_1: 0.2399887868636091 (-0.0017256100832576393) | > avg_loss_disc_real_2: 0.2443715558505394 (-0.0020858366304719878) | > avg_loss_disc_real_3: 0.24354209005832672 (-0.0011895445870681587) | > avg_loss_disc_real_4: 0.24402057222077544 (-0.0031959884603258437) | > avg_loss_disc_real_5: 0.24675788845814456 (-0.0002170666842392921) | > avg_loss_0: 2.906167181444839 (+0.0023257060789720896) | > avg_grad_norm_0: tensor(7.0259, device='cuda:0') (+tensor(0.7124, device='cuda:0')) | > avg_loss_gen: 1.6610669183059477 (+0.0011991715767012412) | > avg_loss_kl: 2.5747504234313965 (+0.019665677782516156) | > avg_loss_feat: 1.271702481827266 (+0.025982684652570365) | > avg_loss_mel: 19.24200439453125 (+0.26586132318201905) | > avg_loss_duration: 1.721077364935002 (-0.006263796712311143) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.470601498241155 (+0.306444812828385) | > avg_grad_norm_1: tensor(201.8349, device='cuda:0') (tensor(-43.3500, device='cuda:0'))  > EPOCH: 32/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:30:56)   --> TIME: 2025-02-25 08:31:18 -- STEP: 45/72 -- GLOBAL_STEP: 1002350 | > loss_disc: 2.884291172027588 (2.9003218015034986) | > loss_disc_real_0: 0.2938843071460724 (0.23647078739272223) | > loss_disc_real_1: 0.2794573903083801 (0.23785293069150712) | > loss_disc_real_2: 0.2398110330104828 (0.24190349049038357) | > loss_disc_real_3: 0.23810052871704102 (0.2425958385070165) | > loss_disc_real_4: 0.2505849003791809 (0.2458759864171346) | > loss_disc_real_5: 0.24859175086021423 (0.24908506770928712) | > loss_0: 2.884291172027588 (2.9003218015034986) | > grad_norm_0: tensor(6.0207, device='cuda:0') (tensor(7.4678, device='cuda:0')) | > loss_gen: 1.6572412252426147 (1.652497861120436) | > loss_kl: 2.621734380722046 (2.615407429801093) | > loss_feat: 1.1476025581359863 (1.2692739990022448) | > loss_mel: 18.350601196289062 (19.220060687594945) | > loss_duration: 1.737242341041565 (1.7137410296334161) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.514421463012695 (26.470981004503038) | > grad_norm_1: tensor(98.5380, device='cuda:0') (tensor(250.6062, device='cuda:0')) | > current_lr_0: 0.00019920154806425444 | > current_lr_1: 0.00019920154806425444 | > step_time: 0.4684 (0.4415375656551785) | > loader_time: 0.0088 (0.007398870256212022) --> EVAL PERFORMANCE | > avg_loader_time: 0.008495404686726311 (+4.566891092643369e-06) | > avg_step_time: 0.4539651870727539 (-0.0013841944681087237) | > avg_loss_disc: 2.906656550689482 (+0.0004893692446428233) | > avg_loss_disc_real_0: 0.23717319482648877 (-0.002988838393923249) | > avg_loss_disc_real_1: 0.23805482622603297 (-0.0019339606375761431) | > avg_loss_disc_real_2: 0.243036741731872 (-0.001334814118667399) | > avg_loss_disc_real_3: 0.24332003001595887 (-0.00022206004236785537) | > avg_loss_disc_real_4: 0.24677020122467633 (+0.002749629003900883) | > avg_loss_disc_real_5: 0.24882020874762198 (+0.0020623202894774206) | > avg_loss_0: 2.906656550689482 (+0.0004893692446428233) | > avg_grad_norm_0: tensor(6.9867, device='cuda:0') (tensor(-0.0392, device='cuda:0')) | > avg_loss_gen: 1.6450922422006096 (-0.015974676105338093) | > avg_loss_kl: 2.568316624198162 (-0.006433799233234616) | > avg_loss_feat: 1.2207241721556221 (-0.05097830967164385) | > avg_loss_mel: 19.14258024726115 (-0.09942414727009918) | > avg_loss_duration: 1.7226826389070968 (+0.0016052739720948672) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.299395923883143 (-0.1712055743580123) | > avg_grad_norm_1: tensor(250.0830, device='cuda:0') (+tensor(48.2481, device='cuda:0'))  > EPOCH: 33/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:31:33)   --> TIME: 2025-02-25 08:31:44 -- STEP: 23/72 -- GLOBAL_STEP: 1002400 | > loss_disc: 2.908190965652466 (2.892306203427522) | > loss_disc_real_0: 0.14946052432060242 (0.24219107563080994) | > loss_disc_real_1: 0.2198607176542282 (0.24348034482935202) | > loss_disc_real_2: 0.25268834829330444 (0.24236279142939526) | > loss_disc_real_3: 0.23690499365329742 (0.2442789518314859) | > loss_disc_real_4: 0.25148075819015503 (0.2464315133250278) | > loss_disc_real_5: 0.2514364421367645 (0.24728670392347418) | > loss_0: 2.908190965652466 (2.892306203427522) | > grad_norm_0: tensor(11.6372, device='cuda:0') (tensor(8.0796, device='cuda:0')) | > loss_gen: 1.5768721103668213 (1.660927264586739) | > loss_kl: 2.687213659286499 (2.5258127813753872) | > loss_feat: 1.1658961772918701 (1.2670804106670877) | > loss_mel: 17.522146224975586 (18.779415794040847) | > loss_duration: 1.7199056148529053 (1.6979675085648247) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.672033309936523 (25.931203759234883) | > grad_norm_1: tensor(71.3032, device='cuda:0') (tensor(246.1433, device='cuda:0')) | > current_lr_0: 0.0001991766478707464 | > current_lr_1: 0.0001991766478707464 | > step_time: 0.4465 (0.4302411286727242) | > loader_time: 0.0074 (0.006673864696336829) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850578764794578 (+1.0382961219468356e-05) | > avg_step_time: 0.44912782521315026 (-0.0048373618596036505) | > avg_loss_disc: 2.894388229074612 (-0.012268321614870015) | > avg_loss_disc_real_0: 0.240359602259918 (+0.003186407433429239) | > avg_loss_disc_real_1: 0.24551359196783792 (+0.007458765741804957) | > avg_loss_disc_real_2: 0.244194003687778 (+0.001157261955905986) | > avg_loss_disc_real_3: 0.2437408602153751 (+0.00042083019941624444) | > avg_loss_disc_real_4: 0.24425759575736355 (-0.002512605467312773) | > avg_loss_disc_real_5: 0.24472065185996847 (-0.0040995568876535105) | > avg_loss_0: 2.894388229074612 (-0.012268321614870015) | > avg_grad_norm_0: tensor(7.2764, device='cuda:0') (+tensor(0.2897, device='cuda:0')) | > avg_loss_gen: 1.665026327254067 (+0.019934085053457506) | > avg_loss_kl: 2.6065539608539 (+0.03823733665573803) | > avg_loss_feat: 1.2614871187948848 (+0.04076294663926272) | > avg_loss_mel: 18.950428438858246 (-0.19215180840290458) | > avg_loss_duration: 1.716140500256713 (-0.0065421386503838885) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.199636190709935 (-0.09975973317320808) | > avg_grad_norm_1: tensor(256.9482, device='cuda:0') (+tensor(6.8652, device='cuda:0'))  > EPOCH: 34/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:32:09)   --> TIME: 2025-02-25 08:32:11 -- STEP: 1/72 -- GLOBAL_STEP: 1002450 | > loss_disc: 2.8423664569854736 (2.8423664569854736) | > loss_disc_real_0: 0.2607113718986511 (0.2607113718986511) | > loss_disc_real_1: 0.24790246784687042 (0.24790246784687042) | > loss_disc_real_2: 0.22919221222400665 (0.22919221222400665) | > loss_disc_real_3: 0.2247556746006012 (0.2247556746006012) | > loss_disc_real_4: 0.23735632002353668 (0.23735632002353668) | > loss_disc_real_5: 0.21411725878715515 (0.21411725878715515) | > loss_0: 2.8423664569854736 (2.8423664569854736) | > grad_norm_0: tensor(4.0799, device='cuda:0') (tensor(4.0799, device='cuda:0')) | > loss_gen: 1.6912994384765625 (1.6912994384765625) | > loss_kl: 2.400812864303589 (2.400812864303589) | > loss_feat: 1.2697324752807617 (1.2697324752807617) | > loss_mel: 18.468006134033203 (18.468006134033203) | > loss_duration: 1.746065378189087 (1.746065378189087) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.575916290283203 (25.575916290283203) | > grad_norm_1: tensor(128.0807, device='cuda:0') (tensor(128.0807, device='cuda:0')) | > current_lr_0: 0.00019915175078976256 | > current_lr_1: 0.00019915175078976256 | > step_time: 0.4235 (0.423541784286499) | > loader_time: 0.0084 (0.008402109146118164)  --> TIME: 2025-02-25 08:32:34 -- STEP: 51/72 -- GLOBAL_STEP: 1002500 | > loss_disc: 2.8109755516052246 (2.896084808835796) | > loss_disc_real_0: 0.26899591088294983 (0.24024933722673678) | > loss_disc_real_1: 0.19879592955112457 (0.24062372481121735) | > loss_disc_real_2: 0.25860342383384705 (0.24542982350377476) | > loss_disc_real_3: 0.2585229277610779 (0.2449799650440029) | > loss_disc_real_4: 0.2529723346233368 (0.24723587579586925) | > loss_disc_real_5: 0.256896048784256 (0.24644788895167558) | > loss_0: 2.8109755516052246 (2.896084808835796) | > grad_norm_0: tensor(5.4665, device='cuda:0') (tensor(7.7320, device='cuda:0')) | > loss_gen: 1.5834040641784668 (1.661729003868851) | > loss_kl: 2.558084726333618 (2.5358614267087445) | > loss_feat: 1.378963828086853 (1.2566610492911994) | > loss_mel: 19.313610076904297 (18.69888025171617) | > loss_duration: 1.7452231645584106 (1.7111848405763215) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.579286575317383 (25.864316641115675) | > grad_norm_1: tensor(137.3377, device='cuda:0') (tensor(203.6909, device='cuda:0')) | > current_lr_0: 0.00019915175078976256 | > current_lr_1: 0.00019915175078976256 | > step_time: 0.4383 (0.4441534070407643) | > loader_time: 0.0088 (0.007602149364995021) --> EVAL PERFORMANCE | > avg_loader_time: 0.008481761099587022 (-2.4026548358757616e-05) | > avg_step_time: 0.4527546251323861 (+0.003626799919235868) | > avg_loss_disc: 2.8963712235571633 (+0.001982994482551348) | > avg_loss_disc_real_0: 0.23736315315038387 (-0.002996449109534144) | > avg_loss_disc_real_1: 0.23939945895067402 (-0.006114133017163903) | > avg_loss_disc_real_2: 0.2442288572939349 (+3.4853606156903805e-05) | > avg_loss_disc_real_3: 0.24536814760993902 (+0.0016272873945639144) | > avg_loss_disc_real_4: 0.24759930225325302 (+0.0033417064958894682) | > avg_loss_disc_real_5: 0.24690124854235582 (+0.002180596682387348) | > avg_loss_0: 2.8963712235571633 (+0.001982994482551348) | > avg_grad_norm_0: tensor(7.3070, device='cuda:0') (+tensor(0.0306, device='cuda:0')) | > avg_loss_gen: 1.6587880547617522 (-0.0062382724923149) | > avg_loss_kl: 2.5481925984503517 (-0.05836136240354817) | > avg_loss_feat: 1.25940705101255 (-0.002080067782334938) | > avg_loss_mel: 18.852543602527028 (-0.09788483633121814) | > avg_loss_duration: 1.7170042336826594 (+0.000863733425946478) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.0359355228048 (-0.16370066790513604) | > avg_grad_norm_1: tensor(222.5524, device='cuda:0') (tensor(-34.3958, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1002521.pth  > EPOCH: 35/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:32:48)   --> TIME: 2025-02-25 08:33:03 -- STEP: 29/72 -- GLOBAL_STEP: 1002550 | > loss_disc: 2.856727123260498 (2.8876393910112053) | > loss_disc_real_0: 0.1809854507446289 (0.23966546099761438) | > loss_disc_real_1: 0.2111508846282959 (0.23144900079431205) | > loss_disc_real_2: 0.23800809681415558 (0.2441820964731019) | > loss_disc_real_3: 0.2589728534221649 (0.24373502977963152) | > loss_disc_real_4: 0.24753575026988983 (0.24467133756341605) | > loss_disc_real_5: 0.22881537675857544 (0.2447178204511774) | > loss_0: 2.856727123260498 (2.8876393910112053) | > grad_norm_0: tensor(6.6879, device='cuda:0') (tensor(8.2423, device='cuda:0')) | > loss_gen: 1.659881591796875 (1.6745629433927864) | > loss_kl: 2.5777201652526855 (2.5365766574596535) | > loss_feat: 1.4067702293395996 (1.3852744513544544) | > loss_mel: 18.895090103149414 (19.264241909158645) | > loss_duration: 1.6840174198150635 (1.6979748545021847) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.223480224609375 (26.558631041954303) | > grad_norm_1: tensor(319.6001, device='cuda:0') (tensor(286.3028, device='cuda:0')) | > current_lr_0: 0.00019912685682091382 | > current_lr_1: 0.00019912685682091382 | > step_time: 0.4345 (0.4327324505510001) | > loader_time: 0.0066 (0.006853580474853516) --> EVAL PERFORMANCE | > avg_loader_time: 0.008461932061423717 (-1.9829038163304835e-05) | > avg_step_time: 0.45157818727090326 (-0.0011764378614828597) | > avg_loss_disc: 2.900446885068652 (+0.004075661511488526) | > avg_loss_disc_real_0: 0.23851767958889544 (+0.0011545264385115772) | > avg_loss_disc_real_1: 0.23914111185241754 (-0.0002583470982564784) | > avg_loss_disc_real_2: 0.24482285241845628 (+0.0005939951245213793) | > avg_loss_disc_real_3: 0.24687115701151566 (+0.0015030094015766327) | > avg_loss_disc_real_4: 0.2470531551770761 (-0.0005461470761769227) | > avg_loss_disc_real_5: 0.24610370565468156 (-0.0007975428876742519) | > avg_loss_0: 2.900446885068652 (+0.004075661511488526) | > avg_grad_norm_0: tensor(7.7387, device='cuda:0') (+tensor(0.4317, device='cuda:0')) | > avg_loss_gen: 1.6797300896174474 (+0.020942034855695235) | > avg_loss_kl: 2.5411424334620087 (-0.0070501649883429884) | > avg_loss_feat: 1.3374154785989036 (+0.07800842758635373) | > avg_loss_mel: 18.930296803863957 (+0.07775320133692887) | > avg_loss_duration: 1.714193342437207 (-0.0028108912454525292) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.202778265509803 (+0.1668427427050041) | > avg_grad_norm_1: tensor(269.7105, device='cuda:0') (+tensor(47.1581, device='cuda:0'))  > EPOCH: 36/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:33:25)   --> TIME: 2025-02-25 08:33:29 -- STEP: 7/72 -- GLOBAL_STEP: 1002600 | > loss_disc: 2.8667664527893066 (2.88627028465271) | > loss_disc_real_0: 0.2500670552253723 (0.2367938416344779) | > loss_disc_real_1: 0.2654327154159546 (0.24287656588213785) | > loss_disc_real_2: 0.29045844078063965 (0.24687757236616953) | > loss_disc_real_3: 0.26566082239151 (0.24493798400674546) | > loss_disc_real_4: 0.27817636728286743 (0.2484007946082524) | > loss_disc_real_5: 0.274057000875473 (0.2513157916920526) | > loss_0: 2.8667664527893066 (2.88627028465271) | > grad_norm_0: tensor(3.3105, device='cuda:0') (tensor(5.4473, device='cuda:0')) | > loss_gen: 1.6583184003829956 (1.6439352887017387) | > loss_kl: 2.5192031860351562 (2.6230450017111644) | > loss_feat: 1.222636342048645 (1.2221744060516357) | > loss_mel: 18.13262176513672 (18.596065521240234) | > loss_duration: 1.7022522687911987 (1.7110803808484758) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.235031127929688 (25.796300615583146) | > grad_norm_1: tensor(231.2368, device='cuda:0') (tensor(160.4467, device='cuda:0')) | > current_lr_0: 0.0001991019659638112 | > current_lr_1: 0.0001991019659638112 | > step_time: 0.4312 (0.426018306187221) | > loader_time: 0.0062 (0.006353071757725307)  --> TIME: 2025-02-25 08:33:52 -- STEP: 57/72 -- GLOBAL_STEP: 1002650 | > loss_disc: 2.8935060501098633 (2.9016979117142525) | > loss_disc_real_0: 0.20483753085136414 (0.23610955084625043) | > loss_disc_real_1: 0.25361406803131104 (0.23755801691297898) | > loss_disc_real_2: 0.22584111988544464 (0.2445740033137171) | > loss_disc_real_3: 0.222314715385437 (0.24277185452611824) | > loss_disc_real_4: 0.22632993757724762 (0.24565634994130386) | > loss_disc_real_5: 0.2158605009317398 (0.2459621220304255) | > loss_0: 2.8935060501098633 (2.9016979117142525) | > grad_norm_0: tensor(4.2383, device='cuda:0') (tensor(6.5752, device='cuda:0')) | > loss_gen: 1.759027123451233 (1.643209783654464) | > loss_kl: 2.350792646408081 (2.591226862187971) | > loss_feat: 1.2179936170578003 (1.2464403991113628) | > loss_mel: 18.89056968688965 (19.03568903605144) | > loss_duration: 1.7459139823913574 (1.71345009092699) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.964298248291016 (26.230015905279863) | > grad_norm_1: tensor(192.7784, device='cuda:0') (tensor(161.7884, device='cuda:0')) | > current_lr_0: 0.0001991019659638112 | > current_lr_1: 0.0001991019659638112 | > step_time: 0.4649 (0.438385055776228) | > loader_time: 0.0099 (0.00777938073141533) --> EVAL PERFORMANCE | > avg_loader_time: 0.008423200795348262 (-3.8731266075455253e-05) | > avg_step_time: 0.44654263576991127 (-0.005035551500991997) | > avg_loss_disc: 2.8969989326638235 (-0.003447952404828314) | > avg_loss_disc_real_0: 0.23634897046525713 (-0.0021687091236383127) | > avg_loss_disc_real_1: 0.2396126647650356 (+0.00047155291261805066) | > avg_loss_disc_real_2: 0.24450705609690976 (-0.0003157963215465187) | > avg_loss_disc_real_3: 0.24280718031903387 (-0.004063976692481791) | > avg_loss_disc_real_4: 0.24537390953218433 (-0.0016792456448917747) | > avg_loss_disc_real_5: 0.24628156360606074 (+0.00017785795137917337) | > avg_loss_0: 2.8969989326638235 (-0.003447952404828314) | > avg_grad_norm_0: tensor(6.4573, device='cuda:0') (tensor(-1.2814, device='cuda:0')) | > avg_loss_gen: 1.6458558734034148 (-0.03387421621403264) | > avg_loss_kl: 2.5713299193852386 (+0.03018748592322984) | > avg_loss_feat: 1.2480308245605147 (-0.08938465403838891) | > avg_loss_mel: 18.981151016665184 (+0.050854212801226595) | > avg_loss_duration: 1.7153057531571725 (+0.0011124107199655953) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.161673102580327 (-0.04110516292947608) | > avg_grad_norm_1: tensor(187.0147, device='cuda:0') (tensor(-82.6958, device='cuda:0'))  > EPOCH: 37/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:34:01)   --> TIME: 2025-02-25 08:34:18 -- STEP: 35/72 -- GLOBAL_STEP: 1002700 | > loss_disc: 2.951052188873291 (2.899826431274414) | > loss_disc_real_0: 0.2397899031639099 (0.23907052448817662) | > loss_disc_real_1: 0.22351621091365814 (0.23524734846183232) | > loss_disc_real_2: 0.255282461643219 (0.2478327989578247) | > loss_disc_real_3: 0.25030407309532166 (0.24767263233661652) | > loss_disc_real_4: 0.25108450651168823 (0.2486804587500436) | > loss_disc_real_5: 0.24816501140594482 (0.24912603497505187) | > loss_0: 2.951052188873291 (2.899826431274414) | > grad_norm_0: tensor(3.5902, device='cuda:0') (tensor(8.8642, device='cuda:0')) | > loss_gen: 1.6397666931152344 (1.672728327342442) | > loss_kl: 2.8145251274108887 (2.5310484068734307) | > loss_feat: 1.199871301651001 (1.311248448916844) | > loss_mel: 19.887409210205078 (19.08282721383231) | > loss_duration: 1.7528481483459473 (1.6962104865482874) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.29442024230957 (26.29406291416713) | > grad_norm_1: tensor(363.2078, device='cuda:0') (tensor(254.1047, device='cuda:0')) | > current_lr_0: 0.0001990770782180657 | > current_lr_1: 0.0001990770782180657 | > step_time: 0.4369 (0.4269232954297747) | > loader_time: 0.008 (0.006995596204485212) --> EVAL PERFORMANCE | > avg_loader_time: 0.008485747055268623 (+6.25462599203614e-05) | > avg_step_time: 0.45271511480841836 (+0.006172479038507095) | > avg_loss_disc: 2.89765229359479 (+0.0006533609309666844) | > avg_loss_disc_real_0: 0.24032508088669308 (+0.003976110421435947) | > avg_loss_disc_real_1: 0.23709680454831727 (-0.0025158602167183186) | > avg_loss_disc_real_2: 0.24631255403370925 (+0.0018054979367994883) | > avg_loss_disc_real_3: 0.2459780412240767 (+0.003170860905042844) | > avg_loss_disc_real_4: 0.2477081558654006 (+0.002334246333216272) | > avg_loss_disc_real_5: 0.24925528466701508 (+0.0029737210609543374) | > avg_loss_0: 2.89765229359479 (+0.0006533609309666844) | > avg_grad_norm_0: tensor(8.3795, device='cuda:0') (+tensor(1.9222, device='cuda:0')) | > avg_loss_gen: 1.6659990330816994 (+0.020143159678284617) | > avg_loss_kl: 2.5085790090157962 (-0.06275091036944236) | > avg_loss_feat: 1.2586709915752141 (+0.010640167014699387) | > avg_loss_mel: 18.70788552727498 (-0.2732654893902051) | > avg_loss_duration: 1.70960293185543 (-0.005702821301742489) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.850737558284276 (-0.31093554429605064) | > avg_grad_norm_1: tensor(251.6705, device='cuda:0') (+tensor(64.6557, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1002737.pth  > EPOCH: 38/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:34:40)   --> TIME: 2025-02-25 08:34:47 -- STEP: 13/72 -- GLOBAL_STEP: 1002750 | > loss_disc: 3.0055484771728516 (2.8755079966325026) | > loss_disc_real_0: 0.22288353741168976 (0.2369491045291607) | > loss_disc_real_1: 0.2985437512397766 (0.23908131970809057) | > loss_disc_real_2: 0.2452249974012375 (0.24233025770920974) | > loss_disc_real_3: 0.21887502074241638 (0.24406903982162476) | > loss_disc_real_4: 0.2258436679840088 (0.24650141940667078) | > loss_disc_real_5: 0.2527886629104614 (0.23852456762240484) | > loss_0: 3.0055484771728516 (2.8755079966325026) | > grad_norm_0: tensor(5.4951, device='cuda:0') (tensor(8.0467, device='cuda:0')) | > loss_gen: 1.61788010597229 (1.6887958783369799) | > loss_kl: 2.606147289276123 (2.524373641380897) | > loss_feat: 1.1971420049667358 (1.3356104630690355) | > loss_mel: 18.116416931152344 (18.67556674663837) | > loss_duration: 1.6831932067871094 (1.706328951395475) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.220779418945312 (25.93067580003005) | > grad_norm_1: tensor(195.8675, device='cuda:0') (tensor(252.2846, device='cuda:0')) | > current_lr_0: 0.00019905219358328844 | > current_lr_1: 0.00019905219358328844 | > step_time: 0.4646 (0.4298483408414401) | > loader_time: 0.0058 (0.006454082635732798)  --> TIME: 2025-02-25 08:35:11 -- STEP: 63/72 -- GLOBAL_STEP: 1002800 | > loss_disc: 2.9186460971832275 (2.887507052648635) | > loss_disc_real_0: 0.33474427461624146 (0.2384207873117356) | > loss_disc_real_1: 0.26759934425354004 (0.23606733008036537) | > loss_disc_real_2: 0.2595338225364685 (0.2444451820282709) | > loss_disc_real_3: 0.2544243037700653 (0.24451400647087704) | > loss_disc_real_4: 0.23641154170036316 (0.24526399539576638) | > loss_disc_real_5: 0.2637921869754791 (0.24631069198487296) | > loss_0: 2.9186460971832275 (2.887507052648635) | > grad_norm_0: tensor(8.5830, device='cuda:0') (tensor(7.2486, device='cuda:0')) | > loss_gen: 1.5917366743087769 (1.669511736385406) | > loss_kl: 2.6982381343841553 (2.5730634227631586) | > loss_feat: 1.1201635599136353 (1.299097646796514) | > loss_mel: 20.49252700805664 (18.709368569510325) | > loss_duration: 1.6882994174957275 (1.7059230236780076) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.59096336364746 (25.9569643111456) | > grad_norm_1: tensor(253.4525, device='cuda:0') (tensor(252.2742, device='cuda:0')) | > current_lr_0: 0.00019905219358328844 | > current_lr_1: 0.00019905219358328844 | > step_time: 0.4721 (0.4446299189612979) | > loader_time: 0.0114 (0.008347288010612368) --> EVAL PERFORMANCE | > avg_loader_time: 0.00869715381676043 (+0.00021140676149180616) | > avg_step_time: 0.4499743253412381 (-0.00274078946718026) | > avg_loss_disc: 2.8915209904522965 (-0.00613130314249366) | > avg_loss_disc_real_0: 0.23723011390424112 (-0.0030949669824519577) | > avg_loss_disc_real_1: 0.23688898623829158 (-0.00020781831002569806) | > avg_loss_disc_real_2: 0.24504428787130705 (-0.001268266162402193) | > avg_loss_disc_real_3: 0.24510972138861536 (-0.0008683198354613453) | > avg_loss_disc_real_4: 0.24584866325620194 (-0.0018594926091986541) | > avg_loss_disc_real_5: 0.246394748628979 (-0.002860536038036071) | > avg_loss_0: 2.8915209904522965 (-0.00613130314249366) | > avg_grad_norm_0: tensor(7.1095, device='cuda:0') (tensor(-1.2700, device='cuda:0')) | > avg_loss_gen: 1.6675650485804383 (+0.0015660154987389152) | > avg_loss_kl: 2.584043287895095 (+0.07546427887929896) | > avg_loss_feat: 1.2865189599319244 (+0.02784796835671033) | > avg_loss_mel: 18.786716380589446 (+0.07883085331446793) | > avg_loss_duration: 1.7107877764903323 (+0.0011848446349023156) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.035631421586157 (+0.1848938633018804) | > avg_grad_norm_1: tensor(255.5769, device='cuda:0') (+tensor(3.9065, device='cuda:0'))  > EPOCH: 39/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:35:16)   --> TIME: 2025-02-25 08:35:37 -- STEP: 41/72 -- GLOBAL_STEP: 1002850 | > loss_disc: 3.0008182525634766 (2.885454154596096) | > loss_disc_real_0: 0.17912451922893524 (0.23796710335626836) | > loss_disc_real_1: 0.16504396498203278 (0.23464490582303302) | > loss_disc_real_2: 0.24343591928482056 (0.2430060600362173) | > loss_disc_real_3: 0.2568128705024719 (0.24328307662068344) | > loss_disc_real_4: 0.25048819184303284 (0.245180285195025) | > loss_disc_real_5: 0.21473179757595062 (0.24634405316376104) | > loss_0: 3.0008182525634766 (2.885454154596096) | > grad_norm_0: tensor(12.6568, device='cuda:0') (tensor(8.0668, device='cuda:0')) | > loss_gen: 1.5556846857070923 (1.6768495774850614) | > loss_kl: 2.6604721546173096 (2.6116410639227885) | > loss_feat: 1.278821587562561 (1.3875658046908494) | > loss_mel: 19.452062606811523 (19.160874669144793) | > loss_duration: 1.6958014965057373 (1.6975565392796585) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.64284324645996 (26.534487840605944) | > grad_norm_1: tensor(52.8584, device='cuda:0') (tensor(214.4780, device='cuda:0')) | > current_lr_0: 0.0001990273120590905 | > current_lr_1: 0.0001990273120590905 | > step_time: 0.4469 (0.4359679919917409) | > loader_time: 0.009 (0.007419440804458246) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591799668862784 (-0.00010535414789764573) | > avg_step_time: 0.45192300433843907 (+0.00194867899720097) | > avg_loss_disc: 2.8873597937570485 (-0.0041611966952479484) | > avg_loss_disc_real_0: 0.23185359676119308 (-0.005376517143048043) | > avg_loss_disc_real_1: 0.23730705468587474 (+0.00041806844758315864) | > avg_loss_disc_real_2: 0.24346858619804113 (-0.001575701673265928) | > avg_loss_disc_real_3: 0.24255247561024948 (-0.0025572457783658853) | > avg_loss_disc_real_4: 0.24492915247527647 (-0.0009195107809254754) | > avg_loss_disc_real_5: 0.2465752172218242 (+0.00018046859284520655) | > avg_loss_0: 2.8873597937570485 (-0.0041611966952479484) | > avg_grad_norm_0: tensor(8.1866, device='cuda:0') (+tensor(1.0771, device='cuda:0')) | > avg_loss_gen: 1.666193728715601 (-0.0013713198648372948) | > avg_loss_kl: 2.5705034564918194 (-0.013539831403275748) | > avg_loss_feat: 1.3344925339792815 (+0.047973574047357026) | > avg_loss_mel: 18.96608290873782 (+0.17936652814837473) | > avg_loss_duration: 1.7086140018113902 (-0.0021737746789420864) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.245886816105372 (+0.2102553945192156) | > avg_grad_norm_1: tensor(222.7237, device='cuda:0') (tensor(-32.8533, device='cuda:0'))  > EPOCH: 40/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:35:53)   --> TIME: 2025-02-25 08:36:03 -- STEP: 19/72 -- GLOBAL_STEP: 1002900 | > loss_disc: 2.9154891967773438 (2.89368704745644) | > loss_disc_real_0: 0.28646570444107056 (0.23665664854802584) | > loss_disc_real_1: 0.21670906245708466 (0.2365753399698358) | > loss_disc_real_2: 0.25536462664604187 (0.23817101669938942) | > loss_disc_real_3: 0.23781736195087433 (0.24707183555552834) | > loss_disc_real_4: 0.24804139137268066 (0.24843331779304303) | > loss_disc_real_5: 0.26004424691200256 (0.2493541021096079) | > loss_0: 2.9154891967773438 (2.89368704745644) | > grad_norm_0: tensor(6.5301, device='cuda:0') (tensor(9.3092, device='cuda:0')) | > loss_gen: 1.6064261198043823 (1.6481602819342362) | > loss_kl: 2.652101755142212 (2.5585928841641077) | > loss_feat: 1.0612691640853882 (1.281225091532657) | > loss_mel: 18.801801681518555 (18.777525751214277) | > loss_duration: 1.6822309494018555 (1.6921884197937815) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.803829193115234 (25.957692196494655) | > grad_norm_1: tensor(213.9314, device='cuda:0') (tensor(201.4572, device='cuda:0')) | > current_lr_0: 0.00019900243364508313 | > current_lr_1: 0.00019900243364508313 | > step_time: 0.4238 (0.42608691516675445) | > loader_time: 0.0078 (0.006770071230436626)  --> TIME: 2025-02-25 08:36:27 -- STEP: 69/72 -- GLOBAL_STEP: 1002950 | > loss_disc: 2.9397640228271484 (2.8920901409093887) | > loss_disc_real_0: 0.31501972675323486 (0.23591444004273068) | > loss_disc_real_1: 0.18162667751312256 (0.23790693391060483) | > loss_disc_real_2: 0.21184244751930237 (0.24180181367673736) | > loss_disc_real_3: 0.22006931900978088 (0.24440574473229007) | > loss_disc_real_4: 0.21064414083957672 (0.24764852960040604) | > loss_disc_real_5: 0.2073180079460144 (0.24803654309632123) | > loss_0: 2.9397640228271484 (2.8920901409093887) | > grad_norm_0: tensor(12.2510, device='cuda:0') (tensor(8.2266, device='cuda:0')) | > loss_gen: 1.6142956018447876 (1.6629310483517852) | > loss_kl: 2.522202730178833 (2.5246390708978623) | > loss_feat: 1.4210301637649536 (1.3104851781458093) | > loss_mel: 18.670785903930664 (18.81560430664947) | > loss_duration: 1.7754371166229248 (1.7090267288512078) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.003751754760742 (26.022686585136082) | > grad_norm_1: tensor(72.7598, device='cuda:0') (tensor(199.9034, device='cuda:0')) | > current_lr_0: 0.00019900243364508313 | > current_lr_1: 0.00019900243364508313 | > step_time: 0.5057 (0.4564408495806266) | > loader_time: 0.0127 (0.008519887924194336) --> EVAL PERFORMANCE | > avg_loader_time: 0.008573874621324137 (-1.7925047538646863e-05) | > avg_step_time: 0.45765567833269144 (+0.0057326739942523686) | > avg_loss_disc: 2.8925361969101595 (+0.0051764031531109644) | > avg_loss_disc_real_0: 0.23482853551985514 (+0.00297493875866206) | > avg_loss_disc_real_1: 0.23980808404969498 (+0.0025010293638202397) | > avg_loss_disc_real_2: 0.24167298661990905 (-0.0017955995781320788) | > avg_loss_disc_real_3: 0.2451263775288219 (+0.00257390191857243) | > avg_loss_disc_real_4: 0.24764981005393283 (+0.0027206575786563603) | > avg_loss_disc_real_5: 0.24815177413779246 (+0.0015765569159682524) | > avg_loss_0: 2.8925361969101595 (+0.0051764031531109644) | > avg_grad_norm_0: tensor(8.2969, device='cuda:0') (+tensor(0.1103, device='cuda:0')) | > avg_loss_gen: 1.6632831096649168 (-0.0029106190506842378) | > avg_loss_kl: 2.5255965582081967 (-0.04490689828362271) | > avg_loss_feat: 1.3134511927483785 (-0.021041341230902955) | > avg_loss_mel: 18.828214161832566 (-0.13786874690525508) | > avg_loss_duration: 1.7089055541535498 (+0.0002915523421596067) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.039450846927267 (-0.2064359691781057) | > avg_grad_norm_1: tensor(208.6986, device='cuda:0') (tensor(-14.0250, device='cuda:0'))  > EPOCH: 41/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:36:30)   --> TIME: 2025-02-25 08:36:53 -- STEP: 47/72 -- GLOBAL_STEP: 1003000 | > loss_disc: 2.947558879852295 (2.8962346239292875) | > loss_disc_real_0: 0.12309382855892181 (0.23628902038995256) | > loss_disc_real_1: 0.24981635808944702 (0.2383508948569602) | > loss_disc_real_2: 0.18802505731582642 (0.24026400231300515) | > loss_disc_real_3: 0.19522930681705475 (0.24284774382063684) | > loss_disc_real_4: 0.18729637563228607 (0.2448044390754497) | > loss_disc_real_5: 0.1825147420167923 (0.24488963250150073) | > loss_0: 2.947558879852295 (2.8962346239292875) | > grad_norm_0: tensor(12.9737, device='cuda:0') (tensor(8.6104, device='cuda:0')) | > loss_gen: 1.5444605350494385 (1.6541968685515382) | > loss_kl: 2.5974507331848145 (2.477771449596324) | > loss_feat: 1.471407175064087 (1.3506589519216663) | > loss_mel: 19.19529151916504 (18.82419456319606) | > loss_duration: 1.7301366329193115 (1.6973669249960717) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.538745880126953 (26.00418882167086) | > grad_norm_1: tensor(227.3046, device='cuda:0') (tensor(223.7910, device='cuda:0')) | > current_lr_0: 0.0001989775583408775 | > current_lr_1: 0.0001989775583408775 | > step_time: 0.4502 (0.436750295314383) | > loader_time: 0.0088 (0.007418632507324219) --> EVAL PERFORMANCE | > avg_loader_time: 0.008436605963908453 (-0.0001372686574156838) | > avg_step_time: 0.448800825736892 (-0.008854852595799434) | > avg_loss_disc: 2.8936255246820584 (+0.0010893277718988514) | > avg_loss_disc_real_0: 0.2343843429231308 (-0.0004441925967243465) | > avg_loss_disc_real_1: 0.23899204945060568 (-0.0008160345990892992) | > avg_loss_disc_real_2: 0.24182242071124868 (+0.00014943409133963415) | > avg_loss_disc_real_3: 0.24301569793425815 (-0.0021106795945637635) | > avg_loss_disc_real_4: 0.24545157396457565 (-0.0021982360893571806) | > avg_loss_disc_real_5: 0.24605546555888486 (-0.0020963085789076075) | > avg_loss_0: 2.8936255246820584 (+0.0010893277718988514) | > avg_grad_norm_0: tensor(8.7471, device='cuda:0') (+tensor(0.4502, device='cuda:0')) | > avg_loss_gen: 1.657123522019722 (-0.006159587645194753) | > avg_loss_kl: 2.471911238952422 (-0.05368531925577491) | > avg_loss_feat: 1.324663667611673 (+0.011212474863294464) | > avg_loss_mel: 18.79912508037729 (-0.029089081455275334) | > avg_loss_duration: 1.7054581625360838 (-0.003447391617465989) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.958281772237427 (-0.08116907468983925) | > avg_grad_norm_1: tensor(229.2313, device='cuda:0') (+tensor(20.5327, device='cuda:0'))  > EPOCH: 42/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:37:06)   --> TIME: 2025-02-25 08:37:19 -- STEP: 25/72 -- GLOBAL_STEP: 1003050 | > loss_disc: 2.818114757537842 (2.904469861984253) | > loss_disc_real_0: 0.23891188204288483 (0.2417030704021454) | > loss_disc_real_1: 0.1539781242609024 (0.23532809734344481) | > loss_disc_real_2: 0.18976281583309174 (0.24730103552341462) | > loss_disc_real_3: 0.23741480708122253 (0.25005557358264924) | > loss_disc_real_4: 0.2372739464044571 (0.2505961400270462) | > loss_disc_real_5: 0.27071112394332886 (0.25353073596954345) | > loss_0: 2.818114757537842 (2.904469861984253) | > grad_norm_0: tensor(12.9953, device='cuda:0') (tensor(9.4887, device='cuda:0')) | > loss_gen: 1.5268961191177368 (1.667661828994751) | > loss_kl: 2.7083396911621094 (2.5137719440460207) | > loss_feat: 1.5528819561004639 (1.312841796875) | > loss_mel: 20.044565200805664 (18.794414749145506) | > loss_duration: 1.6597567796707153 (1.691067295074463) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.49243927001953 (25.97975730895996) | > grad_norm_1: tensor(575.3130, device='cuda:0') (tensor(238.0475, device='cuda:0')) | > current_lr_0: 0.00019895268614608487 | > current_lr_1: 0.00019895268614608487 | > step_time: 0.4436 (0.43138593673706055) | > loader_time: 0.007 (0.006910915374755859) --> EVAL PERFORMANCE | > avg_loader_time: 0.008584868740028059 (+0.00014826277611960578) | > avg_step_time: 0.45127621502943444 (+0.0024753892925424292) | > avg_loss_disc: 2.9084765978262452 (+0.014851073144186877) | > avg_loss_disc_real_0: 0.23877461726816607 (+0.004390274345035278) | > avg_loss_disc_real_1: 0.2395874211485957 (+0.0005953716979900103) | > avg_loss_disc_real_2: 0.24465073372276736 (+0.002828313011518674) | > avg_loss_disc_real_3: 0.2466773443238836 (+0.0036616463896254614) | > avg_loss_disc_real_4: 0.24657023078958754 (+0.0011186568250118878) | > avg_loss_disc_real_5: 0.24840165066047454 (+0.0023461851015896817) | > avg_loss_0: 2.9084765978262452 (+0.014851073144186877) | > avg_grad_norm_0: tensor(9.4743, device='cuda:0') (+tensor(0.7272, device='cuda:0')) | > avg_loss_gen: 1.6820978362795334 (+0.024974314259811337) | > avg_loss_kl: 2.548582647887754 (+0.07667140893533197) | > avg_loss_feat: 1.3363836869387558 (+0.011720019327082776) | > avg_loss_mel: 18.81923457938181 (+0.020109499004519193) | > avg_loss_duration: 1.7128654785559212 (+0.007407316019837351) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.099164103118465 (+0.14088233088103763) | > avg_grad_norm_1: tensor(247.8350, device='cuda:0') (+tensor(18.6037, device='cuda:0'))  > EPOCH: 43/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:37:43)   --> TIME: 2025-02-25 08:37:46 -- STEP: 3/72 -- GLOBAL_STEP: 1003100 | > loss_disc: 2.9403505325317383 (2.96921706199646) | > loss_disc_real_0: 0.23142710328102112 (0.21864000459512076) | > loss_disc_real_1: 0.1648915708065033 (0.1924086461464564) | > loss_disc_real_2: 0.22881264984607697 (0.24120193223158518) | > loss_disc_real_3: 0.2887676954269409 (0.26154786845048267) | > loss_disc_real_4: 0.24470873177051544 (0.24889853099981943) | > loss_disc_real_5: 0.2556806206703186 (0.26012086868286133) | > loss_0: 2.9403505325317383 (2.96921706199646) | > grad_norm_0: tensor(8.4137, device='cuda:0') (tensor(9.3301, device='cuda:0')) | > loss_gen: 1.7896199226379395 (1.6127580006917317) | > loss_kl: 2.411421298980713 (2.50573468208313) | > loss_feat: 1.0142271518707275 (1.2310385306676228) | > loss_mel: 18.20500946044922 (18.530005772908527) | > loss_duration: 1.7441813945770264 (1.731704592704773) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.164457321166992 (25.611241658528645) | > grad_norm_1: tensor(281.9260, device='cuda:0') (tensor(366.4611, device='cuda:0')) | > current_lr_0: 0.0001989278170603166 | > current_lr_1: 0.0001989278170603166 | > step_time: 0.4426 (0.5832704703013102) | > loader_time: 0.0057 (0.006773630777994792)  --> TIME: 2025-02-25 08:38:09 -- STEP: 53/72 -- GLOBAL_STEP: 1003150 | > loss_disc: 2.940631151199341 (2.893475393079362) | > loss_disc_real_0: 0.20699578523635864 (0.2280019348803556) | > loss_disc_real_1: 0.32910919189453125 (0.2376335159225284) | > loss_disc_real_2: 0.2192654013633728 (0.24386557205668036) | > loss_disc_real_3: 0.26762646436691284 (0.2444103275829891) | > loss_disc_real_4: 0.2668669521808624 (0.24631401587207363) | > loss_disc_real_5: 0.26010364294052124 (0.2473062532690336) | > loss_0: 2.940631151199341 (2.893475393079362) | > grad_norm_0: tensor(7.9519, device='cuda:0') (tensor(8.3712, device='cuda:0')) | > loss_gen: 1.6991814374923706 (1.6589474273177813) | > loss_kl: 2.634423017501831 (2.4799524658131156) | > loss_feat: 1.184514045715332 (1.3163384064188544) | > loss_mel: 18.046506881713867 (18.531624344159972) | > loss_duration: 1.7598135471343994 (1.70070865694082) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.324438095092773 (25.687571039739645) | > grad_norm_1: tensor(137.1463, device='cuda:0') (tensor(223.7863, device='cuda:0')) | > current_lr_0: 0.0001989278170603166 | > current_lr_1: 0.0001989278170603166 | > step_time: 0.4484 (0.45179431393461406) | > loader_time: 0.0094 (0.007812814892462964) --> EVAL PERFORMANCE | > avg_loader_time: 0.008599875678478835 (+1.5006938450776858e-05) | > avg_step_time: 0.4600253071583493 (+0.008749092128914837) | > avg_loss_disc: 2.8978457786667513 (-0.010630819159493932) | > avg_loss_disc_real_0: 0.2274057847303404 (-0.011368832537825668) | > avg_loss_disc_real_1: 0.23900813032203996 (-0.0005792908265557228) | > avg_loss_disc_real_2: 0.24399467924950827 (-0.0006560544732590878) | > avg_loss_disc_real_3: 0.2461491860974003 (-0.0005281582264833051) | > avg_loss_disc_real_4: 0.2471614660511554 (+0.000591235261567874) | > avg_loss_disc_real_5: 0.24841774895157612 (+1.6098291101579454e-05) | > avg_loss_0: 2.8978457786667513 (-0.010630819159493932) | > avg_grad_norm_0: tensor(8.5173, device='cuda:0') (tensor(-0.9571, device='cuda:0')) | > avg_loss_gen: 1.6583092867488591 (-0.02378854953067422) | > avg_loss_kl: 2.4748548689022876 (-0.07372777898546623) | > avg_loss_feat: 1.3061906032159296 (-0.03019308372282614) | > avg_loss_mel: 18.540560735783107 (-0.2786738435987033) | > avg_loss_duration: 1.7041007582570467 (-0.00876472029887454) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.68401607996981 (-0.4151480231486566) | > avg_grad_norm_1: tensor(255.6069, device='cuda:0') (+tensor(7.7719, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1003169.pth  > EPOCH: 44/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:38:22)   --> TIME: 2025-02-25 08:38:38 -- STEP: 31/72 -- GLOBAL_STEP: 1003200 | > loss_disc: 2.939303398132324 (2.875089914568009) | > loss_disc_real_0: 0.30073031783103943 (0.24000662613299587) | > loss_disc_real_1: 0.2653368413448334 (0.23572373678607325) | > loss_disc_real_2: 0.31895485520362854 (0.2417568737460721) | > loss_disc_real_3: 0.2817632257938385 (0.245354222193841) | > loss_disc_real_4: 0.2741537392139435 (0.24746703236333786) | > loss_disc_real_5: 0.2894754111766815 (0.24618300555213804) | > loss_0: 2.939303398132324 (2.875089914568009) | > grad_norm_0: tensor(10.1559, device='cuda:0') (tensor(7.5570, device='cuda:0')) | > loss_gen: 1.735785722732544 (1.6912942624861194) | > loss_kl: 2.784484624862671 (2.5395240091508438) | > loss_feat: 1.226628065109253 (1.3979881463512298) | > loss_mel: 18.46434783935547 (18.780331642396987) | > loss_duration: 1.7027747631072998 (1.6862470988304383) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.914020538330078 (26.0953851515247) | > grad_norm_1: tensor(86.6828, device='cuda:0') (tensor(228.9849, device='cuda:0')) | > current_lr_0: 0.00019890295108318404 | > current_lr_1: 0.00019890295108318404 | > step_time: 0.4405 (0.43141011268861834) | > loader_time: 0.0085 (0.00695816163093813) --> EVAL PERFORMANCE | > avg_loader_time: 0.008512527170315596 (-8.73485081632399e-05) | > avg_step_time: 0.44880166859693926 (-0.01122363856141001) | > avg_loss_disc: 2.8766211086595557 (-0.021224670007195634) | > avg_loss_disc_real_0: 0.2340098608967284 (+0.006604076166388001) | > avg_loss_disc_real_1: 0.23750455635534207 (-0.0015035739666978964) | > avg_loss_disc_real_2: 0.24252837854371945 (-0.00146630070578882) | > avg_loss_disc_real_3: 0.24340595745704544 (-0.0027432286403548667) | > avg_loss_disc_real_4: 0.24628941064149562 (-0.0008720554096597888) | > avg_loss_disc_real_5: 0.24574191847317656 (-0.00267583047839956) | > avg_loss_0: 2.8766211086595557 (-0.021224670007195634) | > avg_grad_norm_0: tensor(8.1585, device='cuda:0') (tensor(-0.3587, device='cuda:0')) | > avg_loss_gen: 1.6774607124462935 (+0.019151425697434332) | > avg_loss_kl: 2.5258905686123274 (+0.05103569971003985) | > avg_loss_feat: 1.3680544386447313 (+0.061863835428801695) | > avg_loss_mel: 18.58660174087739 (+0.04604100509428477) | > avg_loss_duration: 1.7027565811721372 (-0.0013441770849094148) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.860763979629734 (+0.17674789965992588) | > avg_grad_norm_1: tensor(268.7787, device='cuda:0') (+tensor(13.1718, device='cuda:0'))  > EPOCH: 45/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:38:59)   --> TIME: 2025-02-25 08:39:05 -- STEP: 9/72 -- GLOBAL_STEP: 1003250 | > loss_disc: 2.646784543991089 (2.8273055023617215) | > loss_disc_real_0: 0.1738201081752777 (0.2386423581176334) | > loss_disc_real_1: 0.16098329424858093 (0.2287102590004603) | > loss_disc_real_2: 0.1747227907180786 (0.24048001401954228) | > loss_disc_real_3: 0.20019173622131348 (0.23843086262543997) | > loss_disc_real_4: 0.20826944708824158 (0.23992736637592316) | > loss_disc_real_5: 0.21118971705436707 (0.23988738656044006) | > loss_0: 2.646784543991089 (2.8273055023617215) | > grad_norm_0: tensor(12.7252, device='cuda:0') (tensor(9.7021, device='cuda:0')) | > loss_gen: 1.6134250164031982 (1.6711890035205417) | > loss_kl: 2.6135306358337402 (2.445402834150526) | > loss_feat: 1.8814626932144165 (1.5001313553916082) | > loss_mel: 19.817413330078125 (18.74275694953071) | > loss_duration: 1.693155288696289 (1.7004101276397705) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.618986129760742 (26.059890535142685) | > grad_norm_1: tensor(56.3285, device='cuda:0') (tensor(244.2708, device='cuda:0')) | > current_lr_0: 0.00019887808821429862 | > current_lr_1: 0.00019887808821429862 | > step_time: 0.4187 (0.43324096997578937) | > loader_time: 0.0056 (0.0067484113905164934)  --> TIME: 2025-02-25 08:39:27 -- STEP: 59/72 -- GLOBAL_STEP: 1003300 | > loss_disc: 2.822441816329956 (2.890110419968427) | > loss_disc_real_0: 0.21276003122329712 (0.23551169770248867) | > loss_disc_real_1: 0.22396120429039001 (0.24144939320572353) | > loss_disc_real_2: 0.22922886908054352 (0.24237462751946207) | > loss_disc_real_3: 0.2114994078874588 (0.24607559280880428) | > loss_disc_real_4: 0.2254524976015091 (0.24736884135310933) | > loss_disc_real_5: 0.23838333785533905 (0.2485586046162298) | > loss_0: 2.822441816329956 (2.890110419968427) | > grad_norm_0: tensor(3.1128, device='cuda:0') (tensor(8.1617, device='cuda:0')) | > loss_gen: 1.663820743560791 (1.6878795462139582) | > loss_kl: 2.4285333156585693 (2.524227510064335) | > loss_feat: 1.3081238269805908 (1.3851286799220714) | > loss_mel: 17.74974822998047 (18.68600185846878) | > loss_duration: 1.7167314291000366 (1.700428063586607) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.86695671081543 (25.98366566027625) | > grad_norm_1: tensor(65.6288, device='cuda:0') (tensor(234.8575, device='cuda:0')) | > current_lr_0: 0.00019887808821429862 | > current_lr_1: 0.00019887808821429862 | > step_time: 0.4677 (0.44011697930804755) | > loader_time: 0.0096 (0.00790944745985128) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847877583033602 (-3.375133997957601e-05) | > avg_step_time: 0.44790293129397113 (-0.0008987373029681334) | > avg_loss_disc: 2.8830115459334684 (+0.006390437273912752) | > avg_loss_disc_real_0: 0.23274323344230652 (-0.0012666274544218814) | > avg_loss_disc_real_1: 0.24128652626359967 (+0.003781969908257604) | > avg_loss_disc_real_2: 0.24059803909818891 (-0.0019303394455305323) | > avg_loss_disc_real_3: 0.24422513221351194 (+0.0008191747564665064) | > avg_loss_disc_real_4: 0.24608030075758275 (-0.00020910988391287244) | > avg_loss_disc_real_5: 0.24622799456119537 (+0.0004860760880188164) | > avg_loss_0: 2.8830115459334684 (+0.006390437273912752) | > avg_grad_norm_0: tensor(8.0972, device='cuda:0') (tensor(-0.0613, device='cuda:0')) | > avg_loss_gen: 1.6842956610128914 (+0.006834948566597943) | > avg_loss_kl: 2.550172234924746 (+0.024281666312418793) | > avg_loss_feat: 1.3746284347184947 (+0.006573996073763411) | > avg_loss_mel: 18.744575204983562 (+0.15797346410617052) | > avg_loss_duration: 1.7050039969699486 (+0.002247415797811403) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.058675524214625 (+0.19791154458489046) | > avg_grad_norm_1: tensor(240.0144, device='cuda:0') (tensor(-28.7643, device='cuda:0'))  > EPOCH: 46/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:39:35)   --> TIME: 2025-02-25 08:39:54 -- STEP: 37/72 -- GLOBAL_STEP: 1003350 | > loss_disc: 2.8979668617248535 (2.8636864455970557) | > loss_disc_real_0: 0.2839125990867615 (0.2249577625377758) | > loss_disc_real_1: 0.26902487874031067 (0.24193409488007828) | > loss_disc_real_2: 0.24222590029239655 (0.24017002735589002) | > loss_disc_real_3: 0.265756756067276 (0.24762066755745862) | > loss_disc_real_4: 0.27062422037124634 (0.25031611766364126) | > loss_disc_real_5: 0.2820366621017456 (0.25236445786179723) | > loss_0: 2.8979668617248535 (2.8636864455970557) | > grad_norm_0: tensor(10.1637, device='cuda:0') (tensor(9.4545, device='cuda:0')) | > loss_gen: 1.6816964149475098 (1.714730591387362) | > loss_kl: 2.445549249649048 (2.649545328037159) | > loss_feat: 1.3527675867080688 (1.5032858848571777) | > loss_mel: 17.754079818725586 (18.91060478622849) | > loss_duration: 1.707586407661438 (1.6976449006312602) | > amp_scaler: 128.0 (128.0) | > loss_1: 24.941679000854492 (26.47581131393845) | > grad_norm_1: tensor(78.7176, device='cuda:0') (tensor(201.2099, device='cuda:0')) | > current_lr_0: 0.00019885322845327182 | > current_lr_1: 0.00019885322845327182 | > step_time: 0.4433 (0.4478271136412749) | > loader_time: 0.0078 (0.007058826652733055) --> EVAL PERFORMANCE | > avg_loader_time: 0.008485461624575332 (+6.685794239312537e-06) | > avg_step_time: 0.45794134072854487 (+0.010038409434573736) | > avg_loss_disc: 2.8597958625202446 (-0.023215683413223775) | > avg_loss_disc_real_0: 0.2283068943611333 (-0.004436339081173218) | > avg_loss_disc_real_1: 0.24022025709420863 (-0.0010662691693910376) | > avg_loss_disc_real_2: 0.24008131384010045 (-0.0005167252580884629) | > avg_loss_disc_real_3: 0.24536776374763167 (+0.0011426315341197257) | > avg_loss_disc_real_4: 0.24800000518140658 (+0.0019197044238238314) | > avg_loss_disc_real_5: 0.24843557997488638 (+0.00220758541369101) | > avg_loss_0: 2.8597958625202446 (-0.023215683413223775) | > avg_grad_norm_0: tensor(9.2330, device='cuda:0') (+tensor(1.1358, device='cuda:0')) | > avg_loss_gen: 1.7133887267448533 (+0.02909306573196191) | > avg_loss_kl: 2.5995849857867603 (+0.04941275086201413) | > avg_loss_feat: 1.49255612366636 (+0.11792768894786532) | > avg_loss_mel: 18.68272692720655 (-0.061848277777013294) | > avg_loss_duration: 1.709373244097535 (+0.004369247127586462) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.19763003604513 (+0.1389545118305051) | > avg_grad_norm_1: tensor(232.7051, device='cuda:0') (tensor(-7.3092, device='cuda:0'))  > EPOCH: 47/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:40:12)   --> TIME: 2025-02-25 08:40:21 -- STEP: 15/72 -- GLOBAL_STEP: 1003400 | > loss_disc: 2.8146872520446777 (2.8671706994374593) | > loss_disc_real_0: 0.29857194423675537 (0.2514100442330042) | > loss_disc_real_1: 0.21484600007534027 (0.23387620747089385) | > loss_disc_real_2: 0.25981175899505615 (0.24032157162825266) | > loss_disc_real_3: 0.2396831065416336 (0.2437069833278656) | > loss_disc_real_4: 0.253944456577301 (0.24540110528469086) | > loss_disc_real_5: 0.24574807286262512 (0.24585119783878326) | > loss_0: 2.8146872520446777 (2.8671706994374593) | > grad_norm_0: tensor(7.4521, device='cuda:0') (tensor(13.1856, device='cuda:0')) | > loss_gen: 1.6995728015899658 (1.7359413067499796) | > loss_kl: 2.767496109008789 (2.5406121095021565) | > loss_feat: 1.558987021446228 (1.5104121367136638) | > loss_mel: 20.031877517700195 (18.562950770060223) | > loss_duration: 1.6662287712097168 (1.690004579226176) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.724163055419922 (26.039921061197916) | > grad_norm_1: tensor(298.5754, device='cuda:0') (tensor(280.3304, device='cuda:0')) | > current_lr_0: 0.00019882837179971516 | > current_lr_1: 0.00019882837179971516 | > step_time: 0.4395 (0.43281795183817545) | > loader_time: 0.0073 (0.006944767634073893)  --> TIME: 2025-02-25 08:40:44 -- STEP: 65/72 -- GLOBAL_STEP: 1003450 | > loss_disc: 2.832529067993164 (2.862437835106483) | > loss_disc_real_0: 0.24491502344608307 (0.23718275061020486) | > loss_disc_real_1: 0.23842865228652954 (0.23644892470194742) | > loss_disc_real_2: 0.22397038340568542 (0.23727310552046849) | > loss_disc_real_3: 0.25183194875717163 (0.24233553340801825) | > loss_disc_real_4: 0.2477438747882843 (0.24501486397706546) | > loss_disc_real_5: 0.26136282086372375 (0.24488739187900838) | > loss_0: 2.832529067993164 (2.862437835106483) | > grad_norm_0: tensor(3.5607, device='cuda:0') (tensor(11.4007, device='cuda:0')) | > loss_gen: 1.7797825336456299 (1.700800035550044) | > loss_kl: 2.5843262672424316 (2.5420049850757307) | > loss_feat: 1.5713251829147339 (1.4675233969321617) | > loss_mel: 19.44862937927246 (18.411784685575046) | > loss_duration: 1.7542240619659424 (1.7000962587503285) | > amp_scaler: 128.0 (128.0) | > loss_1: 27.138288497924805 (25.822209314199593) | > grad_norm_1: tensor(622.8371, device='cuda:0') (tensor(242.7848, device='cuda:0')) | > current_lr_0: 0.00019882837179971516 | > current_lr_1: 0.00019882837179971516 | > step_time: 0.488 (0.4448237822606013) | > loader_time: 0.0117 (0.008327520810640775) --> EVAL PERFORMANCE | > avg_loader_time: 0.008595124096937582 (+0.00010966247236224996) | > avg_step_time: 0.4489479602222711 (-0.00899338050627374) | > avg_loss_disc: 2.860755164858321 (+0.0009593023380762844) | > avg_loss_disc_real_0: 0.23551394842879872 (+0.007207054067665419) | > avg_loss_disc_real_1: 0.23666446236237673 (-0.0035557947318319016) | > avg_loss_disc_real_2: 0.23757033952524964 (-0.0025109743148508112) | > avg_loss_disc_real_3: 0.24223548992418906 (-0.0031322738234426106) | > avg_loss_disc_real_4: 0.24481920289321685 (-0.0031808022881897324) | > avg_loss_disc_real_5: 0.24499900177331038 (-0.0034365782015760016) | > avg_loss_0: 2.860755164858321 (+0.0009593023380762844) | > avg_grad_norm_0: tensor(10.9332, device='cuda:0') (+tensor(1.7002, device='cuda:0')) | > avg_loss_gen: 1.7033580508030637 (-0.010030675941789635) | > avg_loss_kl: 2.553265074609031 (-0.04631991117772927) | > avg_loss_feat: 1.4687315443871727 (-0.023824579279187352) | > avg_loss_mel: 18.482019988583847 (-0.20070693862270161) | > avg_loss_duration: 1.703241229057312 (-0.006132015040223093) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.91061586729238 (-0.28701416875275143) | > avg_grad_norm_1: tensor(240.4052, device='cuda:0') (+tensor(7.7001, device='cuda:0'))  > EPOCH: 48/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:40:49)   --> TIME: 2025-02-25 08:41:09 -- STEP: 43/72 -- GLOBAL_STEP: 1003500 | > loss_disc: 2.843982696533203 (2.864780758702478) | > loss_disc_real_0: 0.20399914681911469 (0.23195321546044462) | > loss_disc_real_1: 0.28321602940559387 (0.23843460616677306) | > loss_disc_real_2: 0.264424204826355 (0.2361950444620709) | > loss_disc_real_3: 0.25498080253601074 (0.24618076619713805) | > loss_disc_real_4: 0.23743127286434174 (0.24892677505349) | > loss_disc_real_5: 0.22532594203948975 (0.2489300938539727) | > loss_0: 2.843982696533203 (2.864780758702478) | > grad_norm_0: tensor(11.2623, device='cuda:0') (tensor(10.0008, device='cuda:0')) | > loss_gen: 1.7409377098083496 (1.6974506599958552) | > loss_kl: 2.48175311088562 (2.4807616056397905) | > loss_feat: 1.3873295783996582 (1.4586920849112575) | > loss_mel: 18.130325317382812 (18.616135619407476) | > loss_duration: 1.6956539154052734 (1.686986659848413) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.43600082397461 (25.940026615941246) | > grad_norm_1: tensor(262.3169, device='cuda:0') (tensor(185.7880, device='cuda:0')) | > current_lr_0: 0.00019880351825324018 | > current_lr_1: 0.00019880351825324018 | > step_time: 0.443 (0.4331376829812693) | > loader_time: 0.0093 (0.00730283870253452) --> EVAL PERFORMANCE | > avg_loader_time: 0.008456777518903705 (-0.00013834657803387697) | > avg_step_time: 0.4479703533817345 (-0.0009776068405366045) | > avg_loss_disc: 2.8676659322120774 (+0.006910767353756508) | > avg_loss_disc_real_0: 0.2313360979019756 (-0.004177850526823124) | > avg_loss_disc_real_1: 0.23902884440522798 (+0.0023643820428512485) | > avg_loss_disc_real_2: 0.23929029808078014 (+0.0017199585555305041) | > avg_loss_disc_real_3: 0.24726918318741758 (+0.005033693263228528) | > avg_loss_disc_real_4: 0.24841686516580447 (+0.0035976622725876206) | > avg_loss_disc_real_5: 0.24857499565876706 (+0.0035759938854566797) | > avg_loss_0: 2.8676659322120774 (+0.006910767353756508) | > avg_grad_norm_0: tensor(11.1977, device='cuda:0') (+tensor(0.2646, device='cuda:0')) | > avg_loss_gen: 1.705435872077942 (+0.002077821274878211) | > avg_loss_kl: 2.4976443270562405 (-0.05562074755279056) | > avg_loss_feat: 1.4733394837715257 (+0.004607939384352999) | > avg_loss_mel: 18.76886794936489 (+0.28684796078104213) | > avg_loss_duration: 1.712100087756842 (+0.008858858699529959) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.157387746891505 (+0.24677187959912672) | > avg_grad_norm_1: tensor(237.7227, device='cuda:0') (tensor(-2.6825, device='cuda:0'))  > EPOCH: 49/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:41:25)   --> TIME: 2025-02-25 08:41:36 -- STEP: 21/72 -- GLOBAL_STEP: 1003550 | > loss_disc: 2.8330655097961426 (2.85355696224031) | > loss_disc_real_0: 0.26272493600845337 (0.2119575118025144) | > loss_disc_real_1: 0.21884548664093018 (0.24097807421570733) | > loss_disc_real_2: 0.23027147352695465 (0.23494964838027954) | > loss_disc_real_3: 0.23666052520275116 (0.24286356426420666) | > loss_disc_real_4: 0.23979555070400238 (0.24558264371894656) | > loss_disc_real_5: 0.24807222187519073 (0.2452851796434039) | > loss_0: 2.8330655097961426 (2.85355696224031) | > grad_norm_0: tensor(6.9243, device='cuda:0') (tensor(10.8866, device='cuda:0')) | > loss_gen: 1.7839207649230957 (1.7015479746318998) | > loss_kl: 2.5677988529205322 (2.5302657853989374) | > loss_feat: 1.7102044820785522 (1.5602056980133057) | > loss_mel: 18.555932998657227 (18.46960213070824) | > loss_duration: 1.7208198308944702 (1.6875181936082386) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.33867645263672 (25.949139640444802) | > grad_norm_1: tensor(123.2232, device='cuda:0') (tensor(266.1277, device='cuda:0')) | > current_lr_0: 0.00019877866781345852 | > current_lr_1: 0.00019877866781345852 | > step_time: 0.4336 (0.4545227686564128) | > loader_time: 0.0065 (0.006838684990292504)  --> TIME: 2025-02-25 08:42:00 -- STEP: 71/72 -- GLOBAL_STEP: 1003600 | > loss_disc: 2.89853572845459 (2.8624046621188306) | > loss_disc_real_0: 0.16972503066062927 (0.2223449048232025) | > loss_disc_real_1: 0.2392481565475464 (0.23915575885436904) | > loss_disc_real_2: 0.21741829812526703 (0.24074578935831364) | > loss_disc_real_3: 0.2162218540906906 (0.2431833649605093) | > loss_disc_real_4: 0.21728208661079407 (0.24512076713669467) | > loss_disc_real_5: 0.21963876485824585 (0.24552265313309682) | > loss_0: 2.89853572845459 (2.8624046621188306) | > grad_norm_0: tensor(10.7640, device='cuda:0') (tensor(11.8857, device='cuda:0')) | > loss_gen: 1.6798393726348877 (1.6979806641457786) | > loss_kl: 2.7096710205078125 (2.467305028942269) | > loss_feat: 1.3938382863998413 (1.5230268323925178) | > loss_mel: 19.081186294555664 (18.26540487584934) | > loss_duration: 1.7335236072540283 (1.7007559655417859) | > amp_scaler: 128.0 (128.0) | > loss_1: 26.598060607910156 (25.654473398772762) | > grad_norm_1: tensor(546.3431, device='cuda:0') (tensor(302.2272, device='cuda:0')) | > current_lr_0: 0.00019877866781345852 | > current_lr_1: 0.00019877866781345852 | > step_time: 0.4798 (0.4585583612952434) | > loader_time: 0.0084 (0.008793545440888742) --> EVAL PERFORMANCE | > avg_loader_time: 0.008793545440888742 (+0.0003367679219850369) | > avg_step_time: 0.4585583612952434 (+0.01058800791350889) | > avg_loss_disc: 2.8624046621188306 (-0.005261270093246839) | > avg_loss_disc_real_0: 0.2223449048232025 (-0.008991193078773108) | > avg_loss_disc_real_1: 0.23915575885436904 (+0.00012691444914106342) | > avg_loss_disc_real_2: 0.24074578935831364 (+0.0014554912775334994) | > avg_loss_disc_real_3: 0.2431833649605093 (-0.004085818226908289) | > avg_loss_disc_real_4: 0.24512076713669467 (-0.003296098029109795) | > avg_loss_disc_real_5: 0.24552265313309682 (-0.003052342525670243) | > avg_loss_0: 2.8624046621188306 (-0.005261270093246839) | > avg_grad_norm_0: tensor(11.8857, device='cuda:0') (+tensor(0.6880, device='cuda:0')) | > avg_loss_gen: 1.6979806641457786 (-0.007455207932163255) | > avg_loss_kl: 2.467305028942269 (-0.030339298113971402) | > avg_loss_feat: 1.5230268323925178 (+0.04968734862099211) | > avg_loss_mel: 18.26540487584934 (-0.5034630735155474) | > avg_loss_duration: 1.7007559655417859 (-0.011344122215056096) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 25.654473398772762 (-0.5029143481187432) | > avg_grad_norm_1: tensor(302.2272, device='cuda:0') (+tensor(64.5045, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1003601.pth  > EPOCH: 50/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:42:04)   --> TIME: 2025-02-25 08:42:28 -- STEP: 49/72 -- GLOBAL_STEP: 1003650 | > loss_disc: 2.8710122108459473 (2.862981450800993) | > loss_disc_real_0: 0.16540732979774475 (0.22800267989538153) | > loss_disc_real_1: 0.22782503068447113 (0.2380243822628138) | > loss_disc_real_2: 0.25439703464508057 (0.2379571734642496) | > loss_disc_real_3: 0.24070747196674347 (0.24563393696230285) | > loss_disc_real_4: 0.24992845952510834 (0.2474924514488298) | > loss_disc_real_5: 0.25572362542152405 (0.2479044685558397) | > loss_0: 2.8710122108459473 (2.862981450800993) | > grad_norm_0: tensor(11.7441, device='cuda:0') (tensor(9.4183, device='cuda:0')) | > loss_gen: 1.7307097911834717 (1.7087044204984392) | > loss_kl: 2.337610960006714 (2.5943098457492115) | > loss_feat: 1.4402676820755005 (1.5127015649055948) | > loss_mel: 18.44094467163086 (18.55779285820163) | > loss_duration: 1.7683517932891846 (1.6979069052910318) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.717884063720703 (26.071415492466517) | > grad_norm_1: tensor(273.9621, device='cuda:0') (tensor(271.0672, device='cuda:0')) | > current_lr_0: 0.00019875382047998183 | > current_lr_1: 0.00019875382047998183 | > step_time: 0.4355 (0.44118268635808205) | > loader_time: 0.0092 (0.0074847425733293805) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438026401358592 (-0.0003555190395301503) | > avg_step_time: 0.4514532727254948 (-0.007105088569748619) | > avg_loss_disc: 2.8556988440768825 (-0.006705818041948142) | > avg_loss_disc_real_0: 0.22597643355248678 (+0.0036315287292842946) | > avg_loss_disc_real_1: 0.23761422407459204 (-0.001541534779777004) | > avg_loss_disc_real_2: 0.23761921680309403 (-0.0031265725552196144) | > avg_loss_disc_real_3: 0.245493739213742 (+0.0023103742532327165) | > avg_loss_disc_real_4: 0.2472763376336702 (+0.0021555704969755396) | > avg_loss_disc_real_5: 0.24717335004202076 (+0.0016506969089239432) | > avg_loss_0: 2.8556988440768825 (-0.006705818041948142) | > avg_grad_norm_0: tensor(9.8929, device='cuda:0') (tensor(-1.9928, device='cuda:0')) | > avg_loss_gen: 1.7078065032690344 (+0.009825839123255742) | > avg_loss_kl: 2.591995823551231 (+0.12469079460896193) | > avg_loss_feat: 1.5046760951968985 (-0.018350737195619304) | > avg_loss_mel: 18.552727578391487 (+0.287322702542145) | > avg_loss_duration: 1.703164278621405 (+0.0024083130796190844) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.060370190042846 (+0.4058967912700844) | > avg_grad_norm_1: tensor(268.3074, device='cuda:0') (tensor(-33.9198, device='cuda:0'))  > EPOCH: 51/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:42:41)   --> TIME: 2025-02-25 08:42:55 -- STEP: 27/72 -- GLOBAL_STEP: 1003700 | > loss_disc: 2.876434803009033 (2.883258916713573) | > loss_disc_real_0: 0.3018462657928467 (0.23213826230278722) | > loss_disc_real_1: 0.3132675886154175 (0.23285501974600334) | > loss_disc_real_2: 0.23092719912528992 (0.24235519159723212) | > loss_disc_real_3: 0.23115064203739166 (0.24308387990351077) | > loss_disc_real_4: 0.2434183806180954 (0.24708536763985953) | > loss_disc_real_5: 0.25741031765937805 (0.24963231495133156) | > loss_0: 2.876434803009033 (2.883258916713573) | > grad_norm_0: tensor(12.3626, device='cuda:0') (tensor(11.1850, device='cuda:0')) | > loss_gen: 1.8585574626922607 (1.7092338932885065) | > loss_kl: 2.629019021987915 (2.4782949641898826) | > loss_feat: 1.5578551292419434 (1.6003786855273778) | > loss_mel: 18.226396560668945 (18.653272770069265) | > loss_duration: 1.6582531929016113 (1.6752867742821023) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.93008041381836 (26.116467157999676) | > grad_norm_1: tensor(105.6782, device='cuda:0') (tensor(270.3232, device='cuda:0')) | > current_lr_0: 0.00019872897625242182 | > current_lr_1: 0.00019872897625242182 | > step_time: 0.4375 (0.42607739236619735) | > loader_time: 0.0077 (0.006783229333383066) --> EVAL PERFORMANCE | > avg_loader_time: 0.008451522236138998 (+1.3495834780406779e-05) | > avg_step_time: 0.44719600005888605 (-0.004257272666608747) | > avg_loss_disc: 2.864859900004427 (+0.00916105592754457) | > avg_loss_disc_real_0: 0.22996641685005645 (+0.003989983297569666) | > avg_loss_disc_real_1: 0.2352440013012416 (-0.0023702227733504444) | > avg_loss_disc_real_2: 0.23638162184769 (-0.0012375949554040422) | > avg_loss_disc_real_3: 0.2443475022282399 (-0.0011462369855021193) | > avg_loss_disc_real_4: 0.24808857789341832 (+0.0008122402597481038) | > avg_loss_disc_real_5: 0.2494460100858984 (+0.0022726600438776334) | > avg_loss_0: 2.864859900004427 (+0.00916105592754457) | > avg_grad_norm_0: tensor(10.4022, device='cuda:0') (+tensor(0.5092, device='cuda:0')) | > avg_loss_gen: 1.7037857888450085 (-0.004020714424025851) | > avg_loss_kl: 2.5242335359815136 (-0.06776228756971747) | > avg_loss_feat: 1.548454284667969 (+0.04377818947107048) | > avg_loss_mel: 18.822659076099665 (+0.26993149770817837) | > avg_loss_duration: 1.7087561311856123 (+0.005591852564207356) | > avg_amp_scaler: 128.0 (+0.0) | > avg_loss_1: 26.307888863791884 (+0.24751867374903824) | > avg_grad_norm_1: tensor(237.9628, device='cuda:0') (tensor(-30.3447, device='cuda:0'))  > EPOCH: 52/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:43:17)   --> TIME: 2025-02-25 08:43:21 -- STEP: 5/72 -- GLOBAL_STEP: 1003750 | > loss_disc: 2.922886371612549 (2.8656113147735596) | > loss_disc_real_0: 0.25867772102355957 (0.23703727424144744) | > loss_disc_real_1: 0.18860110640525818 (0.22338390052318574) | > loss_disc_real_2: 0.2404131144285202 (0.21481280624866486) | > loss_disc_real_3: 0.27346184849739075 (0.22117140293121337) | > loss_disc_real_4: 0.26687008142471313 (0.21719287633895873) | > loss_disc_real_5: 0.2629414498806 (0.21666019260883332) | > loss_0: 2.922886371612549 (2.8656113147735596) | > grad_norm_0: tensor(14.4928, device='cuda:0') (tensor(15.7252, device='cuda:0')) | > loss_gen: 1.8617291450500488 (1.6512876749038696) | > loss_kl: 2.474952459335327 (2.4330774307250977) | > loss_feat: 1.3631813526153564 (1.6117158889770509) | > loss_mel: 17.737857818603516 (18.999121475219727) | > loss_duration: 1.659480333328247 (1.704973578453064) | > amp_scaler: 128.0 (128.0) | > loss_1: 25.09720230102539 (26.40017623901367) | > grad_norm_1: tensor(58.7169, device='cuda:0') (tensor(138.4009, device='cuda:0')) | > current_lr_0: 0.00019870413513039026 | > current_lr_1: 0.00019870413513039026 | > step_time: 0.4391 (0.4278424739837646) | > loader_time: 0.0054 (0.0063037872314453125)  --> TIME: 2025-02-25 08:43:44 -- STEP: 55/72 -- GLOBAL_STEP: 1003800 | > loss_disc: 2.8232173919677734 (2.840140490098432) | > loss_disc_real_0: 0.21242257952690125 (0.22883179865100167) | > loss_disc_real_1: 0.23530729115009308 (0.23032048344612122) | > loss_disc_real_2: 0.2826930582523346 (0.23155391839417544) | > loss_disc_real_3: 0.27391359210014343 (0.24123704406348143) | > loss_disc_real_4: 0.29590296745300293 (0.2451066640290347) | > loss_disc_real_5: 0.31175658106803894 (0.24510091759941793) | > loss_0: 2.8232173919677734 (2.840140490098432) | > grad_norm_0: tensor(4.1290, device='cuda:0') (tensor(12.5051, device='cuda:0')) | > loss_gen: 1.8125609159469604 (1.7180314974351363) | > loss_kl: 2.4199378490448 (2.484680643948641) | > loss_feat: 1.6108816862106323 (1.5938519694588404) | > loss_mel: 17.531253814697266 (18.503572758761326) | > loss_duration: 1.7316795587539673 (1.6944588661193847) | > amp_scaler: 256.0 (165.23636363636365) | > loss_1: 25.106313705444336 (25.994596030495384) | > grad_norm_1: tensor(155.4395, device='cuda:0') (tensor(214.5426, device='cuda:0')) | > current_lr_0: 0.00019870413513039026 | > current_lr_1: 0.00019870413513039026 | > step_time: 0.4623 (0.4477624589746649) | > loader_time: 0.0094 (0.0077034343372691765) --> EVAL PERFORMANCE | > avg_loader_time: 0.008448328770382303 (-3.193465756695657e-06) | > avg_step_time: 0.45633438943137583 (+0.009138389372489786) | > avg_loss_disc: 2.8410626800966927 (-0.023797219907734313) | > avg_loss_disc_real_0: 0.22716005335391407 (-0.0028063634961423833) | > avg_loss_disc_real_1: 0.23368660727856863 (-0.0015573940226729643) | > avg_loss_disc_real_2: 0.22912801550307743 (-0.0072536063446125565) | > avg_loss_disc_real_3: 0.24107383086647785 (-0.0032736713617620428) | > avg_loss_disc_real_4: 0.2442383476546113 (-0.003850230238807012) | > avg_loss_disc_real_5: 0.24458342307889966 (-0.0048625870069987365) | > avg_loss_0: 2.8410626800966927 (-0.023797219907734313) | > avg_grad_norm_0: tensor(12.1434, device='cuda:0') (+tensor(1.7412, device='cuda:0')) | > avg_loss_gen: 1.7130757318416112 (+0.009289942996602685) | > avg_loss_kl: 2.467907771258287 (-0.0563257647232267) | > avg_loss_feat: 1.5856822477260109 (+0.03722796305804188) | > avg_loss_mel: 18.40871156773098 (-0.4139475083686861) | > avg_loss_duration: 1.697152965505358 (-0.011603165680254213) | > avg_amp_scaler: 185.69014084507043 (+57.69014084507043) | > avg_loss_1: 25.872530467073684 (-0.4353583967182004) | > avg_grad_norm_1: tensor(231.7841, device='cuda:0') (tensor(-6.1786, device='cuda:0'))  > EPOCH: 53/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:43:54)   --> TIME: 2025-02-25 08:44:11 -- STEP: 33/72 -- GLOBAL_STEP: 1003850 | > loss_disc: 2.8291704654693604 (2.8410057082320703) | > loss_disc_real_0: 0.22575980424880981 (0.23471452702175488) | > loss_disc_real_1: 0.20638981461524963 (0.24029435352845627) | > loss_disc_real_2: 0.17428001761436462 (0.23350605684699435) | > loss_disc_real_3: 0.2411458045244217 (0.2473932301456278) | > loss_disc_real_4: 0.25171637535095215 (0.2472227956309463) | > loss_disc_real_5: 0.25100642442703247 (0.24994690309871326) | > loss_0: 2.8291704654693604 (2.8410057082320703) | > grad_norm_0: tensor(3.2173, device='cuda:0') (tensor(11.2205, device='cuda:0')) | > loss_gen: 1.6728993654251099 (1.7414439952734746) | > loss_kl: 2.2892262935638428 (2.5096492911830093) | > loss_feat: 1.6173135042190552 (1.606650558384982) | > loss_mel: 17.3621768951416 (18.791937336777195) | > loss_duration: 1.6744953393936157 (1.6780779686841099) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.616111755371094 (26.32775913585316) | > grad_norm_1: tensor(226.8251, device='cuda:0') (tensor(240.6501, device='cuda:0')) | > current_lr_0: 0.00019867929711349895 | > current_lr_1: 0.00019867929711349895 | > step_time: 0.4459 (0.4323695789683949) | > loader_time: 0.0077 (0.006842779390739672) --> EVAL PERFORMANCE | > avg_loader_time: 0.008425145082070793 (-2.3183688311509756e-05) | > avg_step_time: 0.4499679551997655 (-0.006366434231610318) | > avg_loss_disc: 2.8389585152478287 (-0.0021041648488639986) | > avg_loss_disc_real_0: 0.2317033402726684 (+0.00454328691875433) | > avg_loss_disc_real_1: 0.23829698037933295 (+0.0046103731007643145) | > avg_loss_disc_real_2: 0.23518538957750293 (+0.006057374074425498) | > avg_loss_disc_real_3: 0.24537892072973116 (+0.004305089863253314) | > avg_loss_disc_real_4: 0.24740141105483954 (+0.003163063400228233) | > avg_loss_disc_real_5: 0.24704188087456663 (+0.002458457795666974) | > avg_loss_0: 2.8389585152478287 (-0.0021041648488639986) | > avg_grad_norm_0: tensor(10.7259, device='cuda:0') (tensor(-1.4175, device='cuda:0')) | > avg_loss_gen: 1.7423351146805455 (+0.029259382838934256) | > avg_loss_kl: 2.5094609831420462 (+0.04155321188375938) | > avg_loss_feat: 1.6394206654857582 (+0.0537384177597473) | > avg_loss_mel: 18.783744301594474 (+0.37503273386349534) | > avg_loss_duration: 1.6979223321860946 (+0.0007693666807364608) | > avg_amp_scaler: 256.0 (+70.30985915492957) | > avg_loss_1: 26.372883380298884 (+0.5003529132251998) | > avg_grad_norm_1: tensor(252.7056, device='cuda:0') (+tensor(20.9215, device='cuda:0'))  > EPOCH: 54/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:44:31)   --> TIME: 2025-02-25 08:44:37 -- STEP: 11/72 -- GLOBAL_STEP: 1003900 | > loss_disc: 2.8390491008758545 (2.809998555616899) | > loss_disc_real_0: 0.24820472300052643 (0.21746381575411017) | > loss_disc_real_1: 0.29022741317749023 (0.235430584712462) | > loss_disc_real_2: 0.18071788549423218 (0.22483301027254624) | > loss_disc_real_3: 0.25632840394973755 (0.24990602108565244) | > loss_disc_real_4: 0.27307066321372986 (0.25515993210402405) | > loss_disc_real_5: 0.25807955861091614 (0.2486349113962867) | > loss_0: 2.8390491008758545 (2.809998555616899) | > grad_norm_0: tensor(5.1474, device='cuda:0') (tensor(10.5850, device='cuda:0')) | > loss_gen: 1.8149034976959229 (1.7783371101726184) | > loss_kl: 2.713578224182129 (2.561538739637895) | > loss_feat: 1.685654878616333 (1.72059119831432) | > loss_mel: 17.898466110229492 (18.921592365611684) | > loss_duration: 1.6562552452087402 (1.7110092531551013) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.768857955932617 (26.693068417635832) | > grad_norm_1: tensor(211.6792, device='cuda:0') (tensor(207.2721, device='cuda:0')) | > current_lr_0: 0.00019865446220135974 | > current_lr_1: 0.00019865446220135974 | > step_time: 0.4357 (0.4292106411673806) | > loader_time: 0.0056 (0.00659357417713512)  --> TIME: 2025-02-25 08:45:00 -- STEP: 61/72 -- GLOBAL_STEP: 1003950 | > loss_disc: 2.9197168350219727 (2.8314742690227073) | > loss_disc_real_0: 0.2974034547805786 (0.22742814729448224) | > loss_disc_real_1: 0.2563638687133789 (0.2400175638374735) | > loss_disc_real_2: 0.33517763018608093 (0.23498108250195862) | > loss_disc_real_3: 0.22436362504959106 (0.24466327638899693) | > loss_disc_real_4: 0.2417599856853485 (0.24666830188915378) | > loss_disc_real_5: 0.24490393698215485 (0.24746411574668573) | > loss_0: 2.9197168350219727 (2.8314742690227073) | > grad_norm_0: tensor(14.7711, device='cuda:0') (tensor(10.2747, device='cuda:0')) | > loss_gen: 1.6931356191635132 (1.756776653352331) | > loss_kl: 2.4830381870269775 (2.5689633869733965) | > loss_feat: 1.511232852935791 (1.6898288081903927) | > loss_mel: 18.315509796142578 (18.733044421086547) | > loss_duration: 1.7186152935028076 (1.6996793160673047) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.72153091430664 (26.44829256026471) | > grad_norm_1: tensor(303.6135, device='cuda:0') (tensor(211.9670, device='cuda:0')) | > current_lr_0: 0.00019865446220135974 | > current_lr_1: 0.00019865446220135974 | > step_time: 0.4525 (0.44083140717178093) | > loader_time: 0.0107 (0.008123307931618614) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596950853374643 (+0.00017180577130384948) | > avg_step_time: 0.44750347607572316 (-0.0024644791240423514) | > avg_loss_disc: 2.8362633241734034 (-0.002695191074425285) | > avg_loss_disc_real_0: 0.22737950737207707 (-0.004323832900591323) | > avg_loss_disc_real_1: 0.2396739748582034 (+0.0013769944788704636) | > avg_loss_disc_real_2: 0.2349526472914387 (-0.00023274228606423586) | > avg_loss_disc_real_3: 0.2456502870354854 (+0.0002713663057542226) | > avg_loss_disc_real_4: 0.247494605855203 (+9.319480036346084e-05) | > avg_loss_disc_real_5: 0.24822976870436064 (+0.0011878878297940099) | > avg_loss_0: 2.8362633241734034 (-0.002695191074425285) | > avg_grad_norm_0: tensor(10.1320, device='cuda:0') (tensor(-0.5940, device='cuda:0')) | > avg_loss_gen: 1.753883338310349 (+0.011548223629803456) | > avg_loss_kl: 2.562764224871783 (+0.05330324172973677) | > avg_loss_feat: 1.6699942383967654 (+0.030573572911007263) | > avg_loss_mel: 18.761046879728074 (-0.022697421866400447) | > avg_loss_duration: 1.7026982441754408 (+0.004775911989346238) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.450386960741504 (+0.07750358044262029) | > avg_grad_norm_1: tensor(224.8676, device='cuda:0') (tensor(-27.8380, device='cuda:0'))  > EPOCH: 55/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:45:07)   --> TIME: 2025-02-25 08:45:27 -- STEP: 39/72 -- GLOBAL_STEP: 1004000 | > loss_disc: 2.8274149894714355 (2.809869063206208) | > loss_disc_real_0: 0.2420286238193512 (0.22611171962359014) | > loss_disc_real_1: 0.3058733642101288 (0.23887924009408706) | > loss_disc_real_2: 0.24179354310035706 (0.23582057081736052) | > loss_disc_real_3: 0.26911139488220215 (0.24182145870648897) | > loss_disc_real_4: 0.2875801622867584 (0.24515837201705346) | > loss_disc_real_5: 0.27110254764556885 (0.24277650584013033) | > loss_0: 2.8274149894714355 (2.809869063206208) | > grad_norm_0: tensor(11.6332, device='cuda:0') (tensor(10.0473, device='cuda:0')) | > loss_gen: 1.939686894416809 (1.7678647591517522) | > loss_kl: 2.583892345428467 (2.6027668439424954) | > loss_feat: 1.621698021888733 (1.7160679560441237) | > loss_mel: 19.389938354492188 (18.431508895678398) | > loss_duration: 1.6710264682769775 (1.6812487351588714) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.206241607666016 (26.199457266391853) | > grad_norm_1: tensor(316.0740, device='cuda:0') (tensor(288.2885, device='cuda:0')) | > current_lr_0: 0.00019862963039358455 | > current_lr_1: 0.00019862963039358455 | > step_time: 0.4425 (0.44506332201835436) | > loader_time: 0.0079 (0.007094169274354592) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438382350223168 (-0.0001585685031514749) | > avg_step_time: 0.45573922278175893 (+0.008235746706035774) | > avg_loss_disc: 2.8284649815357907 (-0.0077983426376126985) | > avg_loss_disc_real_0: 0.2260677715842153 (-0.0013117357878617841) | > avg_loss_disc_real_1: 0.2382962468644263 (-0.0013777279937771114) | > avg_loss_disc_real_2: 0.23471089169173173 (-0.00024175559970696092) | > avg_loss_disc_real_3: 0.24312621795795333 (-0.002524069077532054) | > avg_loss_disc_real_4: 0.24574244043356935 (-0.0017521654216336446) | > avg_loss_disc_real_5: 0.24631267702075796 (-0.0019170916836026841) | > avg_loss_0: 2.8284649815357907 (-0.0077983426376126985) | > avg_grad_norm_0: tensor(10.2783, device='cuda:0') (+tensor(0.1463, device='cuda:0')) | > avg_loss_gen: 1.747396811633043 (-0.0064865266773059815) | > avg_loss_kl: 2.5468571387546164 (-0.015907086117166624) | > avg_loss_feat: 1.674157758833657 (+0.004163520436891499) | > avg_loss_mel: 18.535234679638492 (-0.2258122000895817) | > avg_loss_duration: 1.6975415609252285 (-0.005156683250212346) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.201187966575084 (-0.2491989941664201) | > avg_grad_norm_1: tensor(266.9349, device='cuda:0') (+tensor(42.0674, device='cuda:0'))  > EPOCH: 56/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:45:44)   --> TIME: 2025-02-25 08:45:53 -- STEP: 17/72 -- GLOBAL_STEP: 1004050 | > loss_disc: 2.850959062576294 (2.8532026094548844) | > loss_disc_real_0: 0.26864439249038696 (0.22590986069510965) | > loss_disc_real_1: 0.27228501439094543 (0.24040743357994976) | > loss_disc_real_2: 0.2502411901950836 (0.2296527212156969) | > loss_disc_real_3: 0.2579598128795624 (0.24400335550308228) | > loss_disc_real_4: 0.23263022303581238 (0.24886796930257013) | > loss_disc_real_5: 0.282838374376297 (0.24547813131528742) | > loss_0: 2.850959062576294 (2.8532026094548844) | > grad_norm_0: tensor(9.7204, device='cuda:0') (tensor(10.7671, device='cuda:0')) | > loss_gen: 1.8409923315048218 (1.7234803508309757) | > loss_kl: 2.61967396736145 (2.5912282466888428) | > loss_feat: 1.5690757036209106 (1.5920091937570011) | > loss_mel: 19.00156593322754 (19.37078588149127) | > loss_duration: 1.6723436117172241 (1.677321567254908) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.703651428222656 (26.954825233010684) | > grad_norm_1: tensor(309.1191, device='cuda:0') (tensor(263.6670, device='cuda:0')) | > current_lr_0: 0.00019860480168978534 | > current_lr_1: 0.00019860480168978534 | > step_time: 0.429 (0.42956994561588063) | > loader_time: 0.0061 (0.006524394540225758)  --> TIME: 2025-02-25 08:46:16 -- STEP: 67/72 -- GLOBAL_STEP: 1004100 | > loss_disc: 2.903355360031128 (2.845446636427694) | > loss_disc_real_0: 0.17568282783031464 (0.22500418771558733) | > loss_disc_real_1: 0.1431373953819275 (0.23561240955075222) | > loss_disc_real_2: 0.2313353717327118 (0.23608715743271272) | > loss_disc_real_3: 0.20376087725162506 (0.242917586618395) | > loss_disc_real_4: 0.24868646264076233 (0.24822743192537508) | > loss_disc_real_5: 0.30446696281433105 (0.24890507246131327) | > loss_0: 2.903355360031128 (2.845446636427694) | > grad_norm_0: tensor(25.5205, device='cuda:0') (tensor(10.8287, device='cuda:0')) | > loss_gen: 1.5615925788879395 (1.7316452229200905) | > loss_kl: 2.4633607864379883 (2.538651679878804) | > loss_feat: 1.5203430652618408 (1.6496383218622919) | > loss_mel: 18.495412826538086 (18.740086484311238) | > loss_duration: 1.7513504028320312 (1.6945284053460876) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.79205894470215 (26.354550176592017) | > grad_norm_1: tensor(170.7461, device='cuda:0') (tensor(224.1139, device='cuda:0')) | > current_lr_0: 0.00019860480168978534 | > current_lr_1: 0.00019860480168978534 | > step_time: 0.5154 (0.44692260115893917) | > loader_time: 0.0126 (0.008289849580223883) --> EVAL PERFORMANCE | > avg_loader_time: 0.00847814116679447 (+3.9758816571302744e-05) | > avg_step_time: 0.4500385069511306 (-0.005700715830628311) | > avg_loss_disc: 2.8434033830400915 (+0.014938401504300813) | > avg_loss_disc_real_0: 0.22634720046755294 (+0.00027942888333765126) | > avg_loss_disc_real_1: 0.23605645878214232 (-0.0022397880822839777) | > avg_loss_disc_real_2: 0.23575562343630993 (+0.0010447317445781978) | > avg_loss_disc_real_3: 0.24135736659379073 (-0.0017688513641626047) | > avg_loss_disc_real_4: 0.24665843475032861 (+0.000915994316759261) | > avg_loss_disc_real_5: 0.249178581254583 (+0.002865904233825045) | > avg_loss_0: 2.8434033830400915 (+0.014938401504300813) | > avg_grad_norm_0: tensor(10.8003, device='cuda:0') (+tensor(0.5220, device='cuda:0')) | > avg_loss_gen: 1.7326455166642094 (-0.01475129496883354) | > avg_loss_kl: 2.535481123857095 (-0.01137601489752127) | > avg_loss_feat: 1.6473720745301583 (-0.02678568430349859) | > avg_loss_mel: 18.746114623378705 (+0.21087994374021335) | > avg_loss_duration: 1.696244669632173 (-0.0012968912930555465) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.35785806682748 (+0.15667010025239492) | > avg_grad_norm_1: tensor(232.9565, device='cuda:0') (tensor(-33.9785, device='cuda:0'))  > EPOCH: 57/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:46:20)   --> TIME: 2025-02-25 08:46:42 -- STEP: 45/72 -- GLOBAL_STEP: 1004150 | > loss_disc: 2.861344575881958 (2.870514620674981) | > loss_disc_real_0: 0.2781733572483063 (0.235092830657959) | > loss_disc_real_1: 0.30178365111351013 (0.23994836409886677) | > loss_disc_real_2: 0.18483597040176392 (0.23300662967893812) | > loss_disc_real_3: 0.27137359976768494 (0.24638639953401353) | > loss_disc_real_4: 0.25539466738700867 (0.2473494162162145) | > loss_disc_real_5: 0.2716522812843323 (0.24620074265533023) | > loss_0: 2.861344575881958 (2.870514620674981) | > grad_norm_0: tensor(9.9816, device='cuda:0') (tensor(10.3528, device='cuda:0')) | > loss_gen: 1.789426326751709 (1.7287935256958007) | > loss_kl: 2.747429609298706 (2.4937745306226944) | > loss_feat: 1.4477758407592773 (1.5944007529152764) | > loss_mel: 18.864137649536133 (18.586970138549802) | > loss_duration: 1.7225972414016724 (1.6838519705666437) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.571367263793945 (26.087790722317166) | > grad_norm_1: tensor(76.5541, device='cuda:0') (tensor(213.7755, device='cuda:0')) | > current_lr_0: 0.0001985799760895741 | > current_lr_1: 0.0001985799760895741 | > step_time: 0.4694 (0.43571342362297905) | > loader_time: 0.011 (0.007364866468641493) --> EVAL PERFORMANCE | > avg_loader_time: 0.008446982209111603 (-3.115895768286692e-05) | > avg_step_time: 0.45034594267186984 (+0.0003074357207392131) | > avg_loss_disc: 2.8594674627545853 (+0.01606407971449375) | > avg_loss_disc_real_0: 0.230784352606451 (+0.0044371521388980595) | > avg_loss_disc_real_1: 0.23900249600410461 (+0.0029460372219622943) | > avg_loss_disc_real_2: 0.23415280018054263 (-0.0016028232557672994) | > avg_loss_disc_real_3: 0.2430287540798456 (+0.0016713874860548594) | > avg_loss_disc_real_4: 0.24613768743797088 (-0.0005207473123577389) | > avg_loss_disc_real_5: 0.2450832875681595 (-0.004095293686423501) | > avg_loss_0: 2.8594674627545853 (+0.01606407971449375) | > avg_grad_norm_0: tensor(10.7078, device='cuda:0') (tensor(-0.0925, device='cuda:0')) | > avg_loss_gen: 1.7237159517449392 (-0.008929564919270216) | > avg_loss_kl: 2.5045993999696123 (-0.03088172388748278) | > avg_loss_feat: 1.5991627112240858 (-0.048209363306072506) | > avg_loss_mel: 18.617052669256502 (-0.12906195412220356) | > avg_loss_duration: 1.6946068296969776 (-0.0016378399351952844) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.13913737552267 (-0.2187206913048101) | > avg_grad_norm_1: tensor(215.7676, device='cuda:0') (tensor(-17.1888, device='cuda:0'))  > EPOCH: 58/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:46:57)   --> TIME: 2025-02-25 08:47:09 -- STEP: 23/72 -- GLOBAL_STEP: 1004200 | > loss_disc: 2.824972629547119 (2.850419179252956) | > loss_disc_real_0: 0.16606342792510986 (0.22450004064518472) | > loss_disc_real_1: 0.2720545828342438 (0.23280823489894037) | > loss_disc_real_2: 0.20776326954364777 (0.24807631321575335) | > loss_disc_real_3: 0.2412312775850296 (0.2443483335816342) | > loss_disc_real_4: 0.24763767421245575 (0.24648144711618838) | > loss_disc_real_5: 0.2551850378513336 (0.24983096964981244) | > loss_0: 2.824972629547119 (2.850419179252956) | > grad_norm_0: tensor(9.6150, device='cuda:0') (tensor(9.8007, device='cuda:0')) | > loss_gen: 1.7068145275115967 (1.723663568496704) | > loss_kl: 2.3551597595214844 (2.5570314137831978) | > loss_feat: 1.731397032737732 (1.6242151882337488) | > loss_mel: 18.156301498413086 (18.76614056462827) | > loss_duration: 1.6812187433242798 (1.6789785623550415) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.630891799926758 (26.350029323412024) | > grad_norm_1: tensor(245.7071, device='cuda:0') (tensor(204.7007, device='cuda:0')) | > current_lr_0: 0.0001985551535925629 | > current_lr_1: 0.0001985551535925629 | > step_time: 0.425 (0.4295343212459398) | > loader_time: 0.0064 (0.0067210715749989386) --> EVAL PERFORMANCE | > avg_loader_time: 0.008725737182187363 (+0.0002787549730757597) | > avg_step_time: 0.45750942700345754 (+0.0071634843315876995) | > avg_loss_disc: 2.8403656046155477 (-0.019101858139037642) | > avg_loss_disc_real_0: 0.22795291958560407 (-0.0028314330208469296) | > avg_loss_disc_real_1: 0.23518175600280225 (-0.003820740001302364) | > avg_loss_disc_real_2: 0.24052547883819525 (+0.0063726786576526195) | > avg_loss_disc_real_3: 0.2434733170019069 (+0.00044456292206132564) | > avg_loss_disc_real_4: 0.24530965719424502 (-0.0008280302437258524) | > avg_loss_disc_real_5: 0.2466594868562591 (+0.0015761992880996) | > avg_loss_0: 2.8403656046155477 (-0.019101858139037642) | > avg_grad_norm_0: tensor(9.4937, device='cuda:0') (tensor(-1.2141, device='cuda:0')) | > avg_loss_gen: 1.731620279835983 (+0.0079043280910438) | > avg_loss_kl: 2.517164173260541 (+0.012564773290928866) | > avg_loss_feat: 1.654944057195959 (+0.05578134597187323) | > avg_loss_mel: 18.51704707615812 (-0.10000559309838053) | > avg_loss_duration: 1.6911450221504964 (-0.003461807546481177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.11192071941537 (-0.027216656107299286) | > avg_grad_norm_1: tensor(189.1198, device='cuda:0') (tensor(-26.6478, device='cuda:0'))  > EPOCH: 59/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:47:34)   --> TIME: 2025-02-25 08:47:36 -- STEP: 1/72 -- GLOBAL_STEP: 1004250 | > loss_disc: 2.872736930847168 (2.872736930847168) | > loss_disc_real_0: 0.17937824130058289 (0.17937824130058289) | > loss_disc_real_1: 0.21341586112976074 (0.21341586112976074) | > loss_disc_real_2: 0.2454749047756195 (0.2454749047756195) | > loss_disc_real_3: 0.2670848071575165 (0.2670848071575165) | > loss_disc_real_4: 0.26246923208236694 (0.26246923208236694) | > loss_disc_real_5: 0.2555696666240692 (0.2555696666240692) | > loss_0: 2.872736930847168 (2.872736930847168) | > grad_norm_0: tensor(7.9025, device='cuda:0') (tensor(7.9025, device='cuda:0')) | > loss_gen: 1.910260796546936 (1.910260796546936) | > loss_kl: 2.208247661590576 (2.208247661590576) | > loss_feat: 1.8697916269302368 (1.8697916269302368) | > loss_mel: 18.896337509155273 (18.896337509155273) | > loss_duration: 1.697526454925537 (1.697526454925537) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.582164764404297 (26.582164764404297) | > grad_norm_1: tensor(188.6122, device='cuda:0') (tensor(188.6122, device='cuda:0')) | > current_lr_0: 0.00019853033419836382 | > current_lr_1: 0.00019853033419836382 | > step_time: 0.4408 (0.4407680034637451) | > loader_time: 0.0061 (0.006110429763793945)  --> TIME: 2025-02-25 08:47:59 -- STEP: 51/72 -- GLOBAL_STEP: 1004300 | > loss_disc: 2.856458902359009 (2.8542189083847345) | > loss_disc_real_0: 0.26557785272598267 (0.23186246407966987) | > loss_disc_real_1: 0.2748275101184845 (0.23637130184500826) | > loss_disc_real_2: 0.2313685417175293 (0.2387404918086295) | > loss_disc_real_3: 0.246766597032547 (0.24443562650213055) | > loss_disc_real_4: 0.23315663635730743 (0.2454245070032045) | > loss_disc_real_5: 0.2389172613620758 (0.2463738561845293) | > loss_0: 2.856458902359009 (2.8542189083847345) | > grad_norm_0: tensor(8.1220, device='cuda:0') (tensor(11.4338, device='cuda:0')) | > loss_gen: 1.7168796062469482 (1.7248358656378353) | > loss_kl: 2.180349349975586 (2.474487519731708) | > loss_feat: 1.799719214439392 (1.644172275767607) | > loss_mel: 17.428001403808594 (18.233774035584688) | > loss_duration: 1.7134040594100952 (1.6805101796692492) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.83835220336914 (25.757780075073242) | > grad_norm_1: tensor(76.7111, device='cuda:0') (tensor(166.6694, device='cuda:0')) | > current_lr_0: 0.00019853033419836382 | > current_lr_1: 0.00019853033419836382 | > step_time: 0.4395 (0.4406360551422718) | > loader_time: 0.0089 (0.0075727257074094285) --> EVAL PERFORMANCE | > avg_loader_time: 0.00844518567474795 (-0.0002805515074394138) | > avg_step_time: 0.45161121663912923 (-0.005898210364328305) | > avg_loss_disc: 2.853910597277359 (+0.013544992661811417) | > avg_loss_disc_real_0: 0.2334076227016852 (+0.005454703116081122) | > avg_loss_disc_real_1: 0.2358943375063614 (+0.0007125815035591565) | > avg_loss_disc_real_2: 0.23870757998714984 (-0.001817898851045413) | > avg_loss_disc_real_3: 0.2434026389054849 (-7.067809642199996e-05) | > avg_loss_disc_real_4: 0.24596219184532972 (+0.0006525346510846963) | > avg_loss_disc_real_5: 0.2466427895804526 (-1.6697275806498713e-05) | > avg_loss_0: 2.853910597277359 (+0.013544992661811417) | > avg_grad_norm_0: tensor(11.4007, device='cuda:0') (+tensor(1.9070, device='cuda:0')) | > avg_loss_gen: 1.7233196648073865 (-0.00830061502859647) | > avg_loss_kl: 2.456272461044956 (-0.060891712215585425) | > avg_loss_feat: 1.622558252912172 (-0.03238580428378701) | > avg_loss_mel: 18.288350924639634 (-0.22869615151848777) | > avg_loss_duration: 1.6869194222168185 (-0.004225599933677993) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.77742084986727 (-0.3344998695481003) | > avg_grad_norm_1: tensor(171.7975, device='cuda:0') (tensor(-17.3223, device='cuda:0'))  > EPOCH: 60/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:48:10)   --> TIME: 2025-02-25 08:48:25 -- STEP: 29/72 -- GLOBAL_STEP: 1004350 | > loss_disc: 2.777634382247925 (2.866813840537236) | > loss_disc_real_0: 0.2598886489868164 (0.23080620991772618) | > loss_disc_real_1: 0.24233366549015045 (0.2322248076570445) | > loss_disc_real_2: 0.22247719764709473 (0.2439245016410433) | > loss_disc_real_3: 0.2227313220500946 (0.24706807578432152) | > loss_disc_real_4: 0.2146698385477066 (0.2522254966456315) | > loss_disc_real_5: 0.19193801283836365 (0.25381252940358784) | > loss_0: 2.777634382247925 (2.866813840537236) | > grad_norm_0: tensor(6.4398, device='cuda:0') (tensor(10.8130, device='cuda:0')) | > loss_gen: 1.7063363790512085 (1.7300617119361614) | > loss_kl: 2.5314719676971436 (2.487925603471954) | > loss_feat: 1.8102024793624878 (1.6217390586589944) | > loss_mel: 17.7999210357666 (18.280840643521014) | > loss_duration: 1.65604567527771 (1.6649644210420806) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.503978729248047 (25.785531734598095) | > grad_norm_1: tensor(323.2068, device='cuda:0') (tensor(228.8155, device='cuda:0')) | > current_lr_0: 0.000198505517906589 | > current_lr_1: 0.000198505517906589 | > step_time: 0.4541 (0.43583541080869476) | > loader_time: 0.0078 (0.006984299626843682) --> EVAL PERFORMANCE | > avg_loader_time: 0.008614419211804027 (+0.00016923353705607797) | > avg_step_time: 0.4520303699332224 (+0.00041915329409314417) | > avg_loss_disc: 2.8619985278223603 (+0.008087930545001232) | > avg_loss_disc_real_0: 0.23158702422195757 (-0.0018205984797276253) | > avg_loss_disc_real_1: 0.23617469655795836 (+0.00028035905159695673) | > avg_loss_disc_real_2: 0.24107376874332698 (+0.00236618875617714) | > avg_loss_disc_real_3: 0.24459158114984003 (+0.0011889422443551179) | > avg_loss_disc_real_4: 0.24853664489699082 (+0.0025744530516610964) | > avg_loss_disc_real_5: 0.25022017640966765 (+0.0035773868292150457) | > avg_loss_0: 2.8619985278223603 (+0.008087930545001232) | > avg_grad_norm_0: tensor(10.9427, device='cuda:0') (tensor(-0.4580, device='cuda:0')) | > avg_loss_gen: 1.7228196657879251 (-0.0004999990194614057) | > avg_loss_kl: 2.4681603606318085 (+0.011887899586852768) | > avg_loss_feat: 1.600060115397816 (-0.02249813751435603) | > avg_loss_mel: 18.33300437389966 (+0.04465344926002501) | > avg_loss_duration: 1.6878136396408079 (+0.0008942174239894296) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.81185813688896 (+0.03443728702169224) | > avg_grad_norm_1: tensor(230.8321, device='cuda:0') (+tensor(59.0346, device='cuda:0'))  > EPOCH: 61/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:48:47)   --> TIME: 2025-02-25 08:48:52 -- STEP: 7/72 -- GLOBAL_STEP: 1004400 | > loss_disc: 2.8701884746551514 (2.8194477217538014) | > loss_disc_real_0: 0.2740136981010437 (0.23684240451880864) | > loss_disc_real_1: 0.22888076305389404 (0.2161211563008172) | > loss_disc_real_2: 0.29014045000076294 (0.23206558184964315) | > loss_disc_real_3: 0.24549447000026703 (0.2325520600591387) | > loss_disc_real_4: 0.24958039820194244 (0.23158423815454757) | > loss_disc_real_5: 0.24731671810150146 (0.23505450359412602) | > loss_0: 2.8701884746551514 (2.8194477217538014) | > grad_norm_0: tensor(7.8413, device='cuda:0') (tensor(11.0581, device='cuda:0')) | > loss_gen: 1.8828563690185547 (1.787545987537929) | > loss_kl: 2.5251543521881104 (2.489682810647147) | > loss_feat: 1.7630857229232788 (1.7972713197980608) | > loss_mel: 19.125898361206055 (18.733982631138392) | > loss_duration: 1.6957038640975952 (1.6939109563827515) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.992698669433594 (26.502393450055802) | > grad_norm_1: tensor(418.0108, device='cuda:0') (tensor(288.0775, device='cuda:0')) | > current_lr_0: 0.00019848070471685067 | > current_lr_1: 0.00019848070471685067 | > step_time: 0.4367 (0.42416347776140484) | > loader_time: 0.0054 (0.006590230124337333)  --> TIME: 2025-02-25 08:49:15 -- STEP: 57/72 -- GLOBAL_STEP: 1004450 | > loss_disc: 2.8416781425476074 (2.8624584591179563) | > loss_disc_real_0: 0.2916838526725769 (0.23484328153886294) | > loss_disc_real_1: 0.2713256776332855 (0.23829672054240578) | > loss_disc_real_2: 0.23875586688518524 (0.23929786865125624) | > loss_disc_real_3: 0.25790461897850037 (0.2404914885236506) | > loss_disc_real_4: 0.25565868616104126 (0.24566401959511272) | > loss_disc_real_5: 0.2640838325023651 (0.2460113870992995) | > loss_0: 2.8416781425476074 (2.8624584591179563) | > grad_norm_0: tensor(9.1305, device='cuda:0') (tensor(9.8427, device='cuda:0')) | > loss_gen: 1.7293109893798828 (1.7306309603808219) | > loss_kl: 2.39324951171875 (2.4945873377615944) | > loss_feat: 1.503561019897461 (1.6345755321937694) | > loss_mel: 17.338451385498047 (18.676382800988975) | > loss_duration: 1.7135088443756104 (1.687729051238612) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.678081512451172 (26.223905630279006) | > grad_norm_1: tensor(93.7430, device='cuda:0') (tensor(227.3337, device='cuda:0')) | > current_lr_0: 0.00019848070471685067 | > current_lr_1: 0.00019848070471685067 | > step_time: 0.4647 (0.4512659206725003) | > loader_time: 0.0097 (0.008141082629822848) --> EVAL PERFORMANCE | > avg_loader_time: 0.008738420378993932 (+0.0001240011671899046) | > avg_step_time: 0.4585682842093454 (+0.006537914276123047) | > avg_loss_disc: 2.8667772521435375 (+0.004778724321177208) | > avg_loss_disc_real_0: 0.2338855912987615 (+0.0022985670768039257) | > avg_loss_disc_real_1: 0.23804532998884229 (+0.0018706334308839223) | > avg_loss_disc_real_2: 0.23925969718207776 (-0.0018140715612492142) | > avg_loss_disc_real_3: 0.24024709450526976 (-0.004344486644570267) | > avg_loss_disc_real_4: 0.24570766825911025 (-0.0028289766378805647) | > avg_loss_disc_real_5: 0.24530047199255983 (-0.0049197044171078175) | > avg_loss_0: 2.8667772521435375 (+0.004778724321177208) | > avg_grad_norm_0: tensor(10.0036, device='cuda:0') (tensor(-0.9391, device='cuda:0')) | > avg_loss_gen: 1.7176188885326116 (-0.005200777255313538) | > avg_loss_kl: 2.4830545942548294 (+0.014894233623020892) | > avg_loss_feat: 1.601419430383494 (+0.001359314985678095) | > avg_loss_mel: 18.560929956570472 (+0.22792558267081375) | > avg_loss_duration: 1.691064068968867 (+0.003250429328059079) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.054086900093186 (+0.24222876320422415) | > avg_grad_norm_1: tensor(214.3399, device='cuda:0') (tensor(-16.4922, device='cuda:0'))  > EPOCH: 62/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:49:24)   --> TIME: 2025-02-25 08:49:41 -- STEP: 35/72 -- GLOBAL_STEP: 1004500 | > loss_disc: 2.9002840518951416 (2.846085330418178) | > loss_disc_real_0: 0.30090266466140747 (0.23294003201382502) | > loss_disc_real_1: 0.2763217091560364 (0.2346105511699404) | > loss_disc_real_2: 0.25452473759651184 (0.2461565222058978) | > loss_disc_real_3: 0.2362712025642395 (0.2458256244659424) | > loss_disc_real_4: 0.24372273683547974 (0.24924805802958352) | > loss_disc_real_5: 0.2305029034614563 (0.2518812733037131) | > loss_0: 2.9002840518951416 (2.846085330418178) | > grad_norm_0: tensor(8.4497, device='cuda:0') (tensor(9.5595, device='cuda:0')) | > loss_gen: 1.556093692779541 (1.7470856870923723) | > loss_kl: 2.717165946960449 (2.5234905924115862) | > loss_feat: 1.51024329662323 (1.6754490579877581) | > loss_mel: 19.380311965942383 (18.637416676112576) | > loss_duration: 1.7056243419647217 (1.6644700390951974) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.86943817138672 (26.247911998203822) | > grad_norm_1: tensor(292.1721, device='cuda:0') (tensor(209.5055, device='cuda:0')) | > current_lr_0: 0.00019845589462876104 | > current_lr_1: 0.00019845589462876104 | > step_time: 0.4381 (0.4339841978890555) | > loader_time: 0.0081 (0.007036215918404715) --> EVAL PERFORMANCE | > avg_loader_time: 0.0084755454264896 (-0.0002628749525043321) | > avg_step_time: 0.4498582054191912 (-0.00871007879015423) | > avg_loss_disc: 2.847654362799416 (-0.01912288934412132) | > avg_loss_disc_real_0: 0.23215246169080198 (-0.001733129607959516) | > avg_loss_disc_real_1: 0.23705532706119645 (-0.0009900029276458389) | > avg_loss_disc_real_2: 0.24340605295040238 (+0.004146355768324617) | > avg_loss_disc_real_3: 0.24483468519969726 (+0.00458759069442749) | > avg_loss_disc_real_4: 0.2485941060831849 (+0.0028864378240746613) | > avg_loss_disc_real_5: 0.2488128009816291 (+0.00351232898906928) | > avg_loss_0: 2.847654362799416 (-0.01912288934412132) | > avg_grad_norm_0: tensor(8.7683, device='cuda:0') (tensor(-1.2352, device='cuda:0')) | > avg_loss_gen: 1.740967122601791 (+0.02334823406917952) | > avg_loss_kl: 2.4985023720163695 (+0.015447777761540049) | > avg_loss_feat: 1.6377336592741416 (+0.036314228890647504) | > avg_loss_mel: 18.577768675038506 (+0.01683871846803342) | > avg_loss_duration: 1.6881357313881458 (-0.0029283375807211964) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.143107454541703 (+0.08902055444851698) | > avg_grad_norm_1: tensor(187.8404, device='cuda:0') (tensor(-26.4995, device='cuda:0'))  > EPOCH: 63/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:50:00)   --> TIME: 2025-02-25 08:50:08 -- STEP: 13/72 -- GLOBAL_STEP: 1004550 | > loss_disc: 2.8554892539978027 (2.8627369770636926) | > loss_disc_real_0: 0.23794470727443695 (0.22462574105996352) | > loss_disc_real_1: 0.26056423783302307 (0.2396454650622148) | > loss_disc_real_2: 0.21130014955997467 (0.23067319622406593) | > loss_disc_real_3: 0.22366821765899658 (0.23704678049454322) | > loss_disc_real_4: 0.2264612317085266 (0.23933845873062426) | > loss_disc_real_5: 0.24977856874465942 (0.24663019868043753) | > loss_0: 2.8554892539978027 (2.8627369770636926) | > grad_norm_0: tensor(3.4827, device='cuda:0') (tensor(10.2442, device='cuda:0')) | > loss_gen: 1.7472915649414062 (1.6893713015776415) | > loss_kl: 2.5043370723724365 (2.454436173805824) | > loss_feat: 1.6029244661331177 (1.6023420095443726) | > loss_mel: 18.427526473999023 (18.728913967426006) | > loss_duration: 1.639389991760254 (1.6723472430155828) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.921470642089844 (26.14741090627817) | > grad_norm_1: tensor(153.3463, device='cuda:0') (tensor(176.8872, device='cuda:0')) | > current_lr_0: 0.00019843108764193245 | > current_lr_1: 0.00019843108764193245 | > step_time: 0.4468 (0.4272147325369028) | > loader_time: 0.0057 (0.006582278471726637)  --> TIME: 2025-02-25 08:50:31 -- STEP: 63/72 -- GLOBAL_STEP: 1004600 | > loss_disc: 2.870388984680176 (2.853951930999756) | > loss_disc_real_0: 0.288534939289093 (0.23153495906837404) | > loss_disc_real_1: 0.23725174367427826 (0.23891032901075152) | > loss_disc_real_2: 0.2877478301525116 (0.23532549161759633) | > loss_disc_real_3: 0.2730226218700409 (0.24101202166269695) | > loss_disc_real_4: 0.25947096943855286 (0.24394385824127804) | > loss_disc_real_5: 0.25441083312034607 (0.24531913394965824) | > loss_0: 2.870388984680176 (2.853951930999756) | > grad_norm_0: tensor(7.0676, device='cuda:0') (tensor(8.8865, device='cuda:0')) | > loss_gen: 1.689012050628662 (1.7053220707272727) | > loss_kl: 2.3674001693725586 (2.4460487782009066) | > loss_feat: 1.5088763236999512 (1.586884337758261) | > loss_mel: 18.23388671875 (18.433041254679363) | > loss_duration: 1.6560373306274414 (1.6812881969270252) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.455211639404297 (25.852584657214937) | > grad_norm_1: tensor(287.7455, device='cuda:0') (tensor(184.3099, device='cuda:0')) | > current_lr_0: 0.00019843108764193245 | > current_lr_1: 0.00019843108764193245 | > step_time: 0.4791 (0.4429842366112603) | > loader_time: 0.0112 (0.008217663992018929) --> EVAL PERFORMANCE | > avg_loader_time: 0.008617474999226318 (+0.00014192957273671813) | > avg_step_time: 0.44994398237953726 (+8.57769603460623e-05) | > avg_loss_disc: 2.854769686578025 (+0.007115323778609017) | > avg_loss_disc_real_0: 0.22978890582289493 (-0.002363555867907041) | > avg_loss_disc_real_1: 0.23935972573891492 (+0.0023043986777184777) | > avg_loss_disc_real_2: 0.2353704334564612 (-0.008035619493941187) | > avg_loss_disc_real_3: 0.24096726049000108 (-0.003867424709696171) | > avg_loss_disc_real_4: 0.24436134540698898 (-0.004232760676195935) | > avg_loss_disc_real_5: 0.24686209626600775 (-0.0019507047156213653) | > avg_loss_0: 2.854769686578025 (+0.007115323778609017) | > avg_grad_norm_0: tensor(9.4041, device='cuda:0') (+tensor(0.6358, device='cuda:0')) | > avg_loss_gen: 1.7092274074823084 (-0.0317397151194827) | > avg_loss_kl: 2.4498759659243303 (-0.048626406092039165) | > avg_loss_feat: 1.5855896002809766 (-0.05214405899316499) | > avg_loss_mel: 18.44052086413746 (-0.13724781090104443) | > avg_loss_duration: 1.6848667171639455 (-0.0032690142242002285) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.870080625507192 (-0.27302682903451014) | > avg_grad_norm_1: tensor(192.5205, device='cuda:0') (+tensor(4.6801, device='cuda:0'))  > EPOCH: 64/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:50:37)   --> TIME: 2025-02-25 08:50:57 -- STEP: 41/72 -- GLOBAL_STEP: 1004650 | > loss_disc: 2.8572237491607666 (2.8409224545083394) | > loss_disc_real_0: 0.21074309945106506 (0.23030883491766163) | > loss_disc_real_1: 0.261011004447937 (0.23335536587529065) | > loss_disc_real_2: 0.2357894331216812 (0.23741284521614633) | > loss_disc_real_3: 0.2752818763256073 (0.24478118993887088) | > loss_disc_real_4: 0.28517016768455505 (0.24545439605305835) | > loss_disc_real_5: 0.2936679422855377 (0.24447020997361438) | > loss_0: 2.8572237491607666 (2.8409224545083394) | > grad_norm_0: tensor(8.1055, device='cuda:0') (tensor(9.5895, device='cuda:0')) | > loss_gen: 1.7107226848602295 (1.73188011529969) | > loss_kl: 2.4179322719573975 (2.4015861429819254) | > loss_feat: 1.5209451913833618 (1.691061118753945) | > loss_mel: 17.831892013549805 (18.35915156108577) | > loss_duration: 1.6808242797851562 (1.6665447572382486) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.162315368652344 (25.85022400646675) | > grad_norm_1: tensor(109.8781, device='cuda:0') (tensor(197.9867, device='cuda:0')) | > current_lr_0: 0.0001984062837559772 | > current_lr_1: 0.0001984062837559772 | > step_time: 0.4423 (0.43671309657213164) | > loader_time: 0.0094 (0.00751876249545958) --> EVAL PERFORMANCE | > avg_loader_time: 0.00865440301492181 (+3.692801569549209e-05) | > avg_step_time: 0.45526038760870274 (+0.005316405229165488) | > avg_loss_disc: 2.856462609600013 (+0.0016929230219879798) | > avg_loss_disc_real_0: 0.22909657499739822 (-0.0006923308254967175) | > avg_loss_disc_real_1: 0.23551096190029466 (-0.0038487638386202616) | > avg_loss_disc_real_2: 0.2426007457182441 (+0.007230312261782917) | > avg_loss_disc_real_3: 0.24255770822645914 (+0.001590447736458056) | > avg_loss_disc_real_4: 0.24403172794362188 (-0.00032961746336709896) | > avg_loss_disc_real_5: 0.24350593795239087 (-0.0033561583136168804) | > avg_loss_0: 2.856462609600013 (+0.0016929230219879798) | > avg_grad_norm_0: tensor(9.7988, device='cuda:0') (+tensor(0.3947, device='cuda:0')) | > avg_loss_gen: 1.7129415693417402 (+0.0037141618594318526) | > avg_loss_kl: 2.4015224315750765 (-0.048353534349253824) | > avg_loss_feat: 1.637951026500111 (+0.05236142621913431) | > avg_loss_mel: 18.231682387875836 (-0.20883847626162577) | > avg_loss_duration: 1.6810328826098375 (-0.0038338345541080177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.665130400321853 (-0.204950225185339) | > avg_grad_norm_1: tensor(229.4055, device='cuda:0') (+tensor(36.8850, device='cuda:0'))  > EPOCH: 65/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:51:14)   --> TIME: 2025-02-25 08:51:24 -- STEP: 19/72 -- GLOBAL_STEP: 1004700 | > loss_disc: 2.751725673675537 (2.852010639090287) | > loss_disc_real_0: 0.2116757333278656 (0.23506777537496468) | > loss_disc_real_1: 0.19135650992393494 (0.22957163421731247) | > loss_disc_real_2: 0.27084484696388245 (0.2385956292089663) | > loss_disc_real_3: 0.22498273849487305 (0.24168984591960907) | > loss_disc_real_4: 0.23547933995723724 (0.24471713680970042) | > loss_disc_real_5: 0.22996635735034943 (0.2479256571907746) | > loss_0: 2.751725673675537 (2.852010639090287) | > grad_norm_0: tensor(11.6683, device='cuda:0') (tensor(9.9817, device='cuda:0')) | > loss_gen: 1.7862660884857178 (1.7088341650209928) | > loss_kl: 2.6650943756103516 (2.504326293342992) | > loss_feat: 1.7522495985031128 (1.5971430414601375) | > loss_mel: 17.57408332824707 (18.433240539149235) | > loss_duration: 1.6519901752471924 (1.667219318841633) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.429685592651367 (25.910763288799085) | > grad_norm_1: tensor(152.1625, device='cuda:0') (tensor(208.7059, device='cuda:0')) | > current_lr_0: 0.00019838148297050769 | > current_lr_1: 0.00019838148297050769 | > step_time: 0.4311 (0.43252350154675934) | > loader_time: 0.0062 (0.006803675701743678)  --> TIME: 2025-02-25 08:51:48 -- STEP: 69/72 -- GLOBAL_STEP: 1004750 | > loss_disc: 2.883967161178589 (2.8524872565615005) | > loss_disc_real_0: 0.22916509211063385 (0.2332861114671265) | > loss_disc_real_1: 0.19253751635551453 (0.23628274103005728) | > loss_disc_real_2: 0.2302854210138321 (0.241100642344226) | > loss_disc_real_3: 0.2712898254394531 (0.24406951449919437) | > loss_disc_real_4: 0.27113524079322815 (0.2469087839126587) | > loss_disc_real_5: 0.2826646864414215 (0.24946702152922534) | > loss_0: 2.883967161178589 (2.8524872565615005) | > grad_norm_0: tensor(5.3920, device='cuda:0') (tensor(9.2762, device='cuda:0')) | > loss_gen: 1.7774717807769775 (1.7270621475966081) | > loss_kl: 2.485835075378418 (2.4471186002095546) | > loss_feat: 1.589167833328247 (1.61906297483306) | > loss_mel: 18.964723587036133 (18.100604416667547) | > loss_duration: 1.7587512731552124 (1.6809539138406948) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.57594871520996 (25.5748021222543) | > grad_norm_1: tensor(467.2714, device='cuda:0') (tensor(213.3185, device='cuda:0')) | > current_lr_0: 0.00019838148297050769 | > current_lr_1: 0.00019838148297050769 | > step_time: 0.5011 (0.449846561404242) | > loader_time: 0.0136 (0.008757017660832063) --> EVAL PERFORMANCE | > avg_loader_time: 0.008819271141374619 (+0.0001648681264528086) | > avg_step_time: 0.45141903447433257 (-0.0038413531343701734) | > avg_loss_disc: 2.8518801037694366 (-0.004582505830576622) | > avg_loss_disc_real_0: 0.23331576103055982 (+0.004219186033161598) | > avg_loss_disc_real_1: 0.23602354946270795 (+0.0005125875624132914) | > avg_loss_disc_real_2: 0.24119206150652658 (-0.0014086842117175258) | > avg_loss_disc_real_3: 0.2435859269659284 (+0.001028218739469261) | > avg_loss_disc_real_4: 0.24697781301720043 (+0.002946085073578547) | > avg_loss_disc_real_5: 0.24951797870682998 (+0.0060120407544391186) | > avg_loss_0: 2.8518801037694366 (-0.004582505830576622) | > avg_grad_norm_0: tensor(9.1970, device='cuda:0') (tensor(-0.6018, device='cuda:0')) | > avg_loss_gen: 1.7252002249301321 (+0.012258655588391898) | > avg_loss_kl: 2.4500964560978855 (+0.04857402452280901) | > avg_loss_feat: 1.619010352752578 (-0.018940673747532832) | > avg_loss_mel: 18.125743879398826 (-0.10593850847700992) | > avg_loss_duration: 1.6806666532032928 (-0.000366229406544738) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.600717598283794 (-0.06441280203805988) | > avg_grad_norm_1: tensor(214.5087, device='cuda:0') (tensor(-14.8968, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1004753.pth  > EPOCH: 66/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:51:53)   --> TIME: 2025-02-25 08:52:16 -- STEP: 47/72 -- GLOBAL_STEP: 1004800 | > loss_disc: 2.961214065551758 (2.870808408615437) | > loss_disc_real_0: 0.1899324208498001 (0.2305303653503986) | > loss_disc_real_1: 0.19863244891166687 (0.24118986472170403) | > loss_disc_real_2: 0.26501980423927307 (0.24783141530574632) | > loss_disc_real_3: 0.20319916307926178 (0.24458488536641954) | > loss_disc_real_4: 0.22528700530529022 (0.24730165968550014) | > loss_disc_real_5: 0.23124881088733673 (0.24659027190918617) | > loss_0: 2.961214065551758 (2.870808408615437) | > grad_norm_0: tensor(9.7361, device='cuda:0') (tensor(11.1613, device='cuda:0')) | > loss_gen: 1.5274100303649902 (1.730169435764881) | > loss_kl: 2.3185319900512695 (2.47647947453438) | > loss_feat: 1.6841219663619995 (1.6108984490658373) | > loss_mel: 19.01351547241211 (18.797482023847863) | > loss_duration: 1.6970081329345703 (1.6725865095219714) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.240589141845703 (26.287615918098613) | > grad_norm_1: tensor(261.0907, device='cuda:0') (tensor(197.0942, device='cuda:0')) | > current_lr_0: 0.00019835668528513637 | > current_lr_1: 0.00019835668528513637 | > step_time: 0.4504 (0.43595314025878906) | > loader_time: 0.0087 (0.00735347828966506) --> EVAL PERFORMANCE | > avg_loader_time: 0.008363381238050863 (-0.00045588990332375533) | > avg_step_time: 0.45075371231831296 (-0.0006653221560196099) | > avg_loss_disc: 2.863606587262221 (+0.011726483492784379) | > avg_loss_disc_real_0: 0.2321007938032419 (-0.0012149672273179257) | > avg_loss_disc_real_1: 0.2384666922226758 (+0.002443142759967848) | > avg_loss_disc_real_2: 0.2448567420663968 (+0.0036646805598702092) | > avg_loss_disc_real_3: 0.24202673347063466 (-0.001559193495293737) | > avg_loss_disc_real_4: 0.24579566654185175 (-0.0011821464753486721) | > avg_loss_disc_real_5: 0.24489610232937503 (-0.004621876377454953) | > avg_loss_0: 2.863606587262221 (+0.011726483492784379) | > avg_grad_norm_0: tensor(11.0308, device='cuda:0') (+tensor(1.8338, device='cuda:0')) | > avg_loss_gen: 1.72164725753623 (-0.0035529673939020956) | > avg_loss_kl: 2.4595747195499045 (+0.009478263452018965) | > avg_loss_feat: 1.6023239803985811 (-0.016686372353996948) | > avg_loss_mel: 18.59838969270948 (+0.4726458133106526) | > avg_loss_duration: 1.6825711676772213 (+0.0019045144739284758) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.064507068043024 (+0.46378946975923085) | > avg_grad_norm_1: tensor(193.0348, device='cuda:0') (tensor(-21.4739, device='cuda:0'))  > EPOCH: 67/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:52:30)   --> TIME: 2025-02-25 08:52:42 -- STEP: 25/72 -- GLOBAL_STEP: 1004850 | > loss_disc: 2.838111162185669 (2.8587242126464845) | > loss_disc_real_0: 0.2646409273147583 (0.23010312795639037) | > loss_disc_real_1: 0.25168710947036743 (0.24857015430927276) | > loss_disc_real_2: 0.21542489528656006 (0.24474615693092347) | > loss_disc_real_3: 0.2494901865720749 (0.24472183346748352) | > loss_disc_real_4: 0.2582859992980957 (0.24486838042736053) | > loss_disc_real_5: 0.27310651540756226 (0.2496925449371337) | > loss_0: 2.838111162185669 (2.8587242126464845) | > grad_norm_0: tensor(6.6539, device='cuda:0') (tensor(8.6613, device='cuda:0')) | > loss_gen: 1.6985331773757935 (1.748354001045227) | > loss_kl: 2.6169345378875732 (2.504551067352296) | > loss_feat: 1.6943621635437012 (1.6653272819519043) | > loss_mel: 19.2904109954834 (18.767690429687505) | > loss_duration: 1.6339256763458252 (1.6678301525115966) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.934167861938477 (26.353753128051757) | > grad_norm_1: tensor(77.7440, device='cuda:0') (tensor(209.6896, device='cuda:0')) | > current_lr_0: 0.00019833189069947573 | > current_lr_1: 0.00019833189069947573 | > step_time: 0.4228 (0.4308187961578369) | > loader_time: 0.0076 (0.006964130401611328) --> EVAL PERFORMANCE | > avg_loader_time: 0.008604456001604106 (+0.0002410747635532423) | > avg_step_time: 0.45123613720208827 (+0.000482424883775312) | > avg_loss_disc: 2.8653656899089555 (+0.0017591026467345117) | > avg_loss_disc_real_0: 0.2361263311664823 (+0.004025537363240406) | > avg_loss_disc_real_1: 0.23941778234193023 (+0.0009510901192544274) | > avg_loss_disc_real_2: 0.2430641575598381 (-0.0017925845065586976) | > avg_loss_disc_real_3: 0.24318679516584102 (+0.0011600616952063547) | > avg_loss_disc_real_4: 0.24618024872222416 (+0.0003845821803724059) | > avg_loss_disc_real_5: 0.24942749780668338 (+0.004531395477308353) | > avg_loss_0: 2.8653656899089555 (+0.0017591026467345117) | > avg_grad_norm_0: tensor(10.1630, device='cuda:0') (tensor(-0.8678, device='cuda:0')) | > avg_loss_gen: 1.727036793467025 (+0.005389535930794942) | > avg_loss_kl: 2.4751670293405024 (+0.015592309790597891) | > avg_loss_feat: 1.59926919198372 (-0.003054788414861065) | > avg_loss_mel: 18.404993379619764 (-0.19339631308971406) | > avg_loss_duration: 1.6836964647534867 (+0.001125297076265408) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.89016291121362 (-0.17434415682940596) | > avg_grad_norm_1: tensor(196.3245, device='cuda:0') (+tensor(3.2897, device='cuda:0'))  > EPOCH: 68/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:53:06)   --> TIME: 2025-02-25 08:53:10 -- STEP: 3/72 -- GLOBAL_STEP: 1004900 | > loss_disc: 2.792980909347534 (2.8005477587381997) | > loss_disc_real_0: 0.2047598510980606 (0.19731472929318747) | > loss_disc_real_1: 0.16917984187602997 (0.18932140370210013) | > loss_disc_real_2: 0.16995784640312195 (0.24051589767138162) | > loss_disc_real_3: 0.21965377032756805 (0.23553876082102457) | > loss_disc_real_4: 0.23361361026763916 (0.24067358672618866) | > loss_disc_real_5: 0.23979723453521729 (0.24127432703971863) | > loss_0: 2.792980909347534 (2.8005477587381997) | > grad_norm_0: tensor(12.8941, device='cuda:0') (tensor(9.7003, device='cuda:0')) | > loss_gen: 1.751641869544983 (1.667905290921529) | > loss_kl: 2.5913009643554688 (2.617438316345215) | > loss_feat: 2.0342490673065186 (1.835221568743388) | > loss_mel: 19.666044235229492 (18.620386759440105) | > loss_duration: 1.7034912109375 (1.6847174564997356) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.746728897094727 (26.425670623779297) | > grad_norm_1: tensor(378.3166, device='cuda:0') (tensor(308.3279, device='cuda:0')) | > current_lr_0: 0.0001983070992131383 | > current_lr_1: 0.0001983070992131383 | > step_time: 0.4231 (0.4295247395833333) | > loader_time: 0.0054 (0.005953311920166016)  --> TIME: 2025-02-25 08:53:33 -- STEP: 53/72 -- GLOBAL_STEP: 1004950 | > loss_disc: 2.8841655254364014 (2.8577777889539613) | > loss_disc_real_0: 0.2060757577419281 (0.22993211701231184) | > loss_disc_real_1: 0.20003622770309448 (0.23140764714411968) | > loss_disc_real_2: 0.22752971947193146 (0.24110156269568317) | > loss_disc_real_3: 0.21575085818767548 (0.2433890242059276) | > loss_disc_real_4: 0.23478594422340393 (0.24618661291194413) | > loss_disc_real_5: 0.22109901905059814 (0.2467680427263368) | > loss_0: 2.8841655254364014 (2.8577777889539613) | > grad_norm_0: tensor(13.7719, device='cuda:0') (tensor(8.8410, device='cuda:0')) | > loss_gen: 1.6201080083847046 (1.7134176222783215) | > loss_kl: 2.3424248695373535 (2.4300130268312863) | > loss_feat: 1.5010193586349487 (1.688233049410694) | > loss_mel: 17.707239151000977 (18.310369491577152) | > loss_duration: 1.7286468744277954 (1.6716177058669757) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.899438858032227 (25.813650814992076) | > grad_norm_1: tensor(61.8485, device='cuda:0') (tensor(219.3347, device='cuda:0')) | > current_lr_0: 0.0001983070992131383 | > current_lr_1: 0.0001983070992131383 | > step_time: 0.4455 (0.4409812891258384) | > loader_time: 0.0091 (0.007657001603324458) --> EVAL PERFORMANCE | > avg_loader_time: 0.008450007774460484 (-0.00015444822714362136) | > avg_step_time: 0.45316291527009345 (+0.001926778068005175) | > avg_loss_disc: 2.8521756823633755 (-0.013190007545579974) | > avg_loss_disc_real_0: 0.22817978011050694 (-0.00794655105597536) | > avg_loss_disc_real_1: 0.2351843645455132 (-0.004233417796417033) | > avg_loss_disc_real_2: 0.24244674305680772 (-0.0006174145030303779) | > avg_loss_disc_real_3: 0.2421194892114317 (-0.00106730595440932) | > avg_loss_disc_real_4: 0.24524168208451338 (-0.0009385666377107749) | > avg_loss_disc_real_5: 0.2457114728823514 (-0.0037160249243319843) | > avg_loss_0: 2.8521756823633755 (-0.013190007545579974) | > avg_grad_norm_0: tensor(8.6315, device='cuda:0') (tensor(-1.5314, device='cuda:0')) | > avg_loss_gen: 1.7205414889563977 (-0.006495304510627298) | > avg_loss_kl: 2.4539920947921114 (-0.021174934548390922) | > avg_loss_feat: 1.6737407395537471 (+0.07447154757002705) | > avg_loss_mel: 18.383905491358803 (-0.021087888260961307) | > avg_loss_duration: 1.6795125863921474 (-0.004183878361339266) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.911692498435436 (+0.02152958722181708) | > avg_grad_norm_1: tensor(225.9081, device='cuda:0') (+tensor(29.5836, device='cuda:0'))  > EPOCH: 69/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:53:43)   --> TIME: 2025-02-25 08:53:59 -- STEP: 31/72 -- GLOBAL_STEP: 1005000 | > loss_disc: 2.88020658493042 (2.8558029743932907) | > loss_disc_real_0: 0.25921809673309326 (0.23349157889043132) | > loss_disc_real_1: 0.1984739750623703 (0.23444981776898907) | > loss_disc_real_2: 0.21922732889652252 (0.23742005853883683) | > loss_disc_real_3: 0.23514579236507416 (0.2429834839797789) | > loss_disc_real_4: 0.21972054243087769 (0.24366771261538228) | > loss_disc_real_5: 0.22057785093784332 (0.24282492264624564) | > loss_0: 2.88020658493042 (2.8558029743932907) | > grad_norm_0: tensor(3.3912, device='cuda:0') (tensor(8.0085, device='cuda:0')) | > loss_gen: 1.7458689212799072 (1.7149726806148406) | > loss_kl: 2.5564374923706055 (2.434836556834559) | > loss_feat: 1.5610660314559937 (1.6764402850981681) | > loss_mel: 19.012895584106445 (18.24597518674789) | > loss_duration: 1.6750812530517578 (1.6553473203412947) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.551349639892578 (25.727572164227887) | > grad_norm_1: tensor(82.0240, device='cuda:0') (tensor(233.6124, device='cuda:0')) | > current_lr_0: 0.00019828231082573666 | > current_lr_1: 0.00019828231082573666 | > step_time: 0.4462 (0.4321405656876103) | > loader_time: 0.0085 (0.00693509655614053) > CHECKPOINT : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/checkpoint_1005000.pth --> EVAL PERFORMANCE | > avg_loader_time: 0.008527282258154639 (+7.727448369415461e-05) | > avg_step_time: 0.45036829693216673 (-0.002794618337926713) | > avg_loss_disc: 2.861509269391987 (+0.009333587028611312) | > avg_loss_disc_real_0: 0.23181007676561113 (+0.003630296655104198) | > avg_loss_disc_real_1: 0.23565732845118348 (+0.0004729639056702817) | > avg_loss_disc_real_2: 0.24018584632537734 (-0.002260896731430373) | > avg_loss_disc_real_3: 0.24245458064784467 (+0.0003350914364129709) | > avg_loss_disc_real_4: 0.24452068919027356 (-0.0007209928942398247) | > avg_loss_disc_real_5: 0.24501653519314778 (-0.0006949376892036174) | > avg_loss_0: 2.861509269391987 (+0.009333587028611312) | > avg_grad_norm_0: tensor(8.2683, device='cuda:0') (tensor(-0.3632, device='cuda:0')) | > avg_loss_gen: 1.7024013089462064 (-0.018140180010191242) | > avg_loss_kl: 2.4353465765294895 (-0.018645518262621952) | > avg_loss_feat: 1.6176793541706784 (-0.05606138538306871) | > avg_loss_mel: 18.107040230657013 (-0.27686526070178985) | > avg_loss_duration: 1.6755447102264618 (-0.003967876165685613) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.538012276233083 (-0.37368022220235275) | > avg_grad_norm_1: tensor(216.6890, device='cuda:0') (tensor(-9.2191, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1005041.pth  > EPOCH: 70/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:54:25)   --> TIME: 2025-02-25 08:54:31 -- STEP: 9/72 -- GLOBAL_STEP: 1005050 | > loss_disc: 2.8284387588500977 (2.8409921328226724) | > loss_disc_real_0: 0.2338678389787674 (0.24406239059236315) | > loss_disc_real_1: 0.2625011205673218 (0.23919519947634804) | > loss_disc_real_2: 0.23587952554225922 (0.23931928310129377) | > loss_disc_real_3: 0.23338717222213745 (0.23986412915918562) | > loss_disc_real_4: 0.26560285687446594 (0.24692904286914402) | > loss_disc_real_5: 0.2600603997707367 (0.24635240932305655) | > loss_0: 2.8284387588500977 (2.8409921328226724) | > grad_norm_0: tensor(4.9568, device='cuda:0') (tensor(9.4487, device='cuda:0')) | > loss_gen: 1.6667253971099854 (1.740802076127794) | > loss_kl: 2.2921295166015625 (2.5866204367743597) | > loss_feat: 1.6664109230041504 (1.62271867858039) | > loss_mel: 18.493732452392578 (18.123147540622288) | > loss_duration: 1.6680991649627686 (1.6830500496758356) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.78709602355957 (25.75633896721734) | > grad_norm_1: tensor(47.8397, device='cuda:0') (tensor(290.1562, device='cuda:0')) | > current_lr_0: 0.00019825752553688343 | > current_lr_1: 0.00019825752553688343 | > step_time: 0.4225 (0.4297192361619737) | > loader_time: 0.0056 (0.0064800050523546)  --> TIME: 2025-02-25 08:54:54 -- STEP: 59/72 -- GLOBAL_STEP: 1005100 | > loss_disc: 2.8100428581237793 (2.8573180376473126) | > loss_disc_real_0: 0.2021401971578598 (0.2312783242787345) | > loss_disc_real_1: 0.1943504959344864 (0.23456210024276022) | > loss_disc_real_2: 0.243153378367424 (0.2409320261518834) | > loss_disc_real_3: 0.23977723717689514 (0.24512599029783474) | > loss_disc_real_4: 0.26211512088775635 (0.24627139507713963) | > loss_disc_real_5: 0.2670755684375763 (0.24910253690460982) | > loss_0: 2.8100428581237793 (2.8573180376473126) | > grad_norm_0: tensor(6.2045, device='cuda:0') (tensor(7.7693, device='cuda:0')) | > loss_gen: 1.8488173484802246 (1.7188405283426835) | > loss_kl: 2.342414379119873 (2.495618266574407) | > loss_feat: 1.5830343961715698 (1.6348160121400477) | > loss_mel: 17.121068954467773 (18.313494827787746) | > loss_duration: 1.6916083097457886 (1.6747149794788685) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.586944580078125 (25.837484941644185) | > grad_norm_1: tensor(213.7298, device='cuda:0') (tensor(175.2543, device='cuda:0')) | > current_lr_0: 0.00019825752553688343 | > current_lr_1: 0.00019825752553688343 | > step_time: 0.485 (0.4408773123207739) | > loader_time: 0.0097 (0.007970797813544837) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855339748758665 (+2.611522943201175e-05) | > avg_step_time: 0.45162401065020497 (+0.0012557137180382316) | > avg_loss_disc: 2.856547436244051 (-0.004961833147935746) | > avg_loss_disc_real_0: 0.2311108794430612 (-0.0006991973225499437) | > avg_loss_disc_real_1: 0.23525757772821776 (-0.00039975072296571934) | > avg_loss_disc_real_2: 0.2407837302332193 (+0.0005978839078419618) | > avg_loss_disc_real_3: 0.24387338295788832 (+0.0014188023100436542) | > avg_loss_disc_real_4: 0.24495098645418462 (+0.00043029726391105583) | > avg_loss_disc_real_5: 0.246791680933724 (+0.0017751457405762172) | > avg_loss_0: 2.856547436244051 (-0.004961833147935746) | > avg_grad_norm_0: tensor(8.2463, device='cuda:0') (tensor(-0.0220, device='cuda:0')) | > avg_loss_gen: 1.7111253402602504 (+0.008724031314043978) | > avg_loss_kl: 2.5006303686491202 (+0.06528379211963076) | > avg_loss_feat: 1.6254788368520603 (+0.007799482681381864) | > avg_loss_mel: 18.29026501615282 (+0.18322478549580623) | > avg_loss_duration: 1.67900430148756 (+0.0034595912610981827) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.80650407495633 (+0.2684917987232467) | > avg_grad_norm_1: tensor(179.5515, device='cuda:0') (tensor(-37.1374, device='cuda:0'))  > EPOCH: 71/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:55:02)   --> TIME: 2025-02-25 08:55:20 -- STEP: 37/72 -- GLOBAL_STEP: 1005150 | > loss_disc: 2.810821533203125 (2.8530902798111373) | > loss_disc_real_0: 0.27082765102386475 (0.230627505763157) | > loss_disc_real_1: 0.21721482276916504 (0.2297666473565875) | > loss_disc_real_2: 0.17970117926597595 (0.23854572627995466) | > loss_disc_real_3: 0.19739145040512085 (0.24240086731073018) | > loss_disc_real_4: 0.20575544238090515 (0.2431505435221904) | > loss_disc_real_5: 0.20660565793514252 (0.24734629892014168) | > loss_0: 2.810821533203125 (2.8530902798111373) | > grad_norm_0: tensor(6.5514, device='cuda:0') (tensor(8.5888, device='cuda:0')) | > loss_gen: 1.6635339260101318 (1.7017445822019832) | > loss_kl: 2.4243862628936768 (2.4802605654742265) | > loss_feat: 1.7951290607452393 (1.6670856250298989) | > loss_mel: 18.675933837890625 (18.6642349346264) | > loss_duration: 1.6867473125457764 (1.6654378175735474) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.245729446411133 (26.178763363812422) | > grad_norm_1: tensor(61.0072, device='cuda:0') (tensor(189.8332, device='cuda:0')) | > current_lr_0: 0.0001982327433461913 | > current_lr_1: 0.0001982327433461913 | > step_time: 0.4419 (0.4342161835850896) | > loader_time: 0.0084 (0.007117761147988809) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864141423937301 (+8.801675178635865e-05) | > avg_step_time: 0.4517893824778812 (+0.0001653718276762306) | > avg_loss_disc: 2.8533990013767303 (-0.0031484348673207485) | > avg_loss_disc_real_0: 0.23298955677260816 (+0.0018786773295469683) | > avg_loss_disc_real_1: 0.2350516878593136 (-0.00020588986890415772) | > avg_loss_disc_real_2: 0.24218710627354367 (+0.0014033760403243667) | > avg_loss_disc_real_3: 0.24298608177144762 (-0.0008873011864407032) | > avg_loss_disc_real_4: 0.24549511977484528 (+0.0005441333206606669) | > avg_loss_disc_real_5: 0.2480824004176637 (+0.0012907194839396896) | > avg_loss_0: 2.8533990013767303 (-0.0031484348673207485) | > avg_grad_norm_0: tensor(8.5679, device='cuda:0') (+tensor(0.3216, device='cuda:0')) | > avg_loss_gen: 1.7186290697312692 (+0.0075037294710187386) | > avg_loss_kl: 2.5099289988128226 (+0.00929863016370236) | > avg_loss_feat: 1.6524961028300535 (+0.027017265977993254) | > avg_loss_mel: 18.59331896607305 (+0.3030539499202298) | > avg_loss_duration: 1.6832626218527136 (+0.004258320365153612) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.15763586339816 (+0.3511317884418297) | > avg_grad_norm_1: tensor(201.9945, device='cuda:0') (+tensor(22.4429, device='cuda:0'))  > EPOCH: 72/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:55:39)   --> TIME: 2025-02-25 08:55:47 -- STEP: 15/72 -- GLOBAL_STEP: 1005200 | > loss_disc: 2.850466728210449 (2.8452314058939616) | > loss_disc_real_0: 0.2432892620563507 (0.22665836612383525) | > loss_disc_real_1: 0.18925677239894867 (0.2336742450793584) | > loss_disc_real_2: 0.2692889869213104 (0.24280764957269033) | > loss_disc_real_3: 0.2821803689002991 (0.24566096564133963) | > loss_disc_real_4: 0.29261624813079834 (0.246475014090538) | > loss_disc_real_5: 0.2873423993587494 (0.24285610616207123) | > loss_0: 2.850466728210449 (2.8452314058939616) | > grad_norm_0: tensor(5.6809, device='cuda:0') (tensor(7.9221, device='cuda:0')) | > loss_gen: 1.7407225370407104 (1.7304233630498251) | > loss_kl: 2.5074830055236816 (2.4565542856852214) | > loss_feat: 1.5671511888504028 (1.693858806292216) | > loss_mel: 20.90087127685547 (18.555100123087566) | > loss_duration: 1.6358002424240112 (1.6670158863067628) | > amp_scaler: 256.0 (256.0) | > loss_1: 28.352027893066406 (26.10295232137044) | > grad_norm_1: tensor(145.7686, device='cuda:0') (tensor(188.5617, device='cuda:0')) | > current_lr_0: 0.00019820796425327303 | > current_lr_1: 0.00019820796425327303 | > step_time: 0.4289 (0.4353130976359049) | > loader_time: 0.0074 (0.006827052434285482)  --> TIME: 2025-02-25 08:56:10 -- STEP: 65/72 -- GLOBAL_STEP: 1005250 | > loss_disc: 2.792635440826416 (2.8572427309476414) | > loss_disc_real_0: 0.2534359395503998 (0.2294968598164045) | > loss_disc_real_1: 0.27439844608306885 (0.23509204823237198) | > loss_disc_real_2: 0.20111443102359772 (0.238500632918798) | > loss_disc_real_3: 0.2269127517938614 (0.24213187946723058) | > loss_disc_real_4: 0.24194316565990448 (0.24541323781013488) | > loss_disc_real_5: 0.23769402503967285 (0.2459368448991042) | > loss_0: 2.792635440826416 (2.8572427309476414) | > grad_norm_0: tensor(10.3410, device='cuda:0') (tensor(10.2111, device='cuda:0')) | > loss_gen: 1.8235875368118286 (1.7118660413301907) | > loss_kl: 2.568059206008911 (2.4649172452779915) | > loss_feat: 1.8164069652557373 (1.642987821652339) | > loss_mel: 19.155546188354492 (18.4417466383714) | > loss_duration: 1.6965388059616089 (1.675748872756958) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.060138702392578 (25.93726639380822) | > grad_norm_1: tensor(272.6232, device='cuda:0') (tensor(191.6394, device='cuda:0')) | > current_lr_0: 0.00019820796425327303 | > current_lr_1: 0.00019820796425327303 | > step_time: 0.4868 (0.44580775040846604) | > loader_time: 0.0116 (0.008385984714214619) --> EVAL PERFORMANCE | > avg_loader_time: 0.008647717220682492 (+6.302981309483022e-06) | > avg_step_time: 0.45084655788582817 (-0.0009428245920530265) | > avg_loss_disc: 2.8548914210897096 (+0.0014924197129793093) | > avg_loss_disc_real_0: 0.23020297141981796 (-0.002786585352790194) | > avg_loss_disc_real_1: 0.23421539618095882 (-0.000836291678354778) | > avg_loss_disc_real_2: 0.23830550762129502 (-0.003881598652248658) | > avg_loss_disc_real_3: 0.24198359104109482 (-0.0010024907303528008) | > avg_loss_disc_real_4: 0.24454964991186706 (-0.000945469862978221) | > avg_loss_disc_real_5: 0.24505568956825094 (-0.0030267108494127504) | > avg_loss_0: 2.8548914210897096 (+0.0014924197129793093) | > avg_grad_norm_0: tensor(10.1709, device='cuda:0') (+tensor(1.6030, device='cuda:0')) | > avg_loss_gen: 1.711098869081954 (-0.0075302006493152) | > avg_loss_kl: 2.463611992312149 (-0.0463170065006735) | > avg_loss_feat: 1.6408635814424972 (-0.011632521387556327) | > avg_loss_mel: 18.429836380649622 (-0.1634825854234272) | > avg_loss_duration: 1.6782698765606947 (-0.004992745292018874) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.92368053382551 (-0.2339553295726482) | > avg_grad_norm_1: tensor(197.5942, device='cuda:0') (tensor(-4.4003, device='cuda:0'))  > EPOCH: 73/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:56:15)   --> TIME: 2025-02-25 08:56:36 -- STEP: 43/72 -- GLOBAL_STEP: 1005300 | > loss_disc: 2.890089273452759 (2.8594547537870185) | > loss_disc_real_0: 0.23613108694553375 (0.23001156989918198) | > loss_disc_real_1: 0.23362191021442413 (0.23614915304405745) | > loss_disc_real_2: 0.23431651294231415 (0.24176117050093274) | > loss_disc_real_3: 0.24075812101364136 (0.24422294282635978) | > loss_disc_real_4: 0.24321606755256653 (0.24827921598456623) | > loss_disc_real_5: 0.2528892755508423 (0.252504637075025) | > loss_0: 2.890089273452759 (2.8594547537870185) | > grad_norm_0: tensor(8.1580, device='cuda:0') (tensor(8.6529, device='cuda:0')) | > loss_gen: 1.6736762523651123 (1.7279714429101278) | > loss_kl: 2.3221781253814697 (2.3956856699877) | > loss_feat: 1.580487608909607 (1.692995739537616) | > loss_mel: 16.75711441040039 (18.388033622919128) | > loss_duration: 1.6722352504730225 (1.6621904428615126) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.005691528320312 (25.866876823957575) | > grad_norm_1: tensor(150.3445, device='cuda:0') (tensor(247.3045, device='cuda:0')) | > current_lr_0: 0.00019818318825774137 | > current_lr_1: 0.00019818318825774137 | > step_time: 0.4518 (0.43354374308918797) | > loader_time: 0.0094 (0.007300077482711437) --> EVAL PERFORMANCE | > avg_loader_time: 0.008452166973705022 (-0.00019555024697747066) | > avg_step_time: 0.4482398335362824 (-0.002606724349545786) | > avg_loss_disc: 2.857885679728548 (+0.0029942586388385095) | > avg_loss_disc_real_0: 0.23090795018303562 (+0.0007049787632176507) | > avg_loss_disc_real_1: 0.23705801787510725 (+0.002842621694148423) | > avg_loss_disc_real_2: 0.243241247996478 (+0.00493574037518299) | > avg_loss_disc_real_3: 0.2445204199199945 (+0.002536828878899694) | > avg_loss_disc_real_4: 0.24669256575510534 (+0.00214291584323828) | > avg_loss_disc_real_5: 0.24935563307412914 (+0.004299943505878201) | > avg_loss_0: 2.857885679728548 (+0.0029942586388385095) | > avg_grad_norm_0: tensor(8.5734, device='cuda:0') (tensor(-1.5976, device='cuda:0')) | > avg_loss_gen: 1.7236461320393521 (+0.012547262957398164) | > avg_loss_kl: 2.4131851548879917 (-0.050426837424157434) | > avg_loss_feat: 1.6663751669333016 (+0.025511585490804345) | > avg_loss_mel: 18.215060543006576 (-0.21477583764304597) | > avg_loss_duration: 1.675156076189498 (-0.0031138003711967954) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.693423015970584 (-0.23025751785492687) | > avg_grad_norm_1: tensor(228.4740, device='cuda:0') (+tensor(30.8799, device='cuda:0'))  > EPOCH: 74/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:56:52)   --> TIME: 2025-02-25 08:57:03 -- STEP: 21/72 -- GLOBAL_STEP: 1005350 | > loss_disc: 2.909548282623291 (2.835058632351103) | > loss_disc_real_0: 0.2182680070400238 (0.2183205158937545) | > loss_disc_real_1: 0.2520752251148224 (0.24671648300829388) | > loss_disc_real_2: 0.24684783816337585 (0.23934777293886458) | > loss_disc_real_3: 0.262095183134079 (0.24675360180082775) | > loss_disc_real_4: 0.2733018696308136 (0.24509248918011076) | > loss_disc_real_5: 0.2780856788158417 (0.24297322758606502) | > loss_0: 2.909548282623291 (2.835058632351103) | > grad_norm_0: tensor(4.4043, device='cuda:0') (tensor(9.8512, device='cuda:0')) | > loss_gen: 1.7130038738250732 (1.7490785746347337) | > loss_kl: 2.2985928058624268 (2.518406867980957) | > loss_feat: 1.3334720134735107 (1.739604910214742) | > loss_mel: 17.30977439880371 (18.675301143101287) | > loss_duration: 1.6956875324249268 (1.6658230395544142) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.35053062438965 (26.348214558192662) | > grad_norm_1: tensor(192.2877, device='cuda:0') (tensor(158.7008, device='cuda:0')) | > current_lr_0: 0.00019815841535920914 | > current_lr_1: 0.00019815841535920914 | > step_time: 0.4536 (0.4349487168448312) | > loader_time: 0.0066 (0.006708167848132905)  --> TIME: 2025-02-25 08:57:27 -- STEP: 71/72 -- GLOBAL_STEP: 1005400 | > loss_disc: 2.9909119606018066 (2.849827692542278) | > loss_disc_real_0: 0.3414369523525238 (0.22777043450886095) | > loss_disc_real_1: 0.19097881019115448 (0.24028252170119488) | > loss_disc_real_2: 0.2844803035259247 (0.2419626643540154) | > loss_disc_real_3: 0.26807576417922974 (0.24357833115147873) | > loss_disc_real_4: 0.27622291445732117 (0.24531069230025923) | > loss_disc_real_5: 0.2762675881385803 (0.24379730476459988) | > loss_0: 2.9909119606018066 (2.849827692542278) | > grad_norm_0: tensor(12.5598, device='cuda:0') (tensor(8.9664, device='cuda:0')) | > loss_gen: 1.785667896270752 (1.7307571139134152) | > loss_kl: 2.533198356628418 (2.473680073106792) | > loss_feat: 1.738784670829773 (1.6685777499642171) | > loss_mel: 18.718751907348633 (18.407632196453257) | > loss_duration: 1.6956455707550049 (1.6827924503407008) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.472047805786133 (25.96343972649373) | > grad_norm_1: tensor(199.3083, device='cuda:0') (tensor(191.6819, device='cuda:0')) | > current_lr_0: 0.00019815841535920914 | > current_lr_1: 0.00019815841535920914 | > step_time: 0.4976 (0.453129167288122) | > loader_time: 0.0083 (0.008542665293518926) --> EVAL PERFORMANCE | > avg_loader_time: 0.008542665293518926 (+9.049831981390415e-05) | > avg_step_time: 0.453129167288122 (+0.004889333751839642) | > avg_loss_disc: 2.849827692542278 (-0.008057987186270044) | > avg_loss_disc_real_0: 0.22777043450886095 (-0.003137515674174668) | > avg_loss_disc_real_1: 0.24028252170119488 (+0.0032245038260876324) | > avg_loss_disc_real_2: 0.2419626643540154 (-0.0012785836424626107) | > avg_loss_disc_real_3: 0.24357833115147873 (-0.0009420887685157864) | > avg_loss_disc_real_4: 0.24531069230025923 (-0.001381873454846111) | > avg_loss_disc_real_5: 0.24379730476459988 (-0.005558328309529265) | > avg_loss_0: 2.849827692542278 (-0.008057987186270044) | > avg_grad_norm_0: tensor(8.9664, device='cuda:0') (+tensor(0.3930, device='cuda:0')) | > avg_loss_gen: 1.7307571139134152 (+0.007110981874063116) | > avg_loss_kl: 2.473680073106792 (+0.060494918218800375) | > avg_loss_feat: 1.6685777499642171 (+0.0022025830309155836) | > avg_loss_mel: 18.407632196453257 (+0.1925716534466808) | > avg_loss_duration: 1.6827924503407008 (+0.007636374151202885) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.96343972649373 (+0.27001671052314435) | > avg_grad_norm_1: tensor(191.6819, device='cuda:0') (tensor(-36.7921, device='cuda:0'))  > EPOCH: 75/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:57:29)   --> TIME: 2025-02-25 08:57:52 -- STEP: 49/72 -- GLOBAL_STEP: 1005450 | > loss_disc: 2.8605105876922607 (2.8710233046084035) | > loss_disc_real_0: 0.281941682100296 (0.23419511287796255) | > loss_disc_real_1: 0.2697787582874298 (0.236409420565683) | > loss_disc_real_2: 0.2415625900030136 (0.24183042438662783) | > loss_disc_real_3: 0.25373634696006775 (0.24508066323338723) | > loss_disc_real_4: 0.2764936089515686 (0.2493748515844345) | > loss_disc_real_5: 0.2728559970855713 (0.24935766507168203) | > loss_0: 2.8605105876922607 (2.8710233046084035) | > grad_norm_0: tensor(6.0903, device='cuda:0') (tensor(9.3501, device='cuda:0')) | > loss_gen: 1.6420700550079346 (1.7235968745484644) | > loss_kl: 2.2953133583068848 (2.427608227243229) | > loss_feat: 1.4242959022521973 (1.670667543703196) | > loss_mel: 17.92823600769043 (18.324967248099192) | > loss_duration: 1.7434343099594116 (1.6677255362880474) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.03335189819336 (25.814565658569336) | > grad_norm_1: tensor(81.7618, device='cuda:0') (tensor(211.8009, device='cuda:0')) | > current_lr_0: 0.00019813364555728923 | > current_lr_1: 0.00019813364555728923 | > step_time: 0.4398 (0.43651318063541333) | > loader_time: 0.0095 (0.007630207100693061) --> EVAL PERFORMANCE | > avg_loader_time: 0.00850583466005997 (-3.683063345895528e-05) | > avg_step_time: 0.44820473899304025 (-0.004924428295081773) | > avg_loss_disc: 2.865070359807619 (+0.015242667265340959) | > avg_loss_disc_real_0: 0.23367451865908126 (+0.005904084150220312) | > avg_loss_disc_real_1: 0.23515723876550165 (-0.005125282935693226) | > avg_loss_disc_real_2: 0.2414244379795773 (-0.0005382263744380944) | > avg_loss_disc_real_3: 0.24412937924055986 (+0.0005510480890811298) | > avg_loss_disc_real_4: 0.24861213795735804 (+0.003301445657098806) | > avg_loss_disc_real_5: 0.2495169182058791 (+0.00571961344127922) | > avg_loss_0: 2.865070359807619 (+0.015242667265340959) | > avg_grad_norm_0: tensor(8.9754, device='cuda:0') (+tensor(0.0089, device='cuda:0')) | > avg_loss_gen: 1.719577109309989 (-0.011180004603426275) | > avg_loss_kl: 2.4273098220287914 (-0.046370251078000635) | > avg_loss_feat: 1.650300687467548 (-0.01827706249666905) | > avg_loss_mel: 18.380199244324597 (-0.027432952128659593) | > avg_loss_duration: 1.6762235601183395 (-0.00656889022236129) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.853610562606598 (-0.10982916388713093) | > avg_grad_norm_1: tensor(216.6683, device='cuda:0') (+tensor(24.9863, device='cuda:0'))  > EPOCH: 76/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:58:05)   --> TIME: 2025-02-25 08:58:19 -- STEP: 27/72 -- GLOBAL_STEP: 1005500 | > loss_disc: 2.7963969707489014 (2.8711316497237593) | > loss_disc_real_0: 0.272580623626709 (0.23786320123407576) | > loss_disc_real_1: 0.19737379252910614 (0.23526284429762098) | > loss_disc_real_2: 0.22215446829795837 (0.2432553497729478) | > loss_disc_real_3: 0.2220640331506729 (0.24332286693431712) | > loss_disc_real_4: 0.23229099810123444 (0.2487132449944814) | > loss_disc_real_5: 0.24183237552642822 (0.2518831232079754) | > loss_0: 2.7963969707489014 (2.8711316497237593) | > grad_norm_0: tensor(8.0373, device='cuda:0') (tensor(7.2512, device='cuda:0')) | > loss_gen: 1.6871098279953003 (1.7223981000758983) | > loss_kl: 2.326436758041382 (2.4240102149822094) | > loss_feat: 1.6185864210128784 (1.6425382340395893) | > loss_mel: 18.007118225097656 (18.115255214549876) | > loss_duration: 1.6438300609588623 (1.6688205798467) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.283079147338867 (25.573022489194518) | > grad_norm_1: tensor(201.0176, device='cuda:0') (tensor(224.3143, device='cuda:0')) | > current_lr_0: 0.00019810887885159456 | > current_lr_1: 0.00019810887885159456 | > step_time: 0.4291 (0.43099780435915347) | > loader_time: 0.008 (0.006773736741807725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008483799410537935 (-2.203524952203509e-05) | > avg_step_time: 0.44895440423992317 (+0.0007496652468829135) | > avg_loss_disc: 2.8616832041404616 (-0.0033871556671574155) | > avg_loss_disc_real_0: 0.23186053923318084 (-0.0018139794259004194) | > avg_loss_disc_real_1: 0.2353427231731549 (+0.00018548440765325802) | > avg_loss_disc_real_2: 0.2418747514906064 (+0.0004503135110290879) | > avg_loss_disc_real_3: 0.2429171105925466 (-0.0012122686480132705) | > avg_loss_disc_real_4: 0.2459047705354825 (-0.0027073674218755506) | > avg_loss_disc_real_5: 0.2474975671986459 (-0.002019351007233189) | > avg_loss_0: 2.8616832041404616 (-0.0033871556671574155) | > avg_grad_norm_0: tensor(8.3858, device='cuda:0') (tensor(-0.5895, device='cuda:0')) | > avg_loss_gen: 1.7168683132655185 (-0.002708796044470496) | > avg_loss_kl: 2.419711089470017 (-0.0075987325587743015) | > avg_loss_feat: 1.6511345463739315 (+0.0008338589063834334) | > avg_loss_mel: 18.077335250209753 (-0.3028639941148441) | > avg_loss_duration: 1.6807423292751043 (+0.004518769156764746) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.545791679704692 (-0.3078188829019055) | > avg_grad_norm_1: tensor(217.2332, device='cuda:0') (+tensor(0.5649, device='cuda:0'))  > EPOCH: 77/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:58:41)   --> TIME: 2025-02-25 08:58:45 -- STEP: 5/72 -- GLOBAL_STEP: 1005550 | > loss_disc: 2.72395920753479 (2.814996528625488) | > loss_disc_real_0: 0.2720147371292114 (0.23146049976348876) | > loss_disc_real_1: 0.23396673798561096 (0.2358856290578842) | > loss_disc_real_2: 0.21150429546833038 (0.2283957153558731) | > loss_disc_real_3: 0.20758260786533356 (0.23739745914936067) | > loss_disc_real_4: 0.23475193977355957 (0.24222363233566285) | > loss_disc_real_5: 0.24117229878902435 (0.24578551054000855) | > loss_0: 2.72395920753479 (2.814996528625488) | > grad_norm_0: tensor(7.3141, device='cuda:0') (tensor(8.3490, device='cuda:0')) | > loss_gen: 1.815287470817566 (1.726225233078003) | > loss_kl: 2.2854294776916504 (2.436672019958496) | > loss_feat: 1.9383132457733154 (1.6435854196548463) | > loss_mel: 18.006906509399414 (17.977390670776366) | > loss_duration: 1.632140874862671 (1.6760910511016847) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.678075790405273 (25.459964752197266) | > grad_norm_1: tensor(139.9421, device='cuda:0') (tensor(184.9381, device='cuda:0')) | > current_lr_0: 0.0001980841152417381 | > current_lr_1: 0.0001980841152417381 | > step_time: 0.4203 (0.41947221755981445) | > loader_time: 0.0061 (0.006465387344360351)  --> TIME: 2025-02-25 08:59:08 -- STEP: 55/72 -- GLOBAL_STEP: 1005600 | > loss_disc: 2.8589742183685303 (2.8442533536390826) | > loss_disc_real_0: 0.2649068236351013 (0.22729939317161388) | > loss_disc_real_1: 0.17416824400424957 (0.23175730217586865) | > loss_disc_real_2: 0.2464473694562912 (0.23964316519823942) | > loss_disc_real_3: 0.2502548098564148 (0.2418990284204483) | > loss_disc_real_4: 0.2625226378440857 (0.24719381440769542) | > loss_disc_real_5: 0.2460917979478836 (0.24682031815702266) | > loss_0: 2.8589742183685303 (2.8442533536390826) | > grad_norm_0: tensor(19.4921, device='cuda:0') (tensor(9.1696, device='cuda:0')) | > loss_gen: 1.7728049755096436 (1.7226262721148404) | > loss_kl: 2.378443717956543 (2.448891271244396) | > loss_feat: 1.4947761297225952 (1.7039690537886185) | > loss_mel: 17.756275177001953 (18.06703903891824) | > loss_duration: 1.7048263549804688 (1.6719985723495483) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.107126235961914 (25.61452435580167) | > grad_norm_1: tensor(47.7852, device='cuda:0') (tensor(191.5772, device='cuda:0')) | > current_lr_0: 0.0001980841152417381 | > current_lr_1: 0.0001980841152417381 | > step_time: 0.4559 (0.4391085841438987) | > loader_time: 0.0092 (0.007928076657381926) --> EVAL PERFORMANCE | > avg_loader_time: 0.008604885826648121 (+0.00012108641611018565) | > avg_step_time: 0.4564982602294062 (+0.007543855989483039) | > avg_loss_disc: 2.8592332450436873 (-0.002449959096774368) | > avg_loss_disc_real_0: 0.23302411728761566 (+0.0011635780544348162) | > avg_loss_disc_real_1: 0.23427298287270773 (-0.0010697403004471806) | > avg_loss_disc_real_2: 0.2407339057871993 (-0.001140845703407084) | > avg_loss_disc_real_3: 0.24313014493861668 (+0.00021303434607009009) | > avg_loss_disc_real_4: 0.24797631758199612 (+0.002071547046513633) | > avg_loss_disc_real_5: 0.24778120891309122 (+0.0002836417144453096) | > avg_loss_0: 2.8592332450436873 (-0.002449959096774368) | > avg_grad_norm_0: tensor(9.1560, device='cuda:0') (+tensor(0.7702, device='cuda:0')) | > avg_loss_gen: 1.721922199491044 (+0.005053886225525606) | > avg_loss_kl: 2.468578597189675 (+0.048867507719657866) | > avg_loss_feat: 1.6862331108308175 (+0.03509856445688597) | > avg_loss_mel: 18.13225888534331 (+0.05492363513355869) | > avg_loss_duration: 1.6778263004732803 (-0.0029160288018239555) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68681910340215 (+0.1410274236974587) | > avg_grad_norm_1: tensor(208.1187, device='cuda:0') (tensor(-9.1146, device='cuda:0'))  > EPOCH: 78/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:59:18)   --> TIME: 2025-02-25 08:59:35 -- STEP: 33/72 -- GLOBAL_STEP: 1005650 | > loss_disc: 2.790376663208008 (2.8566062811649213) | > loss_disc_real_0: 0.17264452576637268 (0.23312380322904297) | > loss_disc_real_1: 0.2591272294521332 (0.23009059781377966) | > loss_disc_real_2: 0.23839600384235382 (0.24054804308847946) | > loss_disc_real_3: 0.23863646388053894 (0.24346944057580197) | > loss_disc_real_4: 0.23291859030723572 (0.24457054743261047) | > loss_disc_real_5: 0.23612362146377563 (0.24668677512443427) | > loss_0: 2.790376663208008 (2.8566062811649213) | > grad_norm_0: tensor(6.6469, device='cuda:0') (tensor(8.9917, device='cuda:0')) | > loss_gen: 1.781074047088623 (1.7125915469545308) | > loss_kl: 2.461029529571533 (2.4425124977574204) | > loss_feat: 1.897339105606079 (1.6338420817346284) | > loss_mel: 19.391494750976562 (18.339348590735234) | > loss_duration: 1.6695436239242554 (1.6657922448533955) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.200481414794922 (25.794087207678594) | > grad_norm_1: tensor(143.2126, device='cuda:0') (tensor(140.2878, device='cuda:0')) | > current_lr_0: 0.00019805935472733287 | > current_lr_1: 0.00019805935472733287 | > step_time: 0.449 (0.43367233420863294) | > loader_time: 0.0078 (0.00736074736624053) --> EVAL PERFORMANCE | > avg_loader_time: 0.008648163835767289 (+4.327800911916799e-05) | > avg_step_time: 0.44983687870939015 (-0.006661381520016052) | > avg_loss_disc: 2.8581796061824747 (-0.0010536388612125691) | > avg_loss_disc_real_0: 0.232213051596158 (-0.0008110656914576686) | > avg_loss_disc_real_1: 0.23258839505659024 (-0.0016845878161174943) | > avg_loss_disc_real_2: 0.24204493450446868 (+0.0013110287172693746) | > avg_loss_disc_real_3: 0.24198980461543715 (-0.0011403403231795284) | > avg_loss_disc_real_4: 0.24413669822921216 (-0.0038396193527839573) | > avg_loss_disc_real_5: 0.24475536044214813 (-0.0030258484709430877) | > avg_loss_0: 2.8581796061824747 (-0.0010536388612125691) | > avg_grad_norm_0: tensor(8.6579, device='cuda:0') (tensor(-0.4980, device='cuda:0')) | > avg_loss_gen: 1.7079949865878468 (-0.013927212903197317) | > avg_loss_kl: 2.421348330000757 (-0.047230267188917985) | > avg_loss_feat: 1.6364348337683878 (-0.049798277062429674) | > avg_loss_mel: 18.217871115241252 (+0.08561222989794004) | > avg_loss_duration: 1.6808108527895431 (+0.002984552316262823) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.66446038367043 (-0.0223587197317201) | > avg_grad_norm_1: tensor(173.9222, device='cuda:0') (tensor(-34.1965, device='cuda:0'))  > EPOCH: 79/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 08:59:55)   --> TIME: 2025-02-25 09:00:01 -- STEP: 11/72 -- GLOBAL_STEP: 1005700 | > loss_disc: 2.7831950187683105 (2.838224107568914) | > loss_disc_real_0: 0.21808256208896637 (0.22882152958349747) | > loss_disc_real_1: 0.2626716196537018 (0.2366937886584889) | > loss_disc_real_2: 0.24606268107891083 (0.2318264530463652) | > loss_disc_real_3: 0.21672222018241882 (0.23693794282999905) | > loss_disc_real_4: 0.24890807271003723 (0.24557337300343948) | > loss_disc_real_5: 0.24879519641399384 (0.2455155524340543) | > loss_0: 2.7831950187683105 (2.838224107568914) | > grad_norm_0: tensor(5.5364, device='cuda:0') (tensor(9.3621, device='cuda:0')) | > loss_gen: 1.7399327754974365 (1.7067810730500654) | > loss_kl: 2.069121837615967 (2.340772520412098) | > loss_feat: 1.7643928527832031 (1.6640893762761897) | > loss_mel: 18.985366821289062 (18.164338805458762) | > loss_duration: 1.6322674751281738 (1.6578261202031916) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.191082000732422 (25.533808274702594) | > grad_norm_1: tensor(382.9642, device='cuda:0') (tensor(135.5414, device='cuda:0')) | > current_lr_0: 0.00019803459730799195 | > current_lr_1: 0.00019803459730799195 | > step_time: 0.4368 (0.4272426041689786) | > loader_time: 0.0057 (0.006252874027598988)  --> TIME: 2025-02-25 09:00:24 -- STEP: 61/72 -- GLOBAL_STEP: 1005750 | > loss_disc: 2.8677265644073486 (2.8578592480206098) | > loss_disc_real_0: 0.29782193899154663 (0.22972287921631923) | > loss_disc_real_1: 0.2489088922739029 (0.23633389443647665) | > loss_disc_real_2: 0.26545628905296326 (0.24194612390682346) | > loss_disc_real_3: 0.24783086776733398 (0.24494173096828772) | > loss_disc_real_4: 0.25405168533325195 (0.2464844877602624) | > loss_disc_real_5: 0.2515145242214203 (0.24723643113355168) | > loss_0: 2.8677265644073486 (2.8578592480206098) | > grad_norm_0: tensor(8.9503, device='cuda:0') (tensor(8.3542, device='cuda:0')) | > loss_gen: 1.6459076404571533 (1.7242548309388708) | > loss_kl: 2.2714312076568604 (2.4939640975389317) | > loss_feat: 1.4693413972854614 (1.6783210996721611) | > loss_mel: 16.473316192626953 (18.340850251619926) | > loss_duration: 1.6963646411895752 (1.6744895079096809) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.55636215209961 (25.91187989907186) | > grad_norm_1: tensor(370.9273, device='cuda:0') (tensor(212.5628, device='cuda:0')) | > current_lr_0: 0.00019803459730799195 | > current_lr_1: 0.00019803459730799195 | > step_time: 0.4632 (0.4423288869076088) | > loader_time: 0.0106 (0.00793285057192943) --> EVAL PERFORMANCE | > avg_loader_time: 0.008439369604621135 (-0.0002087942311461543) | > avg_step_time: 0.44941488118238854 (-0.0004219975270016163) | > avg_loss_disc: 2.857244175924382 (-0.0009354302580928398) | > avg_loss_disc_real_0: 0.22968186584996506 (-0.002531185746192932) | > avg_loss_disc_real_1: 0.2343789050486726 (+0.0017905099920823564) | > avg_loss_disc_real_2: 0.24239188873432052 (+0.00034695422985184243) | > avg_loss_disc_real_3: 0.2448527153948663 (+0.0028629107794291564) | > avg_loss_disc_real_4: 0.2470359017311687 (+0.0028992035019565487) | > avg_loss_disc_real_5: 0.247747315277516 (+0.002991954835367877) | > avg_loss_0: 2.857244175924382 (-0.0009354302580928398) | > avg_grad_norm_0: tensor(8.4869, device='cuda:0') (tensor(-0.1710, device='cuda:0')) | > avg_loss_gen: 1.7191599305246916 (+0.011164943936844862) | > avg_loss_kl: 2.4848850814389505 (+0.06353675143819348) | > avg_loss_feat: 1.6686405564697695 (+0.03220572270138167) | > avg_loss_mel: 18.31249047668886 (+0.0946193614476094) | > avg_loss_duration: 1.6793756132394495 (-0.0014352395500936144) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.864551705373838 (+0.2000913217034075) | > avg_grad_norm_1: tensor(218.1111, device='cuda:0') (+tensor(44.1889, device='cuda:0'))  > EPOCH: 80/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:00:31)   --> TIME: 2025-02-25 09:00:50 -- STEP: 39/72 -- GLOBAL_STEP: 1005800 | > loss_disc: 2.838954210281372 (2.8442571896773114) | > loss_disc_real_0: 0.21383817493915558 (0.22768254425281134) | > loss_disc_real_1: 0.1863125115633011 (0.23590000432271224) | > loss_disc_real_2: 0.2255401313304901 (0.2411131304808152) | > loss_disc_real_3: 0.22537493705749512 (0.24411095946263045) | > loss_disc_real_4: 0.22919875383377075 (0.24481368026672265) | > loss_disc_real_5: 0.22883909940719604 (0.24323275647102258) | > loss_0: 2.838954210281372 (2.8442571896773114) | > grad_norm_0: tensor(5.0043, device='cuda:0') (tensor(8.5572, device='cuda:0')) | > loss_gen: 1.839397668838501 (1.7344961502613165) | > loss_kl: 2.440429449081421 (2.4400625779078555) | > loss_feat: 1.9399868249893188 (1.7316534519195557) | > loss_mel: 18.383113861083984 (18.532332102457683) | > loss_duration: 1.66410493850708 (1.6636276978712816) | > amp_scaler: 256.0 (262.56410256410254) | > loss_1: 26.267032623291016 (26.102171579996746) | > grad_norm_1: tensor(289.8247, device='cuda:0') (tensor(232.0081, device='cuda:0')) | > current_lr_0: 0.00019800984298332845 | > current_lr_1: 0.00019800984298332845 | > step_time: 0.4498 (0.43264116385044193) | > loader_time: 0.0081 (0.007227671452057667) --> EVAL PERFORMANCE | > avg_loader_time: 0.008508981113702479 (+6.961150908134373e-05) | > avg_step_time: 0.44888504458145356 (-0.0005298366009349742) | > avg_loss_disc: 2.853648474518682 (-0.0035957014056999803) | > avg_loss_disc_real_0: 0.22868907535579844 (-0.0009927904941666177) | > avg_loss_disc_real_1: 0.23826176557742373 (+0.003882860528751142) | > avg_loss_disc_real_2: 0.2415032879987233 (-0.0008886007355972114) | > avg_loss_disc_real_3: 0.24453478191100375 (-0.0003179334838625536) | > avg_loss_disc_real_4: 0.24551392797852906 (-0.0015219737526396548) | > avg_loss_disc_real_5: 0.24583382316878144 (-0.0019134921087345658) | > avg_loss_0: 2.853648474518682 (-0.0035957014056999803) | > avg_grad_norm_0: tensor(8.2074, device='cuda:0') (tensor(-0.2795, device='cuda:0')) | > avg_loss_gen: 1.7206603087170023 (+0.0015003781923106363) | > avg_loss_kl: 2.4243003952671107 (-0.06058468617183976) | > avg_loss_feat: 1.6822016877187809 (+0.013561131249011371) | > avg_loss_mel: 18.32088244800836 (+0.008391971319497316) | > avg_loss_duration: 1.6765349448566707 (-0.002840668382778855) | > avg_amp_scaler: 259.6056338028169 (+3.6056338028168966) | > avg_loss_1: 25.824579507532253 (-0.039972197841585455) | > avg_grad_norm_1: tensor(222.6794, device='cuda:0') (+tensor(4.5683, device='cuda:0'))  > EPOCH: 81/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:01:07)   --> TIME: 2025-02-25 09:01:17 -- STEP: 17/72 -- GLOBAL_STEP: 1005850 | > loss_disc: 2.817281723022461 (2.840534238254323) | > loss_disc_real_0: 0.2390151023864746 (0.22502156829132752) | > loss_disc_real_1: 0.22698631882667542 (0.22972943064044504) | > loss_disc_real_2: 0.1927202343940735 (0.23569690479951746) | > loss_disc_real_3: 0.21733137965202332 (0.24076282452134526) | > loss_disc_real_4: 0.22457954287528992 (0.24517312119988835) | > loss_disc_real_5: 0.2415163516998291 (0.24356597837279825) | > loss_0: 2.817281723022461 (2.840534238254323) | > grad_norm_0: tensor(3.6905, device='cuda:0') (tensor(7.3973, device='cuda:0')) | > loss_gen: 1.9419257640838623 (1.732550445724936) | > loss_kl: 2.5454444885253906 (2.410373533473295) | > loss_feat: 1.9768235683441162 (1.746274099630468) | > loss_mel: 18.746431350708008 (18.336212831384994) | > loss_duration: 1.6682466268539429 (1.6495556130128748) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.87887191772461 (25.874966677497415) | > grad_norm_1: tensor(431.0878, device='cuda:0') (tensor(299.2061, device='cuda:0')) | > current_lr_0: 0.00019798509175295552 | > current_lr_1: 0.00019798509175295552 | > step_time: 0.4281 (0.43836090143989115) | > loader_time: 0.0061 (0.006636465297025793)  --> TIME: 2025-02-25 09:01:41 -- STEP: 67/72 -- GLOBAL_STEP: 1005900 | > loss_disc: 2.8841617107391357 (2.847749742109384) | > loss_disc_real_0: 0.21129730343818665 (0.22772507747607446) | > loss_disc_real_1: 0.2889651656150818 (0.23306900863327198) | > loss_disc_real_2: 0.2256956696510315 (0.24679844859820693) | > loss_disc_real_3: 0.21158535778522491 (0.24371891368680926) | > loss_disc_real_4: 0.25270524621009827 (0.24654603849596052) | > loss_disc_real_5: 0.2544425129890442 (0.24607362542579422) | > loss_0: 2.8841617107391357 (2.847749742109384) | > grad_norm_0: tensor(5.2303, device='cuda:0') (tensor(8.4994, device='cuda:0')) | > loss_gen: 1.7967164516448975 (1.7436806248195136) | > loss_kl: 2.382844924926758 (2.4308629391798338) | > loss_feat: 1.7660163640975952 (1.729317279004339) | > loss_mel: 18.498598098754883 (18.22575295861087) | > loss_duration: 1.7327824831008911 (1.6699604027306856) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.176958084106445 (25.79957421146222) | > grad_norm_1: tensor(312.4117, device='cuda:0') (tensor(232.7052, device='cuda:0')) | > current_lr_0: 0.00019798509175295552 | > current_lr_1: 0.00019798509175295552 | > step_time: 0.4969 (0.45248287115524066) | > loader_time: 0.0121 (0.008291201805000877) --> EVAL PERFORMANCE | > avg_loader_time: 0.008468359289034992 (-4.062182466748644e-05) | > avg_step_time: 0.45613767395556815 (+0.007252629374114583) | > avg_loss_disc: 2.8494625057972653 (-0.004185968721416522) | > avg_loss_disc_real_0: 0.22876387016034463 (+7.479480454619658e-05) | > avg_loss_disc_real_1: 0.23244980061557932 (-0.0058119649618444125) | > avg_loss_disc_real_2: 0.2461767576529946 (+0.0046734696542712895) | > avg_loss_disc_real_3: 0.24486444302847687 (+0.00032966111747312343) | > avg_loss_disc_real_4: 0.24723654250863572 (+0.0017226145301066642) | > avg_loss_disc_real_5: 0.24643492656694332 (+0.0006011033981618841) | > avg_loss_0: 2.8494625057972653 (-0.004185968721416522) | > avg_grad_norm_0: tensor(8.6424, device='cuda:0') (+tensor(0.4349, device='cuda:0')) | > avg_loss_gen: 1.7455559498827222 (+0.02489564116571996) | > avg_loss_kl: 2.430800538667491 (+0.006500143400380409) | > avg_loss_feat: 1.7313730313744344 (+0.049171343655653565) | > avg_loss_mel: 18.20157634036641 (-0.11930610764194682) | > avg_loss_duration: 1.6716373386517378 (-0.004897606204932847) | > avg_amp_scaler: 256.0 (-3.6056338028168966) | > avg_loss_1: 25.780943158646703 (-0.043636348885549125) | > avg_grad_norm_1: tensor(229.2429, device='cuda:0') (+tensor(6.5635, device='cuda:0'))  > EPOCH: 82/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:01:44)   --> TIME: 2025-02-25 09:02:07 -- STEP: 45/72 -- GLOBAL_STEP: 1005950 | > loss_disc: 2.8944900035858154 (2.833508666356405) | > loss_disc_real_0: 0.2580908536911011 (0.23095714218086666) | > loss_disc_real_1: 0.24486194550991058 (0.23254381749365066) | > loss_disc_real_2: 0.1967284083366394 (0.23888639509677886) | > loss_disc_real_3: 0.2243981808423996 (0.23963924811946022) | > loss_disc_real_4: 0.2420007586479187 (0.24301706453164418) | > loss_disc_real_5: 0.22485961019992828 (0.24520366423659853) | > loss_0: 2.8944900035858154 (2.833508666356405) | > grad_norm_0: tensor(5.8805, device='cuda:0') (tensor(9.8931, device='cuda:0')) | > loss_gen: 1.4626922607421875 (1.7282514810562133) | > loss_kl: 2.5747323036193848 (2.509362379709878) | > loss_feat: 1.62249755859375 (1.7904404083887735) | > loss_mel: 18.34469985961914 (18.39048733181423) | > loss_duration: 1.6824324131011963 (1.6605517599317763) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.687053680419922 (26.079093509250217) | > grad_norm_1: tensor(316.5132, device='cuda:0') (tensor(208.1762, device='cuda:0')) | > current_lr_0: 0.0001979603436164864 | > current_lr_1: 0.0001979603436164864 | > step_time: 0.4629 (0.43998600112067326) | > loader_time: 0.009 (0.007560364405314128) --> EVAL PERFORMANCE | > avg_loader_time: 0.008591709002642564 (+0.00012334971360757237) | > avg_step_time: 0.45431200551315093 (-0.0018256684424172165) | > avg_loss_disc: 2.8462462492392104 (-0.0032162565580549085) | > avg_loss_disc_real_0: 0.23146004177315135 (+0.0026961716128067192) | > avg_loss_disc_real_1: 0.23306398219625715 (+0.0006141815806778306) | > avg_loss_disc_real_2: 0.23973775392686816 (-0.0064390037261264355) | > avg_loss_disc_real_3: 0.2411585811158301 (-0.0037058619126467685) | > avg_loss_disc_real_4: 0.2431486878596561 (-0.004087854648979622) | > avg_loss_disc_real_5: 0.2452859234222224 (-0.0011490031447209115) | > avg_loss_0: 2.8462462492392104 (-0.0032162565580549085) | > avg_grad_norm_0: tensor(9.9579, device='cuda:0') (+tensor(1.3155, device='cuda:0')) | > avg_loss_gen: 1.7223130917884935 (-0.02324285809422877) | > avg_loss_kl: 2.524731934910088 (+0.09393139624259694) | > avg_loss_feat: 1.740874985573997 (+0.009501954199562457) | > avg_loss_mel: 18.275856609075838 (+0.07428026870942617) | > avg_loss_duration: 1.6743024419730819 (+0.0026651033213440645) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.93807921611088 (+0.15713605746417514) | > avg_grad_norm_1: tensor(238.9791, device='cuda:0') (+tensor(9.7362, device='cuda:0'))  > EPOCH: 83/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:02:21)   --> TIME: 2025-02-25 09:02:33 -- STEP: 23/72 -- GLOBAL_STEP: 1006000 | > loss_disc: 2.738790988922119 (2.844216263812521) | > loss_disc_real_0: 0.24594828486442566 (0.22717083666635596) | > loss_disc_real_1: 0.209656223654747 (0.2344916767400244) | > loss_disc_real_2: 0.26063770055770874 (0.24892906650252963) | > loss_disc_real_3: 0.24813802540302277 (0.2442489013723705) | > loss_disc_real_4: 0.28192153573036194 (0.24740267512590988) | > loss_disc_real_5: 0.2506193220615387 (0.24504438172216) | > loss_0: 2.738790988922119 (2.844216263812521) | > grad_norm_0: tensor(6.8288, device='cuda:0') (tensor(7.6796, device='cuda:0')) | > loss_gen: 1.8838251829147339 (1.7314931102420972) | > loss_kl: 2.4465363025665283 (2.4344017194664995) | > loss_feat: 1.8921529054641724 (1.655924854071244) | > loss_mel: 17.764602661132812 (18.346592778744906) | > loss_duration: 1.6598494052886963 (1.6561098720716394) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.6469669342041 (25.824522516001824) | > grad_norm_1: tensor(64.3773, device='cuda:0') (tensor(261.3699, device='cuda:0')) | > current_lr_0: 0.00019793559857353432 | > current_lr_1: 0.00019793559857353432 | > step_time: 0.4333 (0.43310849562935205) | > loader_time: 0.0064 (0.006783848223478898) --> EVAL PERFORMANCE | > avg_loader_time: 0.008589428915104395 (-2.280087538169759e-06) | > avg_step_time: 0.4509244804650965 (-0.0033875250480544516) | > avg_loss_disc: 2.8558765465105083 (+0.009630297271297827) | > avg_loss_disc_real_0: 0.23311128813615986 (+0.0016512463630085072) | > avg_loss_disc_real_1: 0.2374413850861536 (+0.004377402889896437) | > avg_loss_disc_real_2: 0.24322544688909825 (+0.003487692962230088) | > avg_loss_disc_real_3: 0.2426596850576535 (+0.0015011039418234007) | > avg_loss_disc_real_4: 0.2472593364581256 (+0.004110648598469491) | > avg_loss_disc_real_5: 0.24930379529234395 (+0.004017871870121537) | > avg_loss_0: 2.8558765465105083 (+0.009630297271297827) | > avg_grad_norm_0: tensor(9.0351, device='cuda:0') (tensor(-0.9228, device='cuda:0')) | > avg_loss_gen: 1.7293063821926924 (+0.0069932904041989374) | > avg_loss_kl: 2.396164803437783 (-0.12856713147230492) | > avg_loss_feat: 1.6757589914429356 (-0.06511599413106128) | > avg_loss_mel: 18.290321699330505 (+0.014465090254667246) | > avg_loss_duration: 1.6762725769633977 (+0.0019701349903158416) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76782449534242 (-0.17025472076845816) | > avg_grad_norm_1: tensor(222.6369, device='cuda:0') (tensor(-16.3421, device='cuda:0'))  > EPOCH: 84/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:02:58)   --> TIME: 2025-02-25 09:03:00 -- STEP: 1/72 -- GLOBAL_STEP: 1006050 | > loss_disc: 2.9282140731811523 (2.9282140731811523) | > loss_disc_real_0: 0.2183455228805542 (0.2183455228805542) | > loss_disc_real_1: 0.1405745893716812 (0.1405745893716812) | > loss_disc_real_2: 0.23146413266658783 (0.23146413266658783) | > loss_disc_real_3: 0.23153740167617798 (0.23153740167617798) | > loss_disc_real_4: 0.22167439758777618 (0.22167439758777618) | > loss_disc_real_5: 0.24115917086601257 (0.24115917086601257) | > loss_0: 2.9282140731811523 (2.9282140731811523) | > grad_norm_0: tensor(27.0323, device='cuda:0') (tensor(27.0323, device='cuda:0')) | > loss_gen: 1.614777684211731 (1.614777684211731) | > loss_kl: 2.3541746139526367 (2.3541746139526367) | > loss_feat: 1.8990113735198975 (1.8990113735198975) | > loss_mel: 18.279239654541016 (18.279239654541016) | > loss_duration: 1.673421025276184 (1.673421025276184) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.82062530517578 (25.82062530517578) | > grad_norm_1: tensor(241.6967, device='cuda:0') (tensor(241.6967, device='cuda:0')) | > current_lr_0: 0.00019791085662371262 | > current_lr_1: 0.00019791085662371262 | > step_time: 0.4453 (0.44533205032348633) | > loader_time: 0.006 (0.006005764007568359)  --> TIME: 2025-02-25 09:03:23 -- STEP: 51/72 -- GLOBAL_STEP: 1006100 | > loss_disc: 2.892099142074585 (2.8552628591948865) | > loss_disc_real_0: 0.2790597081184387 (0.2345593191829382) | > loss_disc_real_1: 0.21762381494045258 (0.2368882155301524) | > loss_disc_real_2: 0.2504127621650696 (0.24261855407088412) | > loss_disc_real_3: 0.2264476716518402 (0.24589942804738588) | > loss_disc_real_4: 0.2458173632621765 (0.24601367787987577) | > loss_disc_real_5: 0.24954885244369507 (0.2477161393446081) | > loss_0: 2.892099142074585 (2.8552628591948865) | > grad_norm_0: tensor(7.6902, device='cuda:0') (tensor(8.9535, device='cuda:0')) | > loss_gen: 1.5649921894073486 (1.7404673754000197) | > loss_kl: 2.3304624557495117 (2.451911725249945) | > loss_feat: 1.6212108135223389 (1.6928525625490676) | > loss_mel: 19.034324645996094 (18.51754813100778) | > loss_duration: 1.686582326889038 (1.6717022937886856) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.237573623657227 (26.074481964111328) | > grad_norm_1: tensor(113.5037, device='cuda:0') (tensor(236.1090, device='cuda:0')) | > current_lr_0: 0.00019791085662371262 | > current_lr_1: 0.00019791085662371262 | > step_time: 0.4462 (0.44507090718138453) | > loader_time: 0.009 (0.007654844545850567) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852726546811386 (-6.216344699053536e-05) | > avg_step_time: 0.4578740294550506 (+0.006949548989954146) | > avg_loss_disc: 2.8606332993843187 (+0.004756752873810388) | > avg_loss_disc_real_0: 0.23005860760598115 (-0.0030526805301787086) | > avg_loss_disc_real_1: 0.23606939529868917 (-0.0013719897874644171) | > avg_loss_disc_real_2: 0.24272921702391664 (-0.0004962298651816077) | > avg_loss_disc_real_3: 0.24456377411392374 (+0.0019040890562702362) | > avg_loss_disc_real_4: 0.2457938097732168 (-0.0014655266849087911) | > avg_loss_disc_real_5: 0.24752802999926285 (-0.0017757652930810996) | > avg_loss_0: 2.8606332993843187 (+0.004756752873810388) | > avg_grad_norm_0: tensor(8.7510, device='cuda:0') (tensor(-0.2841, device='cuda:0')) | > avg_loss_gen: 1.723023339056633 (-0.00628304313605943) | > avg_loss_kl: 2.4542252177923496 (+0.058060414354566436) | > avg_loss_feat: 1.663132029519954 (-0.012626961922981517) | > avg_loss_mel: 18.417953047953866 (+0.12763134862336045) | > avg_loss_duration: 1.6779593447564354 (+0.001686767793037669) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.93629291695608 (+0.16846842161366027) | > avg_grad_norm_1: tensor(235.8760, device='cuda:0') (+tensor(13.2391, device='cuda:0'))  > EPOCH: 85/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:03:35)   --> TIME: 2025-02-25 09:03:50 -- STEP: 29/72 -- GLOBAL_STEP: 1006150 | > loss_disc: 2.8701343536376953 (2.834670560113315) | > loss_disc_real_0: 0.1733030378818512 (0.23010013946171465) | > loss_disc_real_1: 0.2905268967151642 (0.23875944162237234) | > loss_disc_real_2: 0.2382815182209015 (0.24070084403301106) | > loss_disc_real_3: 0.2461562156677246 (0.2396396552694255) | > loss_disc_real_4: 0.21975797414779663 (0.2468794707594247) | > loss_disc_real_5: 0.2423006296157837 (0.24146245831045612) | > loss_0: 2.8701343536376953 (2.834670560113315) | > grad_norm_0: tensor(11.2189, device='cuda:0') (tensor(8.4126, device='cuda:0')) | > loss_gen: 1.6390775442123413 (1.7407010752579262) | > loss_kl: 2.2617533206939697 (2.4628754237602495) | > loss_feat: 1.7658865451812744 (1.7774986316417825) | > loss_mel: 18.46258544921875 (18.463224345240096) | > loss_duration: 1.6375397443771362 (1.6496071404424206) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.766841888427734 (26.09390666567046) | > grad_norm_1: tensor(277.0778, device='cuda:0') (tensor(196.2700, device='cuda:0')) | > current_lr_0: 0.00019788611776663464 | > current_lr_1: 0.00019788611776663464 | > step_time: 0.4374 (0.4325218282896897) | > loader_time: 0.0079 (0.006900409172321188) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570288268613144 (+4.302280049928511e-05) | > avg_step_time: 0.45108516787139463 (-0.006788861583655992) | > avg_loss_disc: 2.8457175375710073 (-0.014915761813311335) | > avg_loss_disc_real_0: 0.2306113024832497 (+0.0005526948772685558) | > avg_loss_disc_real_1: 0.23573304020183186 (-0.00033635509685731035) | > avg_loss_disc_real_2: 0.23919713686049823 (-0.0035320801634184107) | > avg_loss_disc_real_3: 0.24238315560448337 (-0.002180618509440374) | > avg_loss_disc_real_4: 0.24614831994117145 (+0.0003545101679546525) | > avg_loss_disc_real_5: 0.24596966257397557 (-0.0015583674252872826) | > avg_loss_0: 2.8457175375710073 (-0.014915761813311335) | > avg_grad_norm_0: tensor(9.1981, device='cuda:0') (+tensor(0.4471, device='cuda:0')) | > avg_loss_gen: 1.7265300566041972 (+0.0035067175475642642) | > avg_loss_kl: 2.4131988639562905 (-0.041026353836059126) | > avg_loss_feat: 1.7306787430400579 (+0.06754671352010377) | > avg_loss_mel: 18.296700034343026 (-0.12125301361083984) | > avg_loss_duration: 1.6727035582905085 (-0.005255786465926837) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.83981121761698 (-0.09648169933910111) | > avg_grad_norm_1: tensor(193.0138, device='cuda:0') (tensor(-42.8622, device='cuda:0'))  > EPOCH: 86/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:04:12)   --> TIME: 2025-02-25 09:04:17 -- STEP: 7/72 -- GLOBAL_STEP: 1006200 | > loss_disc: 2.7610316276550293 (2.7999659946986606) | > loss_disc_real_0: 0.19535070657730103 (0.23302748799324036) | > loss_disc_real_1: 0.2591384947299957 (0.25550136821610586) | > loss_disc_real_2: 0.26773422956466675 (0.24306118488311768) | > loss_disc_real_3: 0.2249167561531067 (0.2381300734622138) | > loss_disc_real_4: 0.2301509827375412 (0.24564845860004425) | > loss_disc_real_5: 0.25502562522888184 (0.23269700578280858) | > loss_0: 2.7610316276550293 (2.7999659946986606) | > grad_norm_0: tensor(7.9260, device='cuda:0') (tensor(7.8157, device='cuda:0')) | > loss_gen: 1.6956878900527954 (1.7945424999509538) | > loss_kl: 2.387800455093384 (2.3400176082338606) | > loss_feat: 2.0046169757843018 (2.0136820588793074) | > loss_mel: 18.0709285736084 (18.946001052856445) | > loss_duration: 1.6587438583374023 (1.670241458075387) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.817779541015625 (26.76448495047433) | > grad_norm_1: tensor(336.9724, device='cuda:0') (tensor(171.0379, device='cuda:0')) | > current_lr_0: 0.0001978613820019138 | > current_lr_1: 0.0001978613820019138 | > step_time: 0.4282 (0.43251729011535645) | > loader_time: 0.0054 (0.006013563701084682)  --> TIME: 2025-02-25 09:04:39 -- STEP: 57/72 -- GLOBAL_STEP: 1006250 | > loss_disc: 2.838599920272827 (2.841506891083299) | > loss_disc_real_0: 0.18903875350952148 (0.23246988654136658) | > loss_disc_real_1: 0.23110732436180115 (0.23908410652687676) | > loss_disc_real_2: 0.2400628924369812 (0.2394902695689285) | > loss_disc_real_3: 0.21258004009723663 (0.24105793316113322) | > loss_disc_real_4: 0.24586927890777588 (0.24517643216409182) | > loss_disc_real_5: 0.24971386790275574 (0.2456591754106053) | > loss_0: 2.838599920272827 (2.841506891083299) | > grad_norm_0: tensor(8.8800, device='cuda:0') (tensor(9.5912, device='cuda:0')) | > loss_gen: 1.6966314315795898 (1.7438703085246838) | > loss_kl: 2.3983519077301025 (2.4276626465613376) | > loss_feat: 1.8148961067199707 (1.7819558235636928) | > loss_mel: 17.692373275756836 (18.209991254304583) | > loss_duration: 1.697073221206665 (1.6659641893286454) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.299325942993164 (25.829444282933284) | > grad_norm_1: tensor(253.9631, device='cuda:0') (tensor(221.2857, device='cuda:0')) | > current_lr_0: 0.0001978613820019138 | > current_lr_1: 0.0001978613820019138 | > step_time: 0.4492 (0.4403759094706753) | > loader_time: 0.0097 (0.00774014623541581) --> EVAL PERFORMANCE | > avg_loader_time: 0.008413106622830244 (-0.0001571816457829004) | > avg_step_time: 0.45151684989391916 (+0.0004316820225245266) | > avg_loss_disc: 2.853258683647908 (+0.007541146076900596) | > avg_loss_disc_real_0: 0.2338726048318433 (+0.003261302348593592) | > avg_loss_disc_real_1: 0.24168113230819432 (+0.0059480921063624625) | > avg_loss_disc_real_2: 0.23946057364974224 (+0.00026343678924400926) | > avg_loss_disc_real_3: 0.24289938261811161 (+0.0005162270136282454) | > avg_loss_disc_real_4: 0.24637979793716486 (+0.00023147799599340657) | > avg_loss_disc_real_5: 0.24729578238977512 (+0.0013261198157995535) | > avg_loss_0: 2.853258683647908 (+0.007541146076900596) | > avg_grad_norm_0: tensor(9.4912, device='cuda:0') (+tensor(0.2931, device='cuda:0')) | > avg_loss_gen: 1.7409763655192416 (+0.014446308915044392) | > avg_loss_kl: 2.4328268128381643 (+0.019627948881873802) | > avg_loss_feat: 1.7557537303843969 (+0.025074987344338995) | > avg_loss_mel: 18.244040636949137 (-0.052659397393888696) | > avg_loss_duration: 1.6717125261333625 (-0.0009910321571460212) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.84531015745351 (+0.005498939836531491) | > avg_grad_norm_1: tensor(235.1172, device='cuda:0') (+tensor(42.1034, device='cuda:0'))  > EPOCH: 87/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:04:49)   --> TIME: 2025-02-25 09:05:06 -- STEP: 35/72 -- GLOBAL_STEP: 1006300 | > loss_disc: 2.8999404907226562 (2.8627241611480714) | > loss_disc_real_0: 0.25515055656433105 (0.24550126876149858) | > loss_disc_real_1: 0.24208100140094757 (0.23429017748151507) | > loss_disc_real_2: 0.2391432821750641 (0.2414703735283443) | > loss_disc_real_3: 0.26938581466674805 (0.2424354293516704) | > loss_disc_real_4: 0.25618529319763184 (0.24593753133501325) | > loss_disc_real_5: 0.2559240460395813 (0.24676197682108197) | > loss_0: 2.8999404907226562 (2.8627241611480714) | > grad_norm_0: tensor(6.3681, device='cuda:0') (tensor(9.4983, device='cuda:0')) | > loss_gen: 1.6077457666397095 (1.7279506547110421) | > loss_kl: 2.7808611392974854 (2.464648757662092) | > loss_feat: 1.6653039455413818 (1.7068392719541277) | > loss_mel: 19.819435119628906 (18.337733404976984) | > loss_duration: 1.714146614074707 (1.652941073690142) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.587493896484375 (25.890113285609655) | > grad_norm_1: tensor(331.2830, device='cuda:0') (tensor(202.1999, device='cuda:0')) | > current_lr_0: 0.00019783664932916355 | > current_lr_1: 0.00019783664932916355 | > step_time: 0.4453 (0.4395534924098424) | > loader_time: 0.008 (0.007381459644862583) --> EVAL PERFORMANCE | > avg_loader_time: 0.00881180964725118 (+0.000398703024420936) | > avg_step_time: 0.4554023474035129 (+0.003885497509593716) | > avg_loss_disc: 2.863372067330589 (+0.01011338368268122) | > avg_loss_disc_real_0: 0.23584418114222272 (+0.0019715763103794193) | > avg_loss_disc_real_1: 0.23384545785440525 (-0.007835674453789077) | > avg_loss_disc_real_2: 0.2421877086162567 (+0.0027271349665144717) | > avg_loss_disc_real_3: 0.24291995860321422 (+2.05759851026055e-05) | > avg_loss_disc_real_4: 0.24707031816663877 (+0.0006905202294739121) | > avg_loss_disc_real_5: 0.24735679424984355 (+6.1011860068432844e-05) | > avg_loss_0: 2.863372067330589 (+0.01011338368268122) | > avg_grad_norm_0: tensor(9.6442, device='cuda:0') (+tensor(0.1530, device='cuda:0')) | > avg_loss_gen: 1.7159030017718462 (-0.02507336374739544) | > avg_loss_kl: 2.4556731774773395 (+0.02284636463917522) | > avg_loss_feat: 1.6968503552423397 (-0.058903375142057124) | > avg_loss_mel: 18.12675617110561 (-0.11728446584352525) | > avg_loss_duration: 1.6736498852850685 (+0.0019373591517060085) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.668832644610337 (-0.17647751284317437) | > avg_grad_norm_1: tensor(205.3715, device='cuda:0') (tensor(-29.7457, device='cuda:0'))  > EPOCH: 88/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:05:25)   --> TIME: 2025-02-25 09:05:33 -- STEP: 13/72 -- GLOBAL_STEP: 1006350 | > loss_disc: 2.8792171478271484 (2.8135959001687856) | > loss_disc_real_0: 0.22929495573043823 (0.22259219678548667) | > loss_disc_real_1: 0.16809841990470886 (0.2348888092316114) | > loss_disc_real_2: 0.2420896291732788 (0.23673973404444182) | > loss_disc_real_3: 0.269193172454834 (0.24563161341043618) | > loss_disc_real_4: 0.25656622648239136 (0.2472249945768943) | > loss_disc_real_5: 0.25637462735176086 (0.24227781364550957) | > loss_0: 2.8792171478271484 (2.8135959001687856) | > grad_norm_0: tensor(10.2062, device='cuda:0') (tensor(9.3042, device='cuda:0')) | > loss_gen: 1.7955272197723389 (1.7903658059927134) | > loss_kl: 2.348168134689331 (2.397504403040959) | > loss_feat: 1.7215862274169922 (1.8663749236326952) | > loss_mel: 17.716148376464844 (18.01930985083947) | > loss_duration: 1.572614073753357 (1.6453483196405263) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.15404510498047 (25.718903908362755) | > grad_norm_1: tensor(248.8394, device='cuda:0') (tensor(233.8456, device='cuda:0')) | > current_lr_0: 0.0001978119197479974 | > current_lr_1: 0.0001978119197479974 | > step_time: 0.4409 (0.42978770916278547) | > loader_time: 0.0058 (0.00686038457430326)  --> TIME: 2025-02-25 09:05:56 -- STEP: 63/72 -- GLOBAL_STEP: 1006400 | > loss_disc: 2.979435443878174 (2.8406670093536377) | > loss_disc_real_0: 0.2522873878479004 (0.23186473997812423) | > loss_disc_real_1: 0.24919335544109344 (0.23501692570391156) | > loss_disc_real_2: 0.2851322889328003 (0.2419796803640941) | > loss_disc_real_3: 0.27651160955429077 (0.24527411967042892) | > loss_disc_real_4: 0.2751944363117218 (0.24504560303120387) | > loss_disc_real_5: 0.2781367003917694 (0.24683719802470433) | > loss_0: 2.979435443878174 (2.8406670093536377) | > grad_norm_0: tensor(5.5146, device='cuda:0') (tensor(9.0713, device='cuda:0')) | > loss_gen: 1.7725441455841064 (1.7551023979035636) | > loss_kl: 2.46433687210083 (2.4066026097252258) | > loss_feat: 1.4583135843276978 (1.7885641181279743) | > loss_mel: 17.801424026489258 (18.151214902363126) | > loss_duration: 1.643946647644043 (1.6602561795522297) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.140567779541016 (25.761740578545464) | > grad_norm_1: tensor(388.3426, device='cuda:0') (tensor(234.3666, device='cuda:0')) | > current_lr_0: 0.0001978119197479974 | > current_lr_1: 0.0001978119197479974 | > step_time: 0.4939 (0.4464765465448773) | > loader_time: 0.0111 (0.008162271408807664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008546302016352264 (-0.00026550763089891634) | > avg_step_time: 0.45245935883320554 (-0.0029429885703073366) | > avg_loss_disc: 2.842169227734418 (-0.021202839596170975) | > avg_loss_disc_real_0: 0.23207849033281835 (-0.003765690809404365) | > avg_loss_disc_real_1: 0.2358920345423927 (+0.002046576687987439) | > avg_loss_disc_real_2: 0.24057673841295107 (-0.0016109702033056394) | > avg_loss_disc_real_3: 0.24463343956101108 (+0.001713480957796859) | > avg_loss_disc_real_4: 0.24587441980838776 (-0.001195898358251013) | > avg_loss_disc_real_5: 0.24740410606626054 (+4.73118164169839e-05) | > avg_loss_0: 2.842169227734418 (-0.021202839596170975) | > avg_grad_norm_0: tensor(9.2561, device='cuda:0') (tensor(-0.3880, device='cuda:0')) | > avg_loss_gen: 1.7452172343160066 (+0.029314232544160435) | > avg_loss_kl: 2.423621298561634 (-0.0320518789157056) | > avg_loss_feat: 1.7629170619266135 (+0.06606670668427372) | > avg_loss_mel: 18.14453032318975 (+0.017774152084136574) | > avg_loss_duration: 1.6651700567191756 (-0.00847982856589291) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.74145626014387 (+0.07262361553353358) | > avg_grad_norm_1: tensor(234.8506, device='cuda:0') (+tensor(29.4790, device='cuda:0'))  > EPOCH: 89/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:06:02)   --> TIME: 2025-02-25 09:06:23 -- STEP: 41/72 -- GLOBAL_STEP: 1006450 | > loss_disc: 2.9040732383728027 (2.84514991830035) | > loss_disc_real_0: 0.25273779034614563 (0.23208104364755677) | > loss_disc_real_1: 0.22609162330627441 (0.23352660347775714) | > loss_disc_real_2: 0.24658502638339996 (0.24462011492833857) | > loss_disc_real_3: 0.2505195438861847 (0.24416981564789283) | > loss_disc_real_4: 0.24195295572280884 (0.24828591070524075) | > loss_disc_real_5: 0.24500368535518646 (0.2476124665359171) | > loss_0: 2.9040732383728027 (2.84514991830035) | > grad_norm_0: tensor(8.3701, device='cuda:0') (tensor(9.7839, device='cuda:0')) | > loss_gen: 1.784749984741211 (1.7591700844648408) | > loss_kl: 2.3241617679595947 (2.435089716097204) | > loss_feat: 1.7413135766983032 (1.7711134247663545) | > loss_mel: 18.207857131958008 (18.154651037076626) | > loss_duration: 1.6634197235107422 (1.6564990078530661) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.72150230407715 (25.776523264443004) | > grad_norm_1: tensor(224.5845, device='cuda:0') (tensor(196.4666, device='cuda:0')) | > current_lr_0: 0.0001977871932580289 | > current_lr_1: 0.0001977871932580289 | > step_time: 0.4396 (0.4368053238566329) | > loader_time: 0.0089 (0.007446056459008194) --> EVAL PERFORMANCE | > avg_loader_time: 0.008635870167906855 (+8.956815155459173e-05) | > avg_step_time: 0.4532308511331048 (+0.0007714922998992568) | > avg_loss_disc: 2.8517555955430165 (+0.00958636780859834) | > avg_loss_disc_real_0: 0.23604993627104962 (+0.003971445938231261) | > avg_loss_disc_real_1: 0.23376037935975572 (-0.002131655182636971) | > avg_loss_disc_real_2: 0.24440345944653094 (+0.0038267210335798663) | > avg_loss_disc_real_3: 0.24316036701202393 (-0.0014730725489871532) | > avg_loss_disc_real_4: 0.24767197752502604 (+0.0017975577166382817) | > avg_loss_disc_real_5: 0.24663576084963032 (-0.0007683452166302174) | > avg_loss_0: 2.8517555955430165 (+0.00958636780859834) | > avg_grad_norm_0: tensor(9.8615, device='cuda:0') (+tensor(0.6054, device='cuda:0')) | > avg_loss_gen: 1.7435075044631958 (-0.0017097298528108151) | > avg_loss_kl: 2.4347989827814236 (+0.011177684219789707) | > avg_loss_feat: 1.730300439915187 (-0.032616622011426566) | > avg_loss_mel: 18.103502031782984 (-0.04102829140676434) | > avg_loss_duration: 1.6719742341780326 (+0.006804177458856975) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68408321326887 (-0.057373046875) | > avg_grad_norm_1: tensor(223.4882, device='cuda:0') (tensor(-11.3624, device='cuda:0'))  > EPOCH: 90/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:06:39)   --> TIME: 2025-02-25 09:06:50 -- STEP: 19/72 -- GLOBAL_STEP: 1006500 | > loss_disc: 2.8341684341430664 (2.8467202814001786) | > loss_disc_real_0: 0.22207742929458618 (0.22897656967765406) | > loss_disc_real_1: 0.21857404708862305 (0.2429870743500559) | > loss_disc_real_2: 0.20325839519500732 (0.24033376025526146) | > loss_disc_real_3: 0.24378570914268494 (0.24023226452501198) | > loss_disc_real_4: 0.22039145231246948 (0.2419865186277189) | > loss_disc_real_5: 0.21705526113510132 (0.2429573151626085) | > loss_0: 2.8341684341430664 (2.8467202814001786) | > grad_norm_0: tensor(5.2269, device='cuda:0') (tensor(8.9425, device='cuda:0')) | > loss_gen: 1.6974616050720215 (1.735395544453671) | > loss_kl: 2.4156100749969482 (2.4058561827007092) | > loss_feat: 1.909623384475708 (1.8726198861473484) | > loss_mel: 17.914682388305664 (18.68711903220729) | > loss_duration: 1.629659652709961 (1.6430472700219405) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.56703758239746 (26.344038009643555) | > grad_norm_1: tensor(116.6928, device='cuda:0') (tensor(221.6084, device='cuda:0')) | > current_lr_0: 0.00019776246985887165 | > current_lr_1: 0.00019776246985887165 | > step_time: 0.4258 (0.4350703515504536) | > loader_time: 0.0065 (0.006777035562615646)  --> TIME: 2025-02-25 09:07:13 -- STEP: 69/72 -- GLOBAL_STEP: 1006550 | > loss_disc: 2.918752908706665 (2.8612664402395063) | > loss_disc_real_0: 0.2478901743888855 (0.23279014704883963) | > loss_disc_real_1: 0.3469572067260742 (0.2405726378378661) | > loss_disc_real_2: 0.26681607961654663 (0.24206166530864826) | > loss_disc_real_3: 0.26818329095840454 (0.24294510246187018) | > loss_disc_real_4: 0.25641316175460815 (0.24615242299826248) | > loss_disc_real_5: 0.28028759360313416 (0.24704133190106656) | > loss_0: 2.918752908706665 (2.8612664402395063) | > grad_norm_0: tensor(8.8906, device='cuda:0') (tensor(8.9738, device='cuda:0')) | > loss_gen: 1.773284912109375 (1.7300319049669348) | > loss_kl: 2.574341058731079 (2.4412692733432935) | > loss_feat: 1.6759419441223145 (1.755070434100386) | > loss_mel: 18.21282196044922 (18.48491508373316) | > loss_duration: 1.7445648908615112 (1.6709448427393816) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.980953216552734 (26.08223149396371) | > grad_norm_1: tensor(330.1239, device='cuda:0') (tensor(248.3322, device='cuda:0')) | > current_lr_0: 0.00019776246985887165 | > current_lr_1: 0.00019776246985887165 | > step_time: 0.5153 (0.4524563913759978) | > loader_time: 0.0126 (0.008469550505928373) --> EVAL PERFORMANCE | > avg_loader_time: 0.008527503886692963 (-0.00010836628121389281) | > avg_step_time: 0.4539017375086395 (+0.0006708863755346961) | > avg_loss_disc: 2.8626308206101547 (+0.010875225067138228) | > avg_loss_disc_real_0: 0.2319525958786548 (-0.004097340392394816) | > avg_loss_disc_real_1: 0.2404449441483323 (+0.006684564788576597) | > avg_loss_disc_real_2: 0.24218082574891373 (-0.0022226336976172156) | > avg_loss_disc_real_3: 0.24360353741007792 (+0.00044317039805399916) | > avg_loss_disc_real_4: 0.24630035816783635 (-0.001371619357189685) | > avg_loss_disc_real_5: 0.24771016059626996 (+0.0010743997466396427) | > avg_loss_0: 2.8626308206101547 (+0.010875225067138228) | > avg_grad_norm_0: tensor(8.9494, device='cuda:0') (tensor(-0.9122, device='cuda:0')) | > avg_loss_gen: 1.7287977698823096 (-0.014709734580886202) | > avg_loss_kl: 2.4455443939692536 (+0.010745411187830012) | > avg_loss_feat: 1.7476395200675643 (+0.017339080152377395) | > avg_loss_mel: 18.445632773385924 (+0.34213074160294) | > avg_loss_duration: 1.670929683765895 (-0.0010445504121376903) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.038544050404724 (+0.3544608371358535) | > avg_grad_norm_1: tensor(245.4543, device='cuda:0') (+tensor(21.9661, device='cuda:0'))  > EPOCH: 91/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:07:16)   --> TIME: 2025-02-25 09:07:39 -- STEP: 47/72 -- GLOBAL_STEP: 1006600 | > loss_disc: 2.830890417098999 (2.8514384158114168) | > loss_disc_real_0: 0.2766723930835724 (0.2301349658915337) | > loss_disc_real_1: 0.23910745978355408 (0.23885884976133387) | > loss_disc_real_2: 0.2891020178794861 (0.24076678492921463) | > loss_disc_real_3: 0.21676598489284515 (0.241262471739282) | > loss_disc_real_4: 0.238656684756279 (0.2443364826922721) | > loss_disc_real_5: 0.23516952991485596 (0.245204359292984) | > loss_0: 2.830890417098999 (2.8514384158114168) | > grad_norm_0: tensor(6.3734, device='cuda:0') (tensor(8.7641, device='cuda:0')) | > loss_gen: 1.6013948917388916 (1.7248294556394537) | > loss_kl: 2.4343888759613037 (2.4210688560567006) | > loss_feat: 1.8378955125808716 (1.7241496963703886) | > loss_mel: 18.268346786499023 (18.192724511978476) | > loss_duration: 1.6874648332595825 (1.6608499694377818) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.829490661621094 (25.723622585864778) | > grad_norm_1: tensor(128.6683, device='cuda:0') (tensor(211.8081, device='cuda:0')) | > current_lr_0: 0.0001977377495501393 | > current_lr_1: 0.0001977377495501393 | > step_time: 0.4503 (0.4409451687589605) | > loader_time: 0.0112 (0.007870638624150703) --> EVAL PERFORMANCE | > avg_loader_time: 0.008754387707777426 (+0.00022688382108446321) | > avg_step_time: 0.452153054761215 (-0.0017486827474245126) | > avg_loss_disc: 2.860117432097314 (-0.002513388512840553) | > avg_loss_disc_real_0: 0.23128781446688612 (-0.0006647814117686779) | > avg_loss_disc_real_1: 0.23891668861180965 (-0.0015282555365226658) | > avg_loss_disc_real_2: 0.2411172129738499 (-0.001063612775063838) | > avg_loss_disc_real_3: 0.24379451761783008 (+0.00019098020775215607) | > avg_loss_disc_real_4: 0.2466701097051862 (+0.00036975153734986055) | > avg_loss_disc_real_5: 0.24689148786202283 (-0.0008186727342471278) | > avg_loss_0: 2.860117432097314 (-0.002513388512840553) | > avg_grad_norm_0: tensor(9.3476, device='cuda:0') (+tensor(0.3982, device='cuda:0')) | > avg_loss_gen: 1.7308207968590963 (+0.0020230269767866904) | > avg_loss_kl: 2.4101092848979255 (-0.035435109071328075) | > avg_loss_feat: 1.71273805557842 (-0.03490146448914433) | > avg_loss_mel: 18.14619045526209 (-0.29944231812383393) | > avg_loss_duration: 1.6787345929884574 (+0.007804909222562495) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.67859321916607 (-0.35995083123865257) | > avg_grad_norm_1: tensor(232.5493, device='cuda:0') (tensor(-12.9049, device='cuda:0'))  > EPOCH: 92/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:07:53)   --> TIME: 2025-02-25 09:08:06 -- STEP: 25/72 -- GLOBAL_STEP: 1006650 | > loss_disc: 2.8274426460266113 (2.8599162006378176) | > loss_disc_real_0: 0.1420454978942871 (0.22999772548675537) | > loss_disc_real_1: 0.24092477560043335 (0.23516590595245362) | > loss_disc_real_2: 0.22451792657375336 (0.24321955740451812) | > loss_disc_real_3: 0.23469123244285583 (0.2417353367805481) | > loss_disc_real_4: 0.242137148976326 (0.24519217133522034) | > loss_disc_real_5: 0.2579938769340515 (0.2469542133808136) | > loss_0: 2.8274426460266113 (2.8599162006378176) | > grad_norm_0: tensor(11.0699, device='cuda:0') (tensor(9.3645, device='cuda:0')) | > loss_gen: 1.6936757564544678 (1.7228529119491578) | > loss_kl: 2.4821853637695312 (2.410423984527588) | > loss_feat: 1.8881858587265015 (1.7513498306274413) | > loss_mel: 17.9317569732666 (17.93092491149902) | > loss_duration: 1.62685227394104 (1.6458809661865235) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.622655868530273 (25.461432571411134) | > grad_norm_1: tensor(159.8380, device='cuda:0') (tensor(181.0744, device='cuda:0')) | > current_lr_0: 0.0001977130323314455 | > current_lr_1: 0.0001977130323314455 | > step_time: 0.4355 (0.43032413482666015) | > loader_time: 0.0071 (0.006814556121826172) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577219197447871 (-0.00017716851032955477) | > avg_step_time: 0.44885055112167144 (-0.0033025036395435414) | > avg_loss_disc: 2.8454757878478145 (-0.014641644249499652) | > avg_loss_disc_real_0: 0.2330808847303122 (+0.0017930702634260665) | > avg_loss_disc_real_1: 0.23895351987489513 (+3.683126308548501e-05) | > avg_loss_disc_real_2: 0.24152690591946455 (+0.0004096929456146592) | > avg_loss_disc_real_3: 0.24091832465688948 (-0.0028761929609406045) | > avg_loss_disc_real_4: 0.2458768944505235 (-0.0007932152546627214) | > avg_loss_disc_real_5: 0.2473212369311024 (+0.0004297490690795547) | > avg_loss_0: 2.8454757878478145 (-0.014641644249499652) | > avg_grad_norm_0: tensor(9.2595, device='cuda:0') (tensor(-0.0881, device='cuda:0')) | > avg_loss_gen: 1.7472589469291795 (+0.0164381500700832) | > avg_loss_kl: 2.395110190754206 (-0.014999094143719471) | > avg_loss_feat: 1.7766044223812265 (+0.06386636680280655) | > avg_loss_mel: 17.9309247110931 (-0.2152657441689918) | > avg_loss_duration: 1.6646953686861925 (-0.014039224302264897) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.514593715399084 (-0.1639995037669877) | > avg_grad_norm_1: tensor(225.2302, device='cuda:0') (tensor(-7.3191, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1006697.pth  > EPOCH: 93/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:08:32)   --> TIME: 2025-02-25 09:08:35 -- STEP: 3/72 -- GLOBAL_STEP: 1006700 | > loss_disc: 2.8367953300476074 (2.8063789208730063) | > loss_disc_real_0: 0.19466467201709747 (0.21596665680408478) | > loss_disc_real_1: 0.2220536470413208 (0.22947006424268088) | > loss_disc_real_2: 0.2399943619966507 (0.23755035797754923) | > loss_disc_real_3: 0.237724170088768 (0.2311839759349823) | > loss_disc_real_4: 0.22815214097499847 (0.22337565819422403) | > loss_disc_real_5: 0.22781401872634888 (0.24018486340840658) | > loss_0: 2.8367953300476074 (2.8063789208730063) | > grad_norm_0: tensor(4.0722, device='cuda:0') (tensor(5.0585, device='cuda:0')) | > loss_gen: 1.7885655164718628 (1.7440050840377808) | > loss_kl: 2.4285025596618652 (2.441890239715576) | > loss_feat: 1.7339575290679932 (1.8061317602793376) | > loss_mel: 17.54124641418457 (18.37698809305827) | > loss_duration: 1.6628772020339966 (1.663833498954773) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.155149459838867 (26.03284962972005) | > grad_norm_1: tensor(269.1956, device='cuda:0') (tensor(450.3511, device='cuda:0')) | > current_lr_0: 0.00019768831820240408 | > current_lr_1: 0.00019768831820240408 | > step_time: 0.4306 (0.4376231829325358) | > loader_time: 0.0051 (0.006353934605916341)  --> TIME: 2025-02-25 09:08:58 -- STEP: 53/72 -- GLOBAL_STEP: 1006750 | > loss_disc: 2.846574306488037 (2.8428224257703096) | > loss_disc_real_0: 0.2150961458683014 (0.23093452178082377) | > loss_disc_real_1: 0.2926884591579437 (0.2367954338496586) | > loss_disc_real_2: 0.2568556070327759 (0.2388865902738751) | > loss_disc_real_3: 0.24152976274490356 (0.24310872268001987) | > loss_disc_real_4: 0.2339305430650711 (0.2461618314374168) | > loss_disc_real_5: 0.23761236667633057 (0.24497585785838794) | > loss_0: 2.846574306488037 (2.8428224257703096) | > grad_norm_0: tensor(5.0923, device='cuda:0') (tensor(8.9066, device='cuda:0')) | > loss_gen: 1.7958133220672607 (1.7454660838505007) | > loss_kl: 2.407600164413452 (2.4061545920821863) | > loss_feat: 1.7982659339904785 (1.808483049554645) | > loss_mel: 18.06688690185547 (18.243879426200426) | > loss_duration: 1.712228536605835 (1.6561295783744667) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.780794143676758 (25.860112640092957) | > grad_norm_1: tensor(261.2036, device='cuda:0') (tensor(278.1767, device='cuda:0')) | > current_lr_0: 0.00019768831820240408 | > current_lr_1: 0.00019768831820240408 | > step_time: 0.4494 (0.44338099011835064) | > loader_time: 0.009 (0.008024984935544571) --> EVAL PERFORMANCE | > avg_loader_time: 0.008722140755451897 (+0.0001449215580040259) | > avg_step_time: 0.4624447621090311 (+0.01359421098735969) | > avg_loss_disc: 2.8467465756644663 (+0.0012707878166517972) | > avg_loss_disc_real_0: 0.23057640522298678 (-0.002504479507325408) | > avg_loss_disc_real_1: 0.23659045511568097 (-0.002363064759214162) | > avg_loss_disc_real_2: 0.24164469603081823 (+0.00011779011135368278) | > avg_loss_disc_real_3: 0.24240049216109263 (+0.001482167504203158) | > avg_loss_disc_real_4: 0.2459575309300087 (+8.063647948519659e-05) | > avg_loss_disc_real_5: 0.24572619732836604 (-0.0015950396027363534) | > avg_loss_0: 2.8467465756644663 (+0.0012707878166517972) | > avg_grad_norm_0: tensor(9.0253, device='cuda:0') (tensor(-0.2342, device='cuda:0')) | > avg_loss_gen: 1.7361758964162477 (-0.011083050512931791) | > avg_loss_kl: 2.3879541376946687 (-0.007156053059537371) | > avg_loss_feat: 1.7827810821398882 (+0.006176659758661662) | > avg_loss_mel: 18.132516592321256 (+0.20159188122815763) | > avg_loss_duration: 1.6633330600362428 (-0.001362308649949684) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.702760669547068 (+0.18816695414798446) | > avg_grad_norm_1: tensor(286.7740, device='cuda:0') (+tensor(61.5438, device='cuda:0'))  > EPOCH: 94/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:09:09)   --> TIME: 2025-02-25 09:09:25 -- STEP: 31/72 -- GLOBAL_STEP: 1006800 | > loss_disc: 2.8985135555267334 (2.8390608449136057) | > loss_disc_real_0: 0.3355867564678192 (0.22906597343183333) | > loss_disc_real_1: 0.2344847321510315 (0.23761290456018141) | > loss_disc_real_2: 0.2637145519256592 (0.2412447621745448) | > loss_disc_real_3: 0.26698336005210876 (0.2425222747748898) | > loss_disc_real_4: 0.2599111795425415 (0.2435181136092832) | > loss_disc_real_5: 0.27140989899635315 (0.24423762098435434) | > loss_0: 2.8985135555267334 (2.8390608449136057) | > grad_norm_0: tensor(11.7400, device='cuda:0') (tensor(9.0351, device='cuda:0')) | > loss_gen: 1.836849331855774 (1.7441184328448387) | > loss_kl: 2.4690277576446533 (2.3537788006567184) | > loss_feat: 1.5864038467407227 (1.8038962848724858) | > loss_mel: 18.240737915039062 (18.149824942311934) | > loss_duration: 1.6816942691802979 (1.641942447231662) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.814712524414062 (25.693560815626576) | > grad_norm_1: tensor(279.7000, device='cuda:0') (tensor(274.9311, device='cuda:0')) | > current_lr_0: 0.00019766360716262876 | > current_lr_1: 0.00019766360716262876 | > step_time: 0.4383 (0.43319436042539533) | > loader_time: 0.0089 (0.00670063111089891) --> EVAL PERFORMANCE | > avg_loader_time: 0.008530143281103858 (-0.00019199747434803932) | > avg_step_time: 0.45175812277995364 (-0.010686639329077485) | > avg_loss_disc: 2.8449327844968986 (-0.0018137911675677465) | > avg_loss_disc_real_0: 0.23155355159665497 (+0.0009771463736681918) | > avg_loss_disc_real_1: 0.23621763414900068 (-0.0003728209666802873) | > avg_loss_disc_real_2: 0.23996328279166154 (-0.0016814132391566872) | > avg_loss_disc_real_3: 0.2410225316252507 (-0.0013779605358419378) | > avg_loss_disc_real_4: 0.24444571976930324 (-0.0015118111607054507) | > avg_loss_disc_real_5: 0.2449525004961121 (-0.0007736968322539228) | > avg_loss_0: 2.8449327844968986 (-0.0018137911675677465) | > avg_grad_norm_0: tensor(9.3399, device='cuda:0') (+tensor(0.3145, device='cuda:0')) | > avg_loss_gen: 1.7298745756417933 (-0.006301320774454444) | > avg_loss_kl: 2.3840578106087698 (-0.003896327085898932) | > avg_loss_feat: 1.7952363004147167 (+0.012455218274828539) | > avg_loss_mel: 18.157969998641754 (+0.025453406320497862) | > avg_loss_duration: 1.6658587136738736 (+0.002525653637630798) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.732997168957347 (+0.03023649941027884) | > avg_grad_norm_1: tensor(248.7052, device='cuda:0') (tensor(-38.0687, device='cuda:0'))  > EPOCH: 95/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:09:46)   --> TIME: 2025-02-25 09:09:52 -- STEP: 9/72 -- GLOBAL_STEP: 1006850 | > loss_disc: 2.892669916152954 (2.8752741283840604) | > loss_disc_real_0: 0.21708309650421143 (0.2615746044450336) | > loss_disc_real_1: 0.24253129959106445 (0.2466398643122779) | > loss_disc_real_2: 0.19138957560062408 (0.23543846110502878) | > loss_disc_real_3: 0.20450440049171448 (0.244712480240398) | > loss_disc_real_4: 0.2548876106739044 (0.24019917514589098) | > loss_disc_real_5: 0.2555510401725769 (0.23660814762115479) | > loss_0: 2.892669916152954 (2.8752741283840604) | > grad_norm_0: tensor(6.3658, device='cuda:0') (tensor(11.6261, device='cuda:0')) | > loss_gen: 1.6023749113082886 (1.724878986676534) | > loss_kl: 2.460114002227783 (2.4189844131469727) | > loss_feat: 1.8752925395965576 (1.6887321339713202) | > loss_mel: 19.30060386657715 (18.058408313327366) | > loss_duration: 1.645197868347168 (1.648135119014316) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.883583068847656 (25.539138582017685) | > grad_norm_1: tensor(67.7587, device='cuda:0') (tensor(207.9868, device='cuda:0')) | > current_lr_0: 0.00019763889921173343 | > current_lr_1: 0.00019763889921173343 | > step_time: 0.4192 (0.43000859684414333) | > loader_time: 0.0055 (0.006274806128607856)  --> TIME: 2025-02-25 09:10:15 -- STEP: 59/72 -- GLOBAL_STEP: 1006900 | > loss_disc: 2.7553534507751465 (2.835366685511702) | > loss_disc_real_0: 0.18585681915283203 (0.2324216999744965) | > loss_disc_real_1: 0.2476375848054886 (0.23611663888066503) | > loss_disc_real_2: 0.24573974311351776 (0.24062686097823968) | > loss_disc_real_3: 0.2772822678089142 (0.2414062010534739) | > loss_disc_real_4: 0.26988962292671204 (0.24400596775240818) | > loss_disc_real_5: 0.2764981687068939 (0.2443834957429918) | > loss_0: 2.7553534507751465 (2.835366685511702) | > grad_norm_0: tensor(5.3082, device='cuda:0') (tensor(9.6949, device='cuda:0')) | > loss_gen: 1.865618348121643 (1.7534478761382022) | > loss_kl: 2.2684638500213623 (2.401330972121934) | > loss_feat: 1.7652645111083984 (1.8352544934062633) | > loss_mel: 18.037229537963867 (18.055635710894055) | > loss_duration: 1.6821601390838623 (1.6587785381381794) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.61873435974121 (25.704447406833456) | > grad_norm_1: tensor(81.6665, device='cuda:0') (tensor(211.3747, device='cuda:0')) | > current_lr_0: 0.00019763889921173343 | > current_lr_1: 0.00019763889921173343 | > step_time: 0.4694 (0.4461076259613037) | > loader_time: 0.0099 (0.008189738807031664) --> EVAL PERFORMANCE | > avg_loader_time: 0.008720921798491143 (+0.00019077851738728488) | > avg_step_time: 0.45346606953043334 (+0.0017079467504796941) | > avg_loss_disc: 2.837461851012539 (-0.007470933484359676) | > avg_loss_disc_real_0: 0.22911332684083724 (-0.0024402247558177326) | > avg_loss_disc_real_1: 0.23517805715681803 (-0.0010395769921826559) | > avg_loss_disc_real_2: 0.23932227387394703 (-0.000641008917714514) | > avg_loss_disc_real_3: 0.24059642011850652 (-0.0004261115067441812) | > avg_loss_disc_real_4: 0.24401527580241084 (-0.00043044396689240205) | > avg_loss_disc_real_5: 0.24470106438851694 (-0.00025143610759517654) | > avg_loss_0: 2.837461851012539 (-0.007470933484359676) | > avg_grad_norm_0: tensor(9.8117, device='cuda:0') (+tensor(0.4718, device='cuda:0')) | > avg_loss_gen: 1.7474645879906667 (+0.017590012348873474) | > avg_loss_kl: 2.4035563301032696 (+0.019498519494499877) | > avg_loss_feat: 1.8321851122547204 (+0.03694881184000365) | > avg_loss_mel: 18.03970745247854 (-0.11826254616321208) | > avg_loss_duration: 1.6639474395295264 (-0.0019112741443472192) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68686079643142 (-0.046136372525925395) | > avg_grad_norm_1: tensor(207.6618, device='cuda:0') (tensor(-41.0434, device='cuda:0'))  > EPOCH: 96/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:10:23)   --> TIME: 2025-02-25 09:10:41 -- STEP: 37/72 -- GLOBAL_STEP: 1006950 | > loss_disc: 2.8688712120056152 (2.843111921001125) | > loss_disc_real_0: 0.22090743482112885 (0.23408113480419726) | > loss_disc_real_1: 0.23979586362838745 (0.23713008374781222) | > loss_disc_real_2: 0.23762498795986176 (0.23916640555536425) | > loss_disc_real_3: 0.29993945360183716 (0.2445331111147597) | > loss_disc_real_4: 0.26724886894226074 (0.24486500628896662) | > loss_disc_real_5: 0.29528436064720154 (0.24817387719412107) | > loss_0: 2.8688712120056152 (2.843111921001125) | > grad_norm_0: tensor(4.3237, device='cuda:0') (tensor(8.9973, device='cuda:0')) | > loss_gen: 1.8229628801345825 (1.7528375387191772) | > loss_kl: 2.4020400047302246 (2.4256414013939933) | > loss_feat: 1.6297051906585693 (1.7923862257519283) | > loss_mel: 17.57550621032715 (18.290893760887354) | > loss_duration: 1.6861450672149658 (1.6469355853828225) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.11635971069336 (25.908694653897673) | > grad_norm_1: tensor(293.0597, device='cuda:0') (tensor(221.7886, device='cuda:0')) | > current_lr_0: 0.00019761419434933197 | > current_lr_1: 0.00019761419434933197 | > step_time: 0.4437 (0.4356011828860721) | > loader_time: 0.0083 (0.0074168282586175045) --> EVAL PERFORMANCE | > avg_loader_time: 0.008675961427285635 (-4.496037120550744e-05) | > avg_step_time: 0.4517632471004003 (-0.0017028224300330574) | > avg_loss_disc: 2.8428470047426897 (+0.0053851537301508046) | > avg_loss_disc_real_0: 0.23257285839235278 (+0.0034595315515155434) | > avg_loss_disc_real_1: 0.23584668422249003 (+0.0006686270656720006) | > avg_loss_disc_real_2: 0.24101419641938007 (+0.0016919225454330444) | > avg_loss_disc_real_3: 0.24190733571287612 (+0.001310915594369605) | > avg_loss_disc_real_4: 0.2443800818332484 (+0.0003648060308375778) | > avg_loss_disc_real_5: 0.2454557949808282 (+0.0007547305923112646) | > avg_loss_0: 2.8428470047426897 (+0.0053851537301508046) | > avg_grad_norm_0: tensor(9.1927, device='cuda:0') (tensor(-0.6190, device='cuda:0')) | > avg_loss_gen: 1.7479556312023754 (+0.0004910432117086749) | > avg_loss_kl: 2.40049899799723 (-0.0030573321060396452) | > avg_loss_feat: 1.7888586571518803 (-0.043326455102840056) | > avg_loss_mel: 18.247278401549433 (+0.20757094907089169) | > avg_loss_duration: 1.6650676878405288 (+0.0011202483110024364) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.849659530209824 (+0.16279873377840204) | > avg_grad_norm_1: tensor(233.2506, device='cuda:0') (+tensor(25.5888, device='cuda:0'))  > EPOCH: 97/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:10:59)   --> TIME: 2025-02-25 09:11:08 -- STEP: 15/72 -- GLOBAL_STEP: 1007000 | > loss_disc: 2.8938674926757812 (2.826928981145223) | > loss_disc_real_0: 0.2119663655757904 (0.22602806190649669) | > loss_disc_real_1: 0.210460364818573 (0.2364251305659612) | > loss_disc_real_2: 0.2578149735927582 (0.2348844329516093) | > loss_disc_real_3: 0.22701798379421234 (0.2398663212855657) | > loss_disc_real_4: 0.25966474413871765 (0.24332582155863444) | > loss_disc_real_5: 0.2730017602443695 (0.24354623953501384) | > loss_0: 2.8938674926757812 (2.826928981145223) | > grad_norm_0: tensor(7.6621, device='cuda:0') (tensor(9.1606, device='cuda:0')) | > loss_gen: 1.6426771879196167 (1.7303386847178142) | > loss_kl: 2.4135079383850098 (2.3973111788431805) | > loss_feat: 1.6746193170547485 (1.822993262608846) | > loss_mel: 19.252784729003906 (18.65646947224935) | > loss_duration: 1.6388096809387207 (1.6615913073221842) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.622398376464844 (26.268704223632813) | > grad_norm_1: tensor(177.3422, device='cuda:0') (tensor(237.9268, device='cuda:0')) | > current_lr_0: 0.0001975894925750383 | > current_lr_1: 0.0001975894925750383 | > step_time: 0.4368 (0.4360663890838623) | > loader_time: 0.0058 (0.006370671590169271)  --> TIME: 2025-02-25 09:11:32 -- STEP: 65/72 -- GLOBAL_STEP: 1007050 | > loss_disc: 2.790252685546875 (2.840381915752704) | > loss_disc_real_0: 0.20105916261672974 (0.2307037245768767) | > loss_disc_real_1: 0.23580436408519745 (0.2377942871588927) | > loss_disc_real_2: 0.2451656311750412 (0.23865460661741403) | > loss_disc_real_3: 0.2621800899505615 (0.24223673802155715) | > loss_disc_real_4: 0.2545139193534851 (0.24498347915135898) | > loss_disc_real_5: 0.23012188076972961 (0.24619783392319314) | > loss_0: 2.790252685546875 (2.840381915752704) | > grad_norm_0: tensor(6.1809, device='cuda:0') (tensor(8.6530, device='cuda:0')) | > loss_gen: 1.8153076171875 (1.7438565529309786) | > loss_kl: 2.380336284637451 (2.3728033982790433) | > loss_feat: 1.9162089824676514 (1.8206022079174335) | > loss_mel: 18.83551597595215 (18.500929612379807) | > loss_duration: 1.697409987449646 (1.6715445811931904) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.644779205322266 (26.10973645723783) | > grad_norm_1: tensor(67.4119, device='cuda:0') (tensor(228.1180, device='cuda:0')) | > current_lr_0: 0.0001975894925750383 | > current_lr_1: 0.0001975894925750383 | > step_time: 0.4986 (0.45002280748807466) | > loader_time: 0.0118 (0.00830127275907076) --> EVAL PERFORMANCE | > avg_loader_time: 0.008570110294180854 (-0.00010585113310478123) | > avg_step_time: 0.45565544719427403 (+0.0038922000938737544) | > avg_loss_disc: 2.843399531404737 (+0.0005525266620471925) | > avg_loss_disc_real_0: 0.22960002405542723 (-0.0029728343369255505) | > avg_loss_disc_real_1: 0.23787932538650405 (+0.0020326411640140263) | > avg_loss_disc_real_2: 0.2386679508736436 (-0.0023462455457364806) | > avg_loss_disc_real_3: 0.24097493983490367 (-0.0009323958779724473) | > avg_loss_disc_real_4: 0.24548503798498233 (+0.0011049561517339213) | > avg_loss_disc_real_5: 0.2482771982609386 (+0.002821403280110407) | > avg_loss_0: 2.843399531404737 (+0.0005525266620471925) | > avg_grad_norm_0: tensor(9.4130, device='cuda:0') (+tensor(0.2203, device='cuda:0')) | > avg_loss_gen: 1.7410573086268466 (-0.006898322575528804) | > avg_loss_kl: 2.3744717550949312 (-0.02602724290229874) | > avg_loss_feat: 1.8143460498729223 (+0.025487392721041946) | > avg_loss_mel: 18.459961098684392 (+0.2126826971349587) | > avg_loss_duration: 1.673531090709525 (+0.008463402868996273) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.063367360074754 (+0.21370782986492998) | > avg_grad_norm_1: tensor(232.8707, device='cuda:0') (tensor(-0.3799, device='cuda:0'))  > EPOCH: 98/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:11:36)   --> TIME: 2025-02-25 09:11:58 -- STEP: 43/72 -- GLOBAL_STEP: 1007100 | > loss_disc: 2.90559458732605 (2.8422831712767134) | > loss_disc_real_0: 0.26194947957992554 (0.2245746837105862) | > loss_disc_real_1: 0.21439257264137268 (0.23159696750862654) | > loss_disc_real_2: 0.2498631626367569 (0.23851790261823078) | > loss_disc_real_3: 0.250087171792984 (0.2420199589673863) | > loss_disc_real_4: 0.240674689412117 (0.24491920270199) | > loss_disc_real_5: 0.2523328959941864 (0.24853708439095076) | > loss_0: 2.90559458732605 (2.8422831712767134) | > grad_norm_0: tensor(11.5058, device='cuda:0') (tensor(11.6345, device='cuda:0')) | > loss_gen: 1.6270594596862793 (1.7236547026523323) | > loss_kl: 2.2883377075195312 (2.491924363513325) | > loss_feat: 1.4845948219299316 (1.841895347417787) | > loss_mel: 18.055747985839844 (18.009279339812522) | > loss_duration: 1.669986605644226 (1.6539397101069606) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.1257266998291 (25.720693765684615) | > grad_norm_1: tensor(243.5566, device='cuda:0') (tensor(219.4954, device='cuda:0')) | > current_lr_0: 0.0001975647938884664 | > current_lr_1: 0.0001975647938884664 | > step_time: 0.4443 (0.438190171884936) | > loader_time: 0.0093 (0.007677139237869618) --> EVAL PERFORMANCE | > avg_loader_time: 0.008736059699260014 (+0.00016594940507915976) | > avg_step_time: 0.4528855001422721 (-0.002769947052001953) | > avg_loss_disc: 2.8565252834642436 (+0.013125752059506723) | > avg_loss_disc_real_0: 0.2263391336924593 (-0.003260890362967922) | > avg_loss_disc_real_1: 0.23354977012520106 (-0.004329555261302992) | > avg_loss_disc_real_2: 0.24125480420992407 (+0.0025868533362804758) | > avg_loss_disc_real_3: 0.24580359291022932 (+0.004828653075325651) | > avg_loss_disc_real_4: 0.2466210580207932 (+0.0011360200358108696) | > avg_loss_disc_real_5: 0.248907498280767 (+0.0006303000198283892) | > avg_loss_0: 2.8565252834642436 (+0.013125752059506723) | > avg_grad_norm_0: tensor(11.2558, device='cuda:0') (+tensor(1.8428, device='cuda:0')) | > avg_loss_gen: 1.7235490839246292 (-0.017508224702217356) | > avg_loss_kl: 2.4719751922177595 (+0.09750343712282827) | > avg_loss_feat: 1.7854540045832243 (-0.02889204528969791) | > avg_loss_mel: 18.11053149800905 (-0.349429600675343) | > avg_loss_duration: 1.6664663972988936 (-0.0070646934106315395) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.757976290205836 (-0.30539106986891795) | > avg_grad_norm_1: tensor(238.2716, device='cuda:0') (+tensor(5.4009, device='cuda:0'))  > EPOCH: 99/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:12:13)   --> TIME: 2025-02-25 09:12:24 -- STEP: 21/72 -- GLOBAL_STEP: 1007150 | > loss_disc: 2.8524246215820312 (2.834554978779384) | > loss_disc_real_0: 0.2061687558889389 (0.23294606450058164) | > loss_disc_real_1: 0.2931002080440521 (0.22785461374691554) | > loss_disc_real_2: 0.21958033740520477 (0.23475337738082522) | > loss_disc_real_3: 0.23738320171833038 (0.23951083776496707) | > loss_disc_real_4: 0.27166032791137695 (0.25006349100953057) | > loss_disc_real_5: 0.2498731017112732 (0.24640616987432754) | > loss_0: 2.8524246215820312 (2.834554978779384) | > grad_norm_0: tensor(5.7035, device='cuda:0') (tensor(9.6345, device='cuda:0')) | > loss_gen: 1.7755186557769775 (1.7582676297142392) | > loss_kl: 2.444927453994751 (2.369567042305356) | > loss_feat: 1.7122039794921875 (1.8887797650836764) | > loss_mel: 17.273435592651367 (18.281419299897692) | > loss_duration: 1.691765546798706 (1.6440935645784651) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.897850036621094 (25.942127136957076) | > grad_norm_1: tensor(85.1113, device='cuda:0') (tensor(228.1577, device='cuda:0')) | > current_lr_0: 0.00019754009828923033 | > current_lr_1: 0.00019754009828923033 | > step_time: 0.4356 (0.43337660744076684) | > loader_time: 0.0072 (0.00678933234441848)  --> TIME: 2025-02-25 09:12:48 -- STEP: 71/72 -- GLOBAL_STEP: 1007200 | > loss_disc: 2.8531057834625244 (2.8485229854852383) | > loss_disc_real_0: 0.28748002648353577 (0.23352243929681643) | > loss_disc_real_1: 0.28929802775382996 (0.23454856935521246) | > loss_disc_real_2: 0.2309664785861969 (0.23719295691436446) | > loss_disc_real_3: 0.24078072607517242 (0.24182128213660817) | > loss_disc_real_4: 0.2560441792011261 (0.24663994702654826) | > loss_disc_real_5: 0.2868243157863617 (0.24710354754622554) | > loss_0: 2.8531057834625244 (2.8485229854852383) | > grad_norm_0: tensor(13.0257, device='cuda:0') (tensor(9.9222, device='cuda:0')) | > loss_gen: 1.7380576133728027 (1.7452263378761184) | > loss_kl: 2.361119508743286 (2.4339760793766496) | > loss_feat: 1.4999747276306152 (1.8194151310853555) | > loss_mel: 16.93572235107422 (18.33741746821874) | > loss_duration: 1.6862624883651733 (1.666449335259451) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.22113609313965 (26.00248422757001) | > grad_norm_1: tensor(80.9031, device='cuda:0') (tensor(189.6463, device='cuda:0')) | > current_lr_0: 0.00019754009828923033 | > current_lr_1: 0.00019754009828923033 | > step_time: 0.4937 (0.4530146726420228) | > loader_time: 0.0079 (0.008605235059496385) --> EVAL PERFORMANCE | > avg_loader_time: 0.008605235059496385 (-0.0001308246397636291) | > avg_step_time: 0.4530146726420228 (+0.000129172499750696) | > avg_loss_disc: 2.8485229854852383 (-0.008002297979005313) | > avg_loss_disc_real_0: 0.23352243929681643 (+0.007183305604357121) | > avg_loss_disc_real_1: 0.23454856935521246 (+0.0009987992300113935) | > avg_loss_disc_real_2: 0.23719295691436446 (-0.004061847295559612) | > avg_loss_disc_real_3: 0.24182128213660817 (-0.003982310773621156) | > avg_loss_disc_real_4: 0.24663994702654826 (+1.8889005755057253e-05) | > avg_loss_disc_real_5: 0.24710354754622554 (-0.001803950734541454) | > avg_loss_0: 2.8485229854852383 (-0.008002297979005313) | > avg_grad_norm_0: tensor(9.9222, device='cuda:0') (tensor(-1.3336, device='cuda:0')) | > avg_loss_gen: 1.7452263378761184 (+0.021677253951489117) | > avg_loss_kl: 2.4339760793766496 (-0.037999112841109906) | > avg_loss_feat: 1.8194151310853555 (+0.03396112650213112) | > avg_loss_mel: 18.33741746821874 (+0.2268859702096897) | > avg_loss_duration: 1.666449335259451 (-1.7062039442627963e-05) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.00248422757001 (+0.2445079373641761) | > avg_grad_norm_1: tensor(189.6463, device='cuda:0') (tensor(-48.6253, device='cuda:0'))  > EPOCH: 100/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:12:50)   --> TIME: 2025-02-25 09:13:14 -- STEP: 49/72 -- GLOBAL_STEP: 1007250 | > loss_disc: 2.823737382888794 (2.839214334682542) | > loss_disc_real_0: 0.16721117496490479 (0.22325584322822337) | > loss_disc_real_1: 0.2602999806404114 (0.2347907150278286) | > loss_disc_real_2: 0.217144176363945 (0.2413884091133974) | > loss_disc_real_3: 0.21508049964904785 (0.238876968013997) | > loss_disc_real_4: 0.23488403856754303 (0.24576900990641848) | > loss_disc_real_5: 0.2557973861694336 (0.2458621102328203) | > loss_0: 2.823737382888794 (2.839214334682542) | > grad_norm_0: tensor(9.6467, device='cuda:0') (tensor(11.1958, device='cuda:0')) | > loss_gen: 1.876655101776123 (1.7452094019675741) | > loss_kl: 2.466073513031006 (2.443119238833991) | > loss_feat: 1.8262351751327515 (1.8607412868616533) | > loss_mel: 18.982280731201172 (18.46838230989417) | > loss_duration: 1.7476279735565186 (1.6567903100227823) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.89887237548828 (26.17424248675911) | > grad_norm_1: tensor(196.3090, device='cuda:0') (tensor(238.9874, device='cuda:0')) | > current_lr_0: 0.00019751540577694416 | > current_lr_1: 0.00019751540577694416 | > step_time: 0.45 (0.44232458970984634) | > loader_time: 0.0094 (0.007754379389237384) --> EVAL PERFORMANCE | > avg_loader_time: 0.00864849627857477 (+4.326121907838494e-05) | > avg_step_time: 0.4542885733322358 (+0.0012739006902130479) | > avg_loss_disc: 2.8436963591777094 (-0.004826626307528947) | > avg_loss_disc_real_0: 0.22759023077890905 (-0.005932208517907378) | > avg_loss_disc_real_1: 0.23530582810791445 (+0.0007572587527019947) | > avg_loss_disc_real_2: 0.24196379999039877 (+0.004770843076034315) | > avg_loss_disc_real_3: 0.24110915963078888 (-0.0007121225058192848) | > avg_loss_disc_real_4: 0.24567506535792014 (-0.0009648816686281259) | > avg_loss_disc_real_5: 0.2462439348160381 (-0.0008596127301874479) | > avg_loss_0: 2.8436963591777094 (-0.004826626307528947) | > avg_grad_norm_0: tensor(10.5195, device='cuda:0') (+tensor(0.5972, device='cuda:0')) | > avg_loss_gen: 1.735730550658535 (-0.009495787217583374) | > avg_loss_kl: 2.43525339180315 (+0.001277312426500199) | > avg_loss_feat: 1.80723443837233 (-0.012180692713025465) | > avg_loss_mel: 18.355855511947414 (+0.018438043728675524) | > avg_loss_duration: 1.6642404794692993 (-0.0022088557901516204) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.998314306769572 (-0.004169920800439542) | > avg_grad_norm_1: tensor(228.1575, device='cuda:0') (+tensor(38.5111, device='cuda:0'))  > EPOCH: 101/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:13:27)   --> TIME: 2025-02-25 09:13:41 -- STEP: 27/72 -- GLOBAL_STEP: 1007300 | > loss_disc: 2.9130473136901855 (2.839447913346467) | > loss_disc_real_0: 0.16437718272209167 (0.22074398784725754) | > loss_disc_real_1: 0.2843588888645172 (0.2388601518339581) | > loss_disc_real_2: 0.24052879214286804 (0.24335202464350947) | > loss_disc_real_3: 0.2558841109275818 (0.24039436490447433) | > loss_disc_real_4: 0.2594814896583557 (0.24758682703530346) | > loss_disc_real_5: 0.25027427077293396 (0.24674482643604279) | > loss_0: 2.9130473136901855 (2.839447913346467) | > grad_norm_0: tensor(13.8868, device='cuda:0') (tensor(8.6741, device='cuda:0')) | > loss_gen: 1.6216976642608643 (1.7444869456467804) | > loss_kl: 2.6685941219329834 (2.4343695375654426) | > loss_feat: 1.60382878780365 (1.8443154405664515) | > loss_mel: 16.319990158081055 (18.282650770964445) | > loss_duration: 1.5954315662384033 (1.6382906701829698) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.80954360961914 (25.944113484135382) | > grad_norm_1: tensor(275.9817, device='cuda:0') (tensor(229.8331, device='cuda:0')) | > current_lr_0: 0.00019749071635122203 | > current_lr_1: 0.00019749071635122203 | > step_time: 0.4402 (0.4376282603652389) | > loader_time: 0.0083 (0.006901449627346463) --> EVAL PERFORMANCE | > avg_loader_time: 0.008516335151564905 (-0.00013216112700986486) | > avg_step_time: 0.45405506080305075 (-0.00023351252918507592) | > avg_loss_disc: 2.849330035733505 (+0.00563367655579583) | > avg_loss_disc_real_0: 0.22454684038817044 (-0.003043390390738615) | > avg_loss_disc_real_1: 0.23716555997519426 (+0.0018597318672798069) | > avg_loss_disc_real_2: 0.24285782705730116 (+0.000894027066902392) | > avg_loss_disc_real_3: 0.24203988049231784 (+0.0009307208615289553) | > avg_loss_disc_real_4: 0.24684574507491688 (+0.0011706797169967476) | > avg_loss_disc_real_5: 0.2479454797338432 (+0.001701544917805109) | > avg_loss_0: 2.849330035733505 (+0.00563367655579583) | > avg_grad_norm_0: tensor(9.3251, device='cuda:0') (tensor(-1.1944, device='cuda:0')) | > avg_loss_gen: 1.735814549553562 (+8.3998895026971e-05) | > avg_loss_kl: 2.37954499687947 (-0.05570839492367963) | > avg_loss_feat: 1.8058968530574315 (-0.0013375853148984707) | > avg_loss_mel: 17.970017795831374 (-0.3858377161160398) | > avg_loss_duration: 1.659783553069746 (-0.004456926399553307) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.551057788687693 (-0.44725651808187905) | > avg_grad_norm_1: tensor(222.7006, device='cuda:0') (tensor(-5.4568, device='cuda:0'))  > EPOCH: 102/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:14:04)   --> TIME: 2025-02-25 09:14:08 -- STEP: 5/72 -- GLOBAL_STEP: 1007350 | > loss_disc: 2.7876462936401367 (2.807144021987915) | > loss_disc_real_0: 0.2054121047258377 (0.2160310924053192) | > loss_disc_real_1: 0.16360267996788025 (0.2235386848449707) | > loss_disc_real_2: 0.270195335149765 (0.2293948858976364) | > loss_disc_real_3: 0.26691734790802 (0.23060463666915892) | > loss_disc_real_4: 0.24212367832660675 (0.24251160323619841) | > loss_disc_real_5: 0.27408209443092346 (0.2507777541875839) | > loss_0: 2.7876462936401367 (2.807144021987915) | > grad_norm_0: tensor(6.3718, device='cuda:0') (tensor(7.0482, device='cuda:0')) | > loss_gen: 1.8364083766937256 (1.7388893365859985) | > loss_kl: 2.1916956901550293 (2.352883529663086) | > loss_feat: 1.907353162765503 (1.8559232234954834) | > loss_mel: 18.04445457458496 (17.834932327270508) | > loss_duration: 1.6220827102661133 (1.6530301094055175) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.601993560791016 (25.43565788269043) | > grad_norm_1: tensor(134.7484, device='cuda:0') (tensor(310.5513, device='cuda:0')) | > current_lr_0: 0.00019746603001167813 | > current_lr_1: 0.00019746603001167813 | > step_time: 0.4404 (0.4325725555419922) | > loader_time: 0.0083 (0.006185340881347656)  --> TIME: 2025-02-25 09:14:31 -- STEP: 55/72 -- GLOBAL_STEP: 1007400 | > loss_disc: 2.8358118534088135 (2.843480027805675) | > loss_disc_real_0: 0.27639225125312805 (0.23068587211045352) | > loss_disc_real_1: 0.21288229525089264 (0.23533127036961643) | > loss_disc_real_2: 0.2381116896867752 (0.23916460302743045) | > loss_disc_real_3: 0.2429223358631134 (0.2400801282037388) | > loss_disc_real_4: 0.2546062469482422 (0.24587540382688697) | > loss_disc_real_5: 0.265064537525177 (0.25039361796595816) | > loss_0: 2.8358118534088135 (2.843480027805675) | > grad_norm_0: tensor(10.1980, device='cuda:0') (tensor(9.6001, device='cuda:0')) | > loss_gen: 1.81718111038208 (1.7457669128071178) | > loss_kl: 2.535944938659668 (2.4278060609644108) | > loss_feat: 1.7169570922851562 (1.8228506109931253) | > loss_mel: 18.783634185791016 (18.075966887040572) | > loss_duration: 1.7034908533096313 (1.6650612766092474) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.557209014892578 (25.73745158802379) | > grad_norm_1: tensor(320.6658, device='cuda:0') (tensor(192.2118, device='cuda:0')) | > current_lr_0: 0.00019746603001167813 | > current_lr_1: 0.00019746603001167813 | > step_time: 0.46 (0.4414457928050648) | > loader_time: 0.0094 (0.00789555636319247) --> EVAL PERFORMANCE | > avg_loader_time: 0.008622176210645219 (+0.00010584105908031417) | > avg_step_time: 0.4517888586286088 (-0.002266202174441956) | > avg_loss_disc: 2.8444476362685083 (-0.004882399464996912) | > avg_loss_disc_real_0: 0.23178553623212894 (+0.007238695843958504) | > avg_loss_disc_real_1: 0.23660522258617508 (-0.0005603373890191721) | > avg_loss_disc_real_2: 0.2382602912019676 (-0.004597535855333568) | > avg_loss_disc_real_3: 0.23974048566650336 (-0.0022993948258144825) | > avg_loss_disc_real_4: 0.24657154314115015 (-0.0002742019337667323) | > avg_loss_disc_real_5: 0.25053579412715526 (+0.00259031439331206) | > avg_loss_0: 2.8444476362685083 (-0.004882399464996912) | > avg_grad_norm_0: tensor(9.5807, device='cuda:0') (+tensor(0.2556, device='cuda:0')) | > avg_loss_gen: 1.741906681530912 (+0.006092131977350146) | > avg_loss_kl: 2.4244095809023145 (+0.04486458402284432) | > avg_loss_feat: 1.800058467287413 (-0.005838385770018606) | > avg_loss_mel: 18.18203036885866 (+0.21201257302728393) | > avg_loss_duration: 1.6695900030539068 (+0.009806449984160803) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.817994937090806 (+0.26693714840311245) | > avg_grad_norm_1: tensor(203.4392, device='cuda:0') (tensor(-19.2614, device='cuda:0'))  > EPOCH: 103/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:14:41)   --> TIME: 2025-02-25 09:14:58 -- STEP: 33/72 -- GLOBAL_STEP: 1007450 | > loss_disc: 2.8544790744781494 (2.858577793294733) | > loss_disc_real_0: 0.15998345613479614 (0.22927260647217432) | > loss_disc_real_1: 0.2641802430152893 (0.23187588832595132) | > loss_disc_real_2: 0.20875151455402374 (0.24021179296753623) | > loss_disc_real_3: 0.22348318994045258 (0.24097912781166309) | > loss_disc_real_4: 0.20963580906391144 (0.24801855421427524) | > loss_disc_real_5: 0.21699146926403046 (0.2478064849521174) | > loss_0: 2.8544790744781494 (2.858577793294733) | > grad_norm_0: tensor(8.9274, device='cuda:0') (tensor(9.2810, device='cuda:0')) | > loss_gen: 1.750420331954956 (1.7334271416519627) | > loss_kl: 2.417335271835327 (2.4565224358529756) | > loss_feat: 1.905555009841919 (1.8400613394650547) | > loss_mel: 18.059268951416016 (18.069743185332328) | > loss_duration: 1.6205530166625977 (1.6349279374787302) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.753131866455078 (25.734681967532996) | > grad_norm_1: tensor(138.8483, device='cuda:0') (tensor(173.9261, device='cuda:0')) | > current_lr_0: 0.00019744134675792665 | > current_lr_1: 0.00019744134675792665 | > step_time: 0.4395 (0.4401511929251931) | > loader_time: 0.0077 (0.0070432605165423766) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534216544997524 (-8.795966564769525e-05) | > avg_step_time: 0.45467884775618433 (+0.002889989127575543) | > avg_loss_disc: 2.860469955793569 (+0.016022319525060613) | > avg_loss_disc_real_0: 0.23413856988641576 (+0.0023530336542868235) | > avg_loss_disc_real_1: 0.2343029109105258 (-0.002302311675649288) | > avg_loss_disc_real_2: 0.24191395749508496 (+0.0036536662931173636) | > avg_loss_disc_real_3: 0.24317574941776168 (+0.0034352637512583273) | > avg_loss_disc_real_4: 0.24730040300899828 (+0.0007288598678481251) | > avg_loss_disc_real_5: 0.248378917365007 (-0.002156876762148263) | > avg_loss_0: 2.860469955793569 (+0.016022319525060613) | > avg_grad_norm_0: tensor(9.7641, device='cuda:0') (+tensor(0.1834, device='cuda:0')) | > avg_loss_gen: 1.7344495914351772 (-0.0074570900957349195) | > avg_loss_kl: 2.44023829446712 (+0.01582871356480542) | > avg_loss_feat: 1.7767246189251753 (-0.02333384836223762) | > avg_loss_mel: 17.937352516281766 (-0.244677852576892) | > avg_loss_duration: 1.659590502859841 (-0.009999500194065813) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.54835550550004 (-0.26963943159076464) | > avg_grad_norm_1: tensor(189.3575, device='cuda:0') (tensor(-14.0817, device='cuda:0'))  > EPOCH: 104/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:15:18)   --> TIME: 2025-02-25 09:15:24 -- STEP: 11/72 -- GLOBAL_STEP: 1007500 | > loss_disc: 2.8669273853302 (2.8435234373266045) | > loss_disc_real_0: 0.2657109200954437 (0.23328714885494925) | > loss_disc_real_1: 0.17984087765216827 (0.24402513558214362) | > loss_disc_real_2: 0.2876445949077606 (0.24051441929557107) | > loss_disc_real_3: 0.21357019245624542 (0.24002885005690835) | > loss_disc_real_4: 0.24179551005363464 (0.23778190531513907) | > loss_disc_real_5: 0.24409635365009308 (0.23762290179729462) | > loss_0: 2.8669273853302 (2.8435234373266045) | > grad_norm_0: tensor(8.0176, device='cuda:0') (tensor(10.0650, device='cuda:0')) | > loss_gen: 1.7640044689178467 (1.7394300265745684) | > loss_kl: 2.5514864921569824 (2.391209689053622) | > loss_feat: 1.92684006690979 (1.8513339974663474) | > loss_mel: 18.010841369628906 (18.01348079334606) | > loss_duration: 1.6136125326156616 (1.6406025994907727) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.866785049438477 (25.636057593605734) | > grad_norm_1: tensor(270.2138, device='cuda:0') (tensor(270.3638, device='cuda:0')) | > current_lr_0: 0.0001974166665895819 | > current_lr_1: 0.0001974166665895819 | > step_time: 0.4205 (0.4293569868261164) | > loader_time: 0.0056 (0.005980751731178977)  --> TIME: 2025-02-25 09:15:48 -- STEP: 61/72 -- GLOBAL_STEP: 1007550 | > loss_disc: 2.837066888809204 (2.8421464435389785) | > loss_disc_real_0: 0.20827746391296387 (0.22294951706636149) | > loss_disc_real_1: 0.24826978147029877 (0.23922080925253572) | > loss_disc_real_2: 0.20882844924926758 (0.24060768011163491) | > loss_disc_real_3: 0.21605843305587769 (0.24360844908190554) | > loss_disc_real_4: 0.25853294134140015 (0.24749178598161603) | > loss_disc_real_5: 0.271754652261734 (0.2489130333310268) | > loss_0: 2.837066888809204 (2.8421464435389785) | > grad_norm_0: tensor(4.8941, device='cuda:0') (tensor(9.8465, device='cuda:0')) | > loss_gen: 1.7650128602981567 (1.7547430171341192) | > loss_kl: 2.431561231613159 (2.4072728704233644) | > loss_feat: 1.7889947891235352 (1.8978632121789651) | > loss_mel: 17.63347816467285 (18.144542881699866) | > loss_duration: 1.6936476230621338 (1.6551161887215788) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.312694549560547 (25.859538312818184) | > grad_norm_1: tensor(185.7710, device='cuda:0') (tensor(220.4965, device='cuda:0')) | > current_lr_0: 0.0001974166665895819 | > current_lr_1: 0.0001974166665895819 | > step_time: 0.4588 (0.4447429883675497) | > loader_time: 0.0109 (0.007955844285058194) --> EVAL PERFORMANCE | > avg_loader_time: 0.008438395782255791 (-9.582076274173255e-05) | > avg_step_time: 0.45467489873859246 (-3.949017591875492e-06) | > avg_loss_disc: 2.8443479034262644 (-0.016122052367304462) | > avg_loss_disc_real_0: 0.22755096391053267 (-0.006587605975883093) | > avg_loss_disc_real_1: 0.23889093499788097 (+0.004588024087355175) | > avg_loss_disc_real_2: 0.2402386982256258 (-0.0016752592694591617) | > avg_loss_disc_real_3: 0.2442008376541272 (+0.0010250882363655178) | > avg_loss_disc_real_4: 0.24613200484866826 (-0.0011683981603300142) | > avg_loss_disc_real_5: 0.24804471978839016 (-0.0003341975766168459) | > avg_loss_0: 2.8443479034262644 (-0.016122052367304462) | > avg_grad_norm_0: tensor(9.8203, device='cuda:0') (+tensor(0.0561, device='cuda:0')) | > avg_loss_gen: 1.7500778990732115 (+0.015628307638034356) | > avg_loss_kl: 2.410270237586868 (-0.029968056880251837) | > avg_loss_feat: 1.8572701115003774 (+0.08054549257520205) | > avg_loss_mel: 18.02633399694739 (+0.08898148066562328) | > avg_loss_duration: 1.6600601740286387 (+0.0004696711687977029) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.70401250812369 (+0.15565700262364857) | > avg_grad_norm_1: tensor(213.6638, device='cuda:0') (+tensor(24.3062, device='cuda:0'))  > EPOCH: 105/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:15:55)   --> TIME: 2025-02-25 09:16:14 -- STEP: 39/72 -- GLOBAL_STEP: 1007600 | > loss_disc: 2.8024344444274902 (2.8461049581185365) | > loss_disc_real_0: 0.1536007523536682 (0.23002698100530183) | > loss_disc_real_1: 0.2104606330394745 (0.23465312520662943) | > loss_disc_real_2: 0.2579183876514435 (0.23968128936413008) | > loss_disc_real_3: 0.27902206778526306 (0.24266690206833375) | > loss_disc_real_4: 0.27571901679039 (0.24720699359209108) | > loss_disc_real_5: 0.27056610584259033 (0.2480660046522434) | > loss_0: 2.8024344444274902 (2.8461049581185365) | > grad_norm_0: tensor(9.3940, device='cuda:0') (tensor(8.3558, device='cuda:0')) | > loss_gen: 1.7897504568099976 (1.7430633459335718) | > loss_kl: 2.4756314754486084 (2.4317407669165187) | > loss_feat: 1.841991662979126 (1.841716048045036) | > loss_mel: 17.357627868652344 (18.143593030098156) | > loss_duration: 1.6514816284179688 (1.6438265519264417) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.116483688354492 (25.80393981933594) | > grad_norm_1: tensor(402.2325, device='cuda:0') (tensor(206.6382, device='cuda:0')) | > current_lr_0: 0.0001973919895062582 | > current_lr_1: 0.0001973919895062582 | > step_time: 0.4744 (0.43547707948929226) | > loader_time: 0.008 (0.007330643825041942) --> EVAL PERFORMANCE | > avg_loader_time: 0.008640702341643857 (+0.00020230655938806644) | > avg_step_time: 0.45082865970235475 (-0.0038462390362377086) | > avg_loss_disc: 2.846992449021675 (+0.002644545595410719) | > avg_loss_disc_real_0: 0.22824404357184827 (+0.0006930796613155987) | > avg_loss_disc_real_1: 0.23581665691355586 (-0.003074278084325116) | > avg_loss_disc_real_2: 0.24078652682438703 (+0.0005478285987612352) | > avg_loss_disc_real_3: 0.24077987230159867 (-0.003420965352528532) | > avg_loss_disc_real_4: 0.24741625408051718 (+0.0012842492318489163) | > avg_loss_disc_real_5: 0.24847173334007533 (+0.00042701355168517363) | > avg_loss_0: 2.846992449021675 (+0.002644545595410719) | > avg_grad_norm_0: tensor(8.4207, device='cuda:0') (tensor(-1.3995, device='cuda:0')) | > avg_loss_gen: 1.738144624401146 (-0.011933274672065508) | > avg_loss_kl: 2.3871273960865724 (-0.023142841500295663) | > avg_loss_feat: 1.825586648054526 (-0.03168346344585138) | > avg_loss_mel: 17.806628818243322 (-0.2197051787040678) | > avg_loss_duration: 1.6586344661846968 (-0.0014257078439419146) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.41612197983433 (-0.2878905282893598) | > avg_grad_norm_1: tensor(228.0280, device='cuda:0') (+tensor(14.3642, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1007633.pth  > EPOCH: 106/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:16:34)   --> TIME: 2025-02-25 09:16:43 -- STEP: 17/72 -- GLOBAL_STEP: 1007650 | > loss_disc: 2.930431365966797 (2.8200972781461826) | > loss_disc_real_0: 0.27856338024139404 (0.2180304211728713) | > loss_disc_real_1: 0.18485286831855774 (0.21779660617603974) | > loss_disc_real_2: 0.2537737488746643 (0.23350243708666632) | > loss_disc_real_3: 0.24093389511108398 (0.23845696098664226) | > loss_disc_real_4: 0.23608139157295227 (0.24469436968074126) | > loss_disc_real_5: 0.2565968334674835 (0.24333354217164657) | > loss_0: 2.930431365966797 (2.8200972781461826) | > grad_norm_0: tensor(10.0925, device='cuda:0') (tensor(11.2876, device='cuda:0')) | > loss_gen: 1.7477271556854248 (1.7287739024442785) | > loss_kl: 2.6169745922088623 (2.355369077009313) | > loss_feat: 2.031388998031616 (1.9615579142290003) | > loss_mel: 19.398773193359375 (18.327765296487247) | > loss_duration: 1.6833162307739258 (1.6315066183314604) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.478179931640625 (26.004972457885742) | > grad_norm_1: tensor(223.1129, device='cuda:0') (tensor(226.0231, device='cuda:0')) | > current_lr_0: 0.0001973673155075699 | > current_lr_1: 0.0001973673155075699 | > step_time: 0.4347 (0.43439900173860435) | > loader_time: 0.006 (0.006933717166676241)  --> TIME: 2025-02-25 09:17:07 -- STEP: 67/72 -- GLOBAL_STEP: 1007700 | > loss_disc: 2.882356882095337 (2.829213153070478) | > loss_disc_real_0: 0.16426804661750793 (0.2238285763939815) | > loss_disc_real_1: 0.23433716595172882 (0.23195294880155307) | > loss_disc_real_2: 0.24507689476013184 (0.23654807548024762) | > loss_disc_real_3: 0.23871085047721863 (0.23857185911776416) | > loss_disc_real_4: 0.2327325940132141 (0.24388691871913512) | > loss_disc_real_5: 0.2213994264602661 (0.24603439022356005) | > loss_0: 2.882356882095337 (2.829213153070478) | > grad_norm_0: tensor(11.6232, device='cuda:0') (tensor(11.4239, device='cuda:0')) | > loss_gen: 1.7739561796188354 (1.7382516594075446) | > loss_kl: 2.3664000034332275 (2.3883745385639705) | > loss_feat: 1.8301893472671509 (1.877578797625072) | > loss_mel: 18.318208694458008 (18.05661275493565) | > loss_duration: 1.7170159816741943 (1.6551833241733152) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.005769729614258 (25.716000912794428) | > grad_norm_1: tensor(354.2353, device='cuda:0') (tensor(248.4685, device='cuda:0')) | > current_lr_0: 0.0001973673155075699 | > current_lr_1: 0.0001973673155075699 | > step_time: 0.4949 (0.4513668053185762) | > loader_time: 0.0122 (0.008428231993717936) --> EVAL PERFORMANCE | > avg_loader_time: 0.00860809944045376 (-3.260290119009787e-05) | > avg_step_time: 0.45521030291705067 (+0.004381643214695918) | > avg_loss_disc: 2.8320605788432376 (-0.014931870178437556) | > avg_loss_disc_real_0: 0.22566790878772736 (-0.002576134784120915) | > avg_loss_disc_real_1: 0.23277710294219808 (-0.0030395539713577724) | > avg_loss_disc_real_2: 0.23742779690614887 (-0.003358729918238157) | > avg_loss_disc_real_3: 0.23893147308222004 (-0.001848399219378627) | > avg_loss_disc_real_4: 0.24398590792232835 (-0.003430346158188824) | > avg_loss_disc_real_5: 0.2460482697671568 (-0.0024234635729185328) | > avg_loss_0: 2.8320605788432376 (-0.014931870178437556) | > avg_grad_norm_0: tensor(11.6722, device='cuda:0') (+tensor(3.2514, device='cuda:0')) | > avg_loss_gen: 1.7369145658654226 (-0.001230058535723444) | > avg_loss_kl: 2.386414615201279 (-0.0007127808852933626) | > avg_loss_feat: 1.8630145485972014 (+0.03742790054267542) | > avg_loss_mel: 17.97258514081928 (+0.16595632257595838) | > avg_loss_duration: 1.6567185160139914 (-0.0019159501707053384) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.615647248818842 (+0.1995252689845124) | > avg_grad_norm_1: tensor(247.1932, device='cuda:0') (+tensor(19.1652, device='cuda:0'))  > EPOCH: 107/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:17:11)   --> TIME: 2025-02-25 09:17:33 -- STEP: 45/72 -- GLOBAL_STEP: 1007750 | > loss_disc: 2.8413710594177246 (2.8494567447238492) | > loss_disc_real_0: 0.12762093544006348 (0.23481813172499338) | > loss_disc_real_1: 0.23570066690444946 (0.23316698670387268) | > loss_disc_real_2: 0.24592141807079315 (0.24081867867045934) | > loss_disc_real_3: 0.26255080103874207 (0.24302663405736288) | > loss_disc_real_4: 0.2657429277896881 (0.24574284685982598) | > loss_disc_real_5: 0.28803807497024536 (0.24730823006894853) | > loss_0: 2.8413710594177246 (2.8494567447238492) | > grad_norm_0: tensor(14.5392, device='cuda:0') (tensor(9.7459, device='cuda:0')) | > loss_gen: 1.7175562381744385 (1.7432745218276977) | > loss_kl: 2.5241432189941406 (2.4205391830868193) | > loss_feat: 1.721670150756836 (1.8707411024305556) | > loss_mel: 17.517959594726562 (18.09235280354818) | > loss_duration: 1.685192346572876 (1.6419787247975668) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.166521072387695 (25.768886184692384) | > grad_norm_1: tensor(101.1369, device='cuda:0') (tensor(205.1009, device='cuda:0')) | > current_lr_0: 0.00019734264459313146 | > current_lr_1: 0.00019734264459313146 | > step_time: 0.4958 (0.4423593150244819) | > loader_time: 0.0091 (0.007435168160332573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008534508691707127 (-7.359074874663252e-05) | > avg_step_time: 0.4549156276273056 (-0.0002946752897450833) | > avg_loss_disc: 2.844775854701727 (+0.012715275858489417) | > avg_loss_disc_real_0: 0.2346844922908595 (+0.009016583503132142) | > avg_loss_disc_real_1: 0.23574947345424707 (+0.0029723705120489885) | > avg_loss_disc_real_2: 0.24084806715098905 (+0.003420270244840179) | > avg_loss_disc_real_3: 0.24280834575773966 (+0.003876872675519616) | > avg_loss_disc_real_4: 0.2452333577921693 (+0.001247449869840933) | > avg_loss_disc_real_5: 0.24671437748720948 (+0.0006661077200526833) | > avg_loss_0: 2.844775854701727 (+0.012715275858489417) | > avg_grad_norm_0: tensor(10.1120, device='cuda:0') (tensor(-1.5602, device='cuda:0')) | > avg_loss_gen: 1.7496709991508805 (+0.012756433285457947) | > avg_loss_kl: 2.426007264096972 (+0.03959264889569303) | > avg_loss_feat: 1.8474668902410587 (-0.01554765835614269) | > avg_loss_mel: 18.083644759487104 (+0.11105961866782366) | > avg_loss_duration: 1.6576116874184408 (+0.0008931714044493333) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76440136869189 (+0.14875411987304688) | > avg_grad_norm_1: tensor(218.6526, device='cuda:0') (tensor(-28.5406, device='cuda:0'))  > EPOCH: 108/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:17:48)   --> TIME: 2025-02-25 09:18:00 -- STEP: 23/72 -- GLOBAL_STEP: 1007800 | > loss_disc: 2.8159573078155518 (2.849819971167523) | > loss_disc_real_0: 0.11489056050777435 (0.21903846121352652) | > loss_disc_real_1: 0.2503643035888672 (0.22743596071782318) | > loss_disc_real_2: 0.21655771136283875 (0.23622417190800543) | > loss_disc_real_3: 0.20800042152404785 (0.24026077856188235) | > loss_disc_real_4: 0.23388463258743286 (0.2483350641053656) | > loss_disc_real_5: 0.22466182708740234 (0.2515014235092246) | > loss_0: 2.8159573078155518 (2.849819971167523) | > grad_norm_0: tensor(17.8659, device='cuda:0') (tensor(9.7046, device='cuda:0')) | > loss_gen: 1.544012427330017 (1.735317209492559) | > loss_kl: 2.370511770248413 (2.4380961293759555) | > loss_feat: 1.957208514213562 (1.8752455504044243) | > loss_mel: 17.714975357055664 (18.29243742901346) | > loss_duration: 1.6613378524780273 (1.6410664682802947) | > amp_scaler: 512.0 (422.95652173913044) | > loss_1: 25.248046875 (25.98216264144234) | > grad_norm_1: tensor(66.0955, device='cuda:0') (tensor(244.5417, device='cuda:0')) | > current_lr_0: 0.0001973179767625573 | > current_lr_1: 0.0001973179767625573 | > step_time: 0.4411 (0.4338551085928212) | > loader_time: 0.0073 (0.006532793459684953) --> EVAL PERFORMANCE | > avg_loader_time: 0.00848100890576 (-5.349978594712708e-05) | > avg_step_time: 0.4532460226139552 (-0.0016696050133503948) | > avg_loss_disc: 2.85054561118005 (+0.005769756478323007) | > avg_loss_disc_real_0: 0.22634762168770106 (-0.008336870603158436) | > avg_loss_disc_real_1: 0.23503579765977994 (-0.0007136757944671279) | > avg_loss_disc_real_2: 0.24185521229052206 (+0.001007145139533011) | > avg_loss_disc_real_3: 0.2433410492581381 (+0.0005327035003984404) | > avg_loss_disc_real_4: 0.24593995343631422 (+0.0007065956441449361) | > avg_loss_disc_real_5: 0.2473746957913251 (+0.0006603183041156147) | > avg_loss_0: 2.85054561118005 (+0.005769756478323007) | > avg_grad_norm_0: tensor(10.4674, device='cuda:0') (+tensor(0.3554, device='cuda:0')) | > avg_loss_gen: 1.7554032718631583 (+0.0057322727122777195) | > avg_loss_kl: 2.432891748320889 (+0.0068844842239168536) | > avg_loss_feat: 1.886845523202923 (+0.03937863296186439) | > avg_loss_mel: 18.15611346338837 (+0.07246870390126503) | > avg_loss_duration: 1.6602891482098001 (+0.002677460791359332) | > avg_amp_scaler: 335.32394366197184 (+79.32394366197184) | > avg_loss_1: 25.891543106294016 (+0.12714173760212688) | > avg_grad_norm_1: tensor(225.0956, device='cuda:0') (+tensor(6.4431, device='cuda:0'))  > EPOCH: 109/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:18:24)   --> TIME: 2025-02-25 09:18:27 -- STEP: 1/72 -- GLOBAL_STEP: 1007850 | > loss_disc: 2.9343760013580322 (2.9343760013580322) | > loss_disc_real_0: 0.2629437744617462 (0.2629437744617462) | > loss_disc_real_1: 0.3002200722694397 (0.3002200722694397) | > loss_disc_real_2: 0.21628569066524506 (0.21628569066524506) | > loss_disc_real_3: 0.21876762807369232 (0.21876762807369232) | > loss_disc_real_4: 0.22328251600265503 (0.22328251600265503) | > loss_disc_real_5: 0.22973254323005676 (0.22973254323005676) | > loss_0: 2.9343760013580322 (2.9343760013580322) | > grad_norm_0: tensor(7.6624, device='cuda:0') (tensor(7.6624, device='cuda:0')) | > loss_gen: 1.5716511011123657 (1.5716511011123657) | > loss_kl: 2.4250073432922363 (2.4250073432922363) | > loss_feat: 1.8564859628677368 (1.8564859628677368) | > loss_mel: 18.498414993286133 (18.498414993286133) | > loss_duration: 1.6567213535308838 (1.6567213535308838) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.00827980041504 (26.00827980041504) | > grad_norm_1: tensor(329.1075, device='cuda:0') (tensor(329.1075, device='cuda:0')) | > current_lr_0: 0.00019729331201546197 | > current_lr_1: 0.00019729331201546197 | > step_time: 0.4156 (0.41564393043518066) | > loader_time: 0.0047 (0.004715442657470703)  --> TIME: 2025-02-25 09:18:50 -- STEP: 51/72 -- GLOBAL_STEP: 1007900 | > loss_disc: 2.860442638397217 (2.8445000414754826) | > loss_disc_real_0: 0.13472649455070496 (0.23257673166546167) | > loss_disc_real_1: 0.290700763463974 (0.23463445112985723) | > loss_disc_real_2: 0.19597584009170532 (0.23984075586001077) | > loss_disc_real_3: 0.22618569433689117 (0.24189897465939617) | > loss_disc_real_4: 0.22608302533626556 (0.24273434018387513) | > loss_disc_real_5: 0.2311989665031433 (0.2454964796117708) | > loss_0: 2.860442638397217 (2.8445000414754826) | > grad_norm_0: tensor(16.4867, device='cuda:0') (tensor(11.0125, device='cuda:0')) | > loss_gen: 1.6950701475143433 (1.7433481099558812) | > loss_kl: 2.269840717315674 (2.4600358850815724) | > loss_feat: 1.9558706283569336 (1.8714287327785117) | > loss_mel: 18.291481018066406 (17.95526799968645) | > loss_duration: 1.7086482048034668 (1.6540989852419086) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.92091178894043 (25.68417956782322) | > grad_norm_1: tensor(233.2939, device='cuda:0') (tensor(242.6102, device='cuda:0')) | > current_lr_0: 0.00019729331201546197 | > current_lr_1: 0.00019729331201546197 | > step_time: 0.4501 (0.45122630923402074) | > loader_time: 0.0088 (0.007701579262228573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008611843619548093 (+0.00013083471378809268) | > avg_step_time: 0.461457759561673 (+0.008211736947717818) | > avg_loss_disc: 2.853520319495402 (+0.0029747083153521814) | > avg_loss_disc_real_0: 0.23144076869521343 (+0.005093147007512372) | > avg_loss_disc_real_1: 0.2353316218080655 (+0.0002958241482855506) | > avg_loss_disc_real_2: 0.24016726289836454 (-0.0016879493921575228) | > avg_loss_disc_real_3: 0.2413092765589835 (-0.0020317726991546103) | > avg_loss_disc_real_4: 0.2433583872838759 (-0.0025815661524383193) | > avg_loss_disc_real_5: 0.2463965012993611 (-0.0009781944919639851) | > avg_loss_0: 2.853520319495402 (+0.0029747083153521814) | > avg_grad_norm_0: tensor(10.4985, device='cuda:0') (+tensor(0.0311, device='cuda:0')) | > avg_loss_gen: 1.7329934079882126 (-0.0224098638749457) | > avg_loss_kl: 2.4384935674533046 (+0.005601819132415642) | > avg_loss_feat: 1.8436361652025035 (-0.043209358000419584) | > avg_loss_mel: 17.883805718220454 (-0.27230774516791456) | > avg_loss_duration: 1.6647333779805142 (+0.004444229770714125) | > avg_amp_scaler: 256.0 (-79.32394366197184) | > avg_loss_1: 25.563662166326818 (-0.3278809399671978) | > avg_grad_norm_1: tensor(237.6201, device='cuda:0') (+tensor(12.5244, device='cuda:0'))  > EPOCH: 110/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:19:02)   --> TIME: 2025-02-25 09:19:17 -- STEP: 29/72 -- GLOBAL_STEP: 1007950 | > loss_disc: 2.768181800842285 (2.826312056903182) | > loss_disc_real_0: 0.2208687961101532 (0.22872266327512675) | > loss_disc_real_1: 0.24305333197116852 (0.23372577541861042) | > loss_disc_real_2: 0.2856875956058502 (0.24451476592442084) | > loss_disc_real_3: 0.256611168384552 (0.24410057119254408) | > loss_disc_real_4: 0.24351386725902557 (0.2451074349469152) | > loss_disc_real_5: 0.2262411266565323 (0.24910007417201996) | > loss_0: 2.768181800842285 (2.826312056903182) | > grad_norm_0: tensor(3.8209, device='cuda:0') (tensor(9.3898, device='cuda:0')) | > loss_gen: 1.6706459522247314 (1.7690532577448879) | > loss_kl: 2.566333293914795 (2.568418182175735) | > loss_feat: 1.8034948110580444 (1.9290122451453373) | > loss_mel: 18.421382904052734 (18.474558468522705) | > loss_duration: 1.648412823677063 (1.6371448779928273) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.11026954650879 (26.37818685071222) | > grad_norm_1: tensor(72.4000, device='cuda:0') (tensor(209.7753, device='cuda:0')) | > current_lr_0: 0.00019726865035146003 | > current_lr_1: 0.00019726865035146003 | > step_time: 0.44 (0.43775509965830833) | > loader_time: 0.0078 (0.006688381063527074) --> EVAL PERFORMANCE | > avg_loader_time: 0.008478144524802625 (-0.0001336990947454677) | > avg_step_time: 0.4561233419767567 (-0.005334417584916318) | > avg_loss_disc: 2.83224136728636 (-0.021278952209042412) | > avg_loss_disc_real_0: 0.22587328771470297 (-0.005567480980510464) | > avg_loss_disc_real_1: 0.2360154392433838 (+0.0006838174353183124) | > avg_loss_disc_real_2: 0.23915932799728823 (-0.0010079349010763128) | > avg_loss_disc_real_3: 0.24128983306213164 (-1.944349685184532e-05) | > avg_loss_disc_real_4: 0.24624605812657047 (+0.002887670842694562) | > avg_loss_disc_real_5: 0.24775645451646455 (+0.0013599532171034434) | > avg_loss_0: 2.83224136728636 (-0.021278952209042412) | > avg_grad_norm_0: tensor(9.8001, device='cuda:0') (tensor(-0.6984, device='cuda:0')) | > avg_loss_gen: 1.7514155078941667 (+0.018422099905954115) | > avg_loss_kl: 2.48492732182355 (+0.04643375437024533) | > avg_loss_feat: 1.853724887673284 (+0.010088722470780409) | > avg_loss_mel: 17.936225810521087 (+0.052420092300632604) | > avg_loss_duration: 1.6574624457829434 (-0.007270932197570801) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.683755767177527 (+0.12009360085070853) | > avg_grad_norm_1: tensor(253.0788, device='cuda:0') (+tensor(15.4587, device='cuda:0'))  > EPOCH: 111/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:19:39)   --> TIME: 2025-02-25 09:19:44 -- STEP: 7/72 -- GLOBAL_STEP: 1008000 | > loss_disc: 2.7599377632141113 (2.8145037037985667) | > loss_disc_real_0: 0.2737751305103302 (0.2267725850854601) | > loss_disc_real_1: 0.23086251318454742 (0.2370507653270449) | > loss_disc_real_2: 0.2784525752067566 (0.2373275033065251) | > loss_disc_real_3: 0.21464698016643524 (0.23563880154064723) | > loss_disc_real_4: 0.23824641108512878 (0.235230558684894) | > loss_disc_real_5: 0.22189265489578247 (0.23892885872295924) | > loss_0: 2.7599377632141113 (2.8145037037985667) | > grad_norm_0: tensor(8.5473, device='cuda:0') (tensor(8.3631, device='cuda:0')) | > loss_gen: 1.8438777923583984 (1.7799374886921473) | > loss_kl: 2.2471580505371094 (2.329643590109689) | > loss_feat: 2.3345141410827637 (2.005366563796997) | > loss_mel: 18.588909149169922 (18.056053706577845) | > loss_duration: 1.6261787414550781 (1.6374057701655798) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.64063835144043 (25.808407374790736) | > grad_norm_1: tensor(185.7324, device='cuda:0') (tensor(244.1194, device='cuda:0')) | > current_lr_0: 0.0001972439917701661 | > current_lr_1: 0.0001972439917701661 | > step_time: 0.4309 (0.4332024710518973) | > loader_time: 0.0055 (0.00607405390058245)  --> TIME: 2025-02-25 09:20:07 -- STEP: 57/72 -- GLOBAL_STEP: 1008050 | > loss_disc: 2.7762787342071533 (2.845695027133875) | > loss_disc_real_0: 0.1834718883037567 (0.22969752079562136) | > loss_disc_real_1: 0.26440584659576416 (0.23637585420357554) | > loss_disc_real_2: 0.23313000798225403 (0.24171577524720578) | > loss_disc_real_3: 0.23119235038757324 (0.2427886361092852) | > loss_disc_real_4: 0.24208411574363708 (0.24561883666013418) | > loss_disc_real_5: 0.23627975583076477 (0.24621669425253281) | > loss_0: 2.7762787342071533 (2.845695027133875) | > grad_norm_0: tensor(7.3399, device='cuda:0') (tensor(9.1556, device='cuda:0')) | > loss_gen: 1.6752504110336304 (1.752515345288996) | > loss_kl: 2.5861504077911377 (2.4282026332721376) | > loss_feat: 2.0930027961730957 (1.886854939293443) | > loss_mel: 18.337343215942383 (17.951652426468694) | > loss_duration: 1.6951284408569336 (1.647788919900593) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.38687515258789 (25.667014239127177) | > grad_norm_1: tensor(56.5068, device='cuda:0') (tensor(211.1691, device='cuda:0')) | > current_lr_0: 0.0001972439917701661 | > current_lr_1: 0.0001972439917701661 | > step_time: 0.4712 (0.44389316073635166) | > loader_time: 0.0107 (0.007927593431974711) --> EVAL PERFORMANCE | > avg_loader_time: 0.008569079385676854 (+9.093486087422865e-05) | > avg_step_time: 0.45336197127758615 (-0.00276137069917054) | > avg_loss_disc: 2.8410918611875724 (+0.00885049390121262) | > avg_loss_disc_real_0: 0.22637461923377614 (+0.0005013315190731671) | > avg_loss_disc_real_1: 0.2360687919065986 (+5.335266321479515e-05) | > avg_loss_disc_real_2: 0.24099671462891806 (+0.0018373866316298282) | > avg_loss_disc_real_3: 0.24196230462739166 (+0.0006724715652600133) | > avg_loss_disc_real_4: 0.244798535402392 (-0.0014475227241784738) | > avg_loss_disc_real_5: 0.2459044955985647 (-0.0018519589178998541) | > avg_loss_0: 2.8410918611875724 (+0.00885049390121262) | > avg_grad_norm_0: tensor(9.2828, device='cuda:0') (tensor(-0.5173, device='cuda:0')) | > avg_loss_gen: 1.7511217594146729 (-0.0002937484794938161) | > avg_loss_kl: 2.4469717724222533 (-0.03795554940129664) | > avg_loss_feat: 1.889943540935785 (+0.036218653262501066) | > avg_loss_mel: 18.026469861957384 (+0.09024405143629721) | > avg_loss_duration: 1.6552323573072192 (-0.0022300884757242567) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.76973923159317 (+0.0859834644156443) | > avg_grad_norm_1: tensor(211.6181, device='cuda:0') (tensor(-41.4607, device='cuda:0'))  > EPOCH: 112/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:20:15)   --> TIME: 2025-02-25 09:20:34 -- STEP: 35/72 -- GLOBAL_STEP: 1008100 | > loss_disc: 2.8356149196624756 (2.8475781645093647) | > loss_disc_real_0: 0.2440621703863144 (0.2438704954726356) | > loss_disc_real_1: 0.2944531738758087 (0.23064357936382293) | > loss_disc_real_2: 0.24804671108722687 (0.23921925510678974) | > loss_disc_real_3: 0.21548964083194733 (0.2403488895722798) | > loss_disc_real_4: 0.229297935962677 (0.24815717126641954) | > loss_disc_real_5: 0.2323397547006607 (0.24794057139328549) | > loss_0: 2.8356149196624756 (2.8475781645093647) | > grad_norm_0: tensor(6.4713, device='cuda:0') (tensor(11.9515, device='cuda:0')) | > loss_gen: 1.663158655166626 (1.744713384764535) | > loss_kl: 2.5191903114318848 (2.3838492461613248) | > loss_feat: 1.848059892654419 (1.87889575277056) | > loss_mel: 19.169713973999023 (18.165097372872488) | > loss_duration: 1.6528213024139404 (1.6294525691441126) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.852943420410156 (25.8020081111363) | > grad_norm_1: tensor(168.7471, device='cuda:0') (tensor(237.4523, device='cuda:0')) | > current_lr_0: 0.0001972193362711948 | > current_lr_1: 0.0001972193362711948 | > step_time: 0.4391 (0.4505204541342599) | > loader_time: 0.0079 (0.007203033992222377) --> EVAL PERFORMANCE | > avg_loader_time: 0.008553196007097268 (-1.5883378579585605e-05) | > avg_step_time: 0.46180758341937 (+0.008445612141783854) | > avg_loss_disc: 2.836773476130525 (-0.0043183850570471805) | > avg_loss_disc_real_0: 0.2368370042720311 (+0.010462385038254973) | > avg_loss_disc_real_1: 0.2366616085801326 (+0.0005928166735340024) | > avg_loss_disc_real_2: 0.2377729657250391 (-0.00322374890387897) | > avg_loss_disc_real_3: 0.2414066189191711 (-0.0005556857082205657) | > avg_loss_disc_real_4: 0.2443257899771274 (-0.00047274542526459795) | > avg_loss_disc_real_5: 0.2446818039031096 (-0.0012226916954551081) | > avg_loss_0: 2.836773476130525 (-0.0043183850570471805) | > avg_grad_norm_0: tensor(11.5204, device='cuda:0') (+tensor(2.2376, device='cuda:0')) | > avg_loss_gen: 1.751807562062438 (+0.000685802647765188) | > avg_loss_kl: 2.3756365037300204 (-0.07133526869223283) | > avg_loss_feat: 1.871145145993837 (-0.018798394941947905) | > avg_loss_mel: 18.008981865896306 (-0.017487996061078093) | > avg_loss_duration: 1.6524010846312618 (-0.002831272675957397) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.659972177424894 (-0.1097670541682767) | > avg_grad_norm_1: tensor(223.8470, device='cuda:0') (+tensor(12.2289, device='cuda:0'))  > EPOCH: 113/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:20:53)   --> TIME: 2025-02-25 09:21:00 -- STEP: 13/72 -- GLOBAL_STEP: 1008150 | > loss_disc: 2.8656907081604004 (2.8588195397303653) | > loss_disc_real_0: 0.3362360894680023 (0.23505998632082573) | > loss_disc_real_1: 0.25221529603004456 (0.2288002371788025) | > loss_disc_real_2: 0.20831698179244995 (0.2330398937830558) | > loss_disc_real_3: 0.22395534813404083 (0.2374524210508053) | > loss_disc_real_4: 0.2422725260257721 (0.24785451934887812) | > loss_disc_real_5: 0.2245326191186905 (0.25007906212256503) | > loss_0: 2.8656907081604004 (2.8588195397303653) | > grad_norm_0: tensor(17.0258, device='cuda:0') (tensor(16.4770, device='cuda:0')) | > loss_gen: 1.8750324249267578 (1.7404053578009973) | > loss_kl: 2.640188694000244 (2.363160280080942) | > loss_feat: 1.9006191492080688 (1.8612838983535767) | > loss_mel: 16.355329513549805 (17.8304936335637) | > loss_duration: 1.556899070739746 (1.6300490636091967) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.328067779541016 (25.425392150878906) | > grad_norm_1: tensor(260.1764, device='cuda:0') (tensor(244.0704, device='cuda:0')) | > current_lr_0: 0.0001971946838541609 | > current_lr_1: 0.0001971946838541609 | > step_time: 0.4571 (0.43451578800494856) | > loader_time: 0.0058 (0.0068054932814378)  --> TIME: 2025-02-25 09:21:24 -- STEP: 63/72 -- GLOBAL_STEP: 1008200 | > loss_disc: 2.8482818603515625 (2.8442244718945218) | > loss_disc_real_0: 0.23444345593452454 (0.23491960905847095) | > loss_disc_real_1: 0.25713810324668884 (0.23603737094099558) | > loss_disc_real_2: 0.26617899537086487 (0.23743958747576152) | > loss_disc_real_3: 0.2514830231666565 (0.2418707606811372) | > loss_disc_real_4: 0.2621331810951233 (0.24466810221709903) | > loss_disc_real_5: 0.26985669136047363 (0.24679783388735757) | > loss_0: 2.8482818603515625 (2.8442244718945218) | > grad_norm_0: tensor(4.2566, device='cuda:0') (tensor(11.4963, device='cuda:0')) | > loss_gen: 1.7138876914978027 (1.758885902071756) | > loss_kl: 2.3827364444732666 (2.412426993960426) | > loss_feat: 1.4119220972061157 (1.9055185696435353) | > loss_mel: 17.538307189941406 (17.830531589568608) | > loss_duration: 1.6412920951843262 (1.6463252097841292) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.688146591186523 (25.553688291519407) | > grad_norm_1: tensor(59.3488, device='cuda:0') (tensor(257.4180, device='cuda:0')) | > current_lr_0: 0.0001971946838541609 | > current_lr_1: 0.0001971946838541609 | > step_time: 0.5306 (0.4482163663894411) | > loader_time: 0.0113 (0.008259273710704987) --> EVAL PERFORMANCE | > avg_loader_time: 0.00863645781933422 (+8.326181223695245e-05) | > avg_step_time: 0.455153334308678 (-0.006654249110692012) | > avg_loss_disc: 2.8457514601694025 (+0.008977984038877285) | > avg_loss_disc_real_0: 0.23207997016503779 (-0.004757034106993324) | > avg_loss_disc_real_1: 0.23616317250359226 (-0.0004984360765403484) | > avg_loss_disc_real_2: 0.23796649235235134 (+0.00019352662731225712) | > avg_loss_disc_real_3: 0.2412748332594482 (-0.0001317856597228828) | > avg_loss_disc_real_4: 0.24458462629519717 (+0.00025883631806977725) | > avg_loss_disc_real_5: 0.24661170198044308 (+0.0019298980773334862) | > avg_loss_0: 2.8457514601694025 (+0.008977984038877285) | > avg_grad_norm_0: tensor(11.2540, device='cuda:0') (tensor(-0.2664, device='cuda:0')) | > avg_loss_gen: 1.755697895103777 (+0.0038903330413389003) | > avg_loss_kl: 2.4216121720596093 (+0.04597566832958888) | > avg_loss_feat: 1.8945605385471398 (+0.023415392553302716) | > avg_loss_mel: 17.853978009291097 (-0.15500385660520877) | > avg_loss_duration: 1.6511857308132547 (-0.001215353818007081) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.577034318950815 (-0.08293785847407875) | > avg_grad_norm_1: tensor(264.2580, device='cuda:0') (+tensor(40.4110, device='cuda:0'))  > EPOCH: 114/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:21:30)   --> TIME: 2025-02-25 09:21:50 -- STEP: 41/72 -- GLOBAL_STEP: 1008250 | > loss_disc: 2.841259479522705 (2.849920644992736) | > loss_disc_real_0: 0.1319190412759781 (0.22731279745334532) | > loss_disc_real_1: 0.21139760315418243 (0.24077644835158094) | > loss_disc_real_2: 0.26856130361557007 (0.24086391126237264) | > loss_disc_real_3: 0.23457330465316772 (0.24424696768202434) | > loss_disc_real_4: 0.22225895524024963 (0.2468769063309925) | > loss_disc_real_5: 0.21643173694610596 (0.24976333212561724) | > loss_0: 2.841259479522705 (2.849920644992736) | > grad_norm_0: tensor(12.9751, device='cuda:0') (tensor(8.4179, device='cuda:0')) | > loss_gen: 1.7291232347488403 (1.7611194500109044) | > loss_kl: 2.3008480072021484 (2.418987593999723) | > loss_feat: 2.0026485919952393 (1.8901751332166719) | > loss_mel: 18.743955612182617 (18.02838044050263) | > loss_duration: 1.6530710458755493 (1.641310741261738) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.429645538330078 (25.739973207799398) | > grad_norm_1: tensor(281.1378, device='cuda:0') (tensor(228.2634, device='cuda:0')) | > current_lr_0: 0.0001971700345186791 | > current_lr_1: 0.0001971700345186791 | > step_time: 0.4432 (0.43552856910519483) | > loader_time: 0.0094 (0.007273860094023914) --> EVAL PERFORMANCE | > avg_loader_time: 0.00852830645064233 (-0.00010815136869189029) | > avg_step_time: 0.45111009772394745 (-0.004043236584730547) | > avg_loss_disc: 2.8528249532404084 (+0.00707349307100591) | > avg_loss_disc_real_0: 0.23097070820734533 (-0.0011092619576924534) | > avg_loss_disc_real_1: 0.23731685618699436 (+0.0011536836834021014) | > avg_loss_disc_real_2: 0.24005949203397187 (+0.002092999681620522) | > avg_loss_disc_real_3: 0.2417385886672517 (+0.00046375540780349556) | > avg_loss_disc_real_4: 0.2451618834280632 (+0.0005772571328660214) | > avg_loss_disc_real_5: 0.24751487122455113 (+0.0009031692441080486) | > avg_loss_0: 2.8528249532404084 (+0.00707349307100591) | > avg_grad_norm_0: tensor(9.8374, device='cuda:0') (tensor(-1.4167, device='cuda:0')) | > avg_loss_gen: 1.7474162595372804 (-0.008281635566496526) | > avg_loss_kl: 2.3741662938829875 (-0.047445878176621825) | > avg_loss_feat: 1.8598317159733302 (-0.0347288225738096) | > avg_loss_mel: 18.00067430146983 (+0.14669629217873137) | > avg_loss_duration: 1.6561956170579077 (+0.005009886244653039) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.638284038489974 (+0.06124971953915903) | > avg_grad_norm_1: tensor(275.3025, device='cuda:0') (+tensor(11.0445, device='cuda:0'))  > EPOCH: 115/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:22:06)   --> TIME: 2025-02-25 09:22:16 -- STEP: 19/72 -- GLOBAL_STEP: 1008300 | > loss_disc: 2.754560947418213 (2.8582475059910823) | > loss_disc_real_0: 0.22257420420646667 (0.21513276507979945) | > loss_disc_real_1: 0.21435615420341492 (0.24457337197504544) | > loss_disc_real_2: 0.1729259490966797 (0.25001145036597) | > loss_disc_real_3: 0.18955859541893005 (0.24219105745616712) | > loss_disc_real_4: 0.19606377184391022 (0.2531610539084987) | > loss_disc_real_5: 0.21630282700061798 (0.25417085933057887) | > loss_0: 2.754560947418213 (2.8582475059910823) | > grad_norm_0: tensor(10.6270, device='cuda:0') (tensor(10.4612, device='cuda:0')) | > loss_gen: 1.7089124917984009 (1.7556414478703548) | > loss_kl: 2.4712159633636475 (2.4230987649214897) | > loss_feat: 2.1064951419830322 (1.8676715022639225) | > loss_mel: 18.084848403930664 (18.082428279675934) | > loss_duration: 1.6007236242294312 (1.6315484046936035) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.97219467163086 (25.760388424522) | > grad_norm_1: tensor(109.4730, device='cuda:0') (tensor(288.1319, device='cuda:0')) | > current_lr_0: 0.00019714538826436426 | > current_lr_1: 0.00019714538826436426 | > step_time: 0.4277 (0.4323293786299856) | > loader_time: 0.0087 (0.006720555456061112)  --> TIME: 2025-02-25 09:22:41 -- STEP: 69/72 -- GLOBAL_STEP: 1008350 | > loss_disc: 2.981799840927124 (2.855333169301351) | > loss_disc_real_0: 0.21996895968914032 (0.22305147965317187) | > loss_disc_real_1: 0.26769956946372986 (0.2384771166936211) | > loss_disc_real_2: 0.2264125794172287 (0.24060446088728699) | > loss_disc_real_3: 0.24062439799308777 (0.23872732381889786) | > loss_disc_real_4: 0.23894385993480682 (0.24589966453503873) | > loss_disc_real_5: 0.2237127125263214 (0.24671128759349603) | > loss_0: 2.981799840927124 (2.855333169301351) | > grad_norm_0: tensor(6.3152, device='cuda:0') (tensor(10.1373, device='cuda:0')) | > loss_gen: 1.7735099792480469 (1.7368394153705542) | > loss_kl: 2.423961639404297 (2.4198045592377158) | > loss_feat: 1.8128573894500732 (1.8498570884483447) | > loss_mel: 17.638233184814453 (17.96442331784013) | > loss_duration: 1.7362972497940063 (1.6552902028180552) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.38486099243164 (25.626214428224426) | > grad_norm_1: tensor(370.8735, device='cuda:0') (tensor(236.2611, device='cuda:0')) | > current_lr_0: 0.00019714538826436426 | > current_lr_1: 0.00019714538826436426 | > step_time: 0.5449 (0.46130937769793084) | > loader_time: 0.0127 (0.008624398190042248) --> EVAL PERFORMANCE | > avg_loader_time: 0.008680007827114049 (+0.000151701376471719) | > avg_step_time: 0.4626445837423835 (+0.011534486018436041) | > avg_loss_disc: 2.855088724216945 (+0.0022637709765365166) | > avg_loss_disc_real_0: 0.2249699410208514 (-0.006000767186493922) | > avg_loss_disc_real_1: 0.2376517607292659 (+0.0003349045422715302) | > avg_loss_disc_real_2: 0.24141322034345547 (+0.001353728309483604) | > avg_loss_disc_real_3: 0.2391456919656673 (-0.002592896701584413) | > avg_loss_disc_real_4: 0.24629530163717941 (+0.0011334182091162215) | > avg_loss_disc_real_5: 0.2467525926694064 (-0.0007622785551447131) | > avg_loss_0: 2.855088724216945 (+0.0022637709765365166) | > avg_grad_norm_0: tensor(10.2623, device='cuda:0') (+tensor(0.4250, device='cuda:0')) | > avg_loss_gen: 1.738913655281067 (-0.00850260425621352) | > avg_loss_kl: 2.422271748663675 (+0.048105454780687396) | > avg_loss_feat: 1.8529737482608204 (-0.006857967712509794) | > avg_loss_mel: 17.966098154094855 (-0.03457614737497394) | > avg_loss_duration: 1.6553005735639115 (-0.0008950434939962726) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.635557685099858 (-0.0027263533901162873) | > avg_grad_norm_1: tensor(239.5593, device='cuda:0') (tensor(-35.7432, device='cuda:0'))  > EPOCH: 116/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:22:44)   --> TIME: 2025-02-25 09:23:07 -- STEP: 47/72 -- GLOBAL_STEP: 1008400 | > loss_disc: 2.8215041160583496 (2.8532977763642657) | > loss_disc_real_0: 0.25183677673339844 (0.23375887296935347) | > loss_disc_real_1: 0.24361024796962738 (0.23573335276005117) | > loss_disc_real_2: 0.2294919341802597 (0.24205463997861174) | > loss_disc_real_3: 0.23350399732589722 (0.24201425053971878) | > loss_disc_real_4: 0.23600399494171143 (0.24631227806527564) | > loss_disc_real_5: 0.24259136617183685 (0.24966364369747487) | > loss_0: 2.8215041160583496 (2.8532977763642657) | > grad_norm_0: tensor(8.0469, device='cuda:0') (tensor(11.7176, device='cuda:0')) | > loss_gen: 1.7453616857528687 (1.7424163590086268) | > loss_kl: 2.489428758621216 (2.394781112670899) | > loss_feat: 1.6858950853347778 (1.82342469945867) | > loss_mel: 17.570348739624023 (18.066239458449342) | > loss_duration: 1.674207091331482 (1.6388860661932763) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.165241241455078 (25.66574770338992) | > grad_norm_1: tensor(134.3820, device='cuda:0') (tensor(224.4611, device='cuda:0')) | > current_lr_0: 0.0001971207450908312 | > current_lr_1: 0.0001971207450908312 | > step_time: 0.4496 (0.4386223833611671) | > loader_time: 0.0088 (0.007549514161779526) --> EVAL PERFORMANCE | > avg_loader_time: 0.008541681397129113 (-0.00013832642998493593) | > avg_step_time: 0.4524253556426142 (-0.010219228099769273) | > avg_loss_disc: 2.848419273403329 (-0.006669450813615807) | > avg_loss_disc_real_0: 0.23384467659282013 (+0.008874735571968717) | > avg_loss_disc_real_1: 0.2374693018869615 (-0.00018245884230438936) | > avg_loss_disc_real_2: 0.242315766047424 (+0.000902545703968527) | > avg_loss_disc_real_3: 0.24288230777626307 (+0.0037366158105957836) | > avg_loss_disc_real_4: 0.2463759337092789 (+8.063207209949019e-05) | > avg_loss_disc_real_5: 0.24930991022519663 (+0.002557317555790223) | > avg_loss_0: 2.848419273403329 (-0.006669450813615807) | > avg_grad_norm_0: tensor(11.3420, device='cuda:0') (+tensor(1.0796, device='cuda:0')) | > avg_loss_gen: 1.7558787742131192 (+0.016965118932052325) | > avg_loss_kl: 2.423644149807138 (+0.0013724011434632644) | > avg_loss_feat: 1.8580498426732883 (+0.005076094412467924) | > avg_loss_mel: 18.250557026392976 (+0.2844588722981207) | > avg_loss_duration: 1.6583979264111586 (+0.0030973528472471035) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.94652761539943 (+0.31096993029957076) | > avg_grad_norm_1: tensor(250.1299, device='cuda:0') (+tensor(10.5706, device='cuda:0'))  > EPOCH: 117/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:23:20)   --> TIME: 2025-02-25 09:23:33 -- STEP: 25/72 -- GLOBAL_STEP: 1008450 | > loss_disc: 2.821437120437622 (2.8368984031677247) | > loss_disc_real_0: 0.1989038586616516 (0.22317952066659927) | > loss_disc_real_1: 0.2357052117586136 (0.23011987924575805) | > loss_disc_real_2: 0.23919562995433807 (0.23893481850624085) | > loss_disc_real_3: 0.2511695623397827 (0.23868126332759856) | > loss_disc_real_4: 0.25425878167152405 (0.2445218765735626) | > loss_disc_real_5: 0.24680757522583008 (0.24983967483043668) | > loss_0: 2.821437120437622 (2.8368984031677247) | > grad_norm_0: tensor(3.4851, device='cuda:0') (tensor(9.2777, device='cuda:0')) | > loss_gen: 1.7281603813171387 (1.7387859916687012) | > loss_kl: 2.4487991333007812 (2.3268232297897335) | > loss_feat: 1.9109336137771606 (1.8551423788070678) | > loss_mel: 16.471696853637695 (18.02052696228027) | > loss_duration: 1.61146080493927 (1.6257069540023803) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.171051025390625 (25.56698539733887) | > grad_norm_1: tensor(194.9069, device='cuda:0') (tensor(283.4067, device='cuda:0')) | > current_lr_0: 0.00019709610499769482 | > current_lr_1: 0.00019709610499769482 | > step_time: 0.4279 (0.42900221824645995) | > loader_time: 0.0071 (0.006830778121948242) --> EVAL PERFORMANCE | > avg_loader_time: 0.00858163162016533 (+3.9950223036216634e-05) | > avg_step_time: 0.4499325248557077 (-0.002492830786906486) | > avg_loss_disc: 2.8437539456595835 (-0.0046653277437456175) | > avg_loss_disc_real_0: 0.2256449359193654 (-0.00819974067345472) | > avg_loss_disc_real_1: 0.23472179703309504 (-0.0027475048538664615) | > avg_loss_disc_real_2: 0.2401134021265406 (-0.00220236392088341) | > avg_loss_disc_real_3: 0.24052443722604025 (-0.0023578705502228237) | > avg_loss_disc_real_4: 0.24523925172611022 (-0.0011366819831686858) | > avg_loss_disc_real_5: 0.24610626172851507 (-0.0032036484966815604) | > avg_loss_0: 2.8437539456595835 (-0.0046653277437456175) | > avg_grad_norm_0: tensor(9.8035, device='cuda:0') (tensor(-1.5385, device='cuda:0')) | > avg_loss_gen: 1.7383264605428133 (-0.017552313670305963) | > avg_loss_kl: 2.3317461399964885 (-0.09189800981064966) | > avg_loss_feat: 1.853635957543279 (-0.004413885130009287) | > avg_loss_mel: 17.803705000541587 (-0.4468520258513884) | > avg_loss_duration: 1.649935655190911 (-0.008462271220247564) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.377349208778064 (-0.569178406621365) | > avg_grad_norm_1: tensor(286.5043, device='cuda:0') (+tensor(36.3744, device='cuda:0')) > BEST MODEL : voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a/best_model_1008497.pth  > EPOCH: 118/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:24:00)   --> TIME: 2025-02-25 09:24:03 -- STEP: 3/72 -- GLOBAL_STEP: 1008500 | > loss_disc: 2.894160747528076 (2.867253303527832) | > loss_disc_real_0: 0.24751628935337067 (0.17545977234840393) | > loss_disc_real_1: 0.2275703400373459 (0.23204304774602255) | > loss_disc_real_2: 0.23340630531311035 (0.2319840838511785) | > loss_disc_real_3: 0.24728922545909882 (0.24403955042362213) | > loss_disc_real_4: 0.24977436661720276 (0.25737373034159344) | > loss_disc_real_5: 0.2749484181404114 (0.2797236243883769) | > loss_0: 2.894160747528076 (2.867253303527832) | > grad_norm_0: tensor(11.2020, device='cuda:0') (tensor(14.6360, device='cuda:0')) | > loss_gen: 1.7761930227279663 (1.7304871479670207) | > loss_kl: 2.474766492843628 (2.480562130610148) | > loss_feat: 1.6419074535369873 (1.8909279902776082) | > loss_mel: 18.33090591430664 (18.736294428507488) | > loss_duration: 1.6798553466796875 (1.658381422360738) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.903629302978516 (26.496653238932293) | > grad_norm_1: tensor(469.8001, device='cuda:0') (tensor(468.3873, device='cuda:0')) | > current_lr_0: 0.0001970714679845701 | > current_lr_1: 0.0001970714679845701 | > step_time: 0.4384 (0.43627746899922687) | > loader_time: 0.0081 (0.0070798397064208984)  --> TIME: 2025-02-25 09:24:26 -- STEP: 53/72 -- GLOBAL_STEP: 1008550 | > loss_disc: 2.85949969291687 (2.832202677456838) | > loss_disc_real_0: 0.2356933057308197 (0.22151349203766518) | > loss_disc_real_1: 0.3041488528251648 (0.2372463786939405) | > loss_disc_real_2: 0.23038825392723083 (0.23736074088879353) | > loss_disc_real_3: 0.2394043356180191 (0.24158462769580338) | > loss_disc_real_4: 0.24375875294208527 (0.2451431062423958) | > loss_disc_real_5: 0.22603552043437958 (0.2492716559823954) | > loss_0: 2.85949969291687 (2.832202677456838) | > grad_norm_0: tensor(8.0741, device='cuda:0') (tensor(10.8261, device='cuda:0')) | > loss_gen: 1.8146692514419556 (1.7561552861951433) | > loss_kl: 2.2114832401275635 (2.3503655577605627) | > loss_feat: 1.7957677841186523 (1.922858305697171) | > loss_mel: 16.021474838256836 (17.86690215344699) | > loss_duration: 1.6887145042419434 (1.6418468974671274) | > amp_scaler: 256.0 (256.0) | > loss_1: 23.532108306884766 (25.538128402997863) | > grad_norm_1: tensor(362.3083, device='cuda:0') (tensor(246.6369, device='cuda:0')) | > current_lr_0: 0.0001970714679845701 | > current_lr_1: 0.0001970714679845701 | > step_time: 0.4446 (0.44638066021901257) | > loader_time: 0.0092 (0.007879189725192089) --> EVAL PERFORMANCE | > avg_loader_time: 0.008624244743669542 (+4.261312350421244e-05) | > avg_step_time: 0.45626184973918216 (+0.00632932488347443) | > avg_loss_disc: 2.8409877662927334 (-0.0027661793668500856) | > avg_loss_disc_real_0: 0.22574731266834366 (+0.00010237674897825566) | > avg_loss_disc_real_1: 0.2349691395188721 (+0.00024734248577706897) | > avg_loss_disc_real_2: 0.2405374654581849 (+0.0004240633316443254) | > avg_loss_disc_real_3: 0.24228677699263668 (+0.0017623397665964269) | > avg_loss_disc_real_4: 0.24624002462541553 (+0.0010007728993053078) | > avg_loss_disc_real_5: 0.25076069882218266 (+0.004654437093667585) | > avg_loss_0: 2.8409877662927334 (-0.0027661793668500856) | > avg_grad_norm_0: tensor(12.2859, device='cuda:0') (+tensor(2.4824, device='cuda:0')) | > avg_loss_gen: 1.7553307338499686 (+0.01700427330715537) | > avg_loss_kl: 2.3723920399034526 (+0.0406458999069641) | > avg_loss_feat: 1.8996559784445963 (+0.04602002090131729) | > avg_loss_mel: 17.88940061649806 (+0.08569561595647457) | > avg_loss_duration: 1.6565859351359622 (+0.006650279945051185) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.573365520423568 (+0.19601631164550426) | > avg_grad_norm_1: tensor(244.4012, device='cuda:0') (tensor(-42.1032, device='cuda:0'))  > EPOCH: 119/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:24:37)   --> TIME: 2025-02-25 09:24:52 -- STEP: 31/72 -- GLOBAL_STEP: 1008600 | > loss_disc: 2.7467639446258545 (2.8261131086657123) | > loss_disc_real_0: 0.19425860047340393 (0.2296447403007938) | > loss_disc_real_1: 0.24392031133174896 (0.23280471082656615) | > loss_disc_real_2: 0.1798020601272583 (0.23257405190698563) | > loss_disc_real_3: 0.17511296272277832 (0.23697300111093828) | > loss_disc_real_4: 0.23907606303691864 (0.24324173936920782) | > loss_disc_real_5: 0.2534257173538208 (0.24372657941233727) | > loss_0: 2.7467639446258545 (2.8261131086657123) | > grad_norm_0: tensor(4.2238, device='cuda:0') (tensor(9.5215, device='cuda:0')) | > loss_gen: 1.7544376850128174 (1.7530966535691292) | > loss_kl: 2.4657609462738037 (2.4795248969908683) | > loss_feat: 2.175786018371582 (1.96461647172128) | > loss_mel: 18.138710021972656 (18.34734744410361) | > loss_duration: 1.6441351175308228 (1.6359678045395882) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.178831100463867 (26.18055312864242) | > grad_norm_1: tensor(106.4552, device='cuda:0') (tensor(198.9850, device='cuda:0')) | > current_lr_0: 0.000197046834051072 | > current_lr_1: 0.000197046834051072 | > step_time: 0.4365 (0.4357381636096585) | > loader_time: 0.0083 (0.006980265340497417) --> EVAL PERFORMANCE | > avg_loader_time: 0.008546624385135279 (-7.762035853426351e-05) | > avg_step_time: 0.4528685522751069 (-0.00339329746407524) | > avg_loss_disc: 2.8348600058488445 (-0.006127760443888963) | > avg_loss_disc_real_0: 0.22853265608280476 (+0.0027853434144611) | > avg_loss_disc_real_1: 0.2318355529241159 (-0.0031335865947562103) | > avg_loss_disc_real_2: 0.23653092287795643 (-0.004006542580228478) | > avg_loss_disc_real_3: 0.24117613930097767 (-0.0011106376916590055) | > avg_loss_disc_real_4: 0.24412164029101252 (-0.0021183843344030062) | > avg_loss_disc_real_5: 0.24438129974083161 (-0.006379399081351045) | > avg_loss_0: 2.8348600058488445 (-0.006127760443888963) | > avg_grad_norm_0: tensor(9.6819, device='cuda:0') (tensor(-2.6040, device='cuda:0')) | > avg_loss_gen: 1.7513314401599722 (-0.003999293689996408) | > avg_loss_kl: 2.403053465023846 (+0.030661425120393382) | > avg_loss_feat: 1.9306956882208166 (+0.03103970977622028) | > avg_loss_mel: 18.034275860853604 (+0.14487524435554278) | > avg_loss_duration: 1.6569933773766101 (+0.0004074422406479261) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.77634969899352 (+0.20298417856995243) | > avg_grad_norm_1: tensor(217.6151, device='cuda:0') (tensor(-26.7861, device='cuda:0'))  > EPOCH: 120/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:25:13)   --> TIME: 2025-02-25 09:25:19 -- STEP: 9/72 -- GLOBAL_STEP: 1008650 | > loss_disc: 2.7602410316467285 (2.8634965419769287) | > loss_disc_real_0: 0.18801233172416687 (0.2279668649037679) | > loss_disc_real_1: 0.19738785922527313 (0.2358551886346605) | > loss_disc_real_2: 0.20161154866218567 (0.24587341811921862) | > loss_disc_real_3: 0.22512730956077576 (0.23812495006455314) | > loss_disc_real_4: 0.25076788663864136 (0.25050120221243966) | > loss_disc_real_5: 0.23611928522586823 (0.2526626338561376) | > loss_0: 2.7602410316467285 (2.8634965419769287) | > grad_norm_0: tensor(8.0837, device='cuda:0') (tensor(8.1664, device='cuda:0')) | > loss_gen: 1.781408429145813 (1.7347343630260892) | > loss_kl: 2.4365992546081543 (2.37141489982605) | > loss_feat: 2.019899845123291 (2.037071466445923) | > loss_mel: 19.11353874206543 (19.352385414971245) | > loss_duration: 1.6509077548980713 (1.644220921728346) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.00235366821289 (27.139827304416233) | > grad_norm_1: tensor(122.5363, device='cuda:0') (tensor(315.0980, device='cuda:0')) | > current_lr_0: 0.00019702220319681561 | > current_lr_1: 0.00019702220319681561 | > step_time: 0.4348 (0.4274527496761746) | > loader_time: 0.0054 (0.006619400448269314)  --> TIME: 2025-02-25 09:25:42 -- STEP: 59/72 -- GLOBAL_STEP: 1008700 | > loss_disc: 2.7980360984802246 (2.850463519662114) | > loss_disc_real_0: 0.15279725193977356 (0.22696075676861455) | > loss_disc_real_1: 0.214533731341362 (0.2373217485718808) | > loss_disc_real_2: 0.2337779700756073 (0.24066312863665112) | > loss_disc_real_3: 0.2007354497909546 (0.2406584433579849) | > loss_disc_real_4: 0.22827160358428955 (0.24645020446534885) | > loss_disc_real_5: 0.23611602187156677 (0.24734556775982097) | > loss_0: 2.7980360984802246 (2.850463519662114) | > grad_norm_0: tensor(9.5139, device='cuda:0') (tensor(9.7824, device='cuda:0')) | > loss_gen: 1.8444968461990356 (1.7421852956383914) | > loss_kl: 2.2195627689361572 (2.3366117073317705) | > loss_feat: 2.0944182872772217 (1.933535630420103) | > loss_mel: 18.1267032623291 (18.24131471019681) | > loss_duration: 1.675676703453064 (1.645889647936417) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.960859298706055 (25.899536973339018) | > grad_norm_1: tensor(335.2340, device='cuda:0') (tensor(256.6339, device='cuda:0')) | > current_lr_0: 0.00019702220319681561 | > current_lr_1: 0.00019702220319681561 | > step_time: 0.4604 (0.44099465063062765) | > loader_time: 0.0097 (0.008014998193514549) --> EVAL PERFORMANCE | > avg_loader_time: 0.008575012986089143 (+2.838860095386378e-05) | > avg_step_time: 0.45131635665893555 (-0.0015521956161713701) | > avg_loss_disc: 2.8518433604441906 (+0.016983354595346167) | > avg_loss_disc_real_0: 0.2311632473703841 (+0.002630591287579337) | > avg_loss_disc_real_1: 0.23977089704762042 (+0.007935344123504523) | > avg_loss_disc_real_2: 0.23958340455108965 (+0.0030524816731332116) | > avg_loss_disc_real_3: 0.2407108960856854 (-0.0004652432152922603) | > avg_loss_disc_real_4: 0.2448109442919073 (+0.0006893040008947859) | > avg_loss_disc_real_5: 0.24511553305135647 (+0.0007342333105248566) | > avg_loss_0: 2.8518433604441906 (+0.016983354595346167) | > avg_grad_norm_0: tensor(10.6144, device='cuda:0') (+tensor(0.9325, device='cuda:0')) | > avg_loss_gen: 1.7457694772263648 (-0.005561962933607445) | > avg_loss_kl: 2.329848477538203 (-0.07320498748564308) | > avg_loss_feat: 1.912504343919351 (-0.01819134430146563) | > avg_loss_mel: 18.176603129212292 (+0.14232726835868803) | > avg_loss_duration: 1.6511461717981688 (-0.005847205578441317) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.815871601373377 (+0.039521902379856755) | > avg_grad_norm_1: tensor(276.5771, device='cuda:0') (+tensor(58.9620, device='cuda:0'))  > EPOCH: 121/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:25:50)   --> TIME: 2025-02-25 09:26:09 -- STEP: 37/72 -- GLOBAL_STEP: 1008750 | > loss_disc: 2.8199124336242676 (2.8423519134521484) | > loss_disc_real_0: 0.1586817502975464 (0.23312612079285286) | > loss_disc_real_1: 0.18874363601207733 (0.24012436496244893) | > loss_disc_real_2: 0.23485703766345978 (0.24036914069910306) | > loss_disc_real_3: 0.24498650431632996 (0.24548189865576253) | > loss_disc_real_4: 0.26509934663772583 (0.24714384650861895) | > loss_disc_real_5: 0.21738648414611816 (0.24922517707218994) | > loss_0: 2.8199124336242676 (2.8423519134521484) | > grad_norm_0: tensor(11.7273, device='cuda:0') (tensor(8.9337, device='cuda:0')) | > loss_gen: 1.6884304285049438 (1.771258215646486) | > loss_kl: 2.2555599212646484 (2.500305820155788) | > loss_feat: 1.8382556438446045 (2.0001140381838822) | > loss_mel: 18.965133666992188 (18.41322898864746) | > loss_duration: 1.6471484899520874 (1.6494729712202743) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.394527435302734 (26.334380072516364) | > grad_norm_1: tensor(224.2146, device='cuda:0') (tensor(247.9037, device='cuda:0')) | > current_lr_0: 0.000196997575421416 | > current_lr_1: 0.000196997575421416 | > step_time: 0.4744 (0.43719295553258947) | > loader_time: 0.008 (0.007230700673283757) --> EVAL PERFORMANCE | > avg_loader_time: 0.008568609264534963 (-6.4037215541795645e-06) | > avg_step_time: 0.4534051922005667 (+0.00208883554163114) | > avg_loss_disc: 2.8616149694147244 (+0.009771608970533752) | > avg_loss_disc_real_0: 0.23760006923071095 (+0.006436821860326847) | > avg_loss_disc_real_1: 0.24123322040262357 (+0.0014623233550031534) | > avg_loss_disc_real_2: 0.24518702189687272 (+0.005603617345783074) | > avg_loss_disc_real_3: 0.2455651814668951 (+0.004854285381209689) | > avg_loss_disc_real_4: 0.24763844865308682 (+0.0028275043611795114) | > avg_loss_disc_real_5: 0.24896077454929622 (+0.0038452414979397442) | > avg_loss_0: 2.8616149694147244 (+0.009771608970533752) | > avg_grad_norm_0: tensor(9.7702, device='cuda:0') (tensor(-0.8442, device='cuda:0')) | > avg_loss_gen: 1.766239901663552 (+0.020470424437187162) | > avg_loss_kl: 2.4737033038072185 (+0.14385482626901558) | > avg_loss_feat: 1.9320362319409008 (+0.019531888021549815) | > avg_loss_mel: 18.34322226887018 (+0.166619139657886) | > avg_loss_duration: 1.6655654554635706 (+0.014419283665401794) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.1807671936465 (+0.36489559227312185) | > avg_grad_norm_1: tensor(252.9825, device='cuda:0') (tensor(-23.5946, device='cuda:0'))  > EPOCH: 122/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:26:27)   --> TIME: 2025-02-25 09:26:35 -- STEP: 15/72 -- GLOBAL_STEP: 1008800 | > loss_disc: 2.923950672149658 (2.831955655415853) | > loss_disc_real_0: 0.37146639823913574 (0.22648312548796337) | > loss_disc_real_1: 0.25366106629371643 (0.2268927663564682) | > loss_disc_real_2: 0.2783689498901367 (0.2472155143817266) | > loss_disc_real_3: 0.24856501817703247 (0.23652251561482748) | > loss_disc_real_4: 0.25629523396492004 (0.24809630513191222) | > loss_disc_real_5: 0.2547435462474823 (0.24711072444915771) | > loss_0: 2.923950672149658 (2.831955655415853) | > grad_norm_0: tensor(20.2905, device='cuda:0') (tensor(9.7726, device='cuda:0')) | > loss_gen: 1.8229321241378784 (1.7345330635706584) | > loss_kl: 2.4480032920837402 (2.2713330427805585) | > loss_feat: 1.8094085454940796 (1.8212752183278402) | > loss_mel: 17.61872100830078 (17.903762435913087) | > loss_duration: 1.5943093299865723 (1.622736668586731) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.293373107910156 (25.353639984130858) | > grad_norm_1: tensor(95.4269, device='cuda:0') (tensor(242.9855, device='cuda:0')) | > current_lr_0: 0.00019697295072448832 | > current_lr_1: 0.00019697295072448832 | > step_time: 0.4258 (0.43217503229777016) | > loader_time: 0.0058 (0.006463766098022461)  --> TIME: 2025-02-25 09:26:59 -- STEP: 65/72 -- GLOBAL_STEP: 1008850 | > loss_disc: 2.838252544403076 (2.846258489902203) | > loss_disc_real_0: 0.1798495501279831 (0.22610808908939362) | > loss_disc_real_1: 0.20493249595165253 (0.2340854117503533) | > loss_disc_real_2: 0.2038046270608902 (0.24309938481220833) | > loss_disc_real_3: 0.21209362149238586 (0.24223960844370035) | > loss_disc_real_4: 0.22142428159713745 (0.24912587450100826) | > loss_disc_real_5: 0.24162685871124268 (0.2497988139207546) | > loss_0: 2.838252544403076 (2.846258489902203) | > grad_norm_0: tensor(12.6263, device='cuda:0') (tensor(8.4953, device='cuda:0')) | > loss_gen: 1.571721076965332 (1.745486017373892) | > loss_kl: 2.4094760417938232 (2.4321760911207924) | > loss_feat: 1.861517071723938 (1.8762896537780762) | > loss_mel: 18.608257293701172 (18.183348171527577) | > loss_duration: 1.6886506080627441 (1.6532926082611084) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.139619827270508 (25.890592604417066) | > grad_norm_1: tensor(91.4543, device='cuda:0') (tensor(225.9605, device='cuda:0')) | > current_lr_0: 0.00019697295072448832 | > current_lr_1: 0.00019697295072448832 | > step_time: 0.4916 (0.4458884422595684) | > loader_time: 0.0116 (0.008319979447584882) --> EVAL PERFORMANCE | > avg_loader_time: 0.008577682602573446 (+9.07333803848269e-06) | > avg_step_time: 0.450555337986476 (-0.0028498542140906746) | > avg_loss_disc: 2.8470702775767154 (-0.014544691838008994) | > avg_loss_disc_real_0: 0.22667445410305345 (-0.010925615127657495) | > avg_loss_disc_real_1: 0.2355778391512347 (-0.005655381251388869) | > avg_loss_disc_real_2: 0.24280611058356058 (-0.0023809113133121396) | > avg_loss_disc_real_3: 0.24281379895311006 (-0.002751382513785039) | > avg_loss_disc_real_4: 0.24810087806741957 (+0.000462429414332749) | > avg_loss_disc_real_5: 0.24879437706000362 (-0.00016639748929259124) | > avg_loss_0: 2.8470702775767154 (-0.014544691838008994) | > avg_grad_norm_0: tensor(8.5918, device='cuda:0') (tensor(-1.1784, device='cuda:0')) | > avg_loss_gen: 1.745465767215675 (-0.020774134447876902) | > avg_loss_kl: 2.4297727094569668 (-0.043930594350251706) | > avg_loss_feat: 1.8608976098853098 (-0.07113862205559096) | > avg_loss_mel: 18.215603626949694 (-0.12761864192048478) | > avg_loss_duration: 1.6563808817259023 (-0.009184573737668345) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.908120625455613 (-0.2726465681908863) | > avg_grad_norm_1: tensor(218.0619, device='cuda:0') (tensor(-34.9206, device='cuda:0'))  > EPOCH: 123/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:27:03)   --> TIME: 2025-02-25 09:27:25 -- STEP: 43/72 -- GLOBAL_STEP: 1008900 | > loss_disc: 2.8561251163482666 (2.838756910590238) | > loss_disc_real_0: 0.13409560918807983 (0.22877460926078086) | > loss_disc_real_1: 0.2331743836402893 (0.23214110212270603) | > loss_disc_real_2: 0.23000793159008026 (0.24140830407308977) | > loss_disc_real_3: 0.2176816761493683 (0.24288022379542507) | > loss_disc_real_4: 0.22239650785923004 (0.24872612745262856) | > loss_disc_real_5: 0.21443088352680206 (0.24781790063824766) | > loss_0: 2.8561251163482666 (2.838756910590238) | > grad_norm_0: tensor(12.8151, device='cuda:0') (tensor(10.5397, device='cuda:0')) | > loss_gen: 1.5940794944763184 (1.7504641399827114) | > loss_kl: 2.1333577632904053 (2.3482635575671527) | > loss_feat: 2.076111078262329 (1.8960026668947796) | > loss_mel: 17.52189826965332 (17.83137064201887) | > loss_duration: 1.636040210723877 (1.637688811435256) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.96148681640625 (25.46378956284634) | > grad_norm_1: tensor(365.6432, device='cuda:0') (tensor(230.1749, device='cuda:0')) | > current_lr_0: 0.00019694832910564775 | > current_lr_1: 0.00019694832910564775 | > step_time: 0.4553 (0.43779050472170805) | > loader_time: 0.0094 (0.007476734560589457) --> EVAL PERFORMANCE | > avg_loader_time: 0.008585778760238435 (+8.096157664989542e-06) | > avg_step_time: 0.45257450157487894 (+0.002019163588402928) | > avg_loss_disc: 2.838785319261148 (-0.008284958315567525) | > avg_loss_disc_real_0: 0.22723000528107226 (+0.0005555511780188094) | > avg_loss_disc_real_1: 0.23440956292857587 (-0.0011682762226588317) | > avg_loss_disc_real_2: 0.24236580645534353 (-0.00044030412821705056) | > avg_loss_disc_real_3: 0.2428501223174619 (+3.6323364351831255e-05) | > avg_loss_disc_real_4: 0.24680267862031158 (-0.0012981994471079894) | > avg_loss_disc_real_5: 0.24652250404928772 (-0.002271873010715908) | > avg_loss_0: 2.838785319261148 (-0.008284958315567525) | > avg_grad_norm_0: tensor(9.4997, device='cuda:0') (+tensor(0.9079, device='cuda:0')) | > avg_loss_gen: 1.7502692937850952 (+0.00480352656942018) | > avg_loss_kl: 2.3477806272641035 (-0.08199208219286325) | > avg_loss_feat: 1.8945434563596484 (+0.03364584647433855) | > avg_loss_mel: 17.88371499827211 (-0.3318886286775822) | > avg_loss_duration: 1.657033898461033 (+0.0006530167351308513) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.53334211967361 (-0.3747785057820039) | > avg_grad_norm_1: tensor(220.7684, device='cuda:0') (+tensor(2.7065, device='cuda:0'))  > EPOCH: 124/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:27:40)   --> TIME: 2025-02-25 09:27:51 -- STEP: 21/72 -- GLOBAL_STEP: 1008950 | > loss_disc: 2.890005111694336 (2.8216183412642706) | > loss_disc_real_0: 0.18214856088161469 (0.22428369664010547) | > loss_disc_real_1: 0.3023427426815033 (0.23201486397357213) | > loss_disc_real_2: 0.2648037374019623 (0.23834200983955747) | > loss_disc_real_3: 0.2893903851509094 (0.2396103214649927) | > loss_disc_real_4: 0.2806307077407837 (0.24531052864733197) | > loss_disc_real_5: 0.27087149024009705 (0.24588077408926828) | > loss_0: 2.890005111694336 (2.8216183412642706) | > grad_norm_0: tensor(8.9487, device='cuda:0') (tensor(8.2139, device='cuda:0')) | > loss_gen: 1.784650206565857 (1.7629162413733346) | > loss_kl: 2.2863447666168213 (2.278300466991606) | > loss_feat: 1.8982492685317993 (2.016915957132975) | > loss_mel: 18.275358200073242 (18.364219574701217) | > loss_duration: 1.6508169174194336 (1.6310684453873407) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.89542007446289 (26.053420566377184) | > grad_norm_1: tensor(145.9300, device='cuda:0') (tensor(194.0081, device='cuda:0')) | > current_lr_0: 0.00019692371056450955 | > current_lr_1: 0.00019692371056450955 | > step_time: 0.428 (0.4254894256591797) | > loader_time: 0.0071 (0.006669021788097564)  --> TIME: 2025-02-25 09:28:15 -- STEP: 71/72 -- GLOBAL_STEP: 1009000 | > loss_disc: 2.9104552268981934 (2.8279456857224585) | > loss_disc_real_0: 0.15555015206336975 (0.22338720253655608) | > loss_disc_real_1: 0.35701116919517517 (0.23550900134822012) | > loss_disc_real_2: 0.27141276001930237 (0.23971073413398905) | > loss_disc_real_3: 0.2282170057296753 (0.24114617222631482) | > loss_disc_real_4: 0.24343590438365936 (0.24586972489323414) | > loss_disc_real_5: 0.23319217562675476 (0.24703326338613532) | > loss_0: 2.9104552268981934 (2.8279456857224585) | > grad_norm_0: tensor(15.8892, device='cuda:0') (tensor(9.0383, device='cuda:0')) | > loss_gen: 1.9135470390319824 (1.7621658788600438) | > loss_kl: 2.427581310272217 (2.328605651855468) | > loss_feat: 1.9769450426101685 (1.9850555322539638) | > loss_mel: 18.264122009277344 (18.105255046360934) | > loss_duration: 1.6481826305389404 (1.6494275381867314) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.230377197265625 (25.8305095081598) | > grad_norm_1: tensor(400.1477, device='cuda:0') (tensor(239.0849, device='cuda:0')) | > current_lr_0: 0.00019692371056450955 | > current_lr_1: 0.00019692371056450955 | > step_time: 0.4987 (0.44952500705987636) | > loader_time: 0.0079 (0.008514505037119691) --> EVAL PERFORMANCE | > avg_loader_time: 0.008514505037119691 (-7.127372311874387e-05) | > avg_step_time: 0.44952500705987636 (-0.003049494515002582) | > avg_loss_disc: 2.8279456857224585 (-0.01083963353868933) | > avg_loss_disc_real_0: 0.22338720253655608 (-0.003842802744516177) | > avg_loss_disc_real_1: 0.23550900134822012 (+0.0010994384196442442) | > avg_loss_disc_real_2: 0.23971073413398905 (-0.002655072321354479) | > avg_loss_disc_real_3: 0.24114617222631482 (-0.0017039500911470717) | > avg_loss_disc_real_4: 0.24586972489323414 (-0.0009329537270774402) | > avg_loss_disc_real_5: 0.24703326338613532 (+0.0005107593368476049) | > avg_loss_0: 2.8279456857224585 (-0.01083963353868933) | > avg_grad_norm_0: tensor(9.0383, device='cuda:0') (tensor(-0.4614, device='cuda:0')) | > avg_loss_gen: 1.7621658788600438 (+0.011896585074948618) | > avg_loss_kl: 2.328605651855468 (-0.019174975408635664) | > avg_loss_feat: 1.9850555322539638 (+0.09051207589431542) | > avg_loss_mel: 18.105255046360934 (+0.22154004808882277) | > avg_loss_duration: 1.6494275381867314 (-0.007606360274301727) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.8305095081598 (+0.29716738848619073) | > avg_grad_norm_1: tensor(239.0849, device='cuda:0') (+tensor(18.3165, device='cuda:0'))  > EPOCH: 125/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:28:17)   --> TIME: 2025-02-25 09:28:41 -- STEP: 49/72 -- GLOBAL_STEP: 1009050 | > loss_disc: 2.818774700164795 (2.8248624266410363) | > loss_disc_real_0: 0.24909698963165283 (0.22409907363507212) | > loss_disc_real_1: 0.13779830932617188 (0.23242102076812665) | > loss_disc_real_2: 0.22807903587818146 (0.2422110420100543) | > loss_disc_real_3: 0.26300811767578125 (0.24065057568404138) | > loss_disc_real_4: 0.23879583179950714 (0.24436758489024882) | > loss_disc_real_5: 0.23150192201137543 (0.24425655208071884) | > loss_0: 2.818774700164795 (2.8248624266410363) | > grad_norm_0: tensor(18.1052, device='cuda:0') (tensor(9.2884, device='cuda:0')) | > loss_gen: 1.7628936767578125 (1.7565648677397747) | > loss_kl: 2.4074044227600098 (2.399165966072861) | > loss_feat: 1.9943798780441284 (1.980989626475743) | > loss_mel: 18.12413787841797 (17.988769219846148) | > loss_duration: 1.730853796005249 (1.6389616338574156) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.019670486450195 (25.764451357783102) | > grad_norm_1: tensor(141.2494, device='cuda:0') (tensor(212.5877, device='cuda:0')) | > current_lr_0: 0.000196899095100689 | > current_lr_1: 0.000196899095100689 | > step_time: 0.4417 (0.4531414022251051) | > loader_time: 0.0095 (0.007816601772697605) --> EVAL PERFORMANCE | > avg_loader_time: 0.008695649429106375 (+0.00018114439198668383) | > avg_step_time: 0.46439408248579 (+0.014869075425913647) | > avg_loss_disc: 2.833773549173919 (+0.00582786345146058) | > avg_loss_disc_real_0: 0.225186879156341 (+0.0017996766197849179) | > avg_loss_disc_real_1: 0.23343821583499372 (-0.0020707855132263975) | > avg_loss_disc_real_2: 0.24146947705409896 (+0.0017587429201099125) | > avg_loss_disc_real_3: 0.24174195905806314 (+0.0005957868317483239) | > avg_loss_disc_real_4: 0.24561249391293863 (-0.0002572309802955086) | > avg_loss_disc_real_5: 0.24814506924488175 (+0.0011118058587464286) | > avg_loss_0: 2.833773549173919 (+0.00582786345146058) | > avg_grad_norm_0: tensor(9.6313, device='cuda:0') (+tensor(0.5930, device='cuda:0')) | > avg_loss_gen: 1.758869881361303 (-0.003295997498740766) | > avg_loss_kl: 2.38931036667085 (+0.06070471481538231) | > avg_loss_feat: 1.9563452844888392 (-0.028710247765124652) | > avg_loss_mel: 17.9992759328493 (-0.10597911351163347) | > avg_loss_duration: 1.6514460318525073 (+0.002018493665775889) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.755247411593587 (-0.07526209656621319) | > avg_grad_norm_1: tensor(235.2316, device='cuda:0') (tensor(-3.8533, device='cuda:0'))  > EPOCH: 126/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:28:54)   --> TIME: 2025-02-25 09:29:08 -- STEP: 27/72 -- GLOBAL_STEP: 1009100 | > loss_disc: 2.841951370239258 (2.8134781078056053) | > loss_disc_real_0: 0.12627965211868286 (0.216663952226992) | > loss_disc_real_1: 0.24331654608249664 (0.2367878661111549) | > loss_disc_real_2: 0.2124910056591034 (0.23998147745927176) | > loss_disc_real_3: 0.24501897394657135 (0.23968066330309268) | > loss_disc_real_4: 0.2212524563074112 (0.24555735565997935) | > loss_disc_real_5: 0.23941448330879211 (0.24693304962582058) | > loss_0: 2.841951370239258 (2.8134781078056053) | > grad_norm_0: tensor(13.5117, device='cuda:0') (tensor(9.1342, device='cuda:0')) | > loss_gen: 1.666780948638916 (1.766521524499964) | > loss_kl: 2.339459180831909 (2.349463939666748) | > loss_feat: 1.9811453819274902 (2.0267884422231606) | > loss_mel: 17.373579025268555 (18.04697333441841) | > loss_duration: 1.5870237350463867 (1.6428396701812744) | > amp_scaler: 256.0 (256.0) | > loss_1: 24.947986602783203 (25.832586853592485) | > grad_norm_1: tensor(103.2812, device='cuda:0') (tensor(249.7748, device='cuda:0')) | > current_lr_0: 0.0001968744827138014 | > current_lr_1: 0.0001968744827138014 | > step_time: 0.4488 (0.43421280825579606) | > loader_time: 0.0082 (0.006987359788682725) --> EVAL PERFORMANCE | > avg_loader_time: 0.008625460342622137 (-7.018908648423819e-05) | > avg_step_time: 0.45181326463188926 (-0.01258081785390075) | > avg_loss_disc: 2.8220823146927523 (-0.011691234481166823) | > avg_loss_disc_real_0: 0.22281127737861284 (-0.0023756017777281646) | > avg_loss_disc_real_1: 0.23461623552819372 (+0.0011780196931999998) | > avg_loss_disc_real_2: 0.24214960865571467 (+0.0006801316016157022) | > avg_loss_disc_real_3: 0.24050946890468328 (-0.0012324901533798671) | > avg_loss_disc_real_4: 0.2457625752603504 (+0.00015008134741176238) | > avg_loss_disc_real_5: 0.2459797246355406 (-0.0021653446093411466) | > avg_loss_0: 2.8220823146927523 (-0.011691234481166823) | > avg_grad_norm_0: tensor(10.9283, device='cuda:0') (+tensor(1.2970, device='cuda:0')) | > avg_loss_gen: 1.7666843155740013 (+0.007814434212698229) | > avg_loss_kl: 2.3826140585079996 (-0.006696308162850606) | > avg_loss_feat: 1.988475646771175 (+0.0321303622823359) | > avg_loss_mel: 17.883978722800684 (-0.1152972100486167) | > avg_loss_duration: 1.661323280401633 (+0.009877248549125639) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.68307594514229 (-0.07217146645129802) | > avg_grad_norm_1: tensor(242.0226, device='cuda:0') (+tensor(6.7910, device='cuda:0'))  > EPOCH: 127/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:29:31)   --> TIME: 2025-02-25 09:29:35 -- STEP: 5/72 -- GLOBAL_STEP: 1009150 | > loss_disc: 2.8277738094329834 (2.8060186386108397) | > loss_disc_real_0: 0.2313452959060669 (0.2341061294078827) | > loss_disc_real_1: 0.18672674894332886 (0.22637152373790742) | > loss_disc_real_2: 0.26778197288513184 (0.22767284214496614) | > loss_disc_real_3: 0.23328058421611786 (0.24511174261569976) | > loss_disc_real_4: 0.262572318315506 (0.24833742082118987) | > loss_disc_real_5: 0.27166369557380676 (0.2655049741268158) | > loss_0: 2.8277738094329834 (2.8060186386108397) | > grad_norm_0: tensor(6.4152, device='cuda:0') (tensor(16.9097, device='cuda:0')) | > loss_gen: 1.8896304368972778 (1.7843027114868164) | > loss_kl: 2.386305570602417 (2.3622018814086916) | > loss_feat: 1.8696812391281128 (2.0358415842056274) | > loss_mel: 17.370006561279297 (17.472645568847657) | > loss_duration: 1.616824984550476 (1.6263823509216309) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.132450103759766 (25.281374359130858) | > grad_norm_1: tensor(305.6591, device='cuda:0') (tensor(344.6317, device='cuda:0')) | > current_lr_0: 0.00019684987340346216 | > current_lr_1: 0.00019684987340346216 | > step_time: 0.4247 (0.4272500514984131) | > loader_time: 0.0071 (0.006715822219848633)  --> TIME: 2025-02-25 09:29:58 -- STEP: 55/72 -- GLOBAL_STEP: 1009200 | > loss_disc: 2.901193618774414 (2.8282512751492583) | > loss_disc_real_0: 0.1904871165752411 (0.2267525705424222) | > loss_disc_real_1: 0.21548619866371155 (0.23644959222186696) | > loss_disc_real_2: 0.26972219347953796 (0.24150672473690726) | > loss_disc_real_3: 0.22535575926303864 (0.24210273135792126) | > loss_disc_real_4: 0.2556599974632263 (0.24528904448856007) | > loss_disc_real_5: 0.2389320582151413 (0.24863870739936828) | > loss_0: 2.901193618774414 (2.8282512751492583) | > grad_norm_0: tensor(5.4132, device='cuda:0') (tensor(10.0025, device='cuda:0')) | > loss_gen: 1.8313612937927246 (1.7792131900787354) | > loss_kl: 2.4013736248016357 (2.367713295329701) | > loss_feat: 2.310512065887451 (2.0266890309073715) | > loss_mel: 19.17637062072754 (17.99987248507413) | > loss_duration: 1.693312406539917 (1.6382179086858577) | > amp_scaler: 256.0 (256.0) | > loss_1: 27.41292953491211 (25.811705675992098) | > grad_norm_1: tensor(566.7577, device='cuda:0') (tensor(323.7551, device='cuda:0')) | > current_lr_0: 0.00019684987340346216 | > current_lr_1: 0.00019684987340346216 | > step_time: 0.4639 (0.4420209191062234) | > loader_time: 0.0093 (0.007883587750521573) --> EVAL PERFORMANCE | > avg_loader_time: 0.008596165079466053 (-2.9295263156084092e-05) | > avg_step_time: 0.4537681525861713 (+0.0019548879542820585) | > avg_loss_disc: 2.8278767155929345 (+0.005794400900182239) | > avg_loss_disc_real_0: 0.2288575096869133 (+0.006046232308300453) | > avg_loss_disc_real_1: 0.23778735856774827 (+0.003171123039554552) | > avg_loss_disc_real_2: 0.23999357076597885 (-0.0021560378897358123) | > avg_loss_disc_real_3: 0.2402767646060863 (-0.00023270429859698072) | > avg_loss_disc_real_4: 0.24575288572781523 (-9.689532535162027e-06) | > avg_loss_disc_real_5: 0.24772927500832248 (+0.0017495503727818773) | > avg_loss_0: 2.8278767155929345 (+0.005794400900182239) | > avg_grad_norm_0: tensor(10.1226, device='cuda:0') (tensor(-0.8057, device='cuda:0')) | > avg_loss_gen: 1.7733929325157487 (+0.006708616941747403) | > avg_loss_kl: 2.377646160797334 (-0.004967897710665525) | > avg_loss_feat: 1.9946658023646189 (+0.006190155593443825) | > avg_loss_mel: 17.94717431404221 (+0.06319559124152718) | > avg_loss_duration: 1.6497476252031997 (-0.011575655198433177) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.742626673738723 (+0.05955072859643451) | > avg_grad_norm_1: tensor(337.1365, device='cuda:0') (+tensor(95.1138, device='cuda:0'))  > EPOCH: 128/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:30:08)   --> TIME: 2025-02-25 09:30:25 -- STEP: 33/72 -- GLOBAL_STEP: 1009250 | > loss_disc: 2.795694589614868 (2.83488181143096) | > loss_disc_real_0: 0.18799112737178802 (0.24212682721289722) | > loss_disc_real_1: 0.21013467013835907 (0.23326307941566815) | > loss_disc_real_2: 0.2391277700662613 (0.23963144918282828) | > loss_disc_real_3: 0.23620478808879852 (0.241625320279237) | > loss_disc_real_4: 0.2484552413225174 (0.24426100064407696) | > loss_disc_real_5: 0.24903687834739685 (0.2468256385940494) | > loss_0: 2.795694589614868 (2.83488181143096) | > grad_norm_0: tensor(4.9793, device='cuda:0') (tensor(10.7701, device='cuda:0')) | > loss_gen: 1.830545425415039 (1.7814846652926821) | > loss_kl: 2.368116617202759 (2.3098214973102915) | > loss_feat: 2.0367233753204346 (2.0415604620268857) | > loss_mel: 17.88150978088379 (17.902870004827324) | > loss_duration: 1.6137900352478027 (1.6200078653566765) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.73068618774414 (25.655744726007637) | > grad_norm_1: tensor(360.7883, device='cuda:0') (tensor(280.6485, device='cuda:0')) | > current_lr_0: 0.00019682526716928672 | > current_lr_1: 0.00019682526716928672 | > step_time: 0.4362 (0.43516073804913147) | > loader_time: 0.0078 (0.007060513351902817) --> EVAL PERFORMANCE | > avg_loader_time: 0.00855813227908712 (-3.803280037893361e-05) | > avg_step_time: 0.45493515108672666 (+0.0011669985005553496) | > avg_loss_disc: 2.844413374511289 (+0.01653665891835443) | > avg_loss_disc_real_0: 0.23211425070611524 (+0.003256741019201953) | > avg_loss_disc_real_1: 0.24124150691737592 (+0.0034541483496276504) | > avg_loss_disc_real_2: 0.2408251982759422 (+0.0008316275099633508) | > avg_loss_disc_real_3: 0.24090599048305567 (+0.0006292258769693693) | > avg_loss_disc_real_4: 0.24498013711311448 (-0.0007727486147007523) | > avg_loss_disc_real_5: 0.24747921106681017 (-0.0002500639415123074) | > avg_loss_0: 2.844413374511289 (+0.01653665891835443) | > avg_grad_norm_0: tensor(9.6806, device='cuda:0') (tensor(-0.4420, device='cuda:0')) | > avg_loss_gen: 1.7587670591515554 (-0.014625873364193298) | > avg_loss_kl: 2.359549495535837 (-0.01809666526149689) | > avg_loss_feat: 1.9531441204984419 (-0.041521681866176996) | > avg_loss_mel: 17.8444962165725 (-0.10267809746970968) | > avg_loss_duration: 1.6467616843505644 (-0.0029859408526353715) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 25.562718780947407 (-0.17990789279131647) | > avg_grad_norm_1: tensor(270.7306, device='cuda:0') (tensor(-66.4059, device='cuda:0'))  > EPOCH: 129/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:30:45)   --> TIME: 2025-02-25 09:30:52 -- STEP: 11/72 -- GLOBAL_STEP: 1009300 | > loss_disc: 2.78977108001709 (2.8036254752766) | > loss_disc_real_0: 0.1699061244726181 (0.21109688011082736) | > loss_disc_real_1: 0.24348947405815125 (0.24026238918304443) | > loss_disc_real_2: 0.2030487060546875 (0.22087119790640744) | > loss_disc_real_3: 0.22969383001327515 (0.24797844344919379) | > loss_disc_real_4: 0.23330380022525787 (0.2543270831758326) | > loss_disc_real_5: 0.26339346170425415 (0.2521623522043228) | > loss_0: 2.78977108001709 (2.8036254752766) | > grad_norm_0: tensor(11.5306, device='cuda:0') (tensor(9.4941, device='cuda:0')) | > loss_gen: 1.8000524044036865 (1.802463022145358) | > loss_kl: 2.1783273220062256 (2.4183027744293213) | > loss_feat: 2.001811981201172 (2.0385421731255273) | > loss_mel: 18.585689544677734 (19.18806093389338) | > loss_duration: 1.6582152843475342 (1.670241724361073) | > amp_scaler: 256.0 (256.0) | > loss_1: 26.224096298217773 (27.117610931396484) | > grad_norm_1: tensor(245.5950, device='cuda:0') (tensor(275.0335, device='cuda:0')) | > current_lr_0: 0.00019680066401089056 | > current_lr_1: 0.00019680066401089056 | > step_time: 0.4353 (0.43292823704806244) | > loader_time: 0.0066 (0.006322622299194336)  --> TIME: 2025-02-25 09:31:15 -- STEP: 61/72 -- GLOBAL_STEP: 1009350 | > loss_disc: 2.7552599906921387 (2.824674696218772) | > loss_disc_real_0: 0.26640552282333374 (0.22035052920462656) | > loss_disc_real_1: 0.2476954609155655 (0.23595801191251786) | > loss_disc_real_2: 0.22691334784030914 (0.23862036500797898) | > loss_disc_real_3: 0.21638743579387665 (0.24299652508047762) | > loss_disc_real_4: 0.22897575795650482 (0.24643884525924434) | > loss_disc_real_5: 0.24214543402194977 (0.2468535264984506) | > loss_0: 2.7552599906921387 (2.824674696218772) | > grad_norm_0: tensor(11.9358, device='cuda:0') (tensor(10.4034, device='cuda:0')) | > loss_gen: 1.7756869792938232 (1.7684538892058075) | > loss_kl: 2.282668113708496 (2.3992686467092543) | > loss_feat: 2.0358409881591797 (2.0037090465670726) | > loss_mel: 18.027223587036133 (18.519225855342686) | > loss_duration: 1.6698039770126343 (1.652024245652996) | > amp_scaler: 256.0 (256.0) | > loss_1: 25.791223526000977 (26.342681665889554) | > grad_norm_1: tensor(557.5346, device='cuda:0') (tensor(236.3007, device='cuda:0')) | > current_lr_0: 0.00019680066401089056 | > current_lr_1: 0.00019680066401089056 | > step_time: 0.4622 (0.44979213495723536) | > loader_time: 0.0107 (0.008213778011134414) --> EVAL PERFORMANCE | > avg_loader_time: 0.008664027066297934 (+0.00010589478721081437) | > avg_step_time: 0.45815833521560884 (+0.0032231841288821728) | > avg_loss_disc: 2.825978222027631 (-0.0184351524836579) | > avg_loss_disc_real_0: 0.22131369304908832 (-0.010800557657026921) | > avg_loss_disc_real_1: 0.23578767772291748 (-0.005453829194458443) | > avg_loss_disc_real_2: 0.23801871000880925 (-0.0028064882671329505) | > avg_loss_disc_real_3: 0.24272167074008727 (+0.0018156802570316044) | > avg_loss_disc_real_4: 0.24572474247133227 (+0.000744605358217798) | > avg_loss_disc_real_5: 0.24673093960318768 (-0.0007482714636224963) | > avg_loss_0: 2.825978222027631 (-0.0184351524836579) | > avg_grad_norm_0: tensor(10.6788, device='cuda:0') (+tensor(0.9982, device='cuda:0')) | > avg_loss_gen: 1.765042889286095 (+0.006275830134539584) | > avg_loss_kl: 2.398116884097247 (+0.03856738856140973) | > avg_loss_feat: 1.991447260682012 (+0.03830314018357006) | > avg_loss_mel: 18.48884128516829 (+0.6443450685957899) | > avg_loss_duration: 1.6573263171692967 (+0.010564632818732322) | > avg_amp_scaler: 256.0 (+0.0) | > avg_loss_1: 26.300774560847753 (+0.7380557799003462) | > avg_grad_norm_1: tensor(239.3034, device='cuda:0') (tensor(-31.4272, device='cuda:0'))  > EPOCH: 130/1500 --> voco_logs/voco-single-February-25-2025_08+10AM-dbf1a08a  > TRAINING (2025-02-25 09:31:22)