Closing here, the GPU usage is still inconsistent but the training problem was mainly due to bad quality data. Thanks @rgwatwormhill for the attention!  I will be opening an issue at the github repo regarding GPU usage.