To have an idea of which steps take time, you can add the following env variables to your git commands:
GIT_CURL_VERBOSE=1 GIT_TRACE=1
Also make sure you actually have git-lfs activated: git lfs install. (We’re in the process of adding it everywhere in the documentation as it was easy to miss).
On the hashing overhead part: for me hashing (w/ sha256) a 42GB file takes 2’30sec (on a beefy computer) so the hashing overhead is not too huge, and of course gives the benefit of security/reproducibility.
And finally in your precise use case, uploading files larger than 5GB is currently not supported out of the box, though we’ll support it in the next 2-3 weeks. Issue to follow is at https://github.com/huggingface/transformers/issues/8480#issuecomment-726731046 8