I want to keep files in my git master, which should not be downloaded when the git is cloned. Only running an additional script should download these optional files (some trained models).
Is there any git way (e.g. a special command)/pythonic way to do this?
I want to keep files in my git master, which should not be downloaded when the git is cloned.

Impossible. Either store the files in a different branch or in an entirely different repository from where a script will clone them.
How to clone/load special data mentioned in gitignore from master