How should I uninstall a static ffmpeg binary? I've downloaded ffmpeg static binary. If I wanted to 'uninstall' it from my computer, is deleting the static binary enough ?

My understanding is that static binary contains all of the required libraries/code within itself and therefore it does not need to 'install' itself per se… Am I correct in saying that ? 

So I never compiled it, never used homebrew, just downloaded the file, extracted it to downloads folder and used it to convert videos.
Yes, since you didn't install either via script, a package manager or an .pkg you can simply delete the file and you're done.