Can I use OS X Server 4 to remotely manage an OS X Server 3.2.1? In other words: Do I have to update both my Macbook (from where I manage the remote OS X Server) and the Mac Mini (where the Server is actually serving) to Yosemite and OS X Server 4 at the same time?  

What do you mean: "Nothing can go wrong: all your Keychains; SSH keys; Certificates; Open Directory settings; web apps; .... will migrate just faultlessly."?

I think my strategy will be to upgrade the Server machine first and only manage it locally until I am sure it's working perfectly. (Not an option for those without local access to their OS X Server.)
No you can't. 


Server 3 won't run on Yosemite.
So management of remote Server 3 is not possible with Yosemite client.
Server 4 client on Yosemite alerts that it "will only connect to remote OS X Server running on OS X 10.10"
In other words: both Server client app and Server server need to be at Server 4 on OS X 10.10 if either end switches over to Yosemite.