Especially for a shared server, I would recommend to have as few things in the global install as possible. That gives user freedom to install whichever variant they want. And prevent them from breaking the whole system if they do bad things.
In particular because installing cuda within conda is very simple and safe.
So I would suggest:
(for everyone): driver -> Conda
(for each user): create env -> install pytorch (that will pull cuda automatically)
Note that you can also make users install their own conda as well if you want to ensure more separation between users.
The benefit of the share conda is that packages will be re-used, reducing disk footprint.