You should not run into an OOM and yes, some algorithms might try to use a huge workspace, if they need to e.g. create copies to permute and realign data.
As a workaround, you could set CUDNN_CONV_WSCAP_DBG to a lower value in MiB to skip algorithms requesting a larger workspace (description of this env var 3, proposal to expose this mechanism 5).