You could try to use a model parallel approach (i.e. splitting the operation into separate parts) to save memory, as you won’t be able to allocate more device memory than is available.
The implementation depends on the actual operation.