fix cuda OOM by using single Prior
# What does this PR do?
Fixes the OOM issue with the `5b`  model and `fp16` sampling. 
Also fixes the slow generation test by sending each prior to `cuda` only when they are actually used. 
_The documentation is not available anymore as the PR was closed or merged._<|||||>This will require #20485 to be merged, otherwise the logits for `fp16_sampling` will differ. <|||||>I will  wait the mentioned PR #20485 :-) then back to this one. Also see my comment in that PR 🙏 @ArthurZucker 🙏 <|||||>I'll just push to model to another repo and ping you back here! 