Fix SAM
# What does this PR do?

Some failures are introduced in #23656. I only checked the TF tests but not the PT tests. Sorry.

cc @Rocketknight1 : it's likely a hardware difference. Always nice if we can get the results from GCP VM, but I understand it makes your workflow a bit difficult.
Forgot to mention: kudo to @younesbelkada  for spotting this <|||||>_The documentation is not available anymore as the PR was closed or merged._<|||||>The tol used in the tests changed in this PR are not changed in #23656, so nothing to be reverted back.