Fix SAM tests and use smaller checkpoints
This PR moves all the SAM tests for both PT and TF to the `sam-vit-base` instead of the `sam-vit-huge` (!) checkpoint they were using before. The huge checkpoint made the tests quite slow and caused OOM issues in TensorFlow.

It also fixes an issue with the `check_pt_tf_equivalence` test in the PyTorch tests, which should now pass correctly.
_The documentation is not available anymore as the PR was closed or merged._<|||||>@ydshieh Ah! That test was also sneakily loading `sam-vit-huge`. I've fixed it, it should work fine now.<|||||>All pass now 🚀 