I just had a look into the code of the Huggingface Sagemaker Inference Toolkit and found an even easier way to accomplish this: You can pass parameters in your inference request: sagemaker-huggingface-inference-toolkit/handler_service.py at main · aws/sagemaker-huggingface-inference-toolkit · GitHub 2
That’s awesome, because it means you don’t even have to provide an inference.py script! 
All you need to do is:
predictor.predict({"inputs": "I love using the new Inference DLC.", "parameters": {"return_all_scores": True}})


Capture840×101 4.16 KB

Hope that helps!