The model roumbles forever
The model is not running correctly. Sometimes it gets into endless loops.
I run the model in the following way:
docker run --runtime nvidia --gpus all --name vllm-mediphi -v ~/.cache/huggingface:/root/.cache/huggingface --env "HUGGING_FACE_HUB_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model microsoft/MediPhi-Instruct --max_model_len 8000
When I prompt it with something like "Hey" it responds forever. Here is what it outputs:
Hello! How can I assist you today? Greetings! I'm Phi, Phi, Phi, Phi for short, at your service. What can I do for you today? Hi there! I'm Phi, Phi, Phi at your service. What's on your mind? Hello! I'm Phi, Phi, Phi and ready to help. What can I do for you today? Esmeralda, esmeralda! How can Phi assist you today? Hello, Phi Phi Phi is here! What questions do you have for me? Greetings! I'm Phi, Phi, Phi ready to lend a hand. What can I do for you today? Hi! Phi Phi Phi at your service. What's on your mind? Seeya! Phi Phi Phi is ready to assist you with any questions or tasks. Hello! Phi Phi Phi is here to help you. What can I do for you today? Hi! Phi Phi Phi is here. How can I assist you today? Hello again! I'm Phi Phi Phi, your friendly assistant. How can I help? Phi Phi Phi here to lend a hand! What can I do for you? Hello! Phi Phi Phi here. How can I assist you today? Hi! I'm Phi Phi Phi. What can I do for you today? Hi! I'm Phi Phi Phi. How can I be of assistance to you today? Hi there! I'm Phi Phi Phi, ready to assist you. What's on your mind? Hello! Phi Phi Phi is here to help you. What can I do for you today? Phi Phi Phi at your service! What can I help you with today? Hello! I'm Phi Phi Phi, ready to assist with any questions you have. What can I do for you today? Hello! Phi Phi Phi is your virtual assistant. How can I assist you today? Greetings! I'm Phi Phi Phi, here to help. What can I do for you today? Hi Phi Phi Phi! What can I help you with today? Hi, I'm Phi Phi Phi, here to help you. How can I assist you? Hi! I'm Phi Phi Phi, ready to assist you with any help you need. What's on your mind? Hello! I'm Phi Phi Phi, Phi, Phi here to assist. How can I be of service today? Hi Phi Phi Phi! How can I assist you today? Phi Phi Phi, here to serve! What can I do for you today? Hi there! I'm Phi Phi Phi. How can Phi assist you today? Greetings! I'm Phi Phi Phi, Phi, Phi ready to assist you. What can I help you with? Hello! I'm Phi Phi Phi, ready to assist you. How can I help with your questions today? Hello! Phi Phi Phi is here to help. What can I help you with today? Hi Phi! I'm Phi Phi Phi, here to help you with any questions you might have. Greetings! I'm Phi Phi Phi, Phi, Phi. How can I assist you today? Hi Phi Phi Phi! How can I assist you today? Phi Phi Phi, Phi, Phi here to help you! What can I do for you? Hi! I'm Phi Phi Phi, ready to assist you today. How can Phi be of help? Phi Phi Phi at your service! How can I be of assistance today? Hi Chan, Phi Phi Phi here to help! What can I do for you today? Hello! Phi Phi Phi is here to assist. Do you have any questions or tasks for me today? Hello, I'm Phi Phi Phi. How can Phi be of assistance today? Greetings! I'm Phi Phi Phi at your service. How can Phi help you today? Hello! Phi Phi Phi, Phi, Phi here to assist you. How can I assist you today? Hi there! I'm Phi, Phi, Phi. How can I help you today...
@gchalakov thanks for raising this issue.
As a temporary fix, please use other stopping criteria such as the template tokens (e.g. <|end|>) instead of the <|endoftext|> token. We will correct this minor issue in a future update.
Best regards,