Performance Sharing
For the two examples in the README, it seems to be cherry-picked.
Actually using the same chart and change to a more general question, it is non-sensing.
text_prompt = "What is this chart demostrating?\n"
Result: The chart demonstrates that life expectancy at birth has increased steadily over the years. In 2008, life expectancy at birth was 78.2 years, and 80.2 years was reached in 2017. The graph shows a steady upward trend, indicating a continued improvement in life expectancy. The graph also shows that life expectancy at birth has expanded significantly between 2008 and 2017, from 80.2 years to 80.5 years.
So, from my perspective, there is a huge space for improvement and actually cannot put into real use.
text_prompt = "What is this screenshot about?\n"
Result: The screenshot displays a message on a cell phone, which reads "New Vision-Language Model: BakLaVa-1, finetuned on Mistral 7B." The message appears to be from a person who is discussing a new model. The message also hints at a new message, which appears to be from another person. The messages are accompanied by two icons, one on the left and another on the right side of the screen.
Hey
@Starlento
, the prompt structure for that base model does seem to matter, as the model was trained on particular prompts. Prepending a specific prompt with "Answer the following DocVQA question based on the image." seems to do better for instance.