title: DeepSeek-VL: Towards Real-World Vision-Language Understanding
summary: We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed
  for real-world vision and language understanding applications. Our approach is
  structured around three key dimensions:
    We strive to ensure our data is diverse, scalable, and extensively covers
  real-world scenarios including web screenshots, PDFs, OCR, charts, and
  knowledge-based content, aiming for a comprehensive representation of practical
  contexts. Further, we create a use case taxonomy from real user scenarios and
  construct an instruction tuning dataset accordingly. The fine-tuning with this
  dataset substantially improves the model's user experience in practical
  applications. Considering efficiency and the demands of most real-world
  scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently
  processes high-resolution images (1024 x 1024), while maintaining a relatively
  low computational overhead. This design choice ensures the model's ability to
  capture critical semantic and detailed information across various visual tasks.
  We posit that a proficient Vision-Language Model should, foremost, possess
  strong language abilities. To ensure the preservation of LLM capabilities
  during pretraining, we investigate an effective VL pretraining strategy by
  integrating LLM training from the beginning and carefully managing the
  competitive dynamics observed between vision and language modalities.
    The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user
  experiences as a vision-language chatbot in real-world applications, achieving
  state-of-the-art or competitive performance across a wide range of
  visual-language benchmarks at the same model size while maintaining robust
  performance on language-centric benchmarks. We have made both 1.3B and 7B
  models publicly accessible to foster innovations based on this foundation
  model.
paper_authors._id: 65ee78abfc59f6e77e7e805b
paper_authors.name: Haoyu Lu
paper_authors._id: 65ee78abfc59f6e77e7e805c
paper_authors.name: Wen Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:47:50.767Z
paper_authors._id: 63198e8802fb322037332f2d
paper_authors.avatarUrl: /avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
paper_authors.fullname: Wen Liu
paper_authors.type: user
paper_authors.user: doubility123
paper_authors._id: 65ee78abfc59f6e77e7e805d
paper_authors.name: Bo Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T13:33:58.649Z
paper_authors._id: 6237f3eb4cf571bb1ed4036b
paper_authors.avatarUrl: /avatars/cbfb8ddda18e65d18a818db9f176e16e.svg
paper_authors.fullname: Bo Zhang
paper_authors.type: user
paper_authors.user: zhangmozhe
paper_authors._id: 65ee78abfc59f6e77e7e805e
paper_authors.name: Bingxuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:48:38.212Z
paper_authors._id: 6523d81d56fe05f216a559f6
paper_authors.avatarUrl: /avatars/07fcf56b5b8a0b64c31bdfe8fbf41cc6.svg
paper_authors.fullname: Bingxuan Wang
paper_authors.type: user
paper_authors.user: YellowDoge
paper_authors._id: 65ee78abfc59f6e77e7e805f
paper_authors.name: Kai Dong
paper_authors._id: 65ee78abfc59f6e77e7e8060
paper_authors.name: Bo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-18T08:36:43.367Z
paper_authors._id: 635e3a76106f984574c36409
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
paper_authors.fullname: Bo Liu
paper_authors.type: user
paper_authors.user: Benjamin-eecs
paper_authors._id: 65ee78abfc59f6e77e7e8061
paper_authors.name: Jingxiang Sun
paper_authors._id: 65ee78abfc59f6e77e7e8062
paper_authors.name: Tongzheng Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:49:40.164Z
paper_authors._id: 64f6b67db6365b4e44e5716c
paper_authors.avatarUrl: /avatars/f5ef15bfa5c42e3c57f301db7efc38a1.svg
paper_authors.fullname: Tongzheng Ren
paper_authors.type: user
paper_authors.user: tongzheng
paper_authors._id: 65ee78abfc59f6e77e7e8063
paper_authors.name: Zhuoshu Li
paper_authors._id: 65ee78abfc59f6e77e7e8064
paper_authors.name: Yaofeng Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:50:32.161Z
paper_authors._id: 656b0b6cc4d6794d7f367fed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vdJ3GU8J-9YfMJ7lI4bif.jpeg
paper_authors.fullname: Yaofeng Sun
paper_authors.type: user
paper_authors.user: yaofengsun
paper_authors._id: 65ee78abfc59f6e77e7e8065
paper_authors.name: Chengqi Deng
paper_authors._id: 65ee78abfc59f6e77e7e8066
paper_authors.name: Hanwei Xu
paper_authors._id: 65ee78abfc59f6e77e7e8067
paper_authors.name: Zhenda Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:50:54.235Z
paper_authors._id: 6539f6ea26df26ecd1393c37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
paper_authors.fullname: Zhenda Xie
paper_authors.type: user
paper_authors.user: zdaxie
paper_authors._id: 65ee78abfc59f6e77e7e8068
paper_authors.name: Chong Ruan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-17T05:38:38.710Z
paper_authors._id: 6398203609f12714ed1935c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
paper_authors.fullname: Chong Ruan
paper_authors.type: user
paper_authors.user: Chester111
paper_title: DeepSeek-VL: Towards Real-World Vision-Language Understanding
paper_summary: We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed
  for real-world vision and language understanding applications. Our approach is
  structured around three key dimensions:
    We strive to ensure our data is diverse, scalable, and extensively covers
  real-world scenarios including web screenshots, PDFs, OCR, charts, and
  knowledge-based content, aiming for a comprehensive representation of practical
  contexts. Further, we create a use case taxonomy from real user scenarios and
  construct an instruction tuning dataset accordingly. The fine-tuning with this
  dataset substantially improves the model's user experience in practical
  applications. Considering efficiency and the demands of most real-world
  scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently
  processes high-resolution images (1024 x 1024), while maintaining a relatively
  low computational overhead. This design choice ensures the model's ability to
  capture critical semantic and detailed information across various visual tasks.
  We posit that a proficient Vision-Language Model should, foremost, possess
  strong language abilities. To ensure the preservation of LLM capabilities
  during pretraining, we investigate an effective VL pretraining strategy by
  integrating LLM training from the beginning and carefully managing the
  competitive dynamics observed between vision and language modalities.
    The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user
  experiences as a vision-language chatbot in real-world applications, achieving
  state-of-the-art or competitive performance across a wide range of
  visual-language benchmarks at the same model size while maintaining robust
  performance on language-centric benchmarks. We have made both 1.3B and 7B
  models publicly accessible to foster innovations based on this foundation
  model.
paper_ai_keywords: Vision-Language (VL) Model real-world vision and language understanding web screenshots PDFs OCR charts knowledge-based content real user scenarios use case taxonomy instruction tuning dataset fine-tuning hybrid vision encoder high-resolution images computational overhead semantic and detailed information strong language abilities LLM capabilities VL pretraining strategy competitive dynamics vision and language modalities vision-language chatbot visual-language benchmarks language-centric benchmarks foundation model