title: Empower Large Language Model to Perform Better on Industrial
    Domain-Specific Question Answering
summary: Large Language Model (LLM) has gained popularity and achieved remarkable
  results in open-domain tasks, but its performance in real industrial
  domain-specific scenarios is average since there is no specific knowledge in
  it. This issue has attracted widespread attention, but there are few relevant
  benchmarks available. In this paper, we provide a benchmark Question Answering
  (QA) dataset named MSQA, which is about Microsoft products and IT technical
  problems encountered by customers. This dataset contains industry
  cloud-specific QA knowledge, which is not available for general LLM, so it is
  well suited for evaluating methods aimed at improving domain-specific
  capabilities of LLM. In addition, we propose a new model interaction paradigm
  that can empower LLM to achieve better performance on domain-specific tasks
  where it is not proficient. Extensive experiments demonstrate that the approach
  following our model fusion framework outperforms the commonly used LLM with
  retrieval methods.
paper_authors._id: 646ad1a8c37ca1e1230c8a23
paper_authors.name: Zezhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:00:36.444Z
paper_authors._id: 642d6b792320338df27dd884
paper_authors.avatarUrl: /avatars/f76290e059a0e15cee1d755bc1e8f0ba.svg
paper_authors.fullname: Zezhong Wnag
paper_authors.type: user
paper_authors.user: AwesomeW
paper_authors._id: 646ad1a8c37ca1e1230c8a24
paper_authors.name: Fangkai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:19:06.996Z
paper_authors._id: 669dcf6200970c3b27aafa5d
paper_authors.avatarUrl: /avatars/bb9ed5ff86326fdaeb184c6b0e40f74f.svg
paper_authors.fullname: kaikai yang
paper_authors.type: user
paper_authors.user: keanudicap
paper_authors._id: 646ad1a8c37ca1e1230c8a25
paper_authors.name: Pu Zhao
paper_authors._id: 646ad1a8c37ca1e1230c8a26
paper_authors.name: Lu Wang
paper_authors._id: 646ad1a8c37ca1e1230c8a27
paper_authors.name: Jue Zhang
paper_authors._id: 646ad1a8c37ca1e1230c8a28
paper_authors.name: Mohit Garg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:01:17.361Z
paper_authors._id: 643425c8ea46c00990480f60
paper_authors.avatarUrl: /avatars/012e77ac7e5d755050ea065b723a6b60.svg
paper_authors.fullname: Mohit Garg
paper_authors.type: user
paper_authors.user: mohitgargai
paper_authors._id: 646ad1a8c37ca1e1230c8a29
paper_authors.name: Qingwei Lin
paper_authors._id: 646ad1a8c37ca1e1230c8a2a
paper_authors.name: Dongmei Zhang
paper_title: Empower Large Language Model to Perform Better on Industrial
    Domain-Specific Question Answering
paper_summary: Large Language Model (LLM) has gained popularity and achieved remarkable
  results in open-domain tasks, but its performance in real industrial
  domain-specific scenarios is average since there is no specific knowledge in
  it. This issue has attracted widespread attention, but there are few relevant
  benchmarks available. In this paper, we provide a benchmark Question Answering
  (QA) dataset named MSQA, which is about Microsoft products and IT technical
  problems encountered by customers. This dataset contains industry
  cloud-specific QA knowledge, which is not available for general LLM, so it is
  well suited for evaluating methods aimed at improving domain-specific
  capabilities of LLM. In addition, we propose a new model interaction paradigm
  that can empower LLM to achieve better performance on domain-specific tasks
  where it is not proficient. Extensive experiments demonstrate that the approach
  following our model fusion framework outperforms the commonly used LLM with
  retrieval methods.
paper_ai_keywords: Large Language Model (LLM) Question Answering (QA) domain-specific scenarios benchmark MSQA cloud-specific QA knowledge model fusion framework retrieval methods