## HBot: A Chatbot for Healthcare Applications in Traditional Chinese Medicine Based on Human Body 3D Visualization

Bolin Zhang Harbin Institute of Technology Harbin, China brolin@hit.edu.cn

Zhiwei Yi Harbin Institute of Technology Weihai, China 22S030149@stu.hit.edu.cn

Jiahao Wang Chinese Academy of Sciences Beijing, China jiahaowang0917@gmail.com

Dianbo Sui Harbin Institute of Technology Weihai, China suidianbo@hit.edu.cn

## Zhiying Tu ∗

Harbin Institute of Technology Weihai, China tzy\_hit@hit.edu.cn

## ABSTRACT

The unique diagnosis and treatment techniques and remarkable clinical efficacy of traditional Chinese medicine (TCM) make it play an important role in the field of elderly care and healthcare, especially in the rehabilitation of some common chronic diseases of the elderly. Therefore, building a TCM chatbot for healthcare application will help users obtain consultation services in a direct and natural way. However, concepts such as acupuncture points (acupoints) and meridians involved in TCM always appear in the consultation, which cannot be displayed intuitively. To this end, we develop a h ealthcare chat bot (HBot) based on a human body model in 3D and knowledge graph, which provides conversational services such as knowledge Q&amp;A, prescription recommendation, moxibustion therapy recommendation, and acupoint search. When specific acupoints are involved in the conversations between user and HBot, the 3D body will jump to the corresponding acupoints and highlight them. Moreover, Hbot can also be used in training scenarios to accelerate the teaching process of TCM by intuitively displaying acupuncture points and knowledge cards. The demonstration video is available at https://www.youtube.com/watch?v=UhQhutSKkTU. Our code and dataset are publicly available at Gitee 1 .

## CCS CONCEPTS

· Information systems → Computing platforms ; · Computing methodologies → Natural language generation .

## KEYWORDS

Dialog systems, Traditional Chinese Medicine, 3D Visualization

## 1 INTRODUCTION

Online medical dialogue systems have been playing an increasingly important role[2] in healthcare and medical care, with increasing adoption of such chatbots by patients, caregivers, and clinicians. Chatbots support a more direct and natural way of human-computer interaction via text-based or voice-based communication methods, which makes them suitable for a variety of target groups from young children to the elderly[13] and is an ideal candidate in the healthcare field for remote interventions on patients after discharge. To this end, the amount of medical dialogue systems is rising, and they

∗ Corresponding Author.

1 https://gitee.com/plabrolin/interactive-3d-acup.git

Dianhui Chu ∗ Harbin Institute of Technology Weihai, China chudh@hit.edu.cn cover a wide range of categories include mental health, physical health, health information, patient assistance, physician assistance, cognitive or developmental health[14].

However, the TCM-related dialogue system has received less attention. Existing works mainly focuses on TCM diagnosis of diseases[6], medical record management[12] and knowledge base construction[4], while they do not have the ability to communicate with humans in natural language. Since the acupuncture points and meridians involved in TCM always appear in the consultation, we hope to tell users the clear location of these acupoints in an intuitive way. Therefore, we develop a h ealthcare chat bot (HBot) based on a human body model in 3D and knowledge graph, which provides conversational services such as knowledge Q&amp;A, prescription recommendation, moxibustion therapy recommendation, and acupoint search.

The user interface for this system is shown in Fig 1. The interface is divided into four parts: i)the top area is the navigation bar where users can search an acupoint visually, select registration or login page, and obtain API open capability. ii) the right area is the dashboard of 3D body where users can switch parts of the human body, zoom in&amp;out and rotate the body, and click on specific acupoints iii) the left-top area is the dialogue frame where users can chat with HBot in text form and give instructions to operate the 3D body, and iv) the left-bottom area is the multimedia board where the pictures or videos related to acupuncture points and meridians will be displayed to help users understand the TCM-knowledge.

The main contributions are summarized as follows:

- · For the localization and display of acupoints involved in dialogues, we propose an interactive human body in 3D.
- · To support the consultation services, we construct a knowledge graph (TCM-KG) and annotate a document-level entity&amp;relation extraction dataset in TCM.
- · Human evaluation results demonstrate the robustness of Hbot. Functions such as intent detection, entity&amp;relation extraction, and knowledge query will be publicly released in our platform as open APIs.

## 2 SYSTEM ARCHITECTURE

The architecture of the HBot system is shown in Fig 2, where illustrates an example of interaction flow. HBot contains 6 key modules,

Figure 1: The interface of HBot.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00481v1-figure-6b3319f7-3343-4ef5-ba6d-8d6e38d03db1.jpg)

Interactive 3D Body, User Intent Detection, Entity&amp;Relation Extraction, LLM Handler, LLM Wrapper, and Knowledge Graph Construction. User Intent Detection module first classifies user queries and the slot filling task will be finished by the entity extraction module. To achieve multi-turns of dialogue, the history of mentioned entities and user intents will be be saved in a list for dialogue state tracking. According to the dialogue tracking history, the LLM Handler decides whether to trigger the module of Interactive 3D Body or retrieves from the knowledge graph (TCM-KG). Then, the execution result of LLM Handler will be fed into the LLM Wrapper to generate the response. Based on Knowledge Graph Construction, the triples obtained from Entity&amp;Relation Extraction will be imported into Neo4j for the knowledge growth of TCM-KG. The module of Interactive 3D Body will be activated when user give an instruction or the acupoints are involved in the conversation. These modules are described in detail in the following sections.

## 3 INTERACTIVE 3D BODY

To visually display the meridians and acupoints of the human, we constructed an interactive 3D model of the human body. Considering compatibility of operating systems, simplicity of installation and diversity of interactions, we choose to embed our model into a web page and users can interact with the 3D body only through a web browser. Moreover, we pack the operations of the 3D body into javascript API for the developer to call. The implementation of the 3D body is the following steps:

1.According to the State Standard of the Location of Acupoints[5], we manually make an adult male body and mark 409 acupoints by using 3D graphics software (blender).

2.Using the GLTFLoader of three.js, the 3D object files (.glb and .hdr) of the body will be loaded in the web page for real-time rendering and displaying.

3.Use the camera controller (controls.js) of three.js to implement 360-degree rotation of the model, switching, zooming and other operations.

4.Using the event listener functions (onclick, onchange, onmouseover, etc) of Javascript, the switching of body components and the selection and restoration of acupuncture points can be realized.

Figure 2: The interaction flow of HBot.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00481v1-figure-a1397260-c281-4b28-8bfe-e7e9fc2b7729.jpg)

## 4 USER INTENT DETECTION

User Intent Detection is framed as a sentence classification task that aims to identify intents from human utterances, and plays the key role in dialogue systems[1]. However, training intent detection models relies upon a large set of user utterances paired with intents, which are predefined by the developers. In practice, the user will express new intents that may not be expected by the tested system, referred to as out-of-define (OOD) intents. To alleviate this issue, we propose a configurable intent detection process by applying three-phase: i)rule-based intent matching ii) model-based intent understanding and iii) similarity-based query searching.

Phase I : an intent-table is maintained in the Redis database, where one predefined intent is mapped to 5 example utterances and the related regular expressions. We defined two types of intents: Body manipulation related (i.e. rotation, zoom in, zoom out) and dialog content related (i.e. asking about treatment, asking about symptoms). When Hbot is running, the user utterance will be matched to a certain intent based on the regular expressions. If the match fails, Hbot will enter the second phase: model-based intent understanding.

Phase II : we applied the complex intent detection model ConcoERNIE[17], which is effective for both multiple intents and implicit intent detection by exploiting co-occurrence patterns between concepts appeared in medical queries and user intents conveyed by these queries. Given a user query, Conco-ERNIE can predict the probability distribution of the utterance on 21 common medical intents. If the probability of each intent is less than the threshold (0.6), Hbot will enter the third phase: similarity-based query searching.

Phase III : We collected a real question-and-answer corpus between patients and doctors (about 100k Q&amp;A pairs) from medical online forum 2 . To ensure the quality of the corpus, we only select the Q&amp;A pairs where the question has a bounty and the answer is adopted by the patient. To find the approximate answers, we trained a sentence embedding model SBERT[11] with siamese network architecture to calculate the similarity between the input sentence and each query in the corpus. Then, the answer to the query most similar to the input sentence will be returned to the user. Followed by [15], the positive samples of the input question are constructed through word repetition operation. Since entities(i.e. disease, symptoms) appeared in the medical queries have a great influence on sentence semantics, we construct negative samples by randomly replacing the entities of the same or different types.

## 5 ENTITY&amp;RELATION EXTRACTION

The task of entity extraction (similar to slot filling) is to identify entities in text with their corresponding type, which is a critical component in spoken dialogue systems[7]. BERT[3] with a CRF layer have been widely used in named entity recognition and achieved good performance. So we trained a BERT-CRF model to identify all all TCM-related entities in user utterances.

The task of relation extraction is to identify the relationship between entities from text[9], which is essential for the construction of TCM knowledge graph. Under the guidance of medical experts, we finely annotated a small-scale but high-quality document-level entity&amp;relation extraction dataset (contains 270 documents, 5996 entities, 4685 relations) based on TCM books. Followed by [16], we treat relation extraction as a multi-label text classification problem. Given an entity pair (head, tail) and the evidence sentences that used to judge its relation, the spliced text of entity pair will be encoded as 𝑒 by BERT and the 𝑖 -th evidence sentence will be encoded as 𝑠 𝑖 by the same BERT. All the representations of evidence sentences [ 𝑠 1 , 𝑠 2 , .., 𝑠 𝑘 ] will be fed into a Bi-GRU layer, then the final state vector ℎ of Bi-GRU layer and 𝑒 will be concatenated together and then use a softmax function to compute the probability for each relation type. The relation extraction model is shown as Fig 3 and the performance is shown in Table 1.

2 http://club.xywy.com/list\_score.htm

Figure 3: The architecture of the RE model.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00481v1-figure-3f5b7903-6b81-4ea5-810a-6f5b54ddb3d3.jpg)

Table 1: The performances of different models on the task of entity&amp;relation extraction

| Task                | Model     |   precision |   recall |    F1 |
|---------------------|-----------|-------------|----------|-------|
| Entity Extraction   | BiGRU+CRF |       57.52 |    65.99 | 61.47 |
| Entity Extraction   | BERT+CRF  |       78.3  |    80.86 | 79.47 |
| Entity Extraction   | ERNIE+CRF |       76.21 |    79.16 | 77.65 |
| Relation Extraction | ERNIE     |       78.46 |    80.67 | 78.72 |
|                     | BERT+GRU  |       82.36 |    82.72 | 81.56 |
|                     | ERNIE+GRU |       80.15 |    81.07 | 79.63 |

## 6 LLM HANDLER AND WRAPPER

Based on the prompt engineering, the LLM Handler utilize the large language model to choose the next action for the given dialogue history. To respond to users' medical questions, the LLM handler retrieves from two types of data sources: documents and knowledge graphs, in order to obtain text-based answers and entity-based answers respectively. The dialogue history, current user intent, and these two types of answers are input into the LLM Wrapper to generate a fluent answer that conforms to the user instructions. Essentially, our LLM wrapper is a RAG (Retrieval-Augmented Generation) system that comprehensively utilizes knowledge from multiple sources and of multiple types. The LLM adapted by both of these modules is ChatGLM3-6B 3 .

## 7 KNOWLEDGE GRAPH CONSTRUCTION

Knowledge graph is a knowledge base that uses a graph-structured model to integrate data, which is the cornerstone of question answering system. To implement question answering in the domain of TCM healthcare, we construct a TCM-related knowledge graph through three steps: i) ontology modeling ii) instance injection and iii) knowledge growth.

Step I : Followed by ontology development 101 [8], we create the ontology model in the domain of TCM. We first enumerated the important terms in TCM-ontology and defined their properties. Then we defined the relations between different terms. After that,

3 https://huggingface.co/THUDM/chatglm3-6b-base

TCM-ontology contained 12 types of entities, 8 types of relations and 20 types of properties.

Step II : Since the entities and relations in the extraction dataset mentioned in Sec. 5 are annotated according to TCM-ontology, it is feasible to restored the knowledge triples aligned with the ontology from the dataset directly, and then injected them into the graph database (Neo4j) after deduplication. Through instance injection, we construct a handcrafted TCM domain KG with 37851 triples. Fig 4 shows a 3-hop subgraph of with facial paralysis (entity type is Disease) as the central node, where gray nodes represent symptoms, orange nodes represent human acupuncture points, purple nodes represent diseases and pink nodes represent moxibustion therapy.

Step III : To obtain a larger-scale TCM-KG, we crawled more documents from the National service platform for academic experience of famous TCM doctors, and used the entity&amp;relation extraction model mentioned in Sec.5 to find more triples from these documents. After manual proofreading, the correct triples are injected into Neo4j. The size of TCM-KG grows from 37851 triples to 48633 triples.

Figure 4: A 3-hop subgraph of our TCM-KG stored in Neo4j.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00481v1-figure-b2bccebb-9cf0-42c4-a236-919990439c5b.jpg)

## 8 HUMANEVALUATION

Followed by the methodologies of software testing [10], we conducted 𝛼 &amp; 𝛽 testing against user requirements to determine whether HBot satisfies the acceptance criteria. 𝛼 -testing performed to identify all possible issues and bugs before releasing the final product to the end users. A total of 5 members in the development team conducted 𝛼 -testing on HBot from three aspects: the feedback on 3D model operation, the accuracy of user instruction understanding, and the robustness of system at runtime. During 𝛼 -testing, we executed 100 test cases and found 21 bugs, including 10 error feedbacks, 8 responses that did not satisfy instructions, and 3 situations where the system could not run stably. After fixing these bugs, we hired 8 members of our lab to conduct 𝛽 -testing on HBot. These people were only informed of the system's features and did not grasp the development details, who can be regarded as end users. During 𝛽 -testing, each member was asked to give 15 different instructions to HBot in the form of text in the dialog box. Each instruction is recorded along with the corresponding feedback or response from HBot and the evaluation of user satisfaction. User satisfaction measures whether the feedback or response of HBot meets the expectations of users:

- · Bad: not at all as expected
- · Fair: as expected, with tolerable errors
- · Good: exactly as expected
- · None: no feedback or response was returned

The performance of HBot on 𝛼 and 𝛽 testing is shown in Fig 5. After a round of bug fixes, the responses of HBot are generally more in line with user expectations. However, there are more cases where the user evaluation is none. This is because the testers are developers in 𝛼 -testing and they know where the capability boundaries of the system are, while the testers in 𝛽 -testing often give instructions that are outside the scope of the domain (e.g. chat with them, amuse them and ask how much is the surgery). To continuously improve system performance, we developed the evaluation interfaces for end-user and the function of record collection. When the scale of the records is large enough, relevant modules in HBot will be updated offline by analyzing these records.

Figure 5: The performance of HBot on 𝛼 and 𝛽 testing.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00481v1-figure-ae4b821d-90d3-401f-ab01-ba6d22d7a3e1.jpg)

## 9 CONCLUSIONS

To provide TCM-related conversational services such as knowledge Q&amp;A, prescription recommendation, moxibustion therapy recommendation, and acupoint search, we develop a h ealthcare chat bot (HBot). Based on an interactive human body in 3D, HBot supports the localization and display of acupoints in conversations. Moreover, we annotate an entity&amp;relation extraction dataset and construct a knowledge graph in TCM to support the slot filling and knowledge retrieval functions of HBot.

## ACKNOWLEDGMENTS

This work is supported by the National Key R&amp;D Program of China (2022YFF0903100), the Special Funding Program of Shandong Taishan Scholars Project, the Key projects of Shandong Natural Science Foundation (ZR2020KF019) and Harbin Institute of Technology Graduate Teaching Reform Project (23Z-DZ039).

## REFERENCES

- [1] Waheed Ahmed Abro, Guilin Qi, Muhammad Aamir, and Zafar Ali. 2022. Joint intent detection and slot filling using weighted finite state transducer and BERT. Appl. Intell.
- [2] Wei Chen et al. 2023. A benchmark for automatic medical consultation system: frameworks, tasks and datasets. Bioinform.

- [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL .
- [4] Xiaosong Han, Xiaoran Li, Yanchun Liang, Xinghao Wang, Dong Xu, and Renchu Guan. 2021. Acupuncture and tuina knowledge graph for ancient literature of traditional chinese medicine. In Proc. of BIBM .
- [5] LX Huang, JS Zhao, ZC Wu, XD Wu, and YS Tan. 2009. [editorial explanation on the state standard the name and location of acupoints (2006. Zhongguo Zhen jiu= Chinese Acupuncture &amp; Moxibustion .
- [6] Huisheng Mao, Baozheng Zhang, Hua Xu, and Kai Gao. 2022. An end-toend traditional chinese medicine constitution assessment system based on multimodal clinical feature representation and fusion. In Proc. of AAAI .
- [7] Grégoire Mesnil et al. 2015. Using recurrent neural networks for slot filling in spoken language understanding. IEEE ACM Trans. Audio Speech Lang. Process.
- [8] Natalya F Noy, Deborah L McGuinness, et al. 2001. Ontology development 101: a guide to creating your first ontology. (2001).
- [9] Stanley Chika Onye, Nazife Dimililer, and Arif A. Akkeles. 2022. Chemical disease relation extraction through the combination of multiple mention levels: relscan+. Turkish J. Electr. Eng. Comput. Sci.
- [10] Ron Patton. 2005. Software Testing (2nd Edition) . Sams, USA. isbn: 0672327988.
- [11] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: sentence embeddings using siamese bert-networks. In Proc. of EMNLP .
- [12] Yu Tian et al. 2022. Research and implementation of real world traditional chinese medicine clinical scientific research information electronic medical record sharing system. In Proc. of BIBM .
- [13] Lorainne Tudor Car, Dhakshenya Ardhithy Dhinagaran, Bhone Myint Kyaw, Tobias Kowatsch, Shafiq Joty, Yin-Leng Theng, and Rifat Atun. 2020. Conversational agents in health care: scoping review and conceptual analysis. Journal of medical Internet research .
- [14] Mina Valizadeh and Natalie Parde. 2022. The AI doctor is in: A survey of task-oriented dialogue systems for healthcare applications. In Proc. of ACL .
- [15] Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022. Esimcse: enhanced sample building method for contrastive learning of unsupervised sentence embedding. In Proc. of COLING .
- [16] Yuan Yao et al. 2019. Docred: A large-scale document-level relation extraction dataset. In Proc. of ACL .
- [17] Bolin Zhang, Zhiying Tu, Shaoshi Hang, Dianhui Chu, and Xiaofei Xu. 2022. Conco-ernie: complex user intent detect model for smart healthcare cognitive bot. ACM Transactions on Internet Technology .
Bolin Zhang Zhiwei Yi Jiahao Wang Dianbo Sui Zhiying Tu Dianhui Chu
HBot: A Chatbot for Healthcare Applications in Traditional Chinese Medicine Based on Human Body 3D Visualization
The unique diagnosis and treatment techniques and remarkable clinical
efficacy of traditional Chinese medicine (TCM) make it play an important role
in the field of elderly care and healthcare, especially in the rehabilitation
of some common chronic diseases of the elderly. Therefore, building a TCM
chatbot for healthcare application will help users obtain consultation services
in a direct and natural way. However, concepts such as acupuncture points
(acupoints) and meridians involved in TCM always appear in the consultation,
which cannot be displayed intuitively. To this end, we develop a
\textbf{h}ealthcare chat\textbf{bot} (HBot) based on a human body model in 3D
and knowledge graph, which provides conversational services such as knowledge
Q\&A, prescription recommendation, moxibustion therapy recommendation, and
acupoint search. When specific acupoints are involved in the conversations
between user and HBot, the 3D body will jump to the corresponding acupoints and
highlight them. Moreover, Hbot can also be used in training scenarios to
accelerate the teaching process of TCM by intuitively displaying acupuncture
points and knowledge cards. The demonstration video is available at
https://www.youtube.com/watch?v=UhQhutSKkTU . Our code and dataset are publicly
available at Gitee: https://gitee.com/plabrolin/interactive-3d-acup.git