Automated Real-Time Identification of Medicinal Plants Species in Natural Environment Using Deep Learning Models—A Case Study from Borneo Region

The identification of plant species is fundamental for the effective study and management of biodiversity. In a manual identification process, different characteristics of plants are measured as identification keys which are examined sequentially and adaptively to identify plant species. However, the manual process is laborious and time-consuming. Recently, technological development has called for more efficient methods to meet species’ identification requirements, such as developing digital-image-processing and pattern-recognition techniques. Despite several existing studies, there are still challenges in automating the identification of plant species accurately. This study proposed designing and developing an automated real-time plant species identification system of medicinal plants found across the Borneo region. The system is composed of a computer vision system that is used for training and testing a deep learning model, a knowledge base that acts as a dynamic database for storing plant images, together with auxiliary data, and a front-end mobile application as a user interface to the identification and feedback system. For the plant species identification task, an EfficientNet-B1-based deep learning model was adapted and trained/tested on a combined public and private plant species dataset. The proposed model achieved 87% and 84% Top-1 accuracies on a test set for the private and public datasets, respectively, which is more than a 10% accuracy improvement compared to the baseline model. During real-time system testing on the actual samples, using our mobile application, the accuracy slightly dropped to 78.5% (Top-1) and 82.6% (Top-5), which may be related to training data and testing conditions variability. A unique feature of the study is the provision of crowdsourcing feedback and geo-mapping of the species in the Borneo region, with the help of the mobile application. Nevertheless, the proposed system showed a promising direction toward real-time plant species identification system.

Introduction
Plant species' identification is a challenging task that has a key role in effectively studying biodiversity and investigating unknown species. The manual identification of plant species is a time-consuming process and requires a lot of expertise in the field. Automated identification systems based on computer vision and machine learning techniques provide an alternative and assistive approach for this task. These systems are useful, but their accuracy varies due to the diversity of the species. A recent survey highlighted the growing application of machine learning (ML) and deep learning (DL) for plants' identification through leaves [1]. Integrated with Mobile applications, ML and DL techniques are increasingly being applied to distinguish between diseased and healthy plants [2][3][4][5][6][7], identification and classification of herbs and medicinal plants [3][4][5][6][7][8][9][10], classification of both generic plants and specific plant species [11][12][13][14][15][16][17][18][19], identification of crop-specific diseases [20], and general identification of plants to guide field tours [21].
Convolutional neural networks (CNNs) and the various deep CNN models have been reported to be the most commonly used methods in the automation process of plantclassification tasks [1]. An automated diagnosis of the 10 most common tomato leaf diseases, using a mobile application, was conducted in Reference [2], using the MobileNet model of CNN, and an accuracy of up to 89% was reported. Similar to Reference [2], where a single crop (tomato) was used, researchers in Reference [7] trained and deployed Residual Neural Networks (ResNets), a deep version of CNN, in a custom-built mobile application to classify wheat disease in the wild, reporting classification accuracy of up to 96%. ResNet and Xception Networks, in combination with the YOLO object detection framework, have also been used to detect early blight tomato disease, with an accuracy of over 99% reported in Reference [3]. DL Neural Networks have also been deployed through an android-based mobile application to detect different common diseases in terrestrial plants found in the Philippines, with an accuracy of 80% [4]. B5 and B4 models of the EfficientNet DL model are reported in Reference [5], which achieved an accuracy of over 99% in the classification for 38 different diseases in various plants. However, the classification was performed in offline mode, and it was not deployed and tested over a mobile application. More recently, transfer learning has been successfully deployed in a mobile application to enable the detection of tomato leaf diseases, reporting detection accuracy of over 96%, using the EffecientNet-B0 DL model [6]. The results of these studies demonstrate the efficacy of deploying ML and DL techniques in mobile applications to detect, classify, and identify plant diseases by using plant leaves.
Medicinal plants and herbs have continued to be used for the traditional management of various illnesses in many societies since time immemorial [22]. With advancements in artificial intelligence (AI) and different Information and Communication Technologies (ICTs), the need to automatically identify medicinal plants from the thousands of plant species can only continue to grow. Researchers in Indonesia [8] utilized local binary patterns to extract the leaf texture of 30 different medicinal plants and then applied probabilistic Neural Networks to automatically classify the herbal leaves, achieving a classification accuracy of just over 56%. In Reference [9], the researchers utilized a support vector machine (SVM) and DL Neural Networks to automatically classify 20 different herbs found in Malaysia, using a mobile application. The mobile application reported spending only 2 s for processing the input leaf image and returning the classification results, with a classification accuracy of 93%. Similarly, in Reference [10], a fusion of fuzzy local binary pattern and fuzzy color histogram, using product decision rules, was performed to enable the automatic identification of 51 medicinal plant species commonly found in Indonesia. Probabilistic Neural Networks were used to classify color histograms, reportedly achieving an accuracy of just over 74%. The promising results achieved in these studies clearly highlight the plausibility of utilizing ML and DL models in mobile applications to automate the classification of medicinal plants.
It has been reported that there could be over 450,000 different plants, with one-third of them facing extinction [23]. The easy and automatic identification of plant species is, therefore, a crucial step toward their preservation. The scientific research community continues to make efforts toward the realization of this step. In Reference [11], the researchers built a joint-classifier by using Backpropagation Neural Networks and weighted K-NN and deployed it in an android-based mobile application to enable automatic classification of 220 plant species (angiosperms) found in China. The joint classifier reportedly achieved a classification accuracy of nearly 93%. K-NN was similarly utilized in Reference [12] to identify 32 different plant species common in Mauritius, using leaf-shape features and a color histogram; accuracy of just over 87% was reportedly achieved. In Reference [13], a custom-built android-based mobile application was able to identify a tree from 126 tree species common in the French Mediterranean area, using tree leaves, with an accuracy of up to 90%. Researchers achieved this through seg-menting tree leaf images to form feature space and then used histogram intersection to predict the class. Similarly, a custom-built mobile application with a back-end classifier, using SIFT features with the Bag of Words model and SVM, was able to classify 20 different plant species common in Sri Lanka with an accuracy of 96.48% [14]. Furthermore, CNN models, including VGG19, MobileNet, and MobileNetV2, deployed in a mobile application were able to identify 33 different types of leaves common in East Hokkaido in Japan, with MobileNetV2 achieving an accuracy of over 99% [15]. In Reference [16], the researchers also deployed a CNN model in an android-based mobile application to classify natural images of leaves belonging to 129 different species crowdsourced from all over the world. Single image classification took 2.5 s, obtaining a Top-3 test error of above 60%. Similarly, in Reference [17], the researchers presented a mobile application that was able to classify plant leaves belonging to five different plant species common in India, based on leaf color and shape. To perform classification by using leaf shapes, the extracted morphological features of the leaves were categorized by using the Sobel and Otsu methods, while the color-based classification was performed by using the dominant-color method. Finally, but not least, in Reference [18], the results of a mobile application for leaf classification utilizing a CNN were presented. This study experimented with different CNN architectures' performances in classifying 15 leaf species of plants common in California and North Carolina. Deep CNNs are reported to have achieved the highest classification accuracy of 81.6%, with the mobile application completing the classification task in 2.5 s after spending about 5.2 s loading the query leaf image in the NN classifier from the gallery. The results reported in the abovementioned works from the literature demonstrate the promising efficiency of mobile application classifiers that were developed by using the ML and DL models to facilitate the automatic identification and classification of plant species. Not only is this a crucial step toward the conservation of species risking extinction, but such mobile applications could eventually serve as field guides during tours to various plantations in the wild. In Reference [16], the researchers demonstrated how a mobile application utilizing leaf morphological features and angle code histogram was able to serve as a tour guide in a wild field with six different plant species, in the USA, with an error rate between 17% and 53%. While this error rate appears to be high, it is a promising result to begin with for a field tour in the open wild.
The current study aimed to develop a mobile application to enable the automatic identification of medicinal plants and plants predominantly found in the Borneo region in real time. The present study builds on the strengths of promising methods established in related works from the literature, while also addressing the weaknesses (gaps) identified in the same. Most of the reviewed studies either utilized their own generated small-sized datasets for training and assessing their ML and DL models or entirely (i.e., training and testing) used some of the open existing image datasets (such as ImageCLEF, Flavia, ICL Plantae, PlantVillage, CVIP100, Flavia, Swedish, etc.), potentially resulting in overfitted and underfitted models, respectively. Additionally, many of the reviewed studies considered clean leaf images captured against a white background, a condition that is unlikely in a real-world setting in which leaf images are often captured with a non-clean background. Moreover, studies that reported excellent (over 95%) classification accuracies mainly involved classifying a single plant (e.g., tomato), thereby limiting their application to a wide range of species. Additionally, the feedback of the mobile application end-user on observed classification results was not taken into account to further enrich the classifier's knowledge base for future classification. The development of the mobile application proposed in the current study is, therefore, a valuable addition to the existing efforts toward the preservation of the herbs and medicinal plants native to the Borneo region, a region primarily believed to be the most species-rich area in the world [24], yet with the immense threat of extinction to some of its plant species [25,26]. The current study experimented with and optimized different EffecientNet deep learning models, as transfer learning has mainly been singled-out to produce the best classification results for real-time multiclass image classification tasks [1,5]. A unique feature of the study is the provision of crowdsourcing feedback and geo-mapping of the species in the Borneo region with the help of the mobile application. This is important for the region since several local plant species (with their local names and benefits) are unknown to the experts. The system provides an adaptive learning approach where the models can be updated based on the newly collected data and people's feedback.
Our contributions in this study are highlighted as follows: • We have proposed a machine vision system that is capable of automating the identification of medicinal plant species in real time.

•
We have developed an end-to-end computer vision system with a convolutional neural network (CNN) model to identify medicinal plant species when given an image.

•
The system works in real time and can accurately identify different plant species given by simply taking a picture with a mobile camera or uploading an existing image from a device.

•
The system provides a feedback mechanism and a knowledge base as a means to continuous lifelong learning of the models to produce a robust plant species identification system.

Proposed System
The overall flow of the proposed system is presented in Figure 1. The system is composed of three main components: (1) a computer vision and deep-learning-based plant species classifier, (2) a knowledge base as a central repository for plant information together with auxiliary and feedback data, and (3) a mobile front-end that provides a user interface to the end-user to interact with the system and displaying of classification results. Details for each component of the system are explained in the subsequent subsections. time multiclass image classification tasks [1,5]. A unique feature of the study is the provision of crowdsourcing feedback and geo-mapping of the species in the Borneo region with the help of the mobile application. This is important for the region since several local plant species (with their local names and benefits) are unknown to the experts. The system provides an adaptive learning approach where the models can be updated based on the newly collected data and people's feedback. Our contributions in this study are highlighted as follows: • We have proposed a machine vision system that is capable of automating the identification of medicinal plant species in real time.

•
We have developed an end-to-end computer vision system with a convolutional neural network (CNN) model to identify medicinal plant species when given an image.

•
The system works in real time and can accurately identify different plant species given by simply taking a picture with a mobile camera or uploading an existing image from a device. • The system provides a feedback mechanism and a knowledge base as a means to continuous lifelong learning of the models to produce a robust plant species identification system.

Proposed System
The overall flow of the proposed system is presented in Figure 1. The system is composed of three main components: (1) a computer vision and deep-learning-based plant species classifier, (2) a knowledge base as a central repository for plant information together with auxiliary and feedback data, and (3) a mobile front-end that provides a user interface to the end-user to interact with the system and displaying of classification results. Details for each component of the system are explained in the subsequent subsections.

Classification Model
To develop the real-time plant species identification system, a number of deep learning models were trained and tested on the given datasets. The details of the datasets, preprocessing steps, and training/testing of the models are described below.

Datasets PlantCLEF 2015
PlantCLEF 2015 [27] is a plant identification challenge dataset that aims to build an image-based plant identification system and evaluates methods and systems at a very large scale that adapts to real-world conditions. The dataset was constructed through a participatory community platform in 2011, consisting of thousands of collaborators. PlantCLEF 2015 consists of curated images from many different contributors, cameras, areas, periods of the year, and individual plants. More precisely, the PlantCLEF 2015 dataset comprises 113,205 pictures belonging to 41,794 observations of 1000 species of trees, herbs, and ferns living in Western European regions. Each image corresponds to one of the seven types of views in the meta-data (entire plant, fruit, leaf, flower, stem, branch, and leaf scan) and is associated with a scientific name.
In this study, the PlantCLEF 2015 dataset was used as an auxiliary dataset to create our plant identification model to improve our classification result. We have further extracted images relevant to our application, that is, the images containing only leaf-related information (e.g., leaf, leaf-scan, and the entire plant). Thus, the extracted dataset consisted of 23,708 images. The hold-out set is built by partitioning 90% of the dataset for training and testing, while the validation data are set to be roughly 10% of the whole dataset (please see Table 1 for more detailed statistics).

UBD Botanical Garden Dataset
The dataset was laboriously collected from the UBD Botanical Garden, using a DSLR camera with manually annotated labels (i.e., Scientific classification of plants), together with their descriptions. The main originality of the plants in the database is that these plants are mainly native to the Borneo regions, comprising primarily medicinal plants and native plants.
The dataset consists of 106 species with a total of 2097 images collected by us ( Table 2). Plants' images were captured from different angles, and instances of plants with a variety of growth conditions were collected. Images are mostly concentrated on the leaf, with only a small fraction of the dataset containing flowers. Examples of images taken from the dataset are illustrated in Figure 2.

Training Details
Due to the insufficient size of our dataset, we opted for transfer learning with ImageNet [28] pretrained weights. We used the EfficientNet-B1 [29] variant for deployment and added dropouts of 0.6 and a softmax layer downstream of the network. A posttraining quantization of Float16 was employed by using the TensorFlow lite, resulting in 1.95 M parameters after quantization. The network was trained for 100 epochs using the Google Cloud GPU compute engine. We initially trained the model by using ImageNet weights to fine-tune to PlantCLEF 2015 dataset and later merged both datasets (i.e., PlantCLEF and UBD datasets) and performed fine-tuning of the weights in an incremental manner for the locally collected species. A total of 18,949 training Images were used from PlantCLEF and a total of 20,640 images after merging both datasets. These images were resized to 224 × 224 px with an AutoAugment [30] augmentation policy.
AutoAugment policies are the optimal augmentation policies that improve the overall performance of models using Reinforcement Learning (RL) based on the ImageNet dataset. A policy consists of 5 sub-policies, and each sub-policy applies two image operations in sequence. Each image operation has two parameters: the probability of applying it and the magnitude of the operation. Applying the same optimal augmentation policy to different datasets while using pretrained ImageNet weights yielded an improvement in the accuracy of the models when experimented by us. Thus, we opted for ImageNet weights as our basis for transfer learning with AutoAugment policies in training our models.

Training Details
Due to the insufficient size of our dataset, we opted for transfer learning with Ima-geNet [28] pretrained weights. We used the EfficientNet-B1 [29] variant for deployment and added dropouts of 0.6 and a softmax layer downstream of the network. A post-training quantization of Float16 was employed by using the TensorFlow lite, resulting in 1.95 M parameters after quantization. The network was trained for 100 epochs using the Google Cloud GPU compute engine. We initially trained the model by using ImageNet weights to fine-tune to PlantCLEF 2015 dataset and later merged both datasets (i.e., PlantCLEF and UBD datasets) and performed fine-tuning of the weights in an incremental manner for the locally collected species. A total of 18,949 training Images were used from PlantCLEF and a total of 20,640 images after merging both datasets. These images were resized to 224 × 224 px with an AutoAugment [30] augmentation policy.
AutoAugment policies are the optimal augmentation policies that improve the overall performance of models using Reinforcement Learning (RL) based on the ImageNet dataset. A policy consists of 5 sub-policies, and each sub-policy applies two image operations in sequence. Each image operation has two parameters: the probability of applying it and the magnitude of the operation. Applying the same optimal augmentation policy to different datasets while using pretrained ImageNet weights yielded an improvement in the accuracy of the models when experimented by us. Thus, we opted for ImageNet weights as our basis for transfer learning with AutoAugment policies in training our models.
To counter the imbalanced distribution of samples from our datasets, we employed two cost-sensitive learning methods to train our model: computing the class weights for each class and focal loss [29]. To summarize, we trained our models by using transfer learning, using ImageNet weights with AutoAugment optimal augmentation policy and cost-sensitive learning methods applied. Figure 3 highlights the imbalance distribution for both datasets with a ratio. To counter the imbalanced distribution of samples from our datasets, we employed two cost-sensitive learning methods to train our model: computing the class weights for each class and focal loss [29]. To summarize, we trained our models by using transfer learning, using ImageNet weights with AutoAugment optimal augmentation policy and cost-sensitive learning methods applied. Figure 3 highlights the imbalance distribution for both datasets with a ratio.

Class Weighted Function
The classical way of training neural networks by using backpropagation involves updating the model weights with respect to errors being made by the model. This method fails when we have imbalanced training samples where examples from each class are treated the same, meaning that, for imbalanced datasets, the model is prone to performing well only for the majority class; that is, it is biased toward the majority class samples.
The backpropagation algorithm can be updated to consider the misclassification of each class by using a cost-sensitive loss function that incorporates the error in proportion to the number of samples present in the training samples. The effect of adding this classweighted function allows the neural network to learn the minority classes such that the model will be penalized more when misclassification of the minority class occurs. We defined the following algorithm in computing the class weights for our dataset as follows: def class_weights(Y , α) #returns a dict. of class weights. The α parameter represents the smoothing parameter, which balances the class weights between the majority and minority classes. It determines how much it is to penalize the model when misclassification for the minority class occurs. Setting α = 1 equates to applying equal class weightage with respect to the number of species present in the dataset and setting α > 0 equates to making the minority class weights higher. We defined α to be 0.4 for both the PlantCLEF 2015 and UBD Botanical datasets and trained by using the multiclass cross-entropy loss.

Focal Loss
The focal loss was introduced in Reference [29] for the object detection task, which deals with the sparse set of foreground examples present in the datasets and prevents the

Class Weighted Function
The classical way of training neural networks by using backpropagation involves updating the model weights with respect to errors being made by the model. This method fails when we have imbalanced training samples where examples from each class are treated the same, meaning that, for imbalanced datasets, the model is prone to performing well only for the majority class; that is, it is biased toward the majority class samples.
The backpropagation algorithm can be updated to consider the misclassification of each class by using a cost-sensitive loss function that incorporates the error in proportion to the number of samples present in the training samples. The effect of adding this classweighted function allows the neural network to learn the minority classes such that the model will be penalized more when misclassification of the minority class occurs. We defined the following algorithm in computing the class weights for our dataset as follows: def class_weights(Y train , α) #returns a dict. of class weights. counter = dict. containing the no. of samples per class from Y train . if α > 0: p = max(counter.values()) * α for class in counter.keys(): counter[class] += p majority = max(counter.values()) return {class: majority/count class ∀class ∈ Y train } The α parameter represents the smoothing parameter, which balances the class weights between the majority and minority classes. It determines how much it is to penalize the model when misclassification for the minority class occurs. Setting α = 1 equates to applying equal class weightage with respect to the number of species present in the dataset and setting α > 0 equates to making the minority class weights higher. We defined α to be 0.4 for both the PlantCLEF 2015 and UBD Botanical datasets and trained by using the multiclass cross-entropy loss.

Focal Loss
The focal loss was introduced in Reference [29] for the object detection task, which deals with the sparse set of foreground examples present in the datasets and prevents the vast number of easy negatives from overwhelming the detector during training. The loss function was reshaped to down-weight easy examples and, thus, focused training on hard negatives by adding a modulating factor (1 − p t ) γ to the cross-entropy loss, with tunable focusing parameter γ ≥ 0 (Equation (1)). In Reference [29], the authors experimented with γ ∈ [0, 5], where γ = 2 worked best in their experiment.
The properties of focal loss are as follows: • As p t →1-meaning that the model is confident that a given sample belongs to class t-the modulating factor goes to 0 given γ > 0, resulting in the down-weight of the loss of the easy examples during training.

•
Parameter γ, or the focusing parameter, controls the rate at which easy examples are down-weighted. As γ ∈ [0, 5], γ = 0 is equivalent to the standard cross-entropy loss with no class weights assigned. As γ→ 5, the modulating factor grows exponentially, resulting in the increase of down-weight of easy examples.
We followed the same approach as the one described in Reference [29] for defining the γ to be 2 to both of our datasets. The experimented results for different γ values are also shown in Tables 2 and 3.

Hyperparameter Tuning Learning Rate Finder
Following the super convergence approach for neural networks, the learning rate range test (LR range test) was performed to find the optimal LR min and LR max to be used in the Cyclical Learning Rate (CLR) [31,32]. The losses for each mini-batch were plotted with respect to the LR range of 10 −10 < LR < 1, increasing exponentially on each mini-batch. The LR range test serves as a guide for how well a network performs through a range of learning rates and spot values that effectively train the network. The stopping criteria for the LR range test was set at a condition when the current loss is 4 times greater than the previous loss, as seen in Figure 4 at LR > 10 2 regime for the PlantCLEF 2015 dataset. The "triangular" learning rate policy rule was opted for this study where learning rates oscillate in a triangular cycle from LR min to LR max in equal lengths of steps size. We set the LR min and LR max to be 10 −2 to 10 −1 and 10 −4 to 10 −2 for the PlantCLEF 2015 and UBD Botanical datasets, respectively.

Mobile Application Details
We developed a mobile application dedicated to the production of our collected botanical dataset through image-based plant identification. The application currently supports Android (API level 16 and above), allowing images of a plant via gallery upload or captured by the camera to be sent to the web server to retrieve a list of predicted species made by our model. Figure 5 shows a graphical user interface of our mobile application system. Below are the details of the architecture of the mobile application system.

Mobile Application Details
We developed a mobile application dedicated to the production of our collected botanical dataset through image-based plant identification. The application currently supports Android (API level 16 and above), allowing images of a plant via gallery upload or captured by the camera to be sent to the web server to retrieve a list of predicted species made by our model. Figure 5 shows a graphical user interface of our mobile application system. Below are the details of the architecture of the mobile application system.

Mobile Application Details
We developed a mobile application dedicated to the production of our collected botanical dataset through image-based plant identification. The application currently supports Android (API level 16 and above), allowing images of a plant via gallery upload or captured by the camera to be sent to the web server to retrieve a list of predicted species made by our model. Figure 5 shows a graphical user interface of our mobile application system. Below are the details of the architecture of the mobile application system. Figure 5. Screenshot of GUI mobile application. Figure 5. Screenshot of GUI mobile application.

Web API and Front-End
Data exchanges between the back end (web server) and the front-end (client-side) are managed through a REST-full web API, using JSON data format and jpeg images. The list of candidate species predicted for a searched plant observation is displayed in increasing order of confidence level, and the results of only Top-3 predictions are shown to the end-user. In this study, for testing the real-time classification of the species, the Google cloud compute engine server was used as the backend (web server) to deploy the deep learning model, and the custom-developed mobile applications worked as the front-end.

Species Prediction
The predictions made by the classifier are based on the scientific or botanical names of the plant. Metadata information (i.e., family, genus classification, plant usage, etc.) is further extracted from a database via a web server. If a candidate species has a weak confidence level by the model, the observation may not correspond to one species inside our database (rejection class). As a result, the list of returned Top-3 predicted results is not displayed, and the end-users are allowed to suggest the name of the observation.

User Response Information Storage
Each top-k label returned by the model may not be relevant or accurate; as a result, we have implemented a verification mechanism that allows the end-users to verify the returned results and store the response. Information stored includes one or several plant images tagged with provenance data (device, author, date, etc.). Each observation is associated with a geo-location and one or more determinations, i.e., possible species names proposed by the author him/herself, and/or by other annotators, and/or by automated classifiers. Observations are stored within a NoSQL database called MongoDB. The observations are later verified by domain experts and can be used to improve the model performance as a continuous lifelong learning process for plant species identification.

Knowledge Base
A knowledge base (KB) is a centralized repository for information related to a particular field or domain. In this study, a KB was created in order to manage the information related to the plant species' profiles and other ancillary data. The KB contains different types of information, including raw and processed image data, domain knowledge, learning models (trained intelligent methods), geo-location of images transferred by users for classification, feedback data, etc. The designed KB is not a static collection of information; instead, it acts as a dynamic resource that has the capacity to learn and evolve with time when new plant images are presented, and new classification/labels are added to the system. Thus, as an integral component of the plant species classification system, this KB has been used to optimize the collection, organization, and retrieval of relevant information for plants. The internal implementation of the KB is a hybrid storage model consisting of a database, image files, and classification models.

Experiments and Results
The trained classification models were evaluated in two stages: Initially, the models were tested offline on a hold-out test set from both datasets. In the next phase, the models were deployed to the mobile application for real-time species identification, wherein their performance was assessed.
We selected accuracy, sensitivity (true-negative rate), and specificity (true-positive rate) as evaluation metrics, computed as the average for all class labels present in the target dataset. Specificity presents the proportion of the predicted negative classes which were correctly predicted. On the other hand, sensitivity presents a proportion of the true-positive classes which were correctly identified.
The equations for these metrics are given below: where k represents total class labels; and TN, FN, FP, and TP are the numbers of the truepositive, true-negative, false-positive, and false-negative predictions for the considered class, respectively.

Offline Testing of the Classification Models (for Both Datasets)
We trained a baseline model by applying the cost-sensitive learning method to our highly imbalanced dataset with standard cross-entropy loss and applied the same training techniques as outlined in Section 2.2.2. The training and validation loss curves for the model are shown in Figure 6. We performed an experiment for a total of three runs and reported its average evaluation metrics, as shown in Tables 3 and 4 where k represents total class labels; and TN, FN, FP, and TP are the numbers of the truepositive, true-negative, false-positive, and false-negative predictions for the considered class, respectively.

Offline Testing of the Classification Models (for Both Datasets)
We trained a baseline model by applying the cost-sensitive learning method to our highly imbalanced dataset with standard cross-entropy loss and applied the same training techniques as outlined in Section 2.2.2. The training and validation loss curves for the model are shown in Figure 6. We performed an experiment for a total of three runs and reported its average evaluation metrics, as shown in Tables 3 and 4      The baseline model showed roughly 80% in Top-5 accuracy. However, it performs poorly in classifying the true-positive and true-negative samples as visible from the low sensitivity and specificity rate. The baseline model was found to be biased toward the majority classes; hence, it had a high accuracy rate (80%) and a low rate of sensitivity or recall (43.2%). Both cost-sensitive learning methods, including the class weighted and focal loss, showed an improvement in the classification performance for the imbalanced dataset, as indicated by the significant increase in the sensitivity and specificity values. With the class weighted function applied, the model is being penalized more and able to make a correct prediction with an improvement of sensitivity rate from 43% to 79%, while using focal loss has the effect of having the model recognize classes that do not correspond to its labels. We later performed an evaluation on the test samples on the merged dataset and ran a total of three runs. The results shown in Table 4 are based on a total of 251 species with 2537 images.
The baseline model appeared to show no improvement, albeit adding more samples from our collected dataset. This is mainly due to the imbalance in the nature of the dataset, as shown in the distribution graph in Figure 3. After applying the cost-sensitive learning methods and adding more external samples, a slight incremental improvement in performance was found. In the next section, we outlined the test performed on real-life plant observation and reported its performance.

Mobile Application Testing
Based on the species identification performance shown in Tables 3 and 4, we selected two models for testing where both are trained on the merged dataset with the highest sensitivity for each of the respective cost-learning methods, i.e., focal loss with γ = 2.0 and class weighted function with α = 0.4. The real-life plants' observation test samples used for this study are the 121 species which are found at the UBD Botanical Garden.
Due to the lack of variability in our dataset, both models' performance deteriorates upon real-time testing, as shown in Table 5. The accuracy of using focal loss is comparable to offline testing, with an average of 80% accuracy for both Top-1 and Top-5 accuracy. However, the sensitivity drops for both methods' classes weighted function and focal loss from 80% to 66.1% and 75.2%, respectively. The resultant of this performance may have been caused by the failure of the model to generalize the dynamic change of state of the plants (i.e., growth cycle), which can be solved when more samples are present in the dataset. Some examples of the real-time testing of the developed mobile application are shown in Figure 7. In the real-time classification system, the image taken through the mobile camera is uploaded to the cloud, and the classification results are sent back to the mobile device. The corresponding information, including the plant's scientific name, description, medical usage, precautions, local family name, common name, and origin, is retrieved from the online database and displayed to the user. Since the system also facilitates the geo-mapping of the species, the longitude and latitude information, where the user took the image, can also be saved in the knowledge base. Moreover, a confirmation option is provided for the users to give feedback to improve the system further, and the domain expert verifies this at the end. This option helps us continuously improve the classification model for plant species identification. Figure 8 depicts an example of the system when used in the offline mode. It also shows a scenario where a plant image can be classified into multiple species with different probability values. The class with the highest probability value is shown first, while the rest are shown in decreasing order of probability values. Similar to the real-time system, a feedback mechanism is provided to report the misclassification of the plant species. when used in the offline mode. It also shows a scenario where a plant image can be classified into multiple species with different probability values. The class with the highest probability value is shown first, while the rest are shown in decreasing order of probability values. Similar to the real-time system, a feedback mechanism is provided to report the misclassification of the plant species.  In contrast to several previous promising studies in which either open-access datasets [2,3,5] or localized primary datasets [4,20] were exclusively utilized, the deep learning models trained and tested in the current study used over 25,000 images, consisting of both secondary open-access datasets, as well as a primary dataset of images collected from a Selected Image In contrast to several previous promising studies in which either open-access datasets [2,3,5] or localized primary datasets [4,20] were exclusively utilized, the deep learning models trained and tested in the current study used over 25,000 images, consisting of both secondary open-access datasets, as well as a primary dataset of images collected from a local botanical garden in Brunei with medicinal plants native to Borneo region. Therefore, this dataset diversity adds to the reproducibility confidence of the obtained results under real-time conditions and the likely generalizability of the trained models. Additionally, the average real-time in-the-wild classification accuracy of 80% for both Top-1 and Top-5 accuracies obtained in the current study is lower than the classification accuracies of 88.4% and 99.9% previously reported in References [2,3], respectively; it should be noted, however, that the latter studies were dealing with diseased leaf detection for only one type of crop (tomatoes) in contrast to the present study, with over 250 medicinal plants species. Furthermore, as a way of building on previous related works [2,4,6,17,20,21] that aimed to train and deploy deep learning plants identification models on mobile applications, the current work has introduced a new feature by enabling confirmation and, thus, verification of the classification results by the mobile application user, as shown in Figure 5. This new feature introduced in this work aids the continuous improvement of the cloud-hosted knowledge base, particularly by domain experts and native users of medicinal plants, thereby facilitating incremental learning of the deployed deep learning model.

Conclusions and Future Work
In this study, a deep-learning-based system was proposed to perform a real-time species identification of medicinal plants found in the Borneo region. The proposed system addressed some of the key challenges when training deep learning models, such as small training samples with long-tile class imbalance distribution of the species data. Techniques such as class weighting and the use of focal loss function were applied to improve the learning process of the model. The results showed that the proposed system could significantly improve the performance of the deep learning model by more than 10% accuracy compared to the baseline model. However, performance accuracy was slightly dropped when the system was tested on the actual samples by using the developed mobile application in real time. In the future, we intend to further improve the system's performance by improving the sample collection of the training data. Furthermore, to make the system more useful, we intend to increase the number of species, as the Borneo region is a high species diversity spot.

Data Availability Statement:
The data that support the findings of this study are available upon request.


Leveraging Hashtag Networks for Multimodal Popularity Prediction of Instagram Posts

With the increasing commercial and social importance of Instagram in recent years, more researchers begin to take multimodal approaches to predict popular content on Instagram. However, existing popularity prediction approaches often reduce hashtags to simple features such as hashtag length or number of hashtags in a post, ignoring the structural and textual information that entangles between hashtags. In this paper, we propose a multimodal framework using post captions, image, hashtag network, and topic model to predict popular influencer posts in Taiwan. Specifically, the hashtag network is constructed as a homogenous graph using the co-occurrence relationship between hashtags, and we extract its structural information with GraphSAGE and semantic information with BERTopic. Finally, the prediction process is defined as a binary classification task (popular/unpopular) using neural networks. Our results show that the proposed framework incorporating hashtag network outperforms all baselines and unimodal models, while information captured from the hashtag network and topic model appears to be complementary.

Introduction
Over the past few decades, social media have become one of the most powerful and effective devices for spreading information. Companies and influencers around the world share millions of ideas and launch advertisement campaigns everyday through social media. In particular, Instagram, a photo and video sharing platform launched in 2010, has achieved widespread popularity among younger generations and Internet influencers these days. As a result, Instagram have gradually gained tremendous commercial and social importance as it unveils novel business strategies and marketing possibilities for not only companies and organizations, but also influencers, politicians, entertainers, and even country governments. Due to the increasing significance of Instagram, multimodal prediction of popular content on Instagram has become of great interest to researchers in the field of digital marketing or social media analytics. On Instagram, users can utilize a digital arsenal of photos, videos, filters, captions, hashtags, or stories to create appealing content for their audience. Each of the modalities in an Instagram post often require different techniques to represent; for example, natural language processing for captions, computer vision for images, both of which involve knowledge in data analysis, neural networks, or deep learning. However, among different modalities, hashtags are often ignored or over-simplified as features such as hashtag length, number of hashtags in a post or hashtag frequency. Some even consider hashtags as part of the post caption, failing to represent hashtags as individual and informative features. With the above being said, the present study proposes a multimodal framework considering post caption, image, and incorporating hashtag networks for predicting popular Taiwanese influencer posts on Instagram. Unlike existing Instagram multimodal frameworks, this study considers the co-occurrence relationship between all hashtags in the corpus and constructs hashtags as a homogeneous graph. The hashtag embeddings are then generated by combining structural embeddings extracted through GraphSAGE (Hamilton et al., 2017) and topic embeddings extracted using BERTopic. In addition to hashtag representations, contextualized representations of post captions are learned using sentence-transformers (Reimers et al., 2019), and image features are generated using the pretrained Inception-V3 model. Finally, following Carta et al. (2020)'s method, this study calculates the engagement moving average (EMA) of Instagram posts as the basis for determining whether a post is popular or unpopular. In sum, the multimodal framework encodes text, image, and hashtags as a set of modality features that are eventually used as inputs to a binary classifier. It is the goal of this study to provide insights into not only social media marketing and influencer behaviors, but also leveraging a new form of hashtag feature -network graph as a part of the multimodal modeling of fragmented information on Instagram. Hopefully, the present study will illuminate new possibilities for representing hashtag connections as an indicator to the popularity of Instagram posts. In addition, as existing research in this field are mostly conducted on English posts, the present study not only provides precious insights into popularity prediction of Chinese Instagram posts, but it also explores Instagram content in different languages or the different posting patterns between Chinese and English Instagram users. Finally, the applications of the present study may extend to general social media analytics for companies, global brands, politicians, and influencers to form new marketing strategies.

Related Work
In this section, previous work that relate to the present study will be reviewed. To be more specific, this section presents studies under two subtopics: popularity prediction in social media and research in hashtags.

Popularity Prediction in Social Media
In recent years, popularity prediction of social media posts has gradually become a hot topic, and researchers in this field have proposed diverse frameworks concerning different datasets, model structures, feature representations, and defining the task as either a regression or classification problem. The mainstream method for social media popularity prediction is to incorporate deep learning models and combine features extracted from different modalities. Zhang et al. (2018) train a dual-attention multimodal model using image caption pairs and topic features extracted from an LDA model on Instagram. De et al. (2017) implement stacked autoencoders connected to an MLP layer to predict popular posts from GQ India's Instagram account. On the other hand, Mazloom et al. (2016), gather posts from Flickr and trains SVR (support vector regression) models using textual information and CNNs to predict popularity scores, hence solving a regression task. However, some research is able to achieve desirable results despite using features that are extracted without involving neural networks or deep learning. Huang et al. (2018) exploits post-related features such as text length, timestamp, geolocation and user-related features such as followers and following number as inputs to several regression models. Carta et al. (2020) collected about 100,000 posts from over 2,500 users on Instagram, utilizing several features such as post metadata, text length, hashtag counts, and average likes number to represent each post. It is worth noticing that they measure popularity using LMA (likes moving average). Unlike previous work considering raw likes or followers as the popularity metric, LMA compares the likes number of a post with the average likes number of its previous k posts to determine a popularity class (popular or unpopular), taking into consider the recent engagement performance of a post. The present study will also follow this method for popular post prediction, which will be explained in detail in section 4.2.2.

Hashtag Representation
As mentioned earlier, this study introduces hashtag networks as a novel method for including hashtags as a part of the multimodal post representation. Therefore, this section provides an overview of the different aspects of hashtags, including hashtags in social media linguistics to hashtags in networks and graph theory.

2.2.1
Linguistic Functions of Hashtags Using a corpus-based approach, Zappavigna (2015) proposed that hashtags serve several different functions in social media, with hashtags as topic markers as the dominant function. To be more specific, hashtags encapsulates the main topic of a post, helping users to effectively highlight and emphasize central ideas. Scott (2015) also mentioned that hashtags are potent search tools. Following Messina (2007)'s proposal, hashtags are originally designed to track and tag content on Twitter, linking posts with similar topics and discourse environments together. In addition, Scott also claimed that hashtags are able to serve as highlighting devices of constituents in a post, enhancing readers' attention towards a specific chunk of text and increasing its saliency in readers' cognitive environments.

Network Representations and Hashtags
Unlike the linguistic aspects of hashtags, few research has touched upon learning hashtag representations in the form of graphs. A recent attempt is Hashtag2vec (Liu et al., 2018), a hashtag representation framework implementing a heterogeneous graph that connects hashtags with tweets and words in the corpus. They evaluate their framework using hashtag clustering tasks and compare the results to several different methods, including content-based, structure-based, and content-structure-based hashtag representation methods. An important discovery by Liu et al. is that content-based models perform better than structure-based models, but when the two are considered simultaneously, the model performs even better. Their claims align with that of Tu et al. (2017) in their CANE (context-aware network embedding) framework, pointing out that text and structure of a network are equally important for the calculation of network embeddings. TADW (text-associated deep walk) by Yang et al. (2015) is proposed based on similar concepts, they alter the DeepWalk algorithm (Perozzi et al.,2014) while taking into account the "sophisticated interactions between network structure and text information." In terms of the structural aspect of hashtag networks, some research first constructs the network through co-occurring hashtags in a post, and then apply network embeddings frameworks such as node2vec (Grover et al., 2016), LINE (Tang et al., 2015), or DeepWalk (Perozzi et al.,2014) to extract node embeddings. Examples include Hashtag2vec (Liu et al., 2018) and Wang et al. (2016), who examined information virality through the co-occurrence relationship between hashtags. As for the textual aspect of hashtag networks or general networks, neural embedding techniques with attention mechanisms or topic modeling techniques such as LDA, PLSA seem to be common choices.

Problem Statement
This section defines the goal of this paper. For a set of n posts, each post is represented by 3 modalities and a set of metadata, denoted as P = (Xt, Xi, Xh, Xm), where Xt, Xi, Xh, Xm denote text features, image features, hashtag features, and metadata respectively. Hashtag features Xh are generated by concatenating the average hashtag node embeddings V ) ! in a post with the post topic embedding XT, represented as ! = V ) ! ⨁ " . For the hashtag network G = (Vh, E), the vertices consist of all hashtags in the corpus, and edges connect two hashtags co-occurring in the same post. For post metadata, 3 features including text length, hashtag counts, and time of post during a day are considered. Finally, each post is assigned a popularity label (explained in section 4.2.2), being either popular (1) or unpopular (0). These binary labels, along with the multimodal representation of each post are trained together as a supervised binary classification model. Given a set of future posts on Instagram, the model can then be used to predict whether the posts will be popular or unpopular.

Data Collection
This section explains in detail the method of data collection and processing used in this study. The corpus is built from scratch using the python package "instagram-scraper" by arc298 on Github (https://github.com/arc298/instagramscraper). The package is able to crawl a diverse array of information, including, post captions, images, followers and following count, hashtags used, post timestamp, etc. The present study gathers Instagram posts from top 100 social media influencers in Taiwan; the list of top 100 influencers is provided by Business Next, an online media focusing on global trends in technology, the Internet, and digital marketing. The ranking list is compiled based on an interdisciplinary metric calculated through engagement factors from Facebook, Instagram, Youtube, and excluding actors or singers whose popularity do not originate from Internet platforms. This study selects only influencer posts as corpus entries for a few reasons. First of all, the main goal of this study is to provide commercial insights into predicting popular posts on Instagram. Therefore, as Instagram influencers usually participate in business activities, their posting behaviors will be of prior consideration compared to those of the general public. Secondly, influencers have a larger base of online audience, so their likes or comments count can easily reflect whether their audience is fond of their content or not. Compared to influencers, an average Instagram user only has a limited number of followers, so the likes and comments count will remain pretty much the same regardless of what is posted, making popularity hard to track. All posts are crawled and stored in the database on 2021/11/01. After the raw influencer posts are crawled, the following steps are taken to further process the posts: 1.
Exclude influencer accounts with less than 50 posts, leaving posts from 83 influencers.

2.
Collect only the most recent 550 posts for each influencer.

3.
Exclude posts submitted between 2021/10/01 -2021/11/01, as they are too recent to the crawling time, making their likes and comments count still unstable.

4.
Exclude videos from the dataset. If a post contains multiple images, consider only the first image (thumbnail image) for image representation. 5.
Construct hashtag network by linking all hashtags that co-occur in the same post. This results in a homogeneous graph with undirected edges. 7.
Perform topic modeling on the whole corpus using BERTopic. 250 topics are discovered and assigned to each post. Details of BERTopic will be explained in section 4.3.3.
The steps above leave the final dataset with a total of 19267 posts, and a hashtag network that contains 17906 unique nodes (hashtags) and 73300 edges. Every post contains 7 features ready for representation learning or model training, as shown in table 1 below. For text modality, each post caption is encoded to a feature vector of length 512 using sentence-transformers. For image modality, each image is represented as a feature vector of length 1024 using the Inception-V3 featurizer. As for hashtag modality, two kinds of embeddings are involved: topic embeddings (of length 512) calculated using BERTopic and average node embeddings (of length 50) calculated from the hashtag network using GraphSAGE, which combines into a hashtag feature vector of length 562. Last but not least, 3 additional metadata are used in the framework, as shown in table 1. The four components -text, image, hashtags modalities and metadata, are then concatenated in the fusion layer, forming a final feature vector of length 2101. Ultimately, the concatenated features are connected to a dense layer with a sigmoid activation function outputting the final prediction probability. The popularity class is then derived from the probabilities accordingly.

Defining Popular Posts
In this section, how popular posts are determined will be explained in detail. The methods used in this study mostly follows that of Carta et al. (2020); however, instead of calculating the likes moving average (LMA) as the popularity metric, the present study calculates the engagement moving average (EMA). Engagement E in this study is defined as the number of likes count plus 2 times the number of comments in each post. Let Pi be the i th post in an influencer account, the formula can be shown as follows: The EMA of a given post Pi can then be calculated by averaging the engagement of its previous k posts, as shown in the following formula: Finally, the popularity class (PC) of a given post Pi can be derived by comparing its engagement with its EMA times a parameter ∆ , as shown in the following conditional formula: According to the formula above, a post is labeled as a popular post (1) if its current engagement is greater than the average engagement of its previous k posts (EMA). Specifically, the parameter ∆ controls the strictness of the threshold; for example, when ∆ = 0.1, the engagement of a post needs to be greater than 1.1 times its EMA in order to be labeled as a popular post. In other words, as ∆ becomes higher, it will also be harder for a post to become a popular post, decreasing the total number of popular posts as well. In Carta's work, they experimented with 3 levels of k (10,30,50), and 4 levels of ∆ (0, 0.05, 0.1, 0.15). It is concluded that when k is larger, the accuracy of their model also rises accordingly. In light of this discovery, the present study simplifies Carta's experiments by directly assigning k as 50 while keeping 3 levels of ∆ (0, 0.1, 0.15).
There are a few benefits to the current method used to determine popular posts. To begin with, this method negates the biases caused by difference in the number of account followers. Some influencer accounts have significantly more followers than others, therefore, their posts will naturally have more likes and comments as the follower base is larger. If raw likes and comments are used as the popularity metric, popular posts will mostly contain those from high-follower accounts. Since EMA is calculated independently for each account, each influencer can still have a fair number of popular posts regardless of how many followers they have. Secondly, this method also avoids issues caused by posts submitted too long ago. As time goes by, influencers gradually gain more followers; as a result, posts that are submitted more recently generally have more engagement than those submitted years ago. This way, popular posts will simply contain posts that are more recent, but not posts whose content are actually appealing to the audience. As EMA considers previous k posts of a given post, posts from a long time ago can still be evaluated on a scale that matches its follower base at that time, hence solving the issue caused by different posting times.

Text Representation
Using the segmented post captions, this study has first attempted and compared a few text representation methods before combining with other modalities: • LSTM models incorporating self-trained word2vec embeddings as the embedding layer weights • The sentence-transformers framework using "distiluse-base-multilingual-cased-v1" as the pretrained model.

Model
Accuracy (∆=0.15) Single-layer LSTM w/ word2vec 59.63% Bidirectional LSTM w/ word2vec 56.93% Sentence-transformers 62.53% Table 2. Model accuracy comparison for text representation Table 2 compares the testing set accuracy from the above models using only post captions (texts) as inputs when ∆=0.15. As shown in table 2, text embeddings produced by sentence-transformers yield the best results compared to other sequence models. Therefore, sentence-transformers is chosen to be the text encoder in this study.

Image Representation
Image is also a very important aspect for Instagram posts. Standard and state-of-the-art image representation methods often involve knowledge in convolutional neural networks (CNN). The present study implements pic2vec, a lightweight image featurizer by datarobot on Github (https://github.com/datarobot/pic2vec), and selects the Inception-V3 pretrained model to encode each image into a feature vector of length 1024.

4.3.3
Hashtag and Topic Representation As mentioned in section 2.2.1, hashtags serve as effective topic markers and search tools in social media. In such context, the present study leverages the topicality and searchability of hashtags and attempt to utilize hashtags to reflect the topic and real-world information in a post. Sometimes, certain posts on social media receive a high amount of engagement not because of attractive captions or images, but because they contain trending hashtags. For example, posts with hashtags like #metoo, #blacklivesmatter, #tokyo2020 easily bring in events or issues happening in the real world, making these posts more relatable to the public and thus earning more engagement. It is clear at this point that hashtags carry valuable information which should not be ignored in a social media post representation. Since this study mainly focuses on influencer posts on Instagram, hashtags are widely presented throughout the corpus, making hashtags suitable targets for topic representation in the multimodal framework. However, as pointed out in section 2.2.2, representing textual or topical content is as important as representing the structural details of the hashtag network. Consequently, the present study combines two elements for hashtag representation: topic embeddings to represent hashtag content (content embedding) and node embeddings to represent the graphical structures between hashtags (structure embedding). Content embeddings are topic embeddings, which are extracted using the python package BERTopic by MaartenGr on Github (https://github.com/MaartenGr/ BERTopic). BERTopic is a topic modeling algorithm that uses the transformers framework and c-TFIDF to create topic representations. In short, BERTopic first creates a series of document embeddings using sentencetransformers, and then use UMAP (McInnes, 2018) for embeddings dimension reduction and HDBSCAN (Campello, 2013) for document clustering. Finally, they calculate the class-based TFIDF for each cluster (topic) and produce the importance scores of each word in every topic. The topic embedding for each topic is calculated by averaging all document embeddings within the same topic. As for structure embeddings, 2 different frameworks are used and compared in this study: node2vec and GraphSAGE. Node2vec generates node sequences using random walk and feeds them to word2vec to create node embeddings; GraphSAGE implements an unsupervised framework that learns a binary classifier, predicting whether arbitrary node pairs are likely to co-occur in a random walk performed on the graph. Both frameworks can produce informative node embeddings, and can be easily implemented using the python package stellar-graph. These two specific frameworks are chosen since they each represent a node representation technique; node2vec represents random walk-based methods, and GraphSAGE represents GCN-based methods. The hashtag network constructed previously are then used as input to the 2 methods, which produces node embeddings for each hashtag. If multiple hashtags (>1) appear in a single post, the structure embedding of a post will be calculated by averaging the node embeddings of all hashtags in the post. On the other hand, if a post contains does not contain any hashtag, then a zero-vector will be assigned as the structure embedding. Table 3 shows a comprehensive comparison between the performance (testing set accuracy) of different hashtag representation strategies. The ⨁ symbol represents the concatenation operation. The accuracy is calculated with ∆=0.15:

Model
Accuracy ( It can be observed in table 3 that the topic ⨁ GraphSAGE combination outperforms all other models. Therefore, hashtags in this study are represented by concatenating the topic embedding of a post with its structure embedding derived from GraphSAGE.

Metadata
As shown in table 1, there are in total 3 kinds of metadata used in the present study. T-tests are performed respectively on text_length and hashtag_num, and both tests yield p-values smaller than 0.001. As for the time of post feature, a day is divided into 4 sections (as in figure 1 (c)) and represented as a categorical feature with 4 levels. A chi-square test performed on timeofpost also yields pvalue smaller than 0.001. Therefore, text_length, hashtag_num, and timeofpost are all statistically significant features between popular and unpopular posts. Before the model training stage, text_length and hashtag_num are both normalized using the StandardScaler() function from scikitlearn.

Evaluation Metrics
In the results section, 2 different kinds of evaluation metrics will be used to present the performance of the models, including accuracy and balanced accuracy. Accuracy is defined as the ratio of the number of correct predictions to the total number of input samples, while balanced accuracy is defined as the average of recall obtained on each class. Balanced accuracy excels at evaluating the performance of a binary classifier since it avoids false high accuracy caused by an imbalanced dataset.

Baselines
To better evaluate the effectiveness of the features used in each model, 4 different baseline methods will be presented along the proposed framework. These baselines do not involve any machine learning techniques, but are instead based on naïve assumptions on the dataset. The first baseline simply predicts all posts in the dataset as unpopular posts: For the other three baselines, the popularity class is derived based on a certain threshold. To be more specific, a post will be predicted as a popular post if its engagement exceeds an engagement threshold T: Baseline n_ %,∆ (P ( ) = 0 1(popular), if E(P ( ) > T 0(unpopular), otherwise ℎ = (30,000, 60,000, 100,000), = (2,3,4)

Results Overview
This section compares the results from combining different modalities and popularity parameters. The proposed framework is evaluated together with the baselines established in section 4.2.2 using the metrics in section 4.4.1. A possible explanation for this trend is that as ∆ rises, the threshold for a post to become a "popular post" rises as well, causing fewer popular posts to appear in the dataset. In this case, the accuracy of predicting a post as unpopular (0) will increase due to imbalanced dataset distribution. However, even when ∆ = 0.15, predicting all instances as unpopular (baseline 1), only yields an accuracy of 61.96%. At the same ∆ level, all other multimodal models outperform the results of baseline 1 by 2~3%. Therefore, it can be inferred that even higher ∆ levels lead to imbalanced dataset, the multimodal models are still able to capture additional information rather than just guessing posts as unpopular.
Out of the three multimodal models, the text + image combination produces the best results when ∆ = 0 and 0.1, but when ∆ = 0.15, combining text, image, and other modalities produces even better results. As a result, it is possible that a stricter popularity threshold could bring out the effectiveness of combining multiple modalities. In addition, figure 2 and table 4 both demonstrate the fact that multimodal models outperform any unimodal models, proving that modality fusion is able to model the interaction between modalities and boost overall performance. On the contrary to accuracy, overall balanced accuracy decreases as ∆ rises, as shown in figure 2 (b). This is probably due to the recall obtained on popular posts decreases as popularity threshold goes up. Precisely, as ∆ rises, the models start to predict posts as unpopular (0) more frequently, and rarely predicts a post as a popular post. Therefore, the recall for popular posts will drop significantly, which in turn lowers the balanced accuracy.
In other words, the model's abilities to predict popular posts are hindered at higher ∆ levels.

Effectiveness of the Hashtag Modality
An important goal of the present study is to harness the information provided from hashtag networks and improve the quality of post representations on social media. As mentioned in the introduction, we hope that hashtag networks can be used as a new form of hashtag representation method for other tasks as well. Consequently, this section describes in detail the effectiveness of hashtag representation framework used in this study.
To begin with, the hashtag modality gets increasingly effective as ∆ rises. Compare the accuracies of text + image and text + image + hashtag models; when ∆ = 0, text + image performs better by 0.99%, and when ∆ = 0.1, text + image still performs better, but only by 0.37%. However, at ∆ = 0.15, text + image + hashtag outperforms text + image by 0.83%. The same trend applies to balanced accuracy, with text + image + hashtag performing better than text + image when ∆ = 0.15. In conclusion, at higher ∆ levels, the addition of hashtag modality can further improve model performance.
Secondly, as mentioned previously, the hashtag modality consists of 2 components: structure (node) embeddings produced by GraphSAGE and content (topic) embeddings produced by BERTopic. From table 3 in section 4.3.3, it can be observed that when the two are combined together, they indeed perform better than GraphSAGE embeddings or topic embeddings alone. The results align with those mentioned in previous works, where they claim that semantic information has to be considered with structural information in a graph to produce better results. The hashtag network used in this study is presented below: Figure 3. A simplified illustration of the hashtag network Figure 3 shows the simplified hashtag network constructed from the co-occurrence relationship between hashtags in a post. Overall, the network shows an island-like distribution; some bigger node clusters are connected in the middle while smaller ones spread on the outer areas. Each hashtag cluster in the network represents and captures latent topics among the posts. However, in the current dataset, there is a considerable number of "lone hashtags", meaning that these hashtags only appear once in the entire dataset and are isolated from any node cluster. These hashtags usually contain a full sentence and are especially long in length; this is perhaps due to the fact that unlike English, words are connected without spaces in Chinese, so the hashtag can run on while maintaining readability. In this study, lone hashtags are still mapped into the hashtag network to  Therefore, when combined together as the hashtag representation, hashtag network and topic model might mutually enhance their abilities to produce more informative embeddings.

Conclusion and Future Work
This paper proposes a multimodal framework incorporating hashtag network to predict popular Instagram posts of Taiwanese influencers. To be more precise, the hashtag modality of a post is constructed by fusing node embeddings from the hashtag network and topic embeddings from BERTopic. Instead of predicting raw likes or engagement, this study follows Carta et al. (2020) and builds a binary classifier that predicts posts as either popular or unpopular. Results indicate that the proposed multimodal framework surpasses all baselines and unimodal models. More importantly, the hashtag modality is shown to improve overall model performance at higher ∆ levels, while its two components complement each other by capturing different post details. There are some future directions for the present study. First of all, the quality of hashtag embeddings can be improved by trying out different node representation algorithms and topic models; in addition, the attention mechanism might also be of much help in order to design a more sophisticated hashtag representation framework. Secondly, lexical analyses should be performed on popular and unpopular posts, as the textual/multimodal information hidden in the posts can not only contribute to model improvement but also benefit influencers and the marketing industry. Finally, the ultimate goal of this study is to develop hashtag networks as a language resource for social media/text analytics. Besides popularity prediction, hashtag networks will also be applied to several other tasks to evaluate their full effectiveness or potential. We hope that hashtag networks can eventually become a convenient yet effective way of representing hashtags, and benefit future research in social media.

Acknowledgments
I would like to thank everyone from the LOPE lab for providing me with invaluable feedback and new research directions. I would also like to thank Tagnology.co for providing me with relavant data and advice on the model structure.