## Highlights

## Identifying the Hierarchical Emotional Areas in the Human Brain Through Information Fusion

Zhongyu Huang,Changde Du,Chaozhuo Li,Kaicheng Fu,Huiguang He

- • Identify hierarchical emotional areas to study brain mechanisms underlying emotion.
- • Conduct an in-depth theoretical analysis based on information fusion and graph theory.
- • Develop a novel framework to improve emotion decoding using the identified areas.
- • Demonstrate the potential of identified areas by cross-dataset emotion decoding.

## Identifying the Hierarchical Emotional Areas in the Human Brain Through Information Fusion

Zhongyu Huang a,b , Changde Du , a Chaozhuo Li , c Kaicheng Fu a,b and Huiguang He a,b , ∗

- a Laboratory of Brain Atlas and Brain-Inspired Intelligence, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
- b School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China
- c Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing, 100876, China

## ARTICLE INFO

## ABSTRACT

Keywords :

Emotion

Human brain

Hierarchical emotional areas

Multi-source information fusion

Graph machine learning

Psychological constructionism

The brain basis of emotion has consistently received widespread attention, attracting a large number of studies to explore this cutting-edge topic. However, the methods employed in these studies typically only model the pairwise relationship between two brain regions, while neglecting the interactions and information fusion among multiple brain regions-one of the key ideas of the psychological constructionist hypothesis. To overcome the limitations of traditional methods, this study provides an in-depth theoretical analysis of how to maximize interactions and information fusion among brain regions. Building on the results of this analysis, we propose to identify the hierarchical emotional areas in the human brain through multi-source information fusion and graph machine learning methods. Comprehensive experiments reveal that the identified hierarchical emotional areas, from lower to higher levels, primarily facilitate the fundamental process of emotion perception, the construction of basic psychological operations, and the coordination and integration of these operations. Overall, our findings provide unique insights into the brain mechanisms underlying specific emotions based on the psychological constructionist hypothesis.

## 1. Introduction

The research on the brain basis and neural representation of emotion mainly focuses on two hypotheses: locationism and psychological constructionism . The locationist hypothesis suggests that discrete emotion categories consistently and specifically correspond to distinct brain regions, implying a one-to-one mapping between brain regions and emotion categories [4]. On the contrary, the psychological constructionist hypothesis suggests that discrete emotion categories are constructed from more general brain networks, i.e., the interactions among various brain regions, rather than specific brain regions [4]. While both hypotheses are of equal age, there is relatively little evidence supporting the locationist hypothesis. In contrast, a large number of studies [413] demonstrate that the neural representation of emotion is distributed, leading to the conclusion that the psychological constructionist hypothesis has greater potential for revealing the brain basis of emotion.

Emotions are complex psychological and physiological states that arise in response to stimuli from internal feelings or external environments. Generally, these external stimuli are perceived in diverse manners, such as visual experiences, audio signals, and textual information. Our brains respond to these stimuli by generating neural activity in various brain regions, thereby producing corresponding psychological and physiological states. Therefore, the fusion of multi-source information from either diverse manners or various brain regions contributes to understanding the brain mechanisms underlying emotion [1-3].

∗ Corresponding author

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00525v1-figure-faa3e47f-3088-40f2-afdf-726aa38ebc55.jpg)

To investigate the brain basis of emotion, many previous studies have employed various methods to identify patterns of brain activity associated with specific emotions, such as multivariate pattern analyses [7, 9], Bayesian approaches [5, 6], and brain networks [8, 12, 13]. However, these methods either fail to leverage the relationships between brain regions or only model the pairwise relationship between two brain regions, while neglecting the interactions and information fusion among multiple brain regions-one of the key ideas of the psychological constructionist hypothesis. As a result, their findings on the brain regions involved in emotion encoding vary, and even a consistent conclusion on the brain mechanisms for specific emotions has yet to be established.

huangzhongyu2020@ia.ac.cn (Z. Huang); huiguang.he@ia.ac.cn (H.

To reveal the brain mechanisms underlying emotion based on the psychological constructionist hypothesis, it is essential to consider the interactions and information fusion among multiple brain regions at a higher level. From a network neuroscience perspective, the brain regions involved in emotion encoding present a modular structure [14, 15]. Modules with similar functions are coupled together, forming a unique area with a specific function. We refer to this unique area as the emotional area , similar to the functional areas in the human brain. Moreover, a series of influential studies [16, 17] demonstrates that brain networks further exhibit hierarchically modular organizations. The main module in a brain network plays a dominant role, with a set of submodules cooperating to perform subordinate functions. In addition to these sub-modules, there exist sub-sub-modules and so forth, each playing an increasingly subtle role. Accordingly, in the context of emotion encoding, the entire brain network can be decomposed into hierarchical subnetworks that correspond to hierarchical emotional areas .

Since emotions are complex subjective experiences, they include multiple psychological operations and cognitive processes, thereby involving various brain regions [4]. When an individual experiences emotional stimuli, the entire brain network is decomposed into hierarchical subnetworks. Each of these subnetworks corresponds to an emotional area and performs distinct functions, contributing to different psychological operations and cognitive processes involved in emotion encoding. Meanwhile, subnetworks in various emotional areas interact and collaborate to create a comprehensive brain network for emotion encoding. Based on the hierarchically modular organization of the brain network, this analysis aligns perfectly with both the key idea of the psychological constructionist hypothesis and the professional knowledge in affective science. Therefore, this work aims to investigate the hierarchical emotional areas in the human brain through multi-source information fusion and graph machine learning methods. Our ultimate goal is to reveal the brain mechanisms underlying specific emotions based on the psychological constructionist hypothesis.

Overall, the key innovations and main contributions of our research can be summarized as follows:

- · We introduce the concept of hierarchical emotional areas to understand the brain mechanisms underlying emotion, aligning perfectly with both the key idea of the psychological constructionist hypothesis and the professional knowledge in affective science.
- · We provide an in-depth theoretical analysis of how to maximize interactions and information fusion among brain regions. Building on the results of this analysis, we identify hierarchical emotional areas in the human brain.
- · We develop a novel framework, Hierarchical Emotion Network (HEmoN), which improves emotion decoding tasks by exploiting the identified hierarchical emotional areas.
- · To verify the rationality and effectiveness of our approach, we conduct experiments on two large-scale datasets, both containing audio-visual multi-source naturalistic stimuli. Furthermore, we accomplish a challenging task: identifying hierarchical emotional areas on one dataset and applying them to crossdataset emotion decoding on another dataset.

## 2. Related work

Emotions arise from the activations of specialized neuronal populations in the human brain. Therefore, researchers have devoted themselves to exploring the neural representation of emotion to gain a deeper understanding of emotion. One of the most prominent research areas is emotion encoding and decoding , which aims to identify the mapping patterns between emotional stimuli and their corresponding brain responses. In essence, encoding and decoding are complementary operations: encoding investigates how to transform stimuli into brain responses, while decoding investigates how to predict information about the stimuli from brain responses [18].

Given their simplicity, efficiency, and strong interpretability, a large number of studies [5-7, 9-11, 27] employ linear regression or multivariate statistical methods to develop models for emotion encoding or decoding. Unfortunately, these methods are unable to leverage the rich relationships between brain regions. To overcome this limitation, some pioneering works [8, 12, 13, 31-36] use fMRI data to construct functional brain networks [37], which can explicitly model the pairwise relationship between brain regions. A series of works [8, 31-33, 35] uses diverse network measures to assess the significance of each region of interest in the brain network, thereby studying the mechanisms of emotion encoding. Another series of works [12, 13, 36] maps various emotion categories into distinct brain networks with more distinguishable graph structural features, thereby improving the performance of emotion decoding. In addition, several recent advances [8, 34, 38, 39] have made remarkable achievements in interpreting emotions through various graph machine learning methods. Huang et al. [8] investigate the connectivity within the brain network and reveal a general emotion pathway connecting neural nodes involved in basic psychological operations. Li et al. [34] design a graph neural network framework for analyzing brain networks and utilize it to infer salient brain regions for emotional tasks. Huang et al. [38] construct an emotion-brain bipartite graph to model the relationships between emotions and brain regions, thereby improving the representation learning process. Lettieri et al. [39] study the encoding rules

In laboratory settings, researchers usually measure and collect the brain signals of subjects when they experience emotional stimuli. By analyzing the collected brain signals and the experienced emotional stimuli, researchers can delve into emotion encoding and decoding. Commonly used brain signals include functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and magnetoencephalography (MEG). Due to its high spatial resolution, fMRI allows precise localization of the active brain regions involved in emotional processing [19], providing an excellent opportunity to study the underlying brain mechanisms for specific emotions [20]. On the other hand, traditional emotional stimuli include images, especially facial expressions [21, 22], and mental imagery [7, 23]. However, these stimuli are abstract, simplified, and, in many ways, fail to reflect the complexity and dynamics inherent in real-life behaviors and stimuli [24]. To investigate real-life emotional experiences, researchers are increasingly turning to naturalistic paradigms, such as watching movies [10, 13], listening to stories [12, 25], or experiencing audio-visual multi-source stimuli [11, 26, 27]. A growing body of evidence suggests that these naturalistic stimuli benefit from the fusion of multi-source information, incorporate richer and more dynamic experiences, evoke stronger emotions, and provide a more reasonable approximation to freeform cognition in our daily lives [24, 28-30].

of emotional instances and discover that they are represented in an extensive network encompassing sensory, prefrontal, and temporal regions.

Although these works have employed graph machine learning methods to investigate emotional information in the brain more effectively, their primary focus remains on identifying key brain regions or capturing pairwise relationships between two objects. These approaches have essentially deviated from the key idea of the psychological constructionist hypothesis, which emphasizes the interactions and information fusion among multiple brain regions throughout the entire methodology. In addition, these works have rarely considered the hierarchically modular organization of the brain network, neglecting important facts in network neuroscience. Consequently, we aim to investigate the hierarchical emotional areas in the human brain by combining graph machine learning methods with information fusion, thereby overcoming the above challenges and revealing the brain mechanisms underlying specific emotions.

## 3. Preliminaries

## 3.1. Graph theory

A walk in graph 𝐺 is a finite sequence of alternating vertices and edges, such as 𝑣 , 𝑒 0 1 , 𝑣 1 , 𝑒 2 , … , 𝑒 𝑚 , 𝑣 𝑚 , in which each edge 𝑒 𝑖 = ( 𝑣 𝑖 -1 , 𝑣 𝑖 ) . A walk may contain repeated edges. A path is a walk in which all vertices, and hence all edges, are distinct (except, possibly, 𝑣 0 = 𝑣 𝑚 ). We denote a path 𝑃 with length 𝑚 as its vertex sequence ( 𝑣 , 𝑣 0 1 , … , 𝑣 𝑚 ) . If 𝑣 0 = 𝑣 𝑚 and 𝑚 ≥ 2 , this path is called a cycle .

Let 𝐺 = (   , ) ∈  be a graph with vertex set  = { 𝑣 , 𝑣 1 2 , … , 𝑣 𝑁𝐺 } and edge set  = { 𝑒 , 𝑒 1 2 , … , 𝑒 𝑀𝐺 } , where 𝑁𝐺 = |  | and 𝑀𝐺 = |  | represent the number of vertices and edges in 𝐺 , respectively. Let 𝑨 ∈ ℝ 𝑁𝐺 × 𝑁𝐺 be the adjacency matrix of 𝐺 , and 𝑿 ∈ ℝ 𝑁𝐺 × 0 𝑐 be its node feature matrix. For each node 𝑣 ∈  , 𝒙 𝑣 ∈ ℝ 𝑐 0 denotes its node feature vector, and 𝑑 𝑣 denotes its degree (i.e., the number of 1-hop neighbors). The set of 𝑣 's 1-hop neighbors is represented as  ( 𝑣 ) .

In an unweighted graph 𝐺 , the shortest path between two vertices is the path with the minimum length between them, and its length is called the (shortest path) distance between these two vertices. We use dist ( 𝑢, 𝑣 ) to denote the distance between a pair of vertices 𝑢 and 𝑣 . The longest shortest path in graph 𝐺 is the shortest path with the maximum length between any pair of vertices. The diameter of graph 𝐺 is the length of the longest shortest path, defined as

$$\mathfrak { d } = \max \left \{ \text{dist} ( u, v ) \, | \, \forall u, v \in \mathcal { V } \right \}.$$

In this paper, we use the normal 𝑑 to denote the node degree and the Gothic 𝔡 to denote the graph diameter .

A tree is a special type of graph that is connected and acyclic. Trees have several essential properties; for instance, there exists exactly one path between any pair of vertices. Additionally, a tree with 𝑁𝑇 vertices has exactly 𝑁𝑇 - 1 edges. A forest is a collection of disjoint trees, and a tree is a special type of forest with only one connected component.

## 3.2. Psychological constructionist hypothesis

The core affect is a mental representation of bodily sensations, enabling an organism to determine whether something in the environment has motivational salience, e.g., whether it is beneficial, detrimental, approachable, or avoidable. The conceptualization is a process that imbues sensations from the body or the external world with meaning in a given context by drawing on representations of previous experiences. These representations are activated by the current physical and psychological situation. The executive attention is a process that selectively enhances some representations while suppressing others. It can regulate the activity in other processes, such as core affect, conceptualization, or language use. The language word serves as an 'essence placeholder,' which helps integrate feelings, behaviors, and expressions into instances of a meaningful emotion category.

The psychological constructionist hypothesis assumes that emotion categories are common-sense categories, with instances emerging from the combination of more basic psychological operations, such as core affect, conceptualization, executive attention, language, etc. These psychological operations are the common ingredients of all mental states.

## 4. Hierarchical emotional areas: Theoretical analysis and methodology

In this section, we present the identification of hierarchical emotional areas and their application in emotion decoding. We start by extracting a brain tree from the brain network to overcome its challenges and facilitate the capture of hierarchical information. Furthermore, we provide an in-depth theoretical analysis based on this brain tree to maximize interactions and information fusion among brain regions, aligning with the key idea of the psychological constructionist hypothesis. Building on the results of this analysis, we identify the hierarchical emotional areas and present corresponding technical implementations. Finally, we develop a novel framework to improve emotion decoding tasks by exploiting the identified hierarchical emotional areas. The pipeline of this study is illustrated in Figure 1.

## 4.1. Extracting the brain tree

However, conventional brain networks may suffer from various limitations, such as threshold problems [42-45], spurious and noisy connections [42-45], scaling effects [42, 44], and methodological biases [43, 44]. Fortunately, many studies [43-45] have demonstrated that the spanning tree can effectively overcome these challenges and capture the essential properties of complex brain networks, leading to

Wefirst construct a brain network, 𝐺 = (   , ) , using the collected fMRI signals. In this network, nodes are defined as brain regions of interest (ROIs), generally derived from an existing brain parcellation, such as the Power Atlas [40] and the Destrieux Atlas [41]. The edges are defined as the functional connectivity between these ROIs, with functional connectivity commonly computed as the pairwise correlation of the fMRI time series between two ROIs.

Figure 1: The pipeline of identifying hierarchical emotional areas in the human brain. To investigate the brain mechanisms underlying emotion (i.e., emotion encoding), we first measure and collect the brain signals of subjects when they experience audio-visual multi-source emotional stimuli. Next, we construct a brain network using the collected fMRI signals and extract a brain tree from this brain network. Then, we decompose the entire brain tree into trunks at different levels (i.e., hierarchical trunks), with each trunk facilitating information fusion among brain regions. Following the decomposition, we revert these hierarchical trunks in the brain tree back to hierarchical emotional areas in the human brain. As a result, we complete the identification of hierarchical emotional areas on a given dataset. Finally, we use the proposed model, HEmoN, which builds on these identified hierarchical emotional areas, to perform cross-dataset emotion decoding on other challenging datasets.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00525v1-figure-82b2e360-d199-471b-b119-09848bc5b203.jpg)

great success in a wide range of applications. Moreover, previous research [46] has indicated that the tree structure is an efficient tool for capturing hierarchical information from data, aligning with our goal of identifying hierarchical emotional areas. Therefore, we propose to extract a brain tree from the original brain network using the spanning tree algorithm.

influence that one brain region (i.e., node) exerts on another, we start our analysis by quantitatively describing the information influence between an arbitrary pair of nodes, 𝑢 and 𝑣 . Based on previous literature [47, 48], we take the random walk as a general rule for information propagation in the brain tree and define node influence as follows.

Specifically, given a brain network, we first sort all edges in descending order according to their weights. The construction process of the brain tree begins by using the edge with the highest weight. Subsequently, we select the edge with the highest weight from the remaining edges. If the newly selected edge forms a cycle with the previously selected edges, it is discarded; otherwise, it is added to the brain tree. This process is executed iteratively until there are 𝑁𝐺 -1 edges in the brain tree. Finally, all edge weights in the brain tree are set to a value of one. In this way, we transform the brain network 𝐺 = (   , ) into a brain tree 𝑇 = (   , ) , where  denotes the vertex set of 𝑇 , identical to that of 𝐺 , and  denotes the edge set of 𝑇 .

## 4.2. Theoretical analysis: Interactions and information fusion in the brain tree

As discussed in Section 1, the psychological constructionist hypothesis emphasizes the interactions and information fusion among multiple brain regions. Since these interactions can be essentially considered as the information

Definition 1 (Node Influence). Let 𝒉 (0) 𝑣 ∈ ℝ 𝑐 0 be the initial feature vector of node 𝑣 , and 𝒉 ( 𝑘 ) 𝑢 be the representation vector of node 𝑢 after 𝑘 steps of the random walk. The node influence 𝐼 𝑢, 𝑣 ( ) of node 𝑣 on node 𝑢 after 𝑘 steps of the random walk is defined as the norm of the Jacobian matrix 𝜕 𝒉 ( 𝑘 ) 𝑢 ∕ 𝜕 𝒉 (0) 𝑣 :

where the norm ‖ ⋅ ‖ is an arbitrary induced norm 1 .

$$I ( u, v ) = \left \| \frac { \partial h _ { u } ^ { ( k ) } } { \partial h _ { v } ^ { ( 0 ) } } \right \|,$$

The node influence quantifies how changes in the information of node 𝑣 affect that of node 𝑢 . The following theorem presents an analytical expression of the node influence for further analysis.

Theorem 1. Given a tree 𝑇 , the node influence 𝐼 𝑇 ( 𝑢, 𝑣 ) of node 𝑣 on node 𝑢 after 𝑘 = dist ( 𝑢, 𝑣 ) steps of the random

1 An induced norm is a matrix norm induced by a vector norm, defined as ‖ 𝑨 ‖ = sup { ‖ 𝑨𝒙 ‖ ∣ ‖ 𝒙 ‖ = 1} .

walk is

$$I _ { T } ( u, v ) = \left | \frac { a _ { u v _ { 1 } } \cdot a _ { v _ { 1 } v _ { 2 } } \cdots a _ { v _ { k - 1 } v } } { \sum _ { i \in \mathcal { N } ( u ) } a _ { u i } \cdot \sum _ { j \in \mathcal { N } ( v _ { 1 } ) } a _ { v _ { 1 } j } \cdots \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } } \right |, \quad \text{inf} ( \text{$\text{$\text{$m$}$} } ).$$

where the only path between nodes 𝑢 and 𝑣 is denoted as ( 𝑢, 𝑣 1 , 𝑣 2 , ⋯ , 𝑣 𝑘 -1 , 𝑣 ) ; 𝑎 𝑣 𝑖 𝑣 𝑗 is the edge weight between nodes 𝑣 𝑖 and 𝑣 𝑗 , and  ( 𝑢 ) is the set of 𝑢 's 1-hop neighbors.

Detailed proof of Theorem 1 is provided in Appendix A. Since all edges in the brain tree have a uniform weight of 1, the result of Theorem 1 can be further simplified to

$$I _ { T } ( u, v ) = \frac { 1 } { d _ { u } d _ { v _ { 1 } } \cdots d _ { v _ { k - 1 } } } \quad \quad ( 1 ) \quad \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$$

where 𝑑 𝑢 is the degree of node 𝑢 . Eq. (1) indicates that node influence decreases as the distance 𝑘 between nodes and node degrees increase, consistent with intuition. More precisely, node influence 𝐼 𝑇 ( 𝑢, 𝑣 ) decreases exponentially with the increase in the distance between nodes 𝑢 and 𝑣 , and decreases polynomially with the increase in node degrees on the path connecting 𝑢 and 𝑣 .

To facilitate effective interactions among brain regions, we aim to maximize the node influence 𝐼 𝑇 ( 𝑢, 𝑣 ) in the brain tree. This objective can be achieved by solving the following optimization problem:

$$\max _ { d _ { u }, d _ { v _ { i } } } & \quad I _ { T } ( u, v ) = \frac { 1 } { d _ { u } d _ { v _ { 1 } } \cdots d _ { v _ { k - 1 } } } \\ \text{s.t.} \quad & d _ { u } \ \geq 1 \\ & \quad d _ { v _ { i } } \geq 2, \ \forall i = 1, 2, \dots, k - 1$$

Clearly, the optimal solution is ̂ 𝑑 𝑢 = 1 , 𝑑 ̂ 𝑣 𝑖 = 2 , which corresponds to the path between nodes 𝑢 and 𝑣 . Since trees have an essential property that there exists exactly one path between any pair of nodes, the path between 𝑢 and 𝑣 is also the shortest path. Accordingly, the optimal solution motivates us to propagate information along the shortest path, thereby facilitating interactions among brain regions.

Furthermore, it is essential to consider the fusion of information among multiple brain regions, enabling them to form a unique emotional area with a specific function. The above analysis indicates that information should be propagated along the (shortest) path. Consequently, we continue our analysis by quantifying the total amount of information carried by a path and defining path information as follows.

Definition 2 (Path Information). Let 𝑃 denote a path of length 𝑚 with vertex sequence ( 𝑣 , 𝑣 0 1 , … , 𝑣 𝑚 ) . The path information 𝐼 𝑃 ( ) of path 𝑃 is defined as the sum of the node influences between every pair of nodes on the path:

$$I ( P ) = \sum _ { \substack { i, j = 0 \\ i \neq j } } ^ { m } I _ { P } ( v _ { i }, v _ { j } ).$$

The path information quantifies the extent of information fusion among nodes on path 𝑃 . We aim to maximize the path information, thereby enhancing the fusion of information among brain regions as much as possible. The following theorem gives the path with the maximum path information.

Theorem 2. Given a tree 𝑇 , the longest shortest path ̂ 𝑃 has the maximum path information

$$I ( \hat { P } ) = \sum _ { k = 1 } ^ { \mathfrak { d } } ( \mathfrak { d } - k + 1 ) \cdot \frac { 2 } { 2 ^ { k - 1 } },$$

where 𝔡 is the diameter of tree 𝑇 .

Detailed proof of Theorem 2 is provided in Appendix B. Theorem 2 indicates that the longest shortest path has the maximum amount of information among all possible paths in a brain tree. Consequently, the theoretical analysis in this subsection demonstrates that propagating information along the longest shortest path on a brain tree maximizes interactions and information fusion among brain regions. In the next subsection, we will exploit the longest shortest path to further identify the emotional area and present the corresponding implementations.

## 4.3. Identifying the hierarchical emotional areas

Specifically, the 1st-level trunk, 𝔱 1 , serves as the main trunk of the entire brain tree, and there is usually only one main trunk at the first level. The trunk at the remaining 𝓁 th ( 𝓁 ≥ 2 ) level, 𝔱 𝓁 , is considered a branch of the entire brain tree, and there are usually multiple branches at the 𝓁 th level. We denote the set of trunks at the 𝓁 th level as  𝓁 . Our previous theoretical analysis suggests that propagating information along the longest shortest path on a brain tree maximizes interactions and information fusion among brain regions. Therefore, we designate the longest shortest path as the main trunk 𝔱 1 . In addition, the brain regions corresponding to all nodes on this main trunk constitute the main emotional area, denoted as  1 . Once the main trunk is obtained and removed from the brain tree, the remaining part becomes a forest. This forest contains at least one (often multiple) connected component(s), and every component is a (sub)tree [49]. To identify the trunks at the 𝓁 th ( 𝓁 ≥ 2 ) level, we search for the longest shortest path in each connected component of the forest after removing the trunks of all previous ( 𝓁 -1) levels. Similarly, the brain regions corresponding to all nodes on these 𝓁 th-level trunks constitute the 𝓁 th-level emotional area, denoted as  𝓁 . This process runs iteratively and ends once all nodes in the tree have been removed. Figure 4 in Appendix C provides a concrete example to illustrate the process above.

In the natural world, the tree trunk is an essential component of a tree. The remaining parts of a tree after removing the tree trunk are also regarded as (sub)trees, each with its own '(sub)tree trunk.' Thus, natural tree trunks exhibit hierarchical organization throughout the entire tree. Inspired by this interesting natural phenomenon, we decompose the entire brain tree into hierarchical trunks and leverage them to identify the hierarchical emotional areas.

Algorithm 1 Identifying the hierarchical emotional areas.

Output: The emotional areas at every level.

Input: A brain tree 𝑇 = (   , ) .

1: 𝓁 ← 0 ;

3: while  ≠ ∅ do

2: 𝐹 0 ← (   , ) ;

4:

𝓁

←

𝓁

+1

for

5:

𝔠

;

,

= 1 …

,

ℭ

do

7: Get the set  𝓁 , 𝔠 of all nodes in 𝔱 𝓁 , 𝔠 ;

𝓁 -1 6: Identify an 𝓁 th-level trunk 𝔱 𝓁 , 𝔠 from the 𝔠 th connected component of 𝐹 𝓁 -1 ;

←

8: Get the set  𝓁 , 𝔠 of all edges in 𝔱 𝓁 , 𝔠 ; 9:  ;



⧵



,

𝔠







12:

𝓁 10: Find the set  𝓁 , 𝔠 of all isolated nodes; 11: ;

←

⧵

𝓁

,

𝔠

end for







15:

13:  𝓁 ← { 𝔱 𝓁 , 1 , … , 𝔱 𝓁 , ℭ 𝓁 -1 } ; 14:

𝓁

←

𝓁

,

1

∪

⋯

∪

𝓁

,

ℭ

;

𝐹

𝓁

-1

 

16:

𝓁

←

(

,

)

end while

;

17: return   1 , 2 , ⋯ ,  𝐿

Suppose there are 𝐿 levels in total. Algorithm 1 describes the detailed process of identifying hierarchical emotional areas. Here, 𝐹 𝓁 represents a forest obtained after removing the trunks of all previous 𝓁 levels and the isolated nodes. Line 2 initializes the forest 𝐹 0 to tree 𝑇 = (   , ) . Line 5 considers all ℭ 𝓁 -1 connected component(s) in 𝐹 𝓁 -1 . Line 6 searches for the longest shortest path in the 𝔠 th connected component of 𝐹 𝓁 -1 to identify an 𝓁 th-level trunk 𝔱 𝓁 , 𝔠 . Lines 7 and 14 identify the emotional area,  𝓁 , at the 𝓁 th level. Lines 8-9 remove all edges in 𝔱 𝓁 , 𝔠 from 𝐹 𝓁 -1 , and Lines 10-11 remove all isolated nodes. Line 15 updates 𝐹 𝓁 based on the results of Lines 9 and 11. If there are no remaining nodes, Line 16 terminates the loop, and finally, Line 17 returns the emotional areas at every level.

## 4.4. Practical application: Emotion decoding

We further develop a novel framework, Hierarchical Emotion Network (HEmoN), to improve emotion decoding tasks by exploiting the identified hierarchical emotional areas. As previously analyzed, we identify hierarchical trunks by iteratively searching for the longest shortest path. Given that nodes on the longest shortest path (i.e., a trunk) form a long sequence, we use the Long Short-Term Memory (LSTM) [50] to effectively capture long-term dependencies and recognize complex patterns, thereby facilitating the representation learning process of emotion decoding. Specifically, after obtaining the trunks at every level according to Line 13 of Algorithm 1, we apply an LSTM to learn the representation for each trunk along its corresponding trunk path, formulated as

$$h _ { T } ^ { ( \ell ) } = \sum _ { t _ { \ell } \in \mathcal { T } _ { \ell } } LSTM \left ( x _ { v _ { 0 } } ^ { ( t _ { \ell } ) }, x _ { v _ { 1 } } ^ { ( t _ { \ell } ) }, \cdots, x _ { v _ { k _ { t _ { \ell } } } } ^ { ( t _ { \ell } ) } \right ) \quad ( 2 ) \quad \text{escape} \quad \text{accol} \\ \quad \text{to sea}$$

where  𝓁 is the set of trunks at the 𝓁 th level, and 𝔱 𝓁 is a single trunk in  𝓁 ; let the length of 𝔱 𝓁 be 𝑘 𝔱 𝓁 ; 𝒙 ( 𝔱 𝓁 ) 𝑣 𝑖 ∈ ℝ 𝑐 0 denotes the feature vector of node 𝑣 𝑖 on trunk 𝔱 𝓁 , and 𝒉 ( 𝓁 ) 𝑇 ∈ ℝ 𝑐 𝓁 denotes the trunk representation at the 𝓁 th level. Since nodes on all trunks in  𝓁 constitute the 𝓁 th-level emotional area, 𝒉 ( 𝓁 ) 𝑇 also serves as the representation of the emotional area at the 𝓁 th level. Then, we combine all the trunk representations to create a representation of the brain tree:

$$h _ { T } = \sum _ { \ell = 1 } ^ { L } W ^ { ( \ell ) } h _ { T } ^ { ( \ell ) }$$

where 𝑾 ( 𝓁 ) represents a learnable weighting matrix for the 𝓁 th level, optimized along with other model weights during the model training process. For emotion classification tasks, the brain tree is associated with an emotion category label 𝑦 𝑇 from 𝐶 emotion categories. Accordingly, we set 𝑾 ( 𝓁 ) ∈ ℝ 𝐶 × 𝑐 𝓁 and further input 𝒉 𝑇 into the softmax function to obtain the predicted class. For emotion regression tasks, the brain tree is associated with an emotion rating vector 𝒚 𝑇 ∈ [0 , 𝑎 ] 𝐶 containing 𝐶 emotion categories, where 𝑎 represents the maximum rating and is usually set to 100. Accordingly, we set 𝑾 ( 𝓁 ) ∈ ℝ 𝐶 × 𝑐 𝓁 and further input 𝒉 𝑇 into the logistic function to obtain the predicted ratings. The above representation learning process is illustrated in Appendix C through a concrete example.

## 5. Experiments

In this section, we conduct comprehensive experiments on two large-scale datasets, both containing audio-visual multi-source naturalistic stimuli. The entire experimental evaluation consists of two stages. In the first stage (Section 5.3), we identify hierarchical emotional areas based on the overall experience of all emotions in Dataset 1 [27]. In the second stage (Section 5.4), we use the proposed model, HEmoN, which builds on the identified hierarchical emotional areas, to perform cross-dataset emotion decoding on Dataset 2 [11]. Furthermore, we provide a thorough analysis and discussion of the experimental results in Section 6. Our code is publicly available at https://github.com/ zhongyu1998/HEmoN .

## 5.1. Dataset description

We conduct comprehensive experiments on two largescale datasets [11, 27] to demonstrate the effectiveness and superiority of our proposed approach. Both datasets contain audio-visual multi-source naturalistic stimuli and use a similar experimental paradigm for data acquisition. Specifically, they collect fMRI responses when subjects watch emotionally charged movies (i.e., emotional stimuli). To obtain the corresponding emotion ratings for these stimuli, they recruit multiple human raters who are not involved in the fMRI experiments. These raters label each emotional stimulus according to specific emotion categories, assigning a score to each emotion category for every stimulus. The final emotion rating for a specific emotion category of a stimulus is

## Identifying the Hierarchical Emotional Areas in the Human Brain Through Information Fusion

Figure 2: The illustration of the identified hierarchical emotional areas, including (a) all basic emotions, (b) happiness, and (c) sadness. The legend at the top lists all 13 functional systems and one uncertain system, as proposed by the Power Atlas [40]. Each subfigure shows the corresponding brain tree and presents the emotional areas at the first three levels. The nodes are colored according to the legend, and the internal connections within the 1st-level, 2nd-level, and 3rd-level emotional areas are highlighted in green, blue, and red, respectively.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00525v1-figure-c3c6838e-ca09-4097-aa8c-d690460e3351.jpg)

determined by averaging the scores across these raters. As for fMRI responses, we use the fMRI data preprocessed by Lettieri et al. [27] and Koide-Majima et al. [11]. Here, we provide an overview of the basic information and fMRI data preprocessing steps for each dataset. Further details on emotion ratings, fMRI data acquisition, and fMRI data preprocessing are available in their original literature.

Dataset 1 This dataset contains fMRI responses collected from 15 subjects when they experience emotional stimuli, featuring 3599 pairs of {fMRI responses, emotion ratings} for each subject. The emotional stimuli are composed of movie segments from Forrest Gump, sourced from the StudyForrest project ( http://studyforrest.org ), and contain six basic emotions [51]: happiness, surprise, fear, sadness, anger, and disgust. The fMRI data are preprocessed using the Advanced Normalization Tools (ANTs) [52] and the Analysis of Functional NeuroImages (AFNI) [53]. This process mainly involves the following steps: spike correction, slice timing correction, motion correction, rigid body transformation, registration to T1-weighted images, phase distortion correction, spatial smoothing with a Gaussian kernel, and physiological noise filtering.

Dataset 2 This dataset contains fMRI responses collected from 8 subjects when they experience emotional stimuli, featuring 5400 pairs of {fMRI responses, emotion ratings} for each subject. The emotional stimuli are composed of movie clips sourced from the video-sharing site Vimeo ( https://vimeo.com/jp ) and contain 80 emotion categories. The complete list of these 80 emotion categories and their sources are available in the original literature [11]. The fMRI data are preprocessed using the Statistical Parameter Mapping toolbox (SPM8, https://www.fil.ion.ucl.ac. uk/spm/software/spm8/ ). This process mainly involves the following steps: motion correction, z-score normalization, median filter convolution, long-term trend removal, and registration to T1-weighted images.

## 5.2. Implementation details

Our model is implemented with Python version 3.6, PyTorch version 1.4.0 [57], and PyTorch Geometric (PyG) version 1.7.0 [58]. All experiments are conducted on a Linux

We initialize the learnable weighting matrix in Eq. (3) and all model weights using the Xavier initializer [54]. Then, we pre-map the emotional stimuli into 64 dimensions with a linear transformation and use a batch of 32 emotional stimuli as the model input. Our HEmoN model consists of 3 LSTM layers (Eq. (2)) and a final combination layer (Eq. (3)). Each LSTM layer contains 64 hidden units, with a dropout [55] ratio of 0.2 applied to the outputs of each LSTM layer except the last one. The model is trained using the Adam optimizer [56] with an initial learning rate of 0.0005. If the performance on the validation set does not improve after 10 epochs, the learning rate is reduced by a factor of 0.5. The training is stopped when the learning rate reaches the minimum value of 2E-5 or when the number of epochs reaches the maximum value of 300.

server equipped with an Intel ® Xeon ® CPU E5-2650 v4 and 8 NVIDIA TITAN RTX GPUs.

## 5.3. Identification of hierarchical emotional areas

To identify the hierarchical emotional areas, we conduct experiments on Dataset 1 and use the Power Atlas [40] to define ROIs. The Power Atlas comprises 264 putative functional areas (i.e., ROIs) distributed across 13 functional systems and one uncertain system. All the systems are listed at the top of Figure 2. The fMRI time series for each ROI is obtained by averaging the activity of voxels within a fixedradius sphere of 5mm, centered on the ROI coordinates defined by the Power Atlas. Subsequently, we construct the brain network, extract the brain tree, and identify hierarchical emotional areas according to the methodologies described in Sections 4.1 and 4.3. We present the identification results for the overall experience of all basic emotions, as well as those for the single experiences of the most representative positive and negative emotions: happiness and sadness. The identified hierarchical emotional areas for these three types are illustrated in Figure 2.

## 5.4. Performance of emotion decoding

Toperform cross-dataset emotion decoding, we build the HEmoNmodelbyexploiting the identified hierarchical emotional areas and evaluate its performance on Dataset 2. Let 𝒚 𝑖 and ̂ 𝒚 𝑖 denote the true and predicted emotion rating vectors for the 𝑖 th stimulus, respectively, where ̂ 𝒚 𝑖 is calculated according to the steps described in Section 4.4. Additionally, let the scalars 𝑦 𝑖𝑗 and ̂ 𝑦 𝑖𝑗 denote the true and predicted ratings for the 𝑗 th emotion category in the 𝑖 th stimulus, respectively. We take the Mean Absolute Error (MAE), which is widely used in regression tasks, as our evaluation metric. The MAE value 𝛼 is calculated as:

$$\alpha = \frac { 1 } { N _ { s } } \sum _ { i = 1 } ^ { N _ { s } } \sum _ { j = 1 } ^ { C } \left | y _ { i j } - \hat { y } _ { i j } \right | \text{ \quad \ \ } \text{(4)}$$

where 𝑁𝑠 denotes the number of stimuli. Since most emotion categories have ratings close to or containing zeros, Eq. (4) performs the sum operation ∑ 𝐶 𝑗 =1 instead of the average operation 1 𝐶 ∑ 𝐶 𝑗 =1 across all 𝐶 emotion categories.

Since Dataset 2 has already been split into training and test sets, we adopt the default split and perform emotion decoding between the fMRI responses to each emotional stimulus and its corresponding emotion ratings. Specifically, in each experimental run, we use the training set consisting of 3600 pairs to train the model and then evaluate its performance on the test set consisting of 1800 pairs. We repeat this process 10 times and report the evaluation result based on the test MAEs of these 10 runs. Furthermore, we compare HEmoN with multiple competitive baselines, including Fully-connected Neural Network (FNN), Graph Convolutional Network (GCN) [59], Graph Isomorphism Network (GIN) [60], BrainNetCNN [61], BrainGNN [34], Brain Network Transformer (BrainNetTF) [62], and Graphenhanced Emotion Decoding (GED) [38]. The evaluation results on Dataset 2 are summarized in Figure 3.

Figure 3: Emotion decoding results (measured by MAE, lower is better) on Dataset 2. The black error bars represent the standard deviation.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00525v1-figure-62a9bcb7-572c-4e57-ba58-1e419729524f.jpg)

## 5.5. Ablation study

The hierarchical emotional areas and the information fusion rule of brain regions play pivotal roles in our HEmoN model. Therefore, we conduct ablation studies to investigate their impacts on model performance. Specifically, we replace the hierarchical emotional areas with a single 1st-level emotional area to assess the impact of hierarchical emotional areas. In addition, we perform a depth-first traversal on the brain tree and fuse information among brain regions based on the traversal result instead of hierarchical emotional areas to assess the impact of the information fusion rule. The corresponding results are shown in Figure 3, where the suffixes '-EA1' and '-DFT' represent the 1st-level emotional area and the depth-first traversal, respectively.

## 6. Discussion

Figure 2a reveals that for the overall experience of all basic emotions, the 1st-level emotional area mainly includes sensory/somatomotor, auditory, default mode, visual, and subcortical systems. These functional systems involve various senses, including sight (visual), sound (auditory), and touch (sensory/somatomotor), enabling individuals to perceive external emotional stimuli and thus creating necessary conditions for the construction of core affect. In addition, a small portion of this area includes salience and ventral attention systems, which also contribute to the perception of external stimuli and the construction of core affect. Moreover, the default mode and subcortical systems play key roles in distinguishing experiences of discrete emotions [6, 7, 10, 12, 63], thereby promoting the recognition, identification, and perception of emotions. In summary, the 1st-level emotional area primarily facilitates the fundamental process of emotion perception. The 2nd-level emotional area mainly includes sensory/somatomotor, cingulo-opercular task control, auditory, default mode, visual, fronto-parietal task control, salience, subcortical, and cerebellar systems. This emotional area primarily facilitates the construction of basic psychological operations. For instance, the anterior cingulate cortex in the cingulo-opercular task control system and the amygdala in the subcortical system both play pivotal roles in core affect [4, 64, 65]. In addition, a small portion of this area includes memory retrieval, ventral attention, and dorsal attention systems, which contribute to the construction of conceptualization and executive attention. Moreover, Broca's and Wernicke's areas contained in this area are essential for the production and comprehension of human language [66-68]. It is worth noting that our finding of the 2nd-level emotional area is consistent with a previous study [8], which identified a general emotion pathway connecting neural nodes

involved in these basic psychological operations. The 3rdlevel emotional area mainly includes sensory/somatomotor, cingulo-opercular task control, default mode, visual, frontoparietal task control, salience, and dorsal attention systems. These functional systems typically participate in advanced cognitive processes, thereby facilitating the coordination and integration of the above basic psychological operations.

Figure 2c reveals that for a single experience of sadness, the 1st-level emotional area mainly includes auditory, default mode, memory retrieval, visual, fronto-parietal task control, ventral attention, and dorsal attention systems. These functional systems differ significantly from those found in all basic emotions. Compared with the basic emotions, these functional systems incorporate memory retrieval, fronto-parietal task control, ventral attention, and dorsal attention systems, while excluding sensory/somatomotor and subcortical systems. This distinction may stem from the fact that sadness can impair our attention, memory, and decision-making abilities [72, 73]. As a result, the above functional systems are the first to be affected when perceiving sadness. The 2nd-level emotional area mainly includes sensory/somatomotor, cingulo-opercular task control, auditory, default mode, visual, fronto-parietal task control, salience, and subcortical systems. The activation of sadness in the brain is associated with multiple brain regions within these functional systems, such as the anterior cingulate cortex, amygdala, and hippocampus in cingulo-opercular task control and subcortical systems [74, 75]. The 3rd-level emotional area mainly includes sensory/somatomotor, cingulo-opercular task control, auditory, default mode, visual, fronto-parietal task control, salience,

Figure 2b reveals that for a single experience of happiness, the 1st-level emotional area mainly includes sensory/somatomotor, default mode, visual, and subcortical systems. In addition, a small portion of this area includes salience and ventral attention systems. These functional systems are similar to those found in all basic emotions, with the exception of the auditory system. A possible explanation for the absence of the auditory system is that visual information is more dominant than auditory information in the perception of happiness. This conjecture is supported by an interesting study [69], which demonstrates that happiness is more easily perceived visually than auditorily. The 2ndlevel emotional area mainly includes sensory/somatomotor, cingulo-opercular task control, default mode, visual, frontoparietal task control, salience, subcortical, ventral attention, and cerebellar systems. In addition, a small portion of this area includes auditory, memory retrieval, and dorsal attention systems. The activation of happiness in the brain is associated with multiple brain regions within these functional systems, such as the anterior cingulate cortex, prefrontal cortex, and insula in cingulo-opercular task control and salience systems [70, 71]. The 3rd-level emotional area mainly includes sensory/somatomotor, auditory, default mode, visual, fronto-parietal task control, salience, ventral attention, and dorsal attention systems.

subcortical, ventral attention, dorsal attention, and cerebellar systems.

Overall, the experimental evaluation of cross-dataset emotion decoding demonstrates the significant potential and strong generalization capability of the identified hierarchical emotional areas. However, it is worth noting the individual differences in decoding performance among various subjects. Investigating and addressing these individual differences constitutes an important topic for future research.

Furthermore, the emotion decoding results in Figure 3 show that our proposed model, HEmoN, consistently outperforms advanced models such as BrainNetTF and GED, achieving outstanding performance in all cases. Competitive baselines, including BrainNetCNN, BrainGNN, and BrainNetTF, utilize convolutional neural networks, graph neural networks, and transformers to analyze brain networks, respectively. However, they all neglect the information fusion among multiple brain regions. Therefore, the superior performance of HEmoN in emotion decoding tasks highlights the importance of integrating information from multiple brain regions. In addition, as expected, we observe an obvious performance degradation in each ablation of HEmoNEA1 and HEmoN-DFT, demonstrating the effectiveness of both the hierarchical emotional areas and the proposed information fusion rule. Interestingly, HEmoN-EA1 is inferior to HEmoN-DFT in all cases, further showing the potential of the identified hierarchical emotional areas.

## 7. Conclusion

In this paper, we investigate the hierarchical emotional areas in the human brain through multi-source information fusion and graph machine learning methods. Furthermore, we provide preliminary insights into the brain mechanisms underlying specific emotions based on the psychological constructionist hypothesis. To gain a deeper understanding of the roles of emotional areas at various levels in emotion encoding, it is promising to conduct meta-analyses using a large number of existing literature in future research.

## CRediT authorship contribution statement

Zhongyu Huang: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Visualization, Writing - original draft. Changde Du: Funding acquisition, Supervision, Validation, Writing - review &amp; editing. Chaozhuo Li: Investigation, Methodology, Writing - review &amp; editing. Kaicheng Fu: Conceptualization, Data curation. Huiguang He: Funding acquisition, Project administration, Supervision, Writing - review &amp; editing.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Data availability

The data are publicly available on the Open Science Framework (OSF) at https://osf.io/tzpdf .

## Acknowledgments

The authors would like to express their sincere gratitude to Prof. Dan Zhang for his helpful discussions and valuable suggestions, which greatly contributed to improving the overall quality of this paper. They also extend their thanks to Giada Lettieri, Naoko Koide-Majima, and Shinji Nishimoto for providing the essential data for this research.

## A. Proof of Theorem 1

Proof. According to the propagation rule, we have

$$h _ { u } ^ { ( k ) } = \sum _ { w \in \mathcal { N } ( u ) } \left ( \frac { a _ { u w } } { \sum _ { i \in \mathcal { N } ( u ) } a _ { u i } } h _ { w } ^ { ( k - 1 ) } \right ) \quad ( A. 1 ) \quad \text{In a}$$

where 𝑎 𝑢𝑤 is the edge weight between nodes 𝑢 and 𝑤 , and  ( 𝑢 ) is the set of 𝑢 's 1-hop neighbors.

After 𝑘 iterations of expansion as in Eq. (A.1), 𝒉 ( 𝑘 ) 𝑢 can be represented as

$$h _ { u } ^ { ( k ) } & = \frac { 1 } { \sum _ { i \in \mathcal { N } ( u ) } a _ { u i } } \sum _ { w \in \mathcal { N } ( u ) } a _ { u w } \cdot \frac { 1 } { \sum _ { j \in \mathcal { N } ( w ) } a _ { w j } } \sum _ { x \in \mathcal { N } ( w ) } a _ { w x } \cdots \\ & \quad \frac { 1 } { \sum _ { l \in \mathcal { N } ( y ) } a _ { y l } } \sum _ { z \in \mathcal { N } ( y ) } a _ { y z } h _ { z } ^ { ( 0 ) } & & ( \text{A.2} )$$

Abbreviate the node influence 𝐼 𝑇 ( 𝑢, 𝑣 ) as 𝐼 . By substituting Eq. (A.2) into the definition of node influence, we obtain

$$& \overline { \sim } _ { i \in \mathcal { N } ( y ) } \varphi _ { i } ^ { \prime } z \in \mathcal { N } ( y ) \\ & \text{Abbreviate the node influence I} _ { T } ( u, v ) as I. By substitut-
ing Eq. ( A. 2 ) \, \text{into the definition of node influence, we obtain} \\ & I = \left \| \frac { \partial h ^ { ( k ) } _ { u } } { \partial h ^ { ( 0 ) } _ { v } } \right \| _ { \substack { \sim } \\ \sim } & \left \| \frac { \partial } { \partial h ^ { ( 0 ) } _ { v } } \left ( \frac { 1 } { \sum _ { i \in \mathcal { N } ( u ) } a _ { u } } \, \sum _ { w \in \mathcal { N } ( u ) } a _ { u w } \cdot \frac { 1 } { \sum _ { j \in \mathcal { N } ( w ) } a _ { w j } } \sum _ { x \in \mathcal { N } ( w ) } a _ { w x } \cdots \\ & \frac { 1 } { \sum _ { l \in \mathcal { N } ( y ) } a _ { y l } } \sum _ { x \in \mathcal { N } ( y ) } a _ { y z } h ^ { ( 0 ) } _ { z } \right ) \right \| _ { \substack { \sim } \\ \sim } & \left \| \frac { \partial } { \sum _ { l \in \mathcal { N } ( y ) } a _ { u } } \, \sum _ { x \in \mathcal { N } ( y ) } a _ { y z } h ^ { ( 0 ) } _ { z } \right ) \right \| _ { \substack { \sim } \\ \sim } & \left \| \frac { 1 } { \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } } a _ { v _ { k - 1 } v _ { k } } h ^ { ( 0 ) } _ { v } \right ) \right \| _ { \substack { ( A. 3 ) } \, \text{Since} \\ \sim } & \left \| \frac { \partial h ^ { ( 0 ) } _ { v } } { \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { u l } } \, \frac { \partial _ { v _ { l } v _ { 2 } } } { \sum _ { j \in \mathcal { N } ( v _ { 1 } ) } a _ { v _ { 1 } j } } \sum _ { x \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } \right | \\ & \left \| \frac { \partial } { \sum _ { l \in \mathcal { N } ( u ) } a _ { u } } \, \sum _ { j \in \mathcal { N } ( v _ { 1 } ) } a _ { v _ { 1 } j } \cdots \frac { \partial _ { v _ { k - 1 } v } } { \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } \right | \\ & \left \| \frac { \partial } { \sum _ { l \in \mathcal { N } ( u ) } a _ { u } } \cdot \frac { \partial _ { v _ { l } v _ { 2 } } } { \sum _ { j \in \mathcal { N } ( v _ { 1 } ) } a _ { v _ { 1 } j } \cdots \frac { \partial } { \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } \right | } } } \left \| _ { \substack { ( A. 4 ) } \, \text{Eq. (B.:} \\ \sum _ { i \in \mathcal { N } ( u ) } a _ { u } } \cdot \frac { \partial _ { v _ { l } v _ { 2 } } } { \sum _ { j \in \mathcal { N } ( v _ { 1 } ) } a _ { v _ { 1 } j } \cdots \frac { \partial } { \sum _ { l \in \mathcal { N } ( v _ { k - 1 } ) } a _ { v _ { k - 1 } l } \right | } } } \left \| _ { \substack { ( A. 4 ) } \, \text{Eq. (B.:} \\ \end{Eq. (B.:} }$$

According to the properties of trees, there exists exactly one path between nodes 𝑢 and 𝑣 ; we denote this path as 𝑃 = ( 𝑢, 𝑣 1 , 𝑣 2 , ⋯ , 𝑣 𝑘 -1 , 𝑣 ) . Eq. (A.3) holds because the partial derivative of nodes that are not in 𝑃 becomes 0, leading to their elimination. Eq. (A.4) holds because the Jacobian matrix 𝜕 𝒉 (0) 𝑣 ∕ 𝜕 𝒉 (0) 𝑣 is the identity matrix 𝐼 𝑐 0 , and its induced norm ‖ 𝑰 𝑐 0 ‖ equals 1.

## B. Proof of Theorem 2

Proof. Without loss of generality, we consider an arbitrary path 𝑃 of length 𝑚 with vertex sequence ( 𝑣 , 𝑣 0 1 , … , 𝑣 𝑚 ) . According to Eq. (1), the node influence 𝐼 𝑃 ( 𝑣 𝑖 , 𝑣 𝑗 ) of any pair of nodes 𝑣 𝑖 and 𝑣 𝑗 with a distance of 𝑘 in path 𝑃 is

$$I _ { P } ( v _ { i }, v _ { j } ) = \frac { 1 } { 2 ^ { k - 1 } }, \text{ s.t. } | i - j | = k \text{ \quad (B.1)}$$

In addition, there are a total of ( 𝑚 -𝑘 + 1) pairs of nodes with a distance of 𝑘 ( 1 ≤ 𝑘 ≤ 𝑚 ) in path 𝑃 . By using the definition of path information and Eq. (B.1), we obtain

$$\iota \, \text{of path information and $Eq.$} (B.1), \, \text{we ot} \\ I ( P ) & = \sum _ { \substack { i, j = 0 \\ i \neq j } } ^ { m } I _ { P } ( v _ { i }, v _ { j } ) \\ & = 2 \cdot \sum _ { k = 1 } ^ { m } \sum _ { i = 0 } ^ { m - k } I _ { P } ( v _ { i }, v _ { i + k } ) \\ & = \sum _ { k = 1 } ^ { m } ( m - k + 1 ) \cdot \frac { 2 } { 2 ^ { k - 1 } } \\ \text{ding a path with the maximum path information}$$

Thus, finding a path with the maximum path information in tree 𝑇 is equivalent to solving the following optimization problem:

𝑎 𝑤𝑥

⋯

$$\max _ { m } & \quad I ( P ) = \sum _ { k = 1 } ^ { m } \left ( m - k + 1 \right ) \cdot \frac { 2 } { 2 ^ { k - 1 } } \quad \text{(B.2)} \\.. \quad \text{ s.t. } \quad 1 \leq m \leq \mathfrak { d } \\ & \quad m \in \mathbb { Z } ^ { + }$$

where 𝔡 is the diameter of tree 𝑇 .

Consider a finite sequence ( 𝑥 𝑚 ) 𝔡 𝑚 =1 that corresponds to the objective function Eq. (B.2), where

$$x _ { m } = \sum _ { k = 1 } ^ { m } \left ( m - k + 1 \right ) \cdot \frac { 2 } { 2 ^ { k - 1 } }$$

Since

$$x _ { m + 1 } - x _ { m } = \sum _ { k = 1 } ^ { m + 1 } \frac { 2 } { 2 ^ { k - 1 } } > 0, \ \forall m \in \mathbb { Z } ^ { + }$$

We can conclude that sequence ( 𝑥 𝑚 ) 𝔡 𝑚 =1 is strictly increasing. Accordingly, the optimization problem described in Eq. (B.2) is equivalent to the following one:

$$\max _ { m }$$

## Identifying the 1st-level trunk A brain tree Identifying the 2nd-level trunks

Figure 4: A brain tree (left, adapted from Figure 1) and the process of identifying its hierarchical trunks (middle and right).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00525v1-figure-729e4600-3dde-43b0-a6b3-3ed87db0aa3d.jpg)

$$\text{s.t.} \quad 1 \leq m \leq \mathfrak { d } \\ m \in \mathbb { Z } ^ { + }$$

Clearly, the optimal solution is ̂ 𝑚 = 𝔡 , which corresponds to the longest shortest path ̂ 𝑃 . In conclusion, the longest shortest path ̂ 𝑃 has the maximum path information 𝐼 𝑃 ̂ ( ) = ∑ 𝔡 𝑘 =1 ( 𝔡 -𝑘 +1) ⋅ 2 2 𝑘 -1 .

After obtaining all the trunks at each level, we learn the trunk representation 𝒉 ( 𝓁 ) 𝑇 for each level 𝓁 , which also serves as the representation of the emotional area at the 𝓁 th level. According to Eq. (2), we have

## C. Illustration of the HEmoN model

In this section, we illustrate the process of identifying hierarchical trunks and the representation learning process of our proposed HEmoN model through a concrete example, as shown in Figure 4. Let 𝑇 = (   , ) denote the brain tree in Figure 4, where

$$\mathcal { V } & = \{ \, v _ { 0 }, v _ { 1 }, \dots, v _ { 6 }, u _ { 0 }, u _ { 1 }, u _ { 2 } \ \} \,, \\ \mathcal { R } & = \{ \, ( v _ { 0 }, v _ { 1 } ), ( v _ { 1 }, v _ { 2 } ), \dots, ( v _ { 5 }, v _ { 6 } ), \\ & \quad ( u _ { 1 }, u _ { 2 } ), ( v _ { 2 }, u _ { 0 } ), ( v _ { 3 }, u _ { 1 } ) \ \} \,.$$

According to the methodology described in Section 4.3, we designate the longest shortest path as the main trunk 𝔱 1 . As a result, we obtain the set of 1st-level trunk  1 = { 𝔱 1 } . Then, we remove  1 from 𝑇 , and the remaining part becomes a forest 𝐹 1 . This forest contains two connected components: connected component 1 and connected component 2, with each component being a (sub)tree. To identify the trunks at the second level, we search for the longest shortest path in each connected component of the forest. Accordingly, we obtain the 2nd-level trunk 𝔱 2 1 , in connected component 1, the 2nd-level trunk 𝔱 2 2 , in connected component 2, and thus the set of 2nd-level trunks  2 = { 𝔱 2 1 , , 𝔱 2 2 , } .

Since there are no remaining nodes after removing  2 from 𝐹 1 , we stop the process of identifying hierarchical trunks and decompose 𝑇 into two levels of trunks, i.e.,  1 = { 𝔱 1 } and  2 = { 𝔱 2 1 , , 𝔱 2 2 , } .

$$\text{rung to eq. (2), we have} \\ h _ { T } ^ { ( 1 ) } & = LSTM \left ( x _ { v _ { 0 } } ^ { ( t _ { 1 } ) }, x _ { v _ { 1 } } ^ { ( t _ { 1 } ) }, \cdots, x _ { v _ { 6 } } ^ { ( t _ { 1 } ) } \right ) \\ h _ { T } ^ { ( 2 ) } & = LSTM \left ( x _ { u _ { 0 } } ^ { ( t _ { 2, 1 } ) }, x _ { v _ { 2 } } ^ { ( t _ { 2, 1 } ) } \right ) + \\ & \quad LSTM \left ( x _ { v _ { 3 } } ^ { ( t _ { 2, 2 } ) }, x _ { u _ { 1 } } ^ { ( t _ { 2, 2 } ) }, x _ { u _ { 2 } } ^ { ( t _ { 2, 2 } ) } \right ) \\ \cdot \ \. \ \. \quad \. \ \. \ \. \ \. \ \. \ \. \. \.$$

Finally, according to Eq. (3), we combine both the trunk representations to create a representation of the brain tree 𝑇 :

$$h _ { T } = \sum _ { \ell = 1 } ^ { 2 } W ^ { ( \ell ) } h _ { T } ^ { ( \ell ) } = W ^ { ( 1 ) } h _ { T } ^ { ( 1 ) } + W ^ { ( 2 ) } h _ { T } ^ { ( 2 ) }.$$

## References

- [2] S. K. Khare, V. Blanes-Vidal, E. S. Nadimi, U. R. Acharya, Emotion recognition and artificial intelligence: A systematic review (20142023) and research recommendations, Information Fusion 102 (2024) 102019.
- [1] Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, W. Zhang, et al., A systematic review on affective computing: Emotion models, databases, and recent advances, Information Fusion 83 (2022) 19-52.
- [3] A. Geetha, T. Mala, D. Priyanka, E. Uma, Multimodal emotion recognition with deep learning: Advancements, challenges, and future directions, Information Fusion 105 (2024) 102218.
- [5] K. S. Kassam, A. R. Markey, V. L. Cherkassky, G. Loewenstein, M. A. Just, Identifying emotions on the basis of neural activation, PLOS One 8 (2013) e66032.
- [4] K. A. Lindquist, T. D. Wager, H. Kober, E. Bliss-Moreau, L. F. Barrett, The brain basis of emotion: A meta-analytic review, Behavioral and Brain Sciences 35 (2012) 121-143.
- [6] T. D. Wager, J. Kang, T. D. Johnson, T. E. Nichols, A. B. Satpute, L. F. Barrett, A Bayesian model of category-specific emotional brain responses, PLOS Computational Biology 11 (2015) e1004066.

- [7] H. Saarimäki, A. Gotsopoulos, I. P. Jääskeläinen, J. Lampinen, P. Vuilleumier, R. Hari, M. Sams, L. Nummenmaa, Discrete neural signatures of basic emotions, Cerebral Cortex 26 (2016) 2563-2573.
- [9] H. Saarimäki, L. F. Ejtehadian, E. Glerean, I. P. Jääskeläinen, P. Vuilleumier, M. Sams, L. Nummenmaa, Distributed affective space represents multiple emotion categories across the human brain, Social Cognitive and Affective Neuroscience 13 (2018) 471-482.
- [8] Y.-A. Huang, J. Jastorff, J. Van den Stock, L. Van de Vliet, P. Dupont, M. Vandenbulcke, Studying emotion theories through connectivity analysis: Evidence from generalized psychophysiological interactions and graph theory, NeuroImage 172 (2018) 250-262.
- [10] T. Horikawa, A. S. Cowen, D. Keltner, Y. Kamitani, The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions, iScience 23 (2020) 101060.
- [12] H. Saarimäki, E. Glerean, D. Smirnov, H. Mynttinen, I. P. Jääskeläinen, M. Sams, L. Nummenmaa, Classification of emotion categories based on functional connectivity patterns of the human brain, NeuroImage 247 (2022) 118800.
- [11] N. Koide-Majima, T. Nakai, S. Nishimoto, Distinct dimensions of emotion in the human brain and their representation on the cortical surface, NeuroImage 222 (2020) 117258.
- [13] S. Xu, Z. Zhang, L. Li, Y. Zhou, D. Lin, M. Zhang, L. Zhang, G. Huang, X. Liu, B. Becker, et al., Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience, NeuroImage 269 (2023) 119941.
- [15] K. M. Heilman, Emotion and the brain: A distributed modular network mediating emotional experience, in: D. W. Zaidel (Ed.), Neuropsychology, Academic Press, San Diego, 1994, pp. 139-158. doi: 10.1016/B978-0-08-092668-1.50013-2 .
- [14] P. E. Griffiths, Modularity, and the psychoevolutionary theory of emotion, Biology and Philosophy 5 (1990) 175-196.
- [16] D. Meunier, R. Lambiotte, A. Fornito, K. Ersche, E. T. Bullmore, Hierarchical modularity in human brain functional networks, Frontiers in Neuroinformatics 3 (2009) 37.
- [18] T. Naselaris, K. N. Kay, S. Nishimoto, J. L. Gallant, Encoding and decoding in fMRI, NeuroImage 56 (2011) 400-410.
- [17] D. Meunier, R. Lambiotte, E. T. Bullmore, Modular and hierarchically modular organization of brain networks, Frontiers in Neuroscience 4 (2010) 200.
- [19] T. Nguyen, T. Zhou, T. Potter, L. Zou, Y. Zhang, The cortical network of emotion regulation: Insights from advanced EEG-fMRI integration analysis, IEEE Transactions on Medical Imaging 38 (2019) 24232433.
- [21] P. Vuilleumier, J. L. Armony, J. Driver, R. J. Dolan, Effects of attention and emotion on face processing in the human brain: An event-related fMRI study, Neuron 30 (2001) 829-841.
- [20] S. Gu, F. Wang, C. Cao, E. Wu, Y.-Y. Tang, J. H. Huang, An integrative way for studying neural basis of basic emotions with fMRI, Frontiers in Neuroscience 13 (2019) 628.
- [22] W. Sato, T. Kochiyama, S. Yoshikawa, E. Naito, M. Matsumura, Enhanced neural activity in response to dynamic facial expressions of emotion: An fMRI study, Cognitive Brain Research 20 (2004) 81-91.
- [24] S. Sonkusare, M. Breakspear, C. Guo, Naturalistic stimuli in neuroscience: Critically acclaimed, Trends in Cognitive Sciences 23 (2019) 699-714.
- [23] E. A. Holmes, A. Mathews, Mental imagery in emotion and emotional disorders, Clinical Psychology Review 30 (2010) 349-362.
- [25] D. Smirnov, H. Saarimäki, E. Glerean, R. Hari, M. Sams, L. Nummenmaa, Emotions amplify speaker-listener neural alignment, Human Brain Mapping 40 (2019) 4777-4788.
- [26] M. Hanke, N. Adelhöfer, D. Kottke, V. Iacovella, A. Sengupta, F. R. Kaule, R. Nigbur, A. Q. Waite, F. Baumgartner, J. Stadler, A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation, Scientific Data 3 (2016) 160092.
- [27] G. Lettieri, G. Handjaras, E. Ricciardi, A. Leo, P. Papale, M. Betta, P. Pietrini, L. Cecchetti, Emotionotopy in the human right temporoparietal cortex, Nature Communications 10 (2019) 5568.
- [29] M. Bernstein-Eliav, I. Tavor, The prediction of brain activity from connectivity: Advances and applications, The Neuroscientist (2022) 10738584221130974.
- [28] H. Saarimäki, Naturalistic stimuli in affective neuroimaging: A review, Frontiers in Human Neuroscience 15 (2021) 675068.
- [30] K. Ezzameli, H. Mahersia, Emotion recognition from unimodal to multimodal analysis: A review, Information Fusion 99 (2023) 101847.
- [32] S.-Y. Lin, C.-C. Lee, Y.-S. Chen, L.-W. Kuo, Investigation of functional brain network reconfiguration during vocal emotional processing using graph-theoretical analysis, Social Cognitive and Affective Neuroscience 14 (2019) 529-538.
- [31] S. Koelsch, S. Skouras, Functional centrality of amygdala, striatum and hypothalamus in a 'small-world' network underlying joy: An fMRI study with music, Human Brain Mapping 35 (2014) 34853498.
- [33] S. Ghahari, N. Farahani, E. Fatemizadeh, A. Motie Nasrabadi, Investigating time-varying functional connectivity derived from the Jackknife Correlation method for distinguishing between emotions in fMRI data, Cognitive Neurodynamics 14 (2020) 457-471.
- [35] Y. Li, C. Li, L. Jiang, Well-being is associated with cortical thickness network topology of human brain, Behavioral and Brain Functions 19 (2023) 16.
- [34] X. Li, Y. Zhou, N. Dvornek, M. Zhang, S. Gao, J. Zhuang, D. Scheinost, L. H. Staib, P. Ventola, J. S. Duncan, BrainGNN: Interpretable brain graph neural network for fMRI analysis, Medical Image Analysis 74 (2021) 102233.
- [36] C. Liu, Y. Wang, X. Sun, Y. Wang, F. Fang, Decoding six basic emotions from brain functional connectivity patterns, Science China Life Sciences 66 (2023) 835-847.
- [38] Z. Huang, C. Du, Y. Wang, K. Fu, H. He, Graph-enhanced emotion neural decoding, IEEE Transactions on Medical Imaging 42 (2023) 2262-2273.
- [37] E. Bullmore, O. Sporns, Complex brain networks: Graph theoretical analysis of structural and functional systems, Nature Reviews Neuroscience 10 (2009) 186-198.
- [39] G. Lettieri, G. Handjaras, E. M. Cappello, F. Setti, D. Bottari, V. Bruno, M. Diano, A. Leo, C. Tinti, F. Garbarini, et al., Dissecting abstract, modality-specific and experience-dependent coding of affect in the human brain, Science Advances 10 (2024) eadk6840.
- [41] C. Destrieux, B. Fischl, A. Dale, E. Halgren, Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature, NeuroImage 53 (2010) 1-15.
- [40] J. D. Power, A. L. Cohen, S. M. Nelson, G. S. Wig, K. A. Barnes, J. A. Church, A. C. Vogel, T. O. Laumann, F. M. Miezin, B. L. Schlaggar, et al., Functional network organization of the human brain, Neuron 72 (2011) 665-678.
- [42] A. Fornito, A. Zalesky, E. T. Bullmore, Network scaling effects in graph analytic studies of human resting-state fMRI data, Frontiers in Systems Neuroscience 4 (2010) 22.
- [44] P. Tewarie, E. van Dellen, A. Hillebrand, C. J. Stam, The minimum spanning tree: An unbiased method for brain network analysis, NeuroImage 104 (2015) 177-188.
- [43] C. Stam, P. Tewarie, E. Van Dellen, E. Van Straaten, A. Hillebrand, P. Van Mieghem, The trees and the forest: Characterization of complex brain networks with minimum spanning trees, International Journal of Psychophysiology 92 (2014) 129-138.
- [45] E. van Dellen, I. E. Sommer, M. M. Bohlken, P. Tewarie, L. Draaisma, A. Zalesky, M. Di Biase, J. A. Brown, L. Douw, W. M. Otte, et al., Minimum spanning tree analysis of the human connectome, Human Brain Mapping 39 (2018) 2455-2471.
- [47] K. Xu, C. Li, Y. Tian, T. Sonobe, K.-i. Kawarabayashi, S. Jegelka, Representation learning on graphs with jumping knowledge networks,
- [46] R. Sonthalia, A. Gilbert, Tree! I am no tree! I am a low dimensional hyperbolic embedding, in: Advances in Neural Information Processing Systems, 2020, pp. 845-856.

- in: Proceedings of the 35th International Conference on Machine Learning, 2018, pp. 5453-5462.
- [49] D. B. West, Introduction to graph theory, Prentice Hall, Upper Saddle River, 2001.
- [48] K. Huang, M. Zitnik, Graph meta learning via local subgraphs, in: Advances in Neural Information Processing Systems, 2020, pp. 58625874.
- [50] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation 9 (1997) 1735-1780.
- [52] B. B. Avants, N. Tustison, G. Song, Advanced normalization tools (ANTS), The Insight Journal (2009) 1-29.
- [51] P. Ekman, An argument for basic emotions, Cognition and Emotion 6 (1992) 169-200.
- [53] R. W. Cox, AFNI: Software for analysis and visualization of functional magnetic resonance neuroimages, Computers and Biomedical Research 29 (1996) 162-173.
- [55] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (2014) 19291958.
- [54] X. Glorot, Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010, pp. 249256.
- [56] D. P. Kingma, J. L. Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Representations, 2015.
- [58] M. Fey, J. E. Lenssen, Fast graph representation learning with PyTorch Geometric, in: ICLR 2019 Workshop on Representation Learning on Graphs and Manifolds, 2019.
- [57] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., PyTorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems, 2019, pp. 80268037.
- [59] T. N. Kipf, M. Welling, Semi-supervised classification with graph convolutional networks, in: International Conference on Learning Representations, 2017.
- [61] J. Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, G. Hamarneh, BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment, NeuroImage 146 (2017) 1038-1049.
- [60] K. Xu, W. Hu, J. Leskovec, S. Jegelka, How powerful are graph neural networks?, in: International Conference on Learning Representations, 2019.
- [62] X. Kan, W. Dai, H. Cui, Z. Zhang, Y. Guo, C. Yang, Brain network transformer, in: Advances in Neural Information Processing Systems, 2022, pp. 25586-25599.
- [64] L. F. Barrett, E. Bliss-Moreau, Affect as a psychological primitive, Advances in Experimental Social Psychology 41 (2009) 167-218.
- [63] L. F. Barrett, The theory of constructed emotion: An active inference account of interoception and categorization, Social Cognitive and Affective Neuroscience 12 (2017) 1-23.
- [65] C. D. Wilson-Mendenhall, L. F. Barrett, L. W. Barsalou, Neural evidence that human emotions share core affective properties, Psychological Science 24 (2013) 947-956.
- [67] L. Fadiga, L. Craighero, A. D'Ausilio, Broca's area in language, action, and music, Annals of the New York Academy of Sciences 1169 (2009) 448-458.
- [66] F. Aboitiz, R. Garcıa, The evolutionary origin of the language areas in the human brain. A neuroanatomical perspective, Brain Research Reviews 25 (1997) 381-396.
- [68] A. Ardila, B. Bernal, M. Rosselli, How localized are language brain areas? A review of Brodmann areas involvement in oral language, Archives of Clinical Neuropsychology 31 (2016) 112-122.
- [69] T. Most, C. Aviner, Auditory, visual, and auditory-visual perception of emotions by individuals with cochlear implants, hearing aids, and normal hearing, Journal of Deaf Studies and Deaf Education 14 (2009) 449-464.
- [70] M. L. Kringelbach, K. C. Berridge, Towards a functional neuroanatomy of pleasure and happiness, Trends in Cognitive Sciences 13 (2009) 479-487.
- [72] J. Everaert, W. Duyck, E. H. Koster, Attention, interpretation, and memory biases in subclinical depression: A proof-of-principle test of the combined cognitive biases hypothesis, Emotion 14 (2014) 331340.
- [71] A. Suardi, I. Sotgiu, T. Costa, F. Cauda, M. Rusconi, The neural correlates of happiness: A review of PET and fMRI studies using autobiographical recall methods, Cognitive, Affective, &amp; Behavioral Neuroscience 16 (2016) 383-392.
- [73] N. A. Hubbard, J. L. Hutchison, M. Turner, J. Montroy, R. P. Bowles, B. Rypma, Depressive thoughts limit working memory capacity in dysphoria, Cognition and Emotion 30 (2016) 193-209.
- [75] J. A. Arias, C. Williams, R. Raghvani, M. Aghajani, S. Baez, C. Belzung, L. Booij, G. Busatto, J. Chiarella, C. H. Fu, et al., The neuroscience of sadness: A multidisciplinary synthesis and collaborative review, Neuroscience &amp; Biobehavioral Reviews 111 (2020) 199-228.
- [74] A. Yoshino, Y. Okamoto, K. Onoda, S. Yoshimura, Y. Kunisato, Y. Demoto, G. Okada, S. Yamawaki, Sadness enhances the experience of pain via neural activation in the anterior cingulate cortex and amygdala: An fMRI study, NeuroImage 50 (2010) 1194-1201.
Zhongyu Huang Changde Du Chaozhuo Li Kaicheng Fu Huiguang He
Identifying the Hierarchical Emotional Areas in the Human Brain Through Information Fusion
The brain basis of emotion has consistently received widespread attention,
attracting a large number of studies to explore this cutting-edge topic.
However, the methods employed in these studies typically only model the
pairwise relationship between two brain regions, while neglecting the
interactions and information fusion among multiple brain
regions$\unicode{x2014}$one of the key ideas of the psychological
constructionist hypothesis. To overcome the limitations of traditional methods,
this study provides an in-depth theoretical analysis of how to maximize
interactions and information fusion among brain regions. Building on the
results of this analysis, we propose to identify the hierarchical emotional
areas in the human brain through multi-source information fusion and graph
machine learning methods. Comprehensive experiments reveal that the identified
hierarchical emotional areas, from lower to higher levels, primarily facilitate
the fundamental process of emotion perception, the construction of basic
psychological operations, and the coordination and integration of these
operations. Overall, our findings provide unique insights into the brain
mechanisms underlying specific emotions based on the psychological
constructionist hypothesis.