## Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model

Yang Jin 1 , Lei Zhang 2 , Shi Yan 2 , Bin Fan 3 , and Binglu Wang 1 †

1 Xi'an University of Architecture and Technology, Xi'an, China {jin91999, wbl921129}@gmail.com † Corresponding author 2 School of Automation, Northwestern Polytechnical University, Xi'an, China zl\_hnly@163.com, yanshi@mail.nwpu.edu.cn 3 National Key Lab of General AI, School of Intelligence Science and Technology, Peking University, Beijing, China binfan@pku.edu.cn

Abstract. Gaze object prediction (GOP) aims to predict the category and location of the object that a human is looking at. Previous methods utilized box-level supervision to identify the object that a person is looking at, but struggled with semantic ambiguity, i.e ., a single box may contain several items since objects are close together. The Vision foundation model (VFM) has improved in object segmentation using box prompts, which can reduce confusion by more precisely locating objects, offering advantages for fine-grained prediction of gaze objects. This paper presents a more challenging gaze object segmentation (GOS) task, which involves inferring the pixel-level mask corresponding to the object captured by human gaze behavior. In particular, we propose that the pixel-level supervision provided by VFM can be integrated into gaze object prediction to mitigate semantic ambiguity. This leads to our gaze object detection and segmentation framework that enables accurate pixellevel predictions. Different from previous methods that require additional head input or ignore head features, we propose to automatically obtain head features from scene features to ensure the model's inference efficiency and flexibility in the real world. Moreover, rather than directly fuse features to predict gaze heatmap as in existing methods, which may overlook spatial location and subtle details of the object, we develop a space-to-object gaze regression method to facilitate human-object gaze interaction. Specifically, it first constructs an initial human-object spatial connection, then refines this connection by interacting with semantically clear features in the segmentation branch, ultimately predicting a gaze heatmap for precise localization. Extensive experiments on GOO-Synth and GOO-Real datasets demonstrate the effectiveness of our method. The code will be available at https://github.com/jinyang06/SamGOP .

Keywords: Gaze object prediction · Vision foundation model · Object segmentation · Space-to-object gaze regression

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-c1c02d0d-7661-4237-bdb7-0c8681754078.jpg)

(a) Illustration of semantic ambiguity and semantic clarity

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-22b32e66-4bbf-43f3-bfbf-2cc88f44374e.jpg)

(b) The main idea of our pipeline

Fig. 1: (a) Box-level supervision often fails to localize objects in dense settings precisely and leads to semantic ambiguity problems, whereas pixel-level supervision excels by providing clear semantic distinction through pixel-by-pixel predictions. (b) Vision foundation models can produce instance masks, thereby segmentation features can be used to improve the gaze regression branch's spatial perception, and the gaze object mask can help the heatmap focus on the gaze object.

## 1 Introduction

The objects stared at by humans contain important semantic information, which can reveal a person's behavior and state of mind. Therefore, identifying the objects that people are looking at has a wide range of applications. For example, in medical diagnosis, whether a child focuses on an object can potentially reveal whether he/she has autism or visual impairment [4,29,35,37]. In assisted driving scenarios, determining what object the driver is looking at can assess their attention and provide corresponding driving guidance [5,16,26]. In general, detecting the object gazed at by humans realizes the interaction between human sight and specific objects, fostering a richer semantic understanding of the scene.

Recently, vision foundational models (VFM) have gained significant attention due to their powerful generalization capabilities, which can achieve accurate pixel-level segmentation using a variety of prompts, i.e ., boxes, points, or masks.

Tomas et al . [38] first proposed the gaze object detection task and the benchmark datasets GOO-Synth and GOO-Real, aiming to predict the object boxes and categories stared at by humans. Recently, Wang et al . [43, 44] conducted an in-depth exploration of this task. Despite achieving promising results, they still suffer from the following limitations: 1) They typically rely on box-level supervision, which is prone to suffer semantic ambiguity in dense object scenes. Especially when multiple objects are densely stacked or placed adjacent to each other, the box-level supervision algorithm may struggle to accurately separate them, as illustrated in Fig. 1(a). 2) The GOO [38] dataset only contains pixellevel annotations of synthetic images, leading to difficulties in generalizing to fine-grained predictions in real scenes. 3) They usually require additional head information as input, affecting the efficiency and flexibility of the model. While global modeling using the Transformer [41,42] can alleviate this problem, it lacks special attention to head features, which are crucial for gaze direction perception. 4) They directly fuse features to regress the gaze heatmap, lacking a full understanding of the spatial location and subtle detailed information of objects.

In real scenes with small and densely packed objects, VFM can generate accurate boundaries and pixel-level masks (See Fig 3) by utilizing boxes as prompt, which reduces semantic ambiguity and thus opens opportunities for gaze object segmentation. Motivated by this, we propose a more challenging task, gaze object segmentation (GOS). This task involves inferring the pixel-level mask of objects captured by human gaze behavior, helping to further enhance the understanding of human gaze behavior. Furthermore, we establish a GOS framework specifically for this task, which integrates the pixel-level supervision provided by VFM into gaze object prediction to alleviate the aforementioned limitations of previous methods [43,44].

Our main contributions could be summarized as follows:

In the proposed model, we introduce the unified detection and segmentation Transformer, i.e . MaskDINO [22], to simultaneously obtain candidate gaze object boxes, segmentation masks, and head boxes of the subject, which are indispensable for enabling the model to infer in the real world and achieve accurate detection. Instead of requiring additional head input [43, 44] or ignoring head features by global modeling [41,42], we propose a RoI reconstruction module to automatically obtain head region features based on holistic features and head box provided by the unified Transformer. This helps capture gaze direction information during gaze regression to improve model efficiency and flexibility. Subsequently, a space-to-object gaze regression strategy is developed to enhance the perception of the spatial location of objects and the understanding of detailed information. Specifically, inspired by the ability of humans to infer others' gaze objects, a dual attention fusion module is presented to establish an initial human-object gaze spatial connection. Then, we propose an object feature interaction module to refine the gaze spatial connection through semantically clear feature interaction between gaze regression features and object mask features, thus strengthening the modeling of the human-object gaze relationship. In the end, an energy aggregation loss is used to further guide the regression of the gaze heatmap. By focusing on pixel-level details and progressively improving the gaze representation, it further reduces semantic ambiguity, leading to the generation of more precise heatmaps for gazed objects.

- -We introduce the vision foundation model into gaze object prediction to provide pixel-level supervision and propose a more challenging gaze object segmentation task, which focuses on identifying the pixel-level masks of objects captured by human gaze behavior.
- -We propose a space-to-object gaze regression approach that enhances the interaction between object and gaze branches, incrementally improving the modeling of the human-object gaze relationship and alleviating semantic ambiguity.
- -We present an all-in-one end-to-end framework for gaze object detection and segmentation without relying on additional head-related inputs, capable of simultaneously addressing gaze estimation, object detection, and segmentation. To the best of our knowledge, our work is the first to tackle gaze object prediction from the image segmentation perspective.

## 2 Related Works

Vision Foundation Model. The vision foundation models (VFMs) [20,27,51] are typically built on large-scale datasets using self-supervised or semi-supervised methods, which have strong generalizability and zero-shot transfer capabilities. Segment Anything Model (SAM) [20] received attention as a foundation model for image segmentation. Its portability enables it to be flexibly applied to various downstream tasks. Then, the medical foundation model MedSAM [27] extends the foundation model to medical diagnosis. In this article, we choose SAM [20] to obtain high-quality masks to boost gaze estimation and gaze object prediction. Gaze Object Detection. The gaze object detection is first proposed in [38], which aims to predict the box and categories of the object stared at by humans in an RGB image. Tomas et al . [38] released a benchmark dataset for GOP. Wang et al . [44] first proposed a GOP method named GaTector. However, GaTector requires additional head-related prior information, which hinders its application in the real world. Recently, the Transformer-based method [43] achieves comparable performance. Although previous methods are effective to some extent, they can only provide instance-level box outputs, leading to potential ambiguities or misjudgments in scenarios with small and densely packed objects.

Gaze Target Detection. Early research on gaze focused on gaze direction estimation [2,5-7,17,19,30,31,34,45,47,49,50,52], which only predict the gaze vector and ignores the impact of the surrounding environment on the sight, resulting in limited application scenarios. Therefore, to capture higher-level gazerelated semantic information, researchers propose gaze target detection [1,8-10, 13-15, 18, 23, 24, 28, 33, 37, 39, 40, 46], which involves to infer the specific gaze point. Recasens et al . [32] introduced the GazeFollow dataset and first proposed gaze target detection. Recently, Tu et al . [41,42] achieved end-to-end gaze target detection using Transformer, with a relatively low detection efficiency. Despite advancements in gaze target estimation, previous methods struggle with gaze regression due to spatial perception ambiguity. In contrast, our space-to-object regression strategy helps the model gradually converge to a certain gaze point, simplifying learning and enhancing detailed interaction with the object.

## 3 Method

Given an RGB image, our model aims to precisely predict the pixel-level mask of the object humans are looking at. We first present the overall structure of our method, then introduce the VFM-based mask supervision generation method, and finally describe each module of the proposed model in detail.

## 3.1 Overview

As shown in Fig. 2, our method consists of four components: a feature extractor for specific feature extraction, a detection and segmentation module predicting candidate gaze objects boxes, masks, and human head location, a gaze regressor

Fig. 2: Overview of the proposed model. (a) The feature extraction module extracts features for detection and regression branches. (b) The detection and segmentation branch identifies object and human head positions. (c) After obtaining head features, the gaze regression branch progressively refines its output: 1) employing a dual attention fusion module for initial human-object correlations; 2) leveraging a feature interaction module to incorporate semantically clear object-aware insights from the segmentation branch; 3) ultimately predicting the gaze heatmap. (d) Supervision signals are applied to both branches only during training.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-b6824aad-86a5-4268-8a47-4ad0d5136040.jpg)

producing gaze object heatmap with a space-to-object strategy, and a supervision information generation module creating pixel-level masks and ground truth heatmap during training. At training, the overall loss function is defined as:

$$\mathcal { L } = \mathcal { L } _ { \text{det} } + \alpha \mathcal { L } _ { \text{dir} } + \beta \mathcal { L } _ { \text{gaze} } + \gamma \mathcal { L } _ { \text{eng} }$$

where α β , , and γ are the weights of the gaze direction loss, gaze heatmap loss, and energy aggregation loss respectively. Before model training, we choose SAM to generate pixel-level masks for the objects. See Sec. 3.2 for details.

Unified detection and segmentation Transformer. In this paper, we employ a unified Transformer for candidate gaze object detection, segmentation, and human head positioning, integrating MaskDINO [22] for candidate gaze object detection, segmentation, and the generation of pixel-level mask feature embeddings and a head decoder for head box regression, which facilitate accurate head-related feature extraction in head feature reconstruction module and

Feature extraction. As shown in 2(a), given an RGB image I ∈ R 3 × H × W , we first use the backbone B to extract multi-scale features { f 1 , f 2 , f 3 , f 4 } . In previous work [44], pixel shuffle [36] significantly improved small object detection by enlarging feature maps for detailed feature capture. Therefore, we integrate it into dense prediction tasks, i.e ., image segmentation. In our model, the multi-scale features { f 1 , f 2 , f 3 , f 4 } are fed into the pixel shuffle module to generate objectspecific features f obj i = ϕ obj ( f i , η ) , where f i denotes the i -th feature layer, η denotes the scale factor, ϕ obj ( ) · denotes the pixel shuffle module, which consists of a pixel shuffle operation and a convolution layer that transforms the feature channels to be consistent with the original features of the unified Transformer [22]. Finally, the scene saliency features f scene ∈ R 1024 × × 7 7 are extracted by a scene residual block using high dimensional features f 4 ∈ R 2048 × × 7 7 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-d691dabb-60cb-4562-9541-e8dd0e86e08b.jpg)

(a) GOO-Synth

(b) GOO-Real

Fig. 3: Illustration of instance mask generated by VFM.

accurate prediction in real-world settings. The training loss is defined as:

$$\mathcal { L } _ { \det } = \mathcal { L } _ { \det } ^ { \text{obj} } + \mathcal { L } _ { \text{mask} } ^ { \text{obj} } + \mathcal { L } _ { \det } ^ { \text{head} }$$

L obj det and L obj mask is the object detection and segmentation loss respectively. L head det is the head decoder loss, which is consistent with the L obj det . See Sec. 3.3 for details. Space-to-object gaze regression. We employ a space-to-object regression strategy for gaze heatmap prediction. The module begins by representing human gaze behavior with a gaze vector. Subsequently, a dual attention fusion module establishes an initial human-object spatial connection. Finally, pixel-level object location knowledge from the mask branch is introduced to achieve semantically clear feature interaction, refining the gaze heatmap regression while alleviating semantic ambiguity. Optimization involves L dir and L gaze for gaze direction and gaze heatmap, respectively. The energy aggregation loss L eng guides the gaze heatmap to focus on the gaze object mask. See Sec. 3.4 for details.

## 3.2 Instance mask generation via VFM.

Given an RGB image I , we choose the vision foundation model SAM [20] to obtain instance masks for model training. Specifically, image I and prompt P box are first fed into the image encoder Ψ Image and prompt encoder Ψ prompt , respectively. Then, the mask P mask and confidence S are output by the mask decoder Φ mask , which can be formulated as:

$$\mathcal { P } _ { \text{mask} }, \mathcal { S } = \Phi _ { \text{mask} } ( \Psi _ { I \text{mage} } ( \mathcal { I } ), \Psi _ { p r o m p t } ( \mathcal { P } _ { b o x } ) )$$

Instance segmentation by using boxes produces accurate masks for most objects, but accurate segmentation of objects with unclear texture or edge features remains challenging. Therefore, we segment again using P mask as a dense prompt to produce final mask M mask , which can be formulated as:

$$\mathcal { M } _ { \text{mask} }, \mathcal { S } = \Phi _ { \text{mask} } ( \Psi _ { I \text{mag} e } ( \mathcal { I } ), \Psi _ { p r o m p t } ( \mathcal { P } _ { \text{mask} } ) )$$

Finally, M mask is used to train the gaze object segmentation model. The generated masks are shown in Fig. 3.

## 3.3 Unified detection and segmentation Transformer

The unified detection and segmentation Transformer aims to detect candidate gaze objects and locate human heads while extracting high-level pixel mask

features, which comprises a detection and segmentation Transformer, a head decoder, and a RoI feature reconstruction module. As shown in Fig. 2(b), the object-specific features { f obj 1 , f obj 2 , f obj 3 , f obj 4 } are fed it to detect object boxes, categories, masks, and human head boxes.

Head decoder Our model is designed for real-world gaze object detection and segmentation without relying on head priors. We introduce a head decoder based on the unified Transformer, using a 3-layer self-attention decoder to generate proposals for head-related feature reconstruction. During training, we filter head queries with IoU &gt; 0.5 as positives. During inference, boxes are selected based on confidence. Similar to other DETR-like detectors [48, 53], the head decoder uses cross-entropy, L 1 , and GIoU loss for classification and box regression.

Object detection and segmentation The global attention of Transformer ensures powerful long-distance modeling and global perception, achieving excellent detection performance through layer-by-layer self-attention refinement. We use MaskDINO [22] as a foundational detection and segmentation structure due to its robust representation capabilities and fast convergence speed.

Head feature reconstruction Previous methods typically extract head features from manually cropped head images, limiting model inference efficiency and flexibility in the real world. To this end, we design a RoI feature reconstruction module to reconstruct head-related features based on holistic scene features and the head box from the unified Transformer. In this module, we first use the head bounding box R = ( x , y 1 1 , x 2 , y 2 ) , which represents the upper left and lower right corner coordinates, as the region proposal to crop the regions of interest (ROIs). Then, those candidate ROIs are refined by ROIAlign [11], which maps the head region to the corresponding pixels in the holistic image features f 4 ∈ R 2048 × × 7 7 and produces a fixed-size feature map f head ∈ R 2048 × × 7 7 . For head location map generation, following previous works [9,10,32,44] to generate a binary image, setting the head area to 1 and the rest to 0. Additionally, a specific residual block extracts gaze saliency features f gaze ∈ R 1024 × × 7 7 , further enhancing the model's attention to the gaze direction.

## 3.4 Space-to-object gaze regression

We propose a space-to-object gaze regression approach to gradually refine the human-object gaze relationship. Fig. 2(c) shows our gaze regression branch with a dual attention fusion module establishing an initial human-object spatial connection and a feature interaction module strengthening this connection using semantically clear feature interaction.

As shown in Fig. 4(a), we first use global average pooling to obtain the global features of f head , followed by fully connected layers to transform the features into

Spatial perception for gaze objects. When humans infer what another person is staring at, they usually first observe the gaze direction of the others and search for the gazed object along this direction. Inspired by this, we propose the dual attention fusion module for the spatial perception of gaze objects, establishing an initial human-object spatial connection.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-528640ad-7c41-46cb-a33e-3a711adff7a2.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-e3be6a4d-eb69-4411-9718-ae8bdff0db57.jpg)

module

(b) Comparison of different attention map generation methods in dual attention fusion module

Fig. 4: Illustration of spatial perception for gaze objects.

a 2D gaze vector v g . To achieve spatial perception, we project it into a gaze cone map M cone ∈ R 1 × × 7 7 , activating the area around the gaze direction:

$$M _ { i, j } ^ { \text{cone} } = M a x ( 0, \cos ( \mathbf v _ { g }, \mathbf v _ { i, j } ) )$$

where v g and v i , j respectively denotes the predicted gaze vector and the vector composed of the eye point and each point on the gaze field. During training, v g is optimized by L dir to obtain a more accurate gaze direction and activation map. L dir is defined as follows:

$$\mathcal { L } _ { \text{dir} } = 1 - \sum _ { i = 1 } ^ { n } ( \mathbf v _ { g } ^ { i } \cdot \mathbf v _ { t } ^ { i } )$$

where vector v g and v t denote the predicted gaze direction and ground truth, respectively. During backpropagation, L dir adjusts parameters of head feature f head , providing more direction information for gaze regression. To boost scene attention, we adopt the method from GaTector [44] to generate an original attention map M ori ∈ R 1 × × 7 7 , activating scene salience area based on head properties. By multiplying M cone with M ori via element-wise dot product, we generate the final spatial perception map M :

$$M = M _ { \text{ori} } \odot M _ { \text{cone} }$$

This compels the model to search spaces more likely gazed at by humans along

As shown in Fig. 2(c), we initially extract detailed mask features from the mask branch. Subsequently, the fused feature f fuse is utilized for pixel-level attention perception. Given that candidate gaze objects can be located anywhere in the image, the Transformer's global modeling excels in accomplishing this task. Therefore, utilize a transformer layer, akin to DETR [3], to capture the global relationship between human gaze behavior and object masks. For fine-grained gaze direction, establishing an initial human-object gaze spatial relationship. Semantically clear feature interaction. Previous methods [43, 44] usually directly fuse features to regress the gaze heatmap based on spatial saliency, lacking interaction with object details and facing challenges in accurately predicting heatmap for the gazed object. In our method, we introduce a feature interaction module, which leverages semantically clear mask knowledge from VFM to achieve feature interaction, refining human-object gaze relationship modeling.

object mask features, a three-layer MLP is applied to process the decoder query Q obj of the unified Transformer, generating mask embeddings:

$$\mathcal { Q } _ { \max k } = \text{MLP} ( \mathcal { Q } _ { \text{obj} } )$$

where the Q mask denote the generated mask embedding. We first apply a layer self-attention for f fuse :

$$f _ { e } = \text{FFN} ( \text{SelfAttn} ( f _ { \text{fuse} } ) )$$

where f fuse , SelfAttn( ) · , and f e denote the fused feature f fuse , self-attention operation, and encoded f fuse , respectively. Then, the Q mask is perceived by the f e as key-value pairs, i.e ., { K mask , V mask } :

$$f _ { \text{reg} } = \text{FFN} ( \text{CrossAttn} ( f _ { \text{e} }, \{ K _ { \text{mask} }, V _ { \text{mask} } \} ) ) \quad \quad (10 )$$

where f reg and CrossAttn( q, kv ) denote the regression feature and cross-attention operation, respectively. This operation refines the perception of the object by providing a pixel-level understanding of the gaze object in each token of the regression feature. It effectively alleviates semantic ambiguity and strengthens the ability of the model to represent the human-object gaze relationship.

Ultimate gaze regression. Finally, following previous work [44] to generate the ground truth gaze heatmap and use L 2 loss to supervise gaze heatmap regression. L eng proposed by [43] guides gaze heatmap energy aggregation towards the gaze object. We use the gaze object mask for more accurate error measurements. Space-to-object gaze regression approach gradually refines the humanobject gaze behavior representation, regressing a more accurate gaze heatmap.

## 4 Experiments

## 4.1 Experiments Settings

Datasets and implementation details are given in supplementary material .

Metrics: Following previous work [43,44], we use AUC (Area Under the Curve), L2 distance, and Angle error for gaze estimation evaluation, and mSoC (min Single over Closure) 4 under different thresholds for gaze object prediction evaluation. Following MaskDINO [22], AP (Average Precision) is used for instance segmentation and object detection evaluation. To evaluate the gaze object segmentation, we use masks to replace boxes when calculating mSoC (details refer to the supplementary material ). For a fair comparison, we reported results for both real-world and non-real-world settings. The non-real-world settings involve inputting additional head images and head location maps while real-world settings only use scene images. Furthermore, to compare with GTR [42], we also adopt the evaluation method proposed by [42] 5 . This method computes gaze estimation metrics only for results with L2 distance &lt; 0.15 and head box IoU &gt; 0.5, which is an incomplete evaluation for AUC, L2 distance, and Angle error.

4 mSoC metric proposed by Wang et al . URL: https://arxiv.org/abs/2112.03549 .

5 Evaluation method proposed by Tu et al ., URL: https://ieeexplore.ieee.org/ document/10262016 .

Table 1: Gaze object detection results on GOO-Synth and GOO-Real datasets. ∗ indicates using additional head detector to achieve real-world settings.

|                               | GOO-Synth                 | GOO-Synth   | GOO-Synth   | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real Pretrain   | GOO-Real Pretrain       | GOO-Real Pretrain   | GOO-Real Pretrain   |           |
|-------------------------------|---------------------------|-------------|-------------|------------------------|------------------------|------------------------|------------------------|---------------------|-------------------------|---------------------|---------------------|-----------|
| Method                        | mSoC                      | 50 75       | 95          | mSoC                   | 50                     | 75                     | 95                     | mSoC                | 50 75                   | 95                  | Params              | FPS       |
| GaTector [44] TransGOP [43]   | 67.94 98.14 92.80 99.00   | 86.25 98.50 | 0.12 51.90  | 62.40 82.60            | 95.10 97.80            | 73.50 89.40            | 0.20 6.50              | 71.20 89.00         | 97.50 88.80 98.90 97.50 | 2.70 33.20          | 60.78M 94.03M       | 14.1 13.7 |
| Ours                          | 91.19 93.97               | 93.46       | 73.27       | 81.75                  | 98.78                  | 96.43                  | 7.13                   | 81.89               | 98.99 96.63             | 7.28                | 80.99M              | 14.5      |
| GaTector [44] ∗ TransGOP [43] | 56.19 88.08 ∗ 91.60 99.00 | 66.18 98.10 | - 48.70     | 40.87 79.70            | 82.05 97.80            | 33.54 91.40            | 0.30 11.20             | 66.40 83.20         | 96.22 80.49 98.80 95.30 | 0.40 14.40          | 75.73M 106.37M      | 9.3 11.6  |
| Ours                          | 90.77                     | 93.92 93.45 | 69.58       | 81.35                  | 98.95                  | 96.02                  | 2.61                   | 81.60               | 99.03 96.54             | 6.03                | 84.26M              | 14.1      |

Table 2: Ours gaze object segmentation results on GOO-Real and GOO-Synth.

Fig. 5: Comparison of TransGOP [43] and our method inference in the real world.

|               | GOO-Synth   | GOO-Synth   | GOO-Synth         | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real No Pretrain   | GOO-Real Pretrain   | GOO-Real Pretrain   | GOO-Real Pretrain   | GOO-Real Pretrain   |           |    |    |
|---------------|-------------|-------------|-------------------|------------------------|------------------------|------------------------|------------------------|---------------------|---------------------|---------------------|---------------------|-----------|----|----|
| Settings      | mSoC        | 50          | 75 95             | mSoC                   | 50                     | 75                     | 95                     | mSoC                | 50                  | 75 95               | Params              | FPS       |    |    |
| Non-Real Real | 82.68 82.73 | 93.70 93.67 | 91.68 26.83 91.79 | 86.39 85.94            | 98.95 98.99            | 97.44 97.29            | 20.07 16.70            | 89.39 86.96         | 98.80 98.94         | 98.02               | 33.03 80.99M 84.26M | 14.5 14.1 |    |    |
|               |             |             | 26.85             |                        |                        |                        |                        |                     |                     | 97.75 22.21         |                     |           |    |    |

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-fb34a69b-f519-4c26-b514-97b1ac69d123.jpg)

## 4.2 Comparison with SOTA

Gaze object detection and segmentation. Table 1 shows the performance of gaze object detection on GOO-Synth and GOO-Real. In real-world settings, previous methods [43, 44] significantly increase parameters and slow down the inference speed due to the additional head detector (As shown in Fig. 5 (a)) and they can only achieve box-level prediction. In contrast, although our method is slightly inferior to TransGOP [43] in detection performance, our model runs significantly more efficiently because it integrates the acquisition of head features into the overall framework (As shown in Fig. 5 (b)) and can output bounding boxes and segmentation masks simultaneously. The above results also show that our space-to-object gaze regression method can effectively model the humanobject gaze relationship. Table 2 shows our gaze object segmentation performance in real-world and non-real-world scenarios. On GOO-Synth, our model achieves 82.68% and 82.73% mSoC in real-world and non-real-world settings. On GOO-Real, without pre-training, our model performs well in both scenarios (86.39% mSoC and 85.94% mSoC). After pre-training, it reaches 89.39% mSoC and 86.96% mSoC. The above experimental results show that the pixel-level mask knowledge provided by VFMs can effectively alleviate semantic ambiguity while achieving pixel-level prediction. Even under the multi-task learning framework, we still achieve an FPS of 14.1. This is mainly attributed to the automatic

Table 3: Instance segmentation results on GOO-Synth and GOO-Real datasets

|               | GOO-Synth   | Dataset     |   Dataset | GOO-Real   | Dataset     |   Dataset | Params   |    |    |
|---------------|-------------|-------------|-----------|------------|-------------|-----------|----------|----|----|
| Method        | AP          | 50 75       |     95    | AP         | 50 75       |     95    | Params   |    |    |
| Non-Real Ours | 79.33       | 93.51 91.09 |     11.72 | 82.91      | 98.85 95.09 |     11.22 | 80.99M   |    |    |
| MaskDINO [22] | 70.85       | 95.54 83.31 |      1.81 | 77.05      | 98.38 90.45 |      4.26 | 52.03M   |    |    |
| Real Ours     | 78.56       | 96.00 90.56 |      8.43 | 82.86      | 99.02 95.57 |     10.09 | 84.26M   |    |    |

Table 4: Object detection results on GOO-Synth and GOO-Real datasets. ∗ indicates using additional head detector to achieve real-world settings.

|                        | GOO-Synth Dataset   |   GOO-Synth Dataset |   GOO-Synth Dataset | GOO-Synth Dataset   | GOO-Real Dataset   |   GOO-Real Dataset |   GOO-Real Dataset | GOO-Real Dataset   | Params   |
|------------------------|---------------------|---------------------|---------------------|---------------------|--------------------|--------------------|--------------------|--------------------|----------|
| Method                 | AP                  |               50    |               75    | 95                  | AP                 |              50    |              75    | 95                 | Params   |
| GaTector [44]          | 56.80               |               95.3  |               62.5  | 0.10                | 52.25              |              91.92 |              55.34 | 0.10               | 60.78M   |
| Non-Real TransGOP [43] | 87.60               |               99    |               97.3  | 25.50               | 77.20              |              97.8  |              89.4  | 6.50               | 94.03M   |
| Ours                   | 88.33               |               93.88 |               93.08 | 50.70               | 77.32              |              98.65 |              93.91 | 1.93               | 80.99M   |
| GaTector [44] ∗        | 43.50               |               86.23 |               37.52 | -                   | 29.21              |              74.02 |              16.13 | -                  | 75.73M   |
| TransGOP [43] ∗        | 86.00               |               98.8  |               96.2  | 23.80               | 73.30              |              96.7  |              86    | 4.10               | 106.37M  |
| MaskDINO [22]          | 72.93               |               96.26 |               88.13 | 2.04                | 74.44              |              98.72 |              91.19 | 0.19               | 52.03M   |
| Ours                   | 87.60               |               93.72 |               73.07 | 45.53               | 75.20              |              98.94 |              93.2  | 0.76               | 84.26M   |

incorporation of head features, avoiding the need for an additional head detector and thus enhancing the efficiency of the model.

Object detection and segmentation. Tables 3 and 4 show our model's instance segmentation and object detection performance. For a fair comparison, we adjusted the input size of MaskDINO [22] to match ours (224 × 224). In Table 3, our model outperforms MaskDINO by 8.66 AP on GOO-Real (70.85 vs. 79.51) and by 5.81 AP on GOO-Synth (77.05 vs. 82.86) in real-world scenarios, with similar advantages in non-real-world settings of our model. These results show the challenge faced by the original MaskDINO [22] in capturing detailed features in scenes with dense objects and the effectiveness of the pixel shuffle of our model in capturing detailed features. Table 4 further demonstrates the superiority of our model in object detection over previous methods and MaskDINO [22].

Gaze estimation. Tables 5 and 6 show the gaze estimation performance on GOO-Synth and GOO-Real, respectively. In Table 5, our method outperforms GTR [42] and TransGOP [43] by 23.9 mAP and 10.6 mAP, respectively. Even directly evaluating all prediction results in the real world, our model reduces L2 distance and angle errors compared to TransGOP [43] by 0.015 and 1.8, respectively. Applying non-real-world methods to the real world significantly degrades performance, highlighting our real-world model is more efficient. Table 6 verifies our method on GOO-Real, achieving state-of-the-art results without pretraining. Even with direct inference in a real-world setting, our method outperforms previous non-real-world methods. After pre-training, our model exhibits more pronounced performance advantages. These results show the effectiveness

Table 5: Gaze estimation results on GOO-Synth dataset. † indicates the evaluation method proposed by [42].

Table 6: Gaze estimation results on GOO-Real dataset. † indicates the evaluation method proposed by [42].

| Method               |   AUC |   Dist. ↓ |   Ang. ↓ | mAP   |
|----------------------|-------|-----------|----------|-------|
| GazeFollow [32]      | 0.929 |     0.162 |     33   | -     |
| Lian [24]            | 0.954 |     0.107 |     19.7 | -     |
| VideoAttention [9]   | 0.952 |     0.075 |     15.1 | -     |
| GaTector [44]        | 0.957 |     0.073 |     14.9 | -     |
| GTR [42]             | 0.96  |     0.071 |     14.5 | -     |
| TransGOP [43]        | 0.963 |     0.079 |     13.3 | -     |
| Ours                 | 0.947 |     0.072 |     14.8 | -     |
| GazeFollow [32] †    | 0.832 |     0.317 |     42.6 | 0.468 |
| Lian [24] †          | 0.903 |     0.153 |     28.1 | 0.454 |
| VideoAttention [9] † | 0.912 |     0.143 |     24.5 | 0.489 |
| GaTector [44] †      | 0.918 |     0.139 |     24.5 | 0.510 |
| GTR [42] †           | 0.962 |     0.068 |     14.2 | 0.597 |
| TransGOP [43] †      | 0.977 |     0.07  |     11.5 | 0.730 |
| TransGOP [43]        | 0.945 |     0.106 |     20.7 | -     |
| Ours †               | 0.978 |     0.057 |     10.2 | 0.836 |
| Ours                 | 0.938 |     0.091 |     18.9 | -     |

|          | Method                                                     |   AUC |   Dist. ↓ | Ang. ↓   | mAP   |
|----------|------------------------------------------------------------|-------|-----------|----------|-------|
| Non-Real | GazeFollow [32] Lian [24] VideoAttention [9] GaTector [44] | 0.85  |     0.22  | 44.4     | - -   |
| Non-Real |                                                            | 0.84  |     0.321 | 43.5     |       |
| Non-Real |                                                            | 0.796 |     0.252 | 51.4     | -     |
| Non-Real |                                                            | 0.927 |     0.196 | 39.5     | -     |
| Non-Real | Tonini [39]                                                | 0.918 |     0.164 | -        | -     |
| Non-Real | TransGOP [43]                                              | 0.947 |     0.097 | 16.7     | -     |
| Non-Real | Ours                                                       | 0.943 |     0.078 | 12.9     | -     |
| Real     | Ours †                                                     | 0.971 |     0.059 | 9.8      | 0.810 |
| Real     | Ours                                                       | 0.944 |     0.088 | 14.7     | -     |
|          | GazeFollow [32]                                            | 0.903 |     0.195 | 39.8     | -     |
|          | Lian [24]                                                  | 0.89  |     0.168 | 32.6     | -     |
| Non-Real | VideoAttention [9]                                         | 0.889 |     0.15  | 29.1     | -     |
| Non-Real | GaTector [44]                                              | 0.94  |     0.087 | 14.8     | -     |
| Non-Real | TransGOP [43]                                              | 0.957 |     0.081 | 14.7     | -     |
| Non-Real | Ours                                                       | 0.963 |     0.073 | 12.4     | -     |
| Real     | Ours †                                                     | 0.978 |     0.051 | 8.8      | 0.824 |
| Real     | Ours                                                       | 0.949 |     0.082 | 13.8     | -     |

Table 7: Ablation study about each component on GOO-Real

|             |         |    | Gaze     | Object Segmentation Instance Segmentation   | Object Segmentation Instance Segmentation   | Object Segmentation Instance Segmentation   | Object Segmentation Instance Segmentation   | Object Segmentation Instance Segmentation   | Object Segmentation Instance Segmentation   | Gaze Estimation         | Gaze Estimation   |
|-------------|---------|----|----------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|---------------------------------------------|-------------------------|-------------------|
| PS RoI-Rec. | DAF FIM |    | EAL mSoC | mSoC 50                                     | mSoC 50                                     | mSoC 75                                     | AP                                          | AP 50                                       | AP 75                                       | AUC Dist. ↓             | Ang. ↓            |
| ✓           |         |    |          | 79.00 84.50                                 | 98.48 98.75                                 | 92.50 96.56                                 | 74.24 80.53                                 | 98.00                                       | 87.62                                       | 0.854 0.304             | 54.2 59.8         |
| ✓ ✓         | ✓       |    | 84.63    | 98.67                                       |                                             | 96.72                                       | 81.70                                       | 98.68 98.83                                 | 93.67 94.66                                 | 0.839 0.334 0.910 0.114 | 20.5              |
| ✓ ✓         |         |    | 85.44    |                                             | 98.92                                       | 96.96                                       | 81.58                                       | 98.76                                       | 94.29                                       | 0.919 0.095             | 16.8              |
| ✓ ✓         | ✓       | ✓  |          | 85.89                                       | 98.80                                       | 97.14                                       | 82.45                                       | 98.92                                       | 94.82                                       | 0.932 0.089             | 14.9              |
| ✓ ✓         | ✓       | ✓  | ✓        | 85.94                                       | 98.99                                       | 97.20                                       | 82.86                                       | 99.02                                       | 95.57                                       | 0.944 0.088             | 14.7              |

of our space-to-object gaze regression method in modeling human-object gaze relationships while achieving more accurate gaze heatmap regression.

## 4.3 Ablation study and model analysis

Ablation study about the effect of each component. In Table 7, we conduct various ablation studies on the GOO-Real dataset. We first establish a baseline using MaskDINO [22] and a gaze regression branch similar to TransGOP [43], and then gradually add the proposed modules and demonstrate their contribution to overall model performance. (i) Pixel shuffle (PS). After adding the pixel shuffle module, instance segmentation performance improved by 6.29% AP (80.53% vs 74.24%) compared to baseline, indicating that large feature maps significantly enhance detailed features in dense prediction tasks. (ii) RoI Reconstruction (RoI-Rec.). The RoI reconstruction module reduces angle error by 39.3 and L2 distance by 0.22, demonstrating the importance of head-related features for gaze direction perception and the ability of the module to learn accurate gaze features in real-world settings. (iii) Dual attention fusion (DAF).

Table 8: Analysis about dual attention fusion module.

|                              | Gaze Object Segmentation Instance Segmentation Gaze   | Gaze Object Segmentation Instance Segmentation Gaze   |   Gaze Object Segmentation Instance Segmentation Gaze | Gaze Object Segmentation Instance Segmentation Gaze   | Gaze Object Segmentation Instance Segmentation Gaze   | Gaze Object Segmentation Instance Segmentation Gaze   | Gaze Object Segmentation Instance Segmentation Gaze   | Estimation   | Estimation   |
|------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|-------------------------------------------------------|--------------|--------------|
| Original Attention Gaze Cone | mSoC mSoC                                             | mSoC mSoC                                             |                                                 50    | mSoC 75                                               | AP AP                                                 | 50 AP 75                                              | AUC Dist. ↓                                           |              | Ang. ↓       |
| ✓                            |                                                       | 85.51                                                 |                                                 98.86 | 97.17                                                 | 82.08 98.87                                           | 82.08 98.87                                           | 94.78                                                 | 0.938 0.089  | 15.6         |
| ✓                            |                                                       | 84.13                                                 |                                                 98.78 | 96.35                                                 | 81.30 98.78                                           | 81.30 98.78                                           | 93.69                                                 | 0.926 0.097  | 16.3         |
| ✓                            | ✓                                                     | 85.94                                                 |                                                 98.99 | 97.20                                                 | 82.75 99.01                                           | 82.75 99.01                                           | 95.17                                                 | 0.944 0.088  | 14.7         |

Table 9: Analysis about feature interaction module

| Encoder Decoder   | Encoder Decoder   | Mask                  | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Estimation   | Gaze Estimation   |
|-------------------|-------------------|-----------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|-------------------|-------------------|
| Query             | Query             | Embeding mSoC mSoC 50 | Embeding mSoC mSoC 50                            | Embeding mSoC mSoC 50                            | mSoC 75                                          | AP                                               | AP 50                                            | AP 75                                            | AUC Dist. ↓       | Ang. ↓            |
| ✓                 |                   |                       | 85.79                                            | 98.38                                            | 96.44                                            | 81.28 98.87                                      | 81.28 98.87                                      | 94.87                                            | 0.928 0.095       | 16.0              |
|                   | ✓                 |                       | 85.20                                            | 98.21                                            | 97.01                                            | 81.65 98.86                                      | 81.65 98.86                                      | 94.65                                            | 0.949 0.093       | 15.1              |
|                   |                   | ✓                     | 85.94                                            | 98.99                                            | 97.20                                            | 82.86 99.02                                      | 82.86 99.02                                      | 95.57                                            | 0.944 0.088       | 14.7              |

Introducing the dual attention fusion improves gaze heatmap quality (AUC: 0.919 vs 0.910), demonstrating that perceiving gaze object spatial location beforehand produces more accurate heatmaps and eases regression. (iv) Feature interaction module (FIM). The feature interaction module improves gaze estimation, reducing angle error by 1.9 and L2 distance by 0.006. Gaze object segmentation achieves an 85.89 mSoC, showing that pixel-level object information enhances feature interaction and human-object gaze relationship modeling.

Analysis about dual attention fusion module. Table 8 analyzed the effect of different spatial perception methods. Compared to using original attention M ori or gaze cone M cone alone, our dual attention fusion performs better. This indicates that combining spatial search along the gaze direction with human head-based scene saliency is more effective. Fig. 4(b) shows that element-wise fusion of M ori and M cone creates an initial human-object spatial connection.

- (v) Energy aggregation loss (EAL). Adding energy aggregation loss improves the performance of all three sub-tasks, showing that the gaze object mask effectively guides the regression of the gaze heatmap towards the gaze object.

Analysis about feature interaction module. In Table 9, we evaluate the efficacy of different object information for gaze regression. Injecting the global encoder query of MaskDINO [22] results in 85.79% mSoC for gaze object segmentation but poor gaze estimation performance. Refining the decoder query with mask supervision reduces the angle error by 0.9, indicating better object representation. Using mask embedding with mask features during gaze regression achieves optimal performance, demonstrating that semantically clear mask features enhance gaze object positioning and the human-object gaze relationship.

## 4.4 Qualitative Results.

Fig. 6(a) presents qualitative results in the real-world setting. In object-dense scenes, our method accurately predicts the gaze object mask without relying on head-related cues. In Fig. 6(b), compared to previous methods, our method ac-

(b) Visualization of our method and previous box-level supervision methods in Non-Real-World setting.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-c3eaf91d-6d64-4978-8ccc-29723868f3ac.jpg)

Fig. 6: Qualitative comparison of ground truth gaze object (GT), gaze object detection (GOD), instance segmentation (IS), gaze object segmentation (GOS)

3616.JPG 3632.JPG 3792.JPG 507.PNG 728.PNG curately locates adjacent objects using a pixel-level mask, effectively alleviating semantic ambiguity. See supplementary material for more visualization.

## 5 Conclusion

In this paper, we introduce the gaze object segmentation task, which aims to infer pixel-level masks for objects captured by human eyesight, and establish a framework based on the pixel-level supervision from VFM to solve this task while alleviating the semantic ambiguity existing in previous box-level supervision methods. In our method, we choose SAM to generate pixel-level supervision information for the model training. The RoI reconstruction module is proposed to reconstruct head features directly from holistic features using the head box provided by the unified Transformer, which improves the inference efficiency and flexibility of our model in the real world. To boost the representation of the human-object gaze relationship, we propose a space-to-object gaze regression strategy, which first uses the dual attention fusion module to establish a humanobject spatial connection, and then a semantically clear mask feature is injected into the gaze regression to strengthen the human-object gaze relationship modeling. Extensive experiments on GOO-Real and GOO-Synth demonstrate the effectiveness of our method in various settings.

Limitation: The proposed method requires box supervision as prompts for the SAM when generating masks, which limits its application in a wider range of scenarios without strong location priors, more flexible approaches for generating masks via the latest VFM models need to be explored in future work.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China. (No. 82272020).

Bin Fan is sponsored by China National Postdoctoral Program for Innovative Talents (No. BX20230013).

## References

- 1. Bao, J., Liu, B., Yu, J.: ESCNet: Gaze target detection with the understanding of 3D scenes. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 14126-14135 (2022)
- 3. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: Endto-end object detection with transformers. In: Eur. Conf. Comput. Vis. pp. 213-229 (2020)
- 2. Cai, X., Zeng, J., Shan, S., Chen, X.: Source-free adaptive gaze estimation by uncertainty reduction. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2203522045 (2023)
- 4. Chen, W., Li, R., Yu, Q., Xu, A., Feng, Y., Wang, R., Zhao, L., Lin, Z., Yang, Y., Lin, D., et al.: Early detection of visual impairment in young children using a smartphone-based deep learning system. Nature Medicine 29 (2), 493-503 (2023)
- 6. Cheng, Y., Lu, F.: Gaze estimation using transformer. In: Int. Conf. Pattern Recog. pp. 3341-3347 (2022)
- 5. Chen, Y., Nan, Z., Xiang, T.: FBLNet: Feedback loop network for driver attention prediction. In: Int. Conf. Comput. Vis. pp. 13371-13380 (2023)
- 7. Cheng, Y., Lu, F., Zhang, X.: Appearance-based gaze estimation via evaluationguided asymmetric regression. In: Eur. Conf. Comput. Vis. pp. 100-115 (2018)
- 9. Chong, E., Wang, Y., Ruiz, N., Rehg, J.M.: Detecting attended visual targets in video. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5396-5406 (2020)
- 8. Chong, E., Ruiz, N., Wang, Y., Zhang, Y., Rozga, A., Rehg, J.M.: Connecting gaze, scene, and attention: Generalized attention estimation via joint modeling of gaze and scene saliency. In: Eur. Conf. Comput. Vis. pp. 383-398 (2018)
- 10. Gupta, A., Tafasca, S., Odobez, J.M.: A modular multimodal architecture for gaze target prediction: Application to privacy-sensitive settings. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5041-5050 (2022)
- 12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 770-778 (2016)
- 11. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2961-2969 (2017)
- 13. Hu, Z., Yang, D., Cheng, S., Zhou, L., Wu, S., Liu, J.: We know where they are looking at from the RGB-D camera: Gaze following in 3D. IEEE Trans. Instrum. Meas. 71 , 1-14 (2022)
- 15. Hu, Z., Zhao, K., Zhou, B., Guo, H., Wu, S., Yang, Y., Liu, J.: Gaze target estimation inspired by interactive attention. IEEE Trans. Circuit Syst. Video Technol. 32 (12), 8524-8536 (2022)
- 14. Hu, Z., Yang, Y., Zhai, X., Yang, D., Zhou, B., Liu, J.: GFIE: A dataset and baseline for gaze-following from 2D to 3D in indoor environments. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 8907-8916 (2023)
- 16. Huang, T., Fu, R.: Driver distraction detection based on the true driver's focus of attention. IEEE Trans. Intell. Transp. Syst. 23 (10), 19374-19386 (2022)

- 17. Jin, S., Wang, Z., Wang, L., Bi, N., Nguyen, T.: ReDirTrans: Latent-to-latent translation for gaze and head redirection. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 5547-5556 (2023)
- 19. Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., Torralba, A.: Gaze360: Physically unconstrained gaze estimation in the wild. In: Int. Conf. Comput. Vis. pp. 6912-6921 (2019)
- 18. Jin, T., Yu, Q., Zhu, S., Lin, Z., Ren, J., Zhou, Y., Song, W.: Depth-aware gazefollowing via auxiliary networks for robotics. Eng. Appl. Artif. Intell. 113 , 104924 (2022)
- 20. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Int. Conf. Comput. Vis. pp. 4015-4026 (2023)
- 22. Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L.M., Shum, H.Y.: Mask dino: Towards a unified transformer-based framework for object detection and segmentation. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 3041-3050 (2023)
- 21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Process. Syst. 25 (2012)
- 23. Li, Y., Shen, W., Gao, Z., Zhu, Y., Zhai, G., Guo, G.: Looking here or there? gaze following in 360-degree images. In: Int. Conf. Comput. Vis. pp. 3742-3751 (2021)
- 25. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
- 24. Lian, D., Yu, Z., Gao, S.: Believe it or not, we know what you are looking at! In: Asian Conf. Comput. Vis. pp. 35-50 (2018)
- 26. Lv, K., Sheng, H., Xiong, Z., Li, W., Zheng, L.: Improving driver gaze prediction with reinforced attention. IEEE Trans. Multimedia 23 , 4198-4207 (2020)
- 28. Miao, Q., Hoai, M., Samaras, D.: Patch-level gaze distribution prediction for gaze following. In: IEEE Winter Conf. Appl. Comput. Vis. pp. 880-889 (2023)
- 27. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15 (1), 654 (2024)
- 29. Mundy, P., Sigman, M., Kasari, C.: A longitudinal study of joint attention and language development in autistic children. J. Autism Dev. Disord. 20 (1), 115-128 (1990)
- 31. Park, S., Zhang, X., Bulling, A., Hilliges, O.: Learning to find eye region landmarks for remote gaze estimation in unconstrained settings. In: Proc. ACM Symp. Eye Track. Res. Appl. pp. 1-10 (2018)
- 30. Park, S., Spurr, A., Hilliges, O.: Deep pictorial gaze estimation. In: Eur. Conf. Comput. Vis. pp. 721-738 (2018)
- 32. Recasens, A., Khosla, A., Vondrick, C., Torralba, A.: Where are they looking? Adv. Neural Inform. Process. Syst. 28 (2015)
- 34. Ruzzi, A., Shi, X., Wang, X., Li, G., De Mello, S., Chang, H.J., Zhang, X., Hilliges, O.: GazeNeRF: 3d-aware gaze redirection with neural radiance fields. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 9676-9685 (2023)
- 33. Recasens, A., Vondrick, C., Khosla, A., Torralba, A.: Following gaze in video. In: Int. Conf. Comput. Vis. pp. 1435-1443 (2017)
- 35. Senju, A., Johnson, M.H.: Atypical eye contact in autism: Models, mechanisms and development. Neurosci. Biobehav. Rev. 33 (8), 1204-1214 (2009)
- 36. Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient subpixel convolutional neural network. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1874-1883 (2016)

- 37. Tafasca, S., Gupta, A., Odobez, J.M.: ChildPlay: A new benchmark for understanding children's gaze behaviour. In: Int. Conf. Comput. Vis. pp. 20935-20946 (2023)
- 39. Tonini, F., Beyan, C., Ricci, E.: Multimodal across domains gaze target detection. In: Int. Conf. Multimodal Interact. pp. 420-431 (2022)
- 38. Tomas, H., Reyes, M., Dionido, R., Ty, M., Mirando, J., Casimiro, J., Atienza, R., Guinto, R.: Goo: A dataset for gaze object prediction in retail environments. In: IEEE Conf. Comput. Vis. Pattern Recog. Worksh. pp. 3125-3133 (2021)
- 40. Tonini, F., Dall'Asen, N., Beyan, C., Ricci, E.: Object-aware gaze target detection. In: Int. Conf. Comput. Vis. pp. 21860-21869 (2023)
- 42. Tu, D., Shen, W., Sun, W., Min, X., Zhai, G., Chen, C.: Un-Gaze: A unified transformer for joint gaze-location and gaze-object detection. IEEE Trans. Circuit Syst. Video Technol. (2023)
- 41. Tu, D., Min, X., Duan, H., Guo, G., Zhai, G., Shen, W.: End-to-end human-gazetarget detection with transformers. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 2192-2200 (2022)
- 43. Wang, B., Guo, C., Jin, Y., Xia, H., Liu, N.: Transgop: Transformer-based gaze object prediction. AAAI Conf. Artif. Intell. (2024)
- 45. Wang, K., Ji, Q.: Real time eye gaze tracking with 3D deformable eye-face model. In: Int. Conf. Comput. Vis. pp. 1003-1011 (2017)
- 44. Wang, B., Hu, T., Li, B., Chen, X., Zhang, Z.: GaTector: A unified framework for gaze object prediction. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 1958819597 (2022)
- 46. Wang, X., Zhang, H., Wang, Z., Nie, W., Yang, Z., Ren, W., Xu, Q., Xu, X., Liu, H.: Dual regression-enhanced gaze target detection in the wild. IEEE Trans. Cybern. 54 (1), 219-229 (2024)
- 48. Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.Y.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection. Int. Conf. Learn. Represent. (2022)
- 47. Wang, Z., Zhao, J., Lu, C., Yang, F., Huang, H., Guo, Y., et al.: Learning to detect head movement in unconstrained remote gaze estimation in the wild. In: IEEE Winter Conf. Appl. Comput. Vis. pp. 3443-3452 (2020)
- 49. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Appearance-based gaze estimation in the wild. In: IEEE Conf. Comput. Vis. Pattern Recog. pp. 4511-4520 (2015)
- 51. Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., Wang, J.: Fast segment anything. arXiv preprint arXiv:2306.12156 (2023)
- 50. Zhang, X., Sugano, Y., Fritz, M., Bulling, A.: Mpiigaze: Real-world dataset and deep appearance-based gaze estimation. IEEE Trans. Pattern Anal. Mach. Intell. 41 (1), 162-175 (2017)
- 52. Zhu, W., Deng, H.: Monocular free-head 3D gaze tracking with deep learning and geometry constraints. In: Int. Conf. Comput. Vis. pp. 3143-3152 (2017)
- 53. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers for end-to-end object detection. In: Int. Conf. Learn. Represent. (2021)

## Supplementary Materials of Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model

## A Gaze object detection compared to GTR [42]

Recently, Tu et al . proposed GTR [42] to jointly predict gaze location and gaze object but only did limited validation for gaze object detection on the GOO-Real dataset. In GTR [42], the AP of object detection is used to evaluate gaze object detection, and mean average precision (mAP) is used to evaluate the detection accuracy of multiple indicators. In the process of calculating mAP, a predicted gaze instance detection is considered a true positive if and only if it is highly accurate ( i.e ., the IoU between the predicted human head box and ground truth is greater than 0.5 while the L2 distance is less than 0.15 and confidence for gaze object is larger than 0.75), and only calculate AUC and L2 distance for those predicted instances considered as true positives, thereby ignoring relatively poor detection instances, resulting in an incomplete evaluation. For a fair comparison, we also report the performance of our model using the same metric in Table 10. Our model outperforms GTR by 19.06 (77.32% vs . 58.26%) and 18.08 (75.20% vs . 57.12%) in real-world and non-real-world settings respectively. At the same time, mAP also increased by 0.183 (0.810 vs . 0.627) in real-world setting compared to GTR, which indicates that compared to directly using the Transformer to globally model the relationship between head and gazed location, our special attention to the human head produced by RoI reconstruction module and space-to-object gaze regression approach can more effectively perceive the gaze direction and search gaze object in the corresponding area, achieving more accurate detection results.

Table 10: Gaze object detection results on GOO-Real dataset. For a fair comparison, we use the same evaluation metrics as GTR [42]. ∗ represents the results provided by GTR [42].

|                      | Non-Real   | Non-Real   | Non-Real   | Real   | Real   | Real   | Real   |
|----------------------|------------|------------|------------|--------|--------|--------|--------|
| Method               | AP         | AP 50      | AP 75      | AP     | AP 50  | AP 75  | mAP    |
| GazeFollow [32] ∗    | 35.5       | 58.9       | 26.7       | 31.6   | 54.3   | 24.1   | 0.291  |
| Lian [24] ∗          | 35.7       | 62.8       | 29.4       | 32.1   | 58.4   | 27.2   | 0.297  |
| VideoAttention [9] ∗ | 36.8       | 61.5       | 30.8       | 32.4   | 56.7   | 27.9   | 0.314  |
| GaTector [44] ∗      | 52.25      | 91.92      | 55.34      | 48.70  | 86.40  | 52.10  | 0.437  |
| GTR [42] ∗           | 58.26      | 97.31      | 58.42      | 57.12  | 96.47  | 58.90  | 0.627  |
| Ours                 | 77.32      | 98.65      | 93.91      | 75.20  | 98.94  | 93.20  | 0.810  |

## B Energy aggregation loss

In this paper, following TransGOP [43], we use an energy aggregation loss to further guide the gaze heatmap to converge toward the location of the gazed object. Since we perform gaze object segmentation, we replace the bounding box with a mask for calculation to more accurately measure the error, which can be formulated as:

$$\mathcal { L } _ { \text{eng} } = 1 - \frac { 1 } { S _ { m a s k } } \sum _ { ( i, j ) \in \text{mask} } \mathbf M _ { i, j } \quad \quad ( 1 1 )$$

where S mask represents the area of the gaze object mask and M represents the gaze heatmap.

## C Experiments Settings

Datasets: We conduct a comprehensive evaluation of our model on the GOOSynth and GOO-Real datasets. The GOO-Synth dataset contains 192,000 synthetic images, 172,000 images for training, and 2,0000 images for testing. Each image contains approximately 80 objects distributed in 24 categories and each object has accurate bounding box annotation. In addition, the bounding box of the human head in the image and its corresponding gaze point and gaze object bounding box are also annotated. The GOO-Real contains 9,552 real-world images and has the same annotation information as the GOO-Synth dataset. Both GOO-Synth and GOO-Real image scenes contain a large number of small and dense objects and the subjects always face away from the camera, which is very challenging for gaze estimation, gaze object segmentation, and detection.

Metrics: Our mask-based mSoC calculation method is defined as follows:

Implementation Details: The backbone B is a ResNet-50 [12] pre-trained on ImageNet [21]. We use MaskDINO [22] as the detection transformer to extract pixel-level mask features and detect the bounding boxes and instance masks of candidate gaze objects. We train all our components with AdamW [25] optimizer and a learning rate of 0.0001 for 75 epochs with a batch size of 2. In the main text Eq. (4), we set α β , , and γ as 10, 1000, and 1 respectively. We set the pixel shuffle scale factor η as 2, the input size as 224 × 224 , and the predicted heatmap size as 64 × 64 . All experiments are implemented based on the PyTorch and one GeForce RTX 3090Ti GPU.

$$\ m S o C _ { m a s k } = \min ( \frac { p _ { m a s k } } { a }, \frac { g _ { m a s k } } { a } ) \times \frac { p _ { m a s k } \cup g _ { m a s k } } { a }. \quad \quad ( 1 2 )$$

where p mask and g mask represent the areas of the prediction mask and ground truth mask respectively, and a represent the area of the minimum enclosing rectangle of p mask and g mask . It is worth noting that compared to using bounding box calculations, mask-based calculation methods are more restrictive because pixel-level positioning is more accurate.

Table 11: Analysis about RoI reconstruction module

|                       | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Object Segmentation Instance Segmentation   | Gaze Estimation   | Gaze Estimation   | Gaze Estimation   |
|-----------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|--------------------------------------------------|-------------------|-------------------|-------------------|
| RoI-Pooling RoI-Align | mSoC mSoC 50                                     | mSoC mSoC 50                                     | mSoC 75                                          | AP                                               | AP 50                                            | AP 75                                            | AUC ↑             | Dist. ↓           | Ang. ↓            |
| ✓                     | 84.21                                            | 98.90                                            | 96.57                                            | 81.36 98.84                                      | 81.36 98.84                                      | 94.52                                            | 0.942             | 0.092             | 15.1              |
| ✓                     | 85.94                                            | 98.99                                            | 97.20                                            | 82.86 99.02                                      | 82.86 99.02                                      | 95.57                                            | 0.944             | 0.088             | 14.7              |

Fig. 7: Impact of hyper-parameter η , α , and γ .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-f3d5f3fd-01dc-4e47-9a12-5401134694b9.jpg)

## D Analysis about RoI reconstruction module

In this paper, we reconstruct head region features based on overall features to improve the efficiency and flexibility of the model in the real world. We explore the effectiveness of different reconstruction methods. As shown in Table 11, we first use RoI pooling to crop head features directly from the original features, which obtains comparable results. Based on this, we further use RoI align to reconstruct the head feature map, which uses more detailed bilinear interpolation to restore feature resolution and achieve more detailed feature representation.

## E Analysis about hyper-parameters η , α , and γ

To explore the optimal performance of the model and sensitivity, we adjusted the hyper-parameters in the model on the GOO-Real dataset. We first adjusted the pixel shuffle scale factor η . As shown in Fig 7(a), the gaze object prediction and gaze estimation performance achieve best results when η = 2. When the feature map continues to be enlarged, the performance decreases instead, which shows that using smaller inputs to generate too large feature maps will introduce noise in the learning process, resulting in unclear representations. We explore the sensitivity of gaze direction loss in Fig 7(b). The model achieves optimal performance when α equals 10. Fig 7(c) shows the impact of aggregation energy

loss weight γ . When γ equals 1, the model performance is optimal. When γ continues to increase, the model performance will affect the performance of multitask learning, resulting in performance degradation.

## F More visualization

Fig 8 shows the visualization results of our method and previous methods. GaTector and TransGOP regress inaccurate gaze heatmaps due to the lack of spatial perception and detailed features understanding of object locations. In contrast, our model achieves more accurate gaze object positioning through pixel-level prediction and space-to-object gaze regression method, thereby alleviating semantic ambiguity while regressing a more accurate gaze heatmap. Fig 9 and Fig 10 show the visualization results of our method on GOO-Real and GOOSynth datasets respectively. Our method can regress accurate gaze heatmap for gaze objects and achieve pixel-level prediction. Our model still achieves relatively accurate predictions even when the person's back is turned to the camera. We also show failure cases, mainly because the person is relatively far away from the object, making it difficult to model the pixel-level gaze relationship, which indicates optimization directions for future works.

Fig. 8: Visualization of our real-world model and previous non-real world model on GOO-Real dataset. We show the results of ground truth gaze object (GT), gaze object detection (GOD), instance segmentation (IS), gaze object segmentation (GOS), and previous methods.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-3544f21d-38e5-4758-8484-4a7035492ce7.jpg)

Fig. 9: Visualization of our method in Real-World setting on GOO-Real dataset. We show the results of ground truth gaze object (GT), gaze object detection (GOD), instance segmentation (IS), and gaze object segmentation (GOS). The last row shows the failure case.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-b9201af2-eca7-4bfb-b6ae-cdbc3734c27d.jpg)

Fig. 10: Visualization of our method in Real-World setting on GOO-Synth dataset. We show the results of ground truth gaze object (GT), gaze object detection (GOD), instance segmentation (IS), and gaze object segmentation (GOS). The last row shows the failure case.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.01044v1-figure-8f876c15-fdea-4e1a-b1dd-c22b542e5170.jpg)
Yang Jin Lei Zhang Shi Yan Bin Fan Binglu Wang
Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model
Gaze object prediction (GOP) aims to predict the category and location of the
object that a human is looking at. Previous methods utilized box-level
supervision to identify the object that a person is looking at, but struggled
with semantic ambiguity, ie, a single box may contain several items since
objects are close together. The Vision foundation model (VFM) has improved in
object segmentation using box prompts, which can reduce confusion by more
precisely locating objects, offering advantages for fine-grained prediction of
gaze objects. This paper presents a more challenging gaze object segmentation
(GOS) task, which involves inferring the pixel-level mask corresponding to the
object captured by human gaze behavior. In particular, we propose that the
pixel-level supervision provided by VFM can be integrated into gaze object
prediction to mitigate semantic ambiguity. This leads to our gaze object
detection and segmentation framework that enables accurate pixel-level
predictions. Different from previous methods that require additional head input
or ignore head features, we propose to automatically obtain head features from
scene features to ensure the model's inference efficiency and flexibility in
the real world. Moreover, rather than directly fuse features to predict gaze
heatmap as in existing methods, which may overlook spatial location and subtle
details of the object, we develop a space-to-object gaze regression method to
facilitate human-object gaze interaction. Specifically, it first constructs an
initial human-object spatial connection, then refines this connection by
interacting with semantically clear features in the segmentation branch,
ultimately predicting a gaze heatmap for precise localization. Extensive
experiments on GOO-Synth and GOO-Real datasets demonstrate the effectiveness of
our method.