GUI-G^2: Gaussian Reward Modeling for GUI Grounding
Graphical User Interface (GUI) grounding maps natural language instructions
to precise interface locations for autonomous interaction. Current
reinforcement learning approaches use binary rewards that treat elements as
hit-or-miss targets, creating sparse signals that ignore the continuous nature
of spatial interactions. Motivated by human clicking behavior that naturally
forms Gaussian distributions centered on target elements, we introduce GUI
Gaussian Grounding Rewards (GUI-G^2), a principled reward framework that
models GUI elements as continuous Gaussian distributions across the interface
plane. GUI-G^2 incorporates two synergistic mechanisms: Gaussian point
rewards model precise localization through exponentially decaying distributions
centered on element centroids, while coverage rewards assess spatial alignment
by measuring the overlap between predicted Gaussian distributions and target
regions. To handle diverse element scales, we develop an adaptive variance
mechanism that calibrates reward distributions based on element dimensions.
This framework transforms GUI grounding from sparse binary classification to
dense continuous optimization, where Gaussian distributions generate rich
gradient signals that guide models toward optimal interaction positions.
Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro
benchmarks demonstrate that GUI-G^2, substantially outperforms
state-of-the-art method UI-TARS-72B, with the most significant improvement of
24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides
superior robustness to interface variations and enhanced generalization to
unseen layouts, establishing a new paradigm for spatial reasoning in GUI
interaction tasks.
687ef89633947f780d9b4aad
Fei Tang
687ef89633947f780d9b4aae
Zhangxuan Gu
claimed_verified
2025-07-23T08:37:19.203Z
60d2a2984956988b63753371
/avatars/412879e82a5bfc88431d8aa561acf26a.svg
zhangxgu
user
zhangxgu
687ef89633947f780d9b4aaf
Zhengxi Lu
claimed_verified
2025-07-22T07:49:15.897Z
676127cf11b19ea602bb202a
/avatars/dfd802a24bd63e509728159ebb1769f6.svg
Zhengxi Lu
user
LZXzju
687ef89633947f780d9b4ab0
Xuyang Liu
687ef89633947f780d9b4ab1
Shuheng Shen
687ef89633947f780d9b4ab2
Changhua Meng
687ef89633947f780d9b4ab3
Wen Wang
687ef89633947f780d9b4ab4
Wenqi Zhang
687ef89633947f780d9b4ab5
Yongliang Shen
claimed_verified
2025-07-22T07:49:12.501Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
687ef89633947f780d9b4ab6
Weiming Lu
687ef89633947f780d9b4ab7
Jun Xiao
687ef89633947f780d9b4ab8
Yueting Zhuang
GUI-G^2: Gaussian Reward Modeling for GUI Grounding
Graphical User Interface (GUI) grounding maps natural language instructions
to precise interface locations for autonomous interaction. Current
reinforcement learning approaches use binary rewards that treat elements as
hit-or-miss targets, creating sparse signals that ignore the continuous nature
of spatial interactions. Motivated by human clicking behavior that naturally
forms Gaussian distributions centered on target elements, we introduce GUI
Gaussian Grounding Rewards (GUI-G^2), a principled reward framework that
models GUI elements as continuous Gaussian distributions across the interface
plane. GUI-G^2 incorporates two synergistic mechanisms: Gaussian point
rewards model precise localization through exponentially decaying distributions
centered on element centroids, while coverage rewards assess spatial alignment
by measuring the overlap between predicted Gaussian distributions and target
regions. To handle diverse element scales, we develop an adaptive variance
mechanism that calibrates reward distributions based on element dimensions.
This framework transforms GUI grounding from sparse binary classification to
dense continuous optimization, where Gaussian distributions generate rich
gradient signals that guide models toward optimal interaction positions.
Extensive experiments across ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro
benchmarks demonstrate that GUI-G^2, substantially outperforms
state-of-the-art method UI-TARS-72B, with the most significant improvement of
24.7% on ScreenSpot-Pro. Our analysis reveals that continuous modeling provides
superior robustness to interface variations and enhanced generalization to
unseen layouts, establishing a new paradigm for spatial reasoning in GUI
interaction tasks.
A new reward framework, GUI-G$^2$, models GUI elements as continuous Gaussian distributions to improve autonomous interaction through dense gradient signals, outperforming existing methods in spatial reasoning tasks.
reinforcement learning binary rewards Gaussian distributions GUI Gaussian Grounding Rewards Gaussian point rewards coverage rewards adaptive variance mechanism continuous optimization spatial reasoning GUI interaction tasks