NeuralOS: Towards Simulating Operating Systems via Neural Generative
  Models
We introduce NeuralOS, a neural framework that simulates graphical user
interfaces (GUIs) of operating systems by directly predicting screen frames in
response to user inputs such as mouse movements, clicks, and keyboard events.
NeuralOS combines a recurrent neural network (RNN), which tracks computer
state, with a diffusion-based neural renderer that generates screen images. The
model is trained on a large-scale dataset of Ubuntu XFCE recordings, which
include both randomly generated interactions and realistic interactions
produced by AI agents. Experiments show that NeuralOS successfully renders
realistic GUI sequences, accurately captures mouse interactions, and reliably
predicts state transitions like application launches. Although modeling
fine-grained keyboard interactions precisely remains challenging, NeuralOS
offers a step toward creating fully adaptive, generative neural interfaces for
future human-computer interaction systems.
6874615e257d4f043537028f
Luke Rivard
6874615e257d4f0435370290
Sun Sun
6874615e257d4f0435370291
Hongyu Guo
6874615e257d4f0435370292
Wenhu Chen
6874615e257d4f0435370293
Yuntian Deng
claimed_verified
2025-07-15T19:10:44.939Z
63081e15a670ed10f9d44229
https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
Yuntian Deng
user
yuntian-deng
NeuralOS: Towards Simulating Operating Systems via Neural Generative
  Models
We introduce NeuralOS, a neural framework that simulates graphical user
interfaces (GUIs) of operating systems by directly predicting screen frames in
response to user inputs such as mouse movements, clicks, and keyboard events.
NeuralOS combines a recurrent neural network (RNN), which tracks computer
state, with a diffusion-based neural renderer that generates screen images. The
model is trained on a large-scale dataset of Ubuntu XFCE recordings, which
include both randomly generated interactions and realistic interactions
produced by AI agents. Experiments show that NeuralOS successfully renders
realistic GUI sequences, accurately captures mouse interactions, and reliably
predicts state transitions like application launches. Although modeling
fine-grained keyboard interactions precisely remains challenging, NeuralOS
offers a step toward creating fully adaptive, generative neural interfaces for
future human-computer interaction systems.
NeuralOS, a neural framework combining RNNs and diffusion-based rendering, simulates realistic GUIs by predicting screen frames in response to user inputs.
recurrent neural network RNN diffusion-based neural renderer GUIs screen frames user inputs mouse movements clicks keyboard events Ubuntu XFCE recordings realistic GUI sequences mouse interactions state transitions application launches