Sign up to save your podcasts
Or
Share NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
🤗 Upvotes: 45 | cs.CV, cs.AI, cs.CL, cs.HC, cs.LG
Authors:
Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng
Title:
NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Arxiv:
http://arxiv.org/abs/2507.08800v1
Abstract:
We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.
View all episodes
By Jingwen Liang, Gengyu Wang🤗 Upvotes: 45 | cs.CV, cs.AI, cs.CL, cs.HC, cs.LG
Authors:
Luke Rivard, Sun Sun, Hongyu Guo, Wenhu Chen, Yuntian Deng
Title:
NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Arxiv:
http://arxiv.org/abs/2507.08800v1
Abstract:
We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a large-scale dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Although modeling fine-grained keyboard interactions precisely remains challenging, NeuralOS offers a step toward creating fully adaptive, generative neural interfaces for future human-computer interaction systems.