This research paper describes a new benchmark called TheAgentCompany, which is like a video game that tests how well AI agents can do tasks you'd find in a real software company. These tasks include things like writing code, managing projects, and working with other people. The researchers built a fake software company with websites, documents, and even pretend coworkers for the AI to interact with. They tested a bunch of different AI models, including some famous ones like Claude and Gemini, but found that even the best AI was only able to fully complete 24% of the tasks. The researchers learned that AI is still not very good at tasks that need common sense, social skills, or the ability to use complicated websites, especially ones with lots of buttons and menus. This research helps us understand what AI is good at and where it still needs to improve before it can really be helpful in our workplaces.

https://arxiv.org/pdf/2412.14161