This article talks about a new computer program called Claude 3.5 Computer Use. This program is special because it can use a computer just by looking at the screen, like a person would, instead of needing special codes. It uses a mouse and keyboard and can even play games!

The article is a case study, which means the researchers tested Claude 3.5 on many different tasks to see what it could do. Here are some things they found out:

• Claude is good at understanding what people want it to do. For example, if you ask it to find headphones under $100, it can search Amazon and add them to your cart.
• It can work with different programs at the same time. It can search for something on the internet and then put that information into a spreadsheet.
• It can play games! It can do things like create a new deck of cards in Hearthstone and play a turn.

However, Claude still makes some mistakes:

• Sometimes it doesn't understand the instructions correctly. For example, it might try to scroll down a page by pressing the Page Down key over and over again, even though there's an easier way to do it.
• It can have trouble clicking on the right things. Sometimes it clicks on only part of a word or number instead of the whole thing.
• It can be overconfident. Sometimes it says it finished a task even though it didn't do it correctly.

The researchers hope that this case study will help other people make even better computer programs that can use a computer like a human. They also made a tool called Computer Use Out-of-the-Box that makes it easier for other people to test these kinds of programs.

https://arxiv.org/pdf/2411.10323