How Anthropics new computer use ability could further AI automation [Video]

Anthropic’s has upgraded its Claude 3.5 Sonnet LLM with a new ability, computer use, opening up new opportunities in robotic process automation (RPA) and more.

Credit: T. Schneider / Shutterstock

Anthropic’s Claude 3.5 Sonnet large language model has gained a new ability: operating a computer.  

The new ability, which the company is calling “computer use,” is currently in beta test. It enables developers to instruct Claude 3.5 Sonnet, through the Anthropic API, to read and interpret what’s on the display, type text, move the cursor, click buttons, and switch between windows or applications — much as today’s robotic process automation (RPA) tools can be instructed — much more laboriously — to do.

To apply its ability to use a computer, Claude 3.5 Sonnet starts from a prompt defining its goal, identifies the steps necessary to reach that goal, and then scans screenshots much as a human would look at the screen of a computer to …