Someone on reddit said, "Remember robots still can't go into a new house and make a coffee." And I thought

I actually wonder whether, if I provided the physical actuation, current LLMs would be capable of doing this? Like, through a conversation like:

Me: I'm in a house. Your job is to instruct me to make a coffee. I can take photos of my surroundings, I can follow basic directions, and if you ask me to do something too complicated I'll ask for clarification. Here is my current surroundings: (photo)

LLM: Okay, we need to find the kitchen. There's a door on the right of the photo, go through that.

Me: Here's where I am now: (photo)

LLM: That looks like the kitchen on the left, go there.

Me: It looks like this: (photo)

LLM: Now we need to find either a coffee maker or a kettle. Look through the cupboards.

Me: I don't know what those things look like.

LLM: Then open the cupboard on the left and show me a photo.

...and so on.

It wouldn't shock me either way if they can or can't do it. I think I weakly predict that the models have the [...]

The original text contained 3 footnotes which were omitted from this narration.

---

First published:

Source:

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.