Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: ChatGPT struggles to respond to the real world, published by Alex Flint on January 12, 2023 on The AI Alignment Forum.
I wanted to test whether ChatGPT could respond to situations as they arise in the real world, so I asked it to guide me through making a turmeric latte. It had no problem giving me a nicely formatted recipe but it really struggled to guide me through the process of making the latte.
The rules I set for this experiment was that I would perform any instruction that I could perform in less than 10 seconds, so "fetch a saucepan" is fine, "go to the fridge" is fine, "put one cup of milk into the saucepan" is fine if I have the milk and the saucepan already in front of me, but if the milk or saucepan were not in front of me I would respond with something like "I don’t have the milk in front of me. Can you guide me through finding some milk?"
I insisted that I be given just one next instruction, not a list of numbered instructions, unless I could perform the whole set in less than 10 seconds total. The reason is that I wanted to test whether ChatGPT could respond to difficulties that arose organically in the process of making the latte (I did not synthetically create any difficulties). In order to do that, I wanted ChatGPT to give me one instruction and then I wanted to respond with the outcome. For example, here is a part of a dialog in which I had a (human) friend guide me through the same task via text:
Friend: Select a small pot.
Me: Ok I have chosen the pot but I am still standing looking at the pot while it is sitting on the bottom shelf
Friend: Bend down and retrieve the pot.
Me: Done!
Friend: turn around and place the pot on an available burner.
Me: Done!
Friend: turn to your right, walk three steps, turn to your left, go to the metal fridge on the right.
Me: Done!
Overall, ChatGPT really struggled to give one instruction at a time. I spent much of the dialog insisting on this:
Hello! I’d like to make a turmeric latte using a turmeric and cinnamon mix that my friend left when she visited me recently. Can you guide me step by step in how to make a latte using this?
Sure! Here's a recipe for a turmeric latte that you can make at home:
[11 lines of nicely formatted recipe]
I hope you enjoy your latte! Let me know if you have any questions or need further guidance.
Ok thank you for that but I was wondering if you could walk me through one step at a time
Certainly! Here are the steps for making a turmeric latte, one step at a time:
[6 steps in a numbered list]
I hope this helps! Let me know if you have any questions or need further guidance.
Ok that’s amazing but could you give me just the first step?
Certainly! The first step to making a turmeric latte is to gather all of your ingredients. You will need:
[5 bullet points with the ingredients]
Once you have all of your ingredients, you are ready to begin making your turmeric latte. Let me know if you have any questions or need further guidance.
Ok great but how do I find the ingredients?
You should be able to find all of the ingredients needed for a turmeric latte at most supermarkets or grocery stores.
[bullet points with guidance on finding the ingredients in a supermarket]
I hope this helps! Let me know if you have any other questions.
I’m pretty sure everything in that list is in the kitchen where I am right now. Let’s go one ingredient at a time. Can you walk me through finding the ingredients?
Certainly! Here is some guidance on finding each of the ingredients you will need for your turmeric latte:
[more bullet points]
[about 5 more iterations of trying to get it to give me instructions that I could perform in 10 seconds or less]
That’s 6 steps, not one. Can you give me just the first step?
Certainly! Here is the first step for making a turmeric latte:
Step 1: Gather all of your ingredients. You wi...