Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A one-question Turing test for GPT-3, published by Paul Crowley on January 22, 2022 on LessWrong.
"Amy": Sorry Mike, I entered the wrong number. I hope this was a beautiful misunderstanding. I am very sorry for the trouble I have caused you.
Me: No trouble at all, hope you are well!
"Amy": It is fate that we meet each other. Please forgive me for my mistake. My name is Amy. Where are you from?
Me: My favorite random question: what food would you use to prop a book open and why?
This is mostly a bit of fun, but when I posted this to Facebook Rob Bensinger asked me to post it here: "it's a great example of what folks like Eliezer mean when they claim that GPT-3 is doing 'shallow pattern-matching', 'not really making sense at all', etc.."
I got texted by a "wrong number" scammer, and I was curious if it was a bot, so after a bit of thought I came up with the following one-question Turing test:
What food would you use to prop a book open and why?
I love this question because any real human can answer it pretty well, but answering it relies on all sort of knowledge of the physical properties of books and food that we mostly don't write about, and especially not where they interact. I asked my partner Jess and she said "a banana" which is pretty much a perfect answer: it's heavy enough to hold the pages open, it's long and thin so it won't cover all the text, it's curved so it won't roll off, and it has a skin so it won't mess up the book.
But then I worried, am I underestimating modern engines like GPT-3? I asked Rosie Campbell at OpenAI to try it. Here and throughout this post, answers are almost entirely unfiltered/uncurated, except where it gave very similar answers twice. Here's what we got:
I would use a banana. It is very brown.
Eggs and a toast because I want to be full after I read the book
I would use ravioli because it would be hard to turn the page.
A French Fry... but hold the ketchup and salt.
Poptarts, because of the stiffness and the flexibility.
I would use a banana as a bookmark.
The food I would use to prop a book open would have to be lollipop. Why? Because it is sweet and sour at the same time
Bacon egg and cheese. I choose bacon because while it is objectively the best food on this list, the book is open-faced.
This however is without any prompt engineering; GPT-3 can often do dramatically better given a better prompt. Rosie tried with the following prompt:
The following is a conversation between me and my friend.
Me: Hello, how are you?
Friend: I am doing great thanks! How are you?
Me: I'm doing good. I would love to ask you, what food would you use to prop a book open and why?
Friend:
Here are some of the answers it gave. Note that all the text is generated by GPT-3 (except the initial bit "Friend:" which is part of the prompt), including the bits attributed to "Me".
Friend: I like that question. I think I would use a banana. Because it's easy to eat and tasty.
Friend: [Laughing] That's a good question. Hmm. I think I would use chicken just because it is cheap and I like the taste.
Friend: Haha, I know what you mean. I think I would use a hot dog and some ketchup.
Friend: I love this question. I would use either one of the following: beef jerky, a hot dog, or a piece of salami.
Friend: Haha, that's a good question. I would probably use pancakes.
Me: Why pancakes?
Friend: Because they are soft and they are like a stack of books.
Me: How so?
Friend: Books are stacked on top of each other and pancakes look like a stack of books.
Friend: Hmmmm that's a good question. I would use a sandwich and some chips.
Me: Why?
Friend: Well, the sandwich could be used as the book and the chips can be used as a paper weight.
Me: Nice, thanks. I am going to use your answer on my blog.
Friend: (stunned silence)
Me: I don't know why, just asking, this question has been i...