LocalLLaMA

4 readers

4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago

MODERATORS

communick@poweruser.forum

Another test of logical ability for LLMs? (alien.top)

submitted 1 year ago by laca_komputilulo@alien.top to c/localllama@poweruser.forum

14 comments fedilink hide all child comments

Found this in a children's book of riddles:

Six brothers were spending their time together.

The first brother was reading a book.
The second brother was playing chess.
The third brother was solving a crossword.
The fourth brother was watering the lawn.
The fifth brother was drawing a picture.

Question: what was the sixth brother doing?

I cant get ChatGPT to answer correctly with the usual tricks, even after hinting to consider one and two-person activities and emphasizing the word "together".

After a bunch of CoT turns we arrive to a conclusion that this is an open ended question and not a riddle :)

After trying 3 times with fresh promots, I got a correct response once, but when prompted to provide supporting reasoning the model backtracked and started apologizing.

Cant test gpt 4 r/n...

you are viewing a single comment's thread
view the rest of the comments

[–] Be-Kind_Always-Learn@alien.top 1 points 1 year ago

Another (less?) open-ended question with the same premise would be "Where are they?" and I expect the answer to be "In a garden".

Perhaps there's a language barrier here, but none of those activities hint to a garden? In my locale, a garden is a small patch used to grow veggies, herbs, and/or flowers. So I would answer this with "their back yard."

This is a much better riddle for children IMO, because it's barely open-ended at all. The original has almost infinite answers without any leaps or tricks, but yours has a very limited domain: a yard/garden. Though if someone were extra clever, the problem space does open back to nearly infinity (if brother 4 is playing a video game).

Open-ended question are the best for evaluating LLM, because they require common sense/world knowledge/doxa/human like behavior.

For personal testing, that's certainly a valid opinion! But it's not very productive from an objective standpoint because it can't be graded and tests a "gotcha" path of thinking, when we're still focusing on fundamentals like uniform context attention, consistency over time, etc.