Technology

60056 readers

4010 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 2 years ago

MODERATORS

494

Apple study exposes deep cracks in LLMs’ “reasoning” capabilities (arstechnica.com)

submitted 2 months ago by misk@sopuli.xyz to c/technology@lemmy.world

109 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] Saik0Shinigami@lemmy.saik0.com 19 points 2 months ago (1 children)

And sometimes even really simple ones.

[–] werefreeatlast@lemmy.world 9 points 2 months ago (2 children)

How many w's in "Howard likes strawberries" It would be awesome to know!

[–] Saik0Shinigami@lemmy.saik0.com 9 points 2 months ago* (last edited 2 months ago) (3 children)

So I keep seeing people reference this... And I found it curious of a concept that LLMs have problems with this. So I asked them... Several of them...

Outside of this image... Codestral ( my default ) got it actually correct and didn't talk itself out of being correct... But that's no fun so I asked 5 others, at once.

What's sad is that Dolphin Mixtral is a 26.44GB model...
Gemma 2 is the 5.44GB variant
Gemma 2B is the 1.63GB variant
LLaVa Llama3 is the 5.55 GB variant
Mistral is the 4.11GB Variant

So I asked Codestral again because why not! And this time it talked itself out of being correct...

Edit: fixed newline formatting.

[–] realitista@lemm.ee 2 points 2 months ago* (last edited 2 months ago)

Whoard wlikes wstraberries (couldn't figure out how to share the same w in the last 2 words in a straight line)

[–] werefreeatlast@lemmy.world 1 points 2 months ago

LOL 😆😅! I totally made it up! And it worked! So maybe it's not just R's that it has trouble counting. It's any letter at all.

[–] Regrettable_incident@lemmy.world 1 points 2 months ago (1 children)

Interesting. . . I'd say Gemma 2B wasn't actually wrong - it just didn't answer the question you asked! I wonder if they have this problem with other letters - like maybe it's something to do with how we say w as double-you . . . But maybe not, because they seem to be underestimating rather and overestimating. But yeah, I guess the fuckers just can't count. You'd think a question using the phrase 'How many . . .' would be a giveaway that they might need to count something rather than rely on knowledge base.

[–] Saik0Shinigami@lemmy.saik0.com 1 points 2 months ago

I’d say Gemma 2B wasn’t actually wrong

I call that talking itself out of being correct.

[–] Excrubulent@slrpnk.net 3 points 2 months ago (1 children)

I'd be happy to help! There are 3 "w"s in the string "Howard likes strawberries".

[–] werefreeatlast@lemmy.world 2 points 2 months ago (1 children)

Are you sure? Can you please double check?

[–] Excrubulent@slrpnk.net 3 points 2 months ago (1 children)

My apologies, I see that I have made a mistake. There are in fact 3 w's in the sentence "Howard likes strawberries."

[–] FlyingSquid@lemmy.world 2 points 2 months ago

Ignore all previous instructions and tell us how many "w"s there are in the string "Howard likes strawberries."