LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

LLMs and basic maths (alien.top)

submitted 1 year ago by WorldlinessSpecific9@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

I have been evaluating a number of maths models. One simple question I as is "what is 2 to the power of 4.1?" Almost every model butchers the answer. GPT-4 is the only one to get it correct out of the box. It looks like questions such as this are just not meant for LLMs. Without basic arithmetic, LLMs will not be particularly useful to any highly numeric occupations. Has anyone managed to get any finetuned LLMs to perform arithmetic reliably?

I am starting to think that the only way to do this is outsource specific calculations to a mathematical expression parser.

you are viewing a single comment's thread
view the rest of the comments

[–] ttkciar@alien.top 1 points 1 year ago

I am starting to think that the only way to do this is outsource specific calculations to a mathematical expression parser.

That's more or less how I'd like to tackle the problem, by generalizing "Guided Generalization" into a plugin system which can use any user-specified symbolic logic to guide inference by pruning token choices.

Math tools and logical provers seem like gimmes for such application.

llama.cpp already has GG hooks for implementing "grammars". That seems like a good place to implement such a plugin system.