LocalLLaMA

3 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago

MODERATORS

communick@poweruser.forum

What are your thoughts on the future of LLMs running mobile? (alien.top)

submitted 11 months ago by Tree-Sheep@alien.top to c/localllama@poweruser.forum

17 comments fedilink hide all child comments

Following the release of Dimensity 9300 and S8G3 phones, I am expecting growth in popularity of LLMs running on mobile phones, as quantized 3B or 7B models can already run on high-end phones from five years ago or later. But despite it being possible, there are a few concerns, including power consumption and storage size. I've seen posts about successfully running LLMs on mobile devices, but seldom see people discussing about future trends. What are your thoughts?

you are viewing a single comment's thread
view the rest of the comments

[–] oe-g@alien.top 1 points 11 months ago (3 children)

My personal take is what are the use cases for user friendly local LLMs on mobile compared to higher performance llm closed models?

Privacy is the only serious benefit I can think of.

[–] Combinatorilliance@alien.top 1 points 11 months ago

It's not going to be just chat. The LLMs are going to be integrated into everything in the OS.

Suggesting emails, finding appointments in e-mail (I believe this already exists somewhat for Apple? In any case it will be private, local and more reliable), improved search, way improved personal assistant, APIs to access the model from any app. Lots of stuff...

[–] GraceRaccoon@alien.top 1 points 11 months ago

Privacy I don't care about too late for that lol. If it becomes as normal to use ai as it is to google something, my worry about be it intentionally using language to fuck with my head, or skew my perspective on something I'm trying to get info on. Social engineering is a spooky thing. Algorithms on social media are already causing damage lol.

[–] NDBellisario@alien.top 1 points 11 months ago (2 children)

Latency is one thing with the internet.

Any model that can run locally doesn’t need a round trip to a datacenter. This can of course depending on computer power

[–] Maykey@alien.top 1 points 11 months ago

At current capabilities it's faster to query server on the opposite hemisphere than to generate locally.

[–] CocksuckerDynamo@alien.top 1 points 11 months ago

round trip latency of an http request (or grpc or whatever pick your poison) is utterly insignificant compared to the time it takes to run the inference process, even for the smallest models with the fastest inference