LocalLLaMA

4 readers

4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago

MODERATORS

communick@poweruser.forum

Has Anyone Successfully Utilized the Neural Networks API on Android for LLMS with EdgeTPU? (alien.top)

submitted 2 years ago by dewijones92@alien.top to c/localllama@poweruser.forum

3 comments fedilink hide all child comments

Hey everyone,

I've been exploring running lightweight machine learning models (LLMS) on Android devices and came across the Neural Networks API, which seems promising for leveraging neural silicon ASICs.

I own a Google Pixel 7, which features an EdgeTPU integrated into the chip. I'm curious if anyone here has experience or success stories using this API for AI inference, particularly in comparison to traditional CPU performance.

My main interest lies in understanding the practical performance gains when using the EdgeTPU for AI tasks. Does it significantly outperform the CPU in terms of inference speed or efficiency?

I'm especially keen to hear from those who have experimented with similar setups or have insights into optimizing LLMS on Android devices using this technology.

Thanks in advance for your insights and experiences!

you are viewing a single comment's thread
view the rest of the comments

[–] Combinatorilliance@alien.top 1 points 2 years ago

I'm very interested in learning more as well.

Do you know how these edge tpus compare to the coral tpu? There are some people who tried it here on localllama