homeassistant

I love fully local stuff, but the LLM part seems very expensive. Even for such a simple thing as managing our lights and music.

Yeah, this is something I found out when I hacked my Amazon Echo and put LineageOS on it. I like the new interface that isn't constantly advertising Amazon bullshit as a screensaver, but it doesn't have a GPU, so attempting to put any sort of voice component takes many seconds to try to process.

And in this post-memory-crisis economy, it's not quite the right time to buy a dedicated LLM processing rig for my house. But, this really needs to be the route in the future. Or just install a NPU on this speaker itself. Phones already do this, and it's been the standard since 2017.

I'm just gonna shamelessly plug my project here: https://github.com/charludo/hass-closest-intent

You do need local STT, but it's fine if it's somewhat bad; the conversation agent linked is sufficient to "get" what you meant, without LLM.

I love fully local stuff, but the LLM part seems very expensive. Even for such a simple thing as managing our lights and music.

So, the simple things should just be voice command (not recognition, limited vocabulary), kitchen light on, what's todays weather (assuming you automatically download it and have it there, needs TTS like whisper). That can run on a recent potato. Somewhat more complex things need full voice recognition e.g. Play me x songname by y artist. Laptop CPU should handle that fine. You'll only need a full LLM for more general inquiries, e.g. What mess has Trump made today?, which will need web access seeing as small LLMs don't have a lot of world knowledge (although you can self host Wikipedia and point it at that).

If you have a desktop computer and a GPU with 16+GB VRAM (or even smaller using RAM and a MoE LLM) that'll do the job with reasonable smarts e.g. Qwen 3.6 35BA3B, Gemma 4 12B or 26BA4B. You may have to wait a minute or two. Mine pulls around 55W at idle (7800XT) and does double duty as a NAS and gaming rig, or you can have it sleep most of the time and let the laptop wake it up when necessary.

I’m thinking the same thing. I hate the idle power draw of my home server and want to migrate back to arm but have the x86 as a fallback node for games in whales and jellyfin transcodes, but it just doesn’t make sense to wake and sleep constantly for speech recognition.

Get a Strix Halo box to be your headless AI server

They're around 4k USD these days though...

I'm working on this exact problem at home now, and finding 24GB of ddr4, a 6th-gen i7, and an old 8GB rx580 is more than enough to run pretty capable models up to ~35B parameters just fine, and equivalent MoE models shockingly fast. Not exactly cheap, with the price of everything so crazy right now, but certainly cheaper altogether than I'd imagined.

http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-6-Plus.html

https://www.notebookcheck.net/Orange-Pi-6-vs-Raspberry-Pi-45-TOPS-AI-dual-2-5-Gbit-LAN-LPDDR5-and-dual-PCIe-4-SSDs.1323900.0.html

Maybe something like this could work for you? (If you actually can find it in stock anywhere lol)

Ai on arm is getting better and better, ive tried it on a radxa 5b and got almost ok chat speed for an 8b model, and that is only 6 tops