I find running an OpenAI style API endpoint (using llama.cpp directly when I want fine control, or StudioLM when I need something quick and easy) is the best way to go in combination with a good chat UI designed to interface with OpenAI models.

To that end, I redirect Chatbox to my local LLM server, and I LOVE IT. Clean but powerful interface, support for markdown, ability to save different agents for quick recall, and more. Highly, HIGHLY recommend it.

It's open source and available on pretty much every platform -- and you can use it to interface with both local LLM and with OpenAI LLM's.

[–] dr_nick_riveria@alien.top 1 points 9 months ago

What are you using for your local LLM server and any pointers you can share on how to redirect/point Chatbox to your local server?