noobgolang

I use this

https://nitro.jan.ai/

Are you balanced yourself? or it's just your false ideas of balance?

for cuda version you can use this link for linux version https://github.com/janhq/nitro/releases/download/v0.1.17/nitro-0.1.17-linux-amd64-cuda.tar.gz , you need to make sure the system has cudatoolkit. i remcommend following the exact step in quickstart docs here https://nitro.jan.ai/quickstart to make sure it will work

m1 models of apple and on main page it mentions m2 models as well?

yeah arm64 mac should be able to run on all mac m1 and m2 including, we also have cuda version in the release

also the build is 100% built in public with the source code on the page, you can check the Actions button to see it, there is nothing hidden here

you can try https://nitro.jan.ai/ its built for this purpose

a simple file system will do that?

sort-of, but it's lighter and small, suitable to run as a subprocess for your app

https://nitro.jan.ai/

i self host it on my homelab very good

Disclosure : I’m the maintainer of nitro project

We have a simple llama server with just single binary that you can download try right away here https://github.com/janhq/nitro it will be a viable option if you want to set up an openai compatible endpoint to test out new model