overview for SupplyChainNext

Has anyone fine tuned a local model on their company database? in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 9 months ago

Fine tune + RAG seems to be the method we’re expected to go in. Need to fine tune for terminology and RAG for policy and procedure knowledge.

Tool to quickly iterate when fine-tuning open-source LLMs in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 9 months ago (1 children)

This is an amazing sub with amazingly talented individuals. I love it here. This is great.

What kind of specs to run local llm and serve to say up to 20-50 users in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 9 months ago

figure out the size and speed you need. Buy the Nvidia pro gpus (A series) x 20-50 + the server cluster hardware and network infrastructure needed to make them run efficiently.

Think in the several hundred thousand dollar range. I’ve looked into it.

Should corporations and smaller businesses be training, refining, and running local LLMs to avoid the future cost of the new cloud services M$ and Amazon are rolling out? in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 9 months ago

Yes.

Wanna upgrade PC for LLMs in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 10 months ago (1 children)

Meh I’m running 13bs on a 13900k 64 gb ddr5 and a 6900xt with LM studio and it’s faster than my office workstations 12900ks 3090ti. Sometimes ram and processor with decent VRM is enough.

What is the easiest way for a noob using a Windows PC to locally host their own AI? in c/localllama@poweruser.forum

[–] SupplyChainNext@alien.top 1 points 10 months ago

LM Studio (don't shoot me people)

Download, download your desired model and if you have the resources you're good to go.