dododragon

joined 10 months ago
[–] dododragon@alien.top 1 points 9 months ago

generate the sha256 hash using sha256sum your_model.gguf

rename your_model.gguf to "sha256:_hash_" (replace _hash_ with the actual hash)

move it to /usr/share/ollama/.ollama/models/blobs folder

copy a manifest from a similar model in /usr/share/ollama/.ollama/models/

manifests/registry.ollama.ai/library and update the hash & filesize to match your model in the "image.model" entry.

repeat last step for the params entry

you can call the manifest folder/file whatever you like

[–] dododragon@alien.top 1 points 9 months ago

Have a look at https://www.runpod.io/ for AI cloud hosting. You could do some testing based on the number of users you want to cater for, and see what capacity you'll get for your $.

Start with a basic plan, run some tests to see what it can handle and compare it as you scale up the number of users with simultaneous queries.

[–] dododragon@alien.top 1 points 10 months ago

I've recently just got into LLM's have you tried these math models? They seem to follow math related instructions reasonably well.

wizard-math:13b-q6_KMathLLM-MathCoder-CL-7B.Q8_0.ggufmetamath-mistral-7b.Q5_K_M.gguf