Well then… Thanks! I’ll use llama.cpp and be happy. Glad to hear llamacpp_hf is crazy and not me. Which tool do you prefer outside of Oobabooga?
silenceimpaired
joined 1 year ago
It’s only been a day but have you changed? I find this model misspells a lot with the gguf i downloaded.
So helpful… but Yi and llamacpp_hf just falls apart for me… complete gibberish on Oobabooga. Exl hf … fine. Llama.cpp fine… Min-P is there and I can apparently use it but temperature last is missing :/
I’m on Pop, lol. I could get it to compile, but I must have missed a step for nvidia acceleration
It is… but koboldcpp doesn’t have a executable for me to run :/
I could never get up and running on Linux with Nvidia. I used Kobold on Windows, but boy is it painful on Linux.
Please try 70b down to ~30b with a llama 2 model. Thanks!
Exciting and worrying… I have gone to great efforts to use safetensors… I would have to see every model packaged in executable format… but then again I have seen comments about llama.cpp behavior changing for the same model and settings (not sure if it is true but that could be bad)