I tried everything at this point i think i am doing something wrong or i have discovered some very strange bug. i was thinking on posting on their github but i am not sure if i am not simply making a very stupid error.
```
in a fresh conda install set up with python 3.12
i used export LLAMA_CUBLAS=1
then i copied this:
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python
it runs without complaint creating a working llama-cpp-python install but without cuda support. I know that i have cuda working in the wsl because nvidia-sim shows cuda version 12.
i have tried to set up multiple environments i tried removing and reinstalling, i tried different things besides cuda that also dont work so something seems to be off with the backend part but i dont know what. Best guess i do something very basic wrong like not setting the environmental variable correctly or somthing.
when i reinstalled i used this option
pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir
Also it does simply not create the llama_cpp_cuda folder in so llama-cpp-python not using NVIDIA GPU CUDA - Stack Overflow does not seem to be the problem.
Hardware:
Ryzen 5800H
RTX 3060
16gb of ddr4 RAM
WSL2 Ubuntu
TO test it i run the following code and look at the gpu mem usage which stays at about 0
from llama_cpp import Llama
llm = Llama(model_path="/mnt/d/Maschine learning/llm models/llama_2_7b/llama27bchat.Q4_K_M.gguf", n_gpu_layers=20,
n_threads=6, n_ctx=3584, n_batch=521, verbose=True)
output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
So any help or idea what could be going on here would be of great help because i am out of ideas. Thank you very much :)
/u/Noxusequal
I just went through this exact issue building Llama.cpp inside an Ubuntu docker container. If you don't have nvcc installed it will compile without error, but wont include CUDA support regardless of what you set for options. Check to make sure nvcc is installed in the machine
jup that was part of it :) its working now tahnk you