I have tried to set up 3 different versions of it, TheBloke GPTQ/AWQ versions and the original deepseek-coder-6.7b-instruct .
I have tried the 34B as well.
My specs are 64GB ram, 3090Ti , i7 12700k
In AWQ I get just bugged response (""""""""""""""") until max tokens,
GPTQ works much better, but all versions seem to add unnecessary * at the end of some lines.
and gives worse results than on the website (deepseek.com) Let's say il ask for a snake game in pygame, it usually gives an unusable version, and after 5-6 tries il get somewhat working version but still il need to ask for a lot of changes.
While on the official website il get the code working on first try, without any problems.
I am using the Alpaca template with adjustment to match the deepseek version (oogabooga webui)
What can cause it? Is the website version different from the huggingface model?
Yeah I basically turn the temperature to 0.1, disable every sampler, and turn the temperature as low as the GUI will allow (I have mine at 1). I'm using Koboldcpp 1.50 and deepseek-coder-instruct 33b is working very well for me. If it's not on par with GPT-4, it's incredibly close. I tested the 7b model and it's pretty good, but it does mess up more frequently requiring you to fix its mistakes. 33b gives me workable code more often than not.
I've been testing it on a bunch of different problems on this site: https://www.w3resource.com/index.php
...and it seems to ace everything I throw at it. Granted those aren't particularly challenging problems, but still, it's very consistent, which means I can use it for work reliably (I work mainly with SQL). The 16k context doesn't hurt either!
Now I just wish I had more than 8gb vram, cuz I'm getting like 1.3 Tokens per second, so I have to be super patient.
Also I just added it to my VSCode using the Continue extension (while the model runs using llamacpp). It works beautifully there too (once you configure the prompt correctly to what the model expects). If you use VSCode at all, you can now have a really good copilot for free.
Oh nice! I'll have to try those settings and compare with the StarChat preset in Oobabooga. I hear ya, I get 1t/s too.. it's unbearable.
I miss the days when high-end gpu's were like $400-500! I'm not made of moneys, and I also use a laptop, so the most I could buy right now would be 16gb vram anyway. I'll probably save up and wait for next gen and see if they make any headway there.
Thank you for the response!
Il try to adjust the temp too, how can I disable samplers in oobabooga? what is the setting?
Is there a way to set rep penalty lower than 1?
unfortunately I haven't used ooba in a few months so I can't tell you, but in koboldcpp it just tells you what values disable the samplers.