LocalLLaMA

1 readers

1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 10 months ago

MODERATORS

communick@poweruser.forum

Having a hard time setting deepseek coder instruct to work (alien.top)

submitted 10 months ago by iChrist@alien.top to c/localllama@poweruser.forum

17 comments fedilink hide all child comments

I have tried to set up 3 different versions of it, TheBloke GPTQ/AWQ versions and the original deepseek-coder-6.7b-instruct .

I have tried the 34B as well.

My specs are 64GB ram, 3090Ti , i7 12700k

In AWQ I get just bugged response (""""""""""""""") until max tokens,

GPTQ works much better, but all versions seem to add unnecessary * at the end of some lines.

and gives worse results than on the website (deepseek.com) Let's say il ask for a snake game in pygame, it usually gives an unusable version, and after 5-6 tries il get somewhat working version but still il need to ask for a lot of changes.

While on the official website il get the code working on first try, without any problems.

I am using the Alpaca template with adjustment to match the deepseek version (oogabooga webui)

What can cause it? Is the website version different from the huggingface model?

you are viewing a single comment's thread
view the rest of the comments

[–] TobyWonKenobi@alien.top 1 points 10 months ago (1 children)

If you are using it on LM Studio, I think you need to upgrade to the latest Beta, which includes a fix.

I ran into the same issues with Deepseek Gguf

[–] nullnuller@alien.top 1 points 10 months ago

I tried 0.28 and it only shows newlines in the server logs and nothing at all in the UI