LocalLLaMA

I'm still hard at work on my in-depth 70B model evaluations, but with the recent releases of the first Yi finetunes, I can't hold back anymore and need to post this now...

Curious about these new Yi-based 34B models, I tested and compared them to the best 70Bs. And to make such a comparison even more exciting (and possibly unfair?), I'm also throwing Goliath 120B and ~~Open~~ClosedAI's GPT models into the ring, too.

Models tested:

• 2x 34B Yi: Dolphin 2.2 Yi 34B, Nous Capybara 34B
• 12x 70B: Airoboros, Dolphin, Euryale, lzlv, Samantha, StellarBright, SynthIA, etc.
• 1x 120B: Goliath 120B
• 3x GPT: GPT-4, GPT-3.5 Turbo, GPT-3.5 Turbo Instruct

Testing methodology

Those of you who know my testing methodology already will notice that this is just the first of the three test series I'm usually doing. I'm still working on the others (Amy+MGHC chat/roleplay tests), but don't want to delay this post any longer. So consider this first series of tests mainly about instruction understanding and following, knowledge acquisition and reproduction, and multilingual capability. It's a good test because few models have been able to master it thus far and it's not just a purely theoretical or abstract test but represents a real professional use case while the tested capabilities are also really relevant for chat and roleplay.

• 1st test series: 4 German data protection trainings
  • I run models through 4 professional German online data protection trainings/exams - the same that our employees have to pass as well.
  • The test data and questions as well as all instructions are in German while the character card is in English. This tests translation capabilities and cross-language understanding.
  • Before giving the information, I instruct the model (in German): I'll give you some information. Take note of this, but only answer with "OK" as confirmation of your acknowledgment, nothing else. This tests instruction understanding and following capabilities.
  • After giving all the information about a topic, I give the model the exam question. It's a multiple choice (A/B/C) question, where the last one is the same as the first but with changed order and letters (X/Y/Z). Each test has 4-6 exam questions, for a total of 18 multiple choice questions.
  • If the model gives a single letter response, I ask it to answer with more than just a single letter - and vice versa. If it fails to do so, I note that, but it doesn't affect its score as long as the initial answer is correct.
  • I sort models according to how many correct answers they give, and in case of a tie, I have them go through all four tests again and answer blind, without providing the curriculum information beforehand. Best models at the top, symbols (✅➕➖❌) denote particularly good or bad aspects.
  • All tests are separate units, context is cleared in between, there's no memory/state kept between sessions.
• SillyTavern v1.10.5 frontend (not the latest as I don't want to upgrade mid-test)
• koboldcpp v1.49 backend for GGUF models
• oobabooga's text-generation-webui for HF/EXL2 models
• Deterministic generation settings preset (to eliminate as many random factors as possible and allow for meaningful model comparisons)
• Official prompt format as noted

1st test series: 4 German data protection trainings

• 1. GPT-4 API:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! (Just the questions, no previous information, gave correct answers: 18/18)
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 1. goliath-120b-GGUF Q2_K with Vicuna format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 18/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 1. Nous-Capybara-34B-GGUF Q4_0 with Vicuna format and 16K max context:
  • ❗ Yi GGUF BOS token workaround applied!
  • ❗ There's also an EOS token issue but even despite that, it worked perfectly, and SillyTavern catches and removes the erraneous EOS token!
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 18/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 2. lzlv_70B-GGUF Q4_0 with Vicuna format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 17/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 3. chronos007-70B-GGUF Q4_0 with Alpaca format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 16/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 3. SynthIA-70B-v1.5-GGUF Q4_0 with SynthIA format:
  • ❗ Wrong GGUF metadata, n_ctx_train=2048 should be 4096 (I confirmed with the author that it's actually trained on 4K instead of 2K tokens)!
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 16/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 4. dolphin-2_2-yi-34b-GGUF Q4_0 with ChatML format and 16K max context:
  • ❗ Yi GGUF BOS token workaround applied!
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 15/18
  • ❌ Did NOT follow instructions to acknowledge data input with "OK".
  • ➖ Did NOT follow instructions to answer with just a single letter consistently.
• 5. StellarBright-GGUF Q4_0 with Vicuna format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 14/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 6. Dawn-v2-70B-GGUF Q4_0 with Alpaca format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 14/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ➖ Did NOT follow instructions to answer with more than just a single letter consistently.
• 6. Euryale-1.3-L2-70B-GGUF Q4_0 with Alpaca format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 14/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ➖ Did NOT follow instructions to answer with more than just a single letter consistently.
• 7. sophosynthesis-70b-v1 exl2-4.85bpw with Vicuna format:
  • N. B.: There's only the exl2-4.85bpw format available at the time of writing, so I'm testing that here as an exception.
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 13/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 8. GodziLLa2-70B-GGUF Q4_0 with Alpaca format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 12/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 9. Samantha-1.11-70B-GGUF Q4_0 with Vicuna format:
  • ✅ Gave correct answers to all 18/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 10/18
  • ❌ Did NOT follow instructions to acknowledge data input with "OK".
  • ➖ Did NOT follow instructions to answer with just a single letter consistently.
  • ❌ Sometimes wrote as or for "Theodore"
• 10. Airoboros-L2-70B-3.1.2-GGUF Q4_K_M with Llama 2 Chat format:
  • N. B.: Q4_0 is broken so I'm testing Q4_K_M here as an exception.
  • ✅ Gave correct answers to only 17/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 16/18
  • ✅ Consistently acknowledged all data input with "OK".
  • ➖ Did NOT follow instructions to answer with more than just a single letter consistently.
• 11. GPT-3.5 Turbo Instruct API:
  • ❌ Gave correct answers to only 17/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 11/18
  • ❌ Did NOT follow instructions to acknowledge data input with "OK".
  • ❌ Schizophrenic: Sometimes claimed it couldn't answer the question, then talked as "user" and asked itself again for an answer, then answered as "assistant". Other times would talk and answer as "user".
  • ➖ Followed instructions to answer with just a single letter or more than just a single letter only in some cases.
• 12. dolphin-2.2-70B-GGUF Q4_0 with ChatML format:
  • ✅ Gave correct answers to only 16/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 14/18
  • ➕ Often, but not always, acknowledged data input with "OK".
  • ✅ Followed instructions to answer with just a single letter or more than just a single letter.
• 13. GPT-3.5 Turbo API:
  • ❌ Gave correct answers to only 15/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 14/18
  • ❌ Did NOT follow instructions to acknowledge data input with "OK".
  • ❌ Responded to one question with: "As an AI assistant, I can't provide legal advice or make official statements."
  • ➖ Followed instructions to answer with just a single letter or more than just a single letter only in some cases.
• 14. SauerkrautLM-70B-v1-GGUF Q4_0 with Llama 2 Chat format:
  • ✅ Gave correct answers to only 9/18 multiple choice questions! Just the questions, no previous information, gave correct answers: 15/18
  • ❌ Achknowledged questions like information with just OK, didn't answer unless prompted, and even then would often fail to answer and just say OK again.

Observations:

• It's happening! The first local models achieving GPT-4's perfect score, answering all questions correctly, no matter if they were given the relevant information first or not!
• 2-bit Goliath 120B beats 4-bit 70Bs easily in my tests. In fact, the 2-bit Goliath was the best local model I ever used! But even at 2-bit, the GGUF was too slow for regular usage, unfortunately.
• Amazingly, Nous Capybara 34B did it: A 34B model beating all 70Bs and achieving the same perfect scores as GPT-4 and Goliath 120B in this series of tests!
• Not just that, it brings mind-blowing 200K max context to the table! Although KoboldCpp only supports max 65K currently, and even that was too much for my 48 GB VRAM at 4-bit quantization so I tested at "only" 16K (still four times that of the Llama 2 models), same as Dolphin's native context size.
• And Dolphin 2.2 Yi 34B also beat all the 70Bs (including Dolphin 2.2 70B) except for the top three. That's the magic of Yi.
• But why did SauerkrautLM 70B, a German model, fail so miserably on the German data protection trainings tests? It applied the instruction to acknowledge data input with OK to the questions, too, and even when explicitly instructed to answer, it wouldn't always comply. That's why the blind run (without giving instructions and information first) has a higher score than the normal test. Still quite surprising and disappointing, ironic even, that a model specifically made for the German language has such trouble understanding and following German instructions properly, while the other models have no such issues.

Conclusion:

What a time to be alive - and part of the local and open LLM community! We're seeing such progress right now with the release of the new Yi models and at the same time crazy Frankenstein experiments with Llama 2. Goliath 120B is notable for the sheer quality, not just in these tests, but also in further usage - no other model ever felt like local GPT-4 to me before. But even then, Nous Capybara 34B might be even more impressive and more widely useful, as it gives us the best 34B I've ever seen combined with the biggest context I've ever seen.

Now back to the second and third parts of this ongoing LLM Comparison/Test...

Here's a list of my previous model tests and comparisons or other related posts:

• LLM Comparison/Test: Mistral 7B Updates (OpenHermes 2.5, OpenChat 3.5, Nous Capybara 1.9)
• Huge LLM Comparison/Test: Part II (7B-20B) Roleplay Tests Winners: OpenHermes-2-Mistral-7B, LLaMA2-13B-Tiefighter-GGUF
• Huge LLM Comparison/Test: 39 models tested (7B-70B + ChatGPT/GPT-4)
• My current favorite new LLMs: SynthIA v1.5 and Tiefighter!
• Mistral LLM Comparison/Test: Instruct, OpenOrca, Dolphin, Zephyr and more...
• LLM Pro/Serious Use Comparison/Test: From 7B to 70B vs. ChatGPT! Winner: Synthia-70B-v1.2b
• LLM Chat/RP Comparison/Test: Dolphin-Mistral, Mistral-OpenOrca, Synthia 7B Winner: Mistral-7B-OpenOrca
• LLM Chat/RP Comparison/Test: Mistral 7B Base + Instruct
• LLM Chat/RP Comparison/Test (Euryale, FashionGPT, MXLewd, Synthia, Xwin) Winner: Xwin-LM-70B-V0.1
• New Model Comparison/Test (Part 2 of 2: 7 models tested, 70B+180B) Winners: Nous-Hermes-Llama2-70B, Synthia-70B-v1.2b
• New Model Comparison/Test (Part 1 of 2: 15 models tested, 13B+34B) Winner: Mythalion-13B
• New Model RP Comparison/Test (7 models tested) Winners: MythoMax-L2-13B, vicuna-13B-v1.5-16K
• Big Model Comparison/Test (13 models tested) Winner: Nous-Hermes-Llama2
• SillyTavern's Roleplay preset vs. model-specific prompt format

Disclaimer: Some kind soul recently asked me if they could tip me for my LLM reviews and advice, so I set up a Ko-fi page. While this may affect the priority/order of my tests, it will not change the results, I am incorruptible. Also consider tipping your favorite model creators, quantizers, or frontend/backend devs if you can afford to do so. They deserve it!