I'm only getting 0.8 tokens/second with my 3060 12gb using Zephyr 7b beta.
I'll admit I barely know what I'm doing, but was I wrong to expect a little more? I was hoping for something at least a quarter the speed of gpt-3.5...