I performed an experiment with eight 33-34B models I use for code evaluation and technical assistance, to see what effect GPU power limiting had on the RTX 3090 inference.
All models were gguf, q4 quants. Each model was run only once due to time constraints. Each model was served the identical prompt to generate a bash script according to instructions.
I'll abstain from attempting an analysis, you can draw your own conclusions.
Test data below:
Set Meas GPU% M1 M2 M3 M4 M5 M6 M7 M8 T/s EFFICIENCY
300 291 79,50% 27,26 26,14 33,21 27,54 34,83 32,56 24,58 31,1 29,65 101
280 274 80,50% 26,78 27,18 33,25 27,39 34,19 30,48 26,31 31,34 29,62 108
260 253 81,50% 26,03 23,61 29,91 26,33 31,73 30,48 26,27 30,39 28,09 111
240 233 82,00% 23,71 23,13 31,49 23,64 30,12 29,72 22,5 30,93 26,91 115
220 217 84,50% 19,76 20,04 25,34 19,99 26,89 24,93 22,18 25,06 23,02 106
200 197 87,50% 15,46 14,35 19,86 15,63 20,45 19,56 16,42 19,43 17,65 89
180 179 89,50% 11,39 10,57 14,58 11,03 14,67 14,13 12,32 13,65 12,79 71
160 161 93,00% 7,93 6,79 9,1 7,42 9,45 8,82 7,94 8,78 8,28 51
140 160 95,00% 7,31 6,8 8,9 6,78 9,14 7,52 7,37 8,27 7,76 48
120 160 95,00% 6,81 6,31 8,19 6,97 8,46 7,56 6,93 8,24 7,43 46
M1 51L airoboros-c34b-3.1.2.Q4_K_M.gguf
M2 51L Zephyrus-L1-33B.q4_K_M.gguf
M3 51L codellama-34b-instruct.Q4_0.gguf
M4 51L phind-codellama-34b-v2.Q4_K_M.gguf
M5 51L tora-code-34b-v1.0.Q4_0.gguf
M6 51L wizardcoder-python-34b-v1.0.Q4_0.gguf
M7 64L yi-34b.Q4_K_M.gguf
M8 51L ziya-coding-34b-v1.0.Q4_0.gguf
โ
โ