I performed an experiment with eight 33-34B models I use for code evaluation and technical assistance, to see what effect GPU power limiting had on the RTX 3090 inference.
All models were gguf, q4 quants. Each model was run only once due to time constraints. Each model was served the identical prompt to generate a bash script according to instructions.
I'll abstain from attempting an analysis, you can draw your own conclusions.
Test data below:
Set Meas GPU% M1 M2 M3 M4 M5 M6 M7 M8 T/s EFFICIENCY
300 291 79,50% 27,26 26,14 33,21 27,54 34,83 32,56 24,58 31,1 29,65 101
280 274 80,50% 26,78 27,18 33,25 27,39 34,19 30,48 26,31 31,34 29,62 108
260 253 81,50% 26,03 23,61 29,91 26,33 31,73 30,48 26,27 30,39 28,09 111
240 233 82,00% 23,71 23,13 31,49 23,64 30,12 29,72 22,5 30,93 26,91 115
220 217 84,50% 19,76 20,04 25,34 19,99 26,89 24,93 22,18 25,06 23,02 106
200 197 87,50% 15,46 14,35 19,86 15,63 20,45 19,56 16,42 19,43 17,65 89
180 179 89,50% 11,39 10,57 14,58 11,03 14,67 14,13 12,32 13,65 12,79 71
160 161 93,00% 7,93 6,79 9,1 7,42 9,45 8,82 7,94 8,78 8,28 51
140 160 95,00% 7,31 6,8 8,9 6,78 9,14 7,52 7,37 8,27 7,76 48
120 160 95,00% 6,81 6,31 8,19 6,97 8,46 7,56 6,93 8,24 7,43 46
M1 51L airoboros-c34b-3.1.2.Q4_K_M.gguf
M2 51L Zephyrus-L1-33B.q4_K_M.gguf
M3 51L codellama-34b-instruct.Q4_0.gguf
M4 51L phind-codellama-34b-v2.Q4_K_M.gguf
M5 51L tora-code-34b-v1.0.Q4_0.gguf
M6 51L wizardcoder-python-34b-v1.0.Q4_0.gguf
M7 64L yi-34b.Q4_K_M.gguf
M8 51L ziya-coding-34b-v1.0.Q4_0.gguf
Euro means he's probably on 220-240v. Those spikes are in the millisecond range - but they do a number on lesser PSUs. I've found to run a 3090 on my lower end 525w psu, I needed to cap power to 280W and frequency to 1650 Mhz. Asus ROG Strix by default was punching up to 1900+Mhz and blackscreening me.
Now things chug along just fine though. 9-year old PSU still inaudible with fanless hydro.