It's impressive, but from what I have heard on this sub the big weakness of mac computers is the time it takes to evaluate the prompt. For example, if you are roleplaying with 8k context in Sillytavern and your prompt is continuously changing as world entries are triggered and character sheets are swapped during a group chat, then the evaluation speed of the prompt matters greatly.
Zugzwang_CYOA
joined 10 months ago
7b models are absolute garbage for RP with large context and world info entries.
Good to know. It sounds like macs are great at asking simple questions of powerful LLMs, but not so great at roleplaying with large context stories. I had hoped that an M2 Max would be viable for RP at 70b or 120b, but I guess not.
How is the prompt processing time on a mac? If I were to work with a prompt that is 8k in size for RP, with big frequent changes in the prompt, would it be able to read my ever-changing prompt in a timely manner and respond?
I would like to use Sillytavern as my front end, and that an result in big prompt changes between replies.
Interesting, so even if the middle of the context changes between replies as world entries appear and disappear in the context, it will still not force a re-evaluation of the greater whole?