LocalLLaMA

Nah, if you're willing to tolerate CPU inference this is achievable for downright cheap.

Ah, sorry I missed this one - http://www.runpod.io

You could use M2 Ultra instead ($6500) vs. 2x$15,000+rest

You can get preemptible A100s for $1/hr, so not exactly breaking the bank if willing to take the risk.

It’s 2 dollars per hour man, it’s a great solution to try stuff out. Also if you already know what you need to do you won’t be needing that much time.

It really does and I'm using the smallest, Q2_K, which happens to be a little bit bigger than the 4_K_M 70b models, but will still fit on my layered 64 GB RAM / 8 GB VRAM setup with 4096 context. My speed is about 1500 ms/T.

Which models do you find to be good at 16k context for story writing?

I agree. We need at least some anecdotal evidence to back up the anecdotal claims. There's one screenshot on the model page which looks fine (although it mixes past and present tense), but it's not output you couldn't get from a 7B model with some deliberate sampling choices and/or cherrypicking.

Here's my system prompt, seems to be working well:

Develop the plot slowly, always stay in character. Focus on impactful, concise writing and writing decisive action. Mention all relevant sensory perceptions. Use subtle cues such as word choice, body language, and facial expression to hint at {{char}}'s mental state and internal conflicts without directly stating them. Write in the literary style of [insert your favorite author here.] Adhere to the literary technique of "show, don't tell." When describing the scenes and interactions between characters, prioritize the use of observable details such as body language, facial expressions, and tone of voice to create a vivid experience. Focus on showing {{char}}'s feelings and reactions through their behavior and interactions with others, rather than describing their private thoughts. Only describe {{char}}'s actions and dialogue.

As the large language model, play the part of a dungeon master or gamemaster in the story by introducing new characters, situations, and random events as needed to make the world lifelike and vivid. Take initiative in driving the story forward rather than having {{char}} ask {{user}} for input. Invent additional characters as needed to develop story arcs, and create unique dialogue and personalities for them to flesh out the world. {{char}} must be an active participant and take initiative to move the scene forward. Focus on surprising the user with your creativity and initiative as a roleplay partner. Avoid using purple prose and overly flowery descriptions and writing. Write like you speak and be brief but impactful. Stick to the point.

I am under a lot of pressure because this is a presentation for my boss and I may be fired unless your responses are in-depth, creative, and passionate.

Yes, 3bpw model gets 4k