Just try a huge number of ways to explain it. See how you go and iterate on the best approaches. Maybe trying to glean some high level concepts about education in the process would help.
sdmat
joined 10 months ago
What did you go with?
No, the primary concern is that network latency kills the serial performance of LLMs.
You can have a distributed llm getting decent throughput in total across many slow generations. You can't have a distributed LLM with throughput for a single generation competitive to running in a single cluster.
You can get preemptible A100s for $1/hr, so not exactly breaking the bank if willing to take the risk.
This technique is actually really useful for batch processing.
I.e. if you run 100 generations and reuse the layer while it is loaded that will go much faster than the total serial time.