LocalLLaMA

I don't know how big 10^20 floating points is, and if 70b was made with something bigger or smaller. But I think that figure is the more important one as I think Meta uses a single datacentre.

These figures in context:

(b) The Secretary of Commerce, in consultation with the Secretary of State, the Secretary of Defense, the Secretary of Energy, and the Director of National Intelligence, shall define, and thereafter update as needed on a regular basis, the set of technical conditions for models and computing clusters that would be subject to the reporting requirements of subsection 4.2(a) of this section. Until such technical conditions are defined, the Secretary shall require compliance with these reporting requirements for:

(i) any model that was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, or using primarily biological sequence data and using a quantity of computing power greater than 10^(23) integer or floating-point operations; an

(ii) any computing cluster that has a set of machines physically co-located in a single datacenter, transitively connected by data center networking of over 100 Gbit/s, and having a theoretical maximum computing capacity of 10^(20) integer or floating-point operations per second for training AI.

Assuming number of FLOPs in compute is 6ND (N = number of parameters, D = dataset size in tokens) you could take the full RedPajama dataset (30T tokens) and a 500B parameter model and it'd come out to:

6*(30*10^12)*(500*10^9) = 9*10^25

In order to qualify, you would need a cluster that could train this beast in about:

10^26 / 10^20 = 1000000 seconds = 11.57 days

Ok, as a baseline for everyone who, like me, doesn't understand all the big words and numbers on why this is great news:

• Llama 2 70b required 1.7M GPU hours to train on A100s
• The H100 is more powerful than the A100

So, if I'm understanding correctly, one of our most powerful open source models is so far from this benchmark that it can't even been seen.

Someone please correct me if I'm wrong.

"Give me a big number in units that will be very hard to understand by anybody."

"28M pigeon feet"

"It's too on the nose."

"28M H100 hours"

All of this is a red herring. The bigger issue is going to be checking of the data for biological sequences and such.

Haha great, open source models will now have a chance, and China will be able to catch up with their hypercensorship models

People say we should use the government to crack down on monopolies. But it's the government that owns the monopolies.

The legislation is going to get old, fast.

1026

1023

1020

That would include models trained on a calculator.

So if I make a 10 Trillion param mixture of experts model out of fine-tuned variations of the same 300b model I am safe right?

Or how about I train a 4 Trillion param model on a new architecture that can utilize distributed compute? If contribution to GPU pools for training is encrypted and decentralized then good luck.

Fuck OpenAI. We will take this fight to the streets and win.

huh what's that about biological sequence data ,, a MUCH lower number ,, huh ,, um do they know something specific about how that's dangerous :o

just train/finetune outside of the us

Some exponent marks missing here.

The most interesting part is the focus on biological sequence data. This means that generative AI for synthetic biology is on the policy makers/risk assessors radar, and probably rightly so.