LocalLLaMA

4 readers

4 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 2 years ago

MODERATORS

communick@poweruser.forum

RWKV v5 7b, Fully Open-Source, 60% trained, approaching Mistral 7b in abilities or surpassing it. (alien.top)

submitted 2 years ago by vatsadev@alien.top to c/localllama@poweruser.forum

32 comments fedilink hide all child comments

So RWKV 7b v5 is 60% trained now, saw that multilingual parts are better than mistral now, and the english capabilities are close to mistral, except for hellaswag and arc, where its a little behind. all the benchmarks are on rwkv discor, and you can google the pro/cons of rwkv, though most of them are v4.

Thoughts?

you are viewing a single comment's thread
view the rest of the comments

[–] ambient_temp_xeno@alien.top 1 points 2 years ago (8 children)

Fully open source?

[–] _Lee_B_@alien.top 1 points 2 years ago (6 children)

The source is actually available (which is good), but sadly the dataset is not (which is bad, and makes it not truly open, since you can're reliably reproduce it).

[–] vatsadev@alien.top 1 points 2 years ago (1 children)

Um The dataset is opensource, its all public HF datasets

[–] _Lee_B_@alien.top 1 points 2 years ago

"World = Some_Pile + Some_SlimPajama + Some_StarCoder + Some_OSCAR + All_Wikipedia + All_ChatGPT_Data_I_can_find"

"some" as in customized.

load more comments (4 replies)

load more comments (5 replies)