this post was submitted on 30 Oct 2023
1 points (100.0% liked)

LocalLLaMA

3 readers
1 users here now

Community to discuss about Llama, the family of large language models created by Meta AI.

founded 1 year ago
MODERATORS
communick@poweruser.forum
1
RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models (alien.top)
submitted 1 year ago by APaperADay@alien.top to c/localllama@poweruser.forum
6 comments fedilink hide all child comments
 

Blog: https://together.ai/blog/redpajama-data-v2

Hugging Face: https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2

GitHub: https://github.com/togethercomputer/RedPajama-Data

Description:

RedPajama-V2 is an open dataset for training large language models. The dataset includes over 100B text documents coming from 84 CommonCrawl snapshots and processed using the CCNet pipeline. Out of these, there are 30B documents in the corpus that additionally come with quality signals, and 20B documents that are deduplicated.

you are viewing a single comment's thread
view the rest of the comments
[–] Feztopia@alien.top 1 points 1 year ago

Does this have the same level of deduplication like slimpajama or do we need a slimpajama v2?