Measuring and improving quality of NLP datasets in a comprehensive way is probably the main migraine there.
You can measure and improve quality by many dimensions that practitioners disagree on... ( accuracy, completeness, consistency, timeliness, validity, and uniqueness are common ways to slice data quality) and there's no consistent single measure for some of those either.
What if the conversation moves away from data quality / contamination to mixing skills?
It seems that dataset size and quality are more important than the raw number of parameters, as many LLMs are under-trained
Though there has been significant recent work allowing larger and larger models to be trained, our analysis suggests an increased focus on dataset scaling is needed. Speculatively, we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality
This under-training appears to be significant
We conducted our investigation as a case study of the OPT-66B model, a 66-billion-parameter LLM that was open-sourced by Meta last year to serve as an open replica of GPT-3 (and was the largest publicly available decoder-only LLM at the time of our study). We found that a significant portion of the model could be discarded without affecting performance, indicating that OPT-66B and quite likely other prominent LLMs are undertrained.
...
We found that important attention heads are primarily clustered in the model’s intermediate layers, and important FFNs are primarily in later layers. The ability to perform zero-/few-shot in-context learning on 14 different natural-language-processing (NLP) datasets/tasks stayed nearly intact when up to 70% (~15.7B parameters in OPT-66B) of the attention heads are removed.
https://www.amazon.science/blog/do-large-language-models-really-need-all-those-layers