It's an old problem, and it, along with many of the answers are in many recent data sets.

I assume training set components have also priorities, low priority data goes to training very few times at the beginning of pretraining, while higher priority data is trained on multiple times until the end.