[QUESTION] Why doesn't GPTDataset build a global shuffle index?

[dynamicheart]

Your question

If the number samples of final epoch are less than 80% of a standard epoch, GPTDataset will separate it.

• https://github.com/NVIDIA/Megatron-LM/blob/7efaa73/megatron/core/datasets/gpt_dataset.py#L388-L392
• https://github.com/NVIDIA/Megatron-LM/blob/7efaa73/megatron/core/datasets/gpt_dataset.py#L617

However, the loss appears to drop sharply when entering the final epoch. I think the problem is due to the different data distribution between the final epoch and the standard epochs. Is this expected in the design of separate_last_epoch?