Data shuffle with streaming

[JonghwanMun]

Reminder

• I have read the README and searched the existing issues.

System Info

No response

Reproduction

I wonder how data shuffling works when using streaming option.
I understand that data shuffling is applied on each buffer.

If I have total 10,000 data and set buffer_size as 1,000.

• Let's consider buffer1 (1-1000 samples), buffer2 (1001-2000 samples), buffer3 (2001-3000 samples), ...
• Then, the buffer order is shuffled or not?

Expected behavior

The buffer order is shuffled.

Others

No response

[hiyouga]

You can refer to the document for details
https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.IterableDataset.shuffle
https://huggingface.co/docs/datasets/en/stream#shuffle
If your dataset has multiple shards, the order of shards will be shuffled also.