[core] Support upper bound in dynamic bucket mode

[liyubin117]

Purpose

In dynamic bucket mode, unlimited buckets lead to an unpredicable number of small files, which lead to stability problems. so we should support upper bound in dynamic bucket mode.

Linked issue: close #4942

Tests

HashBucketAssignerTest
SimpleHashBucketAssignerTest

• Use new feature, Produce 3 buckets after inserting 12 rows

• Not use new feature, Produce 6 buckets after inserting 12 rows

API and Format

Documentation

docs/layouts/shortcodes/generated/core_configuration.html

[liyubin117]

@JingsongLi CI passed, PTAL, thanks!

[JingsongLi]

I think we should limit total buckets?

[liyubin117]

This is also my initial purpose, but I found that the bucket generation logic in BucketAssigner is a instance behavior and could be parallelized, every assigner handles their own buckets. On the other hand, This feature is equivalent to the total bucket limit by default (1 parallellism).
Furthuremore, the universal bucket limit may cause some assigners to work less, resulting in CPU and data skew. WDYT?

[JingsongLi]

I still think introducing a max bucket is better.
The option should be very easy to understand.

[wwj6591812]

+1
Add option should be easy to user understand.

[liyubin117]

Reasonable enough, I have found a way to implement total max bucket feature, thanks for reminding that :)

    public static int[] distributeBucketsToAssigners(int maxBuckets, int assigners) {
        int[] result = new int[assigners];
        int avg = maxBuckets / assigners;
        int remainder = maxBuckets % assigners;
        for (int i = 0; i < assigners; i++) {
            result[i] = avg;
            if (remainder > 0) {
                result[i]++;
                remainder--;
            }
        }
        return result;
    }

[wwj6591812]

Add a util function of PartitionIndex#cacheBucketAndGet, then here use?

[wwj6591812]

+1
Add option should be easy to user understand.