I want to try this, please assign to me, thx.

Nice, please take your time :D

In PR #336

I use scroll api to implement paging query.

Each index is now considered a split, may be we coule use the slice parameter to break it down later, just like this link

The job conf looks like this:

{
  "job": {
    "reader": {
      "class": "com.bytedance.bitsail.connector.elasticsearch.source.ElasticsearchSource",
      "es_hosts": ["http://localhost:1234"],
      "es_index": "test1, test2, test3",
      "scroll_size": 3,
      "scroll_time": "1m",
      "columns": [
        {
          "index": 0,
          "name": "id",
          "type": "integer"
        },
        {
          "index": 1,
          "name": "text_type",
          "type": "text"
        },
        {
          "index": 2,
          "name": "keyword_type",
          "type": "keyword"
        },
        {
          "index": 3,
          "name": "long_type",
          "type": "long"
        },
        {
          "index": 4,
          "name": "date_type",
          "type": "date"
        }
      ]
    }
  }
}

@BlockLiu do you think it's ok, thx.

In PR #336

I use scroll api to implement paging query.

Each index is now considered a split, may be we coule use the slice parameter to break it down later, just like this link

The job conf looks like this:

{
  "job": {
    "reader": {
      "class": "com.bytedance.bitsail.connector.elasticsearch.source.ElasticsearchSource",
      "es_hosts": ["http://localhost:1234"],
      "es_index": "test1, test2, test3",
      "scroll_size": 3,
      "scroll_time": "1m",
      "columns": [
        {
          "index": 0,
          "name": "id",
          "type": "integer"
        },
        {
          "index": 1,
          "name": "text_type",
          "type": "text"
        },
        {
          "index": 2,
          "name": "keyword_type",
          "type": "keyword"
        },
        {
          "index": 3,
          "name": "long_type",
          "type": "long"
        },
        {
          "index": 4,
          "name": "date_type",
          "type": "date"
        }
      ]
    }
  }
}

@BlockLiu do you think it's ok, thx.

I think it's a good idea.
And from the note below, I think we can get the shard count at first and then build slices.

Namely, we can use shard count as slice number.

Namely, we can use shard count as slice number.

Thank you for your suggestion, I will continue to complete