I cannot start elastalert with elastsearch 7.8.0

[hoyindev]

docker-compose file

    version: '3'
    services:
      es01:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
        container_name: es01
        environment:
            - node.name=es01
            - cluster.name=es-docker-cluster
            - discovery.seed_hosts=es01,es02
            - cluster.initial_master_nodes=es01,es02,es03
            - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
            - "path.repo=/usr/share/elasticsearch/backup"
        
     
        volumes:
          - data01:/usr/share/elasticsearch/data
          - elastic-backup:/usr/share/elasticsearch/backup
        ports:
          - 9200:9200
        networks:
          - elastic
      es02:
      es03:
       
      kibana:
        image: "docker.elastic.co/kibana/kibana:7.8.0"
        container_name: kibana
        environment: 
          ELASTICSEARCH_URL: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
          ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    
        ports:
          - "5601:5601"
        networks: 
          - elastic
        volumes:
          - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    
      filebeat:
        image: "docker.elastic.co/beats/filebeat:7.2.0"
        user: root
        entrypoint : "filebeat -e -strict.perms=false"
        volumes: 
            # persist file beat tracking data
          - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
            # needed to access all docker logs 
          - /var/lib/docker:/var/lib/docker:ro
            # needed to access additional info about containers
          - /var/run/docker.sock:/var/run/docker.sock
        deploy:
          mode: global
          restart_policy:
            condition: on-failure
            delay: 30s
        networks: 
          - elastic
    
      elastalert:
        container_name: elastalert
        build:
          context: .
          dockerfile: Dockerfiles/Dockerfile-elastalert
        # image: elastalert:3.0.0-beta.0
        ports:
          - 3030:3030
          - 3333:3333
        restart: always
        volumes:
          - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
          - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
          - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
          - ./elastalert/rules:/opt/elastalert/rules
          - ./elastalert/rule_templates:/opt/elastalert/rule_templates
        environment: 
          ELASTICSEARCH_URL: ${ELASTICSEARCH_URL}
      #   networks: 
      #     - elastic
        depends_on:
          - es01
          - es02
          - es03
    
    volumes:
      data01:
      data02:
      data03:
      elastic-backup:
      
    networks:
      elastic:
        # driver: overlay

service elastalert log

            docker logs 45cb7d68557f
            Giving Elasticsearch at localhost:9200 time to start...
            Elasticsearch is up and healthy at localhost:9200
            Starting ElastAlert!

            > @bitsensor/[email protected] start /opt/elastalert-server
            > sh ./scripts/start.sh

            10:28:30.587Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
            10:28:30.601Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
            10:28:30.658Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
            10:28:31.006Z  INFO elastalert-server: Router:  Listening for GET request on /.
            10:28:31.006Z  INFO elastalert-server: Router:  Listening for GET request on /status.
            10:28:31.007Z  INFO elastalert-server: Router:  Listening for GET request on /status/control/:action.
            10:28:31.007Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
            10:28:31.007Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
            10:28:31.016Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id.
            10:28:31.016Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id.
            10:28:31.016Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id.
            10:28:31.024Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
            10:28:31.025Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id.
            10:28:31.025Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id.
            10:28:31.025Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id.
            10:28:31.025Z  INFO elastalert-server: Router:  Listening for POST request on /test.
            10:28:31.025Z  INFO elastalert-server: Router:  Listening for GET request on /config.
            10:28:31.026Z  INFO elastalert-server: Router:  Listening for POST request on /config.
            10:28:31.026Z  INFO elastalert-server: Router:  Listening for POST request on /download.
            10:28:31.026Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/:type.
            10:28:31.026Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
            10:28:31.027Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
            10:28:31.048Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
            10:28:31.049Z  INFO elastalert-server: ProcessController:  Creating index
            10:28:47.301Z ERROR elastalert-server:
                ProcessController:  Traceback (most recent call last):
                File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
                    "__main__", fname, loader, pkg_name)
                File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
                    exec code in run_globals
                File "/opt/elastalert/elastalert/create_index.py", line 269, in <module>
                    main()
                File "/opt/elastalert/elastalert/create_index.py", line 265, in main
                    create_index_mappings(es_client=es, ea_index=index, recreate=args.recreate, old_ea_index=old_index)
                File "/opt/elastalert/elastalert/create_index.py", line 24, in create_index_mappings
                    esversion = es_client.info()["version"]["number"]
                File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
                    return func(*args, params=params, **kwargs)
                File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/__init__.py", line 259, in info
                    return self.transport.perform_request("GET", "/", params=params)
                File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
                    timeout=timeout,
                File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 127, in perform_request
                    raise ConnectionError("N/A", str(e), e)
                elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f41c6107450>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f41c6107450>: Failed to establish a new connection: [Errno 111] Connection refused',)))
                
            10:28:47.302Z ERROR elastalert-server: ProcessController:  Index create exited with code 1
            10:28:47.302Z  WARN elastalert-server: ProcessController:  ElastAlert will start but might not be able to save its data!
            10:28:47.304Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
            10:28:47.368Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 40)
            10:28:47.387Z  INFO elastalert-server: Server:  Server listening on port 3030
            10:28:47.405Z  INFO elastalert-server: Server:  Websocket listening on port 3333
            10:28:47.406Z  INFO elastalert-server: Server:  Server started
            10:28:52.780Z  INFO elastalert-server:
                ProcessController:  0 rules loaded
                
            10:28:57.978Z ERROR elastalert-server:
                ProcessController:  WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.087s]
                Traceback (most recent call last):
                File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request
                    response = self.session.send(prepared_request, **send_kwargs)
                File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send
                    r = adapter.send(request, **kwargs)
                File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send
                    raise ConnectionError(e, request=request)
                ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f01fb177050>: Failed to establish a new connection: [Errno 111] Connection refused',))

I did some research I found some people (https://www.gitmemory.com/issue/bitsensor/elastalert/177/648136404 ) can use the version 7.8.0 and the elastalert log would be like that after index is created:

13:00:35.048Z  INFO elastalert-server: ProcessController:  Creating index
13:00:40.962Z  INFO elastalert-server:
    ProcessController:  Elastic Version: 7.8.0
    Reading Elastic 6 index mappings:
    Reading index mapping 'es_mappings/6/silence.json'
    Reading index mapping 'es_mappings/6/elastalert_status.json'
    Reading index mapping 'es_mappings/6/elastalert.json'
    Reading index mapping 'es_mappings/6/past_elastalert.json'
    Reading index mapping 'es_mappings/6/elastalert_error.json'
    New index elastalert_status created
    Done!

I tried to follow the elastalert part of the above link, but I found that elastalert still using the elasticsearch-7.0.1. I am not sure if I update the version properly. I did this:

pip install elasticsearch==7.8.0
pip uninstall elastalert
pip install elastalert

But the error still remained after I reinstalled, what did I miss? or elastalert is not ready for elasticsearch? I should use other version?

[nsano-rururu]

@hoyindev

It seems pip install elasticsearch==7.0.0 should be used instead of pip install elasticsearch==7.8.0.

Yelp/elastalert#2593

Also note: I also added a pin for elasticsearch==7.0.0, because apparently 7.1.0 will NOT work with ES < 6.6 due to it not supported _source_include(s?). 7.0.0 does. Tests won't pass otherwise.

[nsano-rururu]

If you are using the public docker image, I think the following command is unnecessary
At first you tried to do it with Docker, but did you install it directly on the server instead of Docker?

pip install elasticsearch==7.8.0
pip uninstall elastalert
pip install elastalert

[nsano-rururu]

ElastAlert installed with bitsensor/elastalert:3.0.0-beta.0 is 0.2.0b2
It seems that version 7.0.1 of elasticsearch, a Python library, was installed with the settings at that time.
Elasticsearch, a Python library, supports Elasticsearch 7.x in version 7.x.
In other words, if the Python library elasticsearch is 7.x, there should be no problem with Elasticsearch 7.x.
In addition, it seems that the latest version of ElastAlert has a problem and the elasticsearch version of the Python library is changed to 7.0.0 fixed.

https://pypi.org/project/elasticsearch/

For Elasticsearch 7.0 and later, use the major version 7 (7.x.y) of the library.
For Elasticsearch 6.0 and later, use the major version 6 (6.x.y) of the library.
For Elasticsearch 5.0 and later, use the major version 5 (5.x.y) of the library.
For Elasticsearch 2.0 and later, use the major version 2 (2.x.y) of the library, and so on.

Docker Images

Docker image name	tag	ElastAlert	Elasticsearch 7 Support	Remarks
bitsensor/elastalert	2.0.1	0.1.39	×
bitsensor/elastalert	lastet	0.1.39	×
bitsensor/elastalert	3.0.0-beta.0	0.2.0b2	〇
bitsensor/elastalert	3.0.0-beta.1	0.2.0b2	〇
servercentral/elastalert	latest	0.2.1	〇	bitsensor/elastalert fork Customize bug fix
daichi703n/elastalert	0.2.1-dev2	0.2.1	〇	servercentral/elastalert fork Customize
johnsusek/elastalert-server	1592081541	0.2.4	〇	servercentral/elastalert fork Customize Library Update bug fix

[nsano-rururu]

It is the result of running with bitsensor/elastalert:3.0.0-beta.0, Elasticsearch 7.8.0, Kibana 7.8.0 that you found, but I posted it.
#177 (comment)

[nsano-rururu]

elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200)

elastalert.yaml and elastalert-test.yaml are ElastAlert settings.
config.json is the setting of ElastAlert Server.

./elastalert/config/elastalert.yaml

　Is localhost set for es_host?

./elastalert/config/elastalert-test.yaml

　Is localhost set for es_host?

./elastalert/config/config.json

"es_host": "elasticsearch"," should be set by default. It is a correction target.

[hoyindev]

./elastalert/config/config.json
"es_host": I used "localhost" instead of the ip address, turns out it caused the connection error
@nsano-rururu thank you for your information, your example also works perfectly.