Skip to content

Latest commit

 

History

History
138 lines (102 loc) · 3.11 KB

Debezium监控系统搭建.md

File metadata and controls

138 lines (102 loc) · 3.11 KB

Kafka&Zookeeper监控

开启Zookeeper监控

修改zk-Server.sh文件

  • JMXPORT端口
-Dcom.sun.management.jmxremote.port=$JMXPORT
  • JMXAUTH
-Dcom.sun.management.jmxremote.authenticate=$JMXAUTH
  • JMXSSL
-Dcom.sun.management.jmxremote.ssl=$JMXSSL
  • JMXLOG4J
-Dzookeeper.jmx.log4j.disable=$JMXLOG4J

修改zkEnv.sh文件

JMXPORT=21811
JMXSSL=false
JMXAUTH=false
JMXLOG4J=false

开启Kafka监控

查看kafka-run-class.sh

  • JMX_PORT
-Dcom.sun.management.jmxremote.port=$JMX_PORT
  • KAFKA_JMX_OPTS
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

修改kafka-start-server.sh

if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
    # export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi

开启Kafka Connect JMX

  • 下载jmx_prometheus_javaagent-0.3.1.jar
  • 配置config.yml
startDelaySeconds: 0
ssl: false
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:19092/jmxrmi
lowercaseOutputName: false
lowercaseOutputLabelNames: false
rules:
- pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
  name: "kafka_connect_connect_worker_metrics_$1"
- pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
  name: "kafka_connect_connect_metrics_$2"
  labels:
    client: "$1"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^,]+), key=([^>]+)><>RowsScanned"
  name: "debezium_metrics_RowsScanned"
  labels:
    plugin: "$1"
    name: "$3"
    context: "$2"
    table: "$4"
- pattern: "debezium.([^:]+)<type=connector-metrics, context=([^,]+), server=([^>]+)>([^:]+)"
  name: "debezium_metrics_$4"
  labels:
    plugin: "$1"
    name: "$3"
    context: "$2"

修改connect-distributed.sh

# 修改KAFKA_HEAP_OPTS环境变量
  export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
# 添加JMX_PORT
 export JMX_PORT="19092"
# 添加KAFKA_OPTS
 export KAFKA_OPTS="-javaagent:/opt/cloudera/parcels/CDH/lib/kafka/monitor/jmx_prometheus_javaagent-0.3.1.jar=8080:/opt/cloudera/parcels/CDH/lib/kafka/monitor/config.yml"

配置监控大盘

配置prometheus

  • 配置prometheus.yml文件
global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'debezium'
    static_configs:
    #  export KAFKA_OPTS="-javaagent:/opt/cloudera/parcels/CDH/lib/kafka/monitor/jmx_prometheus_javaagent-0.3.1.jar=8080:/opt/cloudera/parcels/CDH/lib/kafka/monitor/config.yml"
      - targets: [这里填写的是kafkaconnect的ip地址和jmx暴露的web端口,如上就是8080]
      

配置grafana