Prometheus监控之kafka集群
文章发布较早,内容可能过时,阅读注意甄别。
prometheus 监控 kafka 常见的有两种开源方案,一种是传统的部署 exporter 的方式,一种是通过 jmx 配置监控,本文将采用第二种方式进行配置。
- 项目地址:
- kafka_exporter:https://github.com/danielqsj/kafka_exporter
- jmx_exporter:https://github.com/prometheus/jmx_exporter
# 1、安装部署
现有 kafka 三节点的集群,环境大概如下:
主机 | 组件 |
---|---|
10.3.0.41–node1 | zookeeper,kafka |
10.3.0.42–node2 | zookeeper,kafka |
10.3.20.4–node3 | zookeeper,kafka |
接着分别在如上三台主机上进行如下配置:
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml
wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
1
2
2
分别创建统一的配置放置目录:
mkdir -p /usr/local/kafka/jmx/
mv kafka-2_0_0.yml /usr/local/kafka/jmx/kafka-2_0_0.yml
mv jmx_prometheus_javaagent-0.12.0.jar /usr/local/kafka/jmx/jmx_prometheus_javaagent-0.12.0.jar
1
2
3
2
3
然后在启动的配置文件中添加如下两行:
$ cat bin/kafka-server-start.sh
if [ $# -lt 1 ];
then
echo "USAGE: $0 [-daemon] server.properties [--override property=value]*"
exit 1
fi
base_dir=$(dirname $0)
if [ "x$KAFKA_LOG4J_OPTS" = "x" ]; then
export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
fi
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
fi
EXTRA_ARGS=${EXTRA_ARGS-'-name kafkaServer -loggc'}
#如下两行内容
export JMX_PORT="9999"
export KAFKA_OPTS="-javaagent:/usr/local/kafka/jmx/jmx_prometheus_javaagent-0.12.0.jar=9991:/usr/local/kafka/jmx/kafka-2_0_0.yml"
COMMAND=$1
case $COMMAND in
-daemon)
EXTRA_ARGS="-daemon "$EXTRA_ARGS
shift
;;
*)
;;
esac
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
然后重启 kafka。
# 2,配置 prometheus.yml 添加监控目标
$ vim /usr/local/prometheus/prometheus.yml
- job_name: 'kafka-cluster'
scrape_interval: 5s
static_configs:
- targets: ['10.3.0.41:9991']
- targets: ['10.3.0.42:9991']
- targets: ['10.3.20.4:9991']
1
2
3
4
5
6
7
2
3
4
5
6
7
重启服务。
$ systemctl restart prometheus
1
或者通过命令热加载:
curl -XPOST localhost:9090/-/reload
1
# 5,配置 Grafana 的模板
展示模板通过 ID 进行导入,可用 ID 有:7589
,10466
,11963
等等
上次更新: 2024/07/04, 22:40:37