二丫讲梵 二丫讲梵
首页
  • 最佳实践
  • 迎刃而解
  • Nginx
  • Php
  • Zabbix
  • AWS
  • Prometheus
  • Grafana
  • CentOS
  • Systemd
  • Docker
  • Rancher
  • Ansible
  • Ldap
  • Gitlab
  • GitHub
  • Etcd
  • Consul
  • RabbitMQ
  • Kafka
  • MySql
  • MongoDB
  • OpenVPN
  • KVM
  • VMware
  • Other
  • ELK
  • K8S
  • LLM
  • Nexus
  • Jenkins
  • 随写编年
  • 家人物语
  • 追忆青春
  • 父亲的朋友圈
  • 电影音乐
  • 效率工具
  • 博客相关
  • Shell
  • 前端实践
  • Vue学习笔记
  • Golang学习笔记
  • Golang编程技巧
  • 学习周刊
  • Obsidian插件周刊
关于
友链
  • 本站索引

    • 分类
    • 标签
    • 归档
  • 本站页面

    • 导航
    • 打赏
  • 我的工具

    • 备忘录清单 (opens new window)
    • json2go (opens new window)
    • gopher (opens new window)
    • 微信MD编辑 (opens new window)
    • 国内镜像 (opens new window)
    • 出口IP查询 (opens new window)
    • 代码高亮工具 (opens new window)
  • 外站页面

    • 开往 (opens new window)
    • ldapdoc (opens new window)
    • HowToStartOpenSource (opens new window)
    • vdoing-template (opens new window)
GitHub (opens new window)

二丫讲梵

行者常至,为者常成
首页
  • 最佳实践
  • 迎刃而解
  • Nginx
  • Php
  • Zabbix
  • AWS
  • Prometheus
  • Grafana
  • CentOS
  • Systemd
  • Docker
  • Rancher
  • Ansible
  • Ldap
  • Gitlab
  • GitHub
  • Etcd
  • Consul
  • RabbitMQ
  • Kafka
  • MySql
  • MongoDB
  • OpenVPN
  • KVM
  • VMware
  • Other
  • ELK
  • K8S
  • LLM
  • Nexus
  • Jenkins
  • 随写编年
  • 家人物语
  • 追忆青春
  • 父亲的朋友圈
  • 电影音乐
  • 效率工具
  • 博客相关
  • Shell
  • 前端实践
  • Vue学习笔记
  • Golang学习笔记
  • Golang编程技巧
  • 学习周刊
  • Obsidian插件周刊
关于
友链
  • 本站索引

    • 分类
    • 标签
    • 归档
  • 本站页面

    • 导航
    • 打赏
  • 我的工具

    • 备忘录清单 (opens new window)
    • json2go (opens new window)
    • gopher (opens new window)
    • 微信MD编辑 (opens new window)
    • 国内镜像 (opens new window)
    • 出口IP查询 (opens new window)
    • 代码高亮工具 (opens new window)
  • 外站页面

    • 开往 (opens new window)
    • ldapdoc (opens new window)
    • HowToStartOpenSource (opens new window)
    • vdoing-template (opens new window)
GitHub (opens new window)
  • Nexus系列文章

  • Jenkins系列文章

  • ELK笔记

  • Kubernetes笔记

    • 手动部署kubernetes-1-8-6集群

    • 其他姿势快速部署

    • 手动搭建k8s-1-10-4高可用集群(推荐版).md

      • 手动搭建k8s-1-10-4高可用集群(前言以及准备)
      • 手动搭建k8s-1-10-4之系统初始化
      • 手动搭建k8s-1-10-4之创建CA证书和秘钥
      • 手动搭建k8s-1-10-4之部署kubectl命令行工具
      • 手动搭建k8s-1-10-4之部署Etcd集群
      • 手动搭建k8s-1-10-4之部署flannel网络
        • 1,下载和分发 flanneld 二进制文件
        • 2,创建 flannel 证书和私钥
        • 3,向 etcd 写入集群 Pod 网段信息
        • 4,创建 flanneld 的 systemd unit 文件
        • 5,分发 flanneld systemd unit 文件到所有节点
        • 6,启动 flanneld 服务
        • 7,检查启动结果
        • 8,检查分配给各 flanneld 的 Pod 网段信息
        • 9,验证各节点能通过 Pod 网段互通
      • 手动搭建k8s-1-10-4之部署master节点
      • 手动搭建k8s-1-10-4之部署高可用组件
      • 手动搭建k8s-1-10-4之部署kube-apiserver
      • 手动搭建k8s-1-10-4之部署kube-controller-manager
      • 手动搭建k8s-1-10-4之部署kube-scheduler
      • 手动搭建k8s-1-10-4之部署work节点
      • 手动搭建k8s-1-10-4之部署docker组件
      • 手动搭建k8s-1-10-4之部署kubelet组件
      • 手动搭建k8s-1-10-4之部署kube-proxy组件
      • 手动搭建k8s-1-10-4之验证集群功能
      • 手动搭建k8s-1-10-4集群插件之coredns
      • 手动搭建k8s-1-10-4集群插件之dashboard
      • 手动搭建k8s-1-10-4集群插件之heapster
      • 手动搭建k8s-1-10-4之一键部署脚本
    • 基础学习

    • 从新出发

  • LLM专题

  • 系列专题
  • Kubernetes笔记
  • 手动搭建k8s-1-10-4高可用集群(推荐版).md
二丫讲梵
2018-12-09
目录

手动搭建k8s-1-10-4之部署flannel网络

文章发布较早,内容可能过时,阅读注意甄别。

flannel 第一次启动时,从 etcd 获取 Pod 网段信息,为本节点分配一个未使用的 /24 段地址,然后创建 flannel.1(也可能是其它名称,如 flannel1 等) 接口。

flannel 将分配的 Pod 网段信息写入 /run/flannel/docker 文件,docker 后续使用这个文件中的环境变量设置 docker0 网桥。

# 1,下载和分发 flanneld 二进制文件

到 https://github.com/coreos/flannel/releases 页面下载最新版本的发布包:

mkdir flannel
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel
1
2
3

分发 flanneld 二进制文件到集群所有节点:

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp  flannel/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/
    ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF
1
2
3
4
5
6
7
8
9
10

# 2,创建 flannel 证书和私钥

flannel 从 etcd 集群存取网段分配信息,而 etcd 集群启用了双向 x509 证书认证,所以需要为 flanneld 生成证书和私钥。

创建证书签名请求:

cat > flanneld-csr.json <<EOF
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
  • 该证书只会被 kubectl 当做 client 证书使用,所以 hosts 字段为空;

生成证书和私钥:

$cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem \
  -config=/etc/kubernetes/cert/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
$ls flanneld*pem
1
2
3
4
5

将生成的证书和私钥分发到所有节点(master 和 worker):

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld"
    scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/cert
done
EOF
1
2
3
4
5
6
7
8
9
10

# 3,向 etcd 写入集群 Pod 网段信息

注意:本步骤只需执行一次。

$source /opt/k8s/bin/environment.sh
$etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
1
2
3
4
5
6
7
  • flanneld 当前版本 (v0.10.0) 不支持 etcd v3,故使用 etcd v2 API 写入配置 key 和网段数据;
  • 写入的 Pod 网段 ${CLUSTER_CIDR} 必须是 /16 段地址,必须与 kube-controller-manager 的 --cluster-cidr 参数值一致;

# 4,创建 flanneld 的 systemd unit 文件

$source /opt/k8s/bin/environment.sh
$cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service
[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${VIP_IF}
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
  • mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件,后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥;
  • flanneld 使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用 -iface 参数指定通信接口,如上面的 eth0 接口;
  • flanneld 运行时需要 root 权限;

# 5,分发 flanneld systemd unit 文件到所有节点

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
done
EOF
1
2
3
4
5
6
7
8
9

# 6,启动 flanneld 服务

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl start flanneld"
done
EOF
1
2
3
4
5
6
7
8
9

# 7,检查启动结果

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh k8s@${node_ip} "systemctl status flanneld|grep Active"
done
EOF
1
2
3
4
5
6
7
8
9

看到如下输出:

$bash magic.sh
>>> 192.168.106.3
   Active: active (running) since Fri 2018-11-23 17:11:40 CST; 6h ago
>>> 192.168.106.4
   Active: active (running) since Fri 2018-11-23 17:11:40 CST; 6h ago
>>> 192.168.106.5
   Active: active (running) since Fri 2018-11-23 17:11:41 CST; 6h ago
1
2
3
4
5
6
7

则说明正常,如果失败,则用如下命令查看日志:

journalctl -ux flanneld
1

# 8,检查分配给各 flanneld 的 Pod 网段信息

查看集群 Pod 网段 (/16):

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config
1
2
3
4
5
6
7

输出:

{"Network":"172.30.0.0/16","SubnetLen": 24, "Backend": {"Type": "vxlan"}}
1

查看已分配的 Pod 子网段列表 (/24):

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets
1
2
3
4
5
6
7

输出:

/kubernetes/network/subnets/172.30.84.0-24
/kubernetes/network/subnets/172.30.8.0-24
/kubernetes/network/subnets/172.30.29.0-24
1
2
3

查看某一 Pod 网段对应的节点 IP 和 flannel 接口地址:

注意其中的IP段换成自己的。

source /opt/k8s/bin/environment.sh
etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.8.0-24
1
2
3
4
5
6
7

输出:

{"PublicIP":"192.168.106.4","BackendType":"vxlan","BackendData":{"VtepMAC":"f2:14:20:50:4f:af"}}
1

# 9,验证各节点能通过 Pod 网段互通

在各节点上部署 flannel 后,检查是否创建了 flannel 接口 (名称可能为 flannel0、flannel.0、flannel.1 等):

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
done
EOF
1
2
3
4
5
6
7
8
9

输出:

$bash magic.sh
>>> 192.168.106.3
    inet 172.30.84.0/32 scope global flannel.1
>>> 192.168.106.4
    inet 172.30.8.0/32 scope global flannel.1
>>> 192.168.106.5
    inet 172.30.29.0/32 scope global flannel.1
1
2
3
4
5
6
7

在各节点上 ping 所有 flannel 接口 IP,确保能通:

注意其中的IP段换成自己的。

cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh ${node_ip} "ping -c 1 172.30.8.0"
    ssh ${node_ip} "ping -c 1 172.30.29.0"
    ssh ${node_ip} "ping -c 1 172.30.84.0"
done
EOF
1
2
3
4
5
6
7
8
9
10
11

输出:

$bash magic.sh
>>> 192.168.106.3
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.285 ms
--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.285/0.285/0.285/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.337 ms
--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.337/0.337/0.337/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.062 ms
--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms
>>> 192.168.106.4
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.055 ms
--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.055/0.055/0.055/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.311 ms
--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.311/0.311/0.311/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.395 ms
--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.395/0.395/0.395/0.000 ms
>>> 192.168.106.5
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.325 ms
--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.325/0.325/0.325/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.060 ms
--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.260 ms
--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.260/0.260/0.260/0.000 ms
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
微信 支付宝
上次更新: 2024/06/13, 22:13:45
手动搭建k8s-1-10-4之部署Etcd集群
手动搭建k8s-1-10-4之部署master节点

← 手动搭建k8s-1-10-4之部署Etcd集群 手动搭建k8s-1-10-4之部署master节点→

最近更新
01
学习周刊-总第213期-2025年第22周
05-29
02
学习周刊-总第212期-2025年第21周
05-22
03
从赵心童世锦赛夺冠聊聊我的斯诺克情缘
05-16
更多文章>
Theme by Vdoing | Copyright © 2017-2025 | 点击查看十年之约 | 浙ICP备18057030号
  • 跟随系统
  • 浅色模式
  • 深色模式
  • 阅读模式