手动搭建k8s-1-10-4之系统初始化
文章发布较早，内容可能过时，阅读注意甄别。
# 1，集群机器
- kube-node1：192.168.106.3
- kube-node2：192.168.106.4
- kube-node3：192.168.106.5
- VIP：192.168.106.110
# 2，主机名
- 设置永久主机名称，然后重新登录:
$ sudo hostnamectl set-hostname kube-node1 # 将 kube-node1 替换为当前主机名
$ sudo hostnamectl set-hostname kube-node2 # 将 kube-node2 替换为当前主机名
$ sudo hostnamectl set-hostname kube-node3 # 将 kube-node3 替换为当前主机名
1
2
3
2
3
- 设置的主机名保存在 /etc/hosts 文件中；
$cat > /etc/hosts << EOF
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.106.3 kube-node1
192.168.106.4 kube-node2
192.168.106.5 kube-node3
EOF
1
2
3
4
5
6
7
2
3
4
5
6
7
# 3，添加 k8s 和 docker 账户
- 在每台机器上添加 k8s 账户，可以无密码 sudo：
$ sudo useradd -m k8s
$ echo 123456 | passwd k8s --stdin # 为 k8s 账户设置密码
$ sudo visudo
$ sudo grep '%wheel.*NOPASSWD: ALL' /etc/sudoers
%wheel ALL=(ALL) NOPASSWD: ALL
$ sudo gpasswd -a k8s wheel
1
2
3
4
5
6
2
3
4
5
6
- 在每台机器上添加 docker 账户，将 k8s 账户添加到 docker 组中，同时配置 dockerd 参数：
$ sudo useradd -m docker
$ sudo gpasswd -a k8s docker
$ sudo mkdir -p /etc/docker/
$ cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://hub-mirror.c.163.com","https://docker.mirrors.ustc.edu.cn"],
"max-concurrent-downloads": 20
}
EOF
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 4，无密码 ssh 登录其它节点
如果没有特殊指明，本文档的所有操作均在 kube-node1 节点上执行，然后远程分发文件和执行命令。稍候会介绍一个神器，通过这个，就能够实现基本上全程在节点 1 中部署整个集群。
设置 kube-node1 可以无密码登录所有节点的 k8s 和 root 账户：
[k8s@kube-node1 k8s]$ ssh-keygen -t rsa
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node1
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node2
[k8s@kube-node1 k8s]$ ssh-copy-id root@kube-node3
[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node1
[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node2
[k8s@kube-node1 k8s]$ ssh-copy-id k8s@kube-node3
1
2
3
4
5
6
7
2
3
4
5
6
7
以后的部署操作，也就在 kube-node1 的 k8s 用户家目录下进行了，不要觉得不方便。
# 5，将可执行文件路径 /opt/k8s/bin 添加到 PATH 变量中
在每台机器上添加环境变量：
$ sudo sh -c "echo 'PATH=/opt/k8s/bin:$PATH:$HOME/bin:$JAVA_HOME/bin' >>/root/.bashrc"
$ echo 'PATH=/opt/k8s/bin:$PATH:$HOME/bin:$JAVA_HOME/bin' >>~/.bashrc
1
2
2
# 6，环境变量
现在将要引入一个环境变量，以便于整个流程的部署工作，一开始有点不大适应，但是习惯了这种操作之后，发现这个思路，结合在 k8s 的部署之上，是真的犀利。
定义变量之前，我们先定义所有这次部署 k8s 所工作的目录。
mkdir -p /opt/k8s/bin/
1
变量内容如下：
cat > /opt/k8s/bin/environment.sh << "EOF"
#!/usr/bin/bash
# 生成 EncryptionConfig 所需的加密 key
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
# 最好使用当前未用的网段来定义服务网段和Pod网段
# 服务网段，部署前路由不可达，部署后集群内路由可达(kube-proxy 和 ipvs 保证)
export SERVICE_CIDR="10.254.0.0/16"
# Pod 网段，建议 /16 段地址，部署前路由不可达，部署后集群内路由可达(flanneld 保证)
export CLUSTER_CIDR="172.30.0.0/16"
# 服务端口范围 (NodePort Range)
export NODE_PORT_RANGE="8400-9000"
# 集群各机器 IP 数组
export NODE_IPS=(192.168.106.3 192.168.106.4 192.168.106.5)
# 集群各 IP 对应的 主机名数组
export NODE_NAMES=(kube-node1 kube-node2 kube-node3)
# kube-apiserver 的 VIP（HA 组件 keepalived 发布的 IP）
export MASTER_VIP=192.168.106.110
# kube-apiserver VIP 地址（HA 组件 haproxy 监听 8443 端口）
export KUBE_APISERVER="https://${MASTER_VIP}:8443"
# HA 节点，配置 VIP 的网络接口名称
export VIP_IF="eth0"
# etcd 集群服务地址列表
export ETCD_ENDPOINTS="https://192.168.106.3:2379,https://192.168.106.4:2379,https://192.168.106.5:2379"
# etcd 集群间通信的 IP 和端口
export ETCD_NODES="kube-node1=https://192.168.106.3:2380,kube-node2=https://192.168.106.4:2380,kube-node3=https://192.168.106.5:2380"
# flanneld 网络配置前缀
export FLANNEL_ETCD_PREFIX="/kubernetes/network"
# kubernetes 服务 IP (一般是 SERVICE_CIDR 中第一个IP)
export CLUSTER_KUBERNETES_SVC_IP="10.254.0.1"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
export CLUSTER_DNS_SVC_IP="10.254.0.2"
# 集群 DNS 域名
export CLUSTER_DNS_DOMAIN="cluster.local."
# 将二进制目录 /opt/k8s/bin 加到 PATH 中
export PATH=/opt/k8s/bin:$PATH
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
其中需要注意相关 IP 的设置对应，不要搞错了。
# 7，安装依赖包
在每台机器上安装依赖包：
$ sudo yum install -y epel-release
$ sudo yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp
1
2
2
上边是传统方式的安装方法，现在可以采用一下刚刚定义的方式安装一下：
定义一个脚本
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "yum install -y epel-release conntrack ipvsadm ipset jq sysstat curl iptables libseccomp"
done
EOF
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
注意：在第一个 EOF 中加引号，这样文本当中的变量就不会被替换。
执行这个脚本：
[k8s@kube-node1 ~]$ bash magic.sh
>>> 192.168.106.3
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.aliyun.com
* extras: mirrors.cn99.com
* updates: mirrors.aliyun.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do
>>> 192.168.106.4
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.aliyun.com
* extras: mirrors.cn99.com
* updates: mirrors.163.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do
>>> 192.168.106.5
Loaded plugins: fastestmirror
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirrors.cn99.com
* epel: mirrors.aliyun.com
* extras: mirrors.cn99.com
* updates: mirrors.aliyun.com
Package epel-release-7-11.noarch already installed and latest version
Package conntrack-tools-1.4.4-3.el7_3.x86_64 already installed and latest version
Package ipvsadm-1.27-7.el7.x86_64 already installed and latest version
Package ipset-6.29-1.el7.x86_64 already installed and latest version
Package jq-1.5-1.el7.x86_64 already installed and latest version
Package sysstat-10.1.5-13.el7.x86_64 already installed and latest version
Package curl-7.29.0-46.el7.x86_64 already installed and latest version
Package iptables-1.4.21-24.1.el7_5.x86_64 already installed and latest version
Package libseccomp-2.3.1-3.el7.x86_64 already installed and latest version
Nothing to do
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
因为已经装过了，所以会报没什么可装的。
# 8，关闭防火墙
将新内容导入到对应脚本，而后就这样，一劳永逸的，一步一步搭建下去。
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl stop firewalld && systemctl disable firewalld"
ssh root@${node_ip} "iptables -F && sudo iptables -X && sudo iptables -F -t nat && sudo iptables -X -t nat && iptables -P FORWARD ACCEPT"
done
EOF
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 9，关闭 swap 分区
如果开启了 swap 分区，kubelet 会启动失败 (可以通过将参数 –fail-swap-on 设置为 false 来忽略 swap on)，故需要在每台机器上关闭 swap 分区，为了防止开机自动挂载 swap 分区，也要注释 /etc/fstab 中相应的条目。
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab"
done
EOF
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 10，关闭 SELinux
关闭 SELinux，否则后续 K8S 挂载目录时可能报错 Permission denied：
$ sudo setenforce 0
$ grep SELINUX /etc/selinux/config
SELINUX=disabled
1
2
3
2
3
# 11，加载内核模块
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "modprobe br_netfilter && modprobe ip_vs"
done
EOF
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 12，设置系统参数
cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF
1
2
3
4
5
6
7
8
9
10
11
12
13
14
2
3
4
5
6
7
8
9
10
11
12
13
14
分发给三台主机，并加载。
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp kubernetes.conf root@{node_ip}:/etc/sysctl.d/kubernetes.conf
ssh root@${node_ip} "sysctl -p /etc/sysctl.d/kubernetes.conf"
done
EOF
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 13，安装一些基础包
安装几个常用但系统未必安装的包，同时配置一下时间同步，并把时间同步写入到定时任务当中。
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} 'yum -y install wget ntpdate lrzsz curl rsync && ntpdate -u cn.pool.ntp.org && echo "* * * * * /usr/sbin/ntpdate -u cn.pool.ntp.org &> /dev/null" > /var/spool/cron/root'
done
EOF
1
2
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
# 14，创建目录
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} 'mkdir -p /opt/k8s/bin && chown -R k8s /opt/k8s && mkdir -p /etc/kubernetes/cert && chown -R k8s /etc/kubernetes'
ssh root@${node_ip} 'mkdir -p /etc/etcd/cert && chown -R k8s /etc/etcd/cert && mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd'
done
EOF
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 15，分发环境变量配置
cat > magic.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
scp /opt/k8s/bin/environment.sh k8s@${node_ip}:/opt/k8s/bin/
ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF
1
2
3
4
5
6
7
8
9
10
2
3
4
5
6
7
8
9
10
# 16，参考
上次更新: 2024/11/19, 23:11:42
|