13-K8s自动部署-kubebin¶
- SaltStack自动化部署Kubernetes v1.10.3版本（支持TLS双向认证、RBAC授权、Flannel网络、ETCD集群、Kuber-Proxy使用LVS等）。
版本明细¶
Release-v1.10.3
- 测试通过系统：CentOS 7.4
- salt-ssh: 2017.7.4
- Kubernetes： v1.10.3
- Etcd: v3.3.1
- Docker: 17.12.1-ce
- Flannel： v0.10.0
- CNI-Plugins： v0.7.0 建议部署节点：最少三个节点，请配置好主机名解析（必备）
架构介绍¶
- 使用Salt Grains进行角色定义，增加灵活性。
- 使用Salt Pillar进行配置项管理，保证安全性。
- 使用Salt SSH执行状态，不需要安装Agent，保证通用性。
- 使用Kubernetes当前稳定版本v1.10.3，保证稳定性。
架构图¶
主机规划¶
| 主机名称 | 主机IP | 备注 |
|---|---|---|
| master1 | 192.168.1.26 | |
| node1 | 192.168.1.27 | |
| node2 | 192.168.1.28 | |
| node3(扩容节点) | 192.168.1.29 |
环境准备¶
1.系统初始化(必备)
1.1 设置主机名！！！
master1节点执行：
HOSTNAME=master1
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname
node1节点执行：
HOSTNAME=node1
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname
node2节点执行：
HOSTNAME=node2
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname
1.2 设置/etc/hosts保证主机名能够解析(master1 node1 node2节点执行)
cat >>/etc/hosts<<EOF
192.168.1.26 master1
192.168.1.27 node1
192.168.1.28 node2
EOF
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config && grep 'SELINUX=disabled' /etc/selinux/config
setenforce 0
getenforce
1.4 关闭NetworkManager和防火墙开启自启动
systemctl disable firewalld
systemctl disable NetworkManager
1.5 优化内核参数(master1 node1 node2节点执行)
cat >>/etc/sysctl.conf<<\EOF
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time=120
net.ipv4.ip_forward = 1
# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2
# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1
#iptables透明网桥的实现
# NOTE: kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1，如果不满足要求，那么 kube-proxy 只是将检查信息记录到日志中，kube-proxy 仍然会正常运行，但是这样通过 Kube-proxy 设置的某些 iptables 规则就不会工作。
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF
modprobe br_netfilter
sysctl -p
1.56 修改yum仓库为阿里源(master1 node1 node2节点执行)
rm -f /etc/yum.repos.d/*.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i "/mirrors.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
sed -i "/mirrors.cloud.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
yum clean all
yum repolist
以上必备条件必须严格检查，否则，一定不会部署成功！
开始部署¶
2.安装Salt-SSH并克隆本项目代码。
2.1 设置部署节点到其它所有节点的SSH免密码登录（包括本机）
yum -y install sshpass
cat >/root/.ssh/config<<EOF
Host *
Port 22
User root
StrictHostKeyChecking no
UserKnownHostsFile=/dev/nul
EOF
cd /root/
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
sshpass -pP@sswd ssh-copy-id master1
sshpass -pP@sswd ssh-copy-id node1
sshpass -pP@sswd ssh-copy-id node2
配置saltstack yum仓库
cat >/etc/yum.repos.d/salt-py3-latest.repo<<\EOF
[salt-py3-latest]
name=SaltStack Latest Release Channel Python 3 for RHEL/Centos $releasever
baseurl=https://repo.saltstack.com/py3/redhat/7/$basearch/latest
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/saltstack-signing-key, file:///etc/pki/rpm-gpg/centos7-signing-key
EOF
sed -i "s/repo.saltstack.com/mirrors.aliyun.com\/saltstack/g" /etc/yum.repos.d/salt-py3-latest.repo
sed -i "s#gpgcheck=1#gpgcheck=0#g" /etc/yum.repos.d/salt-py3-latest.repo
2.2 安装Salt SSH（注意：老版本的Salt SSH不支持Roster定义Grains，需要2017.7.4以上版本）
yum install -y salt-ssh git unzip vim
2.3 获取本项目代码，并放置在/srv目录
备用下载地址：链接: https://pan.baidu.com/s/1hyJ7-5LaWL-1dLjGZBVTlg 密码: h04i
git clone https://gitee.com/chriscentos/salt-kubebin.git
cd salt-kubebin/
mv * /srv/
/bin/cp /srv/roster /etc/salt/roster
/bin/cp /srv/master /etc/salt/master
2.4 下载二进制文件，也可以自行官方下载，为了方便国内用户访问，请在百度云盘下载,下载k8s-v1.10.3-auto.zip。 下载完成后，将文件移动到/srv/salt/k8s/目录下，并解压 Kubernetes二进制文件下载地址： 链接: https://pan.baidu.com/s/1BHn1AYrtEg7h24GQvz4Dgw 提取码: gjv9
备用下载地址：链接: https://pan.baidu.com/s/1Na-x8vuZgICnxlt82XpJIQ 密码: gs53
cd /srv/salt/k8s/
unzip k8s-v1.10.3-auto.zip
rm -f k8s-v1.10.3-auto.zip
ls -l files/
最终解压的文件列表如下：
[root@master1 k8s]# ls -l files/
total 0
drwxr-xr-x 2 root root 94 Jun 3 2018 cfssl-1.2
drwxr-xr-x 2 root root 195 Jun 3 2018 cni-plugins-amd64-v0.7.0
drwxr-xr-x 2 root root 33 Jun 3 2018 etcd-v3.3.1-linux-amd64
drwxr-xr-x 2 root root 47 Jun 3 2018 flannel-v0.10.0-linux-amd64
drwxr-xr-x 3 root root 17 Jun 3 2018 k8s-v1.10.3
3.Salt SSH管理的机器以及角色分配
- k8s-role: 用来设置K8S的角色
- etcd-role: 用来设置etcd的角色，如果只需要部署一个etcd，只需要在一台机器上设置即可
- etcd-name: 如果对一台机器设置了etcd-role就必须设置etcd-name
cat >/etc/salt/roster<<\EOF
master1:
host: 192.168.1.26
user: root
priv: /root/.ssh/id_rsa
minion_opts:
grains:
k8s-role: master
etcd-role: node
etcd-name: etcd-node1
node1:
host: 192.168.1.27
user: root
priv: /root/.ssh/id_rsa
minion_opts:
grains:
k8s-role: node
etcd-role: node
etcd-name: etcd-node2
node2:
host: 192.168.1.28
user: root
priv: /root/.ssh/id_rsa
minion_opts:
grains:
k8s-role: node
etcd-role: node
etcd-name: etcd-node3
EOF
4.修改对应的配置参数，本项目使用Salt Pillar保存配置
[root@master1 ~]# vim /srv/pillar/k8s.sls
#设置Master的IP地址(必须修改)
MASTER_IP: "192.168.1.26"
#设置ETCD集群访问地址（必须修改）
ETCD_ENDPOINTS: "https://192.168.1.26:2379,https://192.168.1.27:2379,https://192.168.1.28:2379"
#设置ETCD集群初始化列表（必须修改）
ETCD_CLUSTER: "etcd-node1=https://192.168.1.26:2380,etcd-node2=https://192.168.1.27:2380,etcd-node3=https://192.168.1.28:2380"
#通过Grains FQDN自动获取本机IP地址，请注意保证主机名解析到本机IP地址
NODE_IP: {{ grains['fqdn_ip4'][0] }}
#设置BOOTSTARP的TOKEN，可以自己生成
BOOTSTRAP_TOKEN: "ad6d5bb607a186796d8861557df0d17f"
#配置Service IP地址段
SERVICE_CIDR: "10.1.0.0/16"
#Kubernetes服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_KUBERNETES_SVC_IP: "10.1.0.1"
#Kubernetes DNS 服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_DNS_SVC_IP: "10.1.0.2"
#设置Node Port的端口范围
NODE_PORT_RANGE: "20000-40000"
#设置POD的IP地址段
POD_CIDR: "10.2.0.0/16"
#设置集群的DNS域名
CLUSTER_DNS_DOMAIN: "cluster.local."
#设置Docker Registry地址
DOCKER_REGISTRY: "http://192.168.1.26:5000"
5.执行SaltStack状态
5.1 测试Salt SSH联通性
salt-ssh -i '*' -r 'yum install -y python3 && swapoff -a && ntpdate time1.aliyun.com'
salt-ssh -i '*' test.ping
5.2 部署Etcd，由于Etcd是基础组建，需要先部署，目标为部署etcd的节点。
salt-ssh -L 'master1,node1,node2' state.sls k8s.etcd
5.3 部署K8S集群
salt-ssh '*' state.highstate
检查环境¶
6.测试Kubernetes安装
[root@master1 ~]# source /etc/profile
[root@master1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
[root@master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.56.12 Ready <none> 1m v1.10.3
192.168.56.13 Ready <none> 1m v1.10.3
[root@master1 ~]# kubectl run net-test --image=alpine --replicas=2 sleep 360000
deployment "net-test" created
需要等待拉取镜像，可能稍有的慢，请等待。
[root@master1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-5767cb94df-n9lvk 1/1 Running 0 14s 10.2.12.2 192.168.56.13
net-test-5767cb94df-zclc5 1/1 Running 0 14s 10.2.24.2 192.168.56.12
测试联通性，如果都能ping通，说明Kubernetes集群部署完毕，有问题请QQ群交流。
[root@master1 ~]# ping -c 1 10.2.12.2
PING 10.2.12.2 (10.2.12.2) 56(84) bytes of data.
64 bytes from 10.2.12.2: icmp_seq=1 ttl=61 time=8.72 ms
--- 10.2.12.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.729/8.729/8.729/0.000 ms
[root@master1 ~]# ping -c 1 10.2.24.2
PING 10.2.24.2 (10.2.24.2) 56(84) bytes of data.
64 bytes from 10.2.24.2: icmp_seq=1 ttl=61 time=22.9 ms
--- 10.2.24.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 22.960/22.960/22.960/0.000 ms
确认服务能够执行 logs exec 等指令;kubectl logs -f net-test-5767cb94df-n9lvk,此时会出现如下报错:
[root@master1 ~]# kubectl logs net-test-5767cb94df-n9lvk
error: You must be logged in to the server (the server has asked for the client to provide credentials ( pods/log net-test-5767cb94df-n9lvk))
由于上述权限问题，我们必需创建一个 apiserver-to-kubelet-rbac.yml 来定义权限，以供我们执行 logs、exec 等指令;
[root@master1 ~]# kubectl apply -f /srv/addons/apiserver-to-kubelet-rbac.yml
然后执行kubctl logs验证是否成功.
cd /srv/example/
kubectl create -f nginx-deployment.yaml
查看deployment
[root@linux-node1 ~]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
net-test 2 2 2 2 9m
nginx-deployment 3 3 3 0 24s
查看deployment详情
[root@linux-node1 ~]# kubectl describe deployment nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Sat, 08 Aug 2020 00:27:06 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision=1
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.13.12
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-6c45fc49cb (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m deployment-controller Scaled up replica set nginx-deployment-6c45fc49cb to 3
查看deployment容器是否启动
[root@linux-node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-5767cb94df-mbj9s 1/1 Running 0 12m 10.2.36.3 192.168.56.13
net-test-5767cb94df-rwkf5 1/1 Running 0 12m 10.2.36.2 192.168.56.13
nginx-deployment-6c45fc49cb-2mkdk 1/1 Running 0 3m 10.2.36.5 192.168.56.13
nginx-deployment-6c45fc49cb-gfj65 1/1 Running 0 3m 10.2.36.6 192.168.56.13
nginx-deployment-6c45fc49cb-nsw68 1/1 Running 0 3m 10.2.36.4 192.168.56.13
查看deployment service服务状态
[root@linux-node1 example]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 2h
验证容器是否可用
[root@linux-node1 ~]# curl --head 10.2.36.5
HTTP/1.1 200 OK
Server: nginx/1.13.12
Date: Fri, 07 Aug 2020 16:31:35 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 09 Apr 2018 16:01:09 GMT
Connection: keep-alive
ETag: "5acb8e45-264"
Accept-Ranges: bytes
扩容node¶
matse1节点添加hosts文件解析
cat >>/etc/hosts<<EOF
192.168.1.29 node3
EOF
matse1节点配置ssh免密
sshpass -pP@sswd ssh-copy-id node3
1.1 设置主机名！！！
node3节点执行：
HOSTNAME=node3
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname
1.2 设置/etc/hosts保证主机名能够解析(master1 node1 node2节点执行)
cat >>/etc/hosts<<EOF
192.168.1.26 master1
192.168.1.27 node1
192.168.1.28 node2
192.168.1.29 node3
EOF
1.3 关闭SELinux和防火墙
sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config && grep 'SELINUX=disabled' /etc/selinux/config
setenforce 0
getenforce
1.4 关闭NetworkManager和防火墙开启自启动
systemctl disable firewalld
systemctl disable NetworkManager
1.5 优化内核参数(master1 node1 node2节点执行)
```bash cat >>/etc/sysctl.conf<<\EOF
For more information, see sysctl.conf(5) and sysctl.d(5).¶
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0 net.ipv4.neigh.default.gc_stale_time=120 net.ipv4.ip_forward = 1
see details in https://help.aliyun.com/knowledge_detail/39428.html¶
net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.lo.arp_announce=2 net.ipv4.conf.all.arp_announce=2
see details in https://help.aliyun.com/knowledge_detail/41334.html¶
net.ipv4.tcp_max_tw_buckets = 5000 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_synack_retries = 2 kernel.sysrq = 1
iptables透明网桥的实现¶
NOTE: kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1，如果不满足要求，那么 kube-proxy 只是将检查信息记录到日志中，kube-proxy 仍然会正常运行，但是这样通过 Kube-proxy 设置的某些 iptables 规则就不会工作。¶
net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-arptables = 1 EOF modprobe br_netfilter sysctl -p ```
1.6 修改yum仓库为阿里源(master1 node1 node2节点执行)
rm -f /etc/yum.repos.d/*.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i "/mirrors.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
sed -i "/mirrors.cloud.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
yum clean all
yum repolist
1.7.如何新增Kubernetes节点(master1节点执行)
- 1.设置SSH无密码登录
- 2.在/etc/salt/roster里面，增加对应的机器
- 3.执行SaltStack状态salt-ssh '*' state.highstate。
[root@master1 ~]# vim /etc/salt/roster node3: host: 192.168.1.29 user: root priv: /root/.ssh/id_rsa minion_opts: grains: k8s-role: node etcd-role: node etcd-name: etcd-node4
验证是否可以联通
salt-ssh -i 'node3' -r 'yum install -y python3 && swapoff -a && ntpdate time1.aliyun.com'
salt-ssh -i 'node3' test.ping
开始扩容node节点
salt-ssh 'node3' state.highstate
检查扩容是否成功
[root@master1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.1.27 Ready <none> 1h v1.10.3
192.168.1.28 Ready <none> 1h v1.10.3
192.168.1.29 Ready <none> 1h v1.10.3
k8s删除node节点
https://blog.csdn.net/m0_60496726/article/details/120782349
etcdctl-bin:
file.managed:
- name: /opt/kubernetes/bin/etcdctl
- source: salt://k8s/files/{{ etcd_version }}/etcdctl
- user: root
- group: root
- mode: 755
kubelet-service
/opt/kubernetes/bin/etcdctl
sed -i "s#gpgcheck=1#gpgcheck=0#g" /etc/yum.repos.d/*
sed -i "s#gpgcheck=1#gpgcheck=0#g" /srv/salt/k8s/templates/docker/docker-ce.repo.template
sed -i "s#gpgcheck=1#gpgcheck=0#g" /srv/salt/k8s/templates/docker/kubernetes.repo.template