跳转至

14-K8s自动部署-kubeadm

12-SaltStack自动化部署Kubernetes(kubeadm HA版)

  • 在Kubernetes v1.13版本开始，kubeadm正式可以生产使用，但是kubeadm手动操作依然很繁琐，这里使用SaltStack进行自动化部署。

版本明细

Release-v1.19.6

  • 支持高可用HA
  • 测试通过系统： CentOS 7.9
  • salt-ssh: 3002.2
  • kubernetes： v1.17.16 v1.18.8 v1.19.6
  • docker-ce: 19.03.8

注意：从Kubernetes 1.16版本开始很多API名称发生了变化，例如常用的daemonsets, deployments, replicasets的API从extensions/v1beta1全部更改为apps/v1，所有老的YAML文件直接使用会有报错，请注意修改，详情可参考Kubernetes 1.19 CHANGELOG

架构介绍

建议部署节点：最少三个节点，请配置好主机名解析（必备） 1. 使用Salt Grains进行角色定义，增加灵活性。 2. 使用Salt Pillar进行配置项管理，保证安全性。 3. 使用Salt SSH执行状态，不需要安装Agent，保证通用性。 4. 使用Kubernetes当前稳定版本v1.19.6，保证稳定性。

环境准备

1.系统初始化(必备)

1.1 设置主机名！！！

master1节点执行：

HOSTNAME=master1
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname

node1节点执行：

HOSTNAME=node1
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname

node2节点执行：

HOSTNAME=node2
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname

1.2 设置/etc/hosts保证主机名能够解析(master1 node1 node2节点执行)

cat >>/etc/hosts<<EOF
192.168.1.26 master1
192.168.1.27 node1
192.168.1.28 node2
EOF

1.3 关闭SELinux和防火墙

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config && grep 'SELINUX=disabled' /etc/selinux/config 
setenforce 0
getenforce

1.4 关闭NetworkManager和防火墙开启自启动

systemctl disable firewalld
systemctl disable NetworkManager

1.5 优化内核参数(master1 node1 node2节点执行)

cat >>/etc/sysctl.conf<<\EOF
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time=120
net.ipv4.ip_forward = 1

# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2


# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1

#iptables透明网桥的实现
# NOTE: kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1，如果不满足要求，那么 kube-proxy 只是将检查信息记录到日志中，kube-proxy 仍然会正常运行，但是这样通过 Kube-proxy 设置的某些 iptables 规则就不会工作。

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF
modprobe br_netfilter
sysctl -p

1.5 修改yum仓库为阿里源(master1 node1 node2节点执行)

rm -f /etc/yum.repos.d/*.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i "/mirrors.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
sed -i "/mirrors.cloud.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
sed -i "s#gpgcheck=1#gpgcheck=0#g" /etc/yum.repos.d/*
yum clean all
yum repolist

以上必备条件必须严格检查，否则，一定不会部署成功！

部署手册

请参考开源书籍：Docker和Kubernetes实践指南 第五章节内容。

2.1 设置部署节点到其它所有节点的SSH免密码登录（包括本机）

yum -y install sshpass
cat >/root/.ssh/config<<EOF
Host *
    Port 22
    User root
    StrictHostKeyChecking no
    UserKnownHostsFile=/dev/nul
EOF
cd /root/
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 
sshpass -pP@sswd ssh-copy-id master1
sshpass -pP@sswd ssh-copy-id node1
sshpass -pP@sswd ssh-copy-id node2

配置saltstack yum仓库

cat >/etc/yum.repos.d/salt-py3-latest.repo<<\EOF
[salt-py3-latest]
name=SaltStack Latest Release Channel Python 3 for RHEL/Centos $releasever
baseurl=https://repo.saltstack.com/py3/redhat/7/$basearch/latest
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/saltstack-signing-key, file:///etc/pki/rpm-gpg/centos7-signing-key
EOF
sed -i "s/repo.saltstack.com/mirrors.aliyun.com\/saltstack/g" /etc/yum.repos.d/salt-py3-latest.repo 
sed -i "s#gpgcheck=1#gpgcheck=0#g" /etc/yum.repos.d/salt-py3-latest.repo  
yum clean all
yum repolist

2.2 安装Salt SSH（注意：老版本的Salt SSH不支持Roster定义Grains，需要2017.7.4以上版本）

yum install -y salt-ssh git unzip vim

2.3 获取本项目代码，并放置在/srv目录

git clone https://gitee.com/chriscentos/salt-kubeadm.git
cd salt-kubeadm/
cp -r * /srv/
/bin/cp /srv/roster /etc/salt/roster
/bin/cp /srv/master /etc/salt/master

备用下载地址：链接: https://pan.baidu.com/s/11G-j6WqoD6iqnOXD73birA 密码: 58fd

3.Salt SSH管理的机器以及角色分配

注意：下方单Master部署和多Master部署，选择其中之一执行。

Kubernetes单Master部署 

[root@linux-node1 ~]# vim /etc/salt/roster 
linux-node1:
  host: 192.168.56.11
  user: root
  priv: /root/.ssh/id_rsa
  minion_opts:
    grains:
      k8s-role: master

linux-node2:
  host: 192.168.56.12
  user: root
  priv: /root/.ssh/id_rsa
  minion_opts:
    grains:
      k8s-role: node

linux-node3:
  host: 192.168.56.13
  user: root
  priv: /root/.ssh/id_rsa
  minion_opts:
    grains:
      k8s-role: node

k8s-role: 用来设置K8S的角色

4.修改对应的配置参数，本项目使用Salt Pillar保存配置

[root@linux-node1 ~]# vim /srv/pillar/k8s.sls
#设置需要安装的Kubernetes版本
K8S_VERSION: "1.19.6"

#设置软件包的版本，和安装版本有区别
K8S_PKG_VERSION: "1.19.6-0"

#设置高可用集群VIP地址（部署高可用必须修改）
MASTER_VIP: "192.168.56.10"

#设置Master的IP地址(必须修改)
MASTER_IP: "192.168.56.11"

#通过Grains FQDN自动获取本机IP地址，请注意保证主机名解析到本机IP地址
NODE_IP: {{ grains['fqdn_ip4'][0] }}

#配置Service IP地址段
SERVICE_CIDR: "10.1.0.0/16"

#Kubernetes服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_KUBERNETES_SVC_IP: "10.1.0.1"

#Kubernetes DNS 服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_DNS_SVC_IP: "10.1.0.2"

#设置Node Port的端口范围
NODE_PORT_RANGE: "20000-40000"

#设置POD的IP地址段
POD_CIDR: "10.2.0.0/16"

#设置集群的DNS域名
CLUSTER_DNS_DOMAIN: "cluster.local."

5.集群部署

5.1 测试Salt SSH联通性

[root@linux-node1 ~]# salt-ssh -i '*' -r 'yum install -y python3 && swapoff -a && ntpdate time1.aliyun.com'
[root@linux-node1 ~]# salt-ssh -i '*' test.ping
linux-node2:
    True
linux-node3:
    True
linux-node1:
    True

此步骤是测试salt-ssh可以联通待部署的节点，保证没有问题，都返回True方可继续，如果有异常请先解决异常。保证机器没有SWAP分区，如果存在需要关闭，如果不是全新的系统，请谨慎执行关闭交换分区操作！

5.2 部署K8S集群基础组件

执行高级状态，会根据定义的角色再对应的机器部署对应的服务，例如安装kubeadm、kubelet、docker，加载IPVS内核模板，调整内核参数，生成kubeadm的配置文件等。 

sed -i "s#gpgcheck=1#gpgcheck=0#g" /srv/salt/k8s/templates/docker/docker-ce.repo.template
sed -i "s#gpgcheck=1#gpgcheck=0#g" /srv/salt/k8s/templates/docker/kubernetes.repo.template
sed -i "/gpg/d" /srv/salt/k8s/templates/docker/docker-ce.repo.template

[root@linux-node1 ~]# salt-ssh '*' state.highstate


sed -i "/gpg/d" /etc/yum.repos.d/*
yum repolist

喝杯咖啡休息一下，根据网络环境的不同，该步骤一般时长在5分钟以内，如果执行有失败可以再次执行即可！执行该操作会部署基本的环境，包括初始化需要用到的YAML。执行完毕之后请查看结果，需要保证所有的Failed：为0，说明初始化成功。 

Summary for linux-node3
-------------
Succeeded: 19 (changed=19)
Failed:     0
-------------
Total states run:     19
Total run time:  733.939 s

5.3 Master初始化

注意：下方单Master部署和多Master部署，选择其中之一执行。

  1. 单Master初始化

在上面的操作中，是自动化安装了Kubeadm、kubelet、docker进行了系统初始化，并生成了后续需要的yaml文件，下面的操作手工操作用于了解kubeadm的基本知识。 如果是在实验环境，只有1个CPU，在执行初始化的时候需要增加--ignore-preflight-errors=NumCPU。

你可以对kubeadm.yml进行定制，kubeadm会读取该文件进行初始化操作，这里我修改了负载均衡的配置使用IPVS,存放在/etc/sysconfig/kubeadm.yml

[root@linux-node1 ~]# kubeadm init --config /etc/sysconfig/kubeadm.yml --ignore-preflight-errors=NumCPU

需要下载Kubernetes所有应用服务镜像，根据网络情况，时间可能较长，请等待。可以在新窗口，docker images查看下载镜像进度。

5.4 为kubectl准备配置文件

kubectl默认会在用户的家目录寻找.kube/config配置文件，下面使用管理员的配置

[root@linux-node1 ~]# mkdir -p $HOME/.kube
[root@linux-node1 ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@linux-node1 ~]# chown $(id -u):$(id -g) $HOME/.kube/config

5.6 部署网络插件Flannel

需要保证所有Node的网卡名称一直，如果你的网卡名称不是eth0，请修改对应参数。 - --iface=eth0，修改为对应的网卡名称。

[root@linux-node1 ~]# kubectl create -f /etc/sysconfig/kube-flannel.yml

5.7 节点加入集群

  1. 在Master节点上输出加入集群的命令： 

    [root@linux-node1 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.56.11:6443 --token qnlyhw.cr9n8jbpbkg94szj     --discovery-token-ca-cert-hash sha256:cca103afc0ad374093f3f76b2f91963ac72eabea3d379571e88d403fc7670611

  2. 在Node节点上执行上面输出的命令，进行部署并加入集群。

如果执行的过程中，一直卡着无进度，请检查三台主机的时间是否同步，时间不同步会造成集群不正常，例如证书过期等。

在node1上执行

[root@linux-node2 ~]# kubeadm join 192.168.56.11:6443 --token qnlyhw.cr9n8jbpbkg94szj     --discovery-token-ca-cert-hash sha256:cca103afc0ad374093f3f76b2f91963ac72eabea3d379571e88d403fc7670611

在node2上执行

[root@linux-node3 ~]# kubeadm join 192.168.56.11:6443 --token qnlyhw.cr9n8jbpbkg94szj     --discovery-token-ca-cert-hash sha256:cca103afc0ad374093f3f76b2f91963ac72eabea3d379571e88d403fc7670611

集群测试

6.测试Kubernetes安装

查看节点状态

[root@linux-node1 ~]# kubectl get node
NAME            STATUS    ROLES     AGE       VERSION
192.168.56.11   Ready     master    1m        v1.19.6
192.168.56.12   Ready     <none>    1m        v1.19.6
192.168.56.13   Ready     <none>    1m        v1.19.6

7.测试Kubernetes集群和Flannel网络

  1. 创建Pod进行测试 

    [root@linux-node1 ~]# kubectl run net-test --image=alpine sleep 360000
deployment "net-test" created
需要等待拉取镜像，可能稍有的慢，请等待。

  2. 查看创建状态 

    [root@linux-node1 ~]# kubectl get pod -o wide
NAME       READY   STATUS    RESTARTS   AGE   IP         NODE                      NOMINATED NODE   READINESS GATES
net-test   1/1     Running   0          22s   10.2.12.2  linux-node2.example.com   <none>           <none>

  3. 测试联通性，如果都能ping通，说明Kubernetes集群部署完毕。 

    [root@linux-node1 ~]# ping -c 1 10.2.12.2
PING 10.2.12.2 (10.2.12.2) 56(84) bytes of data.
64 bytes from 10.2.12.2: icmp_seq=1 ttl=61 time=8.72 ms

--- 10.2.12.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.729/8.729/8.729/0.000 ms

扩容node

matse1节点添加hosts文件解析

cat >>/etc/hosts<<EOF
192.168.1.29 node3
EOF

matse1节点配置ssh免密

sshpass -pP@sswd ssh-copy-id node3

1.1 设置主机名！！！

node3节点执行：

HOSTNAME=node3
hostname ${HOSTNAME}
echo ${HOSTNAME} >/etc/hostname

1.2 设置/etc/hosts保证主机名能够解析(master1 node1 node2节点执行)

cat >>/etc/hosts<<EOF
192.168.1.26 master1
192.168.1.27 node1
192.168.1.28 node2
192.168.1.29 node3
EOF

1.3 关闭SELinux和防火墙

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config && grep 'SELINUX=disabled' /etc/selinux/config 
setenforce 0
getenforce

1.4 关闭NetworkManager和防火墙开启自启动

systemctl disable firewalld
systemctl disable NetworkManager

1.5 优化内核参数(master1 node1 node2节点执行)

cat >>/etc/sysctl.conf<<\EOF
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time=120
net.ipv4.ip_forward = 1

# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2


# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1

#iptables透明网桥的实现
# NOTE: kube-proxy 要求 NODE 节点操作系统中要具备 /sys/module/br_netfilter 文件，而且还要设置 bridge-nf-call-iptables=1，如果不满足要求，那么 kube-proxy 只是将检查信息记录到日志中，kube-proxy 仍然会正常运行，但是这样通过 Kube-proxy 设置的某些 iptables 规则就不会工作。

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF
modprobe br_netfilter
sysctl -p

1.6 修改yum仓库为阿里源(master1 node1 node2节点执行)

rm -f /etc/yum.repos.d/*.repo
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
sed -i "/mirrors.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
sed -i "/mirrors.cloud.aliyuncs.com/d" /etc/yum.repos.d/CentOS-Base.repo
yum clean all
yum repolist

1.7.如何新增Kubernetes节点(master1节点执行)

  • 1.设置SSH无密码登录
  • 2.在/etc/salt/roster里面，增加对应的机器
  • 3.执行SaltStack状态salt-ssh '*' state.highstate。
[root@master1 ~]# vim /etc/salt/roster 
node3:
  host: 192.168.1.29
  user: root
  priv: /root/.ssh/id_rsa
  minion_opts:
    grains:
      k8s-role: node
      etcd-role: node
      etcd-name: etcd-node4

验证是否可以联通

salt-ssh -i 'node3' -r 'yum install -y python3 && swapoff -a && ntpdate time1.aliyun.com'
salt-ssh -i 'node3' test.ping

开始扩容node节点

salt-ssh 'node3' state.highstate
  1. 在Master节点上输出加入集群的命令：
[root@linux-node1 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.56.11:6443 --token qnlyhw.cr9n8jbpbkg94szj     --discovery-token-ca-cert-hash sha256:cca103afc0ad374093f3f76b2f91963ac72eabea3d379571e88d403fc7670611
  1. 在Node节点上执行上面输出的命令，进行部署并加入集群。

如果执行的过程中，一直卡着无进度，请检查三台主机的时间是否同步，时间不同步会造成集群不正常，例如证书过期等。

在node1上执行

[root@linux-node2 ~]# kubeadm join 192.168.56.11:6443 --token qnlyhw.cr9n8jbpbkg94szj     --discovery-token-ca-cert-hash sha256:cca103afc0ad374093f3f76b2f91963ac72eabea3d379571e88d403fc7670611

检查扩容是否成功

[root@master1 ~]# kubectl get node
NAME      STATUS   ROLES    AGE   VERSION
master1   Ready    master   33m   v1.19.6
node1     Ready    <none>   28m   v1.19.6
node2     Ready    <none>   27m   v1.19.6
node3     Ready    <none>   55s   v1.19.6

必备插件

1.部署dashboard

kubeadm部署 Dashboard2.0.3 https://blog.csdn.net/weixin_38849917/article/details/107539193

在Service里面添加NodePort访问类型以及端口，我的recommended.yaml文件如下：

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort        #添加访问类型
  ports:
    - port: 443
      nodePort: 30001   #添加端口
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.0.3
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
    spec:
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.4
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            runAsUser: 1001
            runAsGroup: 2001
      serviceAccountName: kubernetes-dashboard
      nodeSelector:
        "kubernetes.io/os": linux
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

应用配置文件：

[root@master ~]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

查看所属node以及端口

kubectl -n kubernetes-dashboard get pod,svc -o wide

通过任意节点ip以及service的端口30001访问dashboard页面

创建create-admin.yaml文件：

[root@master ~]# cat create-admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

运行

kubectl apply -f create-admin.yaml

获取到用户的token以用作登录

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

把上面的token复制下来，登录https://192.168.1.26:30001/

删除dashboard：

kubectl delete -f create-admin.yaml
kubectl delete -f recommended.yaml

2.部署Ingress-Control

[root@master1 ~]# kubectl get node
NAME      STATUS   ROLES    AGE    VERSION
master1   Ready    master   108m   v1.19.6
node1     Ready    <none>   103m   v1.19.6
node2     Ready    <none>   102m   v1.19.6
node3     Ready    <none>   75m    v1.19.6

[root@master1 ~]# kubectl label nodes node1 edgenode=true
[root@master1 ~]# kubectl create -f /srv/addons/traefik-ingress/

3.部署Helm3

HELM是Kubernetes的包管理工具。使用Helm可以快速的安装和部署应用到Kubernetes上。

1.部署Helm

# https://get.helm.sh/helm-v3.2.4-linux-amd64.tar.gz


cd /usr/local/src
wget https://gitee.com/chriscentos/salt-kubeadm/attach_files/1030391/download/helm-v3.2.4-linux-amd64.tar.gz
tar zxf helm-v3.2.4-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/

2.验证安装是否成功

[root@linux-node1 ~]# helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"b29d20baf09943e134c2fa5e1e1cab3bf93315fa", GitTreeState:"clean", GoVersion:"go1.13.7"}