02-K8S分布式集群入门¶
想要快速的体验Kubernetes的功能，官方提供了非常多的部署方案，可以使用官方提供的kubeadm以容器的方式运行Kubernetes集群，也可以使用二进制方式部署更有利于理解Kubernetes的架构。
注意：请不要把目光仅仅放在部署上，要慢慢的了解其本质。
Kubernetes v1.13版本发布后，kubeadm才正式进入GA，可以生产使用。目前Kubernetes的对应镜像仓库，在国内阿里云也有了镜像站点，使用kubeadm部署Kubernetes集群变得简单并且容易了很多，本文使用kubeadm带领大家快速部署Kubernetes v1.13.3版本。
K8s架构介绍¶
master节点服务介绍¶
API Server: 提供Kubernetes API接口，主要处理REST操作以及更新ETCD的对象，所有资源池增删改查的唯一入口。
Scheduler: 资源调度，负责Pod到Node的调度
Controller Manager：所有其他集群级别的功能，目前由控制器Manger执行，资源池对象的自动化控制中心。
ETCD：所有持久化的状态信息存储
在ETCD中。
master节点服务详解¶
Apiserver提供集群管理的REST API接口，包括认证授权，数据校验以及集群状态表更等
-
只有API server才能直接操作etcd
-
其他模块通过API server查询或修改数据
-
提供其他模块之间的数据交互和通信枢纽
Scheduler负责分配调度Pod到集群内的node节点
-
监听kube-apiserver 查询还未分配Node的Pod
-
根据调度策略为这些Pod分配节点
Controller-manager由一系列的控制器组成，它通过apiserver监控整个集群的状态，并确保集群处于预期的工作状态。
node节点服务¶
Kubelet：管理Pods以及容器，镜像，Volume等，实现对集群对节点的管理
Kube-proxy: 提供网络代理以及负载均衡，实现与service通信
Docker Engine：负责节点的容器管理工作
小结：
- k8s支持多个容器得管理技术
Pod¶
1.RC是k8s集群中最早的保证Pod高可用的API对象，通过监控运行中的Pod来保证集群中运行指定数目的Pod副本
2.指定的数目可以是多个也可以是1个，少于指定数目，RC就是启动运行的新的Pod副本；多于指定数目，RC就会杀死多于的Pod副本。
3.即使在指定数目为1的情况下，通过RC运行Pod也比直接运行Pod更明智，因为RC可以发挥高可用的能力，保证永远有1个Pod在运行。
4.RS是新一代RC，提供同样的高可用能力，区别主要在于RS后来居上，能支持更多的匹配模式，副本集对象一般不单独使用，而是作为部署的理想状态参数使用。
5.RS是k8s 1.2中出现的概念，是RC的升级，一般和deployment的共同使用
Deployment¶
Deployment表示用户对k8s集群的一次更新操作，Deployment是一个比RS应用模式更广的API对象。
可以是创建一个新的服务，更新一个新的服务，也可以是滚动升级一个服务，滚动升级一个服务，实际是创建一个新的RS，然后逐渐将新RS中副本数增加到理想状态，将旧RS副本数减小到0的复合操作；
这样一个复合操作用一个RS是不太好描述的，所以用一个更通用的deploymeng来描述
Service¶
RC，RS和Deployment只是保证了支撑服务器的POD的数量，但是没有解决如何访问这些服务的问题，一个Pod只是一个运行服务的实例，随时可能在一个节点上停止，在另一个节点以一个新的IP启动一个新的Pod，因此不能以确定的IP和端口号提供服务。
要稳定地提供服务需要服务发现和负载均衡能力，服务发现完成的工作，是针对客户端访问的服务，找到对应的后端服务实例。
在k8s集群中，客户端需要访问的服务就是service对象，每个service会对应一个集群内部有效的虚拟IP，集群内部通过虚拟IP访问一个服务。
K8s的ip¶
Node IP: 节点设备的ip，如物理机，虚拟机等容器宿主的实际IP
Pod IP：Pod的IP地址，是根据docker0网络IP段进行分配的
Cluster IP: Service的IP，是一个虚拟IP，仅作用于service对象，由k8s管理和分配，需要结合service port才能使用，单独的IP没有通信功能，集群外访问需要一些修改。
在K8S集群内部，nodeip podip clusterip的通信机制是由k8s制定的路由规则，不是ip路由
命名空间¶
Kubernetes 支持多个虚拟集群，它们底层依赖于同一个物理集群。 这些虚拟集群被称为命名空间。
命名空间适用于存在很多跨多个团队或项目的用户的场景。对于只有几到几十个用户的集群，根本不需要创建或考虑命名空间。当需要名称空间提供的功能时，请开始使用它们。 命名空间为名称提供了一个范围。资源的名称需要在命名空间内是唯一的，但不能跨命名空间。命名空间不能相互嵌套，每个 Kubernetes 资源只能在一个命名空间中。 命名空间是在多个用户之间划分集群资源的一种方法（通过资源配额）。 在 Kubernetes 未来版本中，相同命名空间中的对象默认将具有相同的访问控制策略。 不需要使用多个命名空间来分隔轻微不同的资源，例如同一软件的不同版本：使用 labels来区分同一命名空间中的不同资源。
基础环境准备¶
主机规划¶
| 主机名称 | IP地址（NAT） | 描述 |
|---|---|---|
| Linux-node1 | eth0: 192.168.56.11 | Kubernets Master节点/Etcd节点(部署机) |
| Linux-node2 | eth0: 192.168.56.12 | Kubernets Node节点/Etcd节点 |
| Linux-node3 | eth0: 192.168.56.13 | Kubernets Node节点/Etcd节点 |
| 备注 | 如果有条件可以部署多个kubernets node实验效果更佳 |
部署规划¶
-
环境准备
-
软件包准备
-
开始部署
-
所有文件存放在/opt/kubernetes目录下
-
所有源码包放在/usr/local/src/目录下
-
使用二进制方式进行部署
软件包下载地址¶
百度网盘下载地址：
链接: https://pan.baidu.com/s/1nVeiDHy0e4CAYcORPqw9vg 提取码: 563c
备用下载地址：
链接: https://pan.baidu.com/s/1HW9bDn5v4FI4S5i2WIK5vg 提取码: rx85
上传解压软件包¶
Linux-node1(192.168.56.11)机器上执行
将软件包上传至192.168.56.11服务器/usr/local/src目录
scp k8s-v1.10.1-manual.zip 192.168.1.26:/usr/local/src/
[root@linux-node1 ~]# :
cd /usr/local/src/
yum -y install unzip
unzip k8s-v1.10.1-manual.zip
mv k8s-v1.10.1-manual/k8s-v1.10.1/* .
解压后源码包列表如下：
[root@linux-node1 src]# ll
total 1178908
-rw-r--r-- 1 root root 6595195 Mar 30 2016 cfssl-certinfo_linux-amd64
-rw-r--r-- 1 root root 2277873 Mar 30 2016 cfssljson_linux-amd64
-rw-r--r-- 1 root root 10376657 Mar 30 2016 cfssl_linux-amd64
-rw-r--r-- 1 root root 17108856 Apr 12 2018 cni-plugins-amd64-v0.7.1.tgz
-rw-r--r-- 1 root root 10562874 Mar 30 2018 etcd-v3.2.18-linux-amd64.tar.gz
-rw-r--r-- 1 root root 9706487 Jan 24 2018 flannel-v0.10.0-linux-amd64.tar.gz
drwxr-xr-x 3 root root 25 Apr 23 2018 k8s-v1.10.1-manual
-rw-r--r-- 1 root root 593725046 Aug 5 14:07 k8s-v1.10.1-manual.zip
-rw-r--r-- 1 root root 13344537 Apr 13 2018 kubernetes-client-linux-amd64.tar.gz
-rw-r--r-- 1 root root 112427817 Apr 13 2018 kubernetes-node-linux-amd64.tar.gz
-rw-r--r-- 1 root root 428337777 Apr 13 2018 kubernetes-server-linux-amd64.tar.gz
-rw-r--r-- 1 root root 2716855 Apr 13 2018 kubernetes.tar.gz
解压k8s相关软件包
tar zxf kubernetes.tar.gz
tar zxf kubernetes-server-linux-amd64.tar.gz
tar zxf kubernetes-client-linux-amd64.tar.gz
tar zxf kubernetes-node-linux-amd64.tar.gz
配置免密¶
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# ssh-keygen -t rsa
[root@linux-node1 ~]# ssh-copy-id 192.168.56.11
[root@linux-node1 ~]# ssh-copy-id 192.168.56.12
[root@linux-node1 ~]# ssh-copy-id 192.168.56.13
配置YUM仓库¶
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
cd /etc/yum.repos.d
wget http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sed -i "s#https#http#g" /etc/yum.repos.d/docker-ce.repo
sed -i "s#download.docker.com#download.docker.com/docker-ce#g" /etc/yum.repos.d/docker-ce.repo
yum clean all
yum repolist
设置k8s环境变量¶
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/kubernetes/bin/"
echo 'export PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/opt/kubernetes/bin/"' >>/etc/profile
source /etc/profile
创建统一部署目录¶
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}
配置内核参数¶
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
cat >>/etc/sysctl.conf<<EOF
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time=120
net.ipv4.ip_forward = 1
# see details in https://help.aliyun.com/knowledge_detail/39428.html
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_announce=2
# see details in https://help.aliyun.com/knowledge_detail/41334.html
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1
# iptables透明网桥的实现
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-arptables = 1
EOF
modprobe br_netfilter
sysctl -p
Docker安装启动¶
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
yum install -y docker-ce
systemctl enable docker
systemctl start docker
systemctl status docker
CA证书创建和分发¶
Kubernetes系统各组件需要使用TLS证书对通信进行加密
CA证书管理：
Easyrs
Openssl
cfssl
安装CFSSL
[root@linux-node1 yum.repos.d]# cd /usr/local/src
[root@linux-node1 src]# chmod +x cfssl*
[root@linux-node1 src]# mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo
[root@linux-node1 src]# mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljson
[root@linux-node1 src]# mv cfssl_linux-amd64 /opt/kubernetes/bin/cfssl
分发证书到node节点
[root@linux-node1 src]# scp /opt/kubernetes/bin/cfssl* 192.168.56.12:/opt/kubernetes/bin
[root@linux-node1 src]# scp /opt/kubernetes/bin/cfssl* 192.168.56.13:/opt/kubernetes/bin
初始化cfssl
[root@linux-node1 src]# mkdir ssl && cd ssl
[root@linux-node1 ssl]# cfssl print-defaults config > config.json
[root@linux-node1 ssl]# cfssl print-defaults csr > csr.json
创建用来生成 CA 文件的 JSON 配置文件
[root@linux-node1 ssl]# cat >ca-config.json<<EOF
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
}
}
}
}
EOF
创建用来生成 CA 证书签名请求（CSR）的 JSON 配置文件
[root@linux-node1 ssl]# cat >ca-csr.json<<EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成CA证书（ca.pem）和密钥（ca-key.pem）
[root@linux-node1 ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@linux-node1 ssl]# ls -l ca*
-rw-r--r-- 1 root root 290 Aug 7 19:24 ca-config.json
-rw-r--r-- 1 root root 1001 Aug 7 19:25 ca.csr
-rw-r--r-- 1 root root 208 Aug 7 19:25 ca-csr.json
-rw------- 1 root root 1675 Aug 7 19:25 ca-key.pem
-rw-r--r-- 1 root root 1359 Aug 7 19:25 ca.pem
分发证书
[root@linux-node1 ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl
[root@linux-node1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.56.12:/opt/kubernetes/ssl
[root@linux-node1 ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 192.168.56.13:/opt/kubernetes/ssl
到此ca证书创建和分发完毕！
ETCD集群部署¶
准备etcd软件包
[root@linux-node1 ssl]# cd /usr/local/src/
[root@linux-node1 src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz
[root@linux-node1 src]# cd etcd-v3.2.18-linux-amd64
[root@linux-node1 etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/
[root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.12:/opt/kubernetes/bin/
[root@linux-node1 etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 192.168.56.13:/opt/kubernetes/bin/
创建 etcd 证书签名请求
[root@linux-node1 etcd-v3.2.18-linux-amd64]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cat >etcd-csr.json<<EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.56.11",
"192.168.56.12",
"192.168.56.13"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成 etcd 证书和私钥
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
会生成以下证书文件
[root@linux-node1 ssl]# ls -l etcd*
-rw-r--r-- 1 root root 1062 Aug 7 19:32 etcd.csr
-rw-r--r-- 1 root root 287 Aug 7 19:31 etcd-csr.json
-rw------- 1 root root 1679 Aug 7 19:32 etcd-key.pem
-rw-r--r-- 1 root root 1436 Aug 7 19:32 etcd.pem
将证书移动到/opt/kubernetes/ssl目录下
[root@linux-node1 ssl]# cp etcd*.pem /opt/kubernetes/ssl
[root@linux-node1 ssl]# scp etcd*.pem 192.168.56.12:/opt/kubernetes/ssl
[root@linux-node1 ssl]# scp etcd*.pem 192.168.56.13:/opt/kubernetes/ssl
设置ETCD配置文件
Linux-node1(192.168.56.11)机器上执行
cat >/opt/kubernetes/cfg/etcd.conf<<\EOF
#[member]
ETCD_NAME="etcd-node1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.56.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.11:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.11:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.11:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
EOF
Linux-node2(192.168.56.12)机器上执行
cat >/opt/kubernetes/cfg/etcd.conf<<\EOF
#[member]
ETCD_NAME="etcd-node2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.56.12:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.12:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.12:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.12:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
EOF
Linux-node3(192.168.56.13)机器上执行
cat >/opt/kubernetes/cfg/etcd.conf<<\EOF
#[member]
ETCD_NAME="etcd-node3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://192.168.56.13:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.13:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.13:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-node1=https://192.168.56.11:2380,etcd-node2=https://192.168.56.12:2380,etcd-node3=https://192.168.56.13:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.13:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
EOF
创建ETCD系统服务
[root@linux-node1 ssl]# cat >/etc/systemd/system/etcd.service<<\EOF
[Unit]
Description=Etcd Server
After=network.target
[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"
Type=notify
[Install]
WantedBy=multi-user.target
EOF
分发文件到node节点
[root@linux-node1 ssl]# scp /etc/systemd/system/etcd.service 192.168.56.12:/etc/systemd/system/
[root@linux-node1 ssl]# scp /etc/systemd/system/etcd.service 192.168.56.13:/etc/systemd/system/
重新加载系统服务
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
systemctl daemon-reload
systemctl enable etcd
mkdir /var/lib/etcd
systemctl start etcd
systemctl status etcd
下面需要大家在所有的 etcd 节点重复上面的步骤，直到所有机器的 etcd 服务都已启动。
验证集群
[root@linux-node1 ssl]# etcdctl --endpoints=https://192.168.56.11:2379 \
--ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/etcd.pem \
--key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health
member 435fb0a8da627a4c is healthy: got healthy result from https://192.168.56.12:2379
member 6566e06d7343e1bb is healthy: got healthy result from https://192.168.56.11:2379
member ce7b884e428b6c8c is healthy: got healthy result from https://192.168.56.13:2379
cluster is healthy
Master节点部署¶
部署Kubernetes API服务部署¶
准备软件包
[root@linux-node1 ssl]# cd /usr/local/src/kubernetes
[root@linux-node1 kubernetes]# cp server/bin/kube-apiserver /opt/kubernetes/bin/
[root@linux-node1 kubernetes]# cp server/bin/kube-controller-manager /opt/kubernetes/bin/
[root@linux-node1 kubernetes]# cp server/bin/kube-scheduler /opt/kubernetes/bin/
创建生成CSR的 JSON 配置文件
[root@linux-node1 kubernetes]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cat >kubernetes-csr.json<<EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.56.11",
"10.1.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成 kubernetes 证书和私钥
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
[root@linux-node1 ssl]# cp kubernetes*.pem /opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp kubernetes*.pem 192.168.56.12:/opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp kubernetes*.pem 192.168.56.13:/opt/kubernetes/ssl/
创建 kube-apiserver 使用的客户端 token 文件
[root@linux-node1 ~]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
ad6d5bb607a186796d8861557df0d17f #可以自生成，也可以使用我的
[root@linux-node1 ~]# cat >/opt/kubernetes/ssl/bootstrap-token.csv<<EOF
ad6d5bb607a186796d8861557df0d17f,kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
创建基础用户名/密码认证配置
[root@linux-node1 ~]# cat >/opt/kubernetes/ssl/basic-auth.csv<<EOF
admin,admin,1
readonly,readonly,2
EOF
配置Kubernetes API Server
[root@linux-node1 ~]# cat >/usr/lib/systemd/system/kube-apiserver.service<<\EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
--admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
--bind-address=192.168.56.11 \
--insecure-bind-address=127.0.0.1 \
--authorization-mode=Node,RBAC \
--runtime-config=rbac.authorization.k8s.io/v1 \
--kubelet-https=true \
--anonymous-auth=false \
--basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
--service-cluster-ip-range=10.1.0.0/16 \
--service-node-port-range=20000-40000 \
--tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
--tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/kubernetes/ssl/ca.pem \
--etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
--etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
--etcd-servers=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/log/api-audit.log \
--event-ttl=1h \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
启动API Server服务
[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl enable kube-apiserver
[root@linux-node1 ~]# systemctl start kube-apiserver
查看API Server服务状态
[root@linux-node1 ~]# systemctl status kube-apiserver
部署Controller Manager服务¶
配置Controller Manager服务
[root@linux-node1 ~]# cat >/usr/lib/systemd/system/kube-controller-manager.service<<\EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/kubernetes/bin/kube-controller-manager \
--address=127.0.0.1 \
--master=http://127.0.0.1:8080 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=10.1.0.0/16 \
--cluster-cidr=10.2.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--leader-elect=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
启动Controller Manager
[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl enable kube-controller-manager
[root@linux-node1 ~]# systemctl start kube-controller-manager
查看服务状态
[root@linux-node1 ~]# systemctl status kube-controller-manager
部署Kubernetes Scheduler¶
配置Kubernetes Scheduler
[root@linux-node1 ~]# cat >/usr/lib/systemd/system/kube-scheduler.service<<\EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/opt/kubernetes/bin/kube-scheduler \
--address=127.0.0.1 \
--master=http://127.0.0.1:8080 \
--leader-elect=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
部署验证服务
[root@linux-node1 ~]# systemctl daemon-reload
[root@linux-node1 ~]# systemctl enable kube-scheduler
[root@linux-node1 ~]# systemctl start kube-scheduler
[root@linux-node1 ~]# systemctl status kube-scheduler
部署安装 Kubectl client¶
准备二进制命令包
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/client/bin
[root@linux-node1 bin]# cp kubectl /opt/kubernetes/bin/
创建 admin 证书签名请求
[root@linux-node1 ~]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cat >admin-csr.json<<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
生成 admin 证书和私钥：
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes admin-csr.json | cfssljson -bare admin
[root@linux-node1 ssl]# ls -l admin*
-rw-r--r-- 1 root root 1009 Aug 7 22:02 admin.csr
-rw-r--r-- 1 root root 229 Aug 7 22:02 admin-csr.json
-rw------- 1 root root 1679 Aug 7 22:02 admin-key.pem
-rw-r--r-- 1 root root 1399 Aug 7 22:02 admin.pem
[root@linux-node1 ssl]# mv admin*.pem /opt/kubernetes/ssl/
设置集群参数
[root@linux-node1 ssl]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.56.11:6443
Cluster "kubernetes" set.
设置客户端认证参数
[root@linux-node1 src]# kubectl config set-credentials admin \
--client-certificate=/opt/kubernetes/ssl/admin.pem \
--embed-certs=true \
--client-key=/opt/kubernetes/ssl/admin-key.pem
User "admin" set.
设置上下文参数
[root@linux-node1 src]# kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=admin
Context "kubernetes" created.
设置默认上下文
[root@linux-node1 src]# kubectl config use-context kubernetes
Switched to context "kubernetes".
使用kubectl工具
[root@linux-node1 ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
Node节点部署¶
kubelet部署¶
二进制包准备 将软件包从linux-node1复制到linux-node2 linux-node3中去。
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/server/bin/
[root@linux-node1 bin]# cp kubelet kube-proxy /opt/kubernetes/bin/
[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.12:/opt/kubernetes/bin/
[root@linux-node1 bin]# scp kubelet kube-proxy 192.168.56.13:/opt/kubernetes/bin/
创建角色绑定
[root@linux-node1 ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
clusterrolebinding "kubelet-bootstrap" created
创建 kubelet bootstrapping kubeconfig 文件 设置集群参数
[root@linux-node1 ~]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.56.11:6443 \
--kubeconfig=bootstrap.kubeconfig
Cluster "kubernetes" set.
设置客户端认证参数
[root@linux-node1 ~]# kubectl config set-credentials kubelet-bootstrap \
--token=ad6d5bb607a186796d8861557df0d17f \
--kubeconfig=bootstrap.kubeconfig
User "kubelet-bootstrap" set.
设置上下文参数
[root@linux-node1 ~]# kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig
Context "default" created.
选择默认上下文
[root@linux-node1 ~]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
Switched to context "default".
[root@linux-node1 ~]# cp bootstrap.kubeconfig /opt/kubernetes/cfg
[root@linux-node1 ~]# scp bootstrap.kubeconfig 192.168.56.12:/opt/kubernetes/cfg
[root@linux-node1 ~]# scp bootstrap.kubeconfig 192.168.56.13:/opt/kubernetes/cfg
部署kubelet 1.设置CNI支持
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
mkdir -p /etc/cni/net.d
cat >/etc/cni/net.d/10-default.conf<<EOF
{
"name": "flannel",
"type": "flannel",
"delegate": {
"bridge": "docker0",
"isDefaultGateway": true,
"mtu": 1400
}
}
EOF
创建kubelet目录
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
mkdir /var/lib/kubelet
创建kubelet服务配置
Linux-node2(192.168.56.12)机器上执行
[root@linux-node2 ~]# cat >/usr/lib/systemd/system/kubelet.service<<\EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
--address=192.168.56.12 \
--hostname-override=192.168.56.12 \
--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \
--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--cert-dir=/opt/kubernetes/ssl \
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/kubernetes/bin/cni \
--cluster-dns=10.1.0.2 \
--cluster-domain=cluster.local. \
--hairpin-mode hairpin-veth \
--allow-privileged=true \
--fail-swap-on=false \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
EOF
Linux-node3(192.168.56.13)机器上执行
[root@linux-node2 ~]# cat >/usr/lib/systemd/system/kubelet.service<<\EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
--address=192.168.56.13 \
--hostname-override=192.168.56.13 \
--pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \
--experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--cert-dir=/opt/kubernetes/ssl \
--network-plugin=cni \
--cni-conf-dir=/etc/cni/net.d \
--cni-bin-dir=/opt/kubernetes/bin/cni \
--cluster-dns=10.1.0.2 \
--cluster-domain=cluster.local. \
--hairpin-mode hairpin-veth \
--allow-privileged=true \
--fail-swap-on=false \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
EOF
启动Kubelet查看服务状态
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet
查看csr请求 注意是在linux-node1上执行。
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# kubectl get csr
NAME AGE REQUESTOR CONDITION
node-csr-0_w5F1FM_la_SeGiu3Y5xELRpYUjjT2icIFk9gO9KOU 1m kubelet-bootstrap Pending
批准kubelet 的 TLS 证书请求
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve
certificatesigningrequest.certificates.k8s.io "node-csr-4ieDREHpilnBEaH96oSg8Cio1hVTBbrhnXhdS20tzTI" approved
certificatesigningrequest.certificates.k8s.io "node-csr-G5M16H6stvQI0-nlbpEfhd_m-H8CbEHYW7frms7OSR4" approved
执行完毕后，查看节点状态已经是Ready的状态了
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.56.12 Ready <none> 1m v1.10.1
192.168.56.13 Ready <none> 1m v1.10.1
kube-proxy部署¶
配置kube-proxy使用LVS
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
yum install -y ipvsadm ipset conntrack
创建 kube-proxy 证书请求
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cat >kube-proxy-csr.json<<EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成证书
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
分发证书到所有Node节点
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.12:/opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp kube-proxy*.pem 192.168.56.13:/opt/kubernetes/ssl/
创建kube-proxy配置文件
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ssl]# kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=https://192.168.56.11:6443 \
--kubeconfig=kube-proxy.kubeconfig
Cluster "kubernetes" set.
[root@linux-node1 ssl]# kubectl config set-credentials kube-proxy \
--client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \
--client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
User "kube-proxy" set.
[root@linux-node1 ssl]# kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
Context "default" created.
[root@linux-node1 ssl]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
Switched to context "default".
分发kubeconfig配置文件
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/
[root@linux-node1 ssl]# scp kube-proxy.kubeconfig 192.168.56.12:/opt/kubernetes/cfg/
[root@linux-node1 ssl]# scp kube-proxy.kubeconfig 192.168.56.13:/opt/kubernetes/cfg/
创建kube-proxy服务配置
Linux-node2(192.168.56.12)机器上执行
mkdir /var/lib/kube-proxy
cat >/usr/lib/systemd/system/kube-proxy.service<<\EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
--bind-address=192.168.56.12 \
--hostname-override=192.168.56.12 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
--feature-gates=SupportIPVSProxyMode=true \
--proxy-mode=ipvs \
--ipvs-min-sync-period=5s \
--ipvs-sync-period=5s \
--ipvs-scheduler=rr \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
Linux-node2(192.168.56.13)机器上执行
mkdir /var/lib/kube-proxy
cat >/usr/lib/systemd/system/kube-proxy.service<<\EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
--bind-address=192.168.56.13 \
--hostname-override=192.168.56.13 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
--feature-gates=SupportIPVSProxyMode=true \
--proxy-mode=ipvs \
--ipvs-min-sync-period=5s \
--ipvs-sync-period=5s \
--ipvs-scheduler=rr \
--logtostderr=true \
--v=2 \
--logtostderr=false \
--log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
启动Kubernetes Proxy
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy
查看服务状态 查看kube-proxy服务状态
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
[root@linux-node2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.0.1:443 rr persistent 10800
-> 192.168.56.11:6443 Masq 1 0 0
[root@linux-node3 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.0.1:443 rr persistent 10800
-> 192.168.56.11:6443 Masq 1 0 0
如果你在两台实验机器都安装了kubelet和proxy服务，使用下面的命令可以检查状态：
Linux-node1(192.168.56.11)机器上执行
[root@linux-node1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192.168.56.12 Ready <none> 22m v1.10.1
192.168.56.13 Ready <none> 22m v1.10.1
Flannel网络部署¶
Flannel网络部署¶
为Flannel生成证书
[root@linux-node1 ~]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# cat >flanneld-csr.json<<EOF
{
"CN": "flanneld",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
EOF
生成证书
[root@linux-node1 ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
-ca-key=/opt/kubernetes/ssl/ca-key.pem \
-config=/opt/kubernetes/ssl/ca-config.json \
-profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld
分发证书
[root@linux-node1 ssl]# cp flanneld*.pem /opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp flanneld*.pem 192.168.56.12:/opt/kubernetes/ssl/
[root@linux-node1 ssl]# scp flanneld*.pem 192.168.56.13:/opt/kubernetes/ssl/
下载Flannel软件包
[root@linux-node1 ~]# cd /usr/local/src
[root@linux-node1 src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz
[root@linux-node1 src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/
复制到linux-node2节点
[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.12:/opt/kubernetes/bin/
[root@linux-node1 src]# scp flanneld mk-docker-opts.sh 192.168.56.13:/opt/kubernetes/bin/
复制对应脚本到/opt/kubernetes/bin目录下。
[root@linux-node1 ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/
[root@linux-node1 bin]# cp remove-docker0.sh /opt/kubernetes/bin/
[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.12:/opt/kubernetes/bin/
[root@linux-node1 bin]# scp remove-docker0.sh 192.168.56.13:/opt/kubernetes/bin/
配置Flannel
[root@linux-node1 ~]# cat >/opt/kubernetes/cfg/flannel<<EOF
FLANNEL_ETCD="-etcd-endpoints=https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"
EOF
复制配置到其它节点上
[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.12:/opt/kubernetes/cfg/
[root@linux-node1 ~]# scp /opt/kubernetes/cfg/flannel 192.168.56.13:/opt/kubernetes/cfg/
设置Flannel系统服务
[root@linux-node1 ~]# cat >/usr/lib/systemd/system/flannel.service<<\EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker
Type=notify
[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
复制系统服务脚本到其它节点上
# scp /usr/lib/systemd/system/flannel.service 192.168.56.12:/usr/lib/systemd/system/
# scp /usr/lib/systemd/system/flannel.service 192.168.56.13:/usr/lib/systemd/system/
Flannel CNI集成¶
创建cni目录
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
mkdir /opt/kubernetes/bin/cni
下载CNI插件
[root@linux-node1 ~]# cd /usr/local/src/
[root@linux-node1 src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni
# scp -r /opt/kubernetes/bin/cni/* 192.168.56.12:/opt/kubernetes/bin/cni/
# scp -r /opt/kubernetes/bin/cni/* 192.168.56.13:/opt/kubernetes/bin/cni/
创建Etcd的key
Linux-node1(192.168.56.11)机器上执行
/opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \
--no-sync -C https://192.168.56.11:2379,https://192.168.56.12:2379,https://192.168.56.13:2379 \
mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1
启动flannel
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
systemctl daemon-reload
systemctl enable flannel
chmod +x /opt/kubernetes/bin/*
systemctl start flannel
systemctl status flannel
检查flannel是否安装成功
[root@linux-node1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:25:5e:f5 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.11/24 brd 192.168.56.255 scope global eth0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether c6:74:9e:da:33:0e brd ff:ff:ff:ff:ff:ff
inet 10.2.23.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
配置Docker使用Flannel¶
配置docker使用flannel，修改配置文件
[root@linux-node1 ~]# vim /usr/lib/systemd/system/docker.service
[Unit] #在Unit下面修改After和增加Requires
After=network-online.target firewalld.service flannel.service
Wants=network-online.target
Requires=flannel.service
[Service] #增加EnvironmentFile=-/run/flannel/docker
Type=notify
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS
将配置复制到另外两个节点
scp /usr/lib/systemd/system/docker.service 192.168.56.12:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/docker.service 192.168.56.13:/usr/lib/systemd/system/
重启Docker
Linux-node1,2,3(192.168.56.11,192.168.56.12.192.168.56.13)机器上执行
systemctl daemon-reload
systemctl restart docker
检查flanel网卡与docker0 是否为同一个网段
[root@linux-node1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:25:5e:f5 brd ff:ff:ff:ff:ff:ff
inet 192.168.56.11/24 brd 192.168.56.255 scope global eth0
valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether c6:74:9e:da:33:0e brd ff:ff:ff:ff:ff:ff
inet 10.2.23.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:6a:b2:eb:67 brd ff:ff:ff:ff:ff:ff
inet 10.2.23.1/24 brd 10.2.23.255 scope global docker0
valid_lft forever preferred_lft forever
创建第一个K8S应用¶
配置镜像加速器¶
Linux-node2,3(192.168.56.12.192.168.56.13)机器上执行
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://1v0q5mvy.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl status docker
准备yaml文件¶
[root@linux-node1 ~]# yum -y install git
[root@linux-node1 ~]# git clone https://github.com/unixhot/salt-kubebin.git
运行一组test容器¶
创建一个测试用的deployment
[root@linux-node1 ~]# kubectl run net-test --image=alpine --replicas=2 sleep 360000
查看获取IP情况
[root@linux-node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-5767cb94df-mbj9s 0/1 ContainerCreating 0 5s <none> 192.168.56.13
net-test-5767cb94df-rwkf5 0/1 ContainerCreating 0 5s <none> 192.168.56.13
查看deployment任务
[root@linux-node1 ~]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
net-test 2 2 2 0 56s
查看deployment任务详情
[root@linux-node1 ~]# kubectl describe deployment net-test
Name: net-test
Namespace: default
CreationTimestamp: Thu, 06 Aug 2020 19:44:39 +0800
Labels: run=net-test
Annotations: deployment.kubernetes.io/revision=1
Selector: run=net-test
Replicas: 2 desired | 2 updated | 2 total | 0 available | 2 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: run=net-test
Containers:
net-test:
Image: alpine
Port: <none>
Host Port: <none>
Args:
sleep
360000
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available False MinimumReplicasUnavailable
Progressing True ReplicaSetUpdated
OldReplicaSets: <none>
NewReplicaSet: net-test-5767cb94df (2/2 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 1m deployment-controller Scaled up replica set net-test-5767cb94df to 2
容器启动成功
[root@linux-node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-5767cb94df-mbj9s 1/1 Running 0 59s 10.2.36.3 192.168.56.13
net-test-5767cb94df-rwkf5 1/1 Running 0 59s 10.2.36.2 192.168.56.13
我们可以ping测试容器是否可用
[root@linux-node1 ~]# ping 10.2.36.3
PING 10.2.36.3 (10.2.36.3) 56(84) bytes of data.
64 bytes from 10.2.36.3: icmp_seq=1 ttl=63 time=0.895 ms
64 bytes from 10.2.36.3: icmp_seq=2 ttl=63 time=0.237 ms
64 bytes from 10.2.36.3: icmp_seq=3 ttl=63 time=0.274 ms
^C
--- 10.2.36.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.237/0.468/0.895/0.302 ms
[root@linux-node1 ~]# ping 10.2.36.2
PING 10.2.36.2 (10.2.36.2) 56(84) bytes of data.
64 bytes from 10.2.36.2: icmp_seq=1 ttl=63 time=0.480 ms
64 bytes from 10.2.36.2: icmp_seq=2 ttl=63 time=0.476 ms
64 bytes from 10.2.36.2: icmp_seq=3 ttl=63 time=0.342 ms
^C
--- 10.2.36.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.342/0.432/0.480/0.068 ms
运行一组nginx容器¶
[root@linux-node1 ~]# cd /root/salt-kubebin/example
[root@linux-node1 example]# kubectl create -f nginx-deployment.yaml
deployment.apps "nginx-deployment" created
查看deployment
[root@linux-node1 ~]# kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
net-test 2 2 2 2 9m
nginx-deployment 3 3 3 0 24s
查看deployment详情
[root@linux-node1 ~]# kubectl describe deployment nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Sat, 08 Aug 2020 00:27:06 +0800
Labels: app=nginx
Annotations: deployment.kubernetes.io/revision=1
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.13.12
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-6c45fc49cb (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m deployment-controller Scaled up replica set nginx-deployment-6c45fc49cb to 3
查看deployment容器是否启动
[root@linux-node1 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
net-test-5767cb94df-mbj9s 1/1 Running 0 12m 10.2.36.3 192.168.56.13
net-test-5767cb94df-rwkf5 1/1 Running 0 12m 10.2.36.2 192.168.56.13
nginx-deployment-6c45fc49cb-2mkdk 1/1 Running 0 3m 10.2.36.5 192.168.56.13
nginx-deployment-6c45fc49cb-gfj65 1/1 Running 0 3m 10.2.36.6 192.168.56.13
nginx-deployment-6c45fc49cb-nsw68 1/1 Running 0 3m 10.2.36.4 192.168.56.13
查看deployment service服务状态
[root@linux-node1 example]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 2h
验证容器是否可用
[root@linux-node1 ~]# curl --head 10.2.36.5
HTTP/1.1 200 OK
Server: nginx/1.13.12
Date: Fri, 07 Aug 2020 16:31:35 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 09 Apr 2018 16:01:09 GMT
Connection: keep-alive
ETag: "5acb8e45-264"
Accept-Ranges: bytes
运行一个nginx service服务¶
[root@linux-node1 ~]# cd /root/salt-kubebin/example
[root@linux-node1 example]# kubectl create -f nginx-service.yaml
[root@linux-node1 example]# kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 3h
nginx-service ClusterIP 10.1.222.235 <none> 80/TCP 13s
验证 nginx service服务是否正常
[root@linux-node2 ~]# curl --head 10.1.222.235
HTTP/1.1 200 OK
Server: nginx/1.13.12
Date: Fri, 07 Aug 2020 16:53:43 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Mon, 09 Apr 2018 16:01:09 GMT
Connection: keep-alive
ETag: "5acb8e45-264"
Accept-Ranges: bytes
查看lvs服务器状态
[root@linux-node2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.0.1:443 rr persistent 10800
-> 192.168.56.11:6443 Masq 1 0 0
TCP 10.1.222.235:80 rr
-> 10.2.36.11:80 Masq 1 0 0
-> 10.2.71.4:80 Masq 1 0 0
-> 10.2.71.5:80 Masq 1 0 1
命名空间常用命令¶
查看所有命名空间
kubectl get namespace
创建一个命名空间
kubectl create namespace test-env
删除一个命名空间
kubectl delete namespace test-env
K8s常用命令总结¶
参考：https://blog.csdn.net/Mr_daan/article/details/103663163
创建deployment
kubectl create -f nginx-deployment.yaml
查看deployment
kubectl get deployment
查看Pod
kubectl get pod -o wide
测试Pod访问
ping 10.2.36.11
curl --head 10.2.36.11
更新deployment版本
kubectl set image deployment/nginx-deployment nginx=nginx:1.12.2 --record
查看更新后的deployment
kubectl get pod -o wide
curl --head 10.2.36.13
查看更新历史
[root@linux-node1 ~]# kubectl rollout history deployment/nginx-deployment
deployments "nginx-deployment"
REVISION CHANGE-CAUSE
1 <none>
2 kubectl set image deployment/nginx-deployment nginx=nginx:1.12.2 --record=true
查看具体某一个版本的升级历史
[root@linux-node1 ~]# kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=2701970576
Containers:
nginx:
Image: nginx:1.13.12
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
快速回滚到上一个版本
[root@linux-node1 ~]# kubectl rollout undo deployment/nginx-deployment
扩容到5个节点
kubectl scale deployment nginx-deployment --replicas 5
缩容到2台节点
kubectl scale deployment nginx-deployment --replicas 2
常用命令汇总如下：https://www.cnblogs.com/klvchen/p/9585746.html
# 查看所有 pod 列表, -n 后跟 namespace, 查看指定的命名空间
kubectl get pod
kubectl get pod -n kube
kubectl get pod -o wide
# 查看 RC 和 service 列表， -o wide 查看详细信息
kubectl get rc,svc
kubectl get pod,svc -o wide
kubectl get pod <pod-name> -o yaml
# 显示 Node 的详细信息
kubectl describe node 192.168.0.212
# 显示 Pod 的详细信息, 特别是查看 pod 无法创建的时候的日志
kubectl describe pod <pod-name>
eg:
kubectl describe pod redis-master-tqds9
# 根据 yaml 创建资源, apply 可以重复执行，create 不行
kubectl create -f pod.yaml
kubectl apply -f pod.yaml
# 基于 pod.yaml 定义的名称删除 pod
kubectl delete -f pod.yaml
# 删除所有包含某个 label 的pod 和 service
kubectl delete pod,svc -l name=<label-name>
# 删除所有 Pod
kubectl delete pod --all
# 查看 endpoint 列表
kubectl get endpoints
# 执行 pod 的 date 命令
kubectl exec <pod-name> -- date
kubectl exec <pod-name> -- bash
kubectl exec <pod-name> -- ping 10.24.51.9
# 通过bash获得 pod 中某个容器的TTY，相当于登录容器
kubectl exec -it <pod-name> -c <container-name> -- bash
eg:
kubectl exec -it redis-master-cln81 -- bash
# 查看容器的日志
kubectl logs <pod-name>
kubectl logs -f <pod-name> # 实时查看日志
kubectl log <pod-name> -c <container_name> # 若 pod 只有一个容器，可以不加 -c
kubectl logs -l app=frontend # 返回所有标记为 app=frontend 的 pod 的合并日志。
# 查看注释
kubectl explain pod
kubectl explain pod.apiVersion
# 查看节点 labels
kubectl get node --show-labels
# 重启 pod
kubectl get pod <POD名称> -n <NAMESPACE名称> -o yaml | kubectl replace --force -f -
# 修改网络类型
kubectl patch service istio-ingressgateway -n istio-system -p '{"spec":{"type":"NodePort"}}'
# 伸缩 pod 副本
# 可用于将Deployment及其Pod缩小为零个副本，实际上杀死了所有副本。当您将其缩放回1/1时，将创建一个新的Pod，重新启动您的应用程序。
kubectl scale deploy/nginx-1 --replicas=0
kubectl scale deploy/nginx-1 --replicas=1
# 查看前一个 pod 的日志，logs -p 选项
kubectl logs --tail 100 -p user-klvchen-v1.0-6f67dcc46b-5b4qb > pre.log
Kubernetes必备插件¶
CoreDNS组件部署¶
DNS 服务不是独立的系统服务，而是一种 addon ，作为插件来安装的，不是 kubernetes 集群必须的（但是非常推荐安装）。可以把它看做运行在集群上的应用，只不过这个应用比较特殊而已。
DNS 有两种配置方式，在 1.3 之前使用 etcd + kube2sky + skydns 的方式，在 1.3 之后可以使用 kubedns + dnsmasq 的方式。
创建k8s CoreDNS容器
[root@linux-node1 ~]# cd salt-kubebin/addons/coredns/
[root@linux-node1 coredns]# kubectl create -f coredns.yaml
serviceaccount "coredns" created
clusterrole.rbac.authorization.k8s.io "system:coredns" created
clusterrolebinding.rbac.authorization.k8s.io "system:coredns" created
configmap "coredns" created
deployment.extensions "coredns" created
service "coredns" created
查看k8s CoreDNS容器任务
[root@linux-node1 coredns]# kubectl get deployment -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
coredns 2 2 2 2 57s
kubernetes-dashboard 1 1 1 1 34m
查看k8s CoreDNS容器服务
[root@linux-node1 coredns]# kubectl get service -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
coredns ClusterIP 10.1.0.2 <none> 53/UDP,53/TCP 1m
kubernetes-dashboard NodePort 10.1.61.59 <none> 443:30001/TCP 35m
查看CoreDNS容器vip
[root@linux-node2 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.0.1:443 rr persistent 10800
-> 192.168.56.11:6443 Masq 1 1 0
TCP 10.1.0.2:53 rr
-> 10.2.36.12:53 Masq 1 0 0
-> 10.2.71.6:53 Masq 1 0 0
TCP 10.1.222.235:80 rr
-> 10.2.36.11:80 Masq 1 0 0
-> 10.2.71.4:80 Masq 1 0 0
-> 10.2.71.5:80 Masq 1 0 0
UDP 10.1.0.2:53 rr
-> 10.2.36.12:53 Masq 1 0 0
-> 10.2.71.6:53 Masq 1 0 0
查看容器运行状态
[root@linux-node1 coredns]# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
coredns-77c989547b-gpdh2 1/1 Running 0 2m 10.2.71.6 192.168.56.12
coredns-77c989547b-mnmlt 1/1 Running 0 2m 10.2.36.12 192.168.56.13
现在我们可以临时运行一个dns容器测试一下dns解析是否好用
[root@linux-node1 coredns]# kubectl run dns-test --rm -it --image=alpine /bin/sh
现在开始测试百度 搜狐域名是否可以正确解析
If you don't see a command prompt, try pressing enter.
/ # ping www.baidu.com
PING www.baidu.com (39.156.66.14): 56 data bytes
64 bytes from 39.156.66.14: seq=1 ttl=127 time=34.372 ms
64 bytes from 39.156.66.14: seq=2 ttl=127 time=37.612 ms
64 bytes from 39.156.66.14: seq=3 ttl=127 time=30.272 ms
64 bytes from 39.156.66.14: seq=4 ttl=127 time=25.512 ms
^C
--- www.baidu.com ping statistics ---
5 packets transmitted, 4 packets received, 20% packet loss
round-trip min/avg/max = 25.512/31.942/37.612 ms
/ # ping www.sohu.com
PING www.sohu.com (221.179.177.18): 56 data bytes
64 bytes from 221.179.177.18: seq=0 ttl=127 time=306.848 ms
64 bytes from 221.179.177.18: seq=1 ttl=127 time=29.173 ms
64 bytes from 221.179.177.18: seq=2 ttl=127 time=108.006 ms
64 bytes from 221.179.177.18: seq=3 ttl=127 time=31.001 ms
64 bytes from 221.179.177.18: seq=5 ttl=127 time=33.722 ms
如果可以解析，则表示服务部署正常
现在我们可以临时运行一个centos7.6容器测试一下dns解析是否好用
[root@linux-node1 coredns]# kubectl run dns-test --rm -it --image=centos:7.6.1810 /bin/sh
sh-4.2# cat /etc/resolv.conf
nameserver 10.254.0.3
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
sh-4.2# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
sh-4.2# ping www.baidu.com
PING www.a.shifen.com (180.101.49.12) 56(84) bytes of data.
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=1 ttl=47 time=28.2 ms
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=2 ttl=47 time=28.2 ms
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=3 ttl=47 time=28.2 ms
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=4 ttl=47 time=28.3 ms
Dashboard组件部署¶
kubernetes中管理集群中资源的方式通常有四种：命令行、YAML、API和图形界面，四种不同的方式适用于不同的人群和场景，对比如下：
命令行kubectl，kubectl提供了命令行管理kubernetes资源 优点：使用方便、便捷、快速管理集群资源 缺点：功能相对有限，部分操作无法支持，有一定的门槛 YAML资源定义，kubernetes中最终转换形式，推荐使用方式 优点：功能齐备，能够定义kubernetes的所有对象和资源 缺点：门槛较高，需要具备专业技术能力，使用排障难度大 API管理接入，提供各种编程语言SDK接口，方便各种编程语言应用程序接入 优点：适配各种编程语言，如Java，Go，Python，C等，方便开发kubernetes 缺点：门槛较高，适用于开发人员 图形kubernetes-dashboard，提供图形化管理界面，能够利用metric-server实现node和pod的监控 优点：使用简单，便捷，适合大众。 缺点：功能相对简单，功能原生，适用于demo
创建k8s dashboard容器
[root@linux-node1 ~]# cd salt-kubebin/addons/dashboard/
[root@linux-node1 dashboard]# kubectl create -f .
serviceaccount "admin-user" created
clusterrolebinding.rbac.authorization.k8s.io "admin-user" created
secret "kubernetes-dashboard-certs" created
serviceaccount "kubernetes-dashboard" created
role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created
rolebinding.rbac.authorization.k8s.io "kubernetes-dashboard-minimal" created
deployment.apps "kubernetes-dashboard" created
service "kubernetes-dashboard" created
clusterrole.rbac.authorization.k8s.io "ui-admin" created
rolebinding.rbac.authorization.k8s.io "ui-admin-binding" created
clusterrole.rbac.authorization.k8s.io "ui-read" created
rolebinding.rbac.authorization.k8s.io "ui-read-binding" created
查看创建的k8s dashboard容器
[root@linux-node1 dashboard]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kubernetes-dashboard-66c9d98865-r58p2 1/1 Running 0 5m
查看命名空间里面的容器日志
[root@linux-node1 dashboard]# kubectl log pod/kubernetes-dashboard-66c9d98865-r58p2 -n kube-system
log is DEPRECATED and will be removed in a future version. Use logs instead.
2020/08/06 18:42:25 Starting overwatch
2020/08/06 18:42:25 Using in-cluster config to connect to apiserver
2020/08/06 18:42:25 Using service account token for csrf signing
2020/08/06 18:42:25 No request provided. Skipping authorization
2020/08/06 18:42:25 Successful initial request to the apiserver, version: v1.10.1
2020/08/06 18:42:25 Generating JWE encryption key
2020/08/06 18:42:25 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2020/08/06 18:42:25 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2020/08/06 18:42:25 Storing encryption key in a secret
2020/08/06 18:42:25 Creating in-cluster Heapster client
2020/08/06 18:42:25 Auto-generating certificates
2020/08/06 18:42:25 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
2020/08/06 18:42:25 Successfully created certificates
2020/08/06 18:42:25 Serving securely on HTTPS port: 8443
2020/08/06 18:42:55 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
查看创建的k8s dashboard容器服务端口信息
[root@linux-node1 dashboard]# kubectl get service -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.1.61.59 <none> 443:30001/TCP 7m
现在开始登陆proxy节点192.168.56.12的dashboard
打开浏览器输入（建议火狐浏览器打开）：https://192.168.56.12:30001/#!/login
输入令牌，令牌获取方式如下：
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
令牌内容如下
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLThwNXFxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NGQ0YjE3MS1kODE0LTExZWEtYWIxMS0wMDBjMjkyNTVlZjUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.Rna9Gw2Vn_6L9Ux0OQB3gJXa34k-jJB-LZNg0xF-DT_mzy-3s3LXQzLu7g50kddm5p7dRrCvYVJ3Vh_a0LVYLSiIBL4daXQvaIxBHSv6uHtp-be__99o_5q1VjBdmNn_UcbLh5yH-HSM22mWQmffm-zVfAsPEDdDUDUqZ2g8OlY-YLcw9FACbj5c7EtUz6owiSY6wR0R7Y7pLPUQM2loQyP7H4OOzlEWNfPCxQJ7emg64SLGtWUeBUhA8YNG7EeduvdWGjVw2HAyoFKLo6Xs8zmsgO7hMjZqYJiL4r-hylZR2387SRn8PBrtAFNoU4rTWswzxIYiY5tC0AbcoVk-EQ
输入令牌后登陆web界面
heapster组件部署¶
Heapster是容器集群监控和性能分析工具，天然的支持Kubernetes和CoreOS。
Kubernetes有个出名的监控agent—cAdvisor。在每个kubernetes Node上都会运行cAdvisor，它会收集本机以及容器的监控数据(cpu,memory,filesystem,network,uptime)。
在较新的版本中，K8S已经将cAdvisor功能集成到kubelet组件中。每个Node节点可以直接进行web访问。
Heapster是一个收集者，将每个Node上的cAdvisor的数据进行汇总，然后导到第三方工具(如InfluxDB)。
框架图：
Heapster首先从K8S Master获取集群中所有Node的信息，然后通过这些Node上的kubelet获取有用数据，而kubelet本身的数据则是从cAdvisor得到。所有获取到的数据都被推到Heapster配置的后端存储中，并还支持数据的可视化。现在后端存储 + 可视化的方法，如InfluxDB + grafana。
创建k8s heapster service
[root@linux-node1 ~]# kubectl create -f /root/salt-kubebin/addons/heapster/
deployment.apps "monitoring-grafana" created
service "monitoring-grafana" created
serviceaccount "heapster" created
clusterrolebinding.rbac.authorization.k8s.io "heapster" created
deployment.apps "heapster" created
service "heapster" created
deployment.apps "monitoring-influxdb" created
service "monitoring-influxdb" created
查看创建的k8s heapster service
[root@linux-node1 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-77c989547b-8x6t8 1/1 Running 0 1d
coredns-77c989547b-r2hhf 1/1 Running 0 1d
heapster-64f4f9f59d-62qxz 0/1 ContainerCreating 0 19s
kubernetes-dashboard-66c9d98865-hd4m7 1/1 Running 0 1d
monitoring-grafana-844d4fdf8c-zbfxx 0/1 ContainerCreating 0 21s
monitoring-influxdb-644db5c5b6-wch7x 0/1 ContainerCreating 0 20s
打开浏览器输入（建议火狐浏览器打开）：https://192.168.56.12:30001/#!/login
登录Dashboard即可查看到对应的监控图表。
Kubectl top¶
安装 node top 工具
1、kubectl edit deployments.apps -n kube-system metrics-server
修改 image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.5.2
2、kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml