摘要:部署节点节点包含的组件本文档介绍部署一个三节点高可用集群的步骤,分别命名为安装在每台主机安装。出现如下错误解决办法启动参数增加参数增加在物理机上可能并不会出现该问题。至此一个完整的高可用集群搭建完毕。
部署master节点
kubernetes master 节点包含的组件:
kube-apiserver
kube-scheduler
kube-controller-manager
本文档介绍部署一个三节点高可用 master 集群的步骤,分别命名为k8s-host1、k8s-host2、k8s-host3:
k8s-host1:172.16.120.154
k8s-host2:172.16.120.155
k8s-host3:172.16.120.156
安装docker在每台主机安装docker。建议使用docker v1.12 , 但是 v1.11, v1.13 and 17.03 也可以正常工作. 版本 17.06+ 可能工作, 但是没有被k8s团队测试过。
PS:kubelet设置的cgroup driver应该和Docker设置的cgroup driver一致。这边统一为systemd。
yum install -y docker systemctl enable docker && systemctl start docker安装kubeadm, kubelet 和 kubectl
cat </etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF setenforce 0 yum install -y kubelet kubeadm kubectl systemctl enable kubelet && systemctl start kubelet
通过运行命令setenforce 0 禁用selinux,主要为了允许容器可以访问主机文件系统和pod networks的需要。
Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed。所以需要执行下面命令:
cat <安装master kubeadm init/etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system
kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16
apiserver-advertise-address该参数一般指定为haproxy+keepalived 的vip。
pod-network-cidr 主要是在搭建pod network(calico)时候需要在init时候指定。
出现如下错误:
[root@k8s-node04 ~]# kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 [init] Using Kubernetes version: v1.9.0 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING FileExisting-crictl]: crictl not found in system path [preflight] Some fatal errors occurred: [ERROR Swap]: running with swap on is not supported. Please disable swap [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
解决办法:
1:kubelet启动参数增加--fail-swap-on=false
[Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true" Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin" Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local" Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt" Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0" Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd" Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki" Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false" ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS
2:kubeadm init 参数增加ignore-preflight-errors
kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors "Swap"
3:在物理机上可能并不会出现该问题。
最终安装成功
[root@k8s-host1 manifests]# kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors "Swap" [init] Using Kubernetes version: v1.9.0 [init] Using Authorization modes: [Node RBAC] [preflight] Running pre-flight checks. [WARNING Swap]: running with swap on is not supported. Please disable swap [WARNING FileExisting-crictl]: crictl not found in system path [preflight] Starting the kubelet service [certificates] Generated ca certificate and key. [certificates] Generated apiserver certificate and key. [certificates] apiserver serving cert is signed for DNS names [k8s-host1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.120.200] [certificates] Generated apiserver-kubelet-client certificate and key. [certificates] Generated sa key and public key. [certificates] Generated front-proxy-ca certificate and key. [certificates] Generated front-proxy-client certificate and key. [certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki" [kubeconfig] Wrote KubeConfig file to disk: "admin.conf" [kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf" [kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf" [kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf" [controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml" [controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml" [controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml" [etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml" [init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests". [init] This might take a minute or longer if the control plane images have to be pulled. [apiclient] All control plane components are healthy after 30.002346 seconds [uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [markmaster] Will mark node k8s-host1 as master by adding a label and a taint [markmaster] Master k8s-host1 tainted and labelled with key/value: node-role.kubernetes.io/master="" [bootstraptoken] Using token: 26ea24.cb987cb5cf6f29a6 [bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: kube-dns [addons] Applied essential addon: kube-proxy Your Kubernetes master has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of machines by running the following on each node as root: kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab
PS 需要记下 kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab,便于以后node加入到k8s集群中。
将kube-apiserver指向上一节部署的etcd集群首先需要将上节生成的etcd ca等证书cp到mater主机的/etc/etcd/ssl/下
默认kubeadm安装了一个etcd节点,非集群,非加密,所以我们必须对kube-apiserver.yaml做修改。
修改后的kube-apiserver.yaml:
apiVersion: v1 kind: Pod metadata: annotations: scheduler.alpha.kubernetes.io/critical-pod: "" labels: component: kube-apiserver tier: control-plane name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --allow-privileged=true - --requestheader-username-headers=X-Remote-User - --requestheader-group-headers=X-Remote-Group - --requestheader-extra-headers-prefix=X-Remote-Extra- - --service-cluster-ip-range=10.96.0.0/12 - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key - --enable-bootstrap-token-auth=true - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt - --insecure-port=0 - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota - --requestheader-allowed-names=front-proxy-client - --advertise-address=172.16.120.200 - --client-ca-file=/etc/kubernetes/pki/ca.crt - --secure-port=6443 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key - --authorization-mode=Node,RBAC - --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379 - --etcd-cafile=/etc/etcd/ssl/ca.pem - --etcd-certfile=/etc/etcd/ssl/etcd.pem - --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0 livenessProbe: failureThreshold: 8 httpGet: host: 172.16.120.200 path: /healthz port: 6443 scheme: HTTPS initialDelaySeconds: 15 timeoutSeconds: 15 name: kube-apiserver resources: requests: cpu: 250m volumeMounts: - mountPath: /etc/kubernetes/pki name: k8s-certs readOnly: true - mountPath: /etc/ssl/certs name: ca-certs readOnly: true - mountPath: /etc/pki name: ca-certs-etc-pki readOnly: true - mountPath: /etc/etcd/ssl name: ca-certs-etc-etcd-ssl readOnly: true hostNetwork: true volumes: - hostPath: path: /etc/kubernetes/pki type: DirectoryOrCreate name: k8s-certs - hostPath: path: /etc/ssl/certs type: DirectoryOrCreate name: ca-certs - hostPath: path: /etc/pki type: DirectoryOrCreate name: ca-certs-etc-pki - hostPath: path: /etc/etcd/ssl type: DirectoryOrCreate name: ca-certs-etc-etcd-ssl status: {}
注意添加了以下四项:
--etcd-cafile=/etc/etcd/ssl/ca.pem --etcd-certfile=/etc/etcd/ssl/etcd.pem --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379 # 以及挂载 - mountPath: /etc/etcd/ssl name: ca-certs-etc-etcd-ssl readOnly: true - hostPath: path: /etc/etcd/ssl type: DirectoryOrCreate name: ca-certs-etc-etcd-ssl确认第一台master三大组件都成功启动
kubectl get componentstatuses NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy {"health": "true"} etcd-0 Healthy {"health": "true"} etcd-2 Healthy {"health": "true"}安装其他主机
我一般的做法是,依旧使用kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors "Swap"
主要目的是下载k8s相关组件docker镜像等,其实就是让kubeadm把容易忽略的工作都做了。
然后将第一台/etc/kubernetes/下所有文件拷贝到其他master主机对应目录下,直至其他机器master三大组件启动成功。
至此一个完整的master高可用集群搭建完毕。
文章版权归作者所有,未经允许请勿转载,若此文章存在违规行为,您可以联系管理员删除。
转载请注明本文地址:https://www.ucloud.cn/yun/33038.html
摘要:部署节点节点包含的组件本文档介绍部署一个三节点高可用集群的步骤,分别命名为安装在每台主机安装。出现如下错误解决办法启动参数增加参数增加在物理机上可能并不会出现该问题。至此一个完整的高可用集群搭建完毕。 部署master节点 kubernetes master 节点包含的组件: kube-apiserver kube-scheduler kube-controller-manager ...
摘要:部署节点节点包含的组件本文档介绍部署一个三节点高可用集群的步骤,分别命名为安装在每台主机安装。出现如下错误解决办法启动参数增加参数增加在物理机上可能并不会出现该问题。至此一个完整的高可用集群搭建完毕。 部署master节点 kubernetes master 节点包含的组件: kube-apiserver kube-scheduler kube-controller-manager ...
摘要:集群概述整个集群包括大部分集群节点节点集群主要作为集群和网络的数据存储。集群组件版本集群机器主从从后续计划用替换。 前言 k8s部署的方式多种多样,除去各家云厂商提供的工具,在bare metal中,也有二进制部署和一系列的自动化部署工具(kubeadm,kubespary,rke等)。具体二进制部署大家可以参考宋总的系列文章。而rke是由rancher提供的工具,由于刚刚出来,有不少...
摘要:集群概述整个集群包括大部分集群节点节点集群主要作为集群和网络的数据存储。集群组件版本集群机器主从从后续计划用替换。 前言 k8s部署的方式多种多样,除去各家云厂商提供的工具,在bare metal中,也有二进制部署和一系列的自动化部署工具(kubeadm,kubespary,rke等)。具体二进制部署大家可以参考宋总的系列文章。而rke是由rancher提供的工具,由于刚刚出来,有不少...
阅读 2783·2023-04-25 18:06
阅读 2576·2021-11-22 09:34
阅读 1684·2021-11-08 13:16
阅读 1301·2021-09-24 09:47
阅读 3049·2019-08-30 15:44
阅读 2773·2019-08-29 17:24
阅读 2584·2019-08-23 18:37
阅读 2433·2019-08-23 16:55