本地集群需求
-
轻量; (下载的包小)
-
启动快; (最好是docker 方式启动, 而非VM)
-
占用资源少; (最好是docker 方式启动, 而非VM. 且需要是一个拥有Kubernetes完整功能的, 精简的发行版)
用过minikube, VM启动比较慢, 而且下载最新版的时候, 阿里云的mirror都没有最新版本的镜像, 导致一直启动不起来. 非常难受.
基于K3S的K3D完美符合我的以上需求.
K3S简介 – 轻量级 Kubernetes
轻量级 Kubernetes。安装简单,内存只有一半,所有的二进制都不到 200MB。包含K3S的完整镜像大小如下:
REPOSITORY TAG IMAGE ID CREATED SIZE rancher/k3s v1.18.2-k3s1 e9f6bccce7de 6 months ago 151MB
我这边安装完成后, (又安装了traefik和Kubernetes dashboard和一个demo deployment), 消耗如下:
-
CPU: 0.3 Core
-
内存: 1.2 G
适用于:
-
边缘计算-Edge
-
物联网-IoT
-
CI
-
Development
-
ARM
-
嵌入 K8s
-
不想深陷 k8s 运维管理的人
K3s 是一个完全符合 Kubernetes 的发行版,有以下增强功能。
-
打包为
单个二进制
文件。 -
基于
sqlite3
的轻量级存储后端作为默认存储机制。etcd3,MySQL,Postgres 仍然可用。 -
封装在简单的启动程序中,该启动程序处理很多复杂的 TLS 和选项。
-
默认情况下是安全的,对轻量级环境有合理的默认值。
-
添加了简单但功能强大的“batteries-included”功能,例如:本地存储提供程序,服务负载均衡器,
Helm
controller 和
Traefik
ingress controller。 -
所有 Kubernetes 控制平面组件的操作都封装在单个二进制文件和进程中。这使 K3s 可以自动化和管理复杂的集群操作,例如分发证书。
-
外部依赖性已最小化(仅需要现代内核和 cgroup 挂载)。K3s 软件包需要依赖项,包括:
K3D – K3S in docker
k3d创建容器化的k3s集群。这意味着,您可以使用docker在单台计算机上启动多节点k3s集群。
K3D 快速入门
使用 k3d 搭建 k3s 集群. k3d是快速搭建容器化 k3s 集群的工具。可以使用 Docker 在单台计算机上启动多节点 k3s 集群。
???? 备注:
我的计算机环境:
win10专业版 2004
WSL2 + Ubuntu20.04 + docker desktop
-
运行以下指令,启动具有 3 个 worker 节点的本地 k3s 集群。(搭建集群搭吐了, 有现成官方脚本直接用. 亲测国内好用) 使用
root
执行:
curl -fL https://octopus-assets.oss-cn-beijing.aliyuncs.com/k3d/cluster-k3s-spinup.sh | bash -
⚠️ 注意:
如果安装成功,则应该看到以下日志:
please input CTRL+C to stop the local cluster
如果想要停止K3S集群, 请运行
CTRL+C
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 13549 100 13549 0 0 6784 0 0:00:01 0:00:01 --:--:-- 6781[INFO] [1107 17:02:03] cleanup proxy config[INFO] [1107 17:02:03] creating edge cluster with v1.18.2[INFO] [1107 17:02:03] INGRESS_HTTP_PORT is 54836[INFO] [1107 17:02:03] INGRESS_HTTPS_PORT is 54837INFO[0000] Created cluster network with ID ba03de48d65b8e1fbef6ff03cbba0b9e9ad008e7cc81d67d8393c69272a1c4b9INFO[0000] Add TLS SAN for 0.0.0.0INFO[0000] Created docker volume k3d-edge-imagesINFO[0000] Creating cluster [edge]INFO[0000] Creating server using docker.io/rancher/k3s:v1.18.2-k3s1...INFO[0006] SUCCESS: created cluster [edge]INFO[0006] You can now use the cluster with:export KUBECONFIG="$(k3d get-kubeconfig --name='edge')"kubectl cluster-info[WARN] [1107 17:02:09] default kubeconfig has been backup in /root/.kube/config_k3d_bak[INFO] [1107 17:02:09] edge cluster's kubeconfig wrote in /root/.kube/config now[INFO] [1107 17:02:09] waiting node edge-control-plane for readyINFO[0000] Adding 1 agent-nodes to k3d cluster edge...INFO[0000] Created agent-node with ID 752aebb8f9bb1af1c5fcf62ff9313163c243835373872595f38de03004257514[INFO] [1107 17:02:21] waiting node edge-worker for readyINFO[0000] Adding 1 agent-nodes to k3d cluster edge...INFO[0000] Created agent-node with ID 7d0aa70e24f387217d3094911a7c0f5fa2f504c1fe3e106b08d00f3a6b11158c[INFO] [1107 17:02:34] waiting node edge-worker1 for readyINFO[0000] Adding 1 agent-nodes to k3d cluster edge...INFO[0000] Created agent-node with ID 7b880c8966f9b8b252c5385ee10167384d9517c87ff60763989b69f5c3f344ab[INFO] [1107 17:02:47] waiting node edge-worker2 for ready[WARN] [1107 17:02:59] please input CTRL+C to stop the local cluster
-
打开一个新终端,并配置
KUBECONFIG
以访问本地 k3s 集群。
export KUBECONFIG="$(k3d get-kubeconfig --name='edge')"kubectl cluster-info
输出结果如下:
Kubernetes master is running at https://0.0.0.0:54835CoreDNS is running at https://0.0.0.0:54835/api/v1/namespaces/kube-system/services/kube-dns:dns/proxyMetrics-server is running at https://0.0.0.0:54835/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
-
运行
kubectl get node
命令, 检查本地 k3s 集群的节点是否正常:
# kubectl get nodeNAME STATUS ROLES AGE VERSIONedge-worker Ready <none> 3h17m v1.18.2+k3s1edge-worker2 Ready <none> 3h17m v1.18.2+k3s1edge-control-plane Ready master 3h17m v1.18.2+k3s1edge-worker1 Ready <none> 3h17m v1.18.2+k3s1
-
运行
kubectl get pod -A
命令, 检查本地 k3s 集群的pod是否正常: (默认就已经部署好了traefik)
kubectl get pod -ANAMESPACE NAME READY STATUS RESTARTS AGEkube-system metrics-server-7566d596c8-6h776 1/1 Running 0 3h18mkube-system local-path-provisioner-6d59f47c7-sz5tp 1/1 Running 0 3h18mkube-system coredns-8655855d6-lmrkq 1/1 Running 0 3h18mkube-system svclb-traefik-wxp6k 2/2 Running 0 133mkube-system svclb-traefik-jls5w 2/2 Running 0 133mkube-system svclb-traefik-j776k 2/2 Running 0 133mkube-system svclb-traefik-qbfx4 2/2 Running 0 133mkube-system helm-install-traefik-jxptl 0/1 Completed 0 120mkube-system traefik-6cbfb44969-r9fj2 1/1 Running 0 118m
???? 笔记:
K3D的快速启动脚本, 涉及到以下docker镜像: (只有第一个镜像是在外边pull的, 其他镜像其实都是在启动后的k3s 容器里pull的.)
# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZErancher/k3s v1.18.2-k3s1 e9f6bccce7de 6 months ago 151MBrancher/klipper-helm v0.2.5 6207e2a3f522 6 months ago 136MBrancher/library-traefik 1.7.19-amd64 aa764f7db305 12 months ago 85.7MBrancher/metrics-server v0.3.6 9dd718864ce6 13 months ago 39.9MBrancher/local-path-provisioner v0.0.11 9d12f9848b99 13 months ago 36.2MBrancher/coredns-coredns 1.6.3 c4d3d16fe508 14 months ago 44.3MB
K3D的快速启动脚本, 会启动4个docker容器作为4个node节点:
sudo docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES7b880c8966f9 rancher/k3s:v1.18.2-k3s1 "/bin/k3s agent --no…" 3 hours ago Up 3 hours k3d-edge-worker-37d0aa70e24f3 rancher/k3s:v1.18.2-k3s1 "/bin/k3s agent --no…" 3 hours ago Up 3 hours k3d-edge-worker-2752aebb8f9bb rancher/k3s:v1.18.2-k3s1 "/bin/k3s agent --no…" 3 hours ago Up 3 hours k3d-edge-worker-1dca9851cf5d6 rancher/k3s:v1.18.2-k3s1 "/bin/k3s server --h…" 3 hours ago Up 3 hours 0.0.0.0:54835->54835/tcp, 0.0.0.0:54836->80/tcp, 0.0.0.0:54837->443/tcp k3d-edge-server
从上文可以看到, 1个k3s server(就是控制平面), 3个k3s agent. k3s server对外暴露了3个
随机
端口:
-
54835->54835
: K8S API -
54836->80
: K8S Ingress的HTTP端口. -
54837->443
: K8S Ingress的HTTPS端口.
部署Traefik Dashboard
所以我们要访问部署在容器中的应用, 就用这2个随机端口: http://localhost:54836 或https://localhost:54836 . 部署好了后, 默认是没有任何的Ingress的, 所以访问这2个地址都是报:
404
而且默认脚本是没有启用Traefik的Dashboard的, 管理不便. 我们将它启用起来.
首先是进入到k3s server容器里. 这个容器没有
/bin/bash
, 只有
/bin/sh
, 如下:
# docker exec -it <k3s server container id> ls /binaddgroup cat containerd-shim df expr fstrim i2cdetect ipcs kubectl lsof mkswap openvt ptx runcon sha512sum swapoff tr unxz whoamiadduser charon containerd-shim-runc-v2 diff factor fuser i2cdump iplink last lspci mktemp partprobe pwd runlevel shred swapon traceroute unzip xargsar chattr coreutils dir fallocate getopt i2cget ipneigh less lsscsi modprobe passwd rdate sed shuf switch_root true uptime xtables-legacy-multiarch chcon cp dircolors false getty i2cset iproute link lsusb more paste readlink seq sleep sync truncate users xxdarp check-config cpio dirname fbset ginstall id iprule linux32 lzcat mountpoint patch readprofile setarch slirp4netns sysctl tsort usleep xzarping chgrp crictl dmesg fdflush grep ifconfig ipset linux64 lzma mt pathchk realpath setconsole socat syslogd tty uudecode xzcatash chmod crond dnsd fdformat groups ifdown iptables linuxrc lzopcat mv pidof reboot setfattr sort tac ubirename uuencode yesaux chown crontab dnsdomainname fdisk gunzip ifup iptables-restore ln makedevs nameif pigz renice setkeycodes split tail udhcpc vconfig zcatawk chroot csplit dos2unix fgrep gzip inetd iptables-save loadfont md5sum netstat ping reset setlogcons start-stop-daemon tar uevent vdirb2sum chrt ctr du find halt init iptunnel loadkmap mdev nice pinky resize setpriv stat tc umount vibase32 chvt cut dumpkmap flannel hdparm insmod join logger mesg nl pipe_progress resume setserial strings tee uname vlockbase64 cksum date ebtables flock head install k3s login microcom nohup pivot_root rm setsid stty telnet unexpand wbasename clear dc echo fmt hexdump ip k3s-agent logname mkdir nproc portmap rmdir sh su test uniq watchblkid cmp dd egrep fold hexedit ip6tables k3s-server loopback mkdosfs nsenter poweroff rmmod sha1sum sulogin tftp unix2dos watchdogbridge cni deallocvt eject free host-local ip6tables-restore kill losetup mke2fs nslookup pr route sha224sum sum time unlink wcbunzip2 comm delgroup env freeramdisk hostid ip6tables-save killall ls mkfifo nuke printenv run-init sha256sum svc timeout unlzma wgetbusybox conntrack deluser ether-wake fsck hostname ipaddr killall5 lsattr mknod numfmt printf run-parts sha384sum svok top unlzop whichbzcat containerd devmem expand fsfreeze hwclock ipcrm klogd lsmod mkpasswd od ps runc sha3sum swanctl touch unpigz who
所以通过
/bin/sh
进入到容器里:
# docker exec -it <k3s server container id> /bin/sh---------------已经进入容器里--------------/ # cd /var/lib/rancher/k3s/server/manifests/var/lib/rancher/k3s/server/manifests # vi traefik.yaml
编辑后的
traefik.yaml
如下: (增加:
dashboard.enabled: "true"
)
apiVersion: helm.cattle.io/v1kind: HelmChartmetadata: name: traefik namespace: kube-systemspec: chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz valuesContent: |- rbac: enabled: true ssl: enabled: true metrics: prometheus: enabled: true kubernetes: ingressEndpoint: useDefaultPublishedService: true dashboard: enabled: true image: "rancher/library-traefik" tolerations: - key: "CriticalAddonsOnly" operator: "Exists" - key: "node-role.kubernetes.io/master" operator: "Exists" effect: "NoSchedule"
保存后就会重新部署
traefik.yaml
, 如下:
# kubectl get events -n kube-systemLAST SEEN TYPE REASON OBJECT MESSAGE43s Normal Pulled pod/helm-install-traefik-jxptl Successfully pulled image "rancher/klipper-helm:v0.2.5"43s Normal Created pod/helm-install-traefik-jxptl Created container helm43s Normal Started pod/helm-install-traefik-jxptl Started container helm43s Normal ScalingReplicaSet deployment/traefik Scaled up replica set traefik-6cbfb44969 to 143s Normal SuccessfulCreate replicaset/traefik-6cbfb44969 Created pod: traefik-6cbfb44969-r9fj2<unknown> Normal Scheduled pod/traefik-6cbfb44969-r9fj2 Successfully assigned kube-system/traefik-6cbfb44969-r9fj2 to edge-worker242s Normal Pulling pod/traefik-6cbfb44969-r9fj2 Pulling image "rancher/library-traefik:1.7.19"42s Normal Completed job/helm-install-traefik Job completed41s Normal SandboxChanged pod/helm-install-traefik-jxptl Pod sandbox changed, it will be killed and re-created.9s Normal Pulled pod/traefik-6cbfb44969-r9fj2 Successfully pulled image "rancher/library-traefik:1.7.19"9s Normal Created pod/traefik-6cbfb44969-r9fj2 Created container traefik9s Normal Started pod/traefik-6cbfb44969-r9fj2 Started container traefik
部署后, 会自动配置ingress, 如下:
# kubectl get ingress -ANAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGEkube-system traefik-dashboard <none> traefik.example.com 172.18.0.2 80 149m
所以我们配置hosts:
127.0.0.1 traefik.example.com
. 就可以访问: http://traefik.example.com:54836/dashboard/, 如下图所示:
???? 备注:
其实还有另一种方法可以进行访问:
kubectl port-forward
. 如下:$ kubectl port-forward $(kubectl get pods --selector "app=traefik" --output=name -n kube-system) --address 0.0.0.0 8080:8080 -n kube-system
则可以通过http://localhost:8080/dashboard/ 访问到traefik的管理页面.
部署应用
使用
whoami
应用程序部署测试.
$ kubectl create deploy whoami --image containous/whoamideployment.apps/whoami created$ kubectl expose deploy whoami --port 80service/whoami exposed
然后我们定义一个 Ingress 规则来使用我们新的 Traefik,Traefik 既能读取自己的 CRD IngressRoute,也能读取传统的 Ingress 资源。
vi whoami-ingress.yaml
具体内容如下:
apiVersion: networking.k8s.io/v1beta1kind: Ingressmetadata: name: whoami annotations: traefik.ingress.kubernetes.io/router.entrypoints: web,websecure traefik.ingress.kubernetes.io/router.tls: "true"spec: rules: - http: paths: - path: / backend: serviceName: whoami servicePort: 80
kubectl apply
应用:
kubectl apply -f whoami-ingress.yaml -n default
在这个例子中,我们在 HTTP 和 HTTPs 两个入口点上暴露了 whoami 服务,每一个 URL 都会被发送到该服务上,我们可以在 Traefik Dashboard 上看到新的Ingress。
要测试这个应用我们可以直接在浏览器中访问:http://localhost:54836/ 即可,这是因为上面我们安装 Traefik 的时候自动创建了一个 LoadBalancer 的 Service 服务。为啥要加端口号, 因为k3s server在容器里, 映射到外边是
54386
端口.
部署Kubernetes 仪表盘
GITHUB_URL=https://github.com/kubernetes/dashboard/releasesVERSION_KUBE_DASHBOARD=$(curl -w '%{url_effective}' -I -L -s -S ${GITHUB_URL}/latest -o /dev/null | sed -e 's|.*/||')kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/${VERSION_KUBE_DASHBOARD}/aio/deploy/recommended.yaml
输出如下:
namespace/kubernetes-dashboard createdserviceaccount/kubernetes-dashboard createdservice/kubernetes-dashboard createdsecret/kubernetes-dashboard-certs createdsecret/kubernetes-dashboard-csrf createdsecret/kubernetes-dashboard-key-holder createdconfigmap/kubernetes-dashboard-settings createdrole.rbac.authorization.k8s.io/kubernetes-dashboard createdclusterrole.rbac.authorization.k8s.io/kubernetes-dashboard createdrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard createdclusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard createddeployment.apps/kubernetes-dashboard createdservice/dashboard-metrics-scraper createddeployment.apps/dashboard-metrics-scraper created
验证pod已正常启动:
# kubectl get pod -n kubernetes-dashboardNAME READY STATUS RESTARTS AGEdashboard-metrics-scraper-6b4884c9d5-ltk42 1/1 Running 0 14mkubernetes-dashboard-7d8574ffd9-sptn6 1/1 Running 0 98s
仪表盘 RBAC 配置
⚠️ 重要:
本指南中创建的
admin-user
将在仪表板中拥有管理权限。
创建以下资源清单文件:
vi dashboard.admin-user.ymlapiVersion: v1kind: ServiceAccountmetadata: name: admin-user namespace: kubernetes-dashboard
vi dashboard.admin-user-role.ymlapiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: name: admin-userroleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-adminsubjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard
部署
admin-user
配置:
kubectl create -f dashboard.admin-user.yml -f dashboard.admin-user-role.yml
获得 Bearer Token
kubectl -n kubernetes-dashboard describe secret admin-user-token | grep ^token
结果如下:
token: eyJhbGciOiJSUzI1NiIsImtpZCI6Im9XNENjc0VlSzVBTDJGRWpPT2VuY1pkbzNJblYybFFwY2YxQnBvZVlMVlEifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXA1Y253Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI0NmI4NGFkYS02MDQ3LTQzN2EtODk2My1lY2NmZWQ4MjE0ZDQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZXJuZXRlcy1kYXNoYm9hcmQ6YWRtaW4tdXNlciJ9.N8Zhsf2JU5Hoa8yfhrspJbMGP7AFmfs2JeWXVpDksAEMfWf5mI-MXYcqMkbZ9_Qbwp-h9S7k7oZE41lUp8UXlDWi0Ovm4I4fsuoWqq-aJoyt-c060bWNla1edVZ5BzMTanIYzJHPjS7-cOnsxqg-EtXfdN3JRsiE0QevLvJLhYU37HFc7-cImJ8iH8-r-GHCD8MmuBbTV0EBidLmSo-BdWC5hcZoYghgNtfnMkN0p1e3O23EPRO2XDmaw_lVN4TNgZXPS9hirBD1AZxm1ZE1Iyo2mSOgYjCNQOF8IcaUtjTGqt4RzK4R9AWRbL9z-HMbK_JamcQvDz3fnW3aauCezQ
本地访问仪表盘:
kubectl proxy
现在可以通过以下网址访问仪表盘:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ 使用admin-user Bearer Token Sign In
更多仪表盘访问方式
port-forward
方式
port-forward
$ kubectl port-forward $(kubectl get pods --selector "k8s-app=kubernetes-dashboard" --output=name -n kubernetes-dashboard) --address 0.0.0.0 8443:8443 -n kubernetes-dashboard
Helm 部署应用
# helm repo add stable http://mirror.azure.cn/kubernetes/charts# helm repo update# helm install jenkins stable/jenkinsWARNING: This chart is deprecatedNAME: jenkinsLAST DEPLOYED: Sat Nov 7 22:25:02 2020NAMESPACE: defaultSTATUS: deployedREVISION: 1NOTES:***********************DEPRECATED************************* The Jenkins chart is deprecated. Future development has been moved to https://github.com/jenkinsci/helm-charts1. Get your 'admin' user password by running: printf $(kubectl get secret --namespace default jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo2. Get the Jenkins URL to visit by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=jenkins" -o jsonpath="{.items[0].metadata.name}") echo http://127.0.0.1:8080 kubectl --namespace default port-forward $POD_NAME 8080:80803. Login with the password from step 1 and the username: admin4. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demosFor more information on running Jenkins on Kubernetes, visit:https://cloud.google.com/solutions/jenkins-on-container-engineFor more information about Jenkins Configuration as Code, visit:https://jenkins.io/projects/jcasc/
总结
通过K3S/K3D, 有以下优势:
-
创建, 部署, 启动集群快;
-
集群消耗资源少;
-
创建的集群拥有完善的基础功能;
-
可以提供和标准K8S集群一致的: