博客

k8s-存储插件：cfs运行机制

  • Post author:
  • Post category:其他

部署方式：

csi-provisioner-cfsplugin

[kube-system/statefulset]

1.csi-provisioner

社区 sidecar，负责监听 pvc 资源，并通知 cfs 进行文件系统的创建与删除。
2.csi-cfs

自研 csi 插件，负责接收 csi-provisioner 请求，并调用对应腾讯云文件系统服务中文件系统的创建删除接口。
3.tcfs

自研 operator 插件，针对共享类型的sc。

csi-nodeplugin-cfsplugin

[kube-system/daemonset]

1.csi-node-driver-registrar

社区 sidecar，负责向 kubelet 注册对应 csi 插件。
2.csi-cfs

自研 csi 插件，负责接收 kubelet 请求，进行文件系统的 mount 与 umount（共享存储也由该插件进行文件系统的 mount 与 umount）。

csi-attacher-cfsplugin

[kube-system/statefulset]

集群版本小于1.14才安装

1.csi-attacher

通过GetControllerCapabilities接口，检测到CSI不支持attach.dettach，

启动trivialHandler。

trivialHandler，会自动将VA更新为attached.

2.csi-cfs

GetControllerCapabilities接口

1.14以上版本，CSIDriver设置了不需要attach。

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  annotations:
    meta.helm.sh/release-name: cfs
    meta.helm.sh/release-namespace: kube-system
  name: com.tencent.cloud.csi.cfs
spec:
  attachRequired: false
  fsGroupPolicy: File
  podInfoOnMount: false
  requiresRepublish: false
  storageCapacity: false
  volumeLifecycleModes:
  - Persistent

TCFS

tcfs的CRD资源

 

 

1.非共享类型的CFS

1.1 创建SC和PVC

SC


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cfs-normal
parameters:
  pgroupid: pgroup-lrquouvl // 权限组(针对网络访问)
  storagetype: SD // CFS实例的类型：标准/性能
  subnetid: subnet-rtwb42lu // 子网
  vers: "3" // NFS协议版本
  vpcid: vpc-oilua6pt // VPC网络
  zone: ap-guangzhou-3 // 可用区
provisioner: com.tencent.cloud.csi.cfs
reclaimPolicy: Delete
volumeBindingMode: Immediate

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
    volume.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.cfs
  finalizers:
  - kubernetes.io/pvc-protection
  name: cfs-normal
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: cfs-normal
  volumeMode: Filesystem
  volumeName: pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  phase: Bound


1.2 自动创建PV


1）csi-provisioner-cfsplugin/csi-provisioner


调用createvolume接口，创建CFS。


日志：


I0221 09:46:37.456573       1 controller.go:1317] provision "default/cfs-normal" class "cfs-normal": started
I0221 09:46:37.457251       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-normal", UID:"a11e120e-130c-4aff-9901-17b7b91b3f1d", APIVersion:"v1", ResourceVersion:"424215490", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/cfs-normal"
I0221 09:46:44.915159       1 controller.go:655] create volume rep: {CapacityBytes:10737418240 VolumeId:cfs-3esjn2lz VolumeContext:map[fsid:bz8eisu7 host:10.0.33.133 pgroupid:pgroup-lrquouvl storagetype:SD subnetid:subnet-rtwb42lu vers:3 vpcid:vpc-oilua6pt zone:ap-guangzhou-3] ContentSource:<nil> AccessibleTopology:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0221 09:46:44.915242       1 controller.go:737] successfully created PV pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d for PVC cfs-normal and csi volume name cfs-3esjn2lz
I0221 09:46:44.915257       1 controller.go:1420] provision "default/cfs-normal" class "cfs-normal": volume "pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d" provisioned
I0221 09:46:44.915292       1 controller.go:1437] provision "default/cfs-normal" class "cfs-normal": succeeded
E0221 09:46:44.932235       1 controller.go:1443] couldn't create key for object pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d: object has no meta: object does not implement the Object interfaces
I0221 09:46:44.932419       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-normal", UID:"a11e120e-130c-4aff-9901-17b7b91b3f1d", APIVersion:"v1", ResourceVersion:"424215490", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d


2）csi-provisioner-cfsplugin/csi-tencentcloud-cfs


调用CFS接口，


I0221 09:46:37.457486       1 controller.go:42] CreateVolume CreateVolumeRequest is name:"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d" capacity_range:<required_bytes:10737418240 > volume_capabilities:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > parameters:<key:"pgroupid" value:"pgroup-lrquouvl" > parameters:<key:"storagetype" value:"SD" > parameters:<key:"subnetid" value:"subnet-rtwb42lu" > parameters:<key:"vers" value:"3" > parameters:<key:"vpcid" value:"vpc-oilua6pt" > parameters:<key:"zone" value:"ap-guangzhou-3" > :
I0221 09:46:37.457599       1 controller.go:58] req.name is :  pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d   
I0221 09:46:37.457616       1 secret_util.go:20] Get secretID or secretKey from env failed, will use cloud norm!


1.3 创建POD绑定PV


1）kubelet


// attach，不支持，假设已经attached
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.484139    4762 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.484214    4762 reconciler.go:342] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb6v\" (UniqueName: \"kubernetes.io/projected/4ce98f8e-1ff8-4d78-b61c-739997ff026e-kube-api-access-cmb6v\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
​
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.585483    4762 reconciler.go:254] "operationExecutor.MountVolume started for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
// 全局mount，不支持跳过
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.601243    4762 csi_attacher.go:358] kubernetes.io/csi: attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.601309    4762 operation_generator.go:658] "MountVolume.MountDevice succeeded for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/com.tencent.cloud.csi.cfs/bc6b83b6ab76c3374861a4a35a20a7972c79c506e5b47d412f46a1d63e6b9516/globalmount\"" pod="default/tao-86c948ff77-m8t58"
// mount
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.615495    4762 operation_generator.go:703] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb6v\" (UniqueName: \"kubernetes.io/projected/4ce98f8e-1ff8-4d78-b61c-739997ff026e-kube-api-access-cmb6v\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"
2月 21 17:58:43 VM-33-198-centos kubelet[4762]: I0221 17:58:43.656027    4762 operation_generator.go:703] "MountVolume.SetUp succeeded for volume \"pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d\" (UniqueName: \"kubernetes.io/csi/com.tencent.cloud.csi.cfs^cfs-3esjn2lz\") pod \"tao-86c948ff77-m8t58\" (UID: \"4ce98f8e-1ff8-4d78-b61c-739997ff026e\") " pod="default/tao-86c948ff77-m8t58"




2）csi-nodeplugin-cfsplugin/csi-cfs


// mount
I0221 09:58:43.604953       1 node.go:56] NodePublishVolume NodePublishVolumeRequest is: volume_id:"cfs-3esjn2lz" target_path:"/var/lib/kubelet/pods/4ce98f8e-1ff8-4d78-b61c-739997ff026e/volumes/kubernetes.io~csi/pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d/mount" volume_capability:<mount:<> access_mode:<mode:MULTI_NODE_MULTI_WRITER > > volume_context:<key:"fsid" value:"bz8eisu7" > volume_context:<key:"host" value:"10.0.33.133" > volume_context:<key:"pgroupid" value:"pgroup-lrquouvl" > volume_context:<key:"storage.kubernetes.io/csiProvisionerIdentity" value:"1676950325113-8081-com.tencent.cloud.csi.cfs" > volume_context:<key:"storagetype" value:"SD" > volume_context:<key:"subnetid" value:"subnet-rtwb42lu" > volume_context:<key:"vers" value:"3" > volume_context:<key:"vpcid" value:"vpc-oilua6pt" > volume_context:<key:"zone" value:"ap-guangzhou-3" > 
I0221 09:58:43.605662       1 node.go:150] CFS server 10.0.33.133:/bz8eisu7/ mount option is: [vers=3 noresvport nolock,proto=tcp]
I0221 09:59:03.388362       1 node.go:223] Enabling node service capability: GET_VOLUME_STATS
I0221 09:59:03.388377       1 node.go:223] Enabling node service capability: UNKNOWN
// 
I0221 09:59:03.391793       1 node.go:240] NodeGetVolumeStats is: volume_id:"cfs-3esjn2lz" volume_path:"/var/lib/kubelet/pods/4ce98f8e-1ff8-4d78-b61c-739997ff026e/volumes/kubernetes.io~csi/pvc-a11e120e-130c-4aff-9901-17b7b91b3f1d/mount" 




2.共享类型的CFS


10.0.1.128:/m002rxiu/default-cfs-share-pvc-58c574ae-a636-4296-aed9-ff1eec39de56    /var/lib/kubelet/pods/bc08cf37-1e19-4d6c-b350-98e4b5cb86b6/volumes/kubernetes.io~csi/pvc-58c574ae-a636-4296-aed9-ff1eec39de56/mount




在SC/PVC中，driver为“com.tencent.cloud.csi.tcfs.<SC的名字>”


在PV的spec.csi中，driver为“com.tencent.cloud.csi.cfs”。


2.1 创建SC和PVC


SC


apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cfs-share
parameters:
  pgroupid: pgroup-lrquouvl // 权限组(针对网络访问)
  storagetype: SD // CFS实例的类型：标准/性能
  subdir-share: "true" // 共享模式
  vers: "3" // NFS协议版本
  subnetid: subnet-rtwb42lu // 子网
  vpcid: vpc-oilua6pt // VPC网络
  zone: ap-guangzhou-3 // 可用区
provisioner: com.tencent.cloud.csi.tcfs.cfs-share // 以SC的名字，作为CSI类型
reclaimPolicy: Retain
volumeBindingMode: Immediate


PVC




apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.tcfs.cfs-share
    volume.kubernetes.io/storage-provisioner: com.tencent.cloud.csi.tcfs.cfs-share
  finalizers:
  - kubernetes.io/pvc-protection
  name: cfs-share-1
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: cfs-share
  volumeMode: Filesystem
  volumeName: pvc-651c987e-70c5-4542-93c1-2ba8184ad862
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  phase: Bound


2.2 为SC创建tcfs和deployment


1）创建的内容：


deoloyment




 




TCFS


apiVersion: tcfsoperator.k8s.io/v1alpha1
kind: Tcfs
metadata:
  name: cfs-share
  namespace: kube-system
spec:
  pgroupid: pgroup-lrquouvl // 权限组
  provisionername: com.tencent.cloud.csi.tcfs.cfs-share // CSI的名字
  reclaimpolicy: Retain
  storagetype: SD // CFS实例类型
  subnetid: subnet-rtwb42lu
  vpcid: vpc-oilua6pt
  zone: ap-guangzhou-3
status:
  cfsfilesystemid: wmyio3c8 // FSID
  cfsserver: 10.0.33.130 // CFS实例的IP




2）csi-provisioner-cfsplugin中的TCFS容器






storageclass_controller


1）处理PVC


根据PVC，拿到对应的SC。


如果SC为共享模式，创建对于tcfs对象。


2）处理SC


storageClass被删除,删除对应的tcf对象






tcfs_controller


监听CS对象。为SC对应的tcfs对象创建CFS实例，更新status，创建deploy。


SC和TCFS的命名空间的名字相同。


———-


步骤1:查询SC对应的TCFS的实际情况


1.查询TCFS是否存在


2.使用3种方法: 尝试获取TCFS，关联的cfs信息


1）根据tcfs.status的CfsServer和CfsFileSystemID字段


2）根据tcfs.Spec.FromExistCfs中记录的CFS实例ID


3）使用固定的CFS名字“<集群ID>_sharedCFS-<SC 名字>”，查询CFS实例


3.查询tcfs关联的deployment


步骤2:生成期望的TCFS状态


1）生成tcfs对应deploy的YAML文件。


步骤3:同步期望状态和实际状态


1）创建CFS实例【不删除CFS实例】


直接调用CFS接口创建。


2）更新或创建或删除deploy








2.3 SC对于的deploy，为PVC创建PV


针对共享类型的storgeclass，会创建下面的deployment。


com.tencent.cloud.csi.tcfs.<storgeclass的名字>


1）仓库


开源组件：GitHub – kubernetes-sigs/nfs-subdir-external-provisioner: Dynamic sub-dir volume provisioner on a remote NFS server.


2）启动参数




 


环境变量PROVISIONER_NAME：设置了要监听的CSI类型。


3) volume




 




4）作用：


NFS subdir external provisioner是一个自动配置器，使用现存已经配置好的NF服务器，来支持为PVC动态创建PV。


实现了Provisioner接口的两个方法：






Provision


1)在CFS下，创建子路径：


<PVC ns>-<PVC name>-<PV name>/自定于的路径模式


自定于的路径模式：在SC的”pathPattern”指定，可以为空


2)生成PV




 






Delete


1.根据sc的”onDelete”


判断是否删除/保留路径


2.根据sc的”archiveOnDelete”


T：将自路径重命名为“archived-<原路径>”








日志：


I0221 03:38:37.616545       1 controller.go:1317] provision "default/cfs-share-1" class "cfs-share": started
I0221 03:38:37.629793       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-share-1", UID:"651c987e-70c5-4542-93c1-2ba8184ad862", APIVersion:"v1", ResourceVersion:"419591929", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/cfs-share-1"
I0221 03:38:37.633497       1 controller.go:1420] provision "default/cfs-share-1" class "cfs-share": volume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862" provisioned
I0221 03:38:37.633541       1 controller.go:1437] provision "default/cfs-share-1" class "cfs-share": succeeded
I0221 03:38:37.633552       1 volume_store.go:212] Trying to save persistentvolume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862"
I0221 03:38:37.649191       1 volume_store.go:219] persistentvolume "pvc-651c987e-70c5-4542-93c1-2ba8184ad862" saved
I0221 03:38:37.649426       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"cfs-share-1", UID:"651c987e-70c5-4542-93c1-2ba8184ad862", APIVersion:"v1", ResourceVersion:"419591929", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-651c987e-70c5-4542-93c1-2ba8184ad862


2.4 创建POD，PVmount流程


在PV的spec.csi中，driver为“com.tencent.cloud.csi.cfs”。


与非共享类型的流程一致。






版权声明：本文为qq_34482492原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。