User Guide

Set Storage Class

An Example of Storage Class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  name: csi-qingcloud
provisioner: disk.csi.qingcloud.com
parameters:
  type: "0"
  fsType: "ext4"
  replica: "2"
  tags: "tag-y7uu1q2a"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

Parameters in Storage Class

type, maxSize, minSize, stepSize

See details in QingCloud docs。

Disk	type	maxSize(GB)	minSize(GB)	stepSize(GB)
High Performance	0	2000	20	10
High Capacity	2	5000	20	10
Super High Performance	3	2000	20	10
NeonSAN	5	24000	20	10
NeonSAN HDD	6	32000	20	10
Standard	100	2000	10	10
SSD Enterprise	200	2000	20	10

fsType

Support ext3, ext4, xfs. Default is ext4.

replica

1 represents single duplication disk，2 represents multiple duplication disk. Default is 2.

Other Parameters

Set Default Storage Class

In annotation, please set the value of .metadata.annotations.storageclass.beta.kubernetes.io/is-default-class as true. See details in Kubernetes docs

Expand Volume

Set the value of .allowVolumeExpansion as true. See details in Kubernetes docs

Topology Awareness

We can set Immediate or WaitForFirstConsumer as the value of .volumeBindingMode. See details in Kubernetes docs

Disk Type Matrix

Disk	type
High Performance	0
High Capacity	2
Super High Performance	3
NeonSAN	5
NeonSAN HDD	6
NeonSAN RDMA	7
Third Party Storage	20
Standard	100
SSD Enterprise	200

Instance Type Matrix

Instance	type
High Performance	0
Super High Performance	1
Super High Performance SAN	6
High Performance SAN	7
Standard	101
Enterprise1	201
Enterprise2	202
Enterprise3	203
Premium	301
NvidiaAmpereG3	1003

Disk Compatibility Matrix

	[0]High Performance Disk	[2]High Capacity Disk	[3]Super High Performance Disk	[5]NeonSAN Disk	[6]NeonSAN HDD Disk	[7]NeonSAN RDMA Disk	[20]Third Party Storage Disk	[100]Standard Disk	[200]SSD Enterprise Disk
[0]High Performance Instance	✓	✓	-	✓	✓	✓	-	✓	-
[1]Super High Performance Instance	-	✓	✓	✓	✓	✓	-	-	✓
[6]Super High Performance SAN Instance	-	-	-	✓	✓	✓	-	-	-
[7]High Performance SAN Instance	-	-	-	✓	✓	✓	-	-	-
[101]Standard Instance	✓	✓	-	✓	✓	✓	✓	✓	-
[201]Enterprise1 Instance	-	✓	✓	✓	✓	✓	-	-	✓
[202]Enterprise2 Instance	-	✓	✓	✓	✓	✓	-	-	✓
[203]Enterprise3 Instance	-	-	-	✓	✓	✓	-	-	✓
[301]Premium Instance	-	✓	✓	✓	✓	✓	-	-	✓
[1003]NvidiaAmpereG3 Instance	-	✓	-	✓	✓	✓	-	-	-

Set Snapshot Class

An Example of Snapshot Class

apiVersion: snapshot.storage.k8s.io/v1beta1
kind: VolumeSnapshotClass
metadata:
  name: csi-qingcloud
driver: disk.csi.qingcloud.com
parameters:
  tags: "tag-y7uu1q2a"
deletionPolicy: Delete

Parameters in Storage Class

Volume Management

Volume management including dynamical provisioning/deleting volume, attaching/detaching volume. Please reference Example YAML Files。

Prerequisite

Kubernetes 1.16+ Cluster
Installed QingCloud CSI plugin
Created QingCloud CSI storage class

Create Storage Class

Create

$ kubectl create -f sc.yaml

Check

$ kubectl get sc
NAME            PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-qingcloud   disk.csi.qingcloud.com   Delete          Immediate           true                   40h

Create Volume

Create

$ kubectl create -f pvc.yaml 
persistentvolumeclaim/pvc-example created

Check

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-76429525-a930-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   25m

Mount Volume

Create Deployment

$ kubectl create -f deploy-nginx.yaml 
deployment.apps/deploy-nginx created

Check

$ kubectl exec -ti deploy-nginx-84474cf674-zfhbs /bin/bash
# cd /mnt
# ls
lost+found

Unmount Volume

Delete Deployment

$ kubectl delete deploy deploy-nginx
deployment.extensions "deploy-nginx" deleted

Delete Volume

Delete

$ kubectl delete pvc pvc-example
persistentvolumeclaim "pvc-example" deleted

Check

$ kubectl get pvc pvc-example
Error from server (NotFound): persistentvolumeclaims "pvc-example" not found

Expand Volume

This feature could expand the capacity of volume.

Prerequisite

Kubernetes 1.17+ cluster
Enable (Default enabled) ExpandCSIVolumes=true in feature-gates
Set allowVolumeExpansion as true in storage class
Create a Pod mounting a volume

Expand Volume Online

csi-qingcloud support online volume expansion from v1.3.6.

Change volume capacity

$ kubectl patch pvc pvc-example -p '{"spec":{"resources":{"requests":{"storage": "40Gi"}}}}'
persistentvolumeclaim/pvc-example patched

Check volume capacity

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-906f5760-a935-11e9-9a6a-5254ef68c8c1   40Gi       RWO            csi-qingcloud   6m7s
$ kubectl get po
NAME                            READY   STATUS    RESTARTS   AGE
deploy-nginx-6c444c9b7f-d6n29   1/1     Running   0          3m38s

check

$ kubectl exec -ti deploy-nginx-84474cf674-zfhbs /bin/bash
# df -hl | grep /mnt
/dev/vdg         40G   48M   40G   1% /mnt

Expand Volume Offline

For previous versions, csi-qingcloud only supports offline volume expansion. The procedure of offline volume expansion is shown as follows.

Ensure volume in unmounted status
Edit the capacity of PVC
Mount volume on workload. Reference Example YAML files

Unmount Volume

$ kubectl scale deploy deploy-nginx --replicas=0

Expand Volume

Change volume capacity

$ kubectl patch pvc pvc-example -p '{"spec":{"resources":{"requests":{"storage": "40Gi"}}}}'
persistentvolumeclaim/pvc-example patched

Mount volume

$ kubectl scale deploy deploy-nginx --replicas=1

Check volume capacity

$ kubectl get pvc pvc-example
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-906f5760-a935-11e9-9a6a-5254ef68c8c1   40Gi       RWO            csi-qingcloud   6m7s
$ kubectl get po
NAME                            READY   STATUS    RESTARTS   AGE
deploy-nginx-6c444c9b7f-d6n29   1/1     Running   0          3m38s

Check

$ kubectl exec -ti deploy-nginx-6c444c9b7f-d6n29 /bin/bash
root@deploy-nginx-6c444c9b7f-d6n29:/# s
bash: s: command not found
root@deploy-nginx-6c444c9b7f-d6n29:/# df -ah
Filesystem      Size  Used Avail Use% Mounted on
...
/dev/vdc         40G   49M   40G   1% /mnt
...

Clone Volume

A Clone is defined as a duplicate of an existing Kubernetes Volume. Please reference Example YAML files.

Prerequisite

Kubernetes 1.16+ cluster
Enable (Default enabled) VolumePVCDataSource=true feature gate
Install QingCloud CSI plugin
Create QingCloud CSI storage class
Create a volume

Cloning

Find volume

$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-example   Bound    pvc-d1fb263e-b368-4339-8f8b-448446f4b840   20Gi       RWO            csi-qingcloud   32s

Clone volume

$ kubectl create -f pvc-clone.yaml
persistentvolumeclaim/pvc-clone created

Check

$ kubectl get pvc pvc-clone
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-clone   Bound    pvc-529d2502-02bd-442b-a69f-d3eff28316a8   20Gi       RWO            csi-qingcloud   31s

Snapshot Management

Snapshot management contains creating/deleting snapshot and restoring volume from snapshpot. Please reference Example YAML files.

Prerequisite

Kubernetes 1.17+ cluster
Enable (Default enabled) VolumeSnapshotDataSource=true feature gate at kube-apiserver and kube-controller-manager
Install QingCloud CSI plugin
Create QingCloud CSI storage class
Create a volume

Create a volume pre-populated data

Create volume

$ kubectl create -f original-pvc.yaml
persistentvolumeclaim/pvc-snap-1 created

Check

$ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-snap-1   Bound    pvc-28090960-9eeb-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   37s

Write data

$ kubectl create -f deploy-writer.yaml 
deployment.apps/fio created

$ kubectl get po
NAME                   READY   STATUS    RESTARTS   AGE
fio-645b5d6499-8tc7p   1/1     Running   0          23s

$ kubectl exec -ti fio-645b5d6499-8tc7p /bin/bash
root@fio-645b5d6499-8tc7p:/# cd root
root@fio-645b5d6499-8tc7p:/# ./start-test.sh
crtl+c (5 秒后执行此命令，停止写数据)
root@fio-645b5d6499-8tc7p:/# ls -lh /mnt
total 20G
drwx------ 2 root root  16K Jul  5 06:09 lost+found
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.2.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.3.0
-rw-r--r-- 1 root root  10G Jul  5 06:10 seq-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.2.0
-rw-r--r-- 1 root root 9.6G Jul  5 06:10 seq-write.3.0

Create Snapshot

Create Volume Snapshot Class

$ kubectl create -f snapshot-class.yaml 
volumesnapshotclass.snapshot.storage.k8s.io/csi-qingcloud created

$ kubectl get volumesnapshotclass
NAME            AGE
csi-qingcloud   16s

Craete Volume Snapshot

$ kubectl create -f volume-snapshot.yaml 
volumesnapshot.snapshot.storage.k8s.io/snap-1 created

$ kubectl get volumesnapshot
NAME     AGE
snap-1   91s

Restore Volume from Snapshot

Restore

$ kubectl create -f restore-pvc.yaml 
persistentvolumeclaim/pvc-snap-2 created

$ kubectl get pvc pvc-snap-2
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-snap-2   Bound    pvc-b8a05427-9eef-11e9-9a6a-5254ef68c8c1   20Gi       RWO            csi-qingcloud   52s

Check Data

Compare the difference between restored volume with original volume.

$ kubectl create -f deploy-viewer.yaml 
deployment.apps/nginx created

$ kubectl get po |grep snap-example 
snap-example-85dd9b646c-56g85   1/1     Running   0          3m6s

$ kubectl exec -ti snap-example-85dd9b646c-56g85 /bin/bash
root@snap-example-85dd9b646c-56g85:/# ls /mnt -lh
total 20G
drwx------ 2 root root  16K Jul  5 06:09 lost+found
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.2.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 rand-write.3.0
-rw-r--r-- 1 root root  10G Jul  5 06:10 seq-write.0.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.1.0
-rw-r--r-- 1 root root    0 Jul  5 06:10 seq-write.2.0
-rw-r--r-- 1 root root 9.6G Jul  5 06:10 seq-write.3.0

Delete Snapshot

$ kubectl delete volumesnapshot snap-1
volumesnapshot.snapshot.storage.k8s.io "snap-1" deleted

Topology Awareness

Topology awareness is used at Kubernetes clusters whose nodes across different available zones or having different types of instance. Please reference Example YAML files.

Prerequisite

Kubernetes 1.16+ cluster
Enable CSINodeInfo=true feature gate at Kubernetes control plane and Kubelet
Install QingCloud CSI plugin and enable Topology=true feature gate at external-provisioner sidecar container
Set QingCloud CSI storage class

Example Kubernetes Cluster

In QingCloud Pek3 zone, a Kubernetes v1.15 cluster with same types of instance is created and node1 and node2 running in Pek3c, node3 and node4 running in Pek3b, node5 and node6 running in Pek3d.

Create Storage Class

Volume binding mode can be set as WaitForFirstConsumer or Immediate. Please reference Kubernetes docs.

$ kubectl create -f sc.yaml

$ kubectl get sc csi-qingcloud -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-qingcloud
parameters:
  fsType: ext4
  replica: "2"
  type: "100"
provisioner: disk.csi.qingcloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Create Volume

Create

$ kubectl create -f pvc.yaml
persistentvolumeclaim/pvc-topology created

If VolumeBindingMode set as WaitForFirstConsumer, the status of PVC is shown as Pending. After Pod mounted PVC is sheduled, the PVC status will change to Bound.

$ kubectl get pvc pvc-topology
NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-topology   Pending                                      csi-qingcloud   31s

Create Workload

Create Deployment

$ kubectl create -f deploy.yaml
deployment.apps/nginx-topology created

Check if Pods scheduled on node 3

$ kubectl get po -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
nginx-topology-79d8d5d86d-4lvcl    1/1     Running   0          52s   10.233.92.27     node3   <none>           <none>

Check volume bound

$ kubectl get pvc pvc-topology
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-topology   Bound    pvc-5b34120c-6119-4c86-b9de-e152304683e6   20Gi       RWO            csi-qingcloud   2m48s

The volume named pvc-topology contains topology information and can be mounted on special nodes. In this example, the volume can be mounted on node3 or node4.

Static Volume Provisioning

Static volume provisioning is also called pre-provisioning volume. The process is shown below.

Create QingCloud disk
Create PV
Create PVC

Process

Prerequisite

Create Kubernetes cluster on QingCloud IaaS platform
Install QingCloud CSI plugin
Create QingCloud disk

Create Storage Class

$ kubectl get sc csi-qingcloud -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"csi-qingcloud","namespace":""},"parameters":{"fsType":"ext4","maxSize":"500","minSize":"10","stepSize":"10","type":"0"},"provisioner":"csi-qingcloud","reclaimPolicy":"Delete"}
  creationTimestamp: 2018-08-06T02:20:19Z
  name: csi-qingcloud
  resourceVersion: "1355065"
  selfLink: /apis/storage.k8s.io/v1/storageclasses/csi-qingcloud
  uid: 43f25337-991f-11e8-b5aa-525445c0b555
parameters:
  fsType: ext4
  type: "0"
provisioner: csi-qingcloud
reclaimPolicy: Delete
volumeBindingMode: Immediate

Create PV

Edit

$ vi pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi-qingcloud
  name: pv-static
spec:
  capacity:
    storage: 20Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: csi-qingcloud
  csi:
    driver: csi-qingcloud
    fsType: ext4
    volumeAttributes:
      fsType: ext4
      type: "0"
    volumeHandle: vol-jjtedp2i

Create

$ kubectl create -f pv.yaml

Check

$ kubectl get pv pv-static
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM     STORAGECLASS    REASON    AGE
pv-static   20Gi       RWO            Delete           Available             csi-qingcloud             8m

Create PVC

Edit

$ vi pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: csi-qingcloud
  name: pvc-static
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: csi-qingcloud

Create

$ kubectl create -f pvc.yaml

Check

$  kubectl get pvc pvc-static
NAME         STATUS    VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS    AGE
pvc-static   Bound     pv-static   20Gi       RWO            csi-qingcloud   11s

$ kubectl get pv pv-static 
NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                STORAGECLASS    REASON    AGE
pv-static   20Gi       RWO            Delete           Bound     default/pvc-static   csi-qingcloud             12m

FilesExpand file tree

user-guide.md

Latest commit

History

user-guide.md

File metadata and controls

User Guide

Set Storage Class

An Example of Storage Class

Parameters in Storage Class

type, maxSize, minSize, stepSize

fsType

replica

tags

Other Parameters

Set Default Storage Class

Expand Volume

Topology Awareness

Disk Type Matrix

Instance Type Matrix

Disk Compatibility Matrix

Set Snapshot Class

An Example of Snapshot Class

Parameters in Storage Class

tags

Volume Management

Prerequisite

Create Storage Class

Create Volume

Mount Volume

Unmount Volume

Delete Volume

Expand Volume

Prerequisite

Expand Volume Online

Expand Volume Offline

Unmount Volume

Expand Volume

Check

Clone Volume

Prerequisite

Cloning

Snapshot Management

Prerequisite

Create a volume pre-populated data

Create Snapshot

Create Volume Snapshot Class

Craete Volume Snapshot

Restore Volume from Snapshot

Restore

Check Data

Delete Snapshot

Topology Awareness

Prerequisite

Example Kubernetes Cluster

Create Storage Class

Create Volume

Create Workload

Static Volume Provisioning

Process

Prerequisite

Create Storage Class

Create PV

Create PVC