kubernetes 에서 ceph rbd provisioner 활용 방법입니다.


1. 먼저 rbd provisioner 가 사용할 rbac 권한 yaml 파일을 만듭니다.
# vi rbd-rbac.yaml
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: rbd-provisioner
subjects:
  - kind: ServiceAccount
    name: rbd-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: rbd-provisioner
  apiGroup: rbac.authorization.k8s.io

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbd-provisioner
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: rbd-provisioner
  namespace: kube-system
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: rbd-provisioner
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbd-provisioner
subjects:
- kind: ServiceAccount
  name: rbd-provisioner
  namespace: kube-system 

# kubectl create -f rbd-rbac.yaml 



2. kube-system 네임스페이스로 admin secret 과 user secret 을 생성한다.
# ceph auth get client.admin 2>&1 |grep "key = " |awk '{print  $3'} |xargs echo -n > /tmp/secret.admin
# kubectl create secret generic ceph-secret-admin --type=kubernetes.io/rbd --from-file=/tmp/secret.admin --namespace=kube-system

# ceph auth get-key client.kube > /tmp/secret.user
# kubectl create secret generic ceph-secret-user --type=kubernetes.io/rbd --from-file=/tmp/secret.user --namespace=kube-system 




3. rbd provisioner 가 사용할 storage class yaml 파일을 만듭니다.
pool 이름과 userId 를  정확히 입력해야 합니다.

# vi rbd-storageclass.yaml

---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: "rbd"
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: ceph.com/rbd
reclaimPolicy: Delete
parameters:
  monitors: "192.168.30.23:6789,192.168.30.24:6789,192.168.30.25:6789"
  adminId: "admin"
  adminSecretName: "ceph-secret-admin"
  adminSecretNamespace: "kube-system"
  pool: "kubes"
  userId: "kube"
  userSecretName: "ceph-secret-user"
  userSecretNamespace: "kube-system"
  imageFormat: "2"
  imageFeatures: "layering" 


# kubectl create -f rbd-storageclass.yaml 



4. rbd provisioner 를 deployment 타입 yaml 로 생성합니다.
# vi deployment-rbd-provisioner.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbd-provisioner
  namespace: kube-system
  labels:
    app: rbd-provisioner
    version: v2.1.1-k8s1.11
spec:
  replicas: 2
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: rbd-provisioner
      version: v2.1.1-k8s1.11
  template:
    metadata:
      labels:
        app: rbd-provisioner
        version: v2.1.1-k8s1.11
    spec:
      priorityClassName: system-cluster-critical
      serviceAccount: rbd-provisioner
      containers:
        - name: rbd-provisioner
          image: quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11
          imagePullPolicy: IfNotPresent
          env:
            - name: PROVISIONER_NAME
              value: ceph.com/rbd
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          command:
            - "/usr/local/bin/rbd-provisioner"
          args:
            - "-id=${POD_NAME}" 


# kubectl create -f deployment-rbd-provisioner.yaml 




5. 이제 default 네임스페이스에 테스트를 해 보겠습니다.
먼저 default 네임스페이스에 user secret 을 생성합니다. (rbd provisioner 를 사용하고자 하는 네임스페이스에는 user secret 을 만들어 줘야 합니다.)
# ceph auth get-key client.kube > /tmp/secret.user
# kubectl create secret generic ceph-secret-user --type=kubernetes.io/rbd --from-file=/tmp/secret.user --namespace=default


6. 테스트용 pod 생성.

# vi rbd-test.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-test
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd
  resources:
    requests:
      storage: 1Gi

---
apiVersion: v1
kind: Pod
metadata:
  name: rbd-test
  namespace: default
spec:
  containers:
  - name: pod-test
    image: gcr.io/google_containers/busybox:1.24
    command:
    - "/bin/sh"
    args:
    - "-c"
    - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
    - name: pvc
      mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
  - name: pvc
    persistentVolumeClaim:
      claimName: rbd-test 



# kubectl create -f rbd-test.yaml 





Posted by Kubernetes Korea co-leader seungkyua@gmail.com