kubernetes 에서 ceph rbd provisioner 활용 방법입니다.
1. 먼저 rbd provisioner 가 사용할 rbac 권한 yaml 파일을 만듭니다.
# vi rbd-rbac.yaml
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: kube-system
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
namespace: kube-system
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: kube-system
# kubectl create -f rbd-rbac.yaml
2. kube-system 네임스페이스로 admin secret 과 user secret 을 생성한다.
# ceph auth get client.admin 2>&1 |grep "key = " |awk '{print $3'} |xargs echo -n > /tmp/secret.admin
# kubectl create secret generic ceph-secret-admin --type=kubernetes.io/rbd --from-file=/tmp/secret.admin --namespace=kube-system
# ceph auth get-key client.kube > /tmp/secret.user
# kubectl create secret generic ceph-secret-user --type=kubernetes.io/rbd --from-file=/tmp/secret.user --namespace=kube-system
3. rbd provisioner 가 사용할 storage class yaml 파일을 만듭니다.
pool 이름과 userId 를 정확히 입력해야 합니다.
# vi rbd-storageclass.yaml
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: "rbd"
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: ceph.com/rbd
reclaimPolicy: Delete
parameters:
monitors: "192.168.30.23:6789,192.168.30.24:6789,192.168.30.25:6789"
adminId: "admin"
adminSecretName: "ceph-secret-admin"
adminSecretNamespace: "kube-system"
pool: "kubes"
userId: "kube"
userSecretName: "ceph-secret-user"
userSecretNamespace: "kube-system"
imageFormat: "2"
imageFeatures: "layering"
# kubectl create -f rbd-storageclass.yaml
4. rbd provisioner 를 deployment 타입 yaml 로 생성합니다.
# vi deplyment-rbd-provisioner.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: kube-system
labels:
app: rbd-provisioner
version: v2.1.1-k8s1.11
spec:
replicas: 2
strategy:
type: Recreate
selector:
matchLabels:
app: rbd-provisioner
version: v2.1.1-k8s1.11
template:
metadata:
labels:
app: rbd-provisioner
version: v2.1.1-k8s1.11
spec:
priorityClassName: system-cluster-critical
serviceAccount: rbd-provisioner
containers:
- name: rbd-provisioner
image: quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
command:
- "/usr/local/bin/rbd-provisioner"
args:
- "-id=${POD_NAME}"
# kubectl create -f deplyment-rbd-provisioner.yaml
5. 이제 default 네임스페이스에 테스트를 해 보겠습니다.
먼저 default 네임스페이스에 user secret 을 생성합니다. (rbd provisioner 를 사용하고자 하는 네임스페이스에는 user secret 을 만들어 줘야 합니다.)
# ceph auth get-key client.kube > /tmp/secret.user
# kubectl create secret generic ceph-secret-user --type=kubernetes.io/rbd --from-file=/tmp/secret.user --namespace=default
6. 테스트용 pod 생성.
# vi rbd-test.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-test
namespace: default
spec:
accessModes:
- ReadWriteOnce
storageClassName: rbd
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: pod-test
namespace: default
spec:
containers:
- name: pod-test
image: gcr.io/google_containers/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: pvc
persistentVolumeClaim:
claimName: rbd-test
# kubectl create -f rbd-test.yaml
