前言

Redis Cluster集群实现了对Redis的水平扩容，即启动N个redis节点，将整个数据库分布存储在这N个节点中，每个节点存储总数据的1/N。
Redis 集群通过分区（partition）来提供一定程度的可用性（availability）： 即使集群中有一部分节点失效或者无法进行通讯， 集群也可以继续处理命令请求。

本文将在Kubernetes环境中使用PV/PVC，基于StatefulSet控制器创建并运行Redis Cluster。

创建PV

NFS服务器

先在NFS服务器192.168.100.155创建目录并共享：

# 创建6个共享目录，属主属组选nfsnobody。
mkdir -p /data/k8sdata/myserver/redis{0..5}
chown nfsnobody.nfsnobody /data/k8sdata/myserver/redis{0..5}

# 提供共享服务，可共享的IP段为物理机IP，rw可读写，all_squash远程用户连到nfs服务器都压榨成nfsnobody用户。
cat << EOF >> /etc/exports
/data/k8sdata/myserver/redis0 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/redis1 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/redis2 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/redis3 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/redis4 192.168.100.0/24(rw,all_squash)
/data/k8sdata/myserver/redis5 192.168.100.0/24(rw,all_squash)
EOF

# 重新加载，查看挂载情况
exportfs -r
exportfs -v

# nfs涉及端口较多，测试环境可关闭防火墙
systemctl disable --now firewalld

生成PV

redis集群需创建3主3从6个pod，对应需创建6个pv：

apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv0
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis0 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv1
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis1 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv2
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis2 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv3
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis3 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv4
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis4 

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: redis-cluster-pv5
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: 192.168.100.155
    path: /data/k8sdata/myserver/redis5 

生成并查看pv：

sudo kubectl apply -f redis-cluster-pv.yaml
sudo kubectl get pv -n myserver 
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                              STORAGECLASS   REASON   AGE
redis-cluster-pv0        5Gi        RWO            Retain           Available                                                              32s
redis-cluster-pv1        5Gi        RWO            Retain           Available                                                              32s
redis-cluster-pv2        5Gi        RWO            Retain           Available                                                              32s
redis-cluster-pv3        5Gi        RWO            Retain           Available                                                              32s
redis-cluster-pv4        5Gi        RWO            Retain           Available                                                              32s
redis-cluster-pv5        5Gi        RWO            Retain           Available                                                              32s

Redis Cluster 集群

redis配置

开启aof持久化，开启集群，5秒超时，端口6379：

vim redis.conf

appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379

将配置文件保存至configmap：

sudo kubectl create configmap redis-conf --from-file=redis.conf -n myserver
sudo kubectl describe configmap redis-conf -n myserver

创建Redis Cluster

创建redis服务，redis-access服务，redis StatefulSet应用：

apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: myserver
  labels:
    app: redis
spec:
  selector:
    app: redis
    appCluster: redis-cluster
  ports:
  - name: redis
    port: 6379
  clusterIP: None
  
---
apiVersion: v1
kind: Service
metadata:
  name: redis-access
  namespace: myserver
  labels:
    app: redis
spec:
  selector:
    app: redis
    appCluster: redis-cluster
  ports:
  - name: redis-access
    protocol: TCP
    port: 6379
    targetPort: 6379

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
  namespace: myserver
spec:
  serviceName: redis
  replicas: 6
  selector:
    matchLabels:
      app: redis
      appCluster: redis-cluster
  template:
    metadata:
      labels:
        app: redis
        appCluster: redis-cluster
    spec:
      terminationGracePeriodSeconds: 20
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - redis
              topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: redis:4.0.14
        command:
          - "redis-server"
        args:
          - "/etc/redis/redis.conf"
          - "--protected-mode"
          - "no"
        resources:
          requests:
            cpu: "500m"
            memory: "500Mi"
        ports:
        - containerPort: 6379
          name: redis
          protocol: TCP
        - containerPort: 16379
          name: cluster
          protocol: TCP
        volumeMounts:
        - name: conf
          mountPath: /etc/redis
        - name: data
          mountPath: /var/lib/redis
      volumes:
      - name: conf
        configMap:
          name: redis-conf
          items:
          - key: redis.conf
            path: redis.conf
  volumeClaimTemplates:
  - metadata:
      name: data
      namespace: myserver
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

应用yaml，查看创建的pod：

kubectl get pod -A
myserver               redis-0                                      1/1     Running   0               61m
myserver               redis-1                                      1/1     Running   0               61m
myserver               redis-2                                      1/1     Running   0               61m
myserver               redis-3                                      1/1     Running   0               61m
myserver               redis-4                                      1/1     Running   0               61m
myserver               redis-5                                      1/1     Running   0               33m

查看pvc绑定情况：

kubectl get pvc -A
NAMESPACE   NAME                      STATUS   VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
myserver    data-redis-0              Bound    redis-cluster-pv1        5Gi        RWO                           58m
myserver    data-redis-1              Bound    redis-cluster-pv4        5Gi        RWO                           57m
myserver    data-redis-2              Bound    redis-cluster-pv3        5Gi        RWO                           57m
myserver    data-redis-3              Bound    redis-cluster-pv5        5Gi        RWO                           57m
myserver    data-redis-4              Bound    redis-cluster-pv0        5Gi        RWO                           57m
myserver    data-redis-5              Bound    redis-cluster-pv2        5Gi        RWO                           30m

初始化集群

redis集群使用前需要初始化一次，操作系统请选择ubuntu1804，初始化工具4版本用redis-tribe，5版本用redis-cli。

# 启动一次性系统容器ubuntu
kubectl run -it ubuntu1804 --image=ubuntu:18.04 --restart=Never -n myserver bash

以下操作在一次性容器ubuntu里执行：

#先更新安装包列表：
apt update
# 需要下载，时间较久。出现时区提示就选亚洲-上海：
apt install python2.7 python-pip redis-tools dnsutils iputils-ping net-tools
# 更新pip
pip install --upgrade pip
# 安装初始化工具
pip install redis-trib==0.5.1

# 创建3个master，注意拷贝时\后面不能有空格：
redis-trib.py create \
`dig +short redis-0.redis.myserver.svc.cluster.local`:6379 \
`dig +short redis-1.redis.myserver.svc.cluster.local`:6379 \
`dig +short redis-2.redis.myserver.svc.cluster.local`:6379
Redis-trib 0.5.1 Copyright (c) HunanTV Platform developers
INFO:root:Instance at 172.20.104.7:6379 checked
INFO:root:Instance at 172.20.104.6:6379 checked
INFO:root:Instance at 172.20.166.180:6379 checked
INFO:root:Add 5462 slots to 172.20.104.7:6379
INFO:root:Add 5461 slots to 172.20.104.6:6379
INFO:root:Add 5461 slots to 172.20.166.180:6379

# redis-3加入master redis-0
redis-trib.py replicate \
--master-addr `dig +short redis-0.redis.myserver.svc.cluster.local`:6379 \
 --slave-addr `dig +short redis-3.redis.myserver.svc.cluster.local`:6379
INFO:root:Instance at 172.20.166.181:6379 has joined 172.20.104.7:6379; now set replica
INFO:root:Instance at 172.20.166.181:6379 set as replica to eae814d972b63625fdca98bc039877a5d385da8a

# redis-4加入master redis-1
redis-trib.py replicate \
--master-addr `dig +short redis-1.redis.myserver.svc.cluster.local`:6379 \
 --slave-addr `dig +short redis-4.redis.myserver.svc.cluster.local`:6379
INFO:root:Instance at 172.20.135.4:6379 has joined 172.20.166.180:6379; now set replica
INFO:root:Instance at 172.20.135.4:6379 set as replica to ddedfa38fef4fd2323b2bf41f23b359bc50e523e

# redis-5加入master redis-2
redis-trib.py replicate \
--master-addr `dig +short redis-2.redis.myserver.svc.cluster.local`:6379 \
 --slave-addr `dig +short redis-5.redis.myserver.svc.cluster.local`:6379
INFO:root:Instance at 172.20.135.1:6379 has joined 172.20.104.6:6379; now set replica
INFO:root:Instance at 172.20.135.1:6379 set as replica to 6188dedfc8edb962d0e7557038005920437acaf3

验证redis集群

进入其中一个redis pod，检查集群状态：

kubectl exec -it redis-0 bash -n myserver
root@redis-0:/data# redis-cli

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_ping_sent:1550
cluster_stats_messages_pong_sent:1555
cluster_stats_messages_meet_sent:2
cluster_stats_messages_sent:3107
cluster_stats_messages_ping_received:1551
cluster_stats_messages_pong_received:1552
cluster_stats_messages_meet_received:4
cluster_stats_messages_received:3107

127.0.0.1:6379> CLUSTER NODES
1284f940cc0d1c47cf7d7e9c840b1c5259d04c7d 172.20.135.4:6379@16379 slave ddedfa38fef4fd2323b2bf41f23b359bc50e523e 0 1662476305216 4 connected
eae814d972b63625fdca98bc039877a5d385da8a 172.20.104.7:6379@16379 myself,master - 0 1662476305000 0 connected 0-5461
6188dedfc8edb962d0e7557038005920437acaf3 172.20.104.6:6379@16379 master - 0 1662476305718 2 connected 5462-10922
b8bebc35ded8ff313b155c32bf06921d44973b8a 172.20.135.1:6379@16379 slave 6188dedfc8edb962d0e7557038005920437acaf3 0 1662476305518 5 connected
ddedfa38fef4fd2323b2bf41f23b359bc50e523e 172.20.166.180:6379@16379 master - 0 1662476305000 1 connected 10923-16383
32d59439a69804c1b9643d35c5f19a0d5bd7cc03 172.20.166.181:6379@16379 slave eae814d972b63625fdca98bc039877a5d385da8a 0 1662476304000 3 connected

排错：资源不足

但创建pod时，一直处在pending状态：

myserver               redis-4                                      0/1     Pending   0               8m17s

查看pod具体情况，出现Insufficient cpu：

 Warning  FailedScheduling  91s   
 
 default-scheduler 0/4 nodes are available: 
 1 Insufficient cpu, 
 2 Insufficient memory, 
 2 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 
 2 node(s) were unschedulable. 
 
 preemption: 0/4 nodes are available: 
 2 No preemption victims found for incoming pod, 
 2 Preemption is not helpful for scheduling.

首先部署一台新服务器，配置好主机名和IP，免密登录。通过kubeasz将其加入node节点：

# 免密登录
ssh-copy-id 192.168.100.156
# 加入node
ezctl add-node k8s-cluster 192.168.100.156
# 查看节点
kubectl get node

值得注意的是，k8s在添加工作节点node成功后，会自动重新调度并创建pod。