Redis-集群

一、为什么要用集群？

业务量大，单个Master难以承担所有负载

Redis主从（主挂就不能写了）
-> 哨兵（可以进行故障切换，但只有一个主）
-> 集群(多主多从)

Redis集群：

1. 可以在多个Redis节点之间共享数据程序集
2. 支持多个Master(自带Sentinel的故障转移机制，自动将从切为主)
3. 支持读写分离
4. 支持海量的读写存储操作

集群算法-分片-槽位slot

• 数据分片：使用Redis集群时会将数据分散到多台Redis服务器上，一台Redis实例被称为一个分片

• 如何找到给定key的分片
  对key进行CRC16(key)算法 % 总槽位数
  使用确定性哈希函数，对给定的key通过CRC16计算始终得到同一个结果，数据会映射到同一个分片

• 哈希槽
  哈希槽实际就是一个数组，[0,2^14-1] 形成hash slot空间
  2^14-1=16383
  一共是16384个哈希槽

• 解决什么问题

  1. 解决均匀分配的问题
  2. 方便数据移动（扩容、缩容）

二、搭建集群

三主三从
6379 -> 6479
6389 -> 6489
6399 -> 6499

配置

/etc/redisCluster6379.conf
bind 0.0.0.0
daemonize yes 
protected-mode no
port 6379
logfile "/redisdata/cluster/cluster6379.log"
pidfile /redisdata/cluster6379.pid
dir /redisdata/cluster
dbfilename dump6379.rdb
appendonly yes 
appendfilename "appendonly6379.aof"
requirepass 12345678
masterauth 12345678

# 打开集群
cluster-enabled yes 
# 集群配置文件
cluster-config-file nodes-6379.conf
# 节点超时时间
cluster-node-timeout 5000

其他节点的配置文件修改端口即可（:%s/6379/6399/g）

建目录和启动

mkdir -p /redisdata/cluster/
redis-server /etc/redisCluster6379.conf

实践

1. 将之前的所有实例都停掉
2. 按上面的配置文件启动6个Redis实例

查看端口

[root@db ~]# netstat -tulnp |grep redis
tcp        0      0 0.0.0.0:16489           0.0.0.0:*               LISTEN      24203/redis-server  
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      24135/redis-server  
tcp        0      0 0.0.0.0:6479            0.0.0.0:*               LISTEN      24210/redis-server  
tcp        0      0 0.0.0.0:16399           0.0.0.0:*               LISTEN      24189/redis-server  
tcp        0      0 0.0.0.0:16499           0.0.0.0:*               LISTEN      24196/redis-server  
tcp        0      0 0.0.0.0:6389            0.0.0.0:*               LISTEN      24182/redis-server  
tcp        0      0 0.0.0.0:6489            0.0.0.0:*               LISTEN      24203/redis-server  
tcp        0      0 0.0.0.0:16379           0.0.0.0:*               LISTEN      24135/redis-server  
tcp        0      0 0.0.0.0:16479           0.0.0.0:*               LISTEN      24210/redis-server  
tcp        0      0 0.0.0.0:6399            0.0.0.0:*               LISTEN      24189/redis-server  
tcp        0      0 0.0.0.0:6499            0.0.0.0:*               LISTEN      24196/redis-server  
tcp        0      0 0.0.0.0:16389           0.0.0.0:*               LISTEN      24182/redis-server  

–cluster-replicas 1 为每个Master创建一个slave

[root@db ~]# redis-cli -a 12345678 --cluster create --cluster-replicas 1

4 => 2主2从
6 => 3主3主

(前3台设置为Master了，后3台设置为Slave了)

[root@db ~]# redis-cli -a 12345678 --cluster create --cluster-replicas 1 192.168.100.154:6379 192.168.100.154:6479 192.168.100.154:6389 192.168.100.154:6489 192.168.100.154:6399 192.168.100.154:6499

(type 'yes' to accept): yes
>>> Performing Cluster Check (using node 192.168.100.154:6379)
M: 8f8fd5cd518965c80638baf064a4865983f84fa0 192.168.100.154:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: f5e0816814edc79db120827045c5e8b5f6757658 192.168.100.154:6399
   slots: (0 slots) slave
   replicates 12b85989fddcc4bbb40a44b26986d55bfd98e5a6
M: b4bd9800fc2750c51f3b26b906eb07f32874923d 192.168.100.154:6479
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 007663d4755ddea346c0fcffeb62df7b19c12904 192.168.100.154:6499
   slots: (0 slots) slave
   replicates 8f8fd5cd518965c80638baf064a4865983f84fa0
M: 12b85989fddcc4bbb40a44b26986d55bfd98e5a6 192.168.100.154:6389
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: f77a84f86e0d5f6409033b48c0498c9f7a81bda1 192.168.100.154:6489
   slots: (0 slots) slave
   replicates b4bd9800fc2750c51f3b26b906eb07f32874923d

192.168.100.154:6379 [0-5460]      -> 192.168.100.154:6499
192.168.100.154:6479 [5461-10922]  -> 192.168.100.154:6489
192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399

[root@db ~]# cat /redisdata/cluster/nodes-6379.conf
8f8fd5cd518965c80638baf064a4865983f84fa0 192.168.100.154:6379@16379,,tls-port=0,shard-id=e0ec214a38136a970759e00b216c1aa94c06b85c myself,master - 0 0 1 connected 0-5460
f5e0816814edc79db120827045c5e8b5f6757658 192.168.100.154:6399@16399,,tls-port=0,shard-id=73521525899d1cb1e8338ea9d46ce887366d6340 slave 12b85989fddcc4bbb40a44b26986d55bfd98e5a6 0 1757125969175 3 connected
b4bd9800fc2750c51f3b26b906eb07f32874923d 192.168.100.154:6479@16479,,tls-port=0,shard-id=dfacd4e4f6047668583a32a519a4707973b7a84f master - 0 1757125969085 2 connected 5461-10922
007663d4755ddea346c0fcffeb62df7b19c12904 192.168.100.154:6499@16499,,tls-port=0,shard-id=e0ec214a38136a970759e00b216c1aa94c06b85c slave 8f8fd5cd518965c80638baf064a4865983f84fa0 0 1757125969085 1 connected
12b85989fddcc4bbb40a44b26986d55bfd98e5a6 192.168.100.154:6389@16389,,tls-port=0,shard-id=73521525899d1cb1e8338ea9d46ce887366d6340 master - 0 1757125969606 3 connected 10923-16383
f77a84f86e0d5f6409033b48c0498c9f7a81bda1 192.168.100.154:6489@16489,,tls-port=0,shard-id=dfacd4e4f6047668583a32a519a4707973b7a84f slave b4bd9800fc2750c51f3b26b906eb07f32874923d 0 1757125969070 2 connected
vars currentEpoch 6 lastVoteEpoch 0

查看单台的主从信息

[root@db ~]# redis-cli -a 12345678 -p 6499
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6499> info replication
# Replication
role:slave
master_host:192.168.100.154
master_port:6379
master_link_status:up

查看集群节点

127.0.0.1:6499> cluster nodes
f5e0816814edc79db120827045c5e8b5f6757658 192.168.100.154:6399@16399 slave 12b85989fddcc4bbb40a44b26986d55bfd98e5a6 0 1757127996519 3 connected
f77a84f86e0d5f6409033b48c0498c9f7a81bda1 192.168.100.154:6489@16489 slave b4bd9800fc2750c51f3b26b906eb07f32874923d 0 1757127995000 2 connected
b4bd9800fc2750c51f3b26b906eb07f32874923d 192.168.100.154:6479@16479 master - 0 1757127997062 2 connected 5461-10922
8f8fd5cd518965c80638baf064a4865983f84fa0 192.168.100.154:6379@16379 master - 0 1757127996000 1 connected 0-5460
007663d4755ddea346c0fcffeb62df7b19c12904 192.168.100.154:6499@16499 myself,slave 8f8fd5cd518965c80638baf064a4865983f84fa0 0 0 1 connected
12b85989fddcc4bbb40a44b26986d55bfd98e5a6 192.168.100.154:6389@16389 master - 0 1757127995975 3 connected 10923-16383

127.0.0.1:6499> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:3625
cluster_stats_messages_pong_sent:3602
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:7228
cluster_stats_messages_ping_received:3602
cluster_stats_messages_pong_received:3626
cluster_stats_messages_received:7228
total_cluster_links_buffer_limit_exceeded:0

192.168.100.154:6379 [0-5460] -> 192.168.100.154:6499
192.168.100.154:6479 [5461-10922] -> 192.168.100.154:6489
192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399

未加 -c 参数（开启集群模式）时的 MOVED 错误 -> Redis 集群中，每个 key 会通过 CRC16(key) % 16384 计算出对应的槽位（k1 计算后为槽 12706）
6499 节点不负责槽 12706

127.0.0.1:6499> keys *
(empty array)
127.0.0.1:6499> set k1 v1
(error) MOVED 12706 192.168.100.154:6389
k1 => crc16("k1") % 16384 => slot 12706 => 192.168.100.154:6389

加 -c 参数后的自动重定向
（-c 表示开启集群模式,让客户端自动跟随集群的槽路由，无需手动切换节点）

[root@db ~]# redis-cli -a 12345678 -p 6499 -c
127.0.0.1:6499> keys *
(empty array)
127.0.0.1:6499> set k1 v1
-> Redirected to slot [12706] located at 192.168.100.154:6389
OK
192.168.100.154:6389> keys *
1) "k1"
192.168.100.154:6389> set k2 v2
-> Redirected to slot [449] located at 192.168.100.154:6379
OK
192.168.100.154:6379> set k3 v3
OK
192.168.100.154:6379> set k4 v4
-> Redirected to slot [8455] located at 192.168.100.154:6479
OK
192.168.100.154:6479> set k5 v5
-> Redirected to slot [12582] located at 192.168.100.154:6389
OK
192.168.100.154:6389> keys *
1) "k5"
2) "k1"


127.0.0.1:6399> keys *
1) "k5"
2) "k1"

查看某个key属于哪个槽位

192.168.100.154:6389> cluster keyslot k1
(integer) 12706

三、集群测试：故障迁移

192.168.100.154:6379 [0-5460] -> 192.168.100.154:6499
192.168.100.154:6479 [5461-10922] -> 192.168.100.154:6489
192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399

1. 停掉一个主，观察对应的从节点是否会切换成主节点

192.168.100.154:6389> shutdown
not connected> 
[root@db ~]# redis-cli -a 12345678 -p 6399 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6399> info replication
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:77c60421494ea8556cc733c43c2187948d6e4860
master_replid2:38e44e1859b755f33dbeaa80f16427cae5767fb8
master_repl_offset:3931
second_repl_offset:3932
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:15
repl_backlog_histlen:3917

如果主机又上线了，会又成主吗？

[root@db ~]# redis-server /etc/redisCluster6389.conf
[root@db ~]# redis-cli -a 12345678 -p 6389 -c
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6389> info replication
# Replication
role:slave
master_host:192.168.100.154
master_port:6399
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_read_repl_offset:3973

手动调整主从关系
把哪台切换成主，就在哪台上执行

127.0.0.1:6399> cluster failover
127.0.0.1:6389> cluster failover
OK
127.0.0.1:6389> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.100.154,port=6399,state=online,offset=4882,lag=0
master_failover_state:no-failover
master_replid:47e5673b0a8dfdd017ed8854d3014535c1a1c248
master_replid2:77c60421494ea8556cc733c43c2187948d6e4860
master_repl_offset:4882
second_repl_offset:4869
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:3946
repl_backlog_histlen:937

192.168.100.154:6379 [0-5460] -> 192.168.100.154:6499
192.168.100.154:6479 [5461-10922] -> 192.168.100.154:6489
192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399

四、集群扩容（增加主从）

1. 规划：6300主,6400从
   cp 配置文件
   修改一下端口

   cp /etc/redisCluster6379.conf /etc/redisCluster6300.conf
   cp /etc/redisCluster6379.conf /etc/redisCluster6400.conf
   sed -i ‘s/6379/6300/g’ /etc/redisCluster6300.conf
   sed -i ‘s/6379/6400/g’ /etc/redisCluster6400.conf

2. 启动服务

[root@db ~]# redis-server /etc/redisCluster6300.conf
[root@db ~]# redis-server /etc/redisCluster6400.conf

3. 查看服务状态（主从）

[root@db ~]# redis-cli -h 127.0.0.1 -p 6300 -a 12345678
127.0.0.1:6300> info replication
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:f6eed41e41e84f818e52554166cafa57b77f2cec
master_replid2:0000000000000000000000000000000000000000

[root@db ~]# redis-cli -h 127.0.0.1 -p 6400 -a 12345678
127.0.0.1:6400> info replication
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:a25d195fc4c03817abd902a87ce8a8e2ecaff752

4. 操作新master 6300
   192.168.100.154:6379 [0-5460] -> 192.168.100.154:6499
   192.168.100.154:6479 [5461-10922] -> 192.168.100.154:6489
   192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399

• 将新master加入集群6300

redis-cli -a 12345678 --cluster add-node <新ip:端口> <原集群ip:端口>
[root@db ~]# redis-cli -a 12345678 --cluster add-node 192.168.100.154:6300 192.168.100.154:6379

• 新主节点没有slot号
  redis-cli -a 12345678 --cluster check <集群ip:端口>

[root@db ~]# redis-cli -a 12345678 --cluster check 192.168.100.154:6300
192.168.100.154:6300 (f0affdb7...) -> 0 keys | 0 slots | 0 slaves.
192.168.100.154:6379 (8f8fd5cd...) -> 2 keys | 5461 slots | 1 slaves.
192.168.100.154:6479 (b4bd9800...) -> 1 keys | 5462 slots | 1 slaves.
192.168.100.154:6389 (12b85989...) -> 2 keys | 5461 slots | 1 slaves.
[OK] 5 keys in 4 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.100.154:6300)
M: f0affdb76af305b0909b21628f57a7d9cf2e7202 192.168.100.154:6300
   slots: (0 slots) master
S: f77a84f86e0d5f6409033b48c0498c9f7a81bda1 192.168.100.154:6489
   slots: (0 slots) slave
   replicates b4bd9800fc2750c51f3b26b906eb07f32874923d
M: 8f8fd5cd518965c80638baf064a4865983f84fa0 192.168.100.154:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: f5e0816814edc79db120827045c5e8b5f6757658 192.168.100.154:6399
   slots: (0 slots) slave
   replicates 12b85989fddcc4bbb40a44b26986d55bfd98e5a6
M: b4bd9800fc2750c51f3b26b906eb07f32874923d 192.168.100.154:6479
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
M: 12b85989fddcc4bbb40a44b26986d55bfd98e5a6 192.168.100.154:6389
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 007663d4755ddea346c0fcffeb62df7b19c12904 192.168.100.154:6499
   slots: (0 slots) slave
   replicates 8f8fd5cd518965c80638baf064a4865983f84fa0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...

• 给新master添加槽位
  redis-cli -a 12345678 --cluster reshard <集群ip:端口>

# redis-cli -a 12345678 --cluster reshard 192.168.100.154:6300

How many slots do you want to move (from 1 to 16384)?4096
（给新master分配多少个？）
What is the receiving node ID?
（填写新master的hash串）
从哪些机器移动： all
             : yes

# redis-cli -a 12345678 --cluster check 192.168.100.154:6300
# 新加入的master槽位号不连续（从其他机器移动过来的）
# 重新分配成本太高了

[root@db ~]# redis-cli -a 12345678 -p 6300
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6300> keys *
1) "k2"

• 为6300主分配从节点6400
  redis-cli -a 12345678 --cluster add-node <新slave ip:port> <新master ip:port> --cluster-slave --cluster-master-id 新masterID

[root@db ~]#redis-cli -a 12345678 --cluster add-node 192.168.100.154:6400 192.168.100.154:6300 --cluster-slave  --cluster-master-id f0affdb76af305b0909b21628f57a7d9cf2e7202
[root@db ~]# redis-cli -a 12345678 -p 6400
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6400> keys *
1) "k2"
127.0.0.1:6400> 
127.0.0.1:6400> info replication
# Replication
role:slave
master_host:192.168.100.154
master_port:6300
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0

五、集群缩容

192.168.100.154:6379 [0-5460] -> 192.168.100.154:6499
192.168.100.154:6479 [5461-10922] -> 192.168.100.154:6489
192.168.100.154:6389 [10923-16383] -> 192.168.100.154:6399
192.168.100.154:6300 [] -> 192.168.100.154:6400

• 清除从节点
  获取从节点的ID

[root@db ~]# redis-cli -a 12345678 --cluster check 192.168.100.154:6300
S: 7504ffc270a16511df291942638b5c836717943a 192.168.100.154:6400
   slots: (0 slots) slave
   replicates f0affdb76af305b0909b21628f57a7d9cf2e7202

6400的ID: 7504ffc270a16511df291942638b5c836717943a

• 删除从节点

[root@db ~]# redis-cli -a 12345678 --cluster del-node 192.168.100.154:6300 7504ffc270a16511df291942638b5c836717943a

>>> Removing node 7504ffc270a16511df291942638b5c836717943a from cluster 192.168.100.154:6300
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.

• 再查看一下（从节点删除成功）

[root@db ~]# redis-cli -a 12345678 --cluster check 192.168.100.154:6300

• 将主节点槽位号清空

[root@db ~]# redis-cli -a 12345678 --cluster reshard 192.168.100.154:6300
How many slots do you want to move (from 1 to 16384)? 4096 
# 想移动多少个？

What is the receiving node ID?  8f8fd5cd518965c80638baf064a4865983f84fa0
# 由谁接收nodeid(192.168.100.154:6379)

Please enter all the source node IDs. 
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
  Source node #1: f0affdb76af305b0909b21628f57a7d9cf2e7202
  Source node #2: done
# slot id来自谁(192.168.100.154:6300)

Do you want to proceed with the proposed reshard plan (yes/no)? yes

6300变成了从节点

[root@db ~]# redis-cli -a 12345678 --cluster check 192.168.100.154:6300
S: f0affdb76af305b0909b21628f57a7d9cf2e7202 192.168.100.154:6300
   slots: (0 slots) slave
   replicates 8f8fd5cd518965c80638baf064a4865983f84fa0

• 删除主节点

[root@db ~]# redis-cli -a 12345678 --cluster del-node 192.168.100.154:6300 f0affdb76af305b0909b21628f57a7d9cf2e7202
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node f0affdb76af305b0909b21628f57a7d9cf2e7202 from cluster 192.168.100.154:6300
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.

[root@db ~]# redis-cli -a 12345678 --cluster check 192.168.100.154:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
192.168.100.154:6379 (8f8fd5cd...) -> 2 keys | 8192 slots | 1 slaves.
192.168.100.154:6479 (b4bd9800...) -> 1 keys | 4096 slots | 1 slaves.
192.168.100.154:6389 (12b85989...) -> 2 keys | 4096 slots | 1 slaves.

六、压力测试工具：redis-benchmark

redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -c 20 -t get --csv
-n 总请求数
-c 并发连接数
-t set,get,lpush…测试命令

[root@db ~]# redis-benchmark -h 127.0.0.1 -p 6389 -n 100000 -c 20 -t get -a 12345678
[root@db ~]# redis-benchmark -h 127.0.0.1 -p 6389 -n 100000 -c 10000 -t get -a 12345678
Could not connect to Redis at 127.0.0.1:6389: Can’t create socket: Too many open files

redis-benchmark : 可以测试redis读写能力，并发。通过调整并发数量观察响应能力

单台测试

[root@db ~]# redis-benchmark -h 127.0.0.1 -p 6389 -n 100000 -c 100 -t get  -a 12345678 
Summary:
  throughput summary: 90661.83 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.583     0.128     0.415     1.231     3.167    21.599

集群测试

[root@db ~]# redis-benchmark --cluster -h 127.0.0.1 -p 6389 -n 100000 -c 100 -t get  -a 12345678
Summary:
  throughput summary: 99502.48 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        0.656     0.056     0.391     1.487     5.119    33.599

为什么用了集群还差？

• Redis单线程的 -> 不需要上文切换
• 集群 6台 -> 6个线程 -> 需要上下文切换

**** 常用的优化系统内核参数 ******

• Linux内核参数优化 => 部署Redis服务，如何进行Liux内核参数优化

• 修改文件打开数 =>
  临时：ulimit -n 65535

• 永久修改：vim打开配置文件后末尾添加

  [root@db ~]# vim /etc/security/limits.conf
  * soft nofile 1024000
  * hard nofile 1024000
  * soft nproc 1024000
  * hard nproc 1024000
  

  *表示所有用户,*号也可以换成具体的用户名或者用户组的名称
  soft 指的是当前系统生效的设置值
  hard 表明系统中所能设定的最大值
  nofile - 打开文件的最大数目
  noproc - 进程的最大数目