Codis集群安装部署教程

简介

Codis是由豌豆荚团队推出的高性能分布式Redis集群解决方案，它通过透明的代理层（codis-proxy）将数据分片到多台Redis实例，使客户端无需修改即可像操作单机Redis一样使用集群。支持动态扩缩容与自动化数据迁移，配合可视化运维工具，可无缝扩展集群容量并保障服务高可用，已广泛应用于大规模生产环境。

本文介绍Codis的安装部署。

环境信息

服务器关闭防火墙或者手动开放涉及的端口。

#关闭防火墙
systemctl stop firewalld
#设置开机禁用防火墙
systemctl disable firewalld.service

# 手动开放端口
firewall-cmd --add-port=2181/tcp --permanent
firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --add-port=18080/tcp --permanent
firewall-cmd --add-port=11080/tcp --permanent
firewall-cmd --add-port=19000/tcp --permanent
firewall-cmd --add-port=6379/tcp --permanent
firewall-cmd --reload

前置依赖

安装zookeeper-3.4.11，参见 CentOS 7中安装ZooKeeper

go只在编译源码时要用到，本文使用的是已经编译好的包，所以不需要安装go。

下载安装包

可以下载源码自己编译，但是问题较多，这里推荐直接下载编译好的包。

下载地址：https://github.com/CodisLabs/codis/releases/download/3.2.2/codis3.2.2-go1.8.5-linux.tar.gz

得到安装包：codis3.2.2-go1.8.5-linux.tar.gz

启动及配置

解压安装包

tar -xzvf codis3.2.2-go1.8.5-linux.tar.gz
mv codis3.2.2-go1.8.5-linux /usr/local/codis3.2.2-go1.8.5-linux
cd /usr/local/codis3.2.2-go1.8.5-linux/

启动 Codis Dashboard

配置文件 dashboard.toml，我这里的修改点是coordinator_name、coordinator_addr，其他配置按自己实际情况来。

##################################################
#                                                #
#                  Codis-Dashboard               #
#                                                #
##################################################

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# for zookeeper/etcd, coorinator_auth accept "user:password"
# Quick Start
# coordinator_name = "filesystem"
# coordinator_addr = "/tmp/codis"
coordinator_name = "zookeeper"
coordinator_addr = "127.0.0.1:2181"
#coordinator_auth = ""

# Set Codis Product Name/Auth.
product_name = "codis-demo"
product_auth = ""

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"

# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"

# Set configs for redis sentinel.
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""

启动命令

./codis-dashboard --ncpu=4 --config=dashboard.toml --log=dashboard.log --log-level=WARN

启动后可以通过 http://192.168.64.129:18080/ 查看，如果有返回数据说明成功

启动 Codis Proxy

配置文件 proxy.toml，我这里的修改点是jodis_name、jodis_addr，其他配置按自己实际情况来。

##################################################
#                                                #
#                  Codis-Proxy                   #
#                                                #
##################################################

# Set Codis Product Name/Auth.
product_name = "codis-demo"
product_auth = ""

# Set auth for client session
#   1. product_auth is used for auth validation among codis-dashboard,
#      codis-proxy and codis-server.
#   2. session_auth is different from product_auth, it requires clients
#      to issue AUTH <PASSWORD> before processing any other commands.
session_auth = ""

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080"

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4"
proxy_addr = "0.0.0.0:19000"

# Set jodis address & session timeout
#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
#   2. jodis_addr is short for jodis_coordinator_addr
#   3. jodis_auth is short for jodis_coordinator_auth, for zookeeper/etcd, "user:password" is accepted.
#   4. proxy will be registered as node:
#        if jodis_compatible = true (not suggested):
#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
#        or else
#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
jodis_name = "zookeeper"
jodis_addr = "127.0.0.1:2181"
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 1000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 16

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

启动命令

./codis-proxy --ncpu=4 --config=proxy.toml --log=proxy.log --log-level=WARN

启动后可以通过 http://192.168.64.129:11080/ 查看，如果有返回数据说明成功

启动 Codis Server

配置文件 redis.conf（里面一大堆注释被我删了），我这里的修改点是bind、protected-mode no，其他配置按自己实际情况来。

protected-mode no

port 6379

tcp-backlog 511

timeout 0

tcp-keepalive 300

daemonize yes

supervised no

pidfile /tmp/redis_6379.pid

loglevel notice

logfile "/tmp/redis_6379.log"

databases 16

save 900 1
save 300 10
save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir ./

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

repl-disable-tcp-nodelay no

slave-priority 100

appendonly no

appendfilename "appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512
hash-max-ziplist-value 64

list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512

zset-max-ziplist-entries 128
zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

启动命令

./codis-server redis.conf

启动 Codis FE（可选）

启动命令

./codis-fe --ncpu=4 --log=fe.log --log-level=WARN --zookeeper=127.0.0.1:2181 --listen=0.0.0.0:8080

访问管理页面 http://192.168.64.129:8080/

Codis-FE 配置

创建Group，在 New Group 后面输入1，然后点击 New Group；在 New Group 后面输入2，然后点击 New Group

把 Codis Server 加入 Codis 集群，在 Add Server 后面输入 Codis Server 地址（192.168.64.137:6379）和 Group（1）后点占击 Add Server ；在 Add Server 后面输入 Codis Server 地址（192.168.64.138:6379）和 Group（2）后点占击 Add Server

分配槽位，可以手动分配

也可以自动分配 ，点击 Rebalance All Slots 然后确定

添加 Codis Proxy ，填入地址 192.168.64.129:11080

测试 Codis 服务

测试时直接使用 Redis Cli 连接 Codis Proxy（默认端口19000）

redis-cli -p 19000

也可以使用其他客户端工具测试。

Nginx代理多Proxy

Nginx安装时注意需开启stream模块。

#  以下只列出和常规安装有区别的命令，--with-stream 通过该参数开启stream模块
./configure --prefix=/usr/local/nginx --with-stream

添加以下配置，和http同级别

stream {
    # 定义后端Codis Proxy集群
    upstream codis_cluster {
        # 默认轮询负载均衡
        server 192.168.64.129:19000 weight=2;  # 实例1，权重更高
        server 192.168.64.137:19000;           # 实例2
        server 192.168.64.137:19000 backup;    # 备份节点
    }

    # 代理服务器配置
    server {
        listen 19001;                  # 监听端口
        proxy_pass codis_cluster;       # 转发到上游
        proxy_timeout 3s;               # 超时设置
        proxy_connect_timeout 1s;
        
        # 可选：故障转移策略
        proxy_next_upstream on;
        proxy_next_upstream_timeout 0;
        proxy_next_upstream_tries 2;
    }
}

重启nginx

/usr/local/nginx/sbin/nginx -s reload

开放端口

firewall-cmd --add-port=19001/tcp --permanent
firewall-cmd --reload

客户端通过 192.168.64.129:19001 连接codis集群，和连接单节点redis一样。

参考文档

https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md