6.1 Keepalived+LVS集群介绍
Keepalived和LVS共同构建了一个高效的负载均衡和高可用性解决方案：LVS作为负载均衡器，负责在集群中的多个服务器间分配流量，以其高性能和可扩展性确保应用程序能够处理大量的并发请求；而Keepalived则作为高可用性保证，通过VRRP协议监控LVS服务的状态并在主服务器发生故障时自动进行故障转移，确保服务的持续可用性和无感知切换。这种结合利用了两者的优势，为关键的网络服务提供了一个稳定、可靠且高效的运行环境。
| 机器名称 | IP地址 | 子网掩码 | 说明 |
|---|---|---|---|
| Keepalived-01 | 192.168.110.31 | 255.255.255.0 | 负载均衡+高可用 |
| Keepalived-02 | 192.168.110.32 | 255.255.255.0 | 负载均衡+高可用 |
| RS1 | 192.168.110.33 | 255.255.255.0 | 真实服务器1 |
| RS2 | 192.168.110.34 | 255.255.255.0 | 真实服务器2 |
| Client | 192.168.110.35 | 255.255.255.0 | 客户端 |
6.2 后端RS配置
配置系统服务脚本
[root@LVS-RS1 ~]# vim /etc/init.d/LVS_RS #!/bin/bash # Startup script to handle the initialisation of LVS # chkconfig: - 28 72 # description: Initialise the Linux Virtual Server for DR # Provides: ipvsadm # Required-Start: $local_fs $network $named # Required-Stop: $local_fs $remote_fs $network # Short-Description: Initialise the Linux Virtual Server # Description: The Linux Virtual Server is a highly scalable and highly # available server built on a cluster of real servers, with the load # balancer running on Linux. LOCK=/var/lock/ipvsadm.lock VIP=192.168.110.10 . /etc/rc.d/init.d/functions start() { PID=`ifconfig | grep lo:20 | wc -l` if [ $PID -ne 0 ]; then echo "The LVS-DR-RIP Server is already running !" else /sbin/ifconfig lo:20 $VIP netmask 255.255.255.255 broadcast $VIP up /sbin/route add -host $VIP dev lo:20 echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce /bin/touch $LOCK echo "starting LVS-DR-RIP server is ok !" fi } stop() { /sbin/route del -host $VIP dev lo:20 /sbin/ifconfig lo:20 down >/dev/null echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce rm -rf $LOCK echo "stopping LVS-DR-RIP server is ok !" } status() { if [ -e $LOCK ]; then echo "The LVS-DR-RIP Server is already running !" else echo "The LVS-DR-RIP Server is not running !" fi } case "$1" in start) start ;; stop) stop ;; restart) stop start ;; status) status ;; *) echo "Usage: $1 {start|stop|restart|status}" exit 1 esac exit 0 [root@LVS-RS1 ~]# chmod +x /etc/init.d/LVS_RS [root@LVS-RS1 ~]# chkconfig --add LVS_RS [root@LVS-RS1 ~]# systemctl start LVS_RS.service [root@LVS-RS1 ~]# scp /etc/init.d/LVS_RS 192.168.110.34:/etc/init.d/ #发给另一台RS [root@LVS-RS2 ~]# chmod +x /etc/init.d/LVS_RS [root@LVS-RS2 ~]# chkconfig --add LVS_RS [root@LVS-RS2 ~]# systemctl start LVS_RS.service
6.3 Keepalived配置
6.3.1 基于TCP的健康检测
6.3.1.1 Keepalived-01配置
[root@Keepalived-01 ~]# cp /etc/keepalived/keepalived.conf{,.bak} [root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node1 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.33 80 { weight 1 TCP_CHECK { connect_timeout 3 retry 3 delay_before_retry 3 connect_port 80 } } real_server 192.168.110.34 80 { weight 1 TCP_CHECK { connect_timeout 3 retry 3 delay_before_retry 3 connect_port 80 } } } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.1.2 Keepalived-02配置
[root@Keepalived-02 ~]# cp /etc/keepalived/keepalived.conf{,.bak} [root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node2 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.33 80 { weight 1 TCP_CHECK { connect_timeout 3 retry 3 delay_before_retry 3 connect_port 80 } } real_server 192.168.110.34 80 { weight 1 TCP_CHECK { connect_timeout 3 retry 3 delay_before_retry 3 connect_port 80 } } } [root@Keepalived-02 ~]# systemctl start keepalived.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.1.3 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
6.3.1.4 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.1.5 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.34:80 Route 1 0 9 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.2 基于HTTP的健康检测
6.3.2.1 生成远程网页的MD5哈希值
genhash 是一个命令行工具，用于生成远程网页的MD5哈希值。它可以用于监控HTTP和HTTPS服务，特别是在Keepalived配置中，用于健康检查。genhash 可以通过HTTP或HTTPS连接到网页，并生成页面数据的MD5哈希值，该哈希值可以在Keepalived配置文件中使用。
[root@Keepalived-01 ~]# genhash -s 192.168.110.33 -p 80 -u /index.html MD5SUM = fd0508d1ccc6c66c14977e54ffc7faef [root@Keepalived-01 ~]# genhash -s 192.168.110.34 -p 80 -u /index.html MD5SUM = 0632aaa5fb77608b1a4736d47aacb62c [root@Keepalived-02 ~]# genhash -s 192.168.110.33 -p 80 -u /index.html MD5SUM = fd0508d1ccc6c66c14977e54ffc7faef [root@Keepalived-02 ~]# genhash -s 192.168.110.34 -p 80 -u /index.html MD5SUM = 0632aaa5fb77608b1a4736d47aacb62c
6.3.2.2 Keepalived-01配置
[root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node1 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.33 80 { weight 1 HTTP_GET { url { path /index.html digest fd0508d1ccc6c66c14977e54ffc7faef } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.110.34 80 { weight 1 HTTP_GET { url { path /index.html digest 0632aaa5fb77608b1a4736d47aacb62c } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.2.3 Keepalived-02配置
[root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node2 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.34 80 { weight 1 HTTP_GET { url { path /index.html digest fd0508d1ccc6c66c14977e54ffc7faef } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } real_server 192.168.110.34 80 { weight 1 HTTP_GET { url { path /index.html digest 0632aaa5fb77608b1a4736d47aacb62c } connect_timeout 3 nb_get_retry 3 delay_before_retry 3 } } } [root@Keepalived-02 ~]# systemctl start keepalived.service [root@Keepalived-02 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.2.4 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.2.5 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.2.6 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.34:80 Route 1 0 6 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.3 Keepalived健康检查（MISC方式）
6.3.3.1 编写健康脚本
[root@Keepalived-01 ~]# vim /etc/keepalived/chk_web.sh #!/bin/bash # 检查传入的参数数量是否正确 if [ $# -ne 2 ]; then echo "Error, Parameter error" exit 1 else # 使用 nmap 进行端口扫描，获取端口状态 n=$(nmap -Pn -p$2 -sS -vv $1 | grep "^$2" | awk '{print $2}') # 判断端口是否开放 if [ "$n" = "open" ]; then exit 0 else exit 1 fi fi [root@Keepalived-01 ~]# chmod +x /etc/keepalived/chk_web.sh [root@Keepalived-01 ~]# scp /etc/keepalived/chk_web.sh 192.168.110.32:/etc/keepalived/
6.3.3.2 keepalived-01配置
[root@Keepalived-01 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node1 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.33 80 { weight 1 MISC_CHECK { misc_path "/etc/keepalived/chk_web.sh 192.168.110.33 80" misc_timeout 3 } } real_server 192.168.110.34 80 { weight 1 MISC_CHECK { misc_path "/etc/keepalived/chk_web.sh 192.168.110.34 80" misc_timeout 3 } } } [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.33:80 Route 1 0 0 -> 192.168.110.34:80 Route 1 0 0
6.3.3.3 keepalived-02配置
[root@Keepalived-02 ~]# vim /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_node2 } vrrp_instance LVS { state MASTER interface ens160 virtual_router_id 51 priority 80 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 192.168.110.10 } } virtual_server 192.168.110.10 80 { delay_loop 6 lb_algo rr lb_kind DR protocol TCP real_server 192.168.110.33 80 { weight 1 MISC_CHECK { misc_path "/etc/keepalived/chk_web.sh 192.168.110.33 80" misc_timeout 3 } } real_server 192.168.110.34 80 { weight 1 MISC_CHECK { misc_path "/etc/keepalived/chk_web.sh 192.168.110.34 80" misc_timeout 3 } } } [root@Keepalived-02 ~]# systemctl start keepalived.service
6.3.3.4 访问测试
[root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
This is LVS test IP=192.168.110.33 Host=LVS-RS1
This is LVS test IP=192.168.110.34 Host=LVS-RS2
6.3.3.5 模拟故障转移
[root@Keepalived-01 ~]# systemctl stop keepalived.service [root@Keepalived-02 ~]# ip address show ens160 #VIP漂移到Keepalived-02 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:b0:0d:30 brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.32/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:feb0:d30/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done #访问正常 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.33 Host=LVS-RS1 This is LVS test IP=192.168.110.34 Host=LVS-RS2 [root@Keepalived-01 ~]# systemctl start keepalived.service [root@Keepalived-01 ~]# ip address show ens160 #恢复后回来 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:0c:29:d1:a9:eb brd ff:ff:ff:ff:ff:ff altname enp3s0 inet 192.168.110.31/24 brd 192.168.110.255 scope global noprefixroute ens160 valid_lft forever preferred_lft forever inet 192.168.110.10/32 scope global ens160 valid_lft forever preferred_lft forever inet6 fe80::20c:29ff:fed1:a9eb/64 scope link noprefixroute valid_lft forever preferred_lft forever
6.3.3.6 后端服务器故障
[root@LVS-RS1 ~]# systemctl stop nginx.service [root@Keepalived-01 ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.110.10:80 rr -> 192.168.110.34:80 Route 1 0 6 [root@Client ~]# for ((i=1;i<=6;i++)); do curl http://192.168.110.10; done This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2 This is LVS test IP=192.168.110.34 Host=LVS-RS2