淘先锋技术网

首页 1 2 3 4 5 6 7

准备工作

本例中,使用keepalive+nginx实现高可用和负载均衡。

资源要求

服务器

至少2台,如下:

VIPIP主机名nginx端口
10.120.26.11110.120.26.90status80
10.120.26.11110.120.26.91LB280

IP地址资源

除了服务器IP外,还需要额外提供至少1个“虚拟IP”,作为接入IP。

本例中,如上表10.120.26.111。

其他

提前在2台服务器上部署好nginx(此处不演示)。

环境准备

下载官方安装包

如:

wget http://www.keepalived.org/software/keepalived-1.3.9.tar.gz

安装keepalive

本例使用root用户进行操作。

先执行如下指令,进行环境配置,并编译安装:

# 解压
tar xvf keepalived-1.3.9.tar.gz

cd keepalived-1.3.9

# 初始化环境
./configure --prefix=/home/keepalived

# 编译,并将可执行程序安装到/home/keepalived下
make && make install

--prefix,指定安装的路径,即可。

在执行configure过程中,可能出现如下错误:

1、缺少libnl等包,报错如下:

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

此时需要yum安装,如下:

# yum -y install libnl libnl-devel 

2、缺少libnfnetlink包,报错:

configure: error: libnfnetlink headers missing

执行yum安装:

# yum install -y libnfnetlink-devel

防火墙添加vrrp组播规则

执行如下指令:

firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens160 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload

注意:

keepalive是基于vrrp协议工作的,因此必须配置组播规则。

关闭selinux

vi /etc/sysconfig/selinux
#修改:
SELINUX=disabled
#setenforce 0

说明:

开启selinux时,系统的安全性要求较高,可能导致keepvlive失效。

设置开启启动

拷贝文件:

cp /home/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /home/keepalived/sbin/keepalived /etc/init.d/
cp /home/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

说明:缺少的目录需要自行创建。

配置

keepvlive分为2种工作模式:抢占模式和非抢占模式,通过配置文件中的配置项state区分。

抢占模式

当Master宕机、恢复后,会抢夺Master角色,将自己升级为Master。

需要在配置文件中指定抢占模式,编辑/etc/keepalived/keepalived.conf配置文件:

1、MASTER配置(10.120.26.90)

global_defs {
   router_id status  #标识本节点的名称,通常为hostname
}

## keepalived会定时执行脚本并对脚本执行的结果进行分析,动态调整vrrp_instance的优先级。
##如果脚本执行结果为0,并且weight配置的值大于0,则优先级相应的增加。如果脚本执行结果非0,
##并且weight配置的值小于 0,则优先级相应的减少。其他情况,维持原本配置的优先级,即配置文件中priority对应的值。
vrrp_script chk_nginx {
       script "/etc/keepalived/nginx_check.sh"
       interval 2  #每2秒检测一次nginx的运行状态
       weight -20  #失败一次,将自己的优先级-20
}

vrrp_instance VI_1 {
    state MASTER                  # 状态,主节点为MASTER,备份节点为BACKUP
    interface ens160              # 绑定VIP的网络接口,通过ifconfig查看自己的网络接口
    virtual_router_id 51          # 虚拟路由的ID号,两个节点设置必须一样,可选IP最后一段使用,相同的VRID为一个组,他将决定多播的MAC地址
    mcast_src_ip 10.120.26.90    # 本机IP地址
    priority 100                  # 节点优先级,值范围0~254,MASTER要比BACKUP高
    advert_int 1                  # 组播信息发送时间间隔,两个节点必须设置一样,默认为1秒
    # 设置验证信息,两个节点必须一致
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP,两个节点设置必须一样。可以设置多个,一行写一个
    virtual_ipaddress {
        10.120.26.111
    }

    track_script {
       chk_nginx  # nginx存活状态检测脚本,对应vrrp_script配置块
    }
}

2、 BACKUP配置(10.120.26.91)

global_defs {
   router_id LB2          # 差异点1:BACKUP的主机名
}

vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_check.sh"
    interval 2
    weight -20
}

vrrp_instance VI_1 {
    state BACKUP              # 差异点2:路由器状态为BACKUP
    interface ens160          # 根据本机的网卡名称填写
    virtual_router_id 51      
    mcast_src_ip 10.120.26.91 # 差异点3:本机ip
    priority 90               # 差异点4:要小于Master的优先级
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.120.26.111
    }

    track_script {
       chk_nginx
    }
}

非抢占模式

虚拟路由器组中的master从故障中恢复后,不会抢占备份节点的vip。

1、MASTER配置(10.120.26.90)

global_defs {

   router_id LB1

}

vrrp_script chk_nginx {

       script "/etc/keepalived/nginx_check.sh"

       interval 2

       weight -20

}

vrrp_instance VI_1 {

    #state MASTER

    state BACKUP    # 注意这里的不同!

    interface ens160

    virtual_router_id 51

    mcast_src_ip 10.120.26.90

    priority 100       # 非抢占模式下通过priority判断是否为Master

    advert_int 1

    nopreempt

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        10.120.26.111

    }

    track_script {

       chk_nginx

    }

}

2、BACKUP(10.120.26.91)

除了优先级不一样,其余配置与Master一致。

nginx服务检测脚本

分别在主备服务器/etc/keepalived 目录下创建nginx_check.sh脚本,并为其添加执行权限:

/etc/keepalived

touch nginx_check.sh

chmod +x /etc/keepalived/nginx_check.sh

nginx_check.sh用于keepalived定时检测nginx的服务状态,如果nginx停止了,会尝试重新启动nginx,如果启动失败,会将keepalived进程杀死,将vip漂移到备份机器上。脚本内容如下:

#!/bin/bash
A=`ps -C nginx --no-header | wc -l`
if [ $A -eq 0 ];then
    #尝试重新启动nginx
    /opt/webapps/nginx/nginx_lr/sbin/nginx
    sleep 2  #睡眠2秒
    if [ `ps -C nginx --no-header | wc -l` -eq 0 ];then
         systemctl stop keepalived.service #启动失败,将keepalived服务杀死。将vip漂移到其它备份节点
    fi
fi

启动和验证

keepalive服务启动

systemctl start keepalived.service

查看进程状态:

[root@status etc]# ps -ef|grep keepalived
root     127479      1  0 17:58 ?        00:00:00 /home/keepalived/sbin/keepalived -D
root     127480 127479  0 17:58 ?        00:00:00 /home/keepalived/sbin/keepalived -D
root     127481 127479  0 17:58 ?        00:00:00 /home/keepalived/sbin/keepalived -D
root     127561 117202  0 17:58 pts/3    00:00:00 grep --color=auto keepalived

没问题后,分别查看两个节点的ip地址。Master节点:

[root@status etc]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:50:56:93:18:63 brd ff:ff:ff:ff:ff:ff

    inet 10.120.26.90/24 brd 10.120.26.255 scope global ens160

       valid_lft forever preferred_lft forever

    inet 10.120.26.111/32 scope global ens160

       valid_lft forever preferred_lft forever

    inet6 fe80::250:56ff:fe93:1863/64 scope link 

       valid_lft forever preferred_lft forever

可以看到,10.120.26.90拿到了VIP 10.120.26.111。

BACKUP的ip情况:

[root@LB2 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:50:56:93:6e:eb brd ff:ff:ff:ff:ff:ff

    inet 10.120.26.91/24 brd 10.120.26.255 scope global ens160

       valid_lft forever preferred_lft forever

    inet6 fe80::250:56ff:fe93:6eeb/64 scope link 

       valid_lft forever preferred_lft forever

BACKUP上只有10.120.26.91这个ip,正常。

验证

VIP漂移

我们先关闭Master,验证VIP是否会漂移到BACKUP上。

关闭  10.120.26.90 的 keepalived:

systemctl stop keepalived.service

查看 10.120.26.91 的ip状态:

[root@LB2 ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

       valid_lft forever preferred_lft forever

    inet6 ::1/128 scope host 

       valid_lft forever preferred_lft forever

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:50:56:93:6e:eb brd ff:ff:ff:ff:ff:ff

    inet 10.120.26.91/24 brd 10.120.26.255 scope global ens160

       valid_lft forever preferred_lft forever

    inet 10.120.26.111/32 scope global ens160

       valid_lft forever preferred_lft forever

    inet6 fe80::250:56ff:fe93:6eeb/64 scope link 

       valid_lft forever preferred_lft forever

10.120.26.91 的  keepalived日志如下:

Dec 12 20:23:07 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Transition to MASTER STATE

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Entering MASTER STATE

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) setting protocol VIPs.

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens160 for 10.120.26.111

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:08 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

Dec 12 20:23:13 LB2 Keepalived_vrrp[75622]: Sending gratuitous ARP on ens160 for 10.120.26.111

看到“Transition to MASTER STATE”和“Entering MASTER STATE”,说明切换成功。

抢占模式下,VIP恢复

在抢占模式下,如果10.120.26.90(配置项 state=Master)恢复了,那么VIP就会漂移回来。

重启 10.120.26.90 的 keepalived之后,可以看到VIP又回来了:

ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000

    link/ether 00:50:56:93:18:63 brd ff:ff:ff:ff:ff:ff

    inet 10.120.26.90/24 brd 10.120.26.255 scope global ens160

       valid_lft forever preferred_lft forever

    inet 10.120.26.111/32 scope global ens160

       valid_lft forever preferred_lft forever

    inet6 fe80::250:56ff:fe93:1863/64 scope link 

       valid_lft forever preferred_lft forever       

这是,10.120.26.91 的  keepalived日志如下:

Dec 12 20:27:31 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90

Dec 12 20:27:31 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) Entering BACKUP STATE

Dec 12 20:27:31 LB2 Keepalived_vrrp[75622]: VRRP_Instance(VI_1) removing protocol VIPs.

到此,说明我们keepalive配置的没问题了。在Master路由器出现问题时,可以切换到BACKUP。

常见问题

1、主机访问不到nginx的80端口

报错:no route to host....

查看防火墙状态:

[root@LB1 test]# firewall-cmd --zone=public --list-ports

80/tcp

有这一行,表示80端口开放。没有的话需要添加规则:

firewall-cmd --zone=public --add-port=80/tcp --permanent

firewall-cmd --reload

2、抢占模式配置好后,两台机器都变成主机,都占有虚拟ip地址

这种情况可能是防火墙的组播配置没配置好,重新配置一下组播策略。