码迷,mamicode.com
首页 > 其他好文 > 详细

lvs+keepalived+nginx实现高性能负载均衡集群

时间:2019-03-18 13:54:52      阅读:175      评论:0      收藏:0      [点我收藏+]

标签:and   rda   ade   dev   net   添加   read   data   rev   

lvs+keepalived+nginx实现高性能负载均衡集群

一、为什么要使用负载均衡技术?

1、系统高可用性

2??系统可扩展性

3??负载均衡能力

LVS+keepalived能很好的实现以上的要求,LVS提供负载均衡,keepalived提供健康检查,故障转移,提高系统的可用性!采用这样的架构以后很容易对现有系统进行扩展,只要在后端添加或者减少realserver,只要更改lvs的配置文件,并能实现无缝配置变更!

二、LVS+Keepalived介绍

1??LVS

LVS是一个开源的软件,可以实现LINUX平台下的简单负载均衡。LVSLinux Virtual Server的缩写,意思是Linux虚拟服务器。目前有三种IP负载均衡技术(VS/NATVS/TUNVS/DR);八种调度算法(rr,wrr,lc,wlc,lblc,lblcr,dh,sh)。

2??keepalived

Keepalived?是运行在lvs?之上,它的主要功能是实现真实机的故障隔离及负载均衡器间的失败切换,提高系统的可用性

三、环境:

四台服务器,系统全为CentOS6.8

192.168.2.203 master lvs+keepalived

192.168.2.202 backup lvs+keepalived

192.168.2.204 web1nginx

192.168.2.205 web2 nginx

vip192.168.2.13

其中nginx已预装好,这里不再写搭建过程

? ?

四、搭建并配置

1、分别在backup lvsmaster lvs上安装lvs

技术图片

1 root@bogon src]# yum -y install ipvsadm

2 已加载插件:fastestmirror

3 设置安装进程

4 Determining fastest mirrors

5 epel/metalink | 5.4 kB 00:00

6 * base: mirror.lzu.edu.cn

7 ... ...

8 已安装:

9 ipvsadm.x86_64 0:1.26-4.el6

10

11 作为依赖被安装:

12 libnl.x86_64 0:1.1.4-2.el6

13

14 完毕!

技术图片

2、把ipvsadm模块加载进系统

技术图片

1 [root@bogon src]# ipvsadm

2 IP Virtual Server version 1.2.1 (size=4096)

3 Prot LocalAddress:Port Scheduler Flags

4 -> RemoteAddress:Port Forward Weight ActiveConn InActConn

5 [root@bogon src]# lsmod | grep ip_vs

6 ip_vs 126897 0

7 libcrc32c 1246 1 ip_vs

8 ipv6 336282 270 ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

技术图片

3、分别在backup lvsmaster lvs上安装keepalivedkeepalived官网:http://www.keepalived.org/

技术图片

[root@bogon src]# tar zxf keepalived-1.2.24.tar.gz

[root@bogon src]# cd keepalived-1.2.24

[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/

报错:

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

技术图片

系统缺少openssl-devel包所致

安装openssl-devel

root@bogon keepalived-1.2.24]# yum -y install openssl-devel

再次编辑安装

技术图片

[root@bogon keepalived-1.2.24]# ./configure --sysconf=/etc --with-kernel-dir=/lib/modules/2.6.32-642.3.1.el6.x86_64/

Keepalived configuration

------------------------

Keepalived version : 1.2.24

Compiler : gcc

Preprocessor flags : -I/lib/modules/2.6.32-642.3.1.el6.x86_64//include

Compiler flags : -Wall -Wunused -Wstrict-prototypes

Linker flags :

Extra Lib : -ldl -lssl -lcrypto

Use IPVS Framework : Yes

IPVS use libnl : No

IPVS syncd attributes : No

IPVS 64 bit stats : No

fwmark socket support : Yes

Use VRRP Framework : Yes

Use VRRP VMAC : Yes

Use VRRP authentication : Yes

With ip rules/routes : Yes

SNMP keepalived support : No

SNMP checker support : No

SNMP RFCv2 support : No

SNMP RFCv3 support : No

SHA1 support : No

Use Debug flags : No

Stacktrace support : No

Memory alloc check : No

libnl version : None

Use IPv4 devconf : No

Use libiptc : No

Use libipset : No

Build genhash : Yes

Build documentation : No

?

[root@bogon keepalived-1.2.24]# make && make install

技术图片

[root@bogon keepalived-1.2.24]# ln -s /usr/local/sbin/keepalived /sbin/

[root@bogon keepalived-1.2.24]# chkconfig --add keepalived

[root@bogon keepalived-1.2.24]# chkconfig --level 35 keepalived on

4、配置keepalived

lvs-master的配置文件如下

技术图片

[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

?

global_defs {      #全局配置部分

# notification_email {      #email?通知,基本不用此处所以注释掉

# acassen@firewall.loc

# failover@firewall.loc

# sysadmin@firewall.loc

# }

# notification_email_from Alexandre.Cassen@firewall.loc

# smtp_server 192.168.200.1

# smtp_connect_timeout 30

router_id LVS_DEVEL    #?设置lvs的id,在一个网络内应该是唯一的

vrrp_skip_check_adv_addr  

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

?

vrrp_instance VI_1 {    #vrrp实例定义部分

state MASTER    ?#设置lvs的状态,报错MASTER和BACKUP两种,必须大写

interface eth1    #设置对外服务的接口

virtual_router_id 60  #设置虚拟路由标示,这个标示是一个数字,同一个vrrp实例使用唯一标示

priority 100    #定义优先级,数字越大优先级越高,在一个vrrp——instance下,master的优先级必须大于backup

advert_int 1    #设定master与backup负载均衡器之间同步检查的时间间隔,单位是秒

authentication {  #设置验证类型和密码

auth_type PASS  #主要有PASS和AH两种

auth_pass 1111  #验证密码,同一个vrrp_instance下MASTER和BACKUP密码必须相同

}

virtual_ipaddress {  #设置虚拟ip地址,可以设置多个,每行一个

192.168.2.13

}

}

?

virtual_server 192.168.2.13 80 {  #设置虚拟服务器,需要指定虚拟ip和服务端口

delay_loop 3    #健康检查时间间隔

lb_algo rr    ?#负载均衡调度算法

lb_kind DR    #负载均衡转发规则

persistence_timeout 50    #设置会话保持时间,对动态网页非常有用

protocol TCP    #指定转发协议类型,有TCP和UDP两种

?

real_server 192.168.2.204 80 {  #配置服务器节点1,需要指定real server的真实IP地址和端口

weight 1    #设置权重,数字越大权重越高

TCP_CHECK {     #realserver的状态监测设置部分单位秒

connect_timeout 3    #超时时间

nb_get_retry 3      #重试次数

delay_before_retry 3    #重试间隔

connect_port 80   #监测端口

}

}

real_server 192.168.2.205 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

?

}

技术图片

? ?

LVS-backup的配置文件如下

技术图片

[root@bogon keepalived-1.2.24]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

?

global_defs {

# notification_email {

# acassen@firewall.loc

# failover@firewall.loc

# sysadmin@firewall.loc

# }

# notification_email_from Alexandre.Cassen@firewall.loc

# smtp_server 192.168.200.1

smtp_connect_timeout 30

router_id LVS_DEVEL

vrrp_skip_check_adv_addr

vrrp_strict

vrrp_garp_interval 0

vrrp_gna_interval 0

}

?

vrrp_instance VI_1 {

state BACKUP

interface eth1

virtual_router_id 60

priority 80

advert_int 1

authentication {

auth_type PASS

auth_pass 1111

}

virtual_ipaddress {

192.168.2.13

}

}

?

virtual_server 192.168.2.13 80 {

delay_loop 3

lb_algo rr

lb_kind DR

persistence_timeout 3

protocol TCP

?

real_server 192.168.2.204 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

real_server 192.168.2.205 80 {

weight 1

TCP_CHECK {

connect_timeout 3

nb_get_retry 3

delay_before_retry 3

connect_port 80

}

}

?

}

技术图片

? ?

5realserver的配置

两台web服务器都要执行下面脚本

技术图片

[root@bogon www]# cat /etc/rc.d/init.d/realserver.sh

#!/bin/bash

# description: Config realserver lo and apply noarp

?

SNS_VIP=192.168.2.13

?

/etc/rc.d/init.d/functions

?

case "$1" in

start)

ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP

/sbin/route add -host $SNS_VIP dev lo:0

echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p >/dev/null 2>&1

echo "RealServer Start OK"

?

;;

stop)

ifconfig lo:0 down

route del $SNS_VIP >/dev/null 2>&1

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce

echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore

echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce

echo "RealServer Stoped"

;;

*)

echo "Usage: $0 {start|stop}"

exit 1

esac

?

exit 0

技术图片

技术图片

[root@bogon www]# /etc/rc.d/init.d/realserver.sh start

/etc/rc.d/init.d/realserver.sh: line 6: /etc/rc.d/init.d/functions: 权限不够

RealServer Start OK

[root@bogon www]# ifconfig

eth0 Link encap:Ethernet HWaddr 00:0C:29:41:71:DF

inet addr:192.168.12.129 Bcast:192.168.12.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe41:71df/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:728 errors:0 dropped:0 overruns:0 frame:0

TX packets:98 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:137311 (134.0 KiB) TX bytes:7369 (7.1 KiB)

?

eth1 Link encap:Ethernet HWaddr 00:0C:29:41:71:E9

inet addr:192.168.2.204 Bcast:192.168.2.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fe41:71e9/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:119838 errors:0 dropped:0 overruns:0 frame:0

TX packets:31612 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:23411786 (22.3 MiB) TX bytes:2119106 (2.0 MiB)

?

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:2 errors:0 dropped:0 overruns:0 frame:0

TX packets:2 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:182 (182.0 b) TX bytes:182 (182.0 b)

?

lo:0 Link encap:Local Loopback

inet addr:192.168.2.13 Mask:255.255.255.255

UP LOOPBACK RUNNING MTU:65536 Metric:1

技术图片

6、启动keepalived并进行测试

[root@bogon keepalived-1.2.24]# service keepalived start

正在启动 keepalived: [确定]

lvs-master

技术图片

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:46 bogon Keepalived_healthcheckers[6596]: Netlink reflector reports IP 192.168.2.13 added

Oct 21 01:19:46 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

技术图片

技术图片

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 0

-> 192.168.2.205:80 Route 1 0 0

技术图片

访问curl http://192.168.2.13/test.txt

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

关掉web2再次测试

[root@www etc]# curl http://192.168.2.13/test.txt

it is web1

[root@www etc]# curl http://192.168.2.13/test.txt

it is web1

查看lvs-master

技术图片

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 2

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

技术图片

已经自动把web2剔除

打开web2再次查看

技术图片

[root@bogon keepalived-1.2.24]# ipvsadm -L -n

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

TCP 192.168.2.13:80 rr persistent 3

-> 192.168.2.204:80 Route 1 0 0

-> 192.168.2.205:80 Route 1 0 0

[root@bogon keepalived-1.2.24]# tail -f /var/log/messages

Oct 21 01:19:51 bogon Keepalived_vrrp[6597]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 21 01:28:58 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 failed.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Check on service [192.168.2.205]:80 failed after 1 retry.

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Removing service [192.168.2.205]:80 from VS [192.168.2.13]:80

Oct 21 01:29:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

Oct 21 01:29:31 bogon Keepalived_healthcheckers[6596]: Timeout reading data to remote SMTP server [192.168.200.1]:25.

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: TCP connection to [192.168.2.205]:80 success.

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Adding service [192.168.2.205]:80 to VS [192.168.2.13]:80

Oct 21 01:31:01 bogon Keepalived_healthcheckers[6596]: Remote SMTP server [192.168.200.1]:25 connected.

技术图片

恢复后已自动添加进来

关掉lvs masterkeepalived

[root@bogon keepalived-1.2.24]# service keepalived stop

停止 keepalived: [确定]

访问web并查看lvs backup

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

[root@www etc]# curl http://192.168.2.13/test.txt

it is web2

技术图片

[root@lys2 src]# tail -f /var/log/messages

Oct 23 19:03:26 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Transition to MASTER STATE

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Entering MASTER STATE

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) setting protocol VIPs.

Oct 23 19:03:27 lys2 Keepalived_healthcheckers[13123]: Netlink reflector reports IP 192.168.2.13 added

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:27 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

Oct 23 19:03:32 lys2 Keepalived_vrrp[13124]: Sending gratuitous ARP on eth1 for 192.168.2.13

技术图片

技术图片

[root@lys2 src]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:89:0f:e3 brd ff:ff:ff:ff:ff:ff

inet 192.168.144.101/24 brd 192.168.144.255 scope global eth0

inet6 fe80::20c:29ff:fe89:fe3/64 scope link

valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:89:0f:ed brd ff:ff:ff:ff:ff:ff

inet 192.168.2.202/24 brd 192.168.2.255 scope global eth1

inet 192.168.2.13/32 scope global eth1

inet6 fe80::20c:29ff:fe89:fed/64 scope link

valid_lft forever preferred_lft forever

技术图片

? ?

可以看到lvs backup已自动切换成master状态并自动绑定了vip

查看lvs master vip

技术图片

[root@bogon keepalived-1.2.24]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:55:4d:7a brd ff:ff:ff:ff:ff:ff

inet 192.168.12.128/24 brd 192.168.12.255 scope global eth0

inet6 fe80::20c:29ff:fe55:4d7a/64 scope link

valid_lft forever preferred_lft forever

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

link/ether 00:0c:29:55:4d:84 brd ff:ff:ff:ff:ff:ff

inet 192.168.2.203/24 brd 192.168.2.255 scope global eth1

inet6 fe80::20c:29ff:fe55:4d84/64 scope link

valid_lft forever preferred_lft forever

技术图片

已自动解除vip

? ?

到处全部结束

lvs+keepalived+nginx实现高性能负载均衡集群

标签:and   rda   ade   dev   net   添加   read   data   rev   

原文地址:https://www.cnblogs.com/programer-xinmu78/p/10551568.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!