- 18.11 LVS DR模式搭建
- 18.12 keepalived + LVS
- 扩展
- haproxy+keepalived http://blog.csdn.net/xrt95050/article/details/40926255
- nginx、lvs、haproxy比较 http://www.csdn.net/article/2014-07-24/2820837
- keepalived中自定义脚本 vrrp_script http://my.oschina.net/hncscwc/blog/158746
- lvs dr模式只使用一个公网ip的实现方法 http://storysky.blog.51cto.com/628458/338726
# 18.11 LVS DR模式搭建
- 在生产环境用中的比较多的是DR模式，NAT模式有瓶颈，节省公网IP，对小公司来说公网IP也是花钱的，如果是配置的多台机器，每台机器都去配置一个公网IP就是很浪费资源的情况，而且当下公网IP越来越少；
- 另一种方案，搭建内部的lvs，全部都用内网，包括VIP也用内网，用一个公网IP做一个映射；公网的80端口映射到内网VIP的80端口，这样可以节省IP
- DR模式搭建
- 准备工作
- 三台机器
- dir aming-01（dir） 192.168.202.130 分发器，也叫调度器（简写为dir）
- rs1 aming-02（rs1）192.168.202.132
- rs2 aming-03（rs2）192.168.202.133
- vip 192.168.202.200
- 因为前一章节做了NAT模式，现在需要把两台rs 机器的网关 给改回来，原来的dir机器上配置的ens37的网卡就先不理他。
- 确保两台rs公网能ping通外网
- 先修改俩台rs 机器的网关 原来俩台机器的网关是192.168.202.2
- rs1
```
[root@aming-02 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
[root@aming-02 ~]# systemctl restart network.service
[root@aming-02 ~]#
```
- rs2
```
[root@aming-03 ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33
[root@aming-03 ~]# systemctl restart network.service
[root@aming-03 ~]#
```
- dir机器配置
- 在分发器服务器上，创建一个脚本
```
[root@aming-01 ~]# vim /usr/local/sbin/lvs_dr.sh
#! /bin/bash
echo 1 > /proc/sys/net/ipv4/ip_forward
ipv=/usr/sbin/ipvsadm
vip=192.168.202.200
rs1=192.168.202.132
rs2=192.168.202.133
#注意这里的网卡名字
ifdown ens33
ifup ens33
ifconfig ens33:2 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip dev ens33:2
$ipv -C
$ipv -A -t $vip:80 -s rr
$ipv -a -t $vip:80 -r $rs1:80 -g -w 1
$ipv -a -t $vip:80 -r $rs2:80 -g -w 1
~
:wq
[root@aming-01 ~]# vim /usr/local/sbin/lvs_dr.sh
[root@aming-01 ~]#
```
- 启动下脚本
```
[root@aming-01 ~]# sh /usr/local/sbin/lvs_dr.sh
成功断开设备 ‘ens33‘。
成功激活的连接（D-Bus 激活路径：/org/freedesktop/NetworkManager/ActiveConnection/9）
[root@aming-01 ~]#
```
- 俩个rs机器上也要配置脚本
- 两台rs上也编写脚本 vim /usr/local/sbin/lvs_rs.sh//内容如下
- 先在rs1 aming-02上
```
[root@aming-02 ~]# vi /usr/local/sbin/lvs_rs.sh
#/bin/bash
vip=192.168.202.200
#把vip绑定在lo上，是为了实现rs直接把结果返回给客户端
ifdown lo
ifup lo
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数，目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
:wq
[root@aming-02 ~]# vi /usr/local/sbin/lvs_rs.sh
[root@aming-02 ~]# sh /usr/local/sbin/lvs_rs.sh
[root@aming-02 ~]#
[root@aming-02 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.202.200/32 brd 192.168.202.200 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:58:33:e6 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.132/24 brd 192.168.202.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.202.152/24 brd 192.168.202.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::ecdd:28b7:612b:cb7/64 scope link
valid_lft forever preferred_lft forever
[root@aming-02 ~]#
```
- rs2 aming-03 也是一样创建一个脚本添加如下内容
```
[root@aming-03 ~]# vi /usr/local/sbin/lvs_rs.sh
#/bin/bash
vip=192.168.202.200
#把vip绑定在lo上，是为了实现rs直接把结果返回给客户端
ifdown lo
ifup lo
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
#以下操作为更改arp内核参数，目的是为了让rs顺利发送mac地址给客户端
#参考文档www.cnblogs.com/lgfeng/archive/2012/10/16/2726308.html
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
~
:wq
执行脚本
[root@aming-03 ~]# vi /usr/local/sbin/lvs_rs.sh
[root@aming-03 ~]# sh /usr/local/sbin/lvs_rs.sh
[root@aming-03 ~]#
```
- 可以route -n 查看下
```
[root@aming-03 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.202.2 0.0.0.0 UG 100 0 0 ens33
192.168.202.0 0.0.0.0 255.255.255.0 U 100 0 0 ens33
192.168.202.200 0.0.0.0 255.255.255.255 UH 0 0 0 lo
[root@aming-03 ~]#
[root@aming-03 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.202.200/32 brd 192.168.202.200 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:9c:2b:f0 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.133/24 brd 192.168.202.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.202.153/24 brd 192.168.202.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::4500:6d42:8612:4e53/64 scope link
valid_lft forever preferred_lft forever
inet6 fe80::ecdd:28b7:612b:cb7/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
[root@aming-03 ~]#
```
- 去分发器dir 上看下
```
[root@aming-01 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:2e:28:f2 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.130/24 brd 192.168.202.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.202.200/32 brd 192.168.202.200 scope global ens33:2
valid_lft forever preferred_lft forever
inet 192.168.202.150/24 brd 192.168.202.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::ddac:89a0:52f8:d08d/64 scope link
valid_lft forever preferred_lft forever
inet6 fe80::4500:6d42:8612:4e53/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::ecdd:28b7:612b:cb7/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:2e:28:fc brd ff:ff:ff:ff:ff:ff
inet 192.168.142.147/24 brd 192.168.142.255 scope global ens37
valid_lft forever preferred_lft forever
inet 192.168.142.128/24 brd 192.168.142.255 scope global secondary dynamic ens37
valid_lft 1235sec preferred_lft 1235sec
inet6 fe80::20c:29ff:fe2e:28fc/64 scope link
valid_lft forever preferred_lft forever
[root@aming-01 ~]#
```
- 分别在机器上执行这些脚本
- 测试
- 去windows 浏览器访问下192.168.202.200
- 
- 刷新没有效果，主要是浏览器上有缓存，那就在dir机器上crul试试
```
[root@aming-01 ~]# iptables -nvL
Chain INPUT (policy ACCEPT 4266 packets, 452K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 479 packets, 47671 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 17593 packets, 968K bytes)
pkts bytes target prot opt in out source destination
[root@aming-01 ~]# iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 217 packets, 32912 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 49 packets, 9980 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 100 packets, 9178 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 9 packets, 2456 bytes)
pkts bytes target prot opt in out source destination
115 8744 MASQUERADE all -- * * 192.168.202.0/24 0.0.0.0/0
[root@aming-01 ~]# iptables -t nat -F
[root@aming-01 ~]# curl http://192.168.202.200/
^C
^C
[root@aming-01 ~]#
```
- 不能访问的原因是俩边的机器都有vip，在本机上访问不行，最好是再开一个虚拟机，或者自己的windows浏览器访问下
- 
```
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.202.200:80 rr
-> 192.168.202.132:80 Route 1 1 3
-> 192.168.202.133:80 Route 1 1 1
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.202.200:80 rr
-> 192.168.202.132:80 Route 1 1 2
-> 192.168.202.133:80 Route 1 1 1
[root@aming-01 ~]#
```
# 18.12 keepalived lvs
- 完整架构需要两台服务器（角色为dir）分别安装keepalived软件，目的是实现高可用，但keepalived本身也有负载均衡的功能，所以本次实验可以只安装一台keepalived
- 为什么需要把keepalived 加到lvs 中的目的是什么
- 第一个原因：lvs有个分发器角色，如果宕掉以后，后端的rs就没有办法继续使用，所以需要用keepalived做一个高可用，
- 第二个原因：在使用lvs的时候，当后端有一台rs机器宕机时，lvs照样会分发数据到这台宕机机器，这是就会出现访问无效的情况，说明lvs并不聪明；这时使用keepalived，就可以保证集群中其中一台rs宕机了，web还能正常提供，不会出现用户访问时无效链接的结果；一般这种架构，肯定是2台keepalived；
- 因为keepalived内置了ipvsadm的功能，所以不再需要安装ipvsadm的包，也不用再编写和执行.sh脚本
- 准备工作
- 三台机器分别为：
- dir（安装keepalived）202.130
- rs1 202.132
- rs2 202.133
- vip 202.200
- 卸载掉ipvmsamd 清空ipvsadm规则
ipvsadm -C
- 查看一下ipvsadm的规则
```
[root@aming-01 ~]# ipvsadm -C
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@aming-01 ~]#
```
- 清除之前配置的IP
sytemctl restart network
- 编辑keepalived配置文件 vim /etc/keepalived/keepalived.conf
- 内容请到https://coding.net/u/aminglinux/p/aminglinux-book/git/blob/master/D21Z/lvs_keepalived.conf 获取
- 需要更改里面的ip信息
- 先尝试关闭rs2 aming-03的 nginx
```
[root@aming-03 ~]# systemctl stop nginx
[root@aming-03 ~]# ps aux |grep nginx
root 4956 0.0 0.0 112680 980 pts/0 S+ 23:59 0:00 grep --color=auto nginx
[root@aming-03 ~]#
```
-
```
[root@aming-01 ~]# vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
#备用服务器上为 BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
192.168.202.200
virtual_router_id 51
#备用服务器上为90
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass aminglinux
}
virtual_ipaddress {
192.168.202.200
}
}
virtual_server 192.168.202.200 80 {
#(每隔10秒查询realserver状态)
delay_loop 10
#(lvs 算法)
lb_algo wlc
#(DR模式)
lb_kind DR
#(同一IP的连接60秒内被分配到同一台realserver)
persistence_timeout 0
#(用TCP协议检查realserver状态)
protocol TCP
real_server 192.168.202.132 80 {
#(权重)
weight 100
TCP_CHECK {
#(10秒无响应超时)
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
real_server 192.168.202.133 80 {
weight 100
TCP_CHECK {
connect_timeout 10
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}
:wq
[root@aming-01 ~]# vim /etc/keepalived/keepalived.conf
```
- 开启keepalived 服务,看看进程
```
[root@aming-01 ~]# systemctl start keepalived
[root@aming-01 ~]#
[root@aming-01 ~]# ps aux |grep keep
root 25759 0.0 0.0 112680 980 pts/2 R+ 23:47 0:00 grep --color=auto keep
root 124427 0.0 0.1 120720 1400 ? Ss 20:01 0:01 /usr/sbin/keepalived -D
root 124428 0.0 0.2 120720 2756 ? S 20:01 0:01 /usr/sbin/keepalived -D
root 124429 0.0 0.2 124976 2760 ? S 20:01 0:10 /usr/sbin/keepalived -D
[root@aming-01 ~]#
```
- 看下ip
```
[root@aming-01 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:2e:28:f2 brd ff:ff:ff:ff:ff:ff
inet 192.168.202.130/24 brd 192.168.202.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.202.200/32 brd 192.168.202.200 scope global ens33:2
valid_lft forever preferred_lft forever
inet 192.168.202.150/24 brd 192.168.202.255 scope global secondary ens33:0
valid_lft forever preferred_lft forever
inet6 fe80::ddac:89a0:52f8:d08d/64 scope link
valid_lft forever preferred_lft forever
inet6 fe80::4500:6d42:8612:4e53/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
inet6 fe80::ecdd:28b7:612b:cb7/64 scope link tentative dadfailed
valid_lft forever preferred_lft forever
3: ens37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:2e:28:fc brd ff:ff:ff:ff:ff:ff
inet 192.168.142.147/24 brd 192.168.142.255 scope global ens37
valid_lft forever preferred_lft forever
inet 192.168.142.128/24 brd 192.168.142.255 scope global secondary dynamic ens37
valid_lft 1433sec preferred_lft 1433sec
inet6 fe80::20c:29ff:fe2e:28fc/64 scope link
valid_lft forever preferred_lft forever
[root@aming-01 ~]#
```
- 查看下启动后的规则
```
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.202.200:80 wlc
-> 192.168.202.132:80 Route 100 0 0
-> 192.168.202.133:80 Route 100 0 0
[root@aming-01 ~]#
```
- 停掉keepalived服务 再看下就没有规则了
```
[root@aming-01 ~]# systemctl stop keepalived
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@aming-01 ~]#
```
- 再启动keepalived 再看就有了
```
[root@aming-01 ~]# systemctl start keepalived
[root@aming-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.202.200:80 wlc
-> 192.168.202.132:80 Route 100 0 0
-> 192.168.202.133:80 Route 100 0 0
[root@aming-01 ~]#
```
- 同时还需要做两点
- 1.打开dir机器的端口转发
```
echo 1 > /proc/sys/net/ipv4/ip_forward //打开端口转发
- 2.运行前一章在rs机器上创建的lvs_rs.sh脚本
- #把vip绑定在lo上，是为了实现rs直接把结果返回给客户端
ifconfig lo:0 $vip broadcast $vip netmask 255.255.255.255 up
route add -host $vip lo:0
- #以下操作为更改arp内核参数，目的是为了让rs顺利发送mac地址给客户端
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
```
- 总结：
- keepalived 有一个比较好的功能，可以在一台rs宕机的时候，及时把他踢出 ipvsadm 集群，将不再发送数据包给，也就很好的避免的访问无连接的情况发送18.11 LVS DR模式搭建 18.12 keepalived + LVS
原文地址：http://ch71smas.blog.51cto.com/13090095/1982115
