标签：
Flume的一些组件（如Spooling Directory Source、File Channel）能够保证agent挂掉后不丢失数据。
1、负载均衡
1）Load balancing Sink Processor
source里的event流经channel，进入sink组，在sink组内部根据负载算法（round_robin、random）选择sink，后续可以选择不同机器上的agent实现负载均衡。
实例如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
# Name the components on this agenta1.sources = r1a1.sinks = k1 k2a1.channels = c1 # Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.channels=c1a1.sources.r1.command=tail -F /home/flume/xx.log #define sinkgroupsa1.sinkgroups=g1a1.sinkgroups.g1.sinks=k1 k2a1.sinkgroups.g1.processor.type=load_balancea1.sinkgroups.g1.processor.backoff=truea1.sinkgroups.g1.processor.selector=round_robin#define the sink 1a1.sinks.k1.type=avroa1.sinks.k1.hostname=192.168.1.112a1.sinks.k1.port=9876 #define the sink 2a1.sinks.k2.type=avroa1.sinks.k2.hostname=192.168.1.113a1.sinks.k2.port=9876# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1a1.sinks.k2.channel=c1 |
2）Load balancing Log4J Appender
不同的agent处理同一个client产生的数据。
|
1
2
3
|
log4j.rootLogger=INFO,flumelog4j.appender.flume = org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppenderlog4j.appender.flume.Hosts = 192.168.1.111:41414 192.168.1.111:41414 |
2、故障转移
Failover Sink Processor
配置一组sink，这组sink组成一个Failover Sink Processor，当有一个sink处理失败，Flume将这个sink放到一个地方，等待冷却时间，可以正常处理event时再拿回来。
event通过通过一个channel流向一个sink组，在sink组内部根据优先级选择具体的sink，一个失败后再转向另一个sink，流程图如下：
实例如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
# Name the components on this agenta1.sources = r1a1.sinks = k1 k2a1.channels = c1 # Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.channels=c1a1.sources.r1.command=tail -F /home/flume/xx.log#define sinkgroupsa1.sinkgroups=g1a1.sinkgroups.g1.sinks=k1 k2a1.sinkgroups.g1.processor.type=failovera1.sinkgroups.g1.processor.priority.k1=10a1.sinkgroups.g1.processor.priority.k2=5a1.sinkgroups.g1.processor.maxpenalty=10000#define the sink 1a1.sinks.k1.type=avroa1.sinks.k1.hostname=192.168.1.112a1.sinks.k1.port=9876#define the sink 2a1.sinks.k2.type=avroa1.sinks.k2.hostname=192.168.1.113a1.sinks.k2.port=9876# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1a1.sinks.k2.channel=c1 |
Sink Prosessor - Flume的可靠性保证：故障转移、负载均衡
标签：
原文地址：http://www.cnblogs.com/lishouguang/p/4558790.html
