==作者：YB-Chi==
[toc]
添加agent节点
在BDS部署
配置环境变量
修改配置文件
cp flume-env.sh.template flume-env.sh
加入jdk和hadoop的环境变量
flume-ng version验证
创建agent.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
a1.sources = sysSrc
a1.sources.sysSrc.type=syslogudp
a1.sources.sysSrc.bind=0.0.0.0
a1.sources.sysSrc.port=514
a1.sources.sysSrc.channels=fileChannel
a1.channels = fileChannel
a1.channels.fileChannel.type =memory
a1.channels.fileChannel.capacity=100000
a1.sinks = kafkaSink
a1.sinks.kafkaSink.channel=fileChannel
a1.sinks.kafkaSink.type=org.apache.flume.sink.kafka.KafkaSink
a1.sinks.kafkaSink.brokerList=BDS:9092,BDS-1:9092,BDS-2:9092
a1.sinks.kafkaSink.custom.partition.key=kafkaPartition
a1.sinks.kafkaSink.topic=flumetest
a1.sinks.kafkaSink.serializer.class=kafka.serializer.StringEncoder
|
启动
bin/flume-ng agent --conf conf --conf-file conf/agent.conf --name a1 -Dflume.root.logger=INFO,console
启动消费者
1
2
3
| cd /opt/cloudera/parcels/KAFKA-2.1.1-1.2.1.1.p0.18/lib/kafka
bin/kafka-console-consumer.sh --zookeeper BDS:2181,BDS-1:2181,BDS-2:2181 --topic flumetest --from-beginning
|
使用测试工具发送信息进行测试 查看消费者是否收到
