标签：之间 cer 大量 exp ons .sh key org continue
用Python编写WordCount程序任务
|
程序 |
WordCount |
|
输入 |
一个包含大量单词的文本文件 |
|
输出 |
文件中每个单词及其出现次数（频数），并按照单词字母顺序排序，每个单词和其频数占一行，单词和频数之间有间隔 |
1.编写map函数，reduce函数
首先在/home/hadoop路径下建立wc文件夹，在wc文件夹下创建文件mapper.py和reducer.py
|
1
2
3
4
|
cd /home/hadoopmkdir wccd /home/hadoop/wctouch mapper.py |
|
1
|
touch reducer.py |
编写两个函数
mapper.py:
|
1
2
3
4
5
6
7
|
#!/usr/bin/env pythonimport sysfor line in sys.stdin: line = line.strip() words = line.split() for word in words: print ‘%s\t%s‘ % (word,1) |
reducer.py:
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
#!/usr/bin/env pythonfrom operator import itemgetterimport syscurrent_word = Nonecurrent_count = 0word=Nonefor line in sys.stdin: line = line.strip() word, count = line.split(‘\t‘, 1) try: count=int(count) except ValueError: continue if current_word == word: current_count += count else: if current_word: print ‘%s\t%s‘ % (current_word, current_count) current_count = count current_word = wordif current_word == word: print ‘%s\t%s‘ % (current_word, current_count) |
2.将其权限作出相应修改
|
1
2
|
chmod a+x /home/hadoop/wc/mapper.pychmod a+x /home/hadoop/wc/reducer.py |
3.本机上测试运行代码
|
1
2
3
|
echo "foo foo quux labs foo bar quux" | /home/hadoop/wc/mapper.pyecho "foo foo quux labs foo bar quux" | /home/hadoop/wc/mapper.py | sort -k1,1 | /home/hadoop/wc/reducer.py |
4.放到HDFS上运行
下载文本文件或爬取网页内容存成的文本文件：
|
1
2
3
|
cd /home/hadoop/wcwget http://www.gutenberg.org/files/5000/5000-8.txtwget http://www.gutenberg.org/cache/epub/20417/pg20417.txt |
5.下载并上传文件到hdfs上
|
1
|
hdfs dfs -put /home/hadoop/hadoop/gutenberg/*.txt /user/hadoop/input |
6.用Hadoop Streaming命令提交任务
寻找你的streaming的jar文件存放地址：
|
1
|
cd /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.1.jar |
打开环境变量配置文件:
|
1
|
gedit ~/.bashrc |
在里面写入streaming路径:
|
1
|
export STREAM=$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar |
让环境变量生效:
|
1
2
|
source ~/.bashrcecho $STREAM |
建立一个shell名称为run.sh来运行：
|
1
|
gedit run.sh |
|
1
2
3
4
5
6
7
|
hadoop jar $STREAM-file /home/hadoop/wc/mapper.py \-mapper /home/hadoop/wc/mapper.py \-file /home/hadoop/wc/reducer.py \-reducer /home/hadoop/wc/reducer.py \-input /user/hadoop/input/*.txt \-output /user/hadoop/wcoutput |
|
1
|
source run.sh |
标签：之间 cer 大量 exp ons .sh key org continue
原文地址：https://www.cnblogs.com/BOXczx/p/9026183.html
