码迷,mamicode.com
首页 > 其他好文 > 详细

Hadoop Pipes Exception: Illegal text protocol command

时间:2014-06-15 14:37:31      阅读:243      评论:0      收藏:0      [点我收藏+]

标签:style   class   blog   code   http   tar   

Hadoop Pipes Exception: Illegal text protocol command

对于Hadoop pipes 出现这样的错误,基本上编译代码依赖的.so和.a 版本不匹配

网上也没有给出更多信息,我的同事最近回复了解决办法,可以参考

https://groups.google.com/a/cloudera.org/forum/#!msg/cdh-user/j0dpYPDx3_A/S1rotrWGSf0J

 

如果不能翻墙,摘录如下

 
I met with the same issue, and my way to work around it is to apply the attached patch, regenerate the native libraries(libhadooppipes.a, libhadooputils.a, etc.) and recompile user program with the new libraries. Notice that this is only for running against MR1, and don‘t mess up with those against Yarn. Hope this also works in your case. :-)
 
p.s. I used "mvn -Dskiptests -Pdist,native -Dtar clean package" cmd to generate the tarball and got the native libs under "lib/native" dir
Index: hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
===================================================================
--- hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc    (revision 340277)
+++ hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc    (working copy)
@@ -30,7 +30,6 @@
 #include <stdlib.h>
 #include <string.h>
 #include <strings.h>
-#include <unistd.h>
 #include <sys/socket.h>
 #include <pthread.h>
 #include <iostream>
@@ -806,8 +805,8 @@
       }
       if (reducer != NULL) {
         int64_t spillSize = 100;
-        if (jobConf->hasKey("mapreduce.task.io.sort.mb")) {
-          spillSize = jobConf->getInt("mapreduce.task.io.sort.mb");
+        if (jobConf->hasKey("io.sort.mb")) {
+          spillSize = jobConf->getInt("io.sort.mb");
         }
         writer = new CombineRunner(spillSize * 1024 * 1024, this, reducer, 
                                    uplink, partitioner, numReduces);
@@ -1042,7 +1041,7 @@
    */
   void* ping(void* ptr) {
     TaskContextImpl* context = (TaskContextImpl*) ptr;
-    char* portStr = getenv("mapreduce.pipes.command.port");
+    char* portStr = getenv("hadoop.pipes.command.port");
     int MAX_RETRIES = 3;
     int remaining_retries = MAX_RETRIES;
     while (!context->isDone()) {
@@ -1095,7 +1094,7 @@
     try {
       TaskContextImpl* context = new TaskContextImpl(factory);
       Protocol* connection;
-      char* portStr = getenv("mapreduce.pipes.command.port");
+      char* portStr = getenv("hadoop.pipes.command.port");
       int sock = -1;
       FILE* stream = NULL;
       FILE* outStream = NULL;
@@ -1128,8 +1127,8 @@
         HADOOP_ASSERT(setbuf == 0, string("problem with setvbuf for outStream: ")
                                      + strerror(errno));
         connection = new BinaryProtocol(stream, context, outStream);
-      } else if (getenv("mapreduce.pipes.commandfile")) {
-        char* filename = getenv("mapreduce.pipes.commandfile");
+      } else if (getenv("hadoop.pipes.command.file")) {
+        char* filename = getenv("hadoop.pipes.command.file");
         string outFilename = filename;
         outFilename += ".out";
         stream = fopen(filename, "r");

 

Hadoop Pipes Exception: Illegal text protocol command,布布扣,bubuko.com

Hadoop Pipes Exception: Illegal text protocol command

标签:style   class   blog   code   http   tar   

原文地址:http://www.cnblogs.com/xuxm2007/p/3789133.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!