码迷，mamicode.com
首页 > 其他好文 > 详细

《Hive编程指南》14.3 投影变换的实践出错原因分析

时间：2018-05-03 23:27:42      阅读：361      评论：0      收藏：0      [点我收藏+]

标签：debug   audit   sync   help   red   col   occurred   一个   eva   

自己在学习14.3节投影变换执行SQL语句hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA, newB FROM a;时出现了这个错误

Ended Job = job_local1231989520_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

跟作者的输出不一样。
自己一开始时没有管这个错误，直接跳过这个问题，继续往下看了。== 反思1：有逃避畏难心理，没有钻研的心态==
但在执行接下来的语句SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA FROM a;时又出现了这个错误

Ended Job = job_local1771018374_0006 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

没有明确的错误信息，自己先是另开了一个终端尝试了一下：

duxing@duxing-X550JK:~$ /bin/cut
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" | /bin/cut -f1
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5

第一次没有找的出错原因，==反思2：其实这里的第一个命令执行结果就告诉了出错的原因，自己对出错信息不敏感，没有深入思考==。
又在终端尝试了一下，在执行这行语句的时候SELECT TRANSFORM(col1, col2) USING ‘cut -f1‘ AS newA, newB FROM a;得到了预期的结果，同时我又执行ls /bin | grep "cut"并没有cut文件。
现在看来是因为Ubuntu中cut程序并没有放在/bin/目录下导致语句执行出错。
在执行which cut后得到验证，cut放在了/usr/bin/目录下。
相关命令执行记录

  1. Hive终端

    hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA, newB FROM a;
Ended Job = job_local1231989520_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA, newB FROM a;
Ended Job = job_local1383279613_0005 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA FROM a;
Ended Job = job_local1771018374_0006 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/cut -f1‘ AS newA FROM a;
Ended Job = job_local81582517_0007 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/sed s/4/10‘ AS newA, newB AS a;
NoViableAltException(37@[])
at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormat(HiveParser.java:34626)
at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectTrfmClause(HiveParser_SelectClauseParser.java:2021)
at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1216)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:51850)
at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:45661)
at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:45568)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:44584)
at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:44454)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1696)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1178)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:444)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1242)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1384)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:67 cannot recognize input near ‘AS‘ ‘a‘ ‘<EOF>‘ in serde specification
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘/bin/sed s/4/10‘ AS newA, newB FROM a;
/bin/sed: -e expression #1, char 6: unterminated `s‘ command
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:585)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Ended Job = job_local1180910273_0008 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘cut -f1‘ AS newA, newB FROM a;
newA    newB
4   NULL
3   NULL
hive (default)> SELECT TRANSFORM(col1, col2) USING ‘cut -f1‘ AS newA FROM a;
newA
4
3

    终端1：

    duxing@duxing-X550JK:~$ /bin/cut
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" | /bin/cut -f1
bash: /bin/cut: No such file or directory
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |cut -f1
4 5
duxing@duxing-X550JK:~$ echo "4 5" |sed s/4/10
sed: -e expression #1, char 6: unterminated `s‘ command
duxing@duxing-X550JK:~$ sed s/4/10
sed: -e expression #1, char 6: unterminated `s‘ command

    终端2：

    duxing@duxing-X550JK:~$ ls /bin | grep "cut"
duxing@duxing-X550JK:~$ ls /bin
bash          bzmore          dd             fgrep       kbd_mode    ls          nc                ntfsfallocate  ps         sh                    systemd-inhibit                 uname           zfgrep
bunzip2       cat             df             findmnt     kill        lsblk       nc.openbsd        ntfsfix        pwd        sh.distrib            systemd-machine-id-setup        uncompress      zforce
busybox       cgroups-mount   dir            fuser       kmod        lsmod       netcat            ntfsinfo       rbash      sleep                 systemd-notify                  unicode_start   zgrep
bzcat         cgroups-umount  dmesg          fusermount  less        mkdir       netstat           ntfsls         readlink   ss                    systemd-tmpfiles                vdir            zless
bzcmp         chacl           dnsdomainname  getfacl     lessecho    mknod       networkctl        ntfsmove       red        static-sh             systemd-tty-ask-password-agent  vmmouse_detect  zmore
bzdiff        chgrp           domainname     grep        lessfile    mktemp      nisdomainname     ntfstruncate   rm         stty                  tailf                           wdctl           znew
bzegrep       chmod           dumpkeys       gunzip      lesskey     more        ntfs-3g           ntfswipe       rmdir      su                    tar                             which
bzexe         chown           echo           gzexe       lesspipe    mount       ntfs-3g.probe     open           rnano      sync                  tempfile                        whiptail
bzfgrep       chvt            ed             gzip        ln          mountpoint  ntfs-3g.secaudit  openvt         run-parts  systemctl             touch                           ypdomainname
bzgrep        cp              efibootmgr     hciconfig   loadkeys    mt          ntfs-3g.usermap   pidof          sed        systemd               true                            zcat
bzip2         cpio            egrep          hostname    login       mt-gnu      ntfscat           ping           setfacl    systemd-ask-password  udevadm                         zcmp
bzip2recover  dash            false          ip          loginctl    mv          ntfscluster       ping6          setfont    systemd-escape        ulockmgr_server                 zdiff
bzless        date            fgconsole      journalctl  lowntfs-3g  nano        ntfscmp           plymouth       setupcon   systemd-hwdb          umount                          zegrep
duxing@duxing-X550JK:~$ which cut
/usr/bin/cut
duxing@duxing-X550JK:~$ /usr/bin/cut
/usr/bin/cut: you must specify a list of bytes, characters, or fields
Try ‘/usr/bin/cut --help‘ for more information.

    补充：在写本文的时候发现其实在终端执行/bin/cut的时候已经可以得到出错原因了，bash: /bin/cut: No such file or directory已经提示了/bin/cut不存在，而执行/usr/bin/cut的提示是没有参数。

《Hive编程指南》14.3 投影变换的实践出错原因分析

标签：debug   audit   sync   help   red   col   occurred   一个   eva   

原文地址：https://www.cnblogs.com/DataNerd/p/8987990.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！