码迷，mamicode.com
首页 > 其他好文 > 详细

hive bug

时间：2017-11-18 23:35:17      阅读：263      评论：0      收藏：0      [点我收藏+]

标签：stage   shu   nis   soc   its   parameter   ase   sub   ==   

  1. SHELL$ hive -S -e “set” | grep warehouse
  2. hive.metastore.warehouse.dir=/user/hive/warehouse
  3. hive.warehouse.subdir.inherit.perms=false

 

 

Status: Running (Executing on YARN cluster with App id application_1510723060011_0082)

--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 .......... SUCCEEDED 1 1 0 0 0 0
Map 10 ......... SUCCEEDED 1 1 0 0 0 0
Map 2 .......... SUCCEEDED 1 1 0 0 0 0
Map 3 .......... SUCCEEDED 1 1 0 0 0 0
Map 4 .......... SUCCEEDED 1 1 0 0 0 0
Map 5 .......... SUCCEEDED 1 1 0 0 0 0
Map 6 .......... SUCCEEDED 1 1 0 0 0 0
Map 8 .......... SUCCEEDED 21 21 0 0 0 0
Map 9 .......... SUCCEEDED 2 2 0 0 0 0
Reducer 7 ...... SUCCEEDED 1 1 0 0 0 0
--------------------------------------------------------------------------------
VERTICES: 10/10 [==========================>>] 100% ELAPSED TIME: 13.72 s
--------------------------------------------------------------------------------
Loading data to table bridge.sods_to_cio_libor_hold partition (biz_date=20170628)
Failed with exception java.io.IOException: rename for src path: hdfs://mycluster/apps/hive/warehouse/bridge.db/sods_to_cio_libor_hold/biz_date=20170628/.hive-staging_hive_2017-11-16_10-06-19_248_4976600282215103377-1/-ext-10000/000008_0 to dest path:hdfs://mycluster/apps/hive/warehouse/bridge.db/sods_to_cio_libor_hold/biz_date=20170628/000008_0 returned false
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

 

 

 

 

 

 

 

 

 

om bridge got value #279
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: submitDAG took 370ms
17/11/18 16:55:48 [main]: INFO client.TezClient: Submitted dag to TezSession, sessionName=HIVE-f8697cdf-5cb6-4d3b-9ea6-db5bc976d944, applicationId=application_1510723060011_0609, dagName=Insert overwrite table bridge.sod...‘X_CNBD‘(Stage-1)
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG ssl.FileBasedKeyStoresFactory: CLIENT TrustStore: /etc/security/clientKeys/all.jks
17/11/18 16:55:48 [main]: DEBUG impl.TimelineClientImpl: Cannot load customized ssl related configuration. Fallback to system-generic settings.
java.io.FileNotFoundException: /etc/security/clientKeys/all.jks (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.loadTrustManager(ReloadingX509TrustManager.java:168)
at org.apache.hadoop.security.ssl.ReloadingX509TrustManager.<init>(ReloadingX509TrustManager.java:86)
at org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:211)
at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:151)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.initSslConnConfigurator(TimelineClientImpl.java:510)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.initConnConfigurator(TimelineClientImpl.java:486)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:272)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:163)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.tez.client.TezYarnClient.init(TezYarnClient.java:46)
at org.apache.tez.dag.api.client.DAGClientImpl.<init>(DAGClientImpl.java:90)
at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:581)
at org.apache.tez.client.TezClient.submitDAG(TezClient.java:466)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:446)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:181)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: INFO impl.TimelineClientImpl: Timeline service address: http://cnsz131375:8188/ws/v1/timeline/
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.AHSClientImpl entered state INITED
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:163)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: INFO client.AHSProxy: Connecting to Application History server at cnsz131375/10.25.167.173:10200
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.AHSProxy.getProxy(AHSProxy.java:49)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationHistoryProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.AHSClientImpl is started
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /seabox/var/lib/hadoop-hdfs/dn_socket
17/11/18 16:55:48 [main]: DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://mycluster
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = true
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
17/11/18 16:55:48 [main]: DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /seabox/var/lib/hadoop-hdfs/dn_socket
17/11/18 16:55:48 [main]: DEBUG retry.RetryUtils: multipleLinearRandomRetry = MultipleLinearRandomRetry[500x2000ms]
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131375/10.25.167.173:8020
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge: starting, having connections 8
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #280
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #280
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 2ms
17/11/18 16:55:48 [main]: DEBUG impl.FileSystemTimelineWriter: yarn.timeline-service.client.fd-flush-interval-secs=10, yarn.timeline-service.client.fd-clean-interval-secs=60, yarn.timeline-service.client.fd-retain-secs=300, yarn.timeline-service.entity-file.fs-support-append=true, yarn.timeline-service.entity-group-fs-store.active-dir=/ats/active
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl is started
17/11/18 16:55:48 [main]: DEBUG service.AbstractService: Service org.apache.hadoop.yarn.client.api.impl.YarnClientImpl is started
17/11/18 16:55:48 [main]: INFO log.PerfLogger: </PERFLOG method=TezSubmitDag start=1510995348115 end=1510995348697 duration=582 from=org.apache.hadoop.hive.ql.exec.tez.TezTask>


17/11/18 16:55:48 [main]: INFO SessionState:

17/11/18 16:55:48 [main]: INFO log.PerfLogger: <PERFLOG method=TezRunDag from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:48 [main]: INFO log.PerfLogger: <PERFLOG method=TezSubmitToRunningDag from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131375/10.25.167.173:8032
17/11/18 16:55:48 [main]: WARN ipc.Client: Failed to connect to server: cnsz131375/10.25.167.173:8032: retries get failed due to exceeded maximum allowed retries number: 0
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: closing ipc connection to cnsz131375/10.25.167.173:8032: Connection refused
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8032 from bridge: closed
17/11/18 16:55:48 [main]: DEBUG retry.RetryInvocationHandler: Exception while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over rm1. Trying to failover immediately.
java.net.ConnectException: Call From cnsz131372/10.25.167.170 to cnsz131375:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:801)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
at org.apache.hadoop.ipc.Client.call(Client.java:1496)
at org.apache.hadoop.ipc.Client.call(Client.java:1396)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy27.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
at com.sun.proxy.$Proxy28.getApplicationReport(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:436)
at org.apache.tez.client.TezYarnClient.getApplicationReport(TezYarnClient.java:89)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getAppReport(DAGClientRPCImpl.java:231)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.createAMProxyIfNeeded(DAGClientRPCImpl.java:251)
at org.apache.tez.dag.api.client.rpc.DAGClientRPCImpl.getDAGStatus(DAGClientRPCImpl.java:96)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusViaAM(DAGClientImpl.java:367)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatusInternal(DAGClientImpl.java:227)
at org.apache.tez.dag.api.client.DAGClientImpl.getDAGStatus(DAGClientImpl.java:275)
at org.apache.hadoop.hive.ql.exec.tez.TezTask$SyncDagClient.getDAGStatus(TezTask.java:616)
at org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor.monitorExecution(TezJobMonitor.java:295)
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:186)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:89)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1745)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1491)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1146)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:168)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:379)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:684)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:650)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:745)
at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
at org.apache.hadoop.ipc.Client.call(Client.java:1449)
... 42 more
17/11/18 16:55:48 [main]: INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.RMProxy.getProxy(RMProxy.java:163)
17/11/18 16:55:48 [main]: DEBUG ipc.YarnRPC: Creating YarnRPC for org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
17/11/18 16:55:48 [main]: DEBUG ipc.HadoopYarnProtoRPC: Creating a HadoopYarnProtoRpc proxy for protocol interface org.apache.hadoop.yarn.api.ApplicationClientProtocol
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge sending #282
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131376/10.25.167.174:8032 from bridge got value #282
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getApplicationReport took 0ms
17/11/18 16:55:48 [main]: DEBUG rpc.DAGClientRPCImpl: App: application_1510723060011_0609 in state: RUNNING
17/11/18 16:55:48 [main]: DEBUG client.TezClientUtils: Connecting to Tez AM at cnsz131372/10.25.167.170:43571
17/11/18 16:55:48 [main]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.tez.client.TezClientUtils.getAMProxy(TezClientUtils.java:908)
17/11/18 16:55:48 [main]: DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:48 [main]: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:48 [main]: DEBUG ipc.Client: The ping interval is 60000 ms.
17/11/18 16:55:48 [main]: DEBUG ipc.Client: Connecting to cnsz131372/10.25.167.170:43571
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge: starting, having connections 9
17/11/18 16:55:48 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #283
17/11/18 16:55:48 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #283
17/11/18 16:55:48 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getDAGStatus took 165ms
17/11/18 16:55:48 [main]: INFO log.PerfLogger: </PERFLOG method=TezSubmitToRunningDag start=1510995348707 end=1510995348894 duration=187 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
Status: Running (Executing on YARN cluster with App id application_1510723060011_0609)

17/11/18 16:55:48 [main]: INFO SessionState: Status: Running (Executing on YARN cluster with App id application_1510723060011_0609)

--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITIALIZING -1 0 0 -1 0 0
Map 2 INITIALIZING -1 0 0 -1 0 0
Map 3 INITIALIZING -1 0 0 -1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
Map 2 INITED 1 0 0 1 0 0
Map 3 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Map 2 INITED 1 0 0 1 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #284
Map 3 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0/1 Map 3: 0/1 Map 4: 0/1 Map 5: 0/1 Map 6: 0/1 Map 8: 0/21 Map 9: 0/2 Reducer
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
Map 1 INITIALIZING -1 0 0 -1 0 0
Map 10 INITED 1 0 0 1 0 0
Map 2 INITED 1 0 0 1 0 0
Map 3 INITED 1 0 0 1 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Map 4 INITED 1 0 0 1 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #294
Map 5 INITED 1 0 0 1 0 0
Map 6 INITED 1 0 0 1 0 0
Map 8 INITED 21 0 0 21 0 0
Map 9 INITED 2 0 0 2 0 0p: application_1510723060011_0609 dag:dag_1510723060011_0609_1
Reducer 7 INITED 2 0 0 2 0 0t (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #295
--------------------------------------------------------------------------------
VERTICES: 00/10 [>>--------------------------] 0% ELAPSED TIME: 2.44 s
--------------------------------------------------------------------------------
VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
--------------------------------------------------------------------------------
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 1
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #298
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #298T handler]: DEBUG rpc.DAGClientRPCImpl: TryKill for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 19ms0421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #296
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 10z131372/10.25.167.170:43571 fr17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #299
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #299: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms90421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #297
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 2sz131372/10.25.167.170:43571 fr17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #300
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #300
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 3
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #301
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #301
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 4
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #302
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #302
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 5
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #303
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #303
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 6
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #304
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #304
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 8
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #305
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #305
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Map 9
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #306
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #306
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 1ms
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetVertexStatus via AM for app: application_1510723060011_0609 dag: dag_1510723060011_0609_1 vertex: Reducer 7
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #307
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #307
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getVertexStatus took 0ms
Map 1 KILLED -1 0 0 -1 0 0
Map 10 KILLED 1 0 0 1 0 0
Map 2 KILLED 1 0 0 1 0 0
Map 3 KILLED 1 0 0 1 0 0
Map 4 KILLED 1 0 0 1 0 0
Map 5 KILLED 1 0 0 1 0 0
Map 6 KILLED 1 0 0 1 0 0
Map 8 KILLED 21 0 0 21 0 0
Map 9 KILLED 2 0 0 2 0 0
Reducer 7 KILLED 2 0 0 2 0 0
--------------------------------------------------------------------------------
VERTICES: 00/10 [>>--------------------------] 0% ELAPSED TIME: 2.69 s
--------------------------------------------------------------------------------
Status: Killed
17/11/18 16:55:51 [main]: INFO SessionState: Status: Killed
Job received Kill while in RUNNING state.
17/11/18 16:55:51 [main]: ERROR SessionState: Job received Kill while in RUNNING state.
Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]
Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]
17/11/18 16:55:51 [main]: ERROR SessionState: Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]
DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: ERROR SessionState: DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=TezRunDag start=1510995348707 end=1510995351582 duration=2875 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
17/11/18 16:55:51 [main]: DEBUG rpc.DAGClientRPCImpl: GetDAGStatus via AM for app: application_1510723060011_0609 dag:dag_1510723060011_0609_1
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge sending #308
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131372/10.25.167.170:43571 from bridge got value #308
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: getDAGStatus took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #309
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #309
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #310
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #310
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 0ms
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.AHSClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: DEBUG service.AbstractService: Service: org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl entered state STOPPED
17/11/18 16:55:51 [main]: DEBUG impl.FileSystemTimelineWriter: Closing cache
17/11/18 16:55:51 [main]: DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@a77614d
17/11/18 16:55:51 [main]: INFO hooks.ATSHook: Created ATS Hook
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=FailureHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1510995351618 end=1510995351618 duration=0 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [ATS Logger 0]: INFO hooks.ATSHook: Received post-hook notification for :bridge_20171118165543_29ad0d2b-ae72-466b-9139-210b176ff364
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Job received Kill while in RUNNING state.Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: ERROR ql.Driver: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Job received Kill while in RUNNING state.Vertex killed, vertexName=Map 8, vertexId=vertex_1510723060011_0609_1_09, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_09 [Map 8] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 1, vertexId=vertex_1510723060011_0609_1_08, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1510723060011_0609_1_08 [Map 1] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Reducer 7, vertexId=vertex_1510723060011_0609_1_05, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_05 [Reducer 7] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 10, vertexId=vertex_1510723060011_0609_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_01 [Map 10] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 5, vertexId=vertex_1510723060011_0609_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_02 [Map 5] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 4, vertexId=vertex_1510723060011_0609_1_03, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_03 [Map 4] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 6, vertexId=vertex_1510723060011_0609_1_04, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_04 [Map 6] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 9, vertexId=vertex_1510723060011_0609_1_00, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:2, Vertex vertex_1510723060011_0609_1_00 [Map 9] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 3, vertexId=vertex_1510723060011_0609_1_06, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_06 [Map 3] killed/failed due to:DAG_KILL]Vertex killed, vertexName=Map 2, vertexId=vertex_1510723060011_0609_1_07, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to DAG_KILL, failedTasks:0 killedTasks:1, Vertex vertex_1510723060011_0609_1_07 [Map 2] killed/failed due to:DAG_KILL]DAG did not succeed due to DAG_KILL. failedVertices:0 killedVertices:10
17/11/18 16:55:51 [main]: DEBUG ql.Driver: Shutting down query Insert overwrite table bridge.sods_to_cio_libor_hold partition(biz_date=20170606)
SELECT ‘CNY‘,
‘3002‘ AS COMPANY_REPORT, --数据上报公司
‘3002‘ AS COMPANY_MANAGER, --投资管理公司
‘3002‘ AS COMPANY_OWNER, --投资主体公司
P1.MAPPING_KEY AS BUSINESS_LINE, --业务线
CASE
WHEN T1.TRDTYPE in (‘42‘, ‘82‘, ‘84‘) THEN ‘01001002‘
WHEN T1.TRDTYPE in (‘43‘, ‘83‘, ‘85‘) THEN ‘01001003‘
END AS DETAIL_ASSET_TYPE, --明细资产分类
CASE p2.mapping_key
WHEN ‘005‘ THEN ‘001‘
ELSE ‘001‘
END AS FUND_SOURCE, --资金来源, 把005修改成001后上报，001保持不变
P.PORTFOLIO_UNITS AS PORTFOLIO_UNITS, --投资组合
T1.I_CODE AS SECURITY_ID, --证券代码,给流水号
T1.I_CODE AS SECURITY_NAME, --证券名称,给流水号
nvl(T20.TARGET_CODE,‘OTHER‘) AS MARKET_NO, --市场代码
nvl(T20.TARGET_NAME,‘其他‘) AS MARKET_NAME, --市场名称
oc.partyname AS COUNTER_PARTY, --交易对手名称
T1.partyid AS COUNTER_PARTY_ID, --交易对手组织机构代码
T2.AMOUNT AS HOLDING_UNITS, --持仓数量PAR_VALUE
T2.AMOUNT AS BOOK_BALANCE_N, --财务账面余额_本币AMOUNT
T2.AMOUNT AS BOOK_BALANCE_B, --财务账面余额_本币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_N, --参考财务成本_原币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_B, --参考财务成本_本币AMOUNT
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE(‘2017-06-30‘),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_N, --应计利息_原币
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE(‘2017-06-30‘),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_B --应计利息_本币
FROM bridge.FBMS_A_TTRD_OTC_TRADE_IB T1
INNER JOIN bridge.FBMS_A_TTRD_OTC_IBLB_IB T2 ON T2.I_CODE = T1.INTORDID
AND T2.M_TYPE = T1.M_TYPE
AND T2.A_TYPE = T1.A_TYPE
AND T2.ENDSETDATE > TO_DATE(‘2017-06-30‘) --投资组合映射主表
left join (select v1.accountid, v2.murex_leaf, v2.Third_Business_Line as business_line, v2.portfolio_units, v2.fund_source, v2.account_class
from bridge.BRIDGE_APP_PORTFOLIOACCOUNT_ib v1
inner join bridge.bridge_murex_leaf_config v2 on v1.portfolioid = v2.murex_leaf
and ascii(v2.is_send_group) = -26
and v1.biz_date = v2.biz_date
where ‘20170630‘ = v2.biz_date) p on p.accountid = t1.secu_accid
--业务线
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘BUSINESS_LINE‘ and ‘20170630‘ = biz_date) p1 on p.business_line = p1.chinese_name
--资金来源
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘FUND_SOURCE‘ and ‘20170630‘ = biz_date) p2 on p.fund_source = p2.chinese_name
--会计分类
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘ACCOUNT_CLASS‘ and ‘20170630‘ = biz_date) p3 on p.account_class = p3.chinese_name
--交易对手(交易对手SUMMIT使用的是衡泰的数据源，衡泰是从交易中心获取的，可以通用)
left join (select max(partyname) as partyname, partyid from bridge.FBMS_A_TTRD_OTC_COUNTERPARTY_IB where ‘20170630‘ = biz_date group by partyid) oc on t1.partyid = oc.partyid

left join bridge.bridge_security_to_cio_mapping_ib T20
on T1.M_TYPE = T20.SOURCE_CODE
and T20.SOURCE_SYSTEM_CODE = ‘FBMS‘
and T20.MAPPING_TYPE =‘MARKET_NO‘
and T20.SOURCE_COLUMN_NAME = ‘M_TYPE‘
and T20.SOURCE_TABLE_NAME = ‘FBMS_A_TTRD_OTC_TRADE_IB‘
and T20.biz_date=‘20170630‘
WHERE T1.ORDSOURCE NOT IN (‘4‘, ‘11‘) --过滤掉投顾指令单
AND T1.ORDSTATUS = ‘7‘
AND T1.A_TYPE in (‘SPT_IBOR‘)
AND T1.M_TYPE = ‘X_CNBD‘

17/11/18 16:55:51 [ATS Logger 0]: DEBUG security.UserGroupInformation: PrivilegedAction as:bridge (auth:SIMPLE) from:org.apache.hadoop.yarn.client.api.impl.TimelineWriter.doPosting(TimelineWriter.java:112)
17/11/18 16:55:51 [main]: INFO ql.Driver: Resetting the caller context to
17/11/18 16:55:51 [ATS Logger 0]: DEBUG impl.TimelineWriter: POST to http://cnsz131375:8188/ws/v1/timeline/
17/11/18 16:55:51 [main]: INFO metadata.Hive: Dumping metastore api call timing information for : execution phase
17/11/18 16:55:51 [main]: DEBUG metadata.Hive: Total time spent in each metastore function (ms): {isCompatibleWith_(HiveConf, )=0}
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1510995347627 end=1510995351619 duration=3992 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1510995351619 end=1510995351619 duration=0 from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: DEBUG ql.Driver: Shutting down query Insert overwrite table bridge.sods_to_cio_libor_hold partition(biz_date=20170606)
SELECT ‘CNY‘,
‘3002‘ AS COMPANY_REPORT, --数据上报公司
‘3002‘ AS COMPANY_MANAGER, --投资管理公司
‘3002‘ AS COMPANY_OWNER, --投资主体公司
P1.MAPPING_KEY AS BUSINESS_LINE, --业务线
CASE
WHEN T1.TRDTYPE in (‘42‘, ‘82‘, ‘84‘) THEN ‘01001002‘
WHEN T1.TRDTYPE in (‘43‘, ‘83‘, ‘85‘) THEN ‘01001003‘
END AS DETAIL_ASSET_TYPE, --明细资产分类
CASE p2.mapping_key
WHEN ‘005‘ THEN ‘001‘
ELSE ‘001‘
END AS FUND_SOURCE, --资金来源, 把005修改成001后上报，001保持不变
P.PORTFOLIO_UNITS AS PORTFOLIO_UNITS, --投资组合
T1.I_CODE AS SECURITY_ID, --证券代码,给流水号
T1.I_CODE AS SECURITY_NAME, --证券名称,给流水号
nvl(T20.TARGET_CODE,‘OTHER‘) AS MARKET_NO, --市场代码
nvl(T20.TARGET_NAME,‘其他‘) AS MARKET_NAME, --市场名称
oc.partyname AS COUNTER_PARTY, --交易对手名称
T1.partyid AS COUNTER_PARTY_ID, --交易对手组织机构代码
T2.AMOUNT AS HOLDING_UNITS, --持仓数量PAR_VALUE
T2.AMOUNT AS BOOK_BALANCE_N, --财务账面余额_本币AMOUNT
T2.AMOUNT AS BOOK_BALANCE_B, --财务账面余额_本币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_N, --参考财务成本_原币AMOUNT
T2.AMOUNT AS FINANCIAL_COST_ACCOUNT_B, --参考财务成本_本币AMOUNT
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE(‘2017-06-30‘),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_N, --应计利息_原币
round(ABS((T2.AIAMOUNT / T2.TERM * ((datediff(TO_DATE(‘2017-06-30‘),TO_DATE(T2.FSTSETDATE)))+1)) ), 2) ACCRUE_INTEREST_B --应计利息_本币
FROM bridge.FBMS_A_TTRD_OTC_TRADE_IB T1
INNER JOIN bridge.FBMS_A_TTRD_OTC_IBLB_IB T2 ON T2.I_CODE = T1.INTORDID
AND T2.M_TYPE = T1.M_TYPE
AND T2.A_TYPE = T1.A_TYPE
AND T2.ENDSETDATE > TO_DATE(‘2017-06-30‘) --投资组合映射主表
left join (select v1.accountid, v2.murex_leaf, v2.Third_Business_Line as business_line, v2.portfolio_units, v2.fund_source, v2.account_class
from bridge.BRIDGE_APP_PORTFOLIOACCOUNT_ib v1
inner join bridge.bridge_murex_leaf_config v2 on v1.portfolioid = v2.murex_leaf
and ascii(v2.is_send_group) = -26
and v1.biz_date = v2.biz_date
where ‘20170630‘ = v2.biz_date) p on p.accountid = t1.secu_accid
--业务线
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘BUSINESS_LINE‘ and ‘20170630‘ = biz_date) p1 on p.business_line = p1.chinese_name
--资金来源
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘FUND_SOURCE‘ and ‘20170630‘ = biz_date) p2 on p.fund_source = p2.chinese_name
--会计分类
left join (select col_name, tabname_cn as chinese_name, tabno as mapping_key from bridge.bridge_murex_leaf_mapping_ib where col_name = ‘ACCOUNT_CLASS‘ and ‘20170630‘ = biz_date) p3 on p.account_class = p3.chinese_name
--交易对手(交易对手SUMMIT使用的是衡泰的数据源，衡泰是从交易中心获取的，可以通用)
left join (select max(partyname) as partyname, partyid from bridge.FBMS_A_TTRD_OTC_COUNTERPARTY_IB where ‘20170630‘ = biz_date group by partyid) oc on t1.partyid = oc.partyid

left join bridge.bridge_security_to_cio_mapping_ib T20
on T1.M_TYPE = T20.SOURCE_CODE
and T20.SOURCE_SYSTEM_CODE = ‘FBMS‘
and T20.MAPPING_TYPE =‘MARKET_NO‘
and T20.SOURCE_COLUMN_NAME = ‘M_TYPE‘
and T20.SOURCE_TABLE_NAME = ‘FBMS_A_TTRD_OTC_TRADE_IB‘
and T20.biz_date=‘20170630‘
WHERE T1.ORDSOURCE NOT IN (‘4‘, ‘11‘) --过滤掉投顾指令单
AND T1.ORDSTATUS = ‘7‘
AND T1.A_TYPE in (‘SPT_IBOR‘)
AND T1.M_TYPE = ‘X_CNBD‘

17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #311
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #311
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [IPC Parameter Sending Thread #0]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge sending #312
17/11/18 16:55:51 [IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge]: DEBUG ipc.Client: IPC Client (1990421361) connection to cnsz131375/10.25.167.173:8020 from bridge got value #312
17/11/18 16:55:51 [main]: DEBUG ipc.ProtobufRpcEngine: Call: delete took 1ms
17/11/18 16:55:51 [main]: INFO log.PerfLogger: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
17/11/18 16:55:51 [main]: INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1510995351625 end=1510995351625 duration=0 from=org.apache.hadoop.hive.ql.Driver>
hive> 17/11/18 16:56:11 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 executed
17/11/18 16:56:13 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 expired
17/11/18 16:56:13 [LeaseRenewer:bridge@mycluster]: DEBUG hdfs.LeaseRenewer: Lease renewer daemon for [] with renew id 1 exited

hive bug

标签：stage   shu   nis   soc   its   parameter   ase   sub   ==   

原文地址：http://www.cnblogs.com/qiuhong10/p/7842996.html
踩
(0)
赞
(0)
   
举报
评论 一句话评论（0）
登录后才能评论！
分享档案
更多>
2021年07月29日 (22)
2021年07月28日 (40)
2021年07月27日 (32)
2021年07月26日 (79)
2021年07月23日 (29)
2021年07月22日 (30)
2021年07月21日 (42)
2021年07月20日 (16)
2021年07月19日 (90)
2021年07月16日 (35)
周排行
mamicode.com排行更多图片
更多
友情链接
兰亭集智   国之画   百度统计   站长统计   阿里云   chrome插件   新版天听网
关于我们 - 联系我们 - 留言反馈
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码！