码迷,mamicode.com
首页 > 编程语言 > 详细

java.lang.NullPointerException Ignoring exception during close for [email protected]

时间:2020-02-19 01:00:54      阅读:31      评论:0      收藏:0      [点我收藏+]

标签:cer   mapr   没有   targe   inpu   pac   tput   cal   let   

1.问题描述:在调试mapreduce辅助排序(二次排序)的过程中,运行程序总是报错

Ignoring exception during close for [email protected]
java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2021)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8860 [Thread-6] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
8872 [Thread-6] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1887386767_0001
java.lang.Exception: java.lang.NullPointerException
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:157)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1269)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:74)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1491)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

 

2.原因分析:

(1)初步判断是是调用了空指针;

(2)当点击WritableComparator.java:157 会跳转到报错的地方是this.buffer.reset(b1, s1, l1);

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {
try {
this.buffer.reset(b1, s1, l1);
this.key1.readFields(this.buffer);
this.buffer.reset(b2, s2, l2);
this.key2.readFields(this.buffer);
this.buffer.reset((byte[])null, 0, 0);
} catch (IOException var8) {
throw new RuntimeException(var8);
}
(3)也就是说WritableComparator buffer没有初始化
public class WritableComparator implements RawComparator, Configurable {
private static final ConcurrentHashMap<Class, WritableComparator> comparators = new ConcurrentHashMap();
private Configuration conf;
private final Class<? extends WritableComparable> keyClass;
private final WritableComparable key1;
private final WritableComparable key2;
private final DataInputBuffer buffer;

(4)查找程序中使用到WritableComparator 相关的类的定义是分组类ReducerGroupingComparator 和第一字段的比较类FirstSecondComparator 没有默认的构造函数
public static class ReducerGroupingComparator  extends WritableComparator
{//一定要加默认构造函数,且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for [email protected]
public ReducerGroupingComparator()
{
super(IntPair.class,true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
IntPair pairA=(IntPair)a;
IntPair pairB=(IntPair)b;
//年份相同,返回值为0的分为一组
return (int) (pairA.getFirst()-pairB.getFirst());
}
}
//与FirstSecondComparator中定义的默认对比函数功能相同,这个用于显示设置对比类
public static class FirstSecondComparator extends WritableComparator
{
//
//一定要加默认构造函数,且一定要加super,否则会报异常java.lang.NullPointerException Ignoring exception during close for [email protected]
    public FirstSecondComparator()
{
super(IntPair.class,true);
}
@Override
public int compare(WritableComparable a, WritableComparable b) {
IntPair pairA=(IntPair)a;
IntPair pairB=(IntPair)b;
int cmp= (int) (pairA.getFirst()-pairA.getFirst());
if (cmp!=0)
{
return cmp;
}
double A=pairA.getSecond();
double B=pairB.getSecond();
return -((A==B)? 0:(A>B? 1:-1));
}
}
(5)为什么一定要加这个才不报java.lang.NullPointerException异常?点开super(IntPair.class,true);
会跳转到WritableComparator的构造函数
protected WritableComparator(Class<? extends WritableComparable> keyClass, boolean createInstances) {
this(keyClass, (Configuration)null, createInstances);
}
在点击this,进一步跳转到另外一个重载构造函数,看到这里大家应该名明白了,上面报异常的地方,就是这些变量没有申请内存空间
protected WritableComparator(Class<? extends WritableComparable> keyClass, Configuration conf, boolean createInstances) {
this.keyClass = keyClass;
this.conf = conf != null ? conf : new Configuration();
if (createInstances) {
this.key1 = this.newKey();
this.key2 = this.newKey();
this.buffer = new DataInputBuffer();
} else {
this.key1 = this.key2 = null;
this.buffer = null;
}


}

自己开发了一个股票智能分析软件,功能很强大,需要的点击下面的链接获取:

https://www.cnblogs.com/bclshuai/p/11380657.html



 

java.lang.NullPointerException Ignoring exception during close for [email protected]

标签:cer   mapr   没有   targe   inpu   pac   tput   cal   let   

原文地址:https://www.cnblogs.com/bclshuai/p/12329390.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有 京ICP备13008772号-2
迷上了代码!