码迷,mamicode.com
首页 > 其他好文 > 详细

spark streaming优化:spark.default.parallelism调整处理并行度

时间:2015-05-14 11:36:56      阅读:359      评论:0      收藏:0      [点我收藏+]

标签:

官方是这么说的:

Cluster resources can be under-utilized if the number of parallel tasks used in any stage of the computation is not high enough. For example, for distributed reduce operations like reduceByKey and reduceByKeyAndWindow, the default number of parallel tasks is controlled by thespark.default.parallelism configuration property. You can pass the level of parallelism as an argument (see PairDStreamFunctions documentation), or set the spark.default.parallelism configuration property to change the default.

我理想:就是你可以调整spark.default.parallelism来修改默认并行度,或者在使用transformation,action方法时直接往方法传入并行度。

spark streaming优化:spark.default.parallelism调整处理并行度

标签:

原文地址:http://www.cnblogs.com/hark0623/p/4502812.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!