【spark】java.io.NotSerializableException: org.apache.spark.unsafe.types.UTF8String$IntWrapper

关键词：

java.io.NotSerializableException: org.apache.spark.unsafe.types.UTF8String$IntWrapper

spark-2.2.0

Caused by: java.io.NotSerializableException: org.apache.spark.unsafe.types.UTF8String$IntWrapper
Serialization stack:
    - object not serializable (class: org.apache.spark.unsafe.types.UTF8String$IntWrapper, value: org.apache.spark.unsafe.types.UTF8String$IntWrapper@326450e)
    - field (class: org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToInt$1, name: result$2, type: class org.apache.spark.unsafe.types.UTF8String$IntWrapper)
    - object (class org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToInt$1, <function1>)

Caused by: java.io.NotSerializableException: org.apache.spark.unsafe.types.UTF8String$IntWrapper

Serialization stack:

- object not serializable (class: org.apache.spark.unsafe.types.UTF8String$IntWrapper, value: org.apache.spark.unsafe.types.UTF8String$IntWrapper@326450e)

- field (class: org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToInt$1, name: result$2, type: class org.apache.spark.unsafe.types.UTF8String$IntWrapper)

- object (class org.apache.spark.sql.catalyst.expressions.Cast$$anonfun$castToInt$1, <function1>)

在使用本机部署的spark-2.2.0时，

# 报错
insert overwrite table mart_waimai_risk.yangsen07_fact_wm_ps__order_test_10000 partition(dt) select id,wm_poi_id,dt from mart_waimai_risk.fact_wm_ps__order
where dt=20171024 limit 10000

# 可正确执行
insert overwrite table mart_waimai_risk.yangsen07_fact_wm_ps__order_test_10000 partition(dt) select id,wm_poi_id,dt from mart_waimai_risk.fact_wm_ps__order
where  limit 10000

# 去掉dt分区就能正确执行

# 报错

insert overwrite table mart_waimai_risk.yangsen07_fact_wm_ps__order_test_10000 partition(dt) select id,wm_poi_id,dt from mart_waimai_risk.fact_wm_ps__order

where dt=20171024 limit 10000

# 可正确执行

insert overwrite table mart_waimai_risk.yangsen07_fact_wm_ps__order_test_10000 partition(dt) select id,wm_poi_id,dt from mart_waimai_risk.fact_wm_ps__order

where limit 10000

# 去掉dt分区就能正确执行

折腾了很长时间啊，本地的系统又重装一遍都没好。

最后发现，这是个系统bug，在2017.7.18修复。

https://github.com/apache/spark/pull/18660#issue-243560413

而spark-2.2.0的发布版是2017.7.11。

最后，

退回spark-2.1.2，问题就没有了。