博客

执行hsql :

select user_id, prod_id, size(sw_prod) as item_cnt
from i_order_prod_log lateral view explode(sw_prod) prodTable as prod_id;

发生下面的错误：

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"user_id":"33333333333333309035","plat_user_id":"","order_id":"","order_no":"","prod_no":"","prod_name":"","prod_price":null,"prod_cnt":null,"orig_prod_type":null,"prod_type_name":"","order_price":null,"camp_id":null,"oper_time":"1970-01-01 08:00:00","coll_type":null,"ip":"","page_url":"","refer_url":"","prod_type":null,"sw_prod":["10605"],"dstc_id":null,"domain_name":"","refer_domain_name":"","day_id":"20120903","plat_type":"3","inf_id":"305"}

	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
	at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"user_id":"33333333333333309035","plat_user_id":"","order_id":"","order_no":"","prod_no":"","prod_name":"","prod_price":null,"prod_cnt":null,"orig_prod_type":null,"prod_type_name":"","order_price":null,"camp_id":null,"oper_time":"1970-01-01 08:00:00","coll_type":null,"ip":"","page_url":"","refer_url":"","prod_type":null,"sw_prod":["10605"],"dstc_id":null,"domain_name":"","refer_domain_name":"","day_id":"20120903","plat_type":"3","inf_id":"305"}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548)
	at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
	... 8 more
Caused by: java.lang.IllegalArgumentException: Timestamp format must be yyyy-mm-dd hh:mm:ss[.fffffffff]
	at java.sql.Timestamp.valueOf(Timestamp.java:185)
	at org.apache.hadoop.hive.serde2.lazy.LazyTimestamp.init(LazyTimestamp.java:74)
	at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:109)
	at org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getFieldsAsList(ColumnarStructBase.java:231)
	at org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldsDataAsList(ColumnarStructObjectInspector.java:168)
	at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(UnionStructObjectInspector.java:148)
	at org.apache.hadoop.hive.ql.exec.LateralViewJoinOperator.processOp(LateralViewJoinOperator.java:128)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:70)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.LateralViewForwardOperator.processOp(LateralViewForwardOperator.java:37)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531)

	... 9 more

分析原因：

看似是时间戳问题，但该hsql中根本不涉及到时间戳字段；生产环境测试没有问题，测试环境不行。后查配置发现有多一个配置项

 <property>
  <name>hive.input.format</name>
  <value>org.apache.hadoop.hive.ql.io.HiveInputFormat</value>

 </property>

注释掉可成功运行。