hdfs是java开发的吗_java整合hdfs

  • Post author:
  • Post category:java


都提示了呀 Causedby:java.lang.IllegalArgumentException:java.net.UnknownHostException:hive这个hive应该是mysql里面的hive数据库,就是不知道怎么配置这个值

跟hive-site.xml的配置有关,应该是哪里配置错了,把catalog关了试试回复

@sparkman:hive-site.xml贴出来看看刚配置好hive的时候是可以启动hive的,就是在整合spark和hive后,启动hive才报错,我推测应该是哪个参数没有配置,就是不知道怎么配置

http://t.cn/RKsEOy2

最终问题解决了,是因为在配置hive-site.xml时配置参数错了,

hive.exec.scratchdir  hdfs://

master:9000/data/hive/temp

hive.metastore.warehouse.dir

hdfs://

master:9000/data/hive/warehouse

locationofdefaultdatabaseforthewarehouse

刚开始这两个参数没有给出master的地址和端口所以造成无法识别 /hive文件,这里的unknowhosthive并不是指hive元数据库,而是配置的/hive文件

你好按照你上面的修改出现下面的问题,请问你遇到过吗

scala>[hadoop@worker01bin]$./spark-shellSettingdefaultloglevelto”WARN”.Toadjustlogginglevelusesc.setLogLevel(newLevel).ForSparkR,usesetLogLevel(newLevel).17/09/2510:43:26WARNSparkContext:SupportforJava7isdeprecatedasofSpark2.0.017/09/2510:43:29WARNNativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform…usingbuiltin-javaclasseswhereapplicablejava.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveSessionState’: atorg.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981) atorg.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) atorg.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) atorg.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) atorg.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) atscala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) atscala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) atscala.collection.mutable.HashMap.foreach(HashMap.scala:99) atorg.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878) atorg.apache.spark.repl.Main$.createSparkSession(Main.scala:95) …47elidedCausedby:java.lang.reflect.InvocationTargetException:java.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveExternalCatalog’: atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978) …58moreCausedby:java.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveExternalCatalog’: atorg.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169) atorg.apache.spark.sql.internal.SharedState.(SharedState.scala:86) atorg.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) atorg.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) atscala.Option.getOrElse(Option.scala:121) atorg.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101) atorg.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100) atorg.apache.spark.sql.internal.SessionState.(SessionState.scala:157) atorg.apache.spark.sql.hive.HiveSessionState.(HiveSessionState.scala:32) …63moreCausedby:java.lang.reflect.InvocationTargetException:java.lang.reflect.InvocationTargetException:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166) …71moreCausedby:java.lang.reflect.InvocationTargetException:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264) atorg.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:366) atorg.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:270) atorg.apache.spark.sql.hive.HiveExternalCatalog.(HiveExternalCatalog.scala:65) …76moreCausedby:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) atorg.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:192) …84moreCausedby:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atorg.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612) atorg.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) …85more:14:error:notfound:valuespark    importspark.implicits._       ^:14:error:notfound:valuespark    importspark.sql       ^Welcometo   ____       __   /__/__ ________//__  _\\/_\/_`/__/ ‘_/  /___/.__/\_,_/_//_/\_\ version2.1.0   /_/     UsingScalaversion2.11.8(JavaHotSpot(TM)64-BitServerVM,Java1.7.0_79)Typeinexpressionstohavethemevaluated.Type:helpformoreinformation.



版权声明:本文为weixin_42514750原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。