都提示了呀 Causedby:java.lang.IllegalArgumentException:java.net.UnknownHostException:hive这个hive应该是mysql里面的hive数据库,就是不知道怎么配置这个值
跟hive-site.xml的配置有关,应该是哪里配置错了,把catalog关了试试回复
@sparkman:hive-site.xml贴出来看看刚配置好hive的时候是可以启动hive的,就是在整合spark和hive后,启动hive才报错,我推测应该是哪个参数没有配置,就是不知道怎么配置
http://t.cn/RKsEOy2
最终问题解决了,是因为在配置hive-site.xml时配置参数错了,
hive.exec.scratchdir hdfs://
master:9000/data/hive/temp
hive.metastore.warehouse.dir
hdfs://
master:9000/data/hive/warehouse
locationofdefaultdatabaseforthewarehouse
刚开始这两个参数没有给出master的地址和端口所以造成无法识别 /hive文件,这里的unknowhosthive并不是指hive元数据库,而是配置的/hive文件
你好按照你上面的修改出现下面的问题,请问你遇到过吗
scala>[hadoop@worker01bin]$./spark-shellSettingdefaultloglevelto”WARN”.Toadjustlogginglevelusesc.setLogLevel(newLevel).ForSparkR,usesetLogLevel(newLevel).17/09/2510:43:26WARNSparkContext:SupportforJava7isdeprecatedasofSpark2.0.017/09/2510:43:29WARNNativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform…usingbuiltin-javaclasseswhereapplicablejava.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveSessionState’: atorg.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981) atorg.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) atorg.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) atorg.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) atorg.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878) atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) atscala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) atscala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230) atscala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40) atscala.collection.mutable.HashMap.foreach(HashMap.scala:99) atorg.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878) atorg.apache.spark.repl.Main$.createSparkSession(Main.scala:95) …47elidedCausedby:java.lang.reflect.InvocationTargetException:java.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveExternalCatalog’: atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978) …58moreCausedby:java.lang.IllegalArgumentException:Errorwhileinstantiating’org.apache.spark.sql.hive.HiveExternalCatalog’: atorg.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169) atorg.apache.spark.sql.internal.SharedState.(SharedState.scala:86) atorg.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) atorg.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101) atscala.Option.getOrElse(Option.scala:121) atorg.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101) atorg.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100) atorg.apache.spark.sql.internal.SessionState.(SessionState.scala:157) atorg.apache.spark.sql.hive.HiveSessionState.(HiveSessionState.scala:32) …63moreCausedby:java.lang.reflect.InvocationTargetException:java.lang.reflect.InvocationTargetException:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166) …71moreCausedby:java.lang.reflect.InvocationTargetException:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atsun.reflect.NativeConstructorAccessorImpl.newInstance0(NativeMethod) atsun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) atsun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) atjava.lang.reflect.Constructor.newInstance(Constructor.java:526) atorg.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264) atorg.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:366) atorg.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:270) atorg.apache.spark.sql.hive.HiveExternalCatalog.(HiveExternalCatalog.scala:65) …76moreCausedby:java.lang.RuntimeException:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) atorg.apache.spark.sql.hive.client.HiveClientImpl.(HiveClientImpl.scala:192) …84moreCausedby:java.lang.RuntimeException:Therootscratchdir:/tmp/hiveonHDFSshouldbewritable.Currentpermissionsare:rwxrwxr-x atorg.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612) atorg.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554) atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508) …85more:14:error:notfound:valuespark importspark.implicits._ ^:14:error:notfound:valuespark importspark.sql ^Welcometo ____ __ /__/__ ________//__ _\\/_\/_`/__/ ‘_/ /___/.__/\_,_/_//_/\_\ version2.1.0 /_/ UsingScalaversion2.11.8(JavaHotSpot(TM)64-BitServerVM,Java1.7.0_79)Typeinexpressionstohavethemevaluated.Type:helpformoreinformation.