博客

Hfile在hbase表中的底层数据形式：默认default下面

1、Can’t get master address from ZooKeeper; znode data == null

hbase(main):001:0> list

TABLE

ERROR: Can’t get master address from ZooKeeper; znode data == null

Here is some help for this command:

List all tables in hbase. Optional regular expression parameter could

be used to filter the output. Examples:

hbase> list

hbase> list ‘abc.*’

hbase> list ‘ns:abc.*’

hbase> list ‘ns:.*’

2、LoadIncrementalHFiles:623 – Trying to load hfile=hdfs://

RpcRetryingCaller:132 – Call exception, tries=10, retries=20, started=696043

Caused by:



org.apache.hadoop.security.AccessControlException



: Permission denied: user=hbase, access=WRITE,

inode=”/dop/persist/output/1_20170513232854/f1″:hdfs:supergroup:drwxr-xr-x

解决方法：

http://blog.csdn.net/opensure/article/details/51581460

代码中显示设置权限为777

3、zookeeper.ZooKeeper: Client environment:java.class.path

4、Container id: container_1494491520677_0015_01_000006

Exit code: 1

Stack trace: ExitCodeException exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)

at org.apache.hadoop.util.Shell.run(Shell.java:504)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

http://blog.csdn.net/wjcquking/article/details/41242625

5、同样的程序本地和hadoop环境上跑结果不一样

eclipse中：

Linux中数值和eclipse中不一致！

解决方法：主程序中的变量不能带到mr中，只能通过conf方式

6、

17/05/19 10:40:25 INFO mapreduce.HFileOutputFormat2: Looking up current regions for table tel_rec5

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=21, exceptions:

Fri May 19 10:41:13 CST 2017, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=68513: row ‘tel_rec5,,00000000000000’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname=cdh129136,60020,1494926555927, seqNum=0

at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:270)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:225)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:63)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)

at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)

at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)

at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:161)

at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:156)

at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)

at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:187)

at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:89)

at org.apache.hadoop.hbase.client.MetaScanner.listTableRegionLocations(MetaScanner.java:334)

at org.apache.hadoop.hbase.client.HTable.listRegionLocations(HTable.java:702)

at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:685)

at org.apache.hadoop.hbase.client.HTable.getStartKeys(HTable.java:668)

at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.getRegionStartKeys(HFileOutputFormat2.java:297)

at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:416)

at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:386)

at com.tydic.persist.driver.BulkLoadDriver.run(Unknown Source)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at com.tydic.persist.driver.BulkLoadDriver.bulkLoadRun(Unknown Source)

at com.tydic.entrance.GetRuleConf.execute(Unknown Source)

at com.tydic.entrance.HFileGeneratorEntrance.main(Unknown Source)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=68513: row ‘tel_rec5,,00000000000000’ on table ‘hbase:meta’ at region=hbase:meta,,1.1588230740, hostname=cdh129136,60020,1494926555927, seqNum=0

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)

at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to cdh129136/192.168.129.136:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to cdh129136/192.168.129.136:60020 is closing. Call id=9, waitTime=3

at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1252)

at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1223)

at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)

at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651)

at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:372)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:199)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:371)

at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:345)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)

… 4 more

Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to cdh129136/192.168.129.136:60020 is closing. Call id=9, waitTime=3

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1042)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:849)

at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:568)

7、

[19 20:40:03,040 INFO ] org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2 – Looking up current regions for table tel_rec5

[19 20:40:03,280 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:07,891 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:08,046 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:09,599 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:11,093 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:12,399 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Failed to specify server’s Kerberos principal name

[19 20:40:12,400 WARN ] org.apache.hadoop.hbase.ipc.AbstractRpcClient – Couldn’t setup connection for hbase/cdh129130@MYCDH to null

[19 20:40:12,400 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Couldn’t setup connection for hbase/cdh129130@MYCDH to null

[19 20:40:14,467 WARN ] o

解决方法：


configuration


.set(


“hbase.regionserver.kerberos.principal”


,


dynamicPrincipal


);

configuration


.set(


”


hbase.master.kerberos.principal


”


,


dynamicPrincipal


);

8、

[19 22:36:56,639 WARN ] org.apache.hadoop.security.UserGroupInformation –

PriviledgedActionException

as:hbase/cdh129130@MYCDH (auth:KERBEROS)

cause:java.io.IOException

: Can’t get Master Kerberos principal for use as renewer

java.io.IOException

: Can’t get Master Kerberos principal for use as renewer

at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(

TokenCache.java:133

)

at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(

TokenCache.java:100

)

at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(

TokenCache.java:80

)

at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(

FileOutputFormat.java:142

)

at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(

JobSubmitter.java:270

)

at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(

JobSubmitter.java:143

)

at org.apache.hadoop.mapreduce.Job$10.run(

Job.java:1307

)

at org.apache.hadoop.mapreduce.Job$10.run(

Job.java:1304

)

at java.security.AccessController.doPrivileged(

Native Method

)

at javax.security.auth.Subject.doAs(

Subject.java:415

)

at org.apache.hadoop.security.UserGroupInformation.doAs(

UserGroupInformation.java:1707

)

at org.apache.hadoop.mapreduce.Job.submit(

Job.java:1304

)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(

Job.java:1325

)

at com.tydic.persist.driver.BulkLoadDriver.run(

BulkLoadDriver.java:204

)

at org.apache.hadoop.util.ToolRunner.run(

ToolRunner.java:70

)

at com.tydic.persist.driver.BulkLoadDriver.bulkLoadRun(

BulkLoadDriver.java:76

)

at com.tydic.entrance.GetRuleConf.execute(

GetRuleConf.java:99

)

at com.tydic.entrance.HFileGeneratorEntrance.main(

HFileGeneratorEntrance.java:31

)

解决方法：

configuration


.set(


“yarn.resourcemanager.principal”


,


“yarn/_HOST@MYCDH”


);

或者增加yarn的配置文件

9、

For more detailed output, check application tracking page:http://cdh129130:8088/proxy/application_1495209831244_0002/Then, click on links to logs of each attempt.

Diagnostics: Exception from container-launch.

Container id: container_1495209831244_0002_02_000001

Exit code: 1

Stack trace:


ExitCodeException


exitCode=1:

at org.apache.hadoop.util.Shell.runCommand(


Shell.java:543


)

at org.apache.hadoop.util.Shell.run(


Shell.java:460


)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(


Shell.java:720


)

at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(


LinuxContainerExecutor.java:293


)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(


ContainerLaunch.java:302


)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(


ContainerLaunch.java:82


)

at java.util.concurrent.FutureTask.run(


FutureTask.java:262


)

at java.util.concurrent.ThreadPoolExecutor.runWorker(


ThreadPoolExecutor.java:1145


)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(


ThreadPoolExecutor.java:615


)

at java.lang.Thread.run(


Thread.java:745


)

Shell output: main : command provided 1

main : user is hbase

main : requested yarn user is hbase

解决方法：进入yarn界面查询具体job执行日志，发现是job.xml解析问题

hbase连接可能有问题！！！

10、

11、

[20 15:59:13,273 WARN ] org.apache.hadoop.mapred.LocalJobRunner – job_local199199413_0001

java.lang.Exception

: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1

at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(

LocalJobRunner.java:462

)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(

LocalJobRunner.java:529

)

Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1

at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(

Shuffle.java:134

)

at org.apache.hadoop.mapred.ReduceTask.run(

ReduceTask.java:376

)

at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(

LocalJobRunner.java:319

)

at java.util.concurrent.Executors$RunnableAdapter.call(

Executors.java:471

)

at java.util.concurrent.FutureTask.run(

FutureTask.java:262

)

at java.util.concurrent.ThreadPoolExecutor.runWorker(

ThreadPoolExecutor.java:1145

)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(

ThreadPoolExecutor.java:615

)

at java.lang.Thread.run(

Thread.java:744

)

Caused by: java.lang.ExceptionInInitializerError

at org.apache.hadoop.mapred.SpillRecord.<init>(

SpillRecord.java:70

)

at org.apache.hadoop.mapred.SpillRecord.<init>(

SpillRecord.java:62

)

at org.apache.hadoop.mapred.SpillRecord.<init>(

SpillRecord.java:57

)

at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(

LocalFetcher.java:124

)

at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(

LocalFetcher.java:102

)

at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(

LocalFetcher.java:85

)

Caused by:

java.lang.RuntimeException

: Secure IO is not possible without native code extensions.

at org.apache.hadoop.io.SecureIOUtils.<clinit>(

SecureIOUtils.java:75

)

… 6 more

解决方法：

LocalJobRunner本地eclipse跑的任务报，打成jar放到hadoop环境上正常

12、hive分区表如何处理mr

13、


Wrong FS:

java.lang.IllegalArgumentException



: Wrong FS: hdfs://cdh129130:8020/tmp/dop/persist/output/tel_rec_part_4/20170525091224, expected: file:///

at org.apache.hadoop.fs.FileSystem.checkPath(



FileSystem.java:654



)

at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(



RawLocalFileSystem.java:80



)

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(



RawLocalFileSystem.java:529



)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(



RawLocalFileSystem.java:747



)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(



RawLocalFileSystem.java:524



)

at org.apache.hadoop.fs.ChecksumFileSystem.delete(



ChecksumFileSystem.java:534



)

at com.tydic.utils.HdfsUtil.delete(



HdfsUtil.java:452



)

at com.tydic.persist.driver.BulkLoadDriver.deleteHdfsPath(



BulkLoadDriver.java:210



)

at com.tydic.persist.driver.BulkLoadDriver.run(



BulkLoadDriver.java:147



)

at org.apache.hadoop.util.ToolRunner.run(



ToolRunner.java:70



)

at com.tydic.persist.driver.BulkLoadDriver.bulkLoadRun(



BulkLoadDriver.java:73



)

at com.tydic.entrance.GetRuleConf.execute(



GetRuleConf.java:86



)

at com.tydic.entrance.HFileGeneratorEntrance.main(



HFileGeneratorEntrance.java:29



)

解决方法：下载核心配置文件 hdfs-site.xml、core-site.xml

或代码显示指定

Configuration

config

=


new


Configuration();

if


(!

“”

.equals(PropertiesUtil.

getValue

(

“fs.defaultFS”

))){

config

.set(

“fs.defaultFS”

, PropertiesUtil.

getValue

(

“fs.defaultFS”

));

}

14、

[03 19:46:29,270 INFO ] org.apache.hadoop.mapreduce.JobSubmitter – Kind: HBASE_AUTH_TOKEN, Service: 55f3581e-b84a-49b9-84b3-8998de8db847, Ident: (org.apache.hadoop.hbase.security.token.AuthenticationTokenIdentifier@29)

[Fatal Error] job.xml:65:50: 字符引用 “&#

[03 19:46:29,354 INFO ] org.apache.hadoop.mapreduce.JobSubmitter – Cleaning up the staging area file:/tmp/hadoop-hdfs/mapred/staging/eda1250935355/.staging/job_local1250935355_0001

java.lang.RuntimeException



: org.xml.sax.SAXParseException; systemId: file:/tmp/hadoop-hdfs/mapred/staging/eda1250935355/.staging/job_local1250935355_0001/job.xml; lineNumber: 65; columnNumber: 50; 字符引用 “&#

at org.apache.hadoop.conf.Configuration.loadResource(



Configuration.java:2656



)

at org.apache.hadoop.conf.Configuration.loadResources(



Configuration.java:2513



)

at org.apache.hadoop.conf.Configuration.getProps(



Configuration.java:2409



)

at org.apache.hadoop.conf.Configuration.get(



Configuration.java:982



)

at org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(



JobConf.java:2032



)

at org.apache.hadoop.mapred.JobConf.<init>(



JobConf.java:484



)

at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(



LocalJobRunner.java:153



)

at org.apache.hadoop.mapred.LocalJobRunner.submitJob(



LocalJobRunner.java:731



)

at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(



JobSubmitter.java:244



)

at org.apache.hadoop.mapreduce.Job$10.run(



Job.java:1307



)

at org.apache.hadoop.mapreduce.Job$10.run(



Job.java:1304



)

at java.security.AccessController.doPrivileged(



Native Method



)

at javax.security.auth.Subject.doAs(



Subject.java:415



)

at org.apache.hadoop.security.UserGroupInformation.doAs(



UserGroupInformation.java:1707



)

at org.apache.hadoop.mapreduce.Job.submit(



Job.java:1304



)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(



Job.java:1325



)

at com.tydic.persist.driver.BulkLoadDriver.run(



BulkLoadDriver.java:172



)

at org.apache.hadoop.util.ToolRunner.run(



ToolRunner.java:70



)

at com.tydic.persist.driver.BulkLoadDriver.bulkLoadRun(



BulkLoadDriver.java:74



)

at com.tydic.entrance.GetRuleConf.execute(



GetRuleConf.java:104



)

at com.tydic.entrance.HFileGeneratorEntrance.main(



HFileGeneratorEntrance.java:34



)

Caused by: org.xml.sax.SAXParseException; systemId: file:/tmp/hadoop-hdfs/mapred/staging/eda1250935355/.staging/job_local1250935355_0001/job.xml; lineNumber: 65; columnNumber: 50; 字符引用 “&#

at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(



DOMParser.java:257



)

at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(



DocumentBuilderImpl.java:347



)

at javax.xml.parsers.DocumentBuilder.parse(



DocumentBuilder.java:150



)

at org.apache.hadoop.conf.Configuration.parse(



Configuration.java:2491



)

at org.apache.hadoop.conf.Configuration.loadResource(



Configuration.java:2560



)

… 20 more

解决方法：job.xml格式问题：传递参数有误\001，应该写成\\001

15、

17/06/22 14:26:03 INFO client.RpcRetryingCaller: Call exception, tries=10, retries=20, started=68231 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

17/06/22 14:26:23 INFO client.RpcRetryingCaller: Call exception, tries=11, retries=20, started=88394 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

17/06/22 14:26:43 INFO client.RpcRetryingCaller: Call exception, tries=12, retries=20, started=108503 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

17/06/22 14:27:03 INFO client.RpcRetryingCaller: Call exception, tries=13, retries=20, started=128617 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

17/06/22 14:27:23 INFO client.RpcRetryingCaller: Call exception, tries=14, retries=20, started=148763 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

17/06/22 14:27:44 INFO client.RpcRetryingCaller: Call exception, tries=15, retries=20, started=168968 ms ago, cancelled=false, msg=row ” on table ‘test’ at region=test,,1498042202011.6ab53ace6e83876695e0458191c2d5e7., hostname=nm-304-sa5212m4-bigdata-219,60020,1495677223107, seqNum=2

Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /user/tydic/persist/output/test/20170622103558/F1/78a234dbc4ce42d9a4120682c465a6f4

at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)

at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)

at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)

at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)

at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source)

at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254)

at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)

at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

at com.sun.proxy.$Proxy16.getBlockLocations(Unknown Source)

at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)

at com.sun.proxy.$Proxy17.getBlockLocations(Unknown Source)

at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213)

… 25 more

at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1219)

at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216)

at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300)

at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.bulkLoadHFile(ClientProtos.java:32663)

at org.apache.hadoop.hbase.protobuf.ProtobufUtil.bulkLoadHFile(ProtobufUtil.java:1607)

… 10 more

解决方法：

查RegionServer的log

15、内存溢出

解决方法：

降低mapreduce.reduce.shuffle.memory.limit.percent这个参数

16、Exception in thread “main” java.io.IOException: Trying to load more than 32 hfiles to one family of one region

at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:377)

at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:960)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:967)

解决方案：

1) You can change the program by adding following line after setting zookeeper client port.

configuration.setInt(LoadIncrementalHFiles.MAX_FILES_PER_REGION_PER_FAMILY,64);

2) you can add this property to custom configs through ambari and restart the cluster so new configs take affect

3) you can add this property directly to /etc/conf/hbase-site.xml in the machine where you are running the job so that you need not change the program everytime.

<property>

<name>hbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily</name>

<value>64</value>

</property>

hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles

-Dhbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily=1024 <SOURCE> <TABLE>

17、hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles

hdfs://NM-304-SA5212M4-BIGDATA-221:54310/tmp-dike/app_mbl_user_family_cir_info_m/20170702205018

app_mbl_user_family_cir_info_m

解决方法：


逐个文件load到hbase中