文章目录
a.问题报错：
jid=everdc&segmentTxId=145316887&storageInfo=-63%3A575476508%3A0%3ACID-46d57eba-3c5f-413b-bca6-565ab12fc364’ to transaction ID 144656573
2022-10-12 05:39:15,781 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) – Fast-forwarding stream ‘http://everdc6:8480/getJournal?jid=everdc&segmentTxId=145316887&storageInfo=-63%3A575476508%3A0%3ACID-46d57eba-3c5f-413b-bca6-565ab12fc364’ to transaction ID 144656573
2022-10-12 05:39:33,207 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 313/391944 transactions completed. (0%)
2022-10-12 05:39:50,392 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 1254/391944 transactions completed. (0%)
2022-10-12 05:40:07,677 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 6088/391944 transactions completed. (2%)
2022-10-12 05:40:25,628 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 11427/391944 transactions completed. (3%)
2022-10-12 05:40:43,063 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 15076/391944 transactions completed. (4%)
2022-10-12 05:41:00,365 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 18542/391944 transactions completed. (5%)
2022-10-12 05:41:17,768 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 21752/391944 transactions completed. (6%)
2022-10-12 05:41:34,993 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 25132/391944 transactions completed. (6%)
2022-10-12 05:41:56,454 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 28480/391944 transactions completed. (7%)
2022-10-12 05:42:13,742 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 29326/391944 transactions completed. (7%)
2022-10-12 05:42:35,049 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 32871/391944 transactions completed. (8%)
2022-10-12 05:42:52,222 INFO namenode.FSEditLogLoader (FSEditLogLoader.java:loadEditRecords(266)) – replaying edit log: 33456/391944 transactions completed. (9%)
2022-10-12 05:43:13,674 ERROR namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(221)) – Got error reading edit log input stream http://everdc6:8480/getJournal?jid=everdc&segmentTxId=145316887&storageInfo=-63%3A575476508%3A0%3ACID-46d57eba-3c5f-413b-bca6-565ab12fc364; failing over to edit log http://everdc5:8480/getJournal?jid=everdc&segmentTxId=145316887&storageInfo=-63%3A575476508%3A0%3ACID-46d57eba-3c5f-413b-bca6-565ab12fc364
java.io.IOException: got unexpected exception GC overhead limit exceeded
at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp
R
e
a
d
e
r
.
r
e
a
d
O
p
(
F
S
E
d
i
t
L
o
g
O
p
.
j
a
v
a
:
4600
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
t
S
t
r
e
a
m
.
n
e
x
t
O
p
I
m
p
l
(
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
203
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
t
S
t
r
e
a
m
.
n
e
x
t
O
p
(
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
250
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
r
e
a
d
O
p
(
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
85
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
R
e
d
u
n
d
a
n
t
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
n
e
x
t
O
p
(
R
e
d
u
n
d
a
n
t
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
188
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
r
e
a
d
O
p
(
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
85
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
R
e
d
u
n
d
a
n
t
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
n
e
x
t
O
p
(
R
e
d
u
n
d
a
n
t
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
188
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
r
e
a
d
O
p
(
E
d
i
t
L
o
g
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
85
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
E
d
i
t
L
o
g
L
o
a
d
e
r
.
l
o
a
d
E
d
i
t
R
e
c
o
r
d
s
(
F
S
E
d
i
t
L
o
g
L
o
a
d
e
r
.
j
a
v
a
:
190
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
E
d
i
t
L
o
g
L
o
a
d
e
r
.
l
o
a
d
F
S
E
d
i
t
s
(
F
S
E
d
i
t
L
o
g
L
o
a
d
e
r
.
j
a
v
a
:
143
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
.
l
o
a
d
E
d
i
t
s
(
F
S
I
m
a
g
e
.
j
a
v
a
:
898
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
.
l
o
a
d
F
S
I
m
a
g
e
(
F
S
I
m
a
g
e
.
j
a
v
a
:
753
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
.
r
e
c
o
v
e
r
T
r
a
n
s
i
t
i
o
n
R
e
a
d
(
F
S
I
m
a
g
e
.
j
a
v
a
:
329
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
N
a
m
e
s
y
s
t
e
m
.
l
o
a
d
F
S
I
m
a
g
e
(
F
S
N
a
m
e
s
y
s
t
e
m
.
j
a
v
a
:
983
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
N
a
m
e
s
y
s
t
e
m
.
l
o
a
d
F
r
o
m
D
i
s
k
(
F
S
N
a
m
e
s
y
s
t
e
m
.
j
a
v
a
:
686
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
l
o
a
d
N
a
m
e
s
y
s
t
e
m
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
586
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
i
n
i
t
i
a
l
i
z
e
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
646
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
<
i
n
i
t
>
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
820
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
<
i
n
i
t
>
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
804
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
c
r
e
a
t
e
N
a
m
e
N
o
d
e
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
1516
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
N
a
m
e
N
o
d
e
.
m
a
i
n
(
N
a
m
e
N
o
d
e
.
j
a
v
a
:
1582
)
C
a
u
s
e
d
b
y
:
j
a
v
a
.
l
a
n
g
.
O
u
t
O
f
M
e
m
o
r
y
E
r
r
o
r
:
G
C
o
v
e
r
h
e
a
d
l
i
m
i
t
e
x
c
e
e
d
e
d
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
C
o
d
e
d
I
n
p
u
t
S
t
r
e
a
m
.
<
i
n
i
t
>
(
C
o
d
e
d
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
573
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
C
o
d
e
d
I
n
p
u
t
S
t
r
e
a
m
.
n
e
w
I
n
s
t
a
n
c
e
(
C
o
d
e
d
I
n
p
u
t
S
t
r
e
a
m
.
j
a
v
a
:
55
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
p
a
r
s
e
P
a
r
t
i
a
l
F
r
o
m
(
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
j
a
v
a
:
199
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
p
a
r
s
e
P
a
r
t
i
a
l
D
e
l
i
m
i
t
e
d
F
r
o
m
(
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
j
a
v
a
:
241
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
p
a
r
s
e
D
e
l
i
m
i
t
e
d
F
r
o
m
(
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
j
a
v
a
:
253
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
p
a
r
s
e
D
e
l
i
m
i
t
e
d
F
r
o
m
(
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
j
a
v
a
:
259
)
a
t
c
o
m
.
g
o
o
g
l
e
.
p
r
o
t
o
b
u
f
.
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
p
a
r
s
e
D
e
l
i
m
i
t
e
d
F
r
o
m
(
A
b
s
t
r
a
c
t
P
a
r
s
e
r
.
j
a
v
a
:
49
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
p
r
o
t
o
c
o
l
.
p
r
o
t
o
.
X
A
t
t
r
P
r
o
t
o
s
Reader.readOp(FSEditLogOp.java:4600) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:203) at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:188) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:188) at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:898) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:753) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:329) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:820) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:804) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at com.google.protobuf.CodedInputStream.<init>(CodedInputStream.java:573) at com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:199) at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259) at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49) at org.apache.hadoop.hdfs.protocol.proto.XAttrProtos
R
e
a
d
er
.
re
a
d
Op
(
FSE
d
i
t
L
o
g
Op
.
ja
v
a
:
4600
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
tSt
re
am
.
n
e
x
tOp
I
m
pl
(
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
tSt
re
am
.
ja
v
a
:
203
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
tSt
re
am
.
n
e
x
tOp
(
E
d
i
t
L
o
g
F
i
l
e
I
n
p
u
tSt
re
am
.
ja
v
a
:
250
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
re
a
d
Op
(
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
ja
v
a
:
85
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
R
e
d
u
n
d
an
tE
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
n
e
x
tOp
(
R
e
d
u
n
d
an
tE
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
ja
v
a
:
188
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
re
a
d
Op
(
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
ja
v
a
:
85
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
R
e
d
u
n
d
an
tE
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
n
e
x
tOp
(
R
e
d
u
n
d
an
tE
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
ja
v
a
:
188
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
re
a
d
Op
(
E
d
i
t
L
o
g
I
n
p
u
tSt
re
am
.
ja
v
a
:
85
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSE
d
i
t
L
o
gL
o
a
d
er
.
l
o
a
d
E
d
i
tR
ecor
d
s
(
FSE
d
i
t
L
o
gL
o
a
d
er
.
ja
v
a
:
190
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSE
d
i
t
L
o
gL
o
a
d
er
.
l
o
a
d
FSE
d
i
t
s
(
FSE
d
i
t
L
o
gL
o
a
d
er
.
ja
v
a
:
143
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
.
l
o
a
d
E
d
i
t
s
(
FS
I
ma
g
e
.
ja
v
a
:
898
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
.
l
o
a
d
FS
I
ma
g
e
(
FS
I
ma
g
e
.
ja
v
a
:
753
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
.
reco
v
er
T
r
an
s
i
t
i
o
n
R
e
a
d
(
FS
I
ma
g
e
.
ja
v
a
:
329
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSN
am
esys
t
e
m
.
l
o
a
d
FS
I
ma
g
e
(
FSN
am
esys
t
e
m
.
ja
v
a
:
983
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSN
am
esys
t
e
m
.
l
o
a
d
F
ro
m
D
i
s
k
(
FSN
am
esys
t
e
m
.
ja
v
a
:
686
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
l
o
a
d
N
am
esys
t
e
m
(
N
am
e
N
o
d
e
.
ja
v
a
:
586
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
ini
t
ia
l
i
ze
(
N
am
e
N
o
d
e
.
ja
v
a
:
646
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
<
ini
t
>
(
N
am
e
N
o
d
e
.
ja
v
a
:
820
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
<
ini
t
>
(
N
am
e
N
o
d
e
.
ja
v
a
:
804
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
cre
a
t
e
N
am
e
N
o
d
e
(
N
am
e
N
o
d
e
.
ja
v
a
:
1516
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
N
am
e
N
o
d
e
.
main
(
N
am
e
N
o
d
e
.
ja
v
a
:
1582
)
C
a
u
se
d
b
y
:
ja
v
a
.
l
an
g
.
O
u
tO
f
M
e
m
ory
E
rror
:
GC
o
v
er
h
e
a
d
l
imi
t
e
x
cee
d
e
d
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
C
o
d
e
d
I
n
p
u
tSt
re
am
.
<
ini
t
>
(
C
o
d
e
d
I
n
p
u
tSt
re
am
.
ja
v
a
:
573
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
C
o
d
e
d
I
n
p
u
tSt
re
am
.
n
e
w
I
n
s
t
an
ce
(
C
o
d
e
d
I
n
p
u
tSt
re
am
.
ja
v
a
:
55
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
A
b
s
t
r
a
c
tP
a
rser
.
p
a
rse
P
a
r
t
ia
lF
ro
m
(
A
b
s
t
r
a
c
tP
a
rser
.
ja
v
a
:
199
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
A
b
s
t
r
a
c
tP
a
rser
.
p
a
rse
P
a
r
t
ia
l
De
l
imi
t
e
d
F
ro
m
(
A
b
s
t
r
a
c
tP
a
rser
.
ja
v
a
:
241
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
A
b
s
t
r
a
c
tP
a
rser
.
p
a
rseDe
l
imi
t
e
d
F
ro
m
(
A
b
s
t
r
a
c
tP
a
rser
.
ja
v
a
:
253
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
A
b
s
t
r
a
c
tP
a
rser
.
p
a
rseDe
l
imi
t
e
d
F
ro
m
(
A
b
s
t
r
a
c
tP
a
rser
.
ja
v
a
:
259
)
a
t
co
m
.
g
oo
g
l
e
.
p
ro
t
o
b
u
f
.
A
b
s
t
r
a
c
tP
a
rser
.
p
a
rseDe
l
imi
t
e
d
F
ro
m
(
A
b
s
t
r
a
c
tP
a
rser
.
ja
v
a
:
49
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
p
ro
t
oco
l
.
p
ro
t
o
.
X
A
tt
r
P
ro
t
os
XAttrEditLogProto.parseDelimitedFrom(XAttrProtos.java:1282)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.readXAttrsFromEditLog(FSEditLogOp.java:414)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.access
3800
(
F
S
E
d
i
t
L
o
g
O
p
.
j
a
v
a
:
144
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
E
d
i
t
L
o
g
O
p
3800(FSEditLogOp.java:144) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp
3800
(
FSE
d
i
t
L
o
g
Op
.
ja
v
a
:
144
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSE
d
i
t
L
o
g
Op
MkdirOp.readFields(FSEditLogOp.java:1683)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp
R
e
a
d
e
r
.
d
e
c
o
d
e
O
p
(
F
S
E
d
i
t
L
o
g
O
p
.
j
a
v
a
:
4697
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
E
d
i
t
L
o
g
O
p
Reader.decodeOp(FSEditLogOp.java:4697) at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp
R
e
a
d
er
.
d
eco
d
e
Op
(
FSE
d
i
t
L
o
g
Op
.
ja
v
a
:
4697
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FSE
d
i
t
L
o
g
Op
Reader.readOp(FSEditLogOp.java:4583)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:203)
at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:188)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:188)
at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:143)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:898)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:753)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:329)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:820)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:804)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)
2022-10-12 05:43:13,676 INFO namenode.EditLogInputStream (RedundantEditLogInputStream.java:nextOp(176)) – Fast-forwarding stream ‘http://everdc5:8480/getJournal?jid=everdc&segmentTxId=145316887&storageInfo=-63%3A575476508%3A0%3ACID-46d57eba-3c5f-413b-bca6-565ab12fc364’ to transaction ID 145352425
2022-10-12 05:44:50,753 INFO namenode.FSImage (FSImage.java:updateCountForQuota(930)) – Initializing quota with 4 thread(s)
2022-10-12 05:46:58,067 INFO namenode.FSNamesystem (FSNamesystemLock.java:writeUnlock(259)) – FSNamesystem write lock held for 1564861 ms via
java.lang.Thread.getStackTrace(Thread.java:1559)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:950)
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:259)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1511)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1014)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:820)
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:804)
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
Number of suppressed write-lock reports: 0
Longest write-lock held interval: 1564861
2022-10-12 05:46:58,067 ERROR namenode.NameNode (NameNode.java:main(1587)) – Failed to start namenode.
java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.join(ForkJoinTask.java:720)
at org.apache.hadoop.hdfs.server.namenode.FSImage.updateCountForQuota(FSImage.java:937)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:753)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:329)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:686)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:586)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:646)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:820)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:804)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1516)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1582)
Caused by: java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.getException(ForkJoinTask.java:930)
at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:852)
at org.apache.hadoop.hdfs.server.namenode.FSImage
I
n
i
t
Q
u
o
t
a
T
a
s
k
.
c
o
m
p
u
t
e
(
F
S
I
m
a
g
e
.
j
a
v
a
:
983
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
e
x
e
c
(
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
j
a
v
a
:
189
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
E
x
e
c
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
289
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
P
o
o
l
InitQuotaTask.compute(FSImage.java:983) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool
I
ni
tQ
u
o
t
a
T
a
s
k
.
co
m
p
u
t
e
(
FS
I
ma
g
e
.
ja
v
a
:
983
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
e
x
ec
(
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
ja
v
a
:
189
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
E
x
ec
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
289
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
P
oo
l
WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.hadoop.hdfs.server.namenode.INode.getQuotaCounts(INode.java:496)
at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getQuotaCounts(INodeDirectory.java:170)
at org.apache.hadoop.hdfs.server.namenode.INode.isQuotaSet(INode.java:504)
at org.apache.hadoop.hdfs.server.namenode.FSImage
I
n
i
t
Q
u
o
t
a
T
a
s
k
.
c
o
m
p
u
t
e
(
F
S
I
m
a
g
e
.
j
a
v
a
:
986
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
e
x
e
c
(
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
j
a
v
a
:
189
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
E
x
e
c
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
289
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
401
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
i
n
v
o
k
e
A
l
l
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
843
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
InitQuotaTask.compute(FSImage.java:986) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:843) at org.apache.hadoop.hdfs.server.namenode.FSImage
I
ni
tQ
u
o
t
a
T
a
s
k
.
co
m
p
u
t
e
(
FS
I
ma
g
e
.
ja
v
a
:
986
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
e
x
ec
(
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
ja
v
a
:
189
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
E
x
ec
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
289
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
401
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
in
v
o
k
e
A
ll
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
843
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
InitQuotaTask.compute(FSImage.java:983)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:843)
at org.apache.hadoop.hdfs.server.namenode.FSImage
I
n
i
t
Q
u
o
t
a
T
a
s
k
.
c
o
m
p
u
t
e
(
F
S
I
m
a
g
e
.
j
a
v
a
:
983
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
e
x
e
c
(
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
j
a
v
a
:
189
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
E
x
e
c
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
289
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
401
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
i
n
v
o
k
e
A
l
l
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
843
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
InitQuotaTask.compute(FSImage.java:983) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:843) at org.apache.hadoop.hdfs.server.namenode.FSImage
I
ni
tQ
u
o
t
a
T
a
s
k
.
co
m
p
u
t
e
(
FS
I
ma
g
e
.
ja
v
a
:
983
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
e
x
ec
(
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
ja
v
a
:
189
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
E
x
ec
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
289
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
401
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
in
v
o
k
e
A
ll
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
843
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
InitQuotaTask.compute(FSImage.java:983)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:843)
at org.apache.hadoop.hdfs.server.namenode.FSImage
I
n
i
t
Q
u
o
t
a
T
a
s
k
.
c
o
m
p
u
t
e
(
F
S
I
m
a
g
e
.
j
a
v
a
:
983
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
e
x
e
c
(
R
e
c
u
r
s
i
v
e
A
c
t
i
o
n
.
j
a
v
a
:
189
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
E
x
e
c
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
289
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
401
)
a
t
j
a
v
a
.
u
t
i
l
.
c
o
n
c
u
r
r
e
n
t
.
F
o
r
k
J
o
i
n
T
a
s
k
.
i
n
v
o
k
e
A
l
l
(
F
o
r
k
J
o
i
n
T
a
s
k
.
j
a
v
a
:
843
)
a
t
o
r
g
.
a
p
a
c
h
e
.
h
a
d
o
o
p
.
h
d
f
s
.
s
e
r
v
e
r
.
n
a
m
e
n
o
d
e
.
F
S
I
m
a
g
e
InitQuotaTask.compute(FSImage.java:983) at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401) at java.util.concurrent.ForkJoinTask.invokeAll(ForkJoinTask.java:843) at org.apache.hadoop.hdfs.server.namenode.FSImage
I
ni
tQ
u
o
t
a
T
a
s
k
.
co
m
p
u
t
e
(
FS
I
ma
g
e
.
ja
v
a
:
983
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
e
x
ec
(
R
ec
u
rs
i
v
e
A
c
t
i
o
n
.
ja
v
a
:
189
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
E
x
ec
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
289
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
d
o
I
n
v
o
k
e
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
401
)
a
t
ja
v
a
.
u
t
i
l
.
co
n
c
u
rre
n
t
.
F
or
k
J
o
in
T
a
s
k
.
in
v
o
k
e
A
ll
(
F
or
k
J
o
in
T
a
s
k
.
ja
v
a
:
843
)
a
t
or
g
.
a
p
a
c
h
e
.
ha
d
oo
p
.
h
df
s
.
ser
v
er
.
nam
e
n
o
d
e
.
FS
I
ma
g
e
InitQuotaTask.compute(FSImage.java:983)
at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
2022-10-12 05:46:58,068 INFO util.ExitUtil (ExitUtil.java:terminate(124)) – Exiting with status 1
2022-10-12 05:46:58,070 INFO namenode.NameNode (LogAdapter.java:info(47)) – SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at everdc1/10.200.106.1
************************************************************/
b.错误分析，尝试修复：
查看错误是gc超出的内存，首先想到的方案是加大配置中的gc内存大小
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Zmu5T075-1665631753464)(C:\Users\101202105025\AppData\Roaming\Typora\typora-user-images\image-20221013111921522.png)]
重启namenode后依旧报错
重新分析错误，重启namenode，那需要遍历fsimage，这个集群数据量比较大，那应该是fsimage中的节点数目过多，导致的GC Overhead超过限制。
Fsimage是namenode维护的重要文件之一，它包含了整个HDFS文件系统的所有目录信息和文件信息。对于文件来说包含了数据块描述信息、修改时间、访问时间等；对于目录来说，包含了修改时间、访问权限控制信息等。
重新尝试更改hadoop的内存
export HADOOP_NAMENODE_OPTS=“-Dhadoop.security.logger=
H
A
D
O
O
P
S
E
C
U
R
I
T
Y
L
O
G
G
E
R
:
−
I
N
F
O
,
R
F
A
S
−
D
h
d
f
s
.
a
u
d
i
t
.
l
o
g
g
e
r
=
{HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=
H
A
D
OO
P
S
EC
U
R
I
T
Y
L
OGGER
:
−
I
NFO
,
RF
A
S
−
D
h
df
s
.
a
u
d
i
t
.
l
o
gg
er
=
{HDFS_AUDIT_LOGGER:-INFO,NullAppender} -Xmx25600m $HADOOP_NAMENODE_OPTS”
需要用以下命令查看fsimage文件占用的内存大小
1，查看FSimage文件
执行下面的命令
时间有点长，耐心等待一会
.
/
h
d
f
s
o
i
v
−
p
X
M
L
−
p
r
i
n
t
T
o
S
c
r
e
e
n
−
i
/
d
a
t
a
/
h
a
d
o
o
p
/
h
d
s
p
a
c
e
/
d
f
s
/
n
a
m
e
/
c
u
r
r
e
n
t
/
f
s
i
m
a
g
e
0
000000000144656572
−
o
/
t
m
p
/
a
./hdfs oiv -p XML -printToScreen -i /data/hadoop/hd_space/dfs/name/current/fsimage_0000000000144656572 -o /tmp/a
./
h
df
so
i
v
−
pXM
L
−
p
r
in
tT
o
S
cree
n
−
i
/
d
a
t
a
/
ha
d
oo
p
/
h
d
s
p
a
ce
/
df
s
/
nam
e
/
c
u
rre
n
t
/
f
s
ima
g
e
0
000000000144656572
−
o
/
t
m
p
/
a
时间有点长，耐心等待一会
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-HKfVGwbt-1665631753465)(C:\Users\101202105025\AppData\Roaming\Typora\typora-user-images\image-20221013104012806.png)]好了执行
cat /tmp/a | egrep "<inode><id>|<block><id>" | wc -l | awk '{printf "Objects=%d : Suggested Xms=%0dm Xmx=%0dm\n", $1, (($1 / 1000000 )*1024), (($1 / 1000000 )*1024)}'
3.查看文件大小
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lspjtW5P-1665631753466)(C:\Users\101202105025\AppData\Roaming\Typora\typora-user-images\image-20221013104116769.png)]
这里的大小是22840那zai配置中加大，之前给的是10240
贵州的地址是
/opt/apps/hadoop_everdc/etc/hadoop/hadoop-env.sh
查看文件配置
cat /opt/apps/hadoop_everdc/etc/hadoop/hadoop-env.sh
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-ihIkvuOJ-1665631753468)(C:\Users\101202105025\AppData\Roaming\Typora\typora-user-images\image-20221013110545084.png)]
5.停止之前的namenode，重新启动namenode
–注意要切换用户–
贵州的是hdfs用户，
su hdfs
然后执行
/opt/apps/hadoop_everdc/sbin/hadoop-daemon.sh start namenode
重启正常了