症状:某个MQ发送失败
错误日志:
15:00:00.888 [HSFBizProcessor-DEFAULT-6-thread-23] ERROR com.bosideng.bsdyun.msg.service.MqServiceImpl – com.aliyun.openservices.ons.api.exception.ONSClientException: Connect broker failed, Topic=VIP_JOIN_ACTIVITY_PRD, msgId=null
15:00:07.265 [HSFBizProcessor-DEFAULT-6-thread-50] WARN io.netty.channel.AbstractChannel – Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xd2b4893d]
java.util.concurrent.RejectedExecutionException: event executor terminated
排查 /home/admin/logs/rocketmq_client.log 发现出现过oom:
可以使用 jstat -gcutils pid 1000 查看gc情况,pid需换成业务程序的进程id
查看gc.lg 出现多次full gc
定位应用日志发现在14:11出现oom
重点排查业务系统的内存溢出问题,排查中…
Exception in thread “RebalanceService” java.lang.OutOfMemoryError: Exception From Host:[10.101.3.101]GC overhead limit exceeded14:11:25.197 [HSFBizProcessor-DEFAULT-6-thread-49] ERROR com.bosideng.bsdyun.msg.service.MsInterfaceMessageServiceImpl – org.springframework.jdbc.UncategorizedSQLException: Error attempting to get column ‘xml_content’ from result set. Cause: java.sql.SQLException: Error
; uncategorized SQLException for SQL []; SQL state [null]; error code [0]; Error; nested exception is java.sql.SQLException: Error