博客

前几天在本地开发调试Flink程序时，在WebUI页面无法查看jobManager日志或者taskManager日志，点击会在控制台报如下错误：

18:23:07.953 [flink-rest-server-netty-worker-thread-1] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor 9c9607af-a26b-47aa-89e3-7db9d89b89fb.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_341]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_341]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_341]
	
	......

	at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918) [flink-shaded-netty-4.1.39.Final-11.0.jar:?]
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [flink-shaded-netty-4.1.39.Final-11.0.jar:?]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_341]
Caused by: org.apache.flink.util.FlinkException: The file LOG is not available on the TaskExecutor.
	at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByFilePath(TaskExecutor.java:1747) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
	at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByType(TaskExecutor.java:1006) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]

	.......
// #########################
18:28:27.129 [flink-akka.actor.default-dispatcher-2] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerStdoutFileHandler - Failed to transfer file from TaskExecutor dd0863a3-721d-47d2-b5fd-b4f5b778ec45.
java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_341]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_341]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:661) ~[?:1.8.0_341]
	
	......

	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [akka-actor_2.12-2.5.21.jar:2.5.21]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [akka-actor_2.12-2.5.21.jar:2.5.21]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [akka-actor_2.12-2.5.21.jar:2.5.21]
Caused by: org.apache.flink.util.FlinkException: The file STDOUT is not available on the TaskExecutor.
	at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByFilePath(TaskExecutor.java:1747) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
	at org.apache.flink.runtime.taskexecutor.TaskExecutor.requestFileUploadByType(TaskExecutor.java:1006) ~[flink-runtime_2.12-1.11.2.jar:1.11.2]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_341]
	......

解决办法如下：

1、引入日志配置，包括pom文件中的依赖和 src/main/resources 目录下的日志文职文件。下面以log4j2.xml为例展示日志配置：

<?xml version="1.0" encoding="UTF-8"?>
<configuration monitorInterval="5">
    <Properties>
        <property name="LOG_PATTERN" value="%date{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" />
        <property name="LOG_LEVEL" value="INFO" />
    </Properties>

    <appenders>
        <console name="console" target="SYSTEM_OUT">
            <PatternLayout pattern="${LOG_PATTERN}"/>
            <ThresholdFilter level="${LOG_LEVEL}" onMatch="ACCEPT" onMismatch="DENY"/>
        </console>
        <File name="log" fileName="tmp/log/job.log" append="false">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/>
        </File>
    </appenders>

    <loggers>
        <root level="${LOG_LEVEL}">
            <appender-ref ref="console"/>
            <appender-ref ref="log"/>
        </root>
    </loggers>

</configuration>

2、在flink程序开始初始化env时增加log_path相关的配置项，如下所示：

Configuration conf = new Configuration();
conf.setString(RestOptions.BIND_PORT,"8081");
conf.setString(WebOptions.LOG_PATH,"tmp/log/job.log");
conf.setString(ConfigConstants.TASK_MANAGER_LOG_PATH_KEY,"tmp/log/job.log");
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);

然后重新启动程序就可以WebUI正常查看日志了，如图:

总结：引入日志配置（logback、log4j都可以），开启日志文件的appender和logger，然后将日志文件路径引入到初始化env的conf中就可以了。