博客

Flink高可用:

1.Standalone Cluster

2.Flink  on Yarn

1.Flink高可用配置主要对JobManager的高可用配置,Jobmanager是整个集群的管理节点,负责整个集群的任务调度和资源管理.

Flink默认是不开启Jobmanager高可用

Standalone集群高可用配置

Standalone模式中的jobmanager通过Zookeeper完成Jobmanager Leader的选举.

Conf/zoo.cfg配置

# The number of milliseconds of each tick

＃每个刻度的毫秒数

tickTime = 2000

# The number of ticks that the initial  synchronization phase can take

＃初始同步阶段可以使用的滴答声数量

initLimit = 10

# The number of ticks that can pass between  sending a request and getting an acknowledgement

＃在发送请求和获得确认之间可以经过的滴答数

syncLimit = 5

# The directory where the snapshot is stored.

＃快照存储目录

＃DataDir=/tmp/zookeeper

# The port at which the clients will connect

＃客户端将连接的端口

clientPort = 2181

# ZooKeeper quorum peers

＃ZK主机及端口

server.1 =本地主机：2888：3888

# server.2=host:peer-port:leader-port

在Standalone Cluster高可用配置中,还需要对Master和flink-conf.yaml两个配置文件进行修改

Conf/masters配置(指定JobmanagerAddress信息分别为主备节点的jobmanager和web地址和端口)

JobManagerAddress:webUIPort1

JobManagerAddress:webUIPort2

Conf/flink-conf.yaml配置

开启高可用模式,通过Zookeeper支持系统高可用

high-avavilability:zoopeeper

Zookeeper的服务地址,多个IP地址可以使用逗号分割

high-availability.zookeeper.quorum:IP:2181,IP:2181

Zookeeper中Flink服务对应的root路径

high-availability.zookeeper.path.root:/flink