博客

IDEA本地调试azkaban

  • Post author:
  • Post category:其他




准备工作



下载azkaban源码


中央maven太慢，修改为阿里云的仓库，修改build.gradle文件
 

buildscript {
  repositories {
    maven {
      url 'https://maven.aliyun.com/repository/gradle-plugin'
    }
  }

allprojects {
  apply plugin: 'jacoco'

  repositories {
    maven {
      url 'https://maven.aliyun.com/repository/gradle-plugin'
    }
  }
}



本地git工具，在azkaban-master目录下

在这里插入图片描述



新建azkaban-master/conf


将azkaban-web-server下的conf复制过来，修改azkaban.properties
 

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
# 必须要绝对路径
web.resource.dir=D:\\WorkProject\\azkaban-master\\azkaban-web-server\\src\\web
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=18081
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=
mysql.database=azkaban
mysql.user=
mysql.password=
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1



azkaban-exec-server.conf

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/home/azkaban-all/azkaban-web-server/web/
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=/home/azkaban-all/azkaban-exec-server/conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Where the Azkaban web server is located
azkaban.webserver.url=http://localhost:18448
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=/home/azkaban-all/azkaban-exec-server/plugins/jobtypes
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30



azkaban-web-server.conf

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=/home/azkaban-all/azkaban-web-server/web
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=/home/azkaban-all/azkaban-web-server/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=18448

jetty.keystore=/home/azkaban-all/azkaban-web-server/keystore
jetty.password=123456
jetty.keypassword=123456
jetty.truststore=/home/azkaban-all/azkaban-web-server/keystore
jetty.trustpassword=123456
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1



源码导入IDEA进行配置



配置exec-server


在IDEA或eclipse 右键 azkaban.execapp.AzkabanExecutorServer 该类，然后运行/debug运行
在运行属性中设置：
VM options: -Dlog4j.configuration=file:E:\awork\azkaban\azkaban-master\conf\log4j.properties -Dserverpath=E:\awork\azkaban\azkaban-master\azkaban-exec-server\build\install\azkaban-exec-server
Program arguments: -conf E:\awork\azkaban\azkaban-master\conf


# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=D:\\WorkProject\\azkaban-master\\azkaban-web-server\\src\\web
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=8081
# Where the Azkaban web server is located
azkaban.webserver.url=http://localhost:18081
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban plugin settings
azkaban.jobtype.plugin.dir=plugins/jobtypes/
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=127.0.0.1
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
# Azkaban Executor settings
executor.maxThreads=50
executor.flow.threads=30



配置web-server

在IDEA或eclipse 右键 azkaban.webapp.AzkabanWebServer 该类，然后运行/debug运行
在运行属性中设置：
VM options: -Dlog4j.configuration=file:E:\awork\azkaban\azkaban-master\conf\log4j.properties -Dserverpath=E:\awork\azkaban\azkaban-master\azkaban-web-server\build\install\azkaban-web-server
Program arguments: -conf E:\awork\azkaban\azkaban-master\conf

# Azkaban Personalization Settings
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=D:\\WorkProject\\azkaban-master\\azkaban-web-server\\src\\web
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=D:\\WorkProject\\azkaban-master\\conf\\azkaban-users.xml
# Loader for projects
executor.global.properties=D:\\WorkProject\\azkaban-master\\conf\\global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.use.ssl=false
jetty.maxThreads=25
jetty.port=18081
# Azkaban Executor settings
# mail settings
mail.sender=
mail.host=
# User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
# enduser -> myazkabanhost:443 -> proxy -> localhost:8081
# when this parameters set then these parameters are used to generate email links.
# if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
# azkaban.webserver.external_hostname=myazkabanhost.com
# azkaban.webserver.external_ssl_port=443
# azkaban.webserver.external_port=8081
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true
# Azkaban mysql settings by default. Users should configure their own username and password.
database.type=mysql
mysql.port=3306
mysql.host=127.0.0.1
mysql.database=azkaban
mysql.user=root
mysql.password=123456
mysql.numconnections=100
#Multiple Executor
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1



出现问题



Unable to get scheduled flow triggers
2020/05/22 09:52:03.450 +0800  INFO [LoginAbstractAzkabanServlet] [22496252@qtp-16641088-5] [Azkaban] 0 admin "GET /flowtrigger ajax=fetchTrigger&projectId=1&flowId=second HTTP/1.1" browser
2020/05/22 09:52:03.451 +0800 ERROR [FlowTriggerScheduler] [22496252@qtp-16641088-5] [Azkaban] Unable to get scheduled flow triggers
java.lang.NullPointerException
	at azkaban.flowtrigger.quartz.FlowTriggerScheduler.getScheduledFlowTriggerJobs(FlowTriggerScheduler.java:139)
	at azkaban.webapp.servlet.FlowTriggerServlet.ajaxFetchTrigger(FlowTriggerServlet.java:69)
	at azkaban.webapp.servlet.FlowTriggerServlet.handleAJAXAction(FlowTriggerServlet.java:111)
	at azkaban.webapp.servlet.FlowTriggerServlet.handleGet(FlowTriggerServlet.java:59)
	at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:122)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)

这个原因很多。。。

1.

在这里插入图片描述

尝试注释executors的限制

2.我是在IDEA调试的，可能windows系统没有command这个jobflow，部署到服务器的没问题



azkaban.executor.ExecutorManagerException: No active executors found

Navicat查看executors目录

在这里插入图片描述

windows就用postman或者浏览器

请求

http://${executor.host}:${executor.port}/executor?action=activate

linux就直接


curl http://${executor.host}:${executor.port}/executor?action=activate



webUI

如果你按我的方法，那IDEA启动后的本地效果会不好，因为web包是未编译的


解决方法:


在azkaban-master下用git工具，输入./gradlew distTar

在这里插入图片描述

D:\WorkProject\azkaban-master\azkaban-web-server\build\distributions 基本都在这种目录下回有编译好的zip包

在web-server 的 zip包中找到web目录 替换 源代码也就是IDEA中的项目



最终效果

在这里插入图片描述

错误的原因是windows中可能没有command流，因为我只需要调试，所以目的已经达到了。


linux安装部署详细可参考我的另一篇文章



linux安装部署azkaban



linux下效果成功

在这里插入图片描述



博主公众号




求关注


在这里插入图片描述


版权声明：本文为qq_42784606原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。