博客

SpringCloud DataFlow — 0. 本地部署部署



SpringCloud DataFlow — 1. 自定义Processor



SpringCloud DataFlow — 2. 自定义Sink



SpringCloud DataFlow — 3. 暴露properties



SpringCloud DataFlow — 4. Prometheus + Grafana 监控



SpringCloud DataFlow — 5. 多分支负载

以 unpack-processor-kafka 为例

1. pom.xml 中添加依赖

1.1 dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.cloud.stream.app</groupId>
    <artifactId>app-starters-micrometer-common</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.cloud.stream.app</groupId>
    <artifactId>app-starters-security-common</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

1.2 dependencyManagement

<dependency>
    <groupId>org.springframework.cloud.stream.app</groupId>
    <artifactId>app-starters-core-dependencies</artifactId>
    <version>2.1.0.RELEASE</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

2. Prometheus Service Discovery

2.1 下载zip代码包

https://github.com/tzolov/spring-cloud-dataflow-prometheus-service-discovery

2.2 解压缩并切换路径

cd spring-cloud-dataflow-prometheus-service-discovery-master

2.3 打包jar

./mvnw clean package -Dmaven.test.skip=true

2.4 运行服务

java -jar target/spring-cloud-dataflow-prometheus-service-discovery-0.0.5.BUILD-SNAPSHOT.jar --metrics.prometheus.target.discoveryUrl='http://127.0.0.1:9393/runtime/apps' --metrics.prometheus.target.cron='0/30 * * * * *' --metrics.prometheus.target.mode=local --metrics.prometheus.target.overrideIp='127.0.0.1' --server.port=8181

3. Prometheus

3.1 安装prometheus

brew install prometheus

3.2 进入安装目录

cd /usr/local/Cellar/prometheus/2.11.1

3.3 修改homebrew.mxcl.prometheus.plist

# 修改第十一行
<string>/usr/local/opt/prometheus/bin/prometheus --config.file=/usr/local/etc/prometheus.yml</string>

3.4 添加prometheus.yml

vim /usr/local/etc/prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'dataflow'
    metrics_path: '/actuator/prometheus'
    file_sd_configs:
      - files:
          - /tmp/targets.json

3.4 启动服务

brew services start prometheus

3.5 查看

http://localhost:9090/targets

4. Grafana

4.1 安装Grafana

brew install grafana

4.2 启动服务

brew services start grafana

4.3 查看

http://localhost:3000

4.4 配置

4.4.1 data source

Name: prometheus
URL: http://localhost:9090
Access: server(default)

Save & Test

4.4.2 dashboard

# 为了简便, 导入被人写好的配置
1. 访问: https://grafana.com/grafana/dashboards
2. Search: data flow
3. 点击: "SCDF Streams Prometheus"
4. 获得dashboard编号: 9933
5. 访问: http://localhost:3000/dashboard/import
6. Grafana.com Dashboard 输入 9933
7. 点击 "load"
8. ScdfPrometheus 选择4.4.1中创建的数据源数据源 "prometheus"
9. 点击 "Import"