Add environment configuration and documentation for various services
- Created .env.example files for Kafka, Kibana, KodBox, Kong, Langfuse, Logstash, n8n, Nginx, OceanBase, OpenCoze, RocketMQ, TiDB, and TiKV. - Added README.md and README.zh.md files for OceanBase, RocketMQ, TiDB, and TiKV, detailing usage, configuration, and access instructions. - Implemented docker-compose.yaml files for OceanBase, RocketMQ, TiDB, and TiKV, defining service configurations, health checks, and resource limits. - Included broker.conf for RocketMQ to specify broker settings. - Established a consistent timezone (UTC) across all services. - Provided optional port overrides in .env.example files for flexibility in deployment.
This commit is contained in:
9
src/flink/.env.example
Normal file
9
src/flink/.env.example
Normal file
@@ -0,0 +1,9 @@
|
||||
# Flink version
|
||||
FLINK_VERSION=1.20.0-scala_2.12-java11
|
||||
|
||||
# Timezone
|
||||
TZ=UTC
|
||||
|
||||
# Port overrides (optional)
|
||||
# FLINK_JOBMANAGER_RPC_PORT_OVERRIDE=6123
|
||||
# FLINK_JOBMANAGER_UI_PORT_OVERRIDE=8081
|
||||
104
src/flink/README.md
Normal file
104
src/flink/README.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Apache Flink
|
||||
|
||||
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
This setup includes:
|
||||
|
||||
- **JobManager**: Coordinates Flink jobs and manages resources
|
||||
- **TaskManager**: Executes tasks and manages data streams
|
||||
|
||||
## Configuration
|
||||
|
||||
Key environment variables in `FLINK_PROPERTIES`:
|
||||
|
||||
- `jobmanager.rpc.address`: JobManager RPC address
|
||||
- `jobmanager.memory.process.size`: JobManager memory (default: 1600m)
|
||||
- `taskmanager.memory.process.size`: TaskManager memory (default: 1600m)
|
||||
- `taskmanager.numberOfTaskSlots`: Number of task slots per TaskManager (default: 2)
|
||||
|
||||
## Ports
|
||||
|
||||
- `6123`: JobManager RPC port
|
||||
- `8081`: Flink Web UI
|
||||
|
||||
## Access
|
||||
|
||||
### Web UI
|
||||
|
||||
Access Flink Dashboard at: <http://localhost:8081>
|
||||
|
||||
### Submit Jobs
|
||||
|
||||
Submit a Flink job:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
|
||||
```
|
||||
|
||||
Submit a custom job:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/jobs/my-job.jar
|
||||
```
|
||||
|
||||
### Job Management
|
||||
|
||||
```bash
|
||||
# List running jobs
|
||||
docker compose exec jobmanager ./bin/flink list
|
||||
|
||||
# Cancel a job
|
||||
docker compose exec jobmanager ./bin/flink cancel <job-id>
|
||||
|
||||
# Show job details
|
||||
docker compose exec jobmanager ./bin/flink info /path/to/job.jar
|
||||
```
|
||||
|
||||
## Example: WordCount
|
||||
|
||||
Run the built-in WordCount example:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
|
||||
```
|
||||
|
||||
## Scaling TaskManagers
|
||||
|
||||
To scale TaskManagers for more processing capacity:
|
||||
|
||||
```bash
|
||||
docker compose up -d --scale taskmanager=3
|
||||
```
|
||||
|
||||
## Custom Jobs
|
||||
|
||||
Mount your custom Flink jobs by uncommenting the volume in `docker-compose.yaml`:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- ./jobs:/opt/flink/jobs
|
||||
```
|
||||
|
||||
Then place your JAR files in the `./jobs` directory.
|
||||
|
||||
## Notes
|
||||
|
||||
- This is a standalone cluster setup suitable for development
|
||||
- For production, consider using Flink on Kubernetes or YARN
|
||||
- Adjust memory settings based on your workload requirements
|
||||
- Task slots determine parallelism; more slots allow more parallel tasks
|
||||
- Data is persisted in the named volume `flink_data`
|
||||
|
||||
## References
|
||||
|
||||
- [Apache Flink Official Documentation](https://flink.apache.org/docs/stable/)
|
||||
- [Flink Docker Setup](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/resource-providers/standalone/docker/)
|
||||
- [Flink Docker Hub](https://hub.docker.com/_/flink)
|
||||
104
src/flink/README.zh.md
Normal file
104
src/flink/README.zh.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Apache Flink
|
||||
|
||||
Apache Flink 是一个框架和分布式处理引擎,用于对无界和有界数据流进行有状态计算。
|
||||
|
||||
## 使用方法
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## 组件说明
|
||||
|
||||
此配置包含:
|
||||
|
||||
- **JobManager**:协调 Flink 作业并管理资源
|
||||
- **TaskManager**:执行任务并管理数据流
|
||||
|
||||
## 配置说明
|
||||
|
||||
`FLINK_PROPERTIES` 中的关键环境变量:
|
||||
|
||||
- `jobmanager.rpc.address`:JobManager RPC 地址
|
||||
- `jobmanager.memory.process.size`:JobManager 内存(默认:1600m)
|
||||
- `taskmanager.memory.process.size`:TaskManager 内存(默认:1600m)
|
||||
- `taskmanager.numberOfTaskSlots`:每个 TaskManager 的任务槽数量(默认:2)
|
||||
|
||||
## 端口说明
|
||||
|
||||
- `6123`:JobManager RPC 端口
|
||||
- `8081`:Flink Web UI
|
||||
|
||||
## 访问方式
|
||||
|
||||
### Web UI
|
||||
|
||||
访问 Flink Dashboard:<http://localhost:8081>
|
||||
|
||||
### 提交作业
|
||||
|
||||
提交 Flink 作业:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
|
||||
```
|
||||
|
||||
提交自定义作业:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/jobs/my-job.jar
|
||||
```
|
||||
|
||||
### 作业管理
|
||||
|
||||
```bash
|
||||
# 列出运行中的作业
|
||||
docker compose exec jobmanager ./bin/flink list
|
||||
|
||||
# 取消作业
|
||||
docker compose exec jobmanager ./bin/flink cancel <job-id>
|
||||
|
||||
# 显示作业详情
|
||||
docker compose exec jobmanager ./bin/flink info /path/to/job.jar
|
||||
```
|
||||
|
||||
## 示例:WordCount
|
||||
|
||||
运行内置的 WordCount 示例:
|
||||
|
||||
```bash
|
||||
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
|
||||
```
|
||||
|
||||
## 扩展 TaskManager
|
||||
|
||||
要扩展 TaskManager 以获得更多处理能力:
|
||||
|
||||
```bash
|
||||
docker compose up -d --scale taskmanager=3
|
||||
```
|
||||
|
||||
## 自定义作业
|
||||
|
||||
通过取消注释 `docker-compose.yaml` 中的卷来挂载自定义 Flink 作业:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- ./jobs:/opt/flink/jobs
|
||||
```
|
||||
|
||||
然后将 JAR 文件放在 `./jobs` 目录中。
|
||||
|
||||
## 注意事项
|
||||
|
||||
- 这是一个独立集群配置,适合开发环境
|
||||
- 生产环境建议在 Kubernetes 或 YARN 上使用 Flink
|
||||
- 根据工作负载需求调整内存设置
|
||||
- 任务槽决定并行度,更多槽允许更多并行任务
|
||||
- 数据持久化在命名卷 `flink_data` 中
|
||||
|
||||
## 参考资料
|
||||
|
||||
- [Apache Flink 官方文档](https://flink.apache.org/zh/docs/stable/)
|
||||
- [Flink Docker 设置](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/resource-providers/standalone/docker/)
|
||||
- [Flink Docker Hub](https://hub.docker.com/_/flink)
|
||||
78
src/flink/docker-compose.yaml
Normal file
78
src/flink/docker-compose.yaml
Normal file
@@ -0,0 +1,78 @@
|
||||
x-default: &default
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 100m
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
jobmanager:
|
||||
<<: *default
|
||||
image: flink:${FLINK_VERSION:-1.20.0-scala_2.12-java11}
|
||||
hostname: jobmanager
|
||||
command: jobmanager
|
||||
environment:
|
||||
TZ: ${TZ:-UTC}
|
||||
FLINK_PROPERTIES: |
|
||||
jobmanager.rpc.address: jobmanager
|
||||
jobmanager.memory.process.size: 1600m
|
||||
taskmanager.memory.process.size: 1600m
|
||||
taskmanager.numberOfTaskSlots: 2
|
||||
volumes:
|
||||
- flink_data:/opt/flink/data
|
||||
# Custom Flink jobs
|
||||
# - ./jobs:/opt/flink/jobs
|
||||
ports:
|
||||
- "${FLINK_JOBMANAGER_RPC_PORT_OVERRIDE:-6123}:6123"
|
||||
- "${FLINK_JOBMANAGER_UI_PORT_OVERRIDE:-8081}:8081"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 1G
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:8081 || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
taskmanager:
|
||||
<<: *default
|
||||
image: flink:${FLINK_VERSION:-1.20.0-scala_2.12-java11}
|
||||
command: taskmanager
|
||||
environment:
|
||||
TZ: ${TZ:-UTC}
|
||||
FLINK_PROPERTIES: |
|
||||
jobmanager.rpc.address: jobmanager
|
||||
jobmanager.memory.process.size: 1600m
|
||||
taskmanager.memory.process.size: 1600m
|
||||
taskmanager.numberOfTaskSlots: 2
|
||||
volumes:
|
||||
- flink_data:/opt/flink/data
|
||||
# Custom Flink jobs
|
||||
# - ./jobs:/opt/flink/jobs
|
||||
depends_on:
|
||||
jobmanager:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '2.0'
|
||||
memory: 2G
|
||||
reservations:
|
||||
cpus: '0.5'
|
||||
memory: 1G
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "ps aux | grep -v grep | grep -q taskmanager || exit 1"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
|
||||
volumes:
|
||||
flink_data:
|
||||
Reference in New Issue
Block a user