Add environment configuration and documentation for various services

- Created .env.example files for Kafka, Kibana, KodBox, Kong, Langfuse, Logstash, n8n, Nginx, OceanBase, OpenCoze, RocketMQ, TiDB, and TiKV.
- Added README.md and README.zh.md files for OceanBase, RocketMQ, TiDB, and TiKV, detailing usage, configuration, and access instructions.
- Implemented docker-compose.yaml files for OceanBase, RocketMQ, TiDB, and TiKV, defining service configurations, health checks, and resource limits.
- Included broker.conf for RocketMQ to specify broker settings.
- Established a consistent timezone (UTC) across all services.
- Provided optional port overrides in .env.example files for flexibility in deployment.
This commit is contained in:
Sun-ZhenXing
2025-10-22 11:46:50 +08:00
parent 84e8b85990
commit ece59b42bf
49 changed files with 2326 additions and 0 deletions

9
src/flink/.env.example Normal file
View File

@@ -0,0 +1,9 @@
# Flink version
FLINK_VERSION=1.20.0-scala_2.12-java11
# Timezone
TZ=UTC
# Port overrides (optional)
# FLINK_JOBMANAGER_RPC_PORT_OVERRIDE=6123
# FLINK_JOBMANAGER_UI_PORT_OVERRIDE=8081

104
src/flink/README.md Normal file
View File

@@ -0,0 +1,104 @@
# Apache Flink
Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
## Usage
```bash
docker compose up -d
```
## Components
This setup includes:
- **JobManager**: Coordinates Flink jobs and manages resources
- **TaskManager**: Executes tasks and manages data streams
## Configuration
Key environment variables in `FLINK_PROPERTIES`:
- `jobmanager.rpc.address`: JobManager RPC address
- `jobmanager.memory.process.size`: JobManager memory (default: 1600m)
- `taskmanager.memory.process.size`: TaskManager memory (default: 1600m)
- `taskmanager.numberOfTaskSlots`: Number of task slots per TaskManager (default: 2)
## Ports
- `6123`: JobManager RPC port
- `8081`: Flink Web UI
## Access
### Web UI
Access Flink Dashboard at: <http://localhost:8081>
### Submit Jobs
Submit a Flink job:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
```
Submit a custom job:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/jobs/my-job.jar
```
### Job Management
```bash
# List running jobs
docker compose exec jobmanager ./bin/flink list
# Cancel a job
docker compose exec jobmanager ./bin/flink cancel <job-id>
# Show job details
docker compose exec jobmanager ./bin/flink info /path/to/job.jar
```
## Example: WordCount
Run the built-in WordCount example:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
```
## Scaling TaskManagers
To scale TaskManagers for more processing capacity:
```bash
docker compose up -d --scale taskmanager=3
```
## Custom Jobs
Mount your custom Flink jobs by uncommenting the volume in `docker-compose.yaml`:
```yaml
volumes:
- ./jobs:/opt/flink/jobs
```
Then place your JAR files in the `./jobs` directory.
## Notes
- This is a standalone cluster setup suitable for development
- For production, consider using Flink on Kubernetes or YARN
- Adjust memory settings based on your workload requirements
- Task slots determine parallelism; more slots allow more parallel tasks
- Data is persisted in the named volume `flink_data`
## References
- [Apache Flink Official Documentation](https://flink.apache.org/docs/stable/)
- [Flink Docker Setup](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/resource-providers/standalone/docker/)
- [Flink Docker Hub](https://hub.docker.com/_/flink)

104
src/flink/README.zh.md Normal file
View File

@@ -0,0 +1,104 @@
# Apache Flink
Apache Flink 是一个框架和分布式处理引擎,用于对无界和有界数据流进行有状态计算。
## 使用方法
```bash
docker compose up -d
```
## 组件说明
此配置包含:
- **JobManager**:协调 Flink 作业并管理资源
- **TaskManager**:执行任务并管理数据流
## 配置说明
`FLINK_PROPERTIES` 中的关键环境变量:
- `jobmanager.rpc.address`JobManager RPC 地址
- `jobmanager.memory.process.size`JobManager 内存默认1600m
- `taskmanager.memory.process.size`TaskManager 内存默认1600m
- `taskmanager.numberOfTaskSlots`:每个 TaskManager 的任务槽数量默认2
## 端口说明
- `6123`JobManager RPC 端口
- `8081`Flink Web UI
## 访问方式
### Web UI
访问 Flink Dashboard<http://localhost:8081>
### 提交作业
提交 Flink 作业:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
```
提交自定义作业:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/jobs/my-job.jar
```
### 作业管理
```bash
# 列出运行中的作业
docker compose exec jobmanager ./bin/flink list
# 取消作业
docker compose exec jobmanager ./bin/flink cancel <job-id>
# 显示作业详情
docker compose exec jobmanager ./bin/flink info /path/to/job.jar
```
## 示例WordCount
运行内置的 WordCount 示例:
```bash
docker compose exec jobmanager ./bin/flink run /opt/flink/examples/streaming/WordCount.jar
```
## 扩展 TaskManager
要扩展 TaskManager 以获得更多处理能力:
```bash
docker compose up -d --scale taskmanager=3
```
## 自定义作业
通过取消注释 `docker-compose.yaml` 中的卷来挂载自定义 Flink 作业:
```yaml
volumes:
- ./jobs:/opt/flink/jobs
```
然后将 JAR 文件放在 `./jobs` 目录中。
## 注意事项
- 这是一个独立集群配置,适合开发环境
- 生产环境建议在 Kubernetes 或 YARN 上使用 Flink
- 根据工作负载需求调整内存设置
- 任务槽决定并行度,更多槽允许更多并行任务
- 数据持久化在命名卷 `flink_data`
## 参考资料
- [Apache Flink 官方文档](https://flink.apache.org/zh/docs/stable/)
- [Flink Docker 设置](https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/resource-providers/standalone/docker/)
- [Flink Docker Hub](https://hub.docker.com/_/flink)

View File

@@ -0,0 +1,78 @@
x-default: &default
restart: unless-stopped
logging:
driver: json-file
options:
max-size: 100m
max-file: "3"
services:
jobmanager:
<<: *default
image: flink:${FLINK_VERSION:-1.20.0-scala_2.12-java11}
hostname: jobmanager
command: jobmanager
environment:
TZ: ${TZ:-UTC}
FLINK_PROPERTIES: |
jobmanager.rpc.address: jobmanager
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1600m
taskmanager.numberOfTaskSlots: 2
volumes:
- flink_data:/opt/flink/data
# Custom Flink jobs
# - ./jobs:/opt/flink/jobs
ports:
- "${FLINK_JOBMANAGER_RPC_PORT_OVERRIDE:-6123}:6123"
- "${FLINK_JOBMANAGER_UI_PORT_OVERRIDE:-8081}:8081"
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 1G
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:8081 || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
taskmanager:
<<: *default
image: flink:${FLINK_VERSION:-1.20.0-scala_2.12-java11}
command: taskmanager
environment:
TZ: ${TZ:-UTC}
FLINK_PROPERTIES: |
jobmanager.rpc.address: jobmanager
jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1600m
taskmanager.numberOfTaskSlots: 2
volumes:
- flink_data:/opt/flink/data
# Custom Flink jobs
# - ./jobs:/opt/flink/jobs
depends_on:
jobmanager:
condition: service_healthy
deploy:
resources:
limits:
cpus: '2.0'
memory: 2G
reservations:
cpus: '0.5'
memory: 1G
healthcheck:
test: ["CMD-SHELL", "ps aux | grep -v grep | grep -q taskmanager || exit 1"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
volumes:
flink_data: