feat: add redpanda

This commit is contained in:
Sun-ZhenXing
2025-12-04 21:31:46 +08:00
parent 9a079fe79b
commit 3cc5acafbd
9 changed files with 615 additions and 9 deletions

View File

@@ -9,6 +9,9 @@ Apache Pulsar is a cloud-native, distributed messaging and streaming platform. I
### Default (Standalone Mode)
- `pulsar`: Single-node Pulsar instance for development and testing.
- Runs with `--no-functions-worker` flag for simplicity and reduced resource usage
- Uses RocksDB as metadata store by default (since Pulsar 2.11+)
- Includes embedded ZooKeeper and BookKeeper in the same JVM process
### Cluster Mode (profile: `cluster`)
@@ -48,7 +51,25 @@ Please modify the `.env` file as needed for your use case.
docker compose up -d
```
2. Access Pulsar:
2. Wait for Pulsar to be ready (check logs):
```bash
docker compose logs -f pulsar
```
You should see a message like:
```log
INFO org.apache.pulsar.broker.PulsarService - messaging service is ready
```
3. Verify the cluster is healthy:
```bash
docker exec pulsar bin/pulsar-admin brokers healthcheck
```
4. Access Pulsar:
- Broker: `pulsar://localhost:6650`
- Admin API: `http://localhost:8080`
@@ -187,8 +208,39 @@ client.close()
- Standalone mode uses RocksDB as metadata store by default (recommended for single-node).
- Set `PULSAR_STANDALONE_USE_ZOOKEEPER=1` to use ZooKeeper as metadata store.
- Cluster mode is configured for single-node BookKeeper (ensemble size = 1).
- For production, adjust quorum settings and add more bookies.
- Functions worker is disabled by default to reduce resource usage and startup time.
- For production, use cluster mode with dedicated ZooKeeper and BookKeeper instances.
## Troubleshooting
### Standalone Mode Issues
If you encounter connection errors like "NoRouteToHostException" or "Bookie handle is not available":
1. **Clear existing data** (if upgrading or switching metadata store):
```bash
docker compose down -v
docker compose up -d
```
2. **Check container logs**:
```bash
docker compose logs pulsar
```
3. **Verify healthcheck**:
```bash
docker compose ps
docker exec pulsar bin/pulsar-admin brokers healthcheck
```
4. **Ensure sufficient resources**: Standalone mode requires at least:
- 2GB RAM
- 2 CPU cores
- 5GB disk space
## Ports

View File

@@ -9,6 +9,9 @@ Apache Pulsar 是一个云原生的分布式消息和流处理平台。它结合
### 默认(单机模式)
- `pulsar`:单节点 Pulsar 实例,适用于开发和测试。
- 使用 `--no-functions-worker` 标志运行,简化部署并减少资源使用
- 默认使用 RocksDB 作为元数据存储(从 Pulsar 2.11+ 开始)
- 在同一个 JVM 进程中包含内嵌的 ZooKeeper 和 BookKeeper
### 集群模式profile: `cluster`
@@ -48,7 +51,25 @@ Apache Pulsar 是一个云原生的分布式消息和流处理平台。它结合
docker compose up -d
```
2. 访问 Pulsar
2. 等待 Pulsar 就绪(查看日志)
```bash
docker compose logs -f pulsar
```
您应该看到类似以下的消息:
```log
INFO org.apache.pulsar.broker.PulsarService - messaging service is ready
```
3. 验证集群健康状态:
```bash
docker exec pulsar bin/pulsar-admin brokers healthcheck
```
4. 访问 Pulsar
- Broker`pulsar://localhost:6650`
- Admin API`http://localhost:8080`
@@ -187,8 +208,39 @@ client.close()
- 单机模式默认使用 RocksDB 作为元数据存储(推荐用于单节点)。
- 设置 `PULSAR_STANDALONE_USE_ZOOKEEPER=1` 可使用 ZooKeeper 作为元数据存储。
- 集群模式配置为单节点 BookKeeperensemble size = 1
- 生产环境请调整 quorum 设置并添加更多 bookie
- 默认禁用 Functions Worker 以减少资源使用和启动时间
- 生产环境请使用集群模式,配置专用的 ZooKeeper 和 BookKeeper 实例
## 故障排除
### 单机模式问题
如果遇到连接错误,如"NoRouteToHostException"或"Bookie handle is not available"
1. **清除现有数据**(如果升级或切换元数据存储):
```bash
docker compose down -v
docker compose up -d
```
2. **检查容器日志**
```bash
docker compose logs pulsar
```
3. **验证健康检查**
```bash
docker compose ps
docker exec pulsar bin/pulsar-admin brokers healthcheck
```
4. **确保资源充足**:单机模式至少需要:
- 2GB 内存
- 2 个 CPU 核心
- 5GB 磁盘空间
## 端口

View File

@@ -14,6 +14,7 @@ services:
pulsar:
<<: *defaults
image: ${GLOBAL_REGISTRY:-}apachepulsar/pulsar:${PULSAR_VERSION:-4.0.7}
hostname: pulsar-standalone
ports:
- "${PULSAR_BROKER_PORT_OVERRIDE:-6650}:6650"
- "${PULSAR_HTTP_PORT_OVERRIDE:-8080}:8080"
@@ -24,7 +25,7 @@ services:
- TZ=${TZ:-UTC}
- PULSAR_MEM=${PULSAR_MEM:--Xms512m -Xmx512m -XX:MaxDirectMemorySize=256m}
- PULSAR_STANDALONE_USE_ZOOKEEPER=${PULSAR_STANDALONE_USE_ZOOKEEPER:-0}
command: bin/pulsar standalone
command: sh -c "bin/apply-config-from-env.py conf/standalone.conf && bin/pulsar standalone --no-functions-worker"
deploy:
resources:
limits:
@@ -34,11 +35,11 @@ services:
cpus: ${PULSAR_CPU_RESERVATION:-0.50}
memory: ${PULSAR_MEMORY_RESERVATION:-512M}
healthcheck:
test: ["CMD", "bin/pulsar-admin", "brokers", "healthcheck"]
test: ["CMD-SHELL", "bin/pulsar-admin brokers healthcheck || exit 1"]
interval: 30s
timeout: 10s
retries: 5
start_period: 60s
start_period: 90s
# ==================== Cluster Mode (profile: cluster) ====================