feat: add redpanda
This commit is contained in:
Sun-ZhenXing
47
src/redpanda/.env.example
Normal file
47
src/redpanda/.env.example
Normal file
@@ -0,0 +1,47 @@
|
||||
# Redpanda version
|
||||
REDPANDA_VERSION=v24.3.1
|
||||
|
||||
# Redpanda Console version
|
||||
REDPANDA_CONSOLE_VERSION=v2.8.2
|
||||
|
||||
# Global registry prefix (optional, for using mirror registries)
|
||||
# GLOBAL_REGISTRY=your-registry.com/
|
||||
|
||||
# Timezone (default: UTC)
|
||||
TZ=UTC
|
||||
|
||||
# Redpanda Kafka API port (external)
|
||||
REDPANDA_KAFKA_PORT_OVERRIDE=19092
|
||||
|
||||
# Redpanda Schema Registry port (external)
|
||||
REDPANDA_SCHEMA_PORT_OVERRIDE=18081
|
||||
|
||||
# Redpanda HTTP Proxy port (external)
|
||||
REDPANDA_PROXY_PORT_OVERRIDE=18082
|
||||
|
||||
# Redpanda Admin API port (external)
|
||||
REDPANDA_ADMIN_PORT_OVERRIDE=19644
|
||||
|
||||
# Redpanda Console UI port
|
||||
REDPANDA_CONSOLE_PORT_OVERRIDE=8080
|
||||
|
||||
# Redpanda CPU cores for processing (default: 1)
|
||||
REDPANDA_SMP=1
|
||||
|
||||
# Redpanda memory allocation (default: 1G)
|
||||
REDPANDA_MEMORY=1G
|
||||
|
||||
# Redpanda log level (trace, debug, info, warn, error)
|
||||
REDPANDA_LOG_LEVEL=info
|
||||
|
||||
# Redpanda resource limits
|
||||
REDPANDA_CPU_LIMIT=2.0
|
||||
REDPANDA_MEMORY_LIMIT=2G
|
||||
REDPANDA_CPU_RESERVATION=1.0
|
||||
REDPANDA_MEMORY_RESERVATION=1G
|
||||
|
||||
# Redpanda Console resource limits
|
||||
REDPANDA_CONSOLE_CPU_LIMIT=0.5
|
||||
REDPANDA_CONSOLE_MEMORY_LIMIT=512M
|
||||
REDPANDA_CONSOLE_CPU_RESERVATION=0.25
|
||||
REDPANDA_CONSOLE_MEMORY_RESERVATION=256M
|
||||
183
src/redpanda/README.md
Normal file
183
src/redpanda/README.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Redpanda
|
||||
|
||||
[English](README.md) | [中文](README.zh.md)
|
||||
|
||||
Redpanda is a Kafka-compatible streaming data platform built for performance and developer experience. It is designed to be simple to deploy and manage, with no dependencies on JVM or Zookeeper.
|
||||
|
||||
## Features
|
||||
|
||||
- **Kafka API Compatible**: Works with existing Kafka clients and tools
|
||||
- **High Performance**: Built in C++ for maximum throughput and low latency
|
||||
- **No Zookeeper**: Simplified architecture with built-in consensus
|
||||
- **Schema Registry**: Built-in schema registry support
|
||||
- **Developer Friendly**: Easy to deploy and manage with minimal configuration
|
||||
- **Redpanda Console**: Modern web UI for monitoring and management
|
||||
|
||||
## Quick Start
|
||||
|
||||
1. Copy the environment file and customize if needed:
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
2. Start the services:
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
3. Access Redpanda Console at <http://localhost:8080>
|
||||
|
||||
4. Verify the cluster is healthy:
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk cluster health
|
||||
```
|
||||
|
||||
## Service Endpoints
|
||||
|
||||
- **Kafka API** (external): `localhost:19092`
|
||||
- **Schema Registry**: `localhost:18081`
|
||||
- **HTTP Proxy**: `localhost:18082`
|
||||
- **Admin API**: `localhost:19644`
|
||||
- **Redpanda Console**: `http://localhost:8080`
|
||||
|
||||
## Basic Usage
|
||||
|
||||
### Create a topic
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic create my-topic
|
||||
```
|
||||
|
||||
### List topics
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic list
|
||||
```
|
||||
|
||||
### Produce messages
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic produce my-topic
|
||||
```
|
||||
|
||||
Type messages and press Ctrl+C to exit.
|
||||
|
||||
### Consume messages
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic consume my-topic
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| -------------------------------- | --------- | ------------------------------------------- |
|
||||
| `REDPANDA_VERSION` | `v24.3.1` | Redpanda version |
|
||||
| `REDPANDA_CONSOLE_VERSION` | `v2.8.2` | Redpanda Console version |
|
||||
| `TZ` | `UTC` | Timezone |
|
||||
| `REDPANDA_KAFKA_PORT_OVERRIDE` | `19092` | Kafka API external port |
|
||||
| `REDPANDA_SCHEMA_PORT_OVERRIDE` | `18081` | Schema Registry port |
|
||||
| `REDPANDA_PROXY_PORT_OVERRIDE` | `18082` | HTTP Proxy port |
|
||||
| `REDPANDA_ADMIN_PORT_OVERRIDE` | `19644` | Admin API port |
|
||||
| `REDPANDA_CONSOLE_PORT_OVERRIDE` | `8080` | Console UI port |
|
||||
| `REDPANDA_SMP` | `1` | Number of CPU cores for processing |
|
||||
| `REDPANDA_MEMORY` | `1G` | Memory allocation |
|
||||
| `REDPANDA_LOG_LEVEL` | `info` | Log level (trace, debug, info, warn, error) |
|
||||
|
||||
## Resource Configuration
|
||||
|
||||
Default resource limits:
|
||||
|
||||
- **Redpanda**:
|
||||
- CPU: 1.0-2.0 cores
|
||||
- Memory: 1G-2G
|
||||
|
||||
- **Console**:
|
||||
- CPU: 0.25-0.5 cores
|
||||
- Memory: 256M-512M
|
||||
|
||||
Adjust these values in `.env` file based on your workload requirements.
|
||||
|
||||
## Production Considerations
|
||||
|
||||
### Performance Tuning
|
||||
|
||||
1. **CPU Cores**: Increase `REDPANDA_SMP` to match available CPU cores
|
||||
2. **Memory**: Allocate more memory with `REDPANDA_MEMORY` for better performance
|
||||
3. **Storage**: Use high-performance storage (SSD/NVMe) for production
|
||||
|
||||
### Cluster Mode
|
||||
|
||||
This configuration runs Redpanda in single-node development mode. For production clusters:
|
||||
|
||||
1. Remove `--mode dev-container` flag
|
||||
2. Configure multiple Redpanda instances
|
||||
3. Set appropriate replication factors
|
||||
4. Configure proper security (SASL, TLS)
|
||||
|
||||
### Security
|
||||
|
||||
For production use, consider:
|
||||
|
||||
1. Enable SASL authentication
|
||||
2. Configure TLS for encryption
|
||||
3. Set up ACLs for authorization
|
||||
4. Restrict network access
|
||||
|
||||
## Monitoring
|
||||
|
||||
Redpanda Console provides a comprehensive monitoring interface including:
|
||||
|
||||
- Cluster overview and health
|
||||
- Topic management and inspection
|
||||
- Consumer group monitoring
|
||||
- Schema registry management
|
||||
- Message browser
|
||||
|
||||
Additional metrics are available through Redpanda's Admin API at port 19644.
|
||||
|
||||
## Data Persistence
|
||||
|
||||
Data is stored in a Docker named volume `redpanda_data`. To backup or migrate:
|
||||
|
||||
```bash
|
||||
# Backup
|
||||
docker run --rm -v redpanda_redpanda_data:/data -v $(pwd):/backup alpine tar czf /backup/redpanda-backup.tar.gz -C /data .
|
||||
|
||||
# Restore
|
||||
docker run --rm -v redpanda_redpanda_data:/data -v $(pwd):/backup alpine tar xzf /backup/redpanda-backup.tar.gz -C /data
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container fails to start
|
||||
|
||||
1. Check available memory: Redpanda requires at least 1GB
|
||||
2. Verify port availability
|
||||
3. Check logs: `docker compose logs redpanda`
|
||||
|
||||
### Cannot connect to Kafka
|
||||
|
||||
1. Ensure ports are properly mapped
|
||||
2. Verify firewall settings
|
||||
3. Check advertised listeners configuration
|
||||
|
||||
### Performance issues
|
||||
|
||||
1. Increase `REDPANDA_SMP` to use more CPU cores
|
||||
2. Allocate more memory with `REDPANDA_MEMORY`
|
||||
3. Monitor resource usage in Console
|
||||
|
||||
## References
|
||||
|
||||
- [Redpanda Documentation](https://docs.redpanda.com/)
|
||||
- [Redpanda GitHub](https://github.com/redpanda-data/redpanda)
|
||||
- [Redpanda Console](https://docs.redpanda.com/current/manage/console/)
|
||||
- [rpk CLI Reference](https://docs.redpanda.com/current/reference/rpk/)
|
||||
|
||||
## License
|
||||
|
||||
Redpanda is licensed under the [Redpanda Business Source License](https://github.com/redpanda-data/redpanda/blob/dev/licenses/bsl.md).
|
||||
183
src/redpanda/README.zh.md
Normal file
183
src/redpanda/README.zh.md
Normal file
@@ -0,0 +1,183 @@
|
||||
# Redpanda
|
||||
|
||||
[English](README.md) | [中文](README.zh.md)
|
||||
|
||||
Redpanda 是一个与 Kafka 兼容的流数据平台，专为性能和开发者体验而构建。它设计简单易部署和管理，不依赖 JVM 或 Zookeeper。
|
||||
|
||||
## 特性
|
||||
|
||||
- **Kafka API 兼容**：可与现有的 Kafka 客户端和工具配合使用
|
||||
- **高性能**：使用 C++ 构建，实现最大吞吐量和低延迟
|
||||
- **无需 Zookeeper**：内置共识机制的简化架构
|
||||
- **Schema Registry**：内置 Schema Registry 支持
|
||||
- **开发者友好**：易于部署和管理，配置最少
|
||||
- **Redpanda Console**：用于监控和管理的现代化 Web UI
|
||||
|
||||
## 快速开始
|
||||
|
||||
1. 复制环境文件并根据需要自定义：
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
2. 启动服务：
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
3. 访问 Redpanda Console：<http://localhost:8080>
|
||||
|
||||
4. 验证集群健康状态：
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk cluster health
|
||||
```
|
||||
|
||||
## 服务端点
|
||||
|
||||
- **Kafka API**（外部）：`localhost:19092`
|
||||
- **Schema Registry**：`localhost:18081`
|
||||
- **HTTP Proxy**：`localhost:18082`
|
||||
- **Admin API**：`localhost:19644`
|
||||
- **Redpanda Console**：`http://localhost:8080`
|
||||
|
||||
## 基本使用
|
||||
|
||||
### 创建主题
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic create my-topic
|
||||
```
|
||||
|
||||
### 列出主题
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic list
|
||||
```
|
||||
|
||||
### 生产消息
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic produce my-topic
|
||||
```
|
||||
|
||||
输入消息后按 Ctrl+C 退出。
|
||||
|
||||
### 消费消息
|
||||
|
||||
```bash
|
||||
docker compose exec redpanda rpk topic consume my-topic
|
||||
```
|
||||
|
||||
## 环境变量
|
||||
|
||||
| 变量 | 默认值 | 说明 |
|
||||
| -------------------------------- | --------- | ------------------------------------------- |
|
||||
| `REDPANDA_VERSION` | `v24.3.1` | Redpanda 版本 |
|
||||
| `REDPANDA_CONSOLE_VERSION` | `v2.8.2` | Redpanda Console 版本 |
|
||||
| `TZ` | `UTC` | 时区 |
|
||||
| `REDPANDA_KAFKA_PORT_OVERRIDE` | `19092` | Kafka API 外部端口 |
|
||||
| `REDPANDA_SCHEMA_PORT_OVERRIDE` | `18081` | Schema Registry 端口 |
|
||||
| `REDPANDA_PROXY_PORT_OVERRIDE` | `18082` | HTTP Proxy 端口 |
|
||||
| `REDPANDA_ADMIN_PORT_OVERRIDE` | `19644` | Admin API 端口 |
|
||||
| `REDPANDA_CONSOLE_PORT_OVERRIDE` | `8080` | Console UI 端口 |
|
||||
| `REDPANDA_SMP` | `1` | 用于处理的 CPU 核心数 |
|
||||
| `REDPANDA_MEMORY` | `1G` | 内存分配 |
|
||||
| `REDPANDA_LOG_LEVEL` | `info` | 日志级别（trace、debug、info、warn、error） |
|
||||
|
||||
## 资源配置
|
||||
|
||||
默认资源限制：
|
||||
|
||||
- **Redpanda**：
|
||||
- CPU：1.0-2.0 核
|
||||
- 内存：1G-2G
|
||||
|
||||
- **Console**：
|
||||
- CPU：0.25-0.5 核
|
||||
- 内存：256M-512M
|
||||
|
||||
根据您的工作负载需求在 `.env` 文件中调整这些值。
|
||||
|
||||
## 生产环境注意事项
|
||||
|
||||
### 性能调优
|
||||
|
||||
1. **CPU 核心**：增加 `REDPANDA_SMP` 以匹配可用的 CPU 核心
|
||||
2. **内存**：使用 `REDPANDA_MEMORY` 分配更多内存以获得更好的性能
|
||||
3. **存储**：在生产环境中使用高性能存储（SSD/NVMe）
|
||||
|
||||
### 集群模式
|
||||
|
||||
此配置在单节点开发模式下运行 Redpanda。对于生产集群：
|
||||
|
||||
1. 移除 `--mode dev-container` 标志
|
||||
2. 配置多个 Redpanda 实例
|
||||
3. 设置适当的副本因子
|
||||
4. 配置适当的安全性（SASL、TLS）
|
||||
|
||||
### 安全性
|
||||
|
||||
在生产环境中使用时，请考虑：
|
||||
|
||||
1. 启用 SASL 认证
|
||||
2. 配置 TLS 加密
|
||||
3. 设置 ACL 进行授权
|
||||
4. 限制网络访问
|
||||
|
||||
## 监控
|
||||
|
||||
Redpanda Console 提供全面的监控界面，包括：
|
||||
|
||||
- 集群概览和健康状态
|
||||
- 主题管理和检查
|
||||
- 消费者组监控
|
||||
- Schema Registry 管理
|
||||
- 消息浏览器
|
||||
|
||||
通过端口 19644 的 Redpanda Admin API 可以获取其他指标。
|
||||
|
||||
## 数据持久化
|
||||
|
||||
数据存储在 Docker 命名卷 `redpanda_data` 中。备份或迁移：
|
||||
|
||||
```bash
|
||||
# 备份
|
||||
docker run --rm -v redpanda_redpanda_data:/data -v $(pwd):/backup alpine tar czf /backup/redpanda-backup.tar.gz -C /data .
|
||||
|
||||
# 恢复
|
||||
docker run --rm -v redpanda_redpanda_data:/data -v $(pwd):/backup alpine tar xzf /backup/redpanda-backup.tar.gz -C /data
|
||||
```
|
||||
|
||||
## 故障排除
|
||||
|
||||
### 容器启动失败
|
||||
|
||||
1. 检查可用内存：Redpanda 至少需要 1GB
|
||||
2. 验证端口可用性
|
||||
3. 查看日志：`docker compose logs redpanda`
|
||||
|
||||
### 无法连接到 Kafka
|
||||
|
||||
1. 确保端口正确映射
|
||||
2. 验证防火墙设置
|
||||
3. 检查 advertised listeners 配置
|
||||
|
||||
### 性能问题
|
||||
|
||||
1. 增加 `REDPANDA_SMP` 以使用更多 CPU 核心
|
||||
2. 使用 `REDPANDA_MEMORY` 分配更多内存
|
||||
3. 在 Console 中监控资源使用情况
|
||||
|
||||
## 参考资料
|
||||
|
||||
- [Redpanda 文档](https://docs.redpanda.com/)
|
||||
- [Redpanda GitHub](https://github.com/redpanda-data/redpanda)
|
||||
- [Redpanda Console](https://docs.redpanda.com/current/manage/console/)
|
||||
- [rpk CLI 参考](https://docs.redpanda.com/current/reference/rpk/)
|
||||
|
||||
## 许可证
|
||||
|
||||
Redpanda 使用 [Redpanda Business Source License](https://github.com/redpanda-data/redpanda/blob/dev/licenses/bsl.md) 许可。
|
||||
86
src/redpanda/docker-compose.yaml
Normal file
86
src/redpanda/docker-compose.yaml
Normal file
@@ -0,0 +1,86 @@
|
||||
# Redpanda - A Kafka-compatible streaming data platform built for performance
|
||||
# This configuration provides a single-node Redpanda setup with console for monitoring
|
||||
|
||||
x-defaults: &defaults
|
||||
restart: unless-stopped
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 100m
|
||||
max-file: "3"
|
||||
|
||||
services:
|
||||
redpanda:
|
||||
<<: *defaults
|
||||
image: ${GLOBAL_REGISTRY:-}docker.redpanda.com/redpandadata/redpanda:${REDPANDA_VERSION:-v24.3.1}
|
||||
command:
|
||||
- redpanda
|
||||
- start
|
||||
- --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:${REDPANDA_KAFKA_PORT_OVERRIDE:-19092}
|
||||
- --advertise-kafka-addr internal://redpanda:9092,external://localhost:${REDPANDA_KAFKA_PORT_OVERRIDE:-19092}
|
||||
- --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:${REDPANDA_PROXY_PORT_OVERRIDE:-18082}
|
||||
- --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:${REDPANDA_PROXY_PORT_OVERRIDE:-18082}
|
||||
- --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:${REDPANDA_SCHEMA_PORT_OVERRIDE:-18081}
|
||||
- --rpc-addr redpanda:33145
|
||||
- --advertise-rpc-addr redpanda:33145
|
||||
- --mode dev-container
|
||||
- --smp ${REDPANDA_SMP:-1}
|
||||
- --default-log-level=${REDPANDA_LOG_LEVEL:-info}
|
||||
- --reserve-memory 0M
|
||||
- --memory ${REDPANDA_MEMORY:-1G}
|
||||
ports:
|
||||
- "${REDPANDA_SCHEMA_PORT_OVERRIDE:-18081}:8081"
|
||||
- "${REDPANDA_PROXY_PORT_OVERRIDE:-18082}:8082"
|
||||
- "${REDPANDA_KAFKA_PORT_OVERRIDE:-19092}:19092"
|
||||
- "${REDPANDA_ADMIN_PORT_OVERRIDE:-19644}:9644"
|
||||
volumes:
|
||||
- redpanda_data:/var/lib/redpanda/data
|
||||
environment:
|
||||
- TZ=${TZ:-UTC}
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "rpk cluster health | grep -E 'Healthy:.+true' || exit 1"]
|
||||
interval: 15s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${REDPANDA_CPU_LIMIT:-2.0}
|
||||
memory: ${REDPANDA_MEMORY_LIMIT:-2G}
|
||||
reservations:
|
||||
cpus: ${REDPANDA_CPU_RESERVATION:-1.0}
|
||||
memory: ${REDPANDA_MEMORY_RESERVATION:-1G}
|
||||
|
||||
console:
|
||||
<<: *defaults
|
||||
image: ${GLOBAL_REGISTRY:-}docker.redpanda.com/redpandadata/console:${REDPANDA_CONSOLE_VERSION:-v2.8.2}
|
||||
ports:
|
||||
- "${REDPANDA_CONSOLE_PORT_OVERRIDE:-8080}:8080"
|
||||
environment:
|
||||
- KAFKA_BROKERS=redpanda:9092
|
||||
- KAFKA_SCHEMAREGISTRY_ENABLED=true
|
||||
- KAFKA_SCHEMAREGISTRY_URLS=http://redpanda:8081
|
||||
- REDPANDA_ADMINAPI_ENABLED=true
|
||||
- REDPANDA_ADMINAPI_URLS=http://redpanda:9644
|
||||
- TZ=${TZ:-UTC}
|
||||
depends_on:
|
||||
redpanda:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/admin/health"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${REDPANDA_CONSOLE_CPU_LIMIT:-0.5}
|
||||
memory: ${REDPANDA_CONSOLE_MEMORY_LIMIT:-512M}
|
||||
reservations:
|
||||
cpus: ${REDPANDA_CONSOLE_CPU_RESERVATION:-0.25}
|
||||
memory: ${REDPANDA_CONSOLE_MEMORY_RESERVATION:-256M}
|
||||
|
||||
volumes:
|
||||
redpanda_data:
|
||||
Reference in New Issue
Block a user