feat: add more otel services

2026-01-11 23:42:34 +08:00
parent ea5eadfcec
commit 51fd7ea08b
28 changed files with 2358 additions and 70 deletions
--- a/src/loki/.env.example
+++ b/src/loki/.env.example
@@ -0,0 +1,19 @@
+# Global Registry (optional)
+# GLOBAL_REGISTRY=registry.example.com/
+
+# Loki Version
+LOKI_VERSION=3.3.2
+
+# Port Override
+LOKI_PORT_OVERRIDE=3100
+
+# Timezone
+TZ=UTC
+
+# Resource Limits
+LOKI_CPU_LIMIT=1.0
+LOKI_MEMORY_LIMIT=1G
+
+# Resource Reservations
+LOKI_CPU_RESERVATION=0.25
+LOKI_MEMORY_RESERVATION=256M
--- a/src/loki/README.md
+++ b/src/loki/README.md
@@ -0,0 +1,144 @@
+# Grafana Loki
+
+[中文文档](README.zh.md)
+
+Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. Unlike other logging systems, Loki does not index the contents of the logs, but rather a set of labels for each log stream.
+
+## Features
+
+- **Cost-effective**: Only indexes metadata instead of full text, significantly reducing storage costs
+- **LogQL**: Powerful query language similar to PromQL for filtering and aggregating logs
+- **Multi-tenancy**: Built-in support for multi-tenant deployments
+- **Grafana Integration**: Native integration with Grafana for visualization
+- **Scalable**: Can scale horizontally to handle large volumes of logs
+
+## Quick Start
+
+1. Copy the example environment file:
+
+    ```bash
+    cp .env.example .env
+    ```
+
+2. Start the service:
+
+    ```bash
+    docker compose up -d
+    ```
+
+3. Verify the service is running:
+
+    ```bash
+    docker compose ps
+    curl http://localhost:3100/ready
+    ```
+
+## Configuration
+
+### Environment Variables
+
+| Variable                  | Default | Description        |
+| ------------------------- | ------- | ------------------ |
+| `LOKI_VERSION`            | `3.3.2` | Loki version       |
+| `LOKI_PORT_OVERRIDE`      | `3100`  | HTTP API port      |
+| `TZ`                      | `UTC`   | Timezone           |
+| `LOKI_CPU_LIMIT`          | `1.0`   | CPU limit          |
+| `LOKI_MEMORY_LIMIT`       | `1G`    | Memory limit       |
+| `LOKI_CPU_RESERVATION`    | `0.25`  | CPU reservation    |
+| `LOKI_MEMORY_RESERVATION` | `256M`  | Memory reservation |
+
+### Default Configuration
+
+The service includes a basic configuration file (`loki-config.yaml`) that:
+
+- Disables authentication (suitable for development/testing)
+- Uses local filesystem storage
+- Configures a single replica (monolithic mode)
+- Sets up basic caching for query performance
+
+For production deployments, you should customize the configuration based on your requirements.
+
+## Integration with Grafana
+
+1. Add Loki as a data source in Grafana:
+   - URL: `http://loki:3100` (if running in the same Docker network)
+   - Or: `http://localhost:3100` (from host machine)
+
+2. Create dashboards and explore logs using LogQL queries
+
+## Sending Logs to Loki
+
+### Using Promtail
+
+Promtail is the recommended agent for shipping logs to Loki:
+
+```yaml
+services:
+  promtail:
+    image: grafana/promtail:3.3.2
+    volumes:
+      - /var/log:/var/log:ro
+      - ./promtail-config.yaml:/etc/promtail/config.yaml
+    command: -config.file=/etc/promtail/config.yaml
+```
+
+### Using Docker Driver
+
+You can configure Docker to send logs directly to Loki:
+
+```yaml
+logging:
+  driver: loki
+  options:
+    loki-url: "http://localhost:3100/loki/api/v1/push"
+    loki-batch-size: "400"
+```
+
+### Using HTTP API
+
+Send logs directly via HTTP POST:
+
+```bash
+curl -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \
+  '{"streams": [{ "stream": { "job": "test" }, "values": [ [ "$(date +%s)000000000", "test log message" ] ] }]}'
+```
+
+## Storage
+
+Logs and metadata are stored in a Docker volume named `loki_data`.
+
+## Health Check
+
+The service includes a health check that monitors the `/ready` endpoint every 30 seconds.
+
+## Resource Requirements
+
+- **Minimum**: 256MB RAM, 0.25 CPU
+- **Recommended**: 1GB RAM, 1 CPU (for moderate log volumes)
+- **Production**: Scale based on log ingestion rate and retention period
+
+## Security Considerations
+
+The default configuration:
+
+- Runs as non-root user (UID:GID 10001:10001)
+- Disables authentication (suitable for development only)
+- Uses filesystem storage (not suitable for distributed deployments)
+
+For production:
+
+- Enable authentication and multi-tenancy
+- Use object storage (S3, GCS, Azure Blob, etc.)
+- Implement proper network security and access controls
+- Configure retention policies to manage storage costs
+
+## Documentation
+
+- [Official Documentation](https://grafana.com/docs/loki/latest/)
+- [LogQL Query Language](https://grafana.com/docs/loki/latest/query/)
+- [Best Practices](https://grafana.com/docs/loki/latest/operations/best-practices/)
+- [GitHub Repository](https://github.com/grafana/loki)
+
+## License
+
+Loki is licensed under the [AGPLv3 License](https://github.com/grafana/loki/blob/main/LICENSE).
--- a/src/loki/README.zh.md
+++ b/src/loki/README.zh.md
@@ -0,0 +1,144 @@
+# Grafana Loki
+
+[English Documentation](README.md)
+
+Grafana Loki 是一个受 Prometheus 启发的水平可扩展、高可用、多租户日志聚合系统。它被设计为非常高效且易于操作。与其他日志系统不同，Loki 不索引日志的内容，而是为每个日志流索引一组标签。
+
+## 特性
+
+- **成本效益**：仅索引元数据而非全文，显著降低存储成本
+- **LogQL**：类似于 PromQL 的强大查询语言，用于过滤和聚合日志
+- **多租户**：内置对多租户部署的支持
+- **Grafana 集成**：与 Grafana 原生集成进行可视化
+- **可扩展**：可以水平扩展以处理大量日志
+
+## 快速开始
+
+1. 复制示例环境文件：
+
+    ```bash
+    cp .env.example .env
+    ```
+
+2. 启动服务：
+
+    ```bash
+    docker compose up -d
+    ```
+
+3. 验证服务正在运行：
+
+    ```bash
+    docker compose ps
+    curl http://localhost:3100/ready
+    ```
+
+## 配置
+
+### 环境变量
+
+| 变量                      | 默认值  | 描述          |
+| ------------------------- | ------- | ------------- |
+| `LOKI_VERSION`            | `3.3.2` | Loki 版本     |
+| `LOKI_PORT_OVERRIDE`      | `3100`  | HTTP API 端口 |
+| `TZ`                      | `UTC`   | 时区          |
+| `LOKI_CPU_LIMIT`          | `1.0`   | CPU 限制      |
+| `LOKI_MEMORY_LIMIT`       | `1G`    | 内存限制      |
+| `LOKI_CPU_RESERVATION`    | `0.25`  | CPU 预留      |
+| `LOKI_MEMORY_RESERVATION` | `256M`  | 内存预留      |
+
+### 默认配置
+
+该服务包含一个基本配置文件（`loki-config.yaml`），该文件：
+
+- 禁用身份验证（适用于开发/测试）
+- 使用本地文件系统存储
+- 配置单副本（单体模式）
+- 设置基本缓存以提高查询性能
+
+对于生产部署，您应该根据需求自定义配置。
+
+## 与 Grafana 集成
+
+1. 在 Grafana 中添加 Loki 作为数据源：
+   - URL：`http://loki:3100`（如果在同一 Docker 网络中运行）
+   - 或者：`http://localhost:3100`（从主机访问）
+
+2. 使用 LogQL 查询创建仪表板和探索日志
+
+## 向 Loki 发送日志
+
+### 使用 Promtail
+
+Promtail 是向 Loki 发送日志的推荐代理：
+
+```yaml
+services:
+  promtail:
+    image: grafana/promtail:3.3.2
+    volumes:
+      - /var/log:/var/log:ro
+      - ./promtail-config.yaml:/etc/promtail/config.yaml
+    command: -config.file=/etc/promtail/config.yaml
+```
+
+### 使用 Docker 驱动
+
+您可以配置 Docker 直接向 Loki 发送日志：
+
+```yaml
+logging:
+  driver: loki
+  options:
+    loki-url: "http://localhost:3100/loki/api/v1/push"
+    loki-batch-size: "400"
+```
+
+### 使用 HTTP API
+
+通过 HTTP POST 直接发送日志：
+
+```bash
+curl -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \
+  '{"streams": [{ "stream": { "job": "test" }, "values": [ [ "$(date +%s)000000000", "test log message" ] ] }]}'
+```
+
+## 存储
+
+日志和元数据存储在名为 `loki_data` 的 Docker 卷中。
+
+## 健康检查
+
+该服务包含健康检查，每 30 秒监控一次 `/ready` 端点。
+
+## 资源需求
+
+- **最低要求**：256MB RAM，0.25 CPU
+- **推荐配置**：1GB RAM，1 CPU（用于中等日志量）
+- **生产环境**：根据日志摄入速率和保留期限进行扩展
+
+## 安全注意事项
+
+默认配置：
+
+- 以非 root 用户运行（UID:GID 10001:10001）
+- 禁用身份验证（仅适用于开发环境）
+- 使用文件系统存储（不适用于分布式部署）
+
+对于生产环境：
+
+- 启用身份验证和多租户
+- 使用对象存储（S3、GCS、Azure Blob 等）
+- 实施适当的网络安全和访问控制
+- 配置保留策略以管理存储成本
+
+## 文档
+
+- [官方文档](https://grafana.com/docs/loki/latest/)
+- [LogQL 查询语言](https://grafana.com/docs/loki/latest/query/)
+- [最佳实践](https://grafana.com/docs/loki/latest/operations/best-practices/)
+- [GitHub 仓库](https://github.com/grafana/loki)
+
+## 许可证
+
+Loki 使用 [AGPLv3 许可证](https://github.com/grafana/loki/blob/main/LICENSE)。
--- a/src/loki/docker-compose.yaml
+++ b/src/loki/docker-compose.yaml
@@ -0,0 +1,38 @@
+x-defaults: &defaults
+  restart: unless-stopped
+  logging:
+    driver: json-file
+    options:
+      max-size: 100m
+      max-file: "3"
+
+services:
+  loki:
+    <<: *defaults
+    image: ${GLOBAL_REGISTRY:-}grafana/loki:${LOKI_VERSION:-3.3.2}
+    ports:
+      - "${LOKI_PORT_OVERRIDE:-3100}:3100"
+    volumes:
+      - loki_data:/loki
+      - ./loki-config.yaml:/etc/loki/local-config.yaml:ro
+    environment:
+      - TZ=${TZ:-UTC}
+    command: -config.file=/etc/loki/local-config.yaml
+    user: "10001:10001"  # Loki user
+    deploy:
+      resources:
+        limits:
+          cpus: ${LOKI_CPU_LIMIT:-1.0}
+          memory: ${LOKI_MEMORY_LIMIT:-1G}
+        reservations:
+          cpus: ${LOKI_CPU_RESERVATION:-0.25}
+          memory: ${LOKI_MEMORY_RESERVATION:-256M}
+    healthcheck:
+      test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3100/ready"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+
+volumes:
+  loki_data:
--- a/src/loki/loki-config.yaml
+++ b/src/loki/loki-config.yaml
@@ -0,0 +1,50 @@
+auth_enabled: false
+
+server:
+  http_listen_port: 3100
+  grpc_listen_port: 9096
+
+common:
+  instance_addr: 127.0.0.1
+  path_prefix: /loki
+  storage:
+    filesystem:
+      chunks_directory: /loki/chunks
+      rules_directory: /loki/rules
+  replication_factor: 1
+  ring:
+    kvstore:
+      store: inmemory
+
+query_range:
+  results_cache:
+    cache:
+      embedded_cache:
+        enabled: true
+        max_size_mb: 100
+
+schema_config:
+  configs:
+    - from: 2020-10-24
+      store: tsdb
+      object_store: filesystem
+      schema: v13
+      index:
+        prefix: index_
+        period: 24h
+
+ruler:
+  alertmanager_url: http://localhost:9093
+
+# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
+# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
+#
+# Statistics help us better understand how Loki is used, and they show us performance
+# levels for most users. This helps us prioritize features and documentation.
+# For more information on what's sent, look at
+# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
+# Refer to the buildReport method to see what goes into a report.
+#
+# If you would like to disable reporting, uncomment the following lines:
+#analytics:
+#  reporting_enabled: false