compose-anything/apps/openlit/README.md

OpenLIT

OpenLIT is an open-source platform for AI engineering that provides OpenTelemetry-native observability, monitoring, and management tools for large language models (LLMs), GPUs, and vector databases.

Features

• 📈 Analytics Dashboard: Monitor your AI application's health and performance with detailed dashboards that track metrics, costs, and user interactions
• 🔌 OpenTelemetry-native Observability SDKs: Vendor-neutral SDKs to send traces and metrics to your existing observability tools
• 💲 Cost Tracking: Tailor cost estimations for specific models using custom pricing files for precise budgeting
• 🐛 Exceptions Monitoring: Quickly spot and resolve issues by tracking common exceptions and errors
• 💭 Prompt Management: Manage and version prompts using Prompt Hub for consistent and easy access across applications
• 🔑 API Keys and Secrets Management: Securely handle your API keys and secrets centrally
• 🎮 Experiment with different LLMs: Use OpenGround to explore, test and compare various LLMs side by side
• 🚀 Fleet Hub for OpAMP Management: Centrally manage and monitor OpenTelemetry Collectors across your infrastructure using the OpAMP (Open Agent Management Protocol) with secure TLS communication

Quick Start

1. Copy .env.example to .env and customize as needed:

   cp .env.example .env
   

2. Start the services:

   docker compose up -d
   

3. Access OpenLIT at http://localhost:3000

4. Login with default credentials:

   • Email: user@openlit.io
   • Password: openlituser

Components

This deployment includes:

• OpenLIT Platform: Main application providing the UI and API (Port: 3000)
• ClickHouse: Database for storing telemetry data (Ports: 8123, 9000)
• OTLP Receivers:
  • gRPC: Port 4317
  • HTTP: Port 4318

Integration with Your Applications

To integrate OpenLIT with your AI applications:

Python SDK

pip install openlit

import openlit

openlit.init(
    otlp_endpoint="http://localhost:4318"
)

TypeScript SDK

npm install openlit

import Openlit from 'openlit';

Openlit.init({ 
    otlpEndpoint: 'http://localhost:4318'
});

Environment Variables

Key configuration options (see .env.example for all options):

Variable	Description	Default
OPENLIT_VERSION	OpenLIT image version	latest
CLICKHOUSE_VERSION	ClickHouse version	24.4.1
OPENLIT_PORT_OVERRIDE	UI port on host	3000
OPENLIT_OTLP_HTTP_PORT_OVERRIDE	OTLP HTTP port	4318
OPENLIT_OTLP_GRPC_PORT_OVERRIDE	OTLP gRPC port	4317
OPENLIT_DB_PASSWORD	ClickHouse password	OPENLIT
OPAMP_ENVIRONMENT	OpAMP mode (development/production)	production

Resource Requirements

Default resource allocations:

• OpenLIT:
  • Limits: 1 CPU, 2GB RAM
  • Reservations: 0.25 CPU, 512MB RAM
• ClickHouse:
  • Limits: 2 CPU, 4GB RAM
  • Reservations: 0.5 CPU, 2GB RAM

Adjust these in .env file based on your workload.

OAuth Configuration (Optional)

To enable OAuth authentication:

1. Configure Google OAuth:

   GOOGLE_CLIENT_ID=your-client-id
   GOOGLE_CLIENT_SECRET=your-client-secret
   

2. Configure GitHub OAuth:

   GITHUB_CLIENT_ID=your-client-id
   GITHUB_CLIENT_SECRET=your-client-secret
   

OpAMP Fleet Hub

OpenLIT includes an OpAMP server for centralized management of OpenTelemetry Collectors:

• Access Fleet Hub at: http://localhost:3000/fleet-hub
• OpAMP endpoint: wss://localhost:4320/v1/opamp (production mode)
• API endpoint: http://localhost:8080

For production deployments, ensure proper TLS configuration:

OPAMP_ENVIRONMENT=production
OPAMP_TLS_INSECURE_SKIP_VERIFY=false
OPAMP_TLS_REQUIRE_CLIENT_CERT=true

Supported Integrations

OpenLIT provides automatic instrumentation for 50+ LLM providers, vector databases, and frameworks including:

• LLM Providers: OpenAI, Anthropic, Cohere, Azure OpenAI, Google Vertex AI, Bedrock, and more
• Vector Databases: Pinecone, Weaviate, ChromaDB, Qdrant, Milvus, and more
• Frameworks: LangChain, LlamaIndex, Haystack, AutoGen, CrewAI, and more

Health Checks

The OpenLIT service includes health checks to ensure proper startup. The service is considered healthy when:

• The web interface responds on the configured port
• Interval: 30 seconds
• Timeout: 10 seconds
• Start period: 60 seconds

Data Persistence

Data is persisted in Docker volumes:

• clickhouse_data: ClickHouse database files
• openlit_data: OpenLIT application data including SQLite database

Monitoring

Monitor your deployment:

# View logs
docker compose logs -f openlit

# Check service status
docker compose ps

# View resource usage
docker stats

Security Notes

1. Change default password: Update OPENLIT_DB_PASSWORD in production
2. TLS Configuration: For production, use proper TLS certificates with OpAMP
3. OAuth: Consider enabling OAuth for enhanced security
4. Network: By default, services are exposed on all interfaces. Consider using a reverse proxy in production

Troubleshooting

Service won't start

• Check logs: docker compose logs openlit
• Verify ClickHouse is healthy: docker compose ps
• Ensure ports are not in use

Cannot connect to OTLP endpoint

• Verify firewall settings
• Check port configuration in .env
• Ensure the endpoint URL is correct in your application

High resource usage

• Adjust resource limits in .env
• Monitor with: docker stats
• Consider scaling ClickHouse resources for large workloads

Documentation

• Official Documentation: https://docs.openlit.io/
• GitHub Repository: https://github.com/openlit/openlit
• Python SDK: https://github.com/openlit/openlit/tree/main/sdk/python
• TypeScript SDK: https://github.com/openlit/openlit/tree/main/sdk/typescript

License

OpenLIT is available under the Apache-2.0 license.

Support

• Slack Community
• Discord
• GitHub Issues
• X/Twitter