Skip to main contentMonitoring Router Performance
Learn how to track, analyze, and optimize router performance in production applications.
Overview
Effective monitoring is essential for understanding how smart routing behaves in production. This guide covers tracking routing decisions, analyzing performance metrics, and optimizing based on real usage data.
What you’ll learn:
- Tracking routing decisions and model selection
- Measuring routing latency and overhead
- Analyzing cost patterns
- Identifying routing failures
- Building monitoring dashboards
- Optimizing based on metrics
Basic Monitoring
Tracking Selected Models
Track which models the router selects for your requests:
Advanced Monitoring
Comprehensive Metrics Collection
Collect detailed metrics for analysis:
Cost Tracking
Monitoring Routing Costs
Track estimated costs by model and over time:
Prometheus Metrics Export
Export metrics for Prometheus monitoring:
DataDog Integration
Send routing metrics to DataDog:
Alerting and Notifications
Set Up Alerts for Routing Issues
Monitor critical metrics and send alerts:
Best Practices
Monitoring Checklist
✅ Essential Metrics
- Model selection distribution
- Routing latency (P50, P95, P99)
- Error rates and types
- Request volume over time
- Cost per model and total
✅ Performance Monitoring
- Track latency trends
- Monitor for latency regressions
- Alert on high latency (>2s routing time)
- Compare routed vs. fixed model performance
✅ Cost Monitoring
- Track daily/weekly spending
- Monitor cost per model
- Alert on budget thresholds
- Analyze cost optimization opportunities
✅ Error Monitoring
- Track error rates by pool
- Monitor routing failures
- Set up alerts for error spikes
- Log errors for debugging
❌ Avoid
- Monitoring without actionable alerts
- Ignoring cost patterns
- Missing latency regressions
- Not tracking model distribution
Next Steps
