🎯 Problem Statement

Modern Kubernetes environments face observability challenges:

  • Visibility Gap - No centralized view of cluster and application health
  • Troubleshooting - Difficulty correlating metrics, logs, and traces
  • Alert Fatigue - Too many false positives or missing critical alerts
  • Cost - Expensive commercial APM solutions
  • Scalability - Monitoring solution must scale with the cluster

💡 Solution Overview

I designed and deployed a comprehensive open-source observability stack using Prometheus, Grafana, Loki, and Tempo, providing metrics, logs, traces, and alerting in a unified platform.

🏗️ Architecture

┌│├││││││││││││││││││││││││││││││││││││││││││││││││└────────┌│││││└┌│││││││││││││││││││││││││││││││││││└──────────────────┌││└─────────────────A/─────────pm───┌││││└┌│││└┌│││││││└┌│││└┌││└──────pe───────────────────t┬┼│└─────────────────Ar─────---──--──┌││││└──(─────────i─────────────L──-───────c─────SRA──RS───D---───o─────────s─────eel──ol───a───g──C──────┐││┘────rce──ua───tPLT─────o──────────vor──tc───aroe───LA──l──────┌││└───O─irt──ik───okm───og──l───────────b─cd──n/───Smip───kg──e───────A/────s─eiR──gP───oeo───ir──Pc────K───pm────e─PMnu──Aa───ut(───e──rt────u───pe────r─rogl──lg───rhL(───g──os────b───t┬┼│┴─v─one──ee───ceoT───a──m────e──A─Br────a─miRs──rr──G─eugr───t──tl────r──p─i────b─etu──tD──r─sssa───i──ao────n──p─c────i─tol──mu──a─:)c───o──ig────e──l─s────l─hre──at──f─(e───n──ls────t──i┐││┘───i─es──ny──a─Ms───)──────e──c─┬│┼t│▼uC┬│▼a──n─e)──┐│││┘─(f────s──a┌││└───y─sR──g──a─t───Dr──────t──────D──e───r───ao────C──i─A/────N─s──r───i──┌│││└─em────l──o─pm────a──────c─────m────u──n─pe────m──────s───(──oa────s───t┬┼│┴─e──────)───DT──nl────t──P─Cr────s─────────ir──Sl────e──o─i────p─────────sa──e────r──d─c────a─────────tc──tp──────s─s────c─────────Tri──)o──────┐││┘───e─────────ein──d──────────────────mbg──s──────┌││└────────────pu)──────────────────────ot─────────A/─────────────e─────────pm─────────────d─────────pe──────────────────────t┬┼│┘──────┐│││┘───────────Dr───┐│┘┐││┘┐││││││┘──┐││┘──────i───││││││────────c───────────s──────────┐││┘──┐│││┘─────────────────────────────────────────────────────────────││││──────┐┘─────│┐│││││││││││││││││││││││────││││││││││││┘───││││─┐┤││┘│││││││││││││││││││││││││││││││││││││││││││

🛠️ Technologies Used

ComponentTechnologyPurpose
MetricsPrometheusTime-series metrics collection
VisualizationGrafanaDashboards and alerting UI
LogsLoki + PromtailLog aggregation
TracesTempoDistributed tracing
AlertingAlertmanagerAlert routing and notifications
DeploymentHelm + ArgoCDGitOps-based deployment
StorageThanosLong-term metrics storage

📁 Project Structure

k├│││││├││││├││││├│└8─────s─────-oh└d├├├└a├├├└s└kbe─a────l────e─usl─s────e────r─semhrvtr/v├├├└bcpnatnpasiaoa────oloopsoplcpmal────auddpddpoepiburs-ele-l-mlziepgltdtm-i-aiaoialsrroeseeecalclncti/oakm/rtxaleaeiaitmfip-rptertrttoyea-ooioirtitoin/tnv-vcrotsosro.haavestns.n.snye-lar.e-.y-y/-auvulvjrmyaaammsaeuis.eamlmol-lseeojtmlelnvu.swnsrlriaey..oittlsayjncsou.mass.reylmo.yssalnja..msmyylolaanmmll

🔧 Implementation Highlights

Prometheus ServiceMonitor

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
  namespace: observability
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: my-application
  namespaceSelector:
    matchNames:
      - production
      - staging
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics
      scrapeTimeout: 10s

Alert Rules for SLOs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: slo-alerts
  namespace: observability
spec:
  groups:
    - name: slo.rules
      rules:
        - alert: HighErrorRate
          expr: |
            (
              sum(rate(http_requests_total{status=~"5.."}[5m]))
              /
              sum(rate(http_requests_total[5m]))
            ) > 0.01
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate detected"
            description: "Error rate is {{ $value | humanizePercentage }} (>1% threshold)"
            runbook_url: "https://runbooks.example.com/high-error-rate"

        - alert: HighLatencyP99
          expr: |
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
            ) > 0.5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "P99 latency is high"
            description: "P99 latency for {{ $labels.service }} is {{ $value }}s"

Grafana Dashboard as Code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: observability
  labels:
    grafana_dashboard: "1"
data:
  cluster-overview.json: |
    {
      "dashboard": {
        "title": "Cluster Overview",
        "panels": [
          {
            "title": "CPU Usage",
            "type": "graph",
            "targets": [
              {
                "expr": "sum(rate(container_cpu_usage_seconds_total[5m])) by (namespace)",
                "legendFormat": "{{namespace}}"
              }
            ]
          },
          {
            "title": "Memory Usage",
            "type": "graph",
            "targets": [
              {
                "expr": "sum(container_memory_usage_bytes) by (namespace) / 1024 / 1024 / 1024",
                "legendFormat": "{{namespace}} (GB)"
              }
            ]
          }
        ]
      }
    }

Loki Log Queries

1
2
3
4
5
6
7
8

# Error logs from production namespace
{namespace="production"} |= "error" | json | line_format "{{.message}}"

# Slow requests (>1s response time)
{app="api-gateway"} | json | response_time > 1000

# Failed Kubernetes events
{job="kubernetes-events"} | json | type="Warning"

📊 Dashboards Created

DashboardPurpose
Cluster OverviewNode health, resource utilization, pod status
Namespace MetricsPer-namespace resource consumption
Application SLOsError rates, latency percentiles, throughput
Node ExporterDetailed node-level metrics
Kubernetes EventsCluster events and warnings
Cost AnalysisResource costs by namespace/team

🔔 Alerting Strategy

Alert Severity Levels

SeverityResponse TimeNotification Channel
CriticalImmediatePagerDuty + Slack
Warning1 hourSlack
InfoNext business dayEmail

SLO-Based Alerting

  • Availability SLO: 99.9% uptime
  • Latency SLO: P99 < 500ms
  • Error Budget: Alert when 50% consumed in 24h

📈 Results & Metrics

MetricAchievement
MTTRReduced from 2 hours to 15 minutes
Alert NoiseReduced false positives by 80%
Cost Savings$50k/year vs commercial APM
Log Retention30 days with 99.9% query performance
Dashboard Load< 3 seconds for complex queries

🔑 Key Learnings

  1. Start with SLOs - Define SLOs first, then build alerts around them
  2. Reduce Cardinality - High cardinality labels kill Prometheus performance
  3. Retention Tiers - Different retention for different metric types
  4. Runbooks - Every alert should link to a runbook
  5. On-Call Experience - Invest in alert routing and escalation

🔗 Resources

Need help with observability? Contact me!