Observability: Monitoring, Logging & Tracing

Duration: 45 min

Observability: Monitoring, Logging & Tracing

Duration: 45 min

The Three Pillars

1. Metrics — Numeric measurements over time (CPU, request rate, error count) 2. Logs — Timestamped event records from your applications 3. Traces — End-to-end request path across services

Prometheus + Grafana (Metrics)

ServiceMonitor - tell Prometheus to scrape your app

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-metrics
spec:
  selector:
    matchLabels:
      app: api
  endpoints:
    - port: http
      path: /metrics
      interval: 15s


Instrument your Python app

from prometheus_client import Counter, Histogram, start_http_server
REQUEST_COUNT = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
REQUEST_LATENCY = Histogram('http_request_duration_seconds', 'Request latency', ['endpoint'])
@app.middleware("http")
async def metrics_middleware(request, call_next):
    with REQUEST_LATENCY.labels(endpoint=request.url.path).time():
        response = await call_next(request)
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.url.path,
        status=response.status_code
    ).inc()
    return response
Expose metrics on :9090/metrics

start_http_server(9090)

Grafana Dashboard Alert

PrometheusRule - alert when error rate > 5%

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-alerts
spec:
  groups:
    - name: api
      rules:
        - alert: HighErrorRate
          expr: |
            rate(http_requests_total{status=~"5.."}[5m])
            / rate(http_requests_total[5m]) > 0.05
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "High error rate on {{ $labels.endpoint }}"

Centralized Logging (ELK / Loki)

Structured logging in your app (JSON format)


Python example:

import structlog
logger = structlog.get_logger()
logger.info("request_processed",
    method="POST",
    path="/api/users",
    status=201,
    duration_ms=45,
    user_id="u-12345"
)

Output: {"event":"request_processed","method":"POST","path":"/api/users",...}



Loki + Promtail for Kubernetes log collection


promtail-config.yaml

scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace

Distributed Tracing (OpenTelemetry)

Auto-instrument your app with OpenTelemetry

from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
Setup

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
Auto-instrument frameworks

FastAPIInstrumentor.instrument_app(app)
HTTPXClientInstrumentor().instrument()
Manual spans for custom logic

tracer = trace.get_tracer(__name__)
async def process_order(order_id: str):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        # ... your logic here

SLOs and Error Budgets

Define Service Level Objectives


SLI: 99.9% of requests complete in < 500ms


SLO: 99.9% availability over 30 days


Error Budget: 0.1% = ~43 minutes of downtime/month
PromQL for SLO tracking:


Availability SLO:


1 - (sum(rate(http_requests_total{status=~"5.."}[30d]))


     / sum(rate(http_requests_total[30d])))
Latency SLO:


sum(rate(http_request_duration_seconds_bucket{le="0.5"}[30d]))


/ sum(rate(http_request_duration_seconds_count[30d]))

Quiz

Q1: What are the three pillars of observability?

  • A) CPU, Memory, Disk
  • B) Metrics, Logs, Traces ✓
  • C) Build, Test, Deploy
  • D) Dev, Staging, Production

Q2: What does a distributed trace show you?

  • A) Server CPU usage
  • B) The full path of a request across multiple services with timing ✓
  • C) Container memory usage
  • D) Git commit history

Q3: What is an "error budget"?

  • A) Money set aside for fixing bugs
  • B) The allowed amount of downtime/errors before violating your SLO ✓
  • C) The number of developers on the team
  • D) Maximum log file size

← Back ▶ Run in Colab