OpenTelemetry 全链路追踪实战：4步从代码埋点到完整可观测性平台搭建

微服务多了，排查问题靠”猜”？一次请求经过5个服务，哪一环出了问题，你能在5分钟内定位吗？本文带你用 OpenTelemetry 打通全链路追踪，从代码埋点到 Jaeger 可视化，一站式搞定。

---

背景：微服务时代的”侦探困境”

传统单体应用出问题，翻翻日志基本能解决。但进入微服务时代，一个用户请求可能经过：

用户 → API Gateway → 用户服务 → 订单服务 → 库存服务 → 支付服务 → 消息队列

某个环节响应慢了 200ms，到底是哪里的锅？没有全链路追踪，你只能一个服务一个服务地翻日志，效率极低。

OpenTelemetry（OTel） 是 CNCF 的可观测性标准，整合了 Tracing（追踪）、Metrics（指标）、Logs（日志）三大支柱，已成为云原生可观测性的事实标准。

---

核心概念速览

概念	说明	类比
Trace	一次完整请求的生命周期	一趟快递从发货到签收
Span	Trace中的单个操作单元	快递在某个中转站的处理记录
SpanContext	跨服务传播的上下文	快递单号（贯穿全程）
Exporter	数据导出组件	把数据发送到Jaeger/Zipkin/OTLP
Collector	数据收集处理中间层	数据清洗中转站

---

第1步：搭建可观测性后端（Jaeger + OTel Collector）

用 Docker Compose 一键启动全套基础设施：

# docker-compose.yml
version: '3.8'

services:
  # Jaeger 全链路追踪 UI + 存储
  jaeger:
    image: jaegertracing/all-in-one:1.54
    ports:
      - "16686:16686"   # Jaeger UI
      - "14250:14250"   # gRPC接收
      - "14268:14268"   # HTTP接收
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    networks:
      - observability

  # OpenTelemetry Collector - 统一接收、处理、转发
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "4317:4317"    # OTLP gRPC
      - "4318:4318"    # OTLP HTTP
      - "8888:8888"    # Prometheus metrics
    depends_on:
      - jaeger
    networks:
      - observability

  # Prometheus（可选，收集指标）
  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      - observability

networks:
  observability:
    driver: bridge

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8888"
  logging:
    loglevel: warn

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [jaeger, logging]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

# 启动所有服务
docker-compose up -d

# 验证
curl http://localhost:16686  # Jaeger UI
curl http://localhost:9090   # Prometheus

---

第2步：Python 服务接入 OpenTelemetry

安装依赖

pip install opentelemetry-sdk \
            opentelemetry-api \
            opentelemetry-exporter-otlp-proto-grpc \
            opentelemetry-instrumentation-fastapi \
            opentelemetry-instrumentation-requests \
            opentelemetry-instrumentation-sqlalchemy \
            fastapi uvicorn requests

核心配置：初始化 OTel SDK

# otel_setup.py
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader


def setup_telemetry(service_name: str, otel_endpoint: str = "localhost:4317"):
    """初始化 OpenTelemetry，一行代码完成全套配置"""

    # 定义服务资源（这些信息会附加到所有 Span 上）
    resource = Resource.create({
        SERVICE_NAME: service_name,
        "service.version": "1.0.0",
        "deployment.environment": "production",
    })

    # ===== 配置 Tracing =====
    tracer_provider = TracerProvider(resource=resource)
    otlp_exporter = OTLPSpanExporter(
        endpoint=otel_endpoint,
        insecure=True  # 生产环境请配置TLS
    )
    tracer_provider.add_span_processor(
        BatchSpanProcessor(otlp_exporter)
    )
    trace.set_tracer_provider(tracer_provider)

    # ===== 配置 Metrics =====
    metric_reader = PeriodicExportingMetricReader(
        OTLPMetricExporter(endpoint=otel_endpoint, insecure=True),
        export_interval_millis=30000  # 每30秒上报一次
    )
    meter_provider = MeterProvider(
        resource=resource,
        metric_readers=[metric_reader]
    )
    metrics.set_meter_provider(meter_provider)

    print(f"✅ OpenTelemetry 初始化完成: {service_name} -> {otel_endpoint}")
    return trace.get_tracer(service_name), metrics.get_meter(service_name)

订单服务实战代码

# order_service.py
from fastapi import FastAPI, HTTPException
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode
import requests
import time

from otel_setup import setup_telemetry

# 初始化 OTel
tracer, meter = setup_telemetry("order-service")

# 创建自定义指标
order_counter = meter.create_counter(
    "orders_total",
    description="Total number of orders created"
)
order_duration = meter.create_histogram(
    "order_processing_duration_ms",
    description="Order processing time in milliseconds"
)

app = FastAPI(title="Order Service")

# 自动为 FastAPI 所有路由注入 Trace
FastAPIInstrumentor.instrument_app(app)
# 自动为 requests 库注入 Trace（跨服务传播）
RequestsInstrumentor().instrument()


@app.post("/orders")
async def create_order(user_id: str, product_id: str, quantity: int):
    """创建订单 - 完整链路追踪示例"""

    start_time = time.time()

    # 获取当前 Span（由 FastAPIInstrumentor 自动创建）
    current_span = trace.get_current_span()
    current_span.set_attribute("order.user_id", user_id)
    current_span.set_attribute("order.product_id", product_id)
    current_span.set_attribute("order.quantity", quantity)

    try:
        # 子 Span 1：检查库存（会自动传播 TraceID 到库存服务）
        with tracer.start_as_current_span("check_inventory") as inv_span:
            inv_span.set_attribute("inventory.product_id", product_id)
            inv_resp = requests.get(
                f"http://inventory-service/stock/{product_id}",
                timeout=3
            )
            stock = inv_resp.json()["stock"]
            inv_span.set_attribute("inventory.stock", stock)

            if stock < quantity:
                inv_span.set_status(Status(StatusCode.ERROR, "库存不足"))
                raise HTTPException(status_code=400, detail="库存不足")

        # 子 Span 2：创建支付单
        with tracer.start_as_current_span("create_payment") as pay_span:
            pay_resp = requests.post(
                "http://payment-service/payments",
                json={"user_id": user_id, "amount": quantity * 99.9},
                timeout=5
            )
            pay_id = pay_resp.json()["payment_id"]
            pay_span.set_attribute("payment.id", pay_id)

        # 子 Span 3：写入数据库（SQLAlchemy 会自动 instrument）
        with tracer.start_as_current_span("save_order") as db_span:
            order_id = f"ORD_{int(time.time())}"
            db_span.set_attribute("db.order_id", order_id)
            # 实际项目中这里是 db.session.add(order)
            time.sleep(0.01)  # 模拟DB操作

        # 上报指标
        duration_ms = (time.time() - start_time) * 1000
        order_counter.add(1, {"status": "success", "product": product_id})
        order_duration.record(duration_ms, {"status": "success"})

        current_span.set_attribute("order.id", order_id)
        return {"order_id": order_id, "payment_id": pay_id, "status": "created"}

    except Exception as e:
        # 记录异常到 Span
        current_span.record_exception(e)
        current_span.set_status(Status(StatusCode.ERROR, str(e)))
        order_counter.add(1, {"status": "error", "product": product_id})
        raise

---

第3步：Go 服务接入（跨语言链路打通）

// main.go - Go 库存服务接入 OTel
package main

import (
    "context"
    "fmt"
    "net/http"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func initTracer(ctx context.Context) func() {
    // 连接 OTel Collector
    exp, _ := otlptracegrpc.New(ctx,
        otlptracegrpc.WithInsecure(),
        otlptracegrpc.WithEndpoint("localhost:4317"),
    )

    res, _ := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceName("inventory-service"),
            semconv.ServiceVersion("1.0.0"),
        ),
    )

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)

    // 注册 W3C TraceContext 传播器（跨语言链路关键！）
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return func() { tp.Shutdown(ctx) }
}

func stockHandler(w http.ResponseWriter, r *http.Request) {
    tracer := otel.Tracer("inventory-service")
    _, span := tracer.Start(r.Context(), "query_stock")
    defer span.End()

    productID := r.PathValue("productID")
    span.SetAttributes(semconv.DBOperation("SELECT"))

    // 实际查库存逻辑
    fmt.Fprintf(w, `{"product_id": "%s", "stock": 100}`, productID)
}

func main() {
    ctx := context.Background()
    shutdown := initTracer(ctx)
    defer shutdown()

    // otelhttp 自动接收上游传来的 TraceContext
    http.Handle("/stock/{productID}", otelhttp.NewHandler(
        http.HandlerFunc(stockHandler), 
        "stock-query",
    ))

    fmt.Println("库存服务启动: :8082")
    http.ListenAndServe(":8082", nil)
}

---

第4步：在 Jaeger UI 中分析链路

启动服务后，访问 http://localhost:16686：

关键操作

1. 搜索 Trace：选择 order-service，设置时间范围，点击搜索
2. 查看瀑布图：直观看到每个 Span 的耗时，拖慢的环节一目了然
3. 跨服务追踪：同一个 TraceID 会把 Python 订单服务 + Go 库存服务的 Span 串联在一张图上
4. 异常分析：出错的 Span 会标红，点击可看到完整异常堆栈

性能指标对比

场景	排查耗时（传统日志）	排查耗时（OTel 追踪）
单服务 500 错误	5-10 分钟	< 1 分钟
跨3个服务的慢请求	30-60 分钟	2-5 分钟
偶发性超时	几小时（可能复现不了）	10 分钟（从历史 Trace 分析）
数据库慢查询定位	需手动关联慢查询日志	直接在 Span 里看 SQL

---

生产环境注意事项

采样策略（必须配置！）

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased

# 只采样 10% 的请求（高流量场景必须）
sampler = ParentBased(root=TraceIdRatioBased(0.1))

tracer_provider = TracerProvider(
    resource=resource,
    sampler=sampler  # 添加此参数
)

敏感数据过滤

# otel-collector-config.yaml 中添加数据脱敏
processors:
  attributes:
    actions:
      - key: http.request.header.authorization
        action: delete  # 删除鉴权头
      - key: db.statement
        action: update
        value: "[REDACTED]"  # 替换 SQL 语句

---

总结

步骤	内容	时间成本
搭建基础设施	Docker Compose 启动 Jaeger + Collector	10 分钟
Python 服务接入	pip 安装 + 初始化 SDK + 自动 instrument	30 分钟
Go 服务接入	SDK 初始化 + otelhttp 中间件	20 分钟
配置采样和过滤	调整 YAML 配置	15 分钟

OpenTelemetry 已经是可观测性的工业标准，越早接入越好——等系统复杂了再补，成本是现在的 10 倍。

---

👤 作者简介

一枚在大中原腹地（河南）卖公有云的从业者，主营腾讯云/阿里云/火山云，曾踩坑无数，现专注AI大模型应用落地。关注公众号「AI热点日报」，围观AI前沿动态~

博客：y1cloud.cn

你们团队是怎么做可观测性的？Jaeger/Zipkin/SkyWalking 各有优劣，欢迎留言交流！