OpenTelemetry: Трассировка микросервисов — руководство 2026

┌─────────────────────────────────────────────────────────┐

│                  Ваше приложение                         │

│                                                          │

│  ┌─────────────────────────────────────────────────┐    │

│  │  OTel SDK (Python / Go / Java / ...)            │    │

│  │  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │    │

│  │  │  Tracer  │  │  Meter   │  │  Logger      │  │    │

│  │  │ Provider │  │ Provider │  │  Provider    │  │    │

│  │  └────┬─────┘  └────┬─────┘  └──────┬───────┘  │    │

│  │       │              │               │           │    │

│  │  ┌────▼──────────────▼───────────────▼───────┐  │    │

│  │  │          OTLP Exporter (gRPC / HTTP)       │  │    │

│  └──┴────────────────────┬──────────────────────-┘  │    │

└───────────────────────────┼─────────────────────────┘

                            │ OTLP

                    ┌───────▼──────────┐

                    │  OTel Collector   │

                    │  ┌─────────────┐ │

                    │  │  Receivers  │ │  ← принимает OTLP, Jaeger, Zipkin

                    │  │  Processors │ │  ← batch, filter, transform

                    │  │  Exporters  │ │  ← отправляет в бэкенды

                    │  └─────────────┘ │

                    └───────┬──────────┘

               ┌────────────┼─────────────┐

               ▼            ▼             ▼

           Jaeger      Grafana Tempo   Datadog/NR

Порты по умолчанию:

  OTLP gRPC:      4317

  OTLP HTTP:      4318

  Jaeger (legacy) 6831/UDP, 14268/HTTP

  Zipkin (legacy) 9411

: a3f2e1b9c7d5e3f1a2b4c6d8e0f2a4b6   ← уникальный ID всего запроса

│

├── Span: api-gateway (100ms)

│   SpanID: f1e2d3c4b5a6

│   Attributes: http.method=GET, http.url=/api/orders

│   │

│   ├── Span: order-service (60ms)

│   │   SpanID: a1b2c3d4e5f6

│   │   Attributes: rpc.method=GetOrder

│   │   │

│   │   ├── Span: postgres query (15ms)

│   │   │   db.statement: SELECT * FROM orders WHERE id=?

│   │   │

│   │   └── Span: redis get (3ms)

│   │       db.system: redis

│   │

│   └── Span: notification-service (10ms)

│       SpanID: b2c3d4e5f6a1

│       Status: ERROR ← здесь что-то пошло не так

│       Events: exception.message="Connection timeout"

TraceContext заголовки:

  traceparent: 00-a3f2e1b9c7d5e3f1a2b4c6d8e0f2a4b6-f1e2d3c4b5a6-01

  tracestate:  vendor-specific-data

<h2 id="docker-compose-yml">docker-compose.yml</h2>

version: '3.9'



services:

  jaeger:

    image: jaegertracing/all-in-one:1.55

    ports:

      - "16686:16686"   # Jaeger UI

      - "14250:14250"   # gRPC для Collector → Jaeger

    environment:

      - COLLECTOR_OTLP_ENABLED=true



  otel-collector:

    image: otel/opentelemetry-collector-contrib:0.96.0

    command: ["--config=/etc/otel-config.yaml"]

    volumes:

      - ./otel-config.yaml:/etc/otel-config.yaml:ro

    ports:

      - "4317:4317"     # OTLP gRPC (от приложений)

      - "4318:4318"     # OTLP HTTP (от приложений)

      - "8888:8888"     # Collector self-metrics

    depends_on:

      - jaeger



  # Опционально: Grafana + Tempo для альтернативного бэкенда

  tempo:

    image: grafana/tempo:2.4.0

    command: ["-config.file=/etc/tempo.yaml"]

    volumes:

      - ./tempo.yaml:/etc/tempo.yaml:ro

    ports:

      - "3200:3200"     # Tempo HTTP



  grafana:

    image: grafana/grafana:10.3.0

    ports:

      - "3000:3000"

    environment:

      - GF_SECURITY_ADMIN_PASSWORD=admin

<h2 id="otel-config-yaml">otel-config.yaml</h2>

receivers:

  otlp:

    protocols:

      grpc:

        endpoint: 0.0.0.0:4317

      http:

        endpoint: 0.0.0.0:4318



processors:

  batch:

    timeout: 1s

    send_batch_size: 1024

  memory_limiter:

    limit_mib: 256



exporters:

  jaeger:

    endpoint: jaeger:14250

    tls:

      insecure: true

  debug:

    verbosity: normal



service:

  pipelines:

    traces:

      receivers:  [otlp]

      processors: [memory_limiter, batch]

      exporters:  [jaeger, debug]

<h2 id="zapusk-steka">Запуск стека</h2>

docker-compose up -d



<h2 id="proverka-jaeger-ui">Проверка: Jaeger UI</h2>

open http://localhost:16686



<h2 id="proverka-health-collector">Проверка health Collector</h2>

curl http://localhost:8888/metrics | grep otelcol_receiver

<h2 id="otpravka-testovogo-span-cherez-otlp-http">Отправка тестового span через OTLP HTTP</h2>

curl -X POST http://localhost:4318/v1/traces \

  -H "Content-Type: application/json" \

  -d '{

    "resourceSpans": [{

      "resource": {

        "attributes": [{"key": "service.name", "value": {"stringValue": "test-service"}}]

      },

      "scopeSpans": [{

        "spans": [{

          "traceId": "a3f2e1b9c7d5e3f1a2b4c6d8e0f2a4b6",

          "spanId": "f1e2d3c4b5a6f7e8",

          "name": "test-operation",

          "kind": 2,

          "startTimeUnixNano": "1700000000000000000",

          "endTimeUnixNano":   "1700000000500000000",

          "status": {"code": 1}

        }]

      }]

    }]

  }'



<h2 id="cherez-1-2-sekundy-treys-poyavitsya-v-jaeger-ui">Через 1–2 секунды трейс появится в Jaeger UI</h2>

<h2 id="servis-test-service-operatsiya-test-operation">Сервис: test-service, операция: test-operation</h2>

<h2 id="bazovyy-sdk">Базовый SDK</h2>

pip install opentelemetry-sdk opentelemetry-exporter-otlp



<h2 id="avtomaticheskaya-instrumentatsiya-zero-kod">Автоматическая инструментация (зеро-код)</h2>

pip install opentelemetry-instrumentation

pip install opentelemetry-instrumentation-fastapi

pip install opentelemetry-instrumentation-django

pip install opentelemetry-instrumentation-httpx

pip install opentelemetry-instrumentation-sqlalchemy

pip install opentelemetry-instrumentation-redis

pip install opentelemetry-instrumentation-celery



<h2 id="ili-ustanovit-vse-dostupnye-pakety-avtoinstrumentatsii-razom">Или установить все доступные пакеты автоинструментации разом</h2>

opentelemetry-bootstrap --action=install

<h2 id="peremennye-okruzheniya">Переменные окружения</h2>

export OTEL_SERVICE_NAME="order-service"

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"

export OTEL_TRACES_EXPORTER="otlp"

export OTEL_METRICS_EXPORTER="otlp"

export OTEL_LOGS_EXPORTER="otlp"

export OTEL_PYTHON_LOG_CORRELATION="true"



<h2 id="zapusk-s-avto-instrumentatsiey">Запуск с авто-инструментацией</h2>

opentelemetry-instrument uvicorn app.main:app --host 0.0.0.0 --port 8000

<h2 id="app-main-py-kod-prilozheniya-ne-menyaetsya">app/main.py — код приложения НЕ меняется</h2>

from fastapi import FastAPI

import httpx



app = FastAPI()



@app.get("/orders/{order_id}")

async def get_order(order_id: int):

    # OTel автоматически создаст span для этого эндпоинта

    # и для вызова httpx ниже

    async with httpx.AsyncClient() as client:

        resp = await client.get(

            f"http://payment-service/payments/{order_id}"

        )

    return {"order_id": order_id, "status": resp.json()}

<h2 id="app-services-order-service-py">app/services/order_service.py</h2>

from opentelemetry import trace

from opentelemetry.trace import Status, StatusCode



<h2 id="poluchaem-tracer-odin-raz-na-modul">Получаем tracer один раз на модуль</h2>

tracer = trace.get_tracer(__name__, "1.0.0")



async def process_order(order_id: int, user_id: int) -> dict:

    # Создаём span для бизнес-операции

    with tracer.start_as_current_span(

        "order.process",

        attributes={

            "order.id": order_id,

            "user.id": user_id,

            "order.source": "web",

        }

    ) as span:

        try:

            # Вложенный span для проверки инвентаря

            with tracer.start_as_current_span("inventory.check") as inv_span:

                inventory = await check_inventory(order_id)

                inv_span.set_attribute("inventory.available", inventory.count)



            if inventory.count == 0:

                # Добавляем событие (не ошибку) в span

                span.add_event(

                    "inventory.empty",

                    attributes={"order.id": order_id}

                )

                return {"status": "backorder"}



            result = await create_order_record(order_id, user_id)

            span.set_attribute("order.db_id", result.id)

            return result



        except Exception as e:

            # Помечаем span как ошибку

            span.set_status(Status(StatusCode.ERROR, str(e)))

            span.record_exception(e)

            raise

<h2 id="app-telemetry-py">app/telemetry.py</h2>

from opentelemetry import trace

from opentelemetry.sdk.trace import TracerProvider

from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.sdk.resources import Resource, SERVICE_NAME

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor



def setup_telemetry(service_name: str, otlp_endpoint: str) -> None:

    # Ресурс описывает источник телеметрии

    resource = Resource.create({

        SERVICE_NAME: service_name,

        "service.version": "2.1.0",

        "deployment.environment": "production",

        "service.instance.id": os.environ.get("POD_NAME", "local"),

    })



    # Экспортер → Collector

    exporter = OTLPSpanExporter(

        endpoint=otlp_endpoint,

        insecure=True,      # False в продакшне + TLS сертификаты

    )



    # BatchSpanProcessor буферизует spans и отправляет пакетами

    processor = BatchSpanProcessor(

        exporter,

        max_export_batch_size=512,

        export_timeout_millis=5000,

        schedule_delay_millis=500,

    )



    provider = TracerProvider(resource=resource)

    provider.add_span_processor(processor)

    trace.set_tracer_provider(provider)



    # Инструментируем фреймворки

    FastAPIInstrumentor.instrument()

    SQLAlchemyInstrumentor().instrument()

    HTTPXClientInstrumentor().instrument()



<h2 id="app-main-py">app/main.py</h2>

from app.telemetry import setup_telemetry



setup_telemetry(

    service_name="order-service",

    otlp_endpoint="http://otel-collector:4317"

)

app = FastAPI()

<h2 id="app-logging-config-py">app/logging_config.py</h2>

import logging

from opentelemetry import trace



class OTelLogFormatter(logging.Formatter):

    """Добавляет trace_id и span_id в каждую лог-запись."""



    def format(self, record: logging.LogRecord) -> str:

        ctx = trace.get_current_span().get_span_context()

        if ctx.is_valid:

            record.trace_id = format(ctx.trace_id, '032x')

            record.span_id  = format(ctx.span_id,  '016x')

        else:

            record.trace_id = "0" * 32

            record.span_id  = "0" * 16

        return super().format(record)



<h2 id="konfiguratsiya-loggera-s-otel-formatterom">Конфигурация логгера с OTel-форматтером</h2>

logging.basicConfig(

    format='{"time": "%(asctime)s", "level": "%(levelname)s", '

           '"msg": "%(message)s", "trace_id": "%(trace_id)s", '

           '"span_id": "%(span_id)s"}',

)

for handler in logging.root.handlers:

    handler.setFormatter(OTelLogFormatter(handler.formatter._fmt))

go get go.opentelemetry.io/otel

go get go.opentelemetry.io/otel/sdk

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

go get go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql

// internal/telemetry/telemetry.go

package telemetry



import (

    "context"

    "fmt"

    "os"



    "go.opentelemetry.io/otel"

    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

    "go.opentelemetry.io/otel/sdk/resource"

    sdktrace "go.opentelemetry.io/otel/sdk/trace"

    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"

    "google.golang.org/grpc"

    "google.golang.org/grpc/credentials/insecure"

)



// InitTracer инициализирует OTel TracerProvider и возвращает

// функцию shutdown для корректного завершения.

func InitTracer(ctx context.Context, serviceName, version string) (func(context.Context) error, error) {

    // Подключение к Collector

    conn, err := grpc.DialContext(ctx,

        os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"),

        grpc.WithTransportCredentials(insecure.NewCredentials()),

        grpc.WithBlock(),

    )

    if err != nil {

        return nil, fmt.Errorf("grpc dial: %w", err)

    }



    exporter, err := otlptracegrpc.New(ctx,

        otlptracegrpc.WithGRPCConn(conn),

    )

    if err != nil {

        return nil, fmt.Errorf("otlp exporter: %w", err)

    }



    res, err := resource.New(ctx,

        resource.WithAttributes(

            semconv.ServiceName(serviceName),

            semconv.ServiceVersion(version),

            semconv.DeploymentEnvironment(

                os.Getenv("DEPLOYMENT_ENV")),

        ),

        resource.WithOS(),

        resource.WithProcess(),

    )

    if err != nil {

        return nil, fmt.Errorf("resource: %w", err)

    }



    tp := sdktrace.NewTracerProvider(

        sdktrace.WithBatcher(exporter,

            sdktrace.WithMaxExportBatchSize(512),

            sdktrace.WithBatchTimeout(500*time.Millisecond),

        ),

        sdktrace.WithResource(res),

        // Продакшн-сэмплинг: 10% запросов + все ошибки

        sdktrace.WithSampler(sdktrace.ParentBased(

            sdktrace.TraceIDRatioBased(0.1),

        )),

    )



    otel.SetTracerProvider(tp)

    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(

        propagation.TraceContext{},

        propagation.Baggage{},

    ))



    return tp.Shutdown, nil

}

// cmd/server/main.go

package main



import (

    "context"

    "log"

    "net/http"

    "os/signal"

    "syscall"



    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

    "github.com/yourorg/svc/internal/telemetry"

    "github.com/yourorg/svc/internal/handlers"

)



func main() {

    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)

    defer stop()



    shutdown, err := telemetry.InitTracer(ctx, "payment-service", "1.5.0")

    if err != nil {

        log.Fatalf("init tracer: %v", err)

    }

    defer func() {

        if err := shutdown(context.Background()); err != nil {

            log.Printf("tracer shutdown: %v", err)

        }

    }()



    mux := http.NewServeMux()

    mux.Handle("/payments/", handlers.PaymentHandler())



    // otelhttp.NewHandler оборачивает весь mux —

    // каждый входящий запрос получает span автоматически

    server := &http.Server{

        Addr:    ":8080",

        Handler: otelhttp.NewHandler(mux, "payment-service",

            otelhttp.WithFilter(func(r *http.Request) bool {

                return r.URL.Path != "/healthz"  // исключаем health checks

            }),

        ),

    }



    go func() {

        if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {

            log.Fatalf("server: %v", err)

        }

    }()



    <-ctx.Done()

    server.Shutdown(context.Background())

}

// internal/handlers/payment.go

package handlers



import (

    "context"

    "fmt"



    "go.opentelemetry.io/otel"

    "go.opentelemetry.io/otel/attribute"

    "go.opentelemetry.io/otel/codes"

    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"

)



var tracer = otel.Tracer("payment-handler", trace.WithInstrumentationVersion("1.0"))



func ProcessPayment(ctx context.Context, paymentID string, amount float64) error {

    ctx, span := tracer.Start(ctx, "payment.process",

        trace.WithAttributes(

            attribute.String("payment.id", paymentID),

            attribute.Float64("payment.amount", amount),

            attribute.String("payment.currency", "RUB"),

        ),

        trace.WithSpanKind(trace.SpanKindInternal),

    )

    defer span.End()



    // Вложенный span для валидации

    ctx, valSpan := tracer.Start(ctx, "payment.validate")

    if err := validatePayment(paymentID, amount); err != nil {

        valSpan.SetStatus(codes.Error, err.Error())

        valSpan.RecordError(err)

        valSpan.End()

        return fmt.Errorf("validation: %w", err)

    }

    valSpan.End()



    // Добавляем событие (аудит-след)

    span.AddEvent("payment.authorized", trace.WithAttributes(

        attribute.String("auth.provider", "stripe"),

        attribute.String("auth.code",     "ch_3OxK2L"),

    ))



    if err := chargeCard(ctx, paymentID, amount); err != nil {

        span.SetStatus(codes.Error, "card charge failed")

        span.RecordError(err)

        return err

    }



    span.SetAttributes(

        attribute.String("payment.status", "completed"),

        semconv.HTTPStatusCode(200),

    )

    return nil

}

// gRPC сервер с OTel

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"



grpcServer := grpc.NewServer(

    grpc.StatsHandler(otelgrpc.NewServerHandler(

        otelgrpc.WithFilter(func(info *otelgrpc.InterceptorInfo) bool {

            // Исключить gRPC health check из трейсов

            return info.UnaryServerInfo.FullMethod != "/grpc.health.v1.Health/Check"

        }),

    )),

)



// gRPC клиент с OTel

conn, err := grpc.Dial(target,

    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),

    grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),

)

import (

    "database/sql"

    "go.opentelemetry.io/contrib/instrumentation/database/sql/otelsql"

    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"

)



// Регистрируем драйвер с OTel-обёрткой

sql.Register("postgres-otel",

    otelsql.NewDatabaseDriver(driver,

        otelsql.WithAttributes(

            semconv.DBSystemPostgreSQL,

        ),

        otelsql.WithDBStatement(true),   // логировать SQL в spans

    ),

)



db, err := sql.Open("postgres-otel", dsn)

// Теперь каждый SQL-запрос автоматически создаёт span

// с атрибутами db.statement, db.operation, db.name

<h2 id="skachat-agent-aktualnaya-versiya">Скачать агент (актуальная версия)</h2>

curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar \

  -o opentelemetry-javaagent.jar



<h2 id="zapusk-spring-boot-s-agentom">Запуск Spring Boot с агентом</h2>

java \

  -javaagent:opentelemetry-javaagent.jar \

  -Dotel.service.name=user-service \

  -Dotel.exporter.otlp.endpoint=http://localhost:4317 \

  -Dotel.exporter.otlp.protocol=grpc \

  -Dotel.traces.exporter=otlp \

  -Dotel.metrics.exporter=otlp \

  -Dotel.logs.exporter=otlp \

  -Dotel.instrumentation.spring-web.enabled=true \

  -Dotel.instrumentation.jdbc.enabled=true \

  -Dotel.instrumentation.kafka.enabled=true \

  -jar user-service.jar

<!-- pom.xml -->

<dependencies>

  <dependency>

    <groupId>io.opentelemetry</groupId>

    <artifactId>opentelemetry-api</artifactId>

    <version>1.36.0</version>

  </dependency>

  <dependency>

    <groupId>io.opentelemetry</groupId>

    <artifactId>opentelemetry-sdk</artifactId>

    <version>1.36.0</version>

  </dependency>

  <dependency>

    <groupId>io.opentelemetry</groupId>

    <artifactId>opentelemetry-exporter-otlp</artifactId>

    <version>1.36.0</version>

  </dependency>

  <!-- Spring Boot Actuator интеграция -->

  <dependency>

    <groupId>io.opentelemetry.instrumentation</groupId>

    <artifactId>opentelemetry-spring-boot-starter</artifactId>

    <version>2.3.0-alpha</version>

  </dependency>

</dependencies>

<h2 id="application-yml-spring-boot-s-otel-starter">application.yml (Spring Boot с OTel Starter)</h2>

spring:

  application:

    name: user-service



management:

  tracing:

    sampling:

      probability: 0.1    # 10% сэмплинг в продакшне



otel:

  exporter:

    otlp:

      endpoint: http://otel-collector:4317

      protocol: grpc

  service:

    name: ${spring.application.name}

  resource:

    attributes:

      deployment.environment: production

      service.version: "@project.version@"

  instrumentation:

    spring-web:

      enabled: true

    jdbc:

      enabled: true

      statement-sanitizer:

        enabled: true   # скрывать параметры SQL в трейсах

    kafka:

      enabled: true

// OrderService.java

package com.example.orderservice;



import io.opentelemetry.api.GlobalOpenTelemetry;

import io.opentelemetry.api.trace.Span;

import io.opentelemetry.api.trace.StatusCode;

import io.opentelemetry.api.trace.Tracer;

import io.opentelemetry.context.Scope;



@Service

public class OrderService {



    private static final Tracer tracer =

        GlobalOpenTelemetry.getTracer("order-service", "1.0.0");



    public Order processOrder(String orderId, String userId) {

        Span span = tracer.spanBuilder("order.process")

            .setAttribute("order.id",   orderId)

            .setAttribute("user.id",    userId)

            .setAttribute("order.source", "api")

            .startSpan();



        try (Scope scope = span.makeCurrent()) {

            // Вложенная операция создаст дочерний span автоматически

            // если использует тот же Context

            Order order = inventoryService.reserve(orderId);

            span.setAttribute("order.items_count", order.getItemCount());



            paymentService.charge(userId, order.getTotal());

            span.addEvent("payment.completed",

                Attributes.of(

                    AttributeKey.stringKey("payment.id"), order.getPaymentId()

                ));



            return order;

        } catch (Exception e) {

            span.setStatus(StatusCode.ERROR, e.getMessage());

            span.recordException(e);

            throw e;

        } finally {

            span.end();

        }

    }

}

// С OTel Java Agent или SDK+аннотации

import io.opentelemetry.instrumentation.annotations.WithSpan;

import io.opentelemetry.instrumentation.annotations.SpanAttribute;



@Service

public class InventoryService {



    // @WithSpan автоматически создаёт span с именем метода

    @WithSpan("inventory.check")

    public InventoryStatus checkAvailability(

            @SpanAttribute("product.id") String productId,

            @SpanAttribute("quantity") int quantity) {

        // Span создаётся и закрывается автоматически

        // SpanAttribute добавляет аргументы как атрибуты span

        return repository.findAvailability(productId, quantity);

    }

}

<h2 id="otel-collector-production-yaml">otel-collector-production.yaml</h2>

extensions:

  health_check:

    endpoint: 0.0.0.0:13133

  pprof:

    endpoint: 0.0.0.0:1777

  zpages:

    endpoint: 0.0.0.0:55679



receivers:

  otlp:

    protocols:

      grpc:

        endpoint: 0.0.0.0:4317

        max_recv_msg_size_mib: 16

      http:

        endpoint: 0.0.0.0:4318

  # Приём метрик из Prometheus

  prometheus:

    config:

      scrape_configs:

        - job_name: 'otel-collector'

          static_configs:

            - targets: ['0.0.0.0:8888']



processors:

  # Ограничение памяти: сбрасывает данные при достижении лимита

  memory_limiter:

    check_interval: 5s

    limit_percentage: 80

    spike_limit_percentage: 25



  # Батчинг: снижает количество запросов к бэкенду

  batch:

    timeout: 1s

    send_batch_size: 1024

    send_batch_max_size: 2048



  # Обогащение атрибутами Kubernetes Pod

  k8sattributes:

    auth_type: serviceAccount

    passthrough: false

    extract:

      metadata:

        - k8s.pod.name

        - k8s.namespace.name

        - k8s.node.name

        - k8s.deployment.name

      labels:

        - tag_name: app.version

          key: app.kubernetes.io/version

          from: pod



  # Фильтрация: убираем health-check трейсы

  filter/drop-healthchecks:

    traces:

      span:

        - 'attributes["http.route"] == "/healthz"'

        - 'attributes["http.route"] == "/readyz"'

        - 'attributes["http.route"] == "/metrics"'



  # Трансформация: маскируем чувствительные данные

  transform/redact-pii:

    trace_statements:

      - context: span

        statements:

          # Маскировать номера карт в SQL-запросах

          - replace_pattern(attributes["db.statement"],

              "\\b\\d{16}\\b", "---")

          # Удалить Authorization заголовок

          - delete_key(attributes, "http.request.header.authorization")



  # Сэмплинг на уровне Collector (tail-based)

  tail_sampling:

    decision_wait: 10s

    num_traces: 100000

    expected_new_traces_per_sec: 1000

    policies:

      # Всегда сохранять трейсы с ошибками

      - name: errors-policy

        type: status_code

        status_code: {status_codes: [ERROR]}

      # Всегда сохранять медленные запросы (> 2 сек)

      - name: slow-traces-policy

        type: latency

        latency: {threshold_ms: 2000}

      # 5% всех остальных

      - name: default-policy

        type: probabilistic

        probabilistic: {sampling_percentage: 5}



exporters:

  # Jaeger через OTLP

  otlp/jaeger:

    endpoint: jaeger:4317

    tls:

      insecure: false

      ca_file: /certs/ca.crt



  # Grafana Tempo

  otlp/tempo:

    endpoint: tempo:4317

    tls:

      insecure: true



  # Prometheus для метрик

  prometheusremotewrite:

    endpoint: http://prometheus:9090/api/v1/write



  # Дебаггинг (не в продакшне)

  # debug:

  #   verbosity: detailed



service:

  extensions: [health_check, pprof, zpages]

  pipelines:

    traces:

      receivers:  [otlp]

      processors: [memory_limiter, k8sattributes,

                   filter/drop-healthchecks,

                   transform/redact-pii,

                   tail_sampling, batch]

      exporters:  [otlp/jaeger, otlp/tempo]

    metrics:

      receivers:  [otlp, prometheus]

      processors: [memory_limiter, batch]

      exporters:  [prometheusremotewrite]

    logs:

      receivers:  [otlp]

      processors: [memory_limiter, k8sattributes, batch]

      exporters:  [otlp/tempo]

<h2 id="k8s-otel-collector-daemonset-yaml">k8s/otel-collector-daemonset.yaml</h2>

apiVersion: apps/v1

kind: DaemonSet

metadata:

  name: otel-collector-agent

  namespace: monitoring

spec:

  selector:

    matchLabels:

      app: otel-collector-agent

  template:

    metadata:

      labels:

        app: otel-collector-agent

    spec:

      serviceAccountName: otel-collector

      containers:

        - name: collector

          image: otel/opentelemetry-collector-contrib:0.96.0

          args: ["--config=/conf/otel-agent-config.yaml"]

          env:

            - name: MY_POD_IP

              valueFrom:

                fieldRef:

                  fieldPath: status.podIP

            - name: K8S_NODE_NAME

              valueFrom:

                fieldRef:

                  fieldPath: spec.nodeName

          ports:

            - containerPort: 4317  # OTLP gRPC

            - containerPort: 4318  # OTLP HTTP

          resources:

            requests:

              cpu: 100m

              memory: 200Mi

            limits:

              cpu: 500m

              memory: 500Mi

          volumeMounts:

            - name: config

              mountPath: /conf

      volumes:

        - name: config

          configMap:

            name: otel-agent-config

<h2 id="dropped-spans-dannye-teryayutsya-kritichno">Dropped spans (данные теряются — критично!)</h2>

otelcol_processor_dropped_spans_total



<h2 id="queue-size-protsessora-batch">Queue size процессора batch</h2>

otelcol_exporter_queue_size



<h2 id="oshibki-eksportyora">Ошибки экспортёра</h2>

otelcol_exporter_send_failed_spans_total



<h2 id="ispolzovanie-pamyati">Использование памяти</h2>

otelcol_process_memory_rss



<h2 id="latency-eksporta">Latency экспорта</h2>

otelcol_exporter_queue_capacity

<h2 id="docker-compose-jaeger-s-cassandra">docker-compose: Jaeger с Cassandra</h2>

services:

  cassandra:

    image: cassandra:4.1

    environment:

      MAX_HEAP_SIZE: "2G"

      HEAP_NEWSIZE: "512M"

    volumes:

      - cassandra-data:/var/lib/cassandra



  jaeger-collector:

    image: jaegertracing/jaeger-collector:1.55

    environment:

      SPAN_STORAGE_TYPE: cassandra

      CASSANDRA_SERVERS: cassandra

      CASSANDRA_KEYSPACE: jaeger_v1_dc1

      COLLECTOR_OTLP_ENABLED: "true"

    ports:

      - "4317:4317"   # OTLP gRPC

      - "4318:4318"   # OTLP HTTP



  jaeger-query:

    image: jaegertracing/jaeger-query:1.55

    environment:

      SPAN_STORAGE_TYPE: cassandra

      CASSANDRA_SERVERS: cassandra

    ports:

      - "16686:16686"  # Jaeger UI

<h2 id="poisk-medlennyh-treysov-p99-1-sek">Поиск медленных трейсов (p99 &gt; 1 сек)</h2>

Service: order-service

Operation: POST /orders

Min Duration: 1s



<h2 id="poisk-oshibok-za-posledniy-chas">Поиск ошибок за последний час</h2>

Service: payment-service

Tags: error=true

Lookback: 1h



<h2 id="sravnenie-dvuh-treysov-compare-view">Сравнение двух трейсов (Compare view)</h2>

Compare: <traceId1> vs <traceId2>

<h2 id="tempo-yaml">tempo.yaml</h2>

server:

  http_listen_port: 3200



distributor:

  receivers:

    otlp:

      protocols:

        grpc:

          endpoint: 0.0.0.0:4317

        http:

          endpoint: 0.0.0.0:4318



ingester:

  trace_idle_period: 10s

  max_block_bytes: 1_000_000

  max_block_duration: 5m



compactor:

  compaction:

    block_retention: 336h    # 14 дней



storage:

  trace:

    backend: s3              # Или local для разработки

    s3:

      bucket: my-tempo-traces

      endpoint: s3.amazonaws.com

      region: eu-central-1



querier:

  max_concurrent_queries: 20

  search:

    prefer_self: 10



query_frontend:

  search:

    max_duration: 0          # Неограниченный диапазон поиска

<h2 id="vse-treysy-s-oshibkami-v-payment-service">Все трейсы с ошибками в payment-service</h2>

{ resource.service.name = "payment-service" && status = error }



<h2 id="medlennye-zaprosy-k-postgresql-dolshe-500ms">Медленные запросы к PostgreSQL дольше 500ms</h2>

{ span.db.system = "postgresql" } | duration > 500ms



<h2 id="treysy-konkretnogo-polzovatelya">Трейсы конкретного пользователя</h2>

{ span.user.id = "user-12345" }



<h2 id="treysy-s-konkretnym-http-statusom">Трейсы с конкретным HTTP-статусом</h2>

{ span.http.status_code = 500 }



<h2 id="agregatsiya-srednyaya-dlitelnost-po-servisam">Агрегация: средняя длительность по сервисам</h2>

{ } | rate() by(resource.service.name)



<h2 id="nayti-treysy-gde-payment-service-zanyal-1-sek">Найти трейсы, где payment-service занял &gt; 1 сек</h2>

{ resource.service.name = "payment-service" && duration > 1s }

  | select(resource.service.name, duration, rootSpan.name)

<h2 id="python-dobavlenie-exemplar-k-metrike">Python: добавление exemplar к метрике</h2>

from opentelemetry.metrics import get_meter

from opentelemetry import trace



meter  = get_meter("payment-service")

tracer = trace.get_tracer("payment-service")



payment_duration = meter.create_histogram(

    "payment.duration",

    unit="ms",

    description="Время обработки платежа"

)



def process_payment(payment_id: str):

    with tracer.start_as_current_span("payment.process") as span:

        start = time.time()

        try:

            result = _do_process(payment_id)

        finally:

            duration_ms = (time.time() - start) * 1000

            # Exemplar добавляется автоматически — SDK берёт TraceID

            # из текущего контекста при записи метрики

            payment_duration.record(

                duration_ms,

                attributes={"payment.status": "success"}

            )

<h2 id="python-strukturirovannye-logi-s-traceid">Python: структурированные логи с TraceID</h2>

import structlog

from opentelemetry import trace



def add_trace_context(logger, method, event_dict):

    span_context = trace.get_current_span().get_span_context()

    if span_context.is_valid:

        event_dict["trace_id"] = format(span_context.trace_id, "032x")

        event_dict["span_id"]  = format(span_context.span_id,  "016x")

    return event_dict



structlog.configure(

    processors=[

        add_trace_context,

        structlog.processors.JSONRenderer(),

    ]

)

// Go: logrus с OTel-контекстом

import (

    "github.com/sirupsen/logrus"

    "go.opentelemetry.io/otel/trace"

)



func logWithTrace(ctx context.Context, msg string, fields logrus.Fields) {

    span := trace.SpanFromContext(ctx)

    if span.SpanContext().IsValid() {

        fields["trace_id"] = span.SpanContext().TraceID().String()

        fields["span_id"]  = span.SpanContext().SpanID().String()

    }

    logrus.WithFields(fields).Info(msg)

}

. Grafana Dashboard: p99 latency order-service выросло с 50ms до 3s

   → Клик на аномальную точку на графике



2. Exemplar → TraceID: a3f2e1b9c7d5e3f1a2b4c6d8e0f2a4b6

   → Открываем трейс в Grafana Tempo / Jaeger



3. Waterfall-диаграмма трейса показывает:

   order-service (3.1s total)

   └── inventory-check (12ms) ✅

   └── payment-service (2950ms) ← ЗДЕСЬ

       └── postgres-query (2940ms) ← ЗДЕСЬ

           db.statement: "SELECT * FROM transactions WHERE ..."



4. TraceID → Логи в Loki/Elasticsearch:

   trace_id=a3f2e1b9c7d5e3f1a2b4c6d8e0f2a4b6

   → "Sequential scan on 47M rows, missing index on user_id"



5. Причина: деплой 10 минут назад добавил новый запрос без индекса.

   Время от алерта до root cause: 4 минуты.

{

  "title": "Service Health Dashboard",

  "panels": [

    {

      "title": "Request Rate + Errors",

      "type": "timeseries",

      "targets": [

        {

          "expr": "rate(http_server_duration_count{service_name=\"$service\"}[1m])",

          "legendFormat": "req/s"

        },

        {

          "expr": "rate(http_server_duration_count{service_name=\"$service\", http_status_code=~\"5..\"}[1m])",

          "legendFormat": "errors/s"

        }

      ]

    },

    {

      "title": "p50/p95/p99 Latency (с Exemplars)",

      "type": "timeseries",

      "options": {"exemplars": {"enabled": true}},

      "targets": [

        {

          "expr": "histogram_quantile(0.99, rate(http_server_duration_bucket{service_name=\"$service\"}[5m]))",

          "legendFormat": "p99"

        }

      ]

    },

    {

      "title": "Trace Explorer",

      "type": "traces",

      "datasource": "Tempo",

      "targets": [

        {

          "query": "{ resource.service.name = \"$service\" && status = error }"

        }

      ]

    }

  ]

}

<h2 id="python-head-based-sampling-10">Python: head-based sampling 10%</h2>

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBased



sampler = ParentBased(

    root=TraceIdRatioBased(0.10),     # 10% новых трейсов

    remote_parent_sampled=ALWAYS_ON,  # Сохранять если родитель сэмплирован

    remote_parent_not_sampled=ALWAYS_OFF,

)

provider = TracerProvider(sampler=sampler, resource=resource)

<h2 id="v-konfiguratsii-collector-sm-razdel-7-1">В конфигурации Collector (см. раздел 7.1)</h2>

processors:

  tail_sampling:

    policies:

      - name: always-errors

        type: status_code

        status_code: {status_codes: [ERROR]}

      - name: always-slow

        type: latency

        latency: {threshold_ms: 1000}

      - name: sample-rest

        type: probabilistic

        probabilistic: {sampling_percentage: 5}

<h2 id="python-ustanovka-baggage-na-vhode">Python: установка baggage на входе</h2>

from opentelemetry.baggage.propagation import W3CBaggagePropagator

from opentelemetry import baggage, context



@app.middleware("http")

async def add_baggage(request: Request, call_next):

    # Добавляем user_id во весь трейс

    ctx = baggage.set_baggage("user.id", request.headers.get("X-User-Id", "anonymous"))

    ctx = baggage.set_baggage("tenant.id", request.headers.get("X-Tenant-Id", "default"))

    token = context.attach(ctx)

    try:

        response = await call_next(request)

    finally:

        context.detach(token)

    return response



<h2 id="go-chtenie-baggage-v-downstream-servise">Go: чтение baggage в downstream сервисе</h2>

func Handler(w http.ResponseWriter, r *http.Request) {

    ctx := r.Context()

    userId := baggage.FromContext(ctx).Member("user.id").Value()

    tenantId := baggage.FromContext(ctx).Member("tenant.id").Value()



    span := trace.SpanFromContext(ctx)

    span.SetAttributes(

        attribute.String("user.id", userId),

        attribute.String("tenant.id", tenantId),

    )

}

// Go: события в span

span.AddEvent("cache.miss", trace.WithAttributes(

    attribute.String("cache.key", cacheKey),

    attribute.String("cache.type", "redis"),

))



span.AddEvent("retry.attempt", trace.WithAttributes(

    attribute.Int("retry.count", attemptNumber),

    attribute.String("retry.reason", "connection_timeout"),

))



span.AddEvent("payment.3ds.required", trace.WithAttributes(

    attribute.String("3ds.version", "2.0"),

    attribute.Bool("3ds.challenge", true),

))

<h2 id="pravilno-ispolzovat-semanticheskie-konventsii">Правильно: использовать семантические конвенции</h2>

from opentelemetry.semconv.trace import SpanAttributes



span.set_attributes({

    SpanAttributes.HTTP_METHOD:        "POST",

    SpanAttributes.HTTP_URL:           "https://api.example.com/orders",

    SpanAttributes.HTTP_STATUS_CODE:   201,

    SpanAttributes.DB_SYSTEM:          "postgresql",

    SpanAttributes.DB_STATEMENT:       "INSERT INTO orders ...",

    SpanAttributes.MESSAGING_SYSTEM:   "kafka",

    SpanAttributes.MESSAGING_DESTINATION: "orders.created",

})



<h2 id="kastomnye-atributy-ispolzuyte-namespace-organizatsii">Кастомные атрибуты: используйте namespace организации</h2>

span.set_attributes({

    "mycompany.order.id":       order_id,

    "mycompany.order.amount":   amount,

    "mycompany.order.currency": "RUB",

    "mycompany.user.tier":      "premium",

})

<h2 id="ustanovka-operatora">Установка оператора</h2>

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml



<h2 id="proverka">Проверка</h2>

kubectl get pods -n opentelemetry-operator-system

<h2 id="crd-opentelemetrycollector">CRD: OpenTelemetryCollector</h2>

apiVersion: opentelemetry.io/v1alpha1

kind: OpenTelemetryCollector

metadata:

  name: otel-collector

  namespace: monitoring

spec:

  mode: DaemonSet   # Или Deployment, Sidecar, StatefulSet

  image: otel/opentelemetry-collector-contrib:0.96.0

  config: |

    receivers:

      otlp:

        protocols:

          grpc:

            endpoint: 0.0.0.0:4317

    processors:

      batch: {}

      k8sattributes: {}

    exporters:

      otlp:

        endpoint: tempo.monitoring.svc.cluster.local:4317

        tls:

          insecure: true

    service:

      pipelines:

        traces:

          receivers: [otlp]

          processors: [k8sattributes, batch]

          exporters: [otlp]

<h2 id="crd-instrumentation-politika-avto-instrumentatsii">CRD: Instrumentation (политика авто-инструментации)</h2>

apiVersion: opentelemetry.io/v1alpha1

kind: Instrumentation

metadata:

  name: otel-instrumentation

  namespace: production

spec:

  exporter:

    endpoint: http://otel-collector.monitoring:4317

  propagators:

    - tracecontext

    - baggage

  sampler:

    type: parentbased_traceidratio

    argument: "0.1"

  python:

    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest

    env:

      - name: OTEL_PYTHON_LOG_CORRELATION

        value: "true"

  java:

    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest

    env:

      - name: OTEL_INSTRUMENTATION_JDBC_STATEMENT_SANITIZER_ENABLED

        value: "true"

  go:

    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:latest

<h2 id="deployment-dobavlyaem-annotatsiyu-dlya-avto-instrumentatsii">Deployment: добавляем аннотацию для авто-инструментации</h2>

apiVersion: apps/v1

kind: Deployment

metadata:

  name: order-service

  namespace: production

spec:

  template:

    metadata:

      annotations:

        # Выбираем язык инструментации

        instrumentation.opentelemetry.io/inject-python: "true"

        # Или для Java:

        # instrumentation.opentelemetry.io/inject-java: "true"

        # Service name берётся из app.kubernetes.io/name

    spec:

      containers:

        - name: order-service

          image: myregistry/order-service:1.5.0

          # Никаких изменений в контейнере!

<h2 id="rbac-dlya-k8sattributes-processor">RBAC для k8sattributes processor</h2>

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: otel-collector-k8s

rules:

  - apiGroups: [""]

    resources: ["pods", "namespaces", "nodes"]

    verbs: ["get", "watch", "list"]

  - apiGroups: ["apps"]

    resources: ["replicasets"]

    verbs: ["get", "watch", "list"]

.pod.name:          order-service-7d8f9c-xk2pl

k8s.namespace.name:    production

k8s.node.name:         node-3.cluster.local

k8s.deployment.name:   order-service

k8s.container.name:    order-service

app.version:           1.5.0        ← из k8s label

<h2 id="python-optimalnye-nastroyki-batchspanprocessor-dlya-prodakshna">Python: оптимальные настройки BatchSpanProcessor для продакшна</h2>

from opentelemetry.sdk.trace.export import BatchSpanProcessor



processor = BatchSpanProcessor(

    exporter,

    # Максимальный размер очереди spans

    max_queue_size=2048,

    # Интервал отправки (мс) — баланс между latency и batch size

    schedule_delay_millis=500,

    # Максимальный размер одного батча

    max_export_batch_size=512,

    # Таймаут экспорта — не блокировать приложение долго

    export_timeout_millis=3000,

)

// Go: оптимальный BatchSpanProcessor

sdktrace.WithBatcher(exporter,

    sdktrace.WithMaxQueueSize(4096),

    sdktrace.WithBatchTimeout(time.Second),

    sdktrace.WithMaxExportBatchSize(512),

    sdktrace.WithExportTimeout(5*time.Second),

    // Блокировать при переполнении очереди (не терять spans)

    sdktrace.WithBlocking(),

)

<h2 id="ne-nuzhno-sozdavat-spans-dlya">НЕ нужно создавать spans для:</h2>

<h2 id="prostyh-utilitarnyh-funktsiy-getter-setter">- Простых утилитарных функций (getter, setter)</h2>

<h2 id="matematicheskih-vychisleniy-bez-i-o">- Математических вычислений без I/O</h2>

<h2 id="funktsiy-vyzyvaemyh-tysyachi-raz-v-sekundu-bez-vneshnih-zavisimostey">- Функций, вызываемых тысячи раз в секунду без внешних зависимостей</h2>

<h2 id="health-check-endpointov">- Health-check эндпоинтов</h2>




<h2 id="filtruem-health-checks-v-sdk">Фильтруем health-checks в SDK</h2>

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor



FastAPIInstrumentor.instrument_app(

    app,

    excluded_urls="healthz,readyz,metrics,ping"

)

<h2 id="izmerenie-overhead-otel-v-testovoy-srede">Измерение overhead OTel в тестовой среде</h2>

import time

from opentelemetry import trace

from opentelemetry.sdk.trace import TracerProvider

from opentelemetry.sdk.trace.export import SimpleSpanProcessor

from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter



<h2 id="test-bez-otel">Тест без OTel</h2>

def without_otel(n: int):

    start = time.perf_counter()

    for _ in range(n):

        result = sum(range(1000))

    return time.perf_counter() - start



<h2 id="test-s-otel">Тест с OTel</h2>

tracer = trace.get_tracer("bench")

def with_otel(n: int):

    start = time.perf_counter()

    for _ in range(n):

        with tracer.start_as_current_span("bench-span"):

            result = sum(range(1000))

    return time.perf_counter() - start



N = 10_000

t1 = without_otel(N)

t2 = with_otel(N)

print(f"Без OTel: {t1:.3f}s | С OTel: {t2:.3f}s | Overhead: {(t2-t1)/t1*100:.1f}%")

<h2 id="tipichnyy-rezultat-overhead-2-5-pri-100-sampling">Типичный результат: Overhead: 2–5% при 100% sampling</h2>

/api/orders (1240ms)

└── SQLAlchemy: SELECT * FROM orders LIMIT 50 (12ms)

└── SQLAlchemy: SELECT * FROM users WHERE id=1 (8ms)   ← 1

└── SQLAlchemy: SELECT * FROM users WHERE id=7 (9ms)   ← 2

└── SQLAlchemy: SELECT * FROM users WHERE id=12 (11ms) ← 3

... (50 одинаковых запросов к users)

<h2 id="bylo-n-1">Было (N+1):</h2>

orders = db.query(Order).limit(50).all()

return [{"id": o.id, "user": o.user.name} for o in orders]



<h2 id="stalo-1-zapros-s-join">Стало (1 запрос с JOIN):</h2>

orders = db.query(Order).options(

    joinedload(Order.user)

).limit(50).all()

/api/checkout (30001ms → TIMEOUT)

└── order-service: CreateOrder (28ms) ✅

└── inventory-service: Reserve (27ms) ✅

└── payment-service: Charge (29950ms → ERROR)

    └── fraud-detection: Analyze (29900ms → ERROR)

        └── ml-scoring-service: Score (29850ms → TIMEOUT)

            HTTP: POST http://ml-model-v2:8080/score (TIMEOUT)

// Добавлен circuit breaker через gobreaker

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{

    Name:        "ml-scoring",

    MaxRequests: 5,

    Interval:    10 * time.Second,

    Timeout:     5 * time.Second,

    ReadyToTrip: func(counts gobreaker.Counts) bool {

        return counts.ConsecutiveFailures > 3

    },

})



// Добавлен контекст с таймаутом

ctx, cancel := context.WithTimeout(ctx, 500*time.Millisecond)

defer cancel()



score, err := cb.Execute(func() (interface{}, error) {

    return mlClient.Score(ctx, features)

})

.event: "message.received"

span.attributes.kafka.message.payload.size: 47 MB

span.attributes.kafka.message.key: "order-batch-export-2026-01-15"

// Убрали запись payload в span-атрибуты, оставили только метаданные

span.setAttribute("kafka.message.key", record.key());

span.setAttribute("kafka.message.size_bytes", record.serializedValueSize());

// НЕ: span.setAttribute("kafka.message.payload", new String(record.value()));