The Datadog Bill Shock: Why Startups Are Replacing Proprietary APM with OpenTelemetry and ClickHouse in 2026

The $70,000 Observability Surprise

In late 2025 and heading into 2026, a familiar horror story keeps dominating technical leadership discussions: a fast-growing SaaS startup scales its microservices footprint, ships dynamic staging environments, and suddenly receives a Datadog invoice that outstrips its entire AWS EC2 and RDS bill combined.

What begins as an innocent developer experience win—dropping in a proprietary APM agent that auto-instruments runtime traces, infrastructure metrics, and distributed spans—quietly turns into a financial disaster. The economic model of commercial observability platforms relies heavily on pricing dimensions that penalize modern software design:

1. Host-Based and Core-Based Licensing: In an era of autoscaling Kubernetes clusters, serverless workers, and ephemeral test preview environments, static per-host pricing forces engineering teams to constantly prune infrastructure simply to avoid APM penalty fees.
2. Hyper-Cardinality Ingestion Taxes: Attaching granular dimensions like user_id, organization_id, or git_commit to custom metrics triggers exponential cost multipliers that turn high-resolution application debugging into a luxury.
3. Log Ingestion and Retention Multipliers: Ingesting terabytes of structured JSON application logs only to pay separately for indexing, 15-day retention, and live tail filtering creates unpredictable month-over-month invoice volatility.

By 2026, engineering leaders have recognized that proprietary agents do not just capture system telemetry; they capture engineering leverage. Moving away from proprietary APM is no longer about saving a few thousand dollars on tooling—it is an architectural imperative to maintain infrastructure sovereignty.

The Decoupling: OpenTelemetry Collector as the Control Plane

The foundational catalyst enabling this migration is the total maturation of the OpenTelemetry (OTel) standard. In previous years, teams tolerated proprietary APMs because manual telemetry collection and trace context propagation were notoriously difficult. Today, the OpenTelemetry Collector acts as an enterprise-grade ingestion and transformation engine that lives directly within your own VPC.

Instead of transmitting raw spans, profiles, and logs directly to a third-party vendor endpoint, your services emit standard OTLP (OpenTelemetry Protocol) telemetry to a local collector daemonset:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: https://www.google.com/url?q=http://0.0.0.0:4317&source=gmail&ust=1792802485676000&sa=E
      http:
        endpoint: https://www.google.com/url?q=http://0.0.0.0:4318&source=gmail&ust=1792802485676000&sa=E

processors:
  batch:
    timeout: 1s
    send_batch_size: 8192
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 10s
    expected_new_traces_per_sec: 2000
    policies:
      - name: drop_successful_healthchecks
        type: string_attribute
        string_attribute:
          key: http.route
          values: ["/healthz", "/ready", "/metrics"]
          enabled_regex_matching: false
          invert_match: true
      - name: sample_http_errors
        type: status_code
        status_code: { status_codes: [ ERROR ] }
      - name: probabilistic_traffic
        type: probabilistic
        probabilistic: { sampling_percentage: 5.0 }

exporters:
  clickhouse:
    endpoint: "tcp://clickhouse-cluster.internal:9000?database=observability"
    ttl: 30d
    compress: lz4

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [clickhouse]

By placing the OpenTelemetry Collector between application workloads and the storage layer, engineering teams gain decisive superpowers:

• Tail Sampling Over Head Sampling: Instead of blindly sampling 1% of incoming requests at the edge before knowing whether an error occurs, tail sampling buffers traces in collector memory and guarantees that 100% of failed HTTP requests, slow database queries, and unhandled exceptions are captured, while dropping monotonous /healthz polls.
• Dynamic PII Scrubbing: Telemetry sanitization rules run inside your private infrastructure before data ever traverses public internet pipelines, ensuring strict regulatory compliance.
• Vendor Agnosticism: Switching storage backends or dual-routing telemetry during migration requires changing a single exporter configuration file rather than redeploying thousands of application pods.

Why ClickHouse Became the Observability Storage Standard

While OpenTelemetry solved the ingestion and instrumentation layer, storing petabytes of distributed traces, real-time metrics, and semi-structured application logs required an engine capable of handling intense columnar write throughput without collapsing under financial weight.

That engine is ClickHouse.

Traditional observability databases built on top of inverted search indexes (such as Elasticsearch or OpenSearch) suffer from massive RAM consumption and steep index storage amplification. A 100 GB stream of raw application logs frequently requires 150 GB to 200 GB of disk space once index tokens and field maps are calculated.

In contrast, ClickHouse stores log attributes, trace tags, and timeseries data in discrete column files compressed with specialized algorithms like ZSTD and DoubleDelta. In production benchmarks across modern engineering teams:

• 10x to 15x Compression Ratios: High-volume string payloads and JSON log records compress down to 10% or less of their raw size on disk, slashing NVMe storage costs.
• Vectorized SIMD Query Execution: ClickHouse leverages modern CPU vector instructions to scan hundreds of millions of trace rows per second per node. Aggregating p99 latency by API endpoint over billions of records executes in milliseconds rather than minutes.
• Native Support for Hyper-Cardinality: Unlike traditional timeseries databases that choke when tag cardinality exceeds tens of thousands of unique values, ClickHouse treats high-cardinality UUIDs, user IDs, and Git SHAs as standard columnar data types.

Open-source and managed observability platforms like SigNoz, HyperDX, and Grafana Tempo with ClickHouse backends leverage this exact architecture, providing the intuitive trace waterfalls, APM dashboards, and log streaming interfaces developers expect from commercial SaaS at a tiny fraction of the cost.

The Pragmatic Migration Strategy

Migrating an engineering organization off a legacy APM vendor does not require a chaotic big-bang cutover. High-velocity engineering teams execute the transition through three calculated phases:

Phase 1: Instrument with OpenTelemetry SDKs

Replace proprietary tracing libraries with language-standard OpenTelemetry SDKs (available across TypeScript, Go, Python, Java, and Rust). Configure the OTel exporter to route traffic to the local OpenTelemetry Collector. If immediate cutover is impossible, the OpenTelemetry Collector can dual-write spans to both your legacy vendor and your new internal pipeline.

Phase 2: Deploy Tail Sampling and ClickHouse Storage

Provision a dedicated ClickHouse cluster (either managed via ClickHouse Cloud or self-hosted via Kubernetes operators). Implement aggressive tail sampling rules in your collector to capture all 4xx/5xx errors, slow traces exceeding 500ms, and a 2% baseline of healthy transactions. You immediately eliminate 80% of unnecessary trace noise while preserving 100% of diagnostic signal.

Phase 3: Mirror Alerting and Deprecate Agent Footprints

Recreate critical SLI/SLO dashboards and PagerDuty alert rules inside your open-source UI of choice. Once your on-call engineers validate that query latency and incident response workflows match or exceed the legacy experience, systematically decommission the proprietary daemonsets across your cluster.

The New Observability Economics

In 2026, observability is no longer an all-or-nothing trade-off between engineering productivity and run-away operational burn. By adopting OpenTelemetry for vendor-neutral instrumentation and ClickHouse for hyper-efficient columnar analytics, engineering teams reclaim both financial predictability and architectural independence. Stop paying your observability vendor an unearned tax on your company's growth.