The OpenTelemetry Collector makes a good proof of concept easy: receive telemetry, add a processor, export it somewhere and confirm that a trace appears. Production is where the unanswered questions arrive. What happens when the backend slows down? Which component owns retries? Can the pipeline scale without breaking tail sampling? How will anyone know that the Collector itself is dropping data?
I treat the Collector as production infrastructure, not as a transparent pipe. The configuration matters, but topology, failure behavior and operational visibility matter more.
I choose topology from failure boundaries
I normally separate the agent responsibility from the gateway responsibility. An agent close to the workload handles local collection and resource context. A gateway tier handles centralized processing, sampling and exports. The exact shape depends on Kubernetes, virtual machines, networks and compliance boundaries; the point is to make ownership and failure impact explicit.
Before adding processors, I draw the path for metrics, logs and traces. For each hop I document transport, authentication, expected volume, buffering and the behavior when the next hop is unavailable. This catches more production problems than polishing one large YAML file.
Queues and retries need a budget
OpenTelemetry provides sending queues and retry helpers for transient failures. Enabling them is only the start. A queue has finite capacity, consumes memory and eventually overflows. I size it against an explicit interruption window and test what happens after that window is exceeded.
For data that cannot be regenerated, I consider a persistent write-ahead log and validate the storage behavior during restarts. I also decide which signal can be degraded first. Treating every byte as equally critical sounds safe, but it often creates an uncontrolled failure mode instead of a deliberate policy.
Backpressure is a design decision. If nobody made it, the runtime will make it during the incident.
Scaling is not always stateless
Many Collector pipelines can scale horizontally behind a load balancer. Persistent gRPC connections still need appropriate load balancing, because a simple layer-four balancer can leave traffic concentrated on one backend. Stateful processing needs even more care.
Tail sampling, span-to-metrics and other stateful components need related telemetry to reach the same Collector instance. I use consistent routing where required and verify the distribution with load tests. Adding replicas without preserving state affinity can make a pipeline look healthy while silently changing the result.
I monitor the telemetry pipeline itself
A production Collector must expose enough internal telemetry to answer whether it is receiving, refusing, queueing, retrying or dropping data. I watch queue size against capacity, enqueue failures, exporter failures, processor refusals, memory pressure and restart behavior. I put those signals in a small operational view owned by the platform team.
I also attach a version to configuration and deployment events. When telemetry volume or cardinality changes, responders should be able to see whether a Collector release or config change happened at the same time.
- Queue size, capacity and enqueue failures
- Accepted and refused spans, metrics and log records
- Exporter errors, retries and destination latency
- Memory limiter behavior, CPU, memory and restarts
- Configuration version and rollout status
My production gate
Before broad rollout, I run failure tests: slow or stop a destination, restart a Collector, saturate a queue and send an intentionally malformed payload. Then I confirm that the expected alert fires, the stated loss policy is true and recovery does not create a second overload.
Only after that do I expand service coverage. This turns OpenTelemetry from a promising integration into an operated telemetry layer, with limits the team understands and behavior it can trust.
