Sluice
High-throughput telemetry ingestion pipeline in Go — gRPC gateway, Kafka, Redis dedup, TimescaleDB, load-tested to its real ceilings.
Overview
A production-shaped telemetry ingestion path: a gRPC ingest gateway with a bounded worker pool, partitioned idempotent Kafka, a consumer group with Redis dedup and a batched TimescaleDB sink, a dead-letter topic for poison records, Prometheus/Grafana observability, and a Kubernetes deployment path via Helm, Terraform, and Ansible.
Problem
An ingestion path has one job that's easy to say and hard to keep true under load: don't lose data, and know exactly how much load you can take before you do. Most demo pipelines skip the backpressure story or never measure their own ceiling on real hardware.
Approach
Three architectural bets: bounded pool + blocking backpressure, batched writes to every downstream sink, and idempotent-producer + commit-after-write + dedup for effectively-once delivery. Load-tested against real infrastructure to find genuine throughput ceilings — every number in the README comes from running the actual pipeline. Born from operating Kafka streaming infrastructure at HPE: this is the ingestion path I had operated but never built.
Impact
Demonstrates real distributed-systems engineering: durability guarantees, backpressure, dead-lettering, and honest capacity measurement — the operations instincts of production streaming infra, built from scratch.