All work
A real-time analytics pipeline at 8 billion events a day
Ingest, enrich and serve 8B events per day with sub-two-second freshness and no data loss.
8B/day
events ingested
<2s
end-to-end lag
0
events dropped
## The challenge
Reporting was a day behind and getting further behind every week. Batch jobs couldn’t keep up, and customers wanted live numbers.
## What we did
- Built a Rust ingestion tier that does schema validation and enrichment at the edge.
- Streamed through Kafka into Flink for windowed aggregation, with exactly-once semantics.
- Landed everything in ClickHouse with materialised views powering sub-second dashboards.
## Outcome
End-to-end lag dropped from ~18 hours to under two seconds, and the pipeline has ingested billions of events without losing one.