Event-driven Enterprise Data Platform

This blueprint is derived from production experience building a greenfield enterprise data platform at a large retail organisation. Operational data was locked inside SAP and Oracle, served to analytics teams via brittle overnight ETL jobs, 24 to 48 hours stale by the time anyone could query it. The goal was a streaming-first, domain-owned platform delivering sub-minute-fresh data to BI, operational tooling, and AI/ML pipelines, with governance and GDPR compliance built into the ingestion path rather than retrofitted later.

The patterns here are not novel. LinkedIn built Kafka specifically to solve this class of problem at scale and published the architecture in 2011. Netflix, Uber, and Spotify each converged on the same structural decisions independently: immutable event log as the source of truth, domain teams owning their data products, and a thin platform team providing the infrastructure and standards rather than owning the data itself. Zhamak Dehghani formalised the domain-ownership model as Data Mesh in 2019, and since then Zalando, HelloFresh, JPMorgan, and Intuit have published their own implementations following the same principles.

Every decision recorded here reflects a real production trade-off. Where one approach was chosen over another, the reasoning is documented. Where something fails in practice, it is called out. None of this is vendor documentation.

The actual problem

The specific failures that drove this architecture. Not generic "data silos", but the real operational pain that forced a greenfield rebuild:

• 01 24–48 hour analytics lag. All data access to SAP and Oracle was via scheduled DB dump exports into a flat file store. No streaming, no CDC. Inventory data was always stale by the time the BI team got it.
• 02 Two departments reporting different revenue figures. Finance and operations had each built their own ETL logic from the same Oracle source, with different join conditions and fiscal calendar interpretations. Neither was wrong; both were inconsistent.
• 03 Central data team as a bottleneck. Eight engineers owned all ingestion and transformation logic for 40+ product teams. Every new data source required a ticket, a sprint, and a four-week queue.
• 04 Silent pipeline failures. ETL jobs failed without alerting. Business users discovered stale data by noticing that yesterday's sales figures were identical to last week's. MTTR was measured in days.
• 05 GDPR as a fire drill. Subject access requests took 3 days because no one had a reliable map of which database columns contained personal data. Sensitive fields were scattered across 200+ tables with no classification metadata.

Why event-driven

The shift to event-driven ingestion was not a technology preference. It was driven by a specific latency requirement: inventory and order status needed to be available in BI within five minutes of a change in the operational system. Scheduled batch jobs cannot reliably achieve that without over-provisioning, polling at intervals that hammer source databases, and building error-prone watermark logic.

The secondary benefit was forcing explicit domain contracts. When you tell a team "you need to publish an Avro schema and a named event for every business change you want to share," you surface implicit domain knowledge that was previously embedded in SQL join conditions. That process is uncomfortable and slow initially. It pays off when the data mesh scales beyond five domains.

For legacy systems (SAP, Oracle) where publishing application events is impractical, Change Data Capture (CDC) via database log tailing fills the gap. CDC gives you near-real-time row-level changes without modifying application code. The trade-off is that CDC events reflect internal database structure, not business semantics. A transformation step is still needed to produce meaningful domain events from raw CDC rows.

Why data mesh, and where it is genuinely difficult

A centralised data team cannot sustainably own data quality for 40 independent product domains. The data mesh model was chosen not as an architectural ideal but as the only operationally viable option at this scale: push data product ownership to the teams with the deepest domain knowledge, and give the platform team a different mandate: building the tooling and standards that make self-service safe.

The trade-off is real and should not be minimised. Mesh requires significantly more organisational maturity than a centralised lake. Domain teams need to understand SLOs, schema versioning, and access control models. They will resist it initially. The platform team needs to provide tooling that makes compliance easier than non-compliance, not governance via process documents.

If your organisation has fewer than five data-producing domains or fewer than 15 people on data engineering, start with a centralised lake. Introduce mesh incrementally when the central team becomes the demonstrable bottleneck. Premature mesh is worse than no mesh.

Core architectural principles

These governed every design decision. They are opinionated by design.

1. P1
   Bronze is write-once, append-only.

   Raw data is never mutated. Every transformation creates a new artefact in a downstream layer. If something goes wrong in Silver or Gold, you can always reprocess from Bronze. The Bronze layer is also the audit log.

2. P2
   No schema contract, no ingestion.

   Every event source must register an Avro or Protobuf schema before it can publish to the message bus. The schema registry is a hard dependency, not an optional catalogue. The most common production failure is a producer silently changing a field type and crashing every downstream consumer. The schema registry blocks it.

3. P3
   Every pipeline must be re-runnable from Bronze.

   Stateless transforms, idempotent writes. If a pipeline cannot be safely replayed from raw data without producing duplicates or incorrect state, it is not production-ready. Non-negotiable for GDPR correction workflows, where you may need to reprocess data after a field is pseudonymised.

4. P4
   Observability is a deployment gate, not a follow-up task.

   A pipeline without structured telemetry (lag metrics, error rates, record counts at each layer boundary) does not get merged to main. OpenTelemetry instrumentation is part of the pipeline scaffold, not something added later. Silent failures are the most expensive failures.

5. P5
   GDPR classification happens at ingestion.

   Every field is classified as PII, sensitive, or unrestricted before it lands in the lake. The classification drives automatic access policy and determines whether tokenisation or pseudonymisation is required before Bronze storage. Retrofitting this is orders of magnitude more expensive than building it in.

6. P6
   Cost is a first-class architectural constraint.

   Streaming at scale is expensive. BigQuery on-demand pricing becomes untenable at 10 TB+ query volumes. Dataflow autoscaling without maximum worker caps generates surprise invoices. Storage lifecycle policies for Bronze/Silver/Gold must be defined in Terraform from day one, not added when the CFO asks questions.

Technology decisions

GCP is the reference implementation. Every decision has a documented alternative for teams not on GCP or with existing investments in other tooling.