“Create a streaming data pipeline flowchart for web and mobile clickstream events sent to Kafka, checked with a schema registry, routed to a dead-letter topic when fields are missing, enriched with campaign and geography data, privacy-filtered, windowed in Flink, stored in Iceberg, and used for product analytics metrics.”
פענחו את ההחלטות שמאחוריו.
Required event fields are checked before stream processing.
Bad events remain available for producer diagnosis.
Direct identifiers are removed before analytical storage.
Flink turns individual events into session and conversion metrics.
Use this for a governed clickstream pipeline that serves product analytics while preserving bad-event evidence. It is appropriate when a Kafka topic has many independent producers.
שאלות נפוצות
Why keep a dead-letter topic?
It preserves invalid messages for debugging while keeping the main stream usable.
Why remove identifiers before Iceberg?
Analytical use usually needs behavior and context, not direct personal identifiers.
What does Flink aggregate here?
It can calculate event counts, sessions, and conversions over defined time windows.