“Create a daily data pipeline flowchart that extracts changed PostgreSQL orders to Cloud Storage, checks the schema, quarantines failures, deduplicates records, validates revenue and row counts, loads BigQuery, runs a MERGE and dbt marts, then publishes sales dashboard data.”
Leggi le decisioni che ne stanno alla base.
Only orders changed since the previous run are exported.
Unexpected columns or types are isolated before they contaminate analytics.
Row counts and revenue totals must agree before publishing.
MERGE updates the order fact table without creating duplicates.
Use this for a daily transactional-to-warehouse ETL job with a cloud object-store landing zone. It suits reporting pipelines where a failed batch is safer than partial dashboard data.
Domande frequenti
Why land files before loading BigQuery?
A raw landing zone preserves the extract for replay, inspection, and controlled backfills.
Why use MERGE for orders?
Orders can change after creation, so MERGE applies inserts and updates without duplicate order rows.
What should the revenue check compare?
Compare a source aggregate for the extract window with the staged aggregate before publication.