GCP Data Engineering Project

The aim of this project is to execute a modern data engineering workflow using various technologies, performed on a sample grocery store dataset.

Data Source

• This dataset is a sample grocery store dataset that contains information on Order IDs, Categories and Subcategories, Profit and Sales, Region and City, etc.
• More information on the dataset can be found here.
• Provided by Mohamed Harris.

Data Architecture

• GCP Data Architecture

Data Model

• Data Model takes original data schema and follows fact and dimension table formatting.
• Data Model.

Technologies Utilized

• Google Cloud Platform

  • Google Storage
  • Compute Instance
  • BigQuery
  • Looker Studio

• Mage Modern Data Engineering Pipeline Replacement for Airflow.

• Python for ETL

  • ETL Scripts
  • Pipeline

• SQL on BigQuery

  • Queries

Dashboard

• Final Dashboard
• I have since shut down the VM, Storage, and BigQuery, so the interactive portion of the dashboard is unavailable.