Hoa Tran Email & Phone Number

Ho Chi Minh City, Vietnam

Vietnam

Design data architecture and build data platform including data lake, data warehouse and data pipelines in GCP for a financial business project:Build batch data pipelines with Apache Airflow and Python:- Define and schedule task dependencies and workflows with Apache Airflow- Ingest data from OLTP database (MySQL) and Firestore- Extract raw data to Google Cloud Storage with date-time prefixes- Override (WRITE_TRUNCATE) data from GCS to a landing zone in Google BigQuery- Merge source tables in the landing zone into target tables in a staging layer in BigQuery- Transform data within BigQuery using SQL and dbt to build a data mart layer containing tables, standard views and materialized views which are ready for data analysis and visualization.Build streaming data pipelines with CDC and Apache Beam:- Use Google Datastream as the CDC tool to read binary log files from MySQL and load to GCS- Create a trigger notification to produce messages to Cloud Pub/Sub whenever there are new objects in GCS.- Develop Dataflow pipeline jobs to consume messages from Pub/Sub and ingest data from GCS, then load and merge data to BigQuery tables

Design data architecture and build data lakehouse including data pipelines in AWS for an influencer marketing project:- Ingest data from OLTP systems such as Web applications, and a Crawler system.- Load data to S3 Raw bucket, where data is the same as sources, normally in CSV or JSON formats.- Clean, validate data for quality and referential integrity. Transform data with calculations.- Remove unnecessary columns, and form data to typed columns and tables structures.- Format data to Parquet for further aggregation queries before loading it to the Cleansed bucket.- Denormalize by combining data from the Cleansed bucket before loading it to the Curated bucket.- Use Redshift Copy command to load data from S3 Curated bucket, which is normally in a dimensional model to Data Warehouse tables in the Star schema.- Use Redshift SQL to transform, calculate and aggregate data in order to create Data Marts.- Utilize Apache Spark with PySpark DataFrame in AWS EMR for all data processing stages.- Create metadata, table structures in Glue Data Catalog for data in Cleansed and Curated buckets.- Use Athena for large-scale Ad-hoc queries to get data in S3 in order to support data science teams.- Build Airflow DAG and schedule the workflows. Use Airflow Operators to submit Spark jobs to EMR.