Data Engineer
Responsibilities:• Collaborated with cross-functional teams to understand and gather business requirements or financial goals.• Implemented ETL pipeline activities using ADF (Azure Data Factory) to copy data and integrate multiple sources, such as REST API, Database, and Excel to process large data sets.• Completed common-use data format conversion and flattening work, such as CSV, JSON, and Parquet on ADF.• Deployed automated ETL pipelines for batch-processing on Databricks… Show more Responsibilities:• Collaborated with cross-functional teams to understand and gather business requirements or financial goals.• Implemented ETL pipeline activities using ADF (Azure Data Factory) to copy data and integrate multiple sources, such as REST API, Database, and Excel to process large data sets.• Completed common-use data format conversion and flattening work, such as CSV, JSON, and Parquet on ADF.• Deployed automated ETL pipelines for batch-processing on Databricks, using Medallion Architecture, Change Data Feed, and Incremental Processing to optimize the CDC (Change Data Capture) process and version control.• Implemented data management tasks including data entry, cleansing, and quality control.• Conducted database performance tuning and ETL process improvement with skilled SQL (CRUD, Constraints, Indexes).• Applied expertise in Powe Pivot and DAX (Data Analysis Expressions) to enhance data modelling and calculation logic within Power BI.• Utilized Power Pivot, Slicers, SUMIFS, INDEX/MATCH, PivotTable formulas, etc. in Excel to execute analysis and generate insights.• Assisted in designing a data warehouse based on the star schema to store business datasets imported from ETL pipelines for further analysis.• Automated data engineering workflows using Python scripts for increased efficiency.• Ensured compliance with data governance policies and implemented robust security measures. Show less
