Data Engineer
CurrentWorked as a data engineer for Combe Incorporated(Aug'23 - Present), involved in :* Ingesting raw data from source into ADLS Gen2* Transforming, Partitioning & Writing data to Delta tables using PySpark following Medallion architecture* Developing an end-to-end ETL pipeline using Azure Data Factory for orchestration* Data validation, monitoring & performance tuningTech Stack - PySpark, Azure Data Lake Storage Gen2, Delta Lake, Azure Data Factory, RESTWorked as a data engineer for Gojek(Oct'21- May'23), involved in :* Data migration from one project id to another project id in BigQuery * Writing clean & optimised SQL queries to meet business propositions* Partitioning, Clustering and implementing Job schedules for BigQuery tables* Loading federated sheets, creating views in BigQuery* Monitoring & fixing Airflow DAGs* Data validation & performance tuning* Improving data pipeline efficiency by cutting down jobs completion time by 90 minutesTech Stack - Google BigQuery, Apache Airflow