Sr. Data Engineer
Current Collaborated with cross-functional teams to gather data requirements and design data models for new projects. Implemented data quality and governance practices within data engineering projects. Managed ETL (Extract, Transform, Load) tasks ensuring data integrity and pipeline stability. Developed and maintained end-to-end operations of ETL data pipelines with Azure Data Factory (ADF). Designed advanced analytics models, including descriptive, predictive, and machine learning techniques. Converted Pig scripts/components of the ETL process to Spark API for improved performance. Utilized GCP Cloud-focused solutions to architect and implement data pipelines, leveraging services such as BigQuery for efficient data storage and analysis. Implemented optimized Spark jobs to process and analyze large volumes of data. Deployed data factory for creating data pipelines to orchestrate the data into SQL databases. Employed GCP Cloud-focused tools to create scalable and cost-effective data architectures, ensuring optimal performance and resource utilization. Attend Google Cloud Platform (GCP) webinars, conferences, and community events to understand the emerging best practices, updates, and use cases for GCP BigQuery and other associated GCP data services. Executed test plans for connectivity, SIT (System Integration Testing), and UAT (User Acceptance Testing), applying 8 years of data engineering experience to ensure high-quality data solutions. Implemented data security and compliance measures in GCP Cloud-focused environments, ensuring data protection and adherence to regulatory requirements. Developed automated CI/CD and test-driven development pipelines using Azure DevOps. Delivered insights using Azure Power BI, Azure SQL Database, and Data Warehouse. Developed and implemented data quality automation tests using Python and PySpark.