• As an experienced Data Engineer with over 10+ of experience, I bring a wealth of expertise in full lifecycle development, including requirement gathering, data pipeline design, ingestion frameworks, and building data marts and KPIs. I am certified in both GCP and AWS, and possess hands-on experience with a wide array of Azure components such as Azure Data Factory (ADF), Databricks, Synapse, Azure SQL, Azure Functions, Blob Storage, ADLS, Key Vault, Azure Monitor, Application Insights, Azure DevOps, and Power BI. My proficiency extends to AWS cloud components including S3, Lambda, Glue, Step Functions, Redshift, EMR, Athena, and implementing CI/CD with AWS DevOps.My extensive experience includes fine-tuning Spark applications to enhance performance and troubleshooting failures, leading a successful migration project from Teradata to Snowflake using Databricks, and managing end-to-end ETL processes with cloud-based ETL tools and DBT to create efficient and scalable data transformation pipelines. I excel in automating data pipelines with strong resilience and recoverability, and have the capability to work seamlessly across AWS, Azure, and GCP clouds.Additionally, I have a deep understanding of Hadoop, Spark, and their ecosystems for both batch and real-time data processing, along with comprehensive knowledge of HDFS, MapReduce, Hive, HBase, Sqoop, and Spark Streaming. I have developed Agile CI/CD pipelines using GitHub and Jenkins to optimize software development, integration, and deployment processes. My expertise also includes working with various file formats such as Parquet, ORC, Avro, JSON, and CSV, as well as utilizing compression techniques like Avro, Snappy, and LZO. With excellent communication and interpersonal skills, I am adept at quickly learning new technologies and contributing effectively to team efforts.