Orchestrated ETL workflows with AWS Glue, Lambda, and Step Functions for real-time processing.Utilized HiveQL to create Hive tables for structured data storage, enabling efficient data exploration.Implemented various layers of Data Lake architecture on AWS with efficient star schema design.Automated real-time data loading from S3 buckets into Amazon Redshift using AWS Lambda with Python.Led data processing pipelines with Apache Beam, Kubeflow, and AWS Step Functions for robust and scalable processing. DevelopedPython-based AWS Glue ETL jobs for ingesting and processing real-time data from Amazon Kinesis streams into Amazon Redshift.Successfully migrated on-premises workloads to AWS through Proof-of-Concept (POC) utilizing S3, Amazon Redshift, AWS Glue, andAmazon EMR.Developed star schema data models within Snowflake, optimizing data access and query performance for complex analyticsManaged AWS security groups for secure access control to EMR clusters and other resources. Implemented data quality checks andmonitoring procedures, reducing data errors and enhancing data reliability.

Developed and implemented CI/CD pipelines using AWS Code Pipeline, AWS Glue, and AWS Databricks for AWS Big Data solutions.Orchestrated data preparation and loading from Databricks to Amazon Redshift using AWS Glue.Engineered Spark applications for data validation, cleansing, transformation, and aggregation.Constructed and maintained PySpark pipelines for real-time data processing with high accuracy and performance.Optimized PySpark jobs for scalability and speed, delivering near real-time insights.Led migration of SQL databases to Amazon Redshift, Amazon Aurora, Amazon Redshift Spectrum, Amazon Athena, and Amazon RDS.Developed data pipelines using Spark, Hive, Pig, Python, Impala, and HBase for customer data ingestion and integration.Implemented advanced data security measures in compliance with AWS best practices.Conducted performance testing and optimization of data processing workflows, improvement in processing efficiency.Leveraged Snowflake connectors and ELT workflows to integrate data from various sources, ensuring data quality and consistency.Provided technical leadership and mentorship to junior team members, fostering a culture of innovation and excellence.Actively engaged in continuous learning and professional development, staying abreast of the latest trends and technologies in dataengineering and cloud computing.

Built and deployed RESTful APIs using Python frameworks (Flask, Django) for seamless data integration across diverse sources withinthe Azure ecosystem.Leveraged Apache Spark with Python for high-performance Big Data Analytics and Machine Learning applications.Configured and managed Azure Data Factory for scalable execution of data processing tasks.Utilized Apache Airflow on Azure for robust data workflow orchestration, automating data pipelines.Conducted performance tuning and optimization of Azure SQL Database stored procedures.Led data migration initiatives using SQL Server on Azure, Azure Data Factory, and Azure Data Lake, ensuring minimal disruption anddata integrity.Analyzed, designed, and implemented the ETL architecture using Informatica 9.1 on AWS infrastructureProficiently utilized Source Code management tools like GIT and Bit Bucket to establish effective version control mechanisms,promoting collaboration and code quality.