Throughout this role, I contributed to a wide range of data engineering and cloud computing initiatives. I developed and optimized Spark applications using Python and Scala on AWS EMR for both batch and streaming data processing, integrating multiple data sources including RDBMS and streaming platforms like Kafka. I also created ETL pipelines to migrate on-premise data from various sources (Flat Files, Mainframe, Databases) to AWS S3 using PySpark, while automating jobs using Informatica MDM and PowerCenter.Additionally, I worked with Hadoop and AWS ecosystems, writing MapReduce jobs and utilizing AWS services like Lambda, S3, Redshift, and EMR for data ingestion, processing, and analysis. I also used BI tools like Tableau and Power BI for reporting and analytics on AWS data stacks, created Python scripts for AWS automation, and processed real-time data with AWS Kinesis and Kafka.In collaboration with database administrators, I improved SQL performance across platforms like Oracle, MySQL, and MS SQL, and leveraged REST APIs for data ingestion into Google BigQuery. Moreover, I played a key role in building and migrating customer applications to the AWS Cloud, utilizing services like IAM, S3, Lambda, and VPC, while implementing CI/CD pipelines, UNIX shell scripting, and ensuring database performance through stress testing on DynamoDB.In this role, I consistently promoted customer success, collaborated in an agile environment, and continuously optimized algorithms and data pipelines to enhance system performance and scalability.

In this role, I was responsible for designing and implementing ETL processes to transform raw data into an analysis-ready format in HDFS. I ensured data accuracy and consistency by implementing data validation and quality checks. I also created views from HDFS and developed scripts using those views to streamline data processing workflows. I automated the ETL process using Apache Airflow and collaborated with data scientists and analysts to understand business requirements and deliver data assets.I worked with AWS Data Pipeline to configure data loads from S3 into Redshift and used Redshift for extracting, transforming, and loading data from various sources. In addition, I developed ETL solutions using Spark SQL in Azure Databricks to analyze and transform customer usage data from multiple file formats and data sources.I wrote Python scripts to manipulate and transform data as per business requirements and used scheduling tools to automate ETL jobs. I also worked with both SQL and NoSQL databases, such as MongoDB, HBase, Cassandra, SQL Server, and PostgreSQL, and resolved database connectivity and performance issues. Lastly, I optimized the data infrastructure to ensure fast query response times and efficient data processing.

In this role, I designed and deployed robust data pipelines using Azure services like HDInsight, DataLake, DataBricks, Blob Storage, Data Factory, Synapse, and SQL to support large-scale data processing and analytics. I was actively involved in data mapping and system testing, ensuring accurate data extraction, transformation, and transfer across platforms. I worked with diverse data formats, including flat files and relational databases, developing custom ETL solutions and real-time ingestion pipelines using PySpark and Shell scripting for Hadoop clusters. I integrated on-premises data (MySQL, HBase) with Azure cloud systems, applying transformations and loading data into Azure Synapse via Azure Data Factory. I deployed containerized applications using Docker, Azure Container Registry, and Azure Kubernetes Service (AKS), while migrating Hive tables to work efficiently with Azure. I optimized ETL workflows with Apache Airflow and built real-time pipelines using Spark Streaming and Scala. I designed cloud architectures on MS Azure for complex applications, leveraging Spark SQL and other frameworks to improve performance. I also developed custom adapters for ingesting data into HDFS, ensuring efficient data processing and enhanced security by utilizing Azure DevOps, Active Directory, and Apache Ranger. My work with Azure Synapse, Log Analytics, and Ambari Web UI improved both performance and security, and I managed cloud resources and data pipelines using advanced monitoring and CI/CD practices.

In this role I have implemented and optimized big data solutions, focusing on Spark, Scala, and Data Frames for efficient data processing and performance tuning. I ingested and transformed data from various sources like RDBMS and exported it to Cassandra, improving data models to meet business needs. Additionally, I worked on performance tuning of complex scripts and system redesigns to avoid bottlenecks.I led business requirement gathering, data migration, and cleansing strategies, while also developing source-to-target mappings and creating data validation and ETL jobs using Informatica. I utilized Hive for custom functions, partitioning, and bucketing, and managed deployments using Oozie and secure versioning software like Perforce. I also played a key role in migrating data from Oracle and Teradata to HDFS via Sqoop.In terms of data visualization and analysis, I created dashboards using Tableau to generate complex reports, enabling stakeholders to interpret trends and insights. My analytical work involved statistical evaluations with Python, SQL, R, and Excel, while leveraging Excel VBA Macros and Access Forms for automation. I also performed data ingestion and transformation using Azure Data Factory, Databricks, and T-SQL.I collaborated on building ETL processes with Spark for executing business analytical models and communicated project plans, risks, and metrics to stakeholders, ensuring smooth execution and reporting of project progress.