* Engineered end-to-end data solutions by leveraging Big Data technologies, particularly Hadoop ecosystem, to process, store, and analyze massive datasets, ensuring optimal performance and scalability.* Designed and implemented robust data pipelines using tools such as Apache Spark and Apache Flink, streamlining data processing workflows and enhancing real-time analytics capabilities.* Collaborated with cross-functional teams to gather requirements, architect data solutions, and provide technical expertise throughout the project lifecycle.* Optimized Hadoop clusters for efficiency and reliability, employing techniques like performance tuning, resource management, and troubleshooting to meet and exceed performance expectations.* Implemented data security measures and ensured compliance with data governance policies, maintaining the integrity and confidentiality of sensitive information.* Conducted thorough data profiling and analysis to identify patterns, trends, and anomalies, facilitating informed business decision-making.* Provided hands-on training and documentation to empower teams with the knowledge to leverage Big Data tools effectively, fostering a culture of data-driven decision-making within the organization.* Stayed abreast of emerging technologies in the Big Data landscape, continuously enhancing skills to drive innovation and maintain a cutting-edge approach to data engineering solutions.

• Architected and implemented scalable distributed data solutions using AWS services, ETL processes, and Data Lake applications while ensuring data quality, efficient storage, and analytics based on business user requirements.• Collaborated with architects to translate functional and technical requirements into detailed architecture and design, building scalable distributed data solutions using AWS services.• Validated transactional and profile data from RDBMS, transforming and loading it into Data Lake using AWS Cloud Services and automating S3 file system processes.• Developed and tested ETL processes in AWS Glue, migrating campaign data from external sources like S3 and various file formats into AWS Redshift.• Utilized Python and SQL scripts to import and export structured data between relational databases, S3, and AWS RDS using Spark, EC2, and EMR clusters.• Implemented end-to-end Apache Airflow design and development, facilitating communication between middleware and EBI teams and executing critical actions.• Developed monitoring reports and dashboards for Spark jobs, leveraging text analytics and in-memory computing capabilities like Apache Spark with Python, and troubleshooting production-level issues.• Utilized Parquet file format and HBase tables for efficient storage and performance, working with NoSQL databases like HBase.• Managed data from multiple sources, maintaining HDFS and loading structured/unstructured data for diverse processing needs.• Developed RESTful API using Python to track open-source GitHub projects and implemented machine learning methods using Spark, Python, Hadoop, and HBase.• Designed data analysis pipelines with Python, leveraging AWS services like S3, EC2, and Elastic MapReduce for efficient processing and storage.• Applied advanced text analytics using Apache Spark and developed analytic systems with Python and Scala-based ML Libraries.

• Developed and deployed Spark applications using Scala for Hadoop transitions, implemented Microservices architecture with Spring Boot, managed cloud-based storage, and processing in AWS HDFS, and collaborated on Hadoop-based Data Lake initiatives. Also, streamlined data ingestion pipelines monitored Hadoop cluster operations, and migrated MapReduce programs to Spark transformations, utilizing Scala and Python for analysis and optimization, ultimately enhancing data processing efficiency across organization.• Developed Spark applications using Scala to facilitate Hadoop transitions, enhancing performance on the Hortonworks Data Platform.• Utilized Microservices architecture with Spring Boot-based services, building and deploying enterprise-level software products.• Managed cloud-based storage and processing in AWS HDFS, deploying applications using ELBs and EC2 instances.• Created Hive tables and read parquet data using Scala API, Spark, and Spark SQL for faster processing and testing.• Analyzed SQL scripts and designed solutions by implementing Spark programs using Spark optimizing data processing tasks.• Collaborated with the Big Data Architecture team to establish a Hadoop-based Data Lake for organization-wide analytics initiatives.• Extracted real-time data feeds with Spark Streaming, converting them to RDDs and processing data as Data Frames in HDFS.• Developed data pipelines for ingesting customer behavioral data into HDFS using Sqoop, Pig, and Java MapReduce.• Aggregated log data with Apache Flume, staging data in HDFS for further analysis and processing.• Managed Hadoop cluster operations, including installation, upgrades, capacity planning, and troubleshooting MapReduce job execution issues.• Utilized Scala and Python for interactive and batch analysis, developing Spark jobs for efficient data processing tasks.• Migrated MapReduce programs to Spark transformations using Spark, Scala, and Python, enhancing overall data processing efficiency.