•Extracted data using REST APIs of different media and ads platforms like FB, Google, YouTube by developing application using Scala with cats and functional programming to write it to AWS S3 in compressed (gzip) format.•Successfully migrated legacy data warehouses to Amazon Redshift, utilizing AWS Data Migration Service and Redshift Spectrum to enable efficient data querying across both historical and current datasets, enhancing data analytics capabilities.•Hands on experience on Spark Tech, SQL, and Hive to build data ETL pipelines which loads the raw data into Delta Lake•Worked on Jenkins to automate building, testing, and deploying Docker applications and facilitating CI/CD process.•Deployed and managed Kafka clusters on Kubernetes using Helm charts, ensuring high availability and scalability. •Developed scalable data processing pipelines using Node.js, enhancing ETL workflows and reducing data processing time by 30%.•Developed Airflow DAGS using python to trigger the pods in Kubernetes cluster and run spark jobs on Databricks cluster.•Leveraged JSON for efficient data serialization and deserialization in RESTful APIs, improving data retrieval speeds and API response times.•Utilized jQuery for DOM manipulation and event handling to improve the interactivity and user experience of data-heavy web applications.•Implemented and optimized ML algorithms for low latency environments using tools like Apache Kafka for real-time data streaming.Apache Flink for stream processing, and TensorRT for high-performance inference.•Designed and maintained distributed systems leveraging Apache Spark for large-scale data processing, Kubernetes for container orchestration, and Consul for service discovery and configuration management.•Ported legacy ETL from Hive to Databricks Spark, boosting performance by up to 1000% and cutting AWS EC2 costs by 50%.•Utilized Google BigQuery and Dataflow/Data Fusion for seamless data processing and analytics.

•Led the design and implementation of scalable ETL pipelines using SQL Server Integration Services (SSIS), enhancing data integration from multiple source systems into a centralized data warehouse, resulting in a 40% improvement in data processing time. •Established a centralized data warehouse for healthcare data, utilizing Hive for data organization and analysis. •Leveraged Snowflake to design and implement a scalable data warehousing solution that reduced query times by 40% and enhanced data accessibility for business intelligence and analytics teams.•Migrated legacy ETL processes to modern frameworks, including Informatica PowerCenter, improving data consistency and processing speed. •Implemented complex data cleansing and validation processes using Python and Shell scripting, reducing data inconsistencies by 30%. •Utilized Hive and Impala for data querying and analysis, supporting comprehensive data reporting and insights. •Documented ETL processes, including data lineage and transformation logic, ensuring maintainability and compliance. •Conducted thorough data quality assessments and implemented data quality rules, enhancing overall data accuracy and reliability. •Developed and deployed various machine learning algorithms using TensorFlow for deep learning, Scikit-learn for traditional ML models, and PyTorch for advanced research and experimentation.•Collaborated closely with data analysts and business stakeholders to translate business requirements into robust ETL solutions, directly contributing to a 20% increase in data-driven decision-making efficiency. •Played a pivotal role in the development and enhancement of data-driven strategies to improve blood donor identification and engagement for the Find Blood project. •Utilized Agile methodology and participated in regular Scrum calls to manage workflow, ensuring timely completion of data analysis tasks and effective collaboration with team members.

and extracting insights from large datasets.• Led the creation of data pipelines for the DA team's reporting needs and DS team's ML model development. These pipelines ensured smooth data flow, enabling timely insights and efficient model training.• Scripted automated processes for cleansing staging tables in Hive, streamlining data preparation, enhancing quality, and reducing manual effort and errors.• Designed and implemented XML data models for seamless data interchange between diverse enterprise systems, ensuring data integrity and consistency.• Developed and optimized SQL queries for data retrieval and reporting in MySQL databases, improving query performance and data accuracy for business analytics.• Utilized HTML to create dynamic data visualization dashboards, providing real-time insights and interactive reports for stakeholders.• Monitored Production Environment jobs (batch and Spark), addressing failures proactively to minimize disruptions and maintain high system availability.• Applied deep expertise in Hadoop ecosystem best practices (YARN, HDFS, MapReduce) to optimize job performance, resource utilization, and system efficiency, ensuring enhanced reliability and scalability.