•Designed and implemented advanced data processing algorithms using Apache Spark to handle real-time data streams, leading to a 40% improvement in latency for analytics tasks. Conducted performance tuning to optimize Spark jobs for efficiency and reduced operational costs.•Leveraged Java to create modular and reusable data processing components, enhancing the maintainability of ETL workflows. Contributed to the development of microservices architecture for data ingestion and processing.•Developed robust data transformation scripts using Scala to facilitate seamless integration of disparate data sources into centralized data warehouses. Enhanced data quality and consistency through rigorous testing and validation procedures.•Created automation scripts in Python for routine data quality checks and monitoring of ETL jobs, leading to a significant reduction in manual oversight and increased reliability of data pipelines.•Managed GCP resources effectively, utilizing tools like Cloud Functions and Pub/Sub for event-driven architecture, ensuring scalable and reliable data processing. Implemented cost management strategies to optimize cloud expenditure.•Worked closely with data scientists, analysts, and stakeholders to gather requirements and translate business needs into technical specifications, ensuring alignment of data solutions with business objectives.•Implemented data security protocols and best practices to safeguard sensitive data in compliance with regulatory standards, ensuring robust data governance across all ETL processes.•Engineered interactive Tableau dashboards that provided insights into key performance indicators (KPIs) and trends, empowering business units with actionable insights. Conducted user training sessions to enhance the adoption of self-service analytics among stakeholders.

• Configured Google Cloud Platform (GCP) instance templates based on custom machine images and utilized these templates to configure managed instance groups for auto-scaling. Deployed GCP solutions using Compute Engine, Cloud Storage, Cloud SQL, Cloud DNS, Persistent Disks, Load Balancers, and Managed Instance Groups.• Processed and transformed large datasets consisting of structured, semi-structured, and unstructured data using GCP Big Data technologies and Apache Hadoop ecosystem.• Expert in implementing advanced analytics and data processing procedures using in-memory computing capabilities of Apache Spark, written in Scala. Specialized in leveraging Spark with Scala and Spark SQL for efficient data testing and processing, and responsible for data integration from various sources.• Managed data ingestion and integration from multiple data sources into a GCP Data Lake using ETL pipelines developed with Dataflow, Dataproc, and other GCP-native tools, as well as Hadoop-based frameworks like Spark, MapReduce, Pig, and Hive.• Collaborated with Architects to design a GCP Dataproc Spark model as a replacement for the existing MapReduce workflows and migrated these models to Spark using Scala. Developed Scala programs and utilized Spark SQL to perform complex data aggregations and transformations.

•Contributed to the development of key data integration and advanced analytics solutions leveraging Apache Hadoop and other big data technologies for leading organizations using major Hadoop Distributions like Hortonworks.•Responsible for data extraction and data integration from different data sources into Hadoop Data Lake by creating ETL pipelines Using Spark, MapReduce, Pig, and Hive.•Used IMPALA to read, write, and query the Hadoop data in HDFS and configured KAFKA to read and write messages from external programs.•Developed Spark programs with Scala and applied principles of functional programming to process complex unstructured and structured data sets. Processed the spark data with Spark from the Hadoop Distributed File System.•Developed ETL pipelines using notebooks, Spark Data frames, SPARK SQL, and Python scripting.

•Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Bigdata technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive.•Involved in creating Hive Tables, loading with data, and writing Hive queries which will invoke and run Map Reduce jobs in the backend.•Used IMPALA to read, write, and query the Hadoop data in HDFS and configured KAFKA to read and write messages from external programs.•Expertise in implementing Spark Scala applications using higher-order functions for both batch and interactive analysis requirements.