•Proficient in crafting and designing multiple data pipelines, overseeing the complete ETL and ELT process for data ingestion and transformation within the Google Cloud Platform (GCP).•Successfully set up a Continuous Delivery pipeline using Docker and GitHub, streamlining the deployment process.•Re-platformed AWS EMR-based Spark and Hadoop jobs to GCP Dataproc, leveraging GCP's autoscaling and managed cluster features for optimized resource management and cost control.•Transitioned logging and monitoring solutions from AWS CloudWatch to Google Cloud Monitoring and Logging to maintain visibility into the migrated data pipelines and ensure performance metrics are equal.•Developed, deployed, and managed results utilizing Spark and Scala code within a Hadoop cluster hosted on GCP.•Skilled in leveraging Google Cloud components, Google Container Builders, GCP client libraries, and Cloud SDKs to architect and execute data solutions.•Hands-on experience with Google Cloud Function, using Python to load data into BigQuery from incoming CSV files in GCS buckets. Also, proficient in processing and loading both bounded and unbounded data from Google Pub/Sub topics to BigQuery via Cloud Dataflow.•Utilized Spark and Scala APIs to assess the performance of Spark in comparison to Hive and SQL. A•Proficiently stored data in the GCP BigQuery Target Data Warehouse, making it available for various business teams according to their specific use cases.•Successfully deployed applications to GCP using Spinnaker, leveraging rpm-based packages.•Architected several Directed Acyclic Graphs (DAGs) to automate ETL pipelines for seamless data processing.• Leveraged Amazon EMR clusters for processing large-scale datasets, optimizing for performance and cost-efficiency before transitioning to GCP.• Integrated AWS services with GCP components, such as Cloud Dataflow and BigQuery, for a hybrid cloud solution during migration.

• Migration of an entire Oracle database to BigQuery and employed Power BI for robust reporting. Crafted data pipelines within Google Cloud's Airflow to streamline ETL tasks using a variety of airflow operators.• Proficient in a range of Google Cloud Platform (GCP) services including Dataproc, Google Cloud Storage (GCS), Cloud Functions, and BigQuery.• Demonstrated expertise in Google Cloud components, Google Container Builders, GCP client libraries, and Cloud SDKs.• Actively participated in migrating on-premises Hadoop systems to GCP (Google Cloud Platform).• Conducted in-depth analysis of data from diverse domains to enable seamless integration into a Data Marketplace.• Developed Pyspark programs, established data frames, and executed data transformations.• Proficiently employed a variety of GCP services, including GCP Cloud Storage, Dataproc, Data Flow, and BigQuery.• Implemented a Continuous Delivery pipeline using tools such as Maven, Ant, Jenkins, and GCP. Exp• Developed multi-cloud strategies, leveraging the strengths of GCP, especially its Platform as a Service (PaaS) offerings.• Proficiently migrated legacy systems into GCP's cutting-edge technologies.• Implemented daily data file storage in Google Cloud buckets, effectively harnessing DataProc and BigQuery for maintaining cloud-based solutions.• Developed Pyspark scripts to merge static and dynamic files while cleansing the data.

Leveraged Spark RDD, Data Frame API, Data Set API, Data Source API, Spark SQL, and Spark Streaming alongside SQL and DynamoDB for comprehensive data processing• Developed Spark applications using both Python and R, including implementing Apache Spark data processing projects to handle data from various RDBMS and streaming sources•Pioneered a cross-functional team to successfully migrate ETL processes from SSIS to Snowflake, ensuring data integrity and minimal downtime.• Redesigned data workflows to leverage Snowflake’s powerful SQL capabilities, optimizing performance and reducing processing time.• Efficiently extracted data from SQL server, Amazon S3 buckets, and internal SFTP, loading them into AWS S3 buckets in a data warehouse context• Developed Spark jobs for data processing and orchestrated instances and clusters to load data into AWS S3 buckets, thereby creating a DataMart• Leveraged AWS EMR for processing and transforming data to assist the Data Science team based on business requirements• Designed and developed ETL processes in AWS Glue to migrate campaign data from external sources, such as S3, ORC/Parquet/Text files, into AWS Redshift.•Pioneered the migration of an on-premises data warehouse to AWS, using AWS DMS to efficiently transfer terabytes of data from Teradata to Redshift. • Engaged in both batch processing and real-time data processing using Spark Streaming with a Lambda architecture• Developed Python code for various tasks, dependencies, and time sensors in the context of workflow management and automation using the Airflow tool• Collaborated with the DevOps team to implement Nifi Pipelines on EC2 nodes, integrated with Spark, Kafka, and Postgres running on other instances, using SSL handshakes in QA and Production Environments.

Engineered RESTful APIs with Java Spring Boot to orchestrate seamless data extraction via Sqoop, optimizing data retrieval processes for enhanced efficiency and integration within the Hadoop ecosystem.•Leveraged Talend for data integration, cleansing, and transformation, while using dbt to refine raw data into structured datasets, leading to faster processing times and higher data quality•Crafted, developed, and maintained Tableau functional reports according to user specifications, ensuring meaningful data visualization•Deployed Hadoop and Cloudera Distribution for Hadoop (CDH) to optimize the data processing pipeline, including setup, real-time data ingestion Flume, and Spark analytics•Proficiency in Python and Scala, with a knack for creating user-defined functions (UDF) for Hive and Pig using Python•Integrated MongoDB with big data processing frameworks like Hadoop and Spark to build end-to-end data pipelines for batch and stream processing•Configured HBase tables to accommodate various data formats, specifically PII data from diverse portfolios•Developed complex Hive SQL queries to extract, transform, and load data from HDFS into Hive tables•Employed Oracle Data Integrator to load data into Hadoop directly from files or SQL databases, validate and transform data within Hadoop, and then load the processed data from Hadoop into Oracle Database or other targets•Leveraged Oracle PL/SQL to develop robust and efficient data processing routines, ensuring data integrity and reliability