Data Pipeline Development:-Created scalable data pipelines using Cloud Composer (Airflow) for orchestration-Implemented hybrid data solutions connecting on-premise Hadoop with GCP services-Optimized BigQuery performance for data analytics-Developed shell scripts for historical data loads and designed Composer Directed Acyclic Graphs (DAGs) for managing incremental data.-Integrating Streaming and Batch processing using Dataflow-Optimized Spark SQL queries for efficient data transformation and analysisCloud Architecture: -Utilized GCP services including Dataproc, Cloud Storage, BigQuery and Cloud SQL.-Designed and maintained CI/CD pipelines for automated deployment using Jenkins.-Implemented cost-effective solutions for data storage and processingTesting & Quality Assurance: -Developed comprehensive unit test cases for data validation-Led UAT support and provided post-deployment warranty maintenance-Ensured data quality and consistency throughout migration process-Conducted ETL testing using Apache Nifi.-Managed daily status updates with onshore teams and stakeholders-Worked on Jira and ServiceNow in handling client incidents and problems-Conducted thorough testing of ETL processes across multiple data sources -Wrote complex SQL queries for data validation between source and target systems

Data Processing & Analytics: -Led comprehensive big data operations, leveraging Hadoop ecosystem for storage, analysis, and insights extraction -Engineered robust ETL workflows using Apache NiFi and Kafka-Optimized data warehouse solutions using GCP BigQuery and Hive.-Implemented real-time data processing pipelines for streaming analyticsInfrastructure Management: -Administered HBase infrastructure via Cloudera Manager-Implemented monitoring solutions using Grafana for system health tracking-Designed and executed backup strategies using Hadoop replication policiesAutomation & Optimization: -Automated routine tasks through Python scripting and CRON jobs-Optimized query performance in both Hive and BigQuery environmentsTesting & Quality Assurance: -Wrote and conducted automated tests using Robot Framework and Selenium-Contributed to AI data model testing, fine-tuning, and validation, enhancing model accuracy -Reduced regression testing time by 70% through automation