Proficient in Python, PySpark, Hadoop, and Airflow, I manage massive datasets across various formats and conduct thorough data analysis to ensure quality. I specialize in implementing and maintaining efficient Hadoop clusters, leveraging Spark for large-scale processing and analysis tasks. Implemented real-time data replication using IBM InfoSphere Data Replication, ensuring data consistency across multiple environments with minimal latency.Ensured data replication processes adhered to regulatory compliance requirements, including data auditing and tracking changes across replicated systems.Expertise in managing and optimizing IBM DB2 databases, including performance tuning, backup and recovery, and implementing advanced security features

Managed Airflow clusters daily batch runs for data migration between ingestion layers for inventory forecastingOptimized replication processes by fine-tuning parameters and resource allocation, achieving significant improvements in replication speed and efficiency.Created observability metrics with Python scripts for inventory forecasting team to report the Airflow DAGs performanceWrote Airflow DAGs and Spark SQL queries and deployed them to the clusterTimed and tested DAG runs to ensure no conflicts in data movementDeveloped and managed ETL processes using IBM InfoSphere DataStage, enabling efficient data integration and transformation across various data sourcesEnsured data integrity across S3 buckets and EMR clusters with Snowflake, Hue, and ZeppelinMigrated data across EMR clusters using Spark commandsVerified and tested database schemas