• Architected end-to-end data pipelines for seamless migration of data from PostgreSQL to AWS S3 and Redshift, leveraging Talend to improve data processing efficiency by 30%.• Implemented serverless architecture using AWS Lambda and Kinesis, significantly reducing infrastructure costs and enhancing system scalability.• Developed real-time data processing applications using Apache Spark and PySpark, reducing data latency and enabling quicker, data-driven decision-making.• Optimized… Show more • Architected end-to-end data pipelines for seamless migration of data from PostgreSQL to AWS S3 and Redshift, leveraging Talend to improve data processing efficiency by 30%.• Implemented serverless architecture using AWS Lambda and Kinesis, significantly reducing infrastructure costs and enhancing system scalability.• Developed real-time data processing applications using Apache Spark and PySpark, reducing data latency and enabling quicker, data-driven decision-making.• Optimized ETL processes in collaboration with cross-functional teams, achieving a 40% reduction in data integration time.• Enhanced data security and governance by designing and managing AWS S3 buckets and implementing strict access control policies.• Collaborated with data science teams to integrate machine learning models into data pipelines, using Python and custom scripts for predictive analytics. Show less

• Led the migration of data pipelines to the Azure cloud platform, utilizing Azure Data Factory, Azure Data Lake, and Databricks, resulting in a 50% reduction in data processing time.• Developed and optimized Spark SQL scripts for large-scale data processing, enhancing query performance by 20% and accelerating data analysis.• Automated job scheduling and batch processing using Oozie on Azure HDInsight, reducing operational overhead and ensuring timely data processing.• Integrated… Show more • Led the migration of data pipelines to the Azure cloud platform, utilizing Azure Data Factory, Azure Data Lake, and Databricks, resulting in a 50% reduction in data processing time.• Developed and optimized Spark SQL scripts for large-scale data processing, enhancing query performance by 20% and accelerating data analysis.• Automated job scheduling and batch processing using Oozie on Azure HDInsight, reducing operational overhead and ensuring timely data processing.• Integrated multiple data sources into a centralized data lake using Hadoop and Hive, improving data accessibility for analytics teams.• Implemented Azure Key Vault and secure authentication protocols to protect sensitive data and ensure compliance with industry standards.• Provided technical leadership and mentorship to junior engineers, fostering a culture of continuous learning and improvement. Show less

• Developed scalable data ingestion pipelines using Talend, Apache Spark, and Hive, reducing data latency and improving processing capabilities.• Built machine learning models using PySpark and MLlib, providing predictive insights that supported critical business decisions.• Managed AWS S3 data lakes, ensuring reliable and secure storage of large datasets with best practices for data management.• Conducted data transformations using Scala and Spark DataFrames, streamlining data… Show more • Developed scalable data ingestion pipelines using Talend, Apache Spark, and Hive, reducing data latency and improving processing capabilities.• Built machine learning models using PySpark and MLlib, providing predictive insights that supported critical business decisions.• Managed AWS S3 data lakes, ensuring reliable and secure storage of large datasets with best practices for data management.• Conducted data transformations using Scala and Spark DataFrames, streamlining data aggregation and reporting processes.• Developed custom UDFs in Hive to handle complex data transformations, enabling the organization to meet unique business requirements.• Performed performance tuning and troubleshooting of data pipelines, ensuring smooth and efficient data processing systems. Show less

• Implemented and managed Hadoop clusters, including the setup, configuration, and optimization of HDFS, MapReduce, and other Hadoop components, ensuring high availability and performance.• Developed data pipelines using Pig, Hive, and Sqoop for large-scale ETL tasks, enabling efficient data processing and integration with downstream systems.• Configured and managed Hadoop security protocols using Kerberos and Sentry, ensuring secure access to data and compliance with AT&T’s security… Show more • Implemented and managed Hadoop clusters, including the setup, configuration, and optimization of HDFS, MapReduce, and other Hadoop components, ensuring high availability and performance.• Developed data pipelines using Pig, Hive, and Sqoop for large-scale ETL tasks, enabling efficient data processing and integration with downstream systems.• Configured and managed Hadoop security protocols using Kerberos and Sentry, ensuring secure access to data and compliance with AT&T’s security policies.• Led the migration of Hadoop jobs to a Cloudera-based environment, improving job execution times by 30% and enhancing system stability.• Conducted performance tuning and troubleshooting for Hadoop jobs, reducing errors and improving system efficiency.• Developed MapReduce jobs and Pig Latin scripts to transform and aggregate large datasets, supporting data analysis and reporting. Show less

• Worked on Informatica Power enter Designer - Source analyser, Warehouse Designer, Mapping Designer and Transformation developer.• Used various Informatica transformations to recreate data in the data warehouse• Responsible for resolving emergency production issue for the module during the Post Implementation phase. • Had the responsibility for creating the design and implementation documents, effort estimation, planning for coding & implementation, writing and performance tuning… Show more • Worked on Informatica Power enter Designer - Source analyser, Warehouse Designer, Mapping Designer and Transformation developer.• Used various Informatica transformations to recreate data in the data warehouse• Responsible for resolving emergency production issue for the module during the Post Implementation phase. • Had the responsibility for creating the design and implementation documents, effort estimation, planning for coding & implementation, writing and performance tuning the mappings to improve the performance in production environment. • Designed and developed Aggregate, Join, Lookup transformation rules (business rules) to generate consolidated (fact/summary) data identified by dimensions• Used Lookup, Sequence generator, Router and Update Strategy transformations to insert, delete, and update the records for Slowly Changing Dimension tables.• Had the responsibility to lead a 3-member team working in technologies namely Informatica, Unix and Oracle back end Epiphany frontend. Show less