Data Engineer
Current● Developed and implemented ETL jobs and data transformations using Pentaho Data Integration, ensuring efficient data loading, cleansing, and transfer to target tables. ● Designed and executed an end-to-end data pipeline utilizing AWS server-less architecture, incorporating rigorous regression and smoke testing to ensure reliability. ● Conducted load and stress testing by simulating bulk data scenarios to evaluate system performance and stability under high demand. ● Streamlined existing process flows, achieving a 200% reduction in time complexity for improved process execution efficiency. ● Enhanced SQL query performance by employing best practices, including query optimization and indexing strategies, to accelerate data retrieval. ● Currently focused on migrating the existing pipeline architecture to AWS server-less solutions, aiming for improved refresh rates, seamless data flow, and enhanced maintainability and user experience. Leveraging AWS services such as Lambda, S3, Athena, Redshift, Cloud-watch, SQS, and AWS Glue for optimal data management.