Data Engineer
Current• Utilized Python, AWS S3, Lambda, Redshift, and SQL to integrate and manage both in-house and third-party data sources, ensuring seamless data flow from SFTP folders and APIs to a centralized data lake.• Developed and deployed AWS Lambda functions to automate data movement and processing across S3 buckets, facilitating raw data storage, validation, transformation, and efficient loading into the Redshift data warehouse.• Utilized Apache Airflow to orchestrate complex ETL workflows, leveraging various Airflow operators to ensure reliable, automated data pipelines, resulting in a 30% increase in data processing efficiency.• Collaborated with data analysts to provide optimized Redshift endpoints for data analysis, enhancing the accuracy and timeliness of business insights derived from the KPI dashboard.