Data Engineer
Current- Utilized Python, AWS S3, Lambda, Redshift, and SQL to integrate and manage both in-house and third-party data sources, ensuring seamless data flow from SFTP folders and APIs to a centralized data lake.
- Developed and deployed AWS Lambda functions to automate data movement and processing across S3 buckets, facilitating raw data storage, validation, transformation, and efficient loading into the Redshift data warehouse.
- Utilized Apache Airflow to orchestrate complex ETL workflows, leveraging various Airflow operators to ensure reliable, automated data pipelines, resulting in a 30% increase in data processing efficiency.
- Collaborated with data analysts to provide optimized Redshift endpoints for data analysis, enhancing the accuracy and timeliness of business insights derived from the KPI dashboard.